THE ORGANISATIONAL PERFORMANCE IMPACT OF ERP SYSTEMS ON SELECTED COMPANIES

There is a need to determine the performance benefits that firms gain from enterprise resource planning (ERP) systems, and the factors that contribute to success. Consequently, the central research problem is: To determine the impact of ERP systems on organisational performance by analysing achieved ERP benefits within the framework of a suitable performance measurement system (PMS), and to investigate the association between critical success factors (CSFs) and ERP benefits. A list of expected ERP benefits and CSFs is compiled. Performance measurement models are evaluated, and one model is designed to evaluate ERP impact. ERP experts refine identified ERP benefits and CSFs using structured interviews. These are built into a questionnaire, and used to survey South African (SA) companies in manufacturing, mining, and power generation. Results indicate that business benefits are realised by companies that implement ERP systems. By building benefits into the performance measurement model, a positive impact on organisational performance is observed. Although a core list of CSFs is identified, and three associations are found between CSFs and ERP benefits, further research is needed.


INTRODUCTION
There is a lack of available information about the impact of ERP systems, both internationally and in South Africa.This gap in the knowledge base extends from the benefits available to the factors responsible for these benefits being achieved.

PURPOSE OF THE RESEARCH
This study investigates how ERP systems affect organisational performance over time, within selected South African firms.The primary question is:

Do ERP systems have a positive impact on organisational performance in the time periods following implementation?
To investigate this, a link needs to be established between ERP systems and a firm's performance.This leads to the question: Is there a valid and reliable PMS that links ERP performance to organisational performance?
As the most likely influence on a firm's performance may be through the expected ERP benefits, the identified PMS needs to contain these benefits.Therefore, the third question to be investigated is: What benefits are gained from ERP systems?
By investigating these questions, the research establishes the impact of ERP systems on organisational performance.However, without investigating the factors responsible for this impact, the research would be incomplete.Therefore, the fourth research question is: What CSFs are responsible for ERP benefits being achieved?

LITERATURE REVIEW
To explore these questions, the literature review defines an ERP system before investigating the area expected to have the greatest impact on organisational performance: ERP benefits.It then reviews organisational performance measurement systems and the methods used to evaluate ERP performance.The literature review concludes with an investigation into the CSFs responsible for ERP benefits being achieved.

Enterprise resource planning (ERP) systems
ERP systems began to evolve in the 1960s when customised software packages that focused on inventory control were introduced to manufacturing organisations.During the 1970s, the focus shifted to developing information systems to plan and control manufacturing.These systems, known as material requirements planning (MRP) systems, played a role in translating the master production schedule, built for end items, into time-phased net requirements for the sub-assemblies, components, raw materials, and procurement [14].The 1980s saw the evolution of MRP systems into manufacturing resource planning (MRPII) systems.MRPII systems combined MRP outputs with routing information to determine capacity requirements, and served as a control to ensure that MRP plans were feasible.As information systems developed across other functional areas, the need to integrate these systems with MRPII became apparent.In the 1990s, the Gartner Group coined the term 'enterprise resource planning' to describe the next generation of MRPII systems that integrated manufacturing software applications with other functional areas, such as finance and human resources, resulting in a company-wide information system [19].An ERP system can be defined as "a packaged business software system that enables a company to manage the efficient and effective use of resources (materials, human resources, finance, etc) by providing a total, integrated solution for the organisation's information-processing needs" (adapted from [23]).
In the 2000s, ERP systems continue to develop, and -by incorporating technology such as the internet -they are now being designed to aid organisations to integrate functions across their supply chains.

Benefits of ERP systems
The need for real-time and accurate information, the standardisation of business processes, and the integration of applications are the main drivers for ERP adoption [24,29].Markus and Tanus [21] suggest that there should be a connection between reasons for ERP adoption and expected ERP benefits.Research into these expected benefits has produced a number of favourable findings: Chand et al. [4] and Nah et al. [23] note the benefit of timely and accurate information, and Chand et al. [4] and Poston and Grabski [24] discuss the resultant expected decisionmaking benefits.Such benefits arise from increased integration of applications [23,24], and Chand et al. [4] and Shang and Seddon [26] even expect reduced IT operating costs.Additional benefits arise from using ERP implementations as a platform for business process re-engineering (refer to [4,23,24]).These benefits, together with the other benefits proposed in the literature are shown in Table 3.1.An 'ABC' ranking scale indicates the level of support for each benefit.
While some of the literature supports these ERP benefits and cites examples of successful implementations (for example [14,24,29]), there are also cases where benefits have failed to be realised (for example [3,25]).Mixed results indicate that further research is needed, and the reasons for certain implementations producing greater benefits than others need to be clarified.

Organisational and ERP performance measurement
Ghalayini and Noble [11] describe how the literature about organisational performance measurement has evolved in two main phases.The first phase describes the use of primarily financial measures, from the late 1880s through to the 1980s.Eccles [7] describes how, during the 1980s, many executives saw deteriorating financial results due to declines in quality or customer satisfaction, or because of the increased pressure from global competitors.Gomes et al. [13] write that, as a result, a number of integrated performance measurement systems, containing both financial and non-financial measures, were proposed.This started the second phase of performance measurement.They claim the most cited PMSs to be the SMART system [5], the performance measurement matrix [17], the balanced scorecard [16], and the integrated dynamic PMS [12].
While there is much literature describing the evaluation of organisation performance, the literature of ERP performance measurement is still emerging.Similar to the first phase of organisational performance measurement, much of the ERP literature focuses solely on the financial impact (for example [15,24]).However, comparable to the critics of traditional firm measurement techniques, Sarkis and Sundarraj [25] criticise this technique for focusing on only one set of measures.As a result, a number of more balanced ERP PMSs (focusing on both financial and non-financial measures) have emerged.Of these proposed systems, the ones that are assessed to be the most comprehensive are: 1. ES benefit framework [26]; 2. ERPS performance measurement model [31]; and 3. ERP scorecard [4].
In deciding which ERP measurement systems to use, the following decision criteria are considered: A Table 3.1: Expected ERP benefits 1.The system must connect measurements with business goals and organisational strategy.2. The system must consist of a balanced set of financial and non-financial measures that can be linked to strategic objectives.3. Measures must be both internally and externally focused.4. The system must enable performance to be reviewed over specified time horizons against specific goals (i.e.allow for time-factor analysis to be performed).5.The system should be easy to use, to enable improvements to be monitored on a regular basis.
Applying the above criteria, the ERP scorecard is the most comprehensive ERP PMS.However, two weaknesses are noted.First, although a framework is defined to evaluate ERP benefits, the benefits are not specified.Second, the phases over which benefits are evaluated are not confined to fixed time periods.Consequently, the ERP scorecard is modified to produce an ERP time-based balanced scorecard (BSC), shown in Figure 3.1.(The BSC framework has the four perspectives with their benefits.)The degree to which benefits are obtained by implementing firms can be populated for the respective time periods.
Populating the scorecard facilitates the use of progress curves (in the form of perceived net benefit flow (PNBF) graphs [26]), to analyse the benefits and the resulting impact on organisational performance.A

Reasons for ERP successes and failures
Nah et al. [23] maintain that, with the high failure rate of ERP implementations, better understanding of CSFs is needed.Recent ERP research has focused on the causes of ERP successes and failures, and a summary of CSFs identified in the literature is displayed in Table 3.2.As with the ERP benefits list, an 'ABC' ranking system is used to indicate the level of support for each CSF.

Summary of literature findings
The findings of the literature review are now given under each of the questions posed earlier.

What benefits are gained from ERP systems?
From the literature, 27 benefits were compiled.The variability of support for different benefits, the absence of local literature, and the lack of research on the extent to which benefits are achieved, indicate that further research is required.

Is there a valid and reliable PMS that links ERP performance to organisational performance?
The ERP time-based BSC is an appropriate system to measure the impact of ERP systems on organisational performance over time, but as this measurement system is largely untested, its validity and reliability need to be confirmed.

Do ERP systems have a positive impact on organisational performance in the time periods following implementation?
The ERP measurement systems reviewed show limited market testing.Of the testing conducted, conflicting results have been found (e.g.Shang and Seddon [26] and Chand et al.'s [4] results show a positive impact, in conflict with the broader study conducted by Wieder et al. [31]).Conflicting results, combined with the general lack of research in this field, result in this question being unanswered through the literature review.Further work is therefore needed to investigate the impact of ERP systems on organisational performance.

What CSFs are responsible for ERP benefits being achieved?
From the literature, 14 CSFs were identified.With varying levels of support for each CSF, the validity of this list needs to be confirmed.No study confirms an association between CSFs being in place and ERP benefits being achieved.The association is a topic for further investigation.

Research problem
The literature review enabled the questions behind the purpose of the research to be partially answered.Further research is required to address the outstanding questions.
Combining the outstanding questions leads to the research problem for this study: To determine the impact of ERP systems on organisational performance by analysing achieved ERP benefits within the framework of a suitable PMS, and to investigate an association between CSFs and ERP benefits.

Research objectives and hypotheses
Breaking the research problem down into its sub-problems enables the objectives for the research to be determined.Objectives define the sequence in which the research questions are investigated.These objectives are investigated to test the following hypotheses: H1: South African manufacturing firms are experiencing business benefits as a result of ERP implementations.

H2:
The ERP time-based BSC is a valid and reliable ERP PMS.

H3:
The benefits gained through ERP implementations have a positive impact on organisational performance over the three year post-'go-live' period.

H4:
Common CSFs are associated with ERP benefits -and hence an increase in organisational performance -being achieved.

Two phase methodology
To investigate the objectives, a two-phase approach is adopted: Phase 1 conducts expert interviews to validate the list of benefits and CSFs established through the literature review.Phase 2 conducts a sample survey of local ERP implementations (using an email questionnaire) to determine the level to which CSFs have been in place, and the benefits achieved.• Determine the extent to which benefits have been achieved Objective 2 • Determine the reliability of the ERP Time-Based BSC Objective 3 • Consolidate benefits to determine the impact of ERP systems on organisational performance Objective 4 • Investigate an association between CSFs and ERP benefits a seven-point Likert scale to ask respondents the extent to which they believe ERP benefits should be achievable within firms.Responses in Table 5.1 use an 'ABC' ranking applied to each benefit based on the level of support.Results are combined with the literature findings to gauge each benefit's support.

Table 5.1: ABC analysis of benefits
'A' benefits are assessed as valid.'C' benefits are classified as invalid due to lack of support, and are removed from the list.'B' benefits are analysed in more depth to determine their validity.This results in two more benefits -'reduced manufacturing cycle times' and 'improved quality' -being removed.The remaining list of 23 benefits is assessed to be adequate to use in the Phase 2 market research.In populating the ERP time-based BSC with these benefits, the content validity of the scorecard is confirmed.
The second section of the questionnaire asks respondents to rank the list of CSFs according to their perceived importance.Classifying the CSFs in this way results in CSFs 11 to 14 being removed, providing a consolidated list of 10 CSFs for further use.

RESEARCH RESULTS: MARKET RESEARCH
This phase uses the ERP benefits and CSF lists from Phase 1, with the ERP time-based BSC, to investigate further the four objectives and the related hypotheses.To extract the information from senior business executives, a quantitatively-based and pilot-tested email questionnaire was used.
The sample frame for this research consists of SA manufacturing companies that have implemented ERP systems.Gupta and Kohli [14] state that the ERP systems from the top software vendors have a feature overlap of around 60-70 per cent.Consequently, this study does not limit itself to the brand of the ERP system, but distinguishes by company size (annual revenue >R300 million), resulting in an estimated population size of about 150 companies.Gay and Airasian [9] recommend that for a population size of around 500 or less, at least 50 per cent of the population should be sampled.In line with these recommendations, questionnaires were emailed to 79 companies, identified mainly through consultancy and corporate client lists.
A response rate of about 25 per cent was obtained.This figure is in line with the expected email response rate of 20-30 per cent [32], and is deemed sufficient for analysis.Note, however, that the validity of the results may be affected by: 1.The non-random selection of the sample population.This includes sampling from firms that do not have failing systems.Due to the low response rate, the data distribution cannot be confirmed, but the distribution of averages, in line with the central limit theorem, is assumed to follow Student's t distribution.Statistics are selected to analyse the results in line with the four objectives.

Objective 1: ERP benefits
Table 6.1 shows the ERP time-based BSC populated with the benefit averages and 95 per cent confidence interval, beyond which significance is claimed.Results are built into progress curves in the form of PNBF graphs to assess the performance impact over time.Figure 6.1 shows an example of the PNBF graphs for the financial perspective.

Table 6.1: Benefits averages and confidence intervals
Note: cells are highlighted where confidence intervals fall below zero.

Part 1: What are the benefits that companies are gaining from ERP systems?
Survey results show that by the end of 'year 1', on average 19 of the 23 benefits were achieved by the surveyed organisations (i.e. they showed a performance improvement significantly higher than zero).By 'year 3', 22 of the 23 benefits had been achieved by the sample set.'Reduced IT operating costs' is the only benefit not to show significant improvement.

Part 2:
To what extent are benefits being achieved over the three year post-'go-live' period?
PNBF graphs show the main benefits achieved in 'year 1' through to 'year 3' to be mostly internal business benefits.By 'year 3', seven benefits are achieved to a 'medium-to-high' extent, 14 benefits to a 'low-to-medium' extent, and one benefit to a 'low' extent.The gradients for all PNBF graphs are shown to be positive between 'year 1' and 'year 3', indicating that the extent to which benefits are achieved increases over each time period.Having confirmed the content validity of the ERP time-based BSC through the structured interviews, Cronbach's α is used to evaluate the internal consistency reliability (and construct validity).The four BSC perspectives form the latent variables (LV), with the benefits within each category comprising the items to be tested.These latent variables are then collectively analysed to test organisational performance as the final latent variable (using both the LV results and the individual items).Table 6.2 summarises the internal consistency results, rating the results based on George & Mallery's [10] recommendations, viz.>0.9 -excellent, >0.8 -good, >0.7 -acceptable, >0.6 -questionable, >0.5 -poor, and <0.5 -unacceptable.In 'year 1', results show a significant performance improvement for all perspectives, with the exception of the financial perspective.By 'year 2' the performance improvement is significant across all four perspectives.This performance impact is seen to increase on a yearly basis.Consolidating the results of the four perspectives shows overall organisation performance improvements to be significantly above zero for all three periods.The level of improvement is shown to increase at a diminishing rate over time.With the exception of the financial perspective, improvement follows conventional learning theory.

Objective 4 -CSF and benefits association
Linear regression determines an association between CSFs and ERP benefits.Making 'benefits' the dependent variable and 'CSF' the independent variable, the regression equation is shown as: Benefits =  0 +  1 CSF+ error Two aspects of the linear regression results tested when investigating a possible association: 1.The sign and strength of the association is determined by the gradient of the regression line.2. Student t-tests are conducted to determine whether the slope of the regression line differs significantly from zero (significance claimed at the 95 per cent level).
What CSFs are associated with ERP benefits being achieved?
Results reveal four associations for 'year 1' and 'year 2', and five associations for 'year 3' that can be described as strongly positive and significant.Three of the identified associations overlap all three periods.These associations are those between: No other associations from a total of 230 were found to be significant.

Benefits from ERP systems
Objective 1 sets out to determine the main benefits that SA companies are gaining as a result of implementing ERP systems, and to gauge the level to which the benefits are being achieved.The literature review enabled a base list of expected ERP benefits to be established.Validation of this list in Phase 1 results in 23 of the benefits being used in Phase 2. The market research results show that 22 of these benefits are achieved.These benefits are being achieved at varying levels, within the 'low-to-high' range.The research shows a strong correlation between the 'A' benefits, established through Phase 1, and the survey results.Based on these findings, Hypothesis I is assessed to be valid, and it is concluded that: South African manufacturing firms are experiencing business benefits as a result of ERP implementations.

Validity and reliability of the ERP time-based BSC
Objective 2 aims to establish the validity and reliability of the ERP time-based BSC.The content validity of the scorecard is first tested through the structured interviews.This content validity is further confirmed by the results of the market research (with 22 out of 23 benefits being achieved).The overall internal consistency reliability (and hence the construct validity) is 'good-to-excellent'.With a limited sample, Hypothesis II is concluded as: Preliminary research shows the ERP time-based BSC to be a valid and reliable ERP PMS.

Impact of ERP Systems on organisational performance
The aim of objective 3 is to evaluate the impact of ERP systems on organisational performance over the three year post-'go-live' period.Consolidating the benefit results of the market research reveals performance improvements across all four perspectives of the ERP time-based BSC, with overall organisational performance showing a 'medium' performance improvement.These findings support the research of Shang and Seddon [26] and Chand et al. [4].However, the research of Wieder et al. [31] attributes performance improvements to enabling systems instead of ERP systems.As all survey respondents indicated the presence of at least three enabling systems in their organisations, there is a chance that the benefits attributed solely to the ERP systems have resulted (at least in part) from the enabling systems.
Despite the potential for bias in the results, and given the assumptions, sufficient evidence been gathered to show that SA manufacturing companies are achieving organisation performance improvements as a result of the benefits gained from ERP systems.Hypothesis III is therefore confirmed, and the research states that: The benefits gained through ERP implementations have a positive impact on organisational performance over the three year post-'go-live' period.

CSFs for a successful ERP implementation
Objective 4 sets out to determine the CSFs required for successful implementation through their association with ERP benefits being achieved.Three strongly positive associations are identified from the market research results.Similarities can be drawn between these associations and the research of Motwani et al. [22] and Umble et al. [30].However, the identified associations conflict in part with the literature and interview results, as they do not include the most supported and highly ranked CSFs.Considering the above, it is concluded that the research has identified three possible associations between CSFs and ERP benefits.However, with the small sample size, as well as conflict (in part) between the literature and interview findings, there is insufficient evidence to claim findings as complete and valid.Hypothesis IV therefore remains unconfirmed, and further research may validate and build on these findings.

RECOMMENDATIONS FOR FURTHER RESEARCH
Based on the limitations and findings of this study, a number of recommendations for further research are suggested: 1.The low sample size obtained from the market research means that further research should be conducted to confirm and build on the organisational performance findings of this study.2. Further work is required to test the associations between CSFs and ERP benefits, using a larger sample set.3. The literature reviewed places little emphasis on the learning and growth benefits from ERP systems.This study shows that these benefits are being moderately achieved.More focus needs to be placed on investigating the learning and growth benefits available through ERP implementations.4. A source of bias in the results was noted, in that benefits from enabling systems may be attributed solely to ERP systems.Further research should investigate this possibility using the results of this study for comparative purposes.The other source of bias is that only firms with existing (and successful) ERP systems were sampled.Further research should be conducted on firms where ERP systems were unsuccessful.5.If the ERP time-based BSC is to gain further use in business or research, validity and reliability need further testing (specifically for the financial perspective and the learning and growth perspective, where Cronbach's α values were shown to be below 0.8).

Figure 4 . 2 : 1 • 2 • 4 •
Figure 4.2: Two-phase methodology5 RESEARCH RESULTS: STRUCTURED INTERVIEWSTen face-to-face interviews were conducted with SA ERP experts and business users, using a pilot-tested structured interview questionnaire.The first section of the questionnaire uses

Table 5 .
2 summarises the results.A

Table 5 .2: CSF interview results
2. The relatively small sample set.3. Sources of potential bias within the sample set (for example, year of implementation, IT base, and enabling systems in place).A

Table 6 .2: Internal consistency reliability summary Is the ERP time-based BSC a valid and reliable ERP PMS?
Results in Table6.2show the internal consistency reliability of the ERP time-based BSC to be 'good-to-excellent' at an overall level.The internal consistency of the four BSC perspectives is in the 'acceptable-to-good' range.

Latent Variable Cronbach's Apha Internal Consistency Rating 6.3 Objective 3: Organisational performance
Summary results of Table6.1 are built into PNBF graphs to show the performance impact on the four scorecard perspectives.Figure6.3applies an equal weighting average to consolidate these results to the organisational level.