USING MONTE CARLO SIMULATION TO CREATE A RANKED CHECK LIST OF RISKS IN A PORTFOLIO OF RAILWAY CONSTRUCTION PROJECTS

When introducing quantified risk assessments at a South African state-owned freight company, a skills gap was found during the identification and quantified risk analysis. To help with risk identification, a checklist of risks for railway construction projects was developed. The basis of this checklist was 38 individual railway construction project risk registers that were collated into a single risk register. After the risks had been cleaned up and classified, a Monte Carlo simulation using @Risk software was done that produced a ranked check list of risks. This list of risks is valuable because subject matter experts developed it, and can be used as a risk identification checklist by stakeholders in similar projects. The simulation results also showed that project scope is an influencing factor on the ranking of risks.


INTRODUCTION
This article presents, as a case study, a way in which quantified risk registers are used to identify and rank the risks in a portfolio of railway construction projects.It starts by presenting the context in which the case study risk registers were collected.It continues with a concise literature survey on risks that can be found on railway construction projects.The model used during the simulation is then broadly discussed, referring in more detail to the use of MS Excel and to Monte Carlo simulation software such as @Risk.The article continues with the simulation results, where a list of ranked risks found on rail construction projects is presented.The simulation results provide some evidence that project scope is a factor determining the ranking of risk in similarly-scoped projects.

Organisation context and problem statement
The case study organisation presented in this research, Transnet, is wholly owned by the Government of the Republic of South Africa.It is a freight transport and logistics company, and is the custodian of South Africa's freight railway, ports, and pipelines infrastructure [1].Transnet's market demand strategy (MDS) requires that a capital investment of R307.5 billion be made between 2013 and 2020.
The MDS aims to expand South Africa's rail, port, and pipelines infrastructure, resulting in a significant increase in freight volumes, especially in commodities such as iron ore, coal, and manganese.Transnet Capital Projects (TCP) is responsible for the development and execution of Transnet's capital projects related to port, rail, and pipeline infrastructure, and the risk registers used for this research were collected by risk managers in TCP.
As per Cooper et al. [2] and Hillson [3], a risk register template (RRT) was developed for use in the TCP project environment using Monte Carlo simulation and @Risk software.Monte Carlo simulation is a computerised mathematical technique that allows risks in quantitative analysis and decisionmaking to be calculated [3,4].Monte Carlo simulation is used to perform risk analysis by developing models of possible results by substituting a range of values, in the form of a probability distribution, for any factor in a project that has inherent uncertainty [2,5].In the context of simulation software, risk is defined as uncertainty or variability in the outcome of some event or decision.Monte Carlo simulation is used to aggregate variation in a system resulting from variations in the system, for a number of inputs, where each input has a defined distribution and the inputs are related to the output via defined relationships.
In practical terms, a Monte Carlo simulation produces a number in conjunction with a likelihoodfor example: "There is an 80% likelihood that the project will be completed on 4 April 2015" instead of "on 4 April 2015".Other advantages over a deterministic or 'single-point' estimate analysis include, among others, the following:


It enables a sensitivity analysis that identifies which inputs have the biggest effect on the results, and allows scenario analysis to take place [4].


It can provide a measure of the accuracy of the result, and the software is readily available and relatively inexpensive [3].
During the roll-out of this quantified risk approach, a skills gap related to the risk identification analysis was found in the case study organisation.To treat this problem, the available risk registers were used to create checklists that could be used during risk assessment.Of the 106 available project risk registers, 86 were quantified as part of this research.Of these, 38 were related to rail projects and were used to create the simulation model that, in turn, produced the simulation results that were used to create the checklist of risks presented in this paper as part of this research.The next section discusses some appropriate literature on risks used during the research process in this paper.

Literature survey
The first part of the literature survey deals with establishing which risks might be encountered on railway construction projects; and the second part covers the literature on running a concurrent Monte Carlo on a group of projects.

Risks in railway construction projects
ISO31010:2009 [6] identifies six different types of risk assessment tools.These include: (i) lookup methods (checklists, preliminary hazard analysis), (ii) supporting methods (structured interviews and brainstorming), (iii) scenario analysis (root cause analysis, fault tree analysis), (iv) function analysis (FMEA, HAZOP), (v) controls assessment (layers of protection analysis, bow tie analysis) and statistical techniques (Markov analysis, Monte Carlo analysis).Of these techniques, lookup methods are useful because they can be used by non-experts, and can help to ensure that common problems are not forgotten.Limitations include that they tend to limit imagination, and have the potential to ignore 'unknown unknowns'.Checklists are most useful when applied to check that all the important aspects have been covered by more imaginative techniques [3].Therefore, since it can be accepted that checklists are a valid method of risk assessment, some sources for potential checklists should be investigated.
A number of textbooks were identified in this research that deal with risk management, but that do not necessarily confine themselves to construction projects.This includes Chapman and Ward [7], Cooper et al. [2], and Kendrick [8].Chapman and Ward's [7] book was targeted at "board level senior managers responsible for project programme and project portfolio aspects of corporate policy, and their integration with corporate strategy and operations", and contained no specific risks related to construction projects.Barkley [9] focused on risks related to a business on an enterprisewide level, and did not contain any lists of potential project risks.
Flyvbjerg et al. [10] dealt with a wide variety of mega-projects (including a large number of publicprivate partnerships) such as the Channel Tunnel, the Concorde, the Sydney Opera House, and the German MAGLEV train between Berlin and Hamburg.They discussed problems with these projects such as how misinformation was used to justify project implementation, and how inaccurate estimates contributed significantly to project overruns [7].Other books contained lists of potential risks.Cooper et al. [2] included a section called "Examples of Risks and their Treatments" that was obtained from a "wide range of projects".Kendrick [8] concluded his book with a list, although not as exhaustive as that provided by Cooper et al. [2].Of all the books encountered in this research, that of Cooper et al. was the most exhaustive.None of these lists specifically related to rail projects.This does not mean that the risks mentioned were not applicable to rail projects, but simply that these lists did not specifically refer to rail projects, and so might be incomplete.
There have been several survey-based journal articles on the types of risks found in construction projects.These include those by Zou et al. [11], Zou et al. [12], Akintoye and MacLeon [13], and Aritua et al. [14].Although Lam's article [15] mentions a "sectoral review of risks associated with major infrastructure projects", it does not refer to rail projects in any detail.The articles tend to refer to the management of safety, health, the environment, and quality, as demonstrated by Albert and Hallowell [16], Fang et al. [17], and Sousa et al. [18].There is a plethora of articles related to tunnel construction, such as those by Rehbock-Sander [38], Lin et al. [20], and Huang [21].
Upon investigation railway construction projects and their risks, searches in textbooks, journal articles, and conference proceedings did not produce any specific lists that could be used for rail projects.The most complete list of risks was that by Cooper et al. [2], which was applicable to all projects, not just railway construction projects.[23] Determine significant risk factors.

Survey
Frequency of response.
Case study Literature review.Lam et al. [24] Methods on the allocation of risk.
Survey Fuzzy set theory.
Rezakhani [25] Classifying risk factors.Literature review Constructed a risk breakdown structure.Zou et al. [11] Determine significance of risk in relation to project objectives.

Survey
Formula based on product between likelihood and impact.
When considering Table 1 in a project risk management context, the primary limitation of the methods in the table is that they don't generally cost risks in terms of what happens to project costs, should a risk realise.The information from a survey that states that "49% of the respondents agreed that the risk is important" is not as rich as a risk that is described in terms of time delays, rates, and additional direct costs -e.g., "Risk XYZ is a multiple occurrence risk, which can happen four times over the duration of the project, has a time delay of up to four weeks, with a weekly cost of R250 000 and an additional capital cost of between R3 and R6 million".When describing and costing risk in this way, and using Monte Carlo simulation, these models -although more complexrepresent an advantage over the methods employed in Table 1.


These methods in Table 1 do not use variability to state which of the risks cause the most uncertainty in the project.@RISK uses Monte Carlo simulation to identify, measure, and root out the causes of variability in the risk register [4].The methods in Table 1 do not make provision for the occurrence of multiple occurrence risks.


The methods in Table 1 do not enable a sensitivity analysis that identifies which inputs have the biggest effect on the results, and that allows scenario analysis to take place [4].
The next step in the research was to find sources that would give an idea of how a Monte Carlo simulation could be run over a portfolio of projects.As a general comment, standards such as ISO31000:2009 [26], PMBoK [27], and the practice standard for risk management [28], do not provide the 'how to' part of risk simulation.Some other sources were more useful.Vose [29] gave a thorough overview of how to develop simulation models, and devoted a short section of the text to 'portfolios of risks', and referred the reader to the help file of ModelRisk, a programme similar to @Risk and also an MS Excel add-in.As with the @Risk help file [4], Vose used the term 'portfolio' largely as something related to finding an optimal portfolio of investments [29].Nothing in these texts specifically referred to the simulation modelling of a portfolio of capital projects.Kwak and Ingall [30] conceptually explored various applications of Monte Carlo simulation for managing project risks and uncertainties.This included the quantification of the effects of risk and uncertainty in project schedules and budgets, giving the project manager a statistical indicator of project performance such as project completion date and budget targets.A large number of articles referred to the use of Monte Carlo simulation in project schedule simulation, including those by Elshaer [31] and Trietsch and Baker [32].
The paper by Jahangirian et al. [33] contained a review of simulation applications published between 1997 and 2006 in peer-reviewed literature in manufacturing and business, and found that Monte Carlo simulation was mostly used to solve numerical problems of a stochastic nature, such as property valuation and risk management.Cost estimation and cost risk analysis were also covered in great detail by authors such as Pugh and Soden [34], Chapman and Ward [35], Chou [36], Sato and Hirao [37], and Khodakarami and Abdi [38].The search for 'multi-project' and 'programme' also yielded some results.Lytvyn and Rishnyak [39] presented a decision-making algorithm that can be used when the multi-project environment influences a project.This, however, did not attempt to simulate and identify risks on a portfolio or programme level.
Some articles discussed risks common to programmes and multi-project environments.Shehu and Akintoye [40] gave an overview of the challenges experienced in the United Kingdom in the successful practice of programme management.Regarding risks in multi-project environments, Aritua et al. [14] differentiated between risks that are common to programmes, risks that are amplified in programmes, and generic project risks.
To conclude, very little could be found on how to run a concurrent Monte Carlo simulation on a portfolio of projects.This meant that a method to do this had to be developed.

Research boundaries
This research has only considered the inputs from the risk register obtained from the case study organisation, and excludes the integration of the identified risks into each project's estimate and schedules.Work done related to the opportunity part of project risk management is also excluded.Simulation models normally require that risks be correlated with each other before executing simulation results [5,41].Due to the complexity and dynamic nature of such a matrix, it was assumed at this stage of the research that all the risks were independent, thus excluding the use of any correlation matrix.There is evidence that, when considering large numbers of items, realistic correlation modelling is rarely practised [42].The basic method to create the simulation results for the railway projects case study is presented in the next section, in Figure 1.

METHODOLOGY
The methodology followed to create the simulation results as part of the research process in this paper is described in terms of (i) the model used, (ii) the creation of a combined risk register (CRR), (iii) projects divided into various categories, (iv) the importance of cleaning up and consolidating risk names, (v) the creation of named ranges, (vi) the creation of reports, and (vii) error checking and model validation.

Model used
The model used in creating the risk check list is described in terms of likelihood, frequency, and consequence.The model made provision for both single and multiple occurrence risks.For single occurrence risks, Table 2 describes the probability values used in the CRR.These values were selected from the likelihood ranges prescribed by the organisation's enterprise risk management (ERM) policy, since @Risk requires a discrete value to run a binomial or likelihood simulation.The use of likelihood ratings forms the probability part of probability-impact grids (PIG) described by Cooper et al. [2] and Hillson [3].In the RRT, the risk consequence was modelled in terms of the financial impact on the project, using the following equation: =   3   ×  ℎ   +    3   where: The latter is best described using an example.A project has two contractors: Contractor A with a weekly average rate of R50 000, and Contractor B with a weekly average rate of R100 000.During the risk workshop, it has been established that, should a specific risk realise, Contractor A will have a 100% loss and Contractor B will have a 25% loss.

Create combined risk register
To run a concurrent Monte Carlo simulation on all the risk registers, all existing risk registers were combined on a single sheet in one MS Excel workbook.The sheet and related formulas were based on the existing RRT.Various supporting sheets such as Project info, Risk names, and Table references were also created.Various lookup functions were used to link the CRR with information that was contained in these sheets.The final CRR for rail projects in the case study contained 427 different risks, with a total of 138 different risk names.

Categorise projects
Projects have many characteristics and attributes that can be used as criteria to categorise them.These characteristics are summarised by Crawford et al. [43] as follows: (i) application area or product, (ii) stage of life-cycle, (iii) grouped or single, (iv) strategic importance, (v) strategic driver, (vi) geography, (vii) scope, (viii) timing, (ix) uncertainty, (x) risk, (xi) complexity, (xii) customer, (xiii) ownership, and (xiv) contractual.Each of the projects was therefore categorised using one of six scope categories, which included (i) foundation repairs, (ii) new railway lines, (iii) power supply, (iv) signaling and overhead traction equipment (OHTE), (v) tunnels, and (vi) rail yards.These categories were used in the simulation to determine whether scope has an influence on the risk ranking in a set of projects.

Clean up risk names
To ensure the accurate aggregation of risks, risk names such as 'industrial action', 'labour unrest', and 'strikes' were consolidated into 'industrial action', while 'inclement weather', 'heavy rains', and 'bad weather' combined into 'inclement weather', etc.

Create named ranges
To simplify the creation and reading of formulas, the MS Excel name manager tool was used to identify and name the columns in the CRR.Four named ranges were created: (i) project category, (ii) risk name, and (iii) simulation result.

Create reports
To generate the output distributions, a =SumIfs() statement, in conjunction with either a =RiskMakeInput() or =RiskOutput(), was used to generate output distributions based on various sets of simulation results (project type, risk name).After a simulation run, tornado graphs were produced.Tornado graphs are the result of a sensitivity analysis that displays a ranking, in the form of regression coefficients, of the input distributions that impact the simulation results.Inputs with the largest impact on the distribution of the output will have the longest bars in the graph [4].The report (Figure 3) used the simulation results and two other criteria from Column C.This enabled the ranking of the risk names in the various project categories.


Cell C4 used the simulation results from the risk register (CRR_Simulation_Result) and aggregated these for each of the individual project names (CRR_Project_Name) as well as each of the risk names (CRR_Risk_name).In other words, a distribution output was created for each of the individual risk names and project categories. Cell C1 used the inputs that were created by Cells C4 to C71 to create a distribution output that would rank the risks for each of the project categories using a tornado graph.For Cell C1, an output distribution for all the risks related to the project category 'foundation repairs' was created.Similar distribution outputs were created for all of the other project categories.

Yes
Step 1 Step 2 Step 3

Yes
Step 1 Step 2 Step 3 The original risk register methodology, and the way in which the described aggregation takes place, were validated in collaboration with Palisade Corporation, the developers of @Risk [44].

DISCUSSION OF RESULTS
The simulation results for the case study rail portfolios are presented in this section in terms of (i) descriptive statistics, (ii) the risks causing uncertainty in the project portfolio, (iii) the effect of scope on the ranking or risks in a specific project category, and (iv) the application of these results in the project management of rail projects.

Descriptive statistics
As described previously, 38 projects were divided into six different scope-related categories.Table 3 shows that the total project budget was about R28 446 million, of which nearly 80 per cent was contributed by new rail line projects.Of the 405 risks in the risk register, 70.4 per cent were open or could still realise, and contributed to the Monte Carlo simulation.

Risks from the simulation results
Table 4 contains the simulation results, ranked according to the mean of the output distribution, in descending order.The frequency of the open risks appears as column 5.

Application of results in rail project management
There are several ways in which the above results could be used in project management to contribute to meeting project objectives.


These checklists can be used as a checklist during the initial phases of establishing a project risk register for rail projects.In addition, the ranked lists can indicate the level of risk, and can be used to prioritise risks that haven't yet been analysed.Going one step further, the list can be used to identify risks for the project development and the project execution phases of the project.Typical risks that might realise in project development, project execution, or both phases, are as contained in Table 6:


Since checklists can tend to limit imagination and can have the potential to ignore 'unknown unknowns', the ranked list can be used at the end of a risk identification workshop to verify the completeness of the risk register. Table 7, that ranked risks according to different project categories, can be used as a 'top 10 risk' list during the risk identification phase.In addition, it can be an input into tender documentation to indicate which risks would be financed by the project owner and which by the contractor.

CONCLUSION
This article has presented a methodology that can be used to determine the risk ranking in a portfolio of projects.This methodology produced simulation results that in turn were used to create and rank a list of 65 railway project risks (Table 4).This list contributes to the body of knowledge related to risks on railway projects, and can be used either during the risk identification phase of rail construction projects or as a list to verify completeness after the risk identification has been concluded.
The simulation results show that the risk ranking differs for smaller groups of projects that have different scopes.This is useful because it can focus the risk identification process, should there be time and cost constraints during the risk management process.However, it makes no claims about the completeness of such a list.
Taking into consideration the lack of information about risks on railway construction projects, both these results provide a useful research contribution with definite practical implications for the railway construction project management and project risk management environments.With regard to future research, the risk sources (as already contained in the CRR) can be cleaned up, analysed, and used to created questionnaires that could be used to interview the executive management, identify the main causes of these risks, and implement appropriate treatment plans.This would ultimately close the 'communication and consultation', 'monitor', and 'review' aspects of the ISO31000:2009 risk management process related to the outcome of this research.

Figure 1 :Figure 2 Figure 3 : 7 Error
Figure 1: Overall method followed in creating simulation results for the combined risk register

Table 1 ,
constructed as part of this research, contains various techniques that are used to identify and rank risks.It is important for the following reasons:

Table 2 : Likelihood categories
A binomial distribution, together with the probability intervals described in Table2, were used to model single occurrence risks.The binomial distribution is a discrete distribution that returns only integer values greater than or equal to zero[4].Risks such as Industrial action, Inclement weather, and Material deliveries can realise more than once on a project, and were modelled as such.A Poisson distribution was used to model the average frequency of these type of risks.The Poisson distribution is a discrete distribution returning only integer values greater than or equal to zero [4].

Table 4 : Risks ranked according to mean of output distribution
Table5contains the simulation results of the regression coefficients of the top 10 risks from each of the rail project categories, and clearly shows that the risk ranking is different for each project category.