USING A SIMULATION MODEL TO DETERMINE THE OPTIMUM NUMBER OF PILOTS IN AN AIR FORCE 1

The key objective is to present a simulation model to determine the optimum number of pilots in an air force. Background information about the simulation project, and a cursory description of the system under consideration, are provided. The Simul8 simulation software package, the basic concepts of the simulation model, the determination of adequate samples sizes, and simulation model verification and validation are discussed. Generic examples of typical results are shown, and conclusions about the simulation project are presented.


INTRODUCTION
The Pilot Operational Personnel Simulation Model was developed by the Defence Institute for the Directorate Air Capability Plan (DACP) of the South African Air Force (SAAF).The Defence Institute is a division of Armscor Business (Pty) Ltd, and its primary function is to provide support to the Staff Officers of the South African National Defence Force (SANDF).During 2003 the DACP identified that a decision support tool was needed to help them to determine the number of pupil pilots that should enter the system every year in order to keep the flying schools and operational squadrons fully staffed with qualified pilots.After discussions between representatives of the DACP and the Defence Institute, it was decided that a stochastic simulation model would be the correct decision support tool to utilise in this instance, and the simulation project was initiated.(It is important to note that the simulation model is primarily concerned with the operational pilots in the SAAF, and does not include pilots that are in other postings -except that the simulation model keeps track of the number of operational pilots that move on to Staff postings during the period of evaluation.) The simulation model was developed over a period of about two-and-a-half years, from the latter part of 2003 to 2005.The total expenditure in terms of man-hours over this period was about 1700 hours.(This number reflects only the man-hours expended by the Defence Institute, and does not include those expended by the Staff and Operational Officers of the SAAF.)This development history relates closely to a comment from Crowe et al [1], that it may take a few man-years to supply answers to complex problems with a simulation model.The basic simulation model was already well established by the end of 2004, but during the first half of 2005 it was substantially enhanced by the inclusion of the capability to model, to a much higher level of detail, the attrition of pilots who are lost due to non-renewal of contracts and resignations.The further development and use of the simulation model are ongoing processes, and it is anticipated that the simulation model will continue to be enhanced, refined, and used as a decision support tool in the foreseeable future.
Even though the simulation model was originally established to answer the core question regarding the number of pupil pilots that should enter the system every year, its role as a decision support tool has evolved over time.It soon became apparent that several other areas of concern could also be addressed by the simulation model.
Currently the following the key aspects are under consideration: a) The determination of the optimum number of pupil pilots that should enter the system every year.b) The determination of the optimum number of pilots at the flying schools and the operational squadrons.c) The identification of areas of concern (e.g.identifying 'bottlenecks' in the system, monitoring the ratio between pupils and instructors, etc.).
The simulation model is used to determine the optimum number of pilots in the SAAF for every scenario that is evaluated.It is important to note that in this instance http://sajie.journals.ac.za the term 'optimum' does not refer to a formal mathematical optimum, but rather to an answer that is good enough (i.e.acceptable) in the given circumstances, even though it may not be the best possible answer.
Pegden et al [2] propose 12 steps that should be taken in every simulation project.A detailed discussion of the steps of a simulation project, as well as the way that they manifested in this particular instance, does not fall within the scope of this article.However, the system under consideration, the simulation software package that is used, the basic concepts of the simulation model, examples of typical results, and conclusions about the simulation project will be discussed.

SYSTEM DESCRIPTION
In order to facilitate a better understanding of the Pilot Operational Personnel Simulation Model, it is necessary to provide a brief overview of the system under consideration.In this instance the system consists of all the schools and squadrons of the SAAF where operational pilots are trained and used.
The simulation model incorporates the Central Flying School and the three System Groups of the SAAF: the Combat System Group, the Helicopter System Group, and the Air Transport and Maritime System Group (see Figure 1 for a simplified schematic representation of the structure of the simulation model).The Air Transport and Maritime System Group consists of a conversion squadron and three different 'lines': the Passenger line, the Tactical line and the Maritime line.The pupil pilots enter the system through the Central Flying School, and those that qualify as pilots are posted, according to requirements, predisposition, and personal preference, to the System Groups.Upon their arrival at the System Groups, the pilots are first converted to the required aircraft type, and then those that qualify are moved forward on the career path of each System Group.
Each school and squadron comprises sub-groupings of pupil pilots, pilots, co-pilots, commanders, and instructors.For example, 35 Squadron consists of three subgroupings: the C-47 co-pilot group, the C-47 commander group and the C-47 instructor group.Upon arrival at 35 Squadron the pilots are designated C-47 copilots and are trained on the aircraft type by the C-47 instructors.Once the C-47 copilots have completed a tour of duty of around two years and amassed enough flying hours, they qualify as C-47 commanders and complete another tour of duty of about four years.Therefore a pilot will typically spend about six years at 35 Squadron in the roles of co-pilot and commander, while the tour of duty of an instructor at the squadron is about three years.The same basic concept, with simplifications or complications in some instances, applies to all the schools and squadrons.
Along the way -and upon completion of the pilot career paths of the different System Groups -pilots with the required predisposition and experience may return to the Central Flying School to be trained as instructors.Once qualified as instructors they complete a tour of duty at the Central Flying School, and then return to the System Group that they originally came from to be trained and used as instructors at the schools and squadrons.The whole system therefore actually consists of two parallel career paths, one for pilots and the other for instructors.Because of the rigours of flight training, the ratio of pupils (i.e. the pupil pilots at the Central Flying School and the pilots that are trained on the aircraft types at the other schools and squadrons) to instructors is of the utmost importance.

Figure 1: Pilot operational personnel simulation model structure
There is also a multitude of other complexities that have to be accommodated by the simulation model but that are not indicated in the simplified diagram presented in Figure 1.For example, pilots who fail to qualify as combat pilots at 85 Combat Flying School may be rerouted from the Combat System Group to the Air Transport and Maritime System Group, where they will start their conversion to transport pilots at 41 Squadron.

Background
According to Pegden et al [2] very few real-world systems are free from the influence of random variation.When a simulation model makes provision to accommodate the randomness of a system, it can render a very close approximation of the behaviour of the real-world system that is under scrutiny.That is why it was decided that a stochastic simulation model would be the correct decision support tool to use in this instance.
Kelton et al [4] propose that the following three dimensions can be used to classify simulation models: static versus dynamic, discrete versus continuous, and deterministic versus stochastic.According to this proposed system of classification, the Pilot Operational Personnel Simulation Model may be classified as a dynamic, discrete, stochastic simulation model.For the sake of simplicity, such a simulation model is usually referred to just as a stochastic simulation model; the dynamic and discrete parts of the classification are usually assumed or taken for granted.

Software package
The simulation model was developed in the Simul8 simulation software package.Two different simulation software packages were originally considered: Simul8 and Arena. 4Simul8 was chosen for the following primary reasons: a relatively low acquisition cost, no annual licensing fees, a comparatively simple modelling environment, and the inclusion of an internal logic programming language.(Albertyn [5] provides a cursory comparison between the Arena and Simul8 simulation software packages.)The low acquisition cost and simple modelling environment were deemed to be important factors because, in the original planning of the simulation project, it was envisaged that the Defence Institute would be responsible for the development and maintenance of the simulation model, but that the day-today use of the simulation model for scenario analysis would be handled by the DACP.The implementation of that plan would obviously have resulted in the acquisition of an additional Simul8 simulation software package for the DACP.That plan has since been revised, and the current model of operation is that DI develops, maintains, and uses the simulation model for scenario analysis, in close collaboration with the Staff Officers of the DACP.The internal logic programming language of Simul8 is called Visual Logic (VL).The following quotation from the Simul8 ® : Manual and Simulation Guide [6] explains what VL is and how it is used in a simulation model: "Visual Logic (VL) is Simul8's logic building environment.In a simulation of significant complexity you will want to add your own rules for deciding how to process work.VL lets you add very detailed logic to control the operation of your simulation."

Description
At the start of the simulation project, a lot of effort was expended to establish a system description and the boundaries of the system under consideration.More specifically, the major problem was to determine the required resolution or level of detail of the simulation model.The required level of detail should be chosen in accordance with the objectives of the simulation model.Enough detail should be included to validate any inferences drawn from the use of the simulation model, without making the simulation model cumbersome by the inclusion of trivia.During discussions with representatives of the System Groups, the required level of detail was determined, and the expectations of the clients were aligned with the technical possibilities of simulation modelling.
The simulation model consists of the following three hierarchical levels: a) Level 1 -contains the sub-groupings of pupil pilots, pilots, co-pilots, commanders, and instructors of the flying schools and the operational squadrons.b) Level 2 -contains the modelling constructs of the sub-groupings.c) Level 3 -contains the detail to model the attrition of pilots who are lost due to non-renewal of contracts and resignations.
Figure 2 is a screenshot of the simulation model's layout in the Simul8 simulation window.The Central Flying School and the Combat System Group are shown in the view that is presented, while scrolling down in that view on a computer screen will reveal the Helicopter System Group and the Air Transport and Maritime System Group.This view represents the first hierarchical level (Level 1), and each subgrouping of pupil pilots, pilots, co-pilots, commanders, and instructors is represented by an aircraft or helicopter icon.The layout of the icons in the simulation window conforms very closely with the diagrams of the system description that were developed during the initial stages of the simulation project.According to Elder [7] the realistic representation of a simulation model in a layout or configuration that is immediately recognisable is fundamental to the successful familiarisation with, orientation to, and acceptance of, the simulation model by clients and users.
Clicking on an aircraft or helicopter icon will reveal a Simul8 subwindow that contains the modelling constructs of the second hierarchical level (Level 2).Among the constructs of each subwindow is an icon that provides access to yet another subwindow that contains the constructs that model the attrition of pilots who are lost owing to non-renewal of contracts and resignations on the third hierarchical level (Level 3).
Each pilot is created as an entity in the simulation model, and flows forward according to the appropriate career path, qualifying on specific aircraft types, completing tours of duty, qualifying as an instructor, exiting the system, etc.The attributes of each pilot are assigned to different labels that are attached to the entity that represents the pilot.For example, upon successful completion of the instructor course at the Central Flying School, the instructor label of the pilot will be changed from zero to one, signifying that the pilot is qualified as an instructor.
The pilots are routed at decision points in the simulation model with the following two types of decision branches: a) Rule-based decision branches (for example, only qualified instructors may become instructors at the schools and squadrons).b) Percentage-based decision branches (for example, for a specific scenario the pupil pilots who qualify as pilots at the Central Flying School may be posted to the System Groups according to a split of 15%, 50%, and 35% to the Combat, Helicopter, and Air Transport and Maritime System Groups respectively).

Figure 2: Simul8 simulation window of simulation model
The percentage-based decision branches provide the variability or stochastic aspect of the simulation model.According to the Simul8 ® : Manual and Simulation Guide [6], the destinations to which entities are routed are decided randomly, but the exact percentage of entities going to each destination can be specified.
For every scenario that is evaluated, the simulation model has to be set up according to the configuration of the system that is modelled in that specific scenario.This involves ensuring that the simulation model parameters represent the scenario that is evaluated, populating the simulation model with all the pilots that are in the system at the start of the simulation run, and providing the correct schedule for the addition of pupil pilots to the system during the simulation run.The simulation model parameters determine the process flow and process logic (i.e. the rules of operation) of the simulation model.To populate the simulation model with all the pilots that are in the system at the start of the simulation run is no arbitrary task, because each pilot has to be placed (i.e.created as an entity) at exactly the right place in the simulation model and assigned (to the labels of the entity) the qualifications already obtained by that pilot.For example, at the start of the simulation run a specific pilot may be in the second year of a three-year tour of duty as a Cheetah pilot at 2 Squadron, and may be qualified as a pilot on all the aircraft types that lead up to that point in the career of a Combat System Group pilot: Astra, Impala, and Cheetah.The schedule for the addition of pupil pilots to the system during the simulation run consists of two intakes per year of pupil pilots for the duration of the simulation run.
Any logical time period can be evaluated by the simulation model, but it is currently set up to evaluate a period of 10 years.That provides insight into the probable behaviour of the system for a planning horizon of 10 years, or two consecutive fiveyear planning cycles, into the future.

Minimum sufficient sample size
The results of the different replications of a simulation run of a stochastic simulation model are usually not identical because of the random behaviour of the random phenomena -like failures in manufacturing plants, or in this instance, the variability that is introduced into the simulation model by the percentage-based decision branches.This implies that a simulation run consisting of more than one replication has to be completed in order to obtain a mean result that is representative of the simulated scenario.The determination of the minimum number of replications that would yield a mean result that is representative of the simulated scenario is a 'determination of minimum sufficient sample size' problem.An equation from Crow et al [8] can be used to determine the minimum sufficient sample size for sample sizes less than 30, and an equation from Miller et al [9] can be used for sample sizes larger than or equal to 30.
A simulation run comprising 100 replications of a simulated period of 10 years was completed with the simulation model, and the results were evaluated.The results indicate that a sample size of 100 satisfies the requirements for an allowance for a deviation of 5% (i.e. the maximum error of the estimate is 5%) from the values of the variables under consideration, and a 95% confidence interval (using the equation from Miller et al), while a sample size of 20 satisfies the requirements for an allowance for a deviation of 10% from the values of the variables under consideration and a 90% confidence interval (using the equation from Crow et al).
This allows simulation runs comprising 20 replications to be used during the initial stages of scenario analysis when several ideas are bandied about, and simulation runs comprising 100 replications to be used in detailed scenario analysis once the scenarios under consideration have been clearly defined.

Runtime
The runtime of a simulation run comprising 100 replications of a simulated period of 10 years is around 65 minutes, while the runtime of a simulation run comprising 20 replications of a simulated period of 10 years is about 13 minutes.(In both instances the time that it takes to write the results to preformatted spreadsheets is not taken into account.)The principal features of the hardware configuration of the computer that was used are an 800-megahertz processor and 128 megabytes of Random Access Memory (RAM).

Verification and validation
Various authors and manuals stress the importance of comprehensive simulation model verification and validation before the results that are generated by a simulation run can be accepted as representative of the simulated scenario (Harrell and Tumay [10]; Kelton et al [4]; Pegden et al [2]; Simul8 ® : Manual and Simulation Guide [6]).
According to Pegden et al, verification is the process of determining that the simulation model operates as intended, and validation is the process of reaching an acceptable level of confidence that the inferences drawn from the simulation model are correct and applicable to the real-world system that is represented.In this instance the simulation model was verified by meticulously checking that the process flow and process logic of the simulation model function as intended (i.e. in accordance with the system description that is established for each scenario that is evaluated).The validation of the simulation model is more problematic, because the simulation model projects from the present (the current configuration of the system) into the future (the different proposed new configurations of the system in the future), and there is therefore no historical data available against which to validate the outputs of the simulation model.However, this problem was circumvented by analysing the outputs of the simulation model together with representatives of the System Groups to ascertain whether the results realised by the simulation model could reasonably be expected in the given circumstances.

Pilot attrition
Two types of attrition are modelled in the simulation model: the attrition of pilots who are lost in accidents (modelled as a constant percentage over the period of evaluation), and the attrition of pilots who are lost owing to non-renewal of contracts and resignations (modelled as a variable percentage over the period of evaluation).
The simulation model originally modelled the attrition of pilots lost owing to nonrenewal of contracts and resignations as a constant percentage over the period of evaluation, but was later enhanced to allow the modelling of the attrition as a variable percentage over the period of evaluation.This attrition needs to be modelled as a variable percentage because, even though the percentage of pilots who do not renew their contracts every year may be fairly constant, the number of pilots whose contracts are due for renewal each year is not constant for historical reasons (e.g.moving from seven-year contracts to thirteen-year contracts recently).The percentage of pilots who resign (i.e. who leave the system before the end of their contracted period) each year may also be subject to fluctuation owing to external http://sajie.journals.ac.za factors -such as a new commercial airline recruiting aggressively for experienced pilots in a specific year.A comparison between the predicted attrition (derived from historical data and informed estimates) of pilots who are lost due to non-renewal of contracts and resignations, and the results for the attrition from the simulation model for a typical scenario and a 10-year period of evaluation, is shown in Figure 3. (The numerical values of the vertical y-axis are not shown for confidentiality reasons.)It is obvious that the predicted attrition shows large variation over the period of evaluation, and that the simulation model is able to model the predicted attrition relatively closely.(Even though the individual results for some years may show larger deviations between the predicted attrition and the simulation model attrition, the total simulation model attrition over a 10-year period of evaluation typically deviates less than 3% from the total predicted attrition.)It is important to note that the simulation model is not used to predict pilot attrition.The pilot attrition is just one of many concepts that need to be incorporated by the simulation model in order accurately to model the system under consideration.

User-friendliness
On completion of a simulation run, the simulation model automatically writes the results to a set of preformatted Microsoft Excel spreadsheets. 5This feature greatly enhances the user-friendliness of the simulation model, because the user does not have to waste time manipulating blocks of data to find mean values, generate graphs, etc.The spreadsheets are set up to represent the required results in the required format, and lead to a standardised format for the presentation of the results of different simulation runs.

Size
The size of the Simul8 file of the simulation model is 2,619 kilobytes.When viewed in isolation, it is difficult to attach any significance to this numerical value.However, if this simulation model size is compared with the 937 kilobytes size of a Simul8 simulation model of a petrochemical plant (see Albertyn [5] for a detailed discussion of this model), it can be concluded that the Pilot Operational Personnel Simulation Model is fairly sizeable.

MODEL APPLICATION
Currently there are three key areas where the simulation model is used as a decision support tool: the determination of the optimum number of pupil pilots who should enter the system every year, the determination of the optimum number of pilots at the flying schools and the operational squadrons, and the identification of areas of concern (e.g. by identifying 'bottlenecks' in the system, by monitoring the ratio between pupils and instructors, etc.).A large number of scenarios have already been evaluated, and it is foreseen that the simulation model will continue to play a role as a decision support tool in the future.
The results from the different scenarios are used to evaluate the following two basic concepts: a) The staffing structure of a 'design to functional requirements' SAAF.b) The staffing structure of a 'design to budget' SAAF.
The 'design to functional requirements' SAAF represents the system that is needed successfully to complete the operations, missions, and tasks that fall within the sphere of responsibility of the SAAF; and the 'design to budget' SAAF represents the system that is possible within the allocated budget.The goal is to meet all the functional requirements within the restrictions of the allocated budget.
In most instances an iterative process is used when a scenario is evaluated with the simulation model.First the simulation model is set up according to the configuration of the system that is modelled in that specific scenario, and then a simulation run is completed to provide a baseline result.The baseline result is used as the point of departure for the further adjustment and refinement of the values of the parameters that define the configuration of the system that is modelled in the scenario.This process entails the completion of a series of simulation runs where the result of every simulation run is evaluated and then used to determine how the values of the parameters should be adjusted for the next simulation run.Through this iterative process, the optimum number of pilots in the SAAF is determined for every scenario that is evaluated.
Examples of typical results that follow from the simulation model are shown in the http://sajie.journals.ac.za next two figures.Figure 4 shows the results for the number of co-pilots, commanders, and instructors at 35 Squadron for a typical scenario and a 10-year period of evaluation.(The numerical values of the y-axis are not shown for confidentiality reasons.)From Figure 4 it is obvious that the total number of pilots at 35 Squadron first decreases and then increases slightly over the 10-year period of evaluation.This behaviour correlates closely with future requirements at the squadron.It is also evident that, even though the ratio between co-pilots and instructors shows some variation over the 10-year period of evaluation, the final ratio represents a sound pilot operational personnel situation for the squadron.The same principle applies to the ratio between co-pilots and commanders.Figure 5 shows the results for the total number of pilots in the SAAF for a typical scenario and a 10-year period of evaluation.(Once again the numerical values of the y-axis are not shown for confidentiality reasons.)From Figure 5 it is obvious that the general trend for the total number of pilots in the SAAF is upwards, and stabilises around years nine and ten for this scenario.This upward trend can be ascribed to the fact that the SAAF has to grow its pilot numbers to accommodate various new acquisitions to the inventory of the SAAF: the Light Utility Helicopter (LUH), the Hawk and Gripen combat aircraft, and the A400M transport aircraft.The decline in the total number of pilots in the SAAF during year three can be ascribed to the high number of pilots who are lost owing to non-renewal of contracts and resignations in year three (see Figure 3).
It is also interesting to note that the simulation model proved flexible enough to be used as a decision support tool in other areas of concern than the original 'design domain'.For example, several scenarios were evaluated with the simulation model to provide input to the responsible Staff Officer for a study on the racial transformation of the SAAF.

CONCLUSIONS
At this stage in the life cycle of this simulation project, it is the general consensus of all the involved parties that the development and use of the Pilot Operational Personnel Simulation Model may be classified as successful, in that the original goals have been met and the simulation model has proved to be a flexible decision support tool.
The success story of this simulation project may be attributed to the following factors: a) The belief of the director of the DACP in the power of simulation modelling as a decision support tool, and a willingness to invest resources in the simulation project.b) The exposure of the simulation project (through presentations and demonstrations) to various forums in the higher echelons of the SAAF in order to gain recognition for the simulation model and establish its credibility.c) The involvement of the representatives from the System Groups at the start of the simulation project to establish a system description, and during the simulation project to validate the outputs of the simulation model.

d)
The close collaboration between the responsible Staff Officer and the Defence Institute personnel involved with the simulation project.e) The fact that the required level of detail of the simulation model was kept within acceptable limits, and that the expectations of the clients were aligned with the technical possibilities of simulation modelling.f) The fact that there is an established knowledge base in the Defence Institute for simulation modelling and the use of Simul8.
The Simul8 ® : Manual and Simulation Guide [6] indicates that one of the important things to realise about simulation modelling in practice is that some benefit can already be derived just from the process of developing a simulation model, even before the simulation model is used to evaluate alternative scenarios.In this instance the process of establishing a description of the system under consideration forced the representatives of the System Groups to revisit and re-evaluate the career paths of the pilots, and the insights gained from this led to a better understanding of the intricacies of the system for all the parties that are involved.
Kruger [11] states that "... many more [simulation] models suffer from too much detail than suffering from not enough detail.An attempt should be made to keep the [simulation] model as simple as possible ...".The importance of keeping it as simple as possible was once again highlighted by this simulation project, and even though this was applied diligently during the development of the simulation model, the size and complexity of the system under consideration still led to a fairly sizeable and complex simulation model.
Throughout the world the military budgets of countries are coming under increased pressure, leading to a critical need in the military to increase efficiency.Simulation modelling has been identified as one of the key technology areas of future research by the European Union (Geril [12]).It therefore stands to reason that simulation modelling qualifies as a best practice technology that can be used by the military in the quest for higher efficiency.It is encouraging to see that the SAAF has taken cognisance of this fact, and is using simulation modelling as a decision support tool.
The following quotation from the director of the DACP, Brigadier-General P.J. van Zyl, provides the client's perspective on this simulation project: "Our strategic planning environment is characterised by considerable financial constraints, emphasising the need to balance resources to a much higher degree than previously required.The number of pilots in the system cannot exceed that which is affordable within the budget allocation, and that requires strict control over the number of candidates recruited for pilot training.The training cycle from recruitment to wings qualification, however, is so long that any change made in the present will only become evident three to four years hence.
The number of variables in the Air Force pilot system is surprisingly high, and each has the potential to invalidate any correction made in the present.For this reason, it was felt that a simulation model with the ability to represent real life as closely as possible should be able to assist the Air Force in projecting current trends into the future, so that coherent recruitment and training plans could be compiled.
In this respect the simulation model developed by Albertyn & Trichard has proved a major asset in the Air Force's attempts to optimally balance its critical resources.Sufficient time has lapsed since its inception to provide some degree of empirical validation, and an extraordinarily close correlation between original personnel predictions and real life has been observed.This has increased the confidence level in the employment of the model, and has enabled strategic planners to focus on alternative outcomes rather than spending valuable time on manually fine-tuning the large range of variables, as was previously required.

Figure 3 :
Figure 3: Comparison of predicted and simulation model pilot attrition

Comparison of Predicted and Simulation Model Pilot Attrition (Pilots lost due to non-renewal of contracts and resignations)
From a strategic perspective, one of the most valuable aspects of the simulation model is its ability to assist multiple what-if analyses.By successively changing the most significant variables, such as resignations and pass rates at the training schools, and running multiple simulations, it is easy to identify potential bottlenecks, staffing levels at advanced training schools, and recruitment targets for candidate pilots.The ability to do worst-case projections and then relate those projections to cost and personnel implications have proved invaluable to the Air Force.In a relatively short period, this model has become an indispensable strategic planning tool in the SA Air Force, and has made an enormous difference to the accuracy level with which we are able to balance scarce resources."