EVALUATING OR SELECTING A SUITABLE INFORMATION SYSTEM DEVELOPMENT METHODOLOGY : A CASE STUDY

Information system development methodologies have been applied by numerous organisations since the mid-1980s in an attempt to improve the efficiency and effectiveness of designing and developing new information systems. Despite advances in methodologies, tools and techniques, productivity is still low. High quality products are seldom produced and at high cost. The advantages and disadvantages of using a methodological approach is discussed. The author identifies the key drivers for applying an information system development methodology successfully and provides a method for selecting or evaluating a methodology tailored to an organisation’s unique set of organisational, cultural and environmental variables. The framework has been applied to Waymark Infotech, a South African information technology organisation.


INTRODUCTION
Information System Development Methodologies are used by organisations to structure the Information System Development Process.Each methodology contains its own philosophy and a collection of phases, sub-phases, processes, phase-inputs, phase-outputs (deliverables), procedures, techniques, tools and documentation aids.Some methodologies additionally include Project Management components (Project Management phases, processes, tools and techniques).
Various arguments exist for and against the implementation of Information System Development Methodologies.These arguments have been analysed and synthesised into a Methodology Evaluation Method, which could be applied by Software Development Organisations in evaluating or selecting a methodology that would fit their organisation's composition.

MOTIVATION FOR NOT APPLYING A METHODOLOGY
Capers Jones [9] examined the impact of standards and formal development methods in more than 100 large enterprises in the United States and Europe.He found that people had ambiguous opinions regarding the success of applying methodologies.
Fitzgerald [6] performed research on the use of methodologies, the circumstances in which they are used and the contribution of the methodology to the development process.His study indicated that 60% of the respondents were not using methodologies, while only 6% of the respondents reported on following a methodology meticulously.79% of those respondents that did not follow a methodology indicated that they did not intend to adopt one.
Many organisations favour an a methodological approach.They reason that one could hardly apply the same methodology to different projects, since projects have more differences than similarities (De Marco [4]).Descriptive methodologies reduce rather than increase productivity (De Marco and Lister [5]).This is due to loads of paperwork, scarcity of methods, absence of responsibility and a loss of motivation.Boehm [1] performed a study indicating that a methodology is far less important than the ability of developers and the complexity of the project.
The main arguments against the application of Information System Development Methodologies are as follows.

System Design improvement claims have not been proven
According to Middleton [11], a large number of books have been written on various methodologies (for the training market).These books tend to focus on presenting the methodology rather than evaluating or criticising it.
Fitzgerald [6] also states that generalisations are made without the necessary empirical foundation.

Many of the modern methodologies claim to address certain gaps in traditional Information
http://sajie.journals.ac.zaSystems Methodologies.Some empirical studies have indicated that these claims could not be proved.Purvis et al [12] performed empirical studies to compare the effect of the Joint Applications Design (JAD) Methodology with the traditional Information Systems (IS) Design Methodology.The interactions between users and designers, consensus management and user acceptance of design specifications were compared.His research indicates that "designers perceived JAD as being superior to the traditional IS design method with respect to the quality of user-designer interactions, effectiveness of consensus management, and user acceptance of design specifications" (Purvis et al [12]).The users only perceived better userdesigner interactions.The users did not perceive a significant difference in consensus management or user acceptance of design specifications in comparing the different methodologies.

Methodologies are based on certain rigid assumptions and generalisations. Exceptions are not catered for.
As an example, SSADM (Structured Systems Analysis and Design Method) states that "It is assumed that business planning, IS strategy and tactical planning will have been carried out before an SSADM project is initiated.Whether formally or informally, the types of analysis implied by these tasks must be undertaken before an SSADM project can be initiated" (CCTA, [2]).The problem with this assumption is that strategy may change during the development of a new system.
The requirements phase of SSADM also includes the proviso: "…ensure that all requirements, particularly non-functional requirements, have been identified, are described correctly, and are fully detailed."(CCTA,[2]).Attaining such a full set of requirements is almost impossible as users invariably know what they want, they do not always know the possibilities of the technology, their perceptions change and changes in the external environment cannot be fully anticipated.
The rational and sequential processes of the methodology seldom fit all organisations.

Methodologies unlikely counters staff turnover
Some of the very structured methodologies will not counter the effects of staff turnover or inexperienced staff.The methodology based on the traditional mind-set, where knowledge is seen as "well-defined, unambiguous and articulate" can not produce greater staff productivity where a reality mind-set of "ill-defined, inferred, dispersed and entrenched" dominates (Sauer et al [11]).

Methodologies concentrate on technicalities
Most methodologies treat the System Development process as a rational, sequential process without incorporating the social aspects.Individual creativity and learning-over-time are often not recognised.

Most methodologies are unsuitable for rapid development
Fitzgerald [6] indicated that the organisational environment has changed to such an extent that http://sajie.journals.ac.za many of the methodologies are no longer useful.Methodologies rather add to the lethargy of the development process.Today's systems need to be delivered more rapidly.His study indicates that methodologies are used if five or more developers are employed and when the project duration exceeds nine months.

MOTIVATION FOR APPLYING A METHODOLOGY
The main arguments supporting the application of Information System Development Methodologies are as follow:

Providing a standard
One of the main advantages of using a methodological approach is the standardisation of design, development and implementation procedures.
According to Kruchten [10] many organisations do realise the benefits of using a methodology as a standard.Some develop their own methodologies, which often (according to him) "gather dust in nice binders on a developer's shelf -rarely updated, rapidly becoming obsolete, and almost never followed".Some of the new commercial-off-the-shelf methodologies in contrast (e.g.Rational Unified Process) are developed online using Web technology.Regular upgrades are released in a modular form -it could easily be tailored and configured to suit the specific needs of a development organisation.Many methodologies also promote the use of standard sets and formats of documentation as well as coding standards.This ensures interchangeability among developers.

Ensuring quality
A methodology provides a framework of processes (often including measurements and criteria for their execution).Most methodologies specify the quality required for outputs or deliverables (e.g.test plans, use-case realisations and design models).The methodology may also be used by the organisation to acquire ISO-certification.

Controlling change
Most methodologies specify a set of systematic activities for keeping track of system changes and system defects (as identified during the requirements, design and implementation phases).Changes are then synchronised with the available budget and delivery milestones.

Ensuring re-usability
Certain methodologies (e.g. the Rational Unified Process) are designed to support componentbased development.Due to the implementation of the concepts of modularity and encapsulation, these components may be re-used in different Information Systems, reducing the overall development time of new systems.

Other Advantages
Fitzgerald [8] also mentions the following advantages: http://sajie.journals.ac.za • Due to the complexity of Systems Development, methodologies divide the process into a set of logical steps, which facilitate project management and control of the development process.These management and control elements reduce risk and uncertainty.• A persistent framework is provided for the application of techniques and resources during the development process.• Specialisation and division of labour is provided for, which makes determination of remuneration rates straightforward.• The same framework may also be used for acquiring and storing knowledge and experience.

THE CONTRADICTION
From the previous sections it is clear that literature supports motivations for as well as against the application of Information System Development Methodologies.This contradiction will be explained in the following section.
According to Hares [8], the delivery of low-quality deliverables should not be attributed to the application of a methodology but rather the incorrect application of the methodology.
According to Rai [13] poor performance and failures can be attributed to a number of factorsthe management approach applied to system development projects being the major cause of failure.He also states that methodology frameworks are only useful if they are applied to create process models that "enforce discipline within tasks, establish standardised interfaces between tasks and improve the predictability associated with resource requirements".
Rai [13] performed research to increase the understanding of the interrelationship between development process modelling, task uncertainty and quality-oriented development outcomes.
The research results proved the following hypotheses to be true: • The degree to which a process model has been established for a development project is positively related to process quality and product quality.• The degree of task uncertainty in a development project is directly related to a decrease in the development process quality and product quality.• The interaction between process modelling and task uncertainty influences the development process quality and product quality.
Rai's hypotheses support the motivation for a methodological approach in developing Information Systems.The author proposes that the required process quality and product quality will only be realised if appropriate processes are identified for a specific organisation.

SELECTING THE CORRECT MIX OF PROCESSES
According to De Villiers [3] an organisation needs to consider a whole number of issues before choosing or introducing a methodology or a set of processes and tools into an organisation. http://sajie.journals.ac.za The author studied the various issues defined by De Villiers [3] and Fitzgerald et al [7].The author then identified six main categories for grouping issues or parameters that may influence the success of an Information System Development Methodology.These are: • Organisation (including Project Organisation and the Information System Client) Figure 1 indicates the different categorised parameters -each category having a different block-shape.
In selecting, amending or developing a methodology for a specific Organisation or Project, the selected Methodology parameters need to reflect reality in addressing Organisational -, Cultural -, Environmental -and Problem -parameters as well as the Project Management parameters.
Some Organisational parameters may drive the selection of a specific Methodology.For example, organisations that define unclear or unrealistic strategies should apply Information Systems Development methodologies that suit these strategic ambiguities.These organisations would require a methodology (such as Rapid Application Development Methodology OR the Incremental Development Methodology) that continually validates system requirements, software deliverables and embedded strategies.Other organisations may need to strategically release a product with reduced functionality to counter a move by a competitor.In cases such as these, organisations would require an iterative development approach.As an iterative approach tends to become uncontrolled, a methodology could aid in providing guidelines regarding iteration planning (e.g.numbering, allocating duration and objectives as well as tasks and responsibilities to each iteration).
In the following sections of this article, Fitzgerald's framework [7] for comparing methodologies has been expanded to illustrate the different methodology parameters, which will be applied in evaluating the feasibility of a specific methodology for a specific organisation.
Figure 1 illustrates the main components of a methodology.Each component contains a set of related parameters.
• Application Area Domain: This component represents parameters, which restricts the application area for which the methodology may be suitable.• Project Management: This includes project-related parameters.
• Modelling Types: The nature of various methodology models.
• IS Development Methodology Scope: The structural elements of the methodologyphases, sub-phases, processes, inputs and outputs (deliverables).Other scoping parameters are also included: the interaction and iteration of phases, sub-phases and processes; integration with other systems; inter-phase communication and identification and management of design-or development changes. http://sajie.journals.ac.za • Procedures: Step-by-step processes for executing higher-level processes.
• Techniques and Tools: The parameters included here, portray the type of tools and techniques, their interaction and the capability of the methodology to expand the current set of tools and techniques.• Documentation Templates and Aids: The set of electronic documentation templates that may be used to standardise Information System Development-related documentation.
• Practice: This component includes parameters, which describe the number and type of users who currently apply the methodology (in practice) as well as the type of participants responsible for implementing the methodology.
• Product: This component contains the parameters, which describe the methodology software package that may be available as well as the training, support and training documentation accompanying the software package.
Figure 1 indicates a set of 'Other Parameters'.Although these methodology-independent parameters do not form part of the methodology itself, the success of an Information Systems Development Project also relies on these parameters.Since the methodology-dependent parameters are related to the structure and content of the methodology itself, only these parameters will be applied in the Methodology Evaluation Method that follows.

THE METHODOLOGY EVALUATION METHOD
The value proposition of the Methodology Evaluation Method is to provide a quantitative method to: • Evaluate several Information System Development Methodologies to select the most suitable methodology for a specific organisation or project.• Evaluate the suitability of a current Information System Development Methodology (applied by a specific organisation) to highlight low-scoring methodology elements for possible methodology enhancement or amendment.

Defining the Method
The evaluation and selection of a methodology is to a large degree a subjective process.The evaluation method may be simplified by using the Parameter Framework elements of Figure 1 as measures in designing a Methodology Evaluation Table (Table 1).
The Methodology Evaluation Method consists of three processes: 1. Defining the purpose of the evaluation.2. Completing a Methodology Evaluation Table .3. Interpreting the results.

Defining the purpose of the evaluation
The purpose of the evaluation could be to: • Evaluate several Information System Development Methodologies and select the most suitable methodology for the specific organisation.

Completing a Methodology Evaluation Table
This process includes the following steps: • Measures are listed in the first column (corresponding with parameters in Figure 1); • Evaluation Criteria are described in the second column; • Methodology Inclination (third column) describes the tendency of the Methodology regarding the specific parameter; • Organisation Inclination (fourth column) describes the tendency or requirements of the organisation regarding the specific parameter; • % Fit (fifth column) indicates the extent to which the Methodology proposition fits the Organisation's requirements; • Weight (sixth column) indicates the relative importance allocated per measure -the Likert Scale is used ('1' indicates low importance, while '5' indicates high importance).

• The Results of the Methodology Evaluation Table is summarised by calculating an
average score on {'%Fit' multiplied with the corresponding 'Weight' values}.
Note that columns four to six require subjective inputs from organisational or project representatives.The weight-allocations (sixth column) may vary for different organisations according to the organisation's requirements and philosophic propensity.

Interpreting the results
If the organisation evaluates several methodologies for the purpose of selecting the most suitable methodology for the organisation of a project, the methodology that obtained the highest score will be selected.
If the organisation evaluates its current methodology for the purpose of identifying lowscoring elements, the results may indicate priorities for methodology enhancement or amendment.The results may also indicate a low overall score, which may justify adoption of a different methodology.Once-off purchase of electronic methodology material.

APPLYING THE EVALUATION METHOD TO WAYMARK
Due to ease of use, no additional support is required.
Ease of use and teachability.Easy to use -interactive pages.

Organisation
Inclination not applicable.
Training required prior to using the methodology.
Prior training not really required.

Organisation
Inclination not applicable.

•
Evaluate the current Information System Development Methodology elements for possible methodology enhancement or amendment.

Table 1 : Low-scoring methodology elements WAYMARK
is often forced to comply with a completely different methodology (e.g.SUMMIT or Pi-Tech) as prescribed by their clients.The Methodology Evaluation Method may be applied in evaluating each methodology within the organisational and project context.
"5"), but obtained a weighted score (% Fit x Weight) of 2 or less may be targeted for improvement or enhancement.The following table provides a list of these low-scoring parameters.

Table 2 : Methodology Evaluation Table applied to WAYMARK 8. CONCLUSIONS Methodologies
may have a positive effect on the overall effectiveness of a Systems Development Project if a suitable methodology is selected.The author proposes a Methodology Evaluation Method that may be used to facilitate the methodology evaluation process in evaluating or selecting a suitable Information System Development Methodology for a specific organisation.The evaluation process may also highlight certain aspects within the currently applied methodology that may require improvement or enhancement.