SUSTAINABILITY ASSESSMENT CRITERIA FOR PROJECTS AND TECHNOLOGIES : JUDGEMENTS OF INDUSTRY MANAGERS

Industries are increasingly under pressure to incorporate the objectives of sustainable development into company policies and decision-making processes. This study introduces a framework of sustainability assessment criteria that are relevant to projects and developments in industry. The study attempts to establish whether the professional exposure and experience of decision-makers at different management levels influence the individual’s perceptions of the relative importance of the three main dimensions of the framework. The study finds that exposure and experience do not influence the weighting values. The study further highlights the importance of distinguishing between internal decision-making within industry and external decision-making where assessments are used for public reporting. That is, the context within which the three dimensions are weighted greatly determines individual and societal perceptions.


INTRODUCTION
In 1980 the World Conservation Fund, the United Nations Environmental Program (UNEP), and the World Wildlife Fund (WWF) formulated the World Conservation Strategy, which introduced the concept of "sustainable development", although it was not explicitly mentioned [1].In 1987, the World Commission on Environment and Development formally introduced and defined the term "sustainable development" in their report, now generally referred to as Our Common Future.The report stimulated unprecedented levels of worldwide public discussion of the tensions between the environment, society, and the economy [2].In 1992 the United Nations held a World Conference on Environment and Development in Rio de Janeiro.The result of this conference was embodied in two important documents: • Rio Declaration on Environment and Development: A statement of twentyseven principles that sets out the basis upon which states and individuals are to co-operate to further develop international law in the field of sustainable development.
• Agenda 21: A blueprint or action plan for the implementation of sustainable development [3].
In the latter, an entire chapter addresses the need to incorporate the environment and development into decision-making at all levels of government and business management [4].The need to incorporate the concept of sustainable development into decision-making, combined with the World Bank's three-pillar-approach [5] to sustainable development, resulted in the popular business term "triple-bottom-line decision-making".
Techniques to ensure that all three dimensions of sustainable development are considered during decision-making, together with various frameworks to define the three dimensions, have been developed since 1992 [6]. Since sustainable development emphasises evaluation rather than valuation, it has been argued that traditional decision-making techniques based on reducing all information into economic terms cannot be applied, since all social and environmental consequences are not necessarily reducible to economic metrics [7].
This paper focuses on another specific evaluation approach, namely Multi Criteria Decision Analysis (MCDA).MCDA can be defined as a quantitative approach to evaluating decision problems involving multiple and often conflicting variables or criteria.The approach aims to highlight the conflicts and reach compromise by following a transparent process.Many different MCDA methods have been introduced [8].A number of benefits and limitations have been noted in using the MCDA techniques [9,10]: • They allow a systematic approach to evaluating policy options, and help in understanding the problem.
• A mixture of quantitative and qualitative information can be incorporated.MCDA goes beyond the evaluation of purely economic consequences, and allows non-economic criteria to be assessed on an equal basis -i.e.MCDA techniques offer a level of flexibility and inclusiveness that purely economic models tend to lack.

•
Account can be taken of the preferences of the various stakeholder groups with conflicting objectives.
• MCDA methods do not produce the 'best' solution, but a set of preferred solutions or a general ranking of all solutions.Solving such a multi-criteria problem is a compromise, therefore, and depends on the circumstances in which the decision-aiding process is taking place.

•
There is a need for personal judgement and experience in making the decisions.
• MCDA techniques are sometimes very cumbersome and unwieldy.

•
The allocation of weights to each criterion (see below) is subjective.Changing the weights could lead to a different result, i.e. rank reversal.
Nevertheless, the value that the MCDA approach can bring to strategic triple-bottomline decision-making has been demonstrated [11,12], and the general steps that are followed, from a sustainable development assessment perspective, are summarised in Figure 1.
Figure 1 shows that the MCDA approach requires that weighting values for the assessment criteria -for example, of projects and technologies in industry -must be determined.Multi-attribute weighting methods or techniques have been introduced for this task and are well established, such as the Analytical Hierarchy Process (AHP) pair-wise comparison method [13,14,15], direct weighting [16], Simple Multi-Attribute Rating Technique (SMART) [17,18], SWING weighting [18], and TRADEOFF weighting [19].If these techniques are applied to sustainability assessment -that is, weighting values are established for the different criteria of an assessment framework -then weighting values also need to be determined for the three main dimensions of sustainable development: economic, environmental, and social.

Figure 1: The MCDA approach
It has been argued that there are no fundamental differences between the MCDA weighting techniques, since they have the same theoretical background [16].

Aggregate the results
http://sajie.journals.ac.zaHowever, the response scales of the different methods could influence the weighting values in terms of individual preferences.Therefore, comparison of the different MCDA weighting techniques may be appropriate in South Africa, from a project or technology assessment perspective.In South Africa, the direct weighting and the AHP pair-wise comparison techniques (see section 2) have been used before in the sustainability assessment context [20,21,22], and are consequently compared in this paper in the context of project and technology assessment in industry.
Furthermore, the studies done in South Africa to determine weighting values for sustainable development assessment focused exclusively on specific industries or sectors (see section 2), and do not take subjective information about the participants of the studies into account.Although the inherent subjectivity of value judgements for the weightings is a basic assumption of MCDA techniques [23], the influence of the specific personal management context of the participants has not been tested.The research question was therefore raised: Do the professional exposures and experiences of individuals, with respect to the size of the corporation where they are employed and their years of employment, influence their opinions of the relative importance of the sustainable development dimensions?This paper aims to prove two hypotheses: • The size of company where an individual is typical employed, and the respective priorities of the sustainable development dimensions to project and technology assessments, are independent; and

•
The numbers of years that an individual has been employed, and the respective priorities of the sustainable development dimensions to project and technology assessments, are independent.
In addition, following previous research in South Africa [20], the paper aims to show that the direct weighting and the AHP pair-wise comparison techniques yield similar results.

Evaluation of the sustainability of CDM projects in South Africa
The University of Pretoria, in collaboration with the Swiss Federal Institute of Technology (ETH), and with the assistance of PriceWaterhouseCoopers, undertook a survey during 2002 to determine weighting values for assessing the contribution of projects that are potentially eligible for Clean Development Mechanism (CDM) funding under the Kyoto Protocol [24], to overall sustainable development in the South African context [20,22].
Weighting values of the social, environmental and economic sub-criteria were established separately in two South African manufacturing industry sectors, as defined by the Standard Industry Classification [25]: the automobile manufacturing sector, and process industries in the automotive value chain.These manufacturing sectors are introducing sustainable development aspects in company decision-making processes, and are evaluating projects that are potentially eligible for CDM funding.The weighting values had to reflect the importance of the sub-criteria from a project management perspective in industry.Two types of industry participants were subsequently chosen to circulate a survey to those who directly control projects (and related budgets) in the specific sectors [22]: • Managing directors of South African companies in the automotive supply chain, representing first, second, and third tier suppliers.
• Financial directors of organisations or companies, primarily in the processrelated manufacturing industry sector of South Africa.
In addition, individuals representing government, the service industry, NGOs, and academia who are involved with the CDM process in South Africa were either interviewed for the survey, or participated in a workshop [20].The survey was based on the Analytical Hierarchy Process (AHP) pair-wise comparison technique, although the direct weighting method was also used and tested with one-on-one interviews.
The study initially assumed that the three dimensions of sustainable development are of equal importance.However, an analysis of the perceived importance of the subcriteria of the dimensions indicated that economic and social aspects -specifically, macroeconomic stability, employment generation, and capacity development -carry more weight than environmental aspects.This is reflected in the budgetary priorities of the South African government [22].The study concluded that separate weighting values for the main sustainable development dimensions should consequently be determined as well.Also, it was highlighted that the perceptions of other parts of South African society -e.g.government departments, non-government organisations, academia, and businesses not included in the two specific manufacturing sectors or involved with the CDM process -should be obtained.With respect to the application of the AHP pair-wise comparison and direct weighting techniques, it was found that both methods yield similar results, although the AHP techniques tend towards more extreme priorities for the criteria [20].

Evaluation of the sustainability of rural agricultural projects in South Africa
In 2003 the University of Pretoria developed a new set of project selection criteria for the evaluation of project proposals, in order to compile an effective LandCare programme portfolio for the South African government.A hierarchy of selection criteria was developed: these criteria were classified under the three dimensions of sustainable development.The AHP pair-wise comparison technique was primarily used to determine weighting factors for the selection criteria [21].
LandCare coordinators and representatives from all nine provinces of South Africa, as well as members of the LandCare secretariat, attended a workshop where the selection criteria were discussed.A total of 20 officials, all with extensive experience in the planning and implementation of LandCare projects, attended the workshop where each criterion was weighed in a hierarchical fashion to establish its perceived importance [21].
http://sajie.journals.ac.zaIn the overall system analysis, where the main criteria groups were also compared, environmental sustainability was rated the highest (58%), followed by social sustainability (23%) and economic sustainability (19%).These weighting values reflect the importance of the dimensions within the LandCare programme settingi.e. the participants were requested to make judgments based on their experiences with LandCare projects.Natural (environmental) resources are, in general, considered to be the main focus area of the LandCare programme.The study subsequently emphasises that the obtained weighting values do not reflect the importance of the sustainable development dimensions and sub-criteria to South African society in general.In other words, a bottom-up approach is required to determine the importance of sustainable development aspects in the communities where projects are to be implemented [21].

The direct weighting technique
With the direct weighting technique, participants in a study (e.g. a workshop or survey) distribute percentile weights to the criteria that are on the same level in a defined sustainability assessment framework.Figure 2 provides an example of such a framework, which has been introduced to assess the sustainability of project and technologies in industry [26].The total of the distributed numbers must therefore be 100%.Other weighting methods use similar procedures -for example [20]: • SWING: The criterion with the highest importance is assigned 100 points.All the other criteria within the level receive points relative to this most important criterion.

•
SMART: The criterion with the least importance is assigned 10 points.All the other criteria receive points relative to this least important criterion.• SMARTER: The criteria are ranked.
These direct-type weighting procedures require the participants to normalise the values themselves, since the weights they can distribute must add up to a pre-defined total.The advantage is that the participants are urged to focus on the relative importance of the criteria.An increase in the value of one criterion directly leads to a decrease in the other criteria, and trade-offs are more obvious.A disadvantage of this technique is that the inputs of the participants are not tested for inconsistencies.

The AHP pair-wise comparison technique
With the AHP pair-wise comparison weighting technique, participants in a study are required to compare criteria in a level of a defined sustainability assessment framework (e.g. Figure 2) one against the other.Hence, each criterion is weighted relative to all other criteria at the same level.Based on this pair-wise weighting approach (see Figure 3, [13,14]), the criteria contributions are determined as a priority vector.Each pair-wise comparison is rated on a 1 to 9 point scale, which is then translated into relative weights for each criterion using the matrix eigenvalue approach [27].The main advantage of the AHP pair-wise comparison technique is that participants only focus on one comparison at a time.A previous study that compared the direct weighting and AHP pair-wise comparison techniques in South Africa showed to some extent that participants had less difficulty comprehending the comparisons required by the AHP pair-wise comparison method [20].Through the pair-wise procedure, the 'level of inconsistency' is also tested for each participant.The main in that the number of comparisons increases exponentially with the number of criteria at the level of the defined sustainability assessment criteria.

RESEARCH METHODOLOGY
A survey research approach was chosen for this study.The AHP pair-wise comparison technique was used along with the direct weighting technique.The research relied on a survey consisting of different sections, namely: • Section A: General questions relating to work experience.

•
Section B: Criteria evaluation based on AHP pair wise comparisons.

•
Section C: Opinions on relevance of social criteria.

•
Section D: Direct weighting of the three dimensions of sustainable development.
Although the concept of sustainable development is understood intuitively, it remains difficult to express it in concrete operational terms [28].This might be due to the inherent vagueness in the concept itself [29].A sustainable development framework was thus introduced in the survey to ensure that all participants used the same definition of the three dimensions of sustainable development.A number of current integrated frameworks, which are used to assess sustainability at an international, national, local or company level, have been reviewed to determine the relevant aspects (or criteria) that should be considered when assessing industry sustainability.The proposed framework of appropriate criteria to assess the sustainability performances of operational initiatives in industry is shown in Figure 2 [26].The framework is divided into different levels to address the separate aspects of corporate responsibility strategy in terms of sustainability.The rationale of these levels is described in detail elsewhere [26].
The survey was distributed among postgraduate students in the Engineering Faculty at the University of Pretoria over a 12-month period during 2005.Three groups, specialising in engineering, project, and technology management, that followed a similar course in Life Cycle Engineering/Management of Safety, Health, and the Environment, were targeted outside their respective professional environments.The survey was distributed to the participants after the framework had been explained in detail.The advantage of using these postgraduate students was that they had dedicated time allocated to the survey.A disadvantage may be noted in that the students had been exposed to the relevant sustainable development subject matter prior to completing the survey, which may have influenced the value judgements to some degree.

Participant demographics and professional exposures and experiences
A total of 129 students, of whom 75.2% were male and 24.8% female, completed the survey.All the participants had an undergraduate qualification in either engineering or the natural sciences.Twenty-two percent of the participants had been involved in the execution of an Environmental Impact Assessment, while 28% had been involved in Corporate Social Responsibility Projects.Forty-seven percent of the participants indicated that their companies had a Sustainable Development Strategy, and 57% of them knew the strategy in detail.The distribution of the professional exposures and experiences of the participants is shown in

Comparison between direct weighting and AHP pair-wise weighting techniques
Table 3 summarises the results for the three dimensions of sustainable development obtained from using the direct weighting and the AHP pair-wise comparison techniques.For the direct technique, the arithmetical means of the respective participants were taken [30], while the geometric mean is used for the Aggregation of Individual Priorities (AIP), which has been suggested for the AHP technique [31].

Testing of the research hypotheses
The data was arranged in contingency tables and the guidelines for analysis of r-by-c tables [32] were used to perform hypothesis testing on the data.Two tests were performed, as set out in the introductory section of the paper.Table 4 summarises the results.Table 5 provides the statistical calculation for the direct weighting technique.
The hypotheses tests show that at a level of significance of 0.01 the obtained weighting values do not reflect the professional exposures and experiences of the participants.

CONCLUSIONS
The study ascertains that professional exposure and experience does not influence the overall weighting values that technical managers at all levels of a company assign to the three dimensions of sustainable development.This main conclusion can be ascribed to one of two reasons: • The sample of 129 participants was such that no specific industry sector or company perceptions were revealed.
• It was the individual perceptions, outside the realm of professional duties, that determined the weighting values.
When comparing the outcomes with those of previous studies in South Africa, the study highlights the importance of distinguishing between internal decision-making within industry corporations or sectors, and external decision-making, where assessments are used for public reporting.Section 2 showed that specific internal decision-making contexts prioritised the dimensions of sustainable development differently from the general responses across sectors and companies of this study.The context of weighting the three dimensions of sustainable development greatly determines individual (and societal) perceptions, and projects and developments must be evaluated as such.
The study also showed that there is no significant difference between applying the direct weighting technique and the AHP pair-wise comparison technique to determine the weights for the relative importance of criteria to use in MCDA.However, we should emphasise that the number of criteria that have to be compared might influence this conclusion.A large number of criteria may make the application of the direct weighting technique problematic for participants, and the AHP pair-wise comparison technique could show large inconsistencies.A number of specific case studies in industry are required to verify these findings further.

Table 1 : Distribution of companies' sizes Years of Work Experience Number of participants in category
Table 1 (company size) and Table 2 (years employed).

Table 3
highlights the conclusions from previous weighting studies in South Africa, i.e. that the direct and AHP pair-wise comparison weighting techniques yield similar results.Figure4shows the average direct weighting and AHP pair-wise comparison weighting values of all the participants.In this study it was found that the differences in the weighting values for the dimensions are less than 12%.