AN MULTI-CRITERIA DECISION-MAKING BASED COURSE ASSESSMENT FRAMEWORK: DEVELOPMENT AND APPLICATION

Evaluation of the educational process is essential to its development and improvement. This paper aims to create a model for evaluating students’ satisfaction with industrial engineering courses using Multi-Criteria Decision-Making (MCDM) tools. The research began with a course evaluation data survey with 28 questions (variables). Then the assessment model was proposed, based on the previously collected data. The suggested assessment model identified the constructs using exploratory factor analysis (EFA). Then the extracted variables and the resulting constructs were prioritised using the analytic hierarchy process (AHP). Next, a structured local questionnaire was drawn up and distributed to industrial engineering students at a Saudi university to validate the developed model. After that, the suggested framework was applied to the data to assess the students’ overall satisfaction with the course. Finally, it was found that the Operations Research Track, of the five current tracks, scored the highest satisfact ion rate. This study’s results could be helpful in knowing the importance of MCDM in


INTRODUCTION
The process of continuous improvement is essential to maintain a competitive position in every industry or sector. Evaluation is one of the basic parts of such a process, and should be performed periodically to enable development. Furthermore, listening to the targeted audience of any organisation, the voice of the customer (VOC) is also crucial, as it allows the organisation to identify and prioritise the low-hanging fruit and to maintain its potential leading edge; and this applies to all aspects of life. Overall, course satisfaction is essential in the educational industry to attract stakeholders and to compete efficiently and effectively. Therefore, management should continually seek to measure and evaluate overall performance. Such evaluation ensures an objective-based assessment approach to the educational process.
Course evaluation by students is not appreciated owing to a preconceived notion among students, instructors, and other stakeholders. This conflict can seriously hinder course evaluation, in that the vital element -students' satisfaction -is not generally evaluated or chosen to be part of a final evaluation [2]. Yet the importance of course evaluation is not gainsaid by any stakeholder. Course evaluation is considered to be a development-focused approach to check the validity of certain courses for a specific level. This is why it is regarded as a constructive way to measure learning outcomes for each specified group. For this purpose, multiple evaluation policies have been framed and structured to validate the effectiveness of study courses [3].
Previous studies have touted the importance of a student-centred approach, as it can lead to a focus on complex learning. Instructors often implement new techniques to ensure students' satisfaction, as it is that satisfaction that ensures the usefulness and importance of the teaching-and-learning process [4][5]. They have also used various tools to judge the effectiveness of newly adopted ways and techniques. All important decisions are based on the students' satisfaction index, whether it is to check the utility of the teachingand-learning process or to judge the value of the courses being taught [6]. Researchers have used a range of instruments for this, such as multiple attribute decision-making (MADM), multiple objective decisionmaking (MODM), and preference ranking organisation methods for enrichment evaluation (PROMETHEE) [7]. These techniques help to evaluate the thresholds of preferences and of indifference.
Multiple studies have been conducted to elucidate the importance of evaluation in any system, and how strength and growth are connected to evaluating policies [8]. Cloud computing is also one of the popular techniques that are widely applied in evaluation; but research has identified a few major problems with the results of cloud computing. A rapid increase in the use of cloud computing has also been noticed, but it has produced different classifications of different results. This results in unauthentic and inaccurate results. PROMETHEE is another widely applied evaluation technique to evaluate the performance of pairs and groups [9]. It could been considered a multi-criteria technique in which groups were observed when working on any complex problem with accurate data.
Decision-making plays a vital role in daily life [10]. Decisions depend on the weights given to sets of criteria. It thus plays an important part in engineering education. Figure 1 outlines a number of MCDM methods. AHP was applied in the current research to validate students' responses. The main objective of this paper was to create a course satisfaction assessment framework that would help the management to evaluate and improve its educational programmes. To develop a scientifically-based assessment framework, two particular tools were used in this research: exploratory factor analysis (EFA) and the analytic hierarchy process (AHP). These techniques were combined to give weight to students' evaluations and to obtain measured results to improve learning outcomes.

METHODOLOGY
This research aimed to build a scientifically-based course assessment framework. In other words, the goal was to propose an assessment framework that allows industrial engineering courses to be evaluated for overall satisfaction. Such a scientific approach would help a department to identify opportunities for improvement and to maintain a leading position among its competitors. This section ends by briefly discussing the course assessment framework's methodology.
It began by taking universally available survey data consisting of 28 questions and representing more than 5000 students' responses. Then, to identify the latent factors and the variables that represent these factors, an exploratory factor analysis (EFA) was conducted. Before the researchers applied the EFA methodology, its assumptions were tested and found to have been satisfied, based on its SPSS software outcomes. The first assumption was the adequacy of the sample size, which was checked using the Kaiser-Meyer-Olkin (KMO) test. Table 1 shows a KMO of 0.989 (more than the acceptance level of 0.7) [11]; therefore, the sample was adequate to conduct the EFA. In other words, the EFA was appropriate for the research data [12]. The sphericity assumption was tested using Bartlett's test; the result was significant, which meant that at least two variables were correlated. Thus, the EFA's assumptions were satisfied. Finally, the analytic hierarchy process (AHP) was applied to prioritise the variables and constructs. Generally, the validity of the sampling size and the questionnaire is not considered to be a critical issue; but for the current results, the population, the questionnaire, and accuracy mattered a lot for the final evaluation. Similarly, the strengths and weaknesses of the evaluation were based on the accuracy of each construct, tool, and technique being applied. Sig. .000

Exploratory factor analysis
Exploratory factor analysis (EFA) is a dimensionality reduction technique that belongs to multivariate statistical methods to identify the smallest number of the hypothetical constructs -known as latent factors, dimensions, or internal attributes -that can explain the observed covariation among a set of measured variables. In other words, it defines specific factors that explain the order and structure between the measured variables. In the social and behavioural sciences, factors are assumed to be unobservable features of people, demonstrating the variations in the scores obtained on the measured variables. Factor analysis provides methods to evaluate the nature of the interrelationships (correlations) between many variables (such as survey responses or scores) by identifying a set of strongly intercorrelated variables, known as factors. EFA was applied in this study to define the constructs on which this research work would rely [13][14][15]). Table 2 shows the two latent factors that were extracted, based on their eigenvalue (i.e., more than 1). Furthermore, parallel analysis -another technique used to select the number of factors [16] -was applied, and confirmed our decision to extract only two latent factors or constructs.  Table 3 below shows the latent factors and the variables within them. Researchers have found that variables from 1 to 12 belong to the second latent factor, while the remaining variables belong to the first latent factor. Thus, depending on the nature of these sets of questions, a brainstorming session was conducted to label both constructs. The first latent factor was called 'the instructor factor', and the second was called 'the course factor'. A reliability analysis was also conducted, and Cronbach's alpha was found to be more than 0.7 [17] for both constructs. Therefore, the two latent factors and their corresponding variables represented the hypothetical model or structure of the research study.
It should be noted that the missing values for Q1 and Q17 in Table 3 were suppressed, using an option from SPSS software that allows the suppression of values below a specific limit. In this research, the absolute value of 0.4 was used. There is no specific rule to work with; there are many theories about the limit of the absolute value below which the values are suppressed in the extraction process. For example, [18] used 0.32; in another example, [19] used 0.4.

Analytic hierarchy process (AHP)
The analytic hierarchy process (AHP) is a multi-criteria decision-making tool developed by Saaty [20] and used in several applications to handle multi-attribute decision-making problems in real situations. It is a robust technique for defining the problems in detail, and also for solving technical, managerial, and unstructured problems that might have interactions and correlations among different objectives and goals. The AHP helps decision-makers to organise the critical aspects of a problem into a hierarchical structure similar to a family tree; it connects all the levels of the hierarchy, which helps to clarify how changing one criterion might affect other alternatives and criteria; and it starts with pairwise comparison matrices and is based on experts' judgements [21][22][23].
After the two latent factors had been identified and labelled, the AHP was applied within and between both constructs, following three steps. The first step was to build the pairwise comparison matrices using Saaty's scales of relative importance [24]. The scales are proper when the individual preferences are expressed on verbal scales, which is the benefit of using scales of relative importance to convert them into meaningful numerical values. This paper used the scale as an initial step to determine the relative importance of different criteria.  In order to build the pairwise comparison matrices, the researchers conducted a brainstorming session. The second step was to normalise the pairwise comparison matrix to obtain the criteria weights (CW) by dividing each cell in the pairwise comparison matrix by the sum of its column. Then the CWs were calculated by finding the average of each row in the normalised pairwise comparison matrix. The third step was to calculate the consistency ratio (CR). Several calculations were performed using the original non-normalised pairwise comparison matrix. This process began by multiplying each cell in each question by its corresponding CW, then calculating the weighted sum value (WSV) by finding the sum of each row in the matrix. Then each ratio (R) was calculated by dividing each row's WSV by its associated CW. After that, the lambda max ( ) was calculated by finding the average of all the ratios, and the consistency index (CI) was calculated using the following formula: where n is the number of compared elements in the matrix. Finally, the CR was calculated by dividing the CI by the random index (RI) that is suggested for more than fifteen variables [25]. Fortunately, the calculated CRs were less than 10% [26], [27], which meant that the CWs obtained from the normalised pairwise comparison matrices were reliable.
The fuzzy analytic hierarchy process (FAHP), which is similar to AHP, but has been improved by fuzzy logic theory and uses the fuzzy triangle scale [28], was also applied to confirm the results of the AHP and to see whether any different criteria weights might appear. The results were almost the same, and so the researchers decided to continue their work and use the AHP models. Figures 2 and 3 illustrate the prioritisations of both latent factor variables.  Although the techniques applied in this research were very limited, it was based on the authenticity of the data, the questionnaire, the design of the questions, the population, the sample size, and each chosen variable. This research produced limited but accurate results with the quality of the data. This is the reason that the accuracy of the results depends on the accuracy of the data-driven tools and the population.
Once the suggested framework that measured students' satisfaction with courses was ready, a locally based survey, similar to the original one with the addition of an overall course satisfaction question, was designed and distributed, using Google Forms, to the industrial engineering department students at King Abdulaziz University (KAU-IE), from whom 107 responses were collected. After that, the researchers calculated the predicted satisfaction for each industrial engineering track and its courses before comparing them with the ones they had observed, based on the overall course satisfaction feedback.
The collected responses to the 28 main questions were then multiplied by their corresponding weights; and, since the 28 main questions were categorised into two latent factors, the sum of each construct's resulted weight was multiplied by its overall priority. An illustrative example is given below: {((5*0.02) + (5*0.02) + (5*0.06) + (4*0.02) + (3*0.07) + (4*0.04) + (3*0.22) + (5*0.06) + (5*0.08) + (5*0.01) + (4*0.2) + (5*0.2)) * (0.17)} + {((5*0.03) + (3*0.08) + (3*0.02) + (4*0.04) + (5*0.01) + (5*0.12) + (4*0.07) + (4*0.1) + (5*0.05) + (4*0.04) + (4*0.02) + (4*0.08) + (4*0.06) + (4*0.04) + (4*0.06) + (5*0.18)) * (0.83)} = 4 Next, the observed and predicted satisfaction scores were calculated for each course and track; as a statistic, the median was selected owing to its insensitivity to extreme value. In order to calculate the prediction accuracy, the matching responses were counted and then divided by the number of the responses. An illustrative example, from the operations research track, is given here: Number of all responses in track = 33 Number of matching responses (the observed satisfaction matched the predicted satisfaction) = 19 Accuracy = (19 / 33) * 100 = 58% In this research, a five-point Likert scale survey was used. Given the relatively small sample size, and to enhance the accuracy of the results, the responses were coded. In other words, to minimise the gap between the observed and the predicted satisfaction, the observed satisfaction was converted into coded satisfaction, such that responses 1 and 2 were coded as 1; response 3 was coded as 2; and responses 4 and 5 were coded as 3. Table 4 shows the accuracy of the prediction before and after coding the results. The prediction accuracy improved significantly, except for one track, owing to its limited number of responses. Furthermore, the proposed assessment framework showed a better and more reliable performance with the coded responses.  [29] highlighted the importance of students in achieving the best outcomes in the teaching-and-learning process. If students were evaluated as important stakeholders, this could help to shape the future plans of the higher-education institutions. There are many engineering programmes that exist in the same socio-cultural context; this study could be relevant in giving them an outlet. Although the findings of the current research were specific and limited, they could still be used to develop or improve the specifications of any engineering or technical educational programme.

CONCLUSION
This research began with an international student questionnaire with 28 questions that produced more than 5000 responses. The exploratory factor analysis (EFA) technique was applied to identify the latent factors and to aggregate the different variables under each construct. The eigenvalues and a parallel analysis were considered to identify the number of extracted latent factors. Two latent factors were identified: the first was called 'the Instructor factor', with 16 variables associated with it, and the second was called 'the course factor', with 12 variables. The analytic hierarchy process (AHP) was then applied to prioritise constructs and their associated variables. The fuzzy analytic hierarchy process (FAHP) was also applied, and the results were almost the same. Thus, a course assessment framework was proposed based on the resulting constructs and the calculated priorities of their associated variables.
A questionnaire on industrial engineering students' satisfaction with the course was distributed to validate our suggested assessment framework. The locally distributed questionnaire contained the same 28 questions; however, the overall course satisfaction questions and a few other general questions were added. After that, the model calculated or predicted satisfaction compared with the observed satisfaction of all of the participants, by track level and course level. The overall accuracy of the assessment framework was very satisfactory. It was also found that the Operations Research track had the highest satisfaction rate of the five KAU-IE tracks. The students' involvement in evaluating the five tracks ensured the sustainability of those course codes. MCDM was applied to obtain the calculated and measured results for a strategy that would be sustainable in the future. The MCDM technique was helpful in the current research in obtaining rational results, and so could improve the quality of the courses and of education in general.
The researchers recommend that, in future work, a locally based assessment model be constructed, based on local data only, before comparing it with the internationally based assessment model proposed in this paper. Furthermore, it is recommended that other multi-criteria decision-making (MCDM) methods and tools be applied, such as the technique for order preferences by similarity to ideal solutions (TOPSIS), simple additive weighting (SAW), and ELECTRE [30]. Finally, a fully automated system based on the proposed assessment framework might help to expedite the evaluation process