TOWARDS A FRAMEWORK TO GUIDE THE EVALUATION OF INCLUSIVE INNOVATION SYSTEMS

Innovation for inclusive development aims not only to produce products and services for ‘base of the pyramid’ (BoP) markets, but also to improve innovative capacity and empower those that form part of the BoP. Although a wide range of actors aim to support innovation for inclusive development, they lack proper methods to assess the impact that these projects have. This paper sets out to identify an appropriate method or approach for evaluating systems change brought about by innovation for inclusive development projects. Based on a systematic literature review, a process-level approach is proposed. This approach is focused on the components within a system and the changes that occur in the system functions.


INTRODUCTION AND PROBLEM STATEMENT
Hammond & Prahalad [1] defined the four billion poorest people who earn $1 or $2 per day as the 'base of the pyramid' (BoP) population.Alleviating poverty is a challenge that various organisations and institutions continuously try to address.Innovation is seen as a key driver of economic growth -it induces the development of products and services for consumers and increases industrial economic development.Although economic growth has played a vital role in decreasing absolute poverty, there are growing concerns that innovation has also contributed to inequality and exclusion.Although a thriving body of knowledge is developing, the economic periphery such as informal microenterprises, affordable goods and services, and innovation to solve wider social challenges, has traditionally been neglected in innovation policy thinking and practice [2].
Several authors have stated that alternative or non-traditional innovation models could be used to stimulate economic inclusion and growth for those in the BoP [2], [3].The literature highlights two main types of innovation models that can be applied towards these goals.The first type are models where products and services are specifically developed for the benefit of lower income markets.
Here the BoP is viewed as a potential customer, and the goal is mainly to decrease costs and produce more affordable products and services.These innovation modes are often referred to as 'appropriate technology' [4], 'frugal innovation', 'pro-poor innovation', and 'BoP innovation' [5].The second type of innovation model takes a more developmental approach.Examples of this type include 'inclusive innovation', 'innovation for inclusive growth' [6], and 'innovation for inclusive development' [7].These are terms used to describe innovations that "create or enhance opportunities to improve the wellbeing of those at the BoP" [6].Here the aim is that BoP individuals be regarded not only as potential customers but also as business partners who are included in the innovation process, so that, as solutions are conceptualised and developed and goods are manufactured, they can benefit economically from such innovations.For the purpose of this paper, the authors use the term 'innovation for inclusive development'.
Linear (predominantly black-box) evaluation methods fail to provide insights into critical institutional lessons about how to improve innovation as a process [8].Biggs and Underwood [9] identified the need to challenge the current methods for monitoring direct poverty impacts at the project/programme level.They argue that a more holistic approach than that of the status quo can be achieved through the use of the innovation systems (IS) and inclusive innovation systems (IIS) approaches, as these shift the emphasis from a framework of problem-solving to one of learning [9].Inclusive innovation systems are made up of interconnected components that work together to develop and diffuse innovations aimed at resource-poor individuals or groups, in order to make a positive contribution to improved life conditions and upward mobility [6], [10].
An initial review of the multidisciplinary literature on IIS revealed a lack of methods or approaches to evaluate IIS.[11].Several authors have emphasised the potential of innovation system concepts for understanding innovation for marginalised communities [12], but as yet there has been little actual primary or field data-based application of this idea [11].Very often, IIS stakeholders are interested in the impact of innovations for inclusive development.Such impacts need to be determined by means of 'deeper and/or longer-term' evaluation approaches [11], yet the literature lacks both tools for and examples of such evaluations.There is thus a need for more studies to develop such evaluation approaches and to apply such approaches to empirical cases.
Based on these findings from the literature, the IS and IIS approaches were used as a lens through which to view the evaluation of innovation for inclusive development projects.Foster and Heeks [11] argue that the structural representation of IIS needs to be supplemented by a process representation.Such a process level approach to the assessment of the outcomes and impact of inclusive innovation enables the identification of measures that can track institutional changes.This process representation opens the 'black box' of innovation -i.e., looking at inclusions in the innovation process where the inclusive innovation process can be conceptualised to include the following stages: invention, design, development, production, distribution, adoption, and use [13].
In agreement with Heeks et al. [13], we propose that, by opening the 'black box', it is possible to develop a list of impact classification measures that are applicable on both the project-and the systems-level.Using a process focused on both the components within a system and the changes that occur in the system functions will enable the translation of these impacts back to a systems view.We therefore decided to investigate the most appropriate approach by which to open the 'black box' of processes within IIS, thereby paving the way for future empirical studies to develop frameworks for the evaluation of IIS.To do so, the key aim of this systematic review is to review the extant literature to identify methods or approaches that could be used to evaluate IIS.

METHODOLOGY
In order to avoid creating an ad hoc list of randomly-collected publications, we conducted a systematic literature review.This section describes the methodology followed.The review process followed the established three-phase process used by authors such as Kitchenam and Charters [14] and Fink [15].The three main stages are planning, conducting the review, and reporting the review.The authors used the first two stages of the methodology, but added a different third stage: assessment of the evaluation categories.Table 1 describes the methodology followed in this paper.

Stage 1: Planning the review
This section focuses on Stage 1 of the methodology, and is structured as follows: (1) Step 1.1 describes the problem statement and rationale for the review; (2) Step 1.2 describes specific search terms used and the databases searched; and (3) Step 1.3 provides an overview of the pre-defined analytical framework according to which the methods or approaches will be evaluated.Construct criteria according to which the methods/approaches identified from the review will be evaluated.
The criteria according to which the functionality of each approach was evaluated are described.The second part of this step describes how each approach was evaluated in terms of its applicability to be used for the IIS.Identify studies that describe methods/ approaches to evaluate IS /IIS from the literature.
The total yield of the combined searches, using the databases and search terms, and after duplicates had been removed, was 247 references.2.2.Selection of primary studies.
Read through the identified studies in order to gain better understanding and start categorising the studies.
117 studies were identified after non-relevant studies had been excluded.

Data extraction, monitoring and synthesis.
Identify as many methods/approaches as possible from the studies that were included.
The studies were read in order to identify methods/approaches.The different methods/ approaches were coded.The coding process was repeated iteratively as the reviewer started recognising similarities between approaches and could integrate them.This lead to the identification of nine main methods/approaches.

Stage 3: Assessment of evaluation categories
3.1.Evaluate each of the evaluation categories identified from the review.
Evaluate each approach in order to identify the most suitable approach.
Each of the evaluation approaches is evaluated according to the criteria established in Step 1.3.

Selection of the most suitable method or approach.
Select the most suitable method or approach based on the results of step 3.1.
The results of Step 3.1 are analysed, and the most appropriate method or approach is identified.

Step 1.1: Rationale for our study
An initial review of the IIS literature revealed a lack of methods or approaches to evaluate IIS systematically.There were also very few studies that aimed to determine the success factors and understanding of IIS [11].In order to address this gap in the literature, the key aim of this article is to review the extant literature in related fields to identify methods or approaches that could be adapted to evaluate the impact of IIS.

Step 1.2: Specify the review
As previously mentioned, we aimed to identify an appropriate method or approach to guide the evaluation of IIS.The review research questions were therefore: 1. Which methods or approaches are used to evaluate IS and/or IIS? 2. How, and in which context, have these methods or approaches been applied?
Step 1.2 also describes the search strategy used to identify primary studies, including the resources and databases searched and the specific search terms that were used.Digital bibliographic databases, reference lists, and conference proceedings were systematically searched for relevant documents.
Because the literature yields very few available studies of the evaluation of IIS, it was decided to cast a wider net and look at the evaluation of traditional IS as well.Studies were identified through searches of the Web of Science, Scopus, Academic Search Premier, Emerald, Science Direct, and ProQuest Research Library databases.Title, abstract, and keyword information was searched using predefined search terms.A list of search terms was constructed, based on the knowledge gained from the initial literature and the field knowledge of the review team.The terms fall into three broad categories: approaches or methods, innovation systems, and the purpose of the paper.These three categories were specifically chosen to ensure that the studies included in the review described the approach or method used to analyse or evaluate the innovation system in question.To ensure that multi-disciplinary studies were included, results were not filtered by discipline.The main search terms are shown in Table 2.Both peer-reviewed and grey literature were included.Only articles in English were considered.

Purpose
Evaluation* Analys* Title 1 The asterisk (*) functions as a wildcard that represents any group of characters, including no characters.

Step 1.3: Analytical framework to analyse methods identified from the review
The evaluation of innovation systems focuses on how well the system functions rather than on the performance of the system.This implies that system evaluations should emphasise the assessment of knowledge creation, the functions (processes) within the system, and the transparency of the system [16].
Rose and Winter [102] emphasise that there is no 'optimal state' according to which innovation systems can be compared, since each is dependent on the context in which it operates.Thus it is important to identify the areas in which the system fails (gap-analysis).Once the area that hinders innovative performance has been identified, the analysis approach can be specified.In order to evaluate IS/IIS, an integrated analysis must be performed that can provide insight into a system's innovation capacity and functioning.To compare and draw insight from the different methods or approaches in the literature, we identified two sets of criteria against which the methods were analysed.The first set assesses each method or approach according to its functionality as an evaluation methodology.These criteria were based on those defined by Schut et al. [8] and by various other authors from the review.To summarise, these criteria are: 1. Methods or approaches should produce mixed data -both qualitative and quantitative.Qualitative data may provide insights into implicit causes and the historical development of constraints to innovation capacity.It also enables the identification of the different dimensions of complex IS problems and structural conditions that induce or block innovative capacity.
Quantitative data analysis complements qualitative data by providing descriptive trends and statistical data [8].2. Methods or approaches should enable both internal and external analysis.Internal analysis is conducted by stakeholders, but is insufficient on its own, given the lack of a broader view.So it should be accompanied by an external analysis conducted by independent researchers [18].3. Methods or approaches should be applicable to various actors across different levels [8]. 4. Methods or approaches should enable an integrated analysis of complex problems, innovative capacity, and the functioning of the system in question [19]. 5. Methods or approaches must enable the identification of system failures or 'gaps' in order to understand which part of the system is dysfunctional or hinders effective innovation [17].
The second set of criteria describes the requirements that a method or approach must meet in order to be suitable for the evaluation specifically of IIS.Traditional views of innovation regard development as generalised economic growth.Inclusive innovation, however, "explicitly conceives development in terms of active inclusion of those who are excluded from the mainstream of development" [20].Differing in its foundational view of development, inclusive innovation therefore refers to the inclusion, within some aspect of innovation, of groups who are currently marginalised [2].Given these definitions, we deduced that a method or approach to evaluate the impact of IIS will have to be actor-orientated and focus on the complex relationships between the actors that contribute to the emergence of inclusive innovations.Van der Hilst [16] noted that, for the purpose of identifying opportunities for improved interventions, an approach should be used that enables rich qualitative data to be gathered and retained.Several authors have noted that IIS aims constantly to develop and diffuse knowledge and skills; so an evaluation method should be able to analyse processes or functions within the system [21], [22].Finally, although some empirical studies evaluate IS, these are focused on empirical results from developed countries [23].It remains uncertain, therefore, how suitable these approaches are for the evaluation of IIS of less-developed countries.Thus examples were sought where each approach was applied to the IS of a less-developed country whose demographics include people who can be regarded as part of the BoP population.For the purpose of this study, the criteria that a method or approach must meet in order to be suitable for the evaluation of IIS are summarised below: 1. Be actor-orientated -i.e., the method must be able to identify actors within the systems, their roles, and 'who' from the excluded group is to be included.IIS consists of demand-side actors, supply-side actors, and a rich network of innovation intermediaries; and the method must enable the identification of each group [13].2. Focus on the complex relations between the actors -i.e., the method or approach must be able to identify the "informal, loose but socialised relations" that are present in IIS [11].It must focus on how these relationships foster learning and capability transfer, diffusion, and retention to the BoP [13], [24].3. The method or approach must be able to take the context of the system into consideration -it therefore must be used in such a way that rich qualitative data is gathered and retained [16].4. The method or approach must also be able to analyse the processes within the system [21], [22].

Stage 2: Conducting the review
The 117 studies included in our review were individually coded by the authors.This was done in order to standardise the information contained in the studies and to help to identify trends in the literature.The main aspects of the articles that were coded were: 1.The type of paper and the publication type.This was extracted in order to identify the target audience of the study in question, and to gain broad insights into the dimensions of the studies.2. The method or approach used in the article.As previously mentioned, the main objective of this review was to identify methods and approaches used to analyse or evaluate innovation systems.3. The type of innovation system that was analysed or evaluated, and its context or discipline.
This was extracted in order to connect the method or approach used with the type of innovation system to which it was applied, in order to identify patterns.4. The goal of the analysis or evaluation.This again served to provide insights into the types of methods or approaches that are used for specific goals.

Step 2.1 and 2.2: Identification of research and selection of primary studies
The first part of Step 2.1 begins with descriptive statistics of the 117 studies reviewed, including the type of publications in which the studies were published and the chronology of the studies.The second part identifies the main methods or approaches used in the studies.
The total yield of the combined searches, using the databases and search terms described in Stage 1, once duplicates had been removed, was 247 publications.Publications were discarded if they were written in a foreign language.In Step 2.2 the authors reviewed the titles and abstracts, and discarded publications that either were not related to the evaluation of IS/IIS or just mentioned it in passing.This resulted in 117 studies.Table 3 below shows the different types of publications from which the 117 included studies were collected.Figure 1 is a timeline that shows the number of studies (included in this review) that were published per year, in chronological order.From this figure it is clear that there has been an increase in studies on IS since about 2004, and it also depicts the relatively small number of emerging studies in the area of IIS.

Step 2.3: Categories of evaluation methods
The 117 papers were analysed with the use of Atlas.tisoftware.This enabled the authors to code different methods or approaches.The first iteration served as a scoping opportunity.The identification of methods or approaches was an iterative process that was conducted in a 'learningby-doing' manner.It became evident that many approaches are based on the same fundamental characteristics.All the methodologies described in the review were categorised into nine main methods or approaches.These methods or approaches are briefly described below: (1) Macro-economic level approach (Comparative approach): The innovation system performance of geographically different innovation systems is compared in order to guide policy development.It considers the stages of development and the evolution of the national innovation systems, and distinguishes between policies as either supply-side or demand-side in order to create technological dynamism within the country by means of a comparative perspective between two or more countries.It is a comparative framework used to evaluate innovation policy.The framework starts by identifying activities across sectors where a country could build comparative advantages.Next, the innovation chains consisting of both technical and economic interfaces are identified.This evaluation is then used to generate new 'innovation policy' that aims to strengthen the technoeconomic network as a whole, rather than supporting specific activities [25].
(2) Regime (macro, meso, micro) approach: This approach divides the NIS into three levels: macro, micro, and meso.The IS actors and functions are categorised into these levels, and evaluated based on the requirements of each level [26], [27].
(3) Triple-helix model: This model focuses on three main actors within the IS: academy, industry, and government [89].(4) Systems dynamics (SD) approach: System dynamics aims to describe the system through both qualitative and quantitative models.This method entails the construction of causal loop diagrams that describe the causal assumptions of the system in question, and often also include quantitative modelling through a stock and low diagrams.The key concept behind these diagrams is to conceptualise complex behaviours in systems and to model the expected behaviour of systems due to their non-linear nature.These models capture feedback loops and causal thinking, and are hugely useful in improving understanding of the nonlinear behaviours of such systems [28].
(5) Component-based (structural) approach: The structural approach entails the identification of system elements such as all the actors, institutions and organisations within the system, and understanding the relations between these elements [50].
(6) Function-based approach (also called TIS approach): The functional approach was identified as one of those most often used in the literature that forms part of this review.This approach involves the analysis of the system by means of certain activities commonly referred to as 'functions'.Different authors orientate the approach to different focuses, and integrate it with other approaches [21], [29].
(7) Component-function-based approach: This approach is an integration of the component-based and functional approaches.Lamprinopoulou et al. [30] note the argument in the literature that, since structures and functions are mutually dependent, it is better to follow an integrated approach where the component-based and functional approaches are used together.
(8) Data envelopment analysis (DEA): It is one of the most mature efficiency evaluations.This method uses mathematical techniques that can deploy a range of variables and constraints to evaluate the relative efficiency of decision-making units (DMUs).The advantage of the DEA method is that the analysis of the effect of an innovation system on the system outputs can provide managers with useful decision information; and DEA is very objective [31].
(9) Analytical hierarchy process (AHP) method: This method first entails the construction of evaluation indexes for the overall goals and sub-objectives of a specific programme or system, and then it is used to determine the weight of these indexes.This method is an effective way of simplifying complexity in order to determine the relative importance of the objectives of each process within a system [32].. Table 4 shows which studies applied each of the above-mentioned approaches; and Figure 2 shows the frequency of studies that applied each method or approach.

Stage 3: Criteria assessment of evaluation categories
To compare and gain insight into the different methods or approaches in the literature, we identified two sets of criteria according to which the methods were analysed.This section comprises the analysis of the main methods or approaches identified from the literature.A framework for analysing the methods was defined in Step 1.3 of Stage 1.

Step 3.1: Evaluation of each of the identified methods or approaches
Appreciating that no single optimal evaluation method applies to all evaluations, the authors constructed a database of different approaches, the context in which they were applied, and the purpose of each.To be able to compare the functioning and the applicability of each of the methods or approaches to the current study, each was assessed based on whether it meets the criteria for effective evaluation, mentioned in Section 3. The result of this evaluation, and the applicability of each method or approach to the evaluation of IIS, is shown in Table 5, with examples of the application.

Step 3.2: Selection of the component-function-based approach as the most appropriate approach.
The analysis in Table 5 revealed that the approaches meeting most of the pre-defined criteria are the functional approach and the component-function-based approach.Each of these is applicable to various actors, and both use functions of innovation to analyse complex problems, innovative capacity, and the functioning of a system.Both identify system failures by identifying problems that hamper the effective functioning of the system -i.e., that hamper innovation in the system under study.The analysis in Table 5 was also used to determine which of these two approaches would be most applicable to the evaluation of IIS.As previously mentioned, we determined that such an approach should (1) be actor-oriented, (2) analyse complex relationships between actors; (3) use rich qualitative data; and (4) conduct a process-focused analysis.
The study conducted by Van der Hilst [16] suggests that, if the functions are slightly altered, they can apply to the evaluation of IIS.The component-function-based approach meets all four requirements.The combination of the component-and function-based approach ensures that it is actor-orientated and seeks to identify complex relations between actors.The approach does require rich qualitative data to be gathered in order to identify how well system functions are performed.The system processes are analysed by means of system functions.Finally, the review identified a study in which the component-function-based approach was successfully used to analyse an IIS.Van der Hilst [16] developed a tool based on the component-function-based approach to determine the performance of innovation intermediaries within the agricultural IIS of Vietnam.It was developed by altering the method proposed by Wieczorek and Hekkert [114] to be suitable for the evaluation of IIS.
Adopting the component-function-based approach described by Van der Hilst [16], in combination with a process view of innovation suggested by Heeks et al. [11], we have constructed the broad outlines of a framework to guide evaluation.This is an advantageous combination, as the process  Process-orientated approach [11] Inclusion in innovation process:

Define system boundaries
A particular IIS is defined around a single product/service or a sector/sub-sector or a group performing a particular function.

Identify key informants
Key informants need to be identified, as they serve as the primary source of information.

Perform component-functionbased analysis
Use innovation system components as a guide to help us consider actors who are present, and their capabilities.Define in which phase of the innovation process 'inclusion' occurred (invention, design, development, production, distribution, adoption, use).
Use the functions approach, with a specific focus on how this helps actors to engage in the IIS and with other.These are in essence the causes of the outputs and outcomes.Here we open the 'black box' to look at the processes within the innovation system, and to determine the outcomes and outputs generated through system functions.

CONCLUSIONS AND RECCOMMENDATIONS FOR FUTURE STUDIES
The goal of this article was to identify a method or approach that could serve as a basis on which to develop a framework to evaluate IIS.To that end the authors conducted a review of the extant IS evaluation literature.This review identified nine methods that are most frequently used throughout the literature.Each method was evaluated according to pre-defined criteria.The results described in Step 3.2 show that the component-function-based approach is the most appropriate one from which to develop an evaluation framework.It is an actor-oriented approach that enables the identification of the different actors within the system -demand-side, supply-side, and intermediaries.It focuses on the complex relations between actors in order to evaluate knowledge and capability transfer.The combined structural and functional approach is best for the evaluation of IIS, as the functions of a system can only be improved by altering one or more components.Every function is evaluated from the perspective of the components [16].
In the course of this review we have found that more research is needed into aspects of the component-function-based approach, such as indicators, how to assess functionality, and impact classification measures.We propose that further studies apply a systematic learning process in order to improve our understanding of the practical application of such an approach.The componentfunction-based approach needs to be applied using primary data in order to start revising this approach and constructing practical evaluation frameworks.This could help unpack the componentfunction-based approach in order to develop a list of impact classification measures that are applicable on both the project-and the systems-level.Previous research shows that, by opening the 'black box' -i.e., looking at inclusions in the process -it is possible to develop such a list [11].For this we have begun to move towards the construction of a broad framework for understanding and evaluating innovation for inclusive development projects (Figure 3).
As is the case with any study, this paper has limitations.Although the predefined systematic literature review methodology reduces bias in the review process, it only provides specific answers to specific questions [116], [117].The review only included studies that are written in English, thus excluding several studies.The search was also restricted to studies that specifically refer to the search terms listed in Table 2.While this approach was consistent with the goal of identifying a suitable method or approach, it also excluded several research streams that attempt to evaluate IIS.

Figure 1 :
Figure 1: Number of IS and IIS Studies view provides structure (invention, design, development, production, distribution, adoption, and use), and the component-function-based approach enables the evaluation of the functioning of the process.The broad outlines for constructing such a framework are shown in Figure3.

Figure 3 :
Figure 3: Towards a framework to guide the evaluation of IIS