EUROPEAN COMMISSION

Brussels, 29.1.2024

SWD(2024) 29 final

COMMISSION STAFF WORKING DOCUMENT

EVALUATION

Accompanying the document

REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL

Ex-post evaluation of Horizon 2020, the EU Framework Programme for Research and Innovation

{COM(2024) 49 final} - {SEC(2024) 52 final} - {SWD(2024) 30 final}

Contents

1.    Introduction: purpose and scope of the evaluation    

2.    What were the expected outcomes of Horizon 2020?    

2.1    Horizon 2020 and its objectives    

2.2    Points of comparison    

3.    How has the situation evolved during the evaluation period?    

Funding allocation    

4.    Evaluation findings    

4.1    To what extent was Horizon 2020 successful and why?    

4.1.1. Effectiveness: Scientific impacts – To what extent has Horizon 2020 strengthened Europe’s scientific base?    

Strengthening frontier research: Publications, quality of research and scientific advancements    

Scientific breakthroughs and advancements    

Investing in future emerging technologies to accelerate deployment    

Enhancing researchers’ skills, mobility and career development    

Strengthening Research Infrastructures    

Spreading excellence and widening participation    

4.1.2. Effectiveness: ‘Societal impacts’ – To what extent has Horizon 2020 increased the R&I contribution to Societal Challenges?    

Pursuing research and innovation to contribute to Societal Challenges    

Contribution to the Sustainable Development Goals (SDGs)    

The Joint Research Centre’ direct research actions – Science for policy    

Promotion of gender equality in Horizon 2020    

How did international cooperation contribute to the impacts of the programme?    

4.1.3. Effectiveness: Economic impacts – To what extent has Horizon 2020 boosted Europe’s leadership in enabling and industrial technologies and competitiveness?    

Horizon 2020’s innovation outputs    

Facilitating access to risk capital    

Improving Europe’s economic growth and competitiveness    

4.1.4. Dissemination and exploitation of results    

4.1.5. Analysis of the long-term impact of previous framework programmes    

4.2    Efficiency    

4.2.1 Costs, affected stakeholder groups, and overall value-for-money of Horizon 2020    

4.2.2 Performance of Horizon 2020’s simplification measures    

4.2.3 Potential areas for further simplification    

4.3    Coherence    

4.3.1. Internal coherence    

4.3.2. External coherence    

4.4    EU added value    

4.4.1. Horizon 2020 leveraged additional resources for R&I    

4.4.2. Horizon 2020-supported activities that would not have been possible without EU funding    

4.4.3. Horizon 2020 promoted multidisciplinary and European cooperation in R&I    

4.4.4. Horizon 2020 increased excellence in research and innovation, by creating EU-wide competition    

4.4.5. Horizon 2020 helped consolidate the European Research Area    

4.5    Relevance    

5.    What are the conclusions and lessons learned?    

5.1    Conclusions    

5.2    Lessons learned    

Annexes:

·Annex 1 – Procedural information

·Annex 2 – Methodology and analytical models used

·Annex 3 – Evaluation matrix and, where relevant, details on answers to the evaluation questions (by criterion)

·Annex 4 – Overview of benefits and costs

·Annex 5 – Stakeholder consultation: Synopsis report

·Annex 6 – Additional data on Horizon 2020 State of Play

List of Figures

Figure 1: Horizon 2020’s structure    

Figure 2: Horizon 2020 intervention logic    

Figure 3: Applications by country group    

Figure 4: Share of EU funding to Horizon 2020 newcomers by type of action    

Figure 5: Funding allocation by type of action    

Figure 6: Participants and funds by country group in Horizon 2020    

Figure 7: Horizon 2020 key publication metric comparison with other funders    

Figure 8: Horizon 2020 funding to new and emerging areas    

Figure 9: GDP gains linked to Horizon 2020    

Figure 10: The impact of Horizon 2020 on employment    

List of Tables

Table 1: Cross-cutting key performance indicators (KPIs) on SME participation in Horizon 2020    

Table 2: Horizon 2020 KPIs on scientific impact – KPIs, 1, 2, 14, 22, and 23    

Table 3: Number of joint public-private publications, KPI 8 and KPI 17    

Table 4: KPI 4 on the number of researchers undertaking cross-sector and cross-country mobility, including PhD candidates    

Table 5: KPI 5 – number of researchers who have access to research (e-)infrastructures through Union support    

Table 6: KPI 20 on SEWP: evolution of peer-reviewed publications in high-impact journals (ERA Chairs and Twinning activities)    

Table 7: KPIs 14, 15 – Number of publications and patents in the areas of different Societal Challenges    

Table 8: KPI 19 on % of the overall Energy Societal Challenge funds allocated to renewable energy, end user energy efficiency, smart grids and energy storage activities    

Table 9: Women participants across the framework programmes    

Table 10: International cooperation in collaborative projects    

Table 11: KPI 3, KPI 6 and KPI 15 on patent applications    

Table 12: Core EIT KICs key performance indicator totals across the period 2010*-2020 (*=or starting year of the respective KIC)    

Table 13: Cross-cutting key performance indicators (KPIs) on SME participation in Horizon 2020    

Table 14 : KPIs 9-10 on total investments mobilised via Horizon 2020’s equity and debt facilities (InnovFin)    

Table 15: KPI 11 on the number of organisations funded – entities supported by Horizon 2020’s equity and debt facilities (InnovFin)    

Table 16: Total members’ contribution targets for JUs, as per the founding Regulation and legal decisions, and actual contributions, as of 31 December 2021 (2014-20, in EUR million)    

Table 17: KPI 21 on the number of institutional change actions promoted by the programme    

Abbreviations

Glossary

Horizon 2020 is the eighth EU framework programme for research and innovation, set up by Regulation 1291/2013. 2 Following the adoption of the regulation on the European Fund for Strategic Investments 3 , the total budget of Horizon 2020 was set at EUR 75 623 million in current prices. This ex post evaluation of the programme was required to be carried out by 31 December 2023. 4  

The scope of this evaluation is Horizon 2020, including the activities of the European Institute of Innovation and Technology 5 and the direct, non-nuclear, research activities of the Joint Research Centre. 6  It covers activities carried out or begun between 2014 and 2020 in the EU constituted of 28 Member States for the period of 2014-2020 (including the UK 7 ) but also in a set of non-EU countries participating to the programme, including those associated to Horizon 2020. 8 An interim evaluation of Horizon 2020 was published in 2017. 9

The evaluation covers the whole programme Horizon 2020. It addresses in more detail programme parts when they are particularly relevant for reaching one of the programme objectives, for instance the Marie Skłodowska-Curie Actions (programme part), for providing training, career development and transnational mobility of researchers (programme’s specific objective) or InnovFin (programme part) for enhancing access to risk finance in the EU (programme’s specific objective).

Exceptions are seven public-private partnerships 10 (implemented through Joint Undertakings established under Article 187 of the Treaty on the Functioning of the European Union (TFEU)) funded under Horizon 2020 which have legal successors in Horizon Europe. Whereas this evaluation explains how Joint Undertakings contributed to Horizon 2020 and the additional funds they leveraged, they will be fully evaluated as part of the interim evaluation of Horizon Europe according to regulation 2021/2085, which repeals and replaces previous relevant individual regulations of Joint Undertakings 11 . This evaluation also refers to the contribution of three public-public partnerships established under Article 185 of TFEU and already evaluated in 2022 12 : the Partnership for Research & Innovation in the Mediterranean Area (PRIMA), Active and Assisted Living Research and Eurostars 2. 

Data from the seventh framework programme (FP7), that preceded Horizon 2020, are used to assess the long-term impact of the programme, in accordance with the relevant Council conclusions. 13  

All five compulsory evaluation criteria (i.e. effectiveness, efficiency, relevance, coherence, and the European added value of Horizon 2020) are assessed in this evaluation. This evaluation is informed by a set of 12 external studies 14 , Commission monitoring reports, studies and reports issued by other European institutions (the European Court of Auditors, the European Economic and Social Committee, the European Parliament). The evaluation methods used include: (i) a review of documentation and analytical data; (ii) text analysis; (iii) more than 1,000 interviews with beneficiaries, national authorities and implementing bodies; (iv) a survey of successful and unsuccessful applicants 15 ; (v) counterfactual analysis; (vi) a stakeholder consultation which ran from 1 December 2022 to 23 February 2023 gathering 1 818 replies. This mix of qualitative and quantitative methods provided a comprehensive evidence base. 

Limitations in the analysis are due to the sizable share of projects that were still ongoing at the time of preparing this final evaluation: only 21 030 (59%) of the 35 426 projects signed had finished (59%), while 14 396 projects (41%) are still ongoing, in all programme parts and in all three pillars 16 . In addition, it is widely acknowledged in economics research 17  that evaluating R&I activities is challenging because of the nature of knowledge generation and its diffusion process. It takes time for R&I activities to produce results, outcomes, and impacts because of the importance of trial and errors, with an inherent need for risk taking and failures. The question of attribution of the effects observed is another challenge as scientific progress builds on knowledge that cumulates over decades and spreads unexpectedly in multiple domains and applications.

Programme indicators 18  used in this evaluation are referred to in section 4 and in the evaluation matrix, in Annex 3. It should be noted that the performance on all programme indicators is expected to still increase as projects continue delivering upon their completion.

Indicators and their monitoring system suffer from inherent shortcomings. Thorough checks were conducted in order to ensure that data is robust. Nevertheless, the evaluation wants to recognize four challenges in this field. First, as already noted in the interim evaluation, Horizon 2020 indicators refer only to parts of the programme’s intervention logic. Second, indicators could have been better designed, as they were not accompanied systematically by baselines values (i.e. values before the programme) or by target values (i.e. expected values at the end of the programme). Third, monitoring data recorded in the Commission’s IT systems can minimise but not completely exclude cases of multiple counting, e.g. the same publications/patents reported in one year are reported again in other years. Fourth, information on patents is provided voluntarily by beneficiaries with possible cases of erroneous reporting and under-reporting. In addition, Intellectual Property Rights (IPR) from Horizon 2020 and FP7 may include “background patents”, i.e. patent (or other IPR) applications filed before the start of the Horizon 2020 project, for which no causal link can be established with the support received by the programme. 

Data used in this evaluation is publicly available in the Horizon dashboard database, but a direct one-to-one comparation between the figures presented here and the dashboards is not possible due to different reference dates and, for some indicators, different approaches to data cleaning and removal of duplicates.

Section 3 provides data on the implementation of the programme since its launch until 1 January 2023. Section 4 provides an evaluation of the programme based on triangulation of evidence that predates 1 January 2023 19 (e.g. most external studies were carried out in 2022, based on programme data that was extracted at the end of 2021).

The impact assessment 20  of Horizon 2020 identified several weaknesses in the European science and innovation system, which were factors in low productivity, declining competitiveness and inadequate responses to societal challenges. 

While Europe has a historically strong science base, it often lags behind the United States when it comes to highly cited science or top-ranking universities, with increasing competition as well from other countries. An increase in spending on frontier research, associated infrastructure and training and education was identified as necessary to strengthen Europe’s scientific and technological performance, and to provide the basis for the EU’s competitiveness in the future.

The EU had not yet managed to translate its early lead in many green and ‘quality of life’ technologies (in health or security, etc.) into an innovative and competitive lead, experiencing insufficient technological leadership and a lack of innovation capacity in companies.

In addition, a lack of coordination of research to tackle the challenges faced by society led to missed opportunities to generate scale and synergies. Coordination between Member States for R&I was deemed insufficient. On average, in the EU, only some 10% of public budget for R&D is allocated at European level through the FP for R&I, enabling coordination and collaboration across countries. 21

The complexity of administrative procedures to apply for funding and take part in the framework programme were identified as the biggest obstacle to implementation. 22  

In line with the needs described above, Horizon 2020 was designed with the general objective to ‘contribute to building a society and an economy based on knowledge and innovation by leveraging additional R&I funding, and contribute to attaining R&I targets, including the target of 3% of GDP for R&D by 2020. It shall support the implementation of the Europe 2020 strategy and other Union policies, as well as the achievement and functioning of the European Research Area (ERA)’.

The programme’s specific objectives were to:

1.Strengthen Europe’s scientific base (32% of the total budget), through:

a.the European Research Council (ERC) providing attractive and flexible funding to enable EU competition for frontier research. (17% of the total budget);

b.fostering collaboration on radically new ideas to accelerate the development of Future and Emerging Technologies; (4% of the total budget);

c.Marie Skłodowska-Curie actions, providing training, career development and transnational mobility of researchers; (8% of the total budget);

d.Support for excellent European research infrastructures. (3% of the total budget).

2.Boost Europe’s industrial leadership and competitiveness (22% of the total budget) thanks to:

a.Leadership in enabling and industrial technologies, i.e. support for research, development and demonstration of key technologies; (17.5% of the total budget);

b.Access to risk finance, i.e. provision of debt and equity finance for R&I driven companies; (3.5% of the total budget);

c.Innovation in SMEs, support for all forms of innovation in SMEs. (1% of the total budget).

3.Increase R&I’s contribution to tackling Societal Challenges (SC) (39% of the total budget), through basic research, applied research, knowledge transfer and innovation. The focus should be on the EU’s policy priorities, without predetermining the precise choice of technologies or solutions that should be developed in:

a.Health, demographic change and well-being (Societal Challenge (SC)1) (10% of the total budget);

b.Food security, sustainable agriculture and forestry, marine, maritime and inland water research, and the bio economy (SC2) (5% of the total budget);

c.Secure, clean and efficient energy (SC3) (8% of the total budget);

d.Smart, green and integrated transport (SC4) (8% of the total budget);

e.Climate action, environment, resource efficiency and raw materials (SC5) (4% of the total budget);

f.Inclusive, innovative and reflective societies (SC6) (2% of the total budget);

g.Secure societies (SC7) (2% of the total budget).

4.Spread excellence and widen participation (1% of the total budget) via actions aimed at distributing the benefits of an innovation-led economy across the EU, in accordance with the principle of excellence.  

5.Increase the role of science in society (1% of the total budget), to build effective cooperation between science and society. 

6.Optimise the delivery of the programme. 23

Figure 1: Horizon 2020’s structure

Source: European Commission, 2023.

A set of cross-cutting issues were also promoted across the programme, in particular gender equality, social sciences and humanities, international cooperation, responsible research and innovation, widening participation, sustainable development, biodiversity and climate action, the digital agenda, SMEs and broader private sector participation.

From an implementation perspective, the programme’s objective was focused on simplifying and reducing the administrative burden during the phases of preparing a proposal and implementing a project, with targets for the time to award a grant and the time to pay. 24 To boost innovation, a target was set in Leadership in Enabling and Industrial Technologies (LEIT) and Societal Challenges to allocate at least 20% of the budget to SMEs. 25

The JRC (with 2% of Horizon 2020’s total budget) and the EIT (3% of the total budget) were expected to contribute to both the general objective and the specific objectives by (1) providing customer-driven scientific and technical support for EU policies, and (2) integrating the knowledge triangle of higher education, research and innovation. The indicators for assessing the performance of the EIT 26  stem from the activities of universities, businesses and research organisations integrated in the EIT Knowledge and Innovation Communities and collaboration inside the knowledge triangle leading to the development of innovative products, services and processes.

As a result of the annual budget procedures from 2014 to 2020, the programme finally received a voted budget of EUR 75 623.6 million (in current prices 27 ). A total of 78% of the budget was granted to transnational collaborative projects (through research and innovation actions and innovation actions) while support to individual researchers and companies was provided through the ERC grants, some Future and Emerging Technologies (FET) schemes, the Marie Skłodowska-Curie Actions (MSCA) and under the SME instrument. Other types of actions included the procurement of innovative solutions, public-public partnerships (including ERA-NET Co-funds and Article 185 actions), public-private partnerships (including Article 187 partnerships, Joint Technology Initiatives and contractual public-private partnerships), inducement prizes and financial instruments. Coordination and Support Actions (CSA) and procurements were used for studies, expert groups, conferences, and to disseminate and exploit results. A special form of collaborative project was also piloted, the Fast Track to Innovation, which focused on industrial actors. The Commission also undertook direct R&I actions through its Joint Research Centre.

Funding was mostly allocated through biennial work programmes. In evaluating project proposals excellence was the main criterion, next to the quality and efficiency of implementation and the expected impact.

Horizon 2020 brought considerable change compared to FP7 by including all support for innovation in one programme, which had previously been financed by separate EU programmes: the innovation-related part of the Competitiveness and Innovation Programme, and the EIT.

Horizon 2020 attempted to simplify access for participants, with a single web portal for all information and projects, less paperwork when applying, and more focused controls and audits.

Several changes were made during the last two years of the programme, following the interim evaluation. These included a pilot of lump sum funding and two-stage application procedures, the launch of a pilot for the European Innovation Council, flagship initiatives for international cooperation, and cross-cutting calls for proposals on specific policy priorities, such as the 2020 call for proposals to support the European Green Deal through R&I.

Horizon 2020 was expected to deliver scientific, technological and innovation outputs that would translate into scientific, economic and societal impacts related to the specific objectives of the programme. The main impacts expected of Horizon 2020 are illustrated in Figure 2 (on the programme’s intervention logic). It shows that all parts of the programme can bring scientific, economic and/or societal value.

Figure 2: Horizon 2020 intervention logic

Source: European Commission, 2023.

Progress towards the objectives was monitored according to 23 key performance indicators (KPIs) that were either set out in the legal base 28 or subsequently developed by the Commission 29 . The Horizon 2020 indicators represented a step forward in the monitoring process because this was the first time that KPIs were introduced, although they did not cover the entire programme and contained a number of shortcomings (presented in Annex 2).

The performance indicators for assessing progress against the general objective were 30 :

·the research and development target (3% of GDP) of the Europe 2020 strategy;

·the innovation output indicator in the Europe 2020 strategy 31 ; and

·the proportion of researchers in the active population.

Some indicators were accompanied by baseline values and specific targets for the end of Horizon 2020. All indicators are presented in the evaluation matrix in Annex 3.

The Horizon 2020 programme set additional quantitative targets for:

·Sustainable development related investment: at least at 60% of overall investment;

·Renewable energy, end user energy efficiency, smart grids and energy storage activities: at least 85% of the overall Energy Societal Challenge funds; 

·Funding to SME: at least 20% of in the LEIT programme and under Societal Challenges (of which 7% committed through the SME instrument); 

·Climate change-related investment: at least 35% of overall investment.

The main point of comparison is the impact assessment carried out for Horizon 2020. Where the expected effects were quantitatively estimated or targets were set in the Horizon 2020 impact assessment or in the legal base, this evaluation compares the actual Horizon 2020 data to these expectations. This is the case for the analysis of macroeconomic impacts and KPIs on publications, patent applications, mobility of researchers, risk finance, and for targets on simplification, SME participation and climate-related spending, high quality publications by FET, female representation in advisory panels, efficiency metrics (time to contract, time to pay, error rate, etc), leverage factors per selected JU).

In the absence of targets, this evaluation compares the current Horizon 2020 performance to data from the end of FP7. This is the case for the number of publications, patents funded with a contribution from Horizon 2020, female representation in advisory panels, participation from entities located in widening countries, contribution to Sustainable Development Goals, JRC concrete contributions to policies and international cooperation.

Where this approach is not possible, or where additional insights could be gained, this evaluation compares the actual results of Horizon 2020 against those of other relevant EU or international benchmarks. This is the case for the quality and influence of scientific publications, JRC’s publications, share of publications freely and publicly available, number of public private academic co-publications.

If none of the above is possible, newly available data on Horizon 2020 are presented, without any baseline nor benchmark. This is the case for the number of patents and publications for some of the Societal Challenges and leverage factor at programme level.

The overall completion rate of Horizon 2020 projects at the time of this final evaluation is 59%, while the remaining projects are still in progress and due to submit their final report. The cumulative implementation rates at the time of the evaluation were 99.99% for Horizon 2020 commitments and 87.84% for payments.

Implementation delays due to lockdowns linked with the Covid pandemic temporarily affected networking and project dissemination and exploitation opportunities. However, the completion rate of Horizon 2020 is higher than at the time of the final evaluation of the preceding programme, FP7 (50%). While unfinished projects inevitably generate a degree of uncertainty, comparison of results with the FP7 baseline is being done at a similar point in time. Long-term analysis of FP7 also shows that some effects are likely to increase as the final project reports are received and in the years that follow (see section 4.1.5. on long-term impacts).

Calls for proposals

During the lifetime of Horizon 2020, 1 076 calls for proposals, covering a total of 3 706 topics, were launched and evaluated. These calls attracted over 285 000 eligible proposals, which requested EUR 478 billion in EU funding. A total of 1.8% of the proposals submitted were ineligible. 32

Although 46% of the eligible proposals were assessed as being of high quality 33 by external experts, funding could not be granted to all, resulting in an average success rate 34  of 11.9% and an oversubscription rate 35 of 74%.

The percentage of proposals above the quality threshold and the percentage of proposals funded (success rate) were highest under Spreading Excellence and Widening Participation (67.1% and 15.9%), followed by Excellent Science (55.2% and 13.7%), and lowest under Industrial Leadership (34.3% and 8.7%).

Applications to Horizon 2020

The eligible proposals included over 1 million applications 36 . Most applications originated from private for-profit organisations (40%), followed by higher education institutions (36%) and research organisations (17%). SMEs accounted for 24% of applications. More details are provided in Annex 6, including detailed data tables on applications by country, organisation type and pillar, and success rates by country.

88% of the applications originate from EU-28 Member States, half of which originated from entities located in four countries: Spain, Italy, Germany and the United Kingdom. EU-13 countries represent 10% of the applications and - putting the number of applications in perspective with the scientific population of each country - the most active Member States were Cyprus, Greece, Slovenia and Estonia.

Figure 3 presents applications coming from EU-15 (Member States that joined the EU before 2004), EU-13 (Member States that joined in or after 2004), associated countries and third countries.

Figure 3: Applications by country group

Source: CORDA data – cut-off date: 1 January 2023

As regards third countries, Switzerland, Norway, Israel and Turkey accounted for 84% of the 83 377 applications submitted by entities located in countries associated to Horizon 2020. Entities located in non-associated countries outside the EU represent 3.7% of all applications and 1.2% of the requested amount in retained proposals. The United States led with 10 336 applications, almost one third of all applications from non-associated countries, followed by China (2 970 applications) and Canada (2 282 applications).

Funding allocation    

EUR 68.3 billion were allocated through 35 426 signed grants, 31.6% of the funds being allocated to the top 100 beneficiaries.

Around 50% of Horizon 2020 funding was allocated to 341 organisations. Around 10% of funding went to the 50 best-performing universities in Europe, based on the Leiden ranking data. 37  

Around 19% of Horizon 2020 funding (approx. EUR 12.9 billion) went to newcomer organisations, i.e. organisations that had not participated in FP7. Across the programme, around 50% of the total funding to private companies went to newcomers: most newcomers are in fact SMEs. Newcomers are in general more common in types of actions with high industry participation; within Joint Undertakings, 19% of all funding went to newcomers. 38  

Figure 4: Share of EU funding to Horizon 2020 newcomers by type of action 39

Source: CORDA data, 2 August 2022. Taken from European Commission, ‘Newcomers in EU R&I programmes – Main trends in Horizon 2020, first evidence from Horizon Europe’, monitoring & evaluation report,  https://data.europa.eu/doi/10.2777/198795  

The average budgetary size of the signed grants 40  was EUR 2.3 million but this varied substantially between the three pillars, from EUR 1.2 million under Excellent Science to EUR 3.7 million under Societal Challenges. 

SMEs, with an application success rate of 12% (slightly higher than the programme average), accounted for 19.8% of all participations and received EUR 11.4 billion in funding.

The proportion of Horizon 2020 budget spent on climate action 41  reached 32%. 42  

Type of Actions

Most funding was allocated through research and innovation actions (RIAs, 9.830% of the funding), followed by frontier research grants awarded by the ERC (19.7%), innovation actions (17%) and the MSCA (10%). The MSCA accounts for the highest number of signed grants (11 960) followed by the ERC (7 838), the SME instrument (5 734) and RIAs (3 978).

Figure 5: Funding allocation by type of action

Source: CORDA data – cut-off date: 1 January 2023

The new funding scheme for innovation, the innovation actions (IAs), aimed at producing plans and arrangements or designs for new, altered or improved products, processes or services, demonstrations, piloting and prototyping. Despite representing just 6% of all Horizon 2020 projects, IAs represented 17% of the total financial contribution (or EUR 11.4 billion). Except for the first year of the programme, the proportion of funding to IAs constantly increased, from 13% in 2015 to 21% in 2021 43 . IAs are characterised by high participation rates by private for-profit entities, which received over EUR 5 billion in total. In the Industrial Leadership pillar, 32% of all funding was for IAs and 25% in Societal Challenges (35%). The balance between demonstration activities and ‘first-of-a-kind’ innovation activities is heavily lopsided towards the former, by a ratio of almost ten to one (and even higher in the Industrial Leadership pillar). 44  

Beneficiaries

Different organisations received Horizon 2020 funding, 42% of which were SMEs and 69% were newcomers to EU research and innovation programmes. 45

The majority of funding (78% or EUR 53.3 billion) went to collaborative projects, which brought together an average of 11 participants in 14 612 supported projects. Looking at the number of individual grants, 59% (20 814 grants) went to a single beneficiary, mostly for ERC, MSCA and SME instrument representing 22% of the funding (EUR 15 billion).

Higher education institutions come first in terms of the EU contribution received (39% or EUR 26.8 billion), followed by private for-profit entities (28% or EUR 19.3 billion) and research organisations (25% or EUR 17.1 billion).

Participation varied by country group, between EU Member States, associated countries and third countries as shown in Figure 6.

Figure 6: Participants and funds by country group in Horizon 2020

Source: CORDA data – cut-off date: 01/01/23

More than half of the Horizon 2020 funding allocated to Member States went to four countries: Germany (16%), the United Kingdom (13%), France (12%) and Spain (10%). However, when comparing the amounts received from Horizon 2020 to million EUR of gross domestic expenditure on R&D, smaller countries such as Estonia, Greece, Cyprus and Latvia performed better than bigger Member States. More details on this are available in Annex 6, including detailed data tables on Horizon 2020 investment by country, pillar and organisation type.

For non-associated third countries, the share of participations is around 4%. The relatively low figure for the share of EU funding that went to these participants can be explained by the fact that only low- and middle-income third countries are automatically eligible for funding, while high-income countries had to contribute with their own funds to Horizon 2020 projects. 46

Participation of SMEs

The participation of private for-profit entities in Horizon 2020 was higher than in FP7. They accounted for 33.5% of all participations (against 30.3% in FP7), and 28.2% of all EU financial contributions to the programme, 3.3 percentage points more than in FP7. 47

Horizon 2020 has so far awarded 22.2% of EU funding to SMEs in the LEIT programme and under Societal Challenges. The highest rate of SME participation was in Nanotechnologies, advanced materials, biotechnology and advanced manufacturing and processing (NMBP, 29%) 48 , where a few sub-programmes had particularly high 1.5 participation: 42% of funding in nanotechnology topics and 36% in biotechnologies went to SMEs. This was much higher than the SME share in the New Production Technologies theme of FP7 (25.6%). 49 Lower shares of EU funding were seen in SC1 (health, 17%), SC4 (transport, 17%) and SC6 (inclusive societies, 14%).

Around one third of funding to SMEs was provided through the SME instrument. This represents 7.1% of overall Horizon 2020 funding, which is above the 7% target set at the start of the programme (see Table 1).

Table 1: Cross-cutting key performance indicators (KPIs) on SME participation in Horizon 2020

Source: Cross-cutting issues study (2023), elaboration on CORDA data (2022).

Horizon 2020 was also successful in attracting new SME participants. Around half (50.3%) of all EU funding to private for-profit businesses went to newcomers, and two thirds of this amount to SMEs. In general, actions that attract more SMEs – the SME instrument, as well as innovation actions – have high rates of newcomer participation. Conversely, actions that target collaboration with the private sector, but not necessarily SMEs, had fewer new participants. 50  

This section provides an evidence-based assessment of the successes and shortcomings of the Horizon 2020 programme in terms of its effectiveness, efficiency and coherence. It begins by examining how effectively Horizon 2020 achieved its scientific, societal and economic objectives, as well as the programme’s parallel objectives of spreading excellence, widening participation and promoting science in society. This section then considers the cost of pursuing these objectives. Finally, it provides evidence of the degree to which the programme has operated in a coherent way, both internally between its different instruments, and externally with other relevant EU and national programmes.

4.1.1. Effectiveness: Scientific impacts – To what extent has Horizon 2020 strengthened Europe’s scientific base?

51 The objective of Horizon 2020 was to reinforce and extend the excellence of the Union’s science base and to consolidate the European Research Area (ERA) in order to make the Union’s research and innovation system more competitive on a global scale. Throughout the course of Horizon 2020, excellence was assured by means of pan-European competition for funding and a stringent project proposal evaluation process. All actions across all Horizon 2020 pillars are expected to contribute towards achieving scientific impact.

52 53 54 55 The Horizon 2020 Regulation requires that progress on its over-arching general objective is assessed against a number of indicators, including the proportion of researchers in the active population in the EU. Eurostat data show that, in 2021, the number of full-time equivalent (FTE) researchers employed in the EU was 2 million, or 1% of the total active population. This represents an increase of over 570,000 researchers compared with 2012 (1.43 million, 0.7% of the labour force). This 33% increase in the number of researchers in the active EU population between 2012 and 2021 is not considered a direct consequence of Horizon 2020 funding as other external factors (e.g. tax and labour law changes in Member States) also play a role and this evaluation is unable to quantify the direct contribution of Horizon 2020 to this increase. 

56 Horizon 2020 contributed to scientific impact through multiple streams that delivered quality research and contributed to scientific breakthroughs, while reinforcing human capital, research infrastructure, and encouraging participation from countries least performing in research and innovation as described in the sections that follow.

Strengthening frontier research: Publications, quality of research and scientific advancements

57 58 59 60 61 62 63 Horizon 2020 has strengthened the EU’s scientific position worldwide. Bibliometric analysis shows that Horizon 2020 had a significant positive impact, both in terms the overall number and the quality of publications produced, and of the standards of scientific excellence across the entire programme. In the period 2014-2022, Horizon 2020 produced a total of 276 784 peer-reviewed publications. Compared with the previous framework programme (FP7: 219 620 publications), the total number of publications is higher and is still expected to rise when all projects have been completed. Horizon 2020 publications are cited at twice the world average rate for similar publications (with an FWCI of 2.03), while 3.9% of these publications are among the top 1% most cited publications worldwide. Furthermore, evidence from benchmarking exercises demonstrates that the citation scores for Horizon 2020 are higher than those of selected international funders, both as a proportion of the top 1% most cited publications and in terms of its average normalised citation score. Activities funded under the Excellent Science pillar have the highest number of publications, mainly under the European Research Council (ERC) and Marie Skłodowska-Curie Actions (MSCA) (36% and 22% respectively).

Figure 7: Horizon 2020 key publication metric comparison with other funders

Source: Bibliometric analysis, Excellent Science Evaluation study, 2023. NWO (the Netherlands), the French National Research Agency (ANR), the Australian Research Council (ARC), the FCT (Portugal), the Austrian Science Fund (FWF), the Natural Sciences and Engineering Research Council of Canada (NSERC) and the National Science Foundation (USA).

64 The ERC has provided attractive and flexible funding to encourage EU competition in frontier research. Not only did it exceed its Horizon 2020 target (see KPI 1 in Table 3), but it currently accounts for the highest number of peer-reviewed publications across Horizon 2020 (49 496 publications), receiving an average of 24.4 citations per publication. ERC publications have also been cited at over twice the average worldwide rate for similar publications (an FWCI of 2.32).

The Future and Emerging Technologies (FET) programme has met its Horizon 2020 target for high-quality publications (see KPI 2, Table 3). The value is likely to increase further as project implementation is still ongoing. It has also achieved high citation impact, with, on average, 28.2 citations per publication.

The Societal Challenges pillar has made limited progress towards its Horizon 2020 target related to publications (KPI 14). Nevertheless, the Societal Challenge 1 (health, demographic change and well-being) and Societal Challenge 5 (climate action, environment, resource efficiency and raw materials), have achieved high citation rates, with 7.4% and 5.9% of their publications among the top 1% most cited publications, respectively. Moreover, those publications were cited close to three times the global average rate (with FWCIs of 2.86 and 2.61, respectively).

65 66 The scientific impact of JRC publications, as measured by bibliometric indicators, is on a par with leading universities and prestigious research organisations. During the course of Horizon 2020, JRC scientists published over 7 000 peer-reviewed scholarly publications, a third of which feature in the top 10% most highly cited publications in their field. In addition, the JRC produced a wide range of policy-relevant outputs, such as standards, reference materials, technical systems and guidance on policy implementation (KPI 22 below).

Sources: KPI1 - European Research Council Executive Agency, Annual Activity Report 2022, p. 6, https://commission.europa.eu/system/files/2023-06/ERCEA_AAR_2022_en.pdf ; KPIs 2 and 14 - Commission monitoring systems - CORDA, data on 24/04/2023; KPIs 22 and 23 - Joint Research Centre, Ex post evaluation of the activities of the Joint Research Centre under Horizon 2020 and Euratom 2014-2020: final report of the ex post evaluation panel, Publications Office of the European Union, 2022, p. 69, https://data.europa.eu/doi/10.2760/257315  

Horizon 2020 support has enabled European and other supported researchers to reach top-tier status in their field. For Societal Challenges 2, 3, 4 and 5, Horizon 2020 funding boosted citation impact for 2014-2021 publications when measured against a baseline of comparable publications by the same authors. 69  Likewise, Horizon 2020 funding in this area enabled researchers to out-perform other, non-Horizon 2020-funded, publications. 70  

Horizon 2020’s open access principles and requirements had a positive impact on the proportion of publications that were made freely and publicly available online, which rose from 65% in 2014 to 82% in 2022 and compares favourably with similar international programmes. 71  The number of open access datasets arising from Horizon 2020 projects also increased, from 64 open datasets in 2015 to 1 694 open datasets in 2020. However, despite Horizon 2020 producing a larger number of open access datasets, this data did not always meet the principles of findability, accessibility, interoperability and reusability (FAIR) and there were significant variations across disciplines and programme parts. Although over half of all respondents to a survey on the matter 72  considered that their Horizon 2020-funded study was reproducible 73 , complementary research 74 found that many open access datasets did not meet all the FAIR principles. For instance, only 35% of datasets were findable and only 29% were accessible and interoperable. In terms of reusability, only 61% of datasets included a text and data mining licence.

75 ‑‑ 76 77 The total number of public-private academic co-publications produced under Horizon 2020 was 53 813. Although no target was set for this indicator, either in the legal basis for Horizon 2020 or in subsequent programme statements, some desk research suggests that more publicprivate copublications were produced than under FP7 (see Table 4). This rate was also higher than for equivalent non-framework programme publications in other EU countries and internationally. This suggests that Horizon 2020 successfully selected projects and researchers that were able to contribute to knowledge transfer between the business and academic sectors. Some partnerships, such as ECSEL, cPPP 5G, cPPP Photonics, cPPP SPIRE and EIT Raw Materials, also produced high proportions of public-private co-publications.

For the Societal Challenges, direct comparison with FP7 is more difficult. However, some performed especially well, notably SC7 and SC3. Public-private co-publications were also more common under Horizon 2020 projects than under non-framework programme projects. 78

Source: Internal Commission monitoring systems (CORDA), data on 24/04/2023. Numbers are expected to increase when all Horizon 2020 projects are finalised.

Scientific breakthroughs and advancements

External evaluation showed that Horizon 2020 contributed to scientific breakthroughs 79  and advancements in emerging areas of science and technology, especially in the medical sciences, quantum mechanics, chemical engineering and composite materials. Specific examples include: capturing the first ever image of a black hole 80 ; research into metal-halide perovskites 81 (with potential applications in solar cells, light-emitting diodes and other optoelectronic devices); strong coupling plasmonic 82 (with potential applications in quantum-mechanical and classical optical information processing and in fundamental studies of light-matter interaction) and quantum entanglement 83 (observation of the quantum properties of macroscopic objects and quantum cryptography). The programme also made advances in ancient DNA dating (including the discovery of the Denisovans 84 , and evidence of Europe’s first homo sapiens 85 ), predicting new protein structures and interactions 86 (by applying powerful AI tools to structural biology and chemistry), immunotherapy (personalised cancer vaccines 87 ), understanding climate change, as well as developing the Pfizer-BioNTech and Oxford-AstraZeneca vaccines against Covid-19. 88  

The contribution of Horizon 2020 to the advancement of frontier research has been recognised internationally through various rewards and prizes. Horizon 2020 has supported 33 Nobel Prize winners 89 prior to or after the award of their prize (up from 17 reported in the interim evaluation). In addition, between 2014 and 2020, six ERC grantees were awarded a Wolf Prize and one received a Fields Medal. Furthermore, a survey of beneficiaries suggests that they recognise Horizon 2020’s role in helping to promote fundamental and novel research activities, owing to the degree of freedom and originality the programme offered them. 90  Beneficiaries felt that their participation boosted their credibility among their peers and helped to position them at the centre of international networks of R&I experts. 91  The survey findings also confirmed Horizon 2020’s role in helping to generate high-quality research outputs and findings, obtain scientific awards and prizes and produce high-quality publications. 92

Investing in future emerging technologies to accelerate deployment

Bibliometric analysis provides additional evidence on the contribution of Horizon 2020 to new and fast-growing research topics in science. 93  Namely, 26% of all Horizon 2020 publications were linked to these topics, of which 1.5% were also among the top 1% of most highly cited publications worldwide. Pillar 1 performed similarly well, with 24% of publications linked to these hot topics and 1.6% of those among the top 1% most cited publications. Under Pillars 2 and 3, around one third of publications concerned new and fast-growing research fields, of which 2.2% and 1.3%, respectively, were among the top 1% most highly cited. 

Horizon 2020 allocated significant resources to projects in the fields of artificial intelligence, quantum computing and clean energy technologies (see Figure 8). Biological sciences, gene expression and environmental engineering were the top three frontier research areas tackled by ERC grantees, followed by astronomy, theoretical physics, atmospheric sciences, and magnetic fields. 94  

Figure 8: Horizon 2020 funding to new and emerging areas

Source: CORDA data – cut-off date: 04/04/22 from the Excellent Science study

The FET flagships in particular (see two examples in the box below) had a sizeable impact on knowledge creation. The majority of FET projects (83%) dealt with research ideas not previously present in the scientific community and thus deemed to be ‘radically new’. 95 As a consequence, more than one third of the publications stemming from the FET programme are linked to future and emerging research and technology fields with 1.6% being amongst the top-1% most highly cited worldwide. With publications in high impact journals like Science or Nature, 96 FET results were cited in many different scientific fields. 97

Enhancing researchers’ skills, mobility and career development

100 101 102 Horizon 2020 has diversified and improved researchers’ skills and knowledge. The survey of beneficiaries showed that around 85% of those who were awarded MSCA individual fellowships considered the training provided and supervision to be ‘very good’ or ‘good’. The survey also showed that 70% of respondents had received training on new or advanced scientific methods in their own research field. Under ERC projects the skills most frequently developed were ‘scientific methods and/or techniques, ‘project and people management’ and ‘thinking’. The ‘Research infrastructures’ and ‘Science with and for society’ (SwafS) programmes also contributed to skills development, as training was a frequent project component. Specifically, over 80% of the SwafS survey respondents said their projects had improved their research skills and knowledge and given them transferable skills (e.g. project management and teamwork). 

Horizon 2020, mainly via the MSCA, also supported the international and intersectoral mobility of researchers. The corresponding programme target (KPI 4) is expected to be met once all projects are completed.

Source: Baseline from: European Commission, DG for Research and Innovation, Horizon 2020 indicators: assessing the results and impact of Horizon, Publications Office, 2015, . Achieved values: CORDA data, updated on 19/01/2023 (provided by DG EAC). Numbers are expected to increase when all Horizon 2020 projects are finalised.

104 105 106 Horizon 2020 plays a strong role worldwide as regards the international mobility of researchers. According to survey results, around 89% of recipients of individual fellowships under the MSCA said their project offered international mobility opportunities. This proportion was significantly lower (63%) for unsuccessful applicants who still implemented their project, indicating that the programme was successful in this regard. Almost 70% of the ERC principal investigators surveyed, at the very least ‘rather agreed’ that ERC projects contributed to international mobility opportunities. Indeed, half of the programme’s ERC team members were nationals of a country other than that of the host institution, while 40% of the scientific and technical staff moved country when they started working on the ERC grant.

The Horizon 2020 programme also had a positive impact on intersectoral mobility. For example, the MSCA Research and Innovation Staff Exchange (RISE) and the Innovative Training Networks (ITN) actions contributed significantly to the mobility of fellows to the private sector (24.2% of RISE and 15.7% of ITN beneficiary fellows had a private for-profit organisation as their main host institution). Similarly, 36% of those awarded fellowships under RISE and 23% of ERC principal investigators said they had intersectoral mobility opportunities because of their project (compared with 26% and 17%, respectively, among the unsuccessful applicants to the same programmes). 107  Finally, MSCA fellows who engaged in intersectoral mobility were more likely than other MSCA fellows to be employed after the end of the fellowship and found better and more diverse job opportunities outside of academia. 108

109 ‑ 110 MSCA further supported mobility flows between countries by enabling researchers to return to their home country, if they so wished. In particular, the Reintegration Panel of the European Fellowships (IF-EF) helped nationals or long-term residents of Member States or Associated Countries to return to their home country. In several countries where priority was given to widening participation, (Romania, Lithuania, Bulgaria, Latvia, Slovakia, Cyprus, Croatia, Estonia, Poland and Hungary), over half of incoming MSCA researchers awarded individual fellowships were returning to their country of origin. Similarly, over 25% of experienced postdoctoral researchers used the MSCA COFUND to return to Member States where widening participation was prioritised. This proportion rose to 50% for the specific countries mentioned above. In the context of MSCA, 570 EU nationals benefited from the IFEF scheme, or 7.2% of all IF-EF fellows. Of these, 335 fellows (58.8%) used the scheme to return to their home country. Furthermore, the ‘Widening Fellowship’ pilot (renamed ‘ERA fellowships’ under Horizon Europe) significantly increased the inflow of researchers to countries designated as widening countries.

111 112 113 114 Participation in Horizon 2020 improved researchers’ career prospects, particularly those of early career researchers, such as MSCA fellows, ERC Starting and Consolidator grantees, FET grantees and more junior members of teams supported by ‘Spreading Excellence and Widening Participation’ (SEWP) actions. Almost 93% of ERC principal investigators rated their ERC project as having had, at least, a ‘good’ impact on their career prospects, with the impact being particularly evident for early stage researchers. Most FET researchers said that FET projects had advanced their careers and enabled them to venture into previously unfamiliar areas and disciplines and to benefit from an interdisciplinary approach. SEWP interviewees and survey respondents were generally convinced that the widening actions had had a positive impact on research careers across all career stages and that SEWP actions and enabled them to further develop their skills. The main benefits cited were: study visits to institutions in countries where the research was more advanced; knowledge exchange with partners more advanced in the field; training activities and access to and more effective use of high-quality research infrastructure. 

Strengthening Research Infrastructures

Horizon 2020 has enabled the EU to conceive, deliver and upgrade large-scale research infrastructures at a European and global level. The Research Infrastructures programme, in particular, played a role in promoting the development of pan-EU research infrastructures, providing support for collaboration, joint research and services, as well as access to such infrastructures. It also provided support for the development of the pan-EU research infrastructures included in the European roadmaps published by the European Strategy Forum on Research Infrastructures (ESFRI). These roadmaps list the most important research infrastructures in Europe for the next 10-20 years, with the aim of stimulating the implementation or upgrading of these infrastructures. In 2021, 41 infrastructures listed by ESFRI had achieved a state of maturity. Interviews with stakeholders and beneficiaries 115  confirm that this was largely thanks to the contribution of INFRA calls. The INFRA programme is on track to achieve its Horizon 2020 target as project implementation is still ongoing. The three most-used research infrastructures were SOLEIL in France, Diamond in the UK, and MAX IV in Sweden. However, support of Horizon 2020-funded work on infrastructures experienced regular difficulties in implementation, due to legal issues, see section 4.3.2. on synergies.

Source: Baseline from: European Commission, DG for Research and Innovation, Horizon 2020 indicators: assessing the results and impact of Horizon, Publications Office, 2015, https://data.europa.eu/doi/10.2777/71098 . Achieved values: Commission monitoring system (CORDA), data on 24/04/2023. Numbers are expected to increase when all Horizon 2020 projects are finalised.

Spreading excellence and widening participation

Horizon 2020 aimed to spread excellence and build up R&I capacity across the EU. It earmarked EUR 935 million of funding under SEWPs to widen participation in lower R&I-performing countries. 117  Evaluations show that some progress was made in expanding participation and funding to institutions in these countries. 118 Overall, entities from ‘widening countries’ represented 12.3% of Horizon 2020 participations, a 1.3 percentage point increase from the FP7 baseline. 119  All ‘widening countries’, except Croatia and Hungary, increased their participation. In terms of the average number of applications received per 1 000 scientists and engineers as a proportion of the country’s population, or the amount of EU funding received as a percentage of GDP, widening countries performed very well compared to other Member States, especially Cyprus, Estonia and Slovenia. 120

Given its specific focus, participation from widening countries is much higher for SEWP actions than the rest of the programme. 121  There are important differences in the widening countries’ participation patterns: around half of the Horizon 2020 SEWP funding went to four of the 15 countries (Portugal, Cyprus, Poland and Estonia). 122 Also, when the widening funding is normalised considering the population size, it can be observed that the countries benefiting the most from the widening measures are Cyprus, Estonia, Latvia, Slovakia, which also demonstrated higher participation in previous FPs. 123

124 125 126 In a dedicated report on measures aiming to widen participation in Horizon 2020, the European Court of Auditors noted that, although the large-scale effects of the widening instruments will only be visible in the long term, change depends to a large degree on R&I investments and reforms at national level. Similarly, the evaluation study on the cross-cutting issues of Horizon 2020 concluded that the degree to which widening measures were a causal factor in raising levels of participation across Horizon 2020 is difficult to gauge and that they are likely to have long-term rather than immediate impact. 

Nonetheless, initial results of the participation of widening countries over Horizon 2020 have been positive. The SEWP actions funded researchers and research groups with high levels of excellence. This is reflected by the good representation in SEWP actions of institutions listed in the Leiden Europe 250 ranking 127 , both for participants from widening and non-widening countries. At the same time, the participating researchers and research groups from widening countries seemed to be able to improve research production and quality thanks to the participation in SEWP. Beneficiaries of twinning and teaming projects indicated that it is specifically the partnering with the advanced institutions (e.g. through trainings, workshops and staff exchanges with these partners) that contributed the most to the quality of the research. 128  The survey and interviews stressed that the widening projects helped the project partners to structure and institutionalise these collaborations. Specifically, 97% of respondents indicated that these projects strengthened existing collaborations with partner organisations, enabling researchers in the widening countries to access know-how, expertise, infrastructure and equipment that is often not available in their countries. 129

As a result of participation in the programme, widening countries improved their research production and quality. While publications stemming from the SEWP actions account for 3.6% of all Horizon 2020 publications 130 , for widening Member States almost one-third of their total number of publications generated in Horizon 2020 is produced within the widening actions. 131  Among publications stemming from the SEWP actions, highly cited ones increased from 6% in 2014 to 17% in 2020. Moreover, 28% of the highly cited Horizon 2020 publications produced by widening countries were linked to widening actions. This reflects the importance of this programme part for widening countries in terms of producing excellent science. This relative importance is particularly high for Cyprus, Estonia, and Latvia.

However, evaluation findings show that there are few new entrants in the SEWP programme, compared to the share of newcomers in Horizon 2020. This shows that SEWP-funded researchers and research groups had often previously participated in the FP. 132

Source: Commission monitoring system (CORDA), data on 24/04/2023

4.1.2. Effectiveness: ‘Societal impacts’ – To what extent has Horizon 2020 increased the R&I contribution to Societal Challenges?

This section reports on the effects of Horizon 2020 actions grouped under ‘Societal impacts’ in the Intervention Logic (Figure 2). The term primarily refers to the direct, non-market benefits 134 of the programme, which increase the welfare of society, for instance by means of improvements in health, security and the environment. In general, these effects only become visible in the medium-longer term. This also covers a few indirect, non-monetary benefits such as positive impacts on gender equality. The section describes the impact Horizon 2020 had on national, European and international policies including via research actions by the JRC and addresses the programme’s contribution to the Sustainable Development Goals (SDGs). Conclusions on the effectiveness of actions under this pillar are based both on KPI data and the qualitative evidence from case studies provided below.

Pursuing research and innovation to contribute to Societal Challenges

The programme allocated funding in seven broad areas, referred to as Societal Challenges. 135 Table 8 details the baseline, targets and results for the two key performance indicators relevant to all Societal Challenges (KPIs 14 and 15). 

Source: Commission monitoring systems (CORDA), data on 24/04/2023.

The evaluation has made apparent that the Horizon 2020 monitoring and evaluation framework is not adequate to comprehensively capture positive effects under Societal Impacts. The outcomes and even outputs of some projects such as of projects fostering cross-border co-operation or resulting in an influence on a specific policy agenda were incompletely recorded. Potential additional effects from projects cannot be excluded. Case study evidence nevertheless suggests that encouraging trends and clear effects could be observed in specific topic areas, as illustrated by the examples that follow. A more detailed account of societal impacts of Horizon 2020 projects is provided in Annex 3 and in the underlying set of evaluative studies and materials.

In health-related research, societal effects were generated in particular in the areas of rare diseases 138 , orphan medicines 139 and antimicrobial resistance. 140  Following effective efforts to combat Ebola and Zika outbreaks that were already documented in the interim evaluation, Horizon 2020 and previous FPs also funded research instrumental for understanding and combatting COVID-19 which produced societal impacts. Analysis conducted in 2021 documented the contribution of EU funding to publications on COVID-19 research and to the main discoveries and insights on COVID-19. 141  EU funding has contributed to 3 000 papers on COVID-19 coming from almost all parts of the framework programmes (FP7 and Horizon 2020), with Societal Challenge 1, the ERC, and MSCA accounting for about 80% of the total. 142  The EU is the third most frequently acknowledged funding source for COVID-19-related research, after the US Department of Health and Human Services and the National Natural Science Foundation of China. 

The delayed or abandoned implementation of clinical trials negatively affected the benefits from health-related research. In the context of a relatively low share of 79% of Societal Challenge 1 projects meeting all or most of their objectives (compared to 97% under FP7 Health), the area of ‘Treating and managing disease’ was particularly affected, with a share of only 70%. Apart from Covid-19 emergency measures, the reduction reflects an overall trend affecting the increasingly complex clinical trials 143 . 

Under Societal Challenge 2, Horizon 2020 research projects increased knowledge of the marine environment and fishing methods and supported policy, most notably contributing to the further development of the European Common Fisheries Policy. Examples include the DISCARDLESS 144 and MINOUW 145 projects which contributed towards reducing discards, a practice that wastes resources and poses a threat to the health and stability of marine ecosystems. 

As regards the smart European electricity grid, 146  projects successfully established an innovation community directed at next generation technologies and tools for grid automation, integration of storage, energy system integration and increasing the share of renewables in the electricity system.

While the circular economy has been reflected in Horizon 2020 calls since 2014, it gained in importance and presence in the last work programme (2018-2020) 147 , accompanying the successive Circular Economy Action Plans published in 2015 and 2020. The calls for proposals evolved from their initial scope that covered waste management and water innovation to a broader scope aimed at boosting global competitiveness, encouraging sustainable economic growth and generating new jobs. The focus also shifted from supporting SMEs to considering other actors, such as industry, policymakers and the global community for the relevant sectors. This change helped to implement the Circular Economy Action Plan and other high-level EU priorities.

Expenditure on sustainable development 148  exceeded the Horizon 2020 target of 60% 149 , at 64.4% of the total budget. Societal Challenges performed well above target, with 84.7% of all expenditure going to projects contributing to sustainable development. Except for expenditure on Science with and for Society (62.7%), all parts of the remaining programme were below the 60% target, particularly the EIC Pilot, Spreading Excellence, the Excellent Science pillar and the Industrial Leadership pillar.

While many relevant projects were funded under Societal Challenge 3 (Secure, clean and efficient energy), its Horizon 2020 target (KPI 19), detailed in Table 9, was not met.

Source: Commission monitoring systems (CORDA), data on 24/04/2023. Based on policy monitoring flagging by project officers.

In the area of offshore renewable energy, funded projects contributed to an increase in publicly available knowledge and evidence, technical innovations enabling future cost reductions and upscaling, upskilling and training, as well as the diffusion of offshore renewable energy to a wider geographic area. 150 An important contribution was to inspire confidence in policymakers about the feasibility of offshore renewable technologies and the role they can play in the energy system. 151  

In the area of urban transport (under Societal Challenge 4), many projects contributed to the refinement of Sustainable Urban Mobility Plans (SUMP). 152 For example, thanks to the work of the Park4SUMP project, cities could implement well-tailored parking measures and integrate them into sustainable urban mobility planning which improved parking policies in 16 partner cities. The multiple strategies supported by Horizon 2020 contributed to more sustainable and healthy urban transport in different EU cities through a concerted push towards measures such as improving cycling infrastructure.

Projects funded via partnerships played a central role in Horizon 2020 supporting the ambition to make European transport systems sustainable and seamless for all to use 153 . For example, the Fuel Cells and Hydrogen JU 154 drove forward the deployment and scalability of fuel cell buses, as well as the design of key solutions necessary for low-emission air travel also supported by Clean Sky 2 155 .

The Horizon 2020 project IMPACT-SC5 (Climate action, resource efficiency and raw materials) evaluated the progress of 87 projects funded under the Societal Challenge 5 Work Programme for 2014-2015. It found that most of the projects produced policy-related outputs addressing EU policy priorities and the SDGs, particularly in the portfolios of climate, waste, and environment, ecosystems and biodiversity. 156 Climate change mitigation and adaptation, as well as the reduction of waste generation and environmental depletion, were the focus of policy contributions produced by the projects. 157

Horizon 2020 devoted 32% of its investment to climate-related research, falling short of the 35% target. 158  The situation was different in each of the three pillars: 159 Societal Challenges spent around half of its budget on climate action, while Excellent Science and Industrial Leadership fell well below the 35% target (with 17.5% and 21.2% respectively). 160  An aspect to be considered in this context 161 is the over-estimation of the climate contribution from the bottom-up parts of the programme, due to the difficulty to accurately estimate and quantify in the form of ex-ante figures the extent to which programme parts, which aim for excellence and competitiveness of the R&I system, should also address political priorities regarding societal challenges – when some of them, in particular the ERC, are not geared towards specific thematic needs, but provide a bottom-up funding mechanism open to all thematic domains. The lessons learnt in Horizon 2020 should be used to avoid repeating the same underperformance in Horizon Europe where the 35% target has been maintained.

The EU is the second most frequently acknowledged funding source of the research referenced in the Intergovernmental Panel on Climate Change (IPCC)’s 6th Assessment Cycle reports, after the National Science Foundation of the United States. 162 FP7 and Horizon 2020 supported over 4 500 publications cited by the IPCC, coming from over 1 200 projects. These correspond to about 10% of all references cited in the reports. The IPCC was instrumental in creating a broader, evidence-based consensus on climate-related knowledge with a tangible contribution from research funded by the programme.

On migration research (within Societal Challenge 6), interviewees said the programme is the most important funding source on the topic worldwide and a major factor in the emergence of new networks and the increase in the number of young people entering the field of migration research. Of the 41 migration projects, 12 projects have published a total of 44 publications to date, including two that were highly cited. In contrast, determining the policy 163 influence of the programme was challenging. This is because statements about the influence of projects on political agenda-setting are largely based on anecdotal evidence.

Social Sciences and Humanities in Horizon 2020 were addressed both as a programme part (SC6 Europe in a Changing World: Inclusive, Innovative and Reflective Societies), and as a cross-cutting issue. Evidence shows the value of integrating SSH 164 into R&I projects to achieve a multi-disciplinary and/or inter-disciplinary approach in the programme. 165  Horizon 2020 was the first EU research and innovation framework programme into which SSH was systematically integrated 166 , with over 20% of total call budget allocated to SSH-flagged topics. 167

A lack of a well-established monitoring process and clear definitions acted as barriers to efficient implementation of SSH in addressing societal needs. 168 The work on SSH integration under Horizon 2020 produced some positive results, but also revealed such limitations. 169

SC6 funded a variety of themes spanning from migration and socio-economic inequalities to topics on culture and cultural heritage. 170 SC6 projects were reported to have performed well 171 and, in most instances, delivered high-quality results. They were particularly strong in generating outputs that allowed immediate exploitation. Key outputs included peer-reviewed papers, as well as books, online databases, support for evidence-based policymaking and policy advice – a significant share of SC6 projects produced policy evidence and knowledge that was also taken up by EU institutions, agencies, and other organisations. 172 Actions from the culture and cultural heritage sector proposed market-ready products and services. 173 For example, the project ARCHES 174 developed technological solutions that enable inclusiveness so that people with special needs can access and engage with heritage content in cultural spaces.

SSH integration increased over the lifetime of the programme and the budget allocated to SSH partners grew significantly, especially for the 2015-2018 period. 175 Furthermore, the quality of integration – although still uneven between programme parts and in need of improvement in terms of countries and SSH disciplines – was identified to have improved overall over time. 176  

Notably, frontier research funded by the ERC has supported social sciences and humanities via 1 595 grants, with a total value of EUR 2.8 billion. 177 SSH have also been addressed by calls for proposals in the last phase of Horizon 2020, but interviewees reported that their integration into multidisciplinary projects remained challenging, as they were often perceived as an add-on element in the research design. 178 This was confirmed by respondents in the stakeholder consultation.

In civil security research (under Societal Challenge 7), societal impacts relate both to policy uptake and the fact that security research supported the development of an end-product or service (capability driven approach) 179 in areas such as the fight against crime and terrorism, travel facilitation and border surveillance as well as in the field of disaster resilience 180 . While the contribution of specific projects to policy development and implementation is difficult to assess, the positive contribution of security research has been recognized in strategic EU security policy documents 181 and resulted in specific research and innovation provisions in the new mandates of relevant EU Agencies (Frontex, Europol, eu-LISA).

The LEIT programme parts supported advances with societal relevance in the Key Enabling Technologies.. 182  For example, human-centric approaches were espoused by projects like SHERLOCK (under the Factories of the Future cPPP), which enhanced worker satisfaction by developing user-friendly robotic technologies suited to different production environments. 183 Other applications are relevant for sustainability and decarbonisation: for example, the CORALIS project (under the SPIRE cPPP), aimed at optimising use of energy and resources in three industrial parks 184 . These societal goals were addressed by the nanotechnology area as well: for instance, the Open Innovation testbed NewSkin facilitated industrial uptake of more efficient water-repelling surfaces for solar panels. 185

Contribution to the Sustainable Development Goals (SDGs)

There were no specific requirements within Horizon 2020 to meet targets with respect to the SDGs. Compared to FP7, contributions to SDGs have remained largely stable, with most contributions aligning with SDG 3 (Good health and well-being), SDG 7 (Affordable and clean energy), as well as SDG 13 (Climate action). 186 An analysis of LEIT project outputs suggests strong alignment also with SDG 9 (Resilient infrastructure, sustainable industrialization and innovation) for publications, and with SDG 8 (Inclusive and sustainable economic growth) for innovation outputs. 187

A Horizon 2020 monitoring report 188 discloses the results of a similar analysis 189 conducted by the Commission services: looking at 20 994 projects, accounting for EUR 37.7 billion, the report concluded that up to 84% of the current Horizon 2020 investments relate to at least one SDG. 190  Specifically, the Green Deal Call, which was launched in 2020, included EUR 350 million that directly link with the SDGs. 191  

The Joint Research Centre’ direct research actions – Science for policy

The panel of external experts for the ex post evaluation of the Joint Research Centre (JRC) observed that “the JRC has made important contributions to key policy goals and contributed significantly to the Commission’s working methods and funding instruments.” 192  Horizon 2020-funded research by the Joint Research Centre (JRC, representing 2% of the Horizon 2020 budget) was found to have helped shape selected European policies. At the end of FP7, in 2013, JRC’s monitoring data recorded 248 tangible specific impacts on European policies 193  that resulted from the technical and scientific support it provided. By 2020, JRC had surpassed its target of 330 and recorded 513 impacts 194 (see KPI 22 in Table 3).

An analysis of 39 Horizon 2020 case studies evaluated by external experts 195 found that in 82% 196 of the studies, the JRC was instrumental in shaping and implementing EU policies. For example, the JRC’s support for chemicals policy helped translate risk assessments into regulatory limits for the development of legislation on nanomaterials 197 , endocrine disruptors 198 , tattoo inks and permanent make-up 199 , and chemical mixtures. 200  Similarly, the JRC’s evidence on the costs of climate inaction played a significant role in the developing the European Green Deal by providing evidence of the need for more ambitious actions on climate adaptation and urgent steps toward climate neutrality. 201

In addition, most of the Horizon 2020 case studies (67%) demonstrated long-term societal impacts where the JRC contributed to increasing quality of life and community well-being through consumer protection, reducing costs for firms by fighting fraud, and increasing public awareness about worldwide concerns such as climate change. For example, EU legislation on energy labels and eco-design – whose implementation has been largely supported by the JRC – is estimated to bring energy savings of approximately 230 million tonnes of oil equivalent by 2030. For consumers, this means an average saving of up to EUR 285 per year on their household energy bills. Energy efficiency measures will also create EUR 66 billion in extra revenue for European companies. 202

Promotion of gender equality in Horizon 2020

While gender equality has been increasingly addressed since FP5, Horizon 2020 has encouraged gender balance in research teams at all levels, integrating the gender dimension in the content of research and innovation, as a cross‑cutting issue. 203  

To ensure gender balance in research teams at all levels, project proposals that had a more gender-balanced team were favoured if two projects were given the same evaluation score. 204  Only unstructured data in the call evaluation reports is available relating to how many projects were granted preferential treatment following the measures introduced to foster gender equality across the framework programme.

Source: FP7 - H2020 Gender Dashboard - Gender in EU R&I Programmes | Sheet - Qlik Sense (testa.eu) retrieved on 02/08/2023 for all data except data on the contact person for scientific aspects which is from CORDA (cut-off date 07/07/2023)

Throughout Horizon 2020, 42% of project participants (in any role) were women. This corresponds to a slight increase of 1 percentage point compared to FP7, as shown in Table 9.

Comparing the women main contacts (coordinator) figures across FP7 and Horizon 2020 proves difficult due to changes in the reporting system. In FP7, there was a distinction between scientific and administrative contacts which was discontinued in Horizon 2020:

·Throughout FP7, applicants were requested to indicate both a main “contact person” and, in collaborative projects 205 also a ‘contact person for scientific aspects’.

·In Horizon 2020, the application form probed for a ‘main contact’ (only) instead, which corresponds to a primarily scientific role.

·Moreover, to minimise administrative burden, in the Horizon 2020 programming period, beneficiaries were not obliged to provide information on the gender of said main contact, which further complicates a meaningful comparison 206 .

Therefore, the best way to assess the evolution of women coordinators is to compare the Horizon 2020-reported figure of 23% of women coordinators with the FP7-related figure on women contact persons for scientific aspects in collaborative projects (20%) - showing a slight increase of 3 percentage points.

The share of women researchers in Horizon 2020 decreased by 2 percentage points compared to FP7, whereas at European level the proportion of women among Grade A academic staff did show incremental increase between 2010 and 2018 (see box below). Nevertheless, the absolute number of women researchers increased strongly across the programme as a whole.

During both FP7 and Horizon 2020, women’s participation in roles marked as ‘other than researchers’ (potentially administrative, financial or legal) was also monitored: this share increased from 43% in FP7 to 49% in Horizon 2020 (as shown in Table 9).

Gender equality analysis for mono-beneficiary programmes, where outcomes can be more easily associated to one individual, shows the following results:

·The ERC has had a gender equality plan in place since 2009 and the various actions in this plan have succeeded in raising the success rate of women responding to the ERC’s calls for proposals from 8% in all FP7 calls (as opposed to 11% for men) to 13% in Horizon 2020 calls (and with an equal success rate for men and women). The share of principal investigators who are women remains lower than parity at 28% but did increase by 7 percentage points compared to FP7, as shown in Table 9. This reflects the underrepresentation of women in senior academic staff Europe-wide (see box above).

·The MSCA were successful in removing barriers to the mobility of women researchers 211 : 42% of MSCA fellows under Horizon 2020 were women, which is not only higher than the average percentage of women researchers in the EU (33.8% in 2018) 212 , but also corresponds to a 6 percentage points increase from FP7 (36%). 213

·In the SME instrument, women represented 17% of successful applicants. 214 Data on companies started by women that were supported by the EIC pilot is not available to this evaluation. Targeted initiatives encouraging women to engage in innovative entrepreneurship, such as the EIC Prizes and the EU Prize for Women Innovators, were introduced in 2011 with the aim of highlighting women entrepreneurs behind Europe’s most ground-breaking innovations. The number of Women Innovators prize applications rose over time, reaching 64 in 2016 215 to 197 applicants in 2020. 216  This aspect is important to monitor because the ratio of men to women involved in business creation in 2016-20 was 1.61 217  and women remain disadvantaged in view of raising capital in Europe (men accounted for 91% of all capital raised for deep tech in 2020). 218

To ensure gender balance in decision-making 219 , the targets of 50% for advisory panels and 40% for evaluation panels in terms of the representation of women. 220  This was accompanied by a requirement to have at least three women present on each panel. 221 In Horizon 2020, women constituted 43% of advisory group members, so the target of 50% was not met but this still corresponds to a 10 percentage point increase compared to FP7. At the same time, 42% of evaluation panel members were women, thus surpassing the 40% target (a 6-percentage point increase from FP7). 222  

Incorporating the gender dimension into research and innovation content was measured by tracking the number of ‘gender-flagged’ topics across the programme 223 : the share of projects that explicitly indicated in their project proposals that the gender dimension is reflected has increased in every work programme and overall 23% of projects took the gender dimension into account. Comparatively, gender was most comprehensively addressed in relation to Societal Challenge 1 (Health), where 58% of projects incorporated gender-related issues. 224 This monitoring system was introduced in Horizon 2020 so comparable data from FP7 is not available.

How did international cooperation contribute to the impacts of the programme?

In Horizon 2020, international cooperation was mainstreamed across the programme as a cross-cutting priority, in contrast to FP7, which had dedicated international cooperation schemes with ring-fenced budgets. The ambition of Horizon 2020 was to maintain international cooperation at least at the level of FP7.

International cooperation decreased in the first half of the programme due to the discontinuation of dedicated international cooperation schemes, specific eligibility conditions for certain countries, increased competition and geopolitical challenges. Corrective measures, such as clearer emphasis on international cooperation in work programmes, the introduction of co-funding mechanisms, and more awareness raising and support actions, resulted in a recovery of third country participations in the second half of the programme.

As Table 10 illustrates, the 4.1% participation rate of non-associated third countries in collaborative Horizon 2020 projects slightly exceeds the FP7 baseline of 3.6%. In contrast, the share of EU contributions granted to third country parties decreased from 1.3% to 0.8%, mainly due to the discontinuation of the automatic funding for certain countries. 225

Table 10: International cooperation in collaborative projects

Source: R&I Projects Dashboard, data cut-off date 01/01/23

The extent of international cooperation varied across programme parts and topics:

·Under the Excellent Science pillar (Pillar 1), MSCA 226 and ERC 227  remain attractive to international researchers due to their prestige in academic environments. By supporting researchers of 160 different nationalities and the participation of organisations from 139 countries worldwide, the MSCA constituted the most international component of Horizon 2020, accounting alone for over half of all third country participations in the programme. 

·In contrast, participation of third countries in SEWP actions remains limited. SEWP beneficiaries from widening countries report that they often experience difficulties in attracting international talent to their projects.

·Participation of non-EU organisations in Societal Challenges related to Climate, Food, Health, and Inclusive Societies was above the Horizon 2020 average since these topics were perceived as addressing global challenges. 228  

·Conversely, international partnerships were more limited in the LEIT programme due to geopolitical considerations. A survey carried out among the LEIT programme participants revealed that, in their opinion, the programme did not provide sufficient funding opportunities for international collaboration. 229 Interviewed participants reported that developing more structured cooperation with non-EU partners was hindered by lack of interest from large industrial players, as well as by the broader geopolitical developments after 2017, which raised increasing concerns about Europe’s technological and data sovereignty, for instance in the field of 5G. 230