EUROPEAN COMMISSION

Brussels, 29.1.2024

COM(2024) 49 final

REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL

Ex post evaluation of Horizon 2020, the EU framework programme for research and innovation

{SEC(2024) 52 final} - {SWD(2024) 29 final} - {SWD(2024) 30 final}

REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL

Ex post evaluation of Horizon 2020, the EU framework programme for research and innovation

1.Introduction

Since 1984, Europe has invested in research and innovation (R&I) to give the continent a competitive edge, resilience and technological independence. Horizon 2020, the EU's eighth framework programme for R&I, was a major initiative to boost this long-term investment. Spanning from 2014 to 2020, it was endowed with a budget of EUR 75.6 billion.

The aim of this report is to present the findings of the final evaluation 1 of Horizon 2020, building on the interim evaluation of 2017.

Horizon 2020’s core mission was to drive economic growth and create jobs by synergising R&I and putting a strong emphasis on excellent science, industrial leadership and tackling societal challenges. The programme garnered immense interest, attracting over 1 million individual applications from 177 countries over seven years. It funded almost 35 000 projects involving 40 000 organisations. Given this extensive reach, it would have needed an additional EUR 159 billion to fund all the high-quality proposals submitted.

This evaluation looks into the impacts of Horizon 2020. It also assesses the efficiency of the programme, its coherence with other initiatives, relevance to societal needs and the overall added value it provides the EU.

The evaluation is underpinned by an extensive body of evidence, including:

(I)over 1 000 interviews with beneficiaries, national authorities and implementing bodies;

(II)a survey including both successful and unsuccessful applicants;

(III)a mix of quantitative and qualitative evaluation methods;

(IV)a stakeholder consultation that attracted 1 818 replies.

2.What impact did the programme have?

In line with its foundational objectives, Horizon 2020 was instrumental in nurturing a society and economy rooted in knowledge and innovation. It played a key role in mobilising additional R&I funding and it made a significant contribution to the EU’s target of investing 3% of Gross Domestic Product (GDP) in research and development (R&D) by 2020. Nevertheless, Horizon 2020 investments only accounted for 10% of public R&D expenditure in the EU, with the majority of funding originating from the Member States and regional bodies. By the end of 2020, the EU’s investment in R&D had risen to 2.32% of GDP, a 15% increase since the programme was first launched (2.02%).

The programme also boosted other EU policies and was crucial in the development and functioning of the European Research Area (ERA). It demonstrated relevance in meeting the needs of society, exhibiting agility and flexibility in responding to emergencies such as the Ebola or COVID-19 outbreaks. Horizon 2020 was pivotal in producing influential science, particularly in guiding global action to tackle climate change. The evaluation underscores that it is essential to sustain this action on research and innovation across subsequent framework programmes in order to achieve the desired outcomes. It is worth noting that 41% of Horizon 2020 projects were still running at the time of the final evaluation, indicating that the programme’s impacts are ongoing and continue to yield results.

The 2017 interim evaluation prompted refinements to the latter half of the programme, including initiatives to enhance open science, boost international cooperation, and simplify programme implementation. The interim evaluation also led to several novelties, such as the launch of a pilot for the European Innovation Council for the last years of Horizon 2020 and the introduction of EU Missions under the following programme, Horizon Europe (2021-2027).

2.1    Scientific impact

Horizon 2020 was strategically designed to strengthen Europe’s scientific and technological bedrock by investing in knowledge, skills and infrastructure. These long-term investments are critical for the EU’s current and future ability to lead, react or adapt to dynamic changes in scientific and technological advancement and the ever-changing socio-economic environment.

The programme outperformed its predecessor programme (FP7) on scientific output, as evidenced by the number of scientific publications. At the time of evaluation, beneficiaries had reported over 276 000 peer-reviewed publications, with 18% stemming from projects that had received European Research Council (ERC) grants. This number is still expected to rise as more projects reach completion. Notably, Horizon 2020 publications are twice as cited as the global average, and 3.9% rank within the top 1% of the most cited publications worldwide, surpassing the performance of major international funders, including the US National Science Foundation.

Horizon 2020 made substantial contributions to scientific breakthroughs and advancements in nascent domains of science and technology, particularly in medical sciences, quantum mechanics, chemical engineering and composite materials. Funding for transnational R&I projects enabled significant collaborations that might not have been possible otherwise. 26% of all Horizon 2020 publications are linked to new, rapidly evolving research areas. Horizon 2020 allocated over EUR 13 billion to projects relevant for emerging digital technologies, such as artificial intelligence and quantum computing.

The programme also had a profound impact on knowledge circulation, with 82% of its publications being freely and publicly available online, demonstrating a strong commitment to open access. This figure marks a significant increase from 65% at the start of the programme in 2014 and compares favourably with similar international programmes. The trend to provide open access has influenced 19 Member States to adopt similar policies.

Horizon 2020 was pivotal in diversifying and enhancing researchers’ skills and knowledge. It also improved their career prospects, particularly benefiting early-career researchers such as Marie Skłodowska-Curie Actions (MSCA) fellows, ERC Starting and Consolidator grantees, FET grantees and junior members of teams supported by ‘Spreading Excellence and Widening Participation’ projects. With nearly 50 000 researchers supported in cross-sector and cross-country mobility, the programme is on track to achieve its targets on researcher mobility, mainly via MSCA. Erasmus+ and Marie Skłodowska-Curie Actions were exemplary in making synergies between EU programmes, promoting mobility, training and career development for students and researchers.

In addition, the Horizon 2020 programme enabled the EU to develop and upgrade large-scale research infrastructure at both European and global level. Over 24 000 researchers and organisations gained access to these infrastructures, expanding opportunities for collaborative work and scientific advancements. The Leadership in Industrial Technologies (LEIT) programme facilitated access to technology infrastructure such as European Digital Innovation Hubs and open innovation test beds, enabling companies to test innovations in realistic conditions. Another important development was the deployment of common research infrastructure under the roadmap for the European Strategy Forum on Research Infrastructures. These are noteworthy achievements, though the evaluation suggests that greater synergies could be made between EU, national and regional programmes for research infrastructure, in particular to ensure their operations are sustainable.

2.2    Societal impact

Horizon 2020 bolstered research and innovation with the aim of tackling key societal challenges including health, food security, energy, transport, environmental sustainability, climate action, inclusive societies and security. The programme’s contributions to these challenges are significant not just in scope but also in depth, impacting multiple facets of society and the global environment.

A prime example is Horizon 2020’s crucial role in advancing our understanding of climate change. Its investments, building on the foundations laid by FP7, have been influential, with 10% of all scientific publications cited by the UN Intergovernmental Panel on Climate Change (IPCC) originating from these two programmes.

Horizon 2020 has also been instrumental in supporting the development of practical solutions for climate action. A prime example is the progress made in alternative and low-emission fuels. Pioneering work under FP7 and Horizon 2020 propelled the EU to the forefront of testing and deploying hydrogen fuel cells in buses, and is driving innovation on low-emission air travel anticipated for 2030.

In terms of investments in sustainable development, Horizon 2020 exceeded its target of 60%, allocating 64.4% of its total budget to these initiatives. However, despite the significant contributions to climate action, the programme fell short of its 35% budgetary target for climate-specific topics, concluding with a final allocation of 32%. This underscores the importance of continuing the focus on and monitoring of similar targets under Horizon Europe to ensure that future initiatives align more closely with the set objectives.

EU society faces a wide range of complex challenges, for which R&I activities involving social sciences and humanities disciplines, such as sociology, economics, psychology, political science, history, and cultural sciences, are essential. These disciplines are key to gain a deeper understanding of and effective response to complex societal issues. Compared to previous programmes, Horizon 2020 significantly increased the role played by social sciences and humanities disciplines, allocating over 20% of its budget to related topics, evidence of its commitment to interdisciplinary research.

However, the evaluation revealed that the level of integration of social sciences and humanities was uneven across different parts of the programme. This echoes the suggestions for improvement noted in the interim evaluation of Horizon 2020. In response, the Horizon Europe programme was adjusted to bring in new measures to integrate social sciences and humanities in a more robust manner. Proposals now receive lower evaluation scores if they fail to sufficiently incorporate social sciences and humanities disciplines where it was identified as relevant.

Another key challenge identified under Horizon 2020 is the long timespan required for projects to demonstrate their societal impacts, which complicates monitoring and evaluation processes. Experience has shown that monitoring arrangements were too narrow in scope, limiting the ability to capture the full spectrum of societal impacts. In addition, the design of indicators was found to be weak, hindering the effective assessment of outcomes. To remedy these shortcomings, concerted efforts are made to improve data availability and improve the monitoring framework under Horizon Europe.

2.3    Economic impact

Horizon 2020 made a significant contribution to the European economy, not only by stimulating employment and economic output growth, but also by leveraging other funds and raising the productivity of the companies involved. It also led to the development of thousands of innovation outputs. The list of these results is not yet final, as a significant share of the projects supported by the programme was not yet completed at the time of evaluation and continue to yield results.

On top of its nominal budget, Horizon 2020 contributed to increasing R&D spending in Europe by attracting co-investment from both, the public and private sectors. For each euro spent from the Horizon 2020 budget, project participants brought in EUR 0.23 of their own resources to run the projects. The programme was especially effective in the private for-profit sector, where for every euro of Horizon 2020 funding, project participants invested an additional EUR 0.57. The highest degree of financial leverage was achieved in European partnerships: in joint undertakings, private partners’ contributions with resources (in cash or in kind) more than doubled or even tripled the volume of EU funding.

In addition, Horizon 2020 grants had a positive effect on revenue and employment for participating companies, compared to firms with similar characteristics that did not receive funding.

Horizon 2020 has significantly contributed to intellectual property rights (IPR) developments, with its beneficiaries declaring close to 4 000 IPR applications, three quarters of which are for patents, followed by 12% for trademarks. Given the often lengthy patenting process, Horizon 2020 IPR figures are expected to increase significantly even after the programme ends. A long-term analysis has shown that patents stemming from FP7 not only exceed the global average in economic value but also have a clear tendency to be interdisciplinary. In addition, some 40% of patents self-declared by participants of the Leadership in Enabling and Industrial Technologies (LEIT) programme part have contributed to key enabling technologies, including photonics, and to micro- and nanoelectronics. The Societal Challenges pillar has generated some 20% of all innovations under Horizon 2020 and the Excellent Science pillar has contributed 31%, albeit mostly at a lower level of technological readiness.

The Horizon Innovation Radar, a tool for identifying high-potential innovation in the programme, suggests that Horizon 2020 funded potentially ground-breaking technological innovations. The most ready-to-market innovations have emerged from the Industrial Leadership pillar, particularly under the LEIT projects. These projects have shown a higher propensity for market-ready innovations, especially in areas such as emerging digital technologies, high-performance computing and advanced materials.

The interim evaluation of Horizon 2020 identified a notable gap in venture and growth capital in the EU to scale up innovation. To help bridge this gap, a pilot started to run the European Innovation Council (EIC) in the last three years of Horizon 2020. Early indications show that the EIC pilot had a positive impact on the turnover and staffing levels of its beneficiaries. It also tackled a critical funding gap in high-risk areas where limited alternatives are available at national and regional levels. In comparison to countries with similar support structures, the EIC stands out as the only initiative offering enough breadth and funding to support deep-tech companies.

Organisations funded by the programme showed a higher capacity to attract more risk capital than non-funded applicants, though estimates vary. In particular, small and medium-sized enterprises (SMEs) participating in the LEIT part of the programme were successful in drawing equity investments four times higher than the EU contribution they received. The InnovFin financial facility, managed by the European Investment Bank group and supported by EUR 3.7 billion in Horizon 2020 funding, was successful in mobilising private finance for innovative start-ups and other ecosystem players in Europe. It leveraged EUR 77.5 billion in debt and equity for over 38 000 organisations, well above its targets, and fostered the development of venture capital ecosystems and networks.

Although Horizon 2020 made progress in bridging the gap between high-quality European research and market innovation, it has not fully closed this long-standing gap. Measures tracking the spread of innovation suggest that the EU improved its performance during the Horizon 2020 implementation period but still trails behind its main international competitors on this aspect. Horizon Europe, particularly through the EIC, renewed the effort to bring innovations to the market.

3.Who has benefited from programme funding?

Horizon 2020 has had a significant impact on the research and innovation landscape, benefiting a diverse range of participants, from scientists and researchers working in higher education institutions to research organisations and private-for-profit entities such as small and large businesses.

The programme launched over 1 000 calls for proposals, attracting over 285 000 eligible project proposals – double the number received by the predecessor programme, FP7. This surge in interest highlights the programme’s appeal and relevance. However, only 35 426 projects were funded, resulting in a 12% overall success rate. Notably, 74% of proposals assessed as high quality by independent experts could not be funded due to budget constraints. Horizon 2020, with a budget of EUR 75.6 billion, would have needed an additional EUR 159 billion to fund all high-quality proposals.

To give the excellent unfunded proposals a greater chance to secure support at national or regional levels, 20 890 of almost 100 000 high quality proposals not selected for funding received a Seal of Excellence  certificate. Initially applicable only to MSCA or EIC proposals, this EU quality label was later extended under Horizon Europe to other programme parts. An analysis from the European Court of Auditors revealed that in three countries, 26% SME Instrument proposals that received this Seal secured subsequent funding under European Structural Funds. However, Member States’ lack of access to information on proposals awarded with Seals of Excellence was identified as a barrier to maximising their impact. 

In terms of funding distribution, collaborative projects accounted for 78% of the funding, involving an average of 11 participants in nearly 15 000 projects. Single beneficiary grants accounted for 22% of the funding but 59% of all grants, primarily for ERC, MSCA and SME instrument. The average grant size under Horizon 2020 increased to EUR 2.3 million from EUR 1.8 million under FP7, which constitutes an increase also in real terms.

Regarding types of beneficiaries, higher education institutions received the largest share of funding (40%), followed by private-for-profit organisations (28%) and research organisations (25%). SMEs received 17% of the funding, amounting to EUR 11.4 billion. Well-established higher education institutions and research organisations received a high share of the funding, showing a degree of concentration, but less than under FP7. The top 100 beneficiaries received 32% of funds (compared to 34% under FP7). However, the programme also attracted newcomers (organisations that had not participated in FP7), in particular smaller private-for-profit entities. Newcomers received 19% of Horizon 2020 funds, rising to 50% when considering only funding for private companies across the programme. For joint undertakings, 19% of all funding went to newcomers.

Gender equality improved under Horizon 2020, with the share of women in evaluation panels reaching 42%, above the 40% target. However, the share of women in scientific advisory panels and as researchers in projects remained below the 50% target, at 43% and 23% respectively. In response, Horizon Europe was adjusted to integrate the gender dimension across the entire programme.

The programme’s global appeal is evident from the spread of applications from 177 countries. Half of all funding went to just four countries (Germany, UK, France and Spain). However, smaller countries such as Estonia, Greece, Cyprus, and Latvia saw an impressive performance when comparing Horizon 2020 funding to their gross domestic expenditure on R&D.

Widening countries received an 8% share of the total EU contribution, up slightly from FP7. Although this may seem moderate, all these countries except two have increased their participation in the programme. The evaluation identified several challenges for these countries, including limited capacity to manage international R&I projects, brain drain, weak national support systems and the availability of funding alternatives. In response, the Horizon Europe budget for extending country participation tripled to reach 3% and brought in several measures to enhance their involvement, including strengthening the National Contact Point system and offering proposal pre-checks and 'brain circulation' grants.

4.What was the value of the programme for the EU?

Horizon 2020 significantly enhanced the scope and quality of research and innovation in Europe, achieving impacts that extend far beyond what could have been achieved at national or regional level. It supported larger-scale, more complex and more ambitious R&I activities than would have been possible without EU support. It accelerated the development of solutions to pressing global challenges by pooling efforts and resources from across Europe. This was evident in the difficulty faced by unsuccessful applicants, many of whom were unable to implement their projects or had to make significant modifications, primarily due to the lack of alternative funding sources at national or regional level.

A key strength of Horizon 2020 was its multidisciplinary collaboration and pan-European cooperation on R&I. This approach proved effective in consolidating expertise, skills and resources from multiple countries, creating a critical mass to raise the quality of research and innovation outputs. The competitive nature of the EU-wide funding process further enhanced quality, ensuring that research was conducted in areas of significant relevance to European society.

As detailed in the section on economic impact, the programme successfully leveraged both private and public funding for R&I. It is estimated that every euro invested in Horizon 2020 will yield five euro of benefits for EU citizens by 2040. This investment contributed to an increase in the percentage of GDP allocated to R&D, further consolidating the European Research Area.

The long-term value of investing in research and innovation through an EU-level framework programme is clear in setting the scientific and technological foundations necessary for the EU’s future preparedness and strategic autonomy in responding to crises. This was particularly evident in the programme’s fast and effective response to the COVID-19 outbreak, underpinned by long-standing investments in mRNA research.

5.How efficient was Horizon 2020?

Horizon 2020 has demonstrated substantial value-for-money to European society. In terms of economic impact, for every euro spent on the programme (in programme costs and costs to applicants), it is estimated to yield approximately five euro in benefits to EU citizens, as measured through its impact on GDP, up to the year 2040.

A number of simplification measures were effective in reducing the administrative burden for applicants and beneficiaries. Notable improvements include the use of electronic signatures and the annotated model grant agreement. These changes helped accelerate the process to award grants, improved error rates and administrative expenditure that performed well against benchmarks. Key supporting factors mentioned were the new electronic grant management workflow and the elimination of the negotiation stage. However, the evaluation suggests that further tightening the time-to-grant target might not be necessary as it could inadvertently increase financial error risks.

Despite these advancements, the evaluation does not present an overall positive picture regarding the programme’s error rate. The European Court of Auditors pointed out that, particularly in operational expenditure and personnel costs, the level of error remains high and often avoidable.

Looking ahead, there is scope to improve the efficiency of the EU framework programme. Many stakeholders have indicated that participating in Horizon 2020 requires more effort than for other research and innovation funding programmes. This is significant given the programme’s relatively low success rate, as it means that a considerable share of the application cost represents a net loss to EU society. Any effective measure that reduces these costs has a strong potential to improve programme efficiency.

6.conclusions and Lessons Learned

The evaluation concludes that Horizon 2020 significantly contributed to building an EU society and economy rooted in knowledge and innovation. By leveraging additional R&I funding, it supported not only the implementation of other EU policies but also played a crucial role in achieving and maintaining the European Research Area. Horizon 2020’s relevance to societal needs was evident in its rapid and flexible response to emergency situations such as the Ebola and COVID-19 outbreaks, and its influential role in global efforts to tackle climate change. The evaluation underscores the need to sustain action across multiple framework programmes to achieve these outcomes.

While Horizon 2020 met some of its targets or key performance indicators, it did not achieve all of them. This partial success can be attributed to the inherent nature of R&I investments, which often require a lengthy period to yield usable results. Many projects are still ongoing and there were also shortcomings in the initial setup of the programme’s indicators. Despite these challenges, the evaluation confirms that the results produced with Horizon 2020 funding are of high value.

The interim evaluation of Horizon 2020 led to some significant adjustments in the latter half of the programme. New measures to increase open science have borne fruit and the level of international participation was maintained. Further improvements, such as promoting women’s participation, better integrating social sciences and humanities and reducing the administrative burden have been carried forward and reinforced under Horizon Europe. In addition, the monitoring and evaluation framework has been revised to better track impact over time. The effectiveness of these measures will be closely examined in the interim evaluation of Horizon Europe.

This final evaluation of Horizon 2020 highlighted several key areas for further improvement, providing insights that will feed into future enhancements.

·Broadening participation. There is scope to broaden participation in the programme. It would involve engaging with non-traditional players from multiple sectors, scientific disciplines and countries. While national reforms of R&I systems can influence readiness for European-level project participation, the programme itself can be improved by greater simplification, visibility and accessibility.

·Further simplification needed. The programme can benefit from a targeted use of the two-stage application process, especially in areas with low success rates and a high volume of unsuccessful applicants. Extending the use of the Seal of Excellence certification scheme could also enable more applications to be reused for other programmes, reducing wasted effort. There is also further potential for simplification in extending the monitored use of lump-sum funding, as well as in improving outreach, information dissemination, and the user experience of programme tools.

·Dissemination, exploitation and deployment of results. The process of disseminating, exploiting and deploying project results has been uneven and requires more attention. Horizon Europe encourages applicants to give greater thought to the pathway to impact in their applications. Improvements are also needed to ensure the visibility, spread and practical use of project results to unlock broader economic and societal benefits.

·Supporting women in research and innovation. Despite efforts, it remains a challenge to achieve gender balance in research, entrepreneurship and innovation. Stronger measures are needed to support women researchers, entrepreneurs and innovators both Europe-wide or within the framework programme.

·Unlocking more synergies with other initiatives. Synergies with other EU, national and regional initiatives could be strengthened, particularly to support the uptake and use of project results. This includes better alignment to ensure the smooth operation of research infrastructures.

7.Next steps

The insights and key conclusions drawn from this final evaluation of Horizon 2020 are set to play a crucial role not only in shaping the ongoing implementation of Horizon Europe, but also in influencing policy development for future research and innovation initiatives. This will ensure that the lessons learned from Horizon 2020 are effectively integrated into current and future programmes to further improve their efficiency, relevance and impact on Europeans.

(1)  Under Article 32(4) of Regulation (EU) No 1291/2013 of the European Parliament and of the Council of 11 December 2013 establishing Horizon 2020 - the Framework Programme for Research and Innovation (2014-2020), complemented by Regulation 1290/2013 and by Council Decision 2013/743.