Science-based public policies with the help of AI

Software that supports expert analysts, based on artificial intelligence and capable of examining and summarizing the best studies available worldwide in the field of public policy, to provide decision-makers with useful information to make the best possible choices every time, to the full benefit of citizens.

With this goal in mind, researchers from the Center for Digital Society and IRVAPP (Institute for Evaluative Research on Public Policies) of Fondazione Bruno Kessler in Trento have pooled their respective expertise in the course of the PolicyAId project funded by the National Recovery and Resilience Plan (NRRP).

“Over the years, the international evaluation research community,” explains FBK-IRVAPP researcher Dominique Cappelletti, “has produced tens of thousands of impact evaluation studies, which rigorously and scientifically measure whether a given intervention produces the desired effects. We know, for example, which school support programs reduce dropout rates, which active labor policies truly increase employment, and which welfare measures have lasting effects on family well-being. This wealth of knowledge could inform the decisions of those in power. This is the principle underlying evidence-based policymaking: drawing on the most rigorous scientific evidence available to guide the design and selection of the most effective measures.”

The main problem remains access to information: the scientific literature on evaluation is vast, fragmented, and published mainly in English in thousands of academic journals, most of which are only accessible through subscriptions. For a policymaker or public official, navigating this vast landscape requires specialized technical expertise that is not always available. This is where public policy analysts come in, as they have to review large amounts of literature and, above all, select the most methodologically and empirically sound contributions. However, this is a time-consuming task, sometimes taking several months for a single analysis, whereas public decision-makers often need quick answers.

“When we turn to generative language models to increase the productivity of analysis processes,” points out Marco Rovera, a researcher at FBK’s Center for Digital Society, “an additional critical issue arises: generative language models are very versatile tools, but they do not have direct access to primary sources or the most recent scientific publications, and above all, they do not have access to the best practices and methodological expertise of expert analysts, with potentially unreliable outcomes for high-impact decisions.”

This need gave rise to the collaboration between researchers from different fields within Fondazione Bruno Kessler. Public policy evaluation experts have translated their way of reading a scientific article into a checklist of specific questions for each of the categories of methods established in the literature, while computational linguistics researchers have developed a system that translates these questions into a controlled conversation with a language model (LLM).

“The result,” explains FBK-IRVAPP researcher Sergiu Burlacu, “is a detailed analysis of both the method used in a scientific paper and the quality of its application in the specific context. Analysts can thus, for example, sort articles on a topic of interest and base their analysis only on those that offer the most credible and robust evidence. The system is designed to support the analyst, without replacing them, and allows them to trace the chain of responses that support the final assessment of an article.”

In addition to evaluation, the system provides the analyst with tools such as summaries of individual scientific articles and the creation of reports on specific topics based on a set of selected articles. Summaries and reports can always be edited by analysts. “Our intention,” explains Marco Rovera, “is for this design to provide an alternative to the black box model that often characterizes the use of artificial intelligence.”

Recently, the collaboration led to the scientific article “Research Design Tracking and Assessment for the Social Sciences,” authored by FBK Digital Society researcher Marco Rovera and FBK-IRVAPP researchers Sergiu Burlacu, Dominique Cappelletti, Alessio Tomelleri, Sonia Marzadro, Martina Bazzoli, Annalisa Tassi, and Jessica Gagete-Miranda. The article formally defines this task, systematically measures its performance using different technologies and language models, and compares expert assessments with those of the AI system.

The paper was accepted and presented at the 2026 edition of the “Empirical Methods in Natural Language Processing (EMNLP)” conference, held this fall in Budapest, one of the leading international venues for the Computational Linguistics community.

“The tool,” emphasizes Marco Rovera, “is currently being used on an experimental basis by a small group of analysts and policymakers within AMELIA, the digital platform developed by the GRINS Foundation to facilitate access to scientific evidence for public policies.”

The research was made possible thanks to funding from the GRINS (Growing Resilient, INclusive and Sustainable) partnership, under Investment 1.3 of the National Recovery and Resilience Plan (NRRP), funded by the European Union – NextGenerationEU, Mission 4 “Education and Research” – Component 2 “From Research to Enterprise” of the Ministry of University and Research (MUR).