Small nations look for ways to measure research impact

When it comes to research and innovation, a small country can face particular challenges. That's why a group of representatives from six small and technologically advanced economies – including Ireland – met in Dublin last week to discuss how to measure the impacts of research investment.

As part of the Small Advanced Nations Initiative, the meeting involved participants from New Zealand, Singapore, Israel, Denmark, Finland and Ireland having discussions over two days at Farmleigh in the Phoenix Park.

"The common denominator is that all of the countries are small, with a population of less than 10 million, and they are all technologically advanced," explains Prof Mark Ferguson, director general of Science Foundation Ireland, which hosted the event.

“This metrics workshop is really about asking how we can benchmark against each other, learn from each other and how we can tell whether the research we are funding is impactful and what that relates to for economic policy.”

Impact of choices
Small, technologically advanced nations face similar constraints around research policy and they have to pick the right options for investment, according to New Zealand's chief science advisor Sir Peter Gluckman, who was part of the working group at Farmleigh. "We are all struggling with identical problems even though we have got different histories and different places where we come from," he says.

“We all think that the constraints on the science system in a small country are different to a big country, and just looking at a big country’s science statistics to work out what you should do may not be as useful as benchmarking against [another small country].”

But how do you measure the impact of research investment? Counting academic papers can tell you about the “short-term productivity” of science, says Sir Gluckman, but there are wider impacts to consider.

"What the population and their representatives are interested in is whether investing one to two percent of the country's GDP in this area as opposed to another area is making a real difference to the economy, to our society – and that is not an easy question," he says. "What we are trying to do here is make it easier for everybody – the taxpayer, the research funder and the researcher themselves – to understand how valuable the research enterprise is and to make judgements as to where to find more money being spent or spent in different ways."

Impact is not one-size fits all
Small countries face common issues such as the mobility of researchers, the globalisation of enterprises, how to attract foreign multinational companies and how to get more impact out of limited investments, according to Thomas Alslev Christensen, who is head of department at Denmark's Ministry of Science, Innovation and Higher Education.

A big topic of discussion at the Farmleigh meeting was how to measure the non-economic and economic impacts of research in small countries, he says. “We focused on two areas, non-economic impacts – how do we find solutions to challenges in society – and the other aspect is how do we see the impact in terms of productivity, employment and export growth rates.”

The working group also looked at data about research collaboration. “What we see from our impact studies is that researchers who are collaborating with foreign researchers or with companies, they have a higher production of publications, they are earning more money, so they are doing much better,” says Christensen.

Portfolio of research
Countries may vary on how they weight different kinds of impact in their science system, adds Sir Gluckman, and it's important to have a portfolio of research that includes the fundamental and applied.

"You cannot have, in my judgement, a large, sustained critical mass of science in an applied area without having a good foundation of basic science in the same relevant areas to fuel it – otherwise you will run out of gas at some point," he says. "One of the tricks, at least in our thinking in New Zealand about bigger research centres, is to make sure there is a portfolio of research that will not only answer in the short term but will renew and self-renew and develop further."

Future metrics
While the discussions are still at an early stage, Prof Ferguson hopes they will ultimately generate a set of metrics that will allow small advanced economies to compare performances and impacts in science. "Where we are aiming to go is if we could all agree on a core set of metrics," he says. "Each country will have some bespoke things that are appropriate for that country, but if we can have a common core it will allow us to do much more realistic benchmarking."