Author Archives: David Janků

Technological stagnation

17,603 Replies

There are several voices arguing that since the approx 1950s the world is going through a technological stagnation (compared to previous two centuries), with technological progress increasingly slowing down. One of the good summaries of this debate and arguments I’ve read is in this blog post by Jason Crawford. Before I’ve read his summary, I’ve been mostly inclined to believe what he calls the quantitative case: that GDP and TFP growth has been slowing down since approx the 1970s (although he makes a good point that I would like to explore further some day about shift from manufacturing to services being a potential source of systematic underestimation of growth). However, after reading his “qualitative case”, I’ve actually updated to thinking the stagnation might not be there to such a large extent as I thought (contrary to his conclusion). Here is what makes me think so:

In his qualitative case, he categorizes technology/industry in six major categories:

Manufacturing & construction; Agriculture; Energy; Transportation; Information; Medicine.
He argues that in the past we have seen simultaneous revolutions (or disrupting innovations) in more of these categories, whereas since the 1970s we have seen the same level of disruptiveness only in one category – information, via computers, internet, smartphones, etc.. I don’t think we have actually seen less disruptive innovations than in the past – I think there are similarly disruptive technologies in most categories. Jason then goes on to describe what were the closest-to-being-disruptive innovations in each category since the 1970s but for some reason didn’t come to full fruition. I will comment on his views and suggest what could be the revolutionary technologies which makes me more optimistic about the current state of technological progress.


Recent disruptive technologies

Manufacturing: In materials, carbon nanotubes and other nanomaterials are still mostly a research project, and we still have no material to build a space elevator or a space pier. As for processes, atomically precise manufacturing is even more science-fiction than flying cars.

I would argue that perhaps robotization might be a big enough deal. It has been around for some time already, but maybe it is taking off in its most useful form just now. Anyway, robotization sounds like something that would be tracked by common (GDP and TFP) economic indicators, so if that was a big deal already, it would show up (suggesting it wasn’t and only hope left is that it will become a big deal in the future)    

Agriculture: 

Jason doesn’t talk about this category in his post, but here is my take: Green Revolution in general (and genetically modified crops in particular) might have substantial effects on food production and allowing population growth. This might have been missed by the per capita economic indicators, but perhaps would show in the total GDP statistics. As a counter-argument, data from Bloom et al. (2020) on crop yields show no disruptive growth, but rather constant diminishing returns – if green revolution (or genetically modified crops) were such a big deal, there would probably be some disruptive growth shown.
When looking at horizons, perhaps lab grown food (like cultured meat) will be revolutionary enough to significantly change how we create food (and how we treat animals), which perhaps will show in the economic indicators. Vertical farming could be another innovation  with large impacts if scalable, although perhaps not as significant as lab grown food.

Energy: The most obvious stunted technology is nuclear power. In the 1950s, everyone expected a nuclear future. Today, nuclear supplies less than 20% of US electricity and only about 8% of its total energy (and about half those figures in the world at large). Arguably, we should have had nuclear homes, cars and batteries by now.

I agree that nuclear power is the biggest deal. It seems that the problem here is not the problem of invention (the technology itself is quite good) but a problem of distribution/uptake. Why there was (or still is) a problem of distribution/uptake of nuclear energy seems like an important question for progress studies scholars to take a look at.
Another group of technologies on the horizon that might be a big deal are renewable sources of energy. It’s not clear how much this will show up in economic indicators, but the main value it promises is significantly less environmental damage.   

Transportation: In 1969, Apollo 11 landed on the Moon and Concorde took its first supersonic test flight. But they were not followed by a thriving space transportation industry or broadly available supersonic passenger travel. The last Apollo mission flew in 1972, a mere three years later. Concorde was only ever available as a luxury for the elite, was never highly profitable, and was shut down in 2003, after less than thirty years in service. Meanwhile, passenger travel speeds are unchanged over 50 years (actually slightly reduced). And of course, flying cars are still the stuff of science fiction. Self-driving cars may be just around the corner, but haven’t arrived yet.

I think expecting space transportation or supersonic passenger travel might be maybe too high expectations, compared to going from horses -> locomotives and then locomotives -> cars. When talking about passenger travel, the disruptive technology I would mention here that originated in the first half but was distributed in the second half of the 20th century is flying. It seems like air travel has taken off a lot and this seems a comparably large change to locomotives -> cars (in terms of increased mobility, even more in terms of speed). However, again, we would expect this type of technology to show in the standard economic indicators and given that these indicators still point to overall stagnation, perhaps flying was not so big a deal after all. Perhaps economic benefits from increased mobility have hit diminishing returns with cars and flying was this not that important. When talking about cargo, containerization in shipping which was mentioned elsewhere in Jason’s post seems like a big deal.

When looking at a horizon, I think self-driving cars and hyperloop are two potentially very disruptive technologies that will hopefully arrive soon. Again, the question of Why have self-driving cars not arrived yet? and How long will it take for hyperloop to be developed? seem interesting to me from a perspective of progress studies. Anyway, even when these technologies arrive, it’s questionable whether they will be reflected in economic indicators – if there are diminishing returns to economic benefits from mobility, hyperloop will not bring much value this way, even though it might be much faster than airplanes and perhaps more environmentally friendly. Self driving cars will perhaps free up people’s time which could be devoted to leisure or work and will cause more people to survive through decrease in car crashes – it’s not obvious this is going to show up.

Information 

This is the category where Jason seems the only revolutions of the second half of 20th century which the other innovations are not comparable to (i.e. are less impactful): computers, internet and smartphones and other communication technologies which have completely transformed the information processing and communication.  
Moreover, disruptive changes from progress in artificial intelligence seem to be on the horizon (if not happening already). 

Medicine: Cancer and heart disease are still the top causes of death. Solving even one of these, the way we have mostly solved infectious disease and vitamin deficiencies, would have counted as a major breakthrough. Genetic engineering, again, has shown a few excellent early results, but hasn’t yet transformed medicine.

Life expectancy increased a lot since WW2, so we have to be doing something right. Perhaps this is driven by mitigating communicable diseases around the world a lot.
Anyway, looking at the horizon might also bring some more hope: perhaps we will see some progress in life extension (or anti-ageing) efforts; perhaps some progress in genetic medicine as mentioned, and if technologies allow it, maybe we will finally see some instances of personalized medicine. However, I don’t have any sense how far these innovations are. 

Timescales: long term stagnation or short term dip?

To see whether we are in the stagnation, it’s useful to compare the same timescales. If we think about the recent history in terms of 3 industrial revolutions, it could be (as Jason suggests):

1st Industrial Revolution: from the 1700s through the mid-1800s (150 years)
2nd Industrial Revolution: from the mid-1800s to the mid-1900s (100 years)
3rd Industrial Revolution: from the mid-1900s until now (70 years)

If we don’t assume that the length of these revolutions is systematically decreasing (e.g. because the invention and distribution cycles are getting shorter/more efficient), maybe we should wait a little bit before making judgements about stagnation, especially given that there are number of promising technologies on the horizon (as mentioned above). 

Given that this period we look at (data often focusing on 1970-2010, i.e. 40 years) is pretty short from a historical point of view, we may also wonder whether this stagnation is not just an oscillation if we look at the bigger picture. Perhaps there were some oscillations like these in the previous industrial revolutions as well (I’m yet to find some data about this, if you know about some, please let me know), progress is rarely smooth.        

Conclusion

Overall I’ve become less inclined to believe in technological stagnation after realizing what disruptive technologies has humanity recently created or is in the process of creating (btw doing a qualitative best case analysis might not be such a shaky approach as it might sound – it might actually be even more explanatory if we live in a power law world rather than Gaussian/linear world). Anyway, I guess the quantitative case still holds and it is more likely than not (let’s say 65 %) that we are experiencing slow down compared to 100 years ago. However, this slow down might be only common fluctuation (caused e.g. by more complexity involved in creating new disruptive technologies or cultural adaptation to some of those new technologies) before high rise due to new disruptive technologies coming to fruition. The slow down doesn’t seem to be so bad (we have created and continue creating new promising technologies) and has not been here for so long yet.       

How does scientific progress work – distributions of impact

4,235 Replies

Is science advanced by a small group of scientists or is it more of a large scale collaboration? Answer to this question is pretty crucial, because it largely influences what kind of interventions are best to improve science and knowledge creation. For example, if we found that science is advanced by a small group of scientists rather than by every scientist in a large scale collaboration, then we might further think about how to grow this small group or how to empower them and perhaps let off the others if growing is not possible.   

Small group vs large scale collaboration

One of the good treatments of this question I’ve read so far is this post by José Luis Ricón. My aim here is to quickly summarize its findings and then generate some more unexplored questions that might fill in the blank spaces on the map or might help to make the answers more actionable.

This debate could be framed as Newton vs Ortega hypothesis, where Newton hypothesis is a view that top researchers build on the work of other top researchers and most of the other (average) science doesn’t really matter, whereas the Ortega hypothesis is a view that large numbers of average scientists contribute substantially to the advance of science through their research. 

This post by José Luis Ricón looks at some evidence for both of these hypotheses and concludes that Newton hypothesis is likely correct and Ortega hypothesis false. The strongest evidence in favor of this conclusion seems to be a study by Cole & Cole (1972) and later by Bornmann et al. (2010), which show that most cited papers cite mainly other most cited papers and only rarely cite less cited papers. At the same time, other less cited papers also cite mainly the most cited papers and to a lesser extent (but still more than the most cited papers) cite the less cited papers. See this in a figure from Bornmann et al. (2010):

image-20210112094455330

 

The distribution of impact across researchers also seems to be highly skewed, which excludes another potential version of Ortega hypothesis: that everyone tries to get to the frontier and throughout their career, many people succeed, push the frontier a little bit forward but then fall back and stay behind again for the rest of their career – if that was the case, the distribution of citations across researchers would be less skewed and more uniform than it actually seems to be.

Why is that?

We might ask: Why is that? Why do some researchers produce max more impact than others? Here are couple potential answers:

1) It’s genetic – people at the frontier have some superficial combination of genetic traits that are impossible to learn (e.g. combination of very high intelligence, conscientiousness, need for cognition,..).
But if that was true, then we can argue that the raw population numbers increased a lot during the second half of the 20th century and we should see a proportionately large increase in the total number of positions at the frontier (because more people with this superior mix of skills have been born) – did we see that?

2) It’s behavioural – people at the frontier do something differently than all the others and this can be taught/learnt. Maybe it is the choice of which papers to read, how to distribute their attention, how to ask questions that yield more fruitful answers, how to design the research so it is robust, etc… Evidence in this direction could be that top scientists tend to be trained by other top scientists, as observed on Nobel laureates, National Academy members, and other eminent scientists (Cole & Cole, 1972). Similarly, early co-authorship with top scientists predicts success in academic careers (Li, Aste, Caccioli & Livan, 2019). Maybe there is something hidden in the training process which causes researchers to become excellent. This would be good news, because as long as we can figure out what this is, we can scale it up and train way more researchers in this way, adding many more excellent researchers to the pool.

3) It’s systematic – the limited number of positions must be limited because of some feature of the system. For example, attention (expressed via citations) is a limited resource and always just 5 people can get the full attention, regardless of how many other people there are, whether it’s 5 or 500. Knowledge or quality of work doesn’t seem to be limited in this way (e.g. it is permissible to imagine 500 scientists doing the highest quality work at the same time instead of just 5 scientists), but attention is and that may create dynamics as described above. Moreover in deciding where are we going to allocate our attention, we might use heuristics like “following the work and students of those who already proved to be impactful” which would be in line with the notion that top scientists tend to be trained by other top scientists and early co-authorship with top scientists predicts success in academic careers – only this time this relationship would appear not because of the “inner quality” of these apprentices, but rather because of Matthew effect (see more discussion below).

Another version of this is that there is some unique combination of resources (e.g. concentration of talent) that makes it much more likely scientists to rise to the top. This would explain why top scientists often come from the same few universities. Good thing is, that this might be perhaps easier to replicate and thus increase the total number of top scientists.     

Alternative explanation: Matthew effect

The alternative explanation of the Bornmann et al. (2010) data above could be the existence of Matthew effect alas extremizing of the distribution of paper quality via self-reinforcing mechanisms resulting in more skewed distribution not tracking the differences in quality accurately (via Turner & Chubin, 1976). For this, Ricón mentions several studies studying Matthew effect via effects of prestige on the number of citations a given author gets. One way to study this is to look at how the number of citations of an author’s previous work changes after they receive a prestigious award/grant, relative to the control group (see Azoulay et al., 2013; and relatedly Li et al., 2020). Another, perhaps a bit weaker, method, is trying to explicitly control for prestige as a mediator in the relationship between higher scores given by the funder in the grant proposal assessment and higher citation performance later (see Li & Agha, 2015). These studies conclude that the effects of prestige are either very weak or non existent, and thus the main differences in number of citations are likely caused by differences in actual quality of these papers/authors.  

However, prestige might not be the only, or even the most important, way how Matthew’s effect plays out and causes the skewed distribution of impact. Consider this scenario: As a scientist, you have limited time to spend reading papers. Which papers will you choose to read in order to get the best understanding of a given issue? Perhaps the most cited papers since these seem to have been most useful to other scientists. Then your time for reading papers runs out and you write your own paper – which studies are you going to cite? The already top-cited studies, because these are the ones you managed to read. And thus you’ll contribute to the self-reinforcing mechanism of Matthew effect. Anyway, let’s say that already highly cited papers are a bit old because it took time to collect these citations and you want to be ahead. Ideally, you would like to read papers which are not yet highly cited, but will become highly cited in the future – what mechanism are there to find these papers? Perhaps the most established mechanism is journal impact factor ranking. Journals are kind of prediction machines for future citations (indeed, that seems to be their main role now as they are not necessary for the dissemination of knowledge in the internet age when you can upload your paper just about everywhere, they also don’t seem to safeguard against less replicable/true research) so reading papers from high impact factor journal first would make it more likely for you to spot future highly cited papers. Anyways, similarly to the previous case, you are more likely to cite papers from these journals since this is the literature you have read, and thus actually cause the self-fulfilling prophecy of those papers from those journals being more cited. Note that this mechanism seems to operate regardless of the true value of given papers. The gatekeepers of true value are in this case high impact journal editors who kind of decide which papers will get cited in the future. This scenario could be empirically tested by surveying which papers (e.g. in terms of journal IF or number of citations) researchers (ideally broken down to top vs average scientists)  tend to read (e.g. via reference managers like Mendeley, looking at what papers they spend some significant time reading rather than what papers they just save to read later).

What Newton hypothesis suggests if true?

Anyway, let’s imagine the Matthew effect does not play a large role and the differences in contributions to scientific advancement (differences in citations) are actually that skewed with science being advanced by a small group of top researchers, building on each other’s work. What ramifications does that have in real life?

Firstly, we might want to stop focusing on average metrics when assessing science and move to focus on the best case analysis. Metrics like the average number of citations across funded projects or dollars spent per citation are pretty meaningless when it’s driven by outliers who are actually the only ones pushing science forward.
In the same vein, we might want to take a stance on the question of whether to fund smaller number of exceptional researchers and give them extra resources or distribute the funding among greater number of average scientists (which often promises better dollars per citation ratio – but these average citations are perhaps meaningless since they don’t add up to a few pieces of great work), especially when we are able to predict to some extent who will become top scientist.

Further, the Newton hypothesis also diminishes the importance of many meta-science topics – for example, we might not need to push so hard to establish open science behaviours among all researchers. We might be just ok if e.g. the top 10 % cited papers will be all open access to grab all the benefits that open access promises.  We don’t need the remaining 90 % of papers to be accessible because they just matter far less. Perhaps this would also save a lot of money that would otherwise be paid for enabling open access to papers which are not read and cited anyways (that is if we chose to provide open access via the gold way, which is in my opinion wrong strategy anyway. Maybe we should measure progress in open access and open data not via the total ratio of open access/data papers, but rather via looking at how large the ratio of top 10 % cited papers which have open access/data is. Further, we might not care that much about pre-registration, reporting standards, etc.. of all scientists – maybe it’s enough if the top 10 % papers adhere to these standards. Targeting the interventions in this way might also be more manageable than trying to cause a cultural change among all scientists.    

We could also think bigger picture and ask whether it would be sensible to lay off e.g. the less cited half of scientists and either invest the money in something else or redistribute it to the more cited half of scientists, creating better conditions for them and making the science jobs more attractive. I will not go deeper into this discussion but you can find some argumentation in Ricón’s post 

Can we grow it?


Instead of (or simultaneously with) getting rid of less cited scientists, we might focus on how to grow the number of top scientists. Is it even possible to grow it? The answer to this will perhaps depend on the underlying reasons why the top scientists are top, as discussed above. Anyway, we can also look at some pieces of direct empirical evidence: 

One piece of evidence pointing towards “not possible to grow much” is the notion that despite large increases of funding and number of researchers, the rate of economic growth has not increased. However, this tracks only economic impacts of science and is perhaps not a good metric of it anyway.

Another experiment cited by Ricón is Cole & Meyer (1985) who looked at number of hired assistant professors in physics between 1963 and 1975



and compared it to proportion of physicists that ended up being cited:

showing that the proportion of physicists with 1 or more citations stayed the same even though their absolute number increased and then declined in this time period. This suggests that growing the absolute number of cited researchers is possible – however, we might want to use stricter criteria for top scientist than “having 1 or more citations”. 


Conclusion

It seems that distribution of scientific impact is highly skewed both across papers and across researchers and that top papers/scientists are the main contributors to future top papers/scientists, suggesting that Newton hypothesis is true. Anyway, the extent to which this reflects true value is not necessarily clear, because the same dynamic could potentially be created (or at least supported) by the Matthew effect and distribution of attention. Further this finding might be disputed if there is some value causing the scientific progress that is not tracked by citations (which I plan do have some future post about).

If the Newton hypothesis about true distribution of scientific impact is true, then we might want to change the way we measure scientific success (moving to best cases analysis rather than aggregated average metrics) as well as what kind of programmes we support (perhaps stop supporting the average science). We might consider laying off some part (e.g. half) of the less cited scientists and redistributing the funding to support more cited scientists or elsewhere. At the same time, we might want to think more about and experiment with growing the number of top scientists.

In some future post, I would also like to look at another question that could help us understand how the scientific progress works – whether the progress happens via big jumps (similar to Kuhnian revolutions), small incremental steps or some other models and comparing the empirical evidence for this in different fields. This could give a better picture on whether the progress has the same dynamics and shape for science as a whole or whether different fields work differently. 

Open Science

9,755 Replies

Maybe it’s just me, but when I hear discussions about Open Science, it’s often reduced to “is the gold open access model better than subscription based model”? I think this focus is largely ignoring most of the value that open science could bring. Here I will try to sketch out what I think this value is and perhaps also some suggestions how to achieve it. First, let’s look at what all can hide under the Open Science label (figure by OpenAIRE)

Open data

I think that most value of open science lies in open data. The vision behind this is that we would create something like a “data web”, an interconnected repository of all data that is being collected, that we can use to analyze a wide range of questions about the world and come up with new insights that the original authors/data collectors missed or that were not possible to find in each dataset alone. There have already been some success stories in this direction of large scale projects that decided to share their data openly. One of the most striking examples could be the Human Genome Project, which is also considered one the the best investments in terms of economic returns from research, from a large part because they decided to share their data and results openly (read more about this decision here). Another example could be Sloan Digital Sky Survey or Ocean Observatories Initiative (for more details on these stories read Michael Nielsen’s Reinventing Discovery, chapter 6). Anyway persuading individual researchers to share their data seems a bit harder, because the data is the thing that often gives them comparative advantage over other researchers. It seems the incentives are not especially aligned for researchers to share their data. What could change these incentives?

Perhaps if the effort behind collecting data got some more recognition and was not treated only as a way to write good paper. One simple way to increase this recognition would be to plug it in the existing citation-based system, creating citations specifically for the data. Thus when someone else would use your online available data, you would get recognition too.  
The critique that I sometimes hear about sharing the data is that this would “enable some researchers to build their careers only on the data they find online, without ever collecting any data themselves”. I see how frustrating this could be for a researcher who spent a lot of time collecting data with little recognition for their effort, but from a societal perspective it doesn’t seem bad – the end result would be that new insights would be generated form the data which otherwise would not be.    

A half-way solution that I sometimes hear scientists being more open to is delayed release of the data – keeping data private for e.g. 2 years after their collection and releasing them then. This gives the researcher who collected the data some time to extract new insights from it but if they fail to do so, they give others the chance to do it. I think this policy might not be a bad idea, but haven’t yet made my mind about how much value gets lost via this delay (I guess for urgent issues like Covid or some policy relevant questions the delay would mean a large proportion of the value lost, but most research is not so urgent?).

Open access

The main vision of benefits from open access is that it will make it easier to disseminate knowledge and thus get everyone to the frontier of the current knowledge base faster. There is also a moral argument that publicly funded research should be accessible by the public and not be kept behind paywalls. The evidence for these benefits actually realising could be seen e.g. in open access papers being more cited (with paper quality/prestige being controlled for). 

I think these benefits are substantial, but perhaps a bit lower than benefits from open data, since most researchers can find their way around if they want to read a given paper (you can always message the authors asking them for a copy if you don’t want to use pirate services like Sci-Hub). Further, reading someone else’s findings might stimulate your thinking and lead you to new insights, but I guess that new insights drawn from data might be even more important. 

Gold is not the solution

In the gold open access model, the publisher makes all articles and related content available for free immediately on the journal’s website in exchange for a fee paid by the author (or other subjects). Publishers thus switch from being charging readers for access to content to charging authors for the publication service.  

Perhaps the strongest line of critique that I heard is that this system incentivises journals to be less strict in their quality checks (i.e. peer review) – in the subscription based system the earnings of the journal depends on how high quality their content is so that many readers want to access it so they are incentivized to pick the highest quality papers. Not so in the gold open access model, where rejecting an author actually means lower earnings for the journal (in case that the author is not replaced by another author willing to publish in a given journal, which is often the case). Additionally this system could also make it harder to recognise predatory journals (which operate the same way but dont do any real quality checks and don’t provide any other value the standard journals do). I think this might be true and should cause some serious worries, but I’m unsure how much to believe the current quality checks of subscription based journals anyway given that no correlation between replicability and journal status was found and journals with higher impact factor have more papers with erroneous p-values.

Another strong line of critique is that with the current journals’ fees, researchers (or bodies that pay for them like funding agencies or university libraries) would have to increase their budgets (perhaps even several times) to publish all research in this format (see e.g. Green, 2019). Therefore it is not financially sustainable to switch to open science via the gold model. I think this might be true, but it would be wrong to abandon the notion of open access altogether just because it is not possible via the gold model. There are other models which are significantly less expensive and have some additional benefits on the top of the gold model (see below).

Finally, another line of critique is that although one of the goals of open access was to make science results accessible to developing countries which can’t afford paying large subscription fees, high publishing fees in the gold open access model now make it impossible for researchers from those countries to participate by sharing their own findings with the global community. I think that even if this is true, the tradeoff here is pretty skewed – it is much more beneficial for developing countries to be able to consume research of the developed countries (which is perhaps of higher quality) and thus stimulate their economies, than sharing their own findings (which are perhaps of lower quality because fewer resources goes into creating them).     

Overall I think there are strong and sensible arguments against gold open access models and perhaps we should search for alternative models that have fewer downsides and more benefits.

Preprints are

Perhaps the easiest way to make all knowledge open access is to establish norm to share all papers as preprints, ideally in some (centralized) databases that are searchable by Google Scholar, WoS and other search engines. This has not only the advantage of making all papers open access by default and for free, but it also has one additional important advantage – it speeds up the dissemination of knowledge a lot. No more waiting for several months (or sometimes even years!) from the time the paper is finished to the time it is actually published and shared with the world. Findings are shared instantly. This already proved crucial in urgent matters such as Covid research, but I guess the potential of that increased speed is high even for less urgent matters.

Perhaps the only critique of preprints I heard so far is that there is no quality check and thus you can’t know whether a given paper you end up reading doesn’t have some major flaw in it. I agree that there is no quality assurance and perhaps the credit that researchers get for publishing should not be the same for pre-prints and for papers that have gone through some quality checks. Anyway, every researcher can judge for themselves whether a given paper has or doesn’t have any major flaw in it when they are reading it, after all that is how  the peer review works, right? And maybe we can nudge the readers to give a rating to a given pre-print they read so others can get a signal on how good the paper is even before they start reading it. The standard quality checks of journals (i.e. peer review) might still be in place – authors could submit the preprints to the public databases first, but then continue submitting their papers to journals as usual. Journals then can publish revised versions of the paper themselves, or give some “badge of quality” to the preprint version in the database. Funders can value papers with this “badge of quality” higher than standard preprints and thus create incentives for applying for these badges. Or alternatively the initiative might rest on the shoulders of journals – they can search through the preprint databases and decide themselves which papers will they invite for the quality check or give the badge of quality right away. But let’s open our minds still a bit further. Maybe in the future we will find better mechanisms to assess quality than the peer review process. Maybe it will be making use of collective intelligence via the rating scales suggested above or replication prediction polls/markets which would be open to everyone who has read the paper. That sounds like something more scalable than peer review. Maybe it will be some automated AI algorithm assessment which would be even more scalable. In any case, publishing via pre-prints would make these possible in the future and would have some direct benefits even today.    

What is the trend of scientific progress?

2,982 Replies

Is there a problem?

Bloom, Jones, Van Reenen & Webb (2020) found in their multiple case studies, spanning Moore’s law, agricultural crop yields, mortality and life expectancy and research productivity in firm-level data, that each domain suffers from diminishing returns to investment. That means that a constant number of scientists would not keep coming up with a constant number of new ideas throughout the time in a given domain, but rather coming up with new ideas gets harder and harder and therefore we should expect them to come up with fewer and fewer new ideas and less progress.


Ideas in some domains might be getting more and more depleted, but new domains are arising and give opportunities for new and fresh ideas to be harvested. That is perhaps how science progress happens. The more surprising question comes when we try to generalize this to science as a whole, as Bloom et al. seem to be suggesting. Is it the case that it is getting harder and harder to find whole new domains of research? That our ability to find new ideas not in a specific domain, but anywhere, is hitting diminishing returns? Maybe our mechanism of knowledge creation and ability to understand the world is hitting diminishing returns as well?

I would say that evidence for this latter claim is much weaker than for diminishing returns in any given specific domain.
First, we can look at how economists measure the economic benefits of scientific progress. Bloom et al. model that by comparing inputs (investment into National Income and Product Accounts’ “intellectual property products”, a number that is primarily made up of research and development spending but also includes expenditures on creating other nonrival goods like computer software, music, books, and movies) and outputs (total factor productivity, TFP, which is a portion of the growth in output not explained by growth in traditionally measured inputs of labour and capital used in production). However, TFP does not seem to be an especially good measure of scientific progress – it does not track scientific progress per se, but rather all unmeasured factors contributing to productivity. This contains not only the economic benefits of new discoveries but also the diffusion of the existing knowledge, which seems to be qualitatively very different things. We have no way to tell how big proportion of TFP is caused by diffusion versus creation and how this varied in time. Further, some part of scientific progress works via enabling a greater supply of factors measured outside of the TFP (e.g. labor, capital, and land) or are embodied in concrete capital goods (which is perhaps how the benefits from most privately funded research are realised) and these will not be counted in TFP. The counterargument goes that if these biases are roughly constant over time, changes in TFP still would reflect changes in the rate of progress of science and technology (for more details and discussion see Cowen and Southwood, 2019, p.18). Anyway, I don’t think we have a strong reason to assume these biases are constant over time. TFP thus has the advantage of aggregating the science as a whole but seems not to be capturing scientific progress accurately enough to enable drawing strong conclusions about whether the science is slowing down or not. I think that if we wanted to make an argument using TFP, we could still say (although pretty vaguely), that since 1930s, TFP has been declining, whereas investments in research have been increasing, which suggests that at least some part of scientific discovery captured by TFP have been slowing down (or hitting diminishing returns).  


Further, we can take the notion of scientific progress as how much it does contribute to wellbeing measured in other ways than economic benefits (e.g. life satisfaction; chance of survival for the next thousand of years..). Is there any measure like that for scientific progress as a whole? Unfortunately, I’m not aware of any. 

Further yet, we may take a look at scientific progress from a perspective of how much it increases our understanding of the world. For example, measures that do not concern the economic benefits of science, but rather things like “how much a discovery changes our understanding of the world” and “how much does it enable us to make further discoveries”? Each of these interpretations would likely require its own measurement method.
One venture in this direction was done by Patrick Collison and Michael Nielson (2018), who asked top scientists to compare the importance of discoveries awarded by Nobel prizes in each decade. They found that scientists rate older discoveries as more important than the more recent ones. However, they focused again only on a few specific domains in which Nobel prizes are awarded (chemistry, physics, medical sciences), not on science as a whole, not to say newly developed domains of research. Further, the notion of “importance” might mean different things for different scientists –  some might weight economic application of that knowledge more, others might rather weight in how much it changed and formed our current understanding, etc.. Finally scientists might have a bias towards being correct, and it is safer to bet on older discoveries whose applications and importance have already been well demonstrated, rather than on more recent discoveries where applications are yet to be found. To conclude, while an interesting venture, this piece of research does not seem to be telling us much about whether science as a whole is slowing down.            

Diminishing returns to investments seem like a pretty natural way the things work when you want to exploit some resource (like gold mining). It also makes sense based on a heuristic that it’s easier to grow rapidly from small base than from large base (i.e. 10 % returns from investements in 1000 scientists are perhaps easier to achieve than 10 % returns from 1 000 000 scientists, because the latter benefit is just so much larger in absolute terms). Anyway, not all things in the world works this way – there are also things that work on the basis of constant returns or even increasing returns. An argument in this way could go like: the amount of pieces of knowledge is still increasing and by combining different pieces of knowledge, we can create yet another useful piece of knowledge. Therefore, the actual space of possible discoveries is getting larger and larger with each new discovery made, ther than smaller and smaller (as “ideas getting harder to find” hypothesis seems to suggest). However, it doesn’t say anything about (economic) returns to these discoveries – maybe ideas are increasingly easier to find, but the value of these ideas is lower and lower (or perhaps ideas with high economic value are getting harder and harder to find). One piece of evidence that could potentially point to the direction of increasing returns is the correlation evidence that firms and countries that invest more in R&D have higher returns to those investments (Hervas & Amoroso, 2016). However, this could also represent a result of comparative advantage over other countries/firms or there might be reversed causality (i.e. firms and countries that are doing economically better are using their additional resources to fund R&D, but R&D is not a causal mechanims for how they got economically better).


Anyway, let’s imagine science as a whole is really generating less and less benefit per dollar invested. Being true or not, this exercise is helpful in spotting things that might be suboptimal or might become problems in the future, something like writing a pre-mortem. So, why could that be?

Causes

1) New ideas are harder to find

As suggested by Bloom et al. and many others, it could be that just like in many other aspects of our lives, engaging in a given activity repeatedly brings less and less benefit. Good metaphor is the one with low-hanging fruit – we are gradually picking the low hanging fruit and now each new fruit is higher and thus harder and harder to pick. In that case, what kind of ability does this refer to specifically? It might mean that our ability to understand the world is hitting diminishing returns. Or it might be the method of knowledge creation that’s getting depleted. It might also mean that we just need more and more investment to get up to speed with our current level of knowledge. We can see this in several domains: PhD degrees in economics are taking longer to complete, co-authorship is increasing in mathematics (Odlyzko, 2000), economics (Brendel & Schweitzer, 2017) and research teams are growing larger in mathematics (Agrawal, Goldfarb & Teodoridis, 2016), suggesting one head can’t absorb all the required knowledge anymore. The age of the scientists at which they made a discovery for which they later received nobel prize also increased by about 10 years during the 20th century (Jones & Weinberg, 2011). However, note that all these pieces of evidence come from specific, very traditional disciplines, where the diminishing returns are most likely to be observed. They don’t assess science as a whole.  

Picking up the low hanging fruit seems like a natural principle – is it inevitable then? Can we do anything to reverse that or at least slow it down? I will propose some potential solutions in another post.

2) Science as an institution is getting significantly less effective in generating new benefits

This suggests that due to more bureaucracy researchers spend less time and attention on research itself. Or alternatively, with increasing numbers of researchers doing science, it’s getting harder and harder to coordinate and coordination costs cause large decreases in benefits generated.

One possible proxy for measuring the extent of bureaucratisation is looking at the proportion of time researchers spend actually doing research. One older study by Milem, Berger & Dey (2000) found that in the US between 1972 a 1992 the proportion of time devoted to research among academics at research universities actually increased. However, the time spent doing research is operationalized in a way that is likely also contains grant writing. Several more recent studies pointed to the large amount of time that gets wasted by researchers writing grant proposals and harshly competing for funding, with some researchers claiming they spent as much as 60 % of their time seeking funding (Fang & Casadevall, 2009) and overall costs of peer review grant selection process consuming as much as 20–35 per cent of the allocated budget for research (Gluckman, 2012) (for more discussion see e.g. this Nintil post). Moreover, this trend of wasted time seems to be increasing as success rate in grant competitions is getting lower over time (see e.g. this data from NIH or this data from CIHR, I will update with more data in the future to confirm whether this trend holds across domains and countries). Anyway, this trend would significantly contribute to decreasing returns only if the distribution of impact was more uniform, which might not be the case.
Another way to see whether administrative costs are increasing over time is to look whether there is increase in administrative positions (e.g. academic project managers) paid from the R&D budget (will need to find some more data about this).

However, as my friend Aleš suggested, reduced time researchers spent doing research might actually not be the most important form of bureaucratisation. The more important form of bureaucratisation might be expressed via hiring processes. If these processes become too stringent and inflexible, it might miss out some of the most extraordinary talents who won’t get the job and the processes might instead select for more average careerists, which might decrease the amount of innovation produced by academia by far in the long term.     

In the similar category of science institution getting less effective, the problem could also be insufficient research infrastructure. In the EU, the number of researchers grow approx 1,5 % per year, which means doubling once in 46 years (Baumberg, 2018, cited via Cowen a Southwood, 2019, other data since 1980s can be found e.g. via OECD, 2021). It is perhaps possible that institutions of science are finding it hard to keep up with such increases and, because the lack of coordination and infrastructure, the process of knowledge creation gets slower.
   

3) Privatization of research funding and benefits

Benefits from research are getting more and more privatized, oriented towards improving specific private goods rather than generating global public goods. This might explain why the TFP metric does not capture these benefits but also suggests a plausible hypothesis for how the overall value of research decreases. Indeed, the majority of research funding across G20 countries currently comes from business and private sources rather than government spending (1,26 % vs 0,65 % GDP on average) – but was it always like this or has it changed in the past decades?

During the second half of the 20th century, the profile of research funding changed greatly. In the 1960s United States, for example, public investment in research accounted for more than 2/3, while today it accounts for only 1/4 all investments in research. This privatization of research funding also applies to basic research – the proportion of publicly funded basic research has been declining since the 1960s, and in 2013 even more than half of all basic research spending in the US came from private sources (Mervis, 2017). This trend can also be seen in data from other OECD countries and Europe: the ratio of researchers paid from state/public sources vs. private sources is still declining (OECD, 2021).

If privately funded research brings smaller profits to the general public and greater profits to given private investors, then it is possible that this privatization of research funding ultimately reduces the overall economic benefits of investing in research. On the other hand, data from the US NSF show that the total amount of funding devoted to basic research increased 33 times between 1960 and 2000, and the ratio of basic research expenditure to total government expenditure (i.e. compared to applied research and development) increased from around 8% to around 27% over the same period (NSF, 2002). It is therefore true that overall research spending is being privatized, but spending on basic research is still rising – if (publicly funded) basic research is indeed more cost-effective in the long run because it allows knowledge to be more widely distributed and forms the basis for applications, then growth in investments in this type of research should ensure that the overall economic benefits of investing in research will grow, not fall.

Another pointer towards privatisation of research could be the Bayh-Dole Act, which might have encouraged universities to invest their resources to support of monetizable research and perhaps also decrease the disclosure and sharing of their research findings in order to monetize it later. Anyway, if “monetizable”=”more applied”, this should shown in some statistics (although perhaps not the NSF data, because that is strictly governmental funding, not university funding; also, it would be useful to compare how large proportion of research funding actually comes from universities vs governments to see whether that fraction is actually relevant).

Conclusion

To sum up, I think Bloom et al bring some evidence for decreasing economics returns to investments in any given domain, however, this not neccessary generalizes to science as whole. Because we don’t have especially good metric of economic benefits stemming from science as whole, perhaps the argument for diminishing returns for whole science is weak. Anyway, if it is true, there are number of reasons hypothesized – ranging from ideas getting harder to find to science as an institution getting less effective to privatisation of research funding and benefits. Perhaps all these effects are true at the same time and contribute to the observed decrease in benefits, but perhaps they vary in how much they contribute.