Let us perhaps first remember what the academic system is supposed to do, which is: Find out about how nature is, explain that in a transparent way to the public, have a careful set of checks and balances to make sure claims are thoroughly tested before they become canon and allocate funding such that the goal of finding out about nature is best served. This essay would not have been written if I believed that would largely be the case in reality,
A will have had the money to travel to many meetings, hence his work will generally be better known. This translates into a larger probability that people will cite him and that reviewers will think of his work and ask for citations to it be inserted into manuscripts. He also will have had the money to hire a research group, and that implies more publications with his name on them. Finally, since often acquired funding is used as a criterion to judge the track record, he will be able to bring this in his favour as well in the new round of grant applications.
B in contrast will have none of these advantages - she would have to be a much better scientist or present a much more compelling case than A to get the same evaluation based on the usually used proxies - yet note that by definition of the gedankenexperiment, she is just as good a scientist - she only was not as lucky in acquiring her first grant.
This argues that there is inertia and segmentation - there are groups of 'good' and 'bad' scientists, and while not impossible, it is difficult to cross the borders between the groups on merit alone beyond some point- in other words, support predominantly is given to those who already have. Despite the premise that funding is allocated to the best scientists, this is biased by this very effect. This is not to say that being a good scientist has nothing to do with acquiring funding - but it is not the only factor.
Ultimately, the 'in'-group will also supply the people who set the broad outlines of a research filed by acting in advisory committees or selection panels or as journal editors. Hence the way they judge things will have extra impact beyond their own research.
It follows that is it crucial for a young scientist to attract funding early on. Now, let's again make a gedankenexperiment - assuming ability of two scientists (and everything else) being equal, how does one maximize chances to attract funding?
Summarizing all we have discussed previously, the answer is obvious - by focusing on quick, speculative novel ideas, never following them up but rather turning to a new idea, rather than by detailed, systematic studies and in-depth investigations. Producing ideas is easier in the first place, it takes less time and effort to compute a schematic model than a full-scale realistic model, which means more publications over time. Quick ideas allow turfing - which means maximizing citation count as well. They cater well to the expectations of the public - especially if they're accompanied by speculations on implications which are as bold as possible. Given the current academic structure, such focus will imply a competitive edge in funding decisions.
It is clearly not the case that a young scientist can succeed by simply inventing a compelling story out of thin air - the existing checks and balances will usually prevent that - just as the claim of superluminal neutrinos ultimately did not stand. Rather, there is a gradual drift in which moving just to the edge of what is permitted in terms of speculation vs. systematics constitutes an advantage, and as people realize this and pursue their own advantage, the standards of what is acceptable gradually shift.
There is unfortunately no effect which prevents this silent erosion from happening. And as students are educated in the current academic system, they internalize what they see and learn and naturally come to the conclusion that what they observe is science as it should be done. Yet, in the current academic environment, a young scientist who strives to really find out about nature and is willing to prove his own ideas wrong does so at a considerable risk for his career.
Now, this resembles storytelling more than science - exciting results which are so losely based on data that they can not be reproduced but are prominently noticed by other researchers. Once 90% of published results can not be trusted though, what is really learned about nature can not longer be seen in the noise of spurious and false results.
Cancer research is not physics, the problems are much more difficult to isolate and there may be more confunding factors which are difficult to get under control. Yet the slippery slope is real - nothing would have in principle prevented the cancer researchers from trying systematic reproduction before a publication - it just did not happen in practice.
Spectacular results which upon closer investigation do not hold up seem like rare incidents in most of science, but the publicly known cases are just the tip of an iceberg - they're in fact the logical outcome of an academic system which rewards only ideas and speculation and has little left for independent verification, systematic checks and detailed investigations.
The answer inevitably given to such a challenge seems to have to do with market and competition. I suppose the idea of evaluations and project funding over fixed allocation of resources is that researchers perform best if in a constant competition, a market for the best ideas.
Unfortunately, that idea doesn't quite fit. First, most good researchers I know are intrinsically motivated - they do research because they like what they do and usually at the limit of their abilities, they would not do better research if given extrinsic motivation like a raise in paygrade. A researcher who needs an extrinsic motivation in the first place is probably working on the wrong topic. But second, a good strategy in competition often makes for poor science. For instance, withholding crucial information, not putting code into a public repository or letting another research group persist in an errorneous assumption are all valid competitive strategies, but they harm science - science relies on information getting out into the open, competition is often done on proprietary information.
Instead, since proxies such as citations or number of publications are used to gauge scientific performance, the competition partially becomes sensitive to how well researchers can (and are willing to) 'play the system'. A researcher who can publish many papers is not necessarily a researcher doing good science though (yet - since the market advocates measure performance by the indicated proxies, they see indeed an increase in 'performance' as the competition actually drives up citation and publication counts - whether this correlates with better science is an entirely different matter).
From this point, the slippery slope runs on its own with very little signs of problems - a steady stream of ideas arrives on the scene, is digested enthusiastically by a general public excited about the constant scientific progress and spectacular implications, publication and citation counts increase, researchers are happy because their ideas are appreciated rather than critically dissected,... The signs of trouble are only rare, when the public realizes now and then that a spectacularly announced result simply collapsed to nothing. And yet - the outcome of the whole procedure has little to do with Feynman's idea of not fooling oneself or others.
It's also clear that no single scientist can simply decide to do differently without a risk for her career or the field. While nobody is forced to go down the slippery slope rapidly, nobody can stem against the trend either. Technically the situation is a prisoner's dilemma, and its Nash equilibrium is unfortunately at the point of maximum possible story-telling, not at the best science.
What worries me is that a new generation of scientists is educated in the current situation and has no good way to know better. I remember a discussion during a manuscript review when I pointed out to a young researcher that her claims would have implications for other results not discussed in the paper which were in contrast to what was known, hence her conclusions must be wrong. She was deeply offended by the idea that I would hold something that was not discussed in the paper against her results - it became apparent that she was not aware of the whole idea of systematic consistency tests. Yet she is a researcher at a genuine university and will educate students and review papers.
What can be done? I don't have a panacea, but I believe improvements could be made in several areas:
Comments, suggestions or observations from both other scientists and interested readers are welcome at my contact email below - I would like to make this essay as broad in scope as possible. This is not a blog with a moderated comment section, and comments may be edited and/or (with permission by the author) merged into the main bulk of the essay.
Created by Thorsten Renk 2015 - see the disclaimer, privacy statement and contact information .