Issues in Scientific Publishing

Lounge
1

Here are a few paragraphs on the replication crisis for your reflection from the excellent 2020 book Science Fictions by Stuart Ritchie:

…Replication, then, has long been a key part of how science is supposed to work – and incidentally, it’s another of its social aspects, with results only being taken seriously after they’ve been corroborated by multiple observers. But somewhere along the way, between Boyle and modern academia, a great many scientists forgot about the importance of replication. In the collision of our Mertonian ideals with the realities of the scientific publication system – not to mention the realities of human nature – the ideals have proven the more fragile, leaving us with a scientific literature full of untrustworthy, unreliable, unreplicable studies that often do more to confuse than enlighten. In the next chapter, we’ll see just how untrustworthy, unreliable and unreplicable the scientific literature has become…

Ritchie, Stuart. Science Fictions: How Fraud, Bias, Negligence, and Hype Undermine the Search for Truth (p. 24). Henry Holt and Co.. Kindle Edition.

As you might expect, the confluence of failed replications (like the priming studies) and bizarre results (like Bem’s paranormal discoveries), along with revelations of misrepresentation (like Zimbardo’s experiment) and fraud (like Stapel’s fake data) spooked psychologists. Just how many of the studies in their field, they wondered, could be trusted? To get an idea of how bad things were, they started banding together to run large-scale replications of prominent studies across multiple different labs. The highest profile of these involved a large consortium of scientists who chose 100 studies from three top psychology journals and tried to replicate them. The results, published in Science in 2015, made bitter reading: in the end, only 39 percent of the studies were judged to have replicated successfully.26 Another one of these efforts, in 2018, tried to replicate twenty-one social-science papers that had been published in the world’s top two general science journals, Nature and Science.27 This time, the replication rate was 62 per cent. Further collaborations that looked at a variety of different kinds of psychological phenomena found rates of 77 per cent, 54 per cent, and 38 per cent.28 Almost all of the replications, even where successful, found that the original studies had exaggerated the size of their effects. Overall, the replication crisis seems, with a snap of its fingers, to have wiped about half of all psychology research off the map.29

Maybe it’s not quite that bad, for two reasons. First, we would expect some results that really are solid to fail to replicate sometimes, merely due to bad luck.30 Second, some replications might have failed due to their being run with slight changes to the methodology from the original (though if a result is fragile enough that it disappears after minor modifications to the experiment, one might wonder how useful or meaningful it really is).31 For these reasons, it’s sometimes tricky to decide whether a finding is ‘replicable’ or not based just on one or two replication attempts. What’s more, the replication rate seems to differ across different areas of psychology: for example, in the 2015 Science paper, cognitive psychology (studies of memory, perception, language, and so on) did better than social psychology (which includes the sorts of metaphor-priming studies we saw above).32 In general, though, the effect on psychology has been devastating. This wasn’t just a case of fluffy, flashy research like priming and power posing being debunked: a great deal of far more ‘serious’ psychological research (like the Stanford Prison Experiment, and much else besides) was also thrown into doubt. And neither was it a matter of digging up some irrelevant antiques and performatively showing that they were bad – like when Pope Stephen VI, in the year 897, exhumed the corpse of one of his predecessors, Pope Formosus, and put it on trial (it was found guilty). The studies that failed to replicate continued to be routinely cited both by scientists and other writers: entire lines of research, and bestselling popular books, were being built on their foundation. ‘Crisis’ seems to be an apt description…

…Could the sheer complexity of the task make findings in psychology particularly untrustworthy, compared to other sciences?

There is something to this argument: many studies in psychology barely scratch the surface of the phenomena they’re interested in, while other ‘harder’ sciences – say, physics – have better- developed theories and more precise and genuinely objective measurements. But it’s not as if psychology is alone in having problems with replicability – although no other sciences have yet investigated their replication rates as systematically and in as much detail, there are glimmers of the same kinds of problems across very many different fields:

• In economics, a 2016 replication survey of eighteen microeconomic studies (not too different from psychology research, with people coming into the lab and taking part in experiments on their economic behaviour) had a replication rate of only 61 per cent.33
• In neuroscience, a study in 2018 found that standard studies in functional brain imaging, where the brain’s activity is recorded using MRI while the person is completing some kind of task (or just lying in the scanner), were likely only ‘modestly replicable’.34 The world of functional brain-imaging was also rocked by a paper which revealed that a default setting in a software package commonly used to analyse imaging data had a statistical error. It led to a vast number of accidental, uncorrected false-positive results, and it might have compromised around 10 per cent of all studies that had ever been published on the topic.35
• In evolutionary biology and ecology, a series of classic findings, repeated in textbooks and taught to generations of students, have fallen to replication attempts and critical reviews. For instance, the famous ‘domestication syndrome’, where Russian foxes that were selected for tameness started to take on the physical characteristics of domesticated species (like floppy ears and wider faces), turns out to have been hugely exaggerated, with most of the ‘domesticated’ traits existing before the selection even took place.36 And much of what we thought we knew about sexual selection in birds has been shot down by better evidence. For instance, despite what we thought we knew, putting a red band on male finches’ legs probably doesn’t make them super-attractive to females; male sparrows with larger patches of black plumage on their throats (their so-called ‘bib’) probably don’t have higher dominance in the flock; and the evidence that female blue tits are more attracted to particular plumage colours in males is inconclusive.37
• In marine biology, a massive replication attempt in 2020 found that the effects of ocean acidification (one of the consequences of climate change) on fish behaviour were non-existent.38 It thus failed to replicate several highly publicised studies from the previous decade that had apparently shown that more acidic conditions caused fish to become disoriented, and in some cases to swim towards, rather than away from, the chemical cues produced by predators.
• In organic chemistry, the journal Organic Syntheses, which operates an unusual policy where an editorial board member attempts to replicate in their own lab the results of every paper they receive as a submission, reported rejecting 7.5 per cent of submissions because of replication failures.39

There are countless other examples: almost every case I’ll describe in this book involves a scientific ‘finding’ that, upon closer scrutiny, turned out to be either less solid than it seemed, or to be completely untrue. But more worryingly still, these examples are drawn just from the studies that have received that all-important scrutiny. These are just the ones we know about. How many other results, we must ask ourselves, would prove unreplicable if anyone happened to make the attempt?

One reason that we live in such uncertainty is that, as we learned from the Preface, hardly anyone runs replication studies. Though we don’t have the numbers for most fields, scans of the literature from certain subjects draw a bleak conclusion. In economics, a miserable 0.1 per cent of all articles published were attempted replications of prior results; in psychology, the number was better, but still nowhere near good, with an attempted replication rate of just over 1 per cent.40 If everyone is constantly marching onwards to new findings without stopping to check if our previous knowledge is robust, is the above list of replication failures that much of a surprise?

Here’s something that’s perhaps even more alarming. You’d think that if you obtained the exact same dataset as was used in a published study, you’d be able to derive the exact same results that the study reported. Unfortunately, in many subjects, researchers have had terrible difficulty with this seemingly straightforward task. This is a problem sometimes described as a question of reproducibility, as opposed to replicability (the latter term being usually reserved to mean studies that ask the same questions of different data)…

Ritchie, Stuart. Science Fictions: How Fraud, Bias, Negligence, and Hype Undermine the Search for Truth (p. 31). Henry Holt and Co.. Kindle Edition.

Here is some further reading: :slight_smile:

Replication crisis - Wikipedia

Renaldo

1 Like
Physics and Buddhism: 4 elements and rupa kalāpa
What’s at Stake as the Dharma Goes Modern? (Scientism and Dhamma)
2

From the last chapter:

“The process of scientific discovery … will change more over the next twenty years than in the past three hundred years.” Michael Nielsen, The Future of Science (2008)

The majority of the problems with science that we’ve seen in this book are encapsulated in one meta-science paper from 2018. The psychiatry researcher Ymkje Anna de Vries and her colleagues examined all the steps that typically occur between the trialling of a new drug and that drug’s eventual presentation to the world.1 As their sample, they took 105 different trials of antidepressants that had been approved by the US Food and Drug Administration. The ratio of positive to negative results among these trials happened to be almost fifty-fifty, with fifty-three finding that the antidepressant in question worked better than the control or placebo, and fifty-two getting results that were either null (the FDA referred to them as ‘negative’) or ‘questionable’.2 So far, so predictable: some studies find significant results, some don’t. The problem was what happened next.

Looking across all the studies, de Vries and her team observed a process of literature-laundering that transformed a messy, varied set of trials into a much cleaner-looking story of scientific discovery – a process that made the drugs in question look substantially more effective than they really were. The first step was publication bias. 98 per cent of the positive trials (fifty-two of fifty-three) were eventually published, compared to only 48 per cent of the negative ones (twenty-five of fifty-two). In reality, the positive-negative ratio was balanced; after publication, the positives outnumbered the negatives two to one. As we know, scientific journals don’t favour null results – but for a clear-eyed look at the scientific record, we need to see them.

The second step was p-hacking – specifically, outcome-switching, where scientists change the focus of their study upon finding that their main result isn’t statistically significant. Once the outcomes being studied were switched, ten more of the negative studies had changed into positive ones (you’ll recall from Chapter 4 how outcome-switching increases the chances that our result is nothing more than a false positive). At this point there were still fifteen studies with clear negative results – and this was where the spin came in. Ten of those fifteen studies included some kind of spin in their abstract or the main paper to make their results appear more positive.

After this cycle of bias and spin, a mere five studies were left that were unambiguously negative – that is, ten times fewer than existed at the start. And as a final, rotten cherry on top, the positive trials ended up being cited by later research three times as often as the negative ones. The whole dispiriting step-by-step process is illustrated in Figure 4 below.

This isn’t unique to research on antidepressants: the authors found a similar chain of events in trials of new psychotherapies.3 Indeed, the same thing is happening, to a greater or lesser extent, almost everywhere in scientific research. Any future meta-analyst who wants a full picture of a particular research area is going to get a drastically warped perspective – and unless they look in detail through the trial registries (which, incidentally, aren’t required in fields other than medicine), they won’t even know about it. De Vries’s study didn’t investigate the possibility of fraud, or errors in study design or analysis, or the presence of hype in the presentation of the treatments in the media and marketing, but we can safely assume that these are also occurring all the time, further obscuring the reality of the research.

Ritchie, Stuart. Science Fictions: How Fraud, Bias, Negligence, and Hype Undermine the Search for Truth (pp. 200-201). Henry Holt and Co.. Kindle Edition.

It’s very eye opening.

R

3

"Watson and Crick’s 1953 paper describing the double helix structure of DNA was not peer reviewed. Neither was Darwin’s On the Origin of Species. Nor were the foundational papers on quantum mechanics by Planck, Bohr, and Heisenberg. All of the texts that established modern molecular biology, evolutionary theory, and atomic physics entered the scientific record without passing through anonymous review by external experts.

And yet, we still consider them to be science. The leading science journal Nature did not institute formal peer review until 1967. Before that, publication decisions were made by editors who placed “a lot of trust in particular laboratory heads and particular friends in the British scientific community,” as one Nature retrospective noted. If Laurence Bragg said something was good and important, they printed it. The three DNA papers appeared in April 1953 because the editors trusted the judgment of established scientists, not because anonymous reviewers had verified the findings. And really, who was the peer of James Watson in 1953? Peer review as we know it is not an ancient tradition. It is a mid-twentieth-century bureaucratic innovation that gradually transformed from a quality control mechanism into a conformity enforcement system that ensures mediocrity and stasis. Understanding this history is essential to understanding why AI trained on peer-reviewed literature reproduces the pathologies of institutional science.

The myth of equating science with peer review serves an important rhetorical function. When someone says “peer-reviewed studies show,” they invoke an image of careful scrutiny by experts, experiments being repeated and confirmed in a scientific process refined over centuries, and a gold standard of epistemic quality. The phrase carries authority because people imagine that peer review refers to something venerable and rigorous.

The reality is entirely different. Peer review is much closer to what is called “editing” in the publishing industry, and not high-level editing either. Peer review serves as the science industry’s version of the book industry’s slush pile—the mechanism by which publishing houses like my own reject the vast majority of manuscripts submitted to us for publication. This isn’t even an analogy. For most of the history of science, publication decisions were made by editors. The Royal Society of London did establish something resembling peer review in 1665 with the launch of Philosophical Transactions, but this operated more like an exclusive club than a systematic vetting process. The scientific community was small enough that everyone knew everyone, and reputation served as quality control. This system had obvious flaws. It was vulnerable to cronyism, to old-boy networks, and to the suppression of outsiders with valid ideas. But it had one significant advantage over the modern system: editors actually took responsibility for what they published. When Max Planck accepted a revolutionary paper, Planck was putting his reputation on the line. This accountability created incentives for editors to evaluate work carefully and to sometimes take chances on unconventional ideas that might prove important. Planck’s philosophy was “to shun much more the reproach of having suppressed strange opinions than that of having been too gentle in evaluating them.”

The shift to anonymous external review reversed this dynamic. Editors could now deflect responsibility to reviewers. If a paper turned out to be fraudulent or wrong, the editor could point to the reviewers who had approved it. If an important paper was rejected, the editor could blame the anonymous experts. The diffusion of responsibility unsurprisingly resulted in a diffusion of accountability. And the diffusion of accountability has resulted in an explosion of mediocrity, scandals, and fraud. Consider what happened to Linus Pauling, a scientist of towering accomplishment. Pauling had won the Nobel Prize in Chemistry in 1954 for his work on chemical bonding. He had done foundational work on protein structure, sickle cell anemia, and the nature of the chemical bond itself. He was one of the most decorated scientists of the twentieth century. In the 1970s, Pauling became interested in vitamin C. His claims about its benefits for the common cold and cancer treatment were controversial, and the medical establishment was hostile.

When Pauling submitted a review paper to the Journal of the American Medical Association, it was rejected. He revised it to meet the referees’ suggestions. It was rejected again. He revised it again. Rejected again. The manuscript was finally published in a minor journal. A two-time Nobel laureate could not get a review paper past the gatekeepers of mainstream medicine because it violated their scientific consensus. Whether Pauling was ultimately correct about vitamin C is not the point. The point is that the peer review system was totally incapable of evaluating his claims on their merits. Instead, it functioned as an immune system, rejecting foreign ideas that threatened the established paradigm."

Day, Vox. Hardcoded: AI and the End of the Scientific Consensus (The Mathematics of Evolution) (pp. 37-38). Castalia House. Kindle Edition.

R

4

Academic Fraud:

R

5

So, he does a whole ethical piece on academic fraud for the sake of money and prestige and then at the end he touts a VPN that can bypass Geofences and hide your identity (I think because he is reading and downloading articles that he should be paying for)

“I lost access to institutional libraries. And that means I don’t have access to things like scholarly journals. So sometimes I end up going down rabbit holes on the internet trying to find the right PDFs.”

1 Like
6

Correct, but at least he is bringing attention to these fraudsters.

R

7

Publish or perish:

“Publish or perish” is an aphorism describing the pressure to publish academic work in order to succeed in an academic career.[1][2][3] Such institutional pressure is generally strongest at research universities.[4] Some researchers have identified the publish or perish environment as a contributing factor to the replication crisis.[5]

Successful publications bring attention to scholars and their sponsoring institutions, which can help continued funding and their careers. In popular academic perception, scholars who publish infrequently, or who focus on activities that do not result in publications, such as instructing undergraduates, may lose ground in competition for available tenure-track positions. The pressure to publish has been cited as a cause of poor work being submitted to academic journals.[6] The value of published work is often determined by the prestige of the academic journal it is published in. Journals can be measured by their impact factor (IF), which is the average number of citations to articles published in a particular journal over the last two years.[7]

Academic opinion

The pressure to publish has been strongly criticized on the basis that over-emphasis on publishing may decrease the value of resulting scholarship, as scholars must spend more time scrambling to publish whatever they can get into print, rather than spending time developing significant research agendas.[8] Similarly, humanities scholar Camille Paglia has described the publish or perish paradigm as “tyranny” and further writes that “The [academic] profession has become obsessed with quantity rather than quality. … One brilliant article should outweigh one mediocre book.”[9]

The pressure to publish or perish also detracts from the time and effort professors can devote to teaching undergraduate courses and mentoring graduate students. The rewards for exceptional teaching rarely match the rewards for exceptional research, which encourages faculty to favor the latter whenever they conflict.[10]

Also, publish-or-perish is linked to scientific misconduct or at least questionable ethics.[11] It has also been argued that the quality of scientific work has suffered due to publication pressures. Physicist Peter Higgs, namesake of the Higgs boson, was quoted in 2013 as saying that academic expectations since the 1990s would likely have prevented him from both making his groundbreaking research contributions and attaining tenure: “It’s difficult to imagine how I would ever have enough peace and quiet in the present sort of climate to do what I did in 1964 … Today I wouldn’t get an academic job. It’s as simple as that. I don’t think I would be regarded as productive enough.”[12]

According to some researchers, the publish or perish culture might also perpetuate bias in academic institutions. Overall, women publish less frequently than men, and when they do publish their work, it receives fewer citations than their male counterparts - even when published in journals with significantly higher impact factors.[13] Furthermore, one study pointed out that gaps in the promotion and progress of women in academic medicine may be significantly influenced by gender-based variances in article citations.[14]

Research-oriented universities may attempt to manage the unhealthy aspects of the publish or perish practices, but their administrators often argue that some pressure to produce cutting-edge research is necessary to motivate scholars early in their careers to focus on research advancement, and learn to balance its achievement with the other responsibilities of the professorial role.[15]

https://en.wikipedia.org/wiki/Publish_or_perish

R

8

I told a friend about this who is a professor.
She said that you perish because you don’t love your work.
If you are a real professor, you will love to publish.

Now that I’m researching, I can say I really love the work myself and it rings true. I wrote her and told her that I remembered her words. Furthermore, I asked her to review my thesis and she agreed.

1 Like