![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
As a person who makes no secret of having a PhD in Physics and also of being into speculative fiction, I get a lot of URLs thrown at me about the latest in this or that oddball theory. As a result, I see a lot of stuff that scores high on the Crackpot Index. It's a useful tool for non-experts too, to help sift out the potentially groundbreaking new science from, well, crackpots ("psychoceramics").
However, there's one thing I noticed recently that's missing from the Crackpot Index (and I've emailed its creator with a suggestion to add it): claims of peer reviewed articles. This can be pretty insidious, and it's not just lone mad scientists using it, the Intelligent Design movement likes to use it as well. Turning one of the pillars of scientific integrity against science by claiming its protection for themselves.
The thing is, there's some loopholes that the public (assuming they even know about peer review in the first place) isn't aware of, and the "The fools at the Academy laughed at me!" types take full advantage of them. Here's a few, but there's probably others.
A related issue is that of replication studies. A replication study does not have to be successful to be counted as a replication study, you see. It just means someone thought it was worth the time to poke at the original study and see if they could do it themselves. When making a radical new claim, it is often easier to set up something like this than when making an incremental claim, because a study that shows the claim to be false is often easy to devise. Hence, when a crackpot claims X number of replication studies, what they're not telling you is that all X failed to reproduce the effect. Or, often enough, one or two did reproduce the effect due to a fluke or due to inconsistent results combined with wishful thinking, so the crackpot will have someone to send reporters to for confirmation, at least until the successful reproducers try again more carefully and find nothing.
Peer review and replication are two of the biggest self-correcting mechanisms of science, but they're not foolproof in the short term. The system can be gamed, it has holes, and it often takes a long time for replication studies to be absolutely certain one way or another.
Thus, if something seems to be making it past these, you have one other question to ask: how big are the claims? The grander the claims, the less likely they are to be true, in general...just because huge breakthroughs are rare in any case. But big claims, if they have a shred of reality to them, will make it into the scientific news press quickly enough (if not as quickly as the regular press in cases of self-promotion). So check the library for Science News or Physics Today or whatnot, or the webpages of major scientific organizations like the American Physical Society. If nothing shows up within a month of the popular press piece, odds are that the claims are no more than psychoceramics.
However, there's one thing I noticed recently that's missing from the Crackpot Index (and I've emailed its creator with a suggestion to add it): claims of peer reviewed articles. This can be pretty insidious, and it's not just lone mad scientists using it, the Intelligent Design movement likes to use it as well. Turning one of the pillars of scientific integrity against science by claiming its protection for themselves.
The thing is, there's some loopholes that the public (assuming they even know about peer review in the first place) isn't aware of, and the "The fools at the Academy laughed at me!" types take full advantage of them. Here's a few, but there's probably others.
- Crackpots are peers too. There are journals dedicated to some pretty out-there things, and they can use the peer review process just as easily as mainstream journals do. And they have decent reason to, peer review isn't just an up or down vote thing, there's always constructive criticism as well, and an article often gets through the second time after fixing things suggested by the reviewers. Just because you're publishing Atlantean Physics Quarterly doesn't mean you can't benefit from making sure your articles are (relatively) coherent and that your authors are made aware of other work being done in the field. Anyway, having peer-reviewed articles in krankwerks journals doesn't mean your work has merit, but it DOES mean you can tell reporters that you have peer-reviewed articles on the topic.
- Not all journals try to filter based on facts. Some peer-reviewed journals are intended more to provoke discussion than anything else, and their review process is aimed more at the question, "Is this interesting?" than the question, "Is this correct?" You can get your wild ideas published in these and people will point out your errors in followup articles.
- The articles may not be directly about the theory. Let's say you have some new theory that violates Physics As We Know It. But you also have devised several clever instruments in the pursuit of verifying that theory. You can easily get articles accepted about those instruments in journals dedicated to measurement, and then turn around and tell people that you have articles published that relate to your theory. This, while true, will be misleading, since the journals may never have seen your theory, much less evaluated it.
- The articles may not be about the theory at all. In an extreme case, someone who has published articles in "regular" science may simply tell reporters how many peer-reviewed articles they have published in total, without revealing that none of those articles have any bearing on the theory they're expounding on to the press (or at best, the work that was published inspired them in their new direction somehow). This is, of course, just this side of outright lying, but a True Believer often overlooks things like that when trying to promote their ideas, or convinces themself that the older papers really did show the early signs of the new theory.
A related issue is that of replication studies. A replication study does not have to be successful to be counted as a replication study, you see. It just means someone thought it was worth the time to poke at the original study and see if they could do it themselves. When making a radical new claim, it is often easier to set up something like this than when making an incremental claim, because a study that shows the claim to be false is often easy to devise. Hence, when a crackpot claims X number of replication studies, what they're not telling you is that all X failed to reproduce the effect. Or, often enough, one or two did reproduce the effect due to a fluke or due to inconsistent results combined with wishful thinking, so the crackpot will have someone to send reporters to for confirmation, at least until the successful reproducers try again more carefully and find nothing.
Peer review and replication are two of the biggest self-correcting mechanisms of science, but they're not foolproof in the short term. The system can be gamed, it has holes, and it often takes a long time for replication studies to be absolutely certain one way or another.
Thus, if something seems to be making it past these, you have one other question to ask: how big are the claims? The grander the claims, the less likely they are to be true, in general...just because huge breakthroughs are rare in any case. But big claims, if they have a shred of reality to them, will make it into the scientific news press quickly enough (if not as quickly as the regular press in cases of self-promotion). So check the library for Science News or Physics Today or whatnot, or the webpages of major scientific organizations like the American Physical Society. If nothing shows up within a month of the popular press piece, odds are that the claims are no more than psychoceramics.
From:
no subject