Every day, people everywhere are bombarded by messages in the media that suggest that they need to drastically change how they live their lives or to take some particular action due to some sort of scientific finding. But do they really?

The short answer is "it depends." Most often, it depends upon the quality of the science or evidence supporting the action that is being promoted, and the only way to avoid serious disruption to one's daily life is to develop the ability to determine the difference between good science and bad science.

That's not necessarily easy, since bad science comes in many different forms. For instance, there's the "defective" variety, where otherwise good scientists have made an error in their methods that leads to incorrect conclusions, and which is probably the least harmful as it is likely to be quickly corrected, often by the scientists who made the error.

Then there's "deceptive" or "junk" science. Here, we find the veneer of good science, but twisted in a way to support a particular outcome. Taken to the extreme, junk science transforms into outright pseudoscience (or "false" science), where science is just used as a prop to support whatever outrageous claim is being made.

The trick then is to work out what is science and what is pseudoscience. To that end, we've adapted Steve Lower's invaluable guide for recognizing the differences between good and bad science, organizing it by distinguishing aspects and updating it while adding a number of our own observations and comments.

How to Distinguish "Good" Science from "Junk" or "Pseudo" Science
Aspect	Science	Pseudoscience	Comments
Goals	The primary goal of science is to achieve a more complete and more unified understanding of the physical world.	Pseudosciences are more likely to be driven by ideological, cultural or commercial (money-making) goals.	Some examples of pseudosciences include: astrology, UFOlogy, Creation Science and aspects of legitimate fields, such as climate science, nutrition, etc.
Progress	Most scientific fields are the subjects of intense research which result in the continual expansion of knowledge in the discipline.	Pseudoscientific fields generally evolve very little after being first established. What small amount of research and experimentation that is carried out is generally done more to justify the belief than to extend it.	The search for new knowledge is the driving force behind the evolution of any scientific field. Nearly every new finding raises new questions that demand exploration. There is little evidence of this drive in the pseudosciences.
Challenges	Scientists in legitimate fields of study commonly seek out counterexamples or findings that appear to be inconsistent with accepted theories.	A challenge to accepted dogma in the pseudosciences is often considered a hostile act, if not heresy, which leads to bitter disputes or even schisms.	Science advances by accommodating change as new information is obtained. Frequently, the person who shows that a generally accepted belief is incorrect or incomplete is more likely to be considered a hero than a heretic.
Inconsistencies	Observations or data that are not consistent with current scientific understanding generate intense interest for additional study among scientists. Original observations and data are made accessible to all interested parties to support this effort.	Observations of data that are not consistent with established beliefs tend to be ignored or actively suppressed. Original observations and data are often difficult to obtain from pseudoscience practitioners, and is often just anecdotal.	Providing access to all available data allows others to independently reproduce and confirm findings. Failing to make all collected data and analysis available for independent review undermines the validity of any claimed finding. Here's a recent example of the misuse of statistics where contradictory data that would have avoided a pseudoscientific conclusion was improperly screened out, which was found after all the data was made available for independent review.
Models	Using observations backed by experimental results, scientists create models that may be used to anticipate outcomes in the real world. The success of these models is continually challenged with new observations and their effectiveness in anticipating outcomes is thoroughly documented.	Pseudosciences create models to anticipate real world outcomes, but place little emphasis on documenting the forecasting performance of their models, or even in making the methodology used in the models accessible to others.	Have you ever noticed how pseudoscience practitioners always seem eager to announce their new predictions or findings, but never like to talk about how many of their previous predictions or findings were confirmed or found to be valid?
Falsifiability	Science is a process in which each principle must be tested in the crucible of experience and remains subject to being questioned or rejected at any time. In other words, the principles of a true science are always open to challenge and can logically be shown to be false if not backed by observation and experience.	The major principals and tenets of a pseudoscience cannot be tested or challenged in a similar manner and are therefore unlikely to ever be altered or shown to be wrong.	Pseudoscience enthusiasts incorrectly take the logical impossibility of disproving a pseudoscientific principle as evidence of its validity. By the same token, that scientific findings may be challenged and rejected based upon new evidence is taken by pseudoscientists as "proof" that real sciences are fundamentally flawed.
Merit	Scientific ideas and concepts must stand or fall on their own merits, based on existing knowledge and evidence. These ideas and concepts may be created or challenged by anyone with a basic understanding of general scientific principles, without regard to their standing within a particular field.	Pseudoscientific concepts tend to be shaped by individual egos and personalities, almost always by individuals who are not in contact with mainstream science. They often invoke authority (a famous name for example, or perhaps an impressive sounding organization) for support.	Pseudoscience practicioners place an excessive amount of emphasis on credentials, associations and recognition they may have received (even for unrelated matters) to support their pronouncements. They may also may seek to dismiss or disqualify legitimate challenges to their findings because the challengers lack a certain rare pedigree, often uniquely shared by the pseudoscientists.
Clarity	Scientific explanations must be stated in clear, unambiguous terms.	Pseudoscientific explanations tend to be vague and ambiguous, often invoking scientific terms in dubious contexts.	Phrases such as "subtle energy fields" and "sustainable development" may sound impressive, but they are essentially meaningless.
Precision	If numbers are presented in support of a scientific explanation, they must be stated with the precision and accuracy required by their level of significance as determined by known measurement error in the data from which are derived, neither more nor less.	Pseudoscience practitioners will often present numbers with a level of precision and accuracy that exceeds that supported by the known accuracy of real world data in order to give the appearance of greater validity for their claims.	A recent example of pseudoscientific deception by precision include certain economists suggesting that "a Keynesian multiplier of 1.57" specifically applies for government stimulus spending, when a wide range of studies suggest the actual multiplier may be "anywhere from 0 to 1.5" (note the difference in the number of decimal places and potential range of values!)

Other Useful Guides to Identifying Pseudoscience

Here's a short sampling of additional questions you can ask to help determine if you're confronting a claim based on science or pseudoscience.

• How to Recognize Pseudoscience - Part of the entertaining guide to the world of paranormal phenomena, the Paranormal Encyclopedia!
• What Is Pseudoscience? - Stephen G. Saupe's list of questions that need to be asked to separate the science from the pseudoscience.
• Questions to Help Distinguish a Pseudoscience from a Protoscience - Lee Moller's list of 16 questions to ask if you're dealing with a pseudoscience or a new legitimate field of science that isn't yet well established.
• Debunk the Junk - Added 19 September 2015 - Tufts University's Julie Flaherty's summary of the "10 Red Flags of Junk Science" put forth by the Food and Nutrition Science Alliance, a partnership of several professional scientific associations, including the Academy of Nutrition and Dietetics, the American College of Nutrition and the American Society for Nutrition, to help repair the reputation of a field of study that has been badly damaged by numerous questionable and unsupported scientific claims, many of which stood as its near universal consensus for decades, where the field's leading practicioners even actively worked to suppress research that could contradict the core tenets of their so-called "settled" science.
• The Proxmire Amendment May Be the Most Anti-Science Law Ever Passed. It's Still in Effect Today - Added 19 September 2015: Of course, the problems of producing valid science in the field of nutrition didn't happen by accident. RealClearScience's Ross Pomeroy describes how one influential politician made it possible.

Documented Debunkings

Sometimes, those claims you're hearing for the first time are things that have actually been around in one form or another for a long time. Here's a short list of sites that have already done the trustworthiness legwork for you!

• Snopes.com - The Internet's premier reference for urban legends, folklore, myths, rumors and misinformation. Update 31 July 2016: Over time, the quality of Snopes' debunkings has proven to be mixed.
• Quackwatch - If it's medical or health-related and not for real, you'll likely find it here.
• Climate Skeptic - Like Quackwatch, but aimed at the poorly supported aspects of global climate change science.
• JunkScience.com - Steven Milloy's site surveying a number of highly questionable scientific claims made in today's media reporting. Update 19 September 2015: Over time, we find that the site's quality in critiquing a number of science reports is mixed overall, combining a number of valid analyses with others that fall somewhat short. For an example of the latter, see John Whitehead's recent discussion of the site's coverage on the topic of contingent valuation, where negative conclusions about particular studies would appear to have been reached without necessarily being backed by sound evidence, or without consideration that the scientists behind the studies being criticized had addressed their points of criticism.
• Junkfood Science - Sandy Szwarc's blog covering ongoing issues with media reporting of nutrition-based junk science.
• Mythbusters - did you think we'd create a list like this and forget the Mythbusters?
• John Stossel - the media's leading questioner of questionable claims, from consumer issues through politics, who also blogs.
• Biggest Junk Science Stories of 2014 - Added 19 September 2015: Hopefully an annual tradition. RealClearScience offers a summary of the biggest junk science stories of 2014. (Here's 2013's edition.)
• Bad Science - Added 19 September 2015: UK science columnist's Ben Goldacre's site on the topic of science that doesn't measure up.
• Retraction Watch - Added 19 September 2015: An invaluable site that didn't exist when we first began assembling our list of resources. Retraction Watch focuses on the mistakes made by scientists who published erroneous results that subsequently required them to alert their peers and to withdraw their findings by retracting them. That, in itself, is the process of science working as it should, but the reasons for a number of retractions will periodically overlap into junk science territory (for example, conclusions based upon overly small sample sizes for statistical studies is a common theme). Of course, the difference between a real scientist and a junk scientist is that the real scientist has the honesty and integrity to own up to their errors in the interest of advancing understanding and progress within their fields of study. By contrast, the junk scientist will seek to sustain their flawed findings, even when confronted by directly contradictory evidence.
• Wrongful Convictions (Junk Science Category) - Added 19 September 2015 - The negative effects of junk science don't just show up in scientific papers - they also show up in civil and criminal courtrooms. The lawyers behind the Wrongful Convictions blog discuss its impact and how to mitigate against it when it can affect real world judgments of innocence, guilt and liabilities.
• Metabunk.org - Added 19 September 2015: A discussion forum "dedicated to the art and pastime of honest, polite, scientific investigating and debunking". If you've just seen it in the media, it is likely being discussed here!
• Improbable Research - Added 19 September 2015: The home of the Ig Nobel Prize! This site is not about junk science at all, but is instead about recognizing valid scientific studies that would appear to offer precious little value in meeting any of humanity's needs.

Image credit: J.J. at the English language Wikipedia, per the terms of the GNU Free Documentation License, Version 1.2.

Labels: quality

- posted by Ironman at 11:24 AM | Permalink | Home |