Fact-checkers often do not confine themselves to just the facts.

by Matt Grawitch-psychologytoday

KEY POINTS

• A fact should not be capable of being biased, but many fact-check articles are.
• Few fact-checkers limit themselves strictly to checking facts, leading to inductive arguments that are more rebuttals than fact-checks.
• The further afield fact-checkers go from only checking facts, the greater the likelihood they will be influenced by their own biases.
• This may contribute to many people's distrust of fact-checks.

Many news sources, from CNN to Reuters to Breitbart (and many more), offer fact checks on various claims that make their way into popular discussion. However, as an article published on AllSides argued in February 2022, “fact checks often exhibit as much bias as regular news articles and analyses.” This poses a conundrum. A fact is “something that is known to have happened or to exist” (Cambridge Dictionary, 2022). Hence, a fact should not be capable of being biased, but many fact-check articles are.

Facts seldom exist in a vacuum and are generally presented as a part of a more complex claim the arguer is trying to make. When considering any claim, a corresponding argument for that claim often also has several elements. One way to dissect an argument is to use the method developed by Stephen Toulmin (also called the Toulmin method). I won’t dive deeply into the method here (as that is not the purpose of this post), but Toulmin very specifically differentiates between the grounds for a claim (the facts on which the claim is based), the warrant for a claim (the inferences or reasoning that lead from the grounds to the claim),¹ and a rebuttal (which would involve the presentation of contradictory or additional facts that might weaken the connection from the grounds to the claim). Hence, the issue that arises is whether a fact-checker is supposed to:

1. fact-check the grounds (which would make it an actual fact-check) or
2. evaluate the logical inference from grounds to claim and/or present a rebuttal.

Nieman and Sankari (2021) offered two observations when reviewing fact-checks conducted specifically by PolitiFact:

1. Thirty-three percent of fact-checks sought to render a single judgment on a complex claim, which oversimplified the actual (un)truthfulness of the facts underlying that claim.
2. More than 10 percent of fact-checks were for claims that were uncheckable (either future predictions or claims so vague they couldn’t be checked²).

If a fact-check is confined to assessing the accuracy of the grounds, it should be less open to bias because such an argument is more deductive in nature. The term fact-check actually implies a deductive process to prove the truth or untruth of a statement (or a set of statements). However, few fact-checkers limit themselves strictly to the grounds of a claim, which leads them into making more inductive arguments,³ often to rebut the argument made by the public figure.

article continues after advertisement

The AllSides article presents a good example of a fact-checker who failed to focus strictly on the grounds for a claim. It linked to a fact-check conducted in 2015 by the Washington Post on a statement made by Carly Fiorina about her rise from “secretary to CEO.” A fact-check that strictly stuck to checking the facts should have determined this statement to be true or accurate (as she did start her career as a secretary).

However, the fact-checker provided a “3 Pinocchio” rating, arguing that the imagery she was trying to conjure, as an upstart who challenged the status quo, wasn’t accurate (as her father was the Dean of Duke Law School and she went to Stanford). Rather than checking only the facts, the fact-checker assessed whether Fiorina’s success was a result of her pulling herself up by her bootstraps, which was debatable and well beyond simply checking the facts. Thus, the fact-checker blurred the lines between checking the facts and rebutting Carly Fiorina’s argument.

This is a recurring problem not confined only to this Washington Post fact-check. The International Fact-Checking Network @ Poynter, for example, does not specify what the parameters of a fact-check are supposed to be (nor is it found in its code of ethics). The further afield fact-checkers go from only checking facts, the greater the likelihood they will be influenced by their own biases.

This may lead to conclusions about truthfulness that have less to do with the facts (which are not often debatable) and more to do with the fact-checker’s belief that the conclusion logically follows from those facts (which is often debatable), an issue that both Ceci and Williams (2020) and Dotson (2022) discussed. The lack of strict standards for what constitutes a fact-check and the frequency with which fact-checkers stray from checking only the facts may help to explain why a majority of respondents report negative attitudes toward various fact-checking sources (Brandtzaeg & Følstad, 2017), especially when it comes to social media companies (Kemp & Ekins, 2021).

But there’s also the matter of the biases brought by the consumer that bears on this issue. Li and Wagner (2020), for example, found that consumers were less likely to adjust their beliefs because of a fact-check if they possessed strongly held beliefs that contradicted the facts. This could occur as a result of motivated reasoning—our biases toward the desired conclusion can predispose us to look for reasons to discount facts that might dispute that conclusion (especially when those facts are based on statistics).

Still, some of this effect may occur, in part, because many fact-checkers no longer confine themselves to the Sgt. Joe Friday mantra—“All we want are the facts, ma’am”⁴—with too many fact-checkers more interested in rebuttal than fact-checking. A stricter focus on evaluating the facts, with less emphasis on more inductive rebuttals, could improve the fact-checking process for both the fact-checker and the consumer.

References

Footnotes

[1] He also defined backing as a way to further justify a warrant, which could, technically, be another set of grounds (as it is an argument for the warrant).

[2] In some cases because there were no facts offered in support of the claim.

[3] This was an issue I discussed recently, though not about fact-checkers specifically.

[4] Coincidentally, the expression “just the facts, ma’am” is frequently attributed to Joe Friday, but as Mikkelson (2002) reported in a fact-check on Snopes, Sgt. Joe Friday never actually uttered that line exactly, so I used his direct quote.