The checking routine
A summary produced by an artificial intelligence is not the source. Here is a routine in three steps, based on what the best professional fact-checkers do.
A summary produced by an artificial intelligence is not the source. It is a compression, and a compression can cut a nuance without flagging it: a date, a reservation, a disagreement between experts. The problem is not new, it long predates language models. What changes is the speed at which a summary is produced, and the trust its finished form inspires.
A summary written by a person can go wrong by taking a shortcut, but that person generally knows they are simplifying. A language model draws no such distinction: it produces the most probable wording, whether or not it keeps the reservation present in the original text. Nothing in its output tells a faithful summary from one that has lost a nuance along the way.
What the best fact-checkers do
In 2019, Sam Wineburg and Sarah McGrew published a study comparing three groups faced with the same task: judging whether a website deserves to be believed. 10 historians with a doctorate, 10 professional fact-checkers, and 25 undergraduate students. The historians and the students stay on the page and examine it: a logo, a domain name that sounds serious, a careful layout are often enough to convince them. The professional fact-checkers do the opposite: they leave the page within seconds, open other tabs, and look for what other sources say about that site or about the claim itself. This practice has a name, “lateral reading”, as against the “vertical” reading that stays inside the page. The measured result: the professional fact-checkers reach better founded conclusions, in far less time than the other two groups.
A summary produced by a machine strips away precisely the contextual clues that usually help in judging a page: no domain name left to examine, no layout left to assess, only a string of smoothed sentences with no rough edges. The checking routine then moves attention to where it really belongs, the source itself, rather than to the wrapping of the summary, which says nothing about it.
The routine, in three steps
The SIFT method, published the same year by the researcher Mike Caulfield, turns this practice into repeatable moves. Applied to a summary produced by a machine, it comes down to three checks, in this order.
- Reopen the original source and locate the exact passage. A summary rarely quotes a whole sentence: it paraphrases. Finding the precise passage it is talking about makes it possible to check that it exists, and that it does say what the summary makes it say.
- Check whether the source presents the fact as settled or as debated. A sentence can be accurate and yet stripped of a reservation: “some researchers dispute”, “this result remains contested”. A summary that flattens that nuance into certainty turns a hypothesis into a truth.
- Check the date of the source. A fact that was right on one date can be out of date a few years later, above all on a fast-moving subject. An undated source, or one dated without the summary saying so, deserves a further search before being taken up.
Each of these three steps answers a different failure: the first catches an approximate or invented quotation, the second a certainty that was never one, the third a piece of information true yesterday and false today.
Why this is no waste of time
The Wineburg and McGrew study carries a subtitle that sums up its central result: reading less and learning more. The professional fact-checkers do not spend more time than the other two groups: they spend less, because they look for the right information in the right place rather than exhausting the page in front of them. The three steps above follow the same logic: they ask nobody to read everything again, only to reopen the source once, on the precise point the summary puts forward.
This routine does not replace expertise on a technical subject: it makes no claim to turn anyone into a specialist in the field concerned. It catches the most frequent failure: a nuance lost in the course of a summary, not a learned error of reasoning. And when the check fails on a page of The AI Manual itself, there is an address for exactly that: Report an error.
Sources
- Lateral Reading and the Nature of Expertise: Reading Less and Learning More When Evaluating Digital Information Sam Wineburg, Sarah McGrew, Teachers College Record, vol. 121, no. 11, 2019.
- SIFT (The Four Moves) Mike Caulfield, Hapgood, 2019.
This article is published under the CC BY 4.0 licence: copy it, translate it, republish it, crediting ODERSA.