Krakowski, Claude, and *Modern Times*

Many readers wanted to know how Ariel Krakowski ran his AI proofread of Modern Times, so he’s written a guest post to answer your questions. Here’s Ariel:

Outside of the tech world, many people are unfamiliar with how advanced AI has become. Some still think “AI” is just a chat app that spits back text based on predicting the most likely next word. But the AI models have far more training beyond next-token prediction and the AI chat tools can spend time searching and thinking before providing an answer. AI tools can also do much more than just chat - they can develop software, analyze data and navigate web pages. Many of these capabilities require users to leave the regular chat interface and use separate tools, and these tools generally require payment as well.

Recently, Bryan Caplan asked ChatGPT to fact-check Paul Johnson’s book Modern Times. I figured I could do some additional fact-checking within Claude Cowork, a tool which has far more “agentic” capabilities than a regular chat. A chat app can provide a text response to your query, but an agentic tool can create its own plan, use tools, run iterative loops, execute code, and determine the next steps of action. For fact-checking, an agentic tool can carefully go through a book page-by-page. Claude Cowork is geared at non-programmers, so it has a more user-friendly UI than coding-focused tools and runs in a safer “sandboxed” environment by default (although it still has some risks).

I downloaded a PDF of the book from the Internet Archive; it was one of the top results when searching “modern times by paul johnson”. I generally support following copyright rules so I also purchased the book from Amazon, but I needed a DRM-free copy to pass to AI. I then put the PDF in Claude and prompted it:

Please go through the attached book (Modern Times) and evaluate each significant fact or claim for accuracy, and then collect all the errors.

Claude asked some clarifying questions and I told it to do a full sweep of the book and flag potential errors, and then go through the flagged errors afterwards to verify. Going through a 784-page book uses up a lot of tokens, and I’m only on the $20 monthly Claude plan so I hit my session limits a few times. When that happened, I either used additional credits I had in my account, or paused the analysis and resumed later when the limits reset. I was using Claude’s Opus 5 model for the analysis (on the “high” effort setting); it would have been cheaper to use the “Sonnet” model. When the analysis finished I noticed many of the errors were pretty minor so I asked Claude to highlight the important ones. It was also able to publish a nicely styled web page with all the information.

One limitation of the initial analysis is that Claude only reviewed a sample of the flagged claims; it would have taken much longer to review everything. I had it check a random sample of the rest, and it turned out that about 15% of the unverified flags were false positives, a higher rate than the first batch because those had been hand-picked as the clearest cases. This reduced the estimated number of true errors in the book from around 600 to around 500 errors. I asked it to add a “Corrections and calibration” section to the analysis page with this information.

One of the more interesting claims Claude made is that Johnson’s errors have a specific bias:

Where the victims are of the Left’s making, Johnson takes the high estimate (Cambodia, the Cultural Revolution, the Gulag, Soviet war dead); where they are of the Right’s, he takes the low one (Sétif, Philippeville, Chile, the Spanish repression, Franco’s 1947 referendum). That is not itself a factual error, but it shapes what a reader takes away, and it is documented in Part Two.

That’s a claim about an overall pattern rather than a specific fact. To verify this, you’d need to compare Johnson’s numbers to the published estimates, and also investigate whether there’s any bias in the published estimates themselves. More generally, AI is good at checking facts, but can’t be relied on to the same extent for more complex questions and judgment calls.

ChatGPT was more limited in the analysis of the book it did for Bryan, since he was using the chat app, not “ChatGPT Work”. It didn’t use a full agentic loop, so it couldn’t be as thorough in checking for errors or in carefully going over its own findings. For example, one of the top errors it highlighted was:

Johnson says Pentagon Papers participants could not be jailed in America because the press enjoyed constitutional privileges “under the Fourth Amendment.” The press clause is of course in the First Amendment.

Claude pointed out that Johnson actually wrote “First Amendment” on page 651, and the phrase “Fourth Amendment” doesn’t appear anywhere in my 1991 edition of the book (both the PDF and the Kindle version). It seems likely that ChatGPT was misled by an online comment about the text, perhaps a different edition of the book had this mistake. However ChatGPT said it was checking the 1991 edition of the book but wasn’t able to notice this discrepancy, nor did it give any page numbers for any of the mistakes it claimed. This demonstrates the limitations of a chat tool for this kind of task.

It’s also interesting to compare the error rate in Johnson’s book with the error rates in other books. I asked Claude to fact-check A Short History of the World by H.G. Wells, since it’s out of copyright and easy to find a good-quality PDF online. I focused on volume 3, which covers the French Revolution to 1919. In this case the text was small enough that Claude could review all the flagged factual claims. The error rate in this book was over 2x the error rate in Modern Times, even when only counting facts that were known at the time the book was published. However the Wells book has a higher density of facts and Claude was able to check it more closely. The full analysis can be seen here.

Ideally, one should develop consistent standards for this fact-checking so that people can review many different books in a neutral manner. To help with this, I used Claude to create a Claude Skill so any book could be analyzed with a similar approach, and I published it on Github. This can be installed in Claude Code or Claude Cowork and can be modified as needed.

As sophisticated as this analysis was, it still had many limitations. Claude only spot-checked the book’s footnotes rather than working through them systematically, and it doesn’t have access to most of the books cited. It also made mistakes of its own - 9 of the initially flagged errors in Modern Times had to be withdrawn. Modern agentic AI has a lower error rate than earlier versions of AI, since it can look up more information and double check its claims, but it still makes mistakes. When an AI mentions something you’re unsure about, ask it to confirm a specific fact and it will check more closely. To further improve the fact checking process, one could develop a more sophisticated pipeline where every claim in a book is categorized, the sources are checked, one AI does an initial analysis on everything, and then another AI automatically does a second pass. And then run this tool on every book. And while we’re at it, maybe run it on everything published anywhere…

Ariel Krakowski is a software engineer at Google, but the opinions expressed here are his own. You can follow him on Substack at zappable.com or subscribe to his podcast on Spotify or YouTube.