Copy of my FLF Epistemic Case Study Competition

The contest deadline was today. Here's what I submitted. Original here.

Contest submission: Epistack-HowTruthful

This is a submission to FLF's Epistemic Case Study Competition.

If you're not a contest judge and just want agent skills for ingesting and improving arguments, go to README.md.

This is about a 10-12 minute read.

Judging this contest should be an easy job, and almost is

The FLF has asked for tools and methodologies to make reasoning easy to scrutinize. Assume a contestant has made such a tool or methodology. The contestant should be able to use their tool or technology to lay out reasoning for why their submission should win. It's then easy to scrutinize that reasoning, and thus easy to see whether or not their submission should win.

The caveat is, you've never used this tool or methodology before. If such tools and methodologies were common already, why make a contest to create one? They're either nonexistent or uncommon, so even if the tool or methodology is merely a new combination of existing concepts, there's going to be a learning curve.

This is not hypothetical. The submission you're judging now is a methodology and tool for making reasoning easy to scrutinize. The next sections will walk you through the learning curve, and then you'll scrutinize my reasoning for why this submission is a winner.

What "easy to scrutinize" looks like: HowTruthful

Spoiler alert: The tool is called HowTruthful. Rather than give step-by-step instructions for using it, I'm going to explain the reasoning and motivation behind how it works. Then how to use it will click right away.

The obvious way to represent reasoning that everyboy missed

For most of the exactly-200-year history of argument maps, they've been made using ink or pencil on paper. One innovation from their original form was to draw circles around the statements so that they don't run into each other on the paper. Another was to draw directional arrows instead of symmetric lines, so that conclusion-to-premise could be drawn in any direction, not just downward on the paper. Finally, we got computers. There was no longer any edge to the paper, and the circles could be moved to make more room whenever a new one came in.

Arrows connecting circles in two dimensions. That's argument maps since, at the latest, 1958. And when software engineers see arrows connecting circles in two dimensions, they recognize a graph. Software engineers should know that there are other ways to represent graphs besides two-dimensional circle/arrow diagrams. The most prominent example is a hypertext web, ubiquitous to the point where "Internet" and "web" are often used interchangeably.

Somehow, the idea of using hypertext to represent the graph of an argument map is so invisible that even Scott Alexander, a knowledgeable and insightful blogger prominent in the rationalist community, when writing about the abundance of argument-mapping projects, writes as if the circles-and-arrows representation is the only one. "Once you have enough of these circles, aren’t you fighting the argument-mapping idea rather than benefiting from it?" Similar objections are noted on Wikipedia.

When you put statements in circles and connect them with arrows in two dimensions, you run into scaling problems with large numbers of statements. When you put statements in pages and connect them by hypertext, you scale much better. Every statement has a page where you look primarily at the statement, and secondarily at its immediate pro and con connected statements. What you're looking at is essentially a high-level summary. You click into a pro or con statement to dig deeper. It scales to however many statements you want.

Scrutiny and assessing truthfulness are intertwined

Picture yourself looking at the highly-focused format described in the previous section: a statement, and a high-level summary of why you should or shouldn't believe it. Why are you looking at it? You're looking at it in order to decide how truthful it is. Why else would you scrutinize it?

Every statement on HowTruthful is accompanied by a colorful 1-5 rating scale. Everything starts out as a colorless 3, debatable. When there are debatable pros and cons, you click into them, until you reach a statement that's self-evidently true or false, or that has enough non-debatable pros and cons for you to decide its truth. Then a single click changes the colorless 3 into one of the colorful truth values. The process of navigating down through the argument map, and adding color on your way back up, is fun.

For this reason, I've made no attempt in this submission to automate the assessment step with AI. If you really want to let an AI assess truthfulness in a file you want to import to HowTruthful, you can probably just ask it. I haven't tried, though, because the whole point of letting a human scrutinize is to let a human assess.

Where AI proves useful

Clicking the pretty colored rating discs is the fun part of using HowTruthful. The tedious part is creating the graph of statements. You type in "The sky is blue." You click through to its page and stare at it. You decide you need some evidence before you can rate its truthfulness. You click the "Pro" header and type in "It looks blue." Then you click through to that statement's page. You notice that this statement is context-dependent and click it, and edit to "The sky looks blue." You click the Save button and continue.

We have computers. Computers process information. Why not have the computer process the freeform text you were looking at when you decided you wanted to scrutinize reasoning, and transform it into a web of context-independent statements linked by pro and con relationships? If you had asked me this question before modern LLMs came out, I would have laughed and told you computers don't work that way. But today I'd answer that that's a great idea.

Why not integrate AI directly into the HowTruthful web interface?

My vision for HowTruthful is a place where adversaries can meld their arguments and arrive at what, for them, are cruxes. It needs to be a platform people trust. Having a single built-in AI for making the initial draft of an argument would rightly lead people to wonder if bias was secretly being introduced. For this reason, I think it's important to let people drive the AI parts of the process from their own choice of agent, using skills that they can inspect and modify themselves.

That's it for backround and motivation. Now it's time to try it.

Options for trying it out

Large

Use Claude Code or your favorite alternative to open this repo as a project. Follow the README.md instructions to install optional prerequisites and start prompting. It may take several minutes for your LLM to ingest a large corpus. Try pointing it at the contest announcement and asking it to ingest the links for the 3 case studies.

Ask it to import what you ingested. You'll be taken to a HowTruthful page where you scroll down and click Import. Then start clicking statements as described above.

Medium

If you trust me, you can skip trying out the LLM agent skills yourself and just believe my descriptions of how I used the skills to create the examples below. Then do the click / scroll down / import thing to assess how well what I did worked.

With the eggs case study, I used an early version of the skill. It produced an imperfect graph. I fixed up one branch of the argument and rated truthfulness according to my own opinions looking at the evidence.

Click Are eggs good to eat?, scroll down, hit the Import button, and explore.

After vibe coding the howtruthful-ingestion skill for another week, I mapped the LHC argument. Click The LHC will not create a black hole that destroys the Earth, scroll down, hit the Import button, and explore. Click those colored discs to rate truthfulness. This will only be stored in your own web browser on your own device.

With the COVID-19 origins case study, I spent significant time vibing…