How to Talk to Your Robot: Keeping Coding Agents on Track

As the maintainer of Luxury Yacht, I spend a lot of time talking to LLMs. A lot.

I wouldn't say this is fun. Honestly, a lot of it is drudgery. It's keeping the robots in line. It's waiting a long time between prompts, making sure they don't do stupid things, and that the code they spit out isn't total garbage.

No matter how much time and effort I put into documentation, agents files, skills, or whatever the hot thing is this month, LLMs only have so much context, they can't truly learn anything, and those DO NOT IGNORE ME UNDER ANY CIRCUMSTANCES critical instructions you carefully designed are eventually going to get dropped out of the context window to make room for who-knows-what. And that gets more likely the longer a session goes on.

Having said that, Luxury Yacht is largely written by LLMs, and I think it's a pretty good app. I don't like the term "vibe coded" because that implies that I don't care about code quality, and that is absolutely not true. I don't blindly accept whatever the robots churn out. I spend more time in optimization passes than I do in feature work, trying to make the code the best it can be, to the best of my ability as a person with far more experience in platforms and infrastructure than in software development.

With all that in mind, I'd like to share some of the things I've done that seem to help, at least a little bit. I'm not pretending to be an expert here. It's a lot of trial and error, and I have no doubt that there are things I could be doing better. We're all learning, and the AI landscape changes rapidly.

There Are No Silver Bullets

Let's just get this out of the way: for large, complex projects, you'll be wrangling the LLMs a lot. Keeping them on track can be difficult and requires vigilance. LLMs love to be "clever", so you'll be busy making sure that they stick to established patterns, reuse existing code, and don't reinvent yet another wheel. You'll be frequently reminding them to validate that a change they made in one subsystem doesn't have unforeseen consequences elsewhere.

Agents Files

Agents files are the first place an LLM looks for the lay of the land, so this is where you want to put your important rules about how you want the LLM to behave, and tell it where it can find more information.

It won't always follow these instructions, and sometimes it'll forget the very important things you put in here, but it's where to start. If you do nothing else, create an AGENTS.md file in the root of your project or repo.

Curious about what to put in AGENTS.md? Start here.

Claude has to be different

Unfortunately, Claude doesn't want an AGENTS.md file, it wants CLAUDE.md. If you only use Claude, put the instructions in CLAUDE.md instead of AGENTS.md. But, if you use a mix of LLMs including Claude, cover your bases by putting your instructions in the AGENTS.md file, then create a separate CLAUDE.md file with only this line:

@AGENTS.md

Using multiple agents files

For a large project, you might want multiple AGENTS.md files spread around the project. This can help if you're tasking an agent with working on a specific part of the app, and reduce the amount of instructions it has to read vs. dumping everything into a single agents file in the root.

For example, Luxury Yacht has an agents file in the root that talks about overarching rules for the entire project, but then has additional agents files in the backend and frontend directories with more specific rules and info about those areas of the app.

Luxury Yacht is open-source, so you can go look at the github repo to see the agents files I use. You're not going to want to use my exact files for your project, but they might give you some ideas of how to structure yours.

Documentation

I make sure my agents files contain info about where to find documentation for the app. Let the LLMs write their own documentation. Tell them that they are writing it for themselves, so it doesn't need to be human-friendly, but rather optimized for consumption by agents.

Every time the agent modifies the app, it shouldn't consider the task complete until it updates any relevant docs with information about the changes. Instructions for this are of course in the agents files, but I sometimes have to remind the agent to do this before closing out a task.

I put all of my documentation in a docs directory in the root. Unlike the agents files, I don't split it into frontend and backend docs because the split is not always that clean. Many systems have cross-cutting concerns. I usually let the LLMs decide how to organize the docs, since that's who they are for.

I drop a README.md into the docs directory that functions as an index to tell the agents what docs to reference. Rather than try to explain exactly what I mean, I encourage you to look at the file. As with the other docs, the agent is responsible for updating this top-level README any time it makes changes that would impact it.

Skills

Skills are bundles of instructions (a markdown file, and possibly other scripts and reference files) that tell an LLM how you want it to perform a specific task. It's a broad topic that I'm not going to go too deep on in this article. There is a wealth of existing resources for skills, far more comprehensive than I could hope to be here.

If you find yourself frequently repeating a task, you might want to write up a skill for it, or ask an agent to write the skill.

For example, in Luxury Yacht, I have a skill that gives the agent information about how to make changes to the refresh subsystem, the part of the app that makes sure the Kubernetes data being displayed in the app is fresh. This is a complex and fragile system, so it's important to give the agent as much help as I can when it has to interact with it.

There's some crossover with docs here. The way I think about it is that the docs are the what and the why and the skills are the how.

Tests and Linters

A very effective step you can take early on is to enforce tests and linters. Never allow the agent to consider a task to be complete until it has written tests and run a linter.

I find Red/Green TDD (or Red/Green/Refactor) to be particularly effective for LLMs, as it makes them think a little harder about how to properly implement something. If you're unfamiliar with Red/Green tests, it boils down to this:

Red step - write a test that describes what you want. Verify that the test fails.

Green step - write the simplest code to make the test pass.

Refactor - improve the code while ensuring the test keeps passing.

After the tests, you should make sure the LLM runs a linter on the code it generated to catch the easy things that a linter is for. Good tests and a good linter will save you a lot of trouble.

Hooks

One of the newer things I've tried (and I'm not sure about its effectiveness yet) is using hooks to require the agent to perform a task at a certain point in its lifecycle.

The idea to use hooks came after a very frustrating Claude session where I just could not get it to behave. It kept making the same mistakes over and over, ignoring explicit instructions, and I asked it how to get it to stop doing that. It suggested creating a PreToolUse hook to force it to "map the blast radius" before making any changes.

The hook that I use consists of a few parts:

The settings file that tells Claude about the hook

The script that the hook runs

The file that the script writes to, which is meant to be transient so it's in.gitignore.

One reason I say that I'm not sure about the hook's effectiveness is that I currently only have the hook set up for Claude, and not Codex. But I don't think Claude does a better job with the hook than Codex does without it. It really depends on the specific task.

Adversarial Reviews

Robot vs. Robot! 🤖 🫯 🤖

This is maybe the most effective strategy I've found to generate higher-quality results. For any even moderately-complex task, I'll use a second agent to challenge the other. Typical…