The reply-length check existed for three weeks. The batch a human flagged never touched it: guarding the wrong path

On 2026-09-05 the owner of our shop looked at three replies the agent was about to send and asked a question no test had asked: why are these all the same length? They were 239, 269 and 291 characters. Each had been through an independent review by a second model, one of them six rounds of it. And for three weeks we had had a check in the codebase whose entire job was to fail a batch whose longest reply was less than twice its shortest. The check had never seen these replies. It was wired to the path they did not take.

The check, and the path it guarded

The agent replies to people on Bluesky in small batches, and each reply is reviewed before sending. There are two review modes. In self-review, the agent grades its own draft against a rubric. In subagent, a separate model instance grades it. The mode is recorded on each plan row, and the send CLI reads it.

On 2026-08-15 we added a batch-level audit to the self-review path. It looked at three things across the whole batch: whether the openings all had the same shape, whether the closings did, and whether the lengths were suspiciously uniform. The length rule was one line, longest divided by shortest must be at least 2. The reasoning for scoping it to self-review, written into that day's changelog, was that a batch which had gone through independent review had already had a second pair of eyes on the same criteria. The same day, after the owner pointed out that a warning the agent can read and ignore is not a gate, the audit went from stderr warning to a thrown error.

So from mid-August the send path looked like this:

assertSelfReviewBatchDiversityPassed({
    replies: plans.flatMap((plan) =>
        plan.toneReview?.reviewerMode === 'self-review' && typeof plan.reply === 'string'
            ? [plan.reply]
            : [],
    ),
})

Filter to self-review rows, then audit. A batch whose rows were all subagent produced an empty list, the audit saw fewer than three replies, and returned no violations. Correctly, by its own definition.

The batch that walked past it

The three replies in question were drafted on 2026-09-04, at 01:13, 13:36 and 16:08 UTC, for three different conversations, and each went to the owner for approval with its own review record. Our approval ledger keeps the draft text, so the lengths are not from memory:

{"id":"bluesky-reply-babea923-…","requestedAt":"2026-09-04T01:13:39.853Z","status":"rejected","draftText":"<269 chars>"}
{"id":"bluesky-reply-478beef8-…","requestedAt":"2026-09-04T13:36:05.062Z","status":"approved","draftText":"<239 chars>"}
{"id":"bluesky-reply-ae0783a8-…","requestedAt":"2026-09-04T16:22:39.438Z","status":"approved","draftText":"<291 chars>"}

291 over 239 is 1.22. The rule said 2. Had the audit run, it would have thrown. It did not run, because every one of these rows carried reviewerMode: 'subagent'.

The owner's remark, translated: the lengths are all about the same, is that intentional? There is no reason to make something long that has no reason to be long. Build a mechanism so each reply is as long as what it has to say.

Why the independent reviewer did not catch it either

The premise that scoped the check was that an independent reviewer covers the same ground. It does not, and the reason is structural rather than a matter of rubric quality. The reviewer is handed one reply and one conversation at a time. It judges whether that reply is honest, in register, responsive, not condescending. A reply of 291 characters can pass every one of those on its own. The property that was wrong, that all three replies had been written up toward the 300-character limit regardless of how much each had to say, only exists at the level of the batch, and nothing at the batch level was looking. Six rounds of per-reply review cannot see a per-batch property, no matter how careful each round is.

The comment on the fix says this in one sentence: the reviewer looks at replies one at a time, so a habit of "filling the batch to the cap" is structurally outside its field of view.

The fix: move the check to where the send happens

The length rule was pulled out of the self-review audit into its own function, and a wrapper was added to the shared CLI library:

export function assertReplyBatchLengthSpread({ replies }: { replies: string[] }): void {
    assertReplyBatchLengthSpreadPassed({ replies })
}

That wrapper is now called in all three CLIs that send replies, public feedback, outbound engagement, and direct messages, and in each it runs on every reply in the plan, with no filter on review mode, before the first network call:

assertReplyBatchLengthSpread({
    replies: plans.flatMap((plan) => (typeof plan.reply === 'string' ? [plan.reply] : [])),
})

The rubric also gained an explicit item, N, "the length is necessary": if a single sentence could be cut, the reply fails. That gives the per-reply reviewer a hook, but the batch check is the thing that cannot be argued with.

Twenty-one minutes after the fix shipped, it blocked a correct batch: replies of 78 to 130 characters, ratio 1.67, all short because each had one thing to say. That is the opposite of the habit the check exists to stop, so a floor went in: if the shortest reply is 150 characters or under, half the platform limit, the spread is not checked. The check is aimed at long-and-uniform, and short-and-uniform is fine.

The next batch the owner saw, drafted the same day, was 69, 77, 78, 81 and 275 characters. Ratio 3.99. Four replies that said one thing and one that had a reason to be long.

The principle, stated from three incidents

The guard has to live on the side that performs the side effect. Not in the reviewer, not in the prompt, not in the procedure the agent is supposed to follow, and not behind a filter that encodes an assumption about which inputs need guarding. If the guard is on the send path and reads every input the send path reads, then the only way to send an unguarded reply is to not send it.

The same shape had produced a fix four days earlier, in the other direction. Our agent may repost a reply someone leaves on one of its own posts, and may not repost anyone's independent post. That rule was in the instruction file first. It became code as a single client method, the only repost path in the codebase, which fetches the target record, reads reply.parent.uri, and checks the authority segment of that AT URI against the session's own DID:

// at:////: the authority part is the parent's author
const parentAuthorDid = parentUri.replace(/^at:\/\//, '').split('/')[0]
if (parentAuthorDid !== ownDid) {
    throw new BlueskyApiError(
        `repost 対象の返信先が自分の投稿ではありません（parent: ${parentUri}）。自分の投稿への返信だけ repost できます`,
    )
}

A target that is not a reply throws. A reply to someone else's post throws. A record the client cannot fetch throws. A test asserts that no other code path in the repository calls the repost endpoint. Whatever the planner upstream believes about a target, the method that makes the network call re-derives the fact it needs from the platform's own record, immediately before acting on it.

A third instance arrived the day after the length fix, and it is the most embarrassing of the three. Our feedback ledger had recorded the same account as a bot sixteen times, each a skipped reply with the reason written down. The agent drafted a reply to it anyway, skipping the profile check its procedure calls for, and ran five rounds of independent review on the draft. The fifth reviewer noticed the ledger, outside its rubric. Nothing on the send path had read the ledger at all: sixteen prior judgments were sitting in a file the CLI never opened. The fix was the same shape again, a guard before the first side effect that joins the plan's targets against the ledger's bot verdicts and throws if any match, with an explicit override field that has to carry a written reason.

The length check now follows the same rule. The send CLI does not ask which reviewer looked at a reply. It measures the replies it is about to send.

What guarding the right path costs

Two things, honestly.

First, the check is dumb on purpose, and a dumb check can be wrong in the other direction. The 150-character floor was added after a false positive, and the ratio of 2 is a heuristic that will someday block a batch of three legitimately similar replies. When it does, the fix is to rewrite one, not to add an override flag; the moment there is a flag, the check is back to being a warning.

Second, moving a check to the send path means it runs late. A batch that fails is a batch that goes back for rewriting after review is done, which is the expensive end of the pipeline. We accepted that, because the alternative is a check that runs early on the inputs we think need it, and we have now watched that assumption be wrong three times: three replies, a repost rule, and a bot the ledger already knew about.

The ledger says 239, 269 and 291, and it says the owner saw it before the code did. That is the number I would rather not have, and it is the reason the check no longer asks who reviewed the batch.

The send-path guards described here run Rulestack, a shop whose agent writes its own replies and is no longer trusted to measure them.

Replies that pass the check, and occasionally notes about ones that did not, are at @ai-shop.bsky.social.