Scaling security reviews at 1Password: Solving the context and nondeterminism problems

In our last post, we shared how we began to scale our security code review process with SAGE. We discussed how we gathered historical Product Security (ProdSec) review records to create a 1Password-specific ruleset, the three-stage Finder/Critic/Judge pipeline, and the limitations of our v1 implementation. Above all, human ProdSec reviewers still had to bring full context to the findings: where the trust boundaries lie, which directories are sensitive, and whether mitigations exist elsewhere in the codebase.

Our goal for v2 was to help SAGE understand our entire codebase. Many of our GitHub repositories are huge, including our client and server monorepos. That means we have way too much information to fit within any LLM’s context window. We had to find a way to let SAGE perform deeper reasoning about the PR diffs it reviews without the codebase itself.

There was another hurdle. As we built v2, we ran into a fundamental LLM trait: they can’t reliably produce the same output twice. We knew we had to do our best to manage this nondeterminism so we could trust SAGE to be a relatively consistent security reviewer.

We had two things to figure out: how to fit a lot of data into a context window, and how to get consistent output from inherently inconsistent tools. If we could solve those riddles, SAGE wouldn’t just know 1Password, it would finally understand it.

And it would earn the name SuperSAGE.

Compressing context with scaffolding

As it turns out, our Security Research team had already developed a Python proof of concept designed to compress our code context. It was a set of LLM prompts that generated one SCAFFOLDING.md file per source directory. Those scaffolding files carried compressed structural context like sensitivity ratings, attack surfaces, trust boundaries, and file summaries. It was a great foundation; we just had to productionize it as a Go rewrite on top of SAGE v1’s model-agnostic llm.Client harness.

To start, the PoC took inventory of our code structure. Any…