Data Engineers: Just Do Code Reviews

In Code Complete, Steve McConnell writes:

Software testing alone has limited effectiveness — the average defect detection rate is only 25 percent for unit testing, 35 percent for function testing, and 45 percent for integration testing. In contrast, the average effectiveness of design and code inspections are 55 and 60 percent.

That was published in 1993. The evidence has only accumulated since. Peer code review is the single most effective defect-removal technique available to software teams. Nothing else — not tests, not CI pipelines, not observability tooling — comes close on a per-hour basis.

Software engineers have known this for fifty years. Data engineers mostly pretend it doesn’t apply to them.

It does. And the stakes, if anything, are higher.

Why the excuses don’t survive contact with the data

The objections are familiar. We’re not real software engineers. We’re too small a team. We need to move fast. Our tools don’t support it. None of them hold up.

The evidence on review effectiveness is overwhelming. Michael Fagan’s original 1976 IBM study found formal inspections caught 82% of errors before unit testing even began. NASA’s Software Engineering Laboratory found code reading detected roughly twice as many defects per hour as testing. Russell’s 1991 AT&T study found that each hour spent in review avoided 33 hours of downstream maintenance — making inspections up to 20× more cost-effective than testing per defect found.

Modern numbers are just as clear. The SmartBear/Cisco study of 2,500 reviews across 3.2 million lines of code established that reviews conducted within sensible limits — under 400 lines per session, under 500 lines per hour, sessions capped at 60 minutes — yield 70–90% defect removal. Google’s case study of 9 million reviewed changes across 25,000 engineers proved the practice scales to 20,000 commits a day, with median review latency under four hours. The DORA 2023 State of DevOps report, surveying nearly 3,000 practitioners, found that teams with faster code reviews had 50% better software delivery performance. Elite DORA performers ship with a 7× lower change failure rate than low performers.

SmartBear’s State of Code Review asked 800 respondents what most improves quality. Code review ranked first, with unit testing a distant second. Among teams satisfied with their software quality, 80% used tool-based peer review and 82% had documented review guidelines.

This is not a close debate in software engineering. It shouldn’t be one in data engineering either.

Data pipelines fail differently — and that makes reviews more important, not less

Here’s the thing that makes the data engineering situation more precarious: application bugs are loud. They throw exceptions, crash processes, page on-call at 2am. The feedback loop that keeps application engineers honest — run it, watch it break — works quickly.

Data bugs are quiet. The pipeline is green. Row counts look reasonable. The dashboard renders. And the number is wrong.

Benn Stancil describes it precisely: silent data errors “don’t look suspicious and they trigger no warnings, in observability tools or in our manual spot checks. Instead, they often linger undetected, slowly and silently pushing the analytical assets that use them further and further from reality — until, during a board meeting…”

Barr Moses of Monte Carlo has noted that time-to-detection for silent data errors can be measured in months, not minutes. Her 2023 survey of 200 data leaders found the average organisation deals with 67 data incidents per month, with average resolution time up 166% year over year to 15 hours per incident, and 68% of teams taking four or more hours just to detect a problem. In the 2024 follow-up, two-thirds of data leaders had experienced a data incident costing over $100,000 in the previous six months.

The “run it and see if it breaks” safety net that keeps application code marginally honest doesn’t exist for pipelines. The only reliable substitute is another human reading the code before it runs.

The public case studies have already priced in what happens when you skip it

These are not theoretical risks.

Unity Software told investors in Q1 2022 that ingesting bad training data from a large customer cost it roughly $110 million in 2022 revenue, erasing approximately $5 billion in market capitalisation in a day. A data pipeline integrity problem, undetected until it had been compounding for quarters.

Citigroup paid a $400 million OCC fine in 2020 for long-standing failure to establish effective risk management and data governance programs — followed by another $135.6 million in 2024 for insufficient progress on remediation. Data quality and governance failures, sustained over years.

Equifax sent incorrect credit scores to lenders for three weeks in 2022, shifting scores by 25 points or more for around 300,000 consumers. The downstream decisions those scores informed — loan approvals, interest rates, credit limits — cannot be unwound.

Public Health England lost roughly 16,000 positive COVID-19 test results in October 2020 because an .xls pipeline hit Excel’s row limit. Those contacts were not traced.

Every one of these is, at root, a data-pipeline change that no sufficiently rigorous second set of eyes reviewed before it reached production. The review wouldn’t have guaranteed they caught it. The absence of a review guaranteed they had no chance to.

The broader cost is already priced in — you just don’t see the bill

Gartner’s running estimate is that poor data quality costs the average organisation $12.9 million per year. Thomas Redman, writing in Harvard Business Review, put the drag on the US economy at $3.1 trillion annually. His most uncomfortable finding, from measuring 75 executives’ own data: only 3% of company data meets basic quality standards, and 47% of newly-created records contain at least one critical error.

The practitioner surveys agree. Great Expectations’ 2022 study found 77% of data practitioners report quality issues, with 91% saying those issues hurt company performance. dbt Labs’ State of Analytics Engineering 2024 reported 57% of data professionals cite poor data quality as their top challenge — up from 41% in 2022. The problem is getting worse. Monte Carlo’s telemetry puts it at roughly one incident per fifteen tables per year in production. Forrester’s 2023 Data Culture and Literacy Survey found more than a quarter of respondents lose over $5 million a year to poor data quality, with 7% losing more than $25 million.

The worst number: Monte Carlo’s 2022 data suggests data engineers spend roughly 40% of their working week — two full days — firefighting quality issues. Every hour of that is an hour a peer review before merge might have prevented.

What the data community has actually said about this

Michael Kaminsky at Locally Optimistic is clear: “Code review for analytics is often substantively different from code review for software engineering because reviewers need to check the business logic and the analytical methods as well as the code.” Different, meaning harder and more consequential — not optional.

Tristan Handy’s argument that mart models should be treated as stable interfaces, with versions and deprecation windows like APIs, is now the foundation of dbt’s contracts feature. It is also the right frame for what a reviewer is protecting: not just the code, but the downstream trust placed in it.

Emilie Schario’s observation remains accurate: “Data is behind software development when it comes to learning and implementing the best practices of DevOps.” The review is how you close the gap.

And Maxime Beauchemin, who built Airflow and Superset, has documented the cultural cost of data engineering’s second-class status. When the team is small, when velocity is the priority, when “we’re not really software engineers” is the ambient assumption, reviews are the first thing to go. And the work becomes fragile, tribal, and unreviewable — not because of the tools, but because of the habits.

Datafold’s write-up of analytics-engineer review culture names the failure mode precisely: “Faced with the prospect of move fast and break stuff vs. move slow, carefully review, and maybe break less stuff, most reviewers will default to the first option, and ‘LGTM’ their way through the day.”

That is not a review culture. That is a rubber-stamp culture with extra steps.

What a data engineering review should actually cover

The community consensus — distilled from dbt Labs’ PR guidance, the GitLab Data Team Handbook, Locally Optimistic, Datafold, and the dbt Discourse — converges on a specific checklist. One focused on the failure modes unique to data, not imported wholesale from software engineering.

This is not a burdensome list. It’s the checklist a thorough engineer would run through mentally anyway — just written down and shared, so someone else can run through it too.

Just do it

The case for code review in software engineering was settled by Fagan, IBM, NASA, Cisco, Microsoft, Google, and DORA — across decades and millions of commits. Reading code finds roughly 70% of defects before they ship. Teams that review fast deliver 50% better. Silent errors cost enterprises tens of millions per year and mid-market teams their credibility with the business.

The case for code review in data engineering is not weaker than in software engineering. It’s stronger, because pipelines fail silently, errors compound downstream before anyone notices, and the financial consequences show up on earnings calls rather than error logs.

The practitioners who have thought hardest about this — Kaminsky, Stancil, Moses, Sanderson, Beauchemin, Handy — all land in the same place. Every SQL change, every dbt model, every DAG, every schema migration deserves a second set of eyes before it touches production. Not because data engineers are software engineers, but because the cost of pretending the practice doesn’t apply has become impossible to afford.

The tools exist. The checklists exist. The statistics exist. The only thing missing is the decision to stop skipping it.