Cut the Noise

01Last November

Last November felt like a turning point. After months of experimenting with AI coding tools, the models began producing code I felt I could trust. That changed what we needed from verification.

02A reviewer we could trust

This illustrated paraphrase of my November 12 message to Angie captures the problem: we needed better models for code review. Our early reviewer was noisy enough that coworkers deleted its comments. As coding models improved, validation had to keep up.

03Cut the Noise

I’m Joah Gerstenberg, a tech lead at Block working on risk-based agent autonomy. Linters, tests, and compilers only take us so far. This talk covers what I’ve learned while building a reviewer we can trust.

04The old software factory

Writing and reviewing code used to pace the software factory. We knew our libraries closely, verified each other’s work, and learned from one another through review. Features took months to plan, build, and ship.

05Review is the new rate limit

Agents flooded our review inboxes with changes, but we still had to decide whether those changes were safe for customers. Increasing code production did not remove the need for verification. Review became the new rate limit.

06Code reviewed per human

The typical reviewer at Block is reviewing about three times as many lines of code as a year ago. Human attention is spread thin, and review increasingly becomes a ritual of trust: do we trust our teammates’ plans and their agents’ understanding of the bigger picture?

Block internal data

07The code was never the goal

My background in UX engineering taught me that code is a means to a customer experience. The goal is a customer buying a coffee without being double-charged. With agents writing code, understanding that experience matters even more.

Sources: Programming as Theory Building, 1985

08We are still responsible

We want to deliver an experience we are willing to sign our names to. When an agent gets it wrong, pointing at the agent does not help the customer. We still hold responsibility for its output.

09Calibrate your reviewer to past human reviews

Historical reviews give us human-labeled examples to evaluate against. Those comments are outputs of human reasoning, rather than a complete account of that reasoning. They are useful as a measuring stick, but reusing them as model guidance can introduce noise.

Sources: DoorDash Engineering Blog · 2026

10When the reviewer got noisy

We built a pipeline to learn from past human reviews. An April bug fix exposed more of those learned checks, and the reviewer became noisy enough that we shut it down and rebuilt it. The experience changed how we approached precision.

Block internal data

11A reviewer that’s muted catches nothing

DoorDash and Figma described similar lessons: false positives cost a reviewer its audience. DoorDash prioritized precision in its rebuild, while Figma described a minimum precision bar before findings could be posted.

Sources: DoorDash Engineering Blog · 2026 · Figma Blog · 2026

12Offline versus online evals

Offline evaluations compare findings with an answer key of human-labeled examples. Online evaluations observe how the reviewer performs in production. We need both controlled measurements and evidence from the live workflow.

13Precision versus recall

Precision is valid findings divided by all findings: true positives divided by true positives plus false positives. Recall is known defects found divided by all known defects: true positives divided by true positives plus false negatives.

Sources: Google ML Crash Course: precision and recall

14Address rate versus post-merge fix rate

Address rate measures findings acted on out of settled findings. Our post-merge fix rate measures changes made from findings that later require a corrective fix or revert, divided by all changes made from findings.

15Prioritize precision over recall

A useful reviewer stays quiet when it has nothing worth saying. Optimizing precision protects human attention, even if some lower-severity issues are missed. OpenAI’s alignment work describes this same priority for usable code review.

Sources: OpenAI Alignment Blog · 2025

16Online evals: address rate

As we reduced false positives, engineers began taking the reviewer’s comments seriously again. Over time, more findings led to code changes. Address rate helps us monitor whether the reviewer is useful in the live workflow.

Block internal data

17ReviewBench

ReviewBench gives code reviewers a shared benchmark for precision and recall. At launch, Copilot’s grounded recall was 26%, illustrating how much room there is to improve coverage on the benchmark’s known issues.

Sources: ReviewBench

18Different reviewers find different issues

Our reviewer and another reviewer overlapped on only 22% of findings, using the larger finding count as the denominator. Most findings were unique to one reviewer. Different models and perspectives can help cover different blind spots.

Block internal data

19Three specialists improve recall

Our reviewer uses three lenses: Behavior & Contracts, Reliability & Operations, and Security & Trust Boundaries. Each specialist examines the change from a different perspective. Running multiple specialists improves coverage compared with a single reviewer.

Sources: Block review: specialist lenses

Block internal data

20More layers increase precision, more reviewers increase recall

The specialist reviewers report candidates to a coordinator that merges duplicates and tries to disprove each finding. A judge rechecks dropped candidates to restore genuine defects. This gives us broader coverage while filtering noise before findings reach a human.

Sources: DoorDash Engineering Blog · 2026 · Figma Blog · 2026 · Block review: specialist lenses

21Grounded ReviewBench results

Our real ReviewBench run reached 88.7% grounded precision and 32.6% grounded recall. The chart compares block/review with Copilot Code Review, Cursor, Greptile, Codex, and Devin AI using their published grounded scores. Recall is on the horizontal axis and precision is on the vertical axis. A dotted vertical guide marks block/review’s 50% high-severity issue recall, a separate subset metric without an associated precision coordinate.

Sources: block/review on GitHub · ReviewBench

22Risk-based review tiering

Meta’s RADAR describes measuring change risk and routing changes to different review depths. Routine changes can take a lighter path, while changes with greater consequences receive deeper review. A new test and a ledgering-contract change should not require the same treatment.

Sources: Meta · RADAR · 2026 · Block review: specialist lenses

23Where human attention lands

Some changes are still costly enough to get wrong that reading the code is the most efficient way to build confidence. Examples include infrastructure, public claims, SOX compliance, money movement, and personally identifying information.

24What’s my job now?

As agents take on more of the writing and verification, our craft changes. Handing over the puzzles we used to solve can feel difficult. The question becomes which new skills help us guide systems toward outcomes we can stand behind.

25Align and contain

Agents need enough context to understand our goals and recognize when work is going off track. Team messages, issue trackers, and experimentation systems can help build that picture. Risk-based review provides containment as agents take on more autonomy.

26Get review before the code is written

Agent-native planning brings review into the process before implementation. With projects such as Buzz and CoPlan, we can share agent sessions and invite teammates into planning earlier, helping agents build a fuller view of our systems.

Sources: Buzz · Buzz / CoPlan

27Prioritize precision over recall

Build evaluations that measure both valid findings and missed issues. If we want people to pay attention, we must prioritize precision: even a thorough reviewer loses its audience when too many findings are bogus or harmful.

28Give your agents the big picture

Make our goals and preferences legible to agents so they can make better local decisions. Providing the right context helps alignment; an adversarial reviewer double-checking their output helps contain new risks.

29Build alignment before the code is written

Bring coworkers into agent sessions while the approach is still being shaped. Earlier alignment helps us guide the outcome before implementation and lets agents move faster with a shared understanding.

30A workflow we can trust

The factory keeps moving while agents review and humans maintain shared context and responsibility. These slides and source links are available here, and I’d be happy to continue the conversation afterwards.

Sources: joahg.dev/cut-the-noise