This series is about the distance between two things: proving your code is correct before release, and knowing whether it actually performs once real users arrive. Most teams can do the first. Far fewer can do the second.
The product owners were not happy. Near the end of the milestone, they drew a line: fix the performance, or it doesn't ship. Eighteen months of work hung on the answer. The build was green. The new code had deployed cleanly and was serving traffic—yet the product was still too slow to put in front of a customer.
From the development side, every signal we had said success. The suite was green through the whole pipeline. Locally, it felt fine. All the automation we'd built to tell us when to ship—including extensive end-to-end and integration tests—was telling us to ship. And the people who owned the outcome were telling us we couldn't.
Green everything, unhappy product. That gap is the whole problem.
Correctness, performance, and impact
As an industry, we've built a strong toolkit for proving code correct. We have a far less coherent yardstick for what comes next: whether it performs, and whether it matters. There are three main questions worth pulling apart:
- Correctness: did the code behave the way we intended?
- Runtime performance: is it fast and reliable under real traffic?
- Business impact: did it create the outcome anyone wanted?
Tests live almost entirely in the first. This series is about the move to the second, and I want to be careful about the third, because these layers connect. An unusably slow product is a bad business outcome. The same telemetry that shows what is happening (and when) also shows what people are actually using, where the value proposition is strongest, and how well the system serves them. Performance—along with signals like application traffic and service demand—is a business concern, not a separate silo.
Production is where tests fail to predict outcomes
A test harness is a controlled environment by design. Real systems are not. They meet traffic patterns, dependency latency, data shapes, and concurrency that the harness never reproduces. Some behavior only exists in the wild.
The evidence you need lives nowhere but production, and it helps only if it feeds back into how you design, test, and instrument the next change. Production is the part of the loop that most teams never close.
Observability is a capability, not a purchase
Say the word "observability," and the first image is a vendor contract, a dashboard nobody opens, and SOC2-something-or-other. But what if observability isn't something we buy, but something we earn?
I want to reclaim this practice for engineers: the habit of asking the running system questions and being able to get answers. It changes the code you write, because you instrument for the questions you'll need later. It changes how you argue—"I need to refactor the checkout process" becomes "after refactoring the checkout process, the reduced latency and friction has helped reduce abandoned carts by 30% since we shipped the change."
You've arrived when you can answer a hard question with "I don't know, let's see what the data says," and mean it.
A launch that almost didn't happen
Back to that milestone crisis I opened with. Pages were taking twenty to thirty-five seconds to load. Every dev-side check was green. The first question was simple, and we couldn't answer it: where was the time going?
We already had distributed tracing across the stack—a Next.js front end and a Django REST API—and that groundwork is what made the rest possible. The easy offender showed up locally: an accidental N+1, a prefetch missing one attribute. Real, but not the story. The bigger cause was invisible on a laptop, where the dev web server runs single-threaded and never contends with itself. It only appeared under real traffic, for certain customer data shapes: the front end was requesting the same data more than once per page, and those uncoordinated calls queued at the API until the whole app fell behind.
The reflex was to add capacity—spin up more API task containers and get past the milestone. But these were customers with relatively modest data and expectations; the load we were seeing didn't fit. The team already doubted hardware was the answer, and tests to increase scale were still benchmarked poorly. Traces pointed to a different point of contention: a coordination problem, not a capacity one. So instead of scaling to hide it, we resolved the IDs each page needed early, at the page-model layer, and scheduled the fetches to hydrate each page in the fewest calls.
The benchmarked response went from twenty-plus seconds to under 500 milliseconds. The product finally felt like a product, and the milestone shipped on time.
That was a verdict neither the tests nor expert understanding of the system could ever have given us. Only the telemetry could. In fact, this coordination problem was diagnosed and fixed by one of the newest members of the team, despite earlier attempts by more senior engineers to find and fix it.
What the data actually did
The most useful thing wasn't the fix. It was that the traces moved a decision. The call didn't come down to who was most senior or who argued hardest—the evidence settled it. That's the quiet shift that observability buys you. A slow production system is a systems problem,and the fastest way past the finger-pointing is a signal everyone can see.
Where this series goes
Production is knowable, and knowing it is a skill you can build. I'll cover three, each with real examples:
- Benchmarking: making performance a first-class part of design and major code decisions
- Code instrumentation: building the visibility that benchmarking and monitoring both depend on
- Analytical research and visualization: turning the data into decisions, live and in hindsight
You don't need a greenfield project to start. All that you need is to move from treating observability as something that happens to you and to treating it as something you own. That's where the gains are: teams with more agency, more credibility with the people who fund the work, and more confidence that what they ship does what they meant it to.
The next time everything passes, ask the harder question: once real users have it, how will you know it's actually serving them?
Jed Schneider is a Senior Software Consultant at Test Double, with experience in product development, data engineering, observability, and agentic systems.
