Observability and Monitoring: Know Your System Before Users Do

The first you hear about it is a support ticket. A customer says the app is slow, or the checkout is broken, or a feature has stopped working for them. Your engineers open the dashboard, see green lights everywhere, and start asking the customer to clear their cache. By the time someone digs into the logs — if there are logs — the issue has been live for hours, possibly days, and three other customers have already churned without saying anything. This is what shipping without observability looks like. It is one of the most predictable failure patterns in production software, and it is structural: the system has been built with no way to tell whether it is working except by waiting for users to complain.

An observability monitoring framework closes that gap. It is the set of capabilities — structured logging, error tracking, performance monitoring, distributed tracing, and alerting — that lets a team see the health of their system from the inside, in real time, before users do. In the Built to Last™ 2.0 framework, observability sits inside P02 — The Right Infrastructure — alongside the Production Readiness Review™ that gates launch and the security architecture that protects what's in production. Each component depends on the others. A pre-launch review that cannot verify monitoring coverage is checking the wrong boxes. Security architecture without observability has no way to detect the events it was designed to prevent.

This article walks through what an observability monitoring framework actually contains, the four pillars most teams under-build, and how to set it up so it is live before your first user arrives — not bolted on after the first incident.

Why Teams Find Out From Customers

The damage from shipping without observability rarely arrives as a single event. It accumulates. A memory leak runs for three weeks before someone notices the cost line on the cloud bill ticking up. A subset of API calls fail intermittently for one region, and the only signal is a slow drop in conversion on that geo. A scheduled job stops running silently because the alert was wired to a Slack channel nobody monitors. Each issue is recoverable on its own. Together, across six months, they shape the customer's experience of the product more than the features the team is shipping.

The cost shows up across three lines. The first is revenue — every minute a customer-facing issue is live before you know about it is lost transactions, lost trust, or both. The second is engineering — debugging without observability is archaeology, and engineers without telemetry can spend days chasing what proper tracing would have surfaced in twenty minutes. The third is reputation, and it is the hardest to recover. A customer who has churned because the product broke for them silently is not going to wait for the fix.

The pattern that produces this outcome is consistent. Production monitoring gets treated as something to set up after launch, because launch day is already overcrowded with work that feels more urgent. The team adds uptime checks because they are quick. They add a basic error tracker because their cloud provider includes one. They mean to come back and configure structured logging properly, set up traces, and define alerting thresholds — and they never do. By the time the first real incident happens, the team is doing the work that should have happened pre-launch, while production is on fire.

What an Observability Monitoring Framework Actually Contains

"Observability and monitoring" gets used loosely. In practice the framework has four distinct pillars, each catching a different class of failure. A team can have one or two in place and still be blind to the failure modes the others catch. The discipline is to build all four before the first user arrives.

Pillar one: structured logging

Logs are the narrative record of what the system did and when. Unstructured logs — free-form text written by individual engineers as they remembered — are almost useless at scale. By the time you have a hundred thousand log lines an hour, finding the relevant ones requires more work than the issue is worth. Structured logging is the discipline of writing every log line in a consistent, machine-parseable format — typically JSON, with named fields for the request ID, the user ID, the endpoint, the latency, the outcome, and the error type if there was one.

The payoff is that logs become queryable. You can filter by user, by endpoint, by latency band, by error class. You can correlate a customer's complaint with the exact log lines for their session. You can answer "did this happen to anyone else?" in seconds instead of guessing.

Pillar two: metrics and performance monitoring

Metrics are the time-series record of how the system is behaving. Request volume per endpoint, latency distributions, error rates, database connection pool usage, queue depth, memory consumption, CPU utilisation. Metrics are what let you see that something is degrading before it is broken — the API response time has been creeping up for three days, the connection pool is approaching saturation, the background job queue has stopped draining as fast as it fills.

For web products, this also covers user-facing performance metrics — Core Web Vitals, time to first byte, largest contentful paint — which are now scored by search engines and felt directly by users. For mobile, it covers crash rates per OS version, app start time, and frame drop rates per device class. For AI products, it covers tokens per request, model latency, hallucination rate, and cost per call.

Pillar three: distributed tracing

In a system with more than one service, the question "where did this request actually slow down?" cannot be answered from metrics alone. Distributed tracing assigns every request a unique ID and propagates it through every service the request touches. When something goes wrong, the trace shows exactly which service contributed the latency or returned the error, including work that ran asynchronously after the user's request completed.

Tracing is the pillar most teams under-build because it requires discipline at the code level — every service needs to participate, every external call needs to be instrumented. The payoff is debugging a multi-service issue from a single view instead of stitching together logs from four systems. OpenTelemetry is now the de facto standard for instrumentation; building against it from sprint one means the tooling on top — Datadog, Grafana, Honeycomb, or others — becomes interchangeable.

Pillar four: alerting

Alerts are the layer that turns observability into action. The previous three pillars give you the data. Alerts decide when a human needs to look at it. The discipline here is harder than it appears: alerts that fire too often become noise and get ignored, alerts that fire too rarely miss what they were meant to catch, and alerts wired to channels nobody watches might as well not exist.

Good alerts are scoped to the actionable: error rates above a baseline, latency past a defined threshold, queue depth past a saturation point, infrastructure cost trending past forecast. Each alert has a defined owner, a defined response path, and a documented runbook that the on-call engineer can follow at two in the morning without context. The Production Readiness Review verifies that this layer is in place before launch — not just that the alerting tool exists, but that the alerts have been tested and have owners.

The decision the framework produces, and who's in the room

The output of building the framework is a documented monitoring posture: what is logged, what metrics are collected, what is traced end-to-end, what triggers an alert, who responds, and what the runbook says. This document lives in the engineering knowledge base from sprint one and gets reviewed at the Production Readiness Review before launch.

The room that builds it is the engineering lead, the DevOps engineer responsible for the production environment, the product owner (because alerting thresholds are partly product decisions), and the on-call engineer who will actually field the alerts. Including the on-call engineer at design time is what prevents the runbook problem — alerts written by people who will never respond to them.

The failure modes even when the framework is present are worth naming. Alerts that fire correctly but route to a channel nobody monitors. Dashboards that exist but nobody looks at. Logs that are collected but not retained long enough to debug the issues that surface weeks later. Traces instrumented in the main path but missing on the async work where most of the real bugs live. Each is a structural fix, not a tooling problem.

How to Implement It Without Slowing Down Delivery

The reflex to defer observability to "after launch" is understandable. Setting up the pillars properly looks like infrastructure work that does not ship features. In practice, building it in from sprint one is faster than retrofitting it later — because every bug you debug along the way uses the same telemetry the production system will need.

The sequence that works: structured logging from day one — JSON output, consistent field names, request IDs propagated. This is a code-level standard, not a tooling decision, and it has to be set before the first feature ships. Metrics and dashboards come next, ideally wired to the same backend that will run in production. Staging is the wrong place to develop these because the load profile is too different. Distributed tracing should be instrumented as services are written, not added later; instrumenting an existing service is significantly more work than building it instrumented from the start.

Alerts come last, because the thresholds depend on what normal looks like, and normal can only be established once the system is running with realistic load. The Production Readiness Review is the moment those thresholds get reviewed, tested, and signed off. Going live without that review — or with alerts that have not been tested — is the configuration that produces the support-ticket-first failure mode.

What to avoid: over-instrumenting. Every metric, log line, and trace span has a cost — in performance, in storage, in cognitive load on the team trying to debug. The framework is not "collect everything"; it is "collect what answers the questions you will actually need to ask in production." Sampling traces, retaining logs in tiers, and aggregating metrics at sensible intervals keeps the bill and the dashboard useful.

If your team is already past launch with no observability in place, the order of work changes. Structured logging on the most critical paths first — login, payment, the primary user action. Error tracking next so you know what is already broken. Metrics on the database and the API. Tracing last. The full framework is a quarter of work for a mid-sized product; the alternative is months of debugging blind. This work fits naturally inside a structured project delivery framework rather than running as an ad-hoc fix-it sprint.

A Tale of Two Launches

An early-stage SaaS we worked with shipped a customer-facing product with only uptime checks in place. The team had meant to set up the full framework but ran out of time before launch. Three months in, a customer churned and mentioned in the exit interview that the app had been slow and crashing intermittently for them for weeks. The team dug in — they had no structured logs, no traces, no per-user telemetry — and eventually traced the issue to a memory leak in a long-running worker process. It had been affecting a subset of users for the entire period since launch. The exit interview was the alert that the system should have generated weeks earlier.

A second product launched the same week, built to the framework. Structured logging from sprint one, metrics and dashboards live, distributed tracing instrumented as services were written, alerts defined and tested at the Production Readiness Review. In week two of production, a memory metric trended past its threshold. The alert fired before any customer was affected. The on-call engineer followed the runbook, identified the leak in a background job, deployed a fix the same afternoon. No support tickets. No customer-visible impact.

Same engineering capability. Same class of bug. Two completely different customer experiences. The difference was not engineering quality; it was whether the team could see what their system was doing.

When Observability Is Critical, and When You Can Get Away With Less

Observability is non-negotiable when you have paying customers, when your system is part of a revenue path, when downtime has reputational consequences, or when regulatory exposure means you need an audit trail of what happened and when. Anything customer-facing, anything financial, anything with a service-level commitment to a third party belongs here. Build the full framework before the first user arrives.

You can move with a lighter setup for internal tools used by a small audience, for proofs-of-concept explicitly scoped as throwaway, and for very early prototypes where the question is "does anyone want this" rather than "does this work reliably". Even there, structured logging is cheap and pays for itself the first time you need to debug anything; leaving it out is rarely a defensible saving.

The category that catches most teams out is the mid-sized internal tool that quietly becomes load-bearing. A product built for ten internal users ends up handling a critical workflow for a hundred. "We'll add monitoring when we need it" breaks down the moment the tool matters enough that downtime is felt. If a tool is on a trajectory to matter, the framework should land before it does.

What to Do Next

If you are pre-launch, the most useful next step is to run a Production Readiness Review and use its monitoring section as the checklist for what to build first. If you are post-launch with no framework in place, start with structured logging on the highest-traffic path and work outwards. For teams thinking about how observability fits into a broader infrastructure practice, see how we approach DevOps and infrastructure delivery as a discipline that includes monitoring from sprint one.

Frequently Asked Questions