Monitoring and Logging in System Design

A service suddenly feels slow, but the application process is still running. You need evidence about request outcomes and the path behind them, not just a green process-status light.

Metrics summarise observations over time. Logs record events with useful context. Traces connect related operations across a request path. These signals answer different questions and become more useful when their boundaries and identifiers agree.

For an API, record request rate, successful and failed latency, error rate and the constrained resource or queue. A fast error can otherwise make an aggregate response-time average look better during an outage.

A log can include a request identifier, operation and safe outcome context. A trace can show that database waiting dominates an application call. Sampling and missing instrumentation mean a trace view is evidence with limits, not a complete record of every event.

Choose labels carefully. An unbounded user ID or request ID on every metric series can create high cardinality and substantial storage/query cost. Per-request identifiers often belong in suitable logs or traces rather than metric dimensions.

This sample has ten latency observations. Normal mode uses 50 ms for all ten; tail mode changes one to 1,000 ms. The percentile method is nearest rank: p50 is the fifth sorted observation and p95 is the tenth for this tiny sample.

A separate outcome control marks zero or two requests as errors. Latency values do not change when that outcome changes; the lab deliberately shows why timing and success need separate inspection. It does not represent a production p95 estimate or correlate individual failures to spans.

Define a symptom and the user goal it threatens. A growing oldest-job age may matter more than a brief CPU spike if customers are waiting for processing. Track sustained deviations with enough context to choose an action.

An alert needs ownership and an investigation or recovery path. Repeated noisy alerts teach people to ignore them. Review incidents and adjust thresholds, windows and coverage using evidence.

A dashboard is not a substitute for recovery. Ask whether the team would notice a silent data error or a failed daily job, and test that notification path.

OpenTelemetry provides instrumentation and telemetry concepts; it is not by itself the complete backend for storing and querying all observations. Prometheus provides metrics monitoring capabilities. Sources checked 6 October 2026.

Self-hosted tools require capacity and operation; managed services add pricing, retention and data-location choices. Compare ingestion, query work, cardinality and access controls with the workload.

Keep credentials, session tokens and unnecessary personal data out of telemetry. Restrict access and retention, and check that request content cannot create misleading or unsafe log entries.

A Decorator can add local instrumentation, while a Sidecar can supply deployed support. Observe breaker transitions and useful outcomes rather than treating a pattern’s presence as evidence of health.

Combine traffic, latency distributions, errors and resource limits. Use logs and traces to explain the path behind those observations.

Make alerts actionable, budget telemetry volume and protect sensitive context. Validate monitoring as part of the service’s operating design.