I use this workflow in test environments to exercise an observability stack. The example is intentionally generated: a small API receives traffic while a downstream service adds latency. It is not presented as an employer incident. The goal is to make the investigation repeatable before a real alert arrives.

Begin with the user-visible signal.

Suppose an alert says the API’s 95th-percentile latency has stayed above 800 milliseconds for ten minutes. I record the exact start time, affected environment, route, and alert query. Then I check the basic service signals together: request rate, errors, duration, and saturation. Looking at latency alone can hide a traffic spike, retry storm, or capacity limit.

histogram_quantile(
  0.95,
  sum by (le, route) (
    rate(http_request_duration_seconds_bucket{
      job="catalog-api"
    }[5m])
  )
)

I compare the affected route with other routes and the current window with a healthy window. The first question is scope: one endpoint, one pod, one zone, one tenant, or the entire service? A narrow scope changes both the likely cause and the safest mitigation.

Place changes on the same timeline.

Before searching every log, I look for changes around the alert start: deployments, feature flags, dependency releases, node events, and configuration updates.

kubectl rollout history deployment/catalog-api
kubectl get pods -l app=catalog-api -o wide
kubectl get events --sort-by=.lastTimestamp
kubectl describe deployment catalog-api

Correlation is not proof. A deployment near the incident is a useful hypothesis, not an automatic root cause. I compare old and new replicas when both exist, and check whether latency follows a version, node, or availability zone.

Use logs to explain the shape, not rediscover it.

Structured application logs should include a timestamp, request ID, trace ID, route, status, duration, upstream name, and upstream duration. They should not contain credentials or full request bodies by default.

{
  "level": "warn",
  "route": "/v1/catalog",
  "status": 200,
  "duration_ms": 947,
  "upstream": "inventory",
  "upstream_ms": 881,
  "trace_id": "64bd2e..."
}

In this lab, slow requests share high upstream_ms values while application CPU and queue depth remain normal. That narrows the search to the dependency path. A trace using the same trace_id then shows whether the time is DNS, connection setup, server processing, retries, or another child call.

Metrics show that many requests are slow. Logs provide request-level context. Traces explain where one request spent its time. Using each signal for its strength is faster than trying to make one tool answer every question.

Mitigate before completing the investigation.

If user impact is active and the evidence is strong enough, I choose the smallest reversible mitigation: roll back a release, disable a feature flag, reduce concurrency, route around a dependency, or add temporary capacity. I avoid changing several variables at once because that makes recovery harder to verify.

Timeouts and retries deserve special care. Adding retries to a saturated dependency can increase load and extend the incident. A timeout should fit within the caller’s end-to-end budget, and retries should be bounded, use jitter, and apply only to operations safe to repeat.

Validate with the signal that detected the problem.

After mitigation, I watch p95 and p99 latency, error rate, throughput, saturation, and backlog long enough to cover the original alert window. “Pods are green” is not recovery if users still see slow responses.

The final note is short: impact, timeline, trigger, contributing conditions, why detection worked or failed, mitigation, and a small number of owned follow-ups. The repeatable path is simple: confirm impact, define scope, align changes, move from metrics to logs to traces, mitigate safely, and verify recovery.