Structured Logging: Standards Your Engineers Will Actually Follow

Why unstructured logs hold everything back

Free-text logs are written for the person who wrote the code. “Failed to process order 8471 for user bob after 3 retries” reads fine, until you need to count failures per customer, alert on retries, or route by severity. Then someone writes a regular expression, another service words its message differently, and the dashboard works for half your fleet.

Every other part of our logging, ELK, Loki and OpenTelemetry service gets easier when logs are structured: dashboards, routing, alerting and PII scrubbing. That is why this guideline work often comes first, even though it is the least technical-looking item in the list.

What good looks like

• Every service writes one JSON object per line to standard output.
• A small set of fields is present everywhere with identical names and types.
• Levels mean the same thing across teams.
• Messages are stable strings; variable data lives in fields.
• Secrets, tokens and personal data never appear, by rule and by test.

The standard we write

Mandatory fields

Optional but recommended: span_id, request_id, http.method, http.status_code, duration_ms, error.type, error.stack. Use consistent names with OpenTelemetry semantic conventions where they exist, which keeps logs and traces aligned. See OpenTelemetry for background.

Message rules

Keep messages constant and put variables in fields. Compare these two:

{"level":"error","message":"payment authorisation failed","order_id":"8471","provider":"stripe","attempt":3,"duration_ms":1840,"error.type":"timeout"}

versus “Payment for order 8471 failed after 3 attempts (timeout)”. The first can be grouped by message, counted by provider and alerted on by attempt count. The second cannot without parsing.

Levels

We define these in one paragraph each and put them in the repository. Errors must be actionable. If a condition is expected, such as a user mistyping a password, it is not an error. Warnings are for degradation that is handled. Info is sparing: a service handling thousands of requests should not log several info lines for every one in production. Debug stays off by default.

How it looks per language

The goal is a tiny shared setup so developers never hand-build JSON.

// Go (standard library log/slog)
logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo})).
    With("service", "checkout-api", "env", os.Getenv("ENV"), "version", version)
logger.Error("payment authorisation failed", "order_id", orderID, "provider", "stripe")

# Python (structlog)
import structlog
structlog.configure(processors=[
    structlog.contextvars.merge_contextvars,
    structlog.processors.add_log_level,
    structlog.processors.TimeStamper(fmt="iso", utc=True),
    structlog.processors.JSONRenderer(),
])
log = structlog.get_logger().bind(service="checkout-api")
log.error("payment authorisation failed", order_id="8471")

For Node we usually use pino, which writes JSON by default and is fast. For Java we configure Logback or Log4j2 with a JSON encoder and place identifiers in the MDC. In each case a small middleware adds request_id and trace identifiers to every line within a request, so developers do not repeat them. Library and option names should be checked against the versions you run.

Decisions and trade-offs

JSON or logfmt. JSON is universally supported and handles nesting. Logfmt is more readable in a terminal. We default to JSON and give engineers a local pretty-printer for development.

Nested or flat fields. Flat dotted names such as http.status_code index predictably in Elasticsearch and work as Loki parsed fields. Deep nesting complicates mappings.

Stack traces. Put them in a single field as a string, not many lines, so they stay one event. This avoids needing the multiline parsers described on our Fluent Bit page.

Retrofit or leave legacy alone. Migrating a stable legacy service may cost more than it returns. We convert legacy output at the collector where possible and apply the standard to all new and actively changed code.

Common failure modes

• Field type drift. One service logs user_id as a number, another as a string, and Elasticsearch rejects documents. A schema and a type check prevent it.
• Cardinality in labels. Turning high-variance fields into Loki labels causes trouble. Only a few stable fields become labels.
• Logging whole request bodies. This leaks personal data and inflates volume. The standard bans it by default.
• Everything is info. Without level discipline, filters are useless. Code review checks levels.
• Standard without enforcement. A short CI check that parses sample output keeps it alive.

Adopting the standard in an existing codebase

Most teams already have thousands of log statements, and rewriting them all is neither realistic nor necessary. We take a staged approach that delivers value early.

First, switch the output format at the logger configuration level, so every existing message becomes valid JSON with the mandatory fields filled in by the framework. Free text stays in the message field, but timestamps, level, service name and request identifier are now consistent everywhere. This single change usually gives the largest gain for the least effort.

Second, fix the worst offenders. Using search over your existing logs, we list the highest-volume messages and the ones involved in recent incidents, and rewrite those with proper fields rather than interpolated strings.

Third, add the checklist to code review so new code arrives in the right shape. Old statements are improved opportunistically when someone touches them.

What not to convert

Some logs are best left alone. Third-party libraries often write their own formats, and it is cheaper to parse those in the collector than to patch them. Logs from legacy components that will be retired soon do not need a rewrite. The aim is a searchable, consistent system, not a perfect one.

Measuring adoption

We track the share of log lines that parse cleanly as JSON with all mandatory fields present, broken down by service. A simple query on that ratio shows which teams need help and gives leadership a number that moves as the work progresses. Services below an agreed threshold are listed in the sprint review until they catch up.

Delivery and handover

You receive the written standard (kept short), the starter configuration for each of your languages, a review checklist, and a workshop for your engineers. We also provide a validation script that can run in CI. For the wider engineering culture around operations, see how this ties to Managed DevOps practices.

How to tell it is working

The share of services emitting valid structured logs rises, the volume of unparsed lines falls, dashboards need no per-service exceptions, and engineers can find a failing request using one identifier. If you want a guideline fitted to your stack, book a consultation.

References

• Logging Cheat Sheet, OWASP Cheat Sheet Series
• Logs Data Model, OpenTelemetry specification
• Logging Architecture, Kubernetes documentation