Application Metrics Instrumentation for Go, Python, Node and Java

Why instrumentation is the real work

Standing up Prometheus is the easy half of monitoring. The harder half is deciding what each application tells it. Infrastructure metrics say a node is busy; they do not say that checkout failed for customers on one payment method. Only the application knows that, and only if someone taught it to say so.

Teams that skip this end up with beautiful cluster dashboards and no idea whether the product works. As part of our monitoring and observability service we give developers a practical standard and working code, because instrumentation done inconsistently across ten services is nearly as bad as none.

What good instrumentation looks like

• A common core. Every service exposes the same request, error and duration metrics with identical names and labels, so one dashboard and one alert template cover all of them.
• Bounded labels. A route template like /orders/{id}, never the raw URL.
• Right metric types. Histograms for durations. Counters for events. Gauges for state.
• Units in names. _seconds, _bytes, _total suffixes, following Prometheus naming conventions.
• Business signals. Orders placed, payments declined, jobs processed.
• Cheap to emit. Instrumentation must not become the performance problem.

The standard we write

A short document, usually a page or two, in your repository:

Teams may add more, but the core is not optional. This is the table that makes service dashboards and SLO calculations work across the estate.

How it works in practice

Go

var requestDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
    Name:    "http_server_request_duration_seconds",
    Help:    "Request duration in seconds.",
    Buckets: []float64{0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5},
}, []string{"method", "route"})

func instrument(route string, next http.HandlerFunc) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        next(w, r)
        requestDuration.WithLabelValues(r.Method, route).Observe(time.Since(start).Seconds())
    }
}

// expose the endpoint
http.Handle("/metrics", promhttp.Handler())

Python

from prometheus_client import Counter, Histogram, start_http_server

REQUESTS = Counter("http_server_requests_total", "Requests", ["method", "route", "status"])
LATENCY = Histogram("http_server_request_duration_seconds", "Duration", ["method", "route"])

start_http_server(9100)  # separate metrics port

def handle(method, route):
    with LATENCY.labels(method, route).time():
        status = process_request()
    REQUESTS.labels(method, route, str(status)).inc()

With multi-process servers such as Gunicorn, use the library’s multiprocess mode, otherwise each worker reports separately and your numbers will be wrong.

Node.js

const client = require('prom-client');
client.collectDefaultMetrics();

const duration = new client.Histogram({
  name: 'http_server_request_duration_seconds',
  help: 'Request duration in seconds',
  labelNames: ['method', 'route'],
});

app.use((req, res, next) => {
  const end = duration.startTimer();
  res.on('finish', () => end({ method: req.method, route: req.route?.path ?? 'unknown' }));
  next();
});

app.get('/metrics', async (_req, res) => {
  res.set('Content-Type', client.register.contentType);
  res.end(await client.register.metrics());
});

Java

With Spring Boot, Micrometer does most of the work. Add the micrometer-registry-prometheus dependency and expose the endpoint:

management:
  endpoints:
    web:
      exposure:
        include: health,prometheus
  metrics:
    distribution:
      percentiles-histogram:
        http.server.requests: true

That gives request histograms, JVM memory, GC and thread metrics with no code. Custom business metrics use a MeterRegistry bean and Counter.builder("orders.placed").register(registry). For non-Spring Java applications, the JMX Exporter can run as a Java agent.

Decisions and trade-offs

Prometheus client or OpenTelemetry SDK. The Prometheus clients are simple and have fewer moving parts. OpenTelemetry offers one API for all three signals, which pays off if you are also doing tracing. Mixing both in one service is possible but confusing; pick one per language.

Histogram buckets. Buckets are fixed at definition time and should straddle your latency objectives. If an SLO is 300 ms, make sure there is a bucket at or near 0.3. Native histograms are an evolving option, and we only recommend them where your Prometheus version and tooling support them.

What not to measure. Per-user, per-request-ID or per-email metrics belong in logs or traces. Metrics are for aggregates.

Failure modes

• Cardinality bombs from raw paths or user identifiers. We add sample_limit guards and review labels in code review.
• Metrics port exposed publicly. Keep /metrics on an internal port or protect it with network policy (see network policy).
• Multi-process undercounting in Python and Node cluster modes.
• Counters that reset misread as drops; use rate() and increase(), never raw values.
• Silent breakage when a rename changes a metric name. Treat metric names as an API with a deprecation period.

Rolling it out across many services

A standard only matters if it spreads. We start with a pilot service that is important enough to matter but small enough to change quickly. The pilot proves the shared library or middleware, the ServiceMonitor template and the dashboard. Only then do we ask other teams to adopt it.

Wrap it once

Where several services share a language, we recommend a thin internal package that registers the core metrics and middleware, so a new service gets correct names and labels by importing one module. Teams that hand-write the metrics in each service drift apart within a few months, and the shared dashboards quietly stop working for some of them.

Review for cardinality

Add a line to your pull request template asking what values each new label can take. A label whose values come from users, request bodies or identifiers should be rejected. Catching it in review is far cheaper than discovering it as memory growth on the Prometheus server later, and it keeps the cost of the whole monitoring estate predictable as your product grows.

Test the endpoint

A small unit or integration test that scrapes the metrics endpoint and checks the core metric names exist prevents a refactor from silently removing them. It takes minutes to write and protects every dashboard and alert built on top.

Delivery and handover

We deliver the instrumentation standard, tested example code per language, a ServiceMonitor template, a lint check or review checklist for cardinality, and pairing time on a pilot service. Your developers then roll it out service by service, which is faster and more durable than us editing every codebase.

How to tell it is working

Every service has the core metrics, new services pass a metrics checklist at launch, active series per service stay predictable, and SLO dashboards need no per-service query rewrites. If you want help applying this to your languages, talk to us.

References

• Instrumentation practices, Prometheus documentation
• Metric and label naming, Prometheus documentation
• Micrometer Prometheus, Micrometer documentation