What Loki is and why teams adopt it
Loki takes a different position from traditional log search. Instead of indexing every word, it indexes a small set of labels per log stream and stores the lines themselves as compressed chunks, usually in object storage. Finding logs means selecting streams by label, then filtering or parsing the content when you query. That design is why it is inexpensive to run and straightforward to scale.
Teams adopt Loki because they already use Grafana and Prometheus, and Loki uses the same label model. The result is that moving from a metric to the matching logs is natural: the same namespace and app labels apply. Our Loki log aggregation guide for Promtail and Fluent Bit shows the basic setup, and the broader logging service covers design and operation.
How we run Loki in production
Deployment modes
Loki can run as a single binary, as a simple scalable deployment with read, write and backend targets, or as fully distributed microservices. For most startups the simple scalable or single binary mode on Kubernetes is enough. We move to microservices mode only when ingest volume or tenant count demands independent scaling. Whichever mode, the durable state lives in object storage, so compute can be replaced without drama.
Labels
Label design decides whether Loki is pleasant or painful. Each unique combination of label values is a stream, and too many streams make ingestion and queries slow. We keep a short, stable list, typically cluster, namespace, app and environment, and leave request IDs, user IDs and paths inside the log line. This is the same discipline as in metrics, and we apply the structured-logging principles from our structured logging guidelines.
Collection
On Kubernetes, a node-level agent tails container logs and attaches labels from pod metadata. We use Fluent Bit, Grafana Alloy or the OpenTelemetry Collector depending on what else you route logs to; see Fluent Bit as a DaemonSet. Collection should apply back-pressure and retry sensibly so a Loki restart does not lose data or crash nodes.
Querying and alerting
LogQL selects streams, filters lines and can turn logs into metrics. A few examples:
{namespace="payments", app="checkout-api"} |= "error" | json | level="error"
sum by (app) (rate({namespace="payments"} |= "timeout" [5m]))
The second query returns a per-app rate that can drive an alert rule in the Loki ruler or Grafana, routed through Alertmanager. Our notes on log-based alerting cover when this is a good idea and when a metric is the better signal.
Security and access
Loki has no authentication of its own in a default install, so we place it behind a gateway or ingress with authentication and keep it off public networks. Multi-tenancy uses a tenant header, which lets different teams or customers have isolated data when it is enabled. Object storage buckets are private and encrypted, with access granted through workload identity rather than long-lived keys. Because logs often carry personal data, we scrub it at collection time following our PII scrubbing guidance.
Cost and capacity
Storage is cheap because it is object storage, and there is no content index to maintain. The real costs are ingestion compute, query compute for large scans and cross-zone or egress network traffic. Retention is configured through the compactor, and per-tenant limits protect the cluster from a runaway service. Short queries over a narrow label set and time range are fast and inexpensive, while broad scans over days of data are not. We set limits and educate users instead of promising numbers.
Day-2 operations and failure modes
The usual problems are label explosions, rate-limit errors for noisy services, slow queries from regex-heavy filters and misconfigured retention that never deletes. We monitor ingestion errors, stream counts, ingester memory and query latency, and keep a runbook for each. Upgrades are staged, with configuration reviewed against release notes, because Loki’s configuration and storage schema have evolved over time.
Multi-environment and multi-team setups
Most companies have more than one cluster. We usually run one Loki per environment, or a central Loki that receives from several clusters with a cluster label, depending on how strictly production data must be separated. A central instance is easier to query across regions, while separate instances limit blast radius and simplify access rules.
Tenancy is the other lever. Giving each team or product its own tenant ID isolates data and lets you apply different limits and retention, which prevents one chatty service from consuming everyone’s ingestion budget. The trade-off is that cross-tenant queries need explicit configuration, so we only split tenants where there is a real reason, such as customer isolation or differing compliance needs.
Migrating from another logging system
Moving to Loki from Elasticsearch or a hosted service works best as a parallel run. We ship logs to both for a period, rebuild the handful of searches and dashboards engineers actually use, and compare answers during real incidents. Historical data is rarely worth backfilling, so we keep the old system read-only until its retention window expires, then switch it off.
When not to choose Loki
If engineers need to search arbitrary terms across large volumes, run heavy aggregations or investigate security events across content, the ELK Stack is the better fit; our ELK versus Loki comparison goes through the trade-offs. If you want no infrastructure at all, Datadog offers hosted logs with usage-based cost. Many teams run Loki for everyday debugging and send only audit or security logs to a search engine. To plan that split, talk to us.
Official documentation: grafana.com/oss/loki