Deploying ELK or Grafana Loki: Choosing and Building Your Log Platform

Why the platform choice matters

Most teams start with kubectl logs and a lot of luck. That works until a pod restarts and its logs vanish, or an incident spans six services and nobody can line the evidence up. At that point a central log platform stops being optional. The expensive mistake is not picking the “wrong” product, it is picking one without knowing your volume, your questions and your budget, then discovering at 40 GB a day that search is slow, disks are full, or the invoice has doubled.

This page is the platform decision and build. Collection, routing, retention and alerting each have their own page, linked below. The overall programme sits within our logging, ELK, Loki and OpenTelemetry service, and the concept itself is explained in our log aggregation glossary entry.

How the two architectures differ

The core difference is what gets indexed.

Neither is universally better. A team that debugs by service and namespace, and already uses Grafana for metrics, will usually be happier with Loki. A team that analyses customer behaviour from logs, or needs security search across many fields, will usually need Elasticsearch.

What good looks like

• Every production log line is searchable within seconds of being written.
• Logs survive pod, node and even cluster loss, because storage is outside the workload.
• Access is by role: developers see their services, auditors see audit logs, nobody shares an admin password.
• Capacity is known: you can state ingest rate, retention and the point at which you will need to scale.
• The platform is rebuilt from code, not hand-tuned on a running cluster.

How we deploy it in practice

ELK on Kubernetes

For Elasticsearch on Kubernetes we generally use the Elastic operator (ECK) so that upgrades, certificates and node sets are declared rather than scripted. Nodes are split by role: dedicated masters for stability, hot data nodes on fast SSD for current indices, and warm or cold nodes for older data. Our retention and ILM page covers how data moves between them.

A minimal ECK cluster definition looks like this:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: logs
spec:
  version: 8.15.0
  nodeSets:
    - name: master
      count: 3
      config:
        node.roles: ["master"]
    - name: hot
      count: 3
      config:
        node.roles: ["data_hot", "data_content", "ingest"]
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes: ["ReadWriteOnce"]
            storageClassName: gp3
            resources:
              requests:
                storage: 500Gi

Version, node counts and volumes are illustrative. Real values come from your measured volume, and we pin the version you actually run.

Loki on Kubernetes

For Loki we normally run the simple scalable or distributed deployment mode with an object storage bucket for chunks and the index. Compute is stateless apart from short-lived write-ahead data, so scaling and replacing nodes is easier than with Elasticsearch. The bucket, its lifecycle rules and the IAM role are created with Terraform, and configuration is delivered by GitOps.

# Loki storage fragment (illustrative)
schema_config:
  configs:
    - from: "2024-04-01"
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: loki_index_
        period: 24h
limits_config:
  retention_period: 720h
  max_global_streams_per_user: 50000

Labels are the design decision that makes or breaks Loki. A label such as namespace, app or level is fine. A label holding a user ID or request ID creates a separate stream for each value and destroys performance. We agree the label set with your team before rollout.

Decisions and trade-offs

Logstash or Beats or neither. Logstash is powerful for parsing and enrichment but heavy. Lightweight collectors such as Fluent Bit cover most Kubernetes needs, as described in our Fluent Bit DaemonSet page. We keep Logstash for cases that need its filter ecosystem.

Self-hosted or managed. Running Elasticsearch well is real work: shard counts, heap sizing, snapshots, rolling upgrades. If you have fewer than a handful of platform engineers and modest volume, a managed log service can be cheaper in total than your time. We say so when the numbers show it.

When not to build this. If you run a few services on a single VM and cloud-native logging already meets your needs, a full platform is overkill. Likewise, if your real gap is tracing or metrics, start with the monitoring service.

Common failure modes

• Unbounded cardinality. Loki labels or Elasticsearch field mappings that grow without limit. We set mapping limits and label allow-lists.
• Over-sharding. Too many small Elasticsearch shards waste memory. Index templates and rollover rules keep shard size sensible.
• No back-pressure plan. A logging burst should slow collectors, not take down applications. We configure buffers and limits.
• Single-copy data. Disks or buckets without replication or snapshots. We configure snapshots to object storage or bucket versioning.
• Open access. Dashboards exposed without authentication. Everything sits behind single sign-on.

Delivery and handover

You receive the infrastructure and configuration in your own repositories and cloud account, a sizing note with the assumptions behind it, runbooks for common operations, and a walkthrough with your engineers. If you want us to keep running it, that is covered by Managed DevOps. Otherwise your team owns it from day one.

How to tell it is working

Search results arrive quickly, no log source is silently missing, ingest stays below the capacity you planned, and the monthly cost is within the model. If you want an independent view on which platform fits, book a consultation.

References

• Loki overview, Grafana Loki documentation
• Manage storage, Grafana Loki documentation
• Elastic fundamentals, Elastic Docs