What the ELK Stack is and why teams adopt it
The ELK Stack is a log analytics toolset built around Elasticsearch, a distributed search and analytics engine. Logs arrive through shippers or Logstash, are parsed into fields, indexed in Elasticsearch and explored through Kibana. Because content is indexed, you can search for any term, filter by any field and aggregate across millions of events in seconds.
Teams adopt ELK because that flexibility is hard to match. A support engineer can find every request that touched a customer ID, a security analyst can trace a login sequence, and a developer can chart errors by endpoint, all from the same data. The trade-off is operational weight: Elasticsearch is a stateful, memory-hungry cluster that needs deliberate design. Our logging with ELK, Loki and OpenTelemetry service covers the full engagement.
How we run ELK in production
Cluster design
We separate node roles as the cluster grows: dedicated master-eligible nodes for stability, hot data nodes with fast disks for recent logs, warm or cold nodes with cheaper storage for older data, and optionally ingest and coordinating nodes. A small team may begin with fewer roles, but we plan the path so scaling is a configuration change rather than a rebuild.
On Kubernetes we typically use the Elastic operator (ECK), which handles rolling upgrades and certificates, with local or high-performance persistent volumes and anti-affinity so replicas do not share a node. For very small volumes a managed service avoids this work entirely.
Collection and parsing
For Kubernetes we run a node-level collector such as Fluent Bit or Filebeat as a DaemonSet and enrich events with pod, namespace and label metadata. Our guide to Fluent Bit as a Kubernetes DaemonSet explains the pattern. Parsing at the edge is cheaper than parsing centrally, but heavy transformation belongs in Logstash or an ingest pipeline. Structured JSON logging from applications removes most of this work, which is why we push teams towards the conventions in our structured logging guidelines.
Mappings and shards
Dynamic mapping is convenient and dangerous. A single service that logs a new field name per request can create thousands of fields and destabilise a cluster. We define index templates with explicit mappings, set field limits, and use keyword types for identifiers and text types only where full-text search is needed.
Shards are the other classic problem. Each shard carries overhead, so thousands of tiny ones hurt more than they help. We use rollover to create indices by size or age and target shard sizes that match the hardware, as in this lifecycle sketch:
PUT _ilm/policy/app-logs
{
"policy": {
"phases": {
"hot": { "actions": { "rollover": { "max_primary_shard_size": "50gb", "max_age": "7d" } } },
"warm": { "min_age": "7d", "actions": { "shrink": { "number_of_shards": 1 } } },
"delete": { "min_age": "30d", "actions": { "delete": {} } }
}
}
}
Thresholds there are illustrative. We set them from your volume and retention needs, as covered in our log retention and ILM policy guide.
Security and access
Modern Elasticsearch enables TLS and authentication by default, and we keep it that way. Beyond that we apply role-based access with document or field level restrictions where teams must not see each other’s data, connect Kibana to your identity provider, and keep the cluster off the public internet. Logs are a common place for personal data and secrets to leak, so we scrub or hash sensitive fields before indexing. The approach is described in log PII scrubbing for GDPR and HIPAA. Snapshots to object storage with restricted access cover recovery and long-term archive.
Cost and capacity
Elasticsearch cost is driven by retained volume, replica count, indexing overhead and heap. The levers are straightforward: log less noise, drop unused fields, use tiers, shorten hot retention and keep older data in snapshots. We avoid quoting percentages because outcomes depend on your logs. A useful exercise is ranking services by volume, since a handful of chatty ones often dominate the bill.
Day-2 operations and failure modes
Expect to deal with disk watermarks, mapping explosions, overloaded ingest during traffic spikes and slow queries from broad wildcards. We monitor cluster health, heap pressure, indexing rejections and shard counts, use Kafka or a similar buffer when ingest is bursty, and rehearse restores from snapshots. Upgrades follow Elastic’s supported path one step at a time, preceded by deprecation checks and a snapshot.
Kibana and alerting
Kibana is where most engineers meet the stack, so it deserves as much design as the cluster. We create data views per log family, curate saved searches for the services people debug most, and build dashboards for error rates, slow endpoints and ingest health. Spaces separate teams, and role mappings stop one team from editing another’s objects.
Alerting inside Kibana can watch thresholds and query results, which is useful for security events and for patterns that metrics cannot express. We still prefer metric-based alerts for availability and latency, because they are cheaper to evaluate and easier to reason about. Log alerts are best kept for specific, high-signal conditions such as repeated authentication failures, and every rule should name an owner and a runbook.
When not to choose ELK
If your questions are mostly “show me the logs for this service in this window”, Grafana Loki is usually cheaper and simpler, and our comparison of ELK Stack versus Grafana Loki walks through the decision. If you do not want to operate a search cluster at all, a hosted option such as Datadog trades operational effort for usage-based cost. To discuss which fits your volume and team, contact us.
Official documentation: www.elastic.co