What Datadog is and why teams adopt it
Datadog is a software-as-a-service observability platform. You deploy an agent on hosts or as a DaemonSet in Kubernetes, enable integrations for your cloud and software, and metrics, logs, traces and checks flow into one product with dashboards, monitors and search. Storage, scaling and upgrades are Datadog’s job, not yours.
That is the real appeal. A team of five engineers shipping a product does not want to size an Elasticsearch cluster or debug Prometheus memory. They want a graph and an alert. Datadog provides breadth and polish quickly, and linking a trace to its logs and host metrics is a few clicks. The cost of that convenience is a bill that grows with usage and a degree of dependence on one vendor. We are an independent consultancy and not a Datadog partner, so our advice is about fit rather than selling licences.
An honest build or buy view
The question is rarely “which tool is best”. It is “where should your engineers spend their time”. Open source with Prometheus, Grafana and Grafana Loki has no per-metric fee and gives you full control, but someone must run, upgrade, secure and scale it. Datadog turns that engineering cost into a recurring operating cost.
| Factor | Datadog | Self-hosted open source |
|---|---|---|
| Time to value | Fast, one agent | Slower, design and build needed |
| Operational burden | Low | Real, ongoing |
| Cost shape | Usage-based, scales with data sent | Infrastructure plus people time |
| Flexibility | Good within the product | Very high |
| Portability | Moderate, improved by OpenTelemetry | High |
Small teams and fast-growing products often favour Datadog. Larger estates with strong platform teams, strict data residency or very high volumes often favour open source, or a hybrid with Prometheus for metrics and Datadog for traces and incident workflow. We help you compare using your real telemetry volumes, not generic assumptions.
How we run Datadog in production
Agent deployment
On Kubernetes we install the agent with the official Helm chart or the Datadog Operator, with configuration in Git and the API key stored as a secret. We enable only the integrations you need, since each one can add metrics and cost. A minimal values excerpt:
datadog:
apiKeyExistingSecret: datadog-secret
site: datadoghq.com
logs:
enabled: true
containerCollectAll: false
apm:
portEnabled: false
tags:
- "env:production"
- "team:platform"
Note that containerCollectAll: false means logs are collected only from containers you opt in, which is a deliberate volume control. Your site value depends on your Datadog region.
Tagging
Tags are how Datadog organises everything: dashboards filter by them, monitors group by them and cost is attributed through them. We standardise on unified service tags (env, service, version) plus team and cost-centre tags, applied from Kubernetes labels and Terraform. We also avoid tags with unbounded values such as user IDs, because they inflate custom metric counts.
Monitors and dashboards as code
Click-built monitors become unmanageable. We define monitors, dashboards and SLOs in Terraform so they are reviewed and reproducible, with the same alerting principles we use elsewhere: page on user-visible symptoms, link a runbook, name an owner. Our Terraform infrastructure as code service covers the workflow, and the SRE and incident response service covers on-call design.
Security and access
Datadog agents send data outward, so we review what leaves your network. That includes log content, trace attributes and process or command-line data. We configure scrubbing rules for sensitive values, restrict API and application keys by scope, rotate them, and use SSO with role-based access in the Datadog organisation. Cloud integrations use cross-account roles with least privilege rather than long-lived access keys.
Cost and capacity
We do not quote prices because they vary by plan, contract and usage. What we can say is where bills typically grow: custom metrics from high-cardinality tags, indexed log volume, APM spans ingested and retained, and containers or hosts running the agent. Our cost review lists the top contributors in each category, then applies controls such as log exclusion filters, sampling, removing unused custom metrics and tuning ingestion. Cost alerts and usage dashboards stop surprises recurring.
Day-2 operations and failure modes
Typical problems include agents silently failing after a node pool change, tag drift between teams, monitors that nobody owns, alert fatigue from default thresholds and sudden usage spikes after a verbose deploy. We monitor the agent itself, run a regular monitor review, and set usage alerts. Upgrades roll through Helm with staged clusters.
What a first month usually looks like
Rollouts go best when scope is small and sequenced. In the first weeks we usually deploy the agent to one non-production cluster, agree the tag scheme, and build a single service dashboard and a handful of symptom-based monitors for one important service. Only then do we widen to logs and APM, because each new signal changes both usage and the habits of the people reading it.
We also run the account in parallel with whatever you have today, so engineers can compare answers during real incidents. That comparison is more honest than a demo, and it shows quickly whether the product removes work or just moves it. If the trial reveals that a particular signal is not worth its cost, you find out before a contract is committed, not after.
Keeping the exit open
Even if Datadog is the right choice today, instrumenting with OpenTelemetry where practical and defining monitors as code keeps migration cost manageable. Nobody should stay with a vendor only because leaving is painful. If you want an independent review of your current observability spend and setup, book a consultation and see how we work with clients on managed DevOps.
Official documentation: www.datadoghq.com