How personal data ends up in logs
Nobody sets out to log customer emails. It happens through small decisions: a debug statement that prints a whole request object, an exception message that includes a query string, an access log with full URLs containing tokens, a retry handler that dumps the payload that failed. The data then travels. It is indexed, shown on dashboards, copied into snapshots and archived for years.
For teams subject to GDPR or HIPAA, this turns a logging platform into a regulated data store without anyone having decided so. Scrubbing reduces that exposure. It belongs to our logging, ELK, Loki and OpenTelemetry service and complements the structured logging guidelines and audit log collection. We are engineers, not lawyers: your counsel decides what the regulations require of you, and we implement the technical controls.
What good looks like
- A written data classification for log fields: never log, mask, hash, or allowed.
- Application code that does not emit sensitive values in the first place.
- A second layer of redaction in the pipeline for anything that slips through.
- Tests with fake personal data proving redaction works at each destination.
- A tested procedure to find and remove personal data when something leaks.
Layered controls, in the order that works
1. Do not log it
The strongest control is code. Replace logging of whole objects with explicit allowed fields. Configure web frameworks and HTTP clients so request bodies and sensitive headers are excluded. Change error handling so exceptions do not echo input. An allow-list of fields per log event is better than trying to remove bad ones afterwards. This ties into the schema in our guidelines.
2. Tokenise or hash what you need
Debugging often needs to correlate events for one user, not read their email. Replace the value with a keyed hash or an internal ID. A keyed HMAC prevents trivial reversal by dictionary attack, and the key lives in your secrets manager, not in the logging configuration. Keep in mind that, under GDPR, pseudonymised data can still be personal data, which is a question for your counsel.
3. Redact in the pipeline
The collector is where you catch accidents. Options:
Fluent Bit. A Lua filter or the modify filter can rewrite fields. For patterns in text, Lua gives full control.
-- redact.lua: mask email addresses in the "log" field
function redact(tag, timestamp, record)
local msg = record["log"]
if msg ~= nil then
msg = string.gsub(msg, "[%w%._%%%+%-]+@[%w%.%-]+%.%a+", "[redacted-email]")
record["log"] = msg
return 1, timestamp, record
end
return 0, timestamp, record
end
Logstash. The mutate filter with gsub, or the fingerprint filter for hashing, handles field-level changes.
OpenTelemetry Collector. The redaction processor (in the contrib distribution) can block or mask attribute values matching patterns and allow only listed attributes on spans and logs. The transform processor with OTTL can rewrite or delete attributes. Check the processor’s stability level and options for the Collector version you deploy, since they evolve.
Apply the same logic to traces. Span attributes such as http.url, db.statement and user identifiers are a frequent leak, as noted on the OpenTelemetry tracing page.
4. Find leaks you did not predict
Scheduled scanning looks for patterns in stored logs and in object storage. Tools such as Amazon Macie can discover sensitive data in S3, and open-source libraries like Presidio can detect entities in text. The output goes to a ticket and a dashboard, not to a person’s memory. Code scanning with Semgrep can flag risky logging calls in pull requests.
5. Control the destination
Even clean logs deserve access control. Restrict who can read which indices or tenants, encrypt storage, and set short retention for application logs, aligned with the retention policies. If your platform processes health data, vendor agreements such as a business associate agreement may be needed for hosted components, which is a procurement and legal matter.
Decisions and trade-offs
Mask or drop the event. Masking keeps the event useful. Dropping is safer for fields with no debugging value, such as passwords or tokens.
Edge or central redaction. Redacting on the node means the data never leaves the machine, which is the safer position. Central redaction is easier to maintain. For high-risk data, do it at the edge and again centrally.
Regex precision. Aggressive patterns mask innocent numbers and make logs useless. Loose patterns miss data. Tune using real samples and review false positives with developers.
Retention versus audit. Immutable audit storage and erasure duties can pull in different directions, so audit records should carry identifiers, not content.
When this is not the priority. If your system never handles personal data and logs only technical events, a light-touch code review and secret scanning may be enough.
Common failure modes
- Redaction in one place only, while a sidecar, a metrics label or a trace attribute still carries the value.
- Reversible masking such as base64, which is encoding rather than protection.
- Secrets in logs like API keys in URLs, a related and equally serious problem.
- Stack traces with data from SQL errors or deserialisation failures.
- Backups ignored. Snapshots and archives keep the data you thought you removed.
Working with legal and security teams
Technical controls are only part of this work. Someone must decide which fields count as personal or sensitive in your context, how long they may be kept, and who may read them. We do not make those legal decisions for you. We turn them into engineering requirements and bring the right questions to your data protection or compliance lead.
A practical output is a one-page data map listing each log source, the categories of data it may contain, the destination and the retention period. It gives auditors something concrete, and it gives developers a reference when they add a new log line.
You should also agree a response path for the day a leak is found: who is told, how the affected indices or objects are located, how deletion is carried out and verified, and how the event is recorded. Rehearsing it once is far cheaper than inventing it under pressure.
Delivery and handover
You receive the data classification, code changes through pull requests, collector rules in your repository, a synthetic test suite that runs in CI, and a leak response procedure. We hold a short session for developers using their own services as examples. Ongoing review can be included in Managed DevOps.
How to tell it is working
Synthetic markers never appear in any index, bucket or trace. Scheduled scans return clean results or findings that are fixed quickly. New services meet the logging standard before launch. To assess where personal data is in your logs today, contact us.