[!info] All the benchmarking was performed on Macbook Pro M4, 24 GB

I was just exploring if there’s anything super fast that can mask sensitive data in logs, if logged at all.

The Regex

The first thing was obviously the Regex way. I created a regex with all the key-value pairs I wanted to mask, and tested it against log lines of 12KB each. Its the worst log size it can go in my case and wrote a Log4j’s RewritePolicy for it.

The results were shocking. While the baseline was ~2.8B ops/s, regex masking dropped to just ~850 ops/s. A massive, massive drop.

Obviously, I had to figure out something more optimized.

The JSON

One thing worked in my favor that the LogEvent sent to RewritePolicy is always JSON.

So the next approach was to parse the entire log body into a JSON object, then recursively search for keys to mask and set their values.

This was a big jump from regex. masking with the JSON approach hit ~15K ops/s. Still far below baseline, but a massive improvement.

The Concerns

For a single pod producing 500 logs/s, this isn’t a big deal on its own. But here’s the thing:

1. The pod, which is supposed to be doing only business logic, ends up spending CPU on masking too. Ensuring that the maximum portion of the CPU is still goes to core application is a challenge. Solution: move masking out of the pod entirely.
2. Let the pod just write to a file, and let the node level log shipper handle masking instead. But once logs are persisted to disk, the entire purpose of masking to protect sensitive data is already defeated.

The JSON approach should work fine for most services out there, but it’s still a massive drop from baseline.

---

Stay tuned as I’ll keep updating the blog as soon as find new improvements. Feel free to reach out to me on Twitter / X for any suggestion or to solve this problem collaboratively.

— LogLatency