Security gotcha with log collection on Azure Kubernetes cluster.

Azure Kubernetes Service provides a nice way to set up Kubernetes
cluster in the cloud. It’s quite practical as AKS is setup by default
with a rich monitoring and reporting environment. By default, all
container logs are collected, CPU and disk data are gathered. 👍

I used AKS to setup a cluster for my first client as a
freelance. Everything was nice until my client asked me why logs
collection was as expensive as the computer resources.💸

Ouch… 🤦

My first reflex was to reduce the amount of logs produced by all our
containers, i.e. start logging at warn level instead of info
level. This reduced the amount of logs quite a lot.

But this did not reduce the cost of collecting logs, which looks like
to a be a common issue.

Thanks to the documentation provided by Microsoft, I was able to find
that ContainerInventory data table was responsible of more than 60%
of our logging costs.

What is ContainerInventory ? It’s a facility to monitor the content
of all environment variables from all containers.

Wait… What ? ⚠

Should we be worried about our database credentials which are, legacy
oblige, stored in environment variables ?

Unfortunately, the query shown below confirmed that, yes, we should:
the logs aggregated by Azure contains the database credentials of my
client.

ContainerInventory
| where TimeGenerated > ago(1h)

Having credentials collected in logs is lackluster from a security
point of view. 🙄

And we don’t need it because our environment variables do not change.

Well, it’s now time to fix these issues. 🛠

We’re going to:

1. disable the collection of environment variables in Azure, which
   will reduce cost and plug the potential credential leak
2. renew all DB credentials, because the previous credentials can be
   considered as compromised (The renewal of our DB passwords is quite
   easy with the script I provided to my client)
3. pass credentials with files instead of environment variables.

In summary, the service provided by Azure is still nice, but beware of
the default configuration which may contain surprises.

I’m a freelance, available for hire. The https://code-straight.fr site
describes how I can help your projects.

All the best