Deploying on Kubernetes #5: Application Configuration


To read this it’s expected that you’re familiar with Docker, and have perhaps played with building docker containers. Additionally, some experience with docker-compose is perhaps useful, though not immediately related.

Necessary Background

So far we’ve been able:

  1. Create the helm chart to manage the resources
  2. Add the MySQL and Redis dependencies
  3. Create a functional unit of software … sortof.


Applications usually require some sort of configuration to work. Though we have deployed fleet, as well as it’s dependencies MySQL and Redis there is no way that fleet knows how to connect to these services, or the authentication details required.

  • Via environment variables


The fleet documentation indicates that it can accept configuration via files and environment variables also (as well, command line arguments, but we will not use this). This will likely be useful to us as a way to delimit secret configuration from non-secret configuration — but more on that later.

# Run via Docker as I don't need fleet locally$ docker run kolide/fleet fleet config_dump
# /dev/stdout:1-41mysql:
address: localhost:3306
username: kolide
password: kolide
database: kolide
tls_cert: ""
tls_key: ""
tls_ca: ""
tls_server_name: ""
tls_config: ""
address: localhost:6379
password: ""
cert: ./tools/osquery/kolide.crt
key: ./tools/osquery/kolide.key
tls: true
tlsprofile: modern
jwt_key: ""
bcrypt_cost: 12
salt_key_size: 24
token_key_size: 24
invite_token_validity_period: 120h0m0s
key_size: 64
duration: 2160h0m0s
node_key_size: 24
status_log_file: /tmp/osquery_status
result_log_file: /tmp/osquery_result
enable_log_rotation: false
label_update_interval: 1h0m0s
debug: false
json: false
disable_banner: false


Earlier as part of some initial work to define the chart specification, we generated the ConfigMap object. For reference, it currently looks like this:

# templates/configmap.yaml:1-16---
apiVersion: "v1"
kind: "ConfigMap"
app: {{ template "fleet.fullname" . }}
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
heritage: "{{ .Release.Service }}"
release: "{{ .Release.Name }}"
name: {{ template "fleet.fullname" . }}
# hello
# |-
# property.1=value-1
# property.2=value-2
# templates/configmap.yaml:11-16data:
config.yml: |-
address: localhost:3306
# ... and so on
kubectl get configmap --output=yaml

Consuming Configuration

Unfortunately, this doesn’t itself make that configuration available to the application process.

Consuming configuration as a filesystem

As mentioned previously, Kubernetes doesn’t store configuration files specifically, but rather configuration key→value pairs. So, we need to tell it to express this key as a file, with the value as the contents of that file.

  • Configuration
  • Network Filesystems
  • Information from the Kubernetes API itself
# templates/deployment.yaml:19,44:50template:
# The name comes from the configmap. It's also shown earlier
- name: "fleet-configuration"
name: {{ template "fleet.fullname" . }}
# templates/deployment.yaml:119,132:136          ports: 
# - the port stuff goes here
- name: "fleet-configuration"
readOnly: true
mountPath: "/etc/fleet"
# templates/deployment.yaml      containers:
- name: fleet
image: {{ .Values.pod.fleet.image | quote }}
- "fleet"
- "serve"
- "--config"
- "/etc/fleet/config.yml"
$ kubectl get podsNAME                                  READY     STATUS    RESTARTS   AGE
kolide-fleet-fleet-88c9b5876-2dd26 1/1 Running 3 7m
kolide-fleet-mysql-6c859797b4-gf6lk 1/1 Running 4 3d
kolide-fleet-redis-6d95f98b98-qswkz 1/1 Running 4 3d
$ kubectl exec kolide-fleet-fleet-88c9b5876-2dd26 cat /etc/fleet/config.yml---
address: localhost:3306
username: kolide
password: kolide

Updating Configuration

We have taken a sample configuration file, and mounted it into the container where the application is consuming it. Awesome! But ah, it’s the wrong configuration.

Splitting out secrets from configuration

I should mention at the outset, we should not store secrets in configmap resources. There is another for that, which we will cover in a future part of the series.

Known, consistent configuration

There is certain configuration that will always be consistent in releases managed by helm. This includes things like:

  1. Where to find Redis
  2. The status files for Fleet
$ kubectl get svc
$ kubectl get svc
kolide-fleet-mysql ClusterIP <none> 3306/TCP 3d
kolide-fleet-redis ClusterIP <none> 6379/TCP 3d
kubernetes ClusterIP <none> 443/TCP 3d
# templates/configmap.yaml:14-17    ---
# MySQL will resolve at `kolide-fleet-mysql`
address: kolide-fleet-mysql:3306
username: kolide
# templates/configmap.yaml:24-26redis:
address: kolide-fleet-redis:6379
# templates/configmap.yaml:43-47    osquery:
node_key_size: 24
status_log_file: /dev/stdout
result_log_file: /dev/stdout
kubectl logs ${POD}

Inconsistent Configuration

For the rest of the configuration, we simply defer it to the user to supply the required data. Some we will leave for now or simply delete, as there are other ways to handle data. But let’s take a couple of examples:

# /dev/stdout:36-40

debug: false
json: false
# templates/configmap.yml:49-53

debug: {{ default false .Values.fleet.logging.debug }}
json: {{ default false .Values.fleet.logging.json }}
disable_banner: {{ default false .Values.fleet.logging.disable_banner }}
# values.yml:6-15## Fleet application specific settings
## Whether to enable debug logging
debug: false
## Whether logging should be expressed in JSON
json: true
## Whether to disable the banner as part of the logs
disable_banner: true

In Summary

The configuration object allows us to supply various bits of application configuration outside the container, allowing extremely flexible and reusable applications.




Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store