Deploying on Kubernetes #5: Application Configuration

Assumptions

To read this it’s expected that you’re familiar with Docker, and have perhaps played with building docker containers. Additionally, some experience with docker-compose is perhaps useful, though not immediately related.

Necessary Background

So far we’ve been able:

  1. Create the helm chart to manage the resources
  2. Add the MySQL and Redis dependencies
  3. Create a functional unit of software … sortof.

Configuration

Applications usually require some sort of configuration to work. Though we have deployed fleet, as well as it’s dependencies MySQL and Redis there is no way that fleet knows how to connect to these services, or the authentication details required.

  • Via environment variables

Fleet

The fleet documentation indicates that it can accept configuration via files and environment variables also (as well, command line arguments, but we will not use this). This will likely be useful to us as a way to delimit secret configuration from non-secret configuration — but more on that later.

# Run via Docker as I don't need fleet locally$ docker run kolide/fleet fleet config_dump
# /dev/stdout:1-41mysql:
address: localhost:3306
username: kolide
password: kolide
database: kolide
tls_cert: ""
tls_key: ""
tls_ca: ""
tls_server_name: ""
tls_config: ""
redis:
address: localhost:6379
password: ""
server:
address: 0.0.0.0:8080
cert: ./tools/osquery/kolide.crt
key: ./tools/osquery/kolide.key
tls: true
tlsprofile: modern
auth:
jwt_key: ""
bcrypt_cost: 12
salt_key_size: 24
app:
token_key_size: 24
invite_token_validity_period: 120h0m0s
session:
key_size: 64
duration: 2160h0m0s
osquery:
node_key_size: 24
status_log_file: /tmp/osquery_status
result_log_file: /tmp/osquery_result
enable_log_rotation: false
label_update_interval: 1h0m0s
logging:
debug: false
json: false
disable_banner: false

ConfigMap

Earlier as part of some initial work to define the chart specification, we generated the ConfigMap object. For reference, it currently looks like this:

# templates/configmap.yaml:1-16---
apiVersion: "v1"
kind: "ConfigMap"
metadata:
labels:
app: {{ template "fleet.fullname" . }}
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
heritage: "{{ .Release.Service }}"
release: "{{ .Release.Name }}"
name: {{ template "fleet.fullname" . }}
data:
# example.property.1: hello
# example.property.file: |-
# property.1=value-1
# property.2=value-2
# templates/configmap.yaml:11-16data:
config.yml: |-
mysql:
address: localhost:3306
# ... and so on
kubectl get configmap --output=yaml

Consuming Configuration

Unfortunately, this doesn’t itself make that configuration available to the application process.

Consuming configuration as a filesystem

As mentioned previously, Kubernetes doesn’t store configuration files specifically, but rather configuration key→value pairs. So, we need to tell it to express this key as a file, with the value as the contents of that file.

  • Configuration
  • Network Filesystems
  • Information from the Kubernetes API itself
# templates/deployment.yaml:19,44:50template:
spec:
volumes:
# The name comes from the configmap. It's also shown earlier
- name: "fleet-configuration"
configMap:
name: {{ template "fleet.fullname" . }}
# templates/deployment.yaml:119,132:136          ports: 
# - the port stuff goes here
volumeMounts:
- name: "fleet-configuration"
readOnly: true
mountPath: "/etc/fleet"
# templates/deployment.yaml      containers:
- name: fleet
image: {{ .Values.pod.fleet.image | quote }}
args:
- "fleet"
- "serve"
- "--config"
- "/etc/fleet/config.yml"
$ kubectl get podsNAME                                  READY     STATUS    RESTARTS   AGE
kolide-fleet-fleet-88c9b5876-2dd26 1/1 Running 3 7m
kolide-fleet-mysql-6c859797b4-gf6lk 1/1 Running 4 3d
kolide-fleet-redis-6d95f98b98-qswkz 1/1 Running 4 3d
$ kubectl exec kolide-fleet-fleet-88c9b5876-2dd26 cat /etc/fleet/config.yml---
mysql:
address: localhost:3306
username: kolide
password: kolide

Updating Configuration

We have taken a sample configuration file, and mounted it into the container where the application is consuming it. Awesome! But ah, it’s the wrong configuration.

Splitting out secrets from configuration

I should mention at the outset, we should not store secrets in configmap resources. There is another for that, which we will cover in a future part of the series.

Known, consistent configuration

There is certain configuration that will always be consistent in releases managed by helm. This includes things like:

  1. Where to find Redis
  2. The status files for Fleet
$ kubectl get svc
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kolide-fleet-mysql ClusterIP 10.104.173.216 <none> 3306/TCP 3d
kolide-fleet-redis ClusterIP 10.106.51.61 <none> 6379/TCP 3d
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3d
# templates/configmap.yaml:14-17    ---
mysql:
# MySQL will resolve at `kolide-fleet-mysql`
address: kolide-fleet-mysql:3306
username: kolide
# templates/configmap.yaml:24-26redis:
address: kolide-fleet-redis:6379
# templates/configmap.yaml:43-47    osquery:
node_key_size: 24
status_log_file: /dev/stdout
result_log_file: /dev/stdout
kubectl logs ${POD}

Inconsistent Configuration

For the rest of the configuration, we simply defer it to the user to supply the required data. Some we will leave for now or simply delete, as there are other ways to handle data. But let’s take a couple of examples:

# /dev/stdout:36-40

logging:
debug: false
json: false
# templates/configmap.yml:49-53

logging:
debug: {{ default false .Values.fleet.logging.debug }}
json: {{ default false .Values.fleet.logging.json }}
disable_banner: {{ default false .Values.fleet.logging.disable_banner }}
# values.yml:6-15## Fleet application specific settings
fleet:
logging:
## Whether to enable debug logging
debug: false
## Whether logging should be expressed in JSON
json: true
## Whether to disable the banner as part of the logs
disable_banner: true

In Summary

The configuration object allows us to supply various bits of application configuration outside the container, allowing extremely flexible and reusable applications.

--

--

See https://www.andrewhowden.com/

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store