The Importance of Slim Container Builds

Mar 22, 2024 (Last Modified: Mar 22, 2024)

Page content

Introduction

Containers are everywhere nowadays and their adoption is only growing as I type this blog. To quantify that statement, Gartner predicts that “by 2027, more than 90% of global organizations will be running containerized applications in production, which is a significant increase from fewer than 40% in 2021.”. This quote was taken from a Google blog published at https://cloud.google.com/blog/products/containers-kubernetes/a-leader-in-2023-gartner-magic-quadrant-for-container-management. Due to the rapid rise of containerization, it is important to follow known best practices for building and deploying containers.

This specific blog is centered around how to properly build containers. More specifically, how to build containers that are “slim” and why it is important.

Real World Scenario

I have always preferred slim container builds for 3 very specific reasons:

Decreased pull times leads to quicker application deployment times
Smaller container images means less “stuff” installed, which leads to lower attack surface area (more secure)
Smaller container images leads to lower volume consumption on the nodes which run them (less storage consumption)

Recently, I was working with a customer that had incredibly large container builds (upwards to 2GB). While not ideal, sometimes we cannot get around the size of the container based on the application requirements. However, they had a scenario that was causing real world problems. To better understand the problem this was causing, their workflow looked a bit like this:

Large images were deployed to an OpenShift cluster
Each application code change caused a new image to be built
Each application code change caused a new image to be deployed
The development process happened many times per day

On the surface, this does not seem like a problem (outside of the obviously large images). However, the size of the images, coupled with frequent builds and deploys (doubling the node volume size requirements for both the imagefs and containerfs) was causing the Kubernetes Garbage Collection process to enter a state of constantly running. To make matters worse, the image garbage collection process and container garbage collection process in OpenShift are set to the same value and set to hard evict, meaning containers could be potentially killed in order to satisfy the garbage collection process.

In this customer environment, they would see containers disappearing during this process, which would cause application outages. Luckily this was not a production environment, but you can see how this could present itself as a larger problem if discovered in a production scenario.

There is an absolutely fantastic blog I read on this that can better explain the process in more detail > https://www.redhat.com/en/blog/image-garbage-collection-in-openshift.

Fixing in OpenShift

We were able to determine the above by checking the kubelet settings via the API (I simply picked one node):

oc get --raw /api/v1/nodes/$NODE_NAME/proxy/configz | jq

...    
    "imageMinimumGCAge": "2m0s",
    "imageGCHighThresholdPercent": 85,
    "imageGCLowThresholdPercent": 80,
...
    "evictionHard": {
      "imagefs.available": "15%",
...

Notice that the evictionHard percent and the imageGCHighThresholdPercent match (evictionHard is a reversed percent).

Ultimately, OpenShift provides a way to fix this via a KubeletConfig resource. Here is an example, to include commented out defaults and explanations in comments, of how we were able to fix this concern:

apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: gc-kubelet-config
spec:
  kubeletConfig:
    # NOTE: these are the default values
    # imageGCHighThresholdPercent: 85
    # imageGCLowThresholdPercent: 80
    imageGCHighThresholdPercent: 75
    imageGCLowThresholdPercent: 70
    imageMinimumGCAge: "5m30s"
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/worker: ""

The above configuration essentially lowers the value for which image garbage collection occurs in hopes that the hard eviction process can be avoided.

NOTE: I have filed an internal RFE to have this supported officially in ROSA. Today this applies only to self-managed OpenShift.

Ultimate Fix

The ultimate fix is not necessarily updating a few settings in OpenShift (although I would argue that the above configuration is more ideal anyway). It is to fix what this blog is to discuss, and that is to reduce the application image sizes. In our scenario, we were able to reduce our sizes from 2GB down to 500MB.
There is a ton of great information on how to do this, but without repeating what a lot of us already know, there is a great walkthrough guide at https://devopscube.com/reduce-docker-image-size/.

Small Build Example

Here is a sample of a recent container I built, using all of the best practices above, when creating a container for Ansible and using Red Hat’s Universal Base Image (UBI) to accomplish it:

#
# BUILD IMAGE
#
FROM registry.access.redhat.com/ubi9/ubi-minimal:9.3-1552 as build
ENV HOME=/home/ansible

# install required build packages
RUN microdnf --setopt=install_weak_deps=0 --nodocs -y install \
        bash \
        gcc \
        libffi-devel \
        git \
        gpgme-devel \
        libxml2-devel \
        libxslt-devel \
        curl-minimal \
        cargo \
        openssl-devel \
        python3-devel \
        python3-pip \
        cmake \
        gcc-c++ \
        unzip && \
    microdnf clean all

# copy content
COPY . ${HOME}

# add python dependencies
RUN python3 -m pip install --upgrade pip && \
        pip3 install --no-cache-dir -r ${HOME}/requirements/python.txt

# add ansible collection dependencies
RUN ansible-galaxy collection install --force -r ${HOME}/requirements/ansible.yaml && \
        pip3 install --no-cache-dir -r ${HOME}/.ansible/collections/ansible_collections/azure/azcollection/requirements-azure.txt

#
# SYSTEM IMAGE
#   NOTE: this layer lets us configure the system here and copy into a micro install for runtime
#
FROM registry.access.redhat.com/ubi9/ubi-minimal:9.3-1552 as system
ENV HOME=/home/ansible
ENV MICRODNF_OPTS="--config=/etc/dnf/dnf.conf \
    --setopt=install_weak_deps=0 \
    --setopt=cachedir=/var/cache/microdnf \
    --setopt=reposdir=/etc/yum.repos.d \
    --setopt=varsdir=/etc/dnf \
    --releasever=9 \
    --installroot=${HOME}/system \
    --nodocs \
    --noplugins \
    --best \
    --refresh"

# install packages to system root
RUN mkdir -p ${HOME}/system
RUN microdnf -y install ${MICRODNF_OPTS} \
        bash \
        openssl \
        shadow-utils \
        python3 && \
    microdnf clean all ${MICRODNF_OPTS}

# configure runtime user
RUN chroot ${HOME}/system \
        /bin/bash -c "groupadd ansible -g 1000 && useradd -s /bin/bash -g ansible -u 1000 ansible -d ${HOME}"

# remove packages that are not needed at runtime
RUN microdnf -y remove ${MICRODNF_OPTS} \
        shadow-utils && \
    microdnf clean all ${MICRODNF_OPTS}

#
# RUNTIME IMAGE
#
FROM registry.access.redhat.com/ubi9/ubi-micro:9.3-13
ENV HOME=/home/ansible

# copy content to container
COPY --from=system ${HOME}/system /
COPY --chown=ansible:ansible --from=build ${HOME} ${HOME}
COPY --from=build /usr/local/lib/python3.9 /usr/local/lib/python3.9
COPY --from=build /usr/local/lib64/python3.9 /usr/local/lib64/python3.9
COPY --from=build /usr/local/bin /usr/local/bin

# set kubeconfig and ansible options
ENV KUBECONFIG=${HOME}/staging/.kube/config
ENV ANSIBLE_FORCE_COLOR=1

WORKDIR ${HOME}

This produced the following output:

Size: 1.37GB (yes this is large but Ansible containers with lots of dependencies are a pig)
Vulnerabilities: (from Docker Scout) 0C 0H 14M 7L

Large Build Example

Compare this with a non-slim Dockerfile (the same build with multi-stage removed) ignoring best practices to produce a large container image:

FROM registry.access.redhat.com/ubi9/ubi-minimal:9.3-1552 as build
ENV HOME=/home/ansible

# install required build packages
RUN microdnf --setopt=install_weak_deps=0 --nodocs -y install \
        bash \
        gcc \
        libffi-devel \
        git \
        gpgme-devel \
        libxml2-devel \
        libxslt-devel \
        curl-minimal \
        cargo \
        openssl-devel \
        openssl \
        python3-devel \
        python3-pip \
        python3 \
        cmake \
        gcc-c++ \
        shadow-utils \
        unzip && \
    microdnf clean all

# copy content
COPY . ${HOME}

# add python dependencies
RUN python3 -m pip install --upgrade pip && \
        pip3 install --no-cache-dir -r ${HOME}/requirements/python.txt

# add ansible collection dependencies
RUN ansible-galaxy collection install --force -r ${HOME}/requirements/ansible.yaml && \
        pip3 install --no-cache-dir -r ${HOME}/.ansible/collections/ansible_collections/azure/azcollection/requirements-azure.txt

# configure runtime user
RUN groupadd ansible -g 1000 && useradd -s /bin/bash -g ansible -u 1000 ansible -d ${HOME}
USER 1000

# set kubeconfig and ansible options
ENV KUBECONFIG=${HOME}/staging/.kube/config
ENV ANSIBLE_FORCE_COLOR=1

WORKDIR ${HOME}

This produced the following output:

Size: 2.03GB
Vulnerabilities: (from Docker Scout) 0C 28H 391M 86L

Summary

The difference between the small build and the large build is as follows:

Size: 660MB
Vulnerabilities:
- Critical: 0
- High: 28
- Medium: 377
- Low: 79

Alternatives When Building My Own Image is Not Necessary

When building your own container is not necessary, once validated through whatever Software Supply Chain processes your organization has, I am a big advocate of the following priority of container selection:

Scratch/Distroless - where possible, scratch images provides bare minimum image for applications.
Rapidfort - I stumbled across this project which takes the popular Bitnami images, and runs special sauce to remove everything out of them in order to produce only a container that has the necessary bits to run.
Red Hat Universal Base Images - fits in the Red Hat ecosystem well and ideal where support is of utmost importance.
Bitnami - When none of the above is possible, the use of Bitnami images is my fallback. They are not the slimmest images but have a lot of industry best practices built in and a lot of work has gone in to securing and hardening these images.