Pinning your first container

In Why StableBuild? we've looked at a typical Dockerfile, and shown how it's dependent on a variety of services and repositories that are all unstable - which can break your build at any time. As a reminder, here's the Dockerfile:

# Base this Dockerfile on Ubuntu 20.04
FROM ubuntu:20.04
​
# Don't prompt us when installing packages
ARG DEBIAN_FRONTEND=noninteractive
​
# Install some OS level packages
RUN apt update && apt install -y curl software-properties-common
​
# We need a newer Python version, grab it from Deadsnakes PPA
RUN add-apt-repository -y ppa:deadsnakes/ppa && \
    apt update && \
    apt install -y python3.9 python3.9-distutils
​
# Install pip (the Python package manager)
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
    python3.9 get-pip.py
​
# Install a Python package through pip
RUN pip3 install onnx==1.14.0

Let's make this container deterministic and stable with StableBuild!

Signing up for StableBuild

First, head to https://dashboard.stablebuild.com and sign up (there's a free Community plan available). You'll now be on the Dashboard:

StableBuild dashboard

The dashboard gives you access to all StableBuild mirrors and repositories, user and key management, and to billing information.

Pinning the base image

The first line in our Dockerfile is:

FROM ubuntu:20.04

This base image will be overwritten in Docker Hub any time a new Ubuntu 20.04 version is released; and could even be deleted from the registry - leading to indeterministic or broken builds. To pin this image go to Dashboard > Docker mirror. Here you'll find your account-specific Docker mirror server that you can prefix to any FROM call in your Dockerfile.

Your account-specific Docker mirror

It's highly recommended to also use a more specific tag (if available). ubuntu:20.04 is updated very often, but specific snapshots of Ubuntu 20.04 are published under the focal-* tag in Docker Hub. Not pinning ubuntu:20.04 but f.e. on focal-20231128 (the 2023-11-28 version of Ubuntu 20.04) allows you to move to a newer Ubuntu version if needed, without having to remove the pinned image in StableBuild.

Thus, here, change your Dockerfile to:

FROM your-prefix.dockermirror.stablebuild.com/ubuntu:focal-20231128

And rebuild the container. The first time the container is built we'll fetch the ubuntu:focal-20231128 image from Docker Hub (including all architecture versions, not just the one you're building for), and store it in your own StableBuild cache. From now on this image is immutable. You'll now always get the same base image back, even when the upstream image is updated or deleted.

After pulling once the dashboard will update:

The base image is now cached in StableBuild

If you ever need to remove the image from the cache (e.g. because you need a newer Ubuntu 20.04 image) you can do that from "Cached tags".

Pinning packages from the Ubuntu package registry and other PPAs

Next our Dockerfile reads:

# Install some OS level packages
RUN apt update && apt install -y curl software-properties-common

# We need a newer Python version, grab it from Deadsnakes PPA
RUN add-apt-repository -y ppa:deadsnakes/ppa && \
    apt update && \
    apt install -y python3.9 python3.9-distutils

Here we install some packages from the Ubuntu package registry and the deadsnakes PPA, but these packages (both their versions, and whether they're available) can change at any time. To pin them head to Dashboard > Ubuntu repository. Then:

1. Download sb-apt.sh - which is a shell script that sets up the repository (you'll run it from your container); and place it next to your Dockerfile.

2. Select a pin date. We create a full daily copy of the complete Ubuntu (and Debian and Alpine) package registry - including all packages - plus many PPAs, so you'll pin on a date (the date that we created the copy). If you create a new container, just use the default - which is the last mirror date.

3. Select the package repositories you want to load. Here 'Ubuntu' and 'Deadsnakes (Python)'. Note: If you need to use another PPA, then just drop us a message - we'll add it to StableBuild.

Pinning packages from the Ubuntu package registry (and other PPAs)

Then, follow the instructions in Step 4 to update your Dockerfile, which will now read:

# Base this Dockerfile on Ubuntu 20.04 (from 2023-11-28)
FROM your-prefix.dockermirror.stablebuild.com/ubuntu:focal-20231128

ARG SB_API_KEY=your-prefix
ARG APT_PIN_DATE=2023-12-07T10:40:01Z

COPY ./sb-apt.sh /opt/sb-apt.sh
RUN bash /opt/sb-apt.sh load-apt-sources ubuntu deadsnakes

# Install some OS level packages
RUN apt update && apt install -y curl software-properties-common

# We need a newer Python version, grab it from Deadsnakes PPA (no need to manually add the repo anymore)
RUN apt update && apt install -y python3.9 python3.9-distutils

Rebuild the container, and the packages are now pulled from StableBuild's mirror.

The packages are now pulled from StableBuild's mirror

Pinning arbitrary files from the internet

Next, our Dockerfile reads:

# Install pip (the Python package manager)
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
    python3.9 get-pip.py

Here we download and install a file from the internet. To pin arbitrary URLs head to Dashboard > File mirror.

File mirror in StableBuild

Here you'll find a prefix that you can prepend to any URL - which automatically caches that file and make it immutable. After you do this the file is listed under "Cached files", and the exact same file will be returned on subsequent requests:

Viewing cached files in the "my-first-tutorial" cache key.

For most files this is enough. However, some scripts will download other files when running - and for those you'll also have to rewrite any URLs within the script, or set up some HTTP proxy. This is also the case for get-pip.py - it's just a bootstrap script and will download the actual pip package from the PyPi registry.

Fortunately, get-pip has a way to specify the PyPI mirror with the -i flag, and we can use StableBuild's PyPI mirror to pin the actual downloaded package (see "Pinning Python packages" below). Your Dockerfile now reads:

# Base this Dockerfile on Ubuntu 20.04 (from 2023-11-28)
FROM your-prefix.dockermirror.stablebuild.com/ubuntu:focal-20231128

ARG SB_API_KEY=your-prefix
ARG APT_PIN_DATE=2023-12-07T10:40:01Z

COPY ./sb-apt.sh /opt/sb-apt.sh
RUN bash /opt/sb-apt.sh load-apt-sources ubuntu deadsnakes

# Install some OS level packages
RUN apt update && apt install -y curl software-properties-common

# We need a newer Python version, grab it from Deadsnakes PPA (no need to manually add the repo anymore)
RUN apt update && apt install -y python3.9 python3.9-distutils

# Install pip (the Python package manager), by downloading get-pip.py from the
# file cache, then installing the actual package from the PyPI mirror
RUN curl https://your-prefix.httpcache.stablebuild.com/my-first-tutorial/https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
    python3.9 get-pip.py -i https://your-prefix.pypimirror.stablebuild.com/2023-12-07/

Pinning Python packages

Last, our Dockerfile reads:

# Install a Python package through pip
RUN pip3 install onnx==1.14.0

It seems like this would pin the onnx package to an exact version, but onnx itself has unpinned dependencies. To properly pin the package and all its sub-dependencies go to Dashboard > Pypi mirror.

Here you need to select a pin date. Just like the Ubuntu / Debian / PPA package registry we create a daily mirror of the complete Pypi registry - so you can install the complete package list from a specific point in time. If this is a new container, just use the default - but older versions of the PyPI mirror can be super useful to ressurect older Python examples (see Pypi mirror > Getting older Python examples working).

Pinning Python packages using StableBuild's PyPI mirror

To pin, just update the pip3 install line to reference the StableBuild mirror. Your Dockerfile will now read:

# Base this Dockerfile on Ubuntu 20.04 (from 2023-11-28)
FROM your-prefix.dockermirror.stablebuild.com/ubuntu:focal-20231128

ARG SB_API_KEY=your-prefix
ARG APT_PIN_DATE=2023-12-07T10:40:01Z

COPY ./sb-apt.sh /opt/sb-apt.sh
RUN bash /opt/sb-apt.sh load-apt-sources ubuntu deadsnakes

# Install some OS level packages
RUN apt update && apt install -y curl software-properties-common

# We need a newer Python version, grab it from Deadsnakes PPA (no need to manually add the repo anymore)
RUN apt update && apt install -y python3.9 python3.9-distutils

# Install pip (the Python package manager), by downloading get-pip.py from the
# file cache, then installing the actual package from the PyPI mirror
RUN curl https://your-prefix.httpcache.stablebuild.com/my-first-tutorial/https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
    python3.9 get-pip.py -i https://your-prefix.pypimirror.stablebuild.com/2023-12-07/

# Install a Python package through pip
RUN pip3 install \
    -i https://your-prefix.pypimirror.stablebuild.com/2023-12-07/ \
    install onnx==1.14.0

Rebuild the container, and done. This container is now fully pinned; and will always install the same OS version, package list and Python dependencies. 🎉

Tips & tricks

Traffic logs

Anything pulled through StableBuild is subject to traffic limits (or overage charges on paid plans, see the Dashboard for your quota). An overview of traffic for the current billing period is shown on the dashboard, but to see detailed traffic logs head to Dashboard > Billing > Raw usage logs. This lists all raw traffic logs, including service, IP and API key used:

Raw traffic logs in StableBuild

Keep your keys private

The URL to a service (e.g. in https://your-prefix.pypimirror.stablebuild.com/2023-12-07/) is used to authenticate with StableBuild, and is used to bill your account. Keep this URL private and don't commit it to public repositories, as this would allow anyone to pull from StableBuild using your account. If you see weird traffic spikes, check the raw usage logs to see if a key might be leaked. You can rotate keys via Dashboard > Keys.

We're considering creating limited scoped keys (e.g. only allow pulling certain tags) to help you make public repositories stable and deterministic - drop us an email at [e-mail] if that's relevant to you.