File mirror

Your containers most likely depend on some files you pull from the internet: dependencies hosted on GitHub, build toolchains, CLI tooling from your cloud vendor, and many more. For example, in your Dockerfile:

# Install the AWS CLI
RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-$(arch).zip" -o "awscliv2.zip" && \
    unzip awscliv2.zip && \
    ./aws/install

When you aim for reliable and stable builds this is problematic as files might change or get deleted at any time. There's zero guarantee that anything hosted on a URL will remain available; nor that it will always return the same file.

StableBuild solves this by operating a file mirror that acts as a pull-through cache to other servers. You prefix any URL that you download with your StableBuild file mirror URL, e.g.:

curl "https://your-prefix.httpcache.stablebuild.com/my-first-tutorial/https://awscli.amazonaws.com/awscli-exe-linux-$(arch).zip" -o "awscliv2.zip"

The first time this file is requested we'll fetch it from awscli.amazonaws.com, and then store the file in StableBuild. The next time you request this file we'll serve it from cache. The file is now immutable, and you'll always get the exact same file back - regardless whether the file was changed or deleted on the host server.

As different containers might require different versions of this file, you specify a cache key (above my-first-tutorial). Every cache key is its own separate cache.

That's it. Anything you download from the internet is now stable and reliable. 🎉

Authentication / HTTP headers

If you need access to a URL that requires authentication - for example an API key - you can pass the key in through an HTTP header. This is forwarded to the original server whenever the URL is not in cache. For example:

curl -H "x-api-key: xxxxxx" https://your-prefix.httpcache.stablebuild.com/cache-key/https://httpbin.org/headers

Afterwards this URL is in cache, and the URL is the cache key. This means that any headers sent are not interpreted anymore. The URL above will return the same content, even if you change the API key.

Basic authentication

If you use basic authentication, you can pass in the username/password in the URL. If you have the following URL: https://httpbin.org/basic-auth/admin/secret (this is a real URL) - which has as credentials username admin and password secret you can encode the username/password in the URL like this:

curl -v https://your-prefix.httpcache.stablebuild.com/cache-key/https://admin:[e-mail]/basic-auth/admin/secret

Tips & tricks

Deleting files from cache

Cached file are listed under Dashboard > File mirror. Go to "Cached files", click on the cache key, and find your URL there. Click the three dots, then Delete to delete the file from cache. On subsequent requests the file will be re-fetched.

Deleting a file from cache

Caching GitHub repositories

If you need access to a GitHub repository you might have some code like:

git clone https://github.com/libvips/libvips && \
    cd libvips && \
    git checkout 2be083adf2fcf3de5da393a748339bd103960cd3 && \
    # do something with this repository

StableBuild cannot cache git commands, but you can instead download a ZIP file with this commit from GitHub which can be cached. E.g.:

curl https://your-prefix.httpcache.stablebuild.com/my-first-tutorial/https://github.com/libvips/libvips/archive/2be083adf2fcf3de5da393a748339bd103960cd3.zip -o libvips.zip && \
    unzip libvips.zip && \
    mv libvips-2be083adf2fcf3de5da393a748339bd103960cd3 libvips && \
    cd libvips && \
    # do something with this repository

Uploading files manually

If you want to store local files in the file mirror:

1. Go to Dashboard > File mirror

2. Under "Cached files", either select a cache key or create a new cache key using the blue "New cache key" button.

3. Click "Upload file", select your file and a name.

Uploading a file manually to the File mirror

Afterwards, find the uploaded file under "Cached files", select the three dots and select "Copy URL". You can then use that URL in your Dockerfile or build script.

Copy URL for a manually uploaded file

Storage limits

Any cached file is subject to storage limits (or overage fees on paid plans). You can see your current storage usage on Dashboard. To determine if files are no longer fetched from cache (and thus whether you could delete them), you can look at the raw traffic logs under Dashboard > Billing > Raw usage logs.

Large URLs are processed asynchronously

When you make a HTTP request to StableBuild's file mirror, we usually cache the file while you download the first time. But in some cases we process files asynchronously. This happens:

1. If the file returned is >5GB.

2. If the file is >100MB, but no Content-Length header is set.

(This does not apply to files uploaded in the UI, they are always immediately processed.)

When this happens it might take a few minutes (or even more, depending on the size) before the URLs are cached in StableBuild. If this happens you'll see an hourglass icon next to the URL under "Cached files":

Asynchronously caching a file

Note that because this file is processed asynchronously, the file will be fetched a second time (from a different machine in a different data center).