> For the complete documentation index, see [llms.txt](https://stablebuild.gitbook.io/stablebuild-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://stablebuild.gitbook.io/stablebuild-docs/mirrors-and-caches/huggingface-mirror.md).

# HuggingFace mirror

A lot of machine learning code downloads models from HuggingFace at build time or on startup. For example:

```python
from transformers import AutoModel

model = AutoModel.from_pretrained("openai-community/gpt2")
```

When you aim for reliable and stable builds this is problematic. This line downloads whatever is on the repository's `main` branch at that moment. If the author pushes new weights, a new tokenizer or a changed config, your next build picks it up without any warning.

StableBuild's HuggingFace mirror solves this by running an immutable mirror of HuggingFace model repositories. To download through StableBuild, just set `HF_ENDPOINT` to your mirror's domain. For example:

```
export HF_ENDPOINT=https://your-domain.hfmirror.stablebuild.com
```

[`huggingface_hub`](https://pypi.org/project/huggingface-hub/), and every library built on top of it (like [`transformers`](https://pypi.org/project/transformers/) or [`diffusers`](https://pypi.org/project/diffusers/)), reads `HF_ENDPOINT`. So the code above now downloads through StableBuild, without changing a single line.

We recommend pinning the models you use to a revision (a commit hash, a tag, a branch, ...). `main` is like `latest` for Docker images: what it points to keeps changing:

```python
model = AutoModel.from_pretrained("openai-community/gpt2", revision="607a30d")
```

When downloading from the command line, add `--revision`:

```
hf download sentence-transformers/all-MiniLM-L6-v2 model.safetensors --revision 1110a24
```

You can also fetch a single file with just curl. Everything after the mirror's domain is exactly the same path as on huggingface.co:

```
curl -O https://your-domain.hfmirror.stablebuild.com/sentence-transformers/all-MiniLM-L6-v2/resolve/1110a24/model.safetensors
```

Run it again, and this time it comes straight back from the cache. Downloaded revisions are listed under "Cached models" on **Dashboard > HuggingFace mirror**.

If you don't specify a revision, we look up the commit that `main` points to the first time you download the model, and store it. When you download the same model again you get that same commit back, even after the repository has been updated. Every file you download is stored in the cache, so you always get back the exact same model.

That's it. Your models are now stable and reliable. 🎉

### Tips & tricks

#### Updating a model

To move to a new version, **pull it again with the revision you want**. This is the safest way.

We don't recommend deleting the pin first. Once deleted, the model looks up the latest commit again on its next download; if the repository has disappeared from HuggingFace by then, you can't go back to the previous commit.

Only if you track `main` and want to move to the latest commit, delete the model under "Cached models" on **Dashboard > HuggingFace mirror**. The next download looks up the latest commit again.

#### What we mirror

* Public model repositories. Datasets and Spaces are not mirrored.
* We don't mirror gated or private repositories, as that would hand out the model without going through the terms of use the author set.

#### Files larger than 5GB

Files that are larger than 5GB are not cached while you download, but asynchronously. It might take a few minutes (or even more) before these are stored in StableBuild. You can observe the `x-in-sb-cache` header which is returned when downloading files.

#### Storage

Downloaded files count towards your organization's storage usage. If you go over the storage limit on the Community plan, files that aren't cached yet can no longer be downloaded. Files that are already cached remain available.