Add a translation pipeline

[stevhliu]

This PR adds a translate command to translate docs into other languages. There isn't very good coverage for the other languages, they fall out of sync with the English docs, and are difficult to keep up with and maintain because most times we can't review translation PRs.

Design

A nightly scheduled Job under hf-doc-build runs doc-builder translate transformers --lang ja --bucket /bucket which clones Transformers and reads docs/source/en/**. It masks things that shouldn't be translated - like specific doc-builder syntax - and segments the docs. Translated paragraphs are stored in a Bucket, hf-doc-build/doc-translate/cache and reused nightly to compare whether existing changes have been made. If there are no changes, it exits.

If something is new, it loads a model to only translate the prose that's changed using Transformers' native continuous batching. The finished .md files are written to a new Bucket, hf-doc-build/doc-translate/translations.

On the Transformers side, write build_ja_documentation.yml to call on the doc-builder's shared build workflow to sync the translations. It downloads the finished files and swaps them in. doc-builder build uses the existing page-level caching to only build pages whose content changed and then push to hf-doc-build/doc-build.

┌─ 03:00 UTC ─ HF Jobs, hf-doc-build namespace, one a100-large ────────────────┐
│                                                                              │
│  doc-builder translate transformers --lang ja --bucket /bucket               │
│    (bucket mounted as a folder, HF_TOKEN as a secret)                        │
│                                                                              │
│  1. clone transformers, read docs/source/en/**        732 pages              │
│  2. hide code/tags/URLs, split into paragraphs, hash   14,829 IDs            │
│  3. compare against cache/index.json                   ← ONE file read       │
│                                                                              │
│       nothing new → exit here, model never loads       ~$0.15  ⟵ most nights │
│       something new ↓                                                        │
│                                                                              │
│  4. load Gemma, translate ONLY the new paragraphs                     │
│  5. check each page, write the good ones to the bucket                       │
│        failures keep their last good version, or English                     │
└──────────────────────────────────────────────────────────────────────────────┘
                                    ↓
┌─ buckets in hf-doc-build ────────────────────────────────────────────────────┐
│  doc-translate/cache/           translated paragraphs, reused nightly   NEW  │
│  doc-translate/translations/    finished .md files the build reads      NEW  │
│  doc-build-cache/               prerendered HTML                     exists  │
│  doc/transformers/              what hf.co/docs serves               exists  │
└──────────────────────────────────────────────────────────────────────────────┘
                                    ↓
┌─ 04:00 UTC ─ GitHub Actions, huggingface/transformers ───────────────────────┐
│  build_ja_documentation.yml → doc-builder's shared build workflow            │
│                                                                              │
│  A. sync translations/ → docs/source/ja   ← the new step in this PR          │
│       replaces the folder wholesale, clearing the 20 stale orphans           │
│  B. doc-builder build --language ja --html   (page cache skips unchanged)    │
│  C. hf sync → doc/transformers,  push → doc-build dataset                    │
└──────────────────────────────────────────────────────────────────────────────┘
                                    ↓
                        hf.co/docs/transformers/ja

Preview

# get this branch
git checkout doc-builder/ja-translation

# set up preview env
uv venv .venv-tr
VIRTUAL_ENV=.venv-tr uv pip install -e . \
  "transformers @ git+https://github.com/huggingface/transformers"

# get read access to the `hf-doc-build` org
hf auth login

# get latest translation from the Bucket
hf buckets sync \
  hf://buckets/hf-doc-build/doc-translate/translations/transformers/ja \
  /tmp/ja-preview

# start and open the preview
git -C /tmp/ja-preview init -q
./.venv-tr/bin/doc-builder preview transformers /tmp/ja-preview --language ja
open http://localhost:5173/index