Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
9.6 KiB
Language data format
How the Language data other than English is described, checked and published as Language
packs: one downloadable archive per locale, with an index the apps pin. English (and the
interim carry-over listed in language-data/README.md) is
committed and embedded; everything here is fetched from its upstream at a pinned commit and never
committed. Licence classes are ADR 0004; why the data
lives here is ADR 0005. The audit the manifest was seeded
from is dated 2026-10-31.
The manifest: language-data/manifest.toml
One [[file]] per upstream file:
[[file]]
locale = "nl" # lowercase BCP 47
kind = "spelling-aff" # spelling-aff | spelling-dic | hyphenation | licence
url = "https://raw.githubusercontent.com/LibreOffice/dictionaries/<commit>/nl_NL/nl_NL.aff"
commit = "<upstream commit sha>"
sha256 = "<hex>"
size = 12345 # bytes
licence = "BSD-3-Clause" # SPDX expression
class = "Embeddable" # Embeddable | Download-only | Excluded
name = "Nederlands" # optional, first entry per locale wins: endonym
note = "…" # optional: the licence statements, quoted, when evidence is weak
- A pack is every entry of one locale.
spelling-aff/spelling-dicare a Hunspell pair,hyphenationa TeX pattern file (an unmodifiedhyph-*.texfrom tex-hyphen, compiled when loaded),licencea licence or README text that travels with the data. licenceis the licence the file is used under. When upstream offers several, the one in the least restricted class is recorded (ADR 0004). When the evidence is weak or contradictory,licencelists every licence the statements offer,classis Download-only (or Excluded, if any statement is non-commercial or no-derivatives) andnotequotes the statements and where they are (bg, hr, pt-pt, ta, uk, eu, ug). Alicenceentry carries the licence and class of the data it describes.classmay be more restricted than the licence allows, never less; when it is, the data entries need anote.- Licence texts. Every locale has at least one
licenceentry, the upstream notice (a hyphenation file's own header counts: the.texis listed again as alicenceentry when its licence has no SPDX id). On top of that, every licence id its data files name must have a full text in the pack: an SPDX id's standard text islanguage-data/licences/<id>.txt(from spdx/license-list-data at31846481, committed here), added to the archive asLICENCES/<id>.txt; aLicenseRef-id's text is alicenceentry whoselicencenames it. - Excluded entries are recorded (so the decision and its evidence stay visible) and never published.
- Sources are pinned raw files: LibreOffice/dictionaries, tex-hyphen, and an upstream repository or wooorm/dictionaries where the audit names it the best source.
Licence allow-list
Every SPDX id in an expression must be in the checker's allow-list
(crates/craft-i18n-tests/src/packs.rs, LICENCES), which gives each its class. OR takes
the less restricted side and AND the more restricted. A licence with no SPDX id gets a
LicenseRef- id there first (today LicenseRef-BSD-Lebedev, the Russian dictionary's BSD
licence with a "modified versions must be marked" clause, and LicenseRef-hyph-bg, the
Bulgarian patterns' permissive licence). WITH exceptions and + are not allowed: write
-or-later.
Check words: language-data/checks/<locale>.toml
correct = ["huis", "boek", "school", "water", "stad"] # at least 5
incorrect = ["boekk", "schooll", "werkken"] # at least 3
hidden = true # optional
reason = "craft-proof doesn't support compounding yet (COMPOUNDFLAG)"
Common words, written here (this repository's licence). A regional locale without its own file
uses its language's (es-mx uses es). hidden = true keeps a pack out of releases while
craft-proof can't read it, or can't yet use it well; it needs a reason. The reason names the
gap in the pinned craft-proof: an unsupported feature (craft-proof doesn't support <feature> yet (<AFF flag>)), missing word segmentation, or suggestions too slow to use. It is a statement of fact
about the pinned craft-proof, so refresh it when the pin moves.
As of 2026-11-02 (craft-proof at storytold/craft-libs rev 47c9f625, the merge of #497, flag-based
compounding): 80 locales, 3 Excluded (eu, ug, uk), 2 hidden (mn, whose suggestions take 200–580 ms over
its 12,066 affix rules; th, which needs word segmentation), 75 published.
Checks
Every CI run (cargo test --workspace, no network):
- the manifest parses; every entry has a valid lowercase BCP 47 locale (no English), a URL that
parses as
https://raw.githubusercontent.com/<owner>/<repo>/<commit>/<path>with its 40-digitcommitas the third segment (no empty or dot segments, query or fragment), a 64-digit lowercasesha256, a size of 1 byte to 64 MiB (craft-proof's limit), a licence and a class; - every licence parses as SPDX, uses only allow-listed ids, and its class is not less restricted
than the licence allows; a more restricted class has a
note; - no duplicate (locale, kind); licence file names are
[A-Za-z0-9._-], not starting with., unique in a locale and never clash with a standard text's<id>.txt; - a locale with spelling has both
.affand.dic; every locale has a name and a licence entry, and every data licence id has a full text (above); every file inlanguage-data/licences/is an allow-listed SPDX id; - every locale with spelling that isn't Excluded has check words (or is hidden with a reason), none empty, and every checks file names a manifest locale;
- no locale with an Excluded entry is published.
With CRAFT_I18N_FETCH=1 (the weekly and release workflow,
.github/workflows/language-packs.yml):
- every file downloads and matches its
sizeandsha256(every_file_matches_its_pin); - every published pack's
.aff/.dicloads throughcraft-proof's affix engine, accepts everycorrectword and rejects everyincorrectone, and its hyphenation patterns compile (packs_load_and_check). Hidden packs are tried too and reported, so the log says when one could be published.
Files are cached by sha256 in $CRAFT_I18N_CACHE (default target/language-data-cache).
Release assets
A craft-i18n GitHub release language-packs-<n> (pushed tag, or the workflow run by hand with a
release name) holds the assets below. A release is never modified: the workflow refuses a name
that already exists, creates the release as a draft, uploads every asset, then publishes it. A
fix is a new language-packs-<n+1>. A manual run must be on main and refuses a tag that
already exists; runs for the same release name never overlap. If an upload fails, the run
leaves a draft release behind: delete it before running again, or the rerun refuses
the name.
<locale>.tar.zstper published pack (no Excluded entry, not hidden, checks passed). At the archive root:spelling.aff,spelling.dic,hyphenation.pat(any may be absent),LICENCES/<original file name>for eachlicenceentry,LICENCES/<SPDX id>.txtfor each standard text, andpack.toml(locale, name, licence, class, version,upstream(assourcebelow),licence_texts, and a[[source]]per file with its url, commit, sha256, licence and class). Deterministic: sorted names, zero times and owners, mode 0644, zstd level 19.index.json:{ "format": 1, "release": "language-packs-1", "packs": [ { "locale": "nl", "name": "Nederlands", "version": "<craft-i18n commit>", "url": "https://github.com/storytold/craft-i18n/releases/download/language-packs-1/nl.tar.zst", "sha256": "<hex of archive>", "size": 123456, "licence": "BSD-3-Clause", "class": "Embeddable", "spelling": true, "hyphenation": true, "source": "https://github.com/LibreOffice/dictionaries/tree/<commit>/nl_NL" } ] }source(optional, additive to format 1) is the upstream folder of the pack's first data file at its pinned commit, for the app to show before install. A pack'slicenceis its files' distinct licences joined withAND; itsclassits most restricted file's.
Apps pin the index by URL + sha256; packs have no update checks of their own.
The language-packs tool
cargo run --release --bin language-packs -- check # the offline checks
cargo run --release --bin language-packs -- fetch # download + verify every file into the cache
cargo run --release --bin language-packs -- build --out DIR --release NAME --version SHA \
[--base-url URL] [--only nl,fr,pl] [--cache DIR] [--no-download]
--base-url defaults to the release's download URL; --only builds just those locales (and
fails on an unknown, hidden or Excluded one).
Testing an index locally
To try an app against packs before a release is published, build them with archive URLs on a local server and serve the folder:
cargo run --release --bin language-packs -- build --out /tmp/packs \
--release language-packs-test --version "$(git rev-parse HEAD)" \
--base-url http://127.0.0.1:8765/ --only pl,ru,it
python3 -m http.server 8765 --bind 127.0.0.1 --directory /tmp/packs
Then point the app's pack index at http://127.0.0.1:8765/index.json with the sha256 of that
file (sha256sum /tmp/packs/index.json). Leave out --only to build every published pack.