The pipeline

ESSE ships as a Python package, an npm package and a website, all generated from the same JSON sources. This page describes how, and what that dual nature does and does not guarantee.

The stages

schema/ + example/            JSON sources — the authority
        |
        |  npm run set-schema-ids
        |     rewrites every $id to match its path
        v
        |  npm run build-schemas
        |     resolves include() and $ref, merges allOf,
        |     writes resolved copies + schemas.json
        v
     dist/js/{schema,example}/  resolved assets
        |
        +--> datamodel-codegen  -> src/py/mat3ra/esse/models/**   pydantic v2 models
        +--> compile_ts.ts      -> TypeScript types
        +--> npm run build-entity-graph -> site/graph.json     (from sources, not dist)
        |                               -> site/views.json     (the Explorer's category trees)
        |
        v
   npm + PyPI packages          and, on the CI deploy job, the site

The site is assembled in a site/ staging directory by the deploy-docs job — resolved schemas and examples, schemas.json, graph.json, views.json, these documentation pages, the schema explorer and the Ontology map — and published to GitHub Pages, which schemas.mat3ra.com fronts.

Resolution, and what it destroys

build-schemas does two things to every schema:

Resolves references. $ref targets are inlined, so a published schema is self-contained and a consumer needs no resolver.
Merges allOf. The intersection branches are flattened into one object.

This is a genuine convenience — the published material.json shows you every field a material has, in one document. But it is lossy in a specific way: after merging, you can no longer see which mixin a field came from, or that there was a mixin at all.

That is why the ontology map and the schema lint read schema/, never dist/js/schema/. The relationships they exist to describe are precisely what resolution removes. If you write a tool that cares about structure rather than content, read the sources.

The dual-runtime equivalence contract

ESSE is generated by two different runtimes, and the README is candid that they may differ. It is worth being precise about what is guaranteed.

Guaranteed identical, because they are the same files:

The JSON sources in schema/ and example/.
The $id of every schema.
The resolved JSON assets in dist/js, which both packages ship.

Generated per-runtime, and legitimately different:

Python models (src/py/mat3ra/esse/models/**) are produced by datamodel-codegen into pydantic v2 classes. Class names are generated and globally numbered — adding one schema can renumber classes in unrelated files. Do not depend on a generated class name like Units276; depend on the named top-level model.
TypeScript types are produced by json-schema-to-typescript with different naming rules again.
Only the JavaScript build produces the fully resolved, allOf-merged schemas used for this site.

What this means for a consumer: pin the package version, and treat the JSON as the contract. If a Python model and a TypeScript type disagree about a field's name or optionality, the schema is right and one of the generators is being clever; file it as a bug against the generation, not against the format.

Pre-commit

The repository regenerates its derived assets on commit rather than trusting contributors to remember. .husky/pre-commit:

Rebuilds JS and PY assets if any .json changed (npm run transpile-and-build-assets).
Runs pre-commit run --all-files — ruff, black, and datamodel-codegen for the pydantic models — inside the project's .venv, when one exists. Without a .venv the hook skips the Python model regeneration rather than failing.
Runs lint-staged, then npm run transpile.

dist/ is build output and is not committed. It is gitignored and rebuilt on demand: prepublishOnly produces it before an npm publish, and the deploy job builds it before assembling the site. graph.json is a site asset only — it is written into site/, never into dist/, so it does not ship inside the package.

A schema change is still rarely a one-file diff, because the regenerated pydantic models under src/py/ are committed. The codegen churn that appears in unrelated model files is a property of the generator's global class numbering, not a sign something went wrong.

Setting up so the hook runs:

python -m venv .venv
source .venv/bin/activate
pip install -e ".[all]"

CI

.github/workflows/cicd.yml runs the Python linter, Python tests across 3.10–3.13, and the JavaScript validate and test jobs on every push. npm test includes the schema lint, so a broken $ref or a mis-set $id fails the pull request rather than the deploy.

On dev, it then publishes to npm and PyPI and rebuilds the site.