Uncertainty Quantification (UQ)#
gracemaker has three ways to tell when a model is unsure:
- CALM extrapolation grade γ, for GRACE-1L/2L/3L models: one model call, calibrated so that γ ≈ 1 is the boundary of the training data (arXiv:2609.40060). Most of this page is about it.
- D-optimality extrapolation grade, for GRACE/FS models (Phys. Rev. Materials 7, 043801 (2023); arXiv:2212.08716).
- Ensembles (query-by-committee), for any model.
New to UQ? Start with the tutorial
UQ workflows goes step by step from building the CALM
artifact to new training data, with small example models. This page is the
reference: the method, all grace_uq options and the Python API.
1. CALM extrapolation grade#
CALM (calibrated Mahalanobis, Lysogorskiy, Bochkarev and Drautz, arXiv:2609.40060) gives every atom an uncertainty in four steps:
- Map the atom's local environment to a feature vector (the basis-RP feature, below).
- Cluster the training environments per element (streaming k-means).
- Measure the Mahalanobis distance of a new environment to its nearest cluster centre.
- Divide this distance by a threshold calibrated on the training data. The result is γ.
Definitions#
For an atom of element e:
-
Feature z. A fixed, seeded Johnson–Lindenstrauss random projection of the model's rotation-invariant (\(\ell=0\)) basis b, plus log-norm density channels:
\[\mathbf{z} = \Big[\ \frac{\mathbf{b}}{\lVert\mathbf{b}\rVert}\,R\ ,\ \ \log\lVert\mathbf{b}\rVert\ ,\ \ \log\lVert\mathbf{b}^{(1)}\rVert\ ,\ \dots\Big]\]b holds the \(\ell=0\) basis functions that enter the invariant reduce(s) before the atomic energy; \(\mathbf{b}^{(k)}\) is the block of reduce \(k\). \(R = \mathcal{N}(0,1)/\sqrt{d}\) is generated from
rp_seedand stored in the artifact, so the artifact describes its own feature. Withrp_dim= \(d\) (default 128), the feature dimension is D = d + 1 + (number of reduce blocks). The normalization keeps stretched environments visible, the density channels keep compressed ones visible, so both raise γ. -
Cluster assignment. The nearest centroid by Euclidean distance:
\[k^* = \arg\min_k \| \mathbf{z} - \boldsymbol{\mu}_{e,k} \|^2\] -
Sigma (
atomic_sigma). The Mahalanobis distance to that centroid, with the centroid \(\boldsymbol{\mu}_{e,k}\) and covariance \(\Sigma_{e,k}\) of the cluster:\[\sigma = \sqrt{(\mathbf{z} - \boldsymbol{\mu}_{e,k^*})^\top \, \Sigma_{e,k^*}^{-1} \, (\mathbf{z} - \boldsymbol{\mu}_{e,k^*})}\] -
Gamma (
gamma). Sigma divided by the calibrated threshold of the cluster:\[\gamma = \frac{\sigma}{\theta_{e,k^*}}\]
The threshold \(\theta_{e,k}\) is a robust upper fence on the training sigma of the cluster, \(\mathrm{median} + 3 \cdot (1.4826 \cdot \mathrm{MAD})\), about a 3-sigma bound. It is not a p99 quantile, so a heavy outlier tail cannot raise it. If many of your training atoms read \(\gamma > 1\), see Calibrating the threshold.
\(\gamma \approx 1\) means the atom is at the boundary of the training distribution; \(\gamma \gg 1\)
means an environment outside it. gamma is the one UQ signal to use, for screening, active
learning and HAL.
The workflow has two stages:
- Build (once per set of model weights): fit the clusters on features of the training set,
accumulate the covariances and calibrate the thresholds, with
grace_uq build. - Use: load the artifact, or the exported SavedModel or Kokkos
.npzthat carries it, and readgammaper atom in ASE, ingrace_uq predictor in LAMMPS.
2. D-optimality (GRACE/FS models)#
GRACE/FS models have their own extrapolation grade, based on D-optimality, as for ACE
(Lysogorskiy et al., Phys. Rev. Materials 7, 043801 (2023);
arXiv:2212.08716). Build an
active set from the training data with pace_activeset
(how) and use it in
ASE or in LAMMPS. The tools on this
page do not apply to it.
3. Ensembles (query-by-committee)#
An ensemble asks whether independently trained models disagree. It works for any GRACE
model, also one without a CALM artifact. Fit the same input with several seeds
(gracemaker ... --seed 1, --seed 2, …) and pass the models together to one TPCalculator:
from tensorpotential.calculator import TPCalculator
calc_ens = TPCalculator(model=[
"fit/seed/1/saved_model/",
"fit/seed/2/saved_model/",
"fit/seed/3/saved_model/",
])
at.calc = calc_ens
at.get_potential_energy()
calc_ens.results["energy_std"] # std. dev. of total energy over the ensemble
calc_ens.results["forces_std"] # std. dev. of forces
calc_ens.results["stress_std"] # std. dev. of stress
energy, forces and stress are the ensemble mean. An ensemble multiplies the cost of
fitting and inference by the number of models, and its spread is not calibrated: there is no
"γ ≈ 1 is the boundary". Use CALM for routine screening, active learning and HAL, and an ensemble
for a second, independent opinion or for a model without a CALM artifact.
CLI tool: grace_uq#
grace_uq works with CALM and has one subcommand per task:
| Subcommand | Purpose |
|---|---|
build |
Build the CALM artifact from the training data and export a SavedModel with UQ. |
info |
Print a summary of a CALM artifact (.npz). |
predict |
Energies, forces, stresses and per-atom γ for a dataset; several workers; reads pkl.gz, extxyz or a LAMMPS dump. |
select |
Pick the structures for DFT from a pool of candidates. |
hal |
Run uncertainty-biased (HAL) walkers, one or several (--repeats); see UQ workflows. |
grace_uq without a subcommand prints this overview. Each subcommand lists
its options with --help:
grace_uq <subcommand> --help
grace_uq build#
build makes the CALM artifact in three steps, spread over several worker
processes, and then exports a SavedModel with UQ.
In a fit folder, no arguments are needed
gracemaker writes a fit into seed/<N>/. Run grace_uq build there
without arguments:
cd my_fit/seed/1
grace_uq build
It reads the files the fit left in this folder:
| Input | Default |
|---|---|
--model-yaml |
model.yaml |
--checkpoint |
checkpoints/checkpoint.best_test_loss.index |
--train-data |
training_set.pkl.gz, the training split that gracemaker saves (unless data: save_dataset: False). If it is missing: data.filename of ../../input.yaml, but not when that file splits the data (train_size, test_size or test_fraction); then the build stops and asks for --train-data |
It writes gmm_artifacts.npz, the elbow report and saved_model/ into
the same folder, with the default options: 4 workers, GPU 0, k chosen from
1 2 4 8 16, --rp-dim 128. If no training set is found but
gmm_artifacts.npz exists, it only exports the SavedModel.
Basic usage#
grace_uq build --model-yaml model.yaml \
--checkpoint checkpoints/checkpoint.best_test_loss.index \
--train-data training_set.pkl.gz \
--n-workers 4 \
--n-clusters 1 2 4 8 16 \
--artifact-path gmm_artifacts.npz
This runs the three steps with 4 workers, tries k = 1, 2, 4, 8 and 16
clusters and picks k with the elbow method. It then exports a SavedModel with
the compute_uq signatures to saved_model/, next to the artifact.
Export-only mode#
If the artifact exists already, leave out --train-data to export only the
SavedModel:
grace_uq build --model-yaml model.yaml \
--checkpoint checkpoints/checkpoint.best_test_loss.index \
--artifact-path gmm_artifacts.npz
For the Kokkos LAMMPS pair styles
(grace/1l/kk, grace/2l/kk, grace/3l/kk), bake the artifact
into the .npz weights with grace_utils
(details):
grace_utils -p model.yaml -c checkpoints/checkpoint.best_test_loss.index \
export_kokkos -o grace_kokkos.npz --uq-artifacts gmm_artifacts.npz
All options of grace_uq build
| Argument | Description |
|---|---|
--model-yaml |
the model file (default: model.yaml). |
--checkpoint |
the model checkpoint, with or without .index (default: checkpoints/checkpoint.best_test_loss.index). |
--train-data |
one or more training-data files: pickled DataFrames (.pkl.gz, .pkl) or sharded TF datasets (default: training_set.pkl.gz). Not needed for export-only mode. |
--train-data-weighted |
repeatable: a weight (float > 0), then training shard paths; the atoms of these shards count with this weight. For a training mix that differs from the shards on disk. Not together with --train-data; pickle shards only. |
--filter-fn |
module.path:function_name of a function filter_atoms(ase.Atoms) -> bool; structures for which it returns False are dropped (see Structure filtering below). |
--n-workers |
worker processes (default: 4). |
--n-clusters |
clusters per element (default: 1 2 4 8 16). One value is used as is; several values run the elbow method, which picks k. |
--threshold-percentile |
put γ = 1 at the Q-th percentile (0 < Q ≤ 100) of each cluster's training sigma histogram, instead of the default fence. γ is then not comparable with default artifacts (Calibrating the threshold). |
--max-neighbours-per-batch |
target number of neighbour pairs per batch for pickle input (default: 15000); sets the GPU memory use. |
--frac |
use a random fraction (0.0 to 1.0) of the training data, for a faster build. |
--seed |
seed for shuffling the data and for the KMeans start (default: 42). |
--artifact-path |
the artifact file (default: gmm_artifacts.npz); its folder is created if needed. |
--export-path |
the SavedModel folder (default: saved_model/ next to the artifact). |
--no-export |
do not export a SavedModel. |
--gpus |
comma-separated GPU indices for the workers, e.g. 0,1,2,3 (default: 0). |
--threads-per-worker |
CPU threads per worker. auto (default) = cores / workers; an integer overrides it, 0 sets no limit. |
--rp-dim |
projection dimension d of the feature (default: 128). |
--rp-seed |
seed of the projection matrix R (default: 42). |
--restart |
delete the intermediate .step*.npz files and build from scratch. |
--verbose |
print the full worker output and progress. |
Structure filtering (--filter-fn)
To keep some structures out of the artifact, for example chemistries the
model handles poorly, give --filter-fn a function that returns False
for them. It runs on each shard before --frac:
# uq_filter.py
RARE_EARTHS = frozenset({"Gd", "Eu", "Yb"})
def keep(atoms) -> bool:
return not (set(atoms.get_chemical_symbols()) & RARE_EARTHS)
PYTHONPATH=/path/to/filter:$PYTHONPATH \
grace_uq build --train-data /path/shards/*.pkl.gz --filter-fn uq_filter:keep
The workers inherit PYTHONPATH from your shell. Each shard logs how
many structures the filter removed. An element that the filter removes
completely is missing from the artifact; check with grace_uq info.
Pipeline steps#
The method is described in arXiv:2609.40060. Each step spreads the training data over the workers, and the master merges their results:
- Clustering: streaming k-means of the features per element, for every candidate k at once. With several k, the elbow method picks one k; its report and plot are saved next to the artifact. An element with too few atoms gets fewer clusters.
- Covariances: every atom goes to its nearest centroid; the master sums the scatter matrices, inverts the regularized covariances and prints their condition numbers.
- Thresholds: a histogram of σ per cluster gives the threshold.
- Export: a SavedModel with the signatures
compute,compute_uq(withdsigma_dr) and the fastercompute_uq_gamma_only.
The workers save their results after every step, so an interrupted build continues where it
stopped when you run the same command again; --restart starts over.
grace_uq hal#
The hal subcommand runs one uncertainty-biased (HAL) MD walker: the force
is biased along the gradient of Σγ, frames with max γ above a trigger are
collected, and a live copy of the CALM model is updated after each collection
so that the walker keeps moving to new configurations. It takes the model as
model.yaml + checkpoint (not a SavedModel) and the training-set artifact:
grace_uq hal --model-yaml model.yaml \
--checkpoint checkpoints/checkpoint.best_test_loss.index \
--artifact-path gmm_artifacts.npz \
--structure start.extxyz --ensemble nvt -T 300 --n-steps 1000 \
--output-dir hal_w0
Periodic structures use --ensemble nvt, nvt_ramp, npt or npt_stress;
clusters and molecules (pbc=False) use --ensemble cluster. The collected
frames go to hal_w0/hal_collected.extxyz, together with the updated live
artifact, the per-step history, a summary and a diagnostics figure. A
step-by-step introduction with the tutorial models, the Python API
(run_hal), the LAMMPS counterpart and the selection of the collected frames
for DFT are on the UQ workflows
page. This section is the reference.
HAL options#
The first rows are required: the model (--model-dir, or --model-yaml,
--checkpoint and --artifact-path), the starting structure, the
temperature and the number of steps. The other options follow in
alphabetical order. grace_uq hal --help lists the same options:
| Option | Default | Meaning |
|---|---|---|
--model-dir |
— | directory to search for model.yaml, the checkpoint and the CALM artifact (search order in --help); explicit --model-yaml, --checkpoint, --artifact-path override what is found |
--model-yaml, --checkpoint |
(required without --model-dir) |
the model; a SavedModel cannot be used, because the live tower is updated |
--artifact-path |
(required without --model-dir) |
training-set CALM artifact: the frozen tower, and the start of the live tower |
--structure |
(required) | starting structure; any file ASE can read |
-T |
(required) | temperature, K; several values need --repeats (grid) |
--n-steps |
(required) | MD steps |
--alphas |
0.0005 0.001 0.002 0.005 0.01 0.02 0.05 0.1 0.2 0.5 1.0 | weights of the additive live update, tried in the given order (keep them ascending); must end in 1.0 while inflation is on |
--bulk-modulus |
auto |
eV/ų (npt_stress) |
--cluster-wall-k |
1.0 | eV/Ų (cluster) |
--cluster-wall-margin |
2.0 | Å (cluster; containment wall) |
--ensemble |
nvt |
nvt, nvt_ramp, npt, npt_stress, cluster (Ensembles); several values need --repeats (grid) |
--equil-steps |
0 | steps without collection at the start (the bias waits --tau-ema steps either way) |
--feature-gamma-min |
1.0 | keep the CALM features of the atoms with γpristine ≥ this of every collected frame in hal_features.npz, the input of grace_uq select --hal-dirs; none switches it off |
--fragment-mult |
1.25 | bond cutoff for n_fragments: atoms closer than this × (rcov,i + rcov,j) are bonded (recorded only, never used to filter) |
--friction |
0.01 | Langevin friction, 1/fs (nvt, nvt_ramp, cluster) |
--gpus |
— | sets CUDA_VISIBLE_DEVICES for the run |
--history-every |
1000 | flush the history every N steps |
--inflate-max-factor |
4.0 | cap on the covariance scale factor per round |
--inflate-max-rounds |
3 | at most this many inflation rounds per collection |
--live-start-artifact-path |
— | start the live tower from an earlier walker's hal_live_artifacts.npz |
--n-collect |
— | stop after this many collected frames |
--nvt-ramp |
1.0 1.4 | V/V0 at start and end (nvt_ramp) |
--output-dir |
hal_w0 |
output directory; with --repeats the parent of the walker directories (default hal_walkers). The walker refuses to write into a directory that already holds HAL output unless --overwrite is given |
--pair-radii |
— | JSON file with per-pair overrides in Å for the ZBL switch and the rejection distance, e.g. {"Al-Li": {"zbl": [1.3, 1.8], "reject": 1.6}}; either key may be left out |
--param-dtype |
— | override the model dtype (float32 / float64) |
--plot-only DIR |
— | redraw the diagnostics figure of a finished or killed walker |
--pool DIR …, --pool-output, --gamma-pristine-min |
— | pool the collected frames of these walker folders, or of all walker folders below them at any depth, instead of running a walker (pool files) |
--progress-every |
100 | print a status line every N steps; 0 = quiet (run_hal: progress_every, default 0) |
--reject-bond-factor |
0.65 | reject if a pair is closer than this × (rcov,i + rcov,j) |
--reject-gamma |
10 | reject a trigger with max γlive or max γpristine above this |
--remove-update-work / --no-remove-update-work |
on | after each CALM update, take the bias energy the update injects, τr·(Σγbefore − Σγafter), back out of the kinetic energy (nvt, nvt_ramp, cluster; recorded only for npt, npt_stress) |
--save-every |
10 | save the live artifact every N collections |
--seed |
42 | initial velocities and Langevin noise |
--stress-tau |
0.02 | stress-bias strength τs (npt_stress) |
--structure-index |
0 | frame of a multi-frame --structure |
--tau |
0.15 | force-bias strength τ; system dependent, see Choosing τ. Several values run a τ scan: one walker per τ (K with --repeats K) into OUTPUT_DIR/tau_<τ>/hal_w<r>/, at most --walkers-per-gpu at a time, table in OUTPUT_DIR/walkers.csv (grid) |
--tau-ema |
10 | EMA window and bias delay, steps |
--tdamp, --pdamp |
100, 1000 | MTK damping times, fs (npt, npt_stress) |
--timestep |
auto |
fs; auto = 2 fs, or 0.5 fs with H, He, Li, Be |
--trigger-gamma |
1.5 | trigger threshold for γlive |
--trigger-gap |
10 | minimum steps between triggers |
--trigger-min-atoms |
1 | trigger only when at least this many atoms have γlive > --trigger-gamma; 1 = max γlive decides |
--update |
alpha_search |
live-tower update; none keeps it fixed |
--update-inflate / --no-update-inflate |
on | bounded covariance inflation after an α = 1 update that misses the target |
--update-target |
1.25 | α-search target for the frame's max γ |
--update-work-floor |
0.5 | the removal never cools the walker below this fraction of -T |
--volume-guard |
0.75 1.30 | stop when V/V0 leaves this band (npt, npt_stress) |
--walker-id |
0 | written into every collected frame |
--repeats K |
— | run K independent walkers for every setting (one combination of --ensemble, -T and --tau), each in its own process; K walkers in total when each has one value (Many walkers) |
--walkers-per-gpu M |
4 | with --repeats: at most M walkers at once on each visible GPU, the others wait for a free slot |
--zbl |
auto |
auto (universal ZBL) or off |
--zbl-switch |
0.55 0.75 | ZBL fully on below IN, off above OUT, × (rcov,i + rcov,j) |
--trigger-min-atoms makes the trigger less sensitive to a single atom
with a high γ.
The same walker runs from Python with run_hal; see HAL from Python in the Python API.
How the HAL driver works#
HAL adds a force along the gradient of Σγ to the physical force:
The moving averages run over --tau-ema steps, and the bias is off while they fill. The
reported energy is the physical one, without the bias. On every MD step:
- One model call gives the energy, forces, stress, γ, ∇Σγ and the CALM features.
- Two CALM towers. The live tower drives the bias and the trigger and is updated after
every collected frame. The frozen tower is the training-set artifact; its γ
(
gamma_pristine) is checked at every trigger. - Trigger. A frame triggers when at least
--trigger-min-atomsatoms have γlive >--trigger-gamma, and--trigger-gapsteps have passed since the last trigger. - Rejection. A trigger is rejected (no frame, no update) when max γlive or max
γpristine is above
--reject-gamma, or when two atoms are closer than--reject-bond-factor× the sum of their covalent radii. - Live update. The driver takes the first weight α from
--alphasthat brings the frame's max γ below--update-target, and adds α times the frame's covariance to the clusters the frame touches. Centroids and thresholds stay fixed and the training data are never forgotten, so γlive ≤ γpristine: chained walkers do not flag configurations inside the training distribution. If even α = 1 is not enough, the covariances of these clusters are inflated in a few bounded rounds (--inflate-max-rounds,--inflate-max-factor). - ZBL guard. A switched ZBL repulsion keeps atoms apart below
--zbl-switch× the sum of the covalent radii. It is not part of the bias normalization.
The walker stops after --n-steps or --n-collect frames, when the volume leaves
--volume-guard (npt, npt_stress), or on non-finite forces or γ; hal_summary.json gives
the reason. HAL re-inverts both towers in float64, so its γpristine can differ
slightly from the γ of grace_uq predict or LAMMPS.
Limit of the additive update. With one cluster per element, an update along one direction
also covers further drift along it, so a walker that keeps moving the same way collects little.
A higher --update-target makes each update weaker, and the walker collects more.
Temperature control#
Each CALM update adds bias energy, which the walker would turn into heat. In the Langevin
ensembles (nvt, nvt_ramp, cluster) the driver therefore takes this energy out of the
kinetic energy right after each update, but never below --update-work-floor × -T (default
0.5). Walkers that collect often, small molecules most of all, can then run a little below the
target temperature. To switch this off, pass --no-remove-update-work (run_hal:
remove_update_work=False); the Langevin thermostat then removes the extra heat over time. In
npt and npt_stress the energy is only recorded, never removed.
HAL ensembles#
A structure must be either fully periodic (all three pbc true and a full
cell) or fully non-periodic (all pbc false). Mixed pbc, such as a slab with
pbc=(True, True, False), is refused. Non-periodic structures must use
cluster, and cluster refuses periodic cells.
--ensemble |
Structures | Dynamics | Use it for |
|---|---|---|---|
nvt (default) |
periodic | Langevin, --friction 0.01 1/fs, total momentum reset every step |
General exploration at fixed volume. |
nvt_ramp |
periodic | Langevin, cell volume ramped linearly from --nvt-ramp START to END × V0 over the run (default 1.0 → 1.4) |
Expanded, low-density and near-fracture configurations. The ramp costs no extra model call. |
npt |
periodic | MTK NPT at 1 bar, --tdamp 100 fs, --pdamp 1000 fs |
Volume fluctuations and thermal expansion. The walker stops when V/V0 leaves --volume-guard (default 0.75 1.30). Needs an ASE version with ase.md.nose_hoover_chain.MTKNPT. |
npt_stress |
periodic | as npt, plus an uncertainty stress bias σ − τs·B·Ŵ (Ŵ: moving-average unit direction of the Σγ virial) |
Pushing the cell shape and volume towards high γ. τs = --stress-tau (default 0.02); B = --bulk-modulus in eV/ų, or auto (5-point E–V fit with the model at the start). |
cluster |
non-periodic | as nvt, with zero rotation at the start, plus a containment wall |
Clusters, molecules, nanoparticles. |
Containment wall#
Atoms that evaporate from a cluster would fly away and never come back. The
cluster ensemble keeps them with a flat-bottom harmonic restraint on the
distance of each atom from the initial centre of mass c0:
with the margin --cluster-wall-margin (default 2.0 Å) and the stiffness
k = --cluster-wall-k (default 1.0 eV/Ų). Atoms inside Rwall
feel nothing. The wall energy is reported separately (wall_energy in the
history), not in the physical energy, and its forces are added after the
bias. Rwall is written to the settings as cluster_wall_radius.
Collected frames#
Every collected frame is appended to hal_collected.extxyz at once. It carries the per-atom
arrays gamma_live, gamma_pristine and grace_forces, and in info the step, walker id,
temperature, V/V0 (or Rg/Rg0 for cluster), α, τr, grace_energy,
grace_stress, the max γ of both towers, the update work and collection_index, which links the
frame to hal_features.npz. Model predictions are stored only under grace_* keys, never as a
calculator result, so they cannot be mistaken for reference labels.
n_fragments counts bonded fragments (atoms closer than --fragment-mult × the sum of their
covalent radii). It is recorded only, never used to filter. A stretched bond in a hot molecule can
count as a split, and an expanded metal cell can read as many fragments with nothing broken.
Output files#
Per walker in --output-dir:
| File | Written | Content |
|---|---|---|
hal_collected.extxyz |
appended and flushed at every collection | the collected frames (keys listed above) |
hal_live_artifacts.npz |
every --save-every collections and at the end (atomic write); not written with --update none |
the updated live CALM artifact |
hal_features.npz |
at every history flush and at the end (atomic write); not written with --feature-gamma-min none |
per collected frame, the CALM features, γpristine, element and atom index of its atoms with γpristine ≥ --feature-gamma-min, the element counts of the frame, and the sha256 of the training-set artifact; the input of grace_uq select --hal-dirs |
hal_history.npz |
every --history-every steps and at the end; float32, compressed (for diagnostics; run_hal(...).history keeps float64) |
per step: max γ, Σγ, fraction of atoms with γ > 1, τr, the two EMA norms, T, V/V0 and pressure (or Rg/Rg0 and wall energy), shortest pair distance and its ratio to the rejection distance, collected/rejected status, α, max γpristine (trigger steps only), the number of atoms above --trigger-gamma and the update work (collection steps only) |
hal_settings.json |
after setup, before the first step | all settings, including the resolved model paths, the α grid, the training-set artifact sha256 uq_artifact_sha256, the tower re-inversion time reinvert_s, n_fragments0 and fragment_mult, the time step, wall radius and bulk modulus actually used |
hal_summary.json |
at the end | stop reason, step and collection counts, rejections by reason (gamma, gamma_pristine, short_bond); per collection γ, α, n_inflate, inflate_factor, the update work, update_ms (time of the α search, update, inflation and update-work γ evaluation, ms) and n_fragments; n_fragments0 and n_fragmented (collected frames with more fragments than the start); mean temperature (temperature_mean_K), total and removed update work, wall time, steps/s, number of model calls, settings |
hal_diagnostics.png |
at the end, or with --plot-only |
the diagnostics figure |
walkers.csv |
several walkers only (a τ scan or --repeats), in the parent --output-dir, after every walker |
one row per walker: its settings, collected, rejected (by reason), max γlive, max γpristine of the collected frames, n_fragmented, stop reason, mean temperature, speed |
A killed walker (for example at the wall-time limit) keeps every frame it has
collected, the live artifact as of the last save, and the history as of the
last flush; grace_uq hal --plot-only DIR redraws its figure.
Diagnostics figure#
hal_diagnostics.png shows on one step axis: max γlive with the collected and
rejected triggers and max γpristine; τr and the temperature;
V/V0 and pressure (for cluster: Rg/Rg0 and the wall energy); the shortest
pair distance and α per collection; and a box with the settings and counts. An example is in
UQ workflows, HAL from the command line.
What τ means#
By construction, the bias force is τ times the physical force, in summed per-atom magnitude
(averaged over --tau-ema steps). Only its size comes from the physical force; its direction
comes from the γ landscape.
Near equilibrium the bias points mostly against the physical force, because the training data sit
near energy minima. So the bias mainly softens the potential: sampling at T is roughly like
sampling at a higher effective temperature, Teff ≈ T/(1 − 0.8 τ). This is why a small τ
collects little on a well-trained model. At τ ≈ 1.25 the restoring force would cancel, and walkers
run away well before that. Where this happens depends on the system; find a good τ with a
τ scan. On a GPU, runs with the same --seed are not
bit-identical, so the counts differ from run to run.
Many walkers and parameter grids#
--repeats K runs K independent walkers that differ only in the seed (--seed + r) and the
start frame (frame --structure-index + r of a multi-frame --structure).
--ensemble, -T and --tau can take several values: K walkers
then run for every combination. All walkers run as separate processes, at most
--walkers-per-gpu (default 4) at a time on each visible GPU:
grace_uq hal --model-dir MODEL --structure start.extxyz --n-steps 2000 \
--repeats 2 --ensemble nvt npt -T 300 --tau 0.35 0.5 --output-dir grid
This runs 2 × 2 × 2 = 8 walkers into grid/<label>/hal_w<r>/, where the label names the
settings that vary (nvt_tau_0.35, …). With one combination the folders are
OUTPUT_DIR/hal_w<r>/ (default OUTPUT_DIR hal_walkers). Each walker logs to walker.log in
its folder; a failed walker does not stop the others, and grace_uq hal then exits with code 1.
At the end, a table with one row per walker is printed and saved as walkers.csv. Walker ids are
unique over the grid, so one call selects from all of them: grace_uq select --hal-dirs grid
(it searches the subfolders at any depth).
Several --tau values without --repeats are a τ scan with one walker per τ, the same as
--repeats 1. From Python, use run_walkers or scan_tau
(HAL from Python).
Differences from the paper protocols
The driver differs on purpose from the drivers of Fig. 7 and 8 of arXiv:2609.40060, so those figures are reproduced only approximately:
- Additive covariance update, Cnew = Cold + α Cframe,
instead of the paper's convex blend,
Cnew = (1 − α) Cold + α Cframe, which stays available as
GMMUQModel.update_one(..., mode="convex"). The bounded inflation after α = 1 follows the paper; there are no extra compounded α = 1 updates. - A fixed stress-bias strength instead of an adaptive controller, and a volume guard that stops the walker instead of reflecting walls.
- The trigger is max γ by default; the count trigger of Fig. 8 is
--trigger-gamma 1.3 --trigger-min-atomsmax(3, ⌈0.15 N⌉). - Bond rejection by covalent radii, no per-atom bias switch-off and no cap on τr.
- τ = 0.15 with the moving average for
nvt_ramp, instead of 2.0 without it, and one model call per step instead of two.
HAL in LAMMPS#
The TensorFlow pair style grace runs the HAL bias too
(example). pair_style grace tau τ adds
s·∇iΣγ to every GRACE force, with
over the GRACE forces only (not those of zbl or other hybrid/overlay styles). This is the
bias of grace_uq hal without the moving average.
All HAL keywords go on the pair_style line, right after grace:
group li type 2 # only needed for tau_group; define it before pair_style
pair_style hybrid/overlay grace tau 0.4 tau_norm sum tau_group li stress_tau 0.02 bulk_modulus 65 zbl 1.3 1.8
pair_coeff * * grace saved_model Al Li
pair_coeff 1 1 zbl 13 13
pair_coeff 1 2 zbl 13 3
pair_coeff 2 2 zbl 3 3
tauτ: strength of the force bias.stress_tauτs withbulk_modulusB (GPa, default 100) adds the stress bias ofnpt_stress.estimate_bulk_modulus(atoms, calc)intensorpotential.uq.halgives the model's own B (in eV/ų; divide byase.units.GPa).tau_norm sum(default) ormax: the sums in s, or the largest per-atom norms.zbl IN OUTthroughhybrid/overlayguards against too-close atoms; take IN and OUT at about 0.55 and 0.75 × the sum of the covalent radii.fix pairgivesgamma,atomic_sigma,gmm_clusterand ∇Σγ (uncertainty_force).
HAL on a part of the system
tau_group <group> on the pair_style line applies the bias only to the atoms of a LAMMPS
group, and the normalisation sums run over that group only. The rest of the system follows
plain MD. Use it to explore, for example, an interface, a defect or a molecule on a surface.
pair_modify changes the keywords between two run commands: pair_modify tau 0.2
tau_group all with a plain pair_style grace, pair_modify pair grace tau 0.2 tau_group all
under hybrid/overlay.
Compared with grace_uq hal: one MPI rank only; the CALM model stays fixed, so the bias keeps
pushing to regions already dumped; no moving average, no trigger gap, no rejection and no volume
guard; bulk_modulus is set by hand. Frames are written by the dump condition alone, so filter
them with grace_uq select. The keywords are in the grace branch of the GRACE LAMMPS sources
(lammps-ace).
grace_uq predict#
The predict subcommand evaluates a dataset with a UQ-enabled model: energies,
forces, stresses and the per-atom gamma column.
grace_uq predict --model UQ/saved_model \
--artifact UQ/gmm_artifacts.npz \
--dataset candidates.pkl.gz \
--output predicted.pkl.gz \
--features-gamma-min 1.0
| Option | Default | Meaning |
|---|---|---|
--model, --checkpoint, --artifact |
— | a SavedModel with UQ, or a model.yaml with its checkpoint and CALM artifact |
--dataset, --output |
—, predicted.pkl.gz |
input and output; the extension selects the format (.pkl.gz, or .xyz/.extxyz). --dataset also takes a LAMMPS text dump (any file name; recognised by its first line ITEM: TIMESTEP) |
--dump-elements EL … |
— | for a LAMMPS dump without an element column: the element of atom type 1, 2, …; ignored when the dump has the column. A dump with neither is an error |
--features-gamma-min X |
off | keep the CALM features only of the atoms with γ ≥ X, as the candidate columns cand_* of a HAL pool, plus X and the sha256 of --artifact; grace_uq select reads such a file like a HAL pool. With a SavedModel, pass the artifact it was exported with as --artifact to record the hash. Not with --no-ase-atoms or extxyz output |
--save-features |
off | keep the features of every atom (column features); needed by fps-all |
--n-workers |
auto |
worker processes; auto runs min(4, ⌈frames / 200⌉) workers, round robin over the visible GPUs (--gpus, else CUDA_VISIBLE_DEVICES), and a single worker in the same process. An explicit number is spread the same way over the visible GPUs |
--no-ase-atoms |
off | drop the structures from the output |
The output rows keep the order of the input for any number of workers.
grace_uq select#
The select subcommand picks the structures to compute with DFT from a pool
of candidates. It needs the training-set CALM artifact. From HAL walkers it
works on their hal_features.npz sidecars, without the model or a
prediction:
grace_uq select --hal-dirs hal_w0 hal_w1 hal_w2 \
--artifact gmm_artifacts.npz \
--output selected.pkl.gz
A file written by grace_uq predict --features-gamma-min (--predicted) or
a grace_uq hal --pool file (--dataset) is read the same way. For any
other pool, select --dataset predicts first, with --model plus
--checkpoint and --artifact for a model.yaml, or --uq for a
SavedModel; when the artifact is known, this prediction keeps only the
candidate features. The default strategy calm-greedy picks frames largest γ
first and updates a copy of the CALM model after every pick. Without -n it
returns the natural set, the frames needed until no remaining frame has max
γ above --gamma-min under the updated copy; -n caps it. The strategies, the natural set and the
element floors are described below, and the route for
large pools after them;
a worked example is step 4 of the UQ workflows page.
Combine strategies
--strategy takes several parts, NAME[:N] each, run in order on the frames not yet
picked. For example, --strategy calm-greedy random-all:10 takes the natural set as the
backbone and adds 10 random frames from the rest. Later parts avoid configurations close to
earlier picks.
| Option | Default | Meaning |
|---|---|---|
--hal-dirs DIR … |
— | HAL walker folders, or folders that contain them at any depth: select from their hal_features.npz, read only the selected frames |
--dataset |
— | pool as pkl.gz (ase_atoms column), extxyz or LAMMPS text dump; predicted first, unless it holds candidate columns (grace_uq hal --pool, grace_uq predict --features-gamma-min) |
--dump-elements EL … |
— | element of atom type 1, 2, … for a LAMMPS dump --dataset without an element column (as in grace_uq predict) |
--predicted |
— | output of grace_uq predict, with --features-gamma-min (window strategies) or --save-features (all strategies) |
--model, --checkpoint, --artifact, --uq |
— | model for the prediction; the training-set CALM artifact (--artifact or --uq) is needed by calm-greedy and checked against the hash of candidate columns |
--model-dir DIR |
— | fills --model, --checkpoint and --artifact from one directory, searched like grace_uq hal --model-dir; explicit flags win. With --hal-dirs or --predicted only the artifact must be found |
-n |
none for calm-greedy |
number of structures with a single --strategy; required for fps-extrap, fps-all and random-*; for calm-greedy a cap on the natural set. With several strategy parts, give N per part instead |
--strategy NAME[:N] … |
calm-greedy |
calm-greedy, fps-extrap, random-extrap, random-all, fps-all; several parts run in the given order, each on the frames not yet picked, with N frames per part (N optional for calm-greedy). The output column strategy names the part that picked each frame |
--gamma-min, --gamma-max |
1.0, no cap | γ window of the candidate atoms; for calm-greedy, --gamma-min is also the natural-set limit |
--update-target |
1.25 | calm-greedy: α-search target of each update of the selection copy |
--element-stratified, --no-element-stratified |
off for calm-greedy, on for the others |
per-element atom floor of the selection |
--min-per-element |
D + 1 | the floor in atoms per element, clamped to what -n frames of the pool hold; also switches the floor on for calm-greedy |
--fps-max-pool |
all candidates (fps-extrap), 50 000 (fps-all) |
mini-batch FPS cap |
--n-workers, --gpus |
auto |
workers of the prediction inside select, as for grace_uq predict |
--output |
selected.pkl.gz |
output file; the extension selects the format (.pkl.gz, or .xyz/.extxyz without the selection columns) |
Selection strategies#
--strategy |
Picks |
|---|---|
calm-greedy (default) |
frames, largest γ first, each pick scored after updating a copy of the CALM model with the earlier ones; stops at the natural set |
fps-extrap |
farthest-point sampling over the atoms with γ in [--gamma-min, --gamma-max], in the CALM metric |
random-extrap |
random frames among those with at least one atom in the γ window |
random-all |
random frames of the whole pool, no γ filter |
fps-all |
farthest-point sampling over all atoms, no γ filter |
calm-greedypicks the frame with the largest max γ, updates a copy of the training-set CALM model with it (as in HAL,--update-target), and scores the next pick on the updated copy, so frames similar to earlier picks drop out. It stops at the natural set: when no remaining frame has max γ above--gamma-minunder the copy.-ncaps it. This is γ under the selection copy, not after refitting: a refitted model with a rebuilt artifact can still read higher γ. Needs the artifact (--artifact,--uqor--model-dir).fps-extrapspreads the picks evenly over the extrapolative atoms (features whitened per training cluster, distances per element). No stopping point: needs-n.random-extrapis a cheap baseline over the frames with extrapolative atoms. Needs-n, no artifact.random-alldraws frames at random from the whole pool: reference or validation frames that γ would not choose. Needs-n.fps-allspreads the picks over all atoms, without a γ filter. Needs-nand--save-features.
A .pkl.gz output has the columns gamma_at_pick, pick_order and strategy.
γ window. --gamma-min (default 1.0) and --gamma-max (default no cap) select candidate
atoms, not frames: a selected frame can still hold atoms above --gamma-max. To drop whole HAL
frames, pool with --gamma-pristine-min (pool files).
Element floors. A floor gives every element of the pool at least D + 1 atoms in the
selection (D: the CALM feature dimension), as far as the pool allows. It is off by default for
calm-greedy, which covers every element through γ, and on for the other strategies;
--element-stratified, --no-element-stratified and --min-per-element change it. The summary
says whether the floors were met.
Large pools: keep only the extrapolative atoms, then select#
For a large pool, do not store the features of every atom. Keep only the candidates, the atoms with γ above a threshold, and select from them:
- MD frames, LAMMPS dumps or any other dataset:
grace_uq predict --features-gamma-min 1.0keeps the CALM features only of the atoms with γ ≥ 1.0. The file is much smaller than with--save-features, andselect --predictedreads it like a HAL pool. - HAL walkers keep these features already, in
hal_features.npz(--feature-gamma-min, default 1.0). Select straight from the walker folders with--hal-dirs, with no model and no GPU, or pool them first withgrace_uq hal --pool.
# any frames: predict once, keep the candidate atoms, then select
grace_uq predict --model saved_model --artifact gmm_artifacts.npz \
--dataset md_frames.pkl.gz --features-gamma-min 1.0 --output pool_pred.pkl.gz
grace_uq select --predicted pool_pred.pkl.gz --artifact gmm_artifacts.npz --output for_dft.extxyz
# HAL walkers: select from their sidecars directly
grace_uq select --hal-dirs hal_walkers --artifact gmm_artifacts.npz --output for_dft.extxyz
Selecting from such files runs on the CPU and stays fast for pools of many thousands of frames.
Sources and checks#
--hal-dirsreads onlyhal_features.npzand the selected frames ofhal_collected.extxyz: no model, prediction or GPU. Directories without frames or without a sidecar are skipped with a warning: a walker that collected nothing writes a sidecar but nohal_collected.extxyz, and a walker run with--feature-gamma-min nonewrites no sidecar.selectrefuses an artifact whose sha256 differs from the one the walkers (orgrace_uq predict --features-gamma-min) used, a--gamma-minbelow their--feature-gamma-min, and sidecars written with different artifacts.- When
select --datasetpredicts itself and the artifact is known (--artifact,--uqor--model-dir), the prediction keeps only the candidate features; otherwise it keeps the features of every atom. With a SavedModel, pass--model saved_model --uq gmm_artifacts.npz. - The output extension selects the format; the selected structures are written without the model predictions.
Pool files: one file from many walkers
--hal-dirs reads the walkers directly, so a pool file is optional. It helps to:
- mix HAL frames with other sources in one selection: a pool and a
grace_uq predict --features-gamma-minfile have the same candidate columnscand_*, so you can join them withpd.concatand run oneselectover all frames; - keep one file to copy or inspect: every frame with all its
infokeys (T, V/V0, step, γ).--gamma-pristine-minkeeps only frames with max γpristine at or above a value.
grace_uq hal --pool hal_walkers --pool-output hal_pool.pkl.gz
grace_uq select --dataset hal_pool.pkl.gz --artifact gmm_artifacts.npz
select reads a pool without a prediction. Write it as .pkl.gz: an extxyz pool loses the
candidate columns. From Python: pool_frames(dirs, gamma_pristine_min=None) in
tensorpotential.uq.hal.
Calibrating the threshold#
There are two ways to set γ = 1:
- Robust fence (default): median + 3·(1.4826·MAD) of the training σ of each cluster. On a well-behaved training set only about 0.1 % of the training atoms read γ > 1, and a heavy outlier tail cannot move the fence.
-
Percentile:
--threshold-percentile Qputs γ = 1 at the Q-th percentile of each cluster's training σ, so at most about (100 − Q) % of the training atoms read γ > 1:grace_uq build ... --threshold-percentile 99γ then means something else and is not comparable with default artifacts;
grace_uq infoprints whichthreshold estimatorwas used. On a heavy tail the percentile follows the tail, so most atoms read γ far below 1.
If many training atoms (10–20 %) read γ > 1 with the default, the σ distribution of a cluster is
a sharp spike with a heavy tail. Without a narrow_sigma warning from grace_uq info, try more
clusters first (--n-clusters with one value); with it, use the percentile.
The narrow_sigma warning (small models)
The threshold is read from a σ histogram. Each bin is about 1 % of σ wide (artifacts built
before October 2026: a fixed width of 0.4). When the σ spread of a cluster is narrower than
about a bin, the threshold is unreliable, usually too high, so γ reads low.
grace_uq build and grace_uq info warn about such clusters. With the 1 % bins this is
rare; old artifacts of small models trained on few distinct environments often have it.
Rebuild such an artifact with the current grace_uq build. If the warning stays, use fewer
clusters (--n-clusters 1; more clusters make σ narrower) or --threshold-percentile 99,
which hardly depends on the bin width.
Python API#
tensorpotential.uq has a Python API for your own scripts and active-learning loops.
GMMUQModel#
Evaluates σ and γ from features in TensorFlow. It works with tf.GradientTape, so it also gives
the gradients that HAL needs.
from tensorpotential.uq import GMMUQModel
uq_model = GMMUQModel("artifacts.npz")
# Evaluate on features
sigma, total_sigma, cluster_assign = uq_model(features, element_indices)
# Gamma (normalized uncertainty) directly from features, using the calibrated
# per-element interpolation thresholds stored in the artifact (from `grace_uq build`).
gamma, sigma = uq_model.gamma_from_features(features, element_indices)
HAL from Python#
run_hal in tensorpotential.uq.hal runs one walker, like grace_uq hal:
from ase.io import read
from tensorpotential.uq.hal import run_hal
res = run_hal(
read("start.extxyz"),
model_dir="my_fit/seed/1", # model.yaml, checkpoint, gmm_artifacts.npz
ensemble="nvt_ramp",
temperature_K=300,
n_steps=2000,
tau=0.5,
output_dir="hal_py",
)
frames = res.frames # collected Atoms, also in hal_py/hal_collected.extxyz
Its keyword arguments follow the HAL options, with these spellings:
uq_artifact (--artifact-path), live_start_artifact (--live-start-artifact-path),
temperature_K (-T), timestep_fs (--timestep, None = auto), tdamp_fs / pdamp_fs
(--tdamp / --pdamp) and pair_overrides, a dict with the content of the --pair-radii file.
It returns a HALResult with frames (the collected Atoms), history (the per-step arrays of
hal_history.npz), summary (the content of hal_summary.json) and live_model.
In the same module, scan_tau(taus, atoms, output_dir=..., **run_hal_kwargs) runs a τ scan
(run_walkers with one walker per τ),
run_walkers(...) runs many walkers, and pool_frames(dirs) pools their
frames (pool files); each returns a DataFrame.
Calculator from model.yaml#
get_gmm_uq_calculator loads a model from model.yaml and its checkpoint, with the CALM artifact
attached:
from tensorpotential.uq.factories import get_gmm_uq_calculator
calc = get_gmm_uq_calculator(
model_yaml="model.yaml",
checkpoint="checkpoint.best_test_loss",
gmm_artifact_path="gmm_artifacts.npz",
)
atoms.calc = calc
atoms.get_potential_energy()
gamma = calc.results["gamma"]
Calculator results#
A UQ calculator puts these keys into calc.results. The default mode gamma_only gives the
first five; calc.enable_uq(mode="full") adds dsigma_dr and virial_sigma.
| Key | Description |
|---|---|
gamma |
per-atom γ. Only when the artifact has calibrated thresholds. |
atomic_sigma |
per-atom Mahalanobis distance σ. |
total_sigma |
sum of σ over the atoms. |
gmm_cluster |
per-atom index of the nearest cluster of the atom's element. |
features |
the basis-RP features. |
dsigma_dr |
gradient of Σγ with respect to the positions, pointing towards higher γ: the HAL bias direction. Without thresholds in the artifact, the gradient of total_sigma. |
virial_sigma |
uncertainty contribution to the virial. |
Advanced: incremental updates#
In active learning (for example HAL) you can update a CALM model with new environments without a full rebuild:
# Assuming uq_model is an instance of GMMUQModel
uq_model.update_one(new_features, new_element_indices)
uq_model.save("updated_artifacts.npz")