Experiment blocks: stopping + external archive¶
Experimental observation and stopping hooks
These experiment-spec hooks are Experimental. They record traces inside a canonical run artifact, but their observation behavior is not an additional stable algorithm API. See Stability and versioning.
Use these blocks within a CLI experiment configuration. A complete NSGA-II/ZDT1 configuration shows both together. Start with two-objective, unconstrained runs: the hook observes objective vectors and does not independently enforce feasibility.
VAMOS 1.2.0: a block whose values cannot be built (for example
window: 1 or pruning: hypervolume) stops the command with exit code 2
before any run, and --validate-config reports it. VAMOS 1.1.0 and earlier
disabled both hooks with a warning and ran the full evaluation budget without
them.
VAMOS 1.2.0: both blocks merge key by key, nested mappings
included: first defaults, then the top-level block, then problems.<name>.
A key set to null replaces the inherited value, so
problems.zdt1.stopping.hv_convergence: {patience: 10} changes only patience.
Problem keys are case-insensitive: ZDT1 is zdt1, and two keys that differ
only in case are an error. Each value is checked when the spec is validated,
before any run; the values that conflict only after merging (such as
min_points below window + 2) are checked when each problem's hooks are
built, also before any run. In VAMOS 1.1.0 a partial problem override replaced
the whole block, so that example turned the stop off and lost ref_point; a
top-level block replaced the defaults one; problems.ZDT1 was accepted but
never applied; and misspelled or out-of-range values ran as something else
(statistic: medain as the median, epsilon_mode: relative as rel,
confidence: 95 as 1).
stopping.hv_convergence¶
Enable HV-based convergence stopping driven by a hypervolume trace sampled during the run.
Configuration fragment:
stopping:
hv_convergence:
enabled: true
every_k: 200
window: 10
patience: 5
epsilon: 1e-4
epsilon_mode: rel # abs|rel
statistic: median # mean|median|min
min_points: 25
confidence: null # e.g. 0.95 to enable bootstrap CI
bootstrap_samples: 300
ref_point: [2.0, 2.0] # two values (the monitor is two-objective), or "auto"
VAMOS 1.2.0: every_k, patience, min_points and
bootstrap_samples are positive integers and window is an integer of at
least 2; epsilon is a finite number >= 0; epsilon_mode is abs or rel
and statistic is mean, median or min; confidence is null or
strictly between 0 and 1; enabled is true or false; and ref_point is
auto or two finite numbers. PyYAML reads a float without a dot, such as
1e-4, as a string, so epsilon and confidence also accept a string that
is a number. In VAMOS 1.1.0 a ref_point without two values made every HV
sample fail silently, and the stop never triggered.
Canonical record:
manifest.jsonoutcome metrics underhooks.stopping, including the bounded trace
Notes:
- This hook computes HV for exactly two objectives. For more objectives it records an unavailable-HV reason and cannot apply HV-convergence stopping; installing another backend does not change this hook's dimensionality guard.
ref_point: "auto"derives a reference point from the observed set. Use an explicit fixed reference point for comparisons across runs. VAMOS 1.2.0: the automatic reference is fixed at the first sample (its nadir plus 10% of its range in each objective) and recorded asreference_pointin the stopping metadata; later points outside it count as zero. VAMOS 1.1.0 and earlier recomputed it from every sample, so the series was not comparable and convergence was declared too early.- The trace uses the hook archive when enabled, otherwise the observed objective
vectors. Sampling occurs at generation/checkpoint boundaries;
every_kis a sampling interval, not a request for extra objective evaluations.
archive.external¶
Enable an objective-space observation archive with explicit pruning
policies. This hook archive is separate from an algorithm's result archive;
enabling it does not change which decisions or objectives result.X and
result.F return.
Configuration fragment:
archive:
external:
enabled: true
capacity: 200
truncate_size: 200
pruning: crowding # crowding|hv|mc_hv|knn|maxmin|ref_dirs
hv_ref_point: null # optional; hv-based policies use a relative reference without it
rng_seed: null # ref_dirs pruning's stream; null derives it from the run seed
objective_tolerance: 0.0
deduplicate_in: objective # the only value the hook archive accepts
decision_tolerance: 1.0e-32
Canonical record:
manifest.jsonoutcome metrics underhooks.archive, including the bounded trace
Each trace row describes one archive update: evals, archive_size after the
update, inserted (rows offered), pruned (rows the update removed, offered or
already archived), split into dominated_or_duplicate and pruned_by_capacity,
and prune_reason (the pruning policy when the capacity removed rows, otherwise
none). total_inserted, total_pruned, total_dominated_or_duplicate and
total_pruned_by_capacity sum the rows over the whole run, not only over the
bounded trace.
VAMOS 1.2.0: the hook offers the archive only the objective rows
that the previous notification did not contain, and skips the update when there
are none. With a nonzero objective_tolerance, it offers the whole population
whenever a row changed, because a row once dropped as a near-duplicate becomes
eligible when the archived point it was close to is dominated. Steady-state algorithms such as MOEA/D notify after every evaluation
with a population that changed in a row or two; VAMOS 1.1.0 re-inserted the
whole population each time, so inserted summed the population size at every
evaluation. pruned used to mix dominated and duplicated rows with capacity
pruning, and prune_reason named the pruning policy whenever any row was
removed; the two new counters separate them.
Notes:
- Set a finite
capacityexplicitly; the hook parser defaults to 200. It does not inherit population size from a tuning space. VAMOS 1.2.0:capacity: nullkeeps every non-dominated objective vector; VAMOS 1.1.0 rejected it. - The hook stores objective vectors, not a decision-aligned result archive.
VAMOS 1.2.0:
deduplicate_inmust therefore beobjective;decisionandbothare rejected when the spec is validated. VAMOS 1.1.0 accepted them and the run aborted with anIndexErrorat the first archive update. - VAMOS 1.2.0:
capacityandtruncate_sizeare positive integers (nottrueor10.7),rng_seedis an integer, the tolerances are finite numbers>= 0andhv_ref_pointis a list of finite numbers. The same rules apply toExternalArchiveConfigin Python, which also warns about a field that the chosen pruning does not use:hv_ref_pointwithout HV pruning,rng_seedwithoutref_dirs, or a pruning other thancrowdingwithout acapacity. VAMOS 1.1.0 truncated10.7to 10, readtrueas 1 and failed at the first duplicate with aNaNtolerance. objective_toleranceis an absolute Chebyshev distance: two objective vectors within that distance count as duplicates and only the first is kept. Set it in objective units, if at all.- VAMOS 1.2.0:
objective_tolerancedefaults to0.0, exact equality, as in jMetal; it was1.0e-10. The same default applies to the SMPSO leader archive and to the algorithm result archives. With objectives on a scale of about1e-9or smaller, the old tolerance merged distinct non-dominated points; at ordinary scales the archive contents are unchanged. pruning: hvuses exact HV contributions in 2D and, whenmoocoreis installed, exact higher-dimensional contributions. Archive pruning support does not remove the two-objective restriction of the convergence monitor.- VAMOS 1.2.0:
mc_hvis an alias ofhv: both compute the exact contributions; VAMOS 1.1.0 documentedmc_hvas a Monte Carlo estimate, but it never was one.hv_ref_pointis optional. Without it, each truncation measures the contributions againstnadir + (nadir - ideal)of the archived points, the reference 2 of the objectives normalized to[0, 1], so the pruning does not depend on the objective units and keeps the extreme points. A givenhv_ref_pointis used as is, in objective units. VAMOS 1.1.0 and earlier used the largest value seen in each objective plus 1, in objective units: with one objective in hundreds, the archive dropped extreme points that it keeps with the objective in units. - VAMOS 1.2.0: without
moocore, HV pruning with two objectives uses the exact contributions too, and with three or more objectives the run fails with aDependencyErrornaming thecomputeextra. VAMOS 1.1.0 and earlier prune by crowding distance instead, with a single warning per process. The HV subset selection oftop_k(method="hv_greedy")is exact with or withoutmoocore; without it, it is slow with three or more objectives. - VAMOS 1.2.0:
pruning: knnapplies the SPEA2 truncation operator: it repeatedly removes the point whose distances to the others, sorted in ascending order, are lexicographically smallest. VAMOS 1.1.0 and earlier rank points by the distance to their second-nearest neighbour alone, with ties broken by row order, which can remove an extreme point. - For an algorithm result archive, use its public configuration builder and
inspect the resolved
result_mode; see Algorithms and backends. VAMOS 1.3.0: a result archive can report a final subset (final_selectionandfinal_size, see Results and cardinality); the hook archive has no final subset and rejects those fields. Since VAMOS 1.0.1, the SMPSO search leaders and optional external result archive are separate, and explicitpopulationmode returns the final swarm. See the changes since 1.0.0. - Since VAMOS 1.0.1, the tuning CLI excludes result-archive controls from its ordinary search and scores the feasible non-dominated final population. See Tuning version differences.
Reproducibility¶
VAMOS 1.3.0: the random streams of these hooks follow the run
seed. Without an rng_seed, the ref_dirs pruning of the archive and the
bootstrap of stopping.hv_convergence.confidence draw from streams derived
from the run seed: children of its SeedSequence that the algorithm's own
generator never draws from, so the optimization does not change. The runs of an
experiment therefore prune and bootstrap with different streams, and a repeated
seed repeats them. An integer rng_seed fixes the stream, as before. The same
rule applies to ExternalArchiveConfig.rng_seed of an algorithm's external
archive, and optimize records the effective seed in the run's resolved
configuration; its requested specification keeps the null rng_seed, so a
saved run tells a derived stream from a fixed one. The monitor bootstraps with
a numpy.random.Generator. In VAMOS 1.2.0 both rng_seed fields defaulted to
0, so every seed of an experiment shared those streams, and the monitor used
Python's random.Random.
Runs should be launched with fixed seeds and fixed budgets. Early stopping changes executed evaluations, but the run still reports the original max budget in the resolved spec. Use the stopping payload and trace in manifest outcome metrics for analysis.