Studies¶
Study files, tuning with gates, the runner, storage, status and reports.
handoverlab.studies
¶
Studies: equal-budget tuning with gates, paired test stages, stored and resumable results, reports.
Quick start::
from handoverlab.studies import load_study, run_study, write_report
study = load_study("examples/study_smoke.yaml")
run_study(study, workers=2, progress=print) # resumable: re-running skips finished jobs
write_report("results/study-smoke") # report.md, report.html, figures
or on the command line handoverlab study examples/study_smoke.yaml and
handoverlab report results/study-smoke; handoverlab status
results/study-smoke shows what is left and how long it will take. See
.config for the study file, .tuning for the selection rule and
the gates (policies warn, block, off and extend),
.status for the status view, .runner for
jobs and workers, .store for the directory layout and
handoverlab.stats for the statistics.
Study
dataclass
¶
Job
dataclass
¶
One world (condition and seed) with its rules.
rules are dictionaries with name, spec (a rule specification
for make_rule) and optional labels method,
budget and setting. events is the event-log level kept for
every rule run: False (off), True (summary) or "full" (summary
and trace).
reusable(cached: Dict[str, Any]) -> Dict[str, Any]
¶
Stored rule runs this job may reuse (with an event log, if the job keeps event logs).
cost(cached: Dict[str, Any]) -> float
¶
Rough cost for ordering: rule runs still to simulate x users x duration.
payload(cached: Dict[str, Any]) -> Dict[str, Any]
¶
Picklable description for a worker, with the stored results it may reuse.
StudyStore
¶
GateError
¶
StudyError
¶
Bases: RuntimeError
A study cannot continue (failed jobs or an invalid state); the message says why.
load_study(source: Union[str, os.PathLike, Dict[str, Any]]) -> Study
¶
Study from a YAML, TOML or JSON file, or from a dictionary.
study_from_dict(data: Dict[str, Any], base_dir: str = '.') -> Study
¶
Study from a dictionary (see the module text).
write_report(root: str, out: Optional[str] = None, formats: Sequence[str] = ('md', 'html'), figures: bool = True) -> List[str]
¶
Write the report of the study in root; returns the files written.
default_workers(n_cpu: Optional[int] = None) -> int
¶
Default worker count: all logical CPUs but four (half of them on machines with fewer than eight).
run_jobs(jobs: Sequence[Job], root: str, workers: int = 1, cached: Optional[Dict[str, Any]] = None, progress: Optional[Callable[[str], None]] = None) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]
¶
Run the jobs that have no record yet; returns (new records, failures).
With workers > 1 the jobs run in a process pool; a broken pool is
restarted once and its unfinished jobs are tried again.
run_key(scenario_dict: Dict[str, Any], seed: int, rule_key: str, params: Dict[str, Any]) -> str
¶
Content hash of one rule run (see the module text).
read_table(path: str) -> List[Dict[str, Any]]
¶
Rows of a table written by write_table (Parquet or CSV) as dictionaries.
For a .parquet name that does not exist, the CSV of the same name is
read if present.
write_table(rows: Sequence[Dict[str, Any]], path: str, columns: Optional[Sequence[str]] = None) -> str
¶
Write rows as Parquet (.parquet name and pyarrow installed) or CSV; returns the path written.
A .parquet name falls back to the same name with .csv when
pyarrow is missing.
dry_run(study: Study) -> List[str]
¶
What the study would run, without running anything.
run_study(study: Study, out: Optional[str] = None, workers: Optional[int] = None, progress: Optional[Callable[[str], None]] = None, stop_after: Optional[str] = None) -> StudyResult
¶
Run (or resume) all stages of study in out; returns a StudyResult.
stop_after="tuning" stops after the gates. Raises GateError
if a blocking gate fails and StudyError if jobs failed (their
tracebacks are in failed/; re-running retries them).
load_study_dir(root: str) -> Study
¶
The study stored in a study directory (from study.lock.json).
status_data(study: Study, out: Optional[str] = None, workers: Optional[int] = None) -> Dict[str, Any]
¶
Structured status (see the module text): stages, gates, sessions and the time estimate.
study_status(study: Union[str, Study], out: Optional[str] = None, workers: Optional[int] = None) -> List[str]
¶
Lines with the planned, finished, failed and remaining jobs of every stage and the time estimate.
first_round(method: str, grid: Dict[str, Sequence], n: int) -> List[Dict[str, Any]]
¶
n settings drawn at random from grid without repetition (seeded by the method name).
refine(method: str, grid: Dict[str, Sequence], best: Dict[str, Any], taken: Sequence[Dict[str, Any]], n: int, tag: str = '') -> List[Dict[str, Any]]
¶
n new settings: one grid step around best, then two steps, then random ones.
The random fill keeps the budget equal for every method when the
neighbourhood of the best setting is smaller than n.
select(cands: Sequence[Dict[str, float]], budget: float, tolerance: float = 0.05, objective: str = 'unnecessary_pm', constraint: str = 'failures_pm', tie_breaks: Sequence[str] = ('prepared_cells_mean', 'failures_pm')) -> Selection
¶
Apply the selection rule to candidates with mean metrics (see the module text).
handoverlab.studies.config
¶
Study files: conditions, seeds, methods with tuning grids, tuning, comparisons.
A study file (YAML, TOML or JSON) describes a whole comparison::
name: a3-vs-cho
scenario: {base: smoke} # built-in name, {base: ..., ...} or a full scenario
conditions: # full factorial over dotted scenario keys (optional)
layout.site_spacing_m: [100.0, 150.0]
seeds:
tuning: [901, 902] # never used in a test
test: {first: 1, last: 10}
tuning:
settings: 12 # evaluated settings per method: the same for every method
random: 8 # of which drawn at random; the rest refine around the best
reference: {rule: a3, name: A3-ref, off_db: 3.0, ttt_s: 0.16}
budgets: {standard: 1.0, strict: 0.5} # shares of the reference failure rate
group_by: [layout.site_spacing_m] # tune per value of these condition keys (default: all keys)
gates: {grid_edge: block} # warn | block | off per gate (grid_edge also: extend)
methods:
A3:
rule: a3
grid: {off_db: [0, 1, 2, 3, 4, 6, 8], ttt_s: [0.0, 0.04, 0.08, 0.16, 0.32]}
natural_bounds: {ttt_s: lower} # 0 s is a floor, not an edge of the search
CHO:
rule: cho
fixed: {n_max: 4}
grid: {o_exec_db: [0, 1, 2, 3, 4, 6], ttt_exec_s: [0.0, 0.04, 0.08, 0.16]}
compare:
baseline: A3-ref # a method, a tuned method as METHOD@BUDGET, or the reference
metrics: [unnecessary_pm, failures_pm, handovers_pm]
output: results/a3-vs-cho
A method without a grid is not tuned and runs as given. Every key is
checked; an unknown key raises ConfigError.
Study
dataclass
¶
canonical(obj: Any) -> str
¶
Canonical JSON text (sorted keys, no spaces) used for content hashes.
label(cond: Dict[str, Any]) -> str
¶
Short text of a condition, such as layout.site_spacing_m=100; all for none.
study_from_dict(data: Dict[str, Any], base_dir: str = '.') -> Study
¶
Study from a dictionary (see the module text).
load_study(source: Union[str, os.PathLike, Dict[str, Any]]) -> Study
¶
Study from a YAML, TOML or JSON file, or from a dictionary.
handoverlab.studies.tuning
¶
Tuning of handover rules on an equal budget, the selection rule and the gates.
Every tuned method evaluates the same number of settings from its declared
grid (its budget): first random settings drawn without repetition with
a generator seeded by the method name, then settings one grid step around the
best of the first round, then two steps, and, if the neighbourhood is
exhausted, further random settings, until the budget is spent. All settings
run on the tuning seeds, which never appear among the test seeds.
Selection rule (fixed before any result is seen)
The failure budget is a share of the failure rate of a reference rule on
the tuning runs. Among the settings whose failure rate does not exceed
the budget, those within tolerance (5 %) of the lowest
unnecessary-handover rate count as equal; among them the lowest
preparation overhead wins, then the lowest failure rate, then the lowest
unnecessary-handover rate, then the order of evaluation. If no setting
meets the budget, the setting with the lowest failure rate is used and
flagged.
Gates (checked after tuning, before any test run)
grid_edge the selected value of a numeric parameter is the
smallest or largest of its grid, and that end is not
declared a natural bound;
budget_binding more than binding_share of the settings of every
method meet the failure budget, or the reference rule
has fewer than min_reference_failures failures:
the budget does not constrain the trade-off;
budget_met no setting of a method meets the budget;
equal_budget methods evaluated different numbers of settings
(except where a grid is smaller than the budget);
seed_overlap a tuning seed is also a test seed (checked before
anything runs);
flat_optimum more than flat_share of the settings within the
budget lie within the tolerance of the best: the
choice is made by the tie-breaks.
Each gate has a policy: block (the study stops before the test stage
with a message that names the cause), warn (reported in the gate report
and in every table) or off. grid_edge also accepts extend: the
grid of the parameter is widened at the offending end by extend_steps
values with the spacing of the last two values (extension_values),
the methods are tuned again with the same budget, and this repeats up to
max_extensions times; each extension is recorded as a dated amendment.
If the cap is reached, or the grid cannot be widened (for example below
zero), the gate blocks.
Selection
dataclass
¶
Result of the selection rule for one method and budget.
MethodTuning
dataclass
¶
Everything that tuning found for one method in one group.
setting_key(cfg: Dict[str, Any]) -> str
¶
Canonical text of a setting (for duplicates and names).
first_round(method: str, grid: Dict[str, Sequence], n: int) -> List[Dict[str, Any]]
¶
n settings drawn at random from grid without repetition (seeded by the method name).
refine(method: str, grid: Dict[str, Sequence], best: Dict[str, Any], taken: Sequence[Dict[str, Any]], n: int, tag: str = '') -> List[Dict[str, Any]]
¶
n new settings: one grid step around best, then two steps, then random ones.
The random fill keeps the budget equal for every method when the
neighbourhood of the best setting is smaller than n.
select(cands: Sequence[Dict[str, float]], budget: float, tolerance: float = 0.05, objective: str = 'unnecessary_pm', constraint: str = 'failures_pm', tie_breaks: Sequence[str] = ('prepared_cells_mean', 'failures_pm')) -> Selection
¶
Apply the selection rule to candidates with mean metrics (see the module text).
standard_values_gate(methods: Sequence[Any], reference: Dict[str, Any], policy: str) -> GateResult
¶
Gate standard_values (since 2.1.0): every grid value, fixed parameter and the reference rule use values
that TS 38.331 allows (handoverlab.standard), e.g. a time-to-trigger of the TimeToTrigger set.
evaluate_gates(group: str, methods: Sequence[MethodTuning], budgets: Dict[str, float], ref_failures: float, policy: Dict[str, str], binding_share: float = 0.9, min_reference_failures: int = 5, flat_share: float = 0.5, tolerance: float = 0.05, objective: str = 'unnecessary_pm', constraint: str = 'failures_pm') -> List[GateResult]
¶
Gates grid_edge, budget_binding, budget_met, equal_budget and flat_optimum of one group.
ref_failures is the number of failures of the reference rule over all
tuning runs of the group.
extension_values(values: Sequence[Any], side: str, steps: int) -> List[Any]
¶
steps new grid values beyond the side end ("lower" or "upper") with the spacing of the last two.
Integer grids stay integer; float values are rounded to 10 decimals. A grid whose values are all non-negative is not extended below zero.
summary_lines(gates: Sequence[GateResult]) -> List[str]
¶
Human-readable lines for the gates that did not pass.
handoverlab.studies.study
¶
Running a study: tuning rounds, gates, the test stage and the result tables.
Stages
tune1 every condition x tuning seed: the reference rule and the first
(random) settings of every tuned method;
tune2 the same worlds with the refinement settings around the best
setting of round 1 of each tuning group (if the budget is not
spent yet);
(gates) selection per method and failure budget, then the gates of
handoverlab.studies.tuning; a blocking gate stops the
study here with GateError;
test every condition x test seed: the reference, every tuned method
with its selected setting per budget (METHOD@BUDGET) and
every untuned method.
Every stage is resumable: re-running the study skips the jobs whose records
exist and reuses stored rule runs. After the stages the tables
tables/runs and tables/tuning are rebuilt from the job records.
StudyError
¶
Bases: RuntimeError
A study cannot continue (failed jobs or an invalid state); the message says why.
GateError
¶
plan_round2(study: Study, store: StudyStore, round1_jobs: Sequence[Job]) -> List[Job]
¶
Refinement settings around the best setting of round 1 (first budget), per tuning group.
analyse_tuning(study: Study, store: StudyStore, jobs: Sequence[Job]) -> Dict[str, Any]
¶
Selection and gates of every tuning group; writes tuned/<group>.json and tuned/gates.json.
write_gate_report(study: Study, store: StudyStore, gates: Sequence[GateResult]) -> None
¶
Write tuned/gates.json (every gate, the blocking ones, the warnings, the selection rule).
load_amendments(store: StudyStore) -> List[Dict[str, Any]]
¶
Recorded grid extensions of a study directory (one entry per extension round, oldest first).
apply_amendments(study: Study, rounds: Sequence[Dict[str, Any]]) -> Study
¶
A copy of study whose grids include every recorded extension.
plan_extension(study: Study, gates: Sequence[GateResult], steps: int)
¶
Grid extensions for failed grid_edge gates; returns (changes, problems).
One change per method, parameter and side, with the values to add (checked
by constructing the rule with each of them). problems names the edges
that cannot be extended.
job_events(level: str)
¶
Event-log level as stored in a job (False, True or "full"; keeps older job IDs valid).
run_rows(study: Study, records: Sequence[Dict[str, Any]]) -> List[Dict[str, Any]]
¶
One flat row per job and rule.
run_study(study: Study, out: Optional[str] = None, workers: Optional[int] = None, progress: Optional[Callable[[str], None]] = None, stop_after: Optional[str] = None) -> StudyResult
¶
Run (or resume) all stages of study in out; returns a StudyResult.
stop_after="tuning" stops after the gates. Raises GateError
if a blocking gate fails and StudyError if jobs failed (their
tracebacks are in failed/; re-running retries them).
dry_run(study: Study) -> List[str]
¶
What the study would run, without running anything.
handoverlab.studies.status
¶
Status of a study: finished, failed and remaining jobs per stage, and the time still needed.
handoverlab status STUDY (a study file or a study directory) prints, for
every stage that can be planned now, the planned, finished, failed and
remaining jobs and the rule runs that still have to be simulated. Stages
that depend on earlier results (tune2 after tune1, test after the
gates) are counted from the study file until they can be planned.
Time estimate From the finished jobs of the study: the mean time to build a world and the time per simulated rule run, scaled by users x simulated seconds of each job. The remaining CPU time is spread over the workers (default: the worker count of the last session) and is never shorter than the longest remaining job. It is a rough guide: jobs of other conditions or rules can take longer or shorter, and a machine that is busy with other work is slower.
load_study_dir(root: str) -> Study
¶
The study stored in a study directory (from study.lock.json).
resolve(target: Union[str, Study], out: Optional[str] = None)
¶
(study, study directory) for a study file, a study directory or a Study.
measured_rates(records: Sequence[Dict[str, Any]]) -> Optional[Dict[str, float]]
¶
Mean world-build time and engine time per rule run and user-second from finished job records.
status_data(study: Study, out: Optional[str] = None, workers: Optional[int] = None) -> Dict[str, Any]
¶
Structured status (see the module text): stages, gates, sessions and the time estimate.
study_status(study: Union[str, Study], out: Optional[str] = None, workers: Optional[int] = None) -> List[str]
¶
Lines with the planned, finished, failed and remaining jobs of every stage and the time estimate.
handoverlab.studies.runner
¶
Jobs, content hashes and their execution in a pool of worker processes.
A job is one world (a condition and a seed) with a list of rules. Its ID
is a content hash: the SHA-256 of the canonical JSON of the resolved
scenario, the seed, the rule names and the run keys of its rules. A run
key identifies one rule run: the hash of the scenario, the seed, the rule's
registry key and its full parameter set (as reported by params()), and
MODEL_VERSION. Names do not enter a run key, because they do not
change a result.
Resumption and reuse A job whose record exists is skipped. Before a job runs, every rule whose run key is already stored in the study (in any stage) is taken from the stored record instead of being simulated (rules on one world do not influence each other, which the rule contract checks), and identical rules within a job run once. If nothing is left to simulate, the world is not even built.
Workers
One job per process (processes are started with "spawn" on every
platform); BLAS and OpenMP threads are limited to one per process. The default number of workers leaves cores free
(default_workers). Jobs are started longest first. Worker
count and order never change a result: every job is deterministic.
Job
dataclass
¶
One world (condition and seed) with its rules.
rules are dictionaries with name, spec (a rule specification
for make_rule) and optional labels method,
budget and setting. events is the event-log level kept for
every rule run: False (off), True (summary) or "full" (summary
and trace).
reusable(cached: Dict[str, Any]) -> Dict[str, Any]
¶
Stored rule runs this job may reuse (with an event log, if the job keeps event logs).
cost(cached: Dict[str, Any]) -> float
¶
Rough cost for ordering: rule runs still to simulate x users x duration.
payload(cached: Dict[str, Any]) -> Dict[str, Any]
¶
Picklable description for a worker, with the stored results it may reuse.
default_workers(n_cpu: Optional[int] = None) -> int
¶
Default worker count: all logical CPUs but four (half of them on machines with fewer than eight).
scenario_hash_dict(scenario_dict: Dict[str, Any]) -> Dict[str, Any]
¶
Scenario as it enters a hash: everything but its name.
run_key(scenario_dict: Dict[str, Any], seed: int, rule_key: str, params: Dict[str, Any]) -> str
¶
Content hash of one rule run (see the module text).
peak_memory_mb() -> Optional[float]
¶
Peak resident memory of this process so far [MB], or None where it cannot be read.
execute(payload: Dict[str, Any], root: str) -> Dict[str, Any]
¶
Run one job and write its record atomically (the function a worker process calls).
Returns the record, or {"job": id, "error": traceback} if the job failed
(the traceback is also written to failed/<job>.txt).
run_jobs(jobs: Sequence[Job], root: str, workers: int = 1, cached: Optional[Dict[str, Any]] = None, progress: Optional[Callable[[str], None]] = None) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]
¶
Run the jobs that have no record yet; returns (new records, failures).
With workers > 1 the jobs run in a process pool; a broken pool is
restarted once and its unfinished jobs are tried again.
handoverlab.studies.store
¶
Result storage of a study: job records, tables and the directory layout.
::
<output>/
manifest.json versions, platform, start and end times, worker count
study.lock.json the resolved study (every value explicit) and its SHA-256
jobs/<stage>/<job>.json one record per job (atomic write: complete or absent)
failed/<job>.txt traceback of a failed job (removed when it succeeds)
events/<run>.parquet event log of one rule run (test stage, with events: summary or full)
events/<run>-trace.parquet procedure trace of one rule run (events: full)
tuned/<group>.json all evaluated settings, the selection and the gates of a tuning group
tuned/gates.json the gate report of the whole study
tables/runs.parquet one row per job and rule (CSV without pyarrow)
tables/tuning.parquet one row per evaluated setting and tuning group
report/ written by handoverlab report
A job record is the unit of resumption: a job whose record exists is not run
again. Tables are derived from the records and can be rebuilt at any time.
Parquet needs the optional package pyarrow (pip install
handoverlab[parquet]); without it tables are written as CSV and read back
with their types.
StudyStore
¶
atomic_write_json(path: str, obj: Any) -> None
¶
Write JSON to a temporary file and rename it, so a reader sees all or nothing.
write_table(rows: Sequence[Dict[str, Any]], path: str, columns: Optional[Sequence[str]] = None) -> str
¶
Write rows as Parquet (.parquet name and pyarrow installed) or CSV; returns the path written.
A .parquet name falls back to the same name with .csv when
pyarrow is missing.
read_table(path: str) -> List[Dict[str, Any]]
¶
Rows of a table written by write_table (Parquet or CSV) as dictionaries.
For a .parquet name that does not exist, the CSV of the same name is
read if present.
table_path(base: str) -> str
¶
Existing table file for base (without extension): Parquet first, then CSV.
handoverlab.studies.report
¶
Reports of a study: Markdown and HTML with tables and figures.
write_report reads a study directory (the job records of the current
plan, the tuning files and the manifest) and writes report.md,
report.html (self-contained, figures embedded), summary.json (every
number of the report) and figures/*.png into <study>/report or
another directory. Figures need matplotlib (pip install
handoverlab[report]); without it the report has tables only.
Content provenance (versions, sessions, jobs); tuning per group (budgets, selected settings, gates, a figure of every evaluated setting); test results (means with t intervals per rule, pooled and per condition); paired comparisons with the baseline per metric: mean difference with t and bootstrap intervals, relative change, bootstrap interval of the ratio of means, Wilcoxon signed-rank p-value and Holm-adjusted p-value.
Multiplicity: the pooled comparisons of one metric form one Holm family; the per-condition comparisons of one metric (all rules and conditions) form another.
Doc
¶
A small document model rendered as Markdown and as HTML.
test_values(records: Sequence[Dict[str, Any]], metric: str) -> Dict[str, Dict[Tuple[str, int], float]]
¶
rule -> {(condition label, seed): value} of the test stage.
analyse_test(data: Dict[str, Any]) -> Dict[str, Any]
¶
Means and paired comparisons of the test stage (every number of the report).
tuning_figure(gd: Dict[str, Any], objective: str, constraint: str) -> bytes
¶
One panel per method: every evaluated setting (constraint vs objective), budgets, selected settings.
comparison_figure(res: Dict[str, Any], metrics: Sequence[str]) -> bytes
¶
Mean paired difference to the baseline with bootstrap intervals, one panel per metric.
write_report(root: str, out: Optional[str] = None, formats: Sequence[str] = ('md', 'html'), figures: bool = True) -> List[str]
¶
Write the report of the study in root; returns the files written.