Paired statistics

The statistics of the report.

handoverlab.stats

Paired statistics for comparing handover rules.

Rules compared on the same seed run on the same world and the same radio samples, so the unit of analysis is one run (condition and seed) and two rules are compared through their paired differences. Per-run rates are ratios of small counts with many ties and zeros, which is why a rank test with an exact null distribution is the default.

  • paired_differences and t_interval;
  • bootstrap_ci: percentile bootstrap of the mean paired difference, or of the ratio of the two means (pairs are resampled together);
  • wilcoxon_signed_rank: two-sided Wilcoxon signed-rank test, zero differences dropped, ties with mid-ranks; method="exact" builds the null distribution exactly for the ranks present, "approx" uses the normal approximation with tie and continuity corrections, "auto" is exact up to EXACT_MAX non-zero differences;
  • holm: Holm's step-down adjustment of a family of p-values;
  • paired_summary: all of the above for one comparison;
  • noninferior, seeds_needed: a one-sided margin check and a simple planning aid;
  • rnd: decimal rounding half away from zero for printing, so that a value such as 0.0975 always prints the same way.

Only NumPy and SciPy (for the t and normal quantiles) are used.

WilcoxonResult dataclass

Result of wilcoxon_signed_rank.

paired_differences(a: Sequence[float], b: Sequence[float]) -> np.ndarray

a - b element by element; a and b must be paired (same order of runs).

t_interval(d: Sequence[float], level: float = 0.95) -> Tuple[float, float]

Two-sided t interval of the mean of d ((nan, nan) for fewer than two values).

bootstrap_ci(a: Sequence[float], b: Optional[Sequence[float]] = None, stat: str = 'mean_diff', n_boot: int = 2000, level: float = 0.95, seed: int = 0) -> Tuple[float, float]

Percentile bootstrap interval for paired data.

stat="mean_diff": mean of a - b (or of a alone if b is None). stat="ratio": ratio of the means mean(a) / mean(b). Runs are resampled with replacement as pairs, with a seeded generator, so the interval is reproducible. For the ratio, resamples whose baseline mean is zero are dropped: the interval is conditional on a non-zero baseline mean (NaN if no resample is left).

wilcoxon_signed_rank(d: Sequence[float], method: str = 'auto') -> WilcoxonResult

Two-sided Wilcoxon signed-rank test of the paired differences d.

Differences within TIE_TOL of zero are dropped; tied absolute differences get mid-ranks. method="exact" computes the null distribution of the signed-rank sum exactly for the ranks present (each rank joins the positive sum with probability 1/2), which is valid with ties. method="approx" uses the normal approximation with the tie correction of the variance and a continuity correction of 1/2. method="auto" is exact up to EXACT_MAX non-zero differences.

holm(pvals: Sequence[float]) -> list

Holm-adjusted p-values, in the order given (NaN values are kept and not counted).

paired_summary(a: Sequence[float], b: Sequence[float], level: float = 0.95, n_boot: int = 2000, seed: int = 0, method: str = 'auto') -> Dict[str, Any]

Comparison of a (a rule) with b (the baseline) over paired runs.

Returns the number of pairs, both means, the mean difference a - b with its t interval and bootstrap interval, the relative change of the means in per cent, the bootstrap interval of the ratio of the means, the Wilcoxon p-value, the number of non-zero differences and the test method.

noninferior(d: Sequence[float], margin: float, level: float = 0.95) -> bool

True if the upper end of the one-sided level t interval of mean(d) lies below margin.

Use it for statements such as "failures not higher than the baseline by more than margin" with d = rule minus baseline.

seeds_needed(sd: float, half_width: float, level: float = 0.95) -> int

Number of paired runs for a t interval of the given half-width when the differences have spread sd.

rnd(x: Any, nd: int = 3) -> str

x rounded to nd decimals, half away from zero, as a string ("n/a" for None, NaN and inf).