Verification

The checks of handoverlab verify.

handoverlab.verify

Verification of the simulator: handoverlab verify.

Check groups (all run by default, in this order):

pathloss TR 38.901 path loss, breakpoint and LOS probability against hand values radio statistics of the radio model against its parameters (shadowing, LOS share, time variation, L1 error, L3 filter) calibration large-scale calibration of TR 38.901 clause 7.8.1 against the 21 companies of 3GPP R1-165974 (coupling loss, SINR and SIR CDFs of UMa, UMi and the indoor office at 6 and 30 GHz) invariants built-in rules in debug mode, conservation of counts, determinism, independence of rules equivalences CHO with immediate execution = A3; a plug-in rule that reduces to A3 = A3 fingerprints golden event-log fingerprints of short reference runs extended beams, several carriers, TS 38.133 periods, DAPS and LTM against TS 38.133 and published 3GPP evaluations (since 1.10.0) highway the TR 37.885 highway cards: geometry, wrap-around, coupling gain against an independent computation, LOS-only links (since 1.11.0) values RRC parameter values of the built-in cards and rules against TS 38.331 V18.11.0 (since 2.1.0) fading the optional fast fading (TR 38.901 TDL models) and the link-to-system mapping (since 2.2.0)

Optional:

learning the learning rule a3_bandit against an exhaustive search over its arms (--only learning; about 20 minutes; since 1.11.0) ntn NTN link budget, antenna and orbit against TR 38.821 / TR 38.811 (--only ntn; since 2.0.0) ns3 reference comparison with stored ns-3 / 5G-LENA results (--ns3; minutes) stored recompute (a sample of) a stored result file and compare fingerprints (--results FILE)

run_verify returns a Report; report.ok is True if no check failed.

Check dataclass

Outcome of one check: PASS, FAIL or NOTE (information that cannot fail).

run_verify(groups: Optional[Sequence[str]] = None, full: bool = False, ns3: bool = False, ns3_seeds: int = 1, results: Optional[str] = None, sample: Optional[int] = None, printer: Optional[Callable[[str], None]] = print, ns3_reference: Optional[str] = None) -> Report

Run the check groups groups (default: all of GROUPS) and return the report.

full uses three seeds for the radio statistics and 60 s runs for the engine checks instead of one seed and 30 s.

handoverlab.verify.calibration

Large-scale calibration of the radio model against 3GPP (TR 38.901 clause 7.8.1), since 1.8.0.

TR 38.901 clause 7.8.1 defines a drop-based calibration (Table 7.8-1): UMa, UMi-Street Canyon and the indoor office (open office), three sectors per site with the antenna element of Table 7.3-1 in a vertical array of ten elements tilted by DFT weights (Eq. 7.3-1), no fast fading, users attached to the sector with the highest received power, and two metrics, the coupling loss to the serving sector and the geometry (wideband SINR with and without noise). 21 companies reported their CDFs in the 3GPP contribution R1-165974 (RAN1 #85, "E-mail discussion summary of the large scale calibration"); data/r1_165974.csv holds their mean and their range at every percentile.

This module reproduces the drop with the propagation models of HandoverLab (the formulas of TR 38.901 Tables 7.4.1-1 and 7.4.2-1, written here for per-link user heights and, for UMa, the random effective environment height of note 1), the deployment of Tables 7.2-1 and 7.2-2, the O2I building penetration loss of clause 7.4.3.1 (50 % low loss, 50 % high loss) and geographical wrap-around of the 19-site layouts, and compares the CDFs with the 3GPP results.

Versions: the formulas are those of TR 38.901 V19.5.0, except the IRR-glass loss of Table 7.4.3-1. The companies of R1-165974 (May 2016) worked with TR 38.900 V14.0.0, before the Release 19 change of that value; the calibration keeps the pre-Release-19 IRR-glass loss 23 + 0.3 f dB (the value of TR 38.901 V17.0.0, IRR_GLASS_R1_165974), while V19.5.0 has 25.4 + 0.11 f dB (Table 7.4.3-1 note 2: updated in the Release 19 study on 7-24 GHz; IRR_GLASS_V19, which o2i_loss_db takes on request). With the V19.5.0 value the 30 GHz cases leave the company range, the 6 GHz cases do not. It checks the radio model of HandoverLab -- path loss, LOS probability, shadowing, penetration loss -- with the deployment and antenna that 3GPP fixed for calibration; the handover engine itself uses its own antenna abstraction (radio.bs_gain_db) and is not part of this check.

CalScenario dataclass

One calibration case of TR 38.901 Table 7.8-1.

key: str property

Sheet name of R1-165974, e.g. UMa-6GHz.

tx_power_dbm: float property

Table 7.8-1: 44 dBm UMi, 49 dBm UMa at 6 GHz; 35 dBm at 30 and 70 GHz; 24 dBm indoor.

bandwidth_hz: float property

Table 7.8-1: 20 MHz at 6 GHz, 100 MHz at 30 and 70 GHz.

Comparison dataclass

max_dev_db: float property

Largest distance from the industry mean over the compared percentiles [dB].

array_gain_db(theta_deg, tilt_deg: float, m: int = N_ELEMENTS)

Gain of the vertical array with the weights of Eq. 7.3-1 (d_V = lambda / 2), relative to one element.

pathloss_db(name: str, los, d2, d3, fc, hbs, hut, he=1.0)

Path loss of Table 7.4.1-1 for UMa (with effective height he), UMi and InH-Office.

p_los(name: str, d2out, hut)

LOS probability of Table 7.4.2-1: UMa (with C'(h_UT)), UMi, indoor open office.

uma_effective_height(rng, d2, hut)

h_E of note 1 of Table 7.4.1-1.

1 m with probability 1 / (1 + C(d2D, h_UT)), otherwise uniform over 12, 15, ..., h_UT - 1.5 m.

o2i_loss_db(rng, n: int, fc: float, irr_glass: Tuple[float, float] = IRR_GLASS_R1_165974) -> Tuple[np.ndarray, np.ndarray]

Building penetration loss of clause 7.4.3.1 for n indoor users (half low loss, half high loss).

Returns the loss PL_tw + PL_in + N(0, sigma_P) [dB] and d2D-in [m] (minimum of two uniform(0, 25 m)). irr_glass = (a, b) of the IRR-glass loss a + b f (default: the value of R1-165974, see the module text).

hex_sites(isd: float) -> np.ndarray

The 19 sites of a two-ring hexagonal layout (Table 7.2-1), centre site at the origin.

wrap_offsets(isd: float) -> np.ndarray

Translations of the 19-site cluster for geographical wrap-around (the cluster and its six images).

indoor_sites() -> np.ndarray

The 12 ceiling sites of TR 38.901 Figure 7.2-1 in a 120 m x 50 m hall centred on the origin.

simulate_drops(sc: CalScenario, n_users: int = 6000, seed: int = 1, irr_glass: Tuple[float, float] = IRR_GLASS_R1_165974) -> Dict[str, np.ndarray]

Coupling gain (= -coupling loss) to the serving sector, SINR and SIR [dB] of n_users dropped users (irr_glass: see o2i_loss_db).

load_reference(path: str = DATA) -> Dict[str, Dict[str, np.ndarray]]

Mean, minimum and maximum over the companies of R1-165974, per sheet and metric, at 0..100 %.

compare(sc: CalScenario, result: Dict[str, np.ndarray], ref=None, percentiles: Sequence[int] = PERCENTILES) -> List[Comparison]

Our percentiles against the industry mean and range of R1-165974 for the three metrics.

calibration_checks(rep, n_users: int = 10000, seed: int = 1, tol_db: float = 0.5) -> None

Section calibration of handoverlab verify: within the company range of R1-165974 (+- tol_db).

handoverlab.verify.ns3

Reference comparison with stored ns-3 / 5G-LENA results.

data/ns3_urban_a3.json holds results of the packet-level simulator ns-3 with its 5G-LENA module for stock Event-A3 handover in the scenario ns3-urban-3.5ghz (see its scenario card): 24 A3 settings (offset 1-6 dB x time-to-trigger 100-1000 ms) on four conditions (site distance 200 / 400 m x 16 / 30 m/s). HandoverLab runs the same settings and conditions; seeds are different realisations in the two simulators, so only averages over the conditions are compared, setting by setting.

Acceptance limits (version 0.2):

  • median over the settings of the ratio HandoverLab / ns-3 of the handover rate and of the unnecessary-handover rate between 0.8 and 1.25;
  • rank correlation (Spearman) over the settings at least 0.9 for both.

Failure rates and the mean serving SINR are reported without a limit: the two simulators define the load of interfering cells differently, and the failure model of HandoverLab is not calibrated in absolute terms.

The comparison is not part of the quick checks: it simulates 4 x 24 runs of 150 s (a few minutes per seed on one core).

car_loss(speed_mps: float) -> Dict

Overrides that reproduce the in-car loss ns-3 applies at speed_mps.

load_reference(path: Optional[str] = None) -> dict

The stored reference, or the file path in the same format (benchmarks/ns3 writes one; since 2.1.0).

scenario_for(isd_m: int, speed_mps: int, ref: Optional[dict] = None, overrides: Optional[Dict] = None, car: bool = False)

ns3-urban-3.5ghz with the sites of isd_m and constant speed speed_mps (8, 16 or 30).

overrides (dotted keys, such as {"protocol.qout_window_s": 0.12}) change the scenario further, for calibration experiments. car adds the in-car loss that ns-3 applies at this speed (car_loss).

compare(seeds: Sequence[int] = (500,), progress: Optional[Callable[[str], None]] = None, overrides: Optional[Dict] = None, car: bool = False, reference: Optional[str] = None) -> Dict

Run part A of the reference in HandoverLab and return the per-setting table and the summary.

overrides change the scenario of every condition and car adds the in-car loss of ns-3 (see scenario_for); reference is another reference file (load_reference).

holdout(seeds: Sequence[int] = (500,), progress: Optional[Callable[[str], None]] = None, overrides: Optional[Dict] = None, car: bool = False, reference: Optional[str] = None) -> Dict

Run the held-back parts B and C of the reference (see the module text) and compare them cell by cell.

Part B: two A3 settings on six conditions; part C: one setting with one parameter changed per cell (users, preparation delay, T310, site distance, run length, speed). overrides change every scenario and car adds the in-car loss of ns-3, as in compare; tune only on part A.

handoverlab.verify.golden

Golden event-log fingerprints of short reference runs.

data/golden.json stores, for a few short runs per scenario and rule, the fingerprint of the event log (see handoverlab.eventlog), the number of events and some counts, together with the Python, NumPy and SciPy versions and the platform that produced them.

On the same NumPy and SciPy build the fingerprints must be identical: any change of a result shows up here. A change that alters them on purpose needs a new reference (python -m handoverlab.verify.golden --write) and an entry in the changelog. On another build or platform the random streams and floating-point sums may differ in the last bits and move single events, so there the check reports whether the fingerprints are identical and only requires the counts to agree within a tolerance.

cases() -> Dict[str, Dict[str, Any]]

name -> scenario, seed and rules of every golden run.

legacy(scenario)

The 1.x version of a golden scenario: LEGACY_1X, the old values of its card (LEGACY_CARDS, since 2.1.0), and the 1.x urban profile where 2.0 uses v60.

same_build(ref_versions: Dict[str, str]) -> bool

True if NumPy, SciPy and the operating system match those of the reference.

handoverlab.verify.highway

The highway of TR 37.885 (since 1.11.0): handoverlab verify --only highway.

Checks of the two highway cards highway-tr37885-4ghz (below 6 GHz: sites along the road, TR 36.885 clause A.1.3 option 1) and highway-tr37885-30ghz (above 6 GHz: 19 hexagonal sites, option 2):

H1 geometry and link budget equal the values of TR 37.885 V15.3.0, TR 36.885 V14.0.0 and TR 38.901 V19.5.0 (exact); H2 with wrap-around the static received power of every site is continuous where a vehicle re-enters the road: the largest jump there is below 0.5 dB or below the 99.9 % point of the steps elsewhere (since 2.1.0: the shadowing field of TR 38.901 Eq. 7.4-5 is not smooth, so that ordinary steps exceed 0.5 dB); H3 the coupling gain to the strongest site (percentiles 5 to 95) agrees within 0.5 dB with an independent computation: uniform drops on the lanes, the sites and their wrap-around images built from the TR geometry (for the hexagonal layout with the 19-site functions of the calibration module, in the frame of the grid, not turned), the TR 38.901 path loss written out again here (RMa LOS) or taken from the calibration module (UMa LOS), and Gaussian shadowing with the card's standard deviation and the inter-site correlation of the simulator (since 2.1.0 none: TR 38.901 clause 7.5 step 4, which TR 37.885 clause 6.2.2 refers to for the shadowing); H4 with radio.los_model: los every link has the LOS path loss.

TR 37.885 has no published calibration of the base-station-to-vehicle link of the highway (its calibration work is for the sidelink), so H3 checks that the TR assumptions are implemented, not a company result. The plan with these criteria was written before the first run. Since 2.1.0 H3 simulates the shadowing along each vehicle's path (radio.place_share: 0; the same marginal distribution, uncorrelated between sites) instead of the place-tied field: that field has the exponential autocorrelation of TR 38.901 Eq. (7.4-5), whose longer tail leaves few independent samples per road, and with 10 seeds the percentiles of the simulation moved by up to 0.8 dB from one block of seeds to the next. The place-tied field is checked by the radio statistics (group radio) and the unit tests.

rma_los_db(d2, h_bs, h_ut, fc_ghz, h=5.0)

RMa LOS of TR 38.901 Table 7.4.1-1 with the breakpoint of note 5 (written out again for the check).

reference_gain(name: str, sc, n: int, rng) -> np.ndarray

Coupling gain to the strongest site [dB] of n independent drops (see H3).

simulated_gain(sc, seeds, every_s: float = 1.0) -> np.ndarray

Coupling gain to the strongest site [dB] seen by the vehicles of the card, every every_s seconds.

handoverlab.verify.learning

The learning rule a3_bandit against an exhaustive search (since 1.11.0): handoverlab verify --only learning.

The bandit learns, per cell, which of its A3 offsets (arms) earns the highest reward per outcome (useful +1, unnecessary -w_unnecessary, early, late or a link failure -w_failure). Its own objective can also be measured for every arm used as a fixed A3 offset: a bandit with that single arm is A3 with that offset (checked: identical event log), and its record gives the reward per outcome. The arm with the highest pooled reward, b*, is what an exhaustive search (grid search) over the arms finds.

Set-up and criteria, fixed in a plan before the first run: the smoke card with 900 s per run (5 s warm-up), seeds 1, 2 and 3 (the same worlds for every rule), the rule's default arms (1 to 5 dB), TTT 160 ms, epsilon 0.1, epoch 10 and weights 1 and 2.

B1 in the second half of the epochs the share of epochs played on arms within one arm of b* is at least 0.6 (if the best two arms differ by less than the standard error over the seeds, on the arms within one standard error of the best); B2 the bandit's reward per outcome over the whole run is at least the median of the fixed arms; B3 its reward per outcome in the second half of the run is at least best - 0.5 x (best - worst) of the fixed arms.