DCT Laboratory — Volume II, Chapter 15
Enterprise Digital Twins and Autonomous Enterprise Transformation
This laboratory demonstrates
Seed 26215 · Companion to Chapter 15 and AXIOM Lab 2.15 (module AXIOM-15) · Mirrored in DCT_V2_Ch15_Lab.xlsx
This laboratory reproduces Worked Example 15.1, Can a Digital Twin Replace the Engineering Office's Count?, with the book's declarations, and works the five steps of AXIOM Lab 2.15: synchronize, decide, price readings, monitor, and set the envelope. The law, the costs and the hiring rule are Chapter 7's; the new declarations are the skills survey's error and calendar, the twin, its monitor and threshold, the audit on an alarm, and a drift to test the monitor. The seeds are the book's (Online Appendix Remark 15.B.1): 20,000 design paths with seed 1500, 100,000 fresh paths with seed 1501, the program of Figure 15.1(a) seed 1503, the monitor's calibration seed 1505, the drift seed 1507, and the exact filter's check seed 1509. The lab's own seed, 26215, draws only new illustrations: Panel 2's independent paths and Panel 5's ten programs. The book's engine takes about seven and a half minutes; these cells reproduce its numbers in about seventy seconds on two cores, because they draw the same numbers one quarter at a time and search for each design's best target in a window around the book's (Panel 2 says how). Every Meridian number below is the book's (Table 15.3, Figures 15.1–15.3, Online Appendix Example 15.B.2); the final cell checks them. Money is in $ million.
The declarations (Worked Example 15.1, Step 1; Online Appendix Remark 15.B.1)
Chapter 7's model, unchanged. Workforce capability moves as : is the points hired and trained in a quarter, at $4 million a point and at most 2.5 a quarter, and the engineering office's swings, 1.5, 0.75, 0, or points lost with chances 0.05, 0.20, 0.50, 0.20 and 0.05 (variance ). A point short of the floor of 55 for a quarter costs $1.6 million, so a quarter costs , and the board discounts at a quarter. Chapter 7's rule brings planned capability to a target: 55.75 on the engineering office's count, at $230.755 million of cost from 58 (Worked Example 7.1).
The twin (new declarations). The chief operating officer proposes to retire the count, a manual assessment by team leads, and to run the rule on a twin fed by the skills survey. The program starts from an audited 58. The survey runs twice a year, at the start of quarters 2, 4, 6, …, and reads , its error normal with a standard deviation of 2.5 points (95 percent of readings within ; Volume I, Table 5.2). The twin knows the law and its own hires but not the swings. It keeps its estimate by the Kalman filter (Equation 15.1) with , the swings' mean 0 and variance , and a variance of 0 at the audited start, and it hires for a target of planned capability on its estimate. The board's alternatives: the survey read as exact, each reading carried forward by ; a quarterly survey; a survey twice as sharp, 1.25 points; both; and no readings after the audit. Costs are present values over 480 quarters, and shares below the floor are taken over quarters 40–479. Each design's target is chosen on 20,000 design paths and valued on 100,000 fresh ones, every design on the same draws of swings and survey errors (common random numbers, Proposition 15.2). The twin's envelope is Chapter 14's guardrail: never plan below 55 unless hiring at the cap.
The monitor and the drift. The monitor is a one-sided cumulative sum (Equation 15.2) of the surveys' standardized innovations , with reference and the least threshold, on a grid of 0.25 from 1.5, that raises at most one false alarm in 50 surveys. On an alarm the engineering office audits capability: the estimate is set to it with variance 0, the sum restarts, and, once a drift has begun, the law's mean is re-estimated as the drift and the target raised by it. The drift that tests the monitor, a tighter labor market: from the start of quarter 8 each quarter loses an extra half point.
Python Code
import math
import time
import numpy as np
from scipy.stats import norm
import matplotlib.pyplot as plt
plt.rcParams.update({"figure.dpi": 110, "axes.spines.top": False, "axes.spines.right": False})
T_START = time.time()
SEED = 26215 # the lab's seed: new draws only (Panel 2's independent paths, Panel 5's ten programs)
SEED_DESIGN = 1500 # the book's seeds (Online Appendix Remark 15.B.1): the 20,000 design paths,
SEED_FRESH = 1501 # the 100,000 fresh paths,
SEED_PATH = 1503 # the program of Figure 15.1(a),
SEED_CAL = 1505 # the monitor's calibration, 20,000 paths without a drift,
SEED_DRIFT = 1507 # the drift, 100,000 paths,
SEED_EXACT = 1509 # the exact filter's check, 300 programs of 40 quarters
# Chapter 7's declared model, unchanged ($ million, points of capability, quarters)
BETA, ALPHA, K, CAP, FLOOR, P_SHORT = 1.1 ** -0.25, 0.975, 4.0, 2.5, 55.0, 1.6
W = np.array([1.5, 0.75, 0.0, -0.75, -1.5]) # swings: points lost in a quarter
PW = np.array([0.05, 0.20, 0.50, 0.20, 0.05]) # their chances
Q = float((PW * W ** 2).sum()) # a swing's variance, 0.45
C0, T7, CH7 = 58.0, 55.75, 230.755 # the audited start; Chapter 7's target and its cost from 58 (WE 7.1)
# the twin, its readings and its monitor (new declarations, Chapter 15)
SD_SURVEY = 2.5 # the survey's error: 95% of readings within +/-5 points
HORIZON, LONG0 = 480, 40 # quarters simulated; long-run shares over quarters 40-479
DISC = BETA ** np.arange(HORIZON)
N_DESIGN, N_FRESH = 20000, 100000
KAPPA, DRIFT_START, DRIFT = 0.5, 8, 0.5
def schedule(every, horizon=HORIZON):
# survey quarters: the start of quarters every, 2 every, ... (none at the audited start)
s = np.zeros(horizon, dtype=bool)
if every:
s[every::every] = True
return s
def kalman_path(every, sd, horizon=HORIZON, p0=0.0):
# the twin's error variance by quarter (Equation 15.1 in one dimension): prior a^2 P + q, gain P-/(P- + r) at a
# survey, posterior (1 - K) P-; it does not depend on the readings
sv, r = schedule(every, horizon), sd ** 2
prior, post, gain = np.zeros(horizon), np.zeros(horizon), np.zeros(horizon)
p = p0
for t in range(horizon):
pm = p if t == 0 else ALPHA ** 2 * p + Q
k = (pm / (pm + r) if r > 0 else 1.0) if sv[t] else 0.0
prior[t], gain[t] = pm, k
p = (1 - k) * pm
post[t] = p
return prior, post, gain
def reset_path(every, sd, horizon=HORIZON):
# the survey read as exact and carried forward by the hires and the decay: its error variance by quarter
sv, v, p = schedule(every, horizon), np.zeros(horizon), 0.0
for t in range(horizon):
p = p if t == 0 else ALPHA ** 2 * p + Q
if sv[t]:
p = sd ** 2
v[t] = p
return v
def fixed_point(every, sd):
# Theorem 15.1(iii): the one positive fixed point of the every-quarter map, the variance just after a survey
r = sd ** 2
x = r
for _ in range(10000):
pm = x
for _ in range(every):
pm = ALPHA ** 2 * pm + Q
x2 = pm * r / (pm + r)
if abs(x2 - x) < 1e-14:
break
x = x2
return x
def stream(seed, n, horizon=HORIZON):
# the engine's draws(n, seed): rng.random((horizon, n)) for the swings, then rng.standard_normal((horizon, n)) for the
# survey errors. Generated a quarter at a time from two copies of the generator, the second advanced past the
# uniforms: the same numbers, in a quarter's memory instead of a whole program's
ru = np.random.default_rng(seed)
rn = np.random.default_rng(seed)
rn.bit_generator.advance(horizon * n)
cum = np.cumsum(PW)
for t in range(horizon):
yield t, W[np.minimum(np.searchsorted(cum, ru.random(n), side="right"), len(W) - 1)], rn.standard_normal(n)
class Program:
# Target rules run on common draws, one quarter at a time: the engine's simulate(), transcribed. est is 'exact' (the
# engineering office's count), 'twin' (the Kalman filter), 'reset' (the survey read as exact) or 'none' (no readings
# after the audit); one row of paths per target, on the first n paths of the draws. drift = (start, size): from that
# quarter each quarter loses size points more. monitor = (kappa, threshold or one per row): on an alarm the audit sets
# the estimate to the truth with variance 0, the sum restarts, and once the drift has begun the law's mean is
# re-estimated as the drift. oracle: the twin (or the count) is told of the drift when it begins.
def __init__(self, targets, est, every=0, sd=0.0, n=N_DESIGN, drift=None, monitor=None, oracle=False,
horizon=HORIZON, limits=False, keep=None, trace=0, sample=8):
self.T = np.asarray(targets, dtype=float)[:, None]
m = len(self.T)
self.est, self.sd, self.r, self.n, self.drift, self.oracle = est, sd, sd ** 2, n, drift, oracle
self.mon = None if monitor is None else (monitor[0], np.resize(np.asarray(monitor[1], float), m)[:, None])
self.sv = schedule(every, horizon) if est in ("twin", "reset") else np.zeros(horizon, dtype=bool)
self.c, self.ch = np.full((m, n), C0), np.full((m, n), C0)
self.p = np.zeros((m, n)) if monitor is not None else 0.0 # without audits, the same on every path
self.shift = np.zeros((m, n)) if (monitor is not None or oracle) else 0.0 # the extra loss the twin believes
self.G = np.zeros((m, n)) if monitor is not None else None
self.cost, self.below = np.zeros((m, n)), np.zeros((m, n))
self.alarms, self.first = np.zeros((m, n), dtype=np.int32), np.full((m, n), -1, dtype=np.int32)
self.limits, self.keep, self.trace, self.sample = limits, keep, trace, sample
self.at0, self.atcap, self.breaks = np.zeros(m), np.zeros(m), np.zeros(m)
self.kept, self.log, self.zrec, self.zprev = [], [], [], None
def step(self, t, sw, eps):
sw, eps = sw[:self.n], eps[:self.n]
drift_now = self.drift is not None and t >= self.drift[0]
z = None
if self.sv[t]:
z = self.c + self.sd * eps[None, :] # the survey's reading
if self.est == "twin":
pm = self.p
k = pm / (pm + self.r) # the gain
innov = z - self.ch # the innovation
if self.mon is not None:
zs = innov / np.sqrt(pm + self.r) # standardized
self.G = np.maximum(0.0, self.G - zs - self.mon[0])
hit = self.G > self.mon[1]
self.zrec.append((t, zs[0, :self.sample].copy(), self.G[0, :self.sample].copy(), hit[0, :self.sample].copy()))
if hit.any(): # the audit
self.alarms += hit
self.first = np.where(hit & (self.first < 0) & (t >= (self.drift[0] if self.drift else 0)), t,
self.first)
self.ch = np.where(hit, self.c, self.ch + k * innov)
self.p = np.where(hit, 0.0, (1 - k) * pm)
if drift_now:
self.shift = np.where(hit, self.drift[1], self.shift)
self.G = np.where(hit, 0.0, self.G)
else:
self.ch = self.ch + k * innov
self.p = (1 - k) * pm
else:
if self.trace == "z": # sums of z, z^2 and z times the previous survey's z
zz = innov[0] / math.sqrt(pm + self.r)
self.zrec.append((t, float(zz.sum()), float((zz * zz).sum()),
None if self.zprev is None else float((zz * self.zprev).sum()), len(zz)))
self.zprev = zz
self.ch = self.ch + k * innov
self.p = (1 - k) * pm
elif self.est == "reset":
self.ch = z
if self.est == "exact":
self.ch = self.c
if self.oracle and drift_now:
self.shift[:] = self.drift[1]
# decide: bring planned capability, as estimated, to the target plus the extra loss the twin believes in
h = np.clip(self.T + self.shift - ALPHA * self.ch, 0.0, CAP)
self.cost += DISC[t] * (K * h + P_SHORT * np.maximum(FLOOR - self.c, 0.0))
if t >= LONG0:
self.below += self.c < FLOOR
if self.limits:
if t >= LONG0:
self.at0 += (h == 0.0).sum(axis=1)
self.atcap += (h == CAP).sum(axis=1)
self.breaks += ((ALPHA * self.ch + h - self.shift < FLOOR - 1e-9) & (h < CAP)).sum(axis=1)
if self.keep and t < self.keep[1]:
self.kept.append(self.c[0, :self.keep[0]].mean())
if self.trace and self.trace != "z":
pp = self.p if np.isscalar(self.p) else self.p[0, :self.trace].copy()
self.log.append(dict(c=self.c[0, :self.trace].copy(), ch=self.ch[0, :self.trace].copy(), p=pp,
z=None if z is None else z[0, :self.trace].copy(), h=h[0, :self.trace].copy(),
G=None if self.G is None else self.G[0, :self.trace].copy()))
# move: the truth by the swings (and the drift), the twin by its law
wt = sw[None, :] + (self.drift[1] if drift_now else 0.0)
self.c = ALPHA * self.c + h - wt
self.ch = ALPHA * self.ch + h - self.shift
if self.est == "twin":
self.p = ALPHA * ALPHA * self.p + Q
def mean(self):
return self.cost.mean(axis=1)
def share_below(self):
return (self.below / (HORIZON - LONG0)).mean(axis=1)
def run(programs, seed, n, horizon=HORIZON):
# one pass over the book's draws for a seed (n paths, the engine's order); each program reads its first paths
for t, sw, eps in stream(seed, n, horizon):
for pr in programs:
pr.step(t, sw, eps)
return programs
print(f"beta = {BETA:.6f} a quarter (10 percent a year); a swing's variance q = {Q:.2f}, standard deviation "
f"{math.sqrt(Q):.3f} points")
print(f"the survey's error variance r = {SD_SURVEY ** 2:.2f}; surveys at the start of quarters "
f"{', '.join(str(int(x)) for x in np.nonzero(schedule(2, 12))[0])}, ...; quarterly: "
f"{', '.join(str(int(x)) for x in np.nonzero(schedule(1, 6))[0])}, ...")
print(f"{HORIZON} quarters ({HORIZON // 4} years) of present value; long-run shares over quarters {LONG0}-{HORIZON - 1}; "
f"{N_DESIGN:,} design paths and {N_FRESH:,} fresh paths")beta = 0.976454 a quarter (10 percent a year); a swing's variance q = 0.45, standard deviation 0.671 points the survey's error variance r = 6.25; surveys at the start of quarters 2, 4, 6, 8, 10, ...; quarterly: 1, 2, 3, 4, 5, ... 480 quarters (120 years) of present value; long-run shares over quarters 40-479; 20,000 design paths and 100,000 fresh paths
Panel 1 — Synchronize: the error the twin settles at (Theorem 15.1(iii)–(iv); Proposition 15.1; Worked Example 15.1, Step 2; Figure 15.1; Online Appendix Example 15.B.2: the twin's error 1.333 points just after a survey, 1.463 a quarter later and 1.576 just before the next, gain 0.284; a quarterly survey 1.169–1.323, gain 0.219; twice as sharp 0.888–1.261; both 0.791–1.022; the survey read as exact 2.500–2.528; no readings a limit of 3.019; the exact posterior's mean within 0.012 points of the twin's in all 5,700 surveys, mean difference 0.0013, its standard deviation within 0.011)
AXIOM Lab 2.15, step 1. The twin's error variance follows the filter's recursion, which does not involve the readings: it rises by the law between surveys, , and each survey lowers it to with the gain . Theorem 15.1(iii): just after a survey it converges, from any start, to the one positive fixed point of the two-quarter map; part (iv): with no readings it tends to . Proposition 15.1(i): the survey and the law's prediction combine by their precisions. Then a check that the best linear estimate is, here, as good as the best estimate: the exact posterior for the swings' discrete law, computed on a grid of 0.01 points from 40 to 72 with the mass split linearly between grid points, on 300 programs of 40 quarters run by the twin at its best target, 56.70 (seed 1509). Last, Figure 15.1: one program from the audited 58 (seed 1503), and the error by quarter for each design.
Python Code
def errors(every, sd):
# just after a survey, a quarter later (twice a year), just before the next, and the steady gain
fp = fixed_point(every, sd)
before2 = ALPHA ** (2 * every) * fp + Q * sum(ALPHA ** (2 * j) for j in range(every))
return dict(after=math.sqrt(fp), mid=math.sqrt(ALPHA ** 2 * fp + Q) if every == 2 else None,
before=math.sqrt(before2), gain=before2 / (before2 + sd ** 2), fp=fp)
ERR = {"twin": errors(2, SD_SURVEY), "twin_quarterly": errors(1, SD_SURVEY), "twin_sharp": errors(2, SD_SURVEY / 2),
"twin_quarterly_sharp": errors(1, SD_SURVEY / 2)}
ERR["reset"] = dict(after=SD_SURVEY, before=math.sqrt(ALPHA ** 2 * SD_SURVEY ** 2 + Q))
ERR["none"] = dict(limit=math.sqrt(Q / (1 - ALPHA ** 2)))
names = {"twin": "twin, survey twice a year", "twin_quarterly": "twin, quarterly survey", "twin_sharp":
"twin, survey twice as sharp", "twin_quarterly_sharp": "twin, both", "reset": "survey read as exact"}
print("error of the estimate (sd, points) just after a quarter later just before gain")
for key, lab in names.items():
e = ERR[key]
mid = f"{e['mid']:15.3f}" if e.get("mid") else " " * 15
gain = f"{e['gain']:7.3f}" if "gain" in e else ""
print(f" {lab:34s} {e['after']:10.3f} {mid} {e['before']:12.3f} {gain}")
print(f" {'no readings after the audit':34s} tends to sqrt(q/(1 - a^2)) = {ERR['none']['limit']:.3f}")
_, post2, gain2 = kalman_path(2, SD_SURVEY)
_, post2b, _ = kalman_path(2, SD_SURVEY, p0=10.0)
CONVERGE = max(abs(post2[-2] - ERR["twin"]["fp"]), abs(post2b[-2] - ERR["twin"]["fp"]))
print("from the audit (variance 0), just after the surveys of quarters 2, 4, 6, 8, 12, 20: "
+ ", ".join(f"{math.sqrt(post2[q]):.3f}" for q in (2, 4, 6, 8, 12, 20))
+ f"; from a variance of 10 the same fixed point (gap {CONVERGE:.0e}); last gain {gain2[-2]:.4f}")
pm_ = ERR["twin"]["before"] ** 2
PRECISION = 1 / (1 / pm_ + 1 / SD_SURVEY ** 2)
print(f"Proposition 15.1(i): just before a survey the law says {pm_:.4f} (precision {1 / pm_:.4f}) and the survey "
f"{SD_SURVEY ** 2:.2f} ({1 / SD_SURVEY ** 2:.4f}); precisions add: 1/({1 / pm_:.4f} + {1 / SD_SURVEY ** 2:.4f}) = "
f"{PRECISION:.4f} = {ERR['twin']['after']:.3f}^2")
print(f"Theorem 15.1(iv): a twin that stopped reading after the audit settles at {ERR['none']['limit']:.2f} points; the "
f"survey read as exact is good to {ERR['reset']['after']:.2f}-{ERR['reset']['before']:.2f}")
def exact_filter_check(target, n_paths=300, quarters=40, seed=SEED_EXACT, dx=0.01, lo=40.0, hi=72.0):
# the exact posterior on a grid (Bayes' rule; Theorem 15.1(i)) against the Kalman filter, on the same programs
rng = np.random.default_rng(seed)
xs = np.arange(lo, hi + dx / 2, dx)
nx, cum = len(xs), np.cumsum(PW)
diffs, sdd = [], []
for _ in range(n_paths):
c, ch, p = C0, C0, 0.0
post = np.zeros(nx)
post[int(round((C0 - lo) / dx))] = 1.0
for t in range(quarters):
if t > 0 and t % 2 == 0:
z = c + SD_SURVEY * rng.standard_normal()
pm = p
k = pm / (pm + SD_SURVEY ** 2)
ch, p = ch + k * (z - ch), (1 - k) * pm
post = post * np.exp(-0.5 * ((z - xs) / SD_SURVEY) ** 2)
post /= post.sum()
mu = float((post * xs).sum())
diffs.append(ch - mu)
sdd.append(math.sqrt(p) - float(np.sqrt((post * (xs - mu) ** 2).sum())))
h = min(max(target - ALPHA * ch, 0.0), CAP)
w = W[min(int(np.searchsorted(cum, rng.random(), side="right")), len(W) - 1)]
c, ch, p = ALPHA * c + h - w, ALPHA * ch + h, ALPHA * ALPHA * p + Q
new = np.zeros(nx) # predict the grid posterior: 0.975x + h, less each swing
for wm, pw in zip(W, PW):
pos = (np.clip(ALPHA * xs + h - wm, lo, hi) - lo) / dx
j = np.minimum(np.floor(pos).astype(int), nx - 2)
f = pos - j
new += np.bincount(j, pw * post * (1 - f), nx) + np.bincount(j + 1, pw * post * f, nx)
post = new
d, s = np.abs(np.array(diffs)), np.array(sdd)
return dict(max_abs=float(d.max()), mean_abs=float(d.mean()), sd_max=float(np.abs(s).max()), n=len(d))
t0 = time.time()
EXACT = exact_filter_check(56.70)
print(f"\nthe exact posterior against the Kalman filter, {EXACT['n']:,} survey updates on 300 programs (seed {SEED_EXACT}): "
f"means within {EXACT['max_abs']:.4f} points (mean difference {EXACT['mean_abs']:.4f}); standard deviations within "
f"{EXACT['sd_max']:.4f} ({time.time() - t0:.0f} s)")
# Figure 15.1: one program from the audited 58 (seed 1503) and the error by quarter
one = run([Program([56.70], "twin", 2, SD_SURVEY, n=1, horizon=25, trace=1)], SEED_PATH, 1, horizon=25)[0]
cpath = np.array([x["c"][0] for x in one.log])
twin_est = np.array([x["ch"][0] for x in one.log])
twin_sd = np.sqrt(np.array([x["p"] for x in one.log], dtype=float))
reads = [None if x["z"] is None else float(x["z"][0]) for x in one.log]
hires = np.clip(56.70 - ALPHA * twin_est, 0.0, CAP)
as_exact = np.zeros(25)
as_exact[0] = C0
for t in range(1, 25): # the survey read as exact, on the same readings and hires
as_exact[t] = ALPHA * as_exact[t - 1] + hires[t - 1]
if reads[t] is not None:
as_exact[t] = reads[t]
print(f"Figure 15.1(a), the program of seed {SEED_PATH}: capability {cpath.min():.2f} to {cpath.max():.2f}; the twin's "
f"largest error {np.abs(twin_est - cpath).max():.2f}, the survey read as exact {np.abs(as_exact - cpath).max():.2f}")
CURVES = dict(none=np.sqrt(kalman_path(0, SD_SURVEY)[1][:25]), reset2=np.sqrt(reset_path(2, SD_SURVEY)[:25]),
twin2=np.sqrt(post2[:25]), twin1=np.sqrt(kalman_path(1, SD_SURVEY)[1][:25]),
sharp2=np.sqrt(kalman_path(2, SD_SURVEY / 2)[1][:25]))
fig, (ax, bx) = plt.subplots(1, 2, figsize=(7.6, 3.6))
q_ = np.arange(25)
ax.fill_between(q_, twin_est - 2 * twin_sd, twin_est + 2 * twin_sd, color="#1F4E79", alpha=0.15, lw=0)
ax.plot(q_, cpath, color="#333333", lw=1.3, label="capability")
ax.plot(q_, twin_est, color="#1F4E79", lw=1.3, label="twin (2 sd band)")
ax.plot(q_, as_exact, color="#E65100", lw=1.1, ls="--", label="survey as exact")
zt = [t for t in range(25) if reads[t] is not None]
ax.plot(zt, [reads[t] for t in zt], "o", ms=3.5, mfc="white", mec="#E65100", mew=1.0, label="survey reading")
ax.axhline(FLOOR, color="#999999", lw=0.8, ls=":")
ax.set(xlim=(0, 24), ylim=(40, 63), xticks=[0, 4, 8, 12, 16, 20, 24], xlabel="quarter", ylabel="capability (points)")
ax.set_title("(a) one program from an audited 58", fontsize=9)
ax.legend(frameon=False, fontsize=7.5, loc="lower left")
for key, col, ls, lab in (("none", "#999999", ":", "no readings"), ("reset2", "#E65100", "--", "survey as exact"),
("twin2", "#1F4E79", "-", "twin, twice a year"), ("twin1", "#2E7D32", "-.", "twin, quarterly"),
("sharp2", "#6A1B9A", (0, (2, 1)), "twin, sharper survey")):
bx.step(q_, CURVES[key], where="post", color=col, ls=ls, lw=1.3, label=lab)
bx.set(xlim=(0, 24), ylim=(0, 5), xticks=[0, 4, 8, 12, 16, 20, 24], xlabel="quarter", ylabel="error of the estimate (sd, points)")
bx.set_title("(b) the estimate's error by quarter", fontsize=9)
bx.legend(frameon=False, fontsize=7.5, loc="upper right")
fig.suptitle("The twin settles at 1.33 to 1.58 points (book seed 1503)", fontsize=10.5)
plt.tight_layout(); plt.show()error of the estimate (sd, points) just after a quarter later just before gain twin, survey twice a year 1.333 1.463 1.576 0.284 twin, quarterly survey 1.169 1.323 0.219 twin, survey twice as sharp 0.888 1.095 1.261 0.504 twin, both 0.791 1.022 0.401 survey read as exact 2.500 2.528 no readings after the audit tends to sqrt(q/(1 - a^2)) = 3.019 from the audit (variance 0), just after the surveys of quarters 2, 4, 6, 8, 12, 20: 0.877, 1.121, 1.234, 1.287, 1.323, 1.333; from a variance of 10 the same fixed point (gap 1e-14); last gain 0.2844 Proposition 15.1(i): just before a survey the law says 2.4842 (precision 0.4025) and the survey 6.25 (0.1600); precisions add: 1/(0.4025 + 0.1600) = 1.7776 = 1.333^2 Theorem 15.1(iv): a twin that stopped reading after the audit settles at 3.02 points; the survey read as exact is good to 2.50-2.53 the exact posterior against the Kalman filter, 5,700 survey updates on 300 programs (seed 1509): means within 0.0119 points (mean difference 0.0013); standard deviations within 0.0111 (4 s) Figure 15.1(a), the program of seed 1503: capability 53.48 to 58.55; the twin's largest error 3.22, the survey read as exact 7.27

Panel 2 — Decide on the twin: a margin sized to its error (Theorem 15.2(iii); Proposition 15.2; Worked Example 15.1, Step 3; Table 15.3; Figure 15.2(a); Online Appendix Example 15.B.2: best targets 55.75 on the count, 56.70 on the twin, 57.35 on the survey read as exact, 57.80 with no readings; from 58 on the fresh paths $230.844m on the count (standard error 0.040), $241.944m on the twin (0.042), $11.101m more, the survey read as exact $15.950m more, no readings $25.433m more (0.063); Chapter 7's target on the twin $16.313m more and 31.3% of quarters below the floor, on the survey read as exact $27.509m and 38.6%, a target of 60.75 $34.396m and 0.2%; on the design paths 241.852 and 230.747)
AXIOM Lab 2.15, step 2. Each design runs Chapter 7's target rule on its estimate. The book searched every target on a grid of 0.05 from 55 to 61 (to 62 with no readings) on the 20,000 design paths (seed 1500) and kept the least; here each design searches seven targets, 0.15 either side of the book's best, on the same paths, which reproduces the book's design-path costs exactly and confirms the best target locally (it lies inside the window, not at its edge). For the count, the twin, the survey read as exact and no readings, a coarser grid of 0.25 across the whole range draws Figure 15.2(a) and shows that the window holds the least cost. Each best target is then valued on the 100,000 fresh paths (seed 1501), with Chapter 7's 55.75 on the twin and on the survey read as exact, and 60.75, the survey's whole error guarded, on the survey. Theorem 15.2(iii) explains the margins, and Proposition 15.2 why the choice is made on design paths and valued on fresh ones, all on common random numbers. The cell takes about forty seconds.
Python Code
DESIGNS = [("exact", "exact", 0, 0.0), ("twin", "twin", 2, SD_SURVEY), ("reset", "reset", 2, SD_SURVEY),
("twin_quarterly", "twin", 1, SD_SURVEY), ("twin_sharp", "twin", 2, SD_SURVEY / 2),
("twin_quarterly_sharp", "twin", 1, SD_SURVEY / 2), ("none", "none", 0, 0.0)]
LABEL = {"exact": "count (Chapter 7)", "twin": "twin, survey twice a year", "reset": "survey read as exact",
"twin_quarterly": "twin, quarterly survey", "twin_sharp": "twin, twice as sharp", "twin_quarterly_sharp":
"twin, both", "none": "no readings after the audit"}
CENTRE = {"exact": 55.75, "twin": 56.70, "reset": 57.35, "twin_quarterly": 56.50, "twin_sharp": 56.40,
"twin_quarterly_sharp": 56.20, "none": 57.80} # the book's best targets: the windows' centres
CURVED = ("exact", "twin", "reset", "none") # drawn across the whole range in Figure 15.2(a)
t0 = time.time()
DES = {}
for key, est, every, sd in DESIGNS:
win = np.round(CENTRE[key] + 0.05 * np.arange(-3, 4), 2)
grid = np.round(np.arange(55.0, (62.0 if key == "none" else 61.0) + 1e-9, 0.25), 2) if key in CURVED else np.array([])
tg = np.unique(np.concatenate([win, grid]))
DES[key] = dict(est=est, every=every, sd=sd, targets=tg, window=win,
prog=Program(tg, est, every, sd, n=N_DESIGN))
run([d["prog"] for d in DES.values()], SEED_DESIGN, N_DESIGN)
for key, d in DES.items():
mean = d["prog"].mean()
iw = np.isin(d["targets"], d["window"])
jw = int(np.argmin(np.where(iw, mean, np.inf)))
d.update(mean=mean, best=float(d["targets"][jw]), design=float(mean[jw]),
interior=bool(d["window"][0] < d["targets"][jw] < d["window"][-1]),
grid_in_window=bool(d["targets"][int(np.argmin(mean))] in d["window"]) if key in CURVED else None,
costs=d["prog"].cost[jw].copy())
print(f"design paths, {N_DESIGN:,} (seed {SEED_DESIGN}): {time.time() - t0:.0f} s")
print("design window searched best margin design-path cost inside the window")
for key, d in DES.items():
print(f" {LABEL[key]:30s} {d['window'][0]:.2f} to {d['window'][-1]:.2f} {d['best']:8.2f} {d['best'] - FLOOR:7.2f} "
f"{d['design']:15.3f} {d['interior']}" + ("" if d["grid_in_window"] is None else
f"; the 0.25 grid's least cost is in it: {d['grid_in_window']}"))
# valued on the fresh paths, with the board's fixed targets
t0 = time.time()
FR = {key: Program([d["best"]] + ([T7] if key == "twin" else [T7, T7 + 5.0] if key == "reset" else []), d["est"],
d["every"], d["sd"], n=N_FRESH, limits=True) for key, d in DES.items()}
run(list(FR.values()), SEED_FRESH, N_FRESH)
COUNT = float(FR["exact"].mean()[0])
for key, pr in FR.items():
d = DES[key]
d.update(cost=float(pr.mean()[0]), se=float(pr.cost[0].std() / math.sqrt(N_FRESH)), below=float(pr.share_below()[0]),
at0=pr.at0[0] / (N_FRESH * (HORIZON - LONG0)), atcap=pr.atcap[0] / (N_FRESH * (HORIZON - LONG0)),
breaks=int(pr.breaks.sum()))
d["excess"] = d["cost"] - COUNT
FIXED = {}
for name, key, row in (("twin at Chapter 7's 55.75", "twin", 1), ("survey as exact at Chapter 7's 55.75", "reset", 1),
("survey as exact at 60.75, the whole error guarded", "reset", 2)):
pr = FR[key]
FIXED[name] = dict(target=float(pr.T[row, 0]), cost=float(pr.mean()[row]), below=float(pr.share_below()[row]),
excess=float(pr.mean()[row]) - COUNT)
print(f"\nfresh paths, {N_FRESH:,} (seed {SEED_FRESH}): {time.time() - t0:.0f} s")
print("design target cost from 58 (se) above the count below floor decisions at 0 / at the cap")
for key, d in DES.items():
print(f" {LABEL[key]:30s} {d['best']:6.2f} {d['cost']:10.3f} ({d['se']:.3f}) {d['excess']:13.3f} {100 * d['below']:12.1f}% "
f"{100 * d['at0']:13.1f}% / {100 * d['atcap']:.1f}%")
print("the board's fixed targets target cost from 58 above the count below floor")
for name, f in FIXED.items():
print(f" {name:50s} {f['target']:6.2f} {f['cost']:12.3f} {f['excess']:15.3f} {100 * f['below']:12.1f}%")
print(f"the twin costs {DES['twin']['excess']:.2f} more than the count and {DES['reset']['excess'] - DES['twin']['excess']:.2f} "
f"less than the survey read as exact (the abstract's $4.8m); the count is worth "
f"{4 * (1 - BETA) * DES['twin']['excess']:.3f} a year against the twin (Panel 3)")
# Theorem 15.2(iii): the margin is the (1 - k/b)-quantile of the swing less the twin's error
k_hold, b_short = K * (1 - ALPHA * BETA), P_SHORT * BETA
RATIO = k_hold / b_short
Z = float(norm.ppf(1 - RATIO))
cdf = np.cumsum(PW[::-1])[::-1] # P(w >= w_j), swings from the largest loss
QUANT = float(W[::-1][np.argmax(np.cumsum(PW[::-1]) >= 1 - RATIO)])
print(f"\nTheorem 15.2(iii): holding a planned point costs k = 4(1 - 0.975 beta) = {k_hold:.5f} a quarter; a point short next "
f"quarter b = 1.6 beta = {b_short:.4f}; critical ratio k/b = {RATIO:.4f}")
print(f" on the count the margin is the swing's {1 - RATIO:.4f}-quantile: {QUANT:.2f} (P(w > 0.75) = 0.05 <= {RATIO:.4f} <= "
f"P(w >= 0.75) = 0.25), Chapter 7's margin")
print(f" with normal swings and errors, z = {Z:.4f} and the margin z sqrt(q + (0.975 s)^2): the twin's planned level is off "
f"by 0.975 times its error s, so the margin is z times the error a quarter later")
MARGINS = {}
for key, s_list in (("exact", [0.0]), ("twin", [ERR["twin"]["after"], ERR["twin"]["mid"]]),
("twin_quarterly", [ERR["twin_quarterly"]["after"]]),
("twin_sharp", [ERR["twin_sharp"]["after"], math.sqrt(ALPHA ** 2 * ERR["twin_sharp"]["fp"] + Q)]),
("twin_quarterly_sharp", [ERR["twin_quarterly_sharp"]["after"]]),
("reset", [ERR["reset"]["after"], ERR["reset"]["before"]]), ("none", [ERR["none"]["limit"]])):
MARGINS[key] = [Z * math.sqrt(Q + (ALPHA * s) ** 2) for s in s_list]
lim = DES[key]["at0"] + DES[key]["atcap"]
print(f" {LABEL[key]:30s} normal margin {' to '.join(f'{x:.2f}' for x in MARGINS[key]):12s} simulated best "
f"{DES[key]['best'] - FLOOR:.2f}; a hiring limit binds in {100 * lim:4.1f}% of quarters")
print(" the formula holds one quarter at a time; it sizes the count's and the twins' margins, where the limits seldom bind,")
print(" but not the survey read as exact, whose readings swing hires to 0 or to the cap in over half the quarters, nor no")
print(" readings, whose error grows from 0 after the audit")
# Proposition 15.2: choose on design paths, value on fresh ones; common random numbers
print(f"\nProposition 15.2(i): the twin's best target costs {DES['twin']['design']:.3f} on the design paths where it was chosen "
f"and {DES['twin']['cost']:.3f} on fresh paths; the count {DES['exact']['design']:.3f} and {DES['exact']['cost']:.3f}")
indep = run([Program([56.75], "twin", 2, SD_SURVEY, n=N_DESIGN)], SEED, N_DESIGN)[0]
tw = DES["twin"]
j75 = int(np.nonzero(tw["targets"] == 56.75)[0][0])
common = tw["prog"].cost[j75] - tw["costs"]
indiff = indep.cost[0] - tw["costs"]
CRN = dict(diff=float(common.mean()), sd_common=float(common.std()), sd_indep=float(indiff.std()))
print(f"Proposition 15.2(ii): 56.75 against 56.70 on the twin, path by path: on the common design paths the difference has "
f"mean {CRN['diff']:.4f} and standard deviation {CRN['sd_common']:.3f}; against new independent paths (the lab's seed "
f"{SEED}) {CRN['sd_indep']:.3f}: common numbers need {(CRN['sd_indep'] / CRN['sd_common']) ** 2:,.0f} times fewer paths "
f"to tell the targets apart")design paths, 20,000 (seed 1500): 26 s design window searched best margin design-path cost inside the window count (Chapter 7) 55.60 to 55.90 55.75 0.75 230.747 True; the 0.25 grid's least cost is in it: True twin, survey twice a year 56.55 to 56.85 56.70 1.70 241.852 True; the 0.25 grid's least cost is in it: True survey read as exact 57.20 to 57.50 57.35 2.35 246.696 True; the 0.25 grid's least cost is in it: True twin, quarterly survey 56.35 to 56.65 56.50 1.50 239.371 True twin, twice as sharp 56.25 to 56.55 56.40 1.40 237.761 True twin, both 56.05 to 56.35 56.20 1.20 235.668 True no readings after the audit 57.65 to 57.95 57.80 2.80 256.145 True; the 0.25 grid's least cost is in it: True fresh paths, 100,000 (seed 1501): 12 s design target cost from 58 (se) above the count below floor decisions at 0 / at the cap count (Chapter 7) 55.75 230.844 (0.040) 0.000 6.0% 4.7% / 5.1% twin, survey twice a year 56.70 241.944 (0.042) 11.101 13.4% 2.1% / 4.9% survey read as exact 57.35 246.793 (0.041) 15.950 13.3% 22.5% / 33.2% twin, quarterly survey 56.50 239.478 (0.040) 8.634 13.0% 1.0% / 3.8% twin, twice as sharp 56.40 237.849 (0.040) 7.005 12.1% 2.7% / 5.6% twin, both 56.20 235.765 (0.040) 4.921 12.2% 1.3% / 4.4% no readings after the audit 57.80 256.277 (0.063) 25.433 17.6% 0.0% / 0.0% the board's fixed targets target cost from 58 above the count below floor twin at Chapter 7's 55.75 55.75 247.157 16.313 31.3% survey as exact at Chapter 7's 55.75 55.75 258.352 27.509 38.6% survey as exact at 60.75, the whole error guarded 60.75 265.240 34.396 0.2% the twin costs 11.10 more than the count and 4.85 less than the survey read as exact (the abstract's $4.8m); the count is worth 1.045 a year against the twin (Panel 3) Theorem 15.2(iii): holding a planned point costs k = 4(1 - 0.975 beta) = 0.19183 a quarter; a point short next quarter b = 1.6 beta = 1.5623; critical ratio k/b = 0.1228 on the count the margin is the swing's 0.8772-quantile: 0.75 (P(w > 0.75) = 0.05 <= 0.1228 <= P(w >= 0.75) = 0.25), Chapter 7's margin with normal swings and errors, z = 1.1612 and the margin z sqrt(q + (0.975 s)^2): the twin's planned level is off by 0.975 times its error s, so the margin is z times the error a quarter later count (Chapter 7) normal margin 0.78 simulated best 0.75; a hiring limit binds in 9.8% of quarters twin, survey twice a year normal margin 1.70 to 1.83 simulated best 1.70; a hiring limit binds in 7.0% of quarters twin, quarterly survey normal margin 1.54 simulated best 1.50; a hiring limit binds in 4.7% of quarters twin, twice as sharp normal margin 1.27 to 1.46 simulated best 1.40; a hiring limit binds in 8.3% of quarters twin, both normal margin 1.19 simulated best 1.20; a hiring limit binds in 5.7% of quarters survey read as exact normal margin 2.94 to 2.97 simulated best 2.35; a hiring limit binds in 55.7% of quarters no readings after the audit normal margin 3.51 simulated best 2.80; a hiring limit binds in 0.0% of quarters the formula holds one quarter at a time; it sizes the count's and the twins' margins, where the limits seldom bind, but not the survey read as exact, whose readings swing hires to 0 or to the cap in over half the quarters, nor no readings, whose error grows from 0 after the audit Proposition 15.2(i): the twin's best target costs 241.852 on the design paths where it was chosen and 241.944 on fresh paths; the count 230.747 and 230.844 Proposition 15.2(ii): 56.75 against 56.70 on the twin, path by path: on the common design paths the difference has mean 0.0103 and standard deviation 0.268; against new independent paths (the lab's seed 26215) 18.507: common numbers need 4,758 times fewer paths to tell the targets apart
Panel 3 — Price readings (Theorem 15.2(i); Worked Example 15.1, Step 4; Figure 15.2(b); Table 15.3; Online Appendix Example 15.B.2: a quarterly survey $8.634m above the count, one twice as sharp $7.005m, both $4.921m; the count worth about $1.0m a year against the twin, a quarterly survey up to $0.23m, a sharper one $0.39m and both $0.58m; the readings study, twice a year and quarterly, by the survey's error 0.5 to 5: best targets 56.10 to 57.10 and 55.90 to 56.90, costs above the count $3.90m to $16.12m and $2.01m to $13.47m)
AXIOM Lab 2.15, step 3. Theorem 15.2(i): a rule on the true state costs no more than a rule on the twin's readings, which costs no more than one on coarser readings or none, and each gap is the value of the information it adds. Panel 2's fresh paths price the better surveys; a present value is worth a year, forever. The readings study varies the survey's error, twice a year and quarterly, on the first 10,000 design paths and the first 40,000 fresh paths, as the book did (Online Appendix Remark 15.B.1): the book searched targets from 55.5 to 59.5, and each of the eighteen cases here searches seven targets around the book's best. The cell takes about twenty seconds.
Python Code
YEAR = 4 * (1 - BETA) # a present value's yearly equivalent, forever
VALUE = {"count against the twin": DES["twin"]["excess"] * YEAR}
for key, lab in (("twin_quarterly", "a quarterly survey"), ("twin_sharp", "a survey twice as sharp"),
("twin_quarterly_sharp", "both")):
VALUE[lab] = (DES["twin"]["excess"] - DES[key]["excess"]) * YEAR
print("Theorem 15.2(i), from the cheapest knowledge to the dearest (cost from 58 on the fresh paths):")
for key in ("exact", "twin_quarterly_sharp", "twin_sharp", "twin_quarterly", "twin", "reset", "none"):
print(f" {LABEL[key]:30s} {DES[key]['cost']:9.3f} above the count {DES[key]['excess']:7.3f}")
print("worth a year, forever (present value x 4(1 - beta) = x %.5f):" % YEAR)
for lab, v in VALUE.items():
print(f" {lab:24s} {v:.3f}")
READ_CENTRE = {2: (56.10, 56.30, 56.40, 56.45, 56.60, 56.70, 56.80, 57.00, 57.10),
1: (55.90, 56.10, 56.20, 56.25, 56.40, 56.50, 56.60, 56.75, 56.90)} # the book's best targets
SDS = (0.5, 1.0, 1.25, 1.5, 2.0, 2.5, 3.0, 4.0, 5.0)
t0 = time.time()
CASES = []
for every in (2, 1):
for sd, ctr in zip(SDS, READ_CENTRE[every]):
win = np.round(ctr + 0.05 * np.arange(-3, 4), 2)
CASES.append(dict(every=every, sd=sd, window=win, prog=Program(win, "twin", every, sd, n=10000)))
run([c_["prog"] for c_ in CASES], SEED_DESIGN, N_DESIGN)
for c_ in CASES:
mean = c_["prog"].mean()
j = int(np.argmin(mean))
c_.update(best=float(c_["window"][j]), interior=bool(0 < j < len(mean) - 1))
fresh = [Program([c_["best"]], "twin", c_["every"], c_["sd"], n=40000) for c_ in CASES]
count40 = Program([T7], "exact", n=40000)
run(fresh + [count40], SEED_FRESH, N_FRESH)
COUNT40 = float(count40.mean()[0])
for c_, pr in zip(CASES, fresh):
c_["excess"] = float(pr.mean()[0]) - COUNT40
print(f"\nthe readings study: first 10,000 design paths and first 40,000 fresh paths (seeds {SEED_DESIGN}, {SEED_FRESH}); "
f"the count costs {COUNT40:.3f} on these fresh paths ({time.time() - t0:.0f} s)")
print("survey error (sd) " + "".join(f"{sd:8.2f}" for sd in SDS))
for every, lab in ((2, "twice a year"), (1, "quarterly")):
rows = [c_ for c_ in CASES if c_["every"] == every]
print(f" {lab:12s} target " + "".join(f"{c_['best']:8.2f}" for c_ in rows))
print(f" {'':12s} above " + "".join(f"{c_['excess']:8.2f}" for c_ in rows))
print(f"every best target inside its window: {all(c_['interior'] for c_ in CASES)}")
fig, (ax, bx) = plt.subplots(1, 2, figsize=(7.6, 3.6))
for key, col, ls, lab in (("exact", "#333333", "-", "count (exact)"), ("twin", "#1F4E79", "-", "twin"),
("reset", "#E65100", "--", "survey read as exact"), ("none", "#999999", ":", "no readings")):
d = DES[key]
ax.plot(d["targets"], d["mean"], color=col, ls=ls, lw=1.4, label=lab)
ax.plot([d["best"]], [d["design"]], "o", color=col, ms=4.5, zorder=5)
ax.set(xlim=(55, 61), ylim=(225, 315), xlabel="target for planned capability (points)", ylabel="cost from 58 ($ million)")
ax.set_title("(a) design paths: cost against the target", fontsize=9)
ax.legend(frameon=False, fontsize=7.5, loc="upper right")
for every, col, ls, lab in ((2, "#1F4E79", "-", "survey twice a year"), (1, "#2E7D32", "-.", "quarterly survey")):
rows = [c_ for c_ in CASES if c_["every"] == every]
bx.plot(SDS, [c_["excess"] for c_ in rows], color=col, ls=ls, lw=1.4, marker="o", ms=3.5, label=lab)
today = [c_ for c_ in CASES if c_["every"] == 2 and c_["sd"] == 2.5][0]
bx.annotate("today's survey", xy=(2.5, today["excess"]), xytext=(2.9, today["excess"] - 4.0), fontsize=8, color="#1F4E79",
arrowprops=dict(arrowstyle="-", lw=0.6, color="#1F4E79"))
bx.set(xlim=(0, 5.2), ylim=(0, 22), xlabel="survey error (sd, points)", ylabel="cost above the count's ($ million)")
bx.set_title("(b) what better readings are worth", fontsize=9)
bx.legend(frameon=False, fontsize=7.5, loc="upper left")
fig.suptitle("The twin needs a 1.70-point margin and costs \\$11.1m more (book seeds 1500 and 1501)", fontsize=10.5)
plt.tight_layout(); plt.show()Theorem 15.2(i), from the cheapest knowledge to the dearest (cost from 58 on the fresh paths):
count (Chapter 7) 230.844 above the count 0.000
twin, both 235.765 above the count 4.921
twin, twice as sharp 237.849 above the count 7.005
twin, quarterly survey 239.478 above the count 8.634
twin, survey twice a year 241.944 above the count 11.101
survey read as exact 246.793 above the count 15.950
no readings after the audit 256.277 above the count 25.433
worth a year, forever (present value x 4(1 - beta) = x 0.09418):
count against the twin 1.045
a quarterly survey 0.232
a survey twice as sharp 0.386
both 0.582
the readings study: first 10,000 design paths and first 40,000 fresh paths (seeds 1500, 1501); the count costs 230.838 on these fresh paths (16 s)
survey error (sd) 0.50 1.00 1.25 1.50 2.00 2.50 3.00 4.00 5.00
twice a year target 56.10 56.30 56.40 56.45 56.60 56.70 56.80 57.00 57.10
above 3.90 6.00 7.01 7.94 9.63 11.10 12.38 14.47 16.12
quarterly target 55.90 56.10 56.20 56.25 56.40 56.50 56.60 56.75 56.90
above 2.01 4.00 4.92 5.78 7.30 8.63 9.81 11.83 13.47
every best target inside its window: True

Panel 4 — Monitor the law (Theorem 15.3; Worked Example 15.1, Step 5; Figure 15.3; Table 15.3; Online Appendix Example 15.B.2: unmonitored under the drift the twin's bias settles at 2.210 points just after a survey and 3.088 just before the next, its innovations' mean at standard deviations; false alarms a survey, by threshold, 0.0471, 0.0347, 0.0258, 0.0192, 0.0145 and 0.0084, one in 10.6 to 59.8 years, the threshold 2.25; under the drift the count $301.992m and 14.1% of quarters below the floor, the twin that never questions its law $358.465m and 79.9%, with the monitor $320.169m and 14.4%, told of the drift $312.285m and 14.6%; detection on every path, after 12 quarters in the median, 4 to 22 from the 10th to the 90th percentile, 12.3 on average; without a drift $241.638m monitored against $241.837m, 4.6 audits in 120 years)
AXIOM Lab 2.15, step 4. Theorem 15.3(i): a correct twin's innovations have mean zero, are uncorrelated, and have the variances the filter computes; when the law's mean shifts by , a twin that keeps its law acquires a bias that settles, just after a survey, at (two quarters between surveys), and just before the next, which is the innovations' mean with its sign reversed. Part (ii): the cumulative sum with and detects a shift of standard deviations as fast as any test with as few false alarms. The threshold is calibrated on 20,000 paths without a drift (seed 1505), for the six thresholds the book prints; the drift, the count, and the twin unmonitored, monitored and told of the drift run on 100,000 paths (seed 1507). The cell takes about ten seconds.
Python Code
Kss = ERR["twin"]["gain"]
B_AFTER = (1 - Kss) * DRIFT * (1 + ALPHA) / (1 - (1 - Kss) * ALPHA ** 2)
B_BEFORE = ALPHA ** 2 * B_AFTER + DRIFT * (1 + ALPHA)
INNOV = -B_BEFORE / math.sqrt(ERR["twin"]["before"] ** 2 + SD_SURVEY ** 2)
m_, _, gains = 0.0, None, kalman_path(2, SD_SURVEY)[2]
track = []
for t in range(HORIZON): # the mean error quarter by quarter, the gains as they come
m_ *= 1 - gains[t]
track.append(m_)
m_ = ALPHA * m_ + (DRIFT if t >= DRIFT_START else 0.0)
print(f"Theorem 15.3(i): the twin that keeps its law under the drift overestimates capability by b+ = {B_AFTER:.3f} just "
f"after a survey and {B_BEFORE:.3f} just before the next (steady gain {Kss:.4f}); its innovations' mean is "
f"{INNOV:.3f} standard deviations; the recursion from quarter 8 reaches {track[40]:.3f} at quarter 40 and "
f"{track[200]:.3f} at quarter 200")
THRESH = [1.5, 1.75, 2.0, 2.25, 2.5, 3.0]
t0 = time.time()
TT = DES["twin"]["best"]
cal = run([Program([TT] * len(THRESH), "twin", 2, SD_SURVEY, n=N_DESIGN, monitor=(KAPPA, THRESH))], SEED_CAL, N_DESIGN)[0]
RATE = cal.alarms.sum(axis=1) / (N_DESIGN * (HORIZON // 2))
H_STAR = THRESH[int(np.argmax(RATE <= 1 / 50))]
print(f"\nthe monitor's calibration, {N_DESIGN:,} paths without a drift (seed {SEED_CAL}), {time.time() - t0:.0f} s:")
print(" threshold " + "".join(f"{h:8.2f}" for h in THRESH))
print(" false alarms a survey " + "".join(f"{x:8.4f}" for x in RATE))
print(" one in so many years " + "".join(f"{1 / x / 2:8.1f}" for x in RATE))
print(f" the least threshold with at most one false alarm in 50 surveys: {H_STAR}; one in {1 / RATE[THRESH.index(H_STAR)] / 2:.1f} "
f"years")
GAMMA = math.exp(H_STAR * 1.0)
print(f"Theorem 15.3(ii) with delta = 2 kappa = 1: H = ln(gamma)/delta guarantees at least gamma = e^{H_STAR} = {GAMMA:.1f} "
f"surveys between false alarms (here {1 / RATE[THRESH.index(H_STAR)]:.0f}), and a delay of about 2 ln(gamma)/delta^2 = "
f"{2 * math.log(GAMMA):.1f} surveys once the innovations have shifted by a standard deviation")
t0 = time.time()
dr = (DRIFT_START, DRIFT)
DRUNS = dict(count=Program([T7], "exact", n=N_FRESH, drift=dr, oracle=True),
static=Program([TT], "twin", 2, SD_SURVEY, n=N_FRESH, drift=dr, keep=(2000, 40), trace="z"),
monitored=Program([TT], "twin", 2, SD_SURVEY, n=N_FRESH, drift=dr, monitor=(KAPPA, H_STAR), keep=(2000, 40)),
oracle=Program([TT], "twin", 2, SD_SURVEY, n=N_FRESH, drift=dr, oracle=True, keep=(2000, 40)),
monitored_nd=Program([TT], "twin", 2, SD_SURVEY, n=N_FRESH, monitor=(KAPPA, H_STAR)),
twin_nd=Program([TT], "twin", 2, SD_SURVEY, n=N_FRESH, trace="z"))
run(list(DRUNS.values()), SEED_DRIFT, N_FRESH)
DR = {k: dict(cost=float(v.mean()[0]), below=float(v.share_below()[0])) for k, v in DRUNS.items()}
fa = DRUNS["monitored"].first[0]
detected = fa >= DRIFT_START
DELAY = (fa[detected] - DRIFT_START).astype(float)
DLY = dict(detected=float(detected.mean()), median=float(np.median(DELAY)), q10=float(np.quantile(DELAY, 0.1)),
q90=float(np.quantile(DELAY, 0.9)), mean=float(DELAY.mean()))
AUDITS = float(DRUNS["monitored_nd"].alarms.mean())
print(f"\nthe drift from quarter {DRIFT_START}, {N_FRESH:,} paths (seed {SEED_DRIFT}), {time.time() - t0:.0f} s:")
print(" cost from 58 above the count below floor")
for key, lab in (("count", "count, law re-estimated at once"), ("static", "twin, law never questioned"),
("monitored", "twin with the monitor"), ("oracle", "twin told of the drift")):
print(f" {lab:34s} {DR[key]['cost']:12.3f} {DR[key]['cost'] - DR['count']['cost']:15.3f} {100 * DR[key]['below']:12.1f}%")
print(f"the monitor detects the drift on {100 * DLY['detected']:.1f}% of paths, after {DLY['median']:.0f} quarters in the "
f"median ({DLY['q10']:.0f} to {DLY['q90']:.0f} from the 10th to the 90th percentile, {DLY['mean']:.1f} on average)")
print(f"without a drift the monitored twin costs {DR['monitored_nd']['cost']:.3f} against {DR['twin_nd']['cost']:.3f} unmonitored "
f"and audits {AUDITS:.1f} times in {HORIZON // 4} years")
# Theorem 15.3(i) on the simulated innovations: the correct twin's are white; under the drift they lean one way
zn, zd = DRUNS["twin_nd"].zrec, [x for x in DRUNS["static"].zrec if x[0] >= 40]
n_ = sum(x[4] for x in zn)
m_ = sum(x[1] for x in zn) / n_
WHITE = dict(mean=m_, sd=math.sqrt(sum(x[2] for x in zn) / n_ - m_ ** 2),
lag1=sum(x[3] for x in zn[1:]) / sum(x[4] for x in zn[1:]), drift_mean=sum(x[1] for x in zd) / sum(x[4] for x in zd))
print(f"the correct twin's standardized innovations over {len(zn)} surveys x {zn[0][4]:,} paths: mean {WHITE['mean']:.4f}, "
f"sd {WHITE['sd']:.4f}, correlation with the previous survey's {WHITE['lag1']:.4f}; the twin that keeps its law "
f"under the drift, from quarter 40 on: mean {WHITE['drift_mean']:.4f} (the formula: {INNOV:.4f})")
fa_n = DRUNS["monitored_nd"].first[0]
DETECT = [float(np.mean((fa >= 0) & (fa <= t))) for t in range(41)]
FALSE = [float(np.mean((fa_n >= 0) & (fa_n <= t))) for t in range(41)]
print(f"within 40 quarters {100 * FALSE[40]:.1f}% of monitors raise a false alarm without a drift")
fig, (ax, bx) = plt.subplots(1, 2, figsize=(7.6, 3.6))
q_ = np.arange(41)
ax.step(q_, 100 * np.array(DETECT), where="post", color="#E65100", lw=1.6, label=f"drift from quarter {DRIFT_START}")
ax.step(q_, 100 * np.array(FALSE), where="post", color="#1F4E79", lw=1.4, ls="--", label="no drift")
ax.axvline(DRIFT_START, ymax=0.78, color="#999999", lw=0.8, ls=":")
ax.plot([DRIFT_START + DLY["median"]], [50], "o", color="#E65100", ms=4.5, zorder=6)
ax.text(DRIFT_START + DLY["median"] + 1.0, 47, f"median: {DLY['median']:.0f} quarters\nafter the drift", fontsize=8,
color="#E65100", va="top")
ax.set(xlim=(0, 40), ylim=(0, 100), xticks=[0, 8, 16, 24, 32, 40], xlabel="quarter", ylabel="monitors that have alarmed (%)")
ax.set_title("(a) alarms, with and without the drift", fontsize=9)
ax.legend(frameon=False, fontsize=7.5, loc="upper left")
for key, col, ls, lab in (("static", "#B71C1C", "--", "twin, law never questioned"), ("monitored", "#1F4E79", "-",
"twin with escalation"), ("oracle", "#2E7D32", ":", "twin told of the drift")):
bx.plot(np.arange(40), DRUNS[key].kept, color=col, ls=ls, lw=1.5, label=lab)
bx.axhline(FLOOR, color="#999999", lw=0.8, ls=":")
bx.axvline(DRIFT_START, ymax=0.6, color="#999999", lw=0.8, ls=":")
bx.text(39.5, 55.1, "floor", fontsize=8, color="#616161", ha="right", va="bottom")
bx.set(xlim=(0, 40), ylim=(51, 61), xticks=[0, 8, 16, 24, 32, 40], xlabel="quarter", ylabel="mean capability (points)")
bx.set_title("(b) mean capability under the drift", fontsize=9)
bx.legend(frameon=False, fontsize=7.5, loc="upper right")
fig.suptitle("The monitor finds the drift in 12 quarters, in the median (book seed 1507)", fontsize=10.5)
plt.tight_layout(); plt.show()Theorem 15.3(i): the twin that keeps its law under the drift overestimates capability by b+ = 2.210 just after a survey and 3.088 just before the next (steady gain 0.2844); its innovations' mean is -1.045 standard deviations; the recursion from quarter 8 reaches 2.205 at quarter 40 and 2.210 at quarter 200
the monitor's calibration, 20,000 paths without a drift (seed 1505), 2 s:
threshold 1.50 1.75 2.00 2.25 2.50 3.00
false alarms a survey 0.0471 0.0347 0.0258 0.0192 0.0145 0.0084
one in so many years 10.6 14.4 19.4 26.0 34.5 59.8
the least threshold with at most one false alarm in 50 surveys: 2.25; one in 26.0 years
Theorem 15.3(ii) with delta = 2 kappa = 1: H = ln(gamma)/delta guarantees at least gamma = e^2.25 = 9.5 surveys between false alarms (here 52), and a delay of about 2 ln(gamma)/delta^2 = 4.5 surveys once the innovations have shifted by a standard deviation
the drift from quarter 8, 100,000 paths (seed 1507), 8 s:
cost from 58 above the count below floor
count, law re-estimated at once 301.992 0.000 14.1%
twin, law never questioned 358.465 56.473 79.9%
twin with the monitor 320.169 18.177 14.4%
twin told of the drift 312.285 10.293 14.6%
the monitor detects the drift on 100.0% of paths, after 12 quarters in the median (4 to 22 from the 10th to the 90th percentile, 12.3 on average)
without a drift the monitored twin costs 241.638 against 241.837 unmonitored and audits 4.6 times in 120 years
the correct twin's standardized innovations over 239 surveys x 100,000 paths: mean 0.0001, sd 0.9999, correlation with the previous survey's -0.0001; the twin that keeps its law under the drift, from quarter 40 on: mean -1.0449 (the formula: -1.0450)
within 40 quarters 30.8% of monitors raise a false alarm without a drift

Panel 5 — Set the envelope (Definition 15.3; Proposition 15.3; Worked Example 15.1, Step 6: the twin hires alone inside Chapter 14's guardrail and escalates on its monitor's alarm; a trigger on its uncertainty would fire twice a year on schedule; its target, chosen on the design paths at $241.85m, costs $241.94m on the fresh ones, within the simulation's error)
AXIOM Lab 2.15, step 5. The envelope declares what the twin may do alone and when it hands a decision back. Proposition 15.3(ii): a guardrail that holds a best decision for every belief costs nothing at the optimum; every target here plans at least 55.75 on the twin's estimate unless hiring is at the cap, so Chapter 14's guardrail holds on every fresh path and quarter. Part (iii): the error variances do not depend on the readings, so a trigger on the twin's uncertainty fires on a calendar fixed in advance; only the innovations can catch a change in the law. Part (i): the loop is stable when the rule and the filter are. Last, the audit committee's certification of the target on fresh paths, and an illustration with new draws from the lab's seed, 26215: ten programs under the drift, each run by the monitored twin, whose errors keep the same calendar until an alarm sends the engineering office to audit.
Python Code
print(f"Proposition 15.3(ii): planned capability on the estimate below 55 without hiring at the cap, on {N_FRESH:,} fresh "
f"paths x {HORIZON} quarters: {sum(DES[k]['breaks'] for k in DES)} times in all designs")
pr2, po2, _ = kalman_path(2, SD_SURVEY, horizon=40)
sd_dec = np.sqrt(np.where(schedule(2, 40), po2, pr2)) # the error the twin decides with, quarter by quarter
sd_eve = np.sqrt(pr2)
TRIG = 1.5
fires = [t for t in range(40) if schedule(2, 40)[t] and sd_eve[t] > TRIG]
print(f"Proposition 15.3(iii): the twin's error just before each survey, quarters 2-12: "
+ ", ".join(f"{sd_eve[t]:.3f}" for t in range(2, 13, 2)) + f", ... settling at {ERR['twin']['before']:.3f}; just after, "
f"{ERR['twin']['after']:.3f}")
print(f" a trigger 'escalate when the error exceeds {TRIG} points' fires on the eve of every survey from quarter {fires[0]} "
f"on, whatever the surveys say: a calendar the board might as well schedule")
mode_rule = ALPHA - ALPHA # h = T - 0.975 c restores the target in one quarter
mode_filter = (1 - Kss) * ALPHA ** 2 # the error's mode over the half year between surveys
print(f"Proposition 15.3(i): the rule's mode {mode_rule:.3f} (it restores the target in a quarter, inside the limits) and "
f"the filter's {mode_filter:.3f} a half year, (1 - K) a^2: both inside the unit circle, so the loop is stable")
se_d = DES["twin"]["costs"].std() / math.sqrt(N_DESIGN)
GAP = DES["twin"]["cost"] - DES["twin"]["design"]
print(f"Worked Example 15.1, Step 6: chosen on the design paths at {DES['twin']['design']:.2f}, the target costs "
f"{DES['twin']['cost']:.2f} on the fresh ones; the difference {GAP:.3f} is {GAP / math.sqrt(se_d ** 2 + DES['twin']['se'] ** 2):.1f} "
f"standard errors ({se_d:.3f} on the design paths, {DES['twin']['se']:.3f} on the fresh): within the simulation's error")
# new draws with the lab's seed: ten programs under the drift, run by the monitored twin
ten = run([Program([TT], "twin", 2, SD_SURVEY, n=10, horizon=40, drift=dr, monitor=(KAPPA, H_STAR), trace=10, sample=10)],
SEED, 10, horizon=40)[0]
P10 = np.sqrt(np.array([x["p"] for x in ten.log])) # the error each program decides with: [quarter, program]
GT = np.array([t for t, _, _, _ in ten.zrec]) # survey quarters
G10 = np.array([G for _, _, G, _ in ten.zrec]) # the cumulative sum at each survey, before a restart
HIT = np.array([a for _, _, _, a in ten.zrec])
first10 = ten.first[0]
same = bool(np.allclose(P10[:first10.min()], P10[:first10.min(), :1]))
print(f"\nten programs under the drift (seed {SEED}): alarms from quarter {DRIFT_START} at quarters "
+ ", ".join(str(int(x)) for x in first10) + f" ({np.median(first10 - DRIFT_START):.0f} quarters after the drift in the "
f"median); their errors agree, quarter by quarter, until the first audit: {same}")
fig, (ax, bx) = plt.subplots(1, 2, figsize=(7.6, 3.6))
for i in range(10):
ax.step(np.arange(40), P10[:, i], where="post", color="#1F4E79", lw=0.9, alpha=0.55)
ax.plot([first10[i]], [0.0], "o", color="#B71C1C", ms=3.5, zorder=5, clip_on=False,
label="alarm and audit" if i == 0 else None)
upto = GT <= first10[i] # each program's sum until its alarm
bx.plot(GT[upto], G10[upto, i], color="#1F4E79", lw=0.9, alpha=0.55, marker=".", ms=3)
bx.plot([first10[i]], [G10[GT == first10[i], i][0]], "o", color="#B71C1C", ms=4, zorder=5,
label="alarm and audit" if i == 0 else None)
ax.axvline(DRIFT_START, color="#999999", lw=0.8, ls=":")
ax.text(DRIFT_START + 0.5, 1.88, "drift begins", fontsize=8, color="#616161")
ax.set(xlim=(0, 40), ylim=(0, 2.0), xticks=[0, 8, 16, 24, 32, 40], xlabel="quarter", ylabel="the twin's error (sd, points)")
ax.set_title("(a) uncertainty keeps a calendar", fontsize=9)
ax.legend(frameon=False, fontsize=7.5, loc="upper right")
bx.axhline(H_STAR, color="#B71C1C", lw=0.9, ls="--")
bx.text(39.5, H_STAR + 0.15, f"threshold {H_STAR}", fontsize=8, color="#B71C1C", ha="right")
bx.axvline(DRIFT_START, color="#999999", lw=0.8, ls=":")
bx.set(xlim=(0, 40), ylim=(0, 5), xticks=[0, 8, 16, 24, 32, 40], xlabel="quarter", ylabel="cumulative sum G at each survey")
bx.set_title("(b) the surprises raise the alarm", fontsize=9)
bx.legend(frameon=False, fontsize=7.5, loc="upper left")
fig.suptitle("Escalate on surprises, not on uncertainty (seed 26215)", fontsize=10.5)
plt.tight_layout(); plt.show()Proposition 15.3(ii): planned capability on the estimate below 55 without hiring at the cap, on 100,000 fresh paths x 480 quarters: 0 times in all designs Proposition 15.3(iii): the twin's error just before each survey, quarters 2-12: 0.937, 1.254, 1.419, 1.501, 1.541, 1.560, ... settling at 1.576; just after, 1.333 a trigger 'escalate when the error exceeds 1.5 points' fires on the eve of every survey from quarter 8 on, whatever the surveys say: a calendar the board might as well schedule Proposition 15.3(i): the rule's mode 0.000 (it restores the target in a quarter, inside the limits) and the filter's 0.680 a half year, (1 - K) a^2: both inside the unit circle, so the loop is stable Worked Example 15.1, Step 6: chosen on the design paths at 241.85, the target costs 241.94 on the fresh ones; the difference 0.092 is 0.9 standard errors (0.093 on the design paths, 0.042 on the fresh): within the simulation's error ten programs under the drift (seed 26215): alarms from quarter 8 at quarters 20, 16, 12, 10, 22, 20, 26, 12, 26, 22 (12 quarters after the drift in the median); their errors agree, quarter by quarter, until the first audit: True

Panel 6 — Table 15.3: designs for knowing capability, from an audited 58 (Worked Example 15.1)
Table 15.3's rows from the panels above. Target: planned capability (points); error: the estimate's standard deviation just after and just before a survey; cost above the count: present value at 10 percent, $ million, on the 100,000 fresh paths (with the drift, on its 100,000 paths, above the count with the drift); below floor: long-run share of quarters below 55.
Python Code
def err_txt(key):
e = ERR.get(key, {})
if key == "exact":
return "0"
if key == "none":
return f"{e['limit']:.2f}"
return f"{e['after']:.2f}-{e['before']:.2f}"
T153 = [("Count (Chapter 7)", "--", DES["exact"]["best"], "exact", 0.0, DES["exact"]["below"]),
("No readings after the audit", "none", DES["none"]["best"], "none", DES["none"]["excess"], DES["none"]["below"]),
("Survey read as exact", "twice a year", T7, "reset", FIXED["survey as exact at Chapter 7's 55.75"]["excess"],
FIXED["survey as exact at Chapter 7's 55.75"]["below"]),
(" whole error guarded", "twice a year", T7 + 5.0, "reset",
FIXED["survey as exact at 60.75, the whole error guarded"]["excess"],
FIXED["survey as exact at 60.75, the whole error guarded"]["below"]),
(" best target", "twice a year", DES["reset"]["best"], "reset", DES["reset"]["excess"], DES["reset"]["below"]),
("Twin, Chapter 7's target", "twice a year", T7, "twin", FIXED["twin at Chapter 7's 55.75"]["excess"],
FIXED["twin at Chapter 7's 55.75"]["below"]),
("Twin, best target", "twice a year", DES["twin"]["best"], "twin", DES["twin"]["excess"], DES["twin"]["below"]),
("Twin", "quarterly", DES["twin_quarterly"]["best"], "twin_quarterly", DES["twin_quarterly"]["excess"],
DES["twin_quarterly"]["below"]),
("Twin", "twice as sharp", DES["twin_sharp"]["best"], "twin_sharp", DES["twin_sharp"]["excess"],
DES["twin_sharp"]["below"]),
("Twin", "both", DES["twin_quarterly_sharp"]["best"], "twin_quarterly_sharp", DES["twin_quarterly_sharp"]["excess"],
DES["twin_quarterly_sharp"]["below"]),
("Twin, law never questioned", "twice a year", TT, None, DR["static"]["cost"] - DR["count"]["cost"],
DR["static"]["below"]),
("Twin with monitor", "twice a year", TT, None, DR["monitored"]["cost"] - DR["count"]["cost"], DR["monitored"]["below"]),
("Twin told of the drift", "twice a year", TT, None, DR["oracle"]["cost"] - DR["count"]["cost"], DR["oracle"]["below"])]
print(f"{'Design':30s} {'Survey':15s} {'Target':>7s} {'Error (sd)':>11s} {'Cost above the count':>21s} {'Below floor (%)':>16s}")
for i, (name, surv, tgt, key, ex, below) in enumerate(T153):
if i == 10:
print("An extra half point lost each quarter from quarter 8 (cost above the count with the drift):")
print(f"{name:30s} {surv:15s} {tgt:7.2f} {err_txt(key) if key else '--':>11s} {ex:21.2f} {100 * below:16.1f}")
print(f"\nall cells took {time.time() - T_START:.0f} s")Design Survey Target Error (sd) Cost above the count Below floor (%) Count (Chapter 7) -- 55.75 0 0.00 6.0 No readings after the audit none 57.80 3.02 25.43 17.6 Survey read as exact twice a year 55.75 2.50-2.53 27.51 38.6 whole error guarded twice a year 60.75 2.50-2.53 34.40 0.2 best target twice a year 57.35 2.50-2.53 15.95 13.3 Twin, Chapter 7's target twice a year 55.75 1.33-1.58 16.31 31.3 Twin, best target twice a year 56.70 1.33-1.58 11.10 13.4 Twin quarterly 56.50 1.17-1.32 8.63 13.0 Twin twice as sharp 56.40 0.89-1.26 7.01 12.1 Twin both 56.20 0.79-1.02 4.92 12.2 An extra half point lost each quarter from quarter 8 (cost above the count with the drift): Twin, law never questioned twice a year 56.70 -- 56.47 79.9 Twin with monitor twice a year 56.70 -- 18.18 14.4 Twin told of the drift twice a year 56.70 -- 10.29 14.6 all cells took 69 s
Validation — the book's numbers, and agreement with DCT_V2_Ch15_Lab.xlsx
Each checkpoint is a number printed in Chapter 15 (the abstract, Sections 15.2–15.5, Worked Example 15.1, Table 15.3,
Figure 15.3, the Key Takeaways, the LOS Guidance) or in Online Appendix Remark 15.B.1 and Example 15.B.2, with a tolerance
of half a unit in its last printed digit; a few check what a theorem or proposition promises (1 = holds), and the
simulated innovations are checked within 0.01. Shares are in percent. The workbook recomputes the checkpoints marked
workbook with spreadsheet formulas and shows PASS against the same engine values on its Checks sheet.
Python Code
def bk(s):
# a number as the book prints it: its value and half a unit in its last printed digit
t = s.replace(",", "").replace("−", "-")
return float(t), 0.5 * 10.0 ** -(len(t.split(".")[1]) if "." in t else 0) + 1e-9
def c_(name, got, printed):
v, tol = bk(printed)
return (name, float(got), v, tol)
def flag(name, ok):
return (name, float(bool(ok)), 1.0, 0.0)
def count(name, got, n):
return (name, float(got), float(n), 0.0)
pc = lambda x: 100.0 * x
E2, D_ = ERR["twin"], DES
FX = {"twin55": FIXED["twin at Chapter 7's 55.75"], "reset55": FIXED["survey as exact at Chapter 7's 55.75"],
"reset60": FIXED["survey as exact at 60.75, the whole error guarded"]}
checks = [
# the declarations and the twin's error (Step 1-2; Section 15.2; Theorem 15.1; Table 15.3; Example 15.B.2)
c_("a swing's variance q (workbook)", Q, "0.45"),
c_("a swing's standard deviation, Section 15.2 (workbook)", math.sqrt(Q), "0.67"),
c_("no readings: the error's limit (workbook)", ERR["none"]["limit"], "3.019"),
c_("no readings: error, Table 15.3", ERR["none"]["limit"], "3.02"),
c_("no readings: error, Section 15.2 and Step 2", ERR["none"]["limit"], "3.0"),
c_("twin: error just after a survey (workbook)", E2["after"], "1.333"),
c_("twin: error a quarter later (workbook)", E2["mid"], "1.463"),
c_("twin: error just before the next survey (workbook)", E2["before"], "1.576"),
c_("twin: the gain (workbook)", E2["gain"], "0.284"),
c_("twin: the gain, Section 15.2", E2["gain"], "0.28"),
c_("twin: error just after, Table 15.3 and Step 2", E2["after"], "1.33"),
c_("twin: error just before, Table 15.3 and Step 2", E2["before"], "1.58"),
c_("quarterly survey: error just after (workbook)", ERR["twin_quarterly"]["after"], "1.169"),
c_("quarterly survey: error just before (workbook)", ERR["twin_quarterly"]["before"], "1.323"),
c_("quarterly survey: the gain (workbook)", ERR["twin_quarterly"]["gain"], "0.219"),
c_("quarterly survey: Table 15.3, after", ERR["twin_quarterly"]["after"], "1.17"),
c_("quarterly survey: Table 15.3, before", ERR["twin_quarterly"]["before"], "1.32"),
c_("twice as sharp: error just after (workbook)", ERR["twin_sharp"]["after"], "0.888"),
c_("twice as sharp: error just before (workbook)", ERR["twin_sharp"]["before"], "1.261"),
c_("twice as sharp: Table 15.3, after", ERR["twin_sharp"]["after"], "0.89"),
c_("twice as sharp: Table 15.3, before", ERR["twin_sharp"]["before"], "1.26"),
c_("both: error just after (workbook)", ERR["twin_quarterly_sharp"]["after"], "0.791"),
c_("both: error just before (workbook)", ERR["twin_quarterly_sharp"]["before"], "1.022"),
c_("both: Table 15.3, after", ERR["twin_quarterly_sharp"]["after"], "0.79"),
c_("both: Table 15.3, before", ERR["twin_quarterly_sharp"]["before"], "1.02"),
c_("survey read as exact: error just after (workbook)", ERR["reset"]["after"], "2.500"),
c_("survey read as exact: error just before (workbook)", ERR["reset"]["before"], "2.528"),
c_("survey read as exact: Table 15.3 and Step 2, before", ERR["reset"]["before"], "2.53"),
c_("survey read as exact: good to, Section 15.2", ERR["reset"]["after"], "2.5"),
flag("Theorem 15.1(iii): from 0 and from 10 the variance reaches the fixed point", CONVERGE < 1e-12),
flag("Proposition 15.1(i): precisions add at a survey (workbook)", abs(PRECISION - E2["fp"]) < 1e-12),
c_("exact filter: largest difference of the means", EXACT["max_abs"], "0.012"),
c_("exact filter: mean difference", EXACT["mean_abs"], "0.0013"),
c_("exact filter: largest difference of the sds", EXACT["sd_max"], "0.011"),
count("exact filter: survey updates compared", EXACT["n"], 5700),
# the designs (Step 3; Section 15.3; Table 15.3; Example 15.B.2)
] + [c_(f"{LABEL[k]}: best target (workbook)", D_[k]["best"], p) for k, p in
(("exact", "55.75"), ("twin", "56.70"), ("reset", "57.35"), ("twin_quarterly", "56.50"), ("twin_sharp", "56.40"),
("twin_quarterly_sharp", "56.20"), ("none", "57.80"))] + [
flag("every design's best target lies inside its window", all(D_[k]["interior"] for k in D_)),
flag("the 0.25 grid's least cost lies in the window: count, twin, survey as exact, no readings",
all(D_[k]["grid_in_window"] for k in CURVED)),
c_("count: design-path cost (workbook)", D_["exact"]["design"], "230.747"),
c_("twin: design-path cost (workbook)", D_["twin"]["design"], "241.852"),
c_("twin: design-path cost, Step 6", D_["twin"]["design"], "241.85"),
c_("count: cost from 58", D_["exact"]["cost"], "230.844"),
c_("count: cost from 58, Section 15.3 and Step 3", D_["exact"]["cost"], "230.8"),
c_("count: standard error", D_["exact"]["se"], "0.040"),
c_("twin: cost from 58", D_["twin"]["cost"], "241.944"),
c_("twin: cost from 58, Step 6", D_["twin"]["cost"], "241.94"),
c_("twin: cost from 58, Section 15.3", D_["twin"]["cost"], "241.9"),
c_("twin: standard error", D_["twin"]["se"], "0.042"),
c_("survey read as exact: cost from 58", D_["reset"]["cost"], "246.793"),
c_("quarterly survey: cost from 58", D_["twin_quarterly"]["cost"], "239.478"),
c_("twice as sharp: cost from 58", D_["twin_sharp"]["cost"], "237.849"),
c_("both: cost from 58", D_["twin_quarterly_sharp"]["cost"], "235.765"),
c_("no readings: cost from 58", D_["none"]["cost"], "256.277"),
c_("no readings: standard error", D_["none"]["se"], "0.063"),
c_("twin: above the count (workbook)", D_["twin"]["excess"], "11.101"),
c_("twin: above the count, Table 15.3", D_["twin"]["excess"], "11.10"),
c_("twin: above the count, abstract, Step 3, Key Takeaways", D_["twin"]["excess"], "11.1"),
c_("twin: margin, Section 15.3 and the board (workbook)", D_["twin"]["best"] - FLOOR, "1.70"),
c_("survey read as exact: above the count (workbook)", D_["reset"]["excess"], "15.950"),
c_("survey read as exact: above the count, Table 15.3", D_["reset"]["excess"], "15.95"),
c_("survey read as exact: above the count, Step 3", D_["reset"]["excess"], "15.9"),
c_("abstract: the twin costs less than the survey read as exact (workbook)", D_["reset"]["excess"] - D_["twin"]["excess"],
"4.8"),
c_("quarterly survey: above the count (workbook)", D_["twin_quarterly"]["excess"], "8.634"),
c_("quarterly survey: above the count, Table 15.3", D_["twin_quarterly"]["excess"], "8.63"),
c_("quarterly survey: above the count, Step 4", D_["twin_quarterly"]["excess"], "8.6"),
c_("twice as sharp: above the count (workbook)", D_["twin_sharp"]["excess"], "7.005"),
c_("twice as sharp: above the count, Table 15.3", D_["twin_sharp"]["excess"], "7.01"),
c_("twice as sharp: above the count, Step 4", D_["twin_sharp"]["excess"], "7.0"),
c_("both: above the count (workbook)", D_["twin_quarterly_sharp"]["excess"], "4.921"),
c_("both: above the count, Table 15.3", D_["twin_quarterly_sharp"]["excess"], "4.92"),
c_("both: above the count, Step 4", D_["twin_quarterly_sharp"]["excess"], "4.9"),
c_("no readings: above the count (workbook)", D_["none"]["excess"], "25.433"),
c_("no readings: above the count, Table 15.3", D_["none"]["excess"], "25.43"),
c_("no readings: above the count, Step 3", D_["none"]["excess"], "25.4"),
c_("twin at Chapter 7's 55.75: above the count (workbook)", FX["twin55"]["excess"], "16.313"),
c_("twin at Chapter 7's 55.75: Table 15.3", FX["twin55"]["excess"], "16.31"),
c_("twin at Chapter 7's 55.75: Section 15.3 and Step 3", FX["twin55"]["excess"], "16.3"),
c_("survey as exact at 55.75: above the count (workbook)", FX["reset55"]["excess"], "27.509"),
c_("survey as exact at 55.75: Table 15.3", FX["reset55"]["excess"], "27.51"),
c_("survey as exact at 55.75: Step 3", FX["reset55"]["excess"], "27.5"),
c_("survey as exact at 60.75: above the count (workbook)", FX["reset60"]["excess"], "34.396"),
c_("survey as exact at 60.75: Table 15.3", FX["reset60"]["excess"], "34.40"),
c_("survey as exact at 60.75: Step 3", FX["reset60"]["excess"], "34.4"),
c_("count: below the floor, %", pc(D_["exact"]["below"]), "6.0"),
c_("twin: below the floor, %", pc(D_["twin"]["below"]), "13.4"),
c_("survey read as exact: below the floor, %", pc(D_["reset"]["below"]), "13.3"),
c_("quarterly survey: below the floor, %", pc(D_["twin_quarterly"]["below"]), "13.0"),
c_("twice as sharp: below the floor, %", pc(D_["twin_sharp"]["below"]), "12.1"),
c_("both: below the floor, %", pc(D_["twin_quarterly_sharp"]["below"]), "12.2"),
c_("no readings: below the floor, %", pc(D_["none"]["below"]), "17.6"),
c_("twin at 55.75: below the floor, %", pc(FX["twin55"]["below"]), "31.3"),
c_("twin at 55.75: below the floor, %, Section 15.3 and Step 3", pc(FX["twin55"]["below"]), "31"),
c_("survey as exact at 55.75: below the floor, %", pc(FX["reset55"]["below"]), "38.6"),
c_("survey as exact at 60.75: below the floor, %", pc(FX["reset60"]["below"]), "0.2"),
flag("Proposition 15.2(i): the chosen targets cost less on their design paths than on fresh ones",
D_["twin"]["design"] < D_["twin"]["cost"] and D_["exact"]["design"] < D_["exact"]["cost"]),
flag("Proposition 15.2(ii): common paths tell neighbouring targets apart better", CRN["sd_common"] < CRN["sd_indep"]),
flag("Step 6: the design-path cost is within two standard errors of the fresh", abs(GAP) < 2 * math.sqrt(se_d ** 2 + D_["twin"]["se"] ** 2)),
flag("Theorem 15.2(i): the count, better surveys, the twin, the survey as exact, no readings, in order of cost",
D_["exact"]["cost"] < D_["twin_quarterly_sharp"]["cost"] < min(D_["twin_sharp"]["cost"], D_["twin_quarterly"]["cost"])
and max(D_["twin_sharp"]["cost"], D_["twin_quarterly"]["cost"]) < D_["twin"]["cost"] < D_["reset"]["cost"] < D_["none"]["cost"]),
c_("Theorem 15.2(iii): the count's margin, the swing's quantile (workbook)", QUANT, "0.75"),
c_("Chapter 7's critical ratio (workbook)", RATIO, "0.1228"),
# readings (Step 4; Section 15.3; Figure 15.2(b); Example 15.B.2)
c_("the count is worth a year against the twin, Section 15.3 and Step 4 (workbook)", VALUE["count against the twin"], "1.0"),
c_("a quarterly survey is worth a year, Step 4 (workbook)", VALUE["a quarterly survey"], "0.23"),
c_("a survey twice as sharp is worth a year, Step 4 (workbook)", VALUE["a survey twice as sharp"], "0.39"),
c_("both are worth a year, Step 4 (workbook)", VALUE["both"], "0.58"),
] + [c_(f"readings study, {'twice a year' if c['every'] == 2 else 'quarterly'}, error {c['sd']}: best target", c["best"], p)
for c, p in zip(CASES, ("56.10", "56.30", "56.40", "56.45", "56.60", "56.70", "56.80", "57.00", "57.10",
"55.90", "56.10", "56.20", "56.25", "56.40", "56.50", "56.60", "56.75", "56.90"))] + [
c_(f"readings study, {'twice a year' if c['every'] == 2 else 'quarterly'}, error {c['sd']}: above the count",
c["excess"], p)
for c, p in zip(CASES, ("3.90", "6.00", "7.01", "7.94", "9.63", "11.10", "12.38", "14.47", "16.12",
"2.01", "4.00", "4.92", "5.78", "7.30", "8.63", "9.81", "11.83", "13.47"))] + [
flag("readings study: every best target inside its window", all(c["interior"] for c in CASES)),
# the monitor (Step 5; Section 15.4; Theorem 15.3; Figure 15.3; Example 15.B.2)
c_("unmonitored under the drift: bias just after a survey (workbook)", B_AFTER, "2.210"),
c_("unmonitored under the drift: bias just before the next (workbook)", B_BEFORE, "3.088"),
c_("unmonitored under the drift: innovations' mean, sd (workbook)", INNOV, "-1.045"),
c_("Section 15.4: overestimates by about, after", B_AFTER, "2.2"),
c_("Section 15.4: and before", B_BEFORE, "3.1"),
] + [c_(f"calibration: false alarms a survey at {h}", r_, p) for h, r_, p in
zip(THRESH, RATE, ("0.0471", "0.0347", "0.0258", "0.0192", "0.0145", "0.0084"))] + [
c_(f"calibration: one false alarm in so many years at {h} (workbook)", 1 / r_ / 2, p) for h, r_, p in
zip(THRESH, RATE, ("10.6", "14.4", "19.4", "26.0", "34.5", "59.8"))] + [
c_("the threshold (workbook)", H_STAR, "2.25"),
c_("a false alarm about once in so many years, Section 15.4 and Step 5", 1 / RATE[THRESH.index(H_STAR)] / 2, "26"),
c_("Remark 15.B.1: the count raises its target under the drift to", T7 + DRIFT, "56.25"),
c_("drift: the count, cost from 58", DR["count"]["cost"], "301.992"),
c_("drift: the count, Step 5", DR["count"]["cost"], "302.0"),
c_("drift: the count, below the floor, %", pc(DR["count"]["below"]), "14.1"),
c_("drift: law never questioned, cost from 58", DR["static"]["cost"], "358.465"),
c_("drift: law never questioned, above the count (workbook)", DR["static"]["cost"] - DR["count"]["cost"], "56.47"),
c_("drift: law never questioned, above the count, Step 5", DR["static"]["cost"] - DR["count"]["cost"], "56.5"),
c_("drift: law never questioned, below the floor, %", pc(DR["static"]["below"]), "79.9"),
c_("drift: law never questioned, below, %, Section 15.4 and Step 5", pc(DR["static"]["below"]), "80"),
c_("drift: with the monitor, cost from 58", DR["monitored"]["cost"], "320.169"),
c_("drift: with the monitor, above the count (workbook)", DR["monitored"]["cost"] - DR["count"]["cost"], "18.18"),
c_("drift: with the monitor, above the count, Step 5", DR["monitored"]["cost"] - DR["count"]["cost"], "18.2"),
c_("drift: with the monitor, below the floor, %", pc(DR["monitored"]["below"]), "14.4"),
c_("drift: told of the drift, cost from 58", DR["oracle"]["cost"], "312.285"),
c_("drift: told of the drift, above the count (workbook)", DR["oracle"]["cost"] - DR["count"]["cost"], "10.29"),
c_("drift: told of the drift, above the count, Step 5", DR["oracle"]["cost"] - DR["count"]["cost"], "10.3"),
c_("drift: told of the drift, below the floor, %", pc(DR["oracle"]["below"]), "14.6"),
c_("drift: detected on every path, share", DLY["detected"], "1"),
c_("drift: delay, median quarters (workbook)", DLY["median"], "12"),
c_("drift: delay, 10th percentile (workbook)", DLY["q10"], "4"),
c_("drift: delay, 90th percentile (workbook)", DLY["q90"], "22"),
c_("drift: delay, mean (workbook)", DLY["mean"], "12.3"),
c_("abstract and Section 15.4: a median of 12 quarters", DLY["median"], "12"),
c_("no drift: the monitored twin, cost from 58", DR["monitored_nd"]["cost"], "241.638"),
c_("no drift: the twin without a monitor, cost from 58", DR["twin_nd"]["cost"], "241.837"),
c_("no drift: audits in 120 years", AUDITS, "4.6"),
flag("Theorem 15.3(i): the correct twin's innovations, mean 0 (within 0.01)", abs(WHITE["mean"]) < 0.01),
flag("Theorem 15.3(i): the correct twin's innovations, sd 1 (within 0.01)", abs(WHITE["sd"] - 1) < 0.01),
flag("Theorem 15.3(i): the correct twin's innovations, uncorrelated (within 0.01)", abs(WHITE["lag1"]) < 0.01),
flag("Theorem 15.3(i): under the drift the innovations' mean settles at the formula's (within 0.01)",
abs(WHITE["drift_mean"] - INNOV) < 0.01),
# the envelope (Step 6; Section 15.5; Proposition 15.3)
count("Proposition 15.3(ii): the guardrail, breaches on the fresh paths", sum(DES[k]["breaks"] for k in DES), 0),
c_("Section 15.5: least sure just before a survey", E2["before"], "1.58"),
c_("Section 15.5: most sure just after it", E2["after"], "1.33"),
flag("Proposition 15.3(i): the rule's and the filter's modes inside the unit circle (workbook)",
abs(mode_rule) < 1 and abs(mode_filter) < 1),
# LOS Guidance and the board
c_("LOS 15.3: the twin's target", D_["twin"]["best"], "56.70"),
c_("the board: Chapter 7's margin", T7 - FLOOR, "0.75"),
]
ok = 0
for name, got, want, tol in checks:
status = "PASS" if abs(got - want) <= tol else "FAIL"
ok += status == "PASS"
print(f"{name:88s} {got:11.4f} book {want:10.4f} {status}")
print(f"\n{ok} of {len(checks)} checkpoints PASS")a swing's variance q (workbook) 0.4500 book 0.4500 PASS a swing's standard deviation, Section 15.2 (workbook) 0.6708 book 0.6700 PASS no readings: the error's limit (workbook) 3.0189 book 3.0190 PASS no readings: error, Table 15.3 3.0189 book 3.0200 PASS no readings: error, Section 15.2 and Step 2 3.0189 book 3.0000 PASS twin: error just after a survey (workbook) 1.3333 book 1.3330 PASS twin: error a quarter later (workbook) 1.4628 book 1.4630 PASS twin: error just before the next survey (workbook) 1.5761 book 1.5760 PASS twin: the gain (workbook) 0.2844 book 0.2840 PASS twin: the gain, Section 15.2 0.2844 book 0.2800 PASS twin: error just after, Table 15.3 and Step 2 1.3333 book 1.3300 PASS twin: error just before, Table 15.3 and Step 2 1.5761 book 1.5800 PASS quarterly survey: error just after (workbook) 1.1691 book 1.1690 PASS quarterly survey: error just before (workbook) 1.3226 book 1.3230 PASS quarterly survey: the gain (workbook) 0.2187 book 0.2190 PASS quarterly survey: Table 15.3, after 1.1691 book 1.1700 PASS quarterly survey: Table 15.3, before 1.3226 book 1.3200 PASS twice as sharp: error just after (workbook) 0.8877 book 0.8880 PASS twice as sharp: error just before (workbook) 1.2609 book 1.2610 PASS twice as sharp: Table 15.3, after 0.8877 book 0.8900 PASS twice as sharp: Table 15.3, before 1.2609 book 1.2600 PASS both: error just after (workbook) 0.7914 book 0.7910 PASS both: error just before (workbook) 1.0225 book 1.0220 PASS both: Table 15.3, after 0.7914 book 0.7900 PASS both: Table 15.3, before 1.0225 book 1.0200 PASS survey read as exact: error just after (workbook) 2.5000 book 2.5000 PASS survey read as exact: error just before (workbook) 2.5281 book 2.5280 PASS survey read as exact: Table 15.3 and Step 2, before 2.5281 book 2.5300 PASS survey read as exact: good to, Section 15.2 2.5000 book 2.5000 PASS Theorem 15.1(iii): from 0 and from 10 the variance reaches the fixed point 1.0000 book 1.0000 PASS Proposition 15.1(i): precisions add at a survey (workbook) 1.0000 book 1.0000 PASS exact filter: largest difference of the means 0.0119 book 0.0120 PASS exact filter: mean difference 0.0013 book 0.0013 PASS exact filter: largest difference of the sds 0.0111 book 0.0110 PASS exact filter: survey updates compared 5700.0000 book 5700.0000 PASS count (Chapter 7): best target (workbook) 55.7500 book 55.7500 PASS twin, survey twice a year: best target (workbook) 56.7000 book 56.7000 PASS survey read as exact: best target (workbook) 57.3500 book 57.3500 PASS twin, quarterly survey: best target (workbook) 56.5000 book 56.5000 PASS twin, twice as sharp: best target (workbook) 56.4000 book 56.4000 PASS twin, both: best target (workbook) 56.2000 book 56.2000 PASS no readings after the audit: best target (workbook) 57.8000 book 57.8000 PASS every design's best target lies inside its window 1.0000 book 1.0000 PASS the 0.25 grid's least cost lies in the window: count, twin, survey as exact, no readings 1.0000 book 1.0000 PASS count: design-path cost (workbook) 230.7471 book 230.7470 PASS twin: design-path cost (workbook) 241.8519 book 241.8520 PASS twin: design-path cost, Step 6 241.8519 book 241.8500 PASS count: cost from 58 230.8437 book 230.8440 PASS count: cost from 58, Section 15.3 and Step 3 230.8437 book 230.8000 PASS count: standard error 0.0395 book 0.0400 PASS twin: cost from 58 241.9443 book 241.9440 PASS twin: cost from 58, Step 6 241.9443 book 241.9400 PASS twin: cost from 58, Section 15.3 241.9443 book 241.9000 PASS twin: standard error 0.0416 book 0.0420 PASS survey read as exact: cost from 58 246.7935 book 246.7930 PASS quarterly survey: cost from 58 239.4775 book 239.4780 PASS twice as sharp: cost from 58 237.8491 book 237.8490 PASS both: cost from 58 235.7649 book 235.7650 PASS no readings: cost from 58 256.2771 book 256.2770 PASS no readings: standard error 0.0631 book 0.0630 PASS twin: above the count (workbook) 11.1006 book 11.1010 PASS twin: above the count, Table 15.3 11.1006 book 11.1000 PASS twin: above the count, abstract, Step 3, Key Takeaways 11.1006 book 11.1000 PASS twin: margin, Section 15.3 and the board (workbook) 1.7000 book 1.7000 PASS survey read as exact: above the count (workbook) 15.9497 book 15.9500 PASS survey read as exact: above the count, Table 15.3 15.9497 book 15.9500 PASS survey read as exact: above the count, Step 3 15.9497 book 15.9000 PASS abstract: the twin costs less than the survey read as exact (workbook) 4.8491 book 4.8000 PASS quarterly survey: above the count (workbook) 8.6338 book 8.6340 PASS quarterly survey: above the count, Table 15.3 8.6338 book 8.6300 PASS quarterly survey: above the count, Step 4 8.6338 book 8.6000 PASS twice as sharp: above the count (workbook) 7.0054 book 7.0050 PASS twice as sharp: above the count, Table 15.3 7.0054 book 7.0100 PASS twice as sharp: above the count, Step 4 7.0054 book 7.0000 PASS both: above the count (workbook) 4.9212 book 4.9210 PASS both: above the count, Table 15.3 4.9212 book 4.9200 PASS both: above the count, Step 4 4.9212 book 4.9000 PASS no readings: above the count (workbook) 25.4333 book 25.4330 PASS no readings: above the count, Table 15.3 25.4333 book 25.4300 PASS no readings: above the count, Step 3 25.4333 book 25.4000 PASS twin at Chapter 7's 55.75: above the count (workbook) 16.3134 book 16.3130 PASS twin at Chapter 7's 55.75: Table 15.3 16.3134 book 16.3100 PASS twin at Chapter 7's 55.75: Section 15.3 and Step 3 16.3134 book 16.3000 PASS survey as exact at 55.75: above the count (workbook) 27.5087 book 27.5090 PASS survey as exact at 55.75: Table 15.3 27.5087 book 27.5100 PASS survey as exact at 55.75: Step 3 27.5087 book 27.5000 PASS survey as exact at 60.75: above the count (workbook) 34.3961 book 34.3960 PASS survey as exact at 60.75: Table 15.3 34.3961 book 34.4000 PASS survey as exact at 60.75: Step 3 34.3961 book 34.4000 PASS count: below the floor, % 6.0240 book 6.0000 PASS twin: below the floor, % 13.3749 book 13.4000 PASS survey read as exact: below the floor, % 13.2533 book 13.3000 PASS quarterly survey: below the floor, % 12.9642 book 13.0000 PASS twice as sharp: below the floor, % 12.0904 book 12.1000 PASS both: below the floor, % 12.2204 book 12.2000 PASS no readings: below the floor, % 17.6171 book 17.6000 PASS twin at 55.75: below the floor, % 31.3203 book 31.3000 PASS twin at 55.75: below the floor, %, Section 15.3 and Step 3 31.3203 book 31.0000 PASS survey as exact at 55.75: below the floor, % 38.6066 book 38.6000 PASS survey as exact at 60.75: below the floor, % 0.2198 book 0.2000 PASS Proposition 15.2(i): the chosen targets cost less on their design paths than on fresh ones 1.0000 book 1.0000 PASS Proposition 15.2(ii): common paths tell neighbouring targets apart better 1.0000 book 1.0000 PASS Step 6: the design-path cost is within two standard errors of the fresh 1.0000 book 1.0000 PASS Theorem 15.2(i): the count, better surveys, the twin, the survey as exact, no readings, in order of cost 1.0000 book 1.0000 PASS Theorem 15.2(iii): the count's margin, the swing's quantile (workbook) 0.7500 book 0.7500 PASS Chapter 7's critical ratio (workbook) 0.1228 book 0.1228 PASS the count is worth a year against the twin, Section 15.3 and Step 4 (workbook) 1.0455 book 1.0000 PASS a quarterly survey is worth a year, Step 4 (workbook) 0.2323 book 0.2300 PASS a survey twice as sharp is worth a year, Step 4 (workbook) 0.3857 book 0.3900 PASS both are worth a year, Step 4 (workbook) 0.5820 book 0.5800 PASS readings study, twice a year, error 0.5: best target 56.1000 book 56.1000 PASS readings study, twice a year, error 1.0: best target 56.3000 book 56.3000 PASS readings study, twice a year, error 1.25: best target 56.4000 book 56.4000 PASS readings study, twice a year, error 1.5: best target 56.4500 book 56.4500 PASS readings study, twice a year, error 2.0: best target 56.6000 book 56.6000 PASS readings study, twice a year, error 2.5: best target 56.7000 book 56.7000 PASS readings study, twice a year, error 3.0: best target 56.8000 book 56.8000 PASS readings study, twice a year, error 4.0: best target 57.0000 book 57.0000 PASS readings study, twice a year, error 5.0: best target 57.1000 book 57.1000 PASS readings study, quarterly, error 0.5: best target 55.9000 book 55.9000 PASS readings study, quarterly, error 1.0: best target 56.1000 book 56.1000 PASS readings study, quarterly, error 1.25: best target 56.2000 book 56.2000 PASS readings study, quarterly, error 1.5: best target 56.2500 book 56.2500 PASS readings study, quarterly, error 2.0: best target 56.4000 book 56.4000 PASS readings study, quarterly, error 2.5: best target 56.5000 book 56.5000 PASS readings study, quarterly, error 3.0: best target 56.6000 book 56.6000 PASS readings study, quarterly, error 4.0: best target 56.7500 book 56.7500 PASS readings study, quarterly, error 5.0: best target 56.9000 book 56.9000 PASS readings study, twice a year, error 0.5: above the count 3.9006 book 3.9000 PASS readings study, twice a year, error 1.0: above the count 6.0042 book 6.0000 PASS readings study, twice a year, error 1.25: above the count 7.0077 book 7.0100 PASS readings study, twice a year, error 1.5: above the count 7.9396 book 7.9400 PASS readings study, twice a year, error 2.0: above the count 9.6302 book 9.6300 PASS readings study, twice a year, error 2.5: above the count 11.0991 book 11.1000 PASS readings study, twice a year, error 3.0: above the count 12.3757 book 12.3800 PASS readings study, twice a year, error 4.0: above the count 14.4745 book 14.4700 PASS readings study, twice a year, error 5.0: above the count 16.1199 book 16.1200 PASS readings study, quarterly, error 0.5: above the count 2.0088 book 2.0100 PASS readings study, quarterly, error 1.0: above the count 4.0025 book 4.0000 PASS readings study, quarterly, error 1.25: above the count 4.9223 book 4.9200 PASS readings study, quarterly, error 1.5: above the count 5.7756 book 5.7800 PASS readings study, quarterly, error 2.0: above the count 7.2992 book 7.3000 PASS readings study, quarterly, error 2.5: above the count 8.6283 book 8.6300 PASS readings study, quarterly, error 3.0: above the count 9.8097 book 9.8100 PASS readings study, quarterly, error 4.0: above the count 11.8254 book 11.8300 PASS readings study, quarterly, error 5.0: above the count 13.4683 book 13.4700 PASS readings study: every best target inside its window 1.0000 book 1.0000 PASS unmonitored under the drift: bias just after a survey (workbook) 2.2099 book 2.2100 PASS unmonitored under the drift: bias just before the next (workbook) 3.0883 book 3.0880 PASS unmonitored under the drift: innovations' mean, sd (workbook) -1.0450 book -1.0450 PASS Section 15.4: overestimates by about, after 2.2099 book 2.2000 PASS Section 15.4: and before 3.0883 book 3.1000 PASS calibration: false alarms a survey at 1.5 0.0471 book 0.0471 PASS calibration: false alarms a survey at 1.75 0.0347 book 0.0347 PASS calibration: false alarms a survey at 2.0 0.0258 book 0.0258 PASS calibration: false alarms a survey at 2.25 0.0192 book 0.0192 PASS calibration: false alarms a survey at 2.5 0.0145 book 0.0145 PASS calibration: false alarms a survey at 3.0 0.0084 book 0.0084 PASS calibration: one false alarm in so many years at 1.5 (workbook) 10.6107 book 10.6000 PASS calibration: one false alarm in so many years at 1.75 (workbook) 14.4150 book 14.4000 PASS calibration: one false alarm in so many years at 2.0 (workbook) 19.4109 book 19.4000 PASS calibration: one false alarm in so many years at 2.25 (workbook) 26.0010 book 26.0000 PASS calibration: one false alarm in so many years at 2.5 (workbook) 34.4719 book 34.5000 PASS calibration: one false alarm in so many years at 3.0 (workbook) 59.8041 book 59.8000 PASS the threshold (workbook) 2.2500 book 2.2500 PASS a false alarm about once in so many years, Section 15.4 and Step 5 26.0010 book 26.0000 PASS Remark 15.B.1: the count raises its target under the drift to 56.2500 book 56.2500 PASS drift: the count, cost from 58 301.9919 book 301.9920 PASS drift: the count, Step 5 301.9919 book 302.0000 PASS drift: the count, below the floor, % 14.0680 book 14.1000 PASS drift: law never questioned, cost from 58 358.4647 book 358.4650 PASS drift: law never questioned, above the count (workbook) 56.4728 book 56.4700 PASS drift: law never questioned, above the count, Step 5 56.4728 book 56.5000 PASS drift: law never questioned, below the floor, % 79.8706 book 79.9000 PASS drift: law never questioned, below, %, Section 15.4 and Step 5 79.8706 book 80.0000 PASS drift: with the monitor, cost from 58 320.1690 book 320.1690 PASS drift: with the monitor, above the count (workbook) 18.1772 book 18.1800 PASS drift: with the monitor, above the count, Step 5 18.1772 book 18.2000 PASS drift: with the monitor, below the floor, % 14.4315 book 14.4000 PASS drift: told of the drift, cost from 58 312.2846 book 312.2850 PASS drift: told of the drift, above the count (workbook) 10.2927 book 10.2900 PASS drift: told of the drift, above the count, Step 5 10.2927 book 10.3000 PASS drift: told of the drift, below the floor, % 14.5532 book 14.6000 PASS drift: detected on every path, share 1.0000 book 1.0000 PASS drift: delay, median quarters (workbook) 12.0000 book 12.0000 PASS drift: delay, 10th percentile (workbook) 4.0000 book 4.0000 PASS drift: delay, 90th percentile (workbook) 22.0000 book 22.0000 PASS drift: delay, mean (workbook) 12.2681 book 12.3000 PASS abstract and Section 15.4: a median of 12 quarters 12.0000 book 12.0000 PASS no drift: the monitored twin, cost from 58 241.6379 book 241.6380 PASS no drift: the twin without a monitor, cost from 58 241.8370 book 241.8370 PASS no drift: audits in 120 years 4.6057 book 4.6000 PASS Theorem 15.3(i): the correct twin's innovations, mean 0 (within 0.01) 1.0000 book 1.0000 PASS Theorem 15.3(i): the correct twin's innovations, sd 1 (within 0.01) 1.0000 book 1.0000 PASS Theorem 15.3(i): the correct twin's innovations, uncorrelated (within 0.01) 1.0000 book 1.0000 PASS Theorem 15.3(i): under the drift the innovations' mean settles at the formula's (within 0.01) 1.0000 book 1.0000 PASS Proposition 15.3(ii): the guardrail, breaches on the fresh paths 0.0000 book 0.0000 PASS Section 15.5: least sure just before a survey 1.5761 book 1.5800 PASS Section 15.5: most sure just after it 1.3333 book 1.3300 PASS Proposition 15.3(i): the rule's and the filter's modes inside the unit circle (workbook) 1.0000 book 1.0000 PASS LOS 15.3: the twin's target 56.7000 book 56.7000 PASS the board: Chapter 7's margin 0.7500 book 0.7500 PASS 202 of 202 checkpoints PASS
Next. This chapter kept a declared model running: it synchronized the twin with the readings, sized its margin to its error, priced better readings, gave it a monitor, and declared the envelope inside which it acts alone. Chapter 16, Enterprise Applications and Integrated Transformation Case Studies, opens Part V, Applications and Synthesis: its Worked Example 16.1 asks Can Meridian Run Its Whole Program? AXIOM-15 runs the same twin interactively (Lab 2.15); this notebook and the workbook are its reproducible record.


