DCT Laboratory — Volume II, Chapter 14
Artificial Intelligence for Enterprise Optimization
Seed 26214 · Companion to Chapter 14 and AXIOM Lab 2.14 (module AXIOM-14) · Mirrored in DCT_V2_Ch14_Lab.xlsx
This laboratory reproduces Worked Example 14.1, Should Meridian Let an Agent Learn Its Hiring Rule?, with the book's declarations, and works the five steps of AXIOM Lab 2.14: benchmark, train by trial, price exploration, learn a simulator, and learn on the company. The law and the costs are Chapter 7's; the new declarations are the AI team's agent, its quarter-point grid, step size and defaults, the board's guardrail, and the audit committee's certification. The seeds are the book's (Online Appendix Remark 14.B.1): the agents learn in the simulator with seed 1400 and continue with seed 1401, the greedy learners use seed 1403, the bandit's pools of simulated decades seed 1407 and its runs seed 1408, the records of swings seeds 914000 + 1000n + r, the records kept under Volume I's rule seeds 1414000 + r, and learning on the company seed 1411. The lab's own seed, 26214, draws only Panel 6's illustration of the optimizer's curse. The cells take about a minute and a half on two cores, most of it Panel 2's twenty-five agents learning by trial. Every Meridian number below is the book's (Table 14.3, Figures 14.1–14.3, Online Appendix Example 14.B.2); the final cell checks them. Money is in $ million.
The declarations (Worked Example 14.1, Step 1; Online Appendix Remark 14.B.1)
Chapter 7's model, unchanged. Workforce capability is the state and the points hired and trained in a quarter the decision, at $4 million a point and at most 2.5 a quarter. Capability keeps 97.5 percent of itself and moves by the engineering office's swings : , with = 1.5, 0.75, 0, or points lost with chances 0.05, 0.20, 0.50, 0.20 and 0.05 (its record of 40 quarters: counts 2, 8, 20, 8 and 2). A point short of the floor of 55 for a quarter costs $1.6 million, so a quarter's value is , and the board discounts at a quarter. Chapter 7's best rule brings planned capability to 55.75, at $230.755 million of cost from 58 (Worked Example 7.1).
The agent and its world (new declarations). Digital Solutions' AI team proposes an agent that reads capability in quarter-points, 30 to 80 in 201 levels, and hires 0 to 2.5 points in quarter-point steps (11 hires); a decision is admissible when next quarter's capability stays within [30, 80] for every swing. The next capability , with planned capability , is placed on the grid by unbiased rounding: the upper neighbour with chance equal to its fractional position (Chapter 7's interpolation). The agent knows the costs and the discount but not the law of the swings. It learns by trial (Equation 14.2), takes a random admissible decision one time in ten, uses the step size after trials of a pair, starts untried decisions at zero, its software's default, and breaks ties toward the smallest hire. The board's guardrail: never plan below 55 unless hiring at the cap. The audit committee adopts a learned rule only after exact evaluation under the declared model, with the guardrail checked at every capability. A rule is priced exactly by one sparse linear solve of 201 equations, ; a learner's excess is its rule's cost from 58 above the best rule's.
Python Code
import math
import time
import numpy as np
import scipy.sparse as sp
import scipy.sparse.linalg as spla
import matplotlib.pyplot as plt
plt.rcParams.update({"figure.dpi": 110, "axes.spines.top": False, "axes.spines.right": False})
SEED = 26214 # the lab's seed: it draws only Panel 6's illustration of the optimizer's curse
SEED_SIM = 1400 # the book's seeds (Online Appendix Remark 14.B.1): the simulator's first million quarters,
SEED_SIM2 = 1401 # the declared-step agents' second million,
SEED_GREEDY = 1403 # the greedy learners,
SEED_POOLS = 1407 # the bandit's pools of simulated decades,
SEED_BANDIT = 1408 # the bandit's runs,
SEED_LIVE = 1411 # learning on the company, with and without the guardrail,
SEED_SWINGS = 914000 # records of swings: 914000 + 1000 n + r,
SEED_PRACTICE = 1414000 # records kept under Volume I's rule: 1414000 + r
# Chapter 7's declared model, unchanged ($ million, points of capability, quarters)
BETA, ALPHA, K, CAP, FLOOR, P_SHORT = 1.1 ** -0.25, 0.975, 4.0, 2.5, 55.0, 1.6
W = np.array([1.5, 0.75, 0.0, -0.75, -1.5]) # swings: points lost in a quarter
PW = np.array([0.05, 0.20, 0.50, 0.20, 0.05]) # their chances
COUNTS = np.array([2, 8, 20, 8, 2]) # the engineering office's record of 40 quarters
CH7 = 230.755 # Chapter 7's best rule on its 0.05-point grid, cost from 58
# the agent's world: capability in quarter-points, hires in quarter-point steps
LO, HI, DX = 30.0, 80.0, 0.25
X = LO + DX * np.arange(int(round((HI - LO) / DX)) + 1) # 201 levels
S = len(X)
H = DX * np.arange(11) # hires 0, 0.25, ..., 2.5
A = len(H)
C0 = 58.0
I0 = int(round((C0 - LO) / DX))
EPS = 0.10 # one random decision in ten
Y = ALPHA * X[:, None] + H[None, :] # planned capability y = 0.975 c + h
ADM = (Y - W.max() >= LO - 1e-9) & (Y - W.min() <= HI + 1e-9) # next capability stays in [30, 80]
R = -K * H[None, :] - P_SHORT * np.maximum(FLOOR - X, 0.0)[:, None] # a quarter's value
REACH = (ADM & (Y >= FLOOR - 1e-9)).any(axis=1) # some decision plans at least 55
GUARD = ADM & ((Y >= FLOOR - 1e-9) | (~REACH[:, None] & (H[None, :] >= CAP - 1e-9))) # the board's guardrail
def next_dist(pw=PW, w=W):
# for every pair and swing: the lower neighbour of the next capability and the chance of the upper one
out = []
for wm in w:
t = (np.clip(Y - wm, LO, HI) - LO) / DX
j = np.minimum(np.floor(t + 1e-12).astype(int), S - 2)
out.append((j, t - j))
return out
ND = next_dist()
JJ = np.stack([j for j, _ in ND]) # [swing, state, decision]
TT = np.stack([th for _, th in ND])
def transition(rule, pw=PW):
# the sparse transition matrix of a rule (a decision index for every level)
rows, cols, vals = [], [], []
s = np.arange(S)
for (j, th), pm in zip(ND, pw):
jj, tt = j[s, rule], th[s, rule]
rows += [s, s]; cols += [jj, jj + 1]; vals += [pm * (1 - tt), pm * tt]
return sp.csr_matrix((np.concatenate(vals), (np.concatenate(rows), np.concatenate(cols))), shape=(S, S))
def evaluate(rule, pw=PW):
# a rule's exact values: one sparse linear solve of 201 equations
return spla.spsolve(sp.identity(S, format="csc") - BETA * transition(rule, pw).tocsc(), R[np.arange(S), rule])
def q_of(V, pw=PW):
# decision values r + beta E V(next) for every pair (inadmissible: -inf)
EV = np.zeros((S, A))
for (j, th), pm in zip(ND, pw):
EV += pm * ((1 - th) * V[j] + th * V[j + 1])
return np.where(ADM, R + BETA * EV, -np.inf)
def solve(pw=PW, mask=ADM, rule0=None):
# policy iteration: evaluate exactly, improve greedily, keep a decision that is already best
rule = np.argmax(np.where(mask, 0.0, -np.inf), axis=1) if rule0 is None else rule0.copy()
for _ in range(200):
V = evaluate(rule, pw)
Q = np.where(mask, q_of(V, pw), -np.inf)
new = np.argmax(Q, axis=1)
new = np.where(Q[np.arange(S), rule] >= Q[np.arange(S), new] - 1e-9, rule, new)
if np.all(new == rule):
return V, rule, Q
rule = new
raise RuntimeError("policy iteration did not stop")
def solve_min(mask=ADM):
# the worst rule within a set of decisions (policy iteration that minimizes)
rule = np.argmax(np.where(mask, 0.0, -np.inf), axis=1)
for _ in range(200):
V = evaluate(rule)
Q = np.where(mask, q_of(V), np.inf)
new = np.argmin(Q, axis=1)
new = np.where(Q[np.arange(S), rule] <= Q[np.arange(S), new] + 1e-9, rule, new)
if np.all(new == rule):
return V, rule
rule = new
raise RuntimeError("policy iteration did not stop")
def evaluations_to_best(rule0, mask=ADM):
# policy iteration under the declared law from a proposed rule: evaluations until it stops
rule, n = rule0.copy(), 0
for _ in range(200):
V = evaluate(rule); n += 1
Q = np.where(mask, q_of(V), -np.inf)
new = np.argmax(Q, axis=1)
new = np.where(Q[np.arange(S), rule] >= Q[np.arange(S), new] - 1e-9, rule, new)
if np.all(new == rule):
return n, rule
rule = new
raise RuntimeError("policy iteration did not stop")
def target_rule(target, mask=ADM):
# hire to bring planned capability to the target, in quarter-points (nearest), within the limits
j = np.round(np.clip(np.round((target - ALPHA * X) / DX) * DX, 0.0, CAP) / DX).astype(int)
for s in range(S):
if not mask[s, j[s]]:
ok = np.nonzero(mask[s])[0]
j[s] = ok[np.argmin(abs(ok - j[s]))]
return j
def stationary(rule, pw=PW, iters=20000):
# the long-run distribution of capability under a rule, from 58
P = transition(rule, pw).T.tocsr()
pi = np.zeros(S); pi[I0] = 1.0
for _ in range(iters):
new = P @ pi
if abs(new - pi).sum() < 1e-13:
break
pi = new
return pi
def below_floor(rule):
# long-run share of quarters a rule starts below the floor, from 58
return float(stationary(rule)[X < FLOOR - 1e-9].sum())
def planned_target(rule):
y, h = Y[np.arange(S), rule], H[rule]
inner = (h > 1e-9) & (h < CAP - 1e-9)
return float(np.median(y[inner]))
print(f"beta = {BETA:.6f} a quarter; effective horizon 1/(1 - beta) = {1 / (1 - BETA):.2f} quarters")
print(f"swings: mean {PW @ W:.3f}, standard deviation {math.sqrt(PW @ W ** 2):.3f} points a quarter; the record's "
f"frequencies {COUNTS / COUNTS.sum()}")
print(f"the agent's grid: {S} levels from {LO:.0f} to {HI:.0f}, {A} hires from 0 to {CAP}; {ADM.sum():,} admissible "
f"decisions, {GUARD.sum():,} inside the guardrail; {int((~REACH).sum())} levels (30 to {X[~REACH].max():.2f}) cannot "
f"plan 55, and there the guardrail asks for the cap")
print(f"a quarter's value ranges from {R[ADM].min():.0f} to {R[ADM].max() + 0.0:.0f}: rbar = {R[ADM].max() - R[ADM].min():.0f}")beta = 0.976454 a quarter; effective horizon 1/(1 - beta) = 42.47 quarters swings: mean 0.000, standard deviation 0.671 points a quarter; the record's frequencies [0.05 0.2 0.5 0.2 0.05] the agent's grid: 201 levels from 30 to 80, 11 hires from 0 to 2.5; 2,113 admissible decisions, 1,152 inside the guardrail; 96 levels (30 to 53.75) cannot plan 55, and there the guardrail asks for the cap a quarter's value ranges from -50 to 0: rbar = 50
Panel 1 — Benchmark: the best rule, the guardrail and the worst rules (Worked Example 14.1, Step 1; Proposition 14.2(i); Table 14.3; Online Appendix Example 14.B.2: $231.18m from 58, $0.42m above Chapter 7's, hiring at the cap at 54.75 and below and nothing at 57.25 and above, planned capability 55.825 to 55.875, 5.82% of quarters below the floor; 2,113 decisions, 1,152 in the guardrail; Volume I's rule $246.55m, three evaluations; the worst rules $1,260.92m and, inside the guardrail, $373.96m)
AXIOM Lab 2.14, step 1. Solve the declared program on the agent's grid by policy iteration (evaluate exactly, improve greedily, until no decision changes): the benchmark the agent does not know. Count the decisions the guardrail keeps and check that it holds the best rule. Policy iteration started from Volume I's rule, hire to bring planned capability to 58 in quarter-points within the limits, reaches the best rule in three evaluations. The worst rule, found by policy iteration that minimizes, is what a learner confined to a set of decisions can lose at most, whatever it does (Proposition 14.2(i)): outside the guardrail it hires only what the limits force; inside it, as much as the limits allow.
Python Code
Vs, rule_s, Qs = solve()
pi_s = stationary(rule_s)
h_s, y_s = H[rule_s], Y[np.arange(S), rule_s]
inner = (h_s > 1e-9) & (h_s < CAP - 1e-9)
BEST = dict(V58=float(Vs[I0]), cost=-float(Vs[I0]), target=planned_target(rule_s), below=float(pi_s[X < FLOOR - 1e-9].sum()),
cap_upto=float(X[h_s >= CAP - 1e-9].max()), zero_from=float(X[h_s < 1e-9].min()),
y_lo=float(y_s[inner].min()), y_hi=float(y_s[inner].max()))
QV = q_of(Vs)
RESID = float(np.abs(QV.max(axis=1) - Vs).max()) # Bellman residual of the best rule's values
srt = np.sort(np.where(ADM, QV, -1e18), axis=1)
GAP2 = float(np.min(srt[:, -1] - srt[:, -2])) # the best decision's lead over the next
print(f"the best rule on the grid costs {BEST['cost']:.3f} from 58, {BEST['cost'] - CH7:.3f} more than Chapter 7's {CH7} "
f"because hires come in quarter-points")
print(f" it hires at the cap at {BEST['cap_upto']:.2f} and below and nothing at {BEST['zero_from']:.2f} and above; where it "
f"hires below the cap it plans {BEST['y_lo']:.3f} to {BEST['y_hi']:.3f} (about {BEST['target']:.2f}); "
f"{100 * BEST['below']:.2f}% of quarters start below the floor")
print(f" certificate: its values' Bellman residual is {RESID:.1e}; at every level the best decision leads the next by at "
f"least {GAP2:.4f}")
print(" capability " + "".join(f"{c:7.2f}" for c in (53, 54, 54.75, 55, 55.5, 56, 56.5, 57, 57.25, 58)))
ii = [int(round((c - LO) / DX)) for c in (53, 54, 54.75, 55, 55.5, 56, 56.5, 57, 57.25, 58)]
print(" hires " + "".join(f"{h_s[i]:7.2f}" for i in ii))
print(" plans " + "".join(f"{y_s[i]:7.3f}" for i in ii))
HOLD = target_rule(58.0)
V_hold = evaluate(HOLD)
N_HOLD, end_hold = evaluations_to_best(HOLD)
Vg, rule_g, _ = solve(mask=GUARD)
WORST = {}
for name, mask in (("free", ADM), ("guard", GUARD)):
Vw, rw = solve_min(mask)
pw_ = stationary(rw)
WORST[name] = dict(V58=float(Vw[I0]), excess=float(Vs[I0] - Vw[I0]), below=float(pw_[X < FLOOR - 1e-9].sum()),
capability=float(pw_ @ X), rule=rw, pi=pw_)
lim_min = np.array([np.nonzero(ADM[s])[0].min() for s in range(S)])
lim_max = np.array([np.nonzero(GUARD[s])[0].max() for s in range(S)])
print(f"\nthe guardrail keeps {GUARD.sum():,} of the {ADM.sum():,} decisions; the best rule is inside it at every level: "
f"{bool(np.all(GUARD[np.arange(S), rule_s]))}; the best rule inside it costs {-Vg[I0]:.3f}, the same rule: "
f"{bool(np.all(rule_g == rule_s))}")
print(f"{'rule':34s} {'cost from 58':>12s} {'excess':>9s} {'below floor':>12s} {'long-run capability':>20s}")
for name, v58, p_ in (("best rule (declared law)", float(Vs[I0]), pi_s),
("Volume I's rule (hold 58)", float(V_hold[I0]), stationary(HOLD)),
("worst admissible rule", WORST["free"]["V58"], WORST["free"]["pi"]),
("worst rule inside the guardrail", WORST["guard"]["V58"], WORST["guard"]["pi"])):
print(f"{name:34s} {-v58:12.2f} {float(Vs[I0]) - v58:9.2f} {100 * p_[X < FLOOR - 1e-9].sum():11.2f}% {p_ @ X:20.2f}")
print(f"policy iteration from Volume I's rule: {N_HOLD} evaluations to the best rule ({bool(np.all(end_hold == rule_s))})")
print(f"the worst admissible rule hires only what the limits force at every level: {bool(np.all(WORST['free']['rule'] == lim_min))}")
at_cap = H[WORST["guard"]["rule"]] >= CAP - 1e-9
Pw_, d_, cap40 = transition(WORST["guard"]["rule"]).T.tocsr(), np.eye(S)[I0], 0.0
for _ in range(40):
cap40 += d_[at_cap].sum() / 40; d_ = Pw_ @ d_
print(f"the worst guarded rule hires the most the guardrail allows at every level: {bool(np.all(WORST['guard']['rule'] == lim_max))}; "
f"that is the cap at {X[at_cap].max():.2f} and below, and less above, where the cap would carry capability past 80: from "
f"58 it hires at the cap in {100 * cap40:.0f}% of the first 40 quarters and {100 * WORST['guard']['pi'][at_cap].sum():.0f}% "
f"of quarters in the long run")the best rule on the grid costs 231.178 from 58, 0.423 more than Chapter 7's 230.755 because hires come in quarter-points it hires at the cap at 54.75 and below and nothing at 57.25 and above; where it hires below the cap it plans 55.825 to 55.875 (about 55.85); 5.82% of quarters start below the floor certificate: its values' Bellman residual is 2.3e-13; at every level the best decision leads the next by at least 0.0115 capability 53.00 54.00 54.75 55.00 55.50 56.00 56.50 57.00 57.25 58.00 hires 2.50 2.50 2.50 2.25 1.75 1.25 0.75 0.25 0.00 0.00 plans 54.175 55.150 55.881 55.875 55.862 55.850 55.837 55.825 55.819 56.550 the guardrail keeps 1,152 of the 2,113 decisions; the best rule is inside it at every level: True; the best rule inside it costs 231.178, the same rule: True rule cost from 58 excess below floor long-run capability best rule (declared law) 231.18 0.00 5.82% 55.84 Volume I's rule (hold 58) 246.55 15.37 0.00% 58.03 worst admissible rule 1260.92 1029.74 100.00% 31.75 worst rule inside the guardrail 373.96 142.78 0.00% 78.15 policy iteration from Volume I's rule: 3 evaluations to the best rule (True) the worst admissible rule hires only what the limits force at every level: True the worst guarded rule hires the most the guardrail allows at every level: True; that is the cap at 77.75 and below, and less above, where the cap would carry capability past 80: from 58 it hires at the cap in 75% of the first 40 quarters and 31% of quarters in the long run
Panel 2 — Train by trial in a simulator (Theorem 14.1; Worked Example 14.1, Step 2; Figure 14.1; Table 14.3; Online Appendix Example 14.B.2: with the declared step the ten agents' rules cost on average $493.4m, $56.69m, $21.97m, $1.26m and $0.55m more than the best after 1,000 to two million quarters, nine of ten within $1m at the end; decision values wrong by up to $615.6m, by $0.79m at 58; after a million quarters steps of 1/n leave $21.7m, 0.1 leaves $11.9m, and $31.9m)
AXIOM Lab 2.14, step 2. Twenty-five agents learn side by side in a simulator of the declared law, each on its own path of quarters from 58: ten with the declared step size and five with each of three common alternatives, , a constant 0.1 and , for a million quarters (seed 1400); the ten with the declared step continue for a second million (seed 1401). At 1,000, 10,000, 100,000, one million and two million quarters each agent's greedy rule is priced exactly from 58, with its long-run share of quarters below the floor. Theorem 14.1(ii) bounds a greedy rule's loss by twice the largest decision-value error over and, more sharply, by twice the discounted errors at the states the rule itself visits; the cell computes both for the ten final rules (no random draws: the sharper bound is one more linear solve per rule). This cell takes about a minute.
Python Code
class Learners:
# Tabular Q-learning (Equation 14.2) for several agents at once, each on its own path of quarters in the simulator,
# each with a step-size rule: 0 declared 1/(1 + (1 - beta) n), 1 1/n, 2 constant 0.1, 3 n^-0.7
def __init__(self, kinds, eps=EPS, q0=0.0, mask=ADM, start=I0):
n = len(kinds)
self.kinds, self.eps, self.mask = np.asarray(kinds), eps, mask
Q = np.full((n, S, A), q0, dtype=float)
Q[:, ~mask] = -np.inf # decisions the agent cannot take
self.Qf = Q.reshape(-1)
self.Nf = np.zeros(n * S * A, dtype=np.int64) # trials of each pair
self.s = np.full(n, start, dtype=np.int64)
self.base = np.arange(n) * (S * A)
adm = [np.nonzero(mask[k])[0] for k in range(S)]
self.n_adm = np.array([len(x) for x in adm])
pad = np.zeros((S, A), dtype=np.int64)
for k in range(S):
pad[k, :self.n_adm[k]] = adm[k]
self.padf = pad.reshape(-1)
self.t = 0
def Q(self):
return self.Qf.reshape(len(self.kinds), S, A)
def subset(self, sel):
n = len(self.kinds)
Q, N = self.Q()[sel].copy(), self.Nf.reshape(n, S, A)[sel].copy()
self.kinds, self.s = self.kinds[sel], self.s[sel]
self.Qf, self.Nf = Q.reshape(-1), N.reshape(-1)
self.base = np.arange(len(self.kinds)) * (S * A)
def run(self, steps, rng, checkpoints=(), chunk=20000):
out, cps = {}, sorted(c for c in checkpoints if c > self.t)
arA = np.arange(A)
JJf, TTf, Rf = JJ.reshape(-1), TT.reshape(-1), R.reshape(-1)
cum, c1 = np.cumsum(PW), 1.0 - BETA
k0, k1, k2 = self.kinds == 0, self.kinds == 1, self.kinds == 2
end = self.t + steps
while self.t < end:
m = min(chunk, end - self.t, (cps[0] - self.t) if cps else chunk)
n = len(self.kinds)
U1 = rng.random((m, n)) < self.eps # explore this quarter?
U2 = rng.random((m, n)) # which decision, if so
SW = np.minimum(np.searchsorted(cum, rng.random((m, n)), side="right"), len(PW) - 1) * (S * A) # the swing
U4 = rng.random((m, n)) # the rounding of the next capability
s = self.s
for k in range(m):
sA = s * A
row = self.base + sA
a = self.Qf[row[:, None] + arA].argmax(1) # greedy; ties to the smallest hire
ex = U1[k]
if ex.any():
a = np.where(ex, self.padf[sA + (U2[k] * self.n_adm[s]).astype(np.int64)], a)
sa = sA + a
jx = SW[k] + sa
s2 = JJf[jx] + (U4[k] < TTf[jx])
idx = row + a
self.Nf[idx] += 1
nn = self.Nf[idx]
alpha = np.where(k0, 1.0 / (1.0 + c1 * nn), np.where(k1, 1.0 / nn, np.where(k2, 0.1, nn ** -0.7)))
q2 = self.Qf[(self.base + s2 * A)[:, None] + arA].max(1)
self.Qf[idx] += alpha * (Rf[sa] + BETA * q2 - self.Qf[idx]) # Equation 14.2
s = s2
self.s = s
self.t += m
if cps and self.t == cps[0]:
Q = self.Q()
out[self.t] = (np.argmax(np.where(self.mask[None], Q, -np.inf), axis=2), Q.copy(),
(self.Nf.reshape(len(self.kinds), S, A) > 0).sum(axis=(1, 2)))
cps.pop(0)
return out
t0 = time.time()
KINDS = ["declared", "1/n", "constant 0.1", "1/n^0.7"]
BIG = 1_000_000
KIND_OF = [0] * 10 + [1] * 5 + [2] * 5 + [3] * 5
SIM = Learners(KIND_OF)
RES = SIM.run(BIG, np.random.default_rng(SEED_SIM), checkpoints=[1000, 10000, 100000, BIG])
SIM.subset(np.arange(10)) # the ten declared-step agents continue
RES.update(SIM.run(BIG, np.random.default_rng(SEED_SIM2), checkpoints=[2 * BIG]))
print(f"{len(KIND_OF)} agents x {BIG:,} quarters and 10 x {BIG:,} more (seeds {SEED_SIM}, {SEED_SIM2}): {time.time() - t0:.0f} s")
QL = {name: {} for name in KINDS}
for c, (rules, Qall, visited) in RES.items():
kk = np.asarray(KIND_OF)[:len(rules)]
for ki, name in enumerate(KINDS):
sel = np.nonzero(kk == ki)[0]
if len(sel) == 0:
continue
ex = np.array([Vs[I0] - evaluate(rules[i])[I0] for i in sel])
row = dict(excess=ex, mean=float(ex.mean()), q90=float(np.quantile(ex, 0.9)), lo=float(ex.min()),
hi=float(ex.max()), within1=int(np.sum(ex <= 1.0)), visited=float(visited[sel].mean()))
if name == "declared":
row["below"] = float(np.median([below_floor(rules[i]) for i in sel]))
if name == "declared" and c >= BIG:
err = np.abs(Qall[sel][:, ADM] - Qs[ADM][None]) # admissible pairs only
row["qerr"] = float(err.max())
i_, p_ = np.unravel_index(int(np.argmax(err)), err.shape)
row["qerr_at"] = float(X[np.nonzero(ADM)[0][p_]])
row["qerr58"] = float(np.abs(Qall[sel][:, I0, :] - Qs[None, I0, :])[:, ADM[I0]].max())
QL[name][c] = row
print("mean excess from 58 ($m) by step size and simulated quarters (90th percentile; pairs tried, mean)")
for name in KINDS:
print(f" {name:13s} " + " ".join(f"{c:>9,}: {QL[name][c]['mean']:7.2f} ({QL[name][c]['q90']:6.2f}; {QL[name][c]['visited']:5.0f})"
for c in sorted(QL[name])))
D2 = QL["declared"][2 * BIG]
print(f"declared step, every agent's excess, sorted: after 2,000,000 " + ", ".join(f"{x:.2f}" for x in np.sort(D2["excess"]))
+ f"; {D2['within1']} of 10 within $1m")
print(" median long-run share below the floor: " + ", ".join(f"{100 * QL['declared'][c]['below']:.1f}% ({c:,})"
for c in sorted(QL["declared"])))
print(f"largest decision-value error {D2['qerr']:.2f} (at capability {D2['qerr_at']:.2f}); uniform bound "
f"2 x {D2['qerr']:.2f} / (1 - beta) = {2 * D2['qerr'] / (1 - BETA):,.0f}; error at 58 "
f"{QL['declared'][BIG]['qerr58']:.2f} after a million and {D2['qerr58']:.2f} after two")
# Theorem 14.1(ii), more sharply: twice the discounted errors at the states the learned rule visits from 58
FINAL, QFIN = RES[2 * BIG][0], RES[2 * BIG][1]
SHARP = []
for i in range(10):
eps_s = np.where(ADM, np.abs(QFIN[i] - np.where(ADM, Qs, 0.0)), 0.0).max(axis=1) # error at each level
occ = spla.spsolve((sp.identity(S, format="csc") - BETA * transition(FINAL[i]).tocsc()).T.tocsc(), np.eye(S)[I0])
SHARP.append(2 * float(occ @ eps_s))
print("the sharper bound for the ten final rules, from 58: " + ", ".join(f"{x:.1f}" for x in SHARP)
+ f" (their excess {D2['lo']:.2f} to {D2['hi']:.2f})")
fig, (ax, bx) = plt.subplots(1, 2, figsize=(7.6, 3.6))
STY = {"declared": ("#1F4E79", "-", "declared step"), "1/n": ("#E65100", "--", "step 1/n"),
"constant 0.1": ("#2E7D32", ":", "step 0.1"), "1/n^0.7": ("#6A1B9A", "-.", "step 1/n^0.7")}
for name, (col, ls, lab) in STY.items():
cs = sorted(QL[name])
ax.plot(cs, [QL[name][c]["mean"] for c in cs], color=col, ls=ls, lw=1.6, marker="o", ms=3, label=lab)
ax.axhline(1.0, color="#999999", lw=0.8, ls=":")
ax.text(1.0e3, 1.15, "\\$1m", fontsize=8, color="#616161")
ax.annotate(f"\\${D2['mean']:.2f}m", xy=(2e6, D2["mean"]), xytext=(2.2e5, 0.3), fontsize=8, color="#1F4E79",
ha="center", arrowprops=dict(arrowstyle="-", lw=0.6, color="#1F4E79"))
ax.set(xscale="log", yscale="log", xlim=(7e2, 3e6), ylim=(0.15, 1000), xlabel="simulated quarters",
ylabel="mean excess cost from 58 ($ million)")
ax.set_xticks([1e3, 1e4, 1e5, 1e6], ["1,000", "10,000", "100,000", "1m"])
ax.set_yticks([1, 10, 100, 1000], ["1", "10", "100", "1,000"])
ax.set_title("(a) ten or five agents, by step size", fontsize=9)
ax.legend(frameon=False, fontsize=7.5, loc="upper right")
bx2 = bx.twinx()
bx2.fill_between(X, 0, pi_s, step="mid", color="#E0E0E0", lw=0)
bx2.set_ylim(0, 0.6); bx2.set_yticks([])
for side in ("top", "right", "left"):
bx2.spines[side].set_visible(False)
for lab, c, col, ls in (("10,000", 10000, "#999999", ":"), ("100,000", 100000, "#E65100", "--"),
("1,000,000", BIG, "#2E7D32", "-."), ("2,000,000", 2 * BIG, "#1F4E79", "-")):
bx.plot(X, np.maximum(Vs - evaluate(RES[c][0][0]), 1e-3), color=col, ls=ls, lw=1.3, label=lab)
bx.set_zorder(bx2.get_zorder() + 1); bx.patch.set_visible(False)
bx.annotate("where the best\nrule lives", xy=(57.5, 0.02), xytext=(63, 0.02), fontsize=7.5, color="#616161", va="center",
arrowprops=dict(arrowstyle="-", lw=0.6, color="#999999"))
bx.set(yscale="log", xlim=(30, 80), ylim=(0.01, 3000), xlabel="starting capability (points)", ylabel="excess cost ($ million)")
bx.set_yticks([0.01, 0.1, 1, 10, 100, 1000], ["0.01", "0.1", "1", "10", "100", "1,000"])
bx.set_title("(b) one agent, by starting capability", fontsize=9)
bx.legend(frameon=False, fontsize=7.5, loc="lower left", title="after (quarters)", title_fontsize=7.5)
fig.suptitle("A million simulated quarters to come within \\$1.26m (seed 26214)", fontsize=10.5)
plt.tight_layout(); plt.show()25 agents x 1,000,000 quarters and 10 x 1,000,000 more (seeds 1400, 1401): 64 s mean excess from 58 ($m) by step size and simulated quarters (90th percentile; pairs tried, mean) declared 1,000: 493.36 (593.58; 665) 10,000: 56.69 ( 86.11; 1448) 100,000: 21.97 ( 31.63; 2104) 1,000,000: 1.26 ( 1.75; 2110) 2,000,000: 0.55 ( 0.73; 2110) 1/n 1,000: 567.30 (620.95; 679) 10,000: 199.16 (276.07; 1053) 100,000: 31.87 ( 42.24; 1390) 1,000,000: 21.71 ( 40.05; 1550) constant 0.1 1,000: 348.74 (406.82; 733) 10,000: 95.67 (138.85; 1040) 100,000: 51.65 ( 75.10; 1629) 1,000,000: 11.90 ( 20.32; 2113) 1/n^0.7 1,000: 464.79 (536.98; 682) 10,000: 43.47 ( 88.34; 1250) 100,000: 41.20 ( 46.82; 1686) 1,000,000: 31.95 ( 47.48; 2101) declared step, every agent's excess, sorted: after 2,000,000 0.17, 0.29, 0.36, 0.38, 0.44, 0.55, 0.63, 0.65, 0.66, 1.40; 9 of 10 within $1m median long-run share below the floor: 100.0% (1,000), 50.9% (10,000), 7.9% (100,000), 7.6% (1,000,000), 7.8% (2,000,000) largest decision-value error 615.57 (at capability 30.00); uniform bound 2 x 615.57 / (1 - beta) = 52,287; error at 58 1.48 after a million and 0.79 after two the sharper bound for the ten final rules, from 58: 37.5, 40.3, 44.5, 60.4, 40.4, 40.6, 61.4, 39.4, 53.3, 66.2 (their excess 0.17 to 1.40)

Panel 3 — Price exploration: greedy learners and a bandit of target rules (Theorems 14.1(iii) and 14.2; Worked Example 14.1, Step 3; Figure 14.2(a); Table 14.3; Online Appendix Example 14.B.2: greedy learners $121.2m above the best in the median, 99.5% of quarters below the floor at about 52.6; the best target over a decade 55.75, its neighbours $0.16m and $1.01m worse; greedy choice ends on it in 20% of runs, one random target in ten in 63%, the optimistic rule in 84%, losing $1.01m, $0.55m and $0.15m a decade over the last 2,000)
AXIOM Lab 2.14, step 3. A learner that never explores. Twenty agents with no random decisions start every decision at million, far below what good decisions are worth where Meridian operates, and learn for 100,000 simulated quarters (seed 1403): each tries one decision in each capability and keeps it while it looks better than the untried ones (Theorem 14.1(iii)). A bandit. Thirteen target rules, 55 to 58 in quarter-points, each hiring to bring planned capability to its target; a trial is one decade, 40 quarters from 58, and its discounted cost. Each rule's exact decade cost is computed from its transition matrix; 50,000 simulated decades per rule form the pools (seed 1407) from which trials are drawn; 100 runs of 20,000 trials (seed 1408) of greedy choice (each target once, then the least average), one random target in ten, and the optimistic rule, the least average less , its bonus scaled to the decades' spread (Theorem 14.2(iii)). Regret is measured against the exact decade costs; by Theorem 14.2(i) it is the sum of each target's shortfall times its trials.
Python Code
t0 = time.time()
GL = Learners([0] * 20, eps=0.0, q0=-400.0)
G_RULES = list(GL.run(100000, np.random.default_rng(SEED_GREEDY), checkpoints=[100000]).values())[0][0]
G_EX, G_BELOW, G_CAP = [], [], []
for r_ in G_RULES:
p_ = stationary(r_)
G_EX.append(float(Vs[I0] - evaluate(r_)[I0])); G_BELOW.append(float(p_[X < FLOOR - 1e-9].sum())); G_CAP.append(float(p_ @ X))
GREEDY = dict(excess=np.array(G_EX), median=float(np.median(G_EX)), lo=float(min(G_EX)), hi=float(max(G_EX)),
below=float(np.median(G_BELOW)), capability=float(np.median(G_CAP)))
print(f"20 greedy learners, pessimistic start, 100,000 quarters (seed {SEED_GREEDY}): excess {GREEDY['lo']:.2f} to "
f"{GREEDY['hi']:.2f}, median {GREEDY['median']:.2f}; median long-run share below the floor {100 * GREEDY['below']:.1f}%, "
f"at about {GREEDY['capability']:.2f} points ({time.time() - t0:.0f} s)")
TARGETS = np.round(np.arange(55.0, 58.0001, 0.25), 2)
DECADE, POOL = 40, 50000
def decade_means(targets=TARGETS):
# the exact expected discounted cost of 40 quarters from 58 under each target rule
out = []
for tg in targets:
rule = target_rule(tg)
P, r, v = transition(rule), R[np.arange(S), rule], np.zeros(S)
for _ in range(DECADE):
v = r + BETA * (P @ v)
out.append(-float(v[I0]))
return np.array(out)
def decade_pool(target, n, rng):
# n simulated decades from 58 under a target rule: the discounted cost of each
rule = target_rule(target)
s, cost, cum = np.full(n, I0), np.zeros(n), np.cumsum(PW)
for k in range(DECADE):
a = rule[s]
cost -= BETA ** k * R[s, a]
sw = np.minimum(np.searchsorted(cum, rng.random(n), side="right"), len(PW) - 1)
s = JJ[sw, s, a] + (rng.random(n) < TT[sw, s, a])
return cost
def bandit(pools, means, horizon, runs, seed, sigma=12.0, eps=EPS):
# greedy, one random target in ten, and the optimistic rule over the target rules (costs: lower is better),
# all runs at once; each trial draws one of the pool's simulated decades
rng = np.random.default_rng(seed)
k, best = len(means), int(np.argmin(means))
gaps, P, out = means - means[best], np.array(pools), {}
for name in ("greedy", "epsilon-greedy", "UCB"):
n, tot, reg = np.zeros((runs, k)), np.zeros((runs, k)), np.zeros((runs, horizon))
draws = rng.integers(0, POOL, size=(horizon, runs))
u, arm_r = rng.random((horizon, runs)), rng.integers(0, k, size=(horizon, runs))
acc, rr = np.zeros(runs), np.arange(runs)
for t in range(horizon):
if t < k:
a = np.full(runs, t)
else:
mean = tot / n
if name == "UCB":
a = np.argmin(mean - sigma * np.sqrt(2.0 * math.log(t + 1) / n), axis=1)
else:
a = np.argmin(mean, axis=1)
if name == "epsilon-greedy":
a = np.where(u[t] < eps, arm_r[t], a)
n[rr, a] += 1
tot[rr, a] += P[a, draws[t]]
acc += gaps[a]
reg[:, t] = acc
out[name] = dict(regret=reg, final=np.argmax(n, axis=1), n=n, tot=tot)
return out
MEANS = decade_means()
rng_p = np.random.default_rng(SEED_POOLS)
POOLS = [decade_pool(tg, POOL, rng_p) for tg in TARGETS]
SDS = np.array([p.std() for p in POOLS])
HZ, RUNS = 20000, 100
BD = bandit(POOLS, MEANS, HZ, RUNS, SEED_BANDIT)
GAPS = MEANS - MEANS.min()
B_BEST = int(np.argmin(MEANS))
print(f"\nthe bandit: thirteen target rules, a trial is a decade from 58 (pools seed {SEED_POOLS}, runs seed {SEED_BANDIT})")
print(" target " + "".join(f"{t:8.2f}" for t in TARGETS))
print(" decade cost " + "".join(f"{m:8.2f}" for m in MEANS))
print(" its sd " + "".join(f"{x:8.1f}" for x in SDS))
print(f" best target {TARGETS[B_BEST]:.2f}; its neighbours cost {GAPS[B_BEST - 1]:.2f} and {GAPS[B_BEST + 1]:.2f} more a decade")
BAND = {}
for name, lab in (("greedy", "greedy choice"), ("epsilon-greedy", "one random target in ten"), ("UCB", "optimistic rule")):
b = BD[name]
reg = b["regret"]
BAND[name] = dict(final_best=float(np.mean(b["final"] == B_BEST)), regret=float(reg[:, -1].mean()),
per_decade=float((reg[:, -1] - reg[:, -(HZ // 10) - 1]).mean() / (HZ // 10)),
counts=b["n"].mean(axis=0), decomposition=float(GAPS @ b["n"].mean(axis=0)))
x = BAND[name]
print(f" {lab:25s} ends on the best target in {100 * x['final_best']:.0f}% of runs; total regret {x['regret']:,.0f} "
f"(sum of shortfall x trials: {x['decomposition']:,.0f}); over the last 2,000 decades {x['per_decade']:.2f} a decade")
for name, lab in (("greedy", "greedy"), ("epsilon-greedy", "10% random"), ("UCB", "optimistic")):
print(f" mean trials, {lab:11s}" + "".join(f"{x:8.0f}" for x in BAND[name]["counts"]))
UCB_BOUND = np.where(GAPS > 0, 8 * math.log(HZ) / np.maximum(GAPS / 12.0, 1e-12) ** 2 + 1 + math.pi ** 2 / 3, np.nan)
print(" 8 ln T/(gap/12)^2 + 1 + pi^2/3 " + "".join(f"{x:8.0f}" if np.isfinite(x) else " -" for x in UCB_BOUND))
print(f"twenty thousand decades is {HZ * DECADE // 4:,} years")
fig, (ax, bx) = plt.subplots(1, 2, figsize=(7.6, 3.6), gridspec_kw={"width_ratios": [1.1, 1]})
t_ = np.arange(1, HZ + 1)
for name, col, ls, lab in (("greedy", "#E65100", "--", "greedy"), ("epsilon-greedy", "#2E7D32", ":", "10% random"),
("UCB", "#1F4E79", "-", "optimistic")):
ax.plot(t_, BD[name]["regret"].mean(axis=0) / 1000.0, color=col, ls=ls, lw=1.6,
label=f"{lab} (best {100 * BAND[name]['final_best']:.0f}%)")
bx.plot(TARGETS, BAND[name]["counts"], color=col, ls=ls, lw=1.4, marker="o", ms=3, label=lab)
ax.set(xscale="log", xlim=(1, HZ), ylim=(0, 22), xlabel="trials (simulated decades)", ylabel="cumulative regret ($ billion)")
ax.set_xticks([1, 10, 100, 1000, 10000], ["1", "10", "100", "1,000", "10,000"])
ax.set_title("(a) regret against the best target", fontsize=9)
ax.legend(frameon=False, fontsize=7.5, loc="upper left")
bx.axvline(TARGETS[B_BEST], color="#999999", lw=0.8, ls=":")
bx.set(yscale="log", ylim=(0.8, 2e4), xlabel="target for planned capability (points)", ylabel="mean trials of 20,000")
bx.set_yticks([1, 10, 100, 1000, 10000], ["1", "10", "100", "1,000", "10,000"])
bx.set_title("(b) where the trials go", fontsize=9)
fig.suptitle("Optimism ends on the best target in 84% of runs (seed 26214)", fontsize=10.5)
plt.tight_layout(); plt.show()20 greedy learners, pessimistic start, 100,000 quarters (seed 1403): excess 105.66 to 133.51, median 121.19; median long-run share below the floor 99.5%, at about 52.61 points (4 s) the bandit: thirteen target rules, a trial is a decade from 58 (pools seed 1407, runs seed 1408) target 55.00 55.25 55.50 55.75 56.00 56.25 56.50 56.75 57.00 57.25 57.50 57.75 58.00 decade cost 141.49 139.96 138.95 138.79 139.80 140.84 142.06 143.62 145.20 146.78 148.37 149.95 151.53 its sd 13.6 12.8 12.1 11.7 11.4 11.3 11.1 11.0 10.9 11.0 11.0 11.0 10.9 best target 55.75; its neighbours cost 0.16 and 1.01 more a decade greedy choice ends on the best target in 20% of runs; total regret 20,372 (sum of shortfall x trials: 20,372); over the last 2,000 decades 1.01 a decade one random target in ten ends on the best target in 63% of runs; total regret 12,498 (sum of shortfall x trials: 12,498); over the last 2,000 decades 0.55 a decade optimistic rule ends on the best target in 84% of runs; total regret 7,357 (sum of shortfall x trials: 7,357); over the last 2,000 decades 0.15 a decade mean trials, greedy 607 3595 5985 3984 2403 1611 803 803 203 2 2 1 1 mean trials, 10% random 174 726 6010 10815 887 271 183 159 158 154 154 155 154 mean trials, optimistic 275 877 5602 11141 1171 478 191 98 58 39 30 23 17 8 ln T/(gap/12)^2 + 1 + pi^2/3 1579 8353 441771 - 11280 2733 1074 493 282 183 129 96 75 twenty thousand decades is 200,000 years

Panel 4 — Learn a simulator: from the record of swings, and from practice (Theorem 14.3; Worked Example 14.1, Step 4; Figure 14.3; Table 14.3; Online Appendix Example 14.B.2: from 12, 40 and 120 quarters of swings $1.17m, $0.30m and $0.05m on average, the best rule in 86, 137 and 171 of 200 records; from 40 quarters kept under Volume I's rule the first fit , effects of a point hired from to 5.35, no hiring at 55 in 76 records and the worst rule in 66, $42.7m in the median, within $1m in 28)
AXIOM Lab 2.14, step 4. The swings are an outside event, so a simulator can learn their law from a record (Theorem 14.3(ii)): draw records of 12, 40 and 120 quarters from the declared chances, 200 of each (seeds 914000 + 1000n + r), find the rule best for each record's frequencies by policy iteration, and price it under the declared law. A simulator fitted to practice is another matter (Theorem 14.3(iii)). Volume I's rule held planned capability at 58, so in its record hiring moves with capability: draw 200 records of 40 quarters from 58 under that rule (seeds 1414000 + r), fit by least squares, let the fitted simulator draw from the record's residuals, equally likely, and price the rule best for it under the declared law. Theorem 14.3(i)'s bound on what a simulator within of the true law can cost, with , the range of a quarter's value, is about $88 billion times : it guarantees consistency, not a forecast, and the declared model computes the actual losses.
Python Code
t0 = time.time()
SWINGS = {}
for nq in (12, 40, 120):
rows = []
for r_ in range(200):
counts = np.random.default_rng(SEED_SWINGS + 1000 * nq + r_).multinomial(nq, PW)
_, rule, _ = solve(pw=counts / nq, rule0=rule_s)
rows.append(dict(counts=counts, excess=float(Vs[I0] - evaluate(rule)[I0]), same=bool(np.all(rule == rule_s)),
below=below_floor(rule), tv=0.5 * float(np.abs(counts / nq - PW).sum())))
ex = np.array([x["excess"] for x in rows])
SWINGS[nq] = dict(rows=rows, excess=ex, mean=float(ex.mean()), q90=float(np.quantile(ex, 0.9)), hi=float(ex.max()),
same=int(sum(x["same"] for x in rows)), below=float(np.median([x["below"] for x in rows])),
tv=np.array([x["tv"] for x in rows]))
print("simulators learned from records of swings (200 records each):")
print(" quarters mean excess 90th pct worst best rule in median below floor mean distance from the law")
for nq, x in SWINGS.items():
print(f" {nq:8d} {x['mean']:11.2f} {x['q90']:8.2f} {x['hi']:5.2f} {x['same']:9d} {100 * x['below']:14.1f}%"
f" {x['tv'].mean():18.3f}")
RBAR = float(R[ADM].max() - R[ADM].min())
LEMMA = BETA * RBAR / (1 - BETA) ** 2
print(f"Theorem 14.3(i): beta x rbar / (1 - beta)^2 = {LEMMA:,.0f} per unit of distance delta (about ${LEMMA / 1000:.0f} billion)")
for nq in (12, 40, 120):
d5 = math.sqrt(math.log((2 ** 5 - 2) / 0.05) / (2 * nq))
print(f" {nq:3d} quarters: Theorem 14.3(ii) puts the law within {d5:.3f} with 95% confidence, so the rule best for it "
f"loses at most {2 * LEMMA * d5:,.0f}; its records' worst actual loss was {SWINGS[nq]['hi']:.2f}")
def practice_record(n_quarters, rng, target=58.0):
# a record kept while Meridian held planned capability at 58 (hiring in quarter-points)
rule, s, cum = target_rule(target), I0, np.cumsum(PW)
cs, hs, c2 = [], [], []
for _ in range(n_quarters):
a = rule[s]
sw = min(int(np.searchsorted(cum, rng.random(), side="right")), len(PW) - 1)
s2 = JJ[sw, s, a] + (rng.random() < TT[sw, s, a])
cs.append(X[s]); hs.append(H[a]); c2.append(X[s2])
s = s2
return np.array(cs), np.array(hs), np.array(c2)
def learned_dynamics_rule(coef, resid):
# the rule best for the fitted simulator c' = a + b c + d h + e, e drawn from the record's residuals, equally likely
a0, b0, d0 = coef
ybar = a0 + b0 * X[:, None] + d0 * H[None, :]
nd = []
for e in resid:
t = (np.clip(ybar + e, LO, HI) - LO) / DX
j = np.minimum(np.floor(t + 1e-12).astype(int), S - 2)
nd.append((j, t - j))
pe = np.full(len(resid), 1.0 / len(resid))
rule, ss = np.argmax(np.where(ADM, 0.0, -np.inf), axis=1), np.arange(S)
for _ in range(300):
rows, cols, vals = [], [], []
for (j, th), pm in zip(nd, pe):
jj, tt = j[ss, rule], th[ss, rule]
rows += [ss, ss]; cols += [jj, jj + 1]; vals += [pm * (1 - tt), pm * tt]
P = sp.csr_matrix((np.concatenate(vals), (np.concatenate(rows), np.concatenate(cols))), shape=(S, S))
V = spla.spsolve(sp.identity(S, format="csc") - BETA * P.tocsc(), R[ss, rule])
EV = np.zeros((S, A))
for (j, th), pm in zip(nd, pe):
EV += pm * ((1 - th) * V[j] + th * V[j + 1])
Q = np.where(ADM, R + BETA * EV, -np.inf)
new = np.argmax(Q, axis=1)
new = np.where(Q[ss, rule] >= Q[ss, new] - 1e-9, rule, new)
if np.all(new == rule):
return rule
rule = new
return rule
PRACTICE = []
for r_ in range(200):
c_, h_, c2_ = practice_record(40, np.random.default_rng(SEED_PRACTICE + r_))
Xd = np.column_stack([np.ones_like(c_), c_, h_])
coef = np.linalg.lstsq(Xd, c2_, rcond=None)[0]
rule = learned_dynamics_rule(coef, c2_ - Xd @ coef)
PRACTICE.append(dict(coef=coef, excess=float(Vs[I0] - evaluate(rule)[I0]), stop=bool(H[rule[int(round((55 - LO) / DX))]] == 0.0),
below=below_floor(rule), record=(c_, h_, c2_) if r_ == 0 else None))
P_D = np.array([x["coef"][2] for x in PRACTICE]); P_EX = np.array([x["excess"] for x in PRACTICE])
P_STOP = np.array([x["stop"] for x in PRACTICE]); P_WORST = np.abs(P_EX - WORST["free"]["excess"]) < 1e-6
c1_, h1_, c21_ = PRACTICE[0]["record"]
f1 = PRACTICE[0]["coef"]
CORR = float(np.corrcoef(c1_, h1_)[0, 1])
print(f"\nthe first record kept under Volume I's rule (seed {SEED_PRACTICE}): c' = {f1[0]:.2f} + {f1[1]:.3f} c + {f1[2]:.3f} h; "
f"its rule costs {PRACTICE[0]['excess']:.1f} more than the best")
print(f" hiring and capability move together: correlation {CORR:.3f} over its 40 quarters (capability {c1_.min():.2f} to "
f"{c1_.max():.2f}, hires {h1_.min():.2f} to {h1_.max():.2f})")
print(f"200 records: the effect of a point hired runs from {P_D.min():.2f} to {P_D.max():.2f} (truth 1); the rule hires "
f"nothing at 55 in {P_STOP.sum()} records, of which {int((P_WORST & P_STOP).sum())} give the worst admissible rule "
f"({WORST['free']['excess']:.2f}; in {P_WORST.sum()} records in all); within $1m of the best in {int((P_EX <= 1.0).sum())}; "
f"median excess {np.median(P_EX):.2f}, 90th percentile {np.quantile(P_EX, 0.9):.2f}; median long-run share below the "
f"floor {100 * np.median([x['below'] for x in PRACTICE]):.1f}% ({time.time() - t0:.0f} s)")
# Theorem 14.3(iii): the law c' = 58(1 - d) + 0.975 d c + d h - w agrees with the declared one on the pairs h = 58 - 0.975c
gap_rec = np.abs(ALPHA * c1_ + h1_ - 58.0)
print(f" on the record's pairs, 0.975c + h differs from 58 by {gap_rec.mean():.3f} on average (quarter-point rounding), so "
f"laws with any d differ there by |1 - d| x {gap_rec.mean():.3f}; at c = 55 with no hiring they differ by |1 - d| x "
f"{abs(ALPHA * 55 - 58):.3f}")
fig, (ax, bx) = plt.subplots(1, 2, figsize=(7.6, 3.6), gridspec_kw={"width_ratios": [1, 1.15]})
for i, nq in enumerate((12, 40, 120)):
x = SWINGS[nq]
ax.plot([i, i], [x["mean"], x["q90"]], color="#1F4E79", lw=1.2)
ax.plot([i], [x["mean"]], "o", color="#1F4E79", ms=5, label="mean" if i == 0 else None)
ax.plot([i], [x["q90"]], "_", color="#1F4E79", ms=11, mew=1.5, label="90th percentile" if i == 0 else None)
ax.plot([i], [x["hi"]], "x", color="#E65100", ms=6, label="worst record" if i == 0 else None)
ax.text(i + 0.1, x["mean"], f"\\${x['mean']:.2f}m", fontsize=8, color="#1F4E79", va="center")
ax.text(i, -0.55, f"{100 * x['same'] / 200:.0f}% best", fontsize=7.5, color="#616161", ha="center", va="top")
ax.set(xlim=(-0.6, 3.1), ylim=(-1.3, 5.8), xticks=[0, 1, 2], xticklabels=["12", "40", "120"],
xlabel="quarters in the record of swings", ylabel="excess cost from 58 ($ million)")
ax.legend(frameon=False, fontsize=7.5, loc="upper right", bbox_to_anchor=(1.0, 0.86))
ax.set_title("(a) the law of the swings, learned", fontsize=9)
bx.plot(P_D[~P_STOP], np.maximum(P_EX[~P_STOP], 0.01), "o", ms=3, color="#1F4E79", alpha=0.7, label="hires at 55")
bx.plot(P_D[P_STOP], np.maximum(P_EX[P_STOP], 0.01), "o", ms=3, color="#E65100", alpha=0.8, label="none at 55")
bx.axvline(1.0, color="#333333", lw=0.8, ls="--")
bx.text(1.12, 0.015, "truth", fontsize=8, color="#333333")
bx.set(yscale="log", xlim=(-5, 6), ylim=(0.01, 3000), xlabel="fitted effect of a point hired (points)",
ylabel="excess cost from 58 ($ million)")
bx.set_yticks([0.01, 0.1, 1, 10, 100, 1000], ["0.01", "0.1", "1", "10", "100", "1,000"])
bx.legend(frameon=False, fontsize=7.5, loc="center right")
bx.set_title("(b) the effect of hiring, fitted to practice", fontsize=9)
fig.suptitle("Learn the swings, not the effect of hiring (seed 26214)", fontsize=10.5)
plt.tight_layout(); plt.show()simulators learned from records of swings (200 records each):
quarters mean excess 90th pct worst best rule in median below floor mean distance from the law
12 1.17 5.04 5.04 86 5.8% 0.203
40 0.30 0.71 5.04 137 5.8% 0.109
120 0.05 0.12 2.12 171 5.8% 0.065
Theorem 14.3(i): beta x rbar / (1 - beta)^2 = 88,062 per unit of distance delta (about $88 billion)
12 quarters: Theorem 14.3(ii) puts the law within 0.516 with 95% confidence, so the rule best for it loses at most 90,929; its records' worst actual loss was 5.04
40 quarters: Theorem 14.3(ii) puts the law within 0.283 with 95% confidence, so the rule best for it loses at most 49,804; its records' worst actual loss was 5.04
120 quarters: Theorem 14.3(ii) puts the law within 0.163 with 95% confidence, so the rule best for it loses at most 28,754; its records' worst actual loss was 2.12
the first record kept under Volume I's rule (seed 1414000): c' = -63.46 + 2.036 c + 2.219 h; its rule costs 29.6 more than the best
hiring and capability move together: correlation -0.985 over its 40 quarters (capability 56.50 to 59.00, hires 0.50 to 2.50)
200 records: the effect of a point hired runs from -4.54 to 5.35 (truth 1); the rule hires nothing at 55 in 76 records, of which 66 give the worst admissible rule (1029.74; in 66 records in all); within $1m of the best in 28; median excess 42.70, 90th percentile 1029.74; median long-run share below the floor 8.3% (4 s)
on the record's pairs, 0.975c + h differs from 58 by 0.071 on average (quarter-point rounding), so laws with any d differ there by |1 - d| x 0.071; at c = 55 with no hiring they differ by |1 - d| x 4.375

Panel 5 — Learn on the company, inside a guardrail (Proposition 14.2(i); Worked Example 14.1, Step 5; Figure 14.2(b); Table 14.3; Online Appendix Example 14.B.2: without the guardrail $545.5m on average and $622.6m at the 90th percentile, 92.2% of quarters below the floor, capability 32.30 after ten years; inside it $1.18m and $19.04m, 17.8%, 55.73; the best rule's mean path 56.55 after a quarter and 55.84 from the third on)
AXIOM Lab 2.14, step 5. Without a simulator the agent learns on Meridian itself. Four hundred simulated companies run it for 40 quarters from 58, with the declared step, one random decision in ten and untried decisions at zero, without and with the guardrail (seed 1411 for both). Each company's cost is the value of its learning quarters plus at the capability reached, the best rule's value from there; its excess is measured against the best rule's $231.18 million. Starting untried decisions at zero makes every untried decision look best, and ties go to hiring nothing, so without the guardrail the agent hires nothing in every capability it has not yet seen. Whatever a learner does inside the guardrail, it cannot lose more than the worst rule there (Proposition 14.2(i)).
Python Code
def live(n_agents, quarters, mask, seed, eps=EPS, q0=0.0):
# Q-learning on the company for some quarters from 58, then the best rule: the value of the learning quarters plus
# beta^quarters V* at the capability reached, the share of quarters that start below the floor, and the paths
rng = np.random.default_rng(seed)
Q = np.full((n_agents, S, A), q0, dtype=float)
Q[:, ~mask] = -np.inf
N = np.zeros((n_agents, S, A), dtype=np.int64)
s, ag = np.full(n_agents, I0), np.arange(n_agents)
pv, below, path = np.zeros(n_agents), np.zeros(n_agents), np.zeros((quarters + 1, n_agents))
cum = np.cumsum(PW)
adm = [np.nonzero(mask[i])[0] for i in range(S)]
for k in range(quarters):
a = np.argmax(Q[ag, s], axis=1)
u, u2 = rng.random(n_agents), rng.random(n_agents)
for i in np.nonzero(u < eps)[0]:
ch = adm[s[i]]
a[i] = ch[int(u2[i] * len(ch))]
below += X[s] < FLOOR - 1e-9
path[k] = X[s]
r = R[s, a]
pv += BETA ** k * r
sw = np.minimum(np.searchsorted(cum, rng.random(n_agents), side="right"), len(PW) - 1)
s2 = JJ[sw, s, a] + (rng.random(n_agents) < TT[sw, s, a])
N[ag, s, a] += 1
al = 1.0 / (1.0 + (1.0 - BETA) * N[ag, s, a])
Q[ag, s, a] += al * (r + BETA * Q[ag, s2].max(axis=1) - Q[ag, s, a])
s = s2
pv += BETA ** quarters * Vs[s]
path[quarters] = X[s]
ex = Vs[I0] - pv
return dict(excess=float(ex.mean()), excess_q90=float(np.quantile(ex, 0.9)), below=float(below.sum() / (n_agents * quarters)),
final=float(X[s].mean()), mean=path.mean(axis=1), q10=np.quantile(path, 0.1, axis=1),
q90=np.quantile(path, 0.9, axis=1), all=ex)
t0 = time.time()
LIVE = dict(free=live(400, 40, ADM, SEED_LIVE), guard=live(400, 40, GUARD, SEED_LIVE))
Pb, dist, BEST_PATH = transition(rule_s).T.tocsr(), np.eye(S)[I0], []
for _ in range(41):
BEST_PATH.append(float(dist @ X)); dist = Pb @ dist
BEST_PATH = np.array(BEST_PATH)
for key, lab in (("free", "without the guardrail"), ("guard", "inside the guardrail")):
v = LIVE[key]
print(f"{lab:22s}: excess {v['excess']:7.2f} on average, {v['excess_q90']:7.2f} at the 90th percentile; "
f"{100 * v['below']:5.1f}% of quarters below the floor; capability {v['final']:.2f} after 40 quarters")
print(f"the worst rule inside the guardrail costs {WORST['guard']['excess']:.2f} more than the best: the guarded learners' "
f"mean excess is below it ({LIVE['guard']['excess'] <= WORST['guard']['excess']}), and so is every company's "
f"({bool(np.all(LIVE['guard']['all'] <= WORST['guard']['excess'] + 1e-9))}, the largest {LIVE['guard']['all'].max():.2f})")
print("the best rule's mean path from 58: " + ", ".join(f"{x:.2f}" for x in BEST_PATH[:6]) + f", ... {BEST_PATH[-1]:.2f}; "
f"long-run share below the floor {100 * BEST['below']:.1f}% ({time.time() - t0:.0f} s)")
fig, ax = plt.subplots(figsize=(7.6, 3.6))
q_ = np.arange(41)
for key, col, lab in (("free", "#E65100", "no guardrail"), ("guard", "#1F4E79", "guardrail")):
v = LIVE[key]
ax.fill_between(q_, v["q10"], v["q90"], color=col, alpha=0.15, lw=0)
ax.plot(q_, v["mean"], color=col, lw=1.8, label=f"{lab} ({100 * v['below']:.0f}% of quarters below 55)")
ax.plot(q_, BEST_PATH, color="#333333", lw=1.1, ls="--", label="best rule")
ax.axhline(FLOOR, color="#999999", lw=0.8, ls=":")
ax.text(39.5, 54.2, "floor 55", fontsize=8, color="#616161", ha="right", va="top")
ax.set(xlim=(0, 40), ylim=(28, 62), xlabel="quarter of learning on the company", ylabel="capability (points)",
title="Learning on the company: the guardrail keeps capability at 55.7 (seed 26214)")
ax.legend(frameon=False, fontsize=8, loc="lower left"); plt.tight_layout(); plt.show()without the guardrail : excess 545.54 on average, 622.59 at the 90th percentile; 92.2% of quarters below the floor; capability 32.30 after 40 quarters inside the guardrail : excess 1.18 on average, 19.04 at the 90th percentile; 17.8% of quarters below the floor; capability 55.73 after 40 quarters the worst rule inside the guardrail costs 142.78 more than the best: the guarded learners' mean excess is below it (True), and so is every company's (True, the largest 46.05) the best rule's mean path from 58: 58.00, 56.55, 55.89, 55.84, 55.84, 55.84, ... 55.84; long-run share below the floor 5.8% (0 s)

Panel 6 — Certify before adopting (Definition 14.2; Propositions 14.1(i) and 14.2(ii)–(iii); Worked Example 14.1, Step 6; Online Appendix Example 14.B.2: after two million quarters the ten rules cost $0.17m to $1.40m more than the best; each breaks the guardrail at 74 to 84 of the 201 levels, all below 54.5 and most below 45, where the best rule spends at most 2.8% of its quarters; policy iteration from each reaches the best rule in three or four evaluations)
The audit committee's certificate of a proposal (Definition 14.2) is a computation under the declared model that the board can check without trusting the source: the proposal's exact value, one linear solve; the check of the guardrail at every capability; and a bound on what any other rule could add, here the best rule itself, reached by policy iteration started from the proposal (Proposition 14.2(ii)). Proposition 14.1(i) explains why a rule that is poor where Meridian never goes costs little from 58: its excess equals times the average, over the levels the best rule visits from 58, discounted, of the advantage of the best rule's decisions under the learned rule's values. The agents' own decision values certify little: their Bellman residual is large where they seldom went. Last, the optimizer's curse (Proposition 14.2(iii)) with new draws from the lab's seed, 26214: 10,000 boards judge the thirteen target rules of Panel 3 by one simulated decade each, drawn from the pools, and adopt the cheapest-looking one.
Python Code
OCC = spla.spsolve((sp.identity(S, format="csc") - BETA * transition(rule_s).tocsc()).T.tocsc(), np.eye(S)[I0]) # best rule's discounted visits
CERT = []
for i, r_ in enumerate(FINAL):
Vr = evaluate(r_)
bad = ~GUARD[np.arange(S), r_]
n_ev, r_end = evaluations_to_best(r_)
adv = q_of(Vr)[np.arange(S), rule_s] - Vr # the best rule's advantage under the learned values
VL = np.where(ADM, QFIN[i], -np.inf).max(axis=1) # the agent's own values
e_L = float(np.abs(q_of(VL).max(axis=1) - VL).max()) # their Bellman residual
CERT.append(dict(excess=float(Vs[I0] - Vr[I0]), violations=int(bad.sum()), top=float(X[bad].max()),
below45=float(np.mean(X[bad] < 45)), best_time=float(pi_s[bad].sum()), evaluations=n_ev,
reaches=bool(np.all(r_end == rule_s)), pd=float(OCC @ adv), resid=e_L,
differ=float((1 - BETA) * OCC[r_ != rule_s].sum())))
print("agent excess levels breaking the guardrail highest share below 45 best rule's long-run share there "
"evaluations performance difference")
for i, x in enumerate(CERT):
print(f"{i + 1:5d} {x['excess']:6.2f} {x['violations']:29d} {x['top']:7.2f} {100 * x['below45']:13.0f}% "
f"{100 * x['best_time']:31.4f}% {x['evaluations']:11d} {x['pd']:10.4f}")
ex_ = np.array([x["excess"] for x in CERT])
print(f"excess {ex_.min():.2f} to {ex_.max():.2f}; guardrail broken at {min(x['violations'] for x in CERT)} to "
f"{max(x['violations'] for x in CERT)} levels, all below {max(x['top'] for x in CERT) + DX:.2f}; the best rule spends "
f"at most {100 * max(x['best_time'] for x in CERT):.1f}% of its quarters there; policy iteration reaches the best rule "
f"from every one ({all(x['reaches'] for x in CERT)}) in {min(x['evaluations'] for x in CERT)} to "
f"{max(x['evaluations'] for x in CERT)} evaluations")
print(f"Proposition 14.1(i): the performance difference equals the excess for every rule, to "
f"{max(abs(x['pd'] - x['excess']) for x in CERT):.1e}; the best rule's discounted time at levels where a learned "
f"rule decides differently: {100 * min(x['differ'] for x in CERT):.1f}% to {100 * max(x['differ'] for x in CERT):.1f}%")
e0 = CERT[0]["resid"]
print(f"the agents' own values: Bellman residual {min(x['resid'] for x in CERT):.1f} to {max(x['resid'] for x in CERT):.1f}; for "
f"agent 1 that certifies its greedy rule only within 2 beta e/(1 - beta) = {2 * BETA * e0 / (1 - BETA):,.0f}, "
f"against its exact excess of {CERT[0]['excess']:.2f}")
# the optimizer's curse: new draws with the lab's seed
rng_c = np.random.default_rng(SEED)
PO = np.array(POOLS)
pick = rng_c.integers(0, POOL, size=(10000, len(TARGETS)))
est = PO[np.arange(len(TARGETS))[None, :], pick] # each board's one decade per target
k_hat = est.argmin(axis=1) # the cheapest-looking target is adopted
fresh = PO[k_hat, rng_c.integers(0, POOL, size=10000)] # a fresh decade of the adopted target
CURSE = dict(estimate=float(est[np.arange(10000), k_hat].mean()), truth=float(MEANS[k_hat].mean()), fresh=float(fresh.mean()),
best=float(MEANS.min()), share_best=float(np.mean(k_hat == B_BEST)))
print(f"\nthe optimizer's curse (new draws, seed {SEED}): the adopted target's one-decade estimate averages "
f"{CURSE['estimate']:.2f}, below even the best target's exact cost {CURSE['best']:.2f}; the adopted targets truly cost "
f"{CURSE['truth']:.2f} on average, and a fresh decade of each averages {CURSE['fresh']:.2f}; the board adopts the best "
f"target in {100 * CURSE['share_best']:.0f}% of cases")agent excess levels breaking the guardrail highest share below 45 best rule's long-run share there evaluations performance difference
1 0.17 84 53.00 69% 0.0002% 3 0.1734
2 0.29 77 52.00 70% 0.0000% 3 0.2904
3 0.65 74 54.25 69% 2.8102% 3 0.6485
4 1.40 76 53.50 70% 0.0049% 3 1.4001
5 0.38 76 53.00 71% 0.0001% 3 0.3750
6 0.63 75 52.25 68% 0.0000% 4 0.6315
7 0.55 78 52.75 72% 0.0000% 3 0.5484
8 0.36 74 51.25 73% 0.0000% 3 0.3570
9 0.44 82 53.75 68% 0.0643% 3 0.4448
10 0.66 81 53.50 69% 0.0049% 3 0.6562
excess 0.17 to 1.40; guardrail broken at 74 to 84 levels, all below 54.50; the best rule spends at most 2.8% of its quarters there; policy iteration reaches the best rule from every one (True) in 3 to 4 evaluations
Proposition 14.1(i): the performance difference equals the excess for every rule, to 5.0e-13; the best rule's discounted time at levels where a learned rule decides differently: 11.8% to 65.1%
the agents' own values: Bellman residual 200.4 to 292.4; for agent 1 that certifies its greedy rule only within 2 beta e/(1 - beta) = 23,736, against its exact excess of 0.17
the optimizer's curse (new draws, seed 26214): the adopted target's one-decade estimate averages 123.06, below even the best target's exact cost 138.79; the adopted targets truly cost 141.29 on average, and a fresh decade of each averages 141.31; the board adopts the best target in 14% of cases
Panel 7 — Many agents (Proposition 14.3; Online Appendix 14.B: two agents who gain only by choosing differently miscoordinate in every round of fictitious play while their frequencies approach the equilibrium , worth ; two units that share or hoard reach the only equilibrium, both hoarding, worth 2 to the company against 6; and Proposition 14.1(iii)'s local maximum)
Part (ii): in fictitious play each agent, every round, takes its best decision against the frequencies of the other's past decisions; here both start with weights 1 on and on and receive 1 only when their decisions differ. Part (iii): a unit that hoards is better off whatever the other does, so selfish units reach an equilibrium that is worse for the company. Proposition 14.1(iii): a rule that cannot tell two states apart takes with the same chance at both, so , computed here at Meridian's quarterly discount, and gradient ascent started below stops at the local maximum .
Python Code
w1, w2 = np.array([1.0, math.sqrt(2.0)]), np.array([1.0, math.sqrt(2.0)]) # each agent's weights on the other's A, B
rounds, same, n_a, earned = 1000, 0, 0, 0
for t in range(rounds):
d1, d2 = int(np.argmin(w1)), int(np.argmin(w2)) # each takes what the other is less likely to take
same += d1 == d2; n_a += d1 == 0; earned += d1 != d2
w1[d2] += 1.0; w2[d1] += 1.0 # each records the other's decision
FREQ_A = n_a / rounds
print(f"fictitious play, {rounds} rounds: the agents take the same decision in {same} rounds and earn {earned} in all; "
f"frequency of A {FREQ_A:.3f}; the equilibrium (1/2, 1/2) is worth {0.5 * 0.5 + 0.5 * 0.5:.2f} a round to each agent")
PAY = {("share", "share"): (3, 3), ("share", "hoard"): (0, 4), ("hoard", "share"): (4, 0), ("hoard", "hoard"): (1, 1)}
acts = ("share", "hoard")
best_resp = {o: max(acts, key=lambda a: PAY[(a, o)][0]) for o in acts}
eq = [(a, b) for a in acts for b in acts if best_resp[b] == a and best_resp[a] == b]
print(f"share or hoard: best response to sharing {best_resp['share']}, to hoarding {best_resp['hoard']}; equilibria {eq}, "
f"worth {sum(PAY[eq[0]])} to the company against {sum(PAY[('share', 'share')])} if both share")
V_th = lambda th: -th + 4 * BETA * th ** 2
th, lr = 0.10, 5.0
for _ in range(5000):
phi = math.log(th / (1 - th)) + lr * (-1 + 8 * BETA * th) * th * (1 - th) # ascent on theta = 1/(1 + e^-phi)
th = 1 / (1 + math.exp(-phi))
print(f"Proposition 14.1(iii) at beta = {BETA:.4f}: V(0) = 0 is a local maximum, V(1) = 4 beta - 1 = {V_th(1.0):.3f} the "
f"maximum; the gradient is negative below 1/(8 beta) = {1 / (8 * BETA):.3f}; ascent from 0.10 ends at {th:.5f}")fictitious play, 1000 rounds: the agents take the same decision in 1000 rounds and earn 0 in all; frequency of A 0.500; the equilibrium (1/2, 1/2) is worth 0.50 a round to each agent
share or hoard: best response to sharing hoard, to hoarding hoard; equilibria [('hoard', 'hoard')], worth 2 to the company against 6 if both share
Proposition 14.1(iii) at beta = 0.9765: V(0) = 0 is a local maximum, V(1) = 4 beta - 1 = 2.906 the maximum; the gradient is negative below 1/(8 beta) = 0.128; ascent from 0.10 ends at 0.00004
Panel 8 — What each learner learned, priced exactly (Worked Example 14.1; Table 14.3)
Table 14.3's rows from the panels above. Excess: cost from 58 above the best rule's $231.18 million, the mean (median where marked) and the 90th percentile over agents, records or companies; below floor: share of quarters that start below 55, in the long run under the learned rule (the median where marked), or over the 40 quarters on the company.
Python Code
# (learner, experience, excess, 90th percentile, below floor, marks): m the excess is a median, L the "90th percentile"
# is the largest of the 20 greedy learners, b the share below the floor is a median
T143 = [("Best rule (declared law)", "none", 0.0, 0.0, BEST["below"], ""),
("Volume I's rule (hold 58)", "none", float(Vs[I0] - V_hold[I0]), None, below_floor(HOLD), "")]
for c in (10000, 100000, BIG, 2 * BIG):
x = QL["declared"][c]
T143.append(("Q-learning, simulator", f"{c:,} quarters", x["mean"], x["q90"], x["below"], "b"))
T143.append(("Greedy, pessimistic start", "100,000 quarters", GREEDY["median"], GREEDY["hi"], GREEDY["below"], "mLb"))
for nq in (12, 40, 120):
x = SWINGS[nq]
T143.append(("Simulator from swings", f"{nq} quarters", x["mean"], x["q90"], x["below"], "b"))
T143.append(("Simulator from practice", "40 quarters", float(np.median(P_EX)), float(np.quantile(P_EX, 0.9)),
float(np.median([x["below"] for x in PRACTICE])), "mb"))
for key, lab in (("free", "On the company"), ("guard", "On the company, guardrail")):
v = LIVE[key]
T143.append((lab, "40 quarters", v["excess"], v["excess_q90"], v["below"], ""))
print(f"{'Learner':27s} {'Experience':18s} {'Excess':>10s} {'90th pct':>10s} {'Below floor (%)':>16s}")
for name, exp, m, q, b, mk in T143:
m_s = f"{m:,.2f}" + ("*" if "m" in mk else " ")
q_s = "--" if q is None else f"{q:,.2f}" + ("+" if "L" in mk else " ")
print(f"{name:27s} {exp:18s} {m_s:>10s} {q_s:>10s} {100 * b:15.1f}{'*' if 'b' in mk else ' '}")
print("* the median over agents or records; + the largest of the 20 greedy learners")Learner Experience Excess 90th pct Below floor (%) Best rule (declared law) none 0.00 0.00 5.8 Volume I's rule (hold 58) none 15.37 -- 0.0 Q-learning, simulator 10,000 quarters 56.69 86.11 50.9* Q-learning, simulator 100,000 quarters 21.97 31.63 7.9* Q-learning, simulator 1,000,000 quarters 1.26 1.75 7.6* Q-learning, simulator 2,000,000 quarters 0.55 0.73 7.8* Greedy, pessimistic start 100,000 quarters 121.19* 133.51+ 99.5* Simulator from swings 12 quarters 1.17 5.04 5.8* Simulator from swings 40 quarters 0.30 0.71 5.8* Simulator from swings 120 quarters 0.05 0.12 5.8* Simulator from practice 40 quarters 42.70* 1,029.74 8.3* On the company 40 quarters 545.54 622.59 92.2 On the company, guardrail 40 quarters 1.18 19.04 17.8 * the median over agents or records; + the largest of the 20 greedy learners
Validation — the book's numbers, and agreement with DCT_V2_Ch14_Lab.xlsx
Each checkpoint is a number printed in Chapter 14 (the abstract, Sections 14.2–14.5, Worked Example 14.1, Table 14.3, the
Key Takeaways, the LOS Guidance) or in Online Appendix Remark 14.B.1 and Example 14.B.2, with a tolerance of half a unit
in its last printed digit; a few check what a theorem or proposition promises (1 = holds). Shares are in percent. The
workbook recomputes the checkpoints marked workbook with spreadsheet formulas and shows PASS against the same engine
values on its Checks sheet.
Python Code
def bk(s):
# a number as the book prints it: its value and half a unit in its last printed digit
t = s.replace(",", "").replace("−", "-")
return float(t), 0.5 * 10.0 ** -(len(t.split(".")[1]) if "." in t else 0) + 1e-9
def c_(name, got, printed):
v, tol = bk(printed)
return (name, float(got), v, tol)
def flag(name, ok):
return (name, float(bool(ok)), 1.0, 0.0)
def count(name, got, n):
return (name, float(got), float(n), 0.0)
pc = lambda x: 100.0 * x
QD = QL["declared"]
checks = [
# the declarations and the benchmark (Step 1; Section 14.2; Remark 14.B.1; Example 14.B.2)
c_("effective horizon 1/(1 - beta), quarters (workbook)", 1 / (1 - BETA), "42"),
count("capability levels (workbook)", S, 201),
count("admissible decisions (workbook)", ADM.sum(), 2113),
count("decisions inside the guardrail (workbook)", GUARD.sum(), 1152),
flag("the guardrail holds the best rule at every level (workbook)", np.all(GUARD[np.arange(S), rule_s])),
c_("best rule: cost from 58 (workbook)", BEST["cost"], "231.18"),
c_("best rule: above Chapter 7's (workbook)", BEST["cost"] - CH7, "0.42"),
c_("best rule: planned capability, about", BEST["target"], "55.85"),
c_("best rule: hires at the cap at and below (workbook)", BEST["cap_upto"], "54.75"),
c_("best rule: hires nothing at and above (workbook)", BEST["zero_from"], "57.25"),
c_("best rule: planned capability from (workbook)", BEST["y_lo"], "55.825"),
c_("best rule: planned capability to (workbook)", BEST["y_hi"], "55.875"),
c_("best rule: quarters below the floor, % (workbook)", pc(BEST["below"]), "5.82"),
c_("best rule: quarters below the floor, %, Step 1", pc(BEST["below"]), "5.8"),
flag("best rule: Bellman residual below 1e-9 (workbook)", RESID < 1e-9),
c_("Volume I's rule: cost from 58", -V_hold[I0], "246.55"),
c_("Volume I's rule: excess, Table 14.3 (workbook)", Vs[I0] - V_hold[I0], "15.37"),
c_("Volume I's rule: quarters below the floor, %", pc(below_floor(HOLD)), "0.0"),
count("Volume I's rule: policy-iteration evaluations", N_HOLD, 3),
flag("the best rule inside the guardrail is the best rule", np.all(rule_g == rule_s) and abs(Vg[I0] - Vs[I0]) < 1e-9),
c_("worst admissible rule: cost from 58", -WORST["free"]["V58"], "1260.92"),
c_("worst admissible rule: excess (workbook)", WORST["free"]["excess"], "1029.74"),
c_("worst admissible rule: excess, Section 14.5", WORST["free"]["excess"], "1029.7"),
c_("worst admissible rule: long-run capability", WORST["free"]["capability"], "31.75"),
flag("worst admissible rule: hires only what the limits force (workbook)", np.all(WORST["free"]["rule"] == lim_min)),
flag("worst guarded rule: hires the most the guardrail allows (workbook)", np.all(WORST["guard"]["rule"] == lim_max)),
c_("worst guarded rule: cost from 58", -WORST["guard"]["V58"], "373.96"),
c_("worst guarded rule: excess (workbook)", WORST["guard"]["excess"], "142.78"),
c_("worst guarded rule: excess, Section 14.5 and Step 5", WORST["guard"]["excess"], "142.8"),
c_("worst guarded rule: long-run capability", WORST["guard"]["capability"], "78.15"),
# Q-learning in the simulator (Step 2; Section 14.2; Table 14.3; Figure 14.1; Example 14.B.2)
c_("declared step, 1,000 quarters: mean excess (workbook)", QD[1000]["mean"], "493.4"),
c_("declared step, 1,000 quarters: lowest (workbook)", QD[1000]["lo"], "299.8"),
c_("declared step, 1,000 quarters: highest (workbook)", QD[1000]["hi"], "630.1"),
c_("declared step, 1,000 quarters: pairs tried", QD[1000]["visited"], "665"),
c_("declared step, 10,000 quarters: mean excess (workbook)", QD[10000]["mean"], "56.69"),
c_("declared step, 10,000 quarters: 90th percentile (workbook)", QD[10000]["q90"], "86.11"),
c_("declared step, 10,000 quarters: lowest (workbook)", QD[10000]["lo"], "24.7"),
c_("declared step, 10,000 quarters: highest (workbook)", QD[10000]["hi"], "106.3"),
c_("declared step, 10,000 quarters: pairs tried", QD[10000]["visited"], "1448"),
c_("declared step, 10,000 quarters: below the floor, %", pc(QD[10000]["below"]), "50.9"),
c_("declared step, 100,000 quarters: mean excess (workbook)", QD[100000]["mean"], "21.97"),
c_("declared step, 100,000 quarters: mean excess, Step 2", QD[100000]["mean"], "22.0"),
c_("declared step, 100,000 quarters: 90th percentile (workbook)", QD[100000]["q90"], "31.63"),
c_("declared step, 100,000 quarters: lowest (workbook)", QD[100000]["lo"], "13.8"),
c_("declared step, 100,000 quarters: highest (workbook)", QD[100000]["hi"], "37.2"),
c_("declared step, 100,000 quarters: pairs tried", QD[100000]["visited"], "2104"),
c_("declared step, 100,000 quarters: below the floor, %", pc(QD[100000]["below"]), "7.9"),
c_("declared step, a million quarters: mean excess (workbook)", QD[BIG]["mean"], "1.26"),
c_("declared step, a million quarters: 90th percentile (workbook)", QD[BIG]["q90"], "1.75"),
c_("declared step, a million quarters: lowest (workbook)", QD[BIG]["lo"], "0.56"),
c_("declared step, a million quarters: highest (workbook)", QD[BIG]["hi"], "2.19"),
c_("declared step, a million quarters: pairs tried", QD[BIG]["visited"], "2110"),
c_("declared step, a million quarters: below the floor, %", pc(QD[BIG]["below"]), "7.6"),
c_("declared step, two million quarters: mean excess (workbook)", QD[2 * BIG]["mean"], "0.55"),
c_("declared step, two million quarters: 90th percentile (workbook)", QD[2 * BIG]["q90"], "0.73"),
c_("declared step, two million quarters: below the floor, %", pc(QD[2 * BIG]["below"]), "7.8"),
count("declared step, two million quarters: within $1m (workbook)", QD[2 * BIG]["within1"], 9),
] + [c_(f"declared step, two million quarters: {k + 1}th lowest excess (workbook)", x, p) for k, (x, p) in
enumerate(zip(np.sort(QD[2 * BIG]["excess"]), ("0.17", "0.29", "0.36", "0.38", "0.44", "0.55", "0.63", "0.65", "0.66",
"1.40")))] + [
c_("largest decision-value error", QD[2 * BIG]["qerr"], "615.6"),
flag("the largest error lies at a low capability, below 45", QD[2 * BIG]["qerr_at"] < 45),
c_("uniform bound 2 x error / (1 - beta) (workbook)", 2 * QD[2 * BIG]["qerr"] / (1 - BETA), "52287"),
c_("uniform bound, $ billion, Section 14.2 (workbook)", 2 * QD[2 * BIG]["qerr"] / (1 - BETA) / 1000, "52"),
c_("decision-value error at 58 after a million quarters", QD[BIG]["qerr58"], "1.48"),
c_("decision-value error at 58 after two million quarters", QD[2 * BIG]["qerr58"], "0.79"),
flag("Theorem 14.1(ii): the sharper bound holds for every final rule", all(s_ >= e_ for s_, e_ in zip(SHARP, QD[2 * BIG]["excess"]))),
flag("Theorem 14.1(ii): the sharper bound is below the uniform one", max(SHARP) < 2 * QD[2 * BIG]["qerr"] / (1 - BETA)),
c_("step 1/n: mean excess after 1,000 (workbook)", QL["1/n"][1000]["mean"], "567.3"),
c_("step 1/n: mean excess after 10,000 (workbook)", QL["1/n"][10000]["mean"], "199.2"),
c_("step 1/n: mean excess after 100,000 (workbook)", QL["1/n"][100000]["mean"], "31.9"),
c_("step 1/n: mean excess after a million (workbook)", QL["1/n"][BIG]["mean"], "21.7"),
c_("step 0.1: mean excess after 1,000 (workbook)", QL["constant 0.1"][1000]["mean"], "348.7"),
c_("step 0.1: mean excess after 10,000 (workbook)", QL["constant 0.1"][10000]["mean"], "95.7"),
c_("step 0.1: mean excess after 100,000 (workbook)", QL["constant 0.1"][100000]["mean"], "51.7"),
c_("step 0.1: mean excess after a million (workbook)", QL["constant 0.1"][BIG]["mean"], "11.9"),
c_("step n^-0.7: mean excess after 1,000 (workbook)", QL["1/n^0.7"][1000]["mean"], "464.8"),
c_("step n^-0.7: mean excess after 10,000 (workbook)", QL["1/n^0.7"][10000]["mean"], "43.5"),
c_("step n^-0.7: mean excess after 100,000 (workbook)", QL["1/n^0.7"][100000]["mean"], "41.2"),
c_("step n^-0.7: mean excess after a million (workbook)", QL["1/n^0.7"][BIG]["mean"], "31.9"),
c_("two million quarters, years (workbook)", 2 * BIG / 4, "500000"),
# greedy learners and the bandit (Step 3; Section 14.3; Figure 14.2(a); Example 14.B.2)
c_("greedy learners: median excess (workbook)", GREEDY["median"], "121.19"),
c_("greedy learners: median excess, Step 3", GREEDY["median"], "121.2"),
c_("greedy learners: largest excess (workbook)", GREEDY["hi"], "133.51"),
c_("greedy learners: lowest excess (workbook)", GREEDY["lo"], "105.7"),
c_("greedy learners: below the floor, %", pc(GREEDY["below"]), "99.5"),
c_("greedy learners: long-run capability", GREEDY["capability"], "52.6"),
] + [c_(f"bandit: decade cost, target {t:.2f}", m, p) for t, m, p in
zip(TARGETS, MEANS, ("141.49", "139.96", "138.95", "138.79", "139.80", "140.84", "142.06", "143.62", "145.20", "146.78",
"148.37", "149.95", "151.53"))] + [
c_(f"bandit: decade sd, target {t:.2f}", x, p) for t, x, p in
zip(TARGETS, SDS, ("13.6", "12.8", "12.1", "11.7", "11.4", "11.3", "11.1", "11.0", "10.9", "11.0", "11.0", "11.0", "10.9"))] + [
c_("bandit: best target (workbook)", TARGETS[B_BEST], "55.75"),
c_("bandit: the lower neighbour's shortfall (workbook)", GAPS[B_BEST - 1], "0.16"),
c_("bandit: the upper neighbour's shortfall (workbook)", GAPS[B_BEST + 1], "1.01"),
c_("bandit: a decade's sd under the best target, about", SDS[B_BEST], "12"),
c_("bandit: greedy ends on the best, %", pc(BAND["greedy"]["final_best"]), "20"),
c_("bandit: one in ten at random ends on the best, %", pc(BAND["epsilon-greedy"]["final_best"]), "63"),
c_("bandit: optimistic ends on the best, %", pc(BAND["UCB"]["final_best"]), "84"),
c_("bandit: greedy total regret (workbook)", BAND["greedy"]["regret"], "20372"),
c_("bandit: one in ten at random, total regret (workbook)", BAND["epsilon-greedy"]["regret"], "12498"),
c_("bandit: optimistic total regret (workbook)", BAND["UCB"]["regret"], "7357"),
c_("bandit: greedy, a decade over the last 2,000", BAND["greedy"]["per_decade"], "1.01"),
c_("bandit: one in ten, a decade over the last 2,000", BAND["epsilon-greedy"]["per_decade"], "0.55"),
c_("bandit: optimistic, a decade over the last 2,000", BAND["UCB"]["per_decade"], "0.15"),
flag("Theorem 14.2(i): regret = sum of shortfall x trials, every rule (workbook)",
all(abs(BAND[k]["regret"] - BAND[k]["decomposition"]) < 1e-6 for k in BAND)),
c_("twenty thousand decades, years (workbook)", HZ * DECADE / 4, "200000"),
# learned simulators (Step 4; Section 14.4; Figure 14.3; Table 14.3; Example 14.B.2)
c_("Theorem 14.3(i): about $88 billion times delta (workbook)", LEMMA / 1000, "88"),
] + [c_(f"swings, {nq} quarters: {k}{' (workbook)' if k != 'below the floor, %' else ''}", g, p) for nq, vals in
((12, ("1.17", "5.04", "5.04", "86", "5.8")), (40, ("0.30", "0.71", "5.04", "137", "5.8")),
(120, ("0.05", "0.12", "2.12", "171", "5.8")))
for k, g, p in zip(("mean excess", "90th percentile", "worst record", "records with the best rule", "below the floor, %"),
(SWINGS[nq]["mean"], SWINGS[nq]["q90"], SWINGS[nq]["hi"], SWINGS[nq]["same"], pc(SWINGS[nq]["below"])),
vals)] + [
c_("practice, first record: intercept (workbook)", f1[0], "-63.46"),
c_("practice, first record: capability coefficient (workbook)", f1[1], "2.036"),
c_("practice, first record: hiring coefficient (workbook)", f1[2], "2.219"),
c_("practice, first record: its rule's excess", PRACTICE[0]["excess"], "29.6"),
c_("practice: lowest effect of a point hired (workbook)", P_D.min(), "-4.54"),
c_("practice: highest effect of a point hired (workbook)", P_D.max(), "5.35"),
c_("practice: lowest effect, Section 14.4 (workbook)", P_D.min(), "-4.5"),
c_("practice: highest effect, Section 14.4 (workbook)", P_D.max(), "5.4"),
count("practice: records whose rule hires nothing at 55 (workbook)", P_STOP.sum(), 76),
count("practice: records with the worst admissible rule (workbook)", P_WORST.sum(), 66),
flag("practice: every worst rule hires nothing at 55 (workbook)", np.all(P_STOP[P_WORST])),
count("practice: records within $1m of the best (workbook)", (P_EX <= 1.0).sum(), 28),
c_("practice: median excess (workbook)", np.median(P_EX), "42.70"),
c_("practice: 90th percentile (workbook)", np.quantile(P_EX, 0.9), "1029.74"),
c_("practice: below the floor, median, %", pc(np.median([x["below"] for x in PRACTICE])), "8.3"),
# learning on the company (Step 5; Section 14.3; Figure 14.2(b); Table 14.3; Example 14.B.2)
c_("on the company: mean excess", LIVE["free"]["excess"], "545.54"),
c_("on the company: mean excess, the abstract", LIVE["free"]["excess"], "545.5"),
c_("on the company: 90th percentile", LIVE["free"]["excess_q90"], "622.59"),
c_("on the company: quarters below the floor, %", pc(LIVE["free"]["below"]), "92.2"),
c_("on the company: capability after 40 quarters", LIVE["free"]["final"], "32.30"),
c_("on the company: capability in ten years, Section 14.3", LIVE["free"]["final"], "32"),
c_("inside the guardrail: mean excess", LIVE["guard"]["excess"], "1.18"),
c_("inside the guardrail: mean excess, the abstract", LIVE["guard"]["excess"], "1.2"),
c_("inside the guardrail: 90th percentile", LIVE["guard"]["excess_q90"], "19.04"),
c_("inside the guardrail: quarters below the floor, %", pc(LIVE["guard"]["below"]), "17.8"),
c_("inside the guardrail: quarters below the floor, %, Step 5", pc(LIVE["guard"]["below"]), "18"),
c_("inside the guardrail: capability after 40 quarters", LIVE["guard"]["final"], "55.73"),
flag("Proposition 14.2(i): guarded learners lose less than the worst guarded rule (workbook)",
LIVE["guard"]["excess"] <= WORST["guard"]["excess"]),
c_("best rule's mean path after one quarter (workbook)", BEST_PATH[1], "56.55"),
flag("best rule's mean path 55.84 from the third quarter on", np.all(np.round(BEST_PATH[3:], 2) == 55.84)),
# certification (Step 6; Proposition 14.2(ii); Example 14.B.2)
c_("certify: lowest excess of the ten rules", ex_.min(), "0.17"),
c_("certify: highest excess of the ten rules", ex_.max(), "1.40"),
count("certify: fewest levels breaking the guardrail (workbook)", min(x["violations"] for x in CERT), 74),
count("certify: most levels breaking the guardrail (workbook)", max(x["violations"] for x in CERT), 84),
flag("certify: every breach below 54.5 (workbook)", max(x["top"] for x in CERT) < 54.5),
flag("certify: most breaches below 45, every rule (workbook)", min(x["below45"] for x in CERT) > 0.5),
c_("certify: best rule's long-run share at the breaches, at most, % (workbook)", pc(max(x["best_time"] for x in CERT)), "2.8"),
count("certify: fewest policy-iteration evaluations", min(x["evaluations"] for x in CERT), 3),
count("certify: most policy-iteration evaluations", max(x["evaluations"] for x in CERT), 4),
flag("certify: policy iteration from every learned rule reaches the best rule", all(x["reaches"] for x in CERT)),
flag("Proposition 14.1(i): performance difference equals the excess, every rule",
max(abs(x["pd"] - x["excess"]) for x in CERT) < 1e-8),
flag("Proposition 14.2(iii): the adopted estimate understates the cost; a fresh estimate does not",
CURSE["estimate"] < CURSE["best"] <= CURSE["truth"] and abs(CURSE["fresh"] - CURSE["truth"]) < 0.5),
# many agents (Proposition 14.3; Online Appendix 14.B)
count("fictitious play: rounds in which the agents agree (workbook)", same, rounds),
count("fictitious play: payoff earned in all (workbook)", earned, 0),
c_("fictitious play: frequency of A (workbook)", FREQ_A, "0.5"),
c_("share or hoard: the equilibrium's worth to the company (workbook)", sum(PAY[eq[0]]), "2"),
c_("share or hoard: both sharing (workbook)", sum(PAY[("share", "share")]), "6"),
flag("Proposition 14.1(iii): ascent from below 1/(8 beta) falls to zero", th < 1e-3),
# the abstract, Key Takeaways, LOS Guidance
c_("abstract: a million quarters to come within", QD[BIG]["mean"], "1.26"),
c_("Section 14.2: two million for", QD[2 * BIG]["mean"], "0.55"),
c_("Section 14.2: steps of 1/n leave", QL["1/n"][BIG]["mean"], "21.7"),
c_("Section 14.2: a constant step of 0.1 leaves", QL["constant 0.1"][BIG]["mean"], "11.9"),
c_("Section 14.3: the optimistic rule, % of runs", pc(BAND["UCB"]["final_best"]), "84"),
c_("Section 14.4: from 40 quarters of swings", SWINGS[40]["mean"], "0.30"),
count("Section 14.4: records with the worst rule", P_WORST.sum(), 66),
c_("Section 14.5: guarded learning loses", LIVE["guard"]["excess"], "1.2"),
c_("Key Takeaways: on the company", LIVE["free"]["excess"], "545.5"),
c_("LOS 14.2: values wrong elsewhere by", QD[2 * BIG]["qerr"], "615.6"),
c_("LOS 14.4: median from practice", np.median(P_EX), "42.7"),
c_("LOS 14.5: at most, whatever it does inside", WORST["guard"]["excess"], "142.8"),
]
ok = 0
for name, got, want, tol in checks:
status = "PASS" if abs(got - want) <= tol else "FAIL"
ok += status == "PASS"
print(f"{name:82s} {got:12.4f} book {want:11.4f} {status}")
print(f"\n{ok} of {len(checks)} checkpoints PASS")effective horizon 1/(1 - beta), quarters (workbook) 42.4702 book 42.0000 PASS capability levels (workbook) 201.0000 book 201.0000 PASS admissible decisions (workbook) 2113.0000 book 2113.0000 PASS decisions inside the guardrail (workbook) 1152.0000 book 1152.0000 PASS the guardrail holds the best rule at every level (workbook) 1.0000 book 1.0000 PASS best rule: cost from 58 (workbook) 231.1784 book 231.1800 PASS best rule: above Chapter 7's (workbook) 0.4234 book 0.4200 PASS best rule: planned capability, about 55.8500 book 55.8500 PASS best rule: hires at the cap at and below (workbook) 54.7500 book 54.7500 PASS best rule: hires nothing at and above (workbook) 57.2500 book 57.2500 PASS best rule: planned capability from (workbook) 55.8250 book 55.8250 PASS best rule: planned capability to (workbook) 55.8750 book 55.8750 PASS best rule: quarters below the floor, % (workbook) 5.8199 book 5.8200 PASS best rule: quarters below the floor, %, Step 1 5.8199 book 5.8000 PASS best rule: Bellman residual below 1e-9 (workbook) 1.0000 book 1.0000 PASS Volume I's rule: cost from 58 246.5478 book 246.5500 PASS Volume I's rule: excess, Table 14.3 (workbook) 15.3694 book 15.3700 PASS Volume I's rule: quarters below the floor, % 0.0000 book 0.0000 PASS Volume I's rule: policy-iteration evaluations 3.0000 book 3.0000 PASS the best rule inside the guardrail is the best rule 1.0000 book 1.0000 PASS worst admissible rule: cost from 58 1260.9217 book 1260.9200 PASS worst admissible rule: excess (workbook) 1029.7433 book 1029.7400 PASS worst admissible rule: excess, Section 14.5 1029.7433 book 1029.7000 PASS worst admissible rule: long-run capability 31.7512 book 31.7500 PASS worst admissible rule: hires only what the limits force (workbook) 1.0000 book 1.0000 PASS worst guarded rule: hires the most the guardrail allows (workbook) 1.0000 book 1.0000 PASS worst guarded rule: cost from 58 373.9562 book 373.9600 PASS worst guarded rule: excess (workbook) 142.7778 book 142.7800 PASS worst guarded rule: excess, Section 14.5 and Step 5 142.7778 book 142.8000 PASS worst guarded rule: long-run capability 78.1505 book 78.1500 PASS declared step, 1,000 quarters: mean excess (workbook) 493.3573 book 493.4000 PASS declared step, 1,000 quarters: lowest (workbook) 299.7966 book 299.8000 PASS declared step, 1,000 quarters: highest (workbook) 630.0848 book 630.1000 PASS declared step, 1,000 quarters: pairs tried 665.4000 book 665.0000 PASS declared step, 10,000 quarters: mean excess (workbook) 56.6936 book 56.6900 PASS declared step, 10,000 quarters: 90th percentile (workbook) 86.1082 book 86.1100 PASS declared step, 10,000 quarters: lowest (workbook) 24.6562 book 24.7000 PASS declared step, 10,000 quarters: highest (workbook) 106.3108 book 106.3000 PASS declared step, 10,000 quarters: pairs tried 1447.9000 book 1448.0000 PASS declared step, 10,000 quarters: below the floor, % 50.9145 book 50.9000 PASS declared step, 100,000 quarters: mean excess (workbook) 21.9709 book 21.9700 PASS declared step, 100,000 quarters: mean excess, Step 2 21.9709 book 22.0000 PASS declared step, 100,000 quarters: 90th percentile (workbook) 31.6336 book 31.6300 PASS declared step, 100,000 quarters: lowest (workbook) 13.8310 book 13.8000 PASS declared step, 100,000 quarters: highest (workbook) 37.2086 book 37.2000 PASS declared step, 100,000 quarters: pairs tried 2104.2000 book 2104.0000 PASS declared step, 100,000 quarters: below the floor, % 7.8789 book 7.9000 PASS declared step, a million quarters: mean excess (workbook) 1.2606 book 1.2600 PASS declared step, a million quarters: 90th percentile (workbook) 1.7457 book 1.7500 PASS declared step, a million quarters: lowest (workbook) 0.5606 book 0.5600 PASS declared step, a million quarters: highest (workbook) 2.1947 book 2.1900 PASS declared step, a million quarters: pairs tried 2109.9000 book 2110.0000 PASS declared step, a million quarters: below the floor, % 7.5857 book 7.6000 PASS declared step, two million quarters: mean excess (workbook) 0.5525 book 0.5500 PASS declared step, two million quarters: 90th percentile (workbook) 0.7306 book 0.7300 PASS declared step, two million quarters: below the floor, % 7.7783 book 7.8000 PASS declared step, two million quarters: within $1m (workbook) 9.0000 book 9.0000 PASS declared step, two million quarters: 1th lowest excess (workbook) 0.1734 book 0.1700 PASS declared step, two million quarters: 2th lowest excess (workbook) 0.2904 book 0.2900 PASS declared step, two million quarters: 3th lowest excess (workbook) 0.3570 book 0.3600 PASS declared step, two million quarters: 4th lowest excess (workbook) 0.3750 book 0.3800 PASS declared step, two million quarters: 5th lowest excess (workbook) 0.4448 book 0.4400 PASS declared step, two million quarters: 6th lowest excess (workbook) 0.5484 book 0.5500 PASS declared step, two million quarters: 7th lowest excess (workbook) 0.6315 book 0.6300 PASS declared step, two million quarters: 8th lowest excess (workbook) 0.6485 book 0.6500 PASS declared step, two million quarters: 9th lowest excess (workbook) 0.6562 book 0.6600 PASS declared step, two million quarters: 10th lowest excess (workbook) 1.4001 book 1.4000 PASS largest decision-value error 615.5708 book 615.6000 PASS the largest error lies at a low capability, below 45 1.0000 book 1.0000 PASS uniform bound 2 x error / (1 - beta) (workbook) 52286.8528 book 52287.0000 PASS uniform bound, $ billion, Section 14.2 (workbook) 52.2869 book 52.0000 PASS decision-value error at 58 after a million quarters 1.4771 book 1.4800 PASS decision-value error at 58 after two million quarters 0.7932 book 0.7900 PASS Theorem 14.1(ii): the sharper bound holds for every final rule 1.0000 book 1.0000 PASS Theorem 14.1(ii): the sharper bound is below the uniform one 1.0000 book 1.0000 PASS step 1/n: mean excess after 1,000 (workbook) 567.2982 book 567.3000 PASS step 1/n: mean excess after 10,000 (workbook) 199.1551 book 199.2000 PASS step 1/n: mean excess after 100,000 (workbook) 31.8727 book 31.9000 PASS step 1/n: mean excess after a million (workbook) 21.7133 book 21.7000 PASS step 0.1: mean excess after 1,000 (workbook) 348.7388 book 348.7000 PASS step 0.1: mean excess after 10,000 (workbook) 95.6730 book 95.7000 PASS step 0.1: mean excess after 100,000 (workbook) 51.6505 book 51.7000 PASS step 0.1: mean excess after a million (workbook) 11.9013 book 11.9000 PASS step n^-0.7: mean excess after 1,000 (workbook) 464.7897 book 464.8000 PASS step n^-0.7: mean excess after 10,000 (workbook) 43.4681 book 43.5000 PASS step n^-0.7: mean excess after 100,000 (workbook) 41.1986 book 41.2000 PASS step n^-0.7: mean excess after a million (workbook) 31.9493 book 31.9000 PASS two million quarters, years (workbook) 500000.0000 book 500000.0000 PASS greedy learners: median excess (workbook) 121.1860 book 121.1900 PASS greedy learners: median excess, Step 3 121.1860 book 121.2000 PASS greedy learners: largest excess (workbook) 133.5112 book 133.5100 PASS greedy learners: lowest excess (workbook) 105.6603 book 105.7000 PASS greedy learners: below the floor, % 99.4864 book 99.5000 PASS greedy learners: long-run capability 52.6117 book 52.6000 PASS bandit: decade cost, target 55.00 141.4856 book 141.4900 PASS bandit: decade cost, target 55.25 139.9627 book 139.9600 PASS bandit: decade cost, target 55.50 138.9544 book 138.9500 PASS bandit: decade cost, target 55.75 138.7937 book 138.7900 PASS bandit: decade cost, target 56.00 139.7996 book 139.8000 PASS bandit: decade cost, target 56.25 140.8386 book 140.8400 PASS bandit: decade cost, target 56.50 142.0598 book 142.0600 PASS bandit: decade cost, target 56.75 143.6230 book 143.6200 PASS bandit: decade cost, target 57.00 145.1980 book 145.2000 PASS bandit: decade cost, target 57.25 146.7817 book 146.7800 PASS bandit: decade cost, target 57.50 148.3658 book 148.3700 PASS bandit: decade cost, target 57.75 149.9502 book 149.9500 PASS bandit: decade cost, target 58.00 151.5346 book 151.5300 PASS bandit: decade sd, target 55.00 13.5600 book 13.6000 PASS bandit: decade sd, target 55.25 12.8262 book 12.8000 PASS bandit: decade sd, target 55.50 12.1283 book 12.1000 PASS bandit: decade sd, target 55.75 11.6542 book 11.7000 PASS bandit: decade sd, target 56.00 11.3557 book 11.4000 PASS bandit: decade sd, target 56.25 11.2540 book 11.3000 PASS bandit: decade sd, target 56.50 11.0609 book 11.1000 PASS bandit: decade sd, target 56.75 11.0274 book 11.0000 PASS bandit: decade sd, target 57.00 10.9430 book 10.9000 PASS bandit: decade sd, target 57.25 10.9696 book 11.0000 PASS bandit: decade sd, target 57.50 10.9621 book 11.0000 PASS bandit: decade sd, target 57.75 10.9763 book 11.0000 PASS bandit: decade sd, target 58.00 10.9023 book 10.9000 PASS bandit: best target (workbook) 55.7500 book 55.7500 PASS bandit: the lower neighbour's shortfall (workbook) 0.1607 book 0.1600 PASS bandit: the upper neighbour's shortfall (workbook) 1.0059 book 1.0100 PASS bandit: a decade's sd under the best target, about 11.6542 book 12.0000 PASS bandit: greedy ends on the best, % 20.0000 book 20.0000 PASS bandit: one in ten at random ends on the best, % 63.0000 book 63.0000 PASS bandit: optimistic ends on the best, % 84.0000 book 84.0000 PASS bandit: greedy total regret (workbook) 20372.4002 book 20372.0000 PASS bandit: one in ten at random, total regret (workbook) 12497.9564 book 12498.0000 PASS bandit: optimistic total regret (workbook) 7357.3224 book 7357.0000 PASS bandit: greedy, a decade over the last 2,000 1.0115 book 1.0100 PASS bandit: one in ten, a decade over the last 2,000 0.5456 book 0.5500 PASS bandit: optimistic, a decade over the last 2,000 0.1458 book 0.1500 PASS Theorem 14.2(i): regret = sum of shortfall x trials, every rule (workbook) 1.0000 book 1.0000 PASS twenty thousand decades, years (workbook) 200000.0000 book 200000.0000 PASS Theorem 14.3(i): about $88 billion times delta (workbook) 88.0625 book 88.0000 PASS swings, 12 quarters: mean excess (workbook) 1.1697 book 1.1700 PASS swings, 12 quarters: 90th percentile (workbook) 5.0362 book 5.0400 PASS swings, 12 quarters: worst record (workbook) 5.0362 book 5.0400 PASS swings, 12 quarters: records with the best rule (workbook) 86.0000 book 86.0000 PASS swings, 12 quarters: below the floor, % 5.8199 book 5.8000 PASS swings, 40 quarters: mean excess (workbook) 0.3019 book 0.3000 PASS swings, 40 quarters: 90th percentile (workbook) 0.7130 book 0.7100 PASS swings, 40 quarters: worst record (workbook) 5.0362 book 5.0400 PASS swings, 40 quarters: records with the best rule (workbook) 137.0000 book 137.0000 PASS swings, 40 quarters: below the floor, % 5.8199 book 5.8000 PASS swings, 120 quarters: mean excess (workbook) 0.0524 book 0.0500 PASS swings, 120 quarters: 90th percentile (workbook) 0.1158 book 0.1200 PASS swings, 120 quarters: worst record (workbook) 2.1201 book 2.1200 PASS swings, 120 quarters: records with the best rule (workbook) 171.0000 book 171.0000 PASS swings, 120 quarters: below the floor, % 5.8199 book 5.8000 PASS practice, first record: intercept (workbook) -63.4623 book -63.4600 PASS practice, first record: capability coefficient (workbook) 2.0356 book 2.0360 PASS practice, first record: hiring coefficient (workbook) 2.2194 book 2.2190 PASS practice, first record: its rule's excess 29.5559 book 29.6000 PASS practice: lowest effect of a point hired (workbook) -4.5419 book -4.5400 PASS practice: highest effect of a point hired (workbook) 5.3541 book 5.3500 PASS practice: lowest effect, Section 14.4 (workbook) -4.5419 book -4.5000 PASS practice: highest effect, Section 14.4 (workbook) 5.3541 book 5.4000 PASS practice: records whose rule hires nothing at 55 (workbook) 76.0000 book 76.0000 PASS practice: records with the worst admissible rule (workbook) 66.0000 book 66.0000 PASS practice: every worst rule hires nothing at 55 (workbook) 1.0000 book 1.0000 PASS practice: records within $1m of the best (workbook) 28.0000 book 28.0000 PASS practice: median excess (workbook) 42.6994 book 42.7000 PASS practice: 90th percentile (workbook) 1029.7433 book 1029.7400 PASS practice: below the floor, median, % 8.2885 book 8.3000 PASS on the company: mean excess 545.5385 book 545.5400 PASS on the company: mean excess, the abstract 545.5385 book 545.5000 PASS on the company: 90th percentile 622.5892 book 622.5900 PASS on the company: quarters below the floor, % 92.1750 book 92.2000 PASS on the company: capability after 40 quarters 32.2969 book 32.3000 PASS on the company: capability in ten years, Section 14.3 32.2969 book 32.0000 PASS inside the guardrail: mean excess 1.1775 book 1.1800 PASS inside the guardrail: mean excess, the abstract 1.1775 book 1.2000 PASS inside the guardrail: 90th percentile 19.0368 book 19.0400 PASS inside the guardrail: quarters below the floor, % 17.8438 book 17.8000 PASS inside the guardrail: quarters below the floor, %, Step 5 17.8438 book 18.0000 PASS inside the guardrail: capability after 40 quarters 55.7338 book 55.7300 PASS Proposition 14.2(i): guarded learners lose less than the worst guarded rule (workbook) 1.0000 book 1.0000 PASS best rule's mean path after one quarter (workbook) 56.5500 book 56.5500 PASS best rule's mean path 55.84 from the third quarter on 1.0000 book 1.0000 PASS certify: lowest excess of the ten rules 0.1734 book 0.1700 PASS certify: highest excess of the ten rules 1.4001 book 1.4000 PASS certify: fewest levels breaking the guardrail (workbook) 74.0000 book 74.0000 PASS certify: most levels breaking the guardrail (workbook) 84.0000 book 84.0000 PASS certify: every breach below 54.5 (workbook) 1.0000 book 1.0000 PASS certify: most breaches below 45, every rule (workbook) 1.0000 book 1.0000 PASS certify: best rule's long-run share at the breaches, at most, % (workbook) 2.8102 book 2.8000 PASS certify: fewest policy-iteration evaluations 3.0000 book 3.0000 PASS certify: most policy-iteration evaluations 4.0000 book 4.0000 PASS certify: policy iteration from every learned rule reaches the best rule 1.0000 book 1.0000 PASS Proposition 14.1(i): performance difference equals the excess, every rule 1.0000 book 1.0000 PASS Proposition 14.2(iii): the adopted estimate understates the cost; a fresh estimate does not 1.0000 book 1.0000 PASS fictitious play: rounds in which the agents agree (workbook) 1000.0000 book 1000.0000 PASS fictitious play: payoff earned in all (workbook) 0.0000 book 0.0000 PASS fictitious play: frequency of A (workbook) 0.5000 book 0.5000 PASS share or hoard: the equilibrium's worth to the company (workbook) 2.0000 book 2.0000 PASS share or hoard: both sharing (workbook) 6.0000 book 6.0000 PASS Proposition 14.1(iii): ascent from below 1/(8 beta) falls to zero 1.0000 book 1.0000 PASS abstract: a million quarters to come within 1.2606 book 1.2600 PASS Section 14.2: two million for 0.5525 book 0.5500 PASS Section 14.2: steps of 1/n leave 21.7133 book 21.7000 PASS Section 14.2: a constant step of 0.1 leaves 11.9013 book 11.9000 PASS Section 14.3: the optimistic rule, % of runs 84.0000 book 84.0000 PASS Section 14.4: from 40 quarters of swings 0.3019 book 0.3000 PASS Section 14.4: records with the worst rule 66.0000 book 66.0000 PASS Section 14.5: guarded learning loses 1.1775 book 1.2000 PASS Key Takeaways: on the company 545.5385 book 545.5000 PASS LOS 14.2: values wrong elsewhere by 615.5708 book 615.6000 PASS LOS 14.4: median from practice 42.6994 book 42.7000 PASS LOS 14.5: at most, whatever it does inside 142.7778 book 142.8000 PASS 211 of 211 checkpoints PASS
Next. This chapter learned rules by acting, priced every learned rule exactly under the declared law, and kept the board's commitments with a guardrail and a certificate. Chapter 15, Enterprise Digital Twins and Autonomous Enterprise Transformation, continues Part IV. AXIOM-14 runs the same instance interactively (Lab 2.14); this notebook and the workbook are its reproducible record.


