DCT Laboratory — Volume II, Chapter 13
Machine Learning for Enterprise Transformation
This laboratory demonstrates
Seed 26213 · Companion to Chapter 13 and AXIOM Lab 2.13 (module AXIOM-13) · Mirrored in DCT_V2_Ch13_Lab.xlsx
This laboratory reproduces Worked Example 13.1, Is the Market Desk's Warning Worth Paying For?, with the book's declarations, and works the five steps of AXIOM Lab 2.13: read the record, train and validate, price the warning, check and certify, and compress and explain. The law, the levers and the floor are Chapter 9's; the new declarations are the market desk's five readings, its record of 1,000 windows, the two packages and the audit committee's confidence. The seeds are the book's (Online Appendix Remark 13.B.1): Chapter 9's 100,000 paths come from Volume I, Chapter 8's stream with seed 8 and its 100,000 fresh paths from seed 26209; the desk's record uses seed 1300, the book paths' readings seed 1309, the fresh paths' readings the lab's own seed, 26213, the 20,000 fresh programs of the boosting stage curve seed 26214, and the 200 records of the study seeds 913000–913199. The learners are scikit-learn's, as in the book's engine. The cells take about a minute and a half on two cores, most of it the study of 200 records. Every Meridian number below is the book's (Tables 13.2–13.3, Figures 13.1–13.3, Online Appendix Example 13.B.2); the final cell checks them. Money is in $ million.
The declarations (Worked Example 13.1, Step 1; Online Appendix Remark 13.B.1)
Chapter 9's law, levers and floor, unchanged. Execution and market deviations move Meridian's liquidity around the Chapter 1 plan, with of spread $50 million a quarter; funding windows close at random, 0.10 a year, each removing a gamma-distributed amount (mean $120 million, standard deviation $50 million) that is never recovered. A standby line , on a $5 million grid from 0 to $400 million, covers up to of each closure, for fees of 3 percent of ; a quarter of committed delay makes the transformation's draw 4 index points shallower, at $8 million. The board's appetite: at most a 5 percent chance of a breach of the $350 million floor in some quarter of 12. A path survives a package when its critical depth, the deepest draw it can absorb without breaching the floor in any quarter, is at least . Volume I's package, a $170 million line and 0.668 quarter of delay, costs $10.44 million.
The perfect warning (Chapter 9). A path has a closure ahead when a funding window closes in its 12 quarters: under the law, with chance percent. Chapter 9 priced a perfect early warning of closures, known when the board signs its package: one package for the programs with a closure ahead and one for the rest, the 5 percent appetite counted over both (Proposition 9.3(ii)). It was worth $4.12 million.
The market desk's warning (new declarations). The treasury's market desk reads five funding-market indicators when the program starts, in standard units: Meridian's bond spread, the high-yield spread, equity volatility, banks' lending standards and the bank-funding spread. A common financial-conditions factor drives them with loadings ; programs with a closure ahead start higher by ; given closure or not, the readings say nothing more about the path, so Chapter 9's law is unchanged. Each program's readings are , with the common factor and the noises independent standard normal and for a closure ahead. The desk's record holds 1,000 three-year windows of comparable issuers, each with its readings and whether a funding window closed; it draws with chance 25.92 percent. The treasury and the transformation office will sign two packages in advance, one for "warning" and one for "all clear", with the appetite counted over both; the audit committee certifies at one-sided 95 percent confidence.
How a warning is designed and checked (Remark 13.B.1). A warning is a score and a threshold: programs scoring at or above it are warned. A group whose programs have a closure ahead with share is designed on the book's paths, weighted on the paths with a closure ahead and on the others: at each line, the least delay keeps the weighted share of paths whose critical depth falls below within the group's chance of a breach, and the cheapest line and delay are kept. The all-clear group's chance is searched on a grid of 0.25 points up to 10 percent, the warned group's fills the 5 percent appetite, and the pair with the least expected cost is kept. The value of a warning is Volume I's $10.44 million less that expected cost. For each line the paths are sorted once by critical depth, and a group's package for any share and any chance is found by bisection, for every line and every chance at once (Section 13.4, Computational box). Each design is then applied to the fresh paths, each program with its own readings: its true chance of a breach and its expected cost are their fresh averages.
Python Code
import math
import itertools
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
import sklearn
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import StratifiedKFold
plt.rcParams.update({"figure.dpi": 110, "axes.spines.top": False, "axes.spines.right": False})
SEED = 26213 # the lab's seed: the book draws the fresh paths' readings with it (Remark 13.B.1)
BOOK_SEED = 8 # Volume I, Chapter 8's stream: the book's 100,000 paths (Online Appendix 1.B.4)
FRESH_SEED = 26209 # the book's 100,000 fresh paths (Chapter 9's lab seed)
RECORD_SEED = 1300 # the desk's record; also the folds' shuffle and the boosting's random state
BOOK_READ_SEED = 1309 # the readings of the book's paths
STUDY_SEED = 913000 # the study's 200 records: seeds 913000-913199
N, Q = 100_000, 12 # paths; quarters of the program
# Chapter 9's law in USD billion, exactly as Volume I's engine draws it; every result is reported in $ million
FLOOR = 0.35 # the board's floor
A_E, SD_W = 0.6, 0.05 # E' = 0.6 E + w, w with spread $50m a quarter
LAM, J_MEAN, J_SD = 0.10, 0.12, 0.05 # closures a year; gamma sizes, mean $120m and sd $50m, never recovered
GAMMA = ((J_MEAN / J_SD) ** 2, J_SD ** 2 / J_MEAN) # the sizes' shape and scale
DEPTH0, DPQ = 28.0, 4.0 # the draw's depth (index points); a quarter of delay: 4 points shallower
FEE, VQ, ALPHA = 0.03, 8.0, 0.05 # line fees over the program; a quarter of delay ($m); the appetite
LINES = np.arange(0.0, 400.01, 5.0) # the line on a $5m grid from 0 to $400m; the delay is continuous
P_GRID = np.arange(0.0, 0.1001, 0.0025) # the all-clear group's chance of a breach: 0.25 points up to 10%
def relax(x0, x_inf, k, t):
return x_inf + (x0 - x_inf) * np.exp(-k * t)
def trough(depth, tau, t):
return depth * (t / tau) * np.exp(1.0 - t / tau)
TQ = np.arange(1, Q + 1) / 4.0 # the twelve quarters, in years
BASE = 0.62 * relax(100.0, 112.0, 0.35, TQ) / 100.0 # planned liquidity without the draw (index 100 = $0.62bn)
SHAPE = 0.62 * trough(1.0, 2.0, TQ) / 100.0 # the draw per index point of depth, deepest at two years
def draws(seed, n=N):
# Volume I's stream in its order: each quarter the execution shocks, the closure counts, then the sizes path by path
rng = np.random.default_rng(seed)
W, events = np.empty((n, Q)), []
for k in range(Q):
W[:, k] = rng.normal(0.0, SD_W, n)
count = rng.poisson(LAM / 4.0, n)
events.append((np.repeat(np.arange(n), count), rng.gamma(*GAMMA, int(count.sum()))))
E, e = np.zeros((n, Q)), np.zeros(n)
for k in range(Q):
e = A_E * e + W[:, k]
E[:, k] = e
return E, events
class Paths:
# Chapter 9's paths. A line l ($m) covers up to l of each closure; phi quarters of delay make the draw 28 - 4 phi
# deep; a path stays above the floor in every quarter exactly when the draw is no deeper than its critical depth.
def __init__(self, seed, n=N):
self.E, self.events = draws(seed, n)
self.seed, self.n = seed, n
self.ahead = np.zeros(n, bool) # a funding window closes in the program's 12 quarters
for path, _ in self.events:
self.ahead[path] = True
def lost(self, line): # cumulative closures the line does not cover (USD bn), quarter by quarter
J, j = np.zeros((self.n, Q)), np.zeros(self.n)
for k, (path, size) in enumerate(self.events):
j = j + np.bincount(path, np.maximum(size - line / 1000.0, 0.0), self.n)
J[:, k] = j
return J
def crit(self, line): # each path's critical depth under the line
return np.min((BASE[None, :] + self.E - self.lost(line) - FLOOR) / SHAPE[None, :], axis=1)
def cost(line, phi):
return FEE * line + VQ * phi
class Split:
# The book's paths split by closure ahead: for every line, the critical depths in order and the running count of
# paths with a closure ahead. A group whose programs close with share pi is designed on the paths weighted pi / n1
# on the n1 closure paths and (1 - pi) / n0 on the others (Remark 13.B.1).
def __init__(self, P, lines=LINES):
self.lines, self.y, self.n = lines, P.ahead, P.n
self.n1 = int(self.y.sum()); self.n0 = self.n - self.n1
L, n = len(lines), self.n
self.C = np.empty((L, n))
A1 = np.empty((L, n + 1), dtype=np.int32) # running count of closure paths; a spare last column
for i, l in enumerate(lines):
c = P.crit(l)
o = np.argsort(c, kind="stable")
self.C[i], A1[i, :n] = c[o], np.cumsum(self.y[o])
A1[:, n] = self.n1
self.A1, self.off = A1.ravel(), (np.arange(L, dtype=np.int32) * (n + 1))[:, None]
self.steps = [1 << s for s in range(16, -1, -1)] # 65,536 + ... + 1 >= 100,000 paths
def curve(self, pis, targets):
# The cheapest package (cost, line, delay) for groups with closure shares pis and chances of a breach targets,
# one column each. At every line the paths are in order of critical depth, and the draw may be as deep as the
# critical depth of the first path whose weighted running share, pi a1 / n1 + (1 - pi) a0 / n0 (a1 and a0
# paths up to it with and without a closure ahead), exceeds the chance. A binary search finds that path for
# every line and column at once (k: paths known to stay within the chance). inf where no package keeps it.
L, n = len(self.lines), self.n
pis = np.asarray(pis, dtype=float)[None, :]
thr = np.asarray(targets, dtype=float)[None, :] + 1e-12
k = np.zeros((L, thr.shape[1]), dtype=np.int32)
for step in self.steps:
j = k + (step - 1)
jc = np.minimum(j, n)
a1 = np.take(self.A1, jc + self.off)
cw = pis * a1 / self.n1 + (1.0 - pis) * (jc + 1 - a1) / self.n0
k += ((j < n) & (cw <= thr)) * step
d = np.where(k < n, np.minimum(DEPTH0, self.C[np.arange(L)[:, None], np.minimum(k, n - 1)]), DEPTH0)
phi = (DEPTH0 - d) / DPQ
c = np.where(d >= 0, FEE * self.lines[:, None] + VQ * phi, np.inf)
i = np.argmin(c, axis=0)
cols = np.arange(thr.shape[1])
return c[i, cols], self.lines[i], phi[i, cols]
def group(self, pi, pg):
c, l, ph = self.curve([pi], [pg])
return None if not np.isfinite(c[0]) else (float(c[0]), float(l[0]), float(ph[0]))
def two(self, share_w, pi_w, pi_c, p=ALPHA, p_grid=P_GRID):
# Two packages chosen together: the all-clear group's chance pc on the grid, the warned group's from the appetite
if share_w <= 0.0:
g = self.group(pi_c, p)
return None if g is None else dict(cost=g[0], warned=0.0, clear=g, warn=None, pc=p, pw=None)
if share_w >= 1.0:
g = self.group(pi_w, p)
return None if g is None else dict(cost=g[0], warned=1.0, clear=None, warn=g, pc=None, pw=p)
pcs = np.asarray(p_grid, dtype=float)
pws = (p - (1.0 - share_w) * pcs) / share_w
ok = pws >= 0.0
if not ok.any():
return None
pcs, pws = pcs[ok], np.minimum(pws[ok], 1.0)
m = len(pcs)
c_, l_, f_ = self.curve(np.r_[np.full(m, pi_c), np.full(m, pi_w)], np.r_[pcs, pws]) # both groups at once
cc, lc, fc, cw, lw, fw = c_[:m], l_[:m], f_[:m], c_[m:], l_[m:], f_[m:]
tot = share_w * cw + (1.0 - share_w) * cc
if not np.isfinite(tot).any():
return None
j = int(np.argmin(tot))
return dict(cost=float(tot[j]), warned=share_w, clear=(float(cc[j]), float(lc[j]), float(fc[j])),
warn=(float(cw[j]), float(lw[j]), float(fw[j])), pc=float(pcs[j]), pw=float(pws[j]))
def breaches(self, g):
# the book's paths that breach the package g = (cost, line, delay): (with a closure ahead, without)
i = int(round(g[1] / 5.0))
k = int(np.searchsorted(self.C[i], DEPTH0 - DPQ * g[2], side="left"))
a1 = int(self.A1[i * (self.n + 1) + k - 1]) if k else 0
return a1, k - a1
def chance(self, pi, g):
# the group's chance of a breach as the design weighs the paths: pi on closure paths, 1 - pi on the others
b1, b0 = self.breaches(g)
return pi * b1 / self.n1 + (1.0 - pi) * b0 / self.n0
# the desk's readings
NAMES = ["Meridian's bond spread", "high-yield spread", "equity volatility", "banks' lending standards",
"bank-funding spread"]
SHORT = ["bond", "high yield", "volatility", "lending", "funding"]
LOAD = np.array([0.9, 0.85, 0.8, 0.6, 0.3]) # loadings on the common financial-conditions factor
SHIFT = np.array([0.25, 0.25, 0.15, 0.7, 1.9]) # how much higher programs with a closure ahead start
SIGMA = np.outer(LOAD, LOAD) + np.diag(1.0 - LOAD ** 2) # the readings' covariance, given closure or not
P_CLOSE = 1.0 - math.exp(-0.10 * 3.0) # a closure in three years under Chapter 9's law
def readings(y, rng, shift=SHIFT):
# five readings per program: the common factor, each reading's own noise, and the closure shift
n = len(y)
f = rng.normal(size=n)
e = rng.normal(size=(n, 5))
return f[:, None] * LOAD[None, :] + e * np.sqrt(1.0 - LOAD ** 2)[None, :] + y[:, None] * shift[None, :]
def record(seed=RECORD_SEED, n=1000):
# a record of n windows: whether a funding window closed (chance 25.92%), then the readings, from one stream
rng = np.random.default_rng(seed)
y = rng.random(n) < P_CLOSE
return y, readings(y.astype(float), rng)
BOOK = Paths(BOOK_SEED)
SPLIT = Split(BOOK)
PI1 = SPLIT.n1 / SPLIT.n # the book paths' share with a closure ahead
FRESH = Paths(FRESH_SEED)
FRESH_CRIT = np.array([FRESH.crit(l) for l in LINES]) # every fresh path's critical depth under every line
FRESH_X = readings(FRESH.ahead.astype(float), np.random.default_rng(SEED)) # the fresh programs' own readings
def check(warned, design):
# a design on the fresh paths: each program gets its group's package; the true chance of a breach and expected cost
breach, cst = np.zeros(N, bool), 0.0
for mask, g in ((warned, design["warn"]), (~warned, design["clear"])):
if mask.sum() == 0 or g is None:
continue
i = int(round(g[1] / 5.0))
breach[mask] = FRESH_CRIT[i, mask] < DEPTH0 - DPQ * g[2]
cst += mask.mean() * g[0]
return float(breach.mean()), float(cst)
def fresh_breaches(warned, design):
# breaches on the fresh paths, group by group: (warned, all clear)
out = []
for mask, g in ((warned, design["warn"]), (~warned, design["clear"])):
out.append(0 if (g is None or mask.sum() == 0) else
int((FRESH_CRIT[int(round(g[1] / 5.0)), mask] < DEPTH0 - DPQ * g[2]).sum()))
return tuple(out)
NONE = SPLIT.two(0.0, 0.0, PI1) # no warning: one package for every program, Volume I's
C5 = NONE["cost"]
NONE_FRESH = check(np.zeros(N, bool), NONE)
PERFECT = SPLIT.two(PI1, 1.0, 0.0) # Chapter 9's perfect warning: warned exactly when a closure is ahead
PERFECT_FRESH = check(FRESH.ahead.copy(), PERFECT)
print(f"scikit-learn {sklearn.__version__} (the book's engine used scikit-learn 1.9)")
print(f"closure ahead: {100 * PI1:.2f}% of the book's paths, {100 * FRESH.ahead.mean():.2f}% of the fresh ones; "
f"the law: 1 - e^-0.3 = {100 * P_CLOSE:.2f}%")
_, l5, ph5 = NONE["clear"]
print(f"no warning, Volume I's package: line {l5:.0f}, delay {ph5:.4f}, cost {C5:.2f}; chance of a breach "
f"{100 * SPLIT.chance(PI1, NONE['clear']):.2f}% on the book's paths, {100 * NONE_FRESH[0]:.2f}% on the fresh paths")
print("the perfect warning (Chapter 9):")
for name, g, pi in (("all clear, no closure ahead", PERFECT["clear"], 0.0), ("warned, a closure ahead", PERFECT["warn"], 1.0)):
print(f" {name:28s} line {g[1]:3.0f}, delay {g[2]:.3f}, cost {g[0]:5.2f}, chance of a breach "
f"{100 * SPLIT.chance(pi, g):.2f}%")
print(f" expected cost {PERFECT['cost']:.2f}: worth {C5:.2f} - {PERFECT['cost']:.2f} = {C5 - PERFECT['cost']:.2f}; on the "
f"fresh paths a chance of a breach of {100 * PERFECT_FRESH[0]:.2f}% and a cost of {PERFECT_FRESH[1]:.2f}")scikit-learn 1.9.1 (the book's engine used scikit-learn 1.9) closure ahead: 25.91% of the book's paths, 25.96% of the fresh ones; the law: 1 - e^-0.3 = 25.92% no warning, Volume I's package: line 170, delay 0.6676, cost 10.44; chance of a breach 5.00% on the book's paths, 4.91% on the fresh paths the perfect warning (Chapter 9): all clear, no closure ahead line 0, delay 0.608, cost 4.87, chance of a breach 4.75% warned, a closure ahead line 215, delay 0.504, cost 10.49, chance of a breach 5.71% expected cost 6.32: worth 10.44 - 6.32 = 4.12; on the fresh paths a chance of a breach of 4.91% and a cost of 6.33
Panel 1 — Read the record (Theorem 13.2(iii); Worked Example 13.1, Step 1; Figure 13.1(b); Online Appendix Example 13.B.2: 268 closures in 1,000 windows; the law's coefficients −0.323, −0.159, −0.352, 0.769, 1.974, separation 2.029, AUC 0.924; the ceiling, $1.44m, warning 58.86% of programs; $0.63m at an AUC of 0.86, $2.24m at 0.96)
AXIOM Lab 2.13, step 1. Compare the readings of windows with and without a closure, and compute the ceiling: the most any warning on these readings can be worth (Theorem 13.2(iii)). Given closure or not, the readings have the same covariance, , so the log-odds of a closure ahead is linear in them, with coefficients . The log-likelihood ratio is normal with mean for a program with a closure ahead and for the others, and variance , where is the readings' separation; the readings rank a program with a closure ahead above one without with chance . The best warning reads : a threshold splits programs into warned and all clear, the law gives each group's size and closure share exactly, and each of 241 thresholds from to gets its two packages; the cheapest pair prices the ceiling, which is then checked on the fresh paths. Scaling all five shifts traces the most a warning is worth against its readings' AUC (Figure 13.1(b)).
Python Code
YR, XR = record()
print(f"the desk's record: {len(YR):,} windows, {YR.sum()} with a closure ({100 * YR.mean():.1f}%)")
print("mean readings (standard units) " + "".join(f"{s:>12s}" for s in SHORT))
print(" windows with a closure " + "".join(f"{x:12.2f}" for x in XR[YR].mean(0)))
print(" windows without " + "".join(f"{x:12.2f}" for x in XR[~YR].mean(0)))
def posterior_coef(shift=SHIFT, idx=None):
# the declared law's log-odds coefficients beta = Sigma^-1 s and separation Delta, for all readings or a subset
idx = list(range(5)) if idx is None else list(idx)
b = np.linalg.solve(SIGMA[np.ix_(idx, idx)], shift[idx])
return b, float(math.sqrt(shift[idx] @ b))
def auc(Delta):
return float(stats.norm.cdf(Delta / math.sqrt(2.0)))
def posterior_split(Delta, t, pi1):
# a threshold t on the log-likelihood ratio z ~ N(+/- Delta^2/2, Delta^2): the share warned, each group's closure share
if Delta <= 0:
return (0.0, pi1, pi1)
a1 = 1.0 - stats.norm.cdf((t - Delta ** 2 / 2.0) / Delta) # P(warned | closure ahead)
a0 = 1.0 - stats.norm.cdf((t + Delta ** 2 / 2.0) / Delta) # P(warned | none)
sw = pi1 * a1 + (1 - pi1) * a0
pw = pi1 * a1 / sw if sw > 0 else 0.0
pc = pi1 * (1 - a1) / (1 - sw) if sw < 1 else 0.0
return sw, pw, pc
def ceiling(Delta, grid=np.linspace(-6.0, 6.0, 241)):
# the best warning the readings allow, read by the declared law (Theorem 13.2(iii))
best = None
for t in grid * max(Delta, 0.3):
sw, pw, pc = posterior_split(Delta, t, PI1)
d = SPLIT.two(sw, pw, pc)
if d is not None and (best is None or d["cost"] < best["design"]["cost"]):
best = dict(t=float(t), shares=(sw, pw, pc), design=d)
best["value"], best["auc"], best["Delta"] = C5 - best["design"]["cost"], auc(Delta), Delta
return best
BETA_LAW, DELTA = posterior_coef()
print("\nthe declared law: beta = Sigma^-1 s = (" + ", ".join(f"{b:.3f}" for b in BETA_LAW) + f"); separation "
f"Delta = {DELTA:.3f}; AUC = Phi(Delta / sqrt 2) = {auc(DELTA):.3f}")
CEIL = ceiling(DELTA)
Z_FRESH = FRESH_X @ BETA_LAW - DELTA ** 2 / 2.0 # each fresh program's log-likelihood ratio of a closure ahead
CEIL_W = Z_FRESH >= CEIL["t"]
CEIL_FRESH = check(CEIL_W, CEIL["design"])
sw, pw, pc = CEIL["shares"]
cd = CEIL["design"]
print(f"the ceiling: warn when z >= {CEIL['t']:.3f}: {100 * sw:.2f}% of programs warned; closures ahead in {100 * pw:.2f}% "
f"of the warned and {100 * pc:.2f}% of the all clear")
print(f" all clear: line {cd['clear'][1]:.0f}, delay {cd['clear'][2]:.3f}, {cd['clear'][0]:.2f}; warned: line "
f"{cd['warn'][1]:.0f}, delay {cd['warn'][2]:.3f}, {cd['warn'][0]:.2f}; expected {cd['cost']:.2f}; worth "
f"{C5:.2f} - {cd['cost']:.2f} = {CEIL['value']:.2f}, against {C5 - PERFECT['cost']:.2f} for a perfect warning")
print(f" on the fresh paths: {100 * CEIL_W.mean():.2f}% warned, chance of a breach {100 * CEIL_FRESH[0]:.2f}%, worth "
f"{C5 - CEIL_FRESH[1]:.2f}")
SCALES = (0.25, 0.5, 0.75, 1.0, 1.25, 1.5, 2.0, 2.5, 3.0, 4.0)
CURVE = [ceiling(DELTA * c) for c in SCALES]
print("\nthe most a warning is worth against its readings' separation (all five shifts scaled):")
print(" shifts x " + "".join(f"{c:7g}" for c in SCALES))
print(" AUC " + "".join(f"{x['auc']:7.3f}" for x in CURVE))
print(" ceiling " + "".join(f"{x['value']:7.2f}" for x in CURVE))the desk's record: 1,000 windows, 268 with a closure (26.8%) mean readings (standard units) bond high yield volatility lending funding windows with a closure 0.18 0.29 0.17 0.73 2.00 windows without -0.02 0.01 -0.02 -0.06 -0.02 the declared law: beta = Sigma^-1 s = (-0.323, -0.159, -0.352, 0.769, 1.974); separation Delta = 2.029; AUC = Phi(Delta / sqrt 2) = 0.924 the ceiling: warn when z >= -1.826: 58.86% of programs warned; closures ahead in 42.80% of the warned and 1.75% of the all clear all clear: line 0, delay 0.671, 5.37; warned: line 190, delay 0.731, 11.55; expected 9.00; worth 10.44 - 9.00 = 1.44, against 4.12 for a perfect warning on the fresh paths: 58.94% warned, chance of a breach 4.90%, worth 1.43 the most a warning is worth against its readings' separation (all five shifts scaled): shifts x 0.25 0.5 0.75 1 1.25 1.5 2 2.5 3 4 AUC 0.640 0.763 0.859 0.924 0.964 0.984 0.998 1.000 1.000 1.000 ceiling 0.05 0.22 0.63 1.44 2.24 2.88 3.69 4.00 4.09 4.12
Panel 2 — Train and validate (Theorem 13.1; Proposition 13.2; Worked Example 13.1, Steps 2–3; Figure 13.2(a): the logistic model's coefficients −0.428, −0.264, −0.183, 0.968, 2.180 and AUC 0.942 on the record, 0.940 held out, 0.924 fresh; boosting's AUC 1.000 on the record, 0.922 held out, 0.888 fresh, its fresh AUC peaking at 0.910 after 24 stages)
AXIOM Lab 2.13, step 2. Fit a logistic model to the record by unpenalized maximum likelihood, and gradient boosting with 500 stages of depth-4 trees and learning rate 0.1, both with scikit-learn as in the book's engine. Score each on the record it learned, on held-out folds (five folds, stratified and shuffled with the record's seed: each window is scored by the model fitted on the other four folds), and on the 100,000 fresh programs, each with its own readings. The logistic class has six parameters; the boosted model can fit every window of the record, so by Theorem 13.1(iii) its record score says nothing about its fresh error, and by Proposition 13.2(iv) boosting lowers the record's loss at every stage whatever the fresh error does. The stage curve scores the boosted model after each stage on the record and on 20,000 of the fresh programs, drawn with the book's seed 26214 (Remark 13.B.1). Last, the Online Appendix's majority vote that is worse than every member (Proposition 13.2(iii)).
Python Code
def fit_logistic(X, y):
return LogisticRegression(C=1e10, max_iter=5000).fit(X, y) # unpenalized maximum likelihood, as the engine
def fit_boost(X, y, stages=500):
return GradientBoostingClassifier(n_estimators=stages, max_depth=4, learning_rate=0.1,
random_state=RECORD_SEED).fit(X, y)
def oof_scores(make, X, y, seed=RECORD_SEED, folds=5):
# held-out scores: each window scored by the model fitted on the other folds (stratified, shuffled with the seed)
s = np.empty(len(y))
for tr, te in StratifiedKFold(folds, shuffle=True, random_state=seed).split(X, y):
s[te] = make(X[tr], y[tr]).predict_proba(X[te])[:, 1]
return s
LR = fit_logistic(XR, YR)
F_LR = lambda X: LR.predict_proba(X)[:, 1]
S_IN, S_OOF, S_FRESH = F_LR(XR), oof_scores(fit_logistic, XR, YR), F_LR(FRESH_X)
AUC_LR = dict(record=roc_auc_score(YR, S_IN), oof=roc_auc_score(YR, S_OOF), fresh=roc_auc_score(FRESH.ahead, S_FRESH))
ACC_LR = dict(record=float(np.mean((S_IN >= 0.5) == YR)), fresh=float(np.mean((S_FRESH >= 0.5) == FRESH.ahead)))
NEVER = float(1.0 - FRESH.ahead.mean()) # never warning is right whenever no closure is ahead
print("the logistic model: log-odds = " + f"{LR.intercept_[0]:.3f} " + " ".join(
f"{b:+.3f} x {s}" for b, s in zip(LR.coef_[0], SHORT)))
print(f" AUC: {AUC_LR['record']:.3f} on the record, {AUC_LR['oof']:.3f} held out, {AUC_LR['fresh']:.3f} fresh; "
f"right at 0.5: {100 * ACC_LR['record']:.1f}% of the record, {100 * ACC_LR['fresh']:.1f}% of fresh programs; "
f"never warning: {100 * NEVER:.1f}%")
GB = fit_boost(XR, YR)
F_GB = lambda X: GB.predict_proba(X)[:, 1]
G_IN, G_OOF, G_FRESH = F_GB(XR), oof_scores(fit_boost, XR, YR), F_GB(FRESH_X)
AUC_GB = dict(record=roc_auc_score(YR, G_IN), oof=roc_auc_score(YR, G_OOF), fresh=roc_auc_score(FRESH.ahead, G_FRESH))
ACC_GB = dict(record=float(np.mean((G_IN >= 0.5) == YR)), fresh=float(np.mean((G_FRESH >= 0.5) == FRESH.ahead)))
print(f"boosting, 500 stages of depth-4 trees: AUC {AUC_GB['record']:.3f} on the record, {AUC_GB['oof']:.3f} held out, "
f"{AUC_GB['fresh']:.3f} fresh; right at 0.5: {100 * ACC_GB['record']:.1f}% of the record, "
f"{100 * ACC_GB['fresh']:.1f}% of fresh programs")
STAGE_SEED = 26214 # the book's seed for the stage curve's 20,000 fresh programs (Remark 13.B.1)
SUB = np.random.default_rng(STAGE_SEED).choice(N, 20000, replace=False)
ST_REC = np.array([roc_auc_score(YR, p[:, 1]) for p in GB.staged_predict_proba(XR)])
ST_FR = np.array([roc_auc_score(FRESH.ahead[SUB], p[:, 1]) for p in GB.staged_predict_proba(FRESH_X[SUB])])
PEAK = int(np.argmax(ST_FR))
print(f"stage by stage: the fresh AUC peaks at {ST_FR[PEAK]:.3f} after {PEAK + 1} stages and ends at {ST_FR[-1]:.3f}; "
f"the record's is {ST_REC[49]:.3f} after 50 stages and {ST_REC[-1]:.3f} after 500")
# Proposition 13.2(iii): three classifiers each right on 60% of cases, whose majority vote is right on 40%
VOTE = np.zeros((15, 3), bool)
VOTE[:6] = True # 40% of cases: all three right
for j in range(3):
VOTE[6 + 3 * j: 9 + 3 * j, j] = True # 60%: exactly one right, each classifier a third of them
MEMBERS = VOTE.mean(axis=0)
MAJORITY = float((VOTE.sum(axis=1) >= 2).mean())
print("\nProposition 13.2(iii): members right on " + ", ".join(f"{100 * m:.0f}%" for m in MEMBERS)
+ f" of cases; their majority vote on {100 * MAJORITY:.0f}%")
fig, ax = plt.subplots(figsize=(7.6, 3.6))
k_ = np.arange(1, len(ST_REC) + 1)
ax.plot(k_, ST_REC, color="#E65100", lw=1.8, label="on the record")
ax.plot(k_, ST_FR, color="#1F4E79", lw=1.8, label="on 20,000 fresh programs")
ax.axhline(AUC_LR["fresh"], color="#999999", lw=0.9, ls=":")
ax.text(1.1, AUC_LR["fresh"] + 0.004, f"logistic model, fresh {AUC_LR['fresh']:.3f}", fontsize=8, color="#616161")
ax.plot([PEAK + 1], [ST_FR[PEAK]], "o", color="#1F4E79", ms=5)
ax.text(PEAK + 1, ST_FR[PEAK] - 0.008, f"peak {ST_FR[PEAK]:.3f} after {PEAK + 1} stages", fontsize=8, color="#1F4E79",
ha="center", va="top")
ax.text(500, ST_FR[-1] - 0.008, f"{ST_FR[-1]:.3f}", fontsize=8, color="#1F4E79", ha="right", va="top")
ax.set_xscale("log")
ax.set_xticks([1, 10, 100, 500], ["1", "10", "100", "500"])
ax.set(xlim=(1, 500), ylim=(0.84, 1.01), xlabel="boosting stages (log scale)", ylabel="AUC (chance a closure ranks higher)",
title="Boosting's record AUC reaches 1.00 as its fresh AUC falls (seed 26213)")
ax.legend(frameon=False, fontsize=8, loc="center right"); plt.tight_layout(); plt.show()the logistic model: log-odds = -3.435 -0.428 x bond -0.264 x high yield -0.183 x volatility +0.968 x lending +2.180 x funding AUC: 0.942 on the record, 0.940 held out, 0.924 fresh; right at 0.5: 89.3% of the record, 87.3% of fresh programs; never warning: 74.0% boosting, 500 stages of depth-4 trees: AUC 1.000 on the record, 0.922 held out, 0.888 fresh; right at 0.5: 100.0% of the record, 85.6% of fresh programs stage by stage: the fresh AUC peaks at 0.910 after 24 stages and ends at 0.888; the record's is 0.992 after 50 stages and 1.000 after 500 Proposition 13.2(iii): members right on 60%, 60%, 60% of cases; their majority vote on 40%

Panel 3 — Price the warning (Theorem 13.2(ii); Worked Example 13.1, Step 2; Figure 13.1: at accuracy's threshold, 0.5, the logistic warning warns 23.0% of programs, is right 87.3% of the time and is worth $0.53m; where the packages are cheapest, at a 4.09% chance, it warns 59.0%, is right 65.9% of the time and is worth $1.44m; never warning is right 74.0% of the time)
AXIOM Lab 2.13, step 3. Sweep the logistic warning's threshold and price each threshold by its two packages. Here each group's closure share is the law's: the book's 100,000 paths get their own readings (seed 1309), the fitted model scores them, and each group's share of paths with a closure ahead is read off the paths (Figure 13.1(a)). Accuracy counts a false alarm and a miss as equal; the packages do not. By Theorem 13.2(ii), the best rule warns when the chance of a closure exceeds a threshold set by the costs, with the appetite priced in; the cell finds it among 400 quantile steps of the scores. Accuracy and value are read on the fresh paths, each program with its own readings. Last, Theorem 13.2(iv)'s example: a prediction with the smaller squared error that leads to the costlier decision.
Python Code
def shares(score, y, tau):
# warn at or above tau: the share warned and each group's closure share
w = score >= tau
return float(w.mean()), (float(y[w].mean()) if w.any() else 0.0), (float(y[~w].mean()) if (~w).any() else 0.0), w
_CP = {}
def cp_upper(k, m, conf=0.95):
# Theorem 13.3(i): the one-sided Clopper-Pearson upper bound for a chance, k events in m cases
if (k, m, conf) not in _CP:
_CP[k, m, conf] = 1.0 if k >= m else float(stats.beta.ppf(conf, k + 1, m - k))
return _CP[k, m, conf]
def thresholds(score, n_steps=100):
return np.unique(np.concatenate([np.quantile(score, np.linspace(0.0, 1.0, n_steps + 1)), [np.inf]]))
def best_split(score, y, certify=False, n_steps=100):
# the threshold and packages with the least expected cost, each group's closure share estimated from (score, y);
# certify: each share replaced by its one-sided 95% upper bound
best = None
for tau in thresholds(score, n_steps):
sw, pw, pc, w = shares(score, y, tau)
if certify:
m_w, m_c = int(w.sum()), int((~w).sum())
pw = cp_upper(int(y[w].sum()), m_w) if m_w else 0.0
pc = cp_upper(int(y[~w].sum()), m_c) if m_c else 0.0
d = SPLIT.two(sw, pw, pc)
if d is not None and (best is None or d["cost"] < best["design"]["cost"] - 1e-12):
best = dict(tau=float(tau), shares=(sw, pw, pc), design=d)
return best
def evaluate(score_fn, tau, design):
# the design on the fresh paths, each program warned by its own readings
warned = score_fn(FRESH_X) >= tau
breach, cst = check(warned, design)
return dict(breach=breach, cost=cst, value=C5 - cst, fresh_warned=float(warned.mean()),
accuracy=float(np.mean(warned == FRESH.ahead)), believed=C5 - design["cost"], warned_mask=warned)
def show(name, x, value=True):
sw, pw, pc = x["shares"]
d = x["design"]
print(f"{name}: warn at a chance of {100 * x['tau']:.2f}%: {100 * sw:.2f}% warned; closures ahead in {100 * pw:.2f}% of "
f"the warned and {100 * pc:.2f}% of the all clear")
print(f" all clear: line {d['clear'][1]:.0f}, delay {d['clear'][2]:.3f}, {d['clear'][0]:.2f}; warned: line "
f"{d['warn'][1]:.0f}, delay {d['warn'][2]:.3f}, {d['warn'][0]:.2f}; promises {x['believed']:.2f}")
print(f" fresh: {100 * x['fresh_warned']:.1f}% warned, right {100 * x['accuracy']:.1f}% of the time, chance of a "
f"breach {100 * x['breach']:.2f}%, " + (f"worth {x['value']:.2f}" if value else "the appetite broken: no value"))
X_BOOK = readings(BOOK.ahead.astype(float), np.random.default_rng(BOOK_READ_SEED))
S_BOOK = F_LR(X_BOOK)
SWEEP = []
for share in np.linspace(0.0, 1.0, 51):
tau = np.inf if share <= 0 else float(np.quantile(S_BOOK, 1.0 - share))
sw, pw, pc, _ = shares(S_BOOK, BOOK.ahead, tau)
d = SPLIT.two(sw, pw, pc)
ev = evaluate(F_LR, tau, d)
SWEEP.append(dict(share=sw, accuracy=ev["accuracy"], value=ev["value"], breach=ev["breach"], design_value=C5 - d["cost"]))
LAW05 = dict(tau=0.5)
LAW05["shares"] = shares(S_BOOK, BOOK.ahead, 0.5)[:3]
LAW05["design"] = SPLIT.two(*LAW05["shares"])
LAW05.update(evaluate(F_LR, 0.5, LAW05["design"]))
LAWBEST = best_split(S_BOOK, BOOK.ahead, n_steps=400)
LAWBEST.update(evaluate(F_LR, LAWBEST["tau"], LAWBEST["design"]))
print("each group's closure share as the law gives it:")
show("at accuracy's threshold", LAW05)
show("where the packages are cheapest", LAWBEST)
print(f"never warning is right {100 * NEVER:.1f}% of the time: the valuable warning is less accurate than no warning")
print("the sweep, by share of programs warned (%): " + ", ".join(
f"{100 * x['share']:.0f}: {100 * x['accuracy']:.1f}% right, {x['design_value']:.2f}" for x in SWEEP[5:40:5]))
# Theorem 13.2(iv): order u against an outcome Y of 0 or 10 with equal chances; over-ordering costs 1 a unit, under 9
def order_cost(u):
return 0.5 * (1.0 * max(u - 0.0, 0.0) + 9.0 * max(0.0 - u, 0.0)) + 0.5 * (1.0 * max(u - 10.0, 0.0) + 9.0 * max(10.0 - u, 0.0))
NV = {u: (0.5 * (u - 0.0) ** 2 + 0.5 * (u - 10.0) ** 2, order_cost(u)) for u in (5.0, 10.0)}
print("\nTheorem 13.2(iv): " + "; ".join(f"predict {u:g}: squared error {se:g}, its decision costs {c:g}" for u, (se, c) in NV.items()))
fig, (ax, bx) = plt.subplots(1, 2, figsize=(7.6, 3.6), gridspec_kw={"width_ratios": [1.15, 1]})
sh = [100 * x["share"] for x in SWEEP]
ax.plot(sh, [x["design_value"] for x in SWEEP], color="#1F4E79", lw=1.8, label="value of its packages")
ax2 = ax.twinx()
ax2.plot(sh, [100 * x["accuracy"] for x in SWEEP], color="#E65100", lw=1.6, ls="--", label="accuracy, fresh paths")
ax2.axhline(100 * NEVER, color="#999999", lw=0.8, ls=":")
ax2.text(3, 100 * NEVER - 1.5, f"never warn: {100 * NEVER:.1f}%", fontsize=8, color="#616161", va="top")
for x_ in (LAW05, LAWBEST):
ax.plot([100 * x_["shares"][0]], [x_["believed"]], "o", color="#1F4E79", ms=5, zorder=5)
ax2.plot([100 * x_["shares"][0]], [100 * x_["accuracy"]], "o", color="#E65100", ms=5, zorder=5)
ax.text(100 * LAWBEST["shares"][0] + 3, LAWBEST["believed"] + 0.08,
f"\\${LAWBEST['believed']:.2f}m,\n{100 * LAWBEST['shares'][0]:.0f}% warned", fontsize=8, color="#1F4E79")
ax.text(100 * LAW05["shares"][0] + 3, LAW05["believed"] - 0.05, f"\\${LAW05['believed']:.2f}m", fontsize=8,
color="#1F4E79", va="top")
ax2.text(100 * LAW05["shares"][0] - 2, 100 * LAW05["accuracy"] + 2.5, f"threshold 0.5: {100 * LAW05['accuracy']:.1f}%",
fontsize=8, color="#E65100")
ax.set_zorder(ax2.get_zorder() + 1); ax.patch.set_visible(False)
ax.set(xlim=(0, 100), ylim=(0, 2.0), xlabel="programs warned (%)", ylabel="value of the warning ($ million)")
ax2.set(ylim=(20, 100)); ax2.set_ylabel("accuracy on fresh paths (%)", color="#E65100")
ax2.spines["top"].set_visible(False)
ax.set_title("(a) the logistic warning, every threshold", fontsize=9)
bx.plot([x["auc"] for x in CURVE], [x["value"] for x in CURVE], "o-", color="#1F4E79", lw=1.6, ms=3.5,
label="all five shifts scaled")
for txt, a_, v_, xy in ((f"perfect warning: \\${C5 - PERFECT['cost']:.2f}m", 1.0, C5 - PERFECT["cost"], (0.56, 3.9)),
(f"the desk's five readings:\nAUC {CEIL['auc']:.3f}, \\${CEIL['value']:.2f}m", CEIL["auc"], CEIL["value"],
(0.56, 2.1))):
bx.plot([a_], [v_], "o", color="#E65100", ms=6, zorder=5)
bx.annotate(txt, xy=(a_, v_), xytext=xy, fontsize=8, va="center", arrowprops=dict(arrowstyle="-", lw=0.6, color="#333333"))
bx.set(xlim=(0.5, 1.01), ylim=(0, 4.5), xlabel="AUC of the readings", ylabel="most a warning is worth ($ million)")
bx.set_title("(b) the ceiling, by the readings' AUC", fontsize=9)
fig.suptitle("A \\$1.44m warning is right less often than no warning (seed 26213)", fontsize=10.5)
plt.tight_layout(); plt.show()each group's closure share as the law gives it: at accuracy's threshold: warn at a chance of 50.00%: 23.01% warned; closures ahead in 78.37% of the warned and 10.23% of the all clear all clear: line 125, delay 0.712, 9.45; warned: line 205, delay 0.665, 11.47; promises 0.53 fresh: 22.9% warned, right 87.3% of the time, chance of a breach 4.90%, worth 0.53 where the packages are cheapest: warn at a chance of 4.09%: 59.00% warned; closures ahead in 42.71% of the warned and 1.73% of the all clear all clear: line 0, delay 0.669, 5.35; warned: line 190, delay 0.730, 11.54; promises 1.44 fresh: 58.6% warned, right 65.9% of the time, chance of a breach 4.93%, worth 1.46 never warning is right 74.0% of the time: the valuable warning is less accurate than no warning the sweep, by share of programs warned (%): 10: 82.7% right, 0.19, 20: 87.1% right, 0.45, 30: 85.9% right, 0.71, 40: 80.8% right, 0.95, 50: 73.5% right, 1.22, 60: 65.0% right, 1.43, 70: 55.9% right, 1.26 Theorem 13.2(iv): predict 5: squared error 25, its decision costs 25; predict 10: squared error 50, its decision costs 5

Panel 4 — A perfect record and a broken appetite (Theorem 13.1(iii); Proposition 13.2(iv); Worked Example 13.1, Step 3; Table 13.3: trusted, boosting warns 27.0% of the record with closure shares 99.26% and 0%, promises $4.04m and breaches 7.68% on the fresh paths; held out, it warns 58.0% with shares 44.48% and 2.38%, promises $1.16m, breaches 4.96% and is worth $0.59m)
AXIOM Lab 2.13, step 4, first part. From here on each group's closure share is estimated from the desk's record, as a design must be. The trusted design takes the shares from the boosted model's scores on the record it learned; the held-out design takes them from its out-of-fold scores. Each design's threshold is chosen among 100 quantile steps of the scores it is designed from, and each design is checked on the fresh paths.
Python Code
TRUSTED = best_split(G_IN, YR)
TRUSTED.update(evaluate(F_GB, TRUSTED["tau"], TRUSTED["design"]))
GB_HELD = best_split(G_OOF, YR)
GB_HELD.update(evaluate(F_GB, GB_HELD["tau"], GB_HELD["design"]))
show("boosting, trusted", TRUSTED, value=False)
show("boosting, held out", GB_HELD)
print(f"\ntrusted, the boosted warning promises {TRUSTED['believed']:.2f}, against {C5 - PERFECT['cost']:.2f} for a "
f"perfect warning, and breaks the appetite: {100 * TRUSTED['breach']:.2f}% on the fresh paths")boosting, trusted: warn at a chance of 2.51%: 27.00% warned; closures ahead in 99.26% of the warned and 0.00% of the all clear all clear: line 0, delay 0.608, 4.87; warned: line 215, delay 0.513, 10.56; promises 4.04 fresh: 35.7% warned, right 81.0% of the time, chance of a breach 7.68%, the appetite broken: no value boosting, held out: warn at a chance of 0.09%: 58.00% warned; closures ahead in 44.48% of the warned and 2.38% of the all clear all clear: line 0, delay 0.757, 6.06; warned: line 190, delay 0.739, 11.61; promises 1.16 fresh: 68.3% warned, right 55.5% of the time, chance of a breach 4.96%, worth 0.59 trusted, the boosted warning promises 4.04, against 4.12 for a perfect warning, and breaks the appetite: 7.68% on the fresh paths
Panel 5 — Certify before paying (Theorem 13.3(i); Worked Example 13.1, Step 4; Table 13.3: designed from held-out shares, 44.75% and 0.98%, the logistic warning promises $1.73m and breaches 5.02% on the fresh paths; designed for their one-sided 95% upper bounds, 48.21% and 2.22%, it promises $1.17m, breaches 4.73% and is worth $1.06m)
AXIOM Lab 2.13, step 4. The audit committee's rule: a model is scored on cases held out from its fitting, and a design is certified on bounds, not point estimates. The held-out design takes each group's closure share from the logistic model's out-of-fold scores; the certified design replaces each by its one-sided 95 percent Clopper–Pearson upper bound, the 0.95 quantile of the beta law with parameters for closures among windows (Theorem 13.3(i)). Both are checked on the fresh paths. Last, the logistic warning at accuracy's threshold with held-out shares (Online Appendix Example 13.B.2).
Python Code
HELD = best_split(S_OOF, YR)
HELD.update(evaluate(F_LR, HELD["tau"], HELD["design"]))
CERT = best_split(S_OOF, YR, certify=True)
CERT.update(evaluate(F_LR, CERT["tau"], CERT["design"]))
OOF05 = dict(tau=0.5)
OOF05["shares"] = shares(S_OOF, YR, 0.5)[:3]
OOF05["design"] = SPLIT.two(*OOF05["shares"])
OOF05.update(evaluate(F_LR, 0.5, OOF05["design"]))
show("logistic, held out", HELD)
w_ = S_OOF >= CERT["tau"]
KM = dict(k_w=int(YR[w_].sum()), m_w=int(w_.sum()), k_c=int(YR[~w_].sum()), m_c=int((~w_).sum()))
print(f"\nat the same threshold the record holds {KM['k_w']} closures in {KM['m_w']} warned windows and {KM['k_c']} in "
f"{KM['m_c']} all clear: point estimates {100 * KM['k_w'] / KM['m_w']:.2f}% and {100 * KM['k_c'] / KM['m_c']:.2f}%, "
f"one-sided 95% upper bounds {100 * cp_upper(KM['k_w'], KM['m_w']):.2f}% and {100 * cp_upper(KM['k_c'], KM['m_c']):.2f}%")
show("logistic, certified", CERT)
print()
show("logistic at 0.5, held-out shares", OOF05)logistic, held out: warn at a chance of 3.48%: 59.00% warned; closures ahead in 44.75% of the warned and 0.98% of the all clear all clear: line 0, delay 0.683, 5.46; warned: line 185, delay 0.677, 10.96; promises 1.73 fresh: 60.9% warned, right 63.9% of the time, chance of a breach 5.02%, worth 1.63 at the same threshold the record holds 264 closures in 590 warned windows and 4 in 410 all clear: point estimates 44.75% and 0.98%, one-sided 95% upper bounds 48.21% and 2.22% logistic, certified: warn at a chance of 3.48%: 59.00% warned; closures ahead in 48.21% of the warned and 2.22% of the all clear all clear: line 0, delay 0.734, 5.87; warned: line 190, delay 0.742, 11.64; promises 1.17 fresh: 60.9% warned, right 63.9% of the time, chance of a breach 4.73%, worth 1.06 logistic at 0.5, held-out shares: warn at a chance of 50.00%: 25.20% warned; closures ahead in 81.75% of the warned and 8.29% of the all clear all clear: line 115, delay 0.718, 9.19; warned: line 205, delay 0.670, 11.51; promises 0.67 fresh: 22.9% warned, right 87.3% of the time, chance of a breach 5.03%, worth 0.72
Panel 6 — Repeat on 200 records (Theorem 13.3(i); Worked Example 13.1, Step 4; Figure 13.2(b): designs from held-out shares keep the appetite in 47.5% of records, with a mean true chance of 5.02%, a 90th percentile of 5.21% and a mean value of $1.53m; certified designs in 98.0%, 4.75%, 4.89% and $0.95m)
AXIOM Lab 2.13, step 4, last part. The desk's record is one of many it could have been. Draw 200 records of 1,000 windows from the declared law (seeds 913000–913199); on each, fit the logistic model, score it out of fold (folds shuffled with the record's own seed), design the held-out and the certified warnings (thresholds among 50 quantile steps of the scores), and check both on the fresh paths. This cell takes about a minute: 400 designs, each searching 51 thresholds.
Python Code
STUDY = []
for r in range(200):
y_, X_ = record(seed=STUDY_SEED + r)
m_ = fit_logistic(X_, y_)
so_ = oof_scores(fit_logistic, X_, y_, seed=STUDY_SEED + r)
f_ = (lambda mm: (lambda X: mm.predict_proba(X)[:, 1]))(m_)
row = dict(seed=STUDY_SEED + r)
for key, cert in (("held", False), ("cert", True)):
b = best_split(so_, y_, certify=cert, n_steps=50)
e = evaluate(f_, b["tau"], b["design"])
row[key] = dict(breach=e["breach"], value=e["value"], believed=e["believed"])
STUDY.append(row)
SUMMARY = {}
for key, name in (("held", "held-out shares"), ("cert", "certified")):
b_ = np.array([x[key]["breach"] for x in STUDY]); v_ = np.array([x[key]["value"] for x in STUDY])
SUMMARY[key] = dict(meet=float(np.mean(b_ <= ALPHA)), mean=float(b_.mean()), q90=float(np.quantile(b_, 0.9)),
value=float(v_.mean()), believed=float(np.mean([x[key]["believed"] for x in STUDY])))
s = SUMMARY[key]
print(f"{name:16s}: keep 5% in {100 * s['meet']:.1f}% of 200 records; true chance {100 * s['mean']:.2f}% on average, "
f"{100 * s['q90']:.2f}% at the 90th percentile; worth {s['value']:.2f} on average (promised {s['believed']:.2f})")
fig, ax = plt.subplots(figsize=(7.6, 3.6))
for key, col, name in (("held", "#E65100", "held-out shares"), ("cert", "#1F4E79", "certified")):
ax.plot([100 * x[key]["breach"] for x in STUDY], [x[key]["value"] for x in STUDY], "o", ms=3.2, color=col, alpha=0.75,
label=f"{name}: {100 * SUMMARY[key]['meet']:.1f}% keep 5%")
ax.axvline(5.0, color="#999999", lw=0.9, ls=":")
ax.text(5.02, 0.08, "the appetite", fontsize=8, color="#616161")
ax.set(xlim=(4.4, 5.6), ylim=(0, 2.8), xlabel="chance of a breach on the fresh paths (%)",
ylabel="value on the fresh paths ($ million)", title="Certified designs keep the appetite in 98.0% of records (seed 26213)")
ax.legend(frameon=False, fontsize=8, loc="upper left"); plt.tight_layout(); plt.show()held-out shares : keep 5% in 47.5% of 200 records; true chance 5.02% on average, 5.21% at the 90th percentile; worth 1.53 on average (promised 1.53) certified : keep 5% in 98.0% of 200 records; true chance 4.75% on average, 4.89% at the 90th percentile; worth 0.95 on average (promised 0.96)

Panel 7 — Compress and explain (Proposition 13.3; Worked Example 13.1, Step 5; Figure 13.3: the first principal component keeps 56.0% of the variance and allows at most $0.14m; the three market readings hold 51.1% and allow $0.02m, the funding spread alone 30.1% and $1.21m; attributions of the reading (1, 1, 1, 1, 1.5), interventional −0.413, −0.242, −0.176, 0.820, 2.131, conditional 0.048, 0.098, 0.043, 0.462, 1.469)
AXIOM Lab 2.13, step 5. Components. The record's principal components are the eigenvectors of its covariance, in order of their eigenvalues (Definition 13.3). A warning on the top components reads , for the first eigenvectors; its separation under the declared law is , and it is priced like the ceiling. Subsets of the readings are priced the same way. Attributions. The fitted model's log-odds is linear. Under the interventional game, readings outside a coalition are set independently at their record averages, and the Shapley value of reading is (Proposition 13.3(iii)); under the conditional game they are set at their averages given the coalition, from a Gaussian fit to the record, and the Shapley values are computed exactly from the game's values on all 32 coalitions. The reading is one standard unit high on every indicator and 1.5 on the funding spread. Last, Proposition 13.3's two small examples.
Python Code
COV = np.cov(XR.T)
EIG, V = np.linalg.eigh(COV)
order = np.argsort(EIG)[::-1]
EIG, V = EIG[order], V[:, order]
if V[:, 0].sum() < 0:
V[:, 0] *= -1
KEPT = np.cumsum(EIG) / EIG.sum()
PCS = []
for r in range(1, 6):
Vr = V[:, :r]
D_r = float(math.sqrt(SHIFT @ Vr @ np.linalg.solve(Vr.T @ SIGMA @ Vr, Vr.T @ SHIFT)))
PCS.append(dict(r=r, kept=float(KEPT[r - 1]), Delta=D_r, auc=auc(D_r), value=ceiling(D_r)["value"]))
print("the record's principal components: eigenvalues " + ", ".join(f"{x:.3f}" for x in EIG))
print(" the first's loadings " + ", ".join(f"{x:.2f}" for x in V[:, 0]) + ": an equal-weighted financial-conditions index")
print(" top r variance kept separation AUC the most a warning on them is worth")
for x in PCS:
print(f" {x['r']:5d} {100 * x['kept']:12.1f}% {x['Delta']:10.3f} {x['auc']:.3f} {x['value']:6.2f}")
VAR = np.var(XR, axis=0, ddof=1) # the record's variance of each reading
TR = float(np.trace(COV))
SUBSETS = {}
for name, idx in (("the three market readings", [0, 1, 2]), ("lending and funding", [3, 4]), ("funding alone", [4]),
("all but funding", [0, 1, 2, 3])):
b_, D_ = posterior_coef(idx=idx)
SUBSETS[name] = dict(idx=idx, Delta=D_, auc=auc(D_), value=ceiling(D_)["value"],
var=float(np.var(XR[:, idx], axis=0, ddof=1).sum() / TR))
print("\nsubsets of the readings variance held AUC the most a warning on them is worth")
for name, s in SUBSETS.items():
print(f" {name:26s} {100 * s['var']:12.1f}% {s['auc']:.3f} {s['value']:6.2f}")
def shapley_linear(beta, x, mean, cov, game="interventional"):
# exact Shapley values of f(x) = beta . x: readings outside a coalition at their means (interventional) or at their
# Gaussian conditional means given the coalition (conditional)
k = len(x)
if game == "interventional":
return beta * (x - mean)
def v(S):
S = list(S)
if not S:
return float(beta @ mean)
T = [j for j in range(k) if j not in S]
ex = np.array(x, dtype=float).copy()
if T:
ex[T] = mean[T] + cov[np.ix_(T, S)] @ np.linalg.solve(cov[np.ix_(S, S)], x[S] - mean[S])
return float(beta @ ex)
phi = np.zeros(k)
for j in range(k):
others = [i for i in range(k) if i != j]
for r in range(k):
for S in itertools.combinations(others, r):
wgt = math.factorial(r) * math.factorial(k - r - 1) / math.factorial(k)
phi[j] += wgt * (v(S + (j,)) - v(S))
return phi
X0 = np.array([1.0, 1.0, 1.0, 1.0, 1.5])
MU = XR.mean(0)
LOGODDS0 = float(LR.coef_[0] @ X0 + LR.intercept_[0])
PROB0 = float(LR.predict_proba(X0[None, :])[0, 1])
SH_I = shapley_linear(LR.coef_[0], X0, MU, COV, "interventional")
SH_C = shapley_linear(LR.coef_[0], X0, MU, COV, "conditional")
print(f"\nthe reading (1, 1, 1, 1, 1.5): log-odds {LOGODDS0:.3f}, a chance of {100 * PROB0:.1f}%: warned at the best "
f"threshold ({100 * LAWBEST['tau']:.2f}%): {PROB0 >= LAWBEST['tau']}; at 0.5: {PROB0 >= 0.5}")
print(" record averages " + "".join(f"{x:9.2f}" for x in MU))
print(" interventional " + "".join(f"{x:9.3f}" for x in SH_I) + f" sum {SH_I.sum():.3f}")
print(" conditional " + "".join(f"{x:9.3f}" for x in SH_C) + f" sum {SH_C.sum():.3f}")
print(f" the three market readings: {SH_I[:3].sum():.2f} interventional, {SH_C[:3].sum():.2f} conditional; "
f"f(x) - f(mean) = {LOGODDS0 - float(LR.coef_[0] @ MU + LR.intercept_[0]):.3f}")
# Proposition 13.3(ii): independent X1, X2 with variances 4 and 1, and Y = X2
PCA_EX = 4.0 / (4.0 + 1.0)
# Proposition 13.3(iv): f(x) = x1 and X2 = X1 with mean mu; the conditional game credits x2 with (x1 - mu) / 2
mu_, x1_ = 0.0, 2.0
v_ = {(): mu_, (1,): x1_, (2,): x1_, (1, 2): x1_} # v(S) = E[f(X) | X_S = x_S]
PHI2 = 0.5 * (v_[(2,)] - v_[()]) + 0.5 * (v_[(1, 2)] - v_[(1,)])
print(f"\nProposition 13.3(ii): the first component keeps {100 * PCA_EX:.0f}% of the variance and none of Y")
print(f"Proposition 13.3(iv): f ignores x2, yet the conditional game credits it with {PHI2:g} = (x1 - mu) / 2 at x1 = {x1_:g}")
fig, (ax, bx) = plt.subplots(1, 2, figsize=(7.6, 3.6), gridspec_kw={"width_ratios": [1.1, 1]})
ax.plot([100 * x["kept"] for x in PCS], [x["value"] for x in PCS], "o-", color="#1F4E79", lw=1.6, ms=4,
label="top 1–5 principal components")
for x in PCS[:3]:
ax.text(100 * x["kept"] + 1.5, x["value"] - 0.05, f"{x['r']}", fontsize=8, color="#1F4E79", va="top")
for name, xy in (("the three market readings", (6, 0.42)), ("funding alone", (4, 0.9)), ("lending and funding", (4, 1.65))):
s = SUBSETS[name]
ax.plot([100 * s["var"]], [s["value"]], "s", color="#E65100", ms=5, zorder=5)
ax.annotate(name, xy=(100 * s["var"], s["value"]), xytext=xy, fontsize=8,
arrowprops=dict(arrowstyle="-", lw=0.6, color="#333333"))
ax.set(xlim=(0, 105), ylim=(0, 2.0), xlabel="variance of the readings kept (%)", ylabel="most a warning is worth ($ million)")
ax.legend(frameon=False, fontsize=8, loc="upper right")
ax.set_title("(a) variance kept is not value", fontsize=9)
yy = np.arange(5)[::-1]
bx.barh(yy + 0.18, SH_I, height=0.36, color="#1F4E79", label="interventional")
bx.barh(yy - 0.18, SH_C, height=0.36, color="#E65100", label="conditional")
bx.axvline(0, color="#333333", lw=0.7)
bx.set_yticks(yy, SHORT)
bx.set(xlim=(-0.6, 2.3), xlabel="contribution to the log-odds")
bx.legend(frameon=False, fontsize=8, loc="upper right")
bx.set_title("(b) the reading (1, 1, 1, 1, 1.5), two games", fontsize=9)
fig.suptitle("The index keeps 56% of the variance, \\$0.14m of value (seed 26213)", fontsize=10.5)
plt.tight_layout(); plt.show()the record's principal components: eigenvalues 3.255, 1.427, 0.587, 0.321, 0.228
the first's loadings 0.47, 0.46, 0.43, 0.43, 0.44: an equal-weighted financial-conditions index
top r variance kept separation AUC the most a warning on them is worth
1 56.0% 0.843 0.724 0.14
2 80.5% 1.975 0.919 1.34
3 90.6% 2.028 0.924 1.44
4 96.1% 2.028 0.924 1.44
5 100.0% 2.029 0.924 1.44
subsets of the readings variance held AUC the most a warning on them is worth
the three market readings 51.1% 0.577 0.02
lending and funding 48.9% 0.914 1.27
funding alone 30.1% 0.910 1.21
all but funding 69.9% 0.698 0.10
the reading (1, 1, 1, 1, 1.5): log-odds -0.071, a chance of 48.2%: warned at the best threshold (4.09%): True; at 0.5: False
record averages 0.04 0.08 0.03 0.15 0.52
interventional -0.413 -0.242 -0.176 0.820 2.131 sum 2.120
conditional 0.048 0.098 0.043 0.462 1.469 sum 2.120
the three market readings: -0.83 interventional, 0.19 conditional; f(x) - f(mean) = 2.120
Proposition 13.3(ii): the first component keeps 80% of the variance and none of Y
Proposition 13.3(iv): f ignores x2, yet the conditional game credits it with 1 = (x1 - mu) / 2 at x1 = 2

Panel 8 — Every warning side by side (Worked Example 13.1; Table 13.3; Table 13.2)
Worked Example 13.1's warnings, their packages and their checks on Chapter 9's fresh paths, in Table 13.3's layout. Warned: the share of the 100,000 fresh programs warned; packages: line ($ million) and delay (quarters); promised value: Volume I's $10.44 million less the expected cost the design expects; fresh: the true chance of a breach and the value on the fresh paths, each program with its own readings. The two logistic rows marked law take each group's closure share from the law; the other logistic and boosting rows estimate it from the desk's record. A fresh value is shown only where the fresh breach is near the appetite. Table 13.2's last column reads the desk's warning by each measure.
Python Code
def row(name, warned, d, fresh, show_value=True):
cl, wa = d["clear"], d["warn"]
return dict(name=name, warned=float(warned), clear=None if cl is None else (cl[1], cl[2]),
warn=None if wa is None else (wa[1], wa[2]), promised=C5 - d["cost"], breach=fresh[0],
value=(C5 - fresh[1]) if show_value else None)
TABLE = [
row("None (Volume I)", 0.0, NONE, NONE_FRESH),
row("Perfect (Chapter 9)", FRESH.ahead.mean(), PERFECT, PERFECT_FRESH),
row("Declared law (ceiling)", CEIL_W.mean(), CEIL["design"], CEIL_FRESH),
row("Logistic, at 0.5 (law)", LAW05["fresh_warned"], LAW05["design"], (LAW05["breach"], LAW05["cost"])),
row("Logistic, cheapest (law)", LAWBEST["fresh_warned"], LAWBEST["design"], (LAWBEST["breach"], LAWBEST["cost"])),
row("Logistic, held-out", HELD["fresh_warned"], HELD["design"], (HELD["breach"], HELD["cost"])),
row("Logistic, certified", CERT["fresh_warned"], CERT["design"], (CERT["breach"], CERT["cost"])),
row("Boosting, trusted", TRUSTED["fresh_warned"], TRUSTED["design"], (TRUSTED["breach"], TRUSTED["cost"]), False),
row("Boosting, held-out", GB_HELD["fresh_warned"], GB_HELD["design"], (GB_HELD["breach"], GB_HELD["cost"])),
]
pk = lambda g: "--" if g is None else f"{g[0]:.0f}, {g[1]:.2f}"
print(f"{'Warning':26s} {'Warned':>7s} {'All clear':>11s} {'Warned':>11s} {'Promised':>8s} {'Fresh':>7s} {'Fresh':>6s}")
print(f"{'':26s} {'(%)':>7s} {'line, delay':>11s} {'line, delay':>11s} {'value':>8s} {'breach':>7s} {'value':>6s}")
for r in TABLE:
print(f"{r['name']:26s} {100 * r['warned']:7.1f} {pk(r['clear']):>11s} {pk(r['warn']):>11s} {r['promised']:8.2f} "
f"{100 * r['breach']:7.2f} {'--' if r['value'] is None else format(r['value'], '6.2f'):>6s}")
print("\nTable 13.2, the desk's warning by each measure (fresh paths):")
print(f" accuracy {100 * LAW05['accuracy']:.1f}% at 0.5, {100 * LAWBEST['accuracy']:.1f}% where cheapest")
print(f" AUC {AUC_LR['fresh']:.3f}")
print(f" decision value ${CEIL['value']:.2f}m at most (the ceiling)")
print(f" fresh breach {100 * CERT['breach']:.2f}% certified")Warning Warned All clear Warned Promised Fresh Fresh
(%) line, delay line, delay value breach value
None (Volume I) 0.0 170, 0.67 -- 0.00 4.91 0.00
Perfect (Chapter 9) 26.0 0, 0.61 215, 0.50 4.12 4.91 4.12
Declared law (ceiling) 58.9 0, 0.67 190, 0.73 1.44 4.90 1.43
Logistic, at 0.5 (law) 22.9 125, 0.71 205, 0.67 0.53 4.90 0.53
Logistic, cheapest (law) 58.6 0, 0.67 190, 0.73 1.44 4.93 1.46
Logistic, held-out 60.9 0, 0.68 185, 0.68 1.73 5.02 1.63
Logistic, certified 60.9 0, 0.73 190, 0.74 1.17 4.73 1.06
Boosting, trusted 35.7 0, 0.61 215, 0.51 4.04 7.68 --
Boosting, held-out 68.3 0, 0.76 190, 0.74 1.16 4.96 0.59
Table 13.2, the desk's warning by each measure (fresh paths):
accuracy 87.3% at 0.5, 65.9% where cheapest
AUC 0.924
decision value $1.44m at most (the ceiling)
fresh breach 4.73% certified
Validation — the book's numbers, and agreement with DCT_V2_Ch13_Lab.xlsx
Each checkpoint is a number printed in Chapter 13 (the abstract, Sections 13.2–13.4, Worked Example 13.1, Tables
13.2–13.3, Figures 13.1–13.3, the Key Takeaways, the LOS Guidance) or in Online Appendix Remark 13.B.1 and Example
13.B.2, with a tolerance of half a unit in its last printed digit; a few check what a theorem or proposition promises
(1 = holds). Chances and shares are in percent. The workbook recomputes the checkpoints marked workbook with
spreadsheet formulas and shows PASS against the same engine values on its Checks sheet.
Python Code
def bk(s):
# a number as the book prints it: its value and half a unit in its last printed digit
t = s.replace(",", "").replace("−", "-")
dp = len(t.split(".")[1]) if "." in t else 0
return float(t), 0.5 * 10.0 ** -dp + 1e-9
def c_(name, got, printed):
v, tol = bk(printed)
return (name, float(got), v, tol)
def flag(name, ok):
return (name, float(bool(ok)), 1.0, 0.0)
def count(name, got, n):
return (name, float(got), float(n), 0.0)
pc = lambda x: 100.0 * x
def design_checks(label, x, printed, wb=True):
# printed: (warned %, closure share warned %, closure share clear %, clear line, clear delay, clear cost,
# warned line, warned delay, warned cost, promised); None where the book prints nothing
keys = ("warned", "share warned", "share all clear", "all clear line", "all clear delay", "all clear cost",
"warned line", "warned delay", "warned cost", "promised value")
d = x["design"]
got = (pc(x["shares"][0]), pc(x["shares"][1]), pc(x["shares"][2]), d["clear"][1], d["clear"][2], d["clear"][0],
d["warn"][1], d["warn"][2], d["warn"][0], x["believed"] if "believed" in x else C5 - d["cost"])
out = []
for k_, g_, p_ in zip(keys, got, printed):
if p_ is None:
continue
mark = " (workbook)" if wb and k_ not in ("all clear line", "all clear delay", "warned line", "warned delay") else ""
out.append(c_(f"{label}: {k_}{mark}", g_, p_) if not k_.endswith("line") else count(f"{label}: {k_}", g_, int(p_)))
return out
cd, pf = CEIL["design"], PERFECT
checks = [
# Remark 13.B.1; the declarations
c_("closure ahead: book paths, % (Remark 13.B.1)", pc(PI1), "25.91"),
c_("closure ahead: fresh paths, %", pc(FRESH.ahead.mean()), "25.96"),
c_("closure ahead: the law, 1 - e^-0.3, % (workbook)", pc(P_CLOSE), "25.92"),
count("Volume I's package: line", NONE["clear"][1], 170),
c_("Volume I's package: delay", NONE["clear"][2], "0.668"),
c_("Volume I's package: cost (workbook)", C5, "10.44"),
c_("Volume I's package: chance on the book paths, % (workbook)", pc(SPLIT.chance(PI1, NONE["clear"])), "5.00"),
c_("Volume I's package: chance on the fresh paths, % (workbook)", pc(NONE_FRESH[0]), "4.91"),
# the perfect warning (WE 13.1, the situation; Example 13.B.2)
count("perfect warning: all clear line", pf["clear"][1], 0),
c_("perfect warning: all clear delay", pf["clear"][2], "0.608"),
c_("perfect warning: all clear delay, the situation", pf["clear"][2], "0.61"),
c_("perfect warning: all clear cost (workbook)", pf["clear"][0], "4.87"),
c_("perfect warning: all clear chance, % (workbook)", pc(SPLIT.chance(0.0, pf["clear"])), "4.75"),
count("perfect warning: warned line", pf["warn"][1], 215),
c_("perfect warning: warned delay", pf["warn"][2], "0.504"),
c_("perfect warning: warned delay, half a quarter", pf["warn"][2], "0.5"),
c_("perfect warning: warned cost (workbook)", pf["warn"][0], "10.49"),
c_("perfect warning: warned chance, % (workbook)", pc(SPLIT.chance(1.0, pf["warn"])), "5.71"),
c_("perfect warning: expected cost (workbook)", pf["cost"], "6.32"),
c_("perfect warning: value (workbook)", C5 - pf["cost"], "4.12"),
c_("perfect warning: fresh breach, %", pc(PERFECT_FRESH[0]), "4.91"),
c_("perfect warning: fresh cost (workbook)", PERFECT_FRESH[1], "6.33"),
# Panel 1: the record and the ceiling (Step 1; Remark 13.B.1; Example 13.B.2)
count("the record: windows", len(YR), 1000),
count("the record: windows with a closure", YR.sum(), 268),
] + [c_(f"the record: mean {s} reading with a closure", m, p) for s, m, p in
zip(SHORT, XR[YR].mean(0), ("0.18", "0.29", "0.17", "0.73", "2.00"))] + [
c_(f"the record: mean {s} reading without", m, p) for s, m, p in
zip(SHORT, XR[~YR].mean(0), ("-0.02", "0.01", "-0.02", "-0.06", "-0.02"))] + [
c_(f"the declared law: beta, {s} (workbook)", b, p) for s, b, p in
zip(SHORT, BETA_LAW, ("-0.323", "-0.159", "-0.352", "0.769", "1.974"))] + [
c_("the declared law: separation Delta (workbook)", DELTA, "2.029"),
c_("the declared law: AUC (workbook)", auc(DELTA), "0.924"),
c_("ceiling: threshold on the log-likelihood ratio", CEIL["t"], "-1.826"),
c_("ceiling: warned, % (workbook)", pc(CEIL["shares"][0]), "58.86"),
c_("ceiling: closure share, warned, % (workbook)", pc(CEIL["shares"][1]), "42.80"),
c_("ceiling: closure share, all clear, % (workbook)", pc(CEIL["shares"][2]), "1.75"),
count("ceiling: all clear line", cd["clear"][1], 0),
c_("ceiling: all clear delay", cd["clear"][2], "0.671"),
c_("ceiling: all clear cost (workbook)", cd["clear"][0], "5.37"),
count("ceiling: warned line", cd["warn"][1], 190),
c_("ceiling: warned delay", cd["warn"][2], "0.731"),
c_("ceiling: warned cost (workbook)", cd["warn"][0], "11.55"),
c_("ceiling: expected cost (workbook)", cd["cost"], "9.00"),
c_("ceiling: value (workbook)", CEIL["value"], "1.44"),
c_("ceiling: fresh warned, % (workbook)", pc(CEIL_W.mean()), "58.94"),
c_("ceiling: fresh breach, % (workbook)", pc(CEIL_FRESH[0]), "4.90"),
c_("ceiling: fresh value (workbook)", C5 - CEIL_FRESH[1], "1.43"),
] + [c_(f"value against AUC: AUC, shifts x {c:g} (workbook)", x["auc"], p) for c, x, p in
zip(SCALES, CURVE, ("0.640", "0.763", "0.859", "0.924", "0.964", "0.984", "0.998", "1.000", "1.000", "1.000"))] + [
c_(f"value against AUC: ceiling, shifts x {c:g}", x["value"], p) for c, x, p in
zip(SCALES, CURVE, ("0.05", "0.22", "0.63", "1.44", "2.24", "2.88", "3.69", "4.00", "4.09", "4.12"))] + [
c_("Step 1: $0.63m at an AUC of 0.86 (workbook)", CURVE[2]["auc"], "0.86"),
c_("Step 1: $2.24m at an AUC of 0.96 (workbook)", CURVE[4]["auc"], "0.96"),
# Panel 2: the learners (Step 2-3; Section 13.2; Example 13.B.2)
] + [c_(f"logistic: coefficient, {s}", b, p) for s, b, p in
zip(SHORT, LR.coef_[0], ("-0.428", "-0.264", "-0.183", "0.968", "2.180"))] + [
c_("logistic: intercept", LR.intercept_[0], "-3.435"),
c_("Step 5: the funding spread's weight", LR.coef_[0][4], "2.18"),
c_("Step 5: lending standards' weight", LR.coef_[0][3], "0.97"),
flag("Step 5: the three market readings weigh negatively", np.all(LR.coef_[0][:3] < 0)),
c_("logistic: AUC on the record", AUC_LR["record"], "0.942"),
c_("logistic: AUC held out", AUC_LR["oof"], "0.940"),
c_("logistic: AUC fresh", AUC_LR["fresh"], "0.924"),
c_("Step 2: AUC on the record", AUC_LR["record"], "0.94"),
c_("Step 2: AUC held out", AUC_LR["oof"], "0.94"),
c_("Step 2: AUC fresh", AUC_LR["fresh"], "0.92"),
c_("logistic: right at 0.5, the record, % (workbook)", pc(ACC_LR["record"]), "89.3"),
c_("logistic: right at 0.5, fresh, % (workbook)", pc(ACC_LR["fresh"]), "87.3"),
c_("never warning: right on fresh programs, % (workbook)", pc(NEVER), "74.0"),
c_("boosting: AUC on the record", AUC_GB["record"], "1.000"),
c_("boosting: right on the record, % (workbook)", pc(ACC_GB["record"]), "100"),
c_("boosting: AUC held out", AUC_GB["oof"], "0.922"),
c_("boosting: AUC fresh", AUC_GB["fresh"], "0.888"),
c_("boosting: right at 0.5, fresh, % (workbook)", pc(ACC_GB["fresh"]), "85.6"),
c_("Step 3: boosting's AUC held out", AUC_GB["oof"], "0.92"),
c_("Step 3: boosting's AUC fresh", AUC_GB["fresh"], "0.89"),
flag("Step 3: boosting's fresh AUC is below the logistic model's", AUC_GB["fresh"] < AUC_LR["fresh"]),
c_("boosting: fresh AUC at its peak (workbook)", ST_FR[PEAK], "0.910"),
count("boosting: the peak's stage (workbook)", PEAK + 1, 24),
c_("boosting: fresh AUC after 500 stages (workbook)", ST_FR[-1], "0.888"),
c_("boosting: record AUC after 500 stages (workbook)", ST_REC[-1], "1.000"),
flag("Proposition 13.2(iv): after the fresh peak the record's AUC rises and the fresh AUC falls",
ST_REC[-1] > ST_REC[PEAK] and ST_FR[-1] < ST_FR[PEAK]),
c_("Proposition 13.2(iii): each member right on, % (workbook)", pc(MEMBERS.min()), "60"),
c_("Proposition 13.2(iii): the majority vote right on, % (workbook)", pc(MAJORITY), "40"),
# Panel 3: price the warning (Step 2; Section 13.3; Figure 13.1(a); Example 13.B.2)
] + design_checks("logistic at 0.5, law's shares", LAW05,
("23.01", "78.37", "10.23", "125", "0.712", "9.45", "205", "0.665", "11.47", "0.53")) + [
c_("logistic at 0.5, law's shares: fresh warned, % (workbook)", pc(LAW05["fresh_warned"]), "22.9"),
c_("logistic at 0.5, law's shares: right on fresh programs, % (workbook)", pc(LAW05["accuracy"]), "87.3"),
c_("logistic at 0.5, law's shares: fresh breach, % (workbook)", pc(LAW05["breach"]), "4.90"),
c_("logistic at 0.5, law's shares: fresh value (workbook)", LAW05["value"], "0.53"),
c_("Step 2: warns at 0.5, %", pc(LAW05["shares"][0]), "23"),
c_("Step 2: the all-clear group's closures at 0.5, %", pc(LAW05["shares"][2]), "10.2"),
c_("logistic, cheapest: the threshold's chance, %", pc(LAWBEST["tau"]), "4.09"),
c_("Step 2: the threshold's chance, %", pc(LAWBEST["tau"]), "4.1"),
] + design_checks("logistic, cheapest, law's shares", LAWBEST,
("59.0", "42.71", "1.73", "0", "0.669", "5.35", "190", "0.730", "11.54", "1.44")) + [
c_("logistic, cheapest: fresh warned, % (workbook)", pc(LAWBEST["fresh_warned"]), "58.6"),
c_("logistic, cheapest: right on fresh programs, % (workbook)", pc(LAWBEST["accuracy"]), "65.9"),
c_("logistic, cheapest: fresh breach, % (workbook)", pc(LAWBEST["breach"]), "4.93"),
c_("logistic, cheapest: fresh value (workbook)", LAWBEST["value"], "1.46"),
c_("Step 2: warns where cheapest, %", pc(LAWBEST["shares"][0]), "59"),
flag("Step 2: the cheapest threshold is worth the ceiling, to the cent", round(LAWBEST["believed"], 2) == round(CEIL["value"], 2)),
flag("Theorem 13.2(ii): the valuable warning is less accurate than never warning", LAWBEST["accuracy"] < NEVER),
c_("Theorem 13.2(iv): squared error of the prediction 5 (workbook)", NV[5.0][0], "25"),
c_("Theorem 13.2(iv): cost of the decision on 5 (workbook)", NV[5.0][1], "25"),
c_("Theorem 13.2(iv): squared error of the prediction 10 (workbook)", NV[10.0][0], "50"),
c_("Theorem 13.2(iv): cost of the decision on 10 (workbook)", NV[10.0][1], "5"),
# Panel 4: boosting (Step 3; Example 13.B.2)
] + design_checks("boosting, trusted", TRUSTED,
("27.0", "99.26", "0", "0", "0.608", "4.87", "215", "0.513", "10.56", "4.04")) + [
c_("boosting, trusted: fresh warned, % (workbook)", pc(TRUSTED["fresh_warned"]), "35.7"),
c_("boosting, trusted: fresh breach, % (workbook)", pc(TRUSTED["breach"]), "7.68"),
] + design_checks("boosting, held-out", GB_HELD,
("58.0", "44.48", "2.38", "0", "0.757", "6.06", "190", "0.739", "11.61", "1.16")) + [
c_("boosting, held-out: fresh warned, % (workbook)", pc(GB_HELD["fresh_warned"]), "68.3"),
c_("boosting, held-out: fresh breach, % (workbook)", pc(GB_HELD["breach"]), "4.96"),
c_("boosting, held-out: fresh value (workbook)", GB_HELD["value"], "0.59"),
# Panel 5: certify (Step 4; Section 13.3; Example 13.B.2)
c_("logistic, held-out: the threshold's chance, %", pc(HELD["tau"]), "3.48"),
] + design_checks("logistic, held-out", HELD,
("59.0", "44.75", "0.98", "0", "0.683", "5.46", "185", "0.677", "10.96", "1.73")) + [
c_("logistic, held-out: fresh warned, % (workbook)", pc(HELD["fresh_warned"]), "60.9"),
c_("logistic, held-out: right on fresh programs, % (workbook)", pc(HELD["accuracy"]), "63.9"),
c_("logistic, held-out: fresh breach, % (workbook)", pc(HELD["breach"]), "5.02"),
c_("logistic, held-out: fresh value (workbook)", HELD["value"], "1.63"),
c_("Step 4: held-out share, warned, %", pc(HELD["shares"][1]), "44.7"),
c_("Step 4: held-out share, all clear, %", pc(HELD["shares"][2]), "1.0"),
flag("certified: the same threshold as the held-out design", CERT["tau"] == HELD["tau"]),
] + design_checks("logistic, certified", CERT,
(None, "48.21", "2.22", "0", "0.734", "5.87", "190", "0.742", "11.64", "1.17")) + [
c_("logistic, certified: fresh breach, % (workbook)", pc(CERT["breach"]), "4.73"),
c_("logistic, certified: fresh value (workbook)", CERT["value"], "1.06"),
c_("Step 4: certified bound, warned, %", pc(CERT["shares"][1]), "48.2"),
c_("Step 4: certified bound, all clear, %", pc(CERT["shares"][2]), "2.2"),
] + design_checks("logistic at 0.5, held-out shares", OOF05,
("25.2", "81.75", "8.29", None, None, None, None, None, None, "0.67")) + [
c_("logistic at 0.5, held-out shares: fresh breach, % (workbook)", pc(OOF05["breach"]), "5.03"),
# Panel 6: 200 records (Step 4; Figure 13.2(b); Example 13.B.2)
count("the study: records", len(STUDY), 200),
c_("200 records, held-out: keep 5%, % (workbook)", pc(SUMMARY["held"]["meet"]), "47.5"),
c_("200 records, held-out: mean true chance, % (workbook)", pc(SUMMARY["held"]["mean"]), "5.02"),
c_("200 records, held-out: 90th percentile, % (workbook)", pc(SUMMARY["held"]["q90"]), "5.21"),
c_("200 records, held-out: mean value (workbook)", SUMMARY["held"]["value"], "1.53"),
c_("200 records, certified: keep 5%, % (workbook)", pc(SUMMARY["cert"]["meet"]), "98.0"),
c_("200 records, certified: mean true chance, % (workbook)", pc(SUMMARY["cert"]["mean"]), "4.75"),
c_("200 records, certified: 90th percentile, % (workbook)", pc(SUMMARY["cert"]["q90"]), "4.89"),
c_("200 records, certified: mean value (workbook)", SUMMARY["cert"]["value"], "0.95"),
c_("Section 13.3: certified designs keep it in, % of records", pc(SUMMARY["cert"]["meet"]), "98"),
flag("Section 13.3: held-out designs keep the appetite only about half the time", 0.4 <= SUMMARY["held"]["meet"] <= 0.6),
# Panel 7: components and attributions (Step 5; Figure 13.3; Example 13.B.2)
] + [c_(f"principal components: eigenvalue {k + 1}", e, p) for k, (e, p) in
enumerate(zip(EIG, ("3.255", "1.427", "0.587", "0.321", "0.228")))] + [
c_(f"principal components: variance kept by the top {k + 1}, % (workbook)", pc(x), p) for k, (x, p) in
enumerate(zip(KEPT, ("56.0", "80.5", "90.6", "96.1", "100")))] + [
c_(f"the first component's loading, {s}", x, p) for s, x, p in zip(SHORT, V[:, 0], ("0.47", "0.46", "0.43", "0.43", "0.44"))] + [
c_(f"top {x['r']} components: AUC (workbook)", x["auc"], p) for x, p in zip(PCS[:3], ("0.724", "0.919", "0.924"))] + [
c_(f"top {x['r']} components: ceiling", x["value"], p) for x, p in zip(PCS, ("0.14", "1.34", "1.44", "1.44", "1.44"))] + [
c_("Step 5: the first component keeps, %", pc(KEPT[0]), "56"),
]
for name, (a_, v_, s_), short in (("the three market readings", ("0.577", "0.02", "51.1"), "51"),
("lending and funding", ("0.914", "1.27", "48.9"), None),
("funding alone", ("0.910", "1.21", "30.1"), "30"),
("all but funding", ("0.698", "0.10", "69.9"), None)):
s = SUBSETS[name]
checks += [c_(f"subset, {name}: AUC (workbook)", s["auc"], a_), c_(f"subset, {name}: ceiling", s["value"], v_),
c_(f"subset, {name}: variance held, % (workbook)", pc(s["var"]), s_)]
if short:
checks.append(c_(f"Step 5: {name} hold, %", pc(s["var"]), short))
checks += [
c_("the reading (1, 1, 1, 1, 1.5): log-odds (workbook)", LOGODDS0, "-0.071"),
c_("the reading: chance of a closure, % (workbook)", pc(PROB0), "48.2"),
flag("the reading: warned at the best threshold (workbook)", PROB0 >= LAWBEST["tau"]),
flag("the reading: not warned at 0.5 (workbook)", PROB0 < 0.5),
] + [c_(f"the record's mean {s} reading", m, p) for s, m, p in zip(SHORT, MU, ("0.04", "0.08", "0.03", "0.15", "0.52"))] + [
c_(f"interventional attribution, {s} (workbook)", x, p) for s, x, p in
zip(SHORT, SH_I, ("-0.413", "-0.242", "-0.176", "0.820", "2.131"))] + [
c_(f"conditional attribution, {s}", x, p) for s, x, p in zip(SHORT, SH_C, ("0.048", "0.098", "0.043", "0.462", "1.469"))] + [
c_("interventional attributions: sum (workbook)", SH_I.sum(), "2.120"),
c_("conditional attributions: sum (workbook)", SH_C.sum(), "2.120"),
c_("Step 5: the funding spread's interventional credit (workbook)", SH_I[4], "2.13"),
c_("Step 5: the market readings' interventional credit (workbook)", SH_I[:3].sum(), "-0.83"),
c_("Step 5: the funding spread's conditional credit", SH_C[4], "1.47"),
flag("Step 5: the conditional game gives every reading positive credit", np.all(SH_C > 0)),
c_("Proposition 13.3(ii): the first component keeps, % (workbook)", pc(PCA_EX), "80"),
c_("Proposition 13.3(iv): the conditional credit of an ignored feature (workbook)", PHI2, "1"),
]
# Panel 8: Table 13.3 and Table 13.2
T133 = {"None (Volume I)": ("0", None, None, None, None, "0", "4.91", "0"),
"Perfect (Chapter 9)": ("26.0", ("0", "0.61"), ("215", "0.50"), "4.12", "4.91", "4.12"),
"Declared law (ceiling)": ("58.9", ("0", "0.67"), ("190", "0.73"), "1.44", "4.90", "1.43"),
"Logistic, at 0.5 (law)": ("22.9", ("125", "0.71"), ("205", "0.67"), "0.53", "4.90", "0.53"),
"Logistic, cheapest (law)": ("58.6", ("0", "0.67"), ("190", "0.73"), "1.44", "4.93", "1.46"),
"Logistic, held-out": ("60.9", ("0", "0.68"), ("185", "0.68"), "1.73", "5.02", "1.63"),
"Logistic, certified": ("60.9", ("0", "0.73"), ("190", "0.74"), "1.17", "4.73", "1.06"),
"Boosting, trusted": ("35.7", ("0", "0.61"), ("215", "0.51"), "4.04", "7.68", None),
"Boosting, held-out": ("68.3", ("0", "0.76"), ("190", "0.74"), "1.16", "4.96", "0.59")}
for r in TABLE:
b = T133[r["name"]]
if r["name"] == "None (Volume I)":
checks += [c_("Table 13.3, None: warned, %", pc(r["warned"]), "0"),
count("Table 13.3, None: all clear line", r["clear"][0], 170),
c_("Table 13.3, None: all clear delay", r["clear"][1], "0.67"),
c_("Table 13.3, None: promised value (workbook)", r["promised"], "0"),
c_("Table 13.3, None: fresh breach, % (workbook)", pc(r["breach"]), "4.91"),
c_("Table 13.3, None: fresh value (workbook)", r["value"], "0")]
continue
w_, (cl_l, cl_d), (wa_l, wa_d), pr_, br_, fv_ = b
nm = f"Table 13.3, {r['name']}"
checks += [c_(f"{nm}: warned, % (workbook)", pc(r["warned"]), w_),
count(f"{nm}: all clear line", r["clear"][0], int(cl_l)), c_(f"{nm}: all clear delay", r["clear"][1], cl_d),
count(f"{nm}: warned line", r["warn"][0], int(wa_l)), c_(f"{nm}: warned delay", r["warn"][1], wa_d),
c_(f"{nm}: promised value (workbook)", r["promised"], pr_), c_(f"{nm}: fresh breach, % (workbook)", pc(r["breach"]), br_)]
if fv_ is not None:
checks.append(c_(f"{nm}: fresh value (workbook)", r["value"], fv_))
checks += [
c_("Table 13.2: accuracy at 0.5, %", pc(LAW05["accuracy"]), "87.3"),
c_("Table 13.2: accuracy where cheapest, %", pc(LAWBEST["accuracy"]), "65.9"),
c_("Table 13.2: AUC", AUC_LR["fresh"], "0.924"),
c_("Table 13.2: decision value at most", CEIL["value"], "1.44"),
c_("Table 13.2: fresh breach, certified, %", pc(CERT["breach"]), "4.73"),
# the abstract, the board's take-away, Key Takeaways, LOS Guidance
c_("abstract: a perfect warning", C5 - pf["cost"], "4.12"),
c_("abstract: the five readings allow at most", CEIL["value"], "1.44"),
c_("abstract: the certified warning is worth", CERT["value"], "1.06"),
flag("the board's take-away: the ceiling is about a third of a perfect warning", 0.30 <= CEIL["value"] / (C5 - pf["cost"]) <= 0.37),
c_("the board's take-away: trusted boosting promises nearly", TRUSTED["believed"], "4"),
c_("the board's take-away and Key Takeaways: certified, about", CERT["value"], "1"),
c_("LOS 13.2: boosting on the record", AUC_GB["record"], "1.00"),
c_("LOS 13.2: boosting fresh", AUC_GB["fresh"], "0.89"),
c_("LOS 13.3: at accuracy's threshold", LAW05["believed"], "0.53"),
c_("LOS 13.3: at a 4.1% threshold", LAWBEST["believed"], "1.44"),
c_("LOS 13.4: held-out designs keep 5%, % of records", pc(SUMMARY["held"]["meet"]), "47.5"),
c_("LOS 13.4: certified designs keep 5%, % of records", pc(SUMMARY["cert"]["meet"]), "98.0"),
c_("LOS 13.4: at an average value of", SUMMARY["cert"]["value"], "0.95"),
c_("LOS 13.5: the first component's value", PCS[0]["value"], "0.14"),
c_("LOS 13.5: the funding spread alone keeps, %", pc(SUBSETS["funding alone"]["var"]), "30"),
]
ok = 0
for name, got, want, tol in checks:
status = "PASS" if abs(got - want) <= tol else "FAIL"
ok += status == "PASS"
print(f"{name:84s} {got:11.4f} book {want:10.4f} {status}")
print(f"\n{ok} of {len(checks)} checkpoints PASS")closure ahead: book paths, % (Remark 13.B.1) 25.9120 book 25.9100 PASS closure ahead: fresh paths, % 25.9600 book 25.9600 PASS closure ahead: the law, 1 - e^-0.3, % (workbook) 25.9182 book 25.9200 PASS Volume I's package: line 170.0000 book 170.0000 PASS Volume I's package: delay 0.6676 book 0.6680 PASS Volume I's package: cost (workbook) 10.4406 book 10.4400 PASS Volume I's package: chance on the book paths, % (workbook) 5.0000 book 5.0000 PASS Volume I's package: chance on the fresh paths, % (workbook) 4.9070 book 4.9100 PASS perfect warning: all clear line 0.0000 book 0.0000 PASS perfect warning: all clear delay 0.6084 book 0.6080 PASS perfect warning: all clear delay, the situation 0.6084 book 0.6100 PASS perfect warning: all clear cost (workbook) 4.8668 book 4.8700 PASS perfect warning: all clear chance, % (workbook) 4.7498 book 4.7500 PASS perfect warning: warned line 215.0000 book 215.0000 PASS perfect warning: warned delay 0.5044 book 0.5040 PASS perfect warning: warned delay, half a quarter 0.5044 book 0.5000 PASS perfect warning: warned cost (workbook) 10.4851 book 10.4900 PASS perfect warning: warned chance, % (workbook) 5.7116 book 5.7100 PASS perfect warning: expected cost (workbook) 6.3226 book 6.3200 PASS perfect warning: value (workbook) 4.1180 book 4.1200 PASS perfect warning: fresh breach, % 4.9130 book 4.9100 PASS perfect warning: fresh cost (workbook) 6.3253 book 6.3300 PASS the record: windows 1000.0000 book 1000.0000 PASS the record: windows with a closure 268.0000 book 268.0000 PASS the record: mean bond reading with a closure 0.1790 book 0.1800 PASS the record: mean high yield reading with a closure 0.2920 book 0.2900 PASS the record: mean volatility reading with a closure 0.1717 book 0.1700 PASS the record: mean lending reading with a closure 0.7348 book 0.7300 PASS the record: mean funding reading with a closure 1.9953 book 2.0000 PASS the record: mean bond reading without -0.0171 book -0.0200 PASS the record: mean high yield reading without 0.0079 book 0.0100 PASS the record: mean volatility reading without -0.0163 book -0.0200 PASS the record: mean lending reading without -0.0605 book -0.0600 PASS the record: mean funding reading without -0.0167 book -0.0200 PASS the declared law: beta, bond (workbook) -0.3232 book -0.3230 PASS the declared law: beta, high yield (workbook) -0.1589 book -0.1590 PASS the declared law: beta, volatility (workbook) -0.3522 book -0.3520 PASS the declared law: beta, lending (workbook) 0.7694 book 0.7690 PASS the declared law: beta, funding (workbook) 1.9738 book 1.9740 PASS the declared law: separation Delta (workbook) 2.0287 book 2.0290 PASS the declared law: AUC (workbook) 0.9243 book 0.9240 PASS ceiling: threshold on the log-likelihood ratio -1.8258 book -1.8260 PASS ceiling: warned, % (workbook) 58.8639 book 58.8600 PASS ceiling: closure share, warned, % (workbook) 42.7969 book 42.8000 PASS ceiling: closure share, all clear, % (workbook) 1.7504 book 1.7500 PASS ceiling: all clear line 0.0000 book 0.0000 PASS ceiling: all clear delay 0.6708 book 0.6710 PASS ceiling: all clear cost (workbook) 5.3661 book 5.3700 PASS ceiling: warned line 190.0000 book 190.0000 PASS ceiling: warned delay 0.7307 book 0.7310 PASS ceiling: warned cost (workbook) 11.5458 book 11.5500 PASS ceiling: expected cost (workbook) 9.0037 book 9.0000 PASS ceiling: value (workbook) 1.4369 book 1.4400 PASS ceiling: fresh warned, % (workbook) 58.9400 book 58.9400 PASS ceiling: fresh breach, % (workbook) 4.8980 book 4.9000 PASS ceiling: fresh value (workbook) 1.4322 book 1.4300 PASS value against AUC: AUC, shifts x 0.25 (workbook) 0.6401 book 0.6400 PASS value against AUC: AUC, shifts x 0.5 (workbook) 0.7634 book 0.7630 PASS value against AUC: AUC, shifts x 0.75 (workbook) 0.8590 book 0.8590 PASS value against AUC: AUC, shifts x 1 (workbook) 0.9243 book 0.9240 PASS value against AUC: AUC, shifts x 1.25 (workbook) 0.9635 book 0.9640 PASS value against AUC: AUC, shifts x 1.5 (workbook) 0.9843 book 0.9840 PASS value against AUC: AUC, shifts x 2 (workbook) 0.9979 book 0.9980 PASS value against AUC: AUC, shifts x 2.5 (workbook) 0.9998 book 1.0000 PASS value against AUC: AUC, shifts x 3 (workbook) 1.0000 book 1.0000 PASS value against AUC: AUC, shifts x 4 (workbook) 1.0000 book 1.0000 PASS value against AUC: ceiling, shifts x 0.25 0.0487 book 0.0500 PASS value against AUC: ceiling, shifts x 0.5 0.2207 book 0.2200 PASS value against AUC: ceiling, shifts x 0.75 0.6277 book 0.6300 PASS value against AUC: ceiling, shifts x 1 1.4369 book 1.4400 PASS value against AUC: ceiling, shifts x 1.25 2.2374 book 2.2400 PASS value against AUC: ceiling, shifts x 1.5 2.8796 book 2.8800 PASS value against AUC: ceiling, shifts x 2 3.6935 book 3.6900 PASS value against AUC: ceiling, shifts x 2.5 3.9987 book 4.0000 PASS value against AUC: ceiling, shifts x 3 4.0930 book 4.0900 PASS value against AUC: ceiling, shifts x 4 4.1178 book 4.1200 PASS Step 1: $0.63m at an AUC of 0.86 (workbook) 0.8590 book 0.8600 PASS Step 1: $2.24m at an AUC of 0.96 (workbook) 0.9635 book 0.9600 PASS logistic: coefficient, bond -0.4280 book -0.4280 PASS logistic: coefficient, high yield -0.2641 book -0.2640 PASS logistic: coefficient, volatility -0.1825 book -0.1830 PASS logistic: coefficient, lending 0.9675 book 0.9680 PASS logistic: coefficient, funding 2.1804 book 2.1800 PASS logistic: intercept -3.4346 book -3.4350 PASS Step 5: the funding spread's weight 2.1804 book 2.1800 PASS Step 5: lending standards' weight 0.9675 book 0.9700 PASS Step 5: the three market readings weigh negatively 1.0000 book 1.0000 PASS logistic: AUC on the record 0.9419 book 0.9420 PASS logistic: AUC held out 0.9401 book 0.9400 PASS logistic: AUC fresh 0.9240 book 0.9240 PASS Step 2: AUC on the record 0.9419 book 0.9400 PASS Step 2: AUC held out 0.9401 book 0.9400 PASS Step 2: AUC fresh 0.9240 book 0.9200 PASS logistic: right at 0.5, the record, % (workbook) 89.3000 book 89.3000 PASS logistic: right at 0.5, fresh, % (workbook) 87.3240 book 87.3000 PASS never warning: right on fresh programs, % (workbook) 74.0400 book 74.0000 PASS boosting: AUC on the record 1.0000 book 1.0000 PASS boosting: right on the record, % (workbook) 100.0000 book 100.0000 PASS boosting: AUC held out 0.9223 book 0.9220 PASS boosting: AUC fresh 0.8883 book 0.8880 PASS boosting: right at 0.5, fresh, % (workbook) 85.6440 book 85.6000 PASS Step 3: boosting's AUC held out 0.9223 book 0.9200 PASS Step 3: boosting's AUC fresh 0.8883 book 0.8900 PASS Step 3: boosting's fresh AUC is below the logistic model's 1.0000 book 1.0000 PASS boosting: fresh AUC at its peak (workbook) 0.9103 book 0.9100 PASS boosting: the peak's stage (workbook) 24.0000 book 24.0000 PASS boosting: fresh AUC after 500 stages (workbook) 0.8885 book 0.8880 PASS boosting: record AUC after 500 stages (workbook) 1.0000 book 1.0000 PASS Proposition 13.2(iv): after the fresh peak the record's AUC rises and the fresh AUC falls 1.0000 book 1.0000 PASS Proposition 13.2(iii): each member right on, % (workbook) 60.0000 book 60.0000 PASS Proposition 13.2(iii): the majority vote right on, % (workbook) 40.0000 book 40.0000 PASS logistic at 0.5, law's shares: warned (workbook) 23.0120 book 23.0100 PASS logistic at 0.5, law's shares: share warned (workbook) 78.3678 book 78.3700 PASS logistic at 0.5, law's shares: share all clear (workbook) 10.2328 book 10.2300 PASS logistic at 0.5, law's shares: all clear line 125.0000 book 125.0000 PASS logistic at 0.5, law's shares: all clear delay 0.7121 book 0.7120 PASS logistic at 0.5, law's shares: all clear cost (workbook) 9.4469 book 9.4500 PASS logistic at 0.5, law's shares: warned line 205.0000 book 205.0000 PASS logistic at 0.5, law's shares: warned delay 0.6654 book 0.6650 PASS logistic at 0.5, law's shares: warned cost (workbook) 11.4735 book 11.4700 PASS logistic at 0.5, law's shares: promised value (workbook) 0.5273 book 0.5300 PASS logistic at 0.5, law's shares: fresh warned, % (workbook) 22.9240 book 22.9000 PASS logistic at 0.5, law's shares: right on fresh programs, % (workbook) 87.3240 book 87.3000 PASS logistic at 0.5, law's shares: fresh breach, % (workbook) 4.8980 book 4.9000 PASS logistic at 0.5, law's shares: fresh value (workbook) 0.5291 book 0.5300 PASS Step 2: warns at 0.5, % 23.0120 book 23.0000 PASS Step 2: the all-clear group's closures at 0.5, % 10.2328 book 10.2000 PASS logistic, cheapest: the threshold's chance, % 4.0935 book 4.0900 PASS Step 2: the threshold's chance, % 4.0935 book 4.1000 PASS logistic, cheapest, law's shares: warned (workbook) 59.0000 book 59.0000 PASS logistic, cheapest, law's shares: share warned (workbook) 42.7136 book 42.7100 PASS logistic, cheapest, law's shares: share all clear (workbook) 1.7341 book 1.7300 PASS logistic, cheapest, law's shares: all clear line 0.0000 book 0.0000 PASS logistic, cheapest, law's shares: all clear delay 0.6687 book 0.6690 PASS logistic, cheapest, law's shares: all clear cost (workbook) 5.3496 book 5.3500 PASS logistic, cheapest, law's shares: warned line 190.0000 book 190.0000 PASS logistic, cheapest, law's shares: warned delay 0.7305 book 0.7300 PASS logistic, cheapest, law's shares: warned cost (workbook) 11.5437 book 11.5400 PASS logistic, cheapest, law's shares: promised value (workbook) 1.4365 book 1.4400 PASS logistic, cheapest: fresh warned, % (workbook) 58.6460 book 58.6000 PASS logistic, cheapest: right on fresh programs, % (workbook) 65.8620 book 65.9000 PASS logistic, cheapest: fresh breach, % (workbook) 4.9280 book 4.9300 PASS logistic, cheapest: fresh value (workbook) 1.4584 book 1.4600 PASS Step 2: warns where cheapest, % 59.0000 book 59.0000 PASS Step 2: the cheapest threshold is worth the ceiling, to the cent 1.0000 book 1.0000 PASS Theorem 13.2(ii): the valuable warning is less accurate than never warning 1.0000 book 1.0000 PASS Theorem 13.2(iv): squared error of the prediction 5 (workbook) 25.0000 book 25.0000 PASS Theorem 13.2(iv): cost of the decision on 5 (workbook) 25.0000 book 25.0000 PASS Theorem 13.2(iv): squared error of the prediction 10 (workbook) 50.0000 book 50.0000 PASS Theorem 13.2(iv): cost of the decision on 10 (workbook) 5.0000 book 5.0000 PASS boosting, trusted: warned (workbook) 27.0000 book 27.0000 PASS boosting, trusted: share warned (workbook) 99.2593 book 99.2600 PASS boosting, trusted: share all clear (workbook) 0.0000 book 0.0000 PASS boosting, trusted: all clear line 0.0000 book 0.0000 PASS boosting, trusted: all clear delay 0.6084 book 0.6080 PASS boosting, trusted: all clear cost (workbook) 4.8668 book 4.8700 PASS boosting, trusted: warned line 215.0000 book 215.0000 PASS boosting, trusted: warned delay 0.5132 book 0.5130 PASS boosting, trusted: warned cost (workbook) 10.5559 book 10.5600 PASS boosting, trusted: promised value (workbook) 4.0377 book 4.0400 PASS boosting, trusted: fresh warned, % (workbook) 35.7200 book 35.7000 PASS boosting, trusted: fresh breach, % (workbook) 7.6760 book 7.6800 PASS boosting, held-out: warned (workbook) 58.0000 book 58.0000 PASS boosting, held-out: share warned (workbook) 44.4828 book 44.4800 PASS boosting, held-out: share all clear (workbook) 2.3810 book 2.3800 PASS boosting, held-out: all clear line 0.0000 book 0.0000 PASS boosting, held-out: all clear delay 0.7574 book 0.7570 PASS boosting, held-out: all clear cost (workbook) 6.0594 book 6.0600 PASS boosting, held-out: warned line 190.0000 book 190.0000 PASS boosting, held-out: warned delay 0.7385 book 0.7390 PASS boosting, held-out: warned cost (workbook) 11.6083 book 11.6100 PASS boosting, held-out: promised value (workbook) 1.1629 book 1.1600 PASS boosting, held-out: fresh warned, % (workbook) 68.3090 book 68.3000 PASS boosting, held-out: fresh breach, % (workbook) 4.9600 book 4.9600 PASS boosting, held-out: fresh value (workbook) 0.5908 book 0.5900 PASS logistic, held-out: the threshold's chance, % 3.4785 book 3.4800 PASS logistic, held-out: warned (workbook) 59.0000 book 59.0000 PASS logistic, held-out: share warned (workbook) 44.7458 book 44.7500 PASS logistic, held-out: share all clear (workbook) 0.9756 book 0.9800 PASS logistic, held-out: all clear line 0.0000 book 0.0000 PASS logistic, held-out: all clear delay 0.6830 book 0.6830 PASS logistic, held-out: all clear cost (workbook) 5.4643 book 5.4600 PASS logistic, held-out: warned line 185.0000 book 185.0000 PASS logistic, held-out: warned delay 0.6768 book 0.6770 PASS logistic, held-out: warned cost (workbook) 10.9640 book 10.9600 PASS logistic, held-out: promised value (workbook) 1.7315 book 1.7300 PASS logistic, held-out: fresh warned, % (workbook) 60.8910 book 60.9000 PASS logistic, held-out: right on fresh programs, % (workbook) 63.8570 book 63.9000 PASS logistic, held-out: fresh breach, % (workbook) 5.0190 book 5.0200 PASS logistic, held-out: fresh value (workbook) 1.6275 book 1.6300 PASS Step 4: held-out share, warned, % 44.7458 book 44.7000 PASS Step 4: held-out share, all clear, % 0.9756 book 1.0000 PASS certified: the same threshold as the held-out design 1.0000 book 1.0000 PASS logistic, certified: share warned (workbook) 48.2117 book 48.2100 PASS logistic, certified: share all clear (workbook) 2.2186 book 2.2200 PASS logistic, certified: all clear line 0.0000 book 0.0000 PASS logistic, certified: all clear delay 0.7340 book 0.7340 PASS logistic, certified: all clear cost (workbook) 5.8720 book 5.8700 PASS logistic, certified: warned line 190.0000 book 190.0000 PASS logistic, certified: warned delay 0.7419 book 0.7420 PASS logistic, certified: warned cost (workbook) 11.6354 book 11.6400 PASS logistic, certified: promised value (workbook) 1.1683 book 1.1700 PASS logistic, certified: fresh breach, % (workbook) 4.7260 book 4.7300 PASS logistic, certified: fresh value (workbook) 1.0593 book 1.0600 PASS Step 4: certified bound, warned, % 48.2117 book 48.2000 PASS Step 4: certified bound, all clear, % 2.2186 book 2.2000 PASS logistic at 0.5, held-out shares: warned (workbook) 25.2000 book 25.2000 PASS logistic at 0.5, held-out shares: share warned (workbook) 81.7460 book 81.7500 PASS logistic at 0.5, held-out shares: share all clear (workbook) 8.2888 book 8.2900 PASS logistic at 0.5, held-out shares: promised value (workbook) 0.6661 book 0.6700 PASS logistic at 0.5, held-out shares: fresh breach, % (workbook) 5.0330 book 5.0300 PASS the study: records 200.0000 book 200.0000 PASS 200 records, held-out: keep 5%, % (workbook) 47.5000 book 47.5000 PASS 200 records, held-out: mean true chance, % (workbook) 5.0197 book 5.0200 PASS 200 records, held-out: 90th percentile, % (workbook) 5.2131 book 5.2100 PASS 200 records, held-out: mean value (workbook) 1.5333 book 1.5300 PASS 200 records, certified: keep 5%, % (workbook) 98.0000 book 98.0000 PASS 200 records, certified: mean true chance, % (workbook) 4.7545 book 4.7500 PASS 200 records, certified: 90th percentile, % (workbook) 4.8911 book 4.8900 PASS 200 records, certified: mean value (workbook) 0.9502 book 0.9500 PASS Section 13.3: certified designs keep it in, % of records 98.0000 book 98.0000 PASS Section 13.3: held-out designs keep the appetite only about half the time 1.0000 book 1.0000 PASS principal components: eigenvalue 1 3.2547 book 3.2550 PASS principal components: eigenvalue 2 1.4271 book 1.4270 PASS principal components: eigenvalue 3 0.5865 book 0.5870 PASS principal components: eigenvalue 4 0.3207 book 0.3210 PASS principal components: eigenvalue 5 0.2279 book 0.2280 PASS principal components: variance kept by the top 1, % (workbook) 55.9520 book 56.0000 PASS principal components: variance kept by the top 2, % (workbook) 80.4854 book 80.5000 PASS principal components: variance kept by the top 3, % (workbook) 90.5686 book 90.6000 PASS principal components: variance kept by the top 4, % (workbook) 96.0822 book 96.1000 PASS principal components: variance kept by the top 5, % (workbook) 100.0000 book 100.0000 PASS the first component's loading, bond 0.4734 book 0.4700 PASS the first component's loading, high yield 0.4599 book 0.4600 PASS the first component's loading, volatility 0.4311 book 0.4300 PASS the first component's loading, lending 0.4345 book 0.4300 PASS the first component's loading, funding 0.4355 book 0.4400 PASS top 1 components: AUC (workbook) 0.7245 book 0.7240 PASS top 2 components: AUC (workbook) 0.9188 book 0.9190 PASS top 3 components: AUC (workbook) 0.9242 book 0.9240 PASS top 1 components: ceiling 0.1409 book 0.1400 PASS top 2 components: ceiling 1.3390 book 1.3400 PASS top 3 components: ceiling 1.4364 book 1.4400 PASS top 4 components: ceiling 1.4365 book 1.4400 PASS top 5 components: ceiling 1.4369 book 1.4400 PASS Step 5: the first component keeps, % 55.9520 book 56.0000 PASS subset, the three market readings: AUC (workbook) 0.5773 book 0.5770 PASS subset, the three market readings: ceiling 0.0240 book 0.0200 PASS subset, the three market readings: variance held, % (workbook) 51.0963 book 51.1000 PASS Step 5: the three market readings hold, % 51.0963 book 51.0000 PASS subset, lending and funding: AUC (workbook) 0.9143 book 0.9140 PASS subset, lending and funding: ceiling 1.2710 book 1.2700 PASS subset, lending and funding: variance held, % (workbook) 48.9037 book 48.9000 PASS subset, funding alone: AUC (workbook) 0.9104 book 0.9100 PASS subset, funding alone: ceiling 1.2145 book 1.2100 PASS subset, funding alone: variance held, % (workbook) 30.1408 book 30.1000 PASS Step 5: funding alone hold, % 30.1408 book 30.0000 PASS subset, all but funding: AUC (workbook) 0.6976 book 0.6980 PASS subset, all but funding: ceiling 0.0991 book 0.1000 PASS subset, all but funding: variance held, % (workbook) 69.8592 book 69.9000 PASS the reading (1, 1, 1, 1, 1.5): log-odds (workbook) -0.0711 book -0.0710 PASS the reading: chance of a closure, % (workbook) 48.2221 book 48.2000 PASS the reading: warned at the best threshold (workbook) 1.0000 book 1.0000 PASS the reading: not warned at 0.5 (workbook) 1.0000 book 1.0000 PASS the record's mean bond reading 0.0354 book 0.0400 PASS the record's mean high yield reading 0.0840 book 0.0800 PASS the record's mean volatility reading 0.0341 book 0.0300 PASS the record's mean lending reading 0.1527 book 0.1500 PASS the record's mean funding reading 0.5225 book 0.5200 PASS interventional attribution, bond (workbook) -0.4128 book -0.4130 PASS interventional attribution, high yield (workbook) -0.2419 book -0.2420 PASS interventional attribution, volatility (workbook) -0.1763 book -0.1760 PASS interventional attribution, lending (workbook) 0.8198 book 0.8200 PASS interventional attribution, funding (workbook) 2.1313 book 2.1310 PASS conditional attribution, bond 0.0484 book 0.0480 PASS conditional attribution, high yield 0.0980 book 0.0980 PASS conditional attribution, volatility 0.0434 book 0.0430 PASS conditional attribution, lending 0.4615 book 0.4620 PASS conditional attribution, funding 1.4688 book 1.4690 PASS interventional attributions: sum (workbook) 2.1201 book 2.1200 PASS conditional attributions: sum (workbook) 2.1201 book 2.1200 PASS Step 5: the funding spread's interventional credit (workbook) 2.1313 book 2.1300 PASS Step 5: the market readings' interventional credit (workbook) -0.8310 book -0.8300 PASS Step 5: the funding spread's conditional credit 1.4688 book 1.4700 PASS Step 5: the conditional game gives every reading positive credit 1.0000 book 1.0000 PASS Proposition 13.3(ii): the first component keeps, % (workbook) 80.0000 book 80.0000 PASS Proposition 13.3(iv): the conditional credit of an ignored feature (workbook) 1.0000 book 1.0000 PASS Table 13.3, None: warned, % 0.0000 book 0.0000 PASS Table 13.3, None: all clear line 170.0000 book 170.0000 PASS Table 13.3, None: all clear delay 0.6676 book 0.6700 PASS Table 13.3, None: promised value (workbook) 0.0000 book 0.0000 PASS Table 13.3, None: fresh breach, % (workbook) 4.9070 book 4.9100 PASS Table 13.3, None: fresh value (workbook) 0.0000 book 0.0000 PASS Table 13.3, Perfect (Chapter 9): warned, % (workbook) 25.9600 book 26.0000 PASS Table 13.3, Perfect (Chapter 9): all clear line 0.0000 book 0.0000 PASS Table 13.3, Perfect (Chapter 9): all clear delay 0.6084 book 0.6100 PASS Table 13.3, Perfect (Chapter 9): warned line 215.0000 book 215.0000 PASS Table 13.3, Perfect (Chapter 9): warned delay 0.5044 book 0.5000 PASS Table 13.3, Perfect (Chapter 9): promised value (workbook) 4.1180 book 4.1200 PASS Table 13.3, Perfect (Chapter 9): fresh breach, % (workbook) 4.9130 book 4.9100 PASS Table 13.3, Perfect (Chapter 9): fresh value (workbook) 4.1153 book 4.1200 PASS Table 13.3, Declared law (ceiling): warned, % (workbook) 58.9400 book 58.9000 PASS Table 13.3, Declared law (ceiling): all clear line 0.0000 book 0.0000 PASS Table 13.3, Declared law (ceiling): all clear delay 0.6708 book 0.6700 PASS Table 13.3, Declared law (ceiling): warned line 190.0000 book 190.0000 PASS Table 13.3, Declared law (ceiling): warned delay 0.7307 book 0.7300 PASS Table 13.3, Declared law (ceiling): promised value (workbook) 1.4369 book 1.4400 PASS Table 13.3, Declared law (ceiling): fresh breach, % (workbook) 4.8980 book 4.9000 PASS Table 13.3, Declared law (ceiling): fresh value (workbook) 1.4322 book 1.4300 PASS Table 13.3, Logistic, at 0.5 (law): warned, % (workbook) 22.9240 book 22.9000 PASS Table 13.3, Logistic, at 0.5 (law): all clear line 125.0000 book 125.0000 PASS Table 13.3, Logistic, at 0.5 (law): all clear delay 0.7121 book 0.7100 PASS Table 13.3, Logistic, at 0.5 (law): warned line 205.0000 book 205.0000 PASS Table 13.3, Logistic, at 0.5 (law): warned delay 0.6654 book 0.6700 PASS Table 13.3, Logistic, at 0.5 (law): promised value (workbook) 0.5273 book 0.5300 PASS Table 13.3, Logistic, at 0.5 (law): fresh breach, % (workbook) 4.8980 book 4.9000 PASS Table 13.3, Logistic, at 0.5 (law): fresh value (workbook) 0.5291 book 0.5300 PASS Table 13.3, Logistic, cheapest (law): warned, % (workbook) 58.6460 book 58.6000 PASS Table 13.3, Logistic, cheapest (law): all clear line 0.0000 book 0.0000 PASS Table 13.3, Logistic, cheapest (law): all clear delay 0.6687 book 0.6700 PASS Table 13.3, Logistic, cheapest (law): warned line 190.0000 book 190.0000 PASS Table 13.3, Logistic, cheapest (law): warned delay 0.7305 book 0.7300 PASS Table 13.3, Logistic, cheapest (law): promised value (workbook) 1.4365 book 1.4400 PASS Table 13.3, Logistic, cheapest (law): fresh breach, % (workbook) 4.9280 book 4.9300 PASS Table 13.3, Logistic, cheapest (law): fresh value (workbook) 1.4584 book 1.4600 PASS Table 13.3, Logistic, held-out: warned, % (workbook) 60.8910 book 60.9000 PASS Table 13.3, Logistic, held-out: all clear line 0.0000 book 0.0000 PASS Table 13.3, Logistic, held-out: all clear delay 0.6830 book 0.6800 PASS Table 13.3, Logistic, held-out: warned line 185.0000 book 185.0000 PASS Table 13.3, Logistic, held-out: warned delay 0.6768 book 0.6800 PASS Table 13.3, Logistic, held-out: promised value (workbook) 1.7315 book 1.7300 PASS Table 13.3, Logistic, held-out: fresh breach, % (workbook) 5.0190 book 5.0200 PASS Table 13.3, Logistic, held-out: fresh value (workbook) 1.6275 book 1.6300 PASS Table 13.3, Logistic, certified: warned, % (workbook) 60.8910 book 60.9000 PASS Table 13.3, Logistic, certified: all clear line 0.0000 book 0.0000 PASS Table 13.3, Logistic, certified: all clear delay 0.7340 book 0.7300 PASS Table 13.3, Logistic, certified: warned line 190.0000 book 190.0000 PASS Table 13.3, Logistic, certified: warned delay 0.7419 book 0.7400 PASS Table 13.3, Logistic, certified: promised value (workbook) 1.1683 book 1.1700 PASS Table 13.3, Logistic, certified: fresh breach, % (workbook) 4.7260 book 4.7300 PASS Table 13.3, Logistic, certified: fresh value (workbook) 1.0593 book 1.0600 PASS Table 13.3, Boosting, trusted: warned, % (workbook) 35.7200 book 35.7000 PASS Table 13.3, Boosting, trusted: all clear line 0.0000 book 0.0000 PASS Table 13.3, Boosting, trusted: all clear delay 0.6084 book 0.6100 PASS Table 13.3, Boosting, trusted: warned line 215.0000 book 215.0000 PASS Table 13.3, Boosting, trusted: warned delay 0.5132 book 0.5100 PASS Table 13.3, Boosting, trusted: promised value (workbook) 4.0377 book 4.0400 PASS Table 13.3, Boosting, trusted: fresh breach, % (workbook) 7.6760 book 7.6800 PASS Table 13.3, Boosting, held-out: warned, % (workbook) 68.3090 book 68.3000 PASS Table 13.3, Boosting, held-out: all clear line 0.0000 book 0.0000 PASS Table 13.3, Boosting, held-out: all clear delay 0.7574 book 0.7600 PASS Table 13.3, Boosting, held-out: warned line 190.0000 book 190.0000 PASS Table 13.3, Boosting, held-out: warned delay 0.7385 book 0.7400 PASS Table 13.3, Boosting, held-out: promised value (workbook) 1.1629 book 1.1600 PASS Table 13.3, Boosting, held-out: fresh breach, % (workbook) 4.9600 book 4.9600 PASS Table 13.3, Boosting, held-out: fresh value (workbook) 0.5908 book 0.5900 PASS Table 13.2: accuracy at 0.5, % 87.3240 book 87.3000 PASS Table 13.2: accuracy where cheapest, % 65.8620 book 65.9000 PASS Table 13.2: AUC 0.9240 book 0.9240 PASS Table 13.2: decision value at most 1.4369 book 1.4400 PASS Table 13.2: fresh breach, certified, % 4.7260 book 4.7300 PASS abstract: a perfect warning 4.1180 book 4.1200 PASS abstract: the five readings allow at most 1.4369 book 1.4400 PASS abstract: the certified warning is worth 1.0593 book 1.0600 PASS the board's take-away: the ceiling is about a third of a perfect warning 1.0000 book 1.0000 PASS the board's take-away: trusted boosting promises nearly 4.0377 book 4.0000 PASS the board's take-away and Key Takeaways: certified, about 1.0593 book 1.0000 PASS LOS 13.2: boosting on the record 1.0000 book 1.0000 PASS LOS 13.2: boosting fresh 0.8883 book 0.8900 PASS LOS 13.3: at accuracy's threshold 0.5273 book 0.5300 PASS LOS 13.3: at a 4.1% threshold 1.4365 book 1.4400 PASS LOS 13.4: held-out designs keep 5%, % of records 47.5000 book 47.5000 PASS LOS 13.4: certified designs keep 5%, % of records 98.0000 book 98.0000 PASS LOS 13.4: at an average value of 0.9502 book 0.9500 PASS LOS 13.5: the first component's value 0.1409 book 0.1400 PASS LOS 13.5: the funding spread alone keeps, % 30.1408 book 30.0000 PASS 376 of 376 checkpoints PASS
Next. This chapter let data supply one declared part of Meridian's model, a learned warning of funding-window closures, judged it on data it had not seen, and left the board's commitments to the optimizer. Chapter 14, Artificial Intelligence for Enterprise Optimization, continues Part IV. AXIOM-13 runs the same instance interactively (Lab 2.13); this notebook and the workbook are its reproducible record.


