% The Fidelity Gate: a rank-correlation test that guards drill-down /
% multi-fidelity search against silent collapse. Controlled, seeded study on a
% synthetic space with KNOWN ground truth. Every number in this manuscript comes
% from results/results.json (seed 0), produced by scripts/run_all.py.
% Primary arXiv category: cs.LG (Machine Learning).
% Cross-list: q-fin.CP (Computational Finance), stat.ML.
\documentclass[11pt]{article}
\usepackage[margin=1in]{geometry}
\usepackage{amsmath,amssymb}
\usepackage{booktabs}
\usepackage[numbers,round]{natbib}
\usepackage{microtype}
\usepackage[hidelinks]{hyperref}
\setlength{\emergencystretch}{2em}

\newcommand{\rhogate}{\rho_{\mathrm{gate}}}
\newcommand{\code}[1]{\texttt{#1}}

\title{The Fidelity Gate:\\
A Rank-Correlation Test That Keeps Drill-Down Search\\
From Collapsing Silently}
\author{Eugen Soloviov}
\date{July 2026}

\begin{document}
\maketitle

\begin{abstract}
Multi-fidelity, or drill-down, search --- successive halving, Hyperband, ASHA
--- screens many configurations cheaply at low fidelity and promotes only the
survivors to the expensive full evaluation. It is a genuine speedup, and it
collapses \emph{silently} when the cheap proxy does not rank configurations the
way the full evaluation does: winners are discarded at the first rung and noise
is promoted, with no error thrown. We isolate this failure on a fully synthetic
configuration space where the true full-fidelity score of every config is known
by construction, so rank correlation, screening accuracy, and search regret are
\emph{measured} against ground truth rather than estimated. Across three proxy
regimes (per-fold noise $\sigma_0\in\{1,3,6\}$), the Spearman rank correlation
between the cheapest rung and the truth is $\rho\!\approx\!0.687$, $0.299$, and
$0.153$, respectively, and rises monotonically to $1.0$ at full fidelity. In the
degraded regime the cheapest rung's top-$1$ screening hit rate is $0.0025$ ---
exactly the $1/400$ coin flip. Successive halving pruned from rung $1$ then has
mean regret $0.0025$, $0.203$, and $0.409$ against an oracle's $0$ and a
random-promotion baseline's $3.023$: the degraded search recovers only
$13.5\%$ of the way from random toward optimal. Our contribution is a
\emph{fidelity gate}: measure $\rho(\text{low},\text{full})$ first and forbid
pruning below the shallowest rung where $\rho\ge0.5$, raising the minimum
fidelity (and disabling pruning entirely if no rung qualifies). The gate leaves
the good regime untouched (prunes from rung $1$, regret unchanged at $0.0025$),
raises the moderate regime to rung $4$ (regret $0.203\!\to\!0.022$, a $89.2\%$
reduction), and disables pruning in the degraded regime (regret
$0.409\!\to\!0.0$, a $100\%$ reduction). The price is reported honestly: the
minimum-fidelity cost multiplies by the gate rung ($1\times$, $4\times$,
$8\times$). Because the space is synthetic with known truth, we claim
calibration of the gate rule, not a result on any specific real tuning task.
This study accompanies a \href{https://marketmaker.cc}{marketmaker.cc} blog post.
\end{abstract}

\section{Introduction}
\label{sec:intro}

Every large parameter search hits the same wall: the space is enormous and each
honest evaluation is expensive. A backtest scored over a multi-fold walk-forward
split can cost seconds per configuration, so ten thousand configurations is
hours. The standard escape is \emph{multi-fidelity}, or drill-down, search:
evaluate cheaply, promote only the survivors to the expensive test. Screen ten
thousand configs on one fold, keep the best few hundred, run those on all folds.

This family --- successive halving \citep{jamieson2016}, Hyperband
\citep{li2018hyperband}, and its asynchronous form ASHA \citep{li2020asha} ---
has one shape. Define a \emph{fidelity}: a cheap-to-expensive resource dial
(number of folds, length of history, candle resolution, epochs) that at its
maximum returns the true objective. Run all configs at low fidelity, keep the
top $1/\eta$, multiply the resource by $\eta$, and repeat until a few finalists
are evaluated at full fidelity. The elimination rate $\eta$ is the shared knob;
its canonical default is $\eta=3$. Drill-down is one of the genuinely good ideas
in optimization: it can turn a day of compute into an hour.

It also rests on a single assumption that almost nobody measures. Successive
halving does not need the cheap fidelity to give the right objective
\emph{value} --- nobody cares that one fold reports a different Sharpe than six.
It needs something weaker but far more specific: the cheap fidelity must
\emph{rank} configurations the same way the expensive one does. If config $A$
beats $B$ at full fidelity, $A$ should tend to beat $B$ at low fidelity too. The
quantity that governs whether drill-down works is therefore the \emph{rank
correlation} between the low-fidelity and full-fidelity objective across the
space. If it is high, the cheap round's survivors are the same configs the
expensive round would keep, and compute is saved for free. If it is low, the
cheap round is a random filter that discards good configs and promotes bad ones,
and the search spends its budget accelerating toward a worse answer. No setting
of $\eta$ rescues a bad proxy: pruned gently it still leaks winners, pruned
aggressively it hemorrhages them.

The failure is invisible in the usual way. The search completes, reports a
champion, and the champion looks fine in-sample; nothing throws an error. The
proxy's dishonesty surfaces only when the champion falls apart out-of-sample,
and by then the blame lands on the strategy rather than on the search that
selected it. The discipline, then, is to \emph{measure the thing the method
assumes, before trusting the method}. For drill-down, that thing is the rank
correlation, and the measurement is cheap.

\paragraph{Contribution.} We make the measurement precise and turn it into a
gate. On a fully synthetic configuration space where each config's true
full-fidelity score is \emph{generated} rather than estimated, we quantify, from
one seeded run: (i) how the Spearman rank correlation between a proxy rung and
the truth climbs with fidelity, across three proxy-quality regimes; (ii) how
poorly the cheapest rung screens for the true best, relative to a random-guess
baseline; (iii) how successive-halving regret slides from near-optimal to
near-random as the proxy degrades; and (iv) a \emph{fidelity gate} that measures
$\rho$ per rung and forbids pruning below the shallowest rung where
$\rho\ge0.5$, recovering low regret where the naive rung-$1$ search does not, and
reporting the honest cost of doing so.

\paragraph{Scope and honesty constraints.} The space is fully synthetic. The
true full-fidelity ranking is known by construction, so rank correlation,
top-$k$ hit rate, and regret are measured against ground truth, not estimated
from a noisy backtest. We therefore claim the \emph{calibration} of the gate ---
where the cliff sits and what the $\rho\ge0.5$ rule buys --- not a result on any
particular real tuning task, and we measure \emph{no} wall-clock time and make
\emph{no} speedup claim beyond the min-fidelity cost multiplier the gate reports.
All numbers derive from one seeded run of one public harness
(\code{scripts/run\_all.py}), with the methods in \code{scripts/fidelity.py}.

\section{The proxy model and the promotion rule}
\label{sec:model}

\paragraph{A synthetic space with known truth.} Each of $M=400$ configurations
$i$ has a true full-fidelity score drawn once,
\begin{equation}
\label{eq:truth}
s_i \sim \mathcal N(0,1), \qquad i = 1,\dots,M,
\end{equation}
higher is better (think out-of-sample Sharpe). A single cheap ``fold'' estimate
of config $i$ is the truth seen through per-fold noise of scale $\sigma_0$, and
the $r$-fold proxy at fidelity rung $r$ is the cumulative mean of $r$
independent folds:
\begin{equation}
\label{eq:proxy}
\phi_r(i) = \frac1r \sum_{q=1}^{r}\big(s_i + \sigma_0\,\varepsilon_{q,i}\big),
\qquad \varepsilon_{q,i}\sim\mathcal N(0,1)\ \text{i.i.d.}
\end{equation}
Averaging $r$ independent folds shrinks the noise from $\sigma_0$ to
$\sigma_0/\sqrt r$, so as fidelity rises the proxy's \emph{ranking} converges to
the truth's. There are $K=8$ rungs; the top rung is forced to equal the truth
($\phi_K \equiv s$), so full-fidelity rank correlation is exactly $1$. The dial
$\sigma_0=\code{base\_noise}$ moves the regime: we study a \emph{good} proxy
($\sigma_0=1$), a \emph{moderate} one ($\sigma_0=3$), and a \emph{degraded} one
($\sigma_0=6$). Every array is seeded from \code{numpy.default\_rng}, and each
reported statistic is a mean over $400$ seeded model draws.

\paragraph{Successive halving / ASHA and regret.} Successive halving
\citep{jamieson2016} with elimination rate $\eta$, starting at a minimum
resource rung, scores all alive configs at the current rung by $\phi_r$, keeps
the top $1/\eta$, multiplies the rung by $\eta$, and repeats until one config or
the full rung remains; ASHA \citep{li2020asha} is the asynchronous form of the
same rule. We use $\eta=3$. The promoted survivor's \emph{regret} is the true
gap to the true best,
\begin{equation}
\label{eq:regret}
\mathrm{regret} = \max_i s_i - s_{\mathrm{promoted}} \ \ge 0,
\end{equation}
zero exactly when the true best is found. We compare four policies: \emph{naive}
(minimum resource $=1$, prune from the cheapest, noisiest rung --- the library
default); \emph{gated} (minimum resource raised by the fidelity gate of
Section~\ref{sec:gate}); \emph{random} (promote a uniform-random config, the
null a collapsed search decays toward); and the \emph{oracle} (promote the true
best, regret $0$).

\section{Rank correlation and screening quality by regime}
\label{sec:rho}

The core diagnostic is the Spearman rank correlation
$\rho_r = \rho(\phi_r, s)$ between rung $r$ and the truth, averaged over the
seeded draws. Table~\ref{tab:rho} gives it per regime and rung. Two facts drive
everything. First, $\rho$ climbs \emph{monotonically} with fidelity in every
regime, from a low cheap-rung value to $1.0$ at full fidelity --- the
$\sigma_0/\sqrt r$ variance averaging of Eq.~\eqref{eq:proxy} made visible.
Second, the cheapest rung is where the regimes separate: $\rho_1 = 0.687$ for
the good proxy (already trustworthy), $0.299$ for the moderate one, and $0.153$
for the degraded one --- barely above the zero of a random ranking. A drill-down
that prunes at rung $1$ is trusting exactly these numbers.

\begin{table}[t]
\centering
\caption{Spearman rank correlation $\rho(\phi_r, s)$ between each fidelity rung
$r$ and the true full-fidelity ranking, per proxy regime, averaged over $400$
seeded model draws ($M=400$ configs, $K=8$ rungs, seed 0). $\rho$ rises
monotonically to $1.0$ at full fidelity (the top rung equals the truth by
construction). The cheapest rung is trustworthy for the good proxy, marginal for
the moderate one, and near-random for the degraded one.}
\label{tab:rho}
\begin{tabular}{rrrr}
\toprule
Rung $r$ & good ($\sigma_0{=}1$) & moderate ($\sigma_0{=}3$) & degraded ($\sigma_0{=}6$) \\
\midrule
$1$ & $0.687$ & $0.299$ & $0.153$ \\
$2$ & $0.802$ & $0.407$ & $0.216$ \\
$3$ & $0.853$ & $0.478$ & $0.260$ \\
$4$ & $0.883$ & $0.531$ & $0.297$ \\
$5$ & $0.903$ & $0.574$ & $0.329$ \\
$6$ & $0.917$ & $0.610$ & $0.358$ \\
$7$ & $0.928$ & $0.640$ & $0.383$ \\
$8$ & $1.0$   & $1.0$   & $1.0$ \\
\bottomrule
\end{tabular}
\end{table}

A rank correlation is abstract; its consequence for screening is not.
Table~\ref{tab:topk} reports the top-$k$ \emph{hit rate} at the cheapest rung ---
the probability that the true best config is among the $k$ highest-scoring
configs at rung $1$ --- against the uniform-guess baseline $k/M$. For the good
proxy the cheap rung already puts the true best in its top-$1$ $18\%$ of the
time and in its top-$20$ $74.5\%$ of the time, orders of magnitude above the
baseline. For the degraded proxy the top-$1$ hit rate is $0.0025$, which is
\emph{exactly} the random baseline $1/400 = 0.0025$: at the cheapest rung the
degraded proxy finds the true best no more often than a coin flip, and its
entire top-$20$ contains the true best only $14.5\%$ of the time. This is what
``the cheap round is a random filter'' means in numbers.

\begin{table}[t]
\centering
\caption{Top-$k$ hit rate at the cheapest rung ($r=1$): probability the true
best config is among the $k$ highest-scoring configs by the low-fidelity proxy,
averaged over $400$ seeded draws, versus the uniform-guess baseline $k/M$
($M=400$). The good proxy screens far above chance; the degraded proxy's top-$1$
hit rate ($0.0025$) equals the $1/400$ baseline exactly.}
\label{tab:topk}
\begin{tabular}{lrrrr}
\toprule
Regime & top-$1$ & top-$5$ & top-$10$ & top-$20$ \\
\midrule
good ($\sigma_0{=}1$)     & $0.18$   & $0.4175$ & $0.5875$ & $0.745$ \\
moderate ($\sigma_0{=}3$) & $0.0175$ & $0.0975$ & $0.1675$ & $0.255$ \\
degraded ($\sigma_0{=}6$) & $0.0025$ & $0.0375$ & $0.0825$ & $0.145$ \\
\midrule
random baseline $k/M$     & $0.0025$ & $0.0125$ & $0.025$  & $0.05$ \\
\bottomrule
\end{tabular}
\end{table}

\section{The silent collapse: regret versus proxy quality}
\label{sec:collapse}

We now run the search itself. Table~\ref{tab:regret} reports successive-halving
regret (Eq.~\eqref{eq:regret}) under naive rung-$1$ pruning, alongside the
random-promotion and oracle baselines, per regime. The random baseline is the
same across regimes --- promoting uniformly at random has mean regret $3.023$,
independent of proxy quality --- and the oracle is $0$ by definition. The
naive drill-down slides between them as the proxy degrades: mean regret $0.0025$
for the good proxy, $0.203$ for the moderate one, and $0.409$ for the degraded
one. Read as a fraction of the random baseline, the naive collapse ratio climbs
from $0.001$ to $0.067$ to $0.135$: the degraded search has slid $13.5\%$ of the
way from optimal toward random, and its rate of actually finding the true best
falls from $98.8\%$ (good) to $50.8\%$ (moderate) to $27.0\%$ (degraded). The
collapse is silent because every one of these runs completes and returns a
champion; only the known ground truth exposes that the champion is increasingly
a random draw.

\begin{table}[t]
\centering
\caption{Successive-halving regret (true-best minus promoted true score, lower
is better) under naive rung-$1$ pruning, with the random-promotion and oracle
baselines, per regime ($\eta=3$, $400$ seeded draws, seed 0). The collapse ratio
is naive regret as a fraction of the random baseline; ``found best'' is the rate
of promoting the true best. As the proxy degrades the naive search slides from
near-optimal toward the random null.}
\label{tab:regret}
\begin{tabular}{lrrrrr}
\toprule
Regime & naive regret & random & oracle & collapse ratio & found best \\
\midrule
good ($\sigma_0{=}1$)     & $0.0025$ & $3.023$ & $0.0$ & $0.001$ & $0.9875$ \\
moderate ($\sigma_0{=}3$) & $0.203$  & $3.023$ & $0.0$ & $0.067$ & $0.5075$ \\
degraded ($\sigma_0{=}6$) & $0.409$  & $3.023$ & $0.0$ & $0.135$ & $0.27$ \\
\bottomrule
\end{tabular}
\end{table}

\section{The fidelity gate}
\label{sec:gate}

The fix is to measure before trusting. The \emph{fidelity gate} is a small
experiment run once, before the search proper, on the same space: draw a few
hundred configs, evaluate each at every rung (nearly free for a fold-count
fidelity, since the cheap rungs' folds are reused in the expensive one), and
compute $\rho_r=\rho(\phi_r,s)$ for every rung. The gate then returns the
shallowest trustworthy rung,
\begin{equation}
\label{eq:gate}
g(\rho) = \min\{\, r : \rho_r \ge \rhogate \,\}, \qquad \rhogate = 0.5,
\end{equation}
falling back to the full rung $K$ (pruning disabled) if no rung clears the
threshold. The search's minimum resource is set to $g(\rho)$: it may prune only
at rungs it has proven rank like the full evaluation. A drill-down that cannot
prove its proxy simply refuses to prune and degrades gracefully to a
full-fidelity scan.

Table~\ref{tab:gate} shows what the rule buys. In the \emph{good} regime the
cheapest rung already clears the threshold ($\rho_1=0.687\ge0.5$), so the gate
prunes from rung $1$ and changes nothing: regret stays at $0.0025$, and the
minimum-fidelity cost multiplier is $1\times$ --- the gate is free when the proxy
is honest. In the \emph{moderate} regime the cheap rungs are untrustworthy; the
first rung to clear $0.5$ is rung $4$ (see Table~\ref{tab:rho}, where
$\rho_4=0.531$), so the gate raises the minimum resource to $4$ and regret falls
from $0.203$ to $0.022$, an $89.2\%$ reduction, while the rate of finding the
true best rises from $50.8\%$ to $90.8\%$. In the \emph{degraded} regime
\emph{no} rung below full fidelity clears $0.5$ (the highest sub-full rung is
$\rho_7=0.383$), so the gate disables pruning entirely: regret falls from
$0.409$ to $0.0$ ($100\%$ reduction), and the true best is found every time.
The gate's decision is not free in compute --- it multiplies the up-front
minimum fidelity by the gate rung ($1\times$, $4\times$, $8\times$) --- but it
reports that bill explicitly rather than hiding it, and it spends it only where
the proxy has failed to earn trust.

\begin{table}[t]
\centering
\caption{The fidelity gate ($\rhogate=0.5$, $\eta=3$, $400$ seeded draws, seed
0). The gate rung is the shallowest rung with $\rho\ge0.5$, else full fidelity
(pruning off). It leaves the good regime at rung $1$ (free), raises the moderate
regime to rung $4$, and disables pruning in the degraded regime, recovering low
regret in every case. The cost is the min-fidelity multiplier, reported
honestly.}
\label{tab:gate}
\begin{tabular}{lrrrrrr}
\toprule
Regime & $\rho_1$ & gate rung & naive reg. & gated reg. & reduction & min-fid.\ cost \\
\midrule
good ($\sigma_0{=}1$)     & $0.689$ & $1$ (prune)    & $0.0025$ & $0.0025$ & $0\%$    & $1\times$ \\
moderate ($\sigma_0{=}3$) & $0.303$ & $4$            & $0.203$  & $0.022$  & $89.2\%$ & $4\times$ \\
degraded ($\sigma_0{=}6$) & $0.158$ & $8$ (prune off)& $0.409$  & $0.0$    & $100\%$  & $8\times$ \\
\bottomrule
\end{tabular}
\end{table}

\section{Discussion}
\label{sec:discussion}

\paragraph{Measure the assumption, not the output.} A completed search that
returns a plausible champion is not evidence the search worked; on a degraded
proxy that is exactly what a collapsed search also produces. The one number that
separates a real drill-down from a delusional one is the rank correlation
between the cheap and the expensive evaluation, and it is cheap to measure: a few
hundred configs scored at every rung, one Spearman coefficient per rung. Compute
it before the search, not after the champion disappoints out-of-sample.

\paragraph{The gate is free when the proxy is honest and cheap when it is not.}
Where the cheap rung already ranks well (good regime), the gate prunes from rung
$1$ and costs nothing; it activates only when $\rho$ is below threshold, and its
cost --- a higher minimum fidelity --- is exactly proportional to how untrustworthy
the proxy is ($4\times$ at moderate, $8\times$ at degraded). This is the correct
shape for a safety rule: invisible when unneeded, and self-scaling to the danger.
Crucially, the gate never silently throws away winners; in the worst case it
declines to prune and returns the answer a brute-force scan would have.

\paragraph{The $\rho\ge0.5$ threshold.} We use $0.5$ as a deliberately simple,
defensible default: below it a rung explains less than half the rank variance and
the top-$k$ screening (Table~\ref{tab:topk}) is already near chance, while by
$\rho\approx0.53$ (moderate rung $4$) the gated regret is within an order of
magnitude of the good regime's. The threshold is a knob, not a law; a
risk-averse search would raise it, trading more up-front fidelity for tighter
regret. What matters is that the decision is \emph{made from a measurement},
not from the library's rung-$1$ default.

\section{Limitations}
\label{sec:limitations}

\begin{itemize}
\item \textbf{Synthetic proxy model.} The configuration space, the true scores,
  and the fidelity noise are generated, not observed. This is deliberate --- it
  is the only way to measure rank correlation and regret against a \emph{known}
  truth rather than a noisy estimate --- but it means the specific $\rho$, hit-rate,
  and regret numbers are properties of the Gaussian truth and the
  $\sigma_0/\sqrt r$ noise model, not of any real backtest. A real proxy may have
  heavier tails, correlated fold noise, or a systematic \emph{bias} that does not
  average away with more folds; bias is the more dangerous axis and is not
  exercised here (all regimes are noise-only, $\mathrm{bias}=0$).
\item \textbf{One noise family and one truth distribution.} The truth is
  standard normal and the fidelity noise is additive Gaussian averaged over
  independent folds. The \emph{mechanism} --- rank correlation governs whether
  drill-down works, and gating on it prevents silent collapse --- is invariant to
  these choices, but the cliff location (which rung clears $\rho\ge0.5$) is
  data-dependent.
\item \textbf{No wall-clock or speedup claim.} We quantify the gate's cost only
  as the min-fidelity multiplier ($1\times$/$4\times$/$8\times$), an
  evaluation-count proxy. Actual speedup depends on how rung cost scales with
  fidelity in a given problem, which we do not model.
\item \textbf{One promotion rule.} We study synchronous successive halving with
  $\eta=3$ as the reference realization of ASHA on a fixed model. Hyperband's
  outer hedging loop and asynchronous ASHA scheduling would change the constants
  but not the dependence on rank correlation.
\end{itemize}

\section{Conclusion}
\label{sec:conclusion}

Multi-fidelity drill-down search promotes cheap low-fidelity survivors to the
expensive full evaluation, and it is a genuine speedup exactly when the cheap
proxy ranks configurations the way the full evaluation does. When it does not,
the search collapses silently: on a synthetic space with known truth, the
cheapest rung's rank correlation with the truth falls from $0.687$ (good proxy)
to $0.153$ (degraded), the degraded rung's top-$1$ screening hit rate drops to
the $1/400$ coin flip, and successive halving pruned from rung $1$ has mean
regret rising from $0.0025$ to $0.409$ against an oracle's $0$ --- $13.5\%$ of the
way to the random baseline's $3.023$, with every run completing and returning a
plausible champion. The fix is one measurement made into a gate: compute the
Spearman rank correlation between each proxy rung and the full evaluation, and
forbid pruning below the shallowest rung where $\rho\ge0.5$. The gate leaves the
good regime free (rung $1$, regret unchanged), raises the moderate regime to rung
$4$ ($89.2\%$ less regret), and disables pruning in the degraded regime ($100\%$
less regret, the true best found every time), paying an honest min-fidelity cost
of $1\times$, $4\times$, and $8\times$. A drill-down that cannot prove its proxy
should not be allowed to prune on it; the gate is the proof.

\paragraph{Reproducibility.} All numbers derive from one seeded run.
\code{scripts/run\_all.py} regenerates \code{results/results.json} from seed $0$
(Python 3.12.3, NumPy 2.5.1) deterministically, aggregating over $400$ seeded
model draws per regime; the proxy model, the pure-numpy Spearman correlation, the
successive-halving promotion, and the gate are in \code{scripts/fidelity.py}. The
synthetic space uses \code{numpy.random.default\_rng} with standard-normal true
scores and additive per-fold Gaussian noise of scale $\sigma_0\in\{1,3,6\}$ over
$K=8$ rungs and $M=400$ configs. \code{tests/} contains deterministic invariant
tests for the Spearman implementation, the proxy construction, the promotion
rule, and the gate.

\bibliographystyle{plainnat}
\bibliography{refs}

\end{document}
