Introduction

Every large parameter search hits the same wall: the space is enormous and each honest evaluation is expensive. A backtest scored over a multi-fold walk-forward split can cost seconds per configuration, so ten thousand configurations is hours. The standard escape is multi-fidelity, or drill-down, search: evaluate cheaply, promote only the survivors to the expensive test. Screen ten thousand configs on one fold, keep the best few hundred, run those on all folds.

This family — successive halving [1], Hyperband [2], and its asynchronous form ASHA [3] — has one shape. Define a fidelity: a cheap-to-expensive resource dial (number of folds, length of history, candle resolution, epochs) that at its maximum returns the true objective. Run all configs at low fidelity, keep the top 1/\eta, multiply the resource by \eta, and repeat until a few finalists are evaluated at full fidelity. The elimination rate \eta is the shared knob; its canonical default is \eta=3. Drill-down is one of the genuinely good ideas in optimization: it can turn a day of compute into an hour.

It also rests on a single assumption that almost nobody measures. Successive halving does not need the cheap fidelity to give the right objective value — nobody cares that one fold reports a different Sharpe than six. It needs something weaker but far more specific: the cheap fidelity must rank configurations the same way the expensive one does. If config A beats B at full fidelity, A should tend to beat B at low fidelity too. The quantity that governs whether drill-down works is therefore the rank correlation between the low-fidelity and full-fidelity objective across the space. If it is high, the cheap round's survivors are the same configs the expensive round would keep, and compute is saved for free. If it is low, the cheap round is a random filter that discards good configs and promotes bad ones, and the search spends its budget accelerating toward a worse answer. No setting of \eta rescues a bad proxy: pruned gently it still leaks winners, pruned aggressively it hemorrhages them.

The failure is invisible in the usual way. The search completes, reports a champion, and the champion looks fine in-sample; nothing throws an error. The proxy's dishonesty surfaces only when the champion falls apart out-of-sample, and by then the blame lands on the strategy rather than on the search that selected it. The discipline, then, is to measure the thing the method assumes, before trusting the method. For drill-down, that thing is the rank correlation, and the measurement is cheap.

Contribution.

We make the measurement precise and turn it into a gate. On a fully synthetic configuration space where each config's true full-fidelity score is generated rather than estimated, we quantify, from one seeded run: (i) how the Spearman rank correlation between a proxy rung and the truth climbs with fidelity, across three proxy-quality regimes; (ii) how poorly the cheapest rung screens for the true best, relative to a random-guess baseline; (iii) how successive-halving regret slides from near-optimal to near-random as the proxy degrades; and (iv) a fidelity gate that measures \rho per rung and forbids pruning below the shallowest rung where \rho\ge0.5, recovering low regret where the naive rung-1 search does not, and reporting the honest cost of doing so.

Scope and honesty constraints.

The space is fully synthetic. The true full-fidelity ranking is known by construction, so rank correlation, top-k hit rate, and regret are measured against ground truth, not estimated from a noisy backtest. We therefore claim the calibration of the gate — where the cliff sits and what the \rho\ge0.5 rule buys — not a result on any particular real tuning task, and we measure no wall-clock time and make no speedup claim beyond the min-fidelity cost multiplier the gate reports. All numbers derive from one seeded run of one public harness (scripts/run_all.py), with the methods in scripts/fidelity.py.

The proxy model and the promotion rule

A synthetic space with known truth.

Each of M=400 configurations i has a true full-fidelity score drawn once, \begin{equation} \label{eq:truth} s_i \sim \mathcal N(0,1), \qquad i = 1,\dots,M, \end{equation} higher is better (think out-of-sample Sharpe). A single cheap “fold” estimate of config i is the truth seen through per-fold noise of scale \sigma_0, and the r-fold proxy at fidelity rung r is the cumulative mean of r independent folds: \begin{equation} \label{eq:proxy} \phi_r(i) = \frac1r \sum_{q=1}^{r}\big(s_i + \sigma_0\,\varepsilon_{q,i}\big), \qquad \varepsilon_{q,i}\sim\mathcal N(0,1)\ \text{i.i.d.} \end{equation} Averaging r independent folds shrinks the noise from \sigma_0 to \sigma_0/\sqrt r, so as fidelity rises the proxy's ranking converges to the truth's. There are K=8 rungs; the top rung is forced to equal the truth (\phi_K \equiv s), so full-fidelity rank correlation is exactly 1. The dial \sigma_0=\texttt{base\_noise} moves the regime: we study a good proxy (\sigma_0=1), a moderate one (\sigma_0=3), and a degraded one (\sigma_0=6). Every array is seeded from numpy.default_rng, and each reported statistic is a mean over 400 seeded model draws.

Successive halving / ASHA and regret.

Successive halving [1] with elimination rate \eta, starting at a minimum resource rung, scores all alive configs at the current rung by \phi_r, keeps the top 1/\eta, multiplies the rung by \eta, and repeats until one config or the full rung remains; ASHA [3] is the asynchronous form of the same rule. We use \eta=3. The promoted survivor's regret is the true gap to the true best, \begin{equation} \label{eq:regret} \mathrm{regret} = \max_i s_i - s_{\mathrm{promoted}} \ \ge 0, \end{equation} zero exactly when the true best is found. We compare four policies: naive (minimum resource =1, prune from the cheapest, noisiest rung — the library default); gated (minimum resource raised by the fidelity gate of Section 5); random (promote a uniform-random config, the null a collapsed search decays toward); and the oracle (promote the true best, regret 0).

Rank correlation and screening quality by regime

The core diagnostic is the Spearman rank correlation \rho_r = \rho(\phi_r, s) between rung r and the truth, averaged over the seeded draws. Table 1 gives it per regime and rung. Two facts drive everything. First, \rho climbs monotonically with fidelity in every regime, from a low cheap-rung value to 1.0 at full fidelity — the \sigma_0/\sqrt r variance averaging of Eq. (2) made visible. Second, the cheapest rung is where the regimes separate: \rho_1 = 0.687 for the good proxy (already trustworthy), 0.299 for the moderate one, and 0.153 for the degraded one — barely above the zero of a random ranking. A drill-down that prunes at rung 1 is trusting exactly these numbers.

A rank correlation is abstract; its consequence for screening is not. Table 2 reports the top-k hit rate at the cheapest rung — the probability that the true best config is among the k highest-scoring configs at rung 1 — against the uniform-guess baseline k/M. For the good proxy the cheap rung already puts the true best in its top-1 18\% of the time and in its top-20 74.5\% of the time, orders of magnitude above the baseline. For the degraded proxy the top-1 hit rate is 0.0025, which is exactly the random baseline 1/400 = 0.0025: at the cheapest rung the degraded proxy finds the true best no more often than a coin flip, and its entire top-20 contains the true best only 14.5\% of the time. This is what “the cheap round is a random filter” means in numbers.

The silent collapse: regret versus proxy quality

We now run the search itself. Table 3 reports successive-halving regret (Eq. (3)) under naive rung-1 pruning, alongside the random-promotion and oracle baselines, per regime. The random baseline is the same across regimes — promoting uniformly at random has mean regret 3.023, independent of proxy quality — and the oracle is 0 by definition. The naive drill-down slides between them as the proxy degrades: mean regret 0.0025 for the good proxy, 0.203 for the moderate one, and 0.409 for the degraded one. Read as a fraction of the random baseline, the naive collapse ratio climbs from 0.001 to 0.067 to 0.135: the degraded search has slid 13.5\% of the way from optimal toward random, and its rate of actually finding the true best falls from 98.8\% (good) to 50.8\% (moderate) to 27.0\% (degraded). The collapse is silent because every one of these runs completes and returns a champion; only the known ground truth exposes that the champion is increasingly a random draw.

The fidelity gate

The fix is to measure before trusting. The fidelity gate is a small experiment run once, before the search proper, on the same space: draw a few hundred configs, evaluate each at every rung (nearly free for a fold-count fidelity, since the cheap rungs' folds are reused in the expensive one), and compute \rho_r=\rho(\phi_r,s) for every rung. The gate then returns the shallowest trustworthy rung, \begin{equation} \label{eq:gate} g(\rho) = \min\{\, r : \rho_r \ge \rho_{\mathrm{gate}} \,\}, \qquad \rho_{\mathrm{gate}} = 0.5, \end{equation} falling back to the full rung K (pruning disabled) if no rung clears the threshold. The search's minimum resource is set to g(\rho): it may prune only at rungs it has proven rank like the full evaluation. A drill-down that cannot prove its proxy simply refuses to prune and degrades gracefully to a full-fidelity scan.

Table 4 shows what the rule buys. In the good regime the cheapest rung already clears the threshold (\rho_1=0.687\ge0.5), so the gate prunes from rung 1 and changes nothing: regret stays at 0.0025, and the minimum-fidelity cost multiplier is 1\times — the gate is free when the proxy is honest. In the moderate regime the cheap rungs are untrustworthy; the first rung to clear 0.5 is rung 4 (see Table 1, where \rho_4=0.531), so the gate raises the minimum resource to 4 and regret falls from 0.203 to 0.022, an 89.2\% reduction, while the rate of finding the true best rises from 50.8\% to 90.8\%. In the degraded regime no rung below full fidelity clears 0.5 (the highest sub-full rung is \rho_7=0.383), so the gate disables pruning entirely: regret falls from 0.409 to 0.0 (100\% reduction), and the true best is found every time. The gate's decision is not free in compute — it multiplies the up-front minimum fidelity by the gate rung (1\times, 4\times, 8\times) — but it reports that bill explicitly rather than hiding it, and it spends it only where the proxy has failed to earn trust.

Discussion

Measure the assumption, not the output.

A completed search that returns a plausible champion is not evidence the search worked; on a degraded proxy that is exactly what a collapsed search also produces. The one number that separates a real drill-down from a delusional one is the rank correlation between the cheap and the expensive evaluation, and it is cheap to measure: a few hundred configs scored at every rung, one Spearman coefficient per rung. Compute it before the search, not after the champion disappoints out-of-sample.

The gate is free when the proxy is honest and cheap when it is not.

Where the cheap rung already ranks well (good regime), the gate prunes from rung 1 and costs nothing; it activates only when \rho is below threshold, and its cost — a higher minimum fidelity — is exactly proportional to how untrustworthy the proxy is (4\times at moderate, 8\times at degraded). This is the correct shape for a safety rule: invisible when unneeded, and self-scaling to the danger. Crucially, the gate never silently throws away winners; in the worst case it declines to prune and returns the answer a brute-force scan would have.

The \rho\ge0.5 threshold.

We use 0.5 as a deliberately simple, defensible default: below it a rung explains less than half the rank variance and the top-k screening (Table 2) is already near chance, while by \rho\approx0.53 (moderate rung 4) the gated regret is within an order of magnitude of the good regime's. The threshold is a knob, not a law; a risk-averse search would raise it, trading more up-front fidelity for tighter regret. What matters is that the decision is made from a measurement, not from the library's rung-1 default.

Limitations

• Synthetic proxy model. The configuration space, the true scores, and the fidelity noise are generated, not observed. This is deliberate — it is the only way to measure rank correlation and regret against a known truth rather than a noisy estimate — but it means the specific \rho, hit-rate, and regret numbers are properties of the Gaussian truth and the \sigma_0/\sqrt r noise model, not of any real backtest. A real proxy may have heavier tails, correlated fold noise, or a systematic bias that does not average away with more folds; bias is the more dangerous axis and is not exercised here (all regimes are noise-only, \mathrm{bias}=0).

• One noise family and one truth distribution. The truth is standard normal and the fidelity noise is additive Gaussian averaged over independent folds. The mechanism — rank correlation governs whether drill-down works, and gating on it prevents silent collapse — is invariant to these choices, but the cliff location (which rung clears \rho\ge0.5) is data-dependent.

• No wall-clock or speedup claim. We quantify the gate's cost only as the min-fidelity multiplier (1\times/4\times/8\times), an evaluation-count proxy. Actual speedup depends on how rung cost scales with fidelity in a given problem, which we do not model.

• One promotion rule. We study synchronous successive halving with \eta=3 as the reference realization of ASHA on a fixed model. Hyperband's outer hedging loop and asynchronous ASHA scheduling would change the constants but not the dependence on rank correlation.

Conclusion

Multi-fidelity drill-down search promotes cheap low-fidelity survivors to the expensive full evaluation, and it is a genuine speedup exactly when the cheap proxy ranks configurations the way the full evaluation does. When it does not, the search collapses silently: on a synthetic space with known truth, the cheapest rung's rank correlation with the truth falls from 0.687 (good proxy) to 0.153 (degraded), the degraded rung's top-1 screening hit rate drops to the 1/400 coin flip, and successive halving pruned from rung 1 has mean regret rising from 0.0025 to 0.409 against an oracle's 0 — 13.5\% of the way to the random baseline's 3.023, with every run completing and returning a plausible champion. The fix is one measurement made into a gate: compute the Spearman rank correlation between each proxy rung and the full evaluation, and forbid pruning below the shallowest rung where \rho\ge0.5. The gate leaves the good regime free (rung 1, regret unchanged), raises the moderate regime to rung 4 (89.2\% less regret), and disables pruning in the degraded regime (100\% less regret, the true best found every time), paying an honest min-fidelity cost of 1\times, 4\times, and 8\times. A drill-down that cannot prove its proxy should not be allowed to prune on it; the gate is the proof.

Reproducibility.

All numbers derive from one seeded run. scripts/run_all.py regenerates results/results.json from seed 0 (Python 3.12.3, NumPy 2.5.1) deterministically, aggregating over 400 seeded model draws per regime; the proxy model, the pure-numpy Spearman correlation, the successive-halving promotion, and the gate are in scripts/fidelity.py. The synthetic space uses numpy.random.default_rng with standard-normal true scores and additive per-fold Gaussian noise of scale \sigma_0\in\{1,3,6\} over K=8 rungs and M=400 configs. tests/ contains deterministic invariant tests for the Spearman implementation, the proxy construction, the promotion rule, and the gate.

References