Optimal Decision-Making Based on Prediction Sets

Summary

This paper studies the interface between conformal prediction sets and downstream expected-loss decision-making. The starting point is a two-player game in which nature chooses any distribution consistent with the coverage guarantee ($Q(S)\geq 1-\alpha$) and the decision maker chooses an action. The authors derive a clean closed form for the resulting worst-case expected loss, $L_S(a;\alpha)=\ell^{\text{in}}_S(a)+\alpha\,(\ell^{\text{out}}_S(a)-\ell^{\text{in}}_S(a))^+$, decomposing the worst-case risk into an in-set max–min term (recovering the RAC rule of Kiyani et al., 2025) and an $\alpha$-weighted penalty for catastrophic out-of-set outcomes. This decomposition is the paper's key conceptual move: it makes explicit that a purely in-set max–min policy, minimax-optimal for a quantile objective, is not minimax-optimal under expected loss when rare out-of-set errors can dominate the risk. Building on the pointwise characterization, the authors formulate a set-design problem that minimizes the induced robust risk subject to a coverage constraint, and characterize the optimum through Fenchel–Rockafellar duality on normal integrands via a randomized-kernel relaxation and derandomization — a genuinely different duality path from the LP argument of Kiyani et al., yielding a one-dimensional scalar $\beta^\star$. The practical algorithm, ROCP, plugs the oracle quantities into a split-conformal calibration and inherits distribution-free marginal coverage. Experiments on chest X-ray diagnosis (with a safety-critical 10× loss variant) and a toy static BDD100K hazard task are reported to show ROCP reduces critical mistakes relative to RAC precisely in high-stakes regimes. The main conceptual limitation is a gap between theory and deployment: the oracle assumes conditional coverage, but ROCP delivers only marginal coverage (with a $(2-\alpha)$ bound), the variant actually run is a global-$\beta$ split-conformal proxy rather than the transductive form Proposition 4.1 proves, and the robustness is taken over an ambiguity set restricted only by coverage. In execution, the baseline-loss medical results reproduced faithfully at reduced scale, but the two experiments that carry the headline high-stakes claims (the $\Lambda_1$ variant and the entire BDD task) did not complete, so those central claims remain unverified rather than refuted.

Strengths

• Conceptual contribution. The decomposition $L_S(a;\alpha)=\ell^{\text{in}}_S(a)+\alpha\,(\ell^{\text{out}}_S(a)-\ell^{\text{in}}_S(a))^+$ is a genuinely illuminating object: it makes precise why the in-set max–min rule is only correct up to an $\alpha$-weighted out-of-set penalty, and draws a sharp line between the quantile-objective and expected-loss regimes.
• Distinct duality path. The set-design theorem uses Fenchel–Rockafellar duality on normal integrands with a randomized-kernel relaxation and Dvoretzky–Wald–Wolfowitz-style derandomization. This is technically different from the LP duality of Kiyani et al. (2025), and produces the same clean scalar-parameter structure while accommodating an integral objective.
• Finite-sample coverage. Proposition 4.1 gives the transductive ROCP a distribution-free finite-sample marginal coverage guarantee that does not depend on the plug-in model being correct, cleanly separating the coverage side from the risk-targeting side.
• Careful measurability. Appendix C makes measurability assumptions and the invocation of Kuratowski–Ryll-Nardzewski / Rockafellar–Wets 14.37 explicit, and Remark C.1 handles the measurable-selection question often skipped in comparable papers.
• Interpretable, and partly reproduced, empirical differentiation. The paired-curve design (same conformal set with $a_{\text{ROCP}}$ vs. $a_{\text{RA}}$) isolates the decision rule from the set. In the one regime that ran to completion — COVID under the baseline matrix $\Lambda_0$ — every qualitative pattern the paper claims reproduced at reduced scale: overlapping ROCP/RAC worst-case-risk curves, ROCP's slightly higher realized loss, its lower critical-mistake rates, best-response's lowest loss but highest critical mistakes, and policy-invariance of the decision-agnostic sets.

Weaknesses

• Conditional-to-marginal gap. Theorems 2.2 and 3.3 assume the generally-impossible conditional coverage, while ROCP delivers only marginal coverage. Remark 2.3's $(2-\alpha)$ bound is a bound on the same policy, not on the sub-optimality of the actual ROCP output, and it degrades to a factor of $2$ as $\alpha\to0$ — the regime the paper primarily studies. The claim of "targeting the decision-theoretic optimum as the model improves" is therefore heuristic; no consistency theorem is provided, and Section 4's only real theorem is the coverage guarantee.
• Unrestricted adversary. The ambiguity set constrains nature only through coverage, so the out-of-set penalty is set by the single most catastrophic label, however implausible. There is no moderated ambiguity set (bounded density ratio, divergence ball), which risks making ROCP unnecessarily conservative and leaves unclear when the worst-case framing is appropriate.
• Headline metric coincides with the objective. The "average realized worst-case risk" panel plots the very quantity ROCP minimizes, favoring it by construction; and it averages $L_{C(x)}$ over test $x$, whereas Theorem 2.2's minimax risk is a supremum over $x$ — so the plotted certificate is a weaker object than the theorem guarantees.
• Algorithm as run ≠ algorithm as proven. Proposition 4.1 certifies the transductive per-label Algorithm 1, but the experiments use a global-$\beta$ split-conformal proxy whose coverage is claimed only empirically. Readers may wrongly assume the deployed method inherits the proof.
• Comparator gaps and confounds. No expected-loss-native baseline (conformal risk control; Cortes-Gomez et al. 2025) is included, weakening attribution of the gains to the out-of-set term. The Best-Response comparator's failure conflates best-response's conceptual limits with miscalibration of $f_x$, which in the BDD task is a per-bit isotonic map plus a conditional-independence factorization that is not jointly calibrated.
• Sensitivity and scope. The headline claims live entirely in $\Lambda_1$, whose 10× multiplier is ad hoc with no sensitivity sweep. Both experiments have tiny discrete $\mathcal Y$ (4 classes, 8 states); the continuous-$\mathcal Y$ case the theory is written for is never exercised, and the "autonomous driving" experiment is a static single-frame classification.
• Statistical support. Strong verbal claims ("consistently lower," "almost eliminates") rest on ±1 SE bands rather than confidence intervals or paired significance tests across the shared splits.
• Load-bearing assumptions. The clean main-text Theorem 3.3 (scalar equation (13)) collapses only under a singleton-argmin assumption stated in a footnote; ties can arise generically for the discrete loss matrices used. Non-atomicity of $P_X$ is assumed and excludes exactly the discrete-feature regime where the conditional-coverage oracle is realizable.
• Notation. Fig. 2's scalar $M$ clashes with the action-dependent $M(a):=\sup_y\ell(a,y)$; the critical-mistake definition's "attains arg max" leaves ties ambiguous; and the $\ell^{\text{out}}_{\mathcal Y}(a):=\ell^{\text{in}}_{\mathcal Y}(a)$ convention is unmotivated.

Reproducibility & code

• Headline claims unverified by execution. In full-mode replication, only one of the planned runs completed, so the three central benchmark claims were scored not attempted — not contradicted. The COVID $\Lambda_1$ evaluation (the source of "ROCP almost eliminates critical mistakes" and "RAC becomes brittle as $\alpha$ grows") never advanced past a START timestamp, and the BDD evaluation ("ROCP matches or outperforms all baselines … across all $\alpha$") produced only a single split-seed line and no figure. The narrative attributes this to a harness bug (long jobs backgrounded and killed at 0/20 seeds), not a defect in the paper or code; the shipped code plus cached model outputs are judged sufficient given a ~5h single-threaded runtime per run.
• Baseline-loss results reproduced. The COVID $\Lambda_0$ run completed at reduced scale (3 seeds) and confirmed all qualitative patterns the paper claims for that regime: near-overlapping ROCP/RAC worst-case risk, ROCP's slightly higher realized loss, ROCP's lower critical-mistake rates on all three labels, best-response's lowest loss but far higher critical mistakes, and byte-identical $a_{\text{ROCP}}$/$a_{\text{RA}}$ results for LAS/APS/SOCOP. The four-panel figure structure also matches. The only caveat is reduced seed count, so no cross-seed variance is available.
• Deployed calibration diverges from Algorithm 1. Appendix D states, and the shipped code confirms, that the experiments calibrate a single global $\beta$ on the calibration set with no test-point inclusion or per-candidate-label search — a substantively different procedure from the transductive Algorithm 1 whose coverage Proposition 4.1 certifies. This should be signalled clearly.
• Documentation friction. The $\Lambda_1$ matrix — the setting for the strongest claims — is a README-documented manual edit rather than a CLI option, and both variants write to the same output filename (silent overwrite). SOCOP's lambda_param, the BDD split fractions, and the BDD loss constants live only in code, and the STOP/LEFT/RIGHT loss formulas are truncated in Appendix D.1, so the BDD loss cannot be reconstructed from the paper — a real gap given the BDD run could not be re-executed.

Recommended Changes

Essential

• Reconcile Algorithm 1 with what is run. Either prove the global-$\beta$ split-conformal proxy is a valid split-conformal calibration under a stated nonconformity score (so Proposition 4.1 carries over), or state explicitly in Sections 4.2 and 5 that the experimental coverage is heuristic and distinct from the certified transductive procedure.
• Complete and report the $\Lambda_1$ and BDD runs. Provide the full multi-seed $\Lambda_1$ COVID and BDD evaluations that substantiate the headline claims ("ROCP almost eliminates critical mistakes," "RAC becomes brittle as $\alpha$ grows," "ROCP matches or outperforms all baselines"), since these did not complete in replication; expose $\Lambda_1$ via a --loss-matrix {baseline,safety} flag and write to distinct output paths.
• Qualify or formalize the risk-optimality claim. Add a consistency-style result for ROCP $\to$ oracle under regularity (uniform consistency of $\hat Q_t^x$, continuity of the argmin correspondence), or rewrite "targeting the decision-theoretic optimum as the model improves" and "approach the oracle risk performance" as explicit heuristics in the abstract, introduction, and Remark 4.2.
• Add a sensitivity sweep over the $\Lambda_1$ multiplier. Show ROCP vs. RAC vs. Best-Response over a range of severe-mismatch multipliers (e.g. 2×, 5×, 10×, 20×, 50×) so the reader sees where the out-of-set penalty begins to matter rather than a single ad hoc 10× point.
• Document all experiment hyperparameters and losses. Put SOCOP's lambda_param, the BDD split fractions, the BDD loss constants, and the actual STOP/LEFT/RIGHT loss formulas into Appendix D, and document the ~5h single-threaded runtime.

Suggested

• Add an expected-loss-native comparator. Include conformal risk control (Angelopoulos et al. 2022) and/or the Cortes-Gomez et al. (2025) utility-directed method to substantiate that the gains come from the $\alpha$-weighted out-of-set term.
• Isolate the Best-Response confound. Rerun Best-Response (and the ROCP plug-in) with a directly-estimated joint 8-class posterior instead of per-bit isotonic + conditional independence, so the BDD comparison does not conflate miscalibration of $f_x$ with the best-response paradigm.
• Discuss the $(2-\alpha)$ bound honestly. State that it is a bound on the same policy (not on ROCP's marginal output) and that it worsens to $2$ as $\alpha\to0$; provide a tightness example or an improved bound.
• Report paired significance. Replace ±1 SE bands with paired confidence intervals / tests across the shared splits for the "consistently lower" and "almost eliminates" comparisons.
• Consider a moderated ambiguity set. Discuss whether the coverage-only worst case can be tempered (bounded density ratio / divergence ball) and whether the duality machinery survives.
• Address discrete-$X$ and continuous-$\mathcal Y$. Note where non-atomicity of $P_X$ fails and the correct dual object for discrete features, and sketch a continuous-$\mathcal Y$ instantiation of ROCP (candidate discretization, thresholded score calibration).
• Broaden scope and rename the driving task. Add at least one larger-$|\mathcal Y|$ or genuinely sequential experiment, and reframe Section 5.2 as a synthetic hazard-classification benchmark.
• Clarify notation and citations. Justify the $\ell^{\text{out}}_{\mathcal Y}(a):=\ell^{\text{in}}_{\mathcal Y}(a)$ convention, disambiguate $M$ vs. $M(a)$ in Fig. 2, reword the critical-mistake metric as "the chosen action belongs to $\arg\max_a\ell(a,y_c)$," and verify future-dated arXiv identifiers (e.g. Chen et al. 2026).