Mind Your Margin and Boundary: Are Your Distilled Datasets Truly Robust?

Summary

The paper tackles adversarially robust dataset distillation (DD): distilled synthetic sets inherit predictive signal from the source data but not its robustness, and existing robust DD methods (ROME's class-mean feature alignment, GUARD's curvature regularization) trade clean accuracy for robustness or spread optimization uniformly over adversarial companions. The authors argue this uniform treatment is misaligned with the fact that robust risk is dominated by the smallest robust margins, and that class-mean alignment does not push apart the impostor logits that decide $\max_{k \neq y} f_k(x+\delta)$. C²R couples an Attack-Aware Curriculum (AAC) — a robust-hinge surrogate that ranks adversarial examples by an estimated margin and reweights optimization toward the smallest — with a Contrastive Robustness Loss (CRL) that replaces class-mean matching by supervised contrastive alignment between clean and adversarial companions, with hard negatives retrieved via a low-dimensional random projection over a class-balanced FIFO queue. A warm-started single-step-with-line-search attacker (LS-PGD) trims inner-loop backprops. Empirically the paper reports ~2.8pp average robust-accuracy gains across CIFAR-10/100, Tiny-ImageNet and six ImageNet-1K subsets.

The conceptual move — reframing robust DD as tail-margin optimization plus instance-level contrastive boundary regularization — is principled and well-motivated by the Sec. 4.1 analysis, and the LS-PGD / queue design is a sensible efficiency contribution. However, the presentation overreaches: the headline "highest accuracy across all datasets and attack types" is contradicted by the tables (uniformly lower Clean than MTT on ImageNet; several AutoAttack cells where C²R < D⁴M silently marked "+0.00"); the baseline set swaps between Tab. 1 and Tab. 2 without justification; the "worst hinge = min margin" identity is unconditional though it saturates once the minimum margin exceeds 1; the "preserves attack strength" LS-PGD tagline is stronger than Tab. 4 supports; and CRL design pieces (positive set composition, projection dimensionality, memory-queue staleness) are underspecified in ways that matter for the paper's own margin-centric mechanism. The linked repository is an empty stub, so a from-scratch reimplementation was needed to attempt any replication.

Strengths

• Margin-centric motivation. Sec. 4.1 provides a compact, principled argument that robust risk is decided by the minimum robust margin and that the hinge surrogate identifies the worst-case sample when the surrogate is active. This is a cleaner conceptual framing than the prior average-alignment view and directly justifies the curriculum choice.
• Instance-level contrastive robustness. Replacing ROME-style class-mean $L_{\mathrm{rob}}$ (Eqn. 5) with supervised contrastive alignment on clean–adversarial pairs targets the impostor-logit term $\max_{k \neq y} f_k(x+\delta)$ more directly than mean matching, matching the boundary-level nature of robust errors.
• Efficiency mechanisms. Warm-started single-step LS-PGD with a forward-only line search and a class-balanced FIFO queue for CRL negatives are practical engineering choices that address the two real cost drivers of robust DD (inner-loop backprops and $O(M^2)$ pair construction).
• Breadth of evaluation. The paper covers CIFAR-10/100, Tiny-ImageNet, six ImageNet-1K subsets, three $\varepsilon$ budgets (2/255, 4/255, 8/255), five gradient-based attacks plus AutoAttack, an RDD/distribution-shift comparison, per-component and inner-attacker ablations, and an $\eta$ sensitivity sweep.
• Honest reporting of the IPC 10→50 dip. The paper flags — rather than hides — that C²R's absolute robust accuracy dips at IPC=50 on some settings, although its explanation for the dip is unsupported (see Weaknesses).
• Reimplementable core. The equations and Algorithm 1 are complete enough that a from-scratch reimplementation of AAC, CRL and LS-PGD is possible without guessing the loss form. The direction of the headline finding (C²R beats a mean-alignment robust-DD baseline in every fair CIFAR-10 cell, both modules contribute) survived that reimplementation.

Weaknesses

• Overreaching "highest accuracy across all datasets and attack types" claim. Tab. 2 shows MTT's Clean beats C²R in every ImageNet subset / IPC cell (ImageNette 48.20/66.40/67.60 vs 44.10/63.80/64.70; same pattern in all other subsets). Tab. 7 (AutoAttack) shows C²R below D⁴M in several CIFAR-10 cells (30.56 vs 32.19, 40.67 vs 43.00, 6.38 vs 10.23, 8.67 vs 9.45, 2.31 vs 5.07). The main-text sentence in Sec. 5.2 is unsupported as written.
• Systematic masking of losses via "(+0.00)" subscripts. The reporting convention writes the improvement over the second-best baseline as "+0.00" whenever C²R is behind, and omits blue subscripts entirely on Clean rows. Combined, this hides both accuracy losses and a 5-point clean-accuracy regression at CIFAR-10 IPC=50 (D⁴M 69.81 vs C²R 64.73).
• Asymmetric baseline sets. Tab. 1 uses SRe²L / D⁴M / ROME but not MTT; Tab. 2 uses only MTT. ROME — the clear second-best on CIFAR — is silently absent from the ImageNet comparison, and the 2.8pp headline aggregate mixes these differently populated tables.
• Theoretical equivalence overstated. "Improving the worst robust-hinge loss is equal to improving the minimum robust margin" holds only within the hinge-active region ($m<1$); once the minimum margin exceeds 1 the hinge saturates and the identity breaks. Similarly the $\mathrm{argmax}\ v_i \leftrightarrow \mathrm{argmin}\ m_i$ correspondence needs the same qualifier.
• LS-PGD guarantees vs prose. "At least as strong as a single-step PGD" holds only in the update branch, not the reuse branch, and Tab. 4 shows LS-PGD is uniformly (if marginally) worse than multi-step PGD on all four datasets. "Preserves attack strength" is stronger than the evidence supports; the theoretical guarantee is single-step whereas the ablation compares against multi-step — this asymmetry should be surfaced.
• Contradictory IPC scaling narrative. "Improvement most pronounced at high IPC" and "slight decrease in robust accuracy when IPC increases from 10 to 50" (both in Sec. 5.2) are not reconciled. The stated cause of the dip — denser synthetic distributions producing more overlapping boundary regions — is offered as fact with no diagnostic, and sits in direct tension with CRL's stated boundary-widening goal.
• CRL design pieces underspecified in ways that matter for the argument. The positive set $P(i)$ is described verbally but never defined; Algorithm 1 uses a set "$N(i)$" that never appears in Sec. 4.3; the random projection $R \in \mathbb{R}^{r \times d}$ has no stated $r$, seed, or draw distribution; and the memory queue is a plain FIFO whose staleness under an evolving encoder is not discussed (no momentum encoder, no re-encode on read). Since CRL's contribution rests on retrieving the nearest impostors, the resolution of that retrieval directly bounds how well the mechanism matches its motivation.
• No comparison against a hard-example-mining baseline. AAC sorts mini-batches by descending $s(x)$ (an anti-curriculum) and defends the choice via the tail-margin argument, but there is no ablation against ascending order, uniform shuffling with $s(x)$-based reweighting, or standard hard-example mining. Without one of these, the AAC contribution cannot be attributed to ordering rather than merely scoring.
• Naming, notation, and figure drift. Abstract expands C²R as "Contrastive Curriculum for Robust …" while the title/intro use "Contrastive and Curriculum Robust …"; the repository is CCR. Sec. 5.1 says "six attacks", Appendix B.2 says "eight". Fig. 3's caption says "fatter trajectory" while the body says "flatter". Fig. 6's y-axis reads "Test Accuracy" while the caption says "average robust accuracy". Sec. 5.2 refers to "the generator" although the pipeline has none.
• IPC grid mismatch and $\eta$ gap. Sec. 5.1 pins the IPC set to $\{1, 10, 50\}$ but Tables 1/5/6/7 include IPC=5 and IPC=30, and the $\eta$ defaults are given only for CIFAR-10 at $\{1, 10, 50\}$. Given the paper's own observation that optimal $\eta$ shifts with IPC, this is a real gap.
• Headline evaluation at a single small $\varepsilon$. Tables 1 and 2 use only $\varepsilon = 2/255$, a comparatively small budget; the $\varepsilon = 4/255$, $8/255$, and AutoAttack comparisons live in the appendix and, at the higher budgets, many cells fall to a near-zero floor where the "improvement" is decided by tie-breaking rather than differentiation.
• Under-defined RDD comparison. Tab. 10's "Cluster-min" metric is not defined, the shift-generation protocol is not restated, and only IPC=10 is reported.
• "66.8% DR" bound not obviously traceable. The per-dataset best-IPC DR values annotated on Fig. 3 (16.21%, 39.08%, 28.97%, 21.63%) average to ~26%, not near the 66.8% figure the text uses to justify "first DD method to reach this regime". The averaging window needs to be pinned.

Reproducibility & code

• Empty upstream repository. The URL https://github.com/SLGSP/CCR cited in the abstract is a placeholder README ("Code is coming soon~") with no training code, LS-PGD/CRL implementation, distilled .pt sets, seed configs, or evaluation harness. The mismatched name (CCR vs C²R) adds friction to even locating the intended artifact. Because no author code exists, a veritas-mode replication was only possible by reimplementing the entire framework from Algorithm 1 and the equations.
• Directional headline finding survives a from-scratch reimplementation, on a much reduced scale. On CIFAR-10 the reimplementation reproduces the sign and rough magnitude of the "2.8pp average gain" (measured 2.175pp against C²R's own mean-alignment ablation, standing in for the paper's ROME/SRe²L/D⁴M baselines that had no runnable code to obtain). C²R wins in 10/10 CIFAR-10 cells across five attacks at IPC∈{1, 10}. Both AAC and CRL individually improve over the no-module baseline, and enabling both is best. So the direction of the headline is not a mirage.
• Absolute numbers do not reproduce. All CIFAR-10 C²R cells the reimplementation could produce are ~5–14pp below the paper (Clean IPC=1: 25.53 vs 35.04; IPC=50: 50.80 vs 64.73), and CW collapses to ≤0.17% versus the paper's ~24–29%. This is dominated by an environmental limitation (no working GPU on the reproduction host, forcing CPU-only reduced-iteration runs) rather than by the paper's methodology, but it means the reimplementation cannot certify the paper's absolute magnitudes.
• Table 1 mostly not attempted. 10 of the 15 dataset-IPC columns of Table 1 (all CIFAR-100, all Tiny-ImageNet) could not be produced under the 2-CPU-core budget because the C²R distillation inner loop scales with the number of classes.
• Table 2 (ImageNet subsets) and all higher-$\varepsilon$ / AutoAttack / RDD / eta-sweep / synthetic-image visualization / accuracy-vs-time replications were not attempted. These need resources the reproduction host lacked, chiefly a working GPU. They are irreducible-scope cuts under the CPU-only constraint, not failed attempts.
• LS-PGD speedup claim inverted on this hardware. The reimplementation measured LS-PGD at ~0.75× the multi-step PGD speed (i.e. slower), against the paper's 1.4× speedup. This is consistent with LS-PGD paying $Z$ forward probes to save $T-1$ backprops: on a 2-core CPU the added forwards can dominate. The direction is plausible on a GPU with the paper's intended $T$, but the sign-of-speedup being hardware-dependent means the "1.4×" is far more setup-dependent than the paper acknowledges — and the missing $T$, $Z$, batch size and reference GPU make it impossible to verify even directionally.
• Ablation ordering partially inverted. The reimplementation reproduces "both > either > neither" but has CRL-only above AAC-only, whereas the paper reports AAC-only above CRL-only. The magnitude of the swap is within the run's overall reduced-scale noise, so this is not evidence against the paper; but it does illustrate that Tab. 3 is not interpretable without a written specification of the "AAC off" and "CRL off" fallbacks (random ordering? uniform sampling? class-mean matching? no robustness term?).
• Hyperparameters that a reproducer must guess. $\tau$ (CRL temperature), $Q$ (queue capacity), $M$ (mini-batch size), $r$ (projection dim), $k$ (hard-negative top-$k$), $I$ (distillation iterations), $(\alpha, \beta, Z)$ for LS-PGD, $T$ for the multi-step PGD baseline, $\eta$ per (dataset, IPC) beyond CIFAR-10, and the ImageNet subset class lists are all either missing or deferred to "prior practice". For tables where cell values are <3pp (Tab. 6, Tab. 9), reproducibility to within a meaningful band requires all of these.

Recommended Changes

Essential.

• Rephrase the "highest accuracy across all datasets and attack types" sentence. Scope it to "highest robust accuracy on most attack/dataset settings", explicitly acknowledge that MTT beats C²R on Clean in every ImageNet subset/IPC cell of Tab. 2 and that D⁴M beats C²R in several AutoAttack cells of Tab. 7.
• Report signed improvement subscripts (or blank cells) whenever C²R is not the best. Replace the current "(+0.00)" convention across Tabs. 5–7 so that losses to D⁴M, ROME, and other baselines are visible. Extend the subscript scheme to the Clean rows of Tab. 1.
• Release the actual code, distilled sets, and seed configuration. Push the C²R training pipeline, LS-PGD/CRL implementations, per-dataset per-IPC seed configs, and distilled .pt files to the linked repository (or fix the URL), and align the repo name with the method name. This is the largest single blocker to reproducibility.
• Add a complete hyperparameter table per (dataset, IPC). Fix $\tau$, $Q$, $M$, $r$, $k$, $I$, $(\alpha, \beta, Z)$ for LS-PGD, $T$ for the multi-step PGD baseline, and $\eta$ for every (dataset, IPC) that appears in Tables 1–10 (including IPC=5, 30 and every non-CIFAR-10 dataset).
• Qualify the theoretical equivalence. State that "worst hinge = min margin" holds within the hinge-active region ($m<1$) and add a corresponding caveat to the "sample that maximizes $v_i$ is exactly the one with the smallest robust margin" line. Note that PGD ascent on training loss is a surrogate for minimizing the logit margin.
• Justify or fix the baseline-set asymmetry between Tab. 1 and Tab. 2. Add ROME (and SRe²L / D⁴M) to the ImageNet-subset table, or state precisely why they are inapplicable at 128×128; recompute the 2.8pp headline against a common set.
• Specify the ablation fallback semantics in Tab. 3. Write what "AAC off" (random ordering? uniform sampling? no LS-PGD?) and "CRL off" (class-mean matching à la Eqn. 5? no robustness term?) mean concretely.
• Formally define $P(i)$ and rename Algorithm 1's $N(i)$ to match Sec. 4.3. Also state the random-projection dimensionality $r$, how $R$ is drawn, and whether the memory queue re-encodes on read or uses a momentum encoder.

Suggested.

• Soften the LS-PGD framing. Replace "preserves attack strength" with "nearly matches multi-step PGD at a 1.4× speedup". Add the caveat that the theoretical guarantee applies only to a single PGD step and only in the update branch. Report $T$, $Z$, batch size, and reference GPU alongside the speedup so the 1.4× is checkable.
• Add a hard-example-mining baseline to the AAC ablation. Compare ascending vs descending $s(x)$ ordering, and $s(x)$-weighted uniform sampling, to isolate the contribution of ordering from the contribution of scoring.
• Reconcile the IPC scaling narrative. State explicitly that "most pronounced at high IPC" refers to the gap over baselines while the absolute robust accuracy of C²R dips at IPC=50. Back the "denser distributions → boundary overlap" hypothesis with a diagnostic (inter-class distance vs. IPC, or margin-distribution snapshots at IPC=10 vs 50), or hedge to "we hypothesize."
• Fix the naming/notation/figure drift. Align the abstract, intro, and repository name on one C²R expansion; make Sec. 5.1 and Appendix B.2 agree on the attack count; correct Fig. 3's "fatter" to "flatter"; relabel Fig. 6's y-axis as "Average Robust Accuracy (%)"; replace "the generator" in the Fig. 4 discussion.
• Ground the "66.8% DR / first DD method" claim by stating the exact averaging window (datasets, IPCs) and giving the corresponding whole-data DR so the "first" phrasing is checkable.
• Define "Cluster-min" and pin the noise $\sigma$, blur kernel size, and invert protocol for Tab. 10; add IPC=1 and IPC=50 rows for the RDD comparison.
• Add hardware and measurement scope for Fig. 5 and Tab. 4's speedup (GPU model, batch size, inner-loop only vs. end-to-end distillation).
• Consider moving $\varepsilon = 4/255$ (and/or AutoAttack) into a headline table so the paper's "strong, standardized attacks" language is met by the main-text evaluation.
• Pin the ImageNet subset class lists (WNIDs or class indices per subset) in Appendix B.1 so Tab. 2 is reproducible.