Faster Activation Functions at the Edge for Post-Training Speedups

Summary

The paper presents FFCC, a compiler that automatically generates fast approximations of expensive activation functions (SiLU, GELU, tanh) by exploiting the fact that reinterpreting the bit pattern of an IEEE-754 float as an integer yields an approximate log2, and vice versa. This bit-cast trick has a long history in graphics (Quake's fast inverse square root) and has previously been applied by hand to isolated functions such as exp (Schraudolph). The conceptual move here is to lift the trick into an SSA-based DSL with rewrite rules, so that a user can specify an activation as an expression in basic float operators and FFCC will lower it into a bit-cast-based implementation, then holistically tune the residual "magic constants" via gradient descent on MSE. The main empirical claims are (1) 13–26× speedups over exact implementations on Arm M7, Snapdragon 8 Elite, and Intel i7-1260P, including 23× over ggml's hand-tuned SiLU on AVX2; (2) absolute error not exceeding 0.019 across [-4, 4], roughly an order of magnitude better than h-swish; and (3) post-training swap-in preserves accuracy on 7/8 small benchmarks and drops ConvNeXt-Tiny/Femto Top-1 by only 0.1–0.2 pp.

The contribution is genuinely useful: prior hardware approaches (e.g., Flex-SFU) demand ~6% chip area, and the leading software baseline (h-swish) is a single-function approximation with about 10× worse error. Automating a bit-cast rewrite plus holistic tuning removes both the manual per-function derivation and the hardware requirement, and the open-source ffcc release is a real artifact.

The main conceptual limitation is scope. The theory is derived only for positive normal floats; the evaluation is exclusively CNN-based despite GELU being motivated as a transformer activation; INT8/FP16 — the dominant edge deployment mode — is not addressed; and the single documented failure (MLPerf anomaly detection) is left to future work with an unimplemented guard sketch. Several formulas are also stated with errors — h-swish, the Table 1 GELU, and an Adam learning rate of 1e6 — and internal inconsistencies between the 13–26×, 15–22× and 15–16% headline numbers make the speedup story harder to verify than it should be.

Strengths

• Conceptual contribution. Turning the graphics-style bit-cast approximation into a generic DSL pipeline (normalize → apply approximation → holistic constant tune → lower) is a clean idea and clearly the right level of abstraction — one derivation instantiates SiLU, GELU, and tanh from a shared code path.
• Shipped constants make the accuracy story checkable. Listing 1 gives the full FastSiLU/GELU/tanh code with all constants, so the max-absolute-error 0.019 claim and the visual overlays in Figure 8 are deterministically reproducible from the manuscript alone.
• A meaningful strong baseline. Comparing against ggml's hand-tuned inline-assembly SiLU on AVX2 (rather than only against libm) exercises the claim against actually-optimized code, and the 23× speedup — attributed to AVX2's shared int/float register file making bit-casts free — has a physically plausible mechanism.
• Honest reporting of a failure case. The Anomaly Detection column in Table 3 explicitly flags a failure mode (activations exceeding the numerically safe range) rather than dropping the row, and correctly diagnoses it as analogous to a post-training quantization failure.
• Open-source release. Publishing ffcc on GitHub is a real reproducibility asset for the compiler pipeline (as opposed to just the shipped constants).

Weaknesses

• Objective/motivation mismatch. Section 3.5 motivates constant tuning as minimizing maximum relative error and cites Kadlec's 2.7× reduction in that metric to justify holistic tuning. Section 4.3 then implements MSE around zero by Adam. These are different objectives — MSE can leave worst-case bounds uncontrolled — and the paper never justifies the switch or notes that the Kadlec number does not transfer.
• Formulas and constants contain errors. The h-swish formula stated as ReLU6(x)/3 − 0.5 is wrong (correct is ReLU6(x+3)/6), the Table 1 GELU has a garbled erf argument, and the Adam learning rate is given as 1e6, which would immediately diverge and is almost certainly a typo for 1e−6. The FastGELU multiplier of −21513938 in Listing 1 is also ~4% larger in magnitude than a naive derivation from 1.702·log2(e)·L suggests, with no explanation for the gap.
• Internal inconsistencies in the speedup range. The 13–26× headline does not match the body: the M7 improvement is quoted as 15–16% of original (~6×), the Snapdragon/Intel range is 15–22×, and 23× is the ggml-specific figure. Neither the 13× lower bound nor the 26× upper bound is traced to a specific measurement.
• Overclaimed accuracy comparison. "Order of magnitude more accurate than h-swish" is quoted for "our activation functions" (three targets), but h-swish only approximates SiLU. The like-for-like comparison holds only for the SiLU row.
• "Post-training drop-in" story is narrow. All eight rows of Table 3 are CNNs and the ConvNeXt evaluation is also a CNN, despite the paper's motivation centering on GELU in transformers. The one documented failure (Anomaly Detection) is left as future work rather than demonstrating the range-safe guard variant that is sketched inline.
• No quantitative comparison against prior software approximations besides h-swish. Schraudolph's fast-exp macro uses the same bit-cast trick and would be the natural baseline to demonstrate that automated holistic tuning beats a single manual constant; Perini & Reitz, Moroz, and Wulfert et al.'s AIfES are cited but not benchmarked.
• Hardswish column in Table 3 mixes training regimes. Section 5.5 says the Hardswish column is trained with Hardswish while the FastSiLU column swaps at inference. That makes the FastSiLU vs SiLU cell fair but confounds the "Hardswish worse than SiLU/FastSiLU" reading, since the H-Swish gap partly reflects training regime rather than approximation quality.
• Single-run accuracy numbers with no variance. Every accuracy cell is a point estimate without seeds or intervals. The 0.1–0.2 pp deltas that anchor the ConvNeXt claim and the "no observable drop" 7/8 claim are within typical single-seed noise on these benchmarks.
• Scope confined to FP32 normal floats. Section 3.4 restricts the derivation to positive normal numbers; subnormals, zero, and inf/NaN are punted. Most real edge deployment runs INT8 or FP16 rather than FP32, and the paper does not discuss how FFCC composes with either.
• Overstated interpretive language. "Profound impact on accuracy" for holistic tuning is not supported by the authors' own 0.1–0.3 pp end-to-end measurement, and Figure 7's per-stage grid shows values at 0.99 and 1.01 — mild parity or regression — which the "speedups across architectures" prose glosses over.
• Minor formula/exposition issues. Figure 3's caption ("subtracting ε from x") disagrees with the body ("subtracting from B"); the exponent set E is written to include segments that the "normal numbers only" restriction should exclude; and the ±74 instability window is presented as symmetric without deriving why the upper bound is needed for a function whose overflow risk is one-sided.

Reproducibility & code

Code was released (github.com/AntonLydike/ffcc) and inspected but not executed for this review. What can be verified from the manuscript alone vs. what would require the released repo or additional information:

• Deterministic from Listing 1. The 0.019 max-absolute-error claim, the Hardswish vs FastSiLU/GELU/tanh error curves in Figure 8b, the visual overlays in Figure 8a, and the ±74 numerical-safe-range behavior all follow deterministically from the shipped constants and can be evaluated on a grid without any additional code.
• Micro-benchmark harness undisclosed. The paper names hardware and compiler flags (clang -O3 -ffast-math) but not the measurement loop: sample count, warmup, whether inputs are held in registers or streamed from memory, batch size, thread pinning, or frequency governor. On Snapdragon/Intel the reported 15–22× and 23× (vs ggml) are highly sensitive to these choices; the M7 6× is more robust because the code is tiny.
• Baselines not named. The "exact SiLU/GELU" implementations the fast kernels are compared against are not identified on any platform — could be libm, __builtin_expf, CMSIS-NN, vector intrinsics, or SVML, each with different cycle counts. The headline speedup ratios depend on this denominator.
• ggml/llama.cpp baseline not pinned. No commit hash, tag, or date; not stated whether the 23× is scalar-vs-scalar or extracted from a vectorized kernel.
• Constant-tuning setup underspecified. The tuning stage produces the shipped constants, so its determinism matters. Missing: initial constant values, sample distribution over [-6, 6], batch size, random seed, and — as noted above — an obviously-wrong Adam learning rate. A reader cannot re-derive the constants in Listing 1 from the described procedure.
• Dataset labels in Table 3 are ambiguous. "Image Class." / "Kw Spotting" / "Anomaly Detect." rows do not name the dataset; the five CNN backbone rows do not state their training dataset either. Absolute accuracies of 83.86 or 91.94 are not interpretable without pinning these.
• ConvNeXt checkpoint provenance missing. The exact pretrained ConvNeXt-Tiny/Femto checkpoint source (original release vs timm) and ImageNet val preprocessing pipeline are not stated; the 0.1/0.2 pp deltas are anchored to an unspecified baseline.
• Per-stage timing harness for ConvNeXt not described. How the Figure 7 stage-level speedups were extracted (torch hooks, per-stage forward passes, or end-to-end with stage-by-stage GELU swap) is not explained.
• No seeds anywhere. Neither training seeds (Section 5.5) nor evaluation seeds (Tables 3, 4) are reported, and no confidence intervals accompany the sub-0.5 pp gaps that carry the drop-in accuracy story.

Recommended Changes

Essential

• Fix the wrong formulas and constants. Correct the h-swish formula to ReLU6(x+3)/6, restore the Table 1 GELU's erf(x/√2), replace 1e6 with the actual Adam learning rate used, and either justify the FastGELU multiplier −21513938 (documenting pre- and post-tuning values) or correct it. These fixes are prerequisites for the correctness sections and Section 5.4's tuning story to be checkable.
• Reconcile the 13–26× / 15–22× / 15–16% headline numbers. Trace each end of the reported ranges to a specific platform and measurement, or narrow the abstract's claim to a range that matches the body.
• Reconcile the tuning objective with the motivation. Either explain why MSE-around-zero is the right objective for the ML-deployment setting and stop invoking Kadlec's 2.7× max-relative-error result as though it transfers, or actually optimize max relative error and re-report Section 4.3.
• Describe the micro-benchmark harness and name the baselines. State the loop structure, sample count, warmup, thread pinning / frequency governor for the Snapdragon and Intel results, and pin the exact libm / intrinsic / vector kernel used as the "exact" baseline on each platform. Include the ggml commit hash and specify whether the 23× comparison is scalar-vs-scalar or scalar-vs-vector.
• Fully specify the constant-tuning setup. Report initial constant values, the sampling distribution over [-6, 6], batch size, and a seed; ideally point to the exact ffcc script that produced the constants in Listing 1.
• Label the Table 3 datasets and the ConvNeXt checkpoint sources. Name the dataset for each Table 3 row (including the three MLPerf Tiny rows), name the pretrained ConvNeXt-Tiny/Femto checkpoints, and describe the ImageNet val preprocessing pipeline for Table 4.
• Implement and evaluate the range-safe guard. The if (x < -74) ... mitigation sketched inline is a few lines of code; run it on the MLPerf Anomaly Detection network, report the recovered accuracy, and quantify the guard's runtime overhead. This turns the drop-in claim from "7/8 works" into "8/8 with a small overhead."

Suggested

• Add a comparison to Schraudolph's fast-exp and to Flex-SFU. A Schraudolph-style single-constant bit-cast exp baseline in Figures 6 and 8 would demonstrate that FFCC's holistic tuning wins over a hand-tuned constant, not just over h-swish; a numerical comparison to Flex-SFU on a common activation would ground the "no special hardware" positioning.
• Add a small transformer to the post-training evaluation. The abstract motivates GELU by its role in transformers, so a small distilled BERT/ViT eval — even at a single accuracy point — would close the CNN-only scope gap.
• Report multiple seeds or explicitly acknowledge single-run noise. For the 0.1–0.2 pp deltas in Tables 3 and 4, either provide mean ± std across seeds or annotate that these are single-run values and readers should treat sub-0.5 pp gaps as within noise.
• Address quantized deployment. Add a paragraph on how FFCC composes with (or does not compose with) INT8 / FP16 inference, since these are the dominant edge deployment modes.
• Fix the Table 3 Hardswish confound. Add a row (or a footnote) showing Hardswish evaluated as an inference-time swap on a SiLU-trained model, so the FastSiLU-vs-Hardswish comparison is a like-for-like drop-in comparison.
• Correct the smaller exposition issues. Align the Figure 3 caption ("from x") with the body ("from B"); narrow the "positive, normal" exponent set E accordingly in Section 3.4; derive the ±74 guard bound per activation rather than presenting it as symmetric; soften "profound impact" and "speedups across architectures" to match the measured effect sizes; broaden Softplus's "typical usage" beyond VAE models; and fix the typos "perofrmance" and "fastes" in the Figure 7 discussion.