← Blog

CadLLM

Improving the Throughput of Diffusion-based Large Language Models via a Training-Free Confidence-Aware Calibration

Jucheng Shen, Gaurav Sarkar, Yeonju Ro, Sharath Nittur Sridhar, Zhangyang Wang, Aditya Akella, Souvik Kundu

Findings of ACL 2026 | paper | code

A controller reads the model's confidence and sets the decoding block size: large when confident, small when not.

TL;DR: Diffusion LLMs already report how confident they are in each token. CadLLM uses that signal to spend compute where the model is still uncertain and save it where predictions have settled. It needs no retraining, and it improves throughput by 1.1–2.28× over the Fast-dLLM threshold baseline on LLaDA with competitive accuracy.

Why decoding is slow

An autoregressive language model produces tokens in a single left-to-right pass. A masked diffusion LLM instead refines a noisy sequence over many denoising steps, and this repeated refinement can make token generation much slower.

Fast-dLLM speeds this up by unmasking, in parallel, every position whose confidence clears a threshold. But its plan is static: fixed block sizes, a fixed number of steps per block, fixed sampling breadth, and a fixed commit threshold.

Confidence is not uniform

A static plan would be fine if every part of a sequence were equally hard, but it is not. Within a block, confidence rises quickly and then plateaus. Across blocks, some stabilize immediately while others stay uncertain. A fixed schedule therefore over-refines easy blocks and under-serves harder ones.

Three heatmaps of mean token confidence by block and step for GSM8K, HumanEval and MBPP. Confidence varies clearly across blocks and steps.
Per-block, per-step confidence on three datasets. Lighter means higher confidence.

This work grew out of our earlier workshop paper, OSDT, which found that confidence trajectories are nearly identical across inputs from the same dataset, so thresholds calibrated on a single sequence can be reused for later inputs. CadLLM goes further by adapting block size, steps, vocabulary, and threshold during decoding.

Sampling breadth has its own cost. Softmax runs at every step, and running it over the full vocabulary of about 50K tokens is nearly an order of magnitude slower than using a small subset.

Line chart of softmax latency against the number of tokens. Latency stays near 5 microseconds up to 1000 tokens and reaches 49.4 microseconds for the full vocabulary.
Softmax latency against vocabulary size, on a log scale.

One signal, four policies

After each forward pass, CadLLM reads the model's confidence and feeds it back into a closed-loop controller. That single signal, together with generation progress, drives four decisions:

PolicyWhat it controlsHow it adapts
Block size BtHow many tokens to decode in parallelLarger when confidence is high
Steps StHow much refinement to allocateMore when confidence is low
Vocabulary VtHow broad the candidate set isWider early, under low confidence, or when tokens start repeating
Threshold τtWhen to commit a tokenStrict at first, relaxing as generation progresses
Overview diagram. High confidence maps to a large block, few steps, a low threshold and a small vocabulary; low confidence maps to the opposite. Below, the controller updates these parameters in a loop around the diffusion LLM.
The controller replaces static parameters with values updated from confidence during decoding.

Each policy is a simple rule, using linear interpolation with clipping. For example, with c̄ as the recent mean confidence:

Bt = clip(Bmin + (Bmax − Bmin) · c̄, Bmin, Bmax)

This simplicity is deliberate. It isolates the benefit of adaptivity itself and adds only a multiply-add per policy per block at inference time.

Results

We compare against both Fast-dLLM variants on LLaDA, on a single NVIDIA H100.

BenchmarkLengthCadLLMFast-dLLM (factor)Fast-dLLM (threshold)
GSM8K25678.01%
120.07 (1.33×)
76.19%
119.18 (1.32×)
79.00%
90.40
GSM8K51275.44%
107.79 (1.37×)
74.45%
100.66 (1.27×)
75.28%
78.77
MATH25632.06%
106.84 (1.34×)
32.22%
109.97 (1.38×)
32.40%
79.58
MATH51234.94%
117.21 (1.14×)
35.40%
111.86 (1.08×)
32.06%
103.18
MBPP25624.00%
99.86 (1.37×)
21.20%
96.01 (1.31×)
25.60%
73.15
MBPP51213.00%
104.62 (1.35×)
13.20%
100.73 (1.30×)
14.20%
77.71
HumanEval25635.97%
220.81 (2.28×)
32.92%
132.28 (1.37×)
37.19%
96.84
HumanEval51243.29%
163.72 (1.74×)
41.46%
131.14 (1.38×)
45.12%
94.41

Each cell shows accuracy, tokens per second, and improvement relative to the threshold baseline at the same generation length.

CadLLM also reaches 1.1–1.4× gains on DREAM across the same four benchmarks. It works with both prefix-only and dual KV caching.

What each policy contributes

Turning policies off one at a time on GSM8K shows that they play different roles:

ModeTokens/sAccuracyNFE
All policies on121.7278.01%86,816
No adaptive vocabulary119.6774.41%86,666
No adaptive steps136.7676.12%76,955
No adaptive block size111.1978.32%92,344
No adaptive threshold34.5778.17%337,664
All policies off34.3278.01%337,664

GSM8K at generation length 256. NFE is the number of function evaluations.

Limitations

Results can remain sensitive to a few hyperparameters that may need careful tuning, so automated search is a natural next step. The linear rules are also intentionally minimal, and learned policies could tighten the trade-off between efficiency and quality further.

Citation

@inproceedings{shen2026cadllm,
  title     = {Improving the Throughput of Diffusion-based Large Language
               Models via a Training-Free Confidence-Aware Calibration},
  author    = {Shen, Jucheng and Sarkar, Gaurav and Ro, Yeonju and
               Sridhar, Sharath Nittur and Wang, Zhangyang and
               Akella, Aditya and Kundu, Souvik},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  pages     = {9826--9837},
  year      = {2026},
  doi       = {10.18653/v1/2026.findings-acl.478}
}