SPECTRA, an experimental accelerator for speculative decoding, changes how its arithmetic units and memory traffic are organized as a language model moves between drafting and verification. On a 20-tile FPGA prototype, that per-kernel reconfiguration delivered up to a 2.09-fold speedup relative to the best fixed compute design tested by the researchers.
Speculative decoding pairs a small draft model with a larger target model. The draft proposes several tokens, then the target checks them together. This preserves the target model's output distribution while replacing some serial target-model calls with a parallel verification step.
The two parts stress hardware differently. Draft generation processes one token at a time and is usually limited by moving model weights from memory. Batched verification uses larger matrix operations and can keep many multipliers busy. A datapath optimized for either phase therefore wastes resources in the other.
One array switches between two dataflows
Each SPECTRA accelerator tile contains the same 8 by 8 array of 64 processing elements. For matrix-vector work during draft decoding, the array becomes eight parallel dot-product lanes and reads a full 8 by 8 weight tile each cycle. For the matrix-matrix operations used in prefill and verification, it becomes an output-stationary systolic array, passing data through the grid while partial sums remain local.
The host also changes how many tiles execute each kernel and how work is partitioned. It can split an output dimension, a reduction dimension or the prompt dimension, then select matching network operations. That second layer of adaptation matters because adding tiles increases available compute but can also duplicate data movement and push a kernel back toward a memory-bandwidth limit.
The prototype used 14 accelerator tiles, four memory tiles, one RISC-V processor tile and one I/O tile on a proFPGA platform with a Xilinx Virtex UltraScale+ XCVU19P device. The researchers evaluated draft-target pairs from the Pythia, SmolLM2 and GPT-2 families with 32-token prompts. Their main flexibility comparison covered the first eight speculative rounds at a speculation length of eight.
Across those pairs, selecting vector-lane or systolic execution per kernel delivered speedups of 1.25 to 2.09 times over the better of two fixed alternatives. Allowing mixed sharding across phases then improved on the best single sharding family by 1.14 times for Pythia, 1.25 times for SmolLM2 and 1.05 times for GPT-2. The smaller GPT-2 gain shows that a fixed reduction-dimension split was already close to the best mapping for that workload.
Comparison with Jetson GPUs
For one GPT-2-to-GPT-2-Large workload with a 128-token prompt, the FPGA produced 0.69 tokens per second at 100 MHz. The paper reports 0.19 tokens per second for both a Jetson TX2 and Jetson Orin NX, and power efficiency of 0.035 tokens per second per watt for SPECTRA versus 0.012 and 0.008 for the two Jetsons.
Those GPU figures were derived from per-token latencies reported in earlier work, while SPECTRA was measured on the authors' prototype. The comparison therefore does not replace a same-code, same-software-stack benchmark. The FPGA design also pays a hardware cost for flexibility: against its systolic-only version, it used 82% more lookup tables and 79% more flip-flops, although block RAM rose 7.7% and DSP use 10.4%.
Sources
- Paper record and abstract, submitted September 21, 2026
- Full paper with architecture, FPGA setup and evaluation
- Lead image: Figure 2 from the paper, “Overview of the SPECTRA architecture,” published under the paper's CC BY 4.0 license.