Company

Resources

Company

Resources

Technology

Porting a Ternary STT Model to the Synaptics Torq NPU

Porting a Ternary STT Model to the Synaptics Torq NPU

How ENERZAi ran a 250M-parameter 1.58-bit ternary STT model in real time on the Synaptics Torq NPU by keeping 2-bit packed weights intact through the compiler.

Jaeho Kim

October 27, 2026

Hello from ENERZAi! In our previous post, Journey to Torq Compiler (https://medium.com/@enerzai/optimizing-the-mlir-based-torq-npu-compiler-f5ea6628c95d), we looked at the compiler architecture for the Synaptics Torq NPU and at where, and how, we reduced the hidden data-movement cost that comes with splitting an operation across two SLICEs for parallel execution.

This post picks up from there. It covers how we took an extremely quantized (ternary) model, a topic we have been researching for years, all the way onto a real NPU using the same Torq compiler. The key was keeping 2-bit packed weights from being unpacked too early during compilation, so that they stay in that form until just before computation and the storage and data-transfer savings from quantization carry through to execution.

To do this, we connected the packed-weight path already built into the hardware and the compiler’s block quantization path all the way through to a real model. As a result, we were able to run a speech-to-text model with about 250 million parameters and 1.58-bit (ternary) quantization in real time on the Torq NPU in the SL2619, part of the Synaptics Astra SL2610 product line.

These two paths did not connect to a real model seamlessly from the start, though. The Torq hardware and compiler already had mechanisms to keep 2-bit weights from being unpacked, but when we applied them to a real model, there were points where those mechanisms did not carry all the way through. In this post, we first look at those mechanisms, then walk through how we closed the gaps so that 2-bit weights make it through the entire compiler. We finish with measurements in a real on-device environment and the optimization work that still remains.

We use the same terms and dialects introduced in the previous post. If XRAM, LRAM, DMA, SLICE, or linalg, torq_hl, and torq_hw are unfamiliar, reading that post first will help.

Extreme Quantization and the Gap in Compiler Support

Ternary quantization restricts weights to three values, {-1, 0, 1}, which dramatically reduces model size and weight transfer volume; it is often called 1.58-bit quantization. But a smaller model file does not automatically mean those benefits survive at runtime. If values smaller than a byte, such as 2-bit values, are unpacked into a larger data type too early in compilation, the savings in storage and data transfer are lost. The sub-byte form, meaning anything smaller than a byte, therefore had to survive IR transformations, memory allocation, DMA, and kernel generation, all the way until just before computation.

Fortunately, Torq already had a solid foundation for this. The hardware provides a path that can store and read sub-byte integer weights in compressed form, that is, packed. Packing here means storing several sub-byte values together inside a single byte. A 2-bit integer, for example, lets you fit four values in one byte.

The remaining challenge was making full use of this hardware capability across the entire compiler. Even after compressed weights went through multiple transformations, the compiler had to be able to recognize their original form and keep it intact until the actual kernel ran.

The Packed-Weight Path

The benefit of packing goes beyond reducing XRAM usage. Because weights can move from XRAM to LRAM in compressed form, without being unpacked first, the amount of data transferred by DMA also shrinks. In other words, you save both memory and bandwidth at the same time.

Since ternary weights use only three values, {-1, 0, 1}, they can be represented as signed 2-bit integers. On Torq hardware, these weights follow this path:

[1] Compiled model file : packed as 2-bit (4 values per byte)
[2] XRAM                : still 2-bit; ideally 1/8 the size of bf16
[3] DMA to LRAM         : packed bytes are transferred as is
[4] LRAM                : still 2-bit; not pre-expanded to bf16
[5] Read on W path      : expanded from int2 -> bf16 at this point
[6] ALU                 : bf16 weight x bf16 activation

The most important step is [5]. Weights stay in 2-bit form even inside LRAM, and are expanded to bf16 only at the moment the W path reads them for computation. If this path can be used, there is no need for a separate decompression buffer or a standalone dequantization kernel. Even after compilation, the weight operand remains a 2-bit integer type, and the hardware is configured to interpret it as a signed value and expand it to bf16 as it is read.

The effect shows up immediately in storage. In ENERZAi’s ternary STT model, about 198 million of the roughly 250 million parameters are ternary weights. For these weights alone, the theoretical sizes are:

  • bf16: about 396 MB

  • packed 2-bit: about 49.5 MB

These are theoretical sizes, calculated by multiplying the parameter count by the bit width of a single element. We will come back to the memory usage actually measured on the board later.

There is one important constraint, however. This capability is not available equally on every input path. In Torq’s current linalg.matmul lowering, the first operand (lhs) is connected to the W path and the second operand (rhs) to the D path, and sub-byte expansion is supported only on the W path. So to keep compressed weights intact until execution, those weights must take the W role. If they go into the D path instead, the compressed input either cannot be used as is or needs a separate expansion step. The same matrix multiplication, mathematically, may not be the same operation from the hardware's point of view.

A small test case that places the same 2-bit weight in the W role and in the D role makes the difference clear.

// Weight role: packed 2-bit values are passed directly to the matmul
op : "matmul-fast"
  W %arg2 : memref<64x64xi2, lram>
  CFG {weight_format = SI, alu_format = BF, ...}

// Data role: an explicit expansion kernel appears, and widened bf16 data is consumed.
op : "i2f"
  D %arg1 : memref<64x64xi8, lram>
op : "matmul-fast"
  D %arg1 : memref<128x64xbf16, lram

In this experiment, moving the weight into the D role prevented the packed path from surviving. The compiler produced an 8-bit integer input plus a separate i2f path, and the corresponding weight data transferred into LRAM grew to four times the size of the packed 2-bit case.

The Block Quantization Path

When weights are reduced to just three values, finer-grained scales are usually needed to preserve accuracy. Instead of applying a single scale to an entire tensor, block quantization groups a fixed number of weights into blocks and gives each block its own scale. In our model, not only the ternary weights but also the 4-bit weights in the final vocabulary projection, the LM head, are represented this way.

When we joined the project, the core of this path was already implemented in the Torq compiler. The central operation is torq_hl.expand_weights. It takes compressed integer weights and per-block scales, and turns them into a kernel that expands each value as it is read and immediately applies that block's scale.

One particularly interesting detail is how zero points are handled. Weight dequantization is conceptually (q - z) * s, but this does not require a standalone subtraction kernel. Because the zero point is constant, it can be algebraically folded together with the scale into a bias/scale-compatible representation that the ACT path already knows how to consume. The important point is not that the zero point itself is simply stored in some particular memory location, but that the dequantization expression is rewritten into a form the hardware can process efficiently.

The compiler also already knew how to bufferize and serialize packed sub-byte integers into the model file, so the basic path from a 2-bit constant in the file to a hardware kernel was already in place.

We were not starting from a blank slate. The remaining work appeared when we ran a real model through that path and started hitting assumptions that had been reasonable for the original implementation but were too narrow for our model.

The first case showed up in the 4-bit LM head. This layer uses both a scale and a floating-point zero point for each block, while the existing path had been written around integer zero points. We extended the lowering so that a value incorporating the scale could be folded into the bias input of expand_weights even when the zero point was floating point. Rather than redesigning the kernel, we widened the set of quantization parameters that the existing block-quantization path could represent.

The same LM head also introduced unsigned 4-bit weights. Signedness is not always preserved directly in the compiler type, so blindly following the existing signed-weight path could reinterpret more than half of the values as negative. We inspected whether the dequantization expression used unsigned conversion semantics, propagated that property down the pipeline, and configured the hardware to zero-extend those weights instead of sign-extending them.

Finally, we stopped assuming that scale and zero point would always appear at fixed operand positions. Different model exporters can produce equivalent dequantization expressions with different operand ordering. Matching by position can therefore fail silently: the model still compiles, but the optimization simply does not fire. We changed the matcher to infer the roles of scale and zero point from the structure and semantics of the computation instead.

None of these changes added a brand-new operation. They widened an existing block-quantization path so that it could accept a broader range of real model representations. Because Synaptics had already built the foundation, we only needed to extend the boundaries that our model exposed.

Tracking 2-Bit Weights All the Way Through

To make full use of the hardware path described above in a real ternary model, the compiler had to recognize that a weight was originally a packed integer constant, even after it went through multiple IR transformations. Similar problems kept appearing in different forms, so here we follow one representative case from the beginning.

From the compiler’s point of view, a ternary weight is not a special kind of object. It is simply an integer constant with a small bit width, and that information is already in the IR type. For example, the type tensor<128x64xi2> alone tells you that each element is 2 bits.

The existing Torq compiler decided whether a weight was packed only at the final, kernel-generation stage. At that stage, it checked the weight operand of a matrix multiplication, and if it could tell that the value came from a packed integer constant, it sent the weight to the W path.

The problem was that several transformations could sit between the constant and the matrix multiplication. By the time compilation reached kernel generation, the original integer constant could already be hidden behind those transformations. If that happened, the compiler fell back to the ordinary bf16 path. The model file could still be small while the runtime execution lost the packed-weight advantage entirely.

The first problem was a type conversion. The model kept the weight as a 2-bit constant, then converted it to bf16 immediately before the matrix multiplication.

%cst = arith.constant dense_resource<...> : tensor<128x64xi2>   // still 2-bit up to here
...
%w  = tensor.extract_slice %cst[%i, 0] [64, 64] [1, 1]
        : tensor<128x64xi2> to tensor<64x64xi2>                 // still i2 after slicing
%wf = linalg.generic ins(%w) { arith.sitofp %in : i2 to bf16 }  // expanded here
%we = tensor.expand_shape %wf ...
linalg.batch_matmul ins(%we, %act) ...                          // kernel generation sees only bf16

A single sitofp was the whole cause, but that was enough. The %we that kernel generation sees is already a bf16 tensor, and the original packed 2-bit constant is hidden behind the conversion. So we changed the analysis to trace back through shape-only operations and type conversions to find the original integer tensor, and to pass that integer tensor to the W path instead of the bf16-converted value. This defers expansion again until the moment the hardware actually reads the weights.

A ternary 768x768 matmul (50 tokens) with slicing enabled shows the difference clearly. Before the change, ① a separate SLICE program unpacked i8 weights to bf16 and stored them in XRAM (144 KB × 8), and ② each matmul tile read those bf16 weights back into LRAM (144 KB × 8). After the change, ③ the packed i2 weights go straight into the W path (72 KB × 2) and are expanded only when actually read. Total DMA traffic dropped from 3.5 MB to 372 KB, and estimated execution time fell from 1135.9 µs to 274.3 µs.

We thought that solved the problem, but the same issue reappeared after slicing.

When a matrix multiplication was split across two slices, there were cases where an already-sliced weight tile had to be split again. If the IR first expanded the coarse tile to bf16 and only then extracted the smaller slice needed by the matmul, the packed-origin tracking broke again.

// Shape produced by slicing: expand first, extract the finer slice afterward.
%wt = extract_slice %cst[o, 0] [T, K]      : tensor<MxK xi2>     // coarse packed tile
%dt = linalg.generic sitofp %wt            : tensor<TxK xbf16>   // expand the whole tile
%s  = extract_slice %dt[i, 0] [t, K]       : tensor<txK xbf16>   // finer tile used by matmul
linalg.batch_matmul ins(%s, %act) ...

// Rewritten shape: slice first, expand afterward.
%ws = extract_slice %cst[o+i, 0] [t, K]    : tensor<txK xi2>     // slice while still packed
%wd = linalg.generic sitofp %ws            : tensor<txK xbf16>
linalg.batch_matmul ins(%wd, %act) ...

In the first form, the existing recognition logic could not reach the original integer constant. It knew how to look through shape-only transformations, but not through the extract_slice that appeared after expansion. We rewrote the IR so that the packed constant was sliced first and expanded afterward. The mathematical computation is identical, but the new form preserves a trail the packed-weight analysis can follow.

If this second issue had only caused a performance regression, it would actually have been harder to notice. Instead, compilation stopped. The leftover expansion lowered into an independent dequantization kernel whose input was still a packed constant, and kernel generation eventually failed with Compressed iType not supported here.

Once both cases were handled, the result was visible directly in torq_hl: the weight operand of the matrix multiplication was still typed as 2-bit.

"torq_hl.matmul"(%act, %shift, %w, %acc)
: (tensor<1x64x64xbf16>, tensor<2xi32>, tensor<64x64xi2>, tensor<1x64x64xbf16>)
-> tensor<1x64x64xbf16

The second rewrite did more than eliminate a compiler error. By allowing the packed weight to survive slicing, it also restored the intended two-slice execution. In an A/B comparison of compiler builds on the same model, both encoder latency and per-token decoding latency improved by more than 10%, while the transcriptions used for validation remained identical.

Here is a before-and-after comparison for a ternary FFN matmul (3072x768, 50 tokens). With the old compiler, enabling slicing triggered the Compressed iType not supported here error, so the model could only run with slicing disabled, leaving ① SLICE 1 idle. After the change, ② both SLICEs can run the matmul in parallel, and ③ the weights stay in packed i2 form (96 KB × 6, 576 KB in total). The estimated execution time for this matmul fell from 1731.7 µs to 1055.8 µs.

The information had been present in the type from the beginning, and the hardware had already been capable of consuming it. The challenge was to keep recognizing what a value originally was after it had passed through several transformations. For sub-byte support, the hard part was not inventing new arithmetic — it was preserving an existing compact representation all the way through the pipeline.

That covers the compiler work most directly tied to ternary weights, but additional general compiler optimizations accumulated along the way as we pushed the model toward real-time execution. Simply reassigning temporary XRAM addresses according to actual buffer lifetimes reduced total DRAM usage by more than one third. Our earlier overview post goes into more detail on the data movement introduced by splitting operations across slices and the optimizations we used to reduce it.

Results on the Edge

After these changes, we had a ternary speech-to-text model running on the Torq NPU of the Astra SL2619 in the Synaptics SL2610 product line. Measurements as of August 18, 2026 were:

  • Memory: up to 225 MB (host 50 MB + NPU 175 MB)

  • RTF: 0.455 (min 0.423 / max 0.502)

  • WER: 4.58% (9 audio clips, 7 errors out of 153 words)

RTF (real-time factor) is the ratio of processing time to audio length; a value below 1 means audio is processed faster than it comes in. With an RTF comfortably below 1, real-time execution was possible, and we also verified an end-to-end demo using live microphone input. We then applied decoder input prefetching as a separate optimization, which cut per-token decoding time by about another 15%. RTF dropped to 0.417, and recognition results for the same input stayed unchanged.

What Is Still Left to Optimize

Getting the model onto the hardware and extracting all of the hardware’s performance are two different milestones. We reached real-time execution, but several major optimization levers are still available.

1. Data movement between programs

There are still regions where adjacent slice programs move the same data back and forth between XRAM and LRAM. Keeping values resident in LRAM for longer and eliminating unnecessary round trips remains one of the largest optimization opportunities.

2. Fusion for non-matmul operations.

Counting the compiled slice tasks shows that matrix multiplications account for only about 18% of them. The remaining 82% are things such as scale/bias operations, LayerNorm, softmax, and data movement. Decoder GEMV in particular has relatively little arithmetic, so fixed costs such as launching and waiting for programs and moving data account for a larger fraction of latency.

Fusing these operations in ways that match the NPU data path is therefore an important direction — but reducing task count is not the goal by itself. We also need to verify which kernel path the fusion selects and whether data movement and the actual critical path improve.

3. More robust tile/fuse/slice algorithms.

Tile sizes and how work is split across SLICEs still rely heavily on heuristics, so they may need retuning whenever the model changes. In the “Next Steps” section of our previous post(https://medium.com/@enerzai/optimizing-the-mlir-based-torq-npu-compiler-f5ea6628c95d), we described a direction for exploring these decisions outside the compiler across many combinations to find the best configuration.

Closing Thoughts

This Torq project was not about forcing a new feature into a compiler that was never designed with ternary models in mind. The hardware already had a sub-byte data path, and the compiler already had a foundation for handling block quantization. What we did was widen the boundaries that surfaced as we applied a real model, and connect the pieces so that the advantages of packed weights carried through every compiler stage to actual kernel execution.

There are still many problems to solve in fully preserving the benefits of extremely quantized models on real hardware. ENERZAi is working through them one by one as we build ultra-compact AI agents that deliver top performance in real on-device environments with minimal memory. If you’re looking to run compact AI models efficiently on your own NPU or devices, or want to learn more about ENERZAi’s technology and products, feel free to reach out anytime!

ENERZAi - Contact

Optimium

Optimium

Solutions

Resources

ENERZAi

Copyright ⓒ ENERZAi Inc. All Rights Reserved

Business number: 246-86-01405

Email: contact@enerzai.com

Call: +82 (2) 883 1231

Address: 06140 27, Teheran-ro 27-gil, Gangnam-gu, Seoul, Republic of Korea

Optimium

Optimium

Solutions

Resources

ENERZAi

Copyright ⓒ ENERZAi Inc. All Rights Reserved

Business number: 246-86-01405

Email: contact@enerzai.com

Call: +82 (2) 883 1231

Address: 06140 27, Teheran-ro 27-gil, Gangnam-gu, Seoul, Republic of Korea

Optimium

Optimium

Solutions

Resources

ENERZAi

Copyright ⓒ ENERZAi Inc. All Rights Reserved

Business number: 246-86-01405

Email: contact@enerzai.com

Call: +82 (2) 883 1231

Address: 06140 27, Teheran-ro 27-gil, Gangnam-gu, Seoul, Republic of Korea