Google adds a C++ Tensor API to LiteRT for on-device models

Google released the LiteRT Tensor API, a light layer in its LiteRT stack. It is for developers who write on-device AI models in C++. The same graph operates on CPU, GPU and NPU (a chip for AI tasks), and developers do not write it again for each hardware target. Google reports that its sample model, SAM 2, operated at more than 30 frames each second on WebGPU, up from less than 10.

How it works

A developer assembles the graph in C++, one operation at a time. The graph can become a .tflite file for LiteRT, or operate straight on the CPU through XNNPACK with no model file. The input and output steps use the same API, and data stays on the accelerator (the chip that does the AI work) between steps.

The numbers

An agent wrote the SAM 2 model. It operates at more than 30 frames each second at 512x512 on WebGPU, up from less than 10. On a MacBook Pro, Google reports about 1.5 times the speed of Apple MLX. The sample has benchmark commands, and other groups can do the test again.

Not known

The 1.5 times number is for one model on one laptop, and Google made the test. Google tells developers to start with model conversion for most models. The text gives no result for NPU.

Insight: Google makes the whole on-device path

Google also added local models to its Antigravity SDK, and reports that 97.2% of the tokens in a test were local. Agents can use the Tensor API to make on-device graphs. The two moves read as a push to keep the on-device path in the Google stack. Builders who want local AI can test the API and check how much of their path stays with one vendor.