Google Launches LiteRT Tensor API for On-Device ML Model Development

Google has released the LiteRT Tensor API, a new C++ interface that lets developers write a single machine learning computation graph and run it unmodified across CPU, GPU, and NPU hardware, according to InfoQ. In benchmark tests on a SAM 2 segmentation model, the company reported the API ran roughly 1.5 times faster than Apple's MLX framework on a MacBook Pro's GPU. The tool also lets AI coding agents write and optimize the C++ graphs directly, a capability Google demonstrated by having a coding agent build a real-world object tracking application.

What Happened

According to InfoQ, the traditional approach to on-device ML has forced developers to write computations manually in C++ for maximum control, but those implementations are tightly coupled to specific backends such as Metal, OpenCL, or Vulkan and must be rewritten for each new hardware target. Managing zero-copy buffer interoperability and synchronization across CPU, GPU, and NPU, while also handling pre-processing, inference, and post-processing, has demanded heavy manual engineering effort, InfoQ reported.

The LiteRT Tensor API addresses this by letting developers build computation graphs operation by operation in C++, with the same graph running through LiteRT on CPU, GPU, and NPU hardware without a rewrite for each target, InfoQ reported. Beyond full control over model graphs, the API's key features include direct weight ingestion from checkpoints, including Hugging Face.safetensors files; zero-copy buffer interoperability that avoids copying accelerator buffers between processing stages; and compatibility with AI coding agents that can construct, modify, validate, and benchmark model graphs by generating C++ code, according to InfoQ.

Google used a coding agent to build a demonstration app for multi-object segmentation and tracking, which also served as the basis for cross-platform benchmarking, InfoQ reported. Google ran the same C++ code for the SAM 2 model on GPUs across a macOS laptop (Metal), an iOS phone (Metal), and an Android phone (OpenCL), and reported that on the MacBook Pro it ran about 1.5 times faster than Apple MLX for the image encoder plus mask decoder. Google also said the coding agent moved SAM 2's per-frame visibility decisions from CPU code into the graph itself and rewrote operations that had triggered CPU fallback on WebGPU using GPU-compatible arithmetic, according to InfoQ.

Commenting on the launch on LinkedIn, as reported by InfoQ, Arm director of AI Laurence Moroney said developers should generally start with standard model conversion and only reach for the C++ API when they need graph-level control or want pre- and post-processing to stay on the accelerator. He noted that on WebGPU, frame rates rose from under 10 to over 30 frames per second at 512x512 resolution, with a coding agent handling the iteration, which he said shows how much structured primitives help agents perform real optimization work, InfoQ reported. Google also said it is working on direct ML Drift backend support, additional NPU-friendly operations, automatic full-graph quantization, and expanded dynamic-shape support for different image sizes, batch sizes, and sequence lengths.

Why It Matters

The most consequential detail in InfoQ's report is not the performance number but the workflow it enables: an AI coding agent writing, validating, and benchmarking low-level C++ code that directly manipulates GPU and NPU memory buffers. That is a meaningfully different trust boundary than an agent writing application logic in a sandboxed high-level language. Direct weight ingestion from Hugging Face checkpoints also means a model's numerical behavior on-device depends on a file pulled from an external repository, with no indication in InfoQ's reporting of signing, provenance checks, or validation tooling built into the pipeline. For enterprises building compliance-sensitive on-device applications, that combination, agent-authored low-level code plus externally sourced weights, raises questions about code review practices and audit trails that the current API description does not address. This pattern of developer tool adoption also has downstream market effects worth tracking through an equity implications of AI developer tool adoption lens, since cross-platform ML tooling shifts how quickly companies can ship inference products without backend-specific rewrites. Developers evaluating this stack should weigh the productivity gains InfoQ reported against the governance gaps the architecture does not yet solve.

What to Watch

Google has signaled several follow-on investments, including direct ML Drift backend support, additional NPU-friendly operations, automatic full-graph quantization, and expanded dynamic-shape support for varying image sizes, batch sizes, and sequence lengths, according to InfoQ. None of these announced items currently address code validation or weight provenance, so whether Google or third parties build auditing tooling around agent-generated C++ graphs will determine how quickly enterprises with compliance requirements adopt the API. The broader question is whether the performance gains InfoQ reported, including the jump from under 10 to over 30 frames per second on WebGPU, generalize beyond the SAM 2 demonstration to other model architectures developers are likely to deploy on-device.

Further reading and context

For broader context on the accelerator hardware this API targets, see MachineHours' comparison of Google TPU vs Nvidia GPU accelerators, and for the model field underpinning on-device deployment, see Large Language Models in 2026: The State of Play.

Further reading and context

• AWS vs Azure vs Google Cloud for AI Workloads in 2026
• Quantum Annealing vs Gate-Based Quantum Computing — Which Approach Wins?