28 September 2025
A TPU is Google’s AI processor that accelerates matrix/tensor math with systolic arrays, built for fast training and inference in Google Cloud.
A short definition of a TPU
A TPU (Tensor Processing Unit) is a domain-specific chip from Google designed to run the matrix and tensor operations at the heart of modern AI. Unlike general-purpose GPUs, TPUs use on-chip systolic arrays (a grid of multiply-accumulate units) that stream data through in lock-step, delivering very high throughput for deep-learning workloads.
How TPUs work:
Your model (typically via TensorFlow or JAX) is compiled by XLA into TPU programs. The TPU’s matrix units (often called MXUs) crunch large batched matmul/conv ops using formats like bfloat16/INT8 with FP32 accumulation, fed by high-bandwidth on-chip SRAM and HBM. Multiple TPUs link into a pod over a dedicated interconnect to scale training and inference.
When a TPU is a good fit
- Large-scale training or high-throughput inference on Google Cloud
- Models that XLA can fuse well (Transformers, CNNs, SSMs)
- Teams already using TensorFlow/JAX (PyTorch via PyTorch/XLA is also supported)
When it’s not ideal
- On-prem deployments (TPUs are primarily a cloud service)
- Heavy use of custom CUDA-only ops or kernels that XLA can’t optimize
- Very small, latency-sensitive batches better suited to CPUs/NPUs/GPUs at the edge
TPU versus GPU
- GPU: flexible, massively parallel generalist for many workloads and tool chains.
- TPU: purpose-built matrix engine that shines when XLA can turn your model into big, efficient matmuls, especially at cloud scale.
Buying TPU hardware
I am afraid not. TPUs are proprietary to Google - they don't sell the hardware, they sell access to it via their cloud AI offering. “Cloud TPU” is a managed service you rent by the hour (v5e, v5p, etc.) in specific regions; you request quota and spin up TPU VMs like any other cloud resource. They’re not sold as add-in cards for your own servers.
Is "TPU" just Google's brand name for a GPU?
Not quite. A TPU isn’t a “GPU with Google branding”, it’s a different kind of accelerator.
- GPU: a general-purpose parallel processor (born for graphics) with thousands of programmable cores; great at many workloads, including deep learning.
- TPU: Google’s domain-specific ASIC for deep learning. Its heart is a systolic array (huge matmul engine) that streams tensors through multiply-accumulate units at very high utilization.
Analogy: GPU = a versatile power tool set; TPU = a specialized industrial machine that does one operation (matmul) incredibly fast at scale.
