A beginner's guide to torch.profiler from Hugging Face

Hugging Face published a beginner's guide to torch.profiler, the built-in PyTorch tool for measuring where time goes on CPU and GPU. The post is aimed at people who have seen the dense colored trace view and bounced off it. It walks through both the summary table and the Chrome-style timeline view with the question "what am I actually looking at" front and center.

The most useful illustration is the contrast between overhead-bound and compute-bound workloads. Running matmul plus bias on 64×64 tensors, CPU time is 2.314 ms while GPU time is only 23.104 µs, with GPU utilization under 1 percent. Move to 4096×4096 and the numbers shift to 4.908 ms on CPU and 4.495 ms on GPU, with utilization around 50 percent. The point is that small kernels are dominated by Python dispatch and CUDA launch overhead, and the cleanest fix is usually larger batches, not faster kernels.

The post also tracks the dispatch chain step by step from aten::matmul down to the actual ampere_bf16_s16816gemm CUDA kernel, shows how torch.compile fuses operators at the dispatcher level into aten::addmm, and notes that the compile pipeline roughly doubles CPU overhead for a single op because the per-call cost only amortizes over many ops.

Why it matters

If you are training or serving PyTorch models, this is the kind of post you bookmark before your next "the GPU looks idle" incident. It is also the page to send a teammate who has the symptoms but does not yet know how to read a trace.