The layer below the serving stack. Every model operation ultimately runs as GPU kernels, and kernel quality decides how much of the hardware you paid for actually gets used. You rarely write kernels yourself, but reading this layer explains why the systems above it behave the way they do.
GPUs run thousands of threads in lockstep groups (warps), organized into blocks and scheduled onto streaming multiprocessors. Threads in a warp share one instruction stream, so divergent branches serialize and waste lanes. Keeping enough warps resident per SM (occupancy) is what hides memory latency.
Registers, then small fast on-chip shared memory and caches, then large off-chip HBM. The bandwidth gap between on-chip and off-chip memory is orders of magnitude, so kernel performance is mostly a question of how rarely data crosses that boundary and how well accesses coalesce.
Dedicated matrix multiply-accumulate units deliver most of a modern GPU's FLOPS, at reduced precisions (FP16/BF16, FP8, INT8). Kernels that fail to feed tensor cores, or that use precisions the hardware lacks, leave most of the chip idle.
Each kernel is limited by arithmetic throughput or by memory bandwidth, whichever runs out first. LLM prefill is compute-heavy; decode is bandwidth-heavy. Knowing which side of the line an op sits on tells you whether more FLOPS or less data movement will help.
Kernel work is complementary to the systems level: continuous batching or prefix caching change what work is done; kernels change how fast each unit of work runs. See serving optimization for the former.
Standard attention materializes an N x N score matrix in HBM, so memory traffic grows quadratically with sequence length and the operation becomes bandwidth-bound. FlashAttention restructures the computation: it streams tiles of the query, key, and value matrices through on-chip memory, maintains the softmax normalization incrementally as tiles arrive, and never writes the full score matrix at all. Same mathematical result, a fraction of the memory traffic, which is what made long-context serving practical.
Successive versions have been re-tuned for each hardware generation (better work partitioning, asynchronous tensor-core pipelines, low-precision paths), and the technique now ships inside PyTorch and every major serving engine. The lesson generalizes: the biggest kernel wins come from restructuring computation around the memory hierarchy, not from micro-tuning instructions.
Ordered by rising control and rising cost. Start at the top; descend only when profiling proves the level above insufficient.
Decision inputs: how standard your operators are, how much peak performance is worth, which hardware you must support, and whether your team's strength is compilers or kernels. Profile first; the bottleneck is often not where intuition points.