Together AI’s Together Compiler uses AI agents to beat megakernels
Together AI reports that its Together Compiler uses AI agents to rewrite and fuse GPU kernels in model forward passes, and that this approach outperforms its previous hand-written megakernels on B200.
- Together Compiler uses AI agents to rewrite and fuse GPU kernels in a model's forward pass.
- Together AI says Together Compiler beats its hand-written megakernels by 28% on B200 on Llama 3.2 1B.
- Together AI’s earlier megakernels were difficult to use in production and ended up supporting only one Llama 1B setup for one customer.
- Together AI found that using CUDA Graphs and Programmatic Dependent Launch on top of default Torch Inductor kernels could recover 98% of megakernel performance without writing new kernels.
- Together AI reports that AI agents need real data and full execution context to generate optimizations that work in production.
Who it affects This matters to teams working on GPU kernels and AI inference engines.
Source: together.ai ·