A rigorous architectural analysis and benchmarking system for automated low-level performance code generation.
Summer 2026
Research Engineer
Systems AI
Prototyped
As Large Language Models demonstrate increasing proficiency in code generation, their application to highly specialized domains like GPU kernel programming (CUDA, Triton) presents significant opportunities and risks. While LLMs can output syntactically valid code, verifying the correctness, identifying race conditions, and measuring the true hardware efficiency (latency, throughput, occupancy) of these generated kernels remains an unsolved bottleneck in automated optimization pipelines.
The objective was to architect a comprehensive, reproducible benchmarking framework capable of autonomously evaluating AI-generated GPU kernels. This system needed to rapidly compile, test, and profile candidate kernels against established baselines (Triton standard library, CUTLASS, cuBLAS) to determine their viability for production deployment.
Engineered an end-to-end Python framework that orchestrates compilation, correctness verification via PyTorch assertions, and micro-benchmarking using NVIDIA Nsight Compute hooks.
Rigorous A/B testing against hand-tuned CUTLASS and cuBLAS implementations to quantify the performance delta of LLM-authored Triton and TileLang kernels.
Automated profiling of shared memory bank conflicts, warp divergence, and streaming multiprocessor occupancy to filter structurally flawed AI kernels.
Prototyped the ingestion of verified, high-performance LLM-generated kernel candidates directly into the ZFLOW optimization graph, establishing a feedback loop for continuous compiler improvement.
"The bottleneck in AI-driven code synthesis is no longer generation, but rigorous, automated verification at the hardware level."
The framework successfully demonstrated that while SOTA agentic systems can produce syntactically valid Triton code, roughly 70% of initial candidates fail strict latency and occupancy checks compared to cuBLAS baselines. By establishing this automated evaluation harness, we provide a quantifiable metric for LLM kernel quality, paving the way for reinforcement learning approaches to autonomous systems programming.