NVIDIA CUDA

CUDA is the software layer that lets programmers use an NVIDIA graphics processor for things other than graphics. You write functions called kernels, usually in C++ or Python, that run across thousands of GPU threads at once, which suits workloads built from large amounts of parallel math. Scientific simulation, video processing, and above all neural networks fit that description, so CUDA became the foundation for modern AI training and inference.

Most people who use CUDA never write it directly. PyTorch, TensorFlow, and JAX call into NVIDIA’s libraries, including cuDNN for neural network operations, cuBLAS for linear algebra, and NCCL for communication between GPUs. TensorRT and related tools optimize models for deployment. Because these libraries have been tuned for years, a model that runs on CUDA is often faster out of the box than the same code on another vendor’s hardware. That software lead is a large part of why NVIDIA GPUs dominate AI data centers.

CUDA is free to download, but it only works on NVIDIA hardware, which is the main criticism. Code written in CUDA cannot run on AMD or Intel GPUs without porting, and many companies try to avoid that lock-in. AMD’s ROCm, OpenAI’s Triton, SYCL, and Apple’s Metal are the usual alternatives, and each covers only part of what CUDA does. For most developers the practical advice is simple: use a framework that handles CUDA for you, and only drop down to writing kernels when profiling shows you need the extra speed.