The Speed of Thought: Navigate LLM Inference Autoscaling for a Gen AI Application Toward Production

Learn how to choose the autoscaling hyperparameters for your LLM applications by understanding the key metrics during inference. Gain essential tools to optimize latency and throughput by running and dissecting LLM inference benchmarks. See how NVIDIA's benchmarking software can be leveraged to make an informed decision about the Kubernetes deployment of your Gen AI application. You'll acquire best practices and tips to allow you to bring low latency at unmatched cost-effectiveness to your NIM applications for production.
Prerequisite(s):

Comfortable reading code in Python and Pandas.
Basic understanding of Kubernetes.
Basic understanding of LLMs.