IBM's llm-d Framework Brings Large Language Models to Enterprises at Scale

The IBM Research-led llm-d project has made significant strides in addressing the challenges of serving large language models (LLMs) at scale. The open-source framework, developed in collaboration with Red Hat and Google, aims to help organizations deploy LLMs on their own infrastructure, reducing costs and improving efficiency.

Traditional chatbots are straightforward, processing a single text input at a time, but agentic systems like coding assistants require more complex inference infrastructure. They repeatedly process massive contexts, reuse information across interactions, and generate bursts of activity from parallel sub-agents, placing new demands on latency, memory, throughput, and cost.

The llm-d team demonstrated the effectiveness of their framework by deploying a 753-billion-parameter open-weight mixture-of-experts model (GLM-5.2) on 544 NVIDIA H100 GPUs. The system showed remarkable efficiency in serving agentic workloads, delivering over 6.6 million output tokens per minute at peak, with costs 5 to 10 times lower than equivalent commercial API pricing.

The team's research identified key characteristics of agentic workloads: extremely long contexts, heavy reuse of information, and bursts of parallel activity. To address these challenges, the llm-d framework employs six capabilities: prefix-aware routing, tiered KV-cache management, peer-to-peer cache sharing, wide expert parallelism, prefill/decode disaggregation, and multi-token prediction.

The deployment demonstrated that self-hosting LLMs can be a cost-effective solution for organizations. According to Carlos Costa, Distinguished Engineer at IBM Research, 'llm-d has already proven it can handle this at scale.' The team's work aims to provide a production-ready system for serving large language models.