AI Infrastructure Technologies Accelerating Business Adoption of LLMs

The use of generative AI is expanding from single calls to a large language model (LLM) toward "AI agents" that work autonomously by decomposing tasks, consulting external tools and knowledge, and verifying and iterating on their results. The performance and cost of an AI agent depend not only on the capability of the LLM itself, but also on the design of everything around it, such as prompts, the choice of model at each step, memory, and the inference infrastructure.

To accelerate the business adoption of LLMs, including NTT's own LLM "tsuzumi," we evaluate and validate the platform technologies that underpin AI agents. Here we introduce two of these initiatives: (1) AI agent optimization and (2) KV-cache-centric inference infrastructure.

Initiative 1: AI agent optimization

AI agents involve many design elements, including prompts, models, memory, and retrieval settings, and tuning them tends to rely on developers' experience and manual effort. We systematically evaluate open-source technologies such as automatic prompt optimization, step-wise model assignment search, and reinforcement learning, and are establishing a methodology that places small language models (SLMs) where they fit best to achieve accuracy comparable to large models at lower cost and latency. In our search for assigning models of different sizes to each step of multi-step reasoning, we confirmed that mixed assignments improve the performance-cost trade-off compared with single-model configurations.

ai-infrastructure_fig01.jpg

Figure 1: AI agent optimization technologies

Initiative 2: KV-cache-centric inference infrastructure

In LLM inference, storing and reusing the "KV cache," the intermediate data produced while the input is processed (the prefill stage), reduces GPU computation and improves time to first token (TTFT:Time To First Token) and throughput. AI agents, which run many inferences while repeatedly referring to the same context, are workloads where the KV cache is especially effective. Using open-source inference engines and KV cache management software, we verify KV cache management and scheduling technologies for agent workloads on real hardware and accumulate know-how for practical deployment.

We are also researching a "distributed AI inference infrastructure" that shares and transfers KV caches across inference environments spread over multiple GPU servers and sites, so that they can be used as a single inference platform. By avoiding redundant prefill computation across servers and sites, and by smoothing inference demand that varies by location and time of day, we aim to improve both throughput and energy efficiency. The technology can also be applied between remote sites connected by high-bandwidth, low-latency networks such as the IOWN All-Photonics Network (APN).

ai-infrastructure_fig02.jpg

Figure 2: KV-cache-centric inference infrastructure

Going forward, we will apply these technologies to support the adoption of LLMs including tsuzumi, and establish them as know-how and best practices that can be rolled out across the NTT Group, contributing to high-performance, efficient AI infrastructure for the age of AI agents.

Glossary

  • KV cache: Key-Value Cache

Related information