Caring for AI Infrastructure: Lessons from Huawei's Pangu Optimization

When AI Systems Need a Checkup

Think of a large language model as a patient in intensive care. The model itself might be healthy, but the surrounding infrastructure—the networks, the memory, the communication pathways—can quietly develop chronic conditions. Ignore them, and performance degrades, response times balloon, and your 'intelligent' system starts feeling like a sluggish bureaucracy.

That's the mindset behind a recent talk by Li Baichao, a systems engineering expert at Huawei's 2012 Lab. He walked through how the Pangu model team diagnosed and treated communication bottlenecks on Huawei's Ascend hardware. The specifics are technical, but the underlying philosophy is simple: keep an eye on the system, catch problems early, and fix them before they become showstoppers.

In the world of AI operations, we often obsess over model architecture or training data. But as models grow, the plumbing matters just as much. Communication overhead can eat up a third of your time, and in long-context scenarios, even moving data from host to device can become the new bottleneck. That's where the real care work begins.

The AllToAll Bottleneck: A Chronic Ailment

For mixture-of-experts (MoE) models, the AllToAll communication pattern is a persistent pain point. Li's team found that it accounts for over 30% of end-to-end time in some cases. That's like a hospital where the elevators are broken—everything else runs fine, but patients can't get to the right floor.

Traditional optimization tricks, like DeepEP, work on some platforms but fail on others. When they tried applying generic solutions to Huawei's Ascend 910A3, performance actually got worse. That's a familiar story for anyone who's tried to transplant a solution from one environment to another.

The lesson? You have to understand the specific hardware you're caring for. Huawei's Ascend 950 has a dedicated communication accelerator called CCU, and its network topology is different from what other libraries assume. By tailoring the AllToAll implementation to these specifics, the team achieved a 10% performance boost on Pangu's expert-parallel communication domain. It's not glamorous, but it's the kind of steady improvement that keeps systems healthy.

The New Kid on the Block: KV Cache Transfer

Long-context inference—think 1 million tokens—introduces a fresh challenge: moving the KV cache from host memory to the device. This Host-to-Device (H2D) transfer can become the dominant factor in time-to-first-token (TTFT). If you've ever waited forever for a response from a 'smart' assistant, this is often why.

Li's team tackled this with a two-pronged approach. On the hardware side, the Ascend 950 provides a dedicated H2D path for each NPU, reducing contention. On the software side, they built something called Omni Cache, which efficiently offloads and reloads the KV cache. The result? A 10% improvement in TTFT. That might not sound like much, but in real-time applications, every millisecond counts.

Hardware Affinity: The Art of Knowing Your Machine

One of the most striking points in Li's talk was the trade-off between generality and performance. The optimizations they developed are deeply tied to the Ascend 950's architecture. Apply the same tricks to other platforms—like the Ascend 910A2/A3 or NVIDIA H20—and they might fail or even degrade performance.

This is a crucial insight for anyone managing AI infrastructure. It's tempting to chase universal solutions, but the reality is that caring for a system means respecting its unique characteristics. Just as a doctor wouldn't prescribe the same medication to every patient, you can't expect one communication strategy to work everywhere.

For the Pangu team, this meant writing custom communication operators that exploit the CCU's capabilities and the specific topology of the Ascend 950. It's more work, but the payoff is tangible: faster training, faster inference, and a healthier overall system.

Practical Steps for Keeping Your AI System Fit

So what can you take away from Huawei's experience? Here are a few practical tips:

• Profile your communication patterns. If you're using MoE models, measure how much time AllToAll takes. If you're doing long-context inference, check the H2D transfer times.
• Know your hardware's quirks. Read the documentation, talk to the vendor, understand what accelerators or dedicated paths exist. Generic libraries might not be enough.
• Be willing to customize. Sometimes you need to write your own operators or adjust your parallel strategy to fit the hardware. It's an investment, but it can pay off.
• Think about hiding communication. The goal is to overlap communication with computation so that it doesn't sit exposed in your end-to-end latency. Look for ways to fuse operators or use multi-stream parallel pipelines.

The Broader Care Landscape

Huawei's talk was part of the AICon conference in Shenzhen, which covered everything from agent engineering to embodied AI. But the theme of caring for AI systems runs through all of it. As models become more complex and deployed in critical applications, the people who maintain them are becoming the unsung heroes.

It's not just about optimizing for speed; it's about ensuring reliability. When an AI system fails in a hospital, a self-driving car, or a customer service bot, the consequences are real. That's why understanding the communication infrastructure is a form of care—it's preventive medicine for the digital age.

Looking Ahead: The Future of AI Infrastructure Care

Li's team is already thinking about the next steps. They want to minimize unmasked communication time even further, and they're exploring fused operators and multi-stream parallel pipelines that are tuned to Ascend's hardware. The goal is to make communication disappear into the background, so that the model's potential is fully realized.

But there's a broader lesson: as AI evolves, so must our care practices. What works today might not work tomorrow. Keeping an eye on emerging bottlenecks, staying curious about new hardware, and being willing to adapt are essential skills for anyone responsible for AI systems.

In the end, caring for AI is like caring for any living system—it requires attention, patience, and a willingness to get your hands dirty. The next time your model feels slow, don't just blame the algorithm. Look at the communication pathways. That's where the real health of your system lies.