Scaling Distributed AI Infrastructure: Lessons from Atlassian on Integrating Network, Storage, and Compute for High-Performance Workloads

As artificial intelligence models evolve from localized prototypes to massive, multi-node distributed systems, the underlying infrastructure requirements have fundamentally shifted. For engineering organizations like Atlassian, simply provisioning GPUs and deploying Kubernetes clusters no longer constitutes an "AI-ready" environment. As training jobs scale to accommodate models with tens of billions of parameters, infrastructure bottlenecks—specifically in inter-node communication, shared storage, and topology-aware scheduling—have emerged as the primary inhibitors of progress. To maintain competitive velocity, platform teams are finding that performance must be treated as a component of correctness, necessitating a transition from treating infrastructure as a collection of disjointed layers to an integrated, unified platform.

The Evolution of the AI Infrastructure Challenge

For many organizations, the initial approach to AI infrastructure was additive: append GPU nodes to existing Kubernetes clusters and assume the software stack would handle the distribution. However, as Atlassian’s internal machine learning workloads for search and ranking grew, the limitations of this "add-on" model became apparent. When a model’s optimizer state and parameters exceed the memory capacity of a single machine, the system must pivot to distributed training. In this environment, the efficiency of the platform is dictated by the slowest component in the data path.

The challenge is multi-faceted. Training at scale requires high-bandwidth, low-latency communication between GPUs across multiple physical nodes. When these networks are misconfigured or fail to leverage specialized protocols like Remote Direct Memory Access (RDMA), the system silently defaults to standard TCP/IP sockets. This fallback often occurs without throwing an explicit error, leading to "silent degradation." GPU utilization drops, step times become erratic, and training throughput plummets, often remaining undetected by standard monitoring dashboards that track only whether a job "ran" rather than how efficiently it performed.

A Chronology of Platform Optimization

The journey to a high-performance training environment for Atlassian did not happen overnight. It was a measured shift that began with identifying a systemic lack of visibility into hardware performance.

In the early stages, distributed training jobs were running on infrastructure that lacked standardized, high-throughput storage, relying on less efficient paths for checkpointing and dataset access. The turning point occurred when the team discovered that a critical device plugin—responsible for advertising the RDMA fabric to Kubernetes—had been failing on production nodes for 271 days. The failure was masked by a mundane technical error: a container image mismatch that prevented the plugin from executing. Because the system lacked a synthetic validation mechanism to test the fabric, the failure remained hidden until an unrelated GPU operator upgrade triggered a deeper inspection.

Following this discovery, the team adopted a new design philosophy: treat platform performance as a first-class citizen. Between 2025 and 2026, the engineering team integrated RDMA-capable networking and Lustre-based shared storage into their core platform. They moved away from a "configuration-first" mindset toward a "topology-aware" architecture. This meant acknowledging that GPU reservations are not just logical allocations but physical ones tied to specific zones and subnets. By managing these transitions as deliberate platform upgrades—introducing new node pools, migrating workloads, and safely retiring the old paths—the team effectively insulated ML practitioners from the underlying complexity of hyperscaler infrastructure.

Empirical Results and Performance Gains

The shift from a socket-based communication path to an RDMA-capable, Lustre-backed infrastructure yielded significant, quantifiable improvements in training efficiency. Data collected from controlled A/B testing on H200 GPU nodes demonstrates the impact of these changes.

When comparing a standard TCP/IP fallback environment against the optimized RDMA architecture, the differences are stark. In a Qwen2.5-14B FSDP supervised fine-tuning scenario involving 16 H200 GPUs across two nodes, the median step time was slashed from 12.36 seconds to 6.07 seconds. Perhaps more tellingly, a 100-step fine-tuning benchmark saw the total duration drop from approximately 1,950 seconds to roughly 836 seconds.

Furthermore, the peak bus bandwidth reached 355 GB/s using an NCCL all-reduce operation, effectively doubling the training throughput compared to the previous, less-optimized setup. These metrics underscore the core argument put forward by Atlassian’s engineering leadership: speed is not merely a vanity metric; it is a prerequisite for reliable capacity planning and efficient GPU-hour utilization. When the infrastructure operates as an integrated whole, organizations gain the confidence to scale models to larger sizes without fear of hidden performance regressions.

Implications for the Cloud-Native Ecosystem

The broader implication for the CNCF (Cloud Native Computing Foundation) community is clear: AI training is pushing cloud-native platforms into a new frontier. While tools like Kubernetes and Kubeflow have matured to handle the orchestration of traditional microservices, they are now being tasked with managing massive, state-heavy, and network-sensitive workloads.

The industry is moving toward a model where "readiness" and "validation" are as important as "deployment." As Abhi Kulkarni and Shishir Jindal of Atlassian noted, platform teams must stop viewing networking, storage, and scheduling as separate silos. The modern platform must absorb the complexity of the fabric, ensuring that when an ML engineer submits a job, the infrastructure automatically validates that the required performance paths—such as RDMA or high-speed storage—are actually available and functioning as intended.

The Shift Toward Synthetic Validation

A critical takeaway from this operational shift is the necessity of synthetic validation. Dashboards and logs that only report on job status are insufficient for modern AI workloads. If a job runs but performs at 50% of its potential due to a silent network fallback, the system is technically "healthy" but practically failing.

Platform teams are now urged to implement telemetry that explicitly pins performance behavior to the transport path. By verifying the network path and storage throughput independently of the training job itself, engineers can catch misconfigurations—such as the aforementioned container manifest issues or topology mismatches—before they result in thousands of wasted GPU hours.

Conclusion: Designing for the Future of Distributed AI

The move toward integrated, high-performance infrastructure is no longer an optional optimization for companies operating at scale; it is a fundamental requirement for the viability of large-scale AI. By aligning physical topology with logical scheduling and wrapping it all in automated validation, organizations can transform their infrastructure from a source of friction into a catalyst for innovation.

As AI models continue to grow in complexity, the gap between "working" infrastructure and "performant" infrastructure will only widen. For those building the next generation of machine learning platforms, the lesson from Atlassian is definitive: treat networking, storage, and accelerator scheduling as a single, cohesive system. In the era of massive parameter counts, the platform that hides the complexity of the hardware is the platform that wins. As the industry looks toward 2027 and beyond, the focus will undoubtedly shift from merely acquiring more compute power to maximizing the efficiency and reliability of the distributed systems that power modern artificial intelligence.