The rapid proliferation of artificial intelligence and machine learning workloads has placed immense pressure on enterprise infrastructure teams to optimize the use of expensive graphics processing units (GPUs). In a typical cloud-native environment, administrators are forced into a difficult compromise: deploying separate Kubernetes clusters for every team—which leads to massive overhead due to redundant control planes—or consolidating workloads into a single, shared cluster, which inherently lacks the strict security and performance isolation required for multi-tenant environments. Furthermore, Kubernetes’ native scheduling treats GPUs as monolithic, indivisible assets. A workload requiring only a fraction of an 80 GB A100 GPU currently consumes the entire card, leaving the remaining capacity stranded and inaccessible to other processes.
A new architectural approach, demonstrated by CNCF Golden Kubestronaut Anish Bista, offers a compelling solution by integrating two specialized open-source projects: Kamaji and HAMi. By decoupling the control plane from the worker nodes and implementing virtualized GPU scheduling, this framework enables organizations to provision isolated, GPU-enabled Kubernetes environments with high efficiency and lower infrastructure costs.
The Architectural Framework: Kamaji and HAMi
The proposed architecture addresses the two primary bottlenecks of modern GPU infrastructure: control plane management and hardware utilization. Kamaji, a project focused on "Cluster-as-a-Service," allows administrators to run multiple tenant Kubernetes clusters on a single management cluster. Each tenant operates with its own API server and etcd datastore, yet all tenants reside as lightweight pods within the primary management infrastructure. This eliminates the need for full-scale virtual machines or bare-metal servers for every control plane.
Complementing this, HAMi (Heterogeneous AI Computing Virtualization Middleware) provides the necessary abstraction layer to slice physical GPUs into smaller, usable chunks. HAMi intercepts pod scheduling requests and allocates specific memory and compute percentages, allowing multiple containers to share a single physical GPU card without interfering with one another. The integration of these two technologies creates a robust environment where tenants gain the illusion of a dedicated cluster, while the hardware is fully utilized at the granular level.
Proof-of-Concept: Simulated Hardware Validation
To prove the viability of this architecture without the prohibitive cost of physical GPU hardware, the test implementation utilized HAMi’s "mock device plugin." This plugin enables developers to define artificial GPU capacities—including memory limits and compute cores—via Kubernetes node annotations.
The experimental setup involved a management cluster hosting Kamaji, which in turn provisioned two distinct tenant clusters: "Tenant-A" and "Tenant-B." Each tenant was assigned its own worker node container. Through this setup, the research team successfully simulated high-end NVIDIA A100 hardware, demonstrating that Kubernetes could schedule fractional GPU workloads across these virtualized environments. The successful execution of eight discrete test cases confirmed that the communication path between the management-hosted API server and the HAMi webhook—facilitated by Kamaji’s Konnectivity tunnel—remained stable and functional.
Chronology of Implementation
The deployment process follows a structured five-step workflow designed to ensure complete separation between the management layer and the tenant layers:
- Management Cluster Initialization: The foundation is established using a kind (Kubernetes in Docker) cluster. Essential components, including cert-manager for TLS certificate management and MetalLB for load-balancing services, are deployed to handle external connectivity.
- Kamaji Deployment: Kamaji is installed via Helm, setting the stage for tenant control plane management.
- Tenant Provisioning: Using the Kamaji API, two tenant control planes (tenant-a and tenant-b) are instantiated. Each is assigned a specific Kubernetes version, with network policies enforced to ensure traffic isolation.
- Worker Node Integration: Standard kubeadm workflows are used to join worker nodes to their respective tenant clusters. These nodes act as the host for the HAMi scheduling logic.
- GPU Virtualization: HAMi is installed on the tenant clusters, and node annotations are applied to simulate the presence of NVIDIA GPUs. This step transforms the virtual nodes into GPU-aware scheduling targets.
Supporting Data and Performance Metrics
The test results provided clear evidence of resource enforcement. In one test case (TC-03 and TC-05), three separate pods were successfully scheduled on a single "mock" card, each consuming 20,000 MB of memory and 30% of the GPU compute cores. This validated the core premise of fractional GPU sharing.
Conversely, the system demonstrated strict adherence to resource limits during negative testing. When a fourth pod was requested, the HAMi scheduler correctly identified the lack of available compute cores and blocked the deployment, returning a "CardInsufficientCore" error. Similarly, attempts to allocate memory exceeding the 81,920 MB capacity of the simulated A100 resulted in immediate scheduling failure. These tests confirm that the system successfully enforces "hard" limits, preventing "noisy neighbor" scenarios where one tenant could starve another of resources.
Implications for Enterprise IT
The implications of this integrated approach are significant for organizations currently struggling with GPU scarcity and rising cloud costs. By moving away from the "one-cluster-per-team" model, companies can significantly reduce the CPU and RAM overhead associated with running multiple etcd and API server instances.
Furthermore, the ability to pack multiple workloads onto a single GPU increases the "GPU-utilization density" of the data center. For research departments or AI/ML teams, this means more researchers can have access to isolated, GPU-enabled environments without requiring additional hardware procurement.
Potential Limitations and Future Considerations
While the architecture provides a compelling path forward, it is important to distinguish between scheduling isolation and runtime performance isolation. As noted in the study, this setup verifies that the scheduling logic and network routing function as intended; however, it does not inherently guarantee that one process cannot attempt to access the memory space of another at the hardware driver level.
Additionally, while the use of the mock device plugin is highly effective for development and CI/CD testing, production environments would require the actual NVIDIA device drivers and compatible HAMi installations. The "lock contention" issues identified during the experiment—where manual annotations were required to release locks on the mock hardware—are artifacts of the simulation and are not expected to persist in production environments utilizing native hardware drivers.
Conclusion
The combination of Kamaji and HAMi represents a mature evolution of Kubernetes multi-tenancy. By addressing the control plane overhead through Kamaji and the hardware utilization challenge through HAMi, the industry now has a viable template for creating cost-effective, secure, and highly efficient AI computing platforms. As Kubernetes continues to serve as the backbone of modern machine learning pipelines, such modular, open-source integrations will likely become the standard for infrastructure architects seeking to maximize both security and hardware ROI. The successful "first-try" integration of the Konnectivity tunnel with the HAMi webhook serves as a strong indicator that this architecture is not only theoretically sound but also practically implementable in complex enterprise settings.
