GPU-to-GPU traffic
Distributed training and collective communication across nodes using InfiniBand or an Ethernet/RoCE fabric.
PUNAIR / DESIGN & TECHNICAL CAPABILITIES
From the overhead tray
to the GPU fabric.
A connected plan for physical pathways, network architecture and deployment. Explore what goes into a modern AI data center—and the deliverables we can help prepare.
01 / PHYSICAL INFRASTRUCTURE
Good routing begins
before the first pull.
Turn floor plans and rack elevations into a coordinated pathway concept, with access, installation sequence and future growth considered together.
Tray layout & routing support
- Overhead pathways: basket tray and ladder-tray concepts, main routes, branches, intersections and rack entry points.
- Rack drops: waterfall locations, vertical management, service loops and access for installation and maintenance.
- A/B coordination: identify shared routes and failure points; define physical separation where the project requires it.
- Capacity planning: review bundle dimensions, tray fill, loading and growth allowances against manufacturer data and project criteria.
- Trade coordination: flag clashes with containment, lighting, sprinkler systems, busways and cooling infrastructure.
CONCEPT / TOP VIEW — NOT TO SCALE
Illustrative routing only. Actual separation, dimensions, support spacing, bend radii and loading follow the approved site design and manufacturer requirements.
Typical deliverablesMarked-up layoutsRoute & drop scheduleCable-length takeoffMaterial scheduleClash / dependency register
We can prepare conceptual layouts and coordination documents from supplied drawings. Construction release and structural, seismic, firestopping, electrical or mechanical design are reviewed by the responsible qualified professionals.
02 / LOGICAL & PHYSICAL NETWORK DESIGN
The right fabric.
At the right scale.
A topology is a capacity and resilience decision. Start with workload, node count, NIC bandwidth, port availability and the required failure domains.
CORE / SUPER-SPINE 01CORE / SUPER-SPINE 02
↙ ↓ ↘ ↙ ↓ ↘
SPINE 01SPINE 02
LEAF 01LEAF 02LEAF 03
↓ ↓ ↓
GPU NODESGPU NODESGPU NODES
Conceptual topology; one logical fabric shown. Real deployments may use multiple rails, planes or pods.
Leaf–spine: a two-tier Clos fabric.
Nodes attach to leaf switches. Each leaf connects to the spines in its fabric, creating multiple paths between leaves. Uplink capacity and oversubscription must match the workload; a leaf–spine layout alone does not guarantee a non-blocking network.
- Leaf: connects servers and presents their traffic to the fabric.
- Spine: interconnects leaves and carries traffic across the fabric.
- Core / super-spine: can interconnect pods in a larger three-tier Clos. An enterprise border/core may instead provide external connectivity; naming depends on the design.
- Rails & planes: map NICs, GPU affinity and independent network paths to the selected reference architecture.
Data in. Results out.
Dataset access, checkpointing and client traffic, on separate or converged networks according to the approved design.
Independent management
BMC, switch management and console access, with a defined security boundary and recovery path.
Architecture reference: NVIDIA NVL72 network architecture ↗
03 / CONFIGURATION & VALIDATION
Translate the design
into a controlled deployment.
Configuration support starts with the exact platform, software release and approved design. Templates are reviewed and tested before a production change.
Ethernet fabric
- IP addressing, loopbacks and interface descriptions
- BGP underlay and ECMP path planning
- VLANs, VRFs and EVPN/VXLAN where required
- MTU, breakout, speed and FEC compatibility
- RoCE QoS, ECN and PFC settings where specified by the validated design
InfiniBand fabric
- Port maps, link rates and cable compatibility
- Subnet manager and fabric management planning
- Partition / P_Key requirements
- Routing and rail alignment to vendor guidance
- Link counters, topology checks and escalation records
Operations & handover
- Management addressing and access controls
- NTP, logging, telemetry and monitoring plans
- Configuration backups and version records
- Method of procedure and rollback plan
- Acceptance checks and as-built port schedules
Project outputsDesign workbookConfiguration templatesChange procedureValidation checklistAs-built package
Exact scope and platform coverage are agreed before work begins. Workload-level acceptance—such as NCCL or application performance testing—is coordinated with the compute/software team. Link-up alone is not proof of application readiness.
Technical references: Cumulus Linux ↗ · RoCE configuration guidance ↗
04 / UNDERSTANDING THE COMPUTE
GPU. Node. Rack. Cluster.
Different layers. One system.
GPU
The accelerator doing parallel computation. GPU memory and internal interconnect bandwidth are different from external network link rates.
Node
A server or compute tray containing CPUs, GPUs, memory and network adapters. GPU count per node varies by platform.
Rack
A physical equipment assembly. Rack-scale systems may combine multiple trays into a larger GPU communication domain.
Cluster
Multiple nodes or racks connected through a scale-out fabric, with storage, management and software coordinating the workload.
Scale-up connectivity
Tightly couples accelerators within a system or rack-scale domain. NVIDIA NVLink is one example. Its cabling and topology follow the platform design.
Scale-out connectivity
Connects nodes and racks across a cluster using network adapters and InfiniBand or Ethernet switches. This is distinct from the internal GPU interconnect.
Rack-scale systems also depend on power, cooling, floor loading and service clearances. Network planning must coordinate with these disciplines, especially around liquid-cooled equipment and CDU access.
05 / TECHNOLOGY BRIEFING · REVIEWED OCTOBER 2026
Plan for the hardware.
Prepare for what follows.
A selection of current AI infrastructure platforms and technologies. These references describe the technology landscape, not PunAir vendor certifications, partnerships or completed installations.
NVIDIA GB300 NVL72
A liquid-cooled Blackwell Ultra system combining 72 GPUs and 36 Grace CPUs. Its network build must account for compute, storage and management connections.
Platform details ↗NVIDIA Vera Rubin NVL72
NVIDIA’s platform combines 72 Rubin GPUs, 36 Vera CPUs, NVLink 6, ConnectX-9 and BlueField-4. Confirm the selected OEM system’s validated hardware and software matrix.
Platform details ↗AMD Instinct MI350 series
MI355X uses AMD CDNA 4 and 288 GB of HBM3E per GPU. Network and rack requirements depend on the chosen server and system configuration.
MI355X specifications ↗800G InfiniBand & Ethernet
Quantum-X800 provides 800 Gb/s InfiniBand connectivity. Spectrum-X serves AI Ethernet fabrics. Match switch, NIC, breakout, optics, firmware and cable combinations to validated designs.
Quantum-X800 stack ↗Spectrum-X stack ↗Silicon photonics & CPO
Co-packaged optics integrates optical technology closer to the switch silicon. Spectrum-X Photonics introduces high-density optical fabrics with different fiber-management and service considerations.
Technology overview ↗800G to 1.6T roadmaps
Higher-rate networking changes density, thermal and optical requirements. A port rate, adapter aggregate rate and bidirectional total are different metrics—verify the exact SKU and operating mode.
Spectrum-X platform ↗Product names belong to their respective owners. Availability, port modes and software support vary by platform, region and release. Use vendor documentation for procurement and final design.
BRING THE PIECES TOGETHER
A clear plan.
A traceable build.
Start with your floor plan, rack elevations, device list, cable schedules and target milestones. We’ll define the planning, installation and configuration support your project needs.
Build your project brief ↗PunAirnetworks@gmail.com ↗