When Collectives Collide: Preventing Congestion Collapse in Cross-Datacenter LLM Training

Training large language models now spans multiple datacenters, because no single site supplies enough power and cooling for the compute required by the new frontier models. Inter-DC collective communication over long-haul links introduces challenges in existing training frameworks due to the timescale mismatch between intra-DC traffic that is on the order of microseconds and inter-DC traffic on the order of milliseconds. Dropped packets belonging to inter-DC flows suffer long retransmission times due to the high link latency. Existing mechanisms that are used for intra-DC traffic are not immediately compatible with long-haul traffic. And new packet-deflection schemes only spread the congestion to neighboring switches and do not have sufficient buffering capacity for inter-DC flows. We present NetSponge, which, rather than dropping a packet under congestion, deflects it to a sponge node – an external buffer on spare exit-switch ports – that holds the packet and reinjects it once the congestion subsides. NetSponge requires no change to end hosts, intra-dc congestion control and is deployable on already available hardware.

M.Sc. student under the supervision of Prof. Mark Silberstein.