At work, we have built something very similar, a custom topology aware scheduler for workloads, and the most interesting things about such a project are entirely absent from this article, making it more of a fluff/marketing piece than an actually interesting technical blog in this area.
Our next iteration of this scheduler aims to scale to O(millions) of GPUs, with a scheduling cycle time measured in the milliseconds.
/genuine do you recommend any other blog?
Seems very relevant to what you are building. Quoting what Ali said: "Quick PoC?"
Big-O doesn't include constant factors for a reason (never mind just being a constant entirely), but maybe I am misunderstanding what you are trying to say?
This means that if say a job requesting GPUs is submitted with a given topology constraint, we would already computed the physical nodes that satisfy the topology requirement, and begun preemption / begun spinning up processes on the target machines within a few milliseconds of the job being submitted. If there were no preemptions required (which means we need to first wait for graceful termination), the job is fully scheduled in the blink of an eye, whether it's on tens or hundreds of thousands of GPUs.
I like to call it Big-O Magnitude Notation.
"Our next iteration of this scheduler aims to scale to millions of GPUs"
When you consider the problem from first principles, understand how the CPU that runs the program actually processes the data, are aware of things like cache locality, vectorization, etc... and have good placement, preemption, bin packing, and defragmentation algorithms, you learn that in the limit, milliseconds is all you need.
When you are responsible for the scheduling of ~millions of GPUs, every second starts to add up, and you realize that doing this correctly very quickly pays for itself.
Consider a large training run that takes 200k GPUs, if it takes 5 minutes to schedule that job, create pods, etc, that's ~16.6k GPU hours wasted. At a market rate of $11/gb300 GPU hour, that's ~$185k of wasted compute in scheduling overhead.
i suspect you also have some great 60Hz cluster observability software ;)
its good for the world if you do it
A coworker once said of a particular breed of manager who is an emphatic late adopter: it’s as if every time he goes to the dentist he reads an old trade journal in the waiting room and comes back to work with a “new” idea that everyone else is already busy replacing.
Are there any actual newer techniques? For instance, putting 11% of your hardware on reserve for hardware failures seems a bit generous to me for such an expensive cluster. Even if it’s down from some much larger number. One solution to this I’ve seen for handling cpu and memory variability with bin packing is to pool the spare capacity between tasks. For instance a data center with four zones might have 7 backup generators, instead of eight or twelve, counting on clusters of generator failures not to happen frequently.
And of course there’s always running low priority tasks on the spare hardware and evicting them on a fault. However those low priority tasks tend to develop their own shadow priorities when they are in house. But that’s a management problem not a technical one.
"No Topology-Aware Scheduling (TAS) support."
There is another blog talking about this coming soon.