YaFSDP: a sharded data parallelism framework, faster for pre-training LLMs

I was surprised to see that the Ya part meant “Yet another”. I mean, I’ve seen it before in many acronyms. But it’s pretty tongue in cheek of them to do that here since one would expect it was just because it was made by Yandex.

Any idea on what are the main tricks used to achieve gains over fsdp?