Implementing Weight-Decomposed Low-Rank Adaptation (DoRA) from Scratch

Speaking of LoRA, what happened with ZipLoRA? It&#x27;s supposed to be a better way of merging multiple LoRAs, and the results look good in their examples. Is it being used anywhere?<p><a href="https:&#x2F;&#x2F;ziplora.github.io&#x2F;" rel="nofollow">https:&#x2F;&#x2F;ziplora.github.io&#x2F;</a>

This is very cool and will change the way we do lora now.

This is a bit of a misleading title. Why not use the original?<p>&quot;Improving LoRA: Implementing Weight-Decomposed Low-Rank Adaptation (DoRA) from Scratch&quot;<p>(If it&#x27;s too long, just drop the &quot;Improving LoRA: &quot; part)

Hooray, no more confusion with LoRa the radio!