LoRA+: Efficient Low Rank Adaptation of Large Models

The other recent improvement suggested for LoRA is DoRA: <a href="https:&#x2F;&#x2F;magazine.sebastianraschka.com&#x2F;p&#x2F;lora-and-dora-from-scratch" rel="nofollow">https:&#x2F;&#x2F;magazine.sebastianraschka.com&#x2F;p&#x2F;lora-and-dora-from-s...</a>. It really does seem to strongly outperform LoRA - see also <a href="https:&#x2F;&#x2F;www.answer.ai&#x2F;posts&#x2F;2024-04-26-fsdp-qdora-llama3.html" rel="nofollow">https:&#x2F;&#x2F;www.answer.ai&#x2F;posts&#x2F;2024-04-26-fsdp-qdora-llama3.htm...</a>

I’m struggling to understand from this paper whether the approach is better in the general sense (all cases, with wider models seeing greater benefits) or purely for wider models (with narrower models seeing detriment)?<p>If it’s the former this could effectively halve finetuning cost overnight which would go a significant way towards enabling a wider array of use cases for LoRA.

I&#x27;ve had sucess with GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2403.03507" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2403.03507</a>

A better name would&#x27;ve probably been FastLoRA or something

What an unfortunate name... I initially thought this was about wireless communication. <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;LoRa" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;LoRa</a>

This is old, having been released in February... Why do you talk about it now?

In 2024 are folks still swapping out LoRA adapters? Is this still relevant?