Fine tune a 70B language model at home

One thing I forgot to mention in the post which I think is kinda cool: at the NeurIPS Efficiency Challenge this year, where Tim Dettmers and I both did keynotes, every single top-ranked entry used QLoRA! The challenge was to create the most accurate model on a single GPU in 24 hours.<p>I think that is a great example of how important and useful QLoRA is. Maybe we should run a dual-GPU challenge next time not that multi-GPU is working...

I liked that you link to renting a dual 24GPU for 0.60cents&#x2F;hour, but how long could it takes to fine tune a 70b model using your system (4 bits for weights)?<p>If I were a consumer I would be interested in the final price of fine tuning, for example a table with model size, training size, cost of training, and expected loss of quality with this technology.<p>One obvious question: Can you apply your technology with the recent (-1,0,1) encoding?, I think you will answers that the (-1,0,1) model is not available and you can&#x27;t try it, but my question is whether once&#x2F;if that model is available answer.ai will be able to use the same technology that this post to fine tune a big model in two very small GPUs, and then I should ask for a new table with cost&#x2F;benefits analysis.<p>Edited: I should add that I find this kind of work very useful for enhancing individual users like me to be able to compete in the applications of LLM market, this is great work and along the lines of the book &quot;from zero to one&quot; (not that I like or dislike the author) to solve the kind of problem that nobody is trying to solve.<p>Edited: Now that I have a total of 23 points in HN, I will change my password to some random one, just to cure my desire to look for votes, and try to make some work, and again some tomorrow create a new presence in HN.

This is great, but one thing I really hoped would come sooner is fast training on Metal. As things are, you can get an M1&#x2F;M2 Ultra (~800 Gb&#x2F;s memory bandwidth; for comparison, RTX 4090 is ~1050 Gb&#x2F;s) Mac Studio with 128Gb RAM for ~$3500. For large model inference, this is already way more affordable than stacking GPUs while being &quot;fast enough&quot;, but training solutions are basically non-existent. I do wonder why; it feels like a low-hanging fruit.

This might be the most interesting constructive approach in &quot;Open Source&quot; LLMs I&#x27;ve seen. Grounded, reasonable and inviting to replicate! I wish academia took that as a standard.<p>Great job!

This is a fantastic breakthrough for those of us who fine-tune LLMs on limited hardware budgets.<p>I was curious about the choice of FSDP over DeepSpeed. I have been using Axolotl for fine-tuning, and FSDP has been broken there, whilst DeepSpeed is rock solid. Why FSDP over DeepSpeed jph00?

This is such exciting news! Huge thanks to you for your continued work in making sense of AI.<p>I wonder if the recent Bitnet 1.58 paper [the use of ternary bits in lieu of fp&#x2F;int] might be an advancement that could further reduce the computation required for inference?

Would be cool to build an “LLM@home” project like folding@home or SETI@home (rip), where tons of folks could donate their GPUs and train something huge and FOSS. I don’t know enough about how these models are trained though. Could it be chunked up and distributed in that way, then stitched&#x2F;merged back together?

If you are gonna be doing stuff like this I’m damn excited for answer.ai!<p>It’ll be the first time we’ll have someone who knows AI create leverage to open source it.<p>Way to go!

What’s the best way for people to contribute to AI open source? I can’t produce things like this for many reasons so how can I and others like me do our part to keep SOTA AI open?

Have you guys looked at using sparsification? It would probably require true re-training of the foundation model, to go at high sparse ratios (say 90% weights excluded), which could be done once on expensive GPU - but fine tuning such sparse models would require less RAM hopefully.<p>The trick with getting more benefit from sparse approach is to do block sparse (iirc, Tim Dettmers used to work on this as well, a few years ago), but large block size (say 16x16) would require much longer retraining to recover for the lost accuracy…

This is great, however there were many opportunities to use the word &#x27;nibble&#x27; in this post and they were all missed.

So, as I understand it, this is for finetuning a preexisting llm? So not actually training one from scratch. I guess that would be too much to ask for. Nonetheless, cheers to Jeremy and the gang for the work.

&gt; the ability to use multiple GPUs with QLoRA training.<p>Thorough article!<p>Question: What&#x27;s your opinion on:<p>- How viable will NVIDIA&#x27;s consumer cards be in the long run?<p>- Besides <a href="https:&#x2F;&#x2F;tinygrad.org" rel="nofollow">https:&#x2F;&#x2F;tinygrad.org</a>, what other cost-effective future alternatives could there be?

Does anyone have sources, or experience, about fine tuning primarily to teach the model some factual data, especially when it comes to later &quot;higher level&quot; question answering.<p>For example, giving the model a bunch of text (academic papers and such) about 19th century writers, then asking things like &quot;Who were the main influences on writer X&quot;?<p>Obviously simple RAG-like approaches don&#x27;t work, as such information is rarely available in the text as-is, and needs to be &quot;extrapolated&quot; to some extent. Long context models might work (just dumping everything into the prompt), but are way too expensive for my needs.

Nice, I tried to use QLoRA+FSDP in the past with litgpt and obviously at that time it did not work. This is very useful!

This is the best news I’ve seen all month. I think one of the great near-term dangers of AI is the bulk of the economic benefit going mainly to relatively few companies. That risk seems substantially reduced if they have to compete with a great variety of models.

Besides being a great result, the quality and clarity of the technical writing here is excellent.

Any plans on supporting AMD? In Germany, the price of an 7900XTX is HALF of a NV 4090...

Very interesting but hard to interpret until the performance numbers &#x2F; benchmarks are available. I can already fine-tune a 70B language model at home using CPU + RAM, but it would be so slow as to be almost totally impractical (~20x slower than GPU). It would be great to see a comparison to eg 8 x A100 (available for $32&#x2F;hr on AWS on-demand) and also CPU + RAM. Presumably it’s somewhere in between, but hard to predict where!

Maybe I&#x27;ve missed it in the article - but how long would a full training run take on 2 consumer GPUs (local or rented) ? Ballpark - hours, days... ?

This article is very well written and super informative. One thing I didn&#x27;t understand is:<p>&gt; At Answer.AI our north star is making useful AI more accessible. $150,000 to create your own high-quality personalized model definitely doesn’t count as accessible!<p>Renting an A100 on RunPod is ~$1.89 &#x2F; hour. So you&#x27;d need ~80,000 A100 hours to train a useful AI model?

So… why do people want to fine tune LLMs at home? It seems very unlikely to provide value.<p>* you’re probably not going to succeed at injecting new knowledge in a way that feels satisfyingly top of mind to the bot<p>* you’re probably not going to create such a meaningfully new style that it warrants a Lora like in images<p>What’s an example use case?

If I wanted to use this software to finetune a 70b model on two 3090s to write fiction, what is the maximum sequence length that would be practical? I&#x27;m at the dataset collection stage, but I&#x27;m not sure whether to aim for bigger or smaller sequence lengths at the moment.

I wonder whether LoRAs could be useful for U-Net training. Especially thinking of CNN-based U-Net models with pre-trained encoders (but randomly initialized decoders). At least, it seems possible that normal weight updates on the decoder and LoRA training on the encoder could improve efficiency.

Is there any framework&#x2F;system that distributes the work across multiple GPUs on different computers over a network (LAN or WAN)? I&#x27;m not concerned much about latency or generation time, but would love to train or load up huge models and send jobs to run overnight.

OK this is going to come out as moronic because I dont have the proper vocab to phrase it:<p>--<p>Is it possible to &#x27;array&#x27; tokenized wokloads across providers of GPU?<p>I want to farm-out my &#x27;compute&#x27; across [things]<p>More importantly can there be a marketplace for GPU resources that I can effectively point my local job at?

&gt; home<p>&gt; two 24GB GPUs.<p>geez

Nice, i&#x27;ve been hoping this would be possible for a while. I&#x27;ll have to do a new fine-tune of Inkbot on top of one of the 70b models.<p>What are the max context lengths &#x2F; batch sizes you can train at with this method for 2x24gb? What about 4x24gb?

This is great.<p>I don&#x27;t think local will be competitive in future IN GENERAL...but if I have a specific use case and I have a specific training dataset...local with specific training will murder the big commercial models.

Question - Can I use this to retrain an LLM (70B) weights on my own data? I am using RAG as of now for asking questions on my text, but always wonder if I could retrain an LLM on my own text. Thoughts?

Do the two 4090 GPUs need to be on the same machine, or is it possible to somehow use two separate machines, each with its own 4090, and link them somehow via e.g. InfiniBand?

Thank you for the repo and write up. What tools (if any) did you use for performance tuning once you achieved the main goal of being able to finetune the model?

If they can continuously train it, it could be better than a large context as this is how a AI OS would need to work when you have constant updates to your files

Congratulations, fantastic contribution to open source AI. Why does the website headline say &quot;train&quot; instead of &quot;finetune&quot;?

NVlink + two 3090 gives 48GB relatively easy (appears as unified memory). I only skimmed the article briefly, was it considered?

Imagine the potential of a Folding@Home-inspired project for AI development. What kind of powerful model could a community of gamers and GPU owners create.

Is it worth it though when the base model isn&#x27;t even smart enough?

Does this support multimodal language models (E.g.: LLaVA)?

Can&#x27;t believe they didn&#x27;t name this Qolor

This is brilliant. Thank you for doing his!

4x 3080???

Ah yes, 24gb top-of-the-line GPUs, I happen to have a whole warehouse full.<p>&#x2F;s

It would be great if we were a bit more respectful towards our natural resources. Using so much energy to play with language models is a waste of resources.