There are many options and opinions about, what is currently the recommended approach for running an LLM locally (e.g., on my 3090 24Gb)? Are options ‘idiot proof’ yet?
Given you're running a 3090 24gb, go with Oobabooga/Sillytavern, and don't come here for advice on this stuff, go to <a href="https://www.reddit.com/r/LocalLLaMA/" rel="nofollow">https://www.reddit.com/r/LocalLLaMA/</a>, they usually have a "best current local model" thread pinned, and you're less likely to get incorrect advice.
Ollama is really easy.<p><i>brew install ollama</i><p><i>brew services start ollama</i><p><i>ollama pull mistral</i><p>Ollama you can query via http. It provides a consistent interface for prompting, regardless of model.<p><a href="https://github.com/ollama/ollama/blob/main/docs/api.md#request">https://github.com/ollama/ollama/blob/main/docs/api.md#reque...</a>
We run coding assistance models on MacBook Pros locally, so here is my experience:
On hardware side I recommend Apple M1 / M2 / M3 with at least 400Gb/s memory bandwidth. For local coding assistance this is perfect for 7B or 33B models.<p>We also run a Mac Studio with a bigger model (70b), M2 ultra and 192GB ram, as a chat server. It's pretty fast. Here we use Open WebUI as interface.<p>Software wise Ollama is OK as most IDE plugins can work with it now. I personally don't like the go code they have. Also some key features are missing from it that I would need and those are just never getting done, even as multiple people submitted PRs for some.<p>LM Studio is better overall, both as server or as chat interface.<p>I can also recommend CodeGPT plugin for JetBrains products and Continue plugin for VSCode.<p>As a chat server UI as I mentioned Open WebUI works great, I use it with together ai too as backend.
LMStudio has an easy interface, you can browse/search for models, it has a server if you need an API, etc.<p>it's pretty 'idiot proof', if you ask me.<p><a href="https://lmstudio.ai" rel="nofollow">https://lmstudio.ai</a>
Newbie questions:<p>What do you do with one of these?<p>Does it generate images? Write code? Can you ask it generic questions?<p>Do you have to 'train' it?<p>Do you need a large amount of storage to hold the data to train the model on?
Gpt4all and Simon Willison's llm python tool are a nice way to get started; even on a modest laptop. A modest 14" mac book pro with 16GB goes a long way with most of the 7B models. Anything larger you need more ram.
If you want a code centric interface it's pretty easy to use llangchain with local models. For example, you specify a model from hugging face and it will download and run it locally.<p><a href="https://python.langchain.com/docs/get_started/introduction" rel="nofollow">https://python.langchain.com/docs/get_started/introduction</a><p>I like llangchain but it can get complex for use cases beyond a simple "give the llm a string, get a string back". I've found myself spending more time in llangchain docs than working on my actual idea/problem. However, it's still a very good framework and they've done an amazing job IMO.<p>edit: "Are options ‘idiot proof’ yet?" - from my limited experience, Ollama is about as straightforward as it gets.
FWIW I use Ollama and Open WebUI. Ollama uses one of my RTX A6000 GPUs. I also use Ollama on my M3 MacBook Pro. Open WebUI even supports multimodal models.
I use LM Studio[0] on my M-series macs and it's pretty plug and play.<p>I've got an Ollama instance running on a VPS providing a backend for a discord bot.<p>[0] <a href="https://lmstudio.ai" rel="nofollow">https://lmstudio.ai</a>
I still feel that llamafiles[1] are the easiest way to do this, on most architectures. It's basically just running a binary with a few command-line options, seems pretty close to what you describe you want.<p>[1] <a href="https://github.com/Mozilla-Ocho/llamafile">https://github.com/Mozilla-Ocho/llamafile</a>
Jumping on this as a fellow idiot to ask for suggestions on having a local LLM generate a summary from a transcript with multiple speakers. How important is it that the transcript is well formatted (diarization, etc) first. My attempts have failed miserably thus far.<p>Edit: using a P40, whisper as ASR
I created a tool called Together Gift It because my family was sending an insane amount of gift related group texts during the holidays. My favorite one was when someone included the gift recipient in the group text about what gift we were getting for the person.<p>Together Gift It solves the problem the way you’d think: with AI. Just kidding. It solves the problem by keeping everything in one place. No more group texts. There are wish lists and everything you’d want around that type of thing. There is also AI.<p><a href="https://www.togethergiftit.com/" rel="nofollow">https://www.togethergiftit.com/</a>
Ollama is the easiest. For coding, use VSCode Continue extension and point it to ollama server.<p>The thing to watch out for (if you have exposable income) is new RTX 5090. Rumors are floating they are going to have 48gb of ram per card. But if not, the ram bandwidth is going to be a lot faster. People who are on 4090 or 3090s doing ML are going to go to those, so you can pick up a second 3090 for cheap at which point you can load higher parameter models, however you will have to learn hugging face Accelerate library to support multi gpu inference (not hard, just some reading trial/error).
ollama is the easiest, IMO. The CLI interface is pretty good too.<p>gpt4all is decent as well, and also provides a way to retrieve information from local documents.
Hugging Face Transformers is your best bet. It's pretty straightforward and has solid docs, but you'll need to get your hands dirty a bit with setup and configs.<p>For squeezing every bit of performance out of your GPU, check out ONNX or TensorRT. They're not exactly plug-and-play, but they're getting easier to use.<p>And yeah, Docker can make life a bit easier by handling most of the setup mess for you. Just pull a container and you're more or less good to go.<p>It's not quite "idiot-proof" yet, but it's getting there. Just be ready to troubleshoot and tinker a bit.
The current HYPERN // MODERN // AI builds are using flox 1.0.2 to install llama.cpp. The default local model is dolphin-8x7b at ‘Q4_KM’ quantization (it lacks defaults for Linux/NXIDIA, that’s coming soon and it works, you just have to configure it manually and Mac gets more love because that’s what myself and the other main contributors have).<p>flox will also install properly accelerated torch/transformers/sentence-transfomers/diffusers/etc: they were kind enough to give me a preview of their soon-to-be-released SDXL environment suite (please don’t hold them to my “soon”, I just know it looks close to me). So you can do all the modern image stuff pretty much up to whatever is on HuggingFace.<p>I don’t have the time I need to be emphasizing this, but the last piece before I’m going to open source this is I’ve got a halfway decent sketch of a binary replacement/conplement for the OpenAI-compatible JSON/HTTP one everyone is using now.<p>I have incomplete bindings to whisper.cpp and llama.cpp for those modalities, and when it’s good enough I hope the bud.build people will accept it as a donation to the community managed ConnectRPC project suite.<p>We’re really close to a plausible shot at open standards on this before NVIDIA or someone totally locks down the protocol via the RT stuff.<p>edit: I almost forgot to mention. We have decent support for multi-vendor, mostly in practice courtesy of the excellent ‘gptel’, though both nvim and VSCode are planned for out-of-the-box support too.<p>The gap is opening up a bit again between the best closed and best open models.<p>This is speculation but I strongly believe the current opus API-accessible build is more than a point release, it’s a fundamental capability increase (though it has a weird BPE truncation issue that could just be a beta bug, but it could hint at something deeper.<p>It can produce verbatim artifacts from 100s of thousands of tokens ago and restart from any branch in the context, takes dramatically longer when it needs to go deep, and claims it’s accessing a sophisticated memory hierarchy. Personally I’ve never been slackjawed with amazement on anything in AI except my first night with SD and this thing.