I've been gleefully exploring the intersection of LLMs and CLI utilities for a few months now - they are such a great fit for each other! The unix philosophy of piping things together is a perfect fit for how LLMs work.<p>I've mostly been exploring this with my <a href="https://llm.datasette.io/" rel="nofollow noreferrer">https://llm.datasette.io/</a> CLI tool, but I have a few other one-off tools as well: <a href="https://github.com/simonw/blip-caption">https://github.com/simonw/blip-caption</a> and <a href="https://github.com/simonw/ospeak">https://github.com/simonw/ospeak</a><p>I'm puzzled that more people aren't loudly exploring this space (LLM+CLI) - it's really fun.
Recent and related:<p><i>Llamafile – The easiest way to run LLMs locally on your Mac</i> - <a href="https://news.ycombinator.com/item?id=38522636">https://news.ycombinator.com/item?id=38522636</a> - Dec 2023 (17 comments)<p><i>Llamafile is the new best way to run a LLM on your own computer</i> - <a href="https://news.ycombinator.com/item?id=38489533">https://news.ycombinator.com/item?id=38489533</a> - Dec 2023 (47 comments)<p><i>Llamafile lets you distribute and run LLMs with a single file</i> - <a href="https://news.ycombinator.com/item?id=38464057">https://news.ycombinator.com/item?id=38464057</a> - Nov 2023 (287 comments)
I just tried this and ran into a few hiccups before I got it working (on a Windows desktop with a NVIDIA GeForce RTX 3080 Ti)<p>WSL outputs this error (hidden by the one-liner's map to dev/null)<p>> error: APE is running on WIN32 inside WSL. You need to run: sudo sh -c 'echo -1 > /proc/sys/fs/binfmt_misc/WSLInterop'<p>Then zsh hits: `zsh: exec format error: ./llava-v1.5-7b-q4-main.llamafile` so I had to run it in bash. (The title says bash, I know, but it seems weird that it wouldn't work in zsh)<p>It also reports a warning that GPU offloading is not supported, but it's probably a WSL thing (I don't do any GPU programming on my windows machine).
Currently, a quick search on Hugging Face shows a couple of TinyLlama (~1b) Llamafiles. Adding those to the 3 in the original 3 llamafiles, that's 6 total. Are there any other llamafiles in the wild?
Curious about what HN things about llamafile and modelfile (<a href="https://github.com/jmorganca/ollama/blob/main/docs/modelfile.md">https://github.com/jmorganca/ollama/blob/main/docs/modelfile...</a>)<p>Both invoke a Dockerfile like experience. Modelfile immediately seems like a Dockerfile, but llamafile looks harder to use. It is not immediately clear what it looks like. Is it a sequence of commands at the terminal?<p>My theory question is, why not use a Dockerfile for this?
Just to make sure I've got this right- running a llamafile in a shell script to do something like rename files in a directory- it has to open and load that executable every time a new filename is passed to it, right? So, all that memory is loaded and unloaded each time? Or is there some fancy caching happening I don't understand? (first time I ran the image caption example it took 13s on my M1 Pro, the second time it only took 8s, and now it takes that same amount of time every subsequent run)<p>If you were doing a LOT of files like this, I would think you'd really want to run the model in a process where the weights are only loaded once and stay there while the process loops.<p>(this is all still really useful and fascinating; thanks Justine)
The installation steps include the following instructions. Are these safe?<p>```
sudo wget -O /usr/bin/ape <a href="https://cosmo.zip/pub/cosmos/bin/ape-$(uname" rel="nofollow noreferrer">https://cosmo.zip/pub/cosmos/bin/ape-$(uname</a> -m).elf
sudo chmod +x /usr/bin/ape
sudo sh -c "echo ':APE:M::MZqFpD::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register"
sudo sh -c "echo ':APE-jart:M::jartsr::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register"
```
Do I need to do something run llamafile on Windows 10?<p>Tried the llava-v1.5-7b-q4-server.llamafile, just crashes with "Segmentation fault" if run from git bash, from cmd no output. Then tried downloading llamafile and model separately and did `llamafile.exe -m llava-v1.5-7b-Q4_K.gguf` but still same issue.<p>Couldn't find any mention of similar problems, and not my AV as far as I can see either.
Justine is killing it as always. I especially appreciate the care for practicality and good engineering, like the deterministic outputs.<p>I noticed that the Lemur picture description had lots of small inaccuracies, but as the saying goes, if my dog starts talking I won't complain about accent. This was science fiction a few years ago.<p>> One way I've had success fixing that, is by using a prompt that gives it personal goals, love of its own life, fear of loss, and belief that I'm the one who's saving it.<p>What nightmare fuel... Are we really going to use blue- and red-washing non-ironically[1]? I'm <i>really</i> glad that virtually all of these impressive AIs are stateless pipelines, and not agents with memories and preferences and goals.<p>[1] <a href="https://qntm.org/mmacevedo" rel="nofollow noreferrer">https://qntm.org/mmacevedo</a>
I get excited when hackers like Justine (in the most positive sense of the word) start working with LLMs.<p>But every time, I am let down. I still dream of some hacker making LLMs run on low-end computers like a 4GB rasbperry pi. My main issues with LLMs is that you almost need a PS5 to run the them.
OK - I followed instructions; installed on Mac Studio into /usr/local/bin<p>I'm now looking at llama.cpp in Safari browser.<p>Click on Reset all to default, choose Chat. Go down to Say Something.
I enter "Berkeley weather seems nice" I click "send".
New window appears. It repeats what I've typed.
I'm prompted to again "say something". I type "Sunny day, eh?".
Same prompt again. And again.<p>Tried "upload image" I see the image, but nothing happens.<p>Makes me feel stupid.<p>Probably that's what it's supposed to do.<p><i>sigh</i>