Hey HN! Alex and Zack here from Nexa AI. We're excited to share something we've been working on.<p>Our journey began with the Octopus series --- action models for mobile AI agents (<a href="https://huggingface.co/NexaAIDev/Octopus-v2" rel="nofollow">https://huggingface.co/NexaAIDev/Octopus-v2</a>). We focused on making sub-billion parameter models excel at function calling, making high accurate and fast function-calling possible on mobile and edge devices. But as we delved into developing full-fledged on-device applications, we hit a roadblock.<p>We realized that optimizing for function calling (tool-use) alone wasn't enough. Building powerful on-device AI apps requires a diverse set of tools: language models with domain expertise, speech processing, image generation, embedding models and more. That's when we decided to create Nexa SDK --- a comprehensive toolkit that brings together everything developers need to build powerful and efficient AI applications that run entirely on-device.<p>Here's what Nexa SDK offers:<p><pre><code> - Support for both ONNX and GGML models.
- An integrated conversion engine for making custom GGML Quantized Models for different device hardware requirements.
- An inference engine that supports language models, image generation models, TTS, audio generation models, and Vision-Language Models.
- An OpenAI-compatible API server with optimization in function calling.
- A Streamlit UI for rapid prototyping.
- An intuitive CLI for easy model management.
- Backend optimizations for latency and power consumption on edge devices.
</code></pre>
We've designed Nexa SDK to be the go-to solution for developers pushing the boundaries of what's possible with on-device AI applications and AI on edge devices.<p>To showcase its capabilities, we've built several demo apps running entirely on your device (<a href="https://github.com/NexaAI/nexa-sdk/tree/main/examples">https://github.com/NexaAI/nexa-sdk/tree/main/examples</a>):<p><pre><code> - AI soulmate with uncensored model and audio-in/audio-out interaction.
- A quick interface for uploading and chatting with PDFs like your personal finance documents.
- A meeting transcription app supporting multiple languages and real-time translation.
</code></pre>
We're proud to share that the winner of yesterday's (Sep 7) House AGI "AI PC/ GenAI Goes Local" hackathon used Nexa SDK to build a local semantic image search (<a href="https://github.com/asl3/deja-view">https://github.com/asl3/deja-view</a>).<p>But we're just getting started! There are lots of exciting developments in our pipeline, and we can't wait to share them with you soon!<p>Check it out: (<a href="https://github.com/NexaAI/nexa-sdk">https://github.com/NexaAI/nexa-sdk</a>)<p>Docs: (<a href="https://docs.nexaai.com/" rel="nofollow">https://docs.nexaai.com/</a>)<p>If you're excited about the future of on-device AI, we'd really appreciate your support. A star on our GitHub repo goes a long way in helping us reach more developers!<p>Cheers,<p>Alex & Zack