Hi HN,<p>Bodhi App is an open-source local LLM inference solution that takes a different and simpler approach. Instead of re-inventing the wheel, it leverages existing, tried and tested ecosystem and solutions:<p>## Technical Architecture:<p>- llama.cpp as inference engine<p>- Rust/Axum backend for type-safe API layer<p>- Tauri for multiplatform builds<p>- HuggingFace integration<p>- YAML based configurations and update at runtime (no restarts required)<p>- OpenAI/Ollama API compatibility layer<p>## Key Technical Decisions:<p>1. No proprietary model format - directly use of GGUF files from HuggingFace<p>2. Opt-in Authentication, provides RBAC for team access<p>3. API design with proper authentication/authorization<p>4. Built-in Swagger UI with complete OpenAPI specs<p>5. Built-in User guide<p># What Sets It Apart:<p>Designed with non-technical users in mind. So it comes a basic Web-based user interface, allowing users to get started quickly with their first AI-assistant conversation.<p>## Setup Wizard:<p>- App displays a setup wizard when run for first time<p>- Allows user to download popular models in a user friendly way<p>## Built-in Chat UI:<p>- Ships with a complete Chat UI<p>- Chat UI is simple enough for non-technical users to get started with their first AI-conversation<p>- Adapts to power users by providing complete control over request settings<p>- Supports realtime streaming response, markdown rendering, code rendering with syntax highlights<p>- Displays chat stats, request tokens, response tokens, token speed<p>- Allow copying of the AI-response etc.<p>## Built-in UI for Model + App Management + API access:<p>- Manage complete Model lifecycle from the UI<p>- Downloading models, deleting models<p>- Configuring models, request + inference server configurations using Model Alias yaml files<p>- Allows configuring for parallel processing of requests<p>- Configuring App Settings - chosing betwen CPU/GPU, server idle time etc.<p>- API tokens for authenticated/authorized access to APIs by 3rd party<p>## Tech for UI:<p>- Uses Nextjs, Tailwindcss, Shadcn to build powerful, responsive and user friendly UI<p>- Supports Dark/Light mode<p>- Exported using config `output: "export"` to export the entire frontend as static html + javascript<p>- Served by the backend as static asset<p>- Thus no packaged nodejs server, reducing app size, complexity and compute<p># Links<p>Try it out: <a href="https://getbodhi.app/" rel="nofollow">https://getbodhi.app/</a><p>Source: <a href="https://github.com/BodhiSearch/BodhiApp">https://github.com/BodhiSearch/BodhiApp</a><p>Looking forward to technical feedback and discussions.