TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

Ask HN: Where to Host Llama 2?

3 pointsby retrovrvalmost 2 years ago
There's ollama/ggml etc for local setup, but other than Replicate, what are the other options for hosting Llama 2?

1 comment

brucethemoose2almost 2 years ago
vast.ai is a popular and economical option.<p>A single 3090 will host 70B reasonably well, two will fit it completely in vram.<p>Another thing I suggest is hosting on AI horde with koboldcpp, if the UI&#x2F;API works for you and the finetune is appropriate for public use. You will get priority access to your host, but fulfilling other prompts in its spare time will earn you kudos to try other models people are hosting, or to get more burst throughput.<p><a href="https:&#x2F;&#x2F;lite.koboldai.net&#x2F;#" rel="nofollow noreferrer">https:&#x2F;&#x2F;lite.koboldai.net&#x2F;#</a>
评论 #37127167 未加载