Mistral: Our first AI endpoints are available in early access

I&#x27;m surprised this isn&#x27;t firmly attached to the top of HN right now for the entire day.<p>This is a tiny company (appears 30 or so people?) that just scored a 2B valuation, produced easily the most performant 7B model and a 7B*8 MOE model that performs at the level of a 70B requiring the inference power of a 14B.<p>I feel this could be a potential bigger threat to OpenAI than Google or Anthropic. I gather with the huge recent investment they&#x27;ll be able to a) scale out to a reasonable traffic load in the near future and b) attract the best and brightest researchers put off w&#x2F; various chest puffing and drama that has been front and center in this industry.

This is extremely impressive if benchmarks translate to real-world performance [1]. The mistral-medium beats GPT3.5 and also Gemini Pro (Google&#x27;s best available model) with a huge margin on all available comparable benchmarks: <a href="https:&#x2F;&#x2F;screenbud.com&#x2F;shot&#x2F;c0d904e3-24a3-4c23-a1e4-2f18bc0215cf&#x2F;image.png" rel="nofollow noreferrer">https:&#x2F;&#x2F;screenbud.com&#x2F;shot&#x2F;c0d904e3-24a3-4c23-a1e4-2f18bc021...</a><p>[1] I would expect real world-performance gap to be even larger if Mistral 7B is anything to go by. The fact that safety filters are opt-in is a huge benefit (even for safe applications).

<a href="https:&#x2F;&#x2F;docs.mistral.ai&#x2F;platform&#x2F;pricing" rel="nofollow noreferrer">https:&#x2F;&#x2F;docs.mistral.ai&#x2F;platform&#x2F;pricing</a><p>Pricing has been released too.<p>Per 1 million output tokens:<p>Mistral-medium $8<p>Mistral-small $1.94<p>gpt-3.5-turbo-1106 $2<p>gpt-4-1106-preview $30<p>gpt-4 $60<p>gpt-4-32k $120<p>This suggests that they’re reasonably confident that the mistral-medium model is substantially better than gpt3-5

&quot;endpoints are available in early access&quot; is in reality &quot;we have a waitlist (of unspecified length) for early access to endpoints&quot;<p>When I try to access: “Access to our API is currently invitation-only, but we&#x27;ll let you know when you can subscribe to get access to our best models.”

&gt; Mistral-embed, our embedding endpoint, serves an embedding model with a 1024 embedding dimension. Our embedding model has been designed with retrieval capabilities in mind. It achieves a retrieval score of 55.26 on MTEB.<p>Is there any information if this embedding model is or will be open source?

&gt; Our API follows the specifications of the popular chat interface initially proposed by our dearest competitor.<p>I like it, also made me laugh

&gt; Mistral-Medium outperforms GPT-4 in Winogrande benchmark 88% vs 87.5%<p>from: <a href="https:&#x2F;&#x2F;twitter.com&#x2F;yupiop12&#x2F;status&#x2F;1734137238177698106" rel="nofollow noreferrer">https:&#x2F;&#x2F;twitter.com&#x2F;yupiop12&#x2F;status&#x2F;1734137238177698106</a>

It&#x27;s interesting that many platforms, like Lemonfox.ai, offer Mistral finetunes at lower prices. They also already announced a Mistral 8x7B API. This raises the question of whether they&#x27;ll still publish future models as open-source (like the Medium version) if they want to make money.

By chance I noticed that Fabrice Bellard&#x27;s TextSynth server has newly added support for Mistral 7B model.<p>&gt; 2023-10-21: CUDA support in the Windows version, mistral model support. Speculative sampling is supported. BNF grammar and JSON schema sampling.<p>&gt; mistral_7B_instruct_q4 - 3.9GB - Mistral 7B chat model<p><a href="https:&#x2F;&#x2F;bellard.org&#x2F;ts_server&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;bellard.org&#x2F;ts_server&#x2F;</a>

&gt; Mistral-medium. Our highest-quality endpoint currently serves a prototype model, that is currently among the top serviced models available based on standard benchmarks.<p>This is interesting. This model outperforms ChatGPT 3.5. I&#x27;m not sure what type of model it is, and it is not open-sourced.

&gt; Mistral-tiny. Our most cost-effective endpoint currently serves Mistral 7B Instruct v0.2, a new minor release of Mistral 7B Instruct. Mistral-tiny only works in English. It obtains 7.6 on MT-Bench. The instructed model can be downloaded here.<p>&quot;download here&quot; link is to v0.1 [0]. Oversight or are they holding back the state of the art tiny model?<p>[0]: <a href="https:&#x2F;&#x2F;huggingface.co&#x2F;mistralai&#x2F;Mistral-7B-v0.1" rel="nofollow noreferrer">https:&#x2F;&#x2F;huggingface.co&#x2F;mistralai&#x2F;Mistral-7B-v0.1</a>

Wow, beating ChatGPT-3.5 is really an accomplishment. Congrats! That&#x27;s literally the default of OpenAI&#x27;s product. I had to fallback to GPT-3.5 the other day because I ran out of usage on ChatGPT-4 (playing 20 questions lol). So I really hope someone can come up on GPT-4! For me GPT-3.5 isn&#x27;t good enough for daily things, it gets too much wrong.

This actually begs the question:<p>Does anyone know the kind of actual infrastructure something like gpt4-32k actually run on?<p>I mean when I actually type something in the prompt, what actually happens behind the scenes?<p>Is the answer computed on a single NVidia GPU?<p>Or is it dedicated H&#x2F;W not known to the general public?<p>How big is that GPU?<p>How much RAM does it have?<p>Is my conversation run by a single GPU instance that is dedicated to me or is that GPU shared by multiple users?<p>If the latter, how many queries per seconds can a single GPU handle?<p>Where is that GPU?<p>Does it run in an Azure data center?<p>Is the API usage cost actually reflective of the HW cost or is it heavily subsidized?<p>Is a single GPU RAM size the bottleneck for how large a model can be?<p>Is any of that info public ?

I’m surprised no one has commented on the context size limitations of these offerings when comparing to the other models. The sliding window technique really does effectively cripple its recall to approximately just 8k tokens which is just plain insufficient for a lot of tasks.<p>All these llama2 derivatives are only effective if you fine tune them, not just because of the parameter count as people keep harping but perhaps even more so because of the tiny context available.<p>A lot of my GPT3.5&#x2F;4 usage involves “one offs” where it would be faster to do the thing by hand than to train&#x2F;fine-tune first, made possible because of the generous context window and some amount of modest context stuffing (drives up input token costs but still a big win).

Competition is how the world moves forward. I&#x27;m super glad small and big players have competitive models.<p>The thing that makes me a bit sad is how announcements show the benchmarks but they way they test is tweaked to make their metrics favorable. They aren&#x27;t apple to apple benchmarks across different paper publications.<p>Super grateful that they openly share the weights and code with Apache license.<p>25 shot - is having 25 tries and selecting the best answer.<p>Is anyone working on an open benchmark where they take the major models and compare them apples to apples.

Well done, Mistral! &quot;Show, don&#x27;t tell&quot; par excellence.

So, what would be the hardware setup for this?<p>Can it run on 1 GPU and swap between experts.

I just signed up for the API waiting list. I have been enjoying running Mistral-7B on my home system, and it feels right to give them some of my paid for API business.

Until we can verify, I think it’s safe to place the in the smoke category. It’s invite only so until it hits GA it’s impossible to know if the pricing is real and the true capabilities of what they are offering.

Is there anything Mistral + tuned on ChatGPT4?

dupe, see:<p><a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=38598559">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=38598559</a>

Two out of the top three stories today on HN! What an achievement. Is mistral.ai a YC property?