New OpenAI Feature: Predicted Outputs

If you use the Cursor IDE: the folks that wrote it talked about their use of speculative decoding to make &quot;Apply&quot; faster on the Lex Friedman podcast last month.<p>Here it is on YouTube, although you can also find it on Spotify and other podcast platforms:<p><a href="https:&#x2F;&#x2F;youtu.be&#x2F;oFfVt3S51T4?t=1206" rel="nofollow">https:&#x2F;&#x2F;youtu.be&#x2F;oFfVt3S51T4?t=1206</a>

I found the OpenAI page to be more interesting <a href="https:&#x2F;&#x2F;platform.openai.com&#x2F;docs&#x2F;guides&#x2F;latency-optimization#use-predicted-outputs" rel="nofollow">https:&#x2F;&#x2F;platform.openai.com&#x2F;docs&#x2F;guides&#x2F;latency-optimization...</a>

This is like the likely() and unlikely() macros in the Linux kernel! Huge speedup if you&#x27;re right; small penalty if you&#x27;re not.