It's kind of weird to think that in a coding assistant, an LLM is regularly asked to produce a valid block of code top to bottom, or repeat a section of code with changes, when that's not what we do. (There are other intuitively odd things about this, like the amount of compute spent generating 'easy' tokens, e.g. repeating unchanged code.) Some of that might be that models are just weird and intuition doesn't apply. But maybe the way we do it--jumping around, correcting as we go, etc.--is legitimately an efficient use of effort, and a model could do its job better, with less effort, or both if it too used some approach other than generating the whole sequence start-to-finish.<p>There's already stuff in the wild moving that direction without completely rethinking how models work. Cursor and now other tools seem to have models for 'next edit' not just 'next word typed'. Agents can edit a thing and then edit again (in response to lints or whatever else); approaches based on tools and prompting like that can be iterated on without the level of resources needed to train a model. You could also imagine post-training a model specifically to be good at producing edit sequences, so it can actually 'hit backspace' or replace part of what it's written if it becomes clear it wasn't right, or if two parts of the output 'disagree' and need to be reconciled.<p>From a quick search it looks like <a href="https://arxiv.org/abs/2306.05426" rel="nofollow">https://arxiv.org/abs/2306.05426</a> in 2023 discussed backtracking LLMs and <a href="https://arxiv.org/html/2410.02749v3" rel="nofollow">https://arxiv.org/html/2410.02749v3</a> / <a href="https://github.com/upiterbarg/lintseq">https://github.com/upiterbarg/lintseq</a> trained models on synthetic edit sequences. There is probably more out there with some digging. (Not the same topic, but the search also turned up <a href="https://arxiv.org/html/2504.20196" rel="nofollow">https://arxiv.org/html/2504.20196</a> from this Monday(!) about automatic prompt improvement for an internal code-editing tool at Google.)