Simplifying Transformer Blocks

“This complexity leads to brittle architectures, where seemingly minor changes can significantly reduce training speed, or render models untrainable.”<p>I am by no means an expert and I can’t verify the authors’ claims about reduced speed and untrainability, but this reflects an impression I’ve been having on the papers I read and review. The field of ML research is moving so fast that people don’t even take time anymore to explain the design decisions behind their architectures. It’s basically “we got nice results, and here is the architecture of the model” (proceeds to show a figure with a hundred coloured blocks connected together in some seemingly random complex way).<p>It used to be that such a thing would get backlash from reviewers, and they would require you to actually justify the design. I don’t see that anymore. The problem with this for me is that we fail to build a nice, crisp understanding of the effects of each design decision in the final outcomes, which hurts the actual “science” of it. It also opens up the field for bogus and unreproducible claims.<p>But at least other people are picking up on the thread and doing that in follow-up papers, which is good.

&quot;[...] modern neural network (NN) architectures have complex designs with many components [...]&quot;<p>I find the Transformer architecture actually very simple compared to previous models like LSTMs or other recurrent models. You could argue that their vision counterparts like ViT are conceptually maybe even simpler than ConvNets?<p>Also, can someone explain why they are so keen to remove the skip connections? At least when it comes to coding, nothing is simpler than adding a skip connection and computationally the effect should be marginal?

&quot;While we have demonstrated the efficacy of our simplifications across architectures, datasets, and tasks, the models we have considered (100-300M parameters) are small relative to the largest transformers.&quot;<p>University researchers without a big lab&#x27;s backing cannot try out such experiments on really large models.

Not sure if I read it correctly, but it seems like the skip connections are kinda still present in the skipless block because they added I after the softmax and v= the previous hidden state. Still a cool paper if they really managed to get rid of those projections without degrading the quality.

There could be a CISC vs RISC argument for Transformer, which I can see, but my bet is the traditional transformer has established enough fundamentals, it is going to take a long time for alternative architectures to prove they are indeed alternatives on a wide range of tasks.<p>Pretty much at this point, to succeed transformer, one need to showcase, the alternative model could achieve chatgpt level performance with either significant reduction of compute or data requirements.

This is a nice start, but what would really help is for someone who understands the GPU programming model in-depth to give this a shot, with the goal of reducing DRAM bandwidth and fitting the layers exactly onto a GPU&#x27;s memory hierarchy (cache levels, local memory and registers). Basically, sizes of a target HW platform&#x27;s memories should be hyper-parameters of the architecture.

What is &quot;Shaped Attention&quot;? They simplified everything except for changing &quot;Attention&quot; to &quot;Shaped Attention&quot;.

I love reading about papers like these. They raise hopes that novel model architectures might reduce the need for computational resources to train powerful models, which would help lower the barrier of entry.

I wonder why only 15% faster training? Seems like simplying the main element would make a huge difference.

Are anyone aware of similar simplifications?

Do they have a demo of this theory?