Build models like we build open-source software

I feel like there&#x27;s several industries that are practically computer science yet don&#x27;t utilize open source effectively. Data science is definitely one, but the video game industry definitely comes to mind.<p>You could argue game engines are notoriously complex, but the Linux kernel would like a word.

Collaborative incremental improvement of models would be extremely disruptive. While this happens via research, it&#x27;s massively inefficient, particularly as pretrained models get larger and span multiple modalities.

There is definitely a problem re: large parameter models, the issue is I don’t think throwing software dev tools at this is the right solution.<p>The constraint is largely hardware. The incremental post training done via transfer learning is generally not broadly applicable to many use cases.

I&#x27;m curious how Deepmind&#x27;s MOE models Perceiver and Switch might play into managing a open distributed model.