PyTorch Native Architecture Optimization: Torchao

You got to give it to the pytorch team, they're really great at bringing complex optimization schemes (mixed-precision, torch.compile, etc) down to a simple to use API. I'm glad I moved from TF/Kerasto Pytorch around 2018-2019 and never looked back. I'm eager to try this as well.

Hi! I'm Mark from the PyTorch team at Meta and work on torchao. If you have any questions about the library or really anything at all about performance, don't hesitate to ask!

This is a cool project! Understanding lower bits is still on my to do list, perhaps I'll spin this up for a go

Pardon my ignorance, but how do matrix operations on quantized data work? Is hardware support needed?<p>AFAIU int4 matrix multiplication is supported by cuda, but I&#x27;m not sure about other operations. The blog post mentioned fp6, and I don&#x27;t think this is supported by cuda. Or maybe the data are upscaled to something common like fp16 before doing math?

&gt; We’re happy to officially launch torchao, a PyTorch native library that makes models faster and smaller by leveraging low bit dtypes<p>Will this let me use uint8 arrays as indexing arrays? A problem I have is that pytorch forces me to use uint64 for fancy indexing.

Any thoughts on XLA integration across PyTorch instead of keeping it in the separate torch-xla package?

Slightly off-topic: Is there a library in JAX that supports post-training quantization, similar to the one mentioned?

Is this what <i>Mojo</i> is supposed to be?