I don't understand video encoding/decoding algorithms. What I want is to process a video stream and parse each frame individually, either as an image or as a matrix of rgba values. Anything out there that can help me with that?
If you're looking for a library to do this programmatically, try imageio (Python) or OpenCV's VideoCapture (C/C++/Python/Java). I recommend imageio - very easy to deploy, compared to OpenCV. Don't know about other languages.