FWIW common libraries like ffmpeg and what not can contain bugs which video files could potentially abuse and exploit.

Is that a common risk? Probably not. But image libraries have been known to have such exploits.