I'm beginning to think that it has got something to do with the resolution of the encode, maybe ffmpeg switches the baseline without asking me.. I encoded a different clip, this time with a resolution of 928x720 and it also fails to load without any special tricks.