VideoDecoderConfig optimizeForLatency dropped frames and performance #491
Description
Using the code from https://w3c.github.io/webcodecs/samples/mp4-decode/ I am seeing dropped/skipped frames when using optimizeForLatency = false. Testing with the default video I get the following results:
optimizeForLatency = false
frameCount: 19036 elapsed: 44.52 seconds
optimizeForLatency = true
frameCount: 19036 elapsed: 46.74 seconds
And when testing with a higher resolution video with 1 keyframe and 242 P frames, I get the following results:
optimizeForLatency = false
frameCount: 232 elapsed: 5.83 seconds
optimizeForLatency = true
frameCount: 243 elapsed: 11.94 seconds
Interestingly, on the second higher resolution video (3840x2160) the performance difference is greater than 2x when optimizeForLatency = true | false.
My file was created with FFmpeg with the following parameters -c:v libx264 -g 2000 -keyint_min 2000 -bf 0 -profile:v high -level 4.1 -crf 32 -pix_fmt yuv420p and was dashed via this command mp4box -dash 10 out.mp4
Note that I am testing with an upgraded mp4box.all.min.js version 0.5.2 and all of the above was tested on the following machine:
Google Chrome | 101.0.4951.64 (Official Build) (x86_64)
-- | --
Revision | d1daa9897e1bc1d507d6be8f2346e377e5505905-refs/branch-heads/4951@{#1208}
OS | macOS Version 12.3.1 (Build 21E258)
JavaScript | V8 10.1.124.12
User Agent | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36
Any ideas as to what could be causing those frame drops and performance differences? I know VideoEncoderConfig has a latencyMode parameters (Quality vs. Realtime) which is noted that it may cause frame drops in Realtime mode, but I don't see anywhere mentioning frame drops for the VideoDecoderConfig optimizeForLatency setting.
I've attached the corresponding mpd file file in case that helps at all.