This is an already solved problem, but I thought I leave my story here so others won't have to spend as much time debugging as I did.
I'm developing a real-time streaming video app and found that I had severe latency problems with the hardware H264 decoder on Baytrail (and other Intel Android devices). My latency was 135 ms on average for my 60 FPS 720p stream. My video stream is H264 high profile and I cannot change that. I found a page which seems to indicate that changing the profile in the SPS to baseline (66) solves the latency issue (even if the stream isn't really baseline). https://software.intel.com/en-us/android/articles/android-hardware-codec...
However, when I do that, my device displays a corrupted green screen instead of the expected video stream. In addition, I also found that my latency was still the same. Then I ended up reading through the libmix AOSP code for the Nexus Player and found another trigger for the "high latency" path - Adaptive playback. In short, setting MediaFormat.KEY_MAX_WIDTH and MediaFormat.KEY_MAX_HEIGHT prevents triggering the low latency path, even when the profile is baseline. As far as I know, this behavior isn't documented anywhere.
Now that I wasn't using adaptive playback, the latency was down to a very solid 14 ms. I still had the green screen issue. This is where some luck struck. While reading the code, I figured out that the low latency path was only chosen based on the current SPS at the time the first SPS and PPS are submitted to the decoder. As such, I figured I could trick it by submitting a SPS with baseline profile, then after I submitted the PPS, submit the SPS again but with the profile changed back to the correct value. With this fix, I had working low latency video (around 10-20ms)!
I was using Jcodec in my project, so saving and modifying the SPS was as simple as just changing the field in the SpsParameterSet object. It's worth noting that some SPS fields are not defined for baseline profile, so more processing may be required than just changing profile_idc. Jcodec took care of this for me though.
Hopefully this helps some people. I only tried on Moorefield, but these tricks might also work on older chips too.
TL;DR For low latency video:
- Don't set MediaFormat.KEY_MAX_WIDTH and MediaFormat.KEY_MAX_HEIGHT
- Switch the profile in the first SPS to baseline, then resubmit it with the correct profile after the first PPS