When an issue is as reproducible as this it’s usually very attractive for someone to grab and implement a fix quickly. Voilà! Now I had a test case that reproduced this rare issue every single time. To consistently reproduce the problem I managed to get a Wireshark capture of one of the failing calls and fed the capture into the video_replay tool.
Replicating the issue this way was very time consuming, often frustrating, and ultimately did not make it easy for the WebRTC team to find the fix. Initially I was able to develop a test setup that reproduced the issue in about 1 out of every 20 calls. Video coding issues are often among the toughest problems to solve. This caused the internal state of the VP8 decoder to be wrong and output frames that looked like random data. Eventually, after using video_replay to debug, the WebRTC team found that Chrome’s jitter buffer reimplementation introduced a bug that made the video stream corrupt in certain cases. To illustrate, recently I was working on an issue where Chrome suddenly displayed the incoming video as a corrupt image shown above. video_replay takes a captured RTP stream of video as an input file, decodes the stream with the WebRTC framework “offline”, and then displays the resulting output on screen. It’s purpose? To easily replay a capture of a WebRTC call to reproduce an observed behavior. WebRTC contains a nice and little known tool called video_replay which has proven very useful for debugging video decoding issues. Fortunately there are tools to help like video_replay. You have a problem if your WebRTC video stream looks like this.
He has experience in large parts of the media stack with a special interest in video codecs and other types of signal processing, network protocols and error resilience. Stian, who works at Pexip, has been dealing with real-time communication for more than 10 years. Unfortunately this process is not too well documented, so we asked Stian to walk us through the process of capturing the necessary data and using the video_replay tool. With an easy reproduction of the stream, the WebRTC video team at Google made short work of the bug. When I saw another video corruption issue filed by Stian Selnes I told him about that tool. To combat these issues, has a pretty powerful tool to reproduce and analyze them called video_replay.
These issues are hard to debug since they occur only when certain packets are lost. Recent Chrome versions have been plagued by video corruption issues related to a new video jitter buffer introduced in Chrome 58. Decoding video when there is packet loss is not an easy task.
0 kommentar(er)
