RTMP2WebRTC: Stuttering when using flutter WebRTC

[winlinvip]

We have developed a live streaming mobile application using Flutter, which employs RTMP for broadcasting and WebRTC for viewing, based on the logic found in your flutter_live repository (https://github.com/ossrs/flutter_live/blob/main/lib/flutter_live.dart). The application performs well on Android devices, but experiences some lag on iOS devices. We are exploring potential optimizations. From previous tests, we observed that the lag on iOS devices is reduced when the encoder profile is switched from 'main' or 'high' to 'baseline', although some lag still persists.

When using RTMP for baseline streaming, there is less stuttering compared to high settings, but stuttering still occurs.

TRANS_BY_GPT4

[zeldan]

I believe this problem is related to #4050 (see my comment #4050 (comment)), where I am experiencing the same issue (but in the browser, not in the application).

[dancixx]

Same issue for me too. I get the same codec information in all cases which is h264 but on iOS in some cases, I have a huge fps drop. I thought there was some experimental codec issue related to VP9/AV1, but the default SRS settings for transcoding is the copy engine, so actually there is no transcoding process.

[DanielZelei]

@winlinvip

In the previously closed issue, you mentioned that the profile should be set to baseline.

I am unable to make any changes to the drone (there are no available options).

Where should I change this setting? Could you please point out where and what I should change in the SRS configuration?

I also tried transcoding with ultrafast and baseline settings, but it was slow (around 2-3 seconds), and I need the streaming to be under 0.5 seconds.

[suzp1984]

Let's reproduce this issue first on iOS device.

My Env

MacOS 14.4.1 M2 chip
iphone 11 + ios 17.4.1
My MacOS's ip address: 192.168.12.100

Here is my steps to do it.

1. prepare a video source to testing

I used this video source https://www.youtube.com/watch?v=-B2jRTTmYwk, restore it to mp4 file by youtube-dl.
(I like this song, you're welcome).
store the video source as test.mp4.

ffprobe test.mp4 and ffprobe -show_frames | grep -i 'pict_type' to check the mp4 file video codec profile and I/P/B frames.

Above video is h.264 high profile without b-frame.

2. start srs

export CANDIDATE=192.168.12.100
./objs/srs -c conf/rtmp2rtc.conf

replace ip address to yours.

3. rtmp publish to srs

1. publish to srs with b-frame

ffmpeg -re -stream_loop -1 -i test.mp4 -vcodec h264 -profile:v high -acodec copy -f flv rtmp://localhost/live/livestream

2. publish to srs without b-frame

ffmpeg -re -stream_loop -1 -i test.mp4 -vcodec h264 -profile:v high -bf 0 -acodec copy -f flv rtmp://localhost/live/livestream
or
ffmpeg -re -stream_loop -1 -i test.mp4 -vcodec copy -acodec copy -f flv rtmp://localhost/live/livestream
or 
ffmpeg -re -stream_loop -1 -i test.mp4 -vcodec h264 -profile:v baseline -acodec copy -f flv rtmp://localhost/live/livestream

4. build and install flutter_live

use the branch in https://github.com/ossrs/flutter_live/pull/20.
In https://github.com/ossrs/flutter_live/pull/20/files#diff-69396fa518b81aa9d2099ead13c33e18bc8e570098408f2f50ee114adf6898d1, I upload my DEVELOPMENT_TEAM = 4X75RZ9C85; , that's a mistake, if you knows a tiny of xcode, you know how to replace this identify to your own one.

What I did in this draft is to upgrade flutter and flutter_webrtc and remove the camera_with_rtmp dependency.

5. play webrtc videos in ios

click webrtc option.
input: webrtc://192.168.12.100/live/livestream. (replace the ip address to yours).
click play.
compare the video quality.

My conclusion

I think the video stream without b-frame seems not that bad, just the same result to play webrtc in web browser (both safari and chrome in MacOS, check my comments in #4050)

[suzp1984]

Steps to reproduce

1. download this video sample recorded from the problem drone. https://drive.google.com/file/d/1UEMoMxv76D9W5pzOE9RQuBQ1bT4qIp06/view?usp=sharing
2. start srs: ./objs/srs -c conf/rtmp2rtc.conf, don't forget to config env $CANDIDATE.
3. rtmp publish step 1 video: ffmpeg -re -stream_loop -1 -i drone_1.flv -vcodec copy -f flv rtmp://localhost/live/livestream
4. check webrtc play in ios flutter_live app. if the main branch of flutter_live can't compile well with your flutter toolchain, use this branch https://github.com/suzp1984/flutter_live/tree/debug/4052-webrtc
5. compare the video with the chrome pc version. the chrome pc webrtc play well, the ios webrtc play stuttering.
6. check the safari mac: the webrtc play also has same problem, the http-flv can't play only show one picture with error indicator.

What I found have three problems: ios flutter_live webrtc play, mac safari webrtc play and mac safari http-flv & http-ts play.

Note: use export SRS_VHOST_HTTP_REMUX_HAS_AUDIO=off; ./objs/srs -c conf/http.ts.live.conf to reproduce http-ts play problem.

Cause

the ios & mac hardware decoder (Video Toolbox) dislike to feed it so many SEI NALU between NonIDR and IDR NALU packets.
The general NALU type sequence of a Normal rtmp video stream is like this:
SEI -> IDR -> SEI -> IDR -> ... NonIDR -> IDR -> ... NonIDR
But the problem video stream is like this:
SEI -> IDR -> SEI -> IDR -> SEI -> NonIDR -> (repeat SEI, NonIDR | IDR).

Solutions

add filter to delete the SEI NALU for rtc, which is works well.
for the http-flv on safari mac, need to do similar thing to filter the flv video tags with SEI NALU.(not yet verified).