After HttpClient processes HLS files over HTTPS, the process becomes highly CPU intensive

[koutaro-masaki]

• Dart version and tooling diagnostic info (dart info)
  • Dart 3.0.2 (stable) (Tue May 23 08:26:58 2023 +0000) on "macos_x64"
  • on macos / Version 13.5.2 (Build 22G91)
  • locale is ja-JP
• Whether you are using Windows, macOS, or Linux (if applicable)
  • macOS

Start the server with the following program.

import 'dart:io';
   
 void main(List<String> arguments) async {
   final server = await HttpServer.bind(InternetAddress.anyIPv4, 8080)
     ..listen((proxyRequest) async {
       final realUrl = Uri.parse(
           'https://devstreaming-cdn.apple.com/videos/streaming/examples/img_bipbop_adv_example_fmp4${proxyRequest.uri.path}');
 
       print('${proxyRequest.uri} -> $realUrl');
 
       final client = HttpClient();
 
       try {
         final req = await client.openUrl(proxyRequest.method, realUrl);
         final realResponse = await req.close();
 
         proxyRequest.response.statusCode = realResponse.statusCode;
         const headerWhiteList = [
           'date',
           'content-length',
           'content-type',
           'accept-ranges',
           'last-modified',
           'etag',
         ];
         realResponse.headers.forEach((key, value) {
           if (headerWhiteList.contains(key)) {
             proxyRequest.response.headers.add(key, value);
           }
         });
 
         await proxyRequest.response.addStream(realResponse);
         await proxyRequest.response.flush();
         await proxyRequest.response.close();
       } finally {
         client.close();
       }
     });
 
   print('🚀 http://${server.address.host}:${server.port}');
 }

Requesting /master.m3u8 from a mobile app video player (AVPlayer or EXOPlayer) towards this server may load to abnormally high CPU usage on the machine running the server. This elevated load will persist until the server is stopped.

The following results were obtained after validation under several different conditions.

• If requested from a MacBook using VLC or the <video> element in a browser, instead of a mobile app player, there was no occurrence of abnormally high CPU usage.
• When I changed the HLS file requested from the server to use http instead of https, there was no occurrence of abnormally high CPU usage.
• When I changed the HttpClient used internally in the server to CupertinoClient, there was no occurrence of abnormally high CPU usage.

I have also attached a snapshot of the Dart process where the problem is occurring.

[mraleph]

Sorry this fell through.

/cc @brianquinlan @aam

[aam]

Requesting /master.m3u8 from a mobile app video player (AVPlayer or EXOPlayer) towards this server may load to abnormally high CPU usage on the machine running the server.

Do you see elevated network traffic on mobile/server as this request being processed on the server?
Are you able to reproduce it with simple http client on macbook(opening http://0.0.0.0:8000/master.m3u8 in desktop browser)? What mobile device/os are you using? Can you post a links to the version of avplayer/exoplayer you used?

[mraleph]

@aam FWIW I merged another issue into this one which has a reproduction one could run (#54752). That issue is about generic slowness of HTTPS implementation. It might not be the same problem - but certainly a related one. I heave peeked a bit under the covers at our implementation and it certainly leaves an impression of being suboptimal.

[Tienisto]

You can checkout this repo for reproduction: https://github.com/Tienisto/httpserver-benchmark
It isn't specific to device/OS from my impression.
Generally, the "slowness" can be noticed on Android and old iPhones because they have slower CPUs.

[aam]

That issue is about generic slowness of HTTPS implementation. It might not be the same problem - but certainly a related one.

Would be good to confirm that if original reporter could provide more information on repro steps. Somehow it looks more severe than generic slowness(which certainly also worth investigating)
But again looking at the threads snapshot from original report it looks like significant number of cycles are spent in getting bytes from c++ to dart(api message seriazlier, allocation and mor surprisingly stacktraces), so perhaps it is the same thing as comparative dart to node.js slowness .

[aam]

Thank you for the benchmark https://github.com/Tienisto/httpserver-benchmark.

Using that I see that bumping up secure filter buffer size by 100x improves throughput number for 100mb file by over 2x on linux x64 desktop.

Throughput for five runs of node.js client uploading 100mb file to dart server in MB/s all running locally on linux x64.

buffer size	10kb	1000kb
release	14 17 15 17 16	42 43 45 47 46
product	21 21 21 21 21	48 48 50 50 49

Release build of dart sdk is slower than product build (especially for smaller buffers) probably due to stack trace capturing during io service receive ports opening.

Node.js server receiving same upload from same node.js client still almost 10x faster at ~350 MB/s on the same machine.

[koutaro-masaki]

AVPlayer/ExoPlayer was indirectly utilized from https://pub.dev/packages/video_player. Since the original Flutter interface did not support HLS, I used a modified version to accommodate it, although there are no changes to the core player itself.

The Dart server runs on macOS Ventura 13.6.3, Intel Core i7, and mobile testing was performed using an emulator. The issue reproduces regardless of the mobile OS version.

I haven't investigated the increase in traffic. However, there doesn't seem to be any indication of a delay in loading videos.

I am not sure if this is the same problem discussed, as my issue is that the CPU usage remains high semi-permanently after the request completes. It may be that some trigger is activated by the slow HTTPS implementation, causing an infinite loop, though.