Cannot start vm_service for a flutter test.

[polina-c]

Steps:

1. Create counter application.
2. Add a test file to the folder test with the code below
3. Run the tests
4. Observe both tests fail

Copy the test to a dart project and see the test succeed.

The test code:

import 'dart:developer';

import 'package:flutter_test/flutter_test.dart';

Future<Uri> _serviceUri() async {
  Uri? uri = (await Service.getInfo()).serverWebSocketUri;

  if (uri != null) return uri;

  uri = (await Service.controlWebServer(enable: true)).serverWebSocketUri;

  const timeout = Duration(seconds: 5);
  final stopwatch = Stopwatch()..start();

  while (uri == null) {
    if (stopwatch.elapsed > timeout) {
      throw StateError(
        'Could not start VM service.',
      );
    }
    await Future.delayed(const Duration(milliseconds: 1));
    uri = (await Service.getInfo()).serverWebSocketUri;
  }

  return uri;
}

void main() {
  test('Service can be started for flutter test.', () async {
    Uri? uri;

    uri = await _serviceUri();

    print(uri);
  });

  testWidgets('Service can be started for flutter widgets test.',
      (WidgetTester tester) async {
    Uri? uri;

    await tester.runAsync(() async => uri = await _serviceUri());

    print(uri);
  });
}

[mkustermann]

/cc @bkonyi @derekxu16

[bkonyi]

So I've done some digging, and I think this might be a tricky one to resolve.

First off, I've always assumed that invoking Service.controlWebServer with enable = true actually started the VM service isolate if it wasn't already running. As it turns out, that's not actually the case, and the API only controls whether or not the VM service will accept HTTP / WebSocket requests.

Another important point to remember is that the creation of isolates is not controlled completely by the Dart VM, as the embedder is responsible for initializing much of an isolate's state. The service isolate is no exception, and the embedder has a say in how and if the service isolate is initialized.

In the standalone embedder (i.e., dart), the service isolate is always present in configurations that support it (i.e., not PRODUCT mode). The presence of --observe or --enable-vm-service is used to set the value of _autoStart, causing the web server to start here. If the HTTP server is not automatically started, invoking Service.controlWebServer(enable: true) causes the HTTP server to be initialized, making the VM service accessible.

On the other hand, the Flutter engine takes a different approach. Instead of always initializing the service isolate and leaving the HTTP server disabled by default, if --enable-vm-service isn't provided the service isolate simply isn't started. This means that invoking Service.controlWebServer(enable: true) has no effect in Flutter or Flutter tests if --enable-vm-service wasn't provided at launch.

Unfortunately, I don't think resolving this issue will be as simple as always starting the service isolate in the Flutter engine as that will lead to performance regressions in benchmarks that don't explicitly pass --enable-vm-service.

We could potentially change flutter_tester to always start the service isolate in a similar fashion to the standalone VM. This would still impact memory usage (and likely startup times), but would only impact tests instead of all Flutter configurations that support the VM service.

[bkonyi]

I'm going to close this issue and move discussion over to flutter/flutter#132865 as this is not a Dart SDK issue.