[desktop] High memory consumption [Part 2] #95092

ivk1800 · 2021-12-11T19:14:57Z

continue of #90868

in multi modules project consumes too much memory. project contains 40 modules with included dart_code_metrics. I attach test project for reproduce.

Steps fo reproduce:

download archive and extract
open project in intellij idea
open any dart file and enable dart for project
run sync.sh
close idea
open again project
wait until analizier done
check system monitor
check memory, memory grows
wait until stop growing
memory not released

dart_code_metrics: 4.8.1

Dart version 2.15.0, macOS 11.5.2 20G95 darwin-x64

HighMemoryConsumption.zip

memory.mov

The text was updated successfully, but these errors were encountered:

ivk1800 · 2021-12-11T19:15:50Z

@scheglov @incendial

incendial · 2021-12-12T07:44:33Z

@ivk1800 we have an open issue in Dart Code Metrics repo and right now it looks like the problem is with the plugin, not the integration. But I'd really appreciate some help from the Dart team, just to be sure that the integration is solid and doesn't have memory leaks.

incendial · 2021-12-12T12:31:13Z

After some investigation with different plugin setups I'm not that sure that it's our code leaking. Definitely need some help with checking the integration memory consumption.

darshankawar · 2021-12-13T09:00:11Z

I tried the given zip and followed steps to replicate on latest master and stable on mac and can see CPU and memory consumed by dart, as below:

stable, master flutter doctor -v

[✓] Flutter (Channel stable, 2.8.0, on Mac OS X 10.15.4 19E2269 darwin-x64,
    locale en-GB)
    • Flutter version 2.8.0 at /Users/dhs/documents/fluttersdk/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision cf44000065 (10 hours ago), 2021-12-08 14:06:50 -0800
    • Engine revision 40a99c5951
    • Dart version 2.15.0

[✓] Android toolchain - develop for Android devices (Android SDK version 30)
    • Android SDK at /Users/dhs/Library/Android/sdk
    • Platform android-30, build-tools 30.0.3
    • ANDROID_HOME = /Users/dhs/Library/Android/sdk
    • Java binary at: /Users/dhs/Library/Application Support/JetBrains/Toolbox/apps/AndroidStudio/ch-0/202.7486908/Android
      Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)
    • All Android licenses accepted.    

[✓] Xcode - develop for iOS and macOS
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 12.3, Build version 12C33
    • CocoaPods version 1.10.1

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.1)
    • Android Studio at /Users/dhs/Library/Application Support/JetBrains/Toolbox/apps/AndroidStudio/ch-0/202.7486908/Android
      Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build
      1.8.0_242-release-1644-b3-6915495)        

[✓] VS Code (version 1.57.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.21.0

[✓] Connected device (4 available)
    • Darshan's iphone (mobile)  • 21150b119064aecc249dfcfe05e259197461ce23 •
      ios            • iOS 14.4.1 18D61
    • iPhone 12 Pro Max (mobile) • A5473606-0213-4FD8-BA16-553433949729     •
      ios            • com.apple.CoreSimulator.SimRuntime.iOS-14-3 (simulator)
    • macOS (desktop)            • macos                                    •
      darwin-x64     • Mac OS X 10.15.4 19E2269 darwin-x64
    • Chrome (web)               • chrome                                   •
      web-javascript • Google Chrome 93.0.4577.82


• No issues found!

[[✓] Flutter (Channel master, 2.6.0-12.0.pre.1002, on Mac OS X 10.15.4 19E2269
    darwin-x64, locale en-GB)
    • Flutter version 2.6.0-12.0.pre.1002 at
      /Users/dhs/documents/fluttersdk/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 9de8849905 (3 hours ago), 2021-12-12 20:54:15 -0500
    • Engine revision f16c96c31f
    • Dart version 2.16.0 (build 2.16.0-85.0.dev)
    • DevTools version 2.9.1
    
[✓] Android toolchain - develop for Android devices (Android SDK version 30)
    • Android SDK at /Users/dhs/Library/Android/sdk
    • Platform android-30, build-tools 30.0.3
    • ANDROID_HOME = /Users/dhs/Library/Android/sdk
    • Java binary at: /Users/dhs/Library/Application Support/JetBrains/Toolbox/apps/AndroidStudio/ch-0/202.7486908/Android
      Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)
    • All Android licenses accepted.    

[✓] Xcode - develop for iOS and macOS
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 12.5.1, Build version 12E507
    • CocoaPods version 1.10.1

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.1)
    • Android Studio at /Users/dhs/Library/Application Support/JetBrains/Toolbox/apps/AndroidStudio/ch-0/202.7486908/Android
      Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build
      1.8.0_242-release-1644-b3-6915495)        

[✓] VS Code (version 1.57.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.21.0

[[✓] Connected device (4 available)
    • Darshan's iphone (mobile)  • 21150b119064aecc249dfcfe05e259197461ce23 •
      ios            • iOS 14.4.1 18D61
    • iPhone 12 Pro Max (mobile) • A5473606-0213-4FD8-BA16-553433949729     •
      ios            • com.apple.CoreSimulator.SimRuntime.iOS-14-3 (simulator)
    • macOS (desktop)            • macos                                    •
      darwin-x64     • Mac OS X 10.15.4 19E2269 darwin-x64
    • Chrome (web)               • chrome                                   •
      web-javascript • Google Chrome 93.0.4577.82


• No issues found!

darshankawar · 2021-12-13T09:02:15Z

/cc @mraleph

mraleph · 2021-12-13T09:11:27Z

AFAIK plugins run in isolates so I would not be surprised that memory jumps considerably when you load additional analyser plugins. AST representation is likely duplicated between main analyser isolate and each plugin.

@bwilkerson @scheglov should know how to diagnose this further.

scheglov · 2021-12-13T16:53:42Z

OK, I also see high memory consumption, by the plugins isolate.

scheglov · 2021-12-13T17:27:02Z

I cannot analyze the heap snapshot when all work is done on plugins, 10 GB seems too much. But a partial snapshot shows some contrast with the main isolate.

Plugins.

Main isolate.

scheglov · 2021-12-13T18:01:38Z

Looking into individual AnalysisDriver instance shows that both have approximately the same size for FileSystemState, but plugins have much more. Specifically:

Each AnalysisDriver in plugins has its own MemoryByteStore, which means that each of them analysis Flutter independently. This wastes both CPU and memory. Ideally we want to use one shared ByteStore instance.
Each driver also has a large portion of the heap used by the element model. This might be something that I affected recently - we started to keep LibraryContext instances even after analysis is done, to support better code completions. But you don't need this in plugins. Or it could also be because of (1) - we have to resynthesize portions of the element model to link more element models, even though we could just resynthisize all of it from bytes without linking at all.

Main:

bwilkerson · 2021-12-13T19:07:44Z

... to support better code completions. But you don't need this in plugins.

Not sure whether it matters, but plugins are allowed to also contribute to code completions.

scheglov · 2021-12-13T19:37:19Z

It probably does not matter for plugins. We have to keep element models to ensure that element instances are mostly the same, so we can use Expando to cache corresponding completion suggestions, which is important only when we suggest a lot - like a few thousand classes in Flutter. Building a few suggestions for expr.^ context should not be very expensive anyway.

scheglov · 2021-12-13T20:33:17Z

Just clearing the libraries contexts does not help at all. We still have references to these elements via cached results for "priority" files. But I don't think this is a recent change in the analyzer. If the plugin always used to configure AnalysisDrivers to set these priority files, we should have always had such high memory consumption. But maybe this issue existed always.

We cache resolved units for priority files because a priority file is a file that the user has open in the IDE. So, as he moves the caret in the file, we want to produce Quick Assists, and don't want to resolve file each time. It is not obvious to me that plugins need such caching.

bwilkerson · 2021-12-13T20:52:42Z

So, as he moves the caret in the file, we want to produce Quick Assists, and don't want to resolve file each time. It is not obvious to me that plugins need such caching.

Probably the most common use cases for plugins are to provide custom lints and custom fixes. It would be better if the plugin didn't need to re-resolve the priority file when the user clicks in a highlighted error range to get quick fixes.

scheglov · 2021-12-13T21:18:24Z

If the plugin does provide quick fixes - yes. But if only diagnostics - no.

Disabling caching priority files in plugins makes it somewhat better, but not totally. We still use a lot of heap, but most of it is garbage.

Before GC:

After GC:

It is unfortunate that VM continues to hold most of this heap.

scheglov · 2021-12-13T21:35:40Z

Using single static MemoryByteStore, shared between all drivers created in the plugin improves heap usage, and also makes it much, much faster (no number). Which is not surprising - we don't have to link large libraries of Flutter 40 times, and usually do it only once.

Before GC:

After GC:

incendial · 2021-12-14T09:00:48Z

@scheglov first of all, thank you so much for the investigation!

Using single static MemoryByteStore, shared between all drivers created in the plugin improves heap usage, and also makes it much, much faster (no number). Which is not surprising - we don't have to link large libraries of Flutter 40 times, and usually do it only once.

Could you please share how to achieve this with the current API? Right now we use ContextBuilder(resourceProvider: resourceProvider); to instantiate the context, which doesn't take anything else and is defined here. Also, how to make MemoryByteStore sharable between all instances, since the plugin is a separate dependency per package? Does they have some shared code too?

Another question - is it possible to share the MemoryByteStore with the main process?

Disabling caching priority files in plugins makes it somewhat better, but not totally. We still use a lot of heap, but most of it is garbage.

I don't know, if it's related or not, but right now we need to use hacks in order to highlight the errors we send to the analyzer in opened files. It was copied from another plugin. Maybe there is a better way to do it?

m-skolnick · 2021-12-14T14:21:55Z

@scheglov For what it's worth. I noticed the high memory consumption early this year. It's not something new to a recent release.

scheglov · 2021-12-14T20:21:48Z

I don't have great understanding of plugins and the APIs that they should be based on. I will leave it to @bwilkerson regarding plugins API. I just want to note that once you step inside the non-API territory with DriverBasedAnalysisContext, you could just as well started using AnalysisContextCollectionImpl that does allow you to set the byteStore.

As for which implementation of ByteStore to use, I used MemoryByteStore just for hack, but it is "infinite", will never flush, which you don't want to do long term. So, MemoryCachingByteStore(NullByteStore) is probably optimal. Using EvictingFileByteStore might let reusing some data between restarts, but might have its own complications.

Nothing of this is API of course.

My understanding of why you need these hacks is that when a file (that was addFile before) is affected by a changeFile, you want to get its resolved unit and produce additional diagnostics. This looks awfully like a lint rule. Which raises a question - why all this is not in linter?

If we stay in a plugin, a more direct way to achieve this is to a callback to AnalysisDriver that is invoked after a file resolution, e.g. void Function(List<CompilationUnit>) onResolvedLibrary. Then results will still contain only ErrorsResults, but this is fine - you will not use it.

incendial · 2021-12-15T09:39:27Z

I just want to note that once you step inside the non-API territory with DriverBasedAnalysisContext, you could just as well started using AnalysisContextCollectionImpl that does allow you to set the byteStore

Unfortunately, we use implementation imports just because there is no other way to create a driver without them. We'd be really happy to use public API instead, but I got your point, I think we can at least give it a try in order to resolve current performance issue.

As for which implementation of ByteStore to use, I used MemoryByteStore just for hack, but it is "infinite", will never flush, which you don't want to do long term. So, MemoryCachingByteStore(NullByteStore) is probably optimal. Using EvictingFileByteStore might let reusing some data between restarts, but might have its own complications.

Thank you, do we need to define it outside of createAnalysisDriver method?

Nothing of this is API of course.

But why? Are you not interested in investing into plugins API or it's just so low on the priority list and you don't have time for it? I mean, I understand that it's barely used right now, but the potential of custom lint rules for a specific team requirements is underestimated right now.

Which raises a question - why all this is not in linter?

You mean, why the functionality that the package provides is not added to the linter package directly, instead of a separate package with custom lint rules?

DanTup · 2022-07-18T17:13:19Z

Are you talking about some specific logging?

I meant if you're able to run the plugin from source, could you add something like File('/foo/bar.txt').writeAsString('info here', mode = FileMode.append); into your code where you're sending these empty arrays to understand why it's happening?

I haven't tested it, but it might be possible to attach a debugger by running the analysis server with the VM Service enabled (using VS Code's "dart.analyzerVmServicePort" setting, and then the Debug: Attach to Dart Process command). If that would be useful (I suspect so), it's probably worth doing some testing and adding some documentation on this.

incendial · 2022-07-18T18:11:33Z

Funny, but creating files worked 😄 . Never tried this before. Okay, I see the problem:

path from analyzeFile: /Users/dimannich/Desktop/code/master-dart-code-metrics/lib/src/analyzer_plugin/analyzer_plugin.dart
context root path: /Users/dimannich/Desktop/code/master-dart-code-metrics
amount of issues: 4

path from analyzeFile: /Users/dimannich/Desktop/code/master-dart-code-metrics/lib/src/analyzer_plugin/analyzer_plugin.dart
context root path: /Users/dimannich/Desktop/code/master-dart-code-metrics/tools/analyzer_plugin
amount of issues: 0

So, it seems to me that it's like one file is being analyzed by 2 contexts and the second one overrides the result of the first analysis. I'd say that behaviour is unexpected and looks like the analysis context from tools/analyzer_plugin should not be used to analyze files not in the tools/analyzer_plugin.

incendial · 2022-07-18T18:13:25Z

And I still get the plugin removed by the analysis server after some time... 😞

scheglov · 2022-07-18T18:16:21Z

Could be dart-lang/sdk#49404

incendial · 2022-07-18T18:19:59Z

Could be dart-lang/sdk#49404

Actually, yeah, looks like it, thanks! Any ideas why the plugin might be removed after few minutes? We also faced a problem with the files not being analysed when they are not open, but we're collecting more info on that.

incendial · 2022-07-18T18:21:33Z

@DanTup thank you for the help!

scheglov · 2022-07-18T18:26:06Z

Could be dart-lang/sdk#49404

Actually, yeah, looks like it, thanks! Any ideas why the plugin might be removed after few minutes?

Not really, maybe https://github.com/dart-lang/sdk/blob/f6aaa72e58b0d11d4ae9d84037bbacc17807e7d5/pkg/analysis_server/lib/src/plugin/plugin_manager.dart#L881

Maybe @bwilkerson can remember when else plugins might be unloaded.

Maybe you could hack the plugin (right in the pub cache), and write into a file all notifications that are sent to the server. This way you could see which exception happens.

incendial · 2022-09-12T18:24:44Z

After giving the new api to our users we received a report that the problem reported here: #90868 (with multiple plugin instances after pub get) has returned 😞:

1 run: 14.35 GB
2 run: 15.52 GB
3 run: 23.24 GB
4 run: 27.40 GB
5 run: 31.77 GB

@scheglov any chance you can take a look?

incendial · 2022-09-13T07:08:23Z

After giving the new api to our users

Also received reports about flickering, now the plugin is not removed, but flickers or after some time just shows no errors.

Another problem that people started to report the issues from the newer package version when they haven't yet updated. Like they have DCM 4.17.0, but IDE somehow reports an issue from 4.18.0. I'm not sure, if it's even possible, but it's already two reports.

scheglov · 2022-09-13T21:15:43Z

It looks that we have a slightly different problem now.

As you might see below (I had to make it small), there are EvictingFileByteStore._cacheCleanUpFunction isolates stuck running. The reason for this is that EvictingFileByteStore is not supposed to be used by clients, IIRC it has no way to exit.

incendial · 2022-09-14T07:13:08Z

It looks that we have a slightly different problem now.

As you might see below (I had to make it small), there are EvictingFileByteStore._cacheCleanUpFunction isolates stuck running. The reason for this is that EvictingFileByteStore is not supposed to be used by clients, IIRC it has no way to exit.

Oh... Do we need to call the cleanup from the plugin somewhere? We also use EvictingFileByteStore for our commands, do we have the same problem there?

Thank you for looking into that

JELaVallee · 2022-09-14T14:21:20Z

Oh... Do we need to call the cleanup from the plugin somewhere? We also use EvictingFileByteStore for our commands, do we have the same problem there?

@incendial just ran into this same issue in a private custom-lint plugin that was also built with EvictingFileByteStore and when it was scanning a project with >40 root packages.

What we were observing: When dart language-server is running either in VSCode or Intellij/Android Studio, everytime we ran flutter pub get we would see ~5GB memory loss. Could easily get to ~30GB mem utilization just switching branches and refreshing pub a couple of times.

We fixed it by changing our MemoryCachingByteStore initialization to use FileByteStore as @scheglov noted:

    return MemoryCachingByteStore(
      FileByteStore(
        stateLocation.path,
        tempNameSuffix: DateTime.now().millisecondsSinceEpoch.toString(),
      ),
      memoryCacheSize,
    );

Now what we're seeing is that initial analysis after opening the IDE still adds about 4-5GB to the memory utilization of the dart language-server proc, but we can see it properly GC'ing the LRU cache in MemoryCachingByteStores when they reach their memoryCacheSize max. 😄

flutter-triage-bot · 2023-07-20T02:56:06Z

This issue is marked P1 but has had no recent status updates.

The P1 label indicates high-priority issues that are at the top of the work list. This is the highest priority level a bug can have if it isn't affecting a top-tier customer or breaking the build. Bugs marked P1 are generally actively being worked on unless the assignee is dealing with a P0 bug (or another P1 bug). Issues at this level should be resolved in a matter of months and should have monthly updates on GitHub.

Please consider where this bug really falls in our current priorities, and label it or assign it accordingly. This allows people to have a clearer picture of what work is actually planned. Thanks!

a-siva · 2023-07-24T16:36:57Z

//cc @srawlins

srawlins · 2023-07-24T17:37:14Z

I believe we can close this. This thread does not represent a single P1 issue. There is not a clear exit state. Much of the changes have been made to one Dart analyzer plugin. We've mitigated issues that arise from multiple Dart analyzer plugins by limiting to one. We're working on a new plugin API. I just don't think there is much to track here.

github-actions · 2023-08-08T06:00:59Z

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

darshankawar added the in triage Presently being triaged by the triage team label Dec 13, 2021

incendial mentioned this issue Dec 14, 2021

[BUG] Extreme CPU and Memory Usage dart-code-checker/dart-code-metrics#528

Closed

incendial mentioned this issue Sep 13, 2022

Analyzer uses too much memory when there is many lints enabled dart-lang/sdk#41793

Closed

incendial mentioned this issue Oct 7, 2022

[Question] Analyzer version update dart-code-checker/dart-code-metrics#1010

Closed

incendial mentioned this issue Oct 14, 2022

[Analyzer plugin] Only the latest version of the plugin is installed, despite the version in pubspec dart-lang/sdk#50209

Closed

ValentinVignal mentioned this issue Feb 13, 2023

High memory consumption Dart-Code/Dart-Code#4380

Closed

nhannah mentioned this issue May 24, 2023

dart:analysis_server takes 19GB memory dart-lang/sdk#52447

Closed

flutter-triage-bot bot added multiteam-retriage-candidate labels Jul 7, 2023

flutter-triage-bot bot removed the triaged-desktop label Jul 20, 2023

gspencergoog added team-engine Owned by Engine team and removed team-desktop labels Jul 20, 2023

a-siva closed this as completed Jul 24, 2023

darshankawar added the r: solved Issue is closed as solved label Jul 25, 2023

github-actions bot locked as resolved and limited conversation to collaborators Aug 8, 2023

[desktop] High memory consumption [Part 2] #95092

[desktop] High memory consumption [Part 2] #95092

Comments

ivk1800 commented Dec 11, 2021 • edited Loading

ivk1800 commented Dec 11, 2021

incendial commented Dec 12, 2021

incendial commented Dec 12, 2021

darshankawar commented Dec 13, 2021

darshankawar commented Dec 13, 2021

mraleph commented Dec 13, 2021

scheglov commented Dec 13, 2021

scheglov commented Dec 13, 2021

scheglov commented Dec 13, 2021

bwilkerson commented Dec 13, 2021

scheglov commented Dec 13, 2021

scheglov commented Dec 13, 2021

bwilkerson commented Dec 13, 2021

scheglov commented Dec 13, 2021

scheglov commented Dec 13, 2021

incendial commented Dec 14, 2021 • edited Loading

m-skolnick commented Dec 14, 2021

scheglov commented Dec 14, 2021

incendial commented Dec 15, 2021 • edited Loading

DanTup commented Jul 18, 2022

incendial commented Jul 18, 2022 • edited Loading

incendial commented Jul 18, 2022 • edited Loading

scheglov commented Jul 18, 2022

incendial commented Jul 18, 2022 • edited Loading

incendial commented Jul 18, 2022

scheglov commented Jul 18, 2022

incendial commented Sep 12, 2022

incendial commented Sep 13, 2022

scheglov commented Sep 13, 2022

incendial commented Sep 14, 2022

JELaVallee commented Sep 14, 2022

flutter-triage-bot bot commented Jul 20, 2023

a-siva commented Jul 24, 2023

srawlins commented Jul 24, 2023

github-actions bot commented Aug 8, 2023

ivk1800 commented Dec 11, 2021 •

edited

Loading

incendial commented Dec 14, 2021 •

edited

Loading

incendial commented Dec 15, 2021 •

edited

Loading

incendial commented Jul 18, 2022 •

edited

Loading

incendial commented Jul 18, 2022 •

edited

Loading

incendial commented Jul 18, 2022 •

edited

Loading