["BUG"] Foam Graph uses inappropriate amount of CPU

[fmagin]

Describe the bug

As soon as I open the Foam Graph two electron processes starts hogging a questionable high amount of CPU performance. On a workspace with around a dozen notes(i.e. nodes in the graph) they are both between 10% and 15% while on a workspace with ~330 notes one is at roughly 65-75% and the other at a quite stable ~24-26%. Numbers according to htop and on an AMD Ryzen 5 3600

This is without any interaction with the graph and even if VS Code is on a workspace that isn't even being displayed

To Reproduce

1. Check a process manager like htop for the current list of processes requiring the most CPU performance
2. Open VS Code with your Foam Workspace (which should not yet show a CPU load spike)
3. Open the Foam Graph and probably observe two new processes hogging roughly an entire CPU core between them

Expected behavior

Not hog an absurdly high amount of CPU performance. The Markdown Links graph requires a similar amount of CPU when opening it and occasionally during panning, but this returns to baseline shortly after again.

Additional Notes

I assume this is something with the interactivity and live update feature. Potentially this could be deactivated via an option until the exact cause of this issue is found.

[jmg-duarte]

I tried to reproduce and wasn't able, is the repo in question private or would you be able to share it as a means to find out more?

[everydayanchovies]

I have the same issue on all of my projects.

[jmg-duarte]

I really need more information to work on this @everydayanchovies @fmagin

[fmagin]

Can you tell me what you would do to investigate this? I doubt that the specific structure of the graph is the issue, but I might be able to dump the structure of the graph myself, or just attach some profiler to it to find out which functions are actually hogging all that CPU time. I just don't know which profiler is a good choice for that, but I'd expect that to be about as much work for me as cleaning and sharing my repo in a way that doesn't expose private information

[jmg-duarte]

Given, that the repo is private, if you could provide a view over the VSCode processes running when the graph is up and profile with the devtools that would be a great starting point.
If you have a public repo suffering the same issue, that would be great too.

Also please provide details over OS, VSCode version (numbered and "edition" - e.g. VSCodium, etc) and number of notes in your graph (if possible)

[everydayanchovies]

I get it, how can I profile this? I just cloned the foam bubble template repo and opened it without modification, opened the graph, let it rest and checked cpu usage with htop. All four cores showed 9% usage from primarily vs code. Closing the graph made the cpu usage go down to 0% to 1% across all cores. This is reproducible 100% of the time.

CPU: 26.8% 2:34.37 /usr/share/code/code --type=renderer --disable-color-correct-rendering --no-sandbox --field-trial-handle=17570122315750226076,12696500128648274180,131072 --enable-features=WebComponentsV0Enabled --disable-features=Cooki
CPU: 28.5% 1.0 2:53.46 /usr/share/code/code --type=renderer --disable-color-correct-rendering --no-sandbox --field-trial-handle=17570122315750226076,12696500128648274180,131072 --enable-features=WebComponentsV0Enabled --disable-features=Cooki
CPU: 6.4% 0.8 3:12.36 /usr/share/code/code --type=gpu-process --field-trial-handle=17570122315750226076,12696500128648274180,131072 --enable-features=WebComponentsV0Enabled --disable-features=CookiesWithoutSameSiteMustBeSecure,SameSiteByDefa
CPU: 5.5% 1.0 0:27.84 /usr/share/code/code --type=renderer --disable-color-correct-rendering --no-sandbox --field-trial-handle=17570122315750226076,12696500128648274180,131072 --enable-features=WebComponentsV0Enabled --disable-features=Cooki

These are all processes that get spawned with the graph (all at once). Closing the graph removes these processes.

Let me know how I should profile this.

[everydayanchovies]

Sorry, I see you already mentioned using the dev tools.

[fmagin]

Just checked the dev tools, It seems like the Rasterizer Threads are the main issue, they start to appear as a solid green around the second I open the Graph view.

[everydayanchovies]

Here is a profile where I started with the graph closed, opened the graph and waited, and closed the graph.
Profile-20210214T162714.zip

[jmg-duarte]

@everydayanchovies is that graph from VSCode running only the Foam extension(s)?
If not please try again only with the Foam recommended extensions enabled.

[everydayanchovies]

I disabled everything but the recommended extensions and repeated my test, same results. I then disabled everything but foam and repeated, no difference. I really think this is a bug in the graph, could you point me to the code that is responsible for rendering it? I would like to take a look @jmg-duarte , maybe I could help.

[everydayanchovies]

If I had to guess I would say the janitor runs too often (does it run on every file change? Maybe that gets triggered every couple of millis due to some bug), or the graph renderer is refreshing the view constantly. I'm tight on time with other projects going on but I did try to get a feel for the codebase, maybe I can look into this later but some pointers would be appreciated. Is this not reproducible on your end?

[jmg-duarte]

From the Chrome traces, the CPU seems to suffer the most during the opening of the graph, which would make sense.
For my repos, the issue isn't reproducible, hence asking about the system, VSCode versions, graph size and extensions.

I do have an idea though, since the graph runs in WebGL, are you running your project with an integrated GPU?
Regarding the start point, things relating the graph itself are in https://github.com/foambubble/foam/tree/master/packages/foam-vscode/static
Since the graph is a webview I am not sure how it appears in the profiler.
What I'd try first would be simply upgrade the dependencies

• force-graph (https://github.com/vasturiano/3d-force-graph)
• d3 (https://github.com/d3/d3/releases)

[fmagin]

In my case it definitely is not running on an integrated GPU, because I don't have one. CPU is a Ryzen 5 3600, GPU is an RX 5700 XT.

One interesting thing I just noticed: When the window shows the full graph of around 900 nodes, the CPU hogging process with the argument --type=renderer is at slightly above 100% CPU, and the --type=gpu-process is at ~36%.
But if I zoom into some emtpy part of the graph, e.g. outside the the circle of all nodes, this drops to ~80% of for the --type=renderer process and ~10% for the --type=gpu-process.

I agree with @everydayanchovies suspicions that this is the graph constantly being redrawn for some reason.

My specific system information is:
Arch Linux with the code 1.53.1-1 package
Graph Size is around 900 nodes.
I am fairly sure this isn't related to extensions, especially because @everydayanchovies tests also showed the issue with all other extensions disabled.

[fmagin]

Other weirdness: The processes are still taking up the same amount of CPU, even if I pause the main thread via the developer tools. So I guess it can't be something in the main thread that keeps updating the graph? But I know too little about electron in general to make much of this.

[jmg-duarte]

@fmagin could you post a print of the VSCode process explorer? Not the system one, the one VSCode provides.
Sorry for asking so much info, its hard to debug the issue without being able to reproduce.

[fmagin]

Quick screenshot, because I don't have much time right now.
The thing is: The VSCode process explorer doesn't show any CPU load, probably because the relevant process isn't even listed there. I have some suspicions that we might be dealing with a bug in the render itself that is triggered by the Foam graph in specific circumstances, maybe some WebGL issue on specific platforms? What are the details of your system where you can't reproduce it?

[jmg-duarte]

I'm running two systems, both unable to reproduce.
My main system has an i7-6700K, 16GB RAM and a AMD RX 480, I'm running Ubuntu 20.04.2 on it.
My laptop runs on a i7-8550U and the respective integrated GPU (though it has a NVidia MX150 it's not configured), this runs Arch Linux.

I just noticed something. You're running Code OSS and not VSCode (please do make this distinction in the future).
I was able to "almost" reproduce the issue with Code OSS though it only reached 80% with this repo as a graph.
I have some ideas to address the problem which I'll try out ASAP
In the meantime the following links might be of some help:
SO: https://stackoverflow.com/questions/47315688/how-can-i-debug-high-cpu-usage-in-electron
VSCode Issue (not the same thing but possibly related): microsoft/vscode#22900

I suspect that the best we can do is to get another graph library but we'll see

[everydayanchovies]

I’m using the regular VS code package as supplied from Microsofts website, IIRC. Also, I did see the cursor bug you linked earlier and concluded that it did not effect me, but that’s probably because they resolved it already. I assume the same underlying cause could be causing this bug. I will check my VS code version ASAP, am currently afk.

[fmagin]

@jmg-duarte what do you mean by "another graph library"? I think the underlying graph library is not the issue, but the actually rendering itself. But maybe this is still the same library in the case of foam.
I think the SO issue looks like a good lead, the graph getting redrawn (and during that rasterized) at 60 FPS sounds like it could cause the issue.

I grabbed the code-stable-x64-1613044905.tar.gz from the official download page, and I still have the same issue with the same CPU load percentages, so this isn't just an issue with Code OSS. Interesting that you can reproduce this with Code OSS and not VS Code though. Maybe some default config issue?

[riccardoferretti]

is this happening across computers and repos? that is:

• can you replicate the same issue with another repo on the same computer?
• can you replicate the same issue on another computer with the same repo?

Also:

• how many notes are in the repo?
• I have seen a performance hit due to very big notes (in the MBs), do you have any such monster in your repo?

The graph is continuously rendered because the physics engine updates the location of nodes even when there are no changes. Still, I have seen this handle a ton of nodes, so I am really surprised by the issue. Can you check how your computer manages the following:

• https://vasturiano.github.io/force-graph/example/basic/
• https://vasturiano.github.io/force-graph/example/large-graph/

FYI this is the library we use: https://github.com/vasturiano/force-graph
In case you wanna try with more examples.

[fmagin]

Yes, this does happen both on my laptop and my desktop with the same repo. And it also is an issue with multiple repos on my desktop, haven't checked another repo on my laptop yet.

The CPU load is actually quite similar when I open the two graphs you provided with chromium. The basic graph has the render process at 60%, and the gpu-process at 25%, the large graph at 108% and 20%. So I guess this is a library issue after all and not even related to electron itself. This is also an issue in Firefox, though the specific process names are different.

[jmg-duarte]

If the issue was regarding VSCode/Electron/Chromium I'm quite sure that it would've been noticed and fixed already, since we are not the only users of such graph libraries.
To further discuss the point, the rendering pipeline is not managed by us, at all. All Foam does is process the documents, turn it into a graph model and send it to the library to render. We're responsible for the sent data, not how it gets rendered.
I doubt upgrading the library has an impact since the huge graph Riccardo posted also lags on me (though that is to be expected), this is of course if the website is using the latest version.

Regarding the "redraw" issue. The graph has to be drawn to screen every frame, that's just how rendering works. Now the work done for each frame, can and should be adjusted. If the graph is loaded and rendered every frame, that is an issue, since it should only be loaded once.

Finally, about the library, I question if we need "forces" for the graph. It's cool but we also don't really take advantage of it.
For comparison, Cytoscape has this large (not huge) graph that gets loaded very quickly http://www.wineandcheesemap.com/
Moving nodes example https://js.cytoscape.org/demos/colajs-graph/

[fmagin]

Well yes, it has to be redrawn for each frame but not rendered for each frame. But this is splitting hairs with terminology that I am not that familiar with either.
If the position of the nodes didn't change you could cache the entire image, for animations you could cache the base image and apply the animation on top of it, e.g. the flow in the direction of the edge). If the position of the nodes actually changes constantly then this would definitely be a questionable feature, I would assume that there is some minimum amount of force that just gets rounded down to zero and then the graph can be considered stable until the user drags a node around.

The thing is that the CPU load for the huge graph Riccardo linked and my medium size graph of ~1000 nodes in Foam is basically the same, ie. between 105% and 110% for the render process.

There is an open issue about performance here vasturiano/force-graph#169 so it might be worth to move this discussion to that or another issue of the graph layout/rendering library.

[everydayanchovies]

I agree on every one of your points @jmg-duarte , also note that the colajs demo loads very smoothly on my iPhone but stutters significantly on the machine I tested this issue on. There may be something wrong with my graphic acceleration?
I'm using i3wm with compton on a standard Fedora install, and I checked to see if direct rendering is working:
glxinfo | grep "direct rendering"
direct rendering: Yes

[fmagin]

For what it's worth I am also using i3wm (mostly without compton though). Direct rendering is also enabled/supported.

[everydayanchovies]

However, once the movements settle the graph is stationary (obviously), and CPU usage goes down to ~1%. So I would prefer to see colajs used seeing as the forces between nodes and the subtle movements are not really adding anything of value. I'll try to implement it this weekend and see how far I get.

[jmg-duarte]

So I tried this on two other setups, one laptop running XMonad and a desktop running Windows.
The XMonad one showed an high CPU load while Windows did it without hiccups.

I'm wondering how platform specific this might be

[sfradkin]

I can replicate this on my private repo. The markdown links view shows 280 files with 2301 links. Mac OS 10.15.7. Using VSCodium 1.53.2. As soon as I open the Foam Graph view I get two processes that together consume 120-150% of the CPU. They’re labeled VSCodium Helper (Renderer) and VSCodium Helper (GPU).

[everydayanchovies]

I started implementing colajs but didn't have time to finish and I didn't make any progress, I mostly struggled with learning VS code development and typescript. I'll come back to it when I have more time.

[jmg-duarte]

@everydayanchovies Me and @riccardoferretti thought about it but considered that colajs would not be viable in large repos such the one @fmagin has