Cap root module loading via worker pool #256
Conversation
| @@ -40,9 +43,14 @@ func NewRootModuleManager(fs tfconfig.FS) RootModuleManager { | |||
|
|
|||
| func newRootModuleManager(fs tfconfig.FS) *rootModuleManager { | |||
| d := &discovery.Discovery{} | |||
|
|
|||
| defaultSize := 3 * runtime.NumCPU() | |||
There was a problem hiding this comment.
3 is the magic number I'm guessing?
There was a problem hiding this comment.
3 is basically an educated guess. I did some simple testing as shown above with 1000 root modules in a hierarchy (which seemed to be the higher end of range reported by folks in #186 ). That effectively triggers terraform version 1000 times and I ran that on my i7 (Macbook) with 4 cores (= pool size 12) and it finished within a few seconds.
I don't expect this to be the best configuration nor the best way of testing this, but I do expect it to behave better than before just because we trade time+memory for less CPU in principle.
I do expect we will tweak the number in the near future based on the feedback from end users and further (more scoped) testing. This is also why I added the logging.
There was a problem hiding this comment.
Generally the performance will largely depend on the time it takes to finish the loading, which after terraform version depends on the size of the schema and exec time of terraform providers schema -json alone may span from half a second to 10+ seconds in some worst case scenarios, so there is no easy answer to this I'm afraid.
We can however make it configurable and let users deal with edge cases (too slow CPU and/or too many modules) that way. Also it would be even better to just avoid calling terraform version where we don't need to as discussed in #186 (comment)
|
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. |
ghost
locked and limited conversation to collaborators
This represents a foundation for resolving #186
A capped worker pool is used when loading root modules, which helps spread the CPU usage and effectively flatten the spikes and therefore improve the UX.
Charts generated via https://github.com/mkevac/debugcharts
CPU Before
CPU After
pprof Before
pprof After
Deferred ideas
We can also make parallelism configurable, if deemed necessary, but I think that can be discussed and addressed separately.