Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes #33214: Set minimum Puma threads equal to maximum puma threads … #984

Merged
merged 1 commit into from
Aug 10, 2021

Conversation

ehelms
Copy link
Member

@ehelms ehelms commented Aug 4, 2021

…by default

This has been shown to prevent memory leaks that can appear in Foreman
when running under Puma. This only changes the default to match and
users must still be aware of this potential when changing the tunings.

Some guidance from Satellite performance team -- https://github.com/RedHatSatellite/satellite-performance-tuning/blob/devel/docs/satellite-configuration-tuning.rst#puma-tuning

Some info from upstream Puma -- puma/puma#2665

There is most likely one or more memory leaks within the application itself leading to this issue. This alleviates the issue for now by pre-allocating the needed memory and preventing spin-up and spin-down of threads that exacerbates the memory leak.

@ehelms
Copy link
Member Author

ehelms commented Aug 6, 2021

The alternative would be to drop the minimum parameter and always force min = max, that requires a major version change. The upside would be that it prevents users from putting themselves accidentally into a situation of having a memory leak. This is a tough challenge as ideally we would just eliminate all the memory leaks and all users to tune things however they want. Finding the leaks is the hard part.

Copy link
Member

@ekohl ekohl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall 👍 for making them match, but I'd prefer to have it dynamic.

@@ -63,7 +63,7 @@
$foreman_service = 'foreman'
$foreman_service_ensure = 'running'
$foreman_service_enable = true
$foreman_service_puma_threads_min = 0
$foreman_service_puma_threads_min = 16
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about we make it dynamic and set it to undef:

Suggested change
$foreman_service_puma_threads_min = 16
$foreman_service_puma_threads_min = undef

Then use $min = pick($foreman::foreman_service_puma_threads_min, $foreman::foreman_service_puma_threads_max). That way users don't need to specify it twice if they want them to match.

You probably also need to change the type to Optional[Integer[0]].

Also an installer migration that clears the answer if they match will be nice for users.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the dynamic aspect from a code perspective. I am wondering if this will confuse users of the installer who look at it and see that the minimum is set to nothing. Asking out loud if that will lead to support posts or users to think that the minimum is therefore 0 if blank.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can document this in the parameter help that the default (undef) means matching max threads.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ehh, I'll update the documentation of the parameter and let that suffice.

@ekohl
Copy link
Member

ekohl commented Aug 6, 2021

Do you recall if min threads set to 1 also works around the leak? Or does matching mean it doesn't rotate out old threads which means the leak doesn't show up?

@ehelms
Copy link
Member Author

ehelms commented Aug 6, 2021

Do you recall if min threads set to 1 also works around the leak? Or does matching mean it doesn't rotate out old threads which means the leak doesn't show up?

I do not know if 1 also works around the leak. I can test this.

My understanding was, that by setting min = max and pre-allocating the memory for each thread, the leak does not occur as there is no spin-up and spin-down.

@ehelms
Copy link
Member Author

ehelms commented Aug 6, 2021

Another note, the Satellite performance team found generally better overall performance when min = max. Further, the Gitlab project sets their defaults to min = max as well for their helm chart (https://docs.gitlab.com/charts/charts/gitlab/webservice/index.html#installation-command-line-options -- search for threads).

@ekohl
Copy link
Member

ekohl commented Aug 6, 2021

Sounds like we should start with min = max then.

@ehelms
Copy link
Member Author

ehelms commented Aug 9, 2021

Failing tests are getting fixed with theforeman/foreman-packaging#7001

Copy link
Contributor

@wbclark wbclark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still requires the update to param documentation at manifests/init.pp#L163

it would be nice also to see an additional unit test for correct behavior of non-default params

@@ -2,6 +2,6 @@
User=<%= scope['foreman::user'] %>
Environment=FOREMAN_ENV=<%= scope['foreman::rails_env'] %>
Environment=FOREMAN_HOME=<%= scope['foreman::app_root'] %>
Environment=FOREMAN_PUMA_THREADS_MIN=<%= scope['foreman::foreman_service_puma_threads_min'] %>
Environment=FOREMAN_PUMA_THREADS_MIN=<%= @min_puma_threads %>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it feels a little funny to me that puppet's erb templating supports @variable in the local scope from where the template is called. is it more efficient in case the scope object is truly huge? consistently using the same syntax would be nicer to look at otherwise.

@wbclark
Copy link
Contributor

wbclark commented Aug 10, 2021

I read the previous note about a migration that would clear the value of the minimum if it matches the maximum. 👍 to that.

I think we should also in that migration clear the value of the minimum if it used the old default of 0. this would alleviate the memory leaks for the vast majority of users not using any non-default params, while still allowing users to set a non-default value of the minimum if they really desire it.

@ehelms
Copy link
Member Author

ehelms commented Aug 10, 2021

I read the previous note about a migration that would clear the value of the minimum if it matches the maximum. to that.

I think we should also in that migration clear the value of the minimum if it used the old default of 0. this would alleviate the memory leaks for the vast majority of users not using any non-default params, while still allowing users to set a non-default value of the minimum if they really desire it.

That's over here -- theforeman/foreman-installer#713

…by default

This has been shown to prevent memory leaks that can appear in Foreman
when running under Puma. This only changes the default to match and
users must still be aware of this potential when changing the tunings.
Copy link
Contributor

@wbclark wbclark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this looks good to me. The new param documentation mentions the leaks and performance benefits of using min = max. It's possible there is some unknown future condition where separately configuring the min to be different from the max would still be desirable.

@ehelms ehelms merged commit 172a16d into theforeman:master Aug 10, 2021
@wbclark wbclark added the Bug label Aug 12, 2021
@wbclark
Copy link
Contributor

wbclark commented Aug 12, 2021

@ehelms I went with 'bug' for now. It's not so much adding a feature that we traditionally think of as an 'enhancement', and the intent behind it is very much 'fix a thing that wasn't working the way it was intended'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants