Percentage settings for memory ballast

[pavolloffay]

Is your feature request related to a problem? Please describe.

I want to configure ballast size relative to the total system memory. This is helpful in a dynamic environments like docker/k8s.

Describe the solution you'd like

Add --mem-ballast-size-percentage flag and ballast_size_percentage to memory limiter. This will leave us with _mib settings and percentage settings, the mib having higher precedence.

I don't like that the ballast is being configured in two places - flag and memory limiter. Could we just support the blast in the memory limiter?

Describe alternatives you've considered
We have agreed to implement percentage for the ballast.

Additional context

Related to #1078
Created from #1622 (comment)

[pavolloffay]

@bogdandrutu @tigrannajaryan @pjanotti could you please comment on this one? Whether we want to remove the ballast flags nad move the allocation to the memory limiter. Or the memory limiter could read flag values.

[pjanotti]

Could we just support the blast in the memory limiter?

My intention was to have use the capability of retrieving maximum memory to have the ballast enabled by default due its cpu savings, however, it needs to be configurable. The memory limiter is an orthogonal concern that needs access to the ballast setting to operate correctly and for which I had seem practical cases it which it was applied selectively to different pipelines under same configuration. I don't think it makes sense to move the ballast to the memory limiter.

It seems reasonable to expose an application level setting like the ballast to components in general. We discussed this in the past. Would you like to put a proposal on how to do that?

[tigrannajaryan]

Yes, ideally expose an interface for components to read global application settings. The memory limiter can find out the ballast size and configure itself accordingly.

[mxiamxia]

#2516

[mxiamxia]

this issue can be closed! the request has been addressed in #2516

[pavolloffay]

I think this is done, closing