feat: on low wasm memory hook

[mraszyk]

This proposal is based on this forum post and has already been approved by motion proposal 106146.

Canister developers have to actively monitor their canisters to know if they are low on wasm memory. If detected too late, a canister can be completely stuck and forever un-upgradable.

To address this, we introduce a hook called on_low_wasm_memory. When defined, it is triggered whenever the canister's memory usage exceeds some developer-defined threshold.

[mraszyk]

If the canister gets frozen immediately after the function is scheduled for execution, the function will run once the canister's unfrozen if the canister's wasm memory remains above the threshold t.

This property is not established by the implementation.

[mraszyk]

The implementation only considers stable memory and ignores, e.g., the chunk store and snapshot size, when deriving the wasm memory capacity.

[mraszyk]

The hook should not run after an upgrade/reinstall/uninstall/install if the condition is not satisfied after the upgrade (although the condition was satisfied before the upgrade and the hook did not execute yet).

[github-actions]

🤖 Here's your preview: https://xaml7-fiaaa-aaaam-abdka-cai.icp0.io

[dragoljub-duric]

Regarding the point:
"The hook should not run after an upgrade/reinstall/uninstall/install if the condition is not satisfied after the upgrade (although the condition was satisfied before the upgrade and the hook did not execute yet)."
Because the hook condition is checked whenever the canister tries to grow wasm memory, can it ever happen that the canister does not try to grow wasm memory when running upgrade/reinstall/uninstall/install?

[mraszyk]

Because the hook condition is checked whenever the canister tries to grow wasm memory, can it ever happen that the canister does not try to grow wasm memory when running upgrade/reinstall/uninstall/install?

Yes, e.g., if the canister does not have any pre- and post-upgrade hooks (in which case there's no wasm execution during canister installation).

[mraszyk]

It should indeed be rephrased, but I'd suggest: if the canister's remaining wasm memory size in bytes remains below the threshold t.

[dragoljub-duric]

In slack conversation:

Luc Bläser
Scenario:
Canister sets memory threshold at 1 GB.
Canister free memory is 0.99 GB -> hook is triggered
Now, canister sets memory threshold to 0.5 GB, free memory is still 0.99 GB
t1 > t remaining Wasm memory is greater than the Wasm memory threshold
And now canister allocates more memory, until 0.49 GB free memory
t2 > t1 remaining wasm memory falls below Wasm memory threshold
Shouldn't the hook now be retriggered?

Dragoljub Duric
That is a very good example. And I think that case may be missing. The reason is to check for the hook condition to be added after every memory_grow operation or after install/reinstall/uninstall/upgrade of the canister. But in this specific case, when memory is 0.99 GB hook will be executed and the status will be Executed since we do not have to check for the hook condition when we set a new threshold the status will remain Executed. Finally when additional memory is requested and the remaining memory becomes 0.49 GB, the condition for the hook will be satisfied, but because the status of the hook is already Executed we will not execute it again. It looks like we will need to add checking for the hook condition after we update wasm_memory_threshold.

Luc found out about this mistake, and I am going to implement the fix for it. After the wasm_memory_threshold is updated, I will check and update the hook condition, but I am between two alternatives for this:

1. Use the old hook status and the result of the checking for the hook condition with the new wasm_memory_threshold , i.e. update the hook status the same way as we do in any other case
2. Use only the result of the checking for the hook condition with the new wasm_memory_threshold, so the hook status after the update of wasm_memory_threshold can be either ConditionNotSatisfied or Ready. I believe this alternative will require spec changes.

The difference can be observed in the case if, before the update of the wasm_memory_threshold hook status was Executed, the old wasm_memory_threshold was 3GB, remaining_wasm_memory was 1GB, and the new wasm_memory_threshold is 2GB. The former approach will keep the hook status to Executed, while the latter will set the hook status to Ready.

I am leaning towards the latter option but I do not have a strong opinion. Do you have an opinion on this?

[mraszyk]

In slack conversation:

Luc Bläser
Scenario:
Canister sets memory threshold at 1 GB.
Canister free memory is 0.99 GB -> hook is triggered
Now, canister sets memory threshold to 0.5 GB, free memory is still 0.99 GB
t1 > t remaining Wasm memory is greater than the Wasm memory threshold
And now canister allocates more memory, until 0.49 GB free memory
t2 > t1 remaining wasm memory falls below Wasm memory threshold
Shouldn't the hook now be retriggered?
Dragoljub Duric
That is a very good example. And I think that case may be missing. The reason is to check for the hook condition to be added after every memory_grow operation or after install/reinstall/uninstall/upgrade of the canister. But in this specific case, when memory is 0.99 GB hook will be executed and the status will be Executed since we do not have to check for the hook condition when we set a new threshold the status will remain Executed. Finally when additional memory is requested and the remaining memory becomes 0.49 GB, the condition for the hook will be satisfied, but because the status of the hook is already Executed we will not execute it again. It looks like we will need to add checking for the hook condition after we update wasm_memory_threshold.

Luc found out about this mistake, and I am going to implement the fix for it. After the wasm_memory_threshold is updated, I will check and update the hook condition, but I am between two alternatives for this:

1. Use the old hook status and the result of the checking for the hook condition with the new wasm_memory_threshold , i.e. update the hook status the same way as we do in any other case
2. Use only the result of the checking for the hook condition with the new wasm_memory_threshold, so the hook status after the update of wasm_memory_threshold can be either ConditionNotSatisfied or Ready. I believe this alternative will require spec changes.

The difference can be observed in the case if, before the update of the wasm_memory_threshold hook status was Executed, the old wasm_memory_threshold was 3GB, remaining_wasm_memory was 1GB, and the new wasm_memory_threshold is 2GB. The former approach will keep the hook status to Executed, while the latter will set the hook status to Ready.

I am leaning towards the latter option but I do not have a strong opinion. Do you have an opinion on this?

Personally, I prefer the former approach because it is consistent in terms of updating the hook status. If we decide to potentially rerun an already executed hook after updating canister settings, then it is not clear why not rerun an already executed hook also in other special cases.

[dragoljub-duric]

@mraszyk Thank you, that makes sense, I implemented the former approach. (link)