[RFC] Add support for UFFD backed memory when restoring Firecracker VM from snapshot #1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Memory::restore now takes an optional &File and creates file-backed memory or anonymous memory depending on the option. Signed-off-by: Adrian Catangiu <[email protected]>
Add new 'mem_backend_type' parameter that takes either:
- File
- UffdOverUDS
as valid values.
Rename 'mem_file_path' to 'mem_backend_path'. Interpretation of this
field depends on the value of 'mem_backend_type':
- Path to file that contains the guest memory to be loaded,
- Path to UDS where a custom page-fault handler process is listening
and expecting a Uffd to be sent by Firecracker. The Uffd is used
to handle Firecracker's guest memory page faults in this separate
process.
Add plumbing for the new API, the Uffd path is only a stub for now.
Signed-off-by: Adrian Catangiu <[email protected]>
When /snapshot/load specifies memory backend type as 'UffdOverUDS', Firecracker doesn't handle the memory file itself anymore and expects an external process to handle its guest memory page faults. To do this, anonymous memory is mmapped as guest memory while keeping the original memory regions shape. Then a Uffd is created and each guest memory range is registered with the Uffd so that any page faults won't be handled by the kernel, but will come up as events on the Uffd. Firecracker then sends the memory ranges descriptions/mappings along with the Uffd over a UnixDomainSocket specified in 'mem_backend_path' parameter on the API call. It is expected that on the other side there is already a process listening for incoming connections. Once Firecracker's connection is accepted, Firecracker sends the mappings and Uffd. The receiving process is from now responsible for handling any pagefaults on the Uffd. The communication medium is a UDS, the protocol is SOCK_STREAM and the encoding is JSON. Signed-off-by: Adrian Catangiu <[email protected]>
In the happy case we can close the fd since the other process has it open and is using it to serve us pages. The problem is that if other process crashes/exits, firecracker guest memory will simply revert to anon-mem behavior which would lead to silent errors and UB. If the fault serving process exits while we hold a copy of the FD as well, the uffd will still be alive but with no one to serve faults leading to guest freeze (which is an unfortunate, but explicit and defined behavior). Use mem::forget() to keep the uffd open in this process as well. Signed-off-by: Adrian Catangiu <[email protected]>
Add an example process that handles page-faults for Firecracker after a load snapshot operation. This simple pf-handler simply populates full memory regions on page-faults belonging to those regions. Signed-off-by: Adrian Catangiu <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reason for This PR
Allows Firecracker customers to better manage the VMs memory.
Description of Changes
Add new 'mem_backend_type' parameter that takes either:
as valid values.
Rename 'mem_file_path' to 'mem_backend_path'. Interpretation of this field depends on the value of 'mem_backend_type':
When /snapshot/load specifies memory backend type as 'UffdOverUDS', Firecracker doesn't handle the memory file itself anymore and expects an external process to handle its guest memory page faults.
To do this, anonymous memory is mmapped as guest memory while keeping the original memory regions shape. Then a Uffd is created and each guest memory range is registered with the Uffd so that any page faults won't be handled by the kernel, but will come up as events on the Uffd.
Firecracker then sends the memory ranges descriptions/mappings along with the Uffd over a UnixDomainSocket specified in 'mem_backend_path' parameter on the API call.
It is expected that on the other side there is already a process listening for incoming connections. Once Firecracker's connection is accepted, Firecracker sends the mappings and Uffd. The receiving process is from now responsible for handling any pagefaults on the Uffd.
The communication medium is a UDS, the protocol is SOCK_STREAM and the encoding is JSON.
Example client also added
Add an example process that handles page-faults for Firecracker after a load snapshot operation.
This simple pf-handler simply populates full memory regions on page-faults belonging to those regions.
License Acceptance
By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license.
PR Checklist
[Author TODO: Meet these criteria.][Reviewer TODO: Verify that these criteria are met. Request changes if not]git commit -s).unsafecode is properly documented.firecracker/swagger.yaml.CHANGELOG.md.