Startup performance analysis

[jbreitbart]

This is not a bug report, just a collection of measurements I did so they won't get lost. Overall, executing the hello_world binary takes about 131ms with uhyve compiled for release.

Running uhyve with valgrind --tool=callgrind --dump-instr=yes --collect-jumps=yes --simulate-cache=yes shows the following runtime distribuation

• 13% is spent in the runtime linker _dl_start
• 3% is consumed in std::rt::lang_start_internal

When only considering uhyve::main the runtime is spent almost completly (~98%) in vm::Vm::load_kernel. About 7% is used to measure the TSC and 89% is spent in memcpy, some of which are unnecessary as the content from the elf file is copied twice:

• uhyve prepares the elf file so it can be executed in the VM
• the elf crate copies the content of the elf file to its internal data arrays

The later part is unnecessary, as neither uhyve not the functions we use from the elf crate use these data arrays. Removing https://github.com/cole14/rust-elf/blob/master/src/lib.rs#L212-L239 save ~3-4ms with the hello_world example, probably more when running a larger binary.

[jbreitbart]

Goblin added functions to lazy parse elf files. Maybe that can be used to improve the performance. See

• elf.parse: added lazy_parse function m4b/goblin#254
• Use goblin as elf parser and only load segments on demand  occlum/occlum#315