Tree shaking / reducing custom system images

[tshort]

I'd like to generate a custom system image that's pared down to just the code needed to run a script. This is mainly for standalone ahead-of-time compilation. This is based on discussions from #32273.

I tried a simple approach using the following:

~/julia/julia -J ~/julia/usr/lib/julia/sys.ji --output-jit-bc t.bc ~/tmp/t.jl

That's promising because it generated a t.bc with approximately the methods needed to run the code in t.jl. But, it's a .bc file and not a .o file, and it's missing stuff needed for the system image.

I also tried making a custom C function that does part of what the command above does. It segfaults creating the system image data.

JL_DLLEXPORT void jl_outputjit(void)
{
    ios_t *s = jl_create_system_image();
    jl_dump_native("test.bc",
                   "testunopt.bc",
                   "test.o",
                   (const char*)s->buf, (size_t)s->size);
}

Is the approach above the right direction to start. Or, does anyone have more suggestions on how to go about implementing "tree shaking"?

[cdsousa]

I'm not sure, but that may also be taken into account in the WIP PackageCompilerX.jl.

[KristofferC]

Where did you read that? Please don't add expectations to someone else's package like that.

[cdsousa]

I'm sorry @KristofferC, my English was not clear. I meant that what Tom Short is seeking here is something that would be nice to also have in your WIP package (that I, may wrongly, though is a kind of a PackageCompiler 2 tentative), not that it is expected there. I'm not sure it is even possible or desirable though.

[KristofferC]

Sorry if my reply came of as harsher than I intended. My point was just that it is a hard problem and I don't want people to get too high expectations that I cannot uphold.

I agree with you that a solution to this would work very well in a package like the one you linked :).

[aminya]

I think tree-shaking at Julia level would be useful too. People are already requesting loading a subset of a package (https://discourse.julialang.org/t/is-it-possible-to-load-a-subset-of-a-package/31136)

For a target package:

we can write a function that parses the source code of a package and finds the dependencies that are used (methods that are called from dependencies and their related types and module global variables), and then bundles those (inline) into the target package.

References from other languages:

https://github.com/rollup/rollup#tree-shaking : Requires the user to use explicit ES modules import to detect the functions/types that are used, and then only bundles those:

import { aFunction, aType} from "aPackage"

In Julia, we can do a better job by automatically detecting these instead of using explicit import, since in Julia the original module of each function or types is known.

[brenhinkeller]

I wonder if we can consider this partly solved by https://github.com/tshort/StaticCompiler.jl nowadays?