Asset/Resource embedding?

[azazeal]

Is there (or will there be) a way to embed static assets into the final compiled executable?

Should there be such functionality if the answer to the above is no?

[bcardiff]

There was an experiment about embedding files in compile time so they can be retrieved in runtime.
There was an idea of maybe allowing overrides if a file exists.

It may come in the future or as an external library.

The ultimate goal to have some way to leave the assets in the .rodata section by using constant data, but that was a bit far in the roadmap at that time.

I think this goes beyond web apps assets, but that is clearly a nice use case.

[asterite]

Also note that right now you can do it:

# The right hand side reads the file at compile time and turned it into a string literal.
# The file is *not* needed at runtime.
file_data = {{ `cat #{__DIR__}/README.md`.stringify }}
puts file_data

[azazeal]

My first instinct right now would be to use a macro for this (correct me if I'm wrong as I'm still new to Crystal) but yeah, web assets was what I had in mind: being able to deploy a new version with a single file drop makes you feel really... happy 😄.

The point of the question was to figure out if there's gonna be something other than than macros, etc: something specific for asset bundling.

I was covered with "an external library" / "may come in the future".

[bcardiff]

I did use some hand made embedding together with .ecr to include base64 encoded images in css in compile time. Since ecr are compiled those won't allow hot asset replacement, but it's fun. Maybe that is enough for embedding some assets it what you are trying to do right now.

[azazeal]

I'm not currently trying to embed assets into an app, the whole thing popped in my head as a little while ago I had to embed some not-so-typical-in-size files into an app.

My concern is that if I go @asterite's way (or my way - see macro) then the asset will be loaded in memory from the get-go (even when not needed) hence the ask for specific tooling.

[asterite]

@azazeal I think "embed static assets into the final compiled executable" but also "I don't want the asset to be loaded in memory from the get-go" there's a contradiction. You either embed it in the executable or you load it at runtime. For embedding it you can use the run macro, for loading it lazily you use File or some other thing.

Also note that the compiler already embeds some html and js for doc generation and it's still equally fast but much easier to distribute because those data files live inside the executable.

Can we close this issue?

[jhass]

@asterite You say one can just embed it with the run macro, but I don't see how to do that for arbitrary data, I think we miss a literal for that. Could you provide an example to embed, say, a png file if you think otherwise?

In any case I wouldn't mind a macro that boils it down to logo = embed_file "#{__DIR__}/res/logo.png".

[asterite]

Oh, I see. Yes, we are missing some literals. I'll leave this open then.

[mperham]

Another recent solution https://github.com/schovi/baked_file_system

[mverzilli]

Since a couple of versions ago Crystal allows invalid UTF-8 Strings so unless I'm missing sth we could use Ary's suggestion to simply embed by stringifying.

Should we close this issue?

[jhass]

I still think we should provide some macro as the official API for this, at least to be able to swap what it does with something more efficient or supported if needed, for example if we decide to make string literals valid UTF8 again.

[ysbaddaden]

We say that binary data is supposed to be a Slice(UInt8), so I support @jhass here. Putting data into a String should be a hack, until we proper support.

Note that such support could be a macro that uses a String in the underlying implementation, but always returns a Slice.

[mverzilli]

Cool, then let's do this. Unless @jhass or @ysbaddaden want to take a stab at this, the rest of the team isn't likely to take it for now. But I think we all want the feature, so if anyone is interested let us know!

[straight-shoota]

It has been established that data can be embedded quite easily through macros. To make this easier there is a shard schovi/baked_file_system.

What's missing is a literal for arbitrary byte data, this is tracked in #2886 and I think this issue can be closed in favour of that one.

[m-o-e]

Fyi, I've created an alternative shard for this: https://github.com/busyloop/rucksack

The difference versus baked_file_system is that rucksack doesn't read the files into
memory, so the attachment size doesn't matter (even gigabytes can be attached).

In order to achieve that it requires an extra step after compilation, though, to
append the packed data to the final executable. It would be great if Crystal could
provide an "after-compile hook" to run a code after compilation, then Rucksack
could make this part transparent to the user.

[mjblack]

On Windows, you can embed with resource files and utilize API.

https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-updateresourcew

[HertzDevil]

Slice.literal also exists now for arbitrary binary data. But there are some problems:

• There are no AST node methods to go from the StringLiteral returned by read_file to the string's bytes.
• It won't be fast because the compiler must parse the contents of the embedded files again, whether it's a StringLiteral or a Call. Those expanded AST nodes themselves also consume extra memory proportional to the file size; for the latter, one NumberLiteral per file byte is simply unacceptable.

So IMO we need compiler support for embedding very large files (e.g. several MBs), in order to avoid any performance issues associated with converting the file contents back and forth. This could be as simple as defining a new top-level primitive if we don't need compile-time access to the file contents.