Define a virtual FS to use for all subsequent routes

[francislavoie]

Followup on #4909

Consider a config like this:

root * /srv
try_files {path} /index.html
file_server

If you switch to using a virtual filesystem, of course you'd need try_files and file_server to use the same filesystem, otherwise it wouldn't really make sense.

Defining the filesystem to use for each directive that might need it doesn't seem very nice, so we need to come up with a solution for defining it once early in the pipeline, and have any of these routes pick it up from the request context (similarly to how root sets a variable to be used later).

We could introduce an fs directive for this, which takes a virtual FS module and then puts a reference to it in context, which file_server and the file matcher can use by default if not otherwise explicitly configured (and if neither, use the local filesystem of course)

[mholt]

I support this idea.

[keyvan-dadash]

Is this issue still open? i want to try working on this one. thx

[mholt]

@sod-lol I think so! Would be happy to review a PR or proposal :)

[elee1766]

@mholt hi, i'm trying to implement the fs directive, but need some help/input on my solution

context - i have a plugin that can use multiple archives/directories from remote/local sources, and use an overlay of such archives to serve a site. it doesn't work though because i want to serve an SPA, try_files won't use the right filesystem

after some reading of the code, my proposal for implementation would be as follows:

provide a global "filesystems" map, similar to "vars" but scoped like "storage". this "filesystems" map exists in "context". instead of using m.fileSystem = osFS{}, we should do m.fileSystem = ctx.Filesystems().Default(). I imagine the implementation simply just uses some mutation method to add to the map in "Provision", and i assume there is some system in caddy that can make sure it is run first?

add a global "filesystem" directive, the syntax would be filesystem <key> <module...>. possibly can use no key if the first word is a valid module to mean "default"? (is this possible with your config framework?)

"filesystem" directive can be used more than once to declare more than one filesystem. this directive would add filesystems to the filesystems map. overwriting the default key can osFS as the default fs.

add a route directive fs, the syntax would be filesystem <key>. this would add a key like http.vars.fs to the vars, like "root".

also replace the file_server fs option with a string key, which would correspond to a filesystem declared in the global filesystems map.

when a MatchFile is provisioned, we can extract the "http.vars.fs" and then read the filesystem out of the global filesystem map as well.

the idea is to reduce the amount of loading of filesystem modules. the loading is done once at the start, and reused by components. it also introduces the ability for modules to provide their own filesystems that can possibly be used by other modules. this could be used to bundle "static assets" to modules, or share information like certificates, accessible by the unified fs interface.

there is potential improvement in the future to allowing inline declared filesystems (and giving them some deterministic name), but that i think can be added on top of this proposal

let me know if this seems reasonable, if so i can start on implementation.

[francislavoie]

Yeah, that all sounds good to me. Thanks for taking it on! We can iterate further on the PR when there's a proof of concept implementation 👍

the syntax would be filesystem <key> <module...>. possibly can use no key if the first word is a valid module to mean "default"? (is this possible with your config framework?)

Yep, that's definitely possible. We can check if there's a matching registered module and control whether to error or just continue and try the next argument. Checking the argument count on the line is usually helpful to enforce a pattern. Though it can be tricky because the module name can be followed by arguments for the module as well so some care needs to be taken to pass those through.

The codebase is full of examples of Caddyfile arguments parsing in similar ways, so I'm sure you can figure it out by reading around 👍 let us know if you need pointers.

[mholt]

That's way more intense (better) than I was planning 😄 But yeah, I think that's great. Hopefully it is not too complex (we prefer simplicity).

We're also on feature freeze until 2.9 so just be aware of that (and there's no rush).

[francislavoie]

Done in #5833

Thanks for the excellent work @elee1766 🎉