Provide the Godot engine itself

[Jehan]

I was looking into this BaseApp for Godot when looking at helping someone developing a game with Godot. I realized that you are just packing the godot-runner and looking at the 2 manifests on Flathub which seems to use this BaseApp, from what I see, it means they just provide a .pck file.

This is basically a pre-compiled package of the game, and all we are doing here in flathub is therefore to run this already-compiled game.
While it makes sense for proprietary games (and then the main advantage of flatpak is to get some sandboxing), this should not be the proposed process for Free Software.

A whole point of a public repository such as Flathub is to also show the full build process: people have access to the manifest, see what sources were used, which commands were run, and can even look at build logs. If the source is a pre-built binary and the commands/logs is just "copy the binary into /app/bin" (what is currently done for Godot applications), I could as well provide you ELF binary and .so libraries for GIMP. It would be the same thing. Just trust meeee 🐍👁️… Not sure anyone would like it! 😛

So my proposition is to provide godot itself. From my researches, godot can export the project from command line: https://docs.godotengine.org/en/stable/getting_started/workflow/export/exporting_projects.html#exporting-from-the-command-line

This will create the software .pck, as well as a binary which can be directly run. There is not even the need of godot-runner itself to run the produced executable (a proper independent ELF binary). So that means that it is not even useful to package godot-runner in the end. In fact Godot could just be made available in a SDK package, it is not needed for running.

Edit: when I said "From my researches", I implied not just theoretical research, I also tested the command line from Godot docs, then I actually uninstalled Godot and ran the application without any problem.

Edit 2: and for a headless Godot, since we don't care about the GUI obviously in our SDK, they have a procedure in their docs too: https://docs.godotengine.org/en/stable/development/compiling/compiling_for_x11.html#compiling-a-headless-server-build
This can be used to just build Godot projects from Flathub build servers.

[Calinou]

Actually, we should drop this BaseApp entirely and replace it with reusable module files (like https://github.com/flathub/shared-modules). See discussion in #3 where @TingPing agreed this was a better solution. BaseApps only have a substantial benefit for really large frameworks like Electron, and there needs to be some degree of guaranteed compatibility over patch releases (which Godot doesn't really have in practice).

The other issue is that exporting a project in a headless manner takes much more time and can sometimes be unreliable, so I prefer shipping the original PCK as intended by the author.

Note that PCK files don't contain any native code and can be extracted with relative ease, so there wouldn't be much benefit to creating the PCK ourselves anyway. PCK files of non-C# projects are also platform-independent, at least across desktop platforms (since all desktop platforms support S3TC).

PS: Godot doesn't compile a native binary when you export a project. It just copies an export template binary, which we happen to build from source in this BaseApp.

[Jehan]

Actually, we should drop this BaseApp entirely and replace it with reusable module files

That's a possibility, yeah. A SDK containing pre-built godot (over the freedesktop SDK for instance) would make quite some sense to me too. Mostly for usage in the build process.

The other issue is that exporting a project in a headless manner takes much more time and can sometimes be unreliable, so I prefer shipping the original PCK as intended by the author.

About the time, I believe that it's less of a problem in a build system (you don't do it every day!). For the unreliability, maybe, I don't know enough about Godot. Anyway we could still ship a PCK and run it with godot-runner being embedded in the package, for maintainers who prefer this, I don't care. But even when doing this, the PCK should ideally be built from the sources (when available). This is really about trust in the build process and all that.

Actually I am going to quote yourself in #3:

it's generally recommended for Flatpaks to be building applications from source rather than downloading precompiled binaries (unless no source code is available).

😛

Note that PCK files don't contain any native code and can be extracted with relative ease, so there wouldn't be much benefit to creating the PCK ourselves anyway.

Hmmm… From what I can tell, it's still not the same thing as the base source (which would be the clone of the source repository itself or a tarball of these same sources at a given release point). There are still packaging and processing steps done. You can reverse them, but it won't ever be the same as pointing to the sources, the ones the developer really works on. I remind it is mostly a question of trust. And for trust to happen, the whole process has to be as transparent as possible (providing pre-processed intermediate binaries and saying "this is somehow reversible with some other third party tool on github" is not ideal at all for trust).

Note that I am not arguing against having the possibility to just make a flatpak off a .pck, in particular useful when working with proprietary games (just as we could make a flatpak off a native totally opaque binary). I am arguing for also having the possibility to easily make a flatpak from the repository source. And for this, a godot SDK package would be perfect (then we don't have to build it ourselves in the game manifest).

Seriously, I just tested right now, building a headless godot exporter took not too long and is easy as hell. We could just have a SDK based off Freedesktop which has this exporter. And people will be free to create either native binaries off it, or just a PCK if you prefer.

PS: Godot doesn't compile a native binary when you export a project. It just copies an export template binary, which we happen to build from source in this BaseApp.

Pretty sure it does. See:

$ godot-3.2.1-stable/bin/godot_server.x11.opt.tools.64 --export "Linux/X11" bidiboup
$ file bidiboup
bidiboup: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.24, BuildID[sha1]=6063ec5febd33dcd77763311b22bb041697f4ce4, not stripped

And this binary works by running it directly (with godot not installed). I did test. :-)

[Calinou]

Pretty sure it does. See:

$ godot-3.2.1-stable/bin/godot_server.x11.opt.tools.64 --export "Linux/X11" bidiboup
$ file bidiboup
bidiboup: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.24, BuildID[sha1]=6063ec5febd33dcd77763311b22bb041697f4ce4, not stripped

And this binary works by running it directly (with godot not installed). I did test. :-)

When you exported the project, Godot copied that ELF binary from $HOME/.local/share/godot/templates/<version>/linux_x11_64_release which is an export template. It didn't compile anything to export the project. (You can uninstall C++ compilers from your PC and the export process will still work.)

[Jehan]

When you exported the project, Godot copied that ELF binary from $HOME/.local/share/godot/templates//linux_x11_64_release which is an export template. It didn't compile anything to export the project. (You can uninstall C++ compilers from your PC and the export process will still work.)

Ah ok yeah it doesn't compile in this sense, yeah maybe. Didn't check code of what it actually does. 🙂 I use compiling in a more broader sense of "going from one format to another" (transpilation and family).

I'll note that it's not a simple copy though. I have no idea what it actually does and maybe no C/C++ compilers are involved, but the files are not exactly the same (diff shows so) and the generated binary is like twice the size and, it can be fully independent (I tested just moving it around to some random directory, and it ran fine), at least when built with the headless godot (with the GUI godot, it needed other data from the repository, probably the PCK itself).

[Calinou]

@Jehan I guess you left debugging enabled in the final export file dialog, so it copied linux_x11_64_debug instead. To export in release mode, uncheck Export With Debug at the bottom of the export file dialog before clicking the Export button.

When exporting a project, the Windows binary will be modified if you have rcedit configured, but otherwise, the Linux binary will never be modified (unless you choose to embed the PCK within the binary).

[Jehan]

@Jehan I guess you left debugging enabled in the final export file dialog, so it copied linux_x11_64_debug instead.

Nope. Well I have no idea if it used the release or the debug binary as a base, but I did a diff of both (the debug is not double size).

$ ls -ltrh /home/jehan/.local/share/godot/templates/3.2.1.stable/linux_x11_* 
-rwxr-xr-x. 1 jehan jehan 39M 29 mai   16:53 /home/jehan/.local/share/godot/templates/3.2.1.stable/linux_x11_32_debug
-rwxr-xr-x. 1 jehan jehan 35M 29 mai   16:53 /home/jehan/.local/share/godot/templates/3.2.1.stable/linux_x11_32_release
-rwxr-xr-x. 1 jehan jehan 41M 29 mai   16:53 /home/jehan/.local/share/godot/templates/3.2.1.stable/linux_x11_64_debug
-rwxr-xr-x. 1 jehan jehan 38M 29 mai   16:53 /home/jehan/.local/share/godot/templates/3.2.1.stable/linux_x11_64_release
$ ls -ltrh bidiboup 
-rwxr-xr-x. 1 jehan jehan 69M 29 mai   15:43 bidiboup

To export in release mode, uncheck Export With Debug at the bottom of the export file dialog before clicking the Export button.

Since I am only interested into building for flathub (I mean right now for this small project), as I said above, I built the headless godot. No checkbox. 🙂

unless you choose to embed the PCK within the binary

Which I guess is the default behavior of the headless godot (as I said in my previous comment, the same command behaved differently with GUI, used as CLI, and headless godot; and the PCK is not even generated so yes, it has to be embedded).

[Calinou]

Which I guess is the default behavior of the headless godot

It's not the default behavior, but it will be enabled if you checked "Embed Pck" in the export preset. This information is written to the export_presets.cfg file, so it's determined on a per-project basis rather than being set using a CLI argument.

Anyway, I think this is a good idea. Here's my current proposal for creating a Godot Flatpak modules repository that can be integrated as a submodule:

• Both runner and headless binaries will be available. You'll be able to choose between two workflows:
  • Export a PCK from project files using godot-headless and combine it with a godot-runner binary. This would be the recommended option for projects with available source files. To export only a PCK file from godot-headless, use --export-pack instead of --export.
  • Download a PCK and combine it with a godot-runner binary. This would be advised for proprietary projects only.
• Deprecate this repository and archive it.
• Create a new repository with Godot Flatpak manifests for each version.

The repository structure could be as follows:

- godot-runner/
  - 3.2.yaml
  - 3.2.1.yaml
  - ...
- godot-headless/
  - 3.2.yaml
  - 3.2.1.yaml
  - ...

The first workflow will require more work on the maintainer's side, so we'll have to make sure it's clearly documented in the README.

What do you think?