Add audio support to the GLTF module

[aaronfranke]

Describe the project you are working on

I am working on a game that has user-generated content dynamically loaded at runtime, meaning GLTF files imported as scenes. The goal is to increase the scope of content the user is able to store in GLTF files, so they can configure their objects in their content creation tool of choice (Blender, Godot, etc) and export to a general-purpose format, instead of just importing an object with only meshes and having to configure everything else after import in the game.

Another important use case is the ability to export glTF files with audio out of Godot, to allow sending those models to another engine, or into a content creation tool, or even just to load them back into Godot for later in a more portable and longer-living format than Godot scenes. In the future this will allow you to use GLTF as an interchange format between game engines.

Note that this is also useful for games without dynamically loaded content, as this will work in the Godot editor of course.

Describe the problem or limitation you are having in your project

There is currently no support for saving or loading audio to/from GLTF files. This means that users cannot create 3D assets with audio in them. For example, a fountain that makes water noises, a gun that makes custom sounds when fired, the radio from Portal that plays music on a loop, or a tree that includes bird chirping noises or rustling leaves or something.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Add support for GLTF audio. Note that the specs for GLTF audio are not yet finished, so this proposal will not be able to be completely finished until a few weeks after opening this proposal, but I think we can get this in for Godot 4.3.

The specs so far: https://github.com/omigroup/gltf-extensions/tree/main/extensions/2.0/KHR_audio_emitter and here is the PR to the Khronos group: KhronosGroup/glTF#2137

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

PR: godotengine/godot#88204

Freely licensed example file: https://github.com/omigroup/gltf-extensions/tree/main/extensions/2.0/KHR_audio_emitter/examples/boom_box This file contains a boom box, a short looping music clip, OMI physics, and licensing information via KHR_xmp_json_ld. The model is CC0, created by The Khronos Group, and the music is CC-BY 3.0, created by Kevin MacLeod.

If this enhancement will not be used often, can it be worked around with a few lines of script?

The code in the implementation is about a thousand lines. You could workaround it for specific use cases with simpler code, but the whole feature set (import, export, many audio formats, standardized with GLTF audio) is complex.

Traditionally, the workflow for advanced content like this is to use a proprietary engine-specific format, like Unity packages. The same could be done with Godot if you loaded Godot scenes, but that has its own pitfalls like security and future-proofing (you may make a scene with Godot 4.2 today, but will that load in Godot 7.3 in 50 years? probably not). I believe it's better for everyone if we use an open standard format instead of an engine-specific one.

Is there a reason why this should be core and not an add-on in the asset library?

For the most part this could be a third-party addon, but it's a thousand lines of code, and it's in-scope for the goals of the GLTF module. I could totally see it being argued that this should be third-party, but my opinion is that the engine is the best place for this.

[Calinou]

This requires implementing support for runtime WAV and MP3 loading if I'm not mistaken, since we only support runtime Ogg Vorbis loading right now.

[aaronfranke]

@Calinou The branch I linked in the proposal supports runtime WAV and MP3 loading by setting data. But this should probably be given a dedicated function like for Ogg Vorbis.

Out of WAV/MP3/Ogg loading/saving, the only thing that seems to be not possible right now is Ogg Vorbis saving.