Problems with preventing "engine bloat" and the AssetLibrary

[karroffel]

Sorry for the very long post. I'm not mad if you don't want to read the whole thing, so here's a short version of it.

This is a mashup or many different issues. I didn't know where a comment like this would have been appropriate, so here it is in its own issue.

TL;DR

Using assets should feel like using core-functionality. Currently it doesn't. Proposed fixes:

• add proper dependency manangement with system wide installs of assets. Move addons out of the project itself with a way to bring them back for modification
• make it easier to extend custom nodes, allow cross-language script inheritance
• allow scripts to be referenced by class name and namespace to avoid super long paths in case of assets.

---

Related issues:
#19178
#17092
#15661
#13187
#10635
#7402
#6277
#5947

---

Details here ⇓

Godot is amazingly powerful for the size of the binary. Unity and Unreal are bloated in comparison, but they do offer more QoL functionality out of the box.

The situation with Godot is that there are some PRs that add good and useful new nodes which are rejected since they are "easy to code in GDScript and don't need to bloat the engine". This is a very valid and reasonable reaction - if Godot should stay a < 40 Mo download then adding new Nodes and features needs to be justified.

While I do think this is a nice approach that ensures a pretty clean core, it has some problems.

Criticism and problem description

1. Functionality in core is official and undergoes quality control

If there's an InterpolatedCamera Node in Godot, I assume that's the interpolated camera most people use, I assume that it's the most developed and "optimized for the engine" one since it's the one that's part of the engine. It has something official about it.

The good thing about a "bloated engine" with many features is that those features are official and most likely the most widely used ones, more tutorials and documentation about them etc. They have been tested in many projects and there's a single place to discuss how to improve them.

I, as a game developer, prefer official nodes over custom ones written by Some Person. Of course this approach has downsides, like huge download sizes, longer loading times and other nasty things. BUT I think in terms of user friendliness, official features are preferred over custom addons. Which one of the 4 InterpolatedCamera assets should I use? How do they differ? Is the documentation good enough? Where's the issue tracker? etc etc...

2. Downloaded assets from the AssetLibrary are just copy pasted into the project

There is no difference between an Asset from the AssetLibrary and some code/scenes from the "main project". However, there is a difference between "official" Nodes and Nodes imported from the AssetLibrary.

If I wanted to add some custom behavior to one of my InterpolatedCameras, I could add a script on it and add my functionality - all cool and nice!

If I wanted to add some custom behavior to one of my AssetInterpolatedCameras, I could add a script on it and - oh. Attaching a script to a custom Node actually replaces the script. The very same script that is the reason I downloaded the asset. This is an editor problem and a core problem at the same time.

Problems:

1. Script inheritance isn't suggested when attaching a script - override is the default for those cases.
2. Script inheritance doesn't work cross-language. A project that uses C# can't extend the camera with C#, while this would be possible with an official node.

Fixes:

1. Make the editor fill in the path of the already existing script as the base type so script inheritance is the default. But this doens't solve the problem when you want to clear your custom script for that one camera as it would just remove all scripts entirely.
2. Make script inheritance part of the Script API. Currently the scripting languages all do script inheritance on their own - they don't know how to deal with other languages. My suggestion is to add a set_base_script that accepts every type of script. Usually the scripting language implementations just use the Script API on the base class anyway, so this could be generalized.

The problems of "copy-pasting assets into the project gets amplified by the fact that all scripts are only defined by their path.

In the camera case, I could do var camera = InterpolatedCamera.new() with an official Node, but I would basically have to do var camera = preload("res://addons/com.name.interpolated_camera/scripts/interpolated_camera.gd").new().

This is more than suboptimal. I know that reduz opinion is that paths are easier to use than class names, but I know for a fact that multiple users would prefer an optional alternative system that uses namespaces and class names instead.

Especially in the context of downloaded assets, this would be a huge improvement.

Proposed solution

Downloading assets and using them should feel like they are part of the engine. If they engine is kept small and meant to be extended with assets, that workflow has to be made as seamless as possible.

The basic goal is to make using assets feel like using core-features.

My proposed solution is a mix of a few existing discussions, but I think this is the place where all of them would shine together.

1. Make the AssetLibrary more package-manager like

• Having a dependencies.json could be good enough. Using the AssetLibrary UI in the editor would just fill in the json, the download and installation happens at another time. An asset can itself have dependencies.

• Assets are saved with version associated, all stored in .local/godot/assets or something like that. They are no longer part of the project itself but available system wide.

• At export the needed files can be copied into the .pck.

• Have a way to actually copy the files into the project in case modification at the source level is desired. Only then the files should be brought into the project folder itself. "Cloned" assets could live in a hidden folder like .custom.

2. Make using downloaded scripts easier

• add a ScriptDB kind of system that can be used to reference classes without knowing the exact path

• add support for cross-language script inheritance

Conclusion

I know this is a very long post, but I hope to spark some more discussion on the AssetLibrary and also usability side of things. I think making using assets as smooth as possible is important. While things like proper versioning and allowing sub-projects will help, I think it's not enough to shift the focus to a more AssetLibrary focussed use of the engine.

[toger5]

Really like your post.

EDIT:
Damn it overlooked this paragraph

Have a way to actually copy the files into the project in case modification at the source level is desired. Only then the files should be brought into the project folder itself. "Cloned" assets could live in a hidden folder like .custom.

So you basically have covered the concerns I had. (sorry)
I just go more in depth how I imagine the workflow + the behaviour.

POST:
One workflow is overlooked IMO though.
Using assets as just a resources or a first building stones.
The current really transparent and simple approach also lets you just download a couple of assets. delete some files. Than keep some png's and a script which has some of the functionality you want in it. Currently it is possible to just work on them and modify.

If assets would be hidden in a .local/godot/assets folder this functionality would be kind of lost.

Proposal

Assets should still be visible and listed in the in editor file browser. Independent if they are inside the .local/godot/assets folder or the project folder itself. So you can browse the files and understand how it works in case there is no proper documentation, or if you want to learn from it.
Every Folder which is 'an Asset' should also be marked as one with some kind of icon so you have an overview what is an asset and where the actual files are stored.
There should also be a editor window which lists all assets. (in a table)

• Downloaded lists all the assets which were downloaded. They can be added to your project with a click of a button. But other than that they are the same than assets in the asset lib. This list is the same in any project and has no effect on the project itself. (just makes it easy to have an overview. it's also easy to clear this list and redownloaded all the assets when opening a project.)
• Always active (basically just toggles "Used in this Project (active/inactive)" for every project)Those assets are added to EVERY PROJECT on your machine (even newly created ones). (Maybe the asset itself needs to allow for that.) Those are really Editor orientated. Like a clock in the corner, git support, todo lists or to have some more default resources:(a custom environment some more themes and a bunch of sprites to work with at the beginning of each project)...
  They can be manually deactivated per project.
  They are still listed in the project file system though and behave like any other assed/plugin. so you can browse their files and see how they work/what they do.
• Used in this Project (active/inactive) this actually shows the linked folder in the project file system. and "adds" (they are still inside the .local/godot/assets folder, but behave like project local files.) files to the project.
  Here you can also choose which version of the Asset you want to have activated. And switch the version.

As soon as you want to edit/save/update a file from an Asset, you get prompted with two options

Save in .local/godot/assets

either update the asset in the .local/godot/assets which puts it into a custom user version.
It the package-manager is git based, the custom user version could be one additional commit. which gets commit --amended whenever you save.
(3.2.5/user for example) and is than available in all your other projects as well. (when updating the version in those other projects to you user version)
why is this useful? (example)
A theme from the asset store is almost perfect, but you hate the font type...
change the font type save it as a new asset version. Revisit all your other projects and change the asset version to the /user. Now you are happy.
Maybe there also could be one button to publish the user version of the asset.
So in the asset store other ppl have the option to also choose a user version with minor tweaks.
Just as an intermediate to make a pr to the asset git repo. (or the author of the asset deicides to cherry pick the changes form a /user version to a new version on master.)

Make local to project

copies the files from the .local/godot/assets folder into the current project... removes any special asset behaviour (like version...) and now you can mess with it however you want to. (also the folder loses it's custom appearance as an asset.)

This allows the asset store to also cover the use case of templates. And is a perfect choice for almost any art related asset. Since you can work on the assets to make them as you need them to be.
You can always just activate a bunch of already downloaded assets (lets say some basic prototype sprites) make them local and you have them imported.

random thought

This might even be a nice solution for your own local asset/resource lib/collection. A folder in .local/godot/assets lets you add it's content to any Godot project with a nice editor ui and without searching for it in your OS file manager.

[mhilbrunner]

I agree that both the AssetLib as well as the asset lib integration (both in the editor and with GDScript) need love if we keep pushing it as an alternative to including things in the core engine.

However, I personally disagree with how heavily this should be a plugin is pushed.

The first question we ask ourselves on a new PR shouldn't be could this be a plugin? but rather is this going to be used enough to be in core?.

There are problems with having things in plugins:

• People need to know what they're looking for in the first place. Godot is targeted at beginners. Beginner-friendly software is usually trying to have everything someone is going to (reasonably) need, because otherwise they first need to learn what they're missing and how they're going to get it.
  There is a reason beginners use Ubuntu, not Gentoo. Regardless of how nice you make that dependency system.
• Decision paralysis exists. If there is a feature in Godot, great, takes a second to use it. If there are 5 plugins for it, which am I going to use? For example, there are 5+ console plugins. Which one should a beginner choose, and why, and how much time should they reasonably spend on this decision?
• Plugins get abandoned, have less quality, lag behind when being updated. There is even less reason to trust a random plugin won't blow up on me than to trust Godot, where there are enough people behind it.
• See Unity. Their assets are mostly even paid, and it still sucks that for many things you need plugins first which eventually are badly integrated or get broken on updates - or discontinued, even though they are paid and usually can afford at least some support.
• Documentation: We have not enough of this anyway. Every feature not in core, I can't use in official tutorials/docs. Random single people for plugins will likely not provide docs, or only short examples, not the more valuable bit of providing demos or full-blown tutorials on how to integrate that feature with others seamlessly in a project.

Examples:

• Spring arm node. (Added spring arm node #18822) Reasonably small addition, seperated into its own class so shouldn't break things, will be used by many 3D games, if not by every single third person game. Response: should be a plugin.
• Even for the RNG stuff, it was discussed to have it as a plugin. As if not 90% of games need a RNG somewhere.

Edit: To be clear, I'm all for avoiding bloat. But this is a balance, and I want to voice support for not having everything in plugins. Very game specific or not broadly used stuff, proprietary plugins, big dependencies, yes please, put those on the Asset Lib!

[dsaltares]

@karroffel This is an excellent post and I mostly agree with everything you said. I would, however, like to comment on this point:

Assets are saved with version associated, all stored in .local/godot/assets or something like that. They are no longer part of the project itself but available system wide.

This is similar to how Python pip system works. People then realised that the latest version of a package screwed the other projects you have in your system. virtualenv solves that problem. In Javascript's npm ecosystem, modules are installed locally to a project by default and system wide is an opt-in feature (npm install --global). You then check in your package-lock.json to source control to make sure everyone in the team uses the same version of every npm module and avoid conflicts.

System wide packages are good as they reduce bloat on people's systems. Version pinning is good because it helps the team keep a consistent set of versions and can help you identify a full collection of versions that work well together.

If we go the system wide route, I would suggest keeping version pinning. For example:

.local/godot/assets/fsm/1.0
.local/godot/assets/fsm/1.1

Project A could use fsm-1.0 whilst Project B uses fsm-1.1. Project C comes along and could re-use either. Users should be able to specify specific versions of plugins in their dependencies.json files or minimum patch, minor or major versions.

Once dependencies are downloaded, a dependencies-lock.json file could be generated and checked into source control.

[karroffel]

@mhilbrunner I definitely agree, I think the "this should be a plugin" mentality is pushed a bit too hard, exactly those situations sparked those thoughts. I think more common things like the SpringArm are worthy to be in the engine, as well as the InterpolatedCamera, which Juan wants to be removed, is insanely useful and cool! Even thought those things are only a few lines of code, they take a while to get right, so "this is too small" is also a bad justification for pushing towards the AssetLibrary.

My thoughts were mostly centered around "Well, the project lead said it's that way, so what can be done to make the new situation better?"

I would like if the heavy pushing of features as plugins would be lowered a bit, but either way I think the changes presented would help a lot!

---

@saltares Yeah that was my thought. I am a big fan of cargo for Rust. It operates pretty much the same. Libraries are installed in a system wide folder in all the versions that were ever needed locally. The description is in one file and a specific configuration is in a .lock file, it works pretty great!

I'm glad this sparked some discussinon already 😊

[reduz]

Here's my view to the above points:

Pushing stuff out of the engine

Pushing stuff outside the engine is IMO the way to go. There are too many platforms where deploy size matterns (mobile, html5) and bloat is a big concern for a project the size of Godot, so we can't just go happy include everything.

Philosophy to me should be that something should be part of the engine only if ALL the criteria below are met:

1. It's used relatively often
2. It's difficult to do yourself
3. There is only one way to do this feature

This rules out spring arm, terrain and I hope it can help to, in the future, remove features such as InterpolatedCamera, Vehicle, etc. that I believe should be outside the engine.

The main concern about this is that, if something is not official it's not going to be maintained. This is where you guys are wrong.

No one said these things should not be official, we can have an official repository for them and they can be first party extensions. Demos are maintained this way and they are up to date, so i don't see why it wouldn't work for an official extension repo.

Extensions should feel like part of the engine

Regarding these points:

• make it easier to extend custom nodes, allow cross-language script inheritance

I'm very strongly against this. There is no practical benefit at all to doing this, and the implementation cost is very expensive. Sorry guys, I know it feels like a nice feature to have but weighting cost/benefit/risk to it is on the very negative side.

I also think it's positive that custom nodes or resources should appear as-if they were NOT core is a good thing for end users. Users are not stupid and this information should not be hidden for them.

If a new node is added from an extension, and is done with a script, this should be even shown in the create dialog.

• Script inheritance isn't suggested when attaching a script - override is the default for those cases.

This needs to definitely be done and would be a very welcome addition.

• add proper dependency manangement with system wide installs of assets. Move addons out of the project itself with a way to bring them back for modification

My feeling about this is that it can either work fantastic, or result in a complete mess and add-ons that break without an obvious reason. Looking at how other engines work with this, it does not seem like having dependency management to me is so important.

• allow scripts to be referenced by class name and namespace to avoid super long paths in case of assets.

This is something we discussed before, but the conclusion we reached was that it should mostly be script language dependent (and not provided by the engine itself as a core ScriptDB), with a small usability enhancement in the new script dialog in case you want to use a global class.

[PLyczkowski]

My feeling about this is that it can either work fantastic, or result in a complete mess and add-ons that break without an obvious reason.

I guess the add-ons should never update automatically in our case, but instead there should maybe be a list of installed ones with version listed and manual updating, so the user can request update and check right away if anything got broken.

[LikeLakers2]

This comment is separated into collapsed sections due to the large length of it.

Preamble

I came from Game Maker Studio to Godot because, despite having paid for GMS, I still felt I couldn't implement all I want without having to resort to some *really* hacky code or coding in C++. When I came to Godot, not only did I come to find that it was much easier to create new functionality (I've only found *one* thing so far that I could not easily code in GDscript, which is getting spectrum data for creating audio-reactive experiences), I was incredibly impressed at how little I actually had to code to get 95% of my ideas working.

Want a camera? Godot's got a camera that can follow a player, with little to no code from you. Want to make an animation in Godot? Not only can you make character animations, that same system can (almost) replace that cut-scene system you worked so hard on, only needing maybe five or so lines of code from you. The editor has even been used to make other editors like RPG in a Box. It's fantastic just how much you can create in Godot.

---

Now with that out of the way, hopefully anybody reading this can understand what I'm thinking coming into this as I try to respond to some of reduz's points regarding pushing things out of the engine. (This won't be replying to the "Extensions should feel like part of the engine" section as I have no thoughts on that currently.)

Removing stuff from the engine, and Guidelines

Pushing stuff outside the engine is IMO the way to go. There are too many platforms where deploy size matterns (mobile, html5) and bloat is a big concern for a project the size of Godot, so we can't just go happy include everything.

I can agree that bloat is a concern, but when I see what you're wanting to remove so far, I ask myself if it's really even worth it to have that stuff removed. Even if you remove nodes like Vehicle, InterpolatedCamera, etc... How much of a file-size difference will that really make? Maybe half a megabyte? Maybe a couple of megabytes? Even if it saves several megabytes worth, what's the difference between a 35MB download and a 45MB download anyways? The game assets will quickly fill that 10MB void regardless! Unless you plan on removing half the engine, I don't see the point.

If you really are concerned about saving as much space as possible, might I suggest making it easier for the end user to remove the nodes they don't need from the engine instead? Let the people who are going to use the engine decide whether they're okay with the bloat -- don't decide for them.

Now of course we can't just include every single suggestion given to us...

1. It's used relatively often
2. It's difficult to do yourself
3. There is only one way to do this feature

...and these guidelines are a great starting point for scrutinizing what should be allowed in the engine. There is no doubt that all three points are already being considered for every single feature request that comes through the issue tracker.

something should be part of the engine only if ALL the criteria below are met

But c'mon. The fate of a feature or node should not rely that heavily on being a core feature. Your suggested guidelines are great, don't get me wrong, but we shouldn't be using them as a checklist. That's what YoYo Games does when you suggest a feature on their forums, and is another part of why I was pushed away from their engine -- can it be made using an extension? Yes? Then our response is make an extension. Guess how many feature requests have been denied because they could be made into extensions instead, regardless of how reasonable it would be for it to be in the engine, of how easy in comparison it would be to implement, or of how the community feels about that feature.

What I'm saying is that you can definitely keep those guidelines in mind. You can consider them every time a feature is suggested. But whether all of them apply should NOT be the deciding factor in whether a feature gets in. Weigh the idea against those guidelines, consider how important each of those guidelines are and apply weight to those factors... Just because an idea doesn't seem like it's used too often doesn't automatically make it a bad idea (keep in mind you might not know the true scale of how it's used, even with several years of experience). Just because an idea is easy to implement ourselves doesn't automatically mean it's a bad idea. And just because we can implement an idea in more than one way doesn't mean it's a bad idea! Not to mention that maybe someone's suggesting idea X because they think other people would benefit from it.

Remember that this is an engine that bills itself as being beginner-friendly, and every time a beginner has to code something non-specific to their game that's not part of the engine, or go look in some asset library to get the code they need, or go some long route to get what they want out of the engine, it could push them a little further away towards another engine, or sometimes even away from game development as a whole.

tl;dr: Guidelines are great when they're used as guidelines and not checklists.

---

An official repository for first party extensions

The main concern about this is that, if something is not official it's not going to be maintained. This is where you guys are wrong.

No one said these things should not be official, we can have an official repository for them and they can be first party extensions. Demos are maintained this way and they are up to date, so i don't see why it wouldn't work for an official extension repo.

I personally disagree with this idea. Alongside with what I've said above (that removing it from the vanilla engine won't save you much on space), I have to ask if we're going to convert them all to GDscript, or if we're going to require them to re-compile the engine just to get those nodes.

• If we keep them as C++ and require a re-compile... why though? There's no good reason we should ever be required to recompile the engine unless we're changing a core part of it, or adding our own functionality. That's just additional time taken away from developing, because we're waiting for the compilation process to finish, alongside requiring the average user to install other tools they might not even use ever again just to get some first-party nodes. Now that sounds like bloat!

• If we convert them to GDscript, then that's going to come with all the limitations of GDscript. Limitations like the potentially overhead of additional child nodes (especially if you're asking the user to import this first-party node as a scene), having to use paths rather than class names to refer to their classes (though you addressed this, it seems, but I feel I need to bring it up), lower performance (regardless of how little the difference might be, it's still lower), etc.. It's a bit more ideal than C++ because you've gone from re-compiling to drag-n-drop, but...

Either option also means the files for these nodes will have to be imported into the engine with every new user (for C++) or into the project folder with every new project (for GDscript) that wants to use them. I don't know about everyone else, but that sounds like a hassle to me, and sounds like it'd make your first guideline ("It's used relatively often") very debatable. I'll try to explain why I think this.

By requiring additional effort to use those nodes, I'm fairly sure the number of usages of this first-party solution will drop, and this first-party solution really will no longer be used "relatively often" where it may have been before. That, in turn, will slowly increase the number of alternatives that recreate the same features when people stop learning there's a first-party solution -- when I'm fairly sure we'd prefer existing versions (in this case, our own) to be improved, rather than have a new one created with feature X added, a new one with Feature Y, and a new one with both of those features but not as optimized, and so on... It's creating this market for alternatives when people don't know there's a first-party solution. It's creating a mess.

Additionally, you say they'll stay maintained. That's great! But who's going to maintain them? What are our guidelines going to be for adding new things to this repository? And are we going to go to the effort of ensuring this repository of first-party extensions will always work, whether with a stable release or with master? Basically, is it worth the effort of what is effectively a separate project?

---

Summary

What I'm trying to get at here is that I don't feel, with what your message has told me, that the time's been taken to consider whether those are really the right ways to handle things. Honestly, and this is more than likely the annoyance within me speaking (so please forgive me as this will probably sound rude), it sounds more like "what reduz wants" rather than "what reduz thinks is best for the engine", with little to no consideration for "what the community wants". Sounds a lot like YoYo Games and Game Maker Studio all over again, at least to me.

[reduz]

by the way, I much prefer to implement multiple script per node support rather than cross language inheritance. The former is just a couple lines of code..

…

On Sun, 10 Jun 2018 at 13:51 PLyczkowski ***@***.***> wrote: My feeling about this is that it can either work fantastic, or result in a complete mess and add-ons that break without an obvious reason. I guess the add-ons should never update automatically in our case, but instead there should maybe be a list of installed ones with version listed and manual updating, so the user can request update and check right away if anything got broken. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#19486 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AF-Z21jtcnUhRyhdyEEnaM_hfTACsxOiks5t7U6HgaJpZM4UhqDC> .

[Zireael07]

Multiple script per node has been attempted a couple of times and always failing.

[reduz]

what was attempted is the multiscript node, but it was a hassle

…

On Sun, 10 Jun 2018 at 15:01 Zireael07 ***@***.***> wrote: Multiple script per node has been attempted a couple of times and always failing. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#19486 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AF-Z26u8Rqc1yL-BGX16CAx_iWsYEo1oks5t7V8CgaJpZM4UhqDC> .

[vnen]

I understand that assets should not look the same as the built-in stuff. But I also think they should be at a higher-class than user-made scripts. They should have a clear separation so it's easy to replace/upgrade assets without having to mine them out of your project.

Also, adding a layer for assets and let them integrate better with the engine to extend it is great way to keep the core small and clean, while easily provide new functionality via the asset library. If assets can register new types not only for nodes/resources, but also classes to be extended, in a way that is globally accessible (i.e. not path dependent), it would be a great benefit for frameworks to build on top of the engine.

Frameworks like Escoria could benefit a lot from it, and there could be other frameworks for things like RPGs, platformers, visual novels, etc.

All I'm saying here is quite abstract, but having a clear separation of what's built-in, what's third-party asset, and what is user code (however this separation is presented) will make the "keep the core small and provide extensions" go forward much easily.

[neikeq]

Cross-language inheritance sounds like a horrible idea to me.

[willnationsdev]

In the work I've been doing with the TypeDB in the EditorData, the I've been keeping a namespaced-typename to filepath HashMap that gets converted to a Dictionary in the ProjectSettings upon modification. In my branch, the GDScript parser and compiler are using this data to create global names for types. At this time, I've got custom type scripts showing up. The scripts can access and inherit them by name. What's more, this data is accessible to anything that can see ProjectSettings. So a CSharpScript seen by the Editor can generate a mapping that is entered into the ProjectSettings Dictionary, allowing types declared in C# to be accessible to GDScript. There's probably also a way to wrap the ProjectSettings Dictionary data in a C# Object that returns Script objects when accessed, that way C# would be able to access GDScript types by name too.

This technique keeps the shared namespaces out of core and purely within the Editor context, whereby languages that opt-in can prepare their own means of accessing the data.

---

I do like @karroffel's suggestion for being able to have cross-language script inheritance though (don't even know if that would be possible). Somehow automatically duplicate methods that delegate calls to the base script and also define all the same properties/signals/constants, etc.

I too would like a more streamlined access to content in the AssetLibrary, with the ability to keep some plugins for the editor overall and other plugins for just a single project.

The TypeDB system I'm making will also give editor support for...

1. Having all of one's custom scripts show up in the CreateDialog
2. Have extending a custom type script automatically make you extend that script when adding a script to the node
3. Have removing a script that extends a custom type script reinstate the custom type script after removal (so you can't remove a custom script - you must use the "Change Type" command in the Editor instead).

Also, all of the interface changes I'm making would keep the custom type scripts clearly understood as being scripts as well as keeping the script in question accessible. Users will always know what is a script and allow them to get to the script, but it still makes them feel like in-engine objects to an extent. Even within GDScript, the names are not globally accessed, as if they were a core type, but rather extracted from a global Dictionary called Scripts. So far I have it working with custom type scripts, but I would like to eventually expand to having the Dictionary register custom type scenes and any typical script or scene available in the project too, likely via the Import system if the script language doesn't have a simple means of acquiring the namespaced typename.

[QbieShay]

As I have been part of the thinking process of this proposal , i would like to point out few things:

My vision of Godot

TLDR: I imagine Godot as a tool that can be highly extended with assets and plugins in a very easy, modular and beginner friendly way.

The motive behind the rejection of my SpringArm pr are more than reasonable, and that's why @karroffel and I discussed about the direction that the engine could take to keep being lightweight and, at the same time, being user friendly.

What I'd love to see is a core engine clean, small and fast and a well integrated ecosystem of addons and plugins.
As ArchLinux has more than one package repository, I think it would be great if Godot had an "official"(mantained by Godot) repository for core functionality (and Nodes that are now core) and another repository for community. In the Godot core "extension repository" I would put all the nodes that are a pain to code but not enough used to be shipped with the vanilla version of the Engine.
Side note, what it's often not taken into account is that some features might be easy to code but might also require a lot of research before laying down these few lines.

I imagine users being able to download plugins and assets that feel like they are "extending" the engine itself: depending on what you are downloading, you have Godot-fps or Godot-platformer or Godot-racing.

Godot could be the first engine to feel really Unix-like and modular, surpassing even Unity with its editor extensions. Many of you know how good it feels when your Linux system is not "that distro", it's your very own and customized system with your very personal functionalities.

Advantages

TLDR: this approach would allow Godot management to manage separately hardcore technical core features and more user-friendly plugins without worrying too much about bloating the engine

What is good about this proposal is also the separation of concern and the handling of the contributor. I feel that, for now, it's not clear what is okay to be in the core and what is not. There is no clear rule nor statistical data to speak about. Some might say a certain feature is widely used and some other think it's not.

Dividing Godot in different "rings" gives the opportunity for a more fine grained management of PRs and features, and can give more time to @reduz and other very technical developers to focus on hardcore technical stuff, while another team would manage the extension ring, taking more into account user feedback and usability rather than having to face engine bloating, since the "extension ring" would be bloat on demand.

About multilanguage inheritance

As Godot is growing, the asset store can easily become a flaming mess. The strict division between scripting language can literally destroy any attempt that people make to sell assets and grow an ecosystem on the asset store, killing many opportunity for freelance toolmakers to make and sell tools for Godot.
Language bridges need to be more complete if we want any chanche to develop a lively ecosystem.
For now cross-language inheritance sounds great to me, but i am not technical enough to say anything on this matter. I just want to point out a problem that will likely present itself soon, expecially since there is such a huge community around C#.

Note about usability

I've been using Godot for almost a year now, and lately i feel like it happens that people that decide about features are not using the engine for anything bigger than small demos. @LikeLakers2 pointed out a problem that i feel is shared by some of the users that are not deep into the developing core team.

[swarnimarun]

I think that what is considered bloat is a trivial topic in the greater scheme of things, it's not like the decision we make now is what we will need to abide by for the whole life of Godot. Being open-source allows us to make changes whenever we like the only problem is that the more we wait the more heavy the decision of change is going to be.

Now that's also a discussion for another day. Right now in the present the facts are,

1. Godot is a compact, fast and portable game engine. And I like it alot but,....
2. Godot's executable is around 45-60MB depending on various factors. But the export template can be as large as 300MB. Now that's a big blow to the first point.
3. Godot has lot's of amazing features that make it a treat for the beginners like Camera follow, Parallax BG and Vehicles, this is a big plus over other similar game engines. As it makes creating games much simpler in Godot. And makes Godot a home for lots of newbie devs, who don't have the skills to write their own implementations for such tools.

Now what will happen if we remove features that some people consider bloat. Well the engine will surely be more compact but is that required? I don't see a problem with myself downloading a 100MB Engine if it fulfils my needs better. But what about exporting, won't a 100MB engine export bigger games?

Probably, yes, but in a world where desktops don't seem to use anything less than 256BG SSD + 1TB HDD and even Phones have space starting from a minimum of 16GB all of which which is more than likely to increase, would it be that BAD?
I mean to say we don't need to remove anything yet, we just need to start managing a few things better and that should be it.
Features should have some sort of importance list to be decided if they are supposed to be in the core or not but that doesn't mean that it becomes a law but rather a rule of thumb.

And for the nit-pickers we could remove some extremely unused tools but that's it. Features don't bloat an engine unless we are careless. And even all the bloat in Unity has not stopped people from using it, has it.
And I absolutely don't mean we should not care about bloat but being careful and being paranoid are different matters all-together.

As for multi-language inheritance, I don't really see a use for it here. Sure we should be able to interact with scripts of multiple languages but inheritance! That's likely more work than I could imagine with benefits that just don't seem to be worth it.
Whereas, multi-script might be just the solution for lot's of problems from, addons creation to better modularity. Although that is my perspective.

[nobuyukinyuu]

There's something to be said about limiting the number of node and resource types that come in the box. By squirreling away any lesser-used assets to an external download, you make it 1. a lot easier for new people to get a grip on the system just by "Messing around" without being overwhelmed by the breadth of features (decision paralysis), and 2. don't end up creating as much legacy baggage if it turns out some of the more narrow use-case-scenario assets end up having a workflow that isn't as good as something that comes afterwards with a different API.

When the latter happens with built-in's, you're stuck with them. When it happens with downloadable assets, the most popular ones tend to become de-facto standard over time, but don't have their fate permanently entwined with that of Godot. That means they can fade into obscurity without having to worry as much as breaking expected core compatibility.

[neikeq]

The cross-language inheritance point needs some explanation. Not only I consider it to be a bad idea, but I don't even see how it is remotely possible.

2. Godot's executable is around 45-60MB depending on various factors. But the export template can be as large as 300MB. Now that's a big blow to the first point.

300MB the export templates? Do you mean with assets?

[karroffel]

@neikeq the thing is that if your whole project is in C# but you are using a custom node downloaded from the AssetLibrary then that Node is just a regular node, most likely with just a GDScript attached.

If you want to "extend" the node, you can't! I guess you can add a child node to add new functionality, but that way it's not possible to override things.

---

Implementation

Currently the scripting languages use a really similar pattern for inherited calls which could be generalized.

godot/modules/gdnative/nativescript/nativescript.cpp

Lines 696 to 712 in 76875ba

	while (script_data) {
	Map<StringName, NativeScriptDesc::Method>::Element *E = script_data->methods.find(p_method);
	if (E) {
	godot_variant result;
	result = E->get().method.method((godot_object *)owner,
	E->get().method.method_data,
	userdata,
	p_argcount,
	(godot_variant **)p_args);
	Variant res = *(Variant *)&result;
	godot_variant_destroy(&result);
	r_error.error = Variant::CallError::CALL_OK;
	return res;
	}
	script_data = script_data->base_data;
	}

godot/modules/gdscript/gdscript.cpp

Lines 638 to 651 in 76875ba

	GDScript *top = this;
	while (top) {
	Map<StringName, GDScriptFunction *>::Element *E = top->member_functions.find(p_method);
	if (E) {
	if (!E->get()->is_static()) {
	WARN_PRINT(String("Can't call non-static function: '" + String(p_method) + "' in script.").utf8().get_data());
	}
	return E->get()->call(NULL, p_args, p_argcount, r_error);
	}
	top = top->_base;
	}

godot/modules/mono/csharp_script.cpp

Lines 1175 to 1191 in 76875ba

	while (top && top != script->native) {
	GDMonoMethod *method = top->get_method(p_method, p_argcount);
	if (method) {
	MonoObject *return_value = method->invoke(mono_object, p_args);
	r_error.error = Variant::CallError::CALL_OK;
	if (return_value) {
	return GDMonoMarshal::mono_object_to_variant(return_value);
	} else {
	return Variant();
	}
	}
	top = top->get_parent_class();
	}

The pattern here is

Class *c = bottom;
while (c) {
    if (c->has_method(m)) {
        c->call(m);
        break;
    }
    c = c->base;
}

So it starts at the bottom of the script inheritance tree and then tries to call the upper levels if there's no such method found.

If each script had a Ref<Script> parent_script instead of the language-specific inheritance structure, this could be generalized to

if (this_class->has_method(m)) {
    this_class->call(m);
} else {
    // method not declared here, go to base class
    parent_script->call(m);
}

---

This might not be a good idea after all, but not having the option to override stuff cross-language might be a problem over time if we push the AssetLib more.

[swarnimarun]

@neikeq See this,

[akien-mga]

@swarnimarun That's export templates for all platforms, all supported arches, debug and release, their size is completely irrelevant to the "bloat" of the engine. We can decide to support only Linux X11 64-bit release and this file will be 20 MB...

[swarnimarun]

@akien-mga I was not intending to call it, bloat.
But a new user will download it full-size they won't be experienced enough to know that they can build separately or download for a specific platform.

How is it irrelevant when this is what we provide to beginner users?

[Zireael07]

@akien-mga: Export templates size matters since your argument for removing nodes was

deploy size matterns (mobile, html5)

[akien-mga]

@swarnimarun @Zireael07 Yes export templates size matter individually, but my point is that measuring the size of a bundle of templates doesn't give us any indication. The size of one template (e.g. the wasm release template, 3.4 MB zipped, or the Android release APK, 23 MB zipped with armv7 and x86 libs) is much more relevant than "downloading everything needed to develop takes 350 MB" (+ maybe 10 GB for Visual Studio if you need to build something...). What matters if what size a minimal game will take on platforms where size matters (web, mobile).

[QbieShay]

Although i think that deploy size is really important, i also think that what needs to be taken into account is also developer time. For example, Unreal has literally any feature you could be looking for, but the fact that there are so many makes the development process highly inefficient because every time you need to do something with it, you first need to figure out which builti-in functionality you need to use and then learn how to properly use it.

This is something i really love about Godot. You can code your feature straight away and then someone will come and tell you "yo, did you know that we had this built-in thing you could use?"

The problem when an engine has too many features all together is that developing with that engine becomes a huge tiring experience. Yes, it's nice to have all the features you need but it's a gigantic pain to go through all of them to figure out what you actually need. That's bloat. And that's why we are proposing to have on-demand well integrated features.

[Ranoller]

add proper dependency manangement with system wide installs of assets. Move addons out of the project itself with a way to bring them back for modification.

This is the same that losing portability. My plugins are part of my project, i need that in whatever computer i´m working and i need the concrete modification of every plugin for every project... I don´t see any benefict in have my plugins stored in .local/whatever. This is kind of obfuscation my projects.

Actual system of res://addons is good... free for the user and portable around computers. If there is no complains about this system, is necesary to change that? There are a lot of other things that users really complain...

[QbieShay]

@Ranoller you would be able at any moment to copy the plugin into your game folder.
Anyway, in that case your project.godot would list your plugin as a dependency and it would be downloaded to your new computer with the correct version

[Ranoller]

Downloaded? My plugins are inside the project that is inside the pendrive, and are part of the git repository ... I don't understand proposal... Plugins change the same that other code... I change more code from plugins that gamecode, i don't wan't to number plugins by version and work outside from the project... Every one line code change will need to manually change every plugin in every computer?... Really plugins feels like part of the project... Give plugins out from git repository is nonsense to me, but i really think that i do not understand something about that...

[karroffel]

@Ranoller the idea is to move those plugins outside of the project that you don't need to modify. For example, the InterpolatedCamera will just be a node you want to use as is, not change the code of it.

The same could be the case with editor-only plugins like TODO lists or git integration etc. Those are things you want to use but not change.

This is a lot like a package manager and libraries work in every language that's not C and C++. You have a file that specifies all the dependencies and then the package manager will fetch those dependencies. This happens even when you change PC - the package manager will download any not-yet-existing packages.

When you want to modify something, then you bring the code into the project. Then it's in your git and the project with all its source code etc. But that's not always the case.

My proposal is to have things outside the project by default until you actually need to modify something.

Currently many assets are mostly templates that you do need to modify, having such a system in place might change that so things are more general purpose via APIs, not via source modification.

[Zylann]

@Two-Tone if plugin A needs version 1 of X and plugin B needs version 2 of X, I would say Godot should not try to magically fix it. Here are simple solutions to this:

• Normally, version 2 of X should be backward-compatible with version 1, and that's how things should be to keep it simple (remember when I mentionned the mindset when developing plugins).
• If you can't use latest version of X, wait for plugin A to support it and download previous version (we need the assetlib API to handle getting different versions, not just the latest one, that shouldnt be hard to do). It's like mods, it doesn't have to be perfect.
• The plugin maker of X uploads it as a different plugin (because the API breaks compatibility), just like we had to when Godot 3 went out. So then you will have twice the same plugin, but it's fine because they are too different to be really the same ones anyways, and stuff depending on it will work too. It can even just be a folder rename by the plugin developer, not even the need to have two asset lib entries.

In the worst case scenario you would have to make this work manually (and PR compatibility to the plugin maker eventually).

If not these, that would involve two versions with different path being installed everytime (screw all references at each update), or core changes being added to the engine to alter how paths work (complicated).
I think if we can just get simple dependency management it will be quite nice already, since it would basically use the functionality and structure Godot already has (so it would be quite simple to implement as well), and do automatically things the user currently has to do manually anyways.

I'm not considering dependency management as a thing that should always work and be the entire responsibility of the engine (like npm), because that would be a lot of work. I consider it primarily as a convenience for things you currently have to do by hand with existing functionality, and plugin makers can release their work based on that simple system in mind.
If someone manages to implement nicely a system going way further than this, then it can be done later, because what I described above is workable before and is mostly simple improvements to the assetlib and the editor only.

[Two-Tone]

@Zylann How are any of those options better than what I've suggested? And there is nothing magical about searching for and appending text to a specific string.

Option 1 forces developers try and make sure their version updates don't break any other plugin that depends on it, which adds a lot of extra work on them. Who wants to develop plugins for something like that?

Option 2 forces users to not use the plugin they want because they have to either wait for the plugin to update, if it's even still being actively developed, or just not use it at all.

Option 3 once again makes things more difficult for the plugin developers.

Two versions with different paths isn't a hard problem. If they ever update, check to see what install packages have them as a dependency, rerun the search and replace (taking into account the old version number) on all the scripts for those packages, then finally unpack the new version.

I feel like people are trying to make this out to be more complicated than it needs to be.

[LikeLakers2]

Proper dependency management and conflict avoidance is an important and hard issue. It's being discussed so much because of that

@Two-Tone What I meant is that you're thinking way too hard on the issue even for it being a hard issue. A lot of the conversation (including my replies, admittedly, so please excuse me as this is going to sound hypocritical) have been about what's wrong with the ideas presented, not so much about what can be done to change it and make it a better plan. The ideas being suggested can't only have downsides, can they? It seems like everyone wants their way and their way only as well, which doesn't help matters.

Trying to just push this through without figuring out all the problems that dependency brings would be a very bad idea and just cause way more problems. This is a hard issue and it will take time.

I suggested coming up with a few simple ideas with the implication that we'd iron them out and then decide once we feel we've got a good system. I didn't think I had to explicitly point that out, but there you go.

---

I'm kinda annoyed at how long this conversation has gone on without any progress. Seriously.

[Zylann]

Two versions with different paths isn't a hard problem

That alone isn't, probably. But having two versions of the same plugin at a given time in Godot implies other things that become confusing to solve.

• Your paths no longer are what you think they are / changes need to be done to core, which is a concern for core devs mostly
• If the plugin exposes a custom type, which one does the editor displays?
• If the plugin is C#, how do you compile it without conflicting with others copies?
• If the plugin registers autoloads, how do you tell them apart?
• If the user creates data which uses the plugin, which one will it be using?
• Which one will show UI in the editor?

I think before we can even find simple solutions to those that fit the engine (and there may be more I didnt think about), we can already improve existing stuff in the editor and asset lib without breaking compatibility or introducing core changes.

[paulhocker]

+1 on everything in this thread - extending the plugin module and increase use of the asset lib is the way to go

[chanon]

After having trying out and learning Unity, I now have a clearer idea on what this should be like.

First is Unity now has a 'Package Manager' which they use to let users select which components of the engine they would like to install/uninstall. It looks like they may also use this for the Asset Store in the future.

Something like this would really help Godot to be able to "reduce bloat" while having "official addons". Unity is still super bloated even with their Package Manager while if Godot had one it could get even smaller. The "one exe" method of distribution might have to go though.

Also, normal "modules" maybe should be built as shared libraries (dll/.so files) so that they can be selected for install/uninstall in the package manager.

Projects and packages should have manifest files listing packages it depends on and Godot / the package manager would ensure that required packages are installed.

A public registry of packages would be needed.

This would enable the community to easily extend Godot's capabilities and easily build upon each other's works. I think it would promote community development a lot.

[mhilbrunner]

Not disagreeing with the overall merit of moving to the Package Manager approach, I have not made up my mind on that yet, but I want to point out that it has its weaknesses as well - one single executable can be way more integrated and well-tested, at least more easily.

With a Package manager, the testing surface grows exponentially for each plugin and version. This isn't that bad if the plugins are mostly small additions (like currently on the AssetLib) not hooking too deep into the engine core, but it quickly becomes so if you split up the core parts of the engine.

Unity already runs into issues, crashes and compatibility problems with different versions of their packages, their editors, or with having some specific package combinations installed (or not installed), or even with the order they were installed in.

This is a lot of additional complexity that needs to be managed well, lest we end up in dependency hell :)

[Eoin-ONeill-Yokai]

Regarding dependency hell, why not take a page from something like AppImage and try to avoid dependency problems by making plugin distributors responsible for managing asset / file dependencies? Here's how I see this problem...

Plug-ins should (ideally) be self contained elements of either logic (custom editors, custom nodes, dynamically linked libraries) or assets (models, materials, fonts, textures, etc) that should have their own sets of dependencies. Trying to figure out shared dependencies is a conundrum that, while people might like it, it means a lot more effort in order to maintain it. If you were to actually allow shared dependencies, you would have to approach this problem similar to the steam runtime and maintain a bunch of different libraries, plugins and assets as their own unique packages. You would also need to store multiple of the same packages on a server for multiple versions -- it would be a maintenance nightmare and it's no surprise that modern linux distros are more or less moving away from package managers.

A plugin could essentially be a single tar archive of all of the elements needed to for that plugin to function in isolation. They should work natively on their own with all dependencies archived inside as well. When godot sees this file, it checks for a manifest file inside of the root of the archived folder. This manifest could contain basic info (name, description, godot-version, icon, authors) but should also contain things like: update-info (link to download the latest version, maintained by plugin developers) which godot can use to automatically update the plugin (with users permission), project_website (link to repository or website to let users quickly find the source of a package), and class_names (list of classes within this addon that can be extended, used for scripting.) It could also point to the addons it uses as dependencies using relative path to an internal plugin file.

If we allow plugins within plugins, then the dependency problems will be (more or less) solved. Differing version of the same library will be in different directory paths and could be treated as different 'paths' on a given fork. The problems with this system, of course, is that you end up having a lot of duplicate information on a given disk. However, when packaging a game, godot could ideally go through all the plugin directories and merge file references that match via checksum (this can included tar-represented file structures?). This way, if two files are identital, godot will optimize the eventual output binary by merging the reference to a single file.

I'm sure this is far too naive of a solution for a problem this complicated, but I was thinking about the plugin system more recently due to my custom font patch. I think the best user experience is one where they don't have to worry about dependencies, and the developers are the ones who should be concerned with managing their own dependencies. This makes it easier for the user to manage: a project can have their addons represented as a single file in the file system dock, you can right click and get additional options (extract files (for development), update plugin (for easy updates), go to website (for easy community organization)) and also reference classes within a given 'container' (extends "[path-to-addon-tar]://ClassName")

Just my 2 cents on the matter, I'm sure there are some flaws with the way I'm thinking of it as well, but it could be an interesting solution and also be a decent way to cut down on the maintenance necessity for the Godot team.

[bojidar-bg]

Just a 2 cents based on the previous 2 cents (does this make it 4 milles?):
I guess .tar archives would be bit unfamiliar to most users (ahem, windows users) so it might be better to use .zip archives, as they have somewhat wider support.

That being said, what about distributing plugins in Godot's own .pck format? I does have some cons (such as being unrecognized by most applications), but that might be a chance to make it more standard and used.

[Eoin-ONeill-Yokai]

I guess .tar archives would be bit unfamiliar to most users (ahem, windows users) so it might be better to use .zip archives, as they have somewhat wider support.

That being said, what about distributing plugins in Godot's own .pck format? I does have some cons (such as being unrecognized by most applications), but that might be a chance to make it more standard and used.

Sure, I mostly meant .tar as an example of a generic packed archive type. It could and probably should use godot's .pck format.

[Calinou]

Why not use ZIP since it offers compression and is a standard format?

[willnationsdev]

@Calinou I agree. If we mandate that .pck files must be used, then people will have to use Godot Engine in order to pack things rather than just using whatever their preferred packing tool is to bundle up their plugin for usage. GitHub also downloads repositories as .zip files and it houses the largest collection of Godot plugins by far.

[paulhocker]

I agree here. I don't think another file type needs to be created. Zip files seem to work very well, and folks have the option of installing using the Godot interface thru the asset library, or they can download manually themselves. a .pck file does not allow that choice.

[golddotasksquestions]

In my ideal world, I would like to have two download buttons at https://godotengine.org/download:

• One sleek bare minimum download that meets reduz three requirements, and
• One bloat download with all approved and and regularly maintained plugins which have surpassed some quality control, so I know their API works at least as well with the core as the API from the minimum version.

Even more ideal:
Select and mix. Upon downloading the bare minimum, I can select minimum version + Plugin "Themes". Which are collections of common plugins needed for common usecases. (so for instance: 2DPixelart, 3D FirstPersonShooter, 3D ThirdPerson, RTS, RPG, 2.5D game, 2DCutout, Rougelike, ManagementSim, Sidescroller, Eventpackage, Shaderpackage, Scriptpackage, Assetpackace, ect). So when I start the engines essential plugins are already installed and ready to go and I know they work because they have been approved.
More advanced users could untick individual plugins within those Theme collections if they know they don't need them, or combine two or more Plugin Themes, or add additional single plugins to Themes from the pool of approved plugins to make a custom package.

Ideally, the result would be that https://godotengine.org/download creates from those choices one download link on the fly. A download link I could copy and share with whoever, who then could with one click download the exact same Godot plugin configuration as me.

Why? Because this would be a massive time saver and quality assurance for people who work in teams or groups, for educators and teachers in schools who have to deploy on multiple computers, for people who follow tutorials and want the same config as the tutor, for individuals who are happy there is one official approved plugin by the dev team for something that is commonly used, but not by everyone, and so not have to dig through Github and Reddit for days to find what they need, not knowing which is the most supported option out of the 6 they found.

A one-click bare minimum download should still exist for those who need the small size, as well as a max feature download who do not care about size but want maximum features to see what the engine can do.

[menip]

A simpler solution would be to ask to install additional [official only?] plugins when running editor for first time. Currently we have option to open AssetLib if project list is empty; why not have a dialogue popup to install official plugins if running Godot for first time?

[Calinou]

Closing in favor of godotengine/godot-proposals#142, godotengine/godot-proposals#577 and godotengine/godot-proposals#607, as feature proposals are now tracked on the Godot proposals repository.

Edit from aaronfranke: Also see godotengine/godot-proposals#554