Add auto-formatter for gdscript exposed via command line switch and editor menu item

[frugs]

Here's a stab at going some way towards resolving issue #7392.

I've implemented a very basic auto-formatter for GDScript using the existing GDScript Tokenizer. I took the least intrusive and easiest way of exposing this and I've exposed it as part of the main binary, enabled using the --format and --script SCRIPT command line switches.

EDIT: This feature has now also been exposed in the script editor UI as a menu item in the edit menu.

[Chaosus]

Thanks for contribution ! Please squash the commits together (using git commit --amend).

[frugs]

@Chaosus Thank you for responding! I'm happy to squash them before merging, but for now my reasoning was that splitting into these self-contained chunks would make it easier to review.

[reduz]

I am not too sure of this, honestly the changes to the tokenizer are not too serious and it may work.
The formatter should be activated via a ScriptLanguage function I guess, so the editor can use it too.

I'm not really sure how well the formatter works, or how useful this really is in the end, so I would prefer it's not up to me to decide on this PR.

Also please leave the renderer string print in there, it helps a lot with bug reports, so for now I would not change it.

[frugs]

@reduz Thanks for taking a look! I can revert the change to the GLES3 renderer print statement, though this means that it will appear at the top of the formatted output (though I suppose I can just handle this in the python wrapper-script I am using to handle command line arguments). I can also move the format function from the Script class to the ScriptLanguage class, though I'll need to have a look at how the ScriptLanguage class is used so I can have a better idea of how best to do this.

The formatter is very basic, and only really handles spacing between tokens (and will conservatively not format portions of the original code between tokens if they contain non-whitespace). I could provide some examples of its inputs and outputs, if that would be helpful.

As to how useful this is - I certainly found it useful for applying to my codebase after a few commits by a contributer who hadn't read the style guide, and I intend to continue using it (with my python wrapper) during the development of my own game. Given that there was an outstanding issue (#7392) which this would go some way towards resolving, I thought I might as well submit it as a pull request.

[frugs]

I've made the suggested changes. It should now be fairly trivial to integrate this into the editor UI.

[frugs]

Example of the auto-formatter in action: https://pastebin.com/vXJYNC65

[frugs]

This feature has now also been exposed in the script editor UI as a menu item in the edit menu.

[frugs]

I've rebased my changes on to the master branch and applied the changes requested.

[akien-mga]

Sorry for not having been able to re-review this more in depth yet. This is still a wanted feature and this seems to be a good PR, but it will be too late now to merge for 3.2 which has entered release freeze, so I'll move it to the 4.0 milestone.

It might still be cherry-picked to the 3.2 stable branch once merged and tested in master.

[LaurentOngaro]

The feature is really nice, and it works pretty well, but the source code is outdated.
I've tried to manually merge a more recent code from the master branch (commit dc114fa) and the changes from frugs's repo into my own repo.
I managed to build and run the editor but the gdscript tokenizer is broken.
My code can be found in github here

I'm not well at ease with godot code yet to be able to fix it, or to merge Frugs's code in the latest (4.x) godot code. Perhaps someone (frugs ?) can just fix the code in the file 'modules/gdscript/gdscript_tokenizer.cpp' and we will be able to use it with the updated source code

I've joined the content of my gdscript_tokenizer.cpp file
gdscript_tokenizer.zip

Many thanks for your work

[LaurentOngaro]

I've just made some changes to the 'gdscript_tokenizer.cpp' file, still have bugs, but less.
I still have issues with constants definition and indents below functions definition

The code in my repo has been updated, and the joined file too.
gdscript_tokenizer.zip

[frugs]

@LaurentOngaro I've now rebased my changes on to the latest changes on the master branch. If you still see bugs with this, please let me know!

Thank you for taking an interest in this :)

[LaurentOngaro]

@frugs , I'm currently testing it and incidentally trying to understand my mistakes.
I'll tell you if I find some issues.
Once again, many thanks for your work. It's very useful for me, because misformatted code give me some headaches, and adding spaces around tokens is boring me.
So auto formatting is my best friend

EDIT: After a few tests, All seems OK !

I'll try to maintain my repo as synchronized as possible with the godot master branch to avoid a too big divergence

[LaurentOngaro]

@frugs , I've found a bug in he modified GDscript parser (with auto formatter).

if you use standard python multi-line comments ( introduced by 3 quotation marks), the script editor will show the following error :

Parse Error: Parse error: Unexpected EOL at String.

To reproduce: just create a multi-line comment in a GDscript file, for instance

extends Node2D
"""
this is a comment 
"""

[LaurentOngaro]

@frugs , I found another small issue.
After formatting (using menu or KB shortcut), the text cursor is sent to the top left of the page instead of staying in place.
That could be painful because it breaks the editing flow

[LaurentOngaro]

I've just synchronized my code with the official master branch.
The updated code (with GDscript auto formatting feature) can be found in github

[aaronfranke]

@vnen is currrently working on rewriting GDScript, so this PR would most likely have to be rewritten from scratch for it to be merged. Therefore, I'm closing this and adding "salvageable".

This kind of thing can very likely be an add-on in the asset library, or even a completely separate tool. Further work in this area should not be done unless a proposal is created and approved.

If such a proposal is created, it's likely that it would be approved, considering what @akien-mga said above about this feature being desired. At the time that comment was made, the plan to rewrite GDScript did not exist yet, and things have changed such that this isn't good as-is. Thanks for understanding, and I do indeed look forward to having an official GDScript formatter (whether that be in the engine or otherwise).

[frugs]

@aaronfranke excited to hear there's a rewrite in the works! I imagine a cleaner implementation of this feature would be possible if the tokeniser is rewritten.