Support semantic tokens

[Eskibear]

depends on eclipse-jdtls/eclipse.jdt.ls#1408

To enable Semantic Highlighting, this PR registers a semantic tokens provider, which actually forward the request to the language server via a delegate command. After semantic highlighting becomes part of LSP, the provider and related delegate commands should be removed, and LS should respond directly to corresponding LSP requests

[fbricon]

@Eskibear can you please fix the conflict?

[fbricon]

isSemanticHighlightingEnabled

[fbricon]

this is too early, the server is not started yet, so causes:

rejected promise not handled within 1 second: Error: command 'java.execute.workspaceCommand' not found
extensionHostProcess.js:920
stack trace: Error: command 'java.execute.workspaceCommand' not found
    at e._tryExecuteCommand (file:///Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:3621:469)
    at file:///Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:3621:350
    at processTicksAndRejections (internal/process/task_queues.js:85:5)
extensionHostProcess.js:920

[Eskibear]

my bad, it should be put in the callback of languageClient.onReady().

[fbricon]

@Eskibear So I can see the tokens with the scope inspector, but, from a user standpoint, it makes no difference, since no colors are assigned to those tokens. Do we need to modify the Java grammar upstream or is this something additional we need to provide?

[fbricon]

if vscode.languages.registerDocumentSemanticTokensProvider is not supported, we should bail (else it'll break in Theia)

[Eskibear]

I see, then I should check if the API exists first, i.e.:

if (languages.registerDocumentSemanticTokensProvider) {
// ...
}

[Eskibear]

no colors are assigned to those tokens

vscode maps token classification to textmate scopes, and then the styles are assigned.
For token type, we added "method" and "variable" here, which are assigned the same color as before.
For modifiers, vscode's default rules doesn't cover that. That should be defined in themes. I imagine that only after we provide token types and modifiers, theme authors are able to add and tune the corresponding rules.

Do we need to modify the Java grammar upstream or is this something additional we need to provide?

We just provide token type and modifiers. Theme authors are responsible for assigning textmate scope and styles for the tokens. Here is a guide for theme authors to add rules.

According to this doc about token styling, users can add custom rules before it's supported by the theme. e.g.

"editor.tokenColorCustomizationsExperimental": {
    "*.static":{
        "fontStyle": "italic"
    },
    "*.final":{
        "fontStyle": "bold"
    },
    "*.deprecated": {
        "fontStyle": "underline"
    },
}

Of course we are able to provide default value of this setting. If provided, ok the users immediately see the difference and know the feature, which is good. The only concern is, it looks a little out of our scope (and might overwrite uesr's theme), and I'm not sure whether we should do that. What do you think?

[fbricon]

If we could provide default semantic highlighting values, that can be disabled with a pref key, that would allow users to benefit from the feature right now and disable our defaults once a theme supports it

[fbricon]

BTW vscode shows deprecated code with strikethrough already, so only static and final would need to be added

[Eskibear]

That makes sense. However, I just tried directly adding above style into configurationDefaults but failed. The contribution point is for default language-specific editor configurations, but editor.tokenColorCustomizationsExperimental is not a language-specific one.

Let me investigate a liitle bit to see if we can embed a simple theme here.

[fbricon]

@Eskibear I don't know if it's possible but if we could at least just document how to do it using editor.tokenColorCustomizations, that'd be enough for me. eg. from https://code.visualstudio.com/docs/cpp/colorization-cpp:

"editor.tokenColorCustomizations": {
        "textMateRules": [
            {
                "scope": "entity.name.type",
                "settings": {
                    "foreground": "#FF0000",
                    "fontStyle": "italic bold underline"
                }
            }
        ]
    }

[Eskibear]

I'm ok with just documenting it, adding vscode's wiki about Semantic-Highlighting-Overview#token-styling as reference. How about adding a section in README? Or below description of pref key java.semanticHighlighting.enabled?

[fbricon]

+1 for a section in the README.

[fbricon]

@Eskibear there's still room for improvement ;-)

[Eskibear]

there's still room for improvement ;-)

Yes, I've also seen the same case occasionally, especially when you change the setting editor.tokenColorCustomizationsExperimental. I once checked the request and response but found no error. It also likely to be an issue of vscode itself about semantic highlighting.

Well it's a good thing that developers are trying it and creating issues for vscode. Now I think it's good to just document how to customize the style. Developers who want to try it could dogfood and help us find potential issues, while others won't be blocked in the meantime.

[fbricon]

Thanks @Eskibear!