Semantic token modifiers do not appear to affect colors

[stamblerre]

The Go language server uses the semantic tokens feature and applies modifiers to certain semantic tokens, such as strings. The goal is to achieve special colorization for formatting directives and special characters, e.g.:

However, these modifiers do not seem to affect the colorization of the tokens. Setting a specific color for the modifier in the VS Code settings via editor.semanticTokenColorCustomizations did work correctly, however.

From microsoft/language-server-protocol#1281:

Yes, this would have to be filed against VSCode. You want VScode to add a TextMate scope mapping for string.modification.

string.modification is not (yet) an established way to represent placeholders in strings, so I'm not sure we will just add that in VSCode. But extensions can also such TextMate scope mappings: https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide#custom-textmate-scope-mappings (second example)

VS Code version:

1.57.0-insider
286b643
x64

OS version:

Mac OS Big Sur 11.3.1

[vscodebot]

(Experimental duplicate detection)
Thanks for submitting this issue. Please also check if it is already covered by an existing one, like:

• Semantic Highlighting is not tokenizing in untitled Javascript file (#121337)

[aeschli]

I tried to reproduce with the go extension, but did not see any semantic tokens.

@stamblerre What semantic tokens is go emitting?

Also can you comment on this:

string.modification is not (yet) an established way to represent placeholders in strings, so I'm not sure we will just add that in VSCode. But extensions can also such TextMate scope mappings: https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide#custom-textmate-scope-mappings (second example)

[stamblerre]

The Go extension has semantic tokens disabled by default, since we need to resolve these issues before enabling them. This feature can be enabled through the following setting:

"gopls": {
	"ui.semanticTokens": true
}

Also can you comment on this:

string.modification is not (yet) an established way to represent placeholders in strings, so I'm not sure we will just add that in VSCode. But extensions can also such TextMate scope mappings: https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide#custom-textmate-scope-mappings (second example)

I think the larger issue is that none of the modifiers (modification was just an example I chose) appear to have any effect on string colorization, so we're not exactly sure how to correctly provide semantic tokens to distinguish formatting directives and newlines from other characters. Our thinking was that these are still strings, so we should report the whole string as a string token, but is the expectation that we should use a different token type?

We can try to use the customization, but we were just surprised that modifiers would have no effect in the default theme.

[aeschli]

Various issues here:

It realize our Dark+/Light+ themes don't colorize string placeholders with TextMate grammars. The constant.other.placeholder scope that is used by the go grammar and other grammars is not covered by the themes.
I fixed that by updating our default color themes (dark+ and light+): 2fb5525

On semantic highlighting:

• You can't expect that all combinations of type/modifiers have a meaning and result in different coloring.. We have a set of default types and modifier combinations for which we have a mapping to TextMate theme (see here). In the end it depends on the actual theme if there's a color defined for that scope.
• We don't have a type to represent string placeholders. That's a valid request. We have [semantic] proposals for new standard semantic token types  #97063 as issue to collect such requests and when there's multiple requests for a given type, we will add it as a new standard token. I'd suggest to call it stringPlaceholder. In the meantime, your extension can define a custom type and scope mapping.
• Strings and string placeholders might not be the typical use case for semantic highlighting . I'd recommend to leave that to the TextMate grammar. The semantic highlighting is something to go on top of syntax highlighting and typically focuses on symbols where a full AST and resolution is needed to evaluate the type of a symbol. E.g. only when resolving an identifier in the file or project it's possible to know if its member, parameter, function and so on.
  But of course that depends on the language. For some languages it's very hard to have a good TextMate grammar, so doing more with semantic highlighting makes sense.

[stamblerre]

Thank you for the detailed explanation. We will try to experiment with customizations for our extension and suggest the stringPlaceholder token type.

[aeschli]

Closing the issue then. Please reopen if you find something that needs to be fixed.