Clarify purpose of superType under semanticTokenTypes

[radeksimko]

I am not sure if this should be filed as a bug or a feature, so to begin I'd like docs to be updated to clarify what superType is meant to be used for and what implications do the values have.

Currently the docs seem very vague on this topic:
https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide#custom-token-types-and-modifiers

vscode/src/vs/workbench/services/themes/common/tokenClassificationExtensionPoint.ts

Lines 42 to 47 in 94c9ea4

	superType: {
	type: 'string',
	description: nls.localize('contributes.semanticTokenTypes.superType', 'The super type of the semantic token type'),
	pattern: typeAndModifierIdPattern,
	patternErrorMessage: nls.localize('contributes.semanticTokenTypes.superType.format', 'Super types should be in the form letterOrDigit[_-letterOrDigit]*'),
	},

My assumption was that superType would be used as an alternative/fallback token type if the theme doesn't support the custom token type.

"semanticTokenTypes": [
      {"id": "hcl-attrName", "superType": "property"},
      {"id": "hcl-blockType", "superType": "type"},
      {"id": "hcl-blockLabel", "superType": "enumMember"}
]

In the example above I expected VSCode to use e.g. hcl-blockType if the theme supports it or type if it doesn't. Instead it just always assigns the custom types regardless of whether themes support it or not.

Instead I had to reverse engineer the token type mapping to TM scopes from here

vscode/src/vs/platform/theme/common/tokenClassificationRegistry.ts

Lines 541 to 582 in 02e8bd1

	// default token types
	registerTokenType('comment', nls.localize('comment', "Style for comments."), [['comment']]);
	registerTokenType('string', nls.localize('string', "Style for strings."), [['string']]);
	registerTokenType('keyword', nls.localize('keyword', "Style for keywords."), [['keyword.control']]);
	registerTokenType('number', nls.localize('number', "Style for numbers."), [['constant.numeric']]);
	registerTokenType('regexp', nls.localize('regexp', "Style for expressions."), [['constant.regexp']]);
	registerTokenType('operator', nls.localize('operator', "Style for operators."), [['keyword.operator']]);
	registerTokenType('namespace', nls.localize('namespace', "Style for namespaces."), [['entity.name.namespace']]);
	registerTokenType('type', nls.localize('type', "Style for types."), [['entity.name.type'], ['support.type']]);
	registerTokenType('struct', nls.localize('struct', "Style for structs."), [['entity.name.type.struct']]);
	registerTokenType('class', nls.localize('class', "Style for classes."), [['entity.name.type.class'], ['support.class']]);
	registerTokenType('interface', nls.localize('interface', "Style for interfaces."), [['entity.name.type.interface']]);
	registerTokenType('enum', nls.localize('enum', "Style for enums."), [['entity.name.type.enum']]);
	registerTokenType('typeParameter', nls.localize('typeParameter', "Style for type parameters."), [['entity.name.type.parameter']]);
	registerTokenType('function', nls.localize('function', "Style for functions"), [['entity.name.function'], ['support.function']]);
	registerTokenType('member', nls.localize('member', "Style for member functions"), [], 'method', 'Deprecated use `method` instead');
	registerTokenType('method', nls.localize('method', "Style for method (member functions)"), [['entity.name.function.member'], ['support.function']]);
	registerTokenType('macro', nls.localize('macro', "Style for macros."), [['entity.name.function.preprocessor']]);
	registerTokenType('variable', nls.localize('variable', "Style for variables."), [['variable.other.readwrite'], ['entity.name.variable']]);
	registerTokenType('parameter', nls.localize('parameter', "Style for parameters."), [['variable.parameter']]);
	registerTokenType('property', nls.localize('property', "Style for properties."), [['variable.other.property']]);
	registerTokenType('enumMember', nls.localize('enumMember', "Style for enum members."), [['variable.other.enummember']]);
	registerTokenType('event', nls.localize('event', "Style for events."), [['variable.other.event']]);
	registerTokenType('decorator', nls.localize('decorator', "Style for decorators & annotations."), [['entity.name.decorator'], ['entity.name.function']]);
	registerTokenType('label', nls.localize('labels', "Style for labels. "), undefined);
	// default token modifiers
	registry.registerTokenModifier('declaration', nls.localize('declaration', "Style for all symbol declarations."), undefined);
	registry.registerTokenModifier('documentation', nls.localize('documentation', "Style to use for references in documentation."), undefined);
	registry.registerTokenModifier('static', nls.localize('static', "Style to use for symbols that are static."), undefined);
	registry.registerTokenModifier('abstract', nls.localize('abstract', "Style to use for symbols that are abstract."), undefined);
	registry.registerTokenModifier('deprecated', nls.localize('deprecated', "Style to use for symbols that are deprecated."), undefined);
	registry.registerTokenModifier('modification', nls.localize('modification', "Style to use for write accesses."), undefined);
	registry.registerTokenModifier('async', nls.localize('async', "Style to use for symbols that are async."), undefined);
	registry.registerTokenModifier('readonly', nls.localize('readonly', "Style to use for symbols that are readonly."), undefined);

and then re-declare this mapping within semanticTokenScope to achieve this.

"semanticTokenScopes": [
      {
        "scopes": {
          "hcl-attrName": ["variable.other.property"],
          "hcl-blockType": ["entity.name.type"],
          "hcl-blockLabel": ["variable.other.enummember"],
        }
      }
]

If this is not how it was meant to work - what is the purpose of that field? If this is how it's meant to work - I guess we can turn this into a bug.

---

This may be related to #103109 although I feel like I may be asking/expecting the exact opposite - that is for VSCode to figure out what the right token to use, and both extensions and themes to stay as simple as possible.

[aeschli]

Yes, you still have to provide token scopes for your custom token type, even if you have a super type. Typically, you don't take exactly the same scopes, but a more specific scopes. e.g. instead of variable.other.property you will have vaiable.other.property.attr.
To see the scopes that the built-in types use, see https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide#predefined-textmate-scope-mappings

What the super type will help you with is when the theme has rules for semanticTokenColors.
E.g. https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide#semantic-coloring-in-color-themes

{
  "name": "Red Theme",
  "semanticTokenColors": {
    "property.readonly:java": "#ff0011"
  }
}

So here also your custom property hcl-attrName will be matched.

Does that make sense?

I guess we could change the behavior, but I'm a bit afraid of breaking existing users.

So I would prefer to just update the docs.