Javascript new ES6 identifiers are not handled as such

[jr-grenoble]

There are 2 issues related to ES6 identifiers:

1. In ES6, identifiers are less constrained than in ES5. For instance, you can define a class method named 𑀓ꘌ where 𑀓 is the Brahmi letter Ka \u{11013} and ꘌ is the Vai syllable lengthener \u{a60c}.
   Note that you can check that 𑀓ꘌ is a valid ES6 identifier. Mathias Bynens from Google has created a script that generates the proper regex for ES6 (and ES5) identifiers.

However, the Sunlight javascript rules for syntax highlighting do not handle such identifiers properly and treat them as punctuation, e.g. d.𑀓ꘌ ( { days: 7 } ); will display with 𑀓 and ꘌ highlighted as sunlight-punctuation.

2. It also seems that the processing of such characters discards some of them: they appear as � in the resulting code. E.g. this is the case for Brahmi letter Ka 𑀓.

It is not a font issue, as Consolas is capable of displaying this letter.

Note that there are other UTF-8 characters that exhibit this second issue, this is only an example.

[lwchkg]

@jr-grenoble What's your use case of sunlight highlighter? If you're highlighting code from Node.js, you should probably check https://github.com/lwchkg/sunlight-x for a version that will actually be maintained.

BTW, why not also to file a issue in https://github.com/lwchkg/sunlight-x/issues? I'd both languages and webpack in my todos, but if you file them I'd give these more priority.

[jr-grenoble]

@lwchkg In fact I'm an indirect user of sunlight: I use it through jsdoc templated by docstrap, indeed to document and highlight node code.

[lwchkg]

@jr-grenoble I see. I'll add an issue at docstrap to see if they're interested. Anyway, I don't see any reason to close the issue here though.

[jr-grenoble]

You are correct,, I'm reopening it here. Thanks !