fix(elixir) fix regular expression detection

[joshgoebel]

Resolves #3206 and other cleanups.

Changes

• clean up grammar a bit, add more regex sigil examples
• add regex sigils that classify as regex
• add char.escape
• remove dead code

Checklist

• Added markup tests, or they don't apply here because...
• Updated the changelog at CHANGES.md

[angelikatyborska]

I know that this PR merely reformats the list, but it caught my attention because there are a lot of keywords in it that I have never seen in Elixir. Also let my starts by saying that I am not sure if there is a formal definition of a "keyword". I'm using that term rather intuitively.

As far as I know, those are not Elixir keywords at all:

• begin
• break (Inspect.Algebra.break/1 is a function in the standard lib, but it has nothing to do with breaking out of loops like it might do in other languages)
• defined
• ensure
• include
• module
• next (OptionParser.next/2 is a function in the standard lib, it has nothing to do with skipping to another iteration in a loop like it might do in other languages)
• redo
• retry
• return
• until
• while

I'm unsure about:

• self. It exists, but IMO it is a normal macro, not a keyword. I wouldn't expect it to be colored differently than other normal macros (like rem, round, trunc etc).
• then. It's a new macro added in Elixir 1.12 together with tap, I don't think those are keywords.
• with|0. I'm not sure what the pipe and zero mean here. with on its own is definitely a keyword.

Missing IMO:

• if case, cond, and unless qualify as keywords, if and else should too.
• if quote qualifies as a keyword, unquote and unquote_splicing should too.
• receive/after https://hexdocs.pm/elixir/Kernel.SpecialForms.html#receive/1
• try/rescue/catch/after https://hexdocs.pm/elixir/Kernel.SpecialForms.html#try/1
• raise https://hexdocs.pm/elixir/Kernel.html#raise/1
• reraise https://hexdocs.pm/elixir/Kernel.html#reraise/2

[neenjaw]

alias is also listed twice in the list.

[angelikatyborska]

Will this also mark regex modifiers as part of the regex? E.g.

~r(hello)
~r(hello)i
~r(hello)ui
~r(hello)U

[joshgoebel]

No, but we should add that. What are all the valid modifiers in Elixir?

[angelikatyborska]

for sigils r/R, combinations of those letters:

uismxfU

for sigils: w/W, combinations of those letters:

sac

Treating any group of lower and/or uppercase letters immediately after the closing delimiter as a modifier and thus part of the sigil could also be enough.

[joshgoebel]

I added modifiers to ~r/~R. I'll save w for another day/PR since it's not already broken out. Really if we're going to customize these any further we perhaps need a simple data table and then programmatically auto-generate all the sigil rules from that table. Are R and W really the only ones with modifiers?

[angelikatyborska]

Are R and W really the only ones with modifiers?

At the moment yes, but the language syntax already allows for any sigil to have any modifier that it wants. That actually makes me realize that anyone can define their own custom sigil with their own modifiers 😅.

[joshgoebel]

Problems for another day. :) Does the PR look workable for now you think?

[angelikatyborska]

I find it hard to judge the code, but from playing around with tools/developer.html I discovered that there is a problem with uppercase R sigil. It still needs to support escaping the closing delimiter.

For example for those two pairs the output HTML should literally only differ by a single letter (r <-> R), but right now it tries to end the R sigil too early:

Regex.match?(~r|foo\|bar|, "foo")
Regex.match?(~R|foo\|bar|, "foo")


Regex.match?(~r(hello( there\)*!)u, "hello!")
Regex.match?(~R(hello( there\)*!)u, "hello!")

[joshgoebel]

Because I thought uppercase sigils didn't allow escaping. I guess escaping the end sigil character is the exception to the rule?

[angelikatyborska]

Yes, that is the exception. Quoting the docs about the sigil R:

It returns a regular expression pattern without interpolations and without escape characters. Note it still supports escape of Regex tokens (such as escaping + or ?) and it also requires you to escape the closing sigil character itself if it appears on the Regex.

[angelikatyborska]

Thanks a lot 💜

[joshgoebel]

@angelikatyborska Thanks for all the help.