[HOLD for payment 2023-06-01] [$1000] Mentions are being rendered inside of code and codefence

[puneetlath]

Mentions are not supposed to get rendered inside codeblocks and codefences. This is working as expected when the mention is the only text inside the code, however, it doesn't work when mentions are included with other text.

The typed text for these four was:

```
@[email protected]
```

```
hey @[email protected] what's up?
```

`@[email protected]`

`hey @[email protected] what's up?`

All of these should get rendered without the <mention-user> or <mention-here> tags, but 2 and 4 are getting rendered with those tags.

cc @getusha happy to give you the first shot at this since you implemented the original.

Upwork Automation - Do Not Edit

• Upwork Job URL: https://www.upwork.com/jobs/~01408a5d65345b1b64
• Upwork Job ID: 1658230072642449408
• Last Price Increase: 2023-05-15

Issue reporter @dhanashree-sawant
Also reported by @parasharrajat

[melvin-bot]

Triggered auto assignment to @Christinadobrzyn (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details.

[melvin-bot]

Job added to Upwork: https://www.upwork.com/jobs/~01408a5d65345b1b64

[melvin-bot]

Bug0 Triage Checklist (Main S/O)

• This "bug" occurs on a supported platform (ensure Platforms in OP are ✅)
• This bug is not a duplicate report (check E/App issues and #expensify-bugs)
  • If it is, comment with a link to the original report, close the issue and add any novel details to the original issue instead
• This bug is reproducible using the reproduction steps in the OP. S/O
  • If the reproduction steps are clear and you're unable to reproduce the bug, check with the reporter and QA first, then close the issue.
  • If the reproduction steps aren't clear and you determine the correct steps, please update the OP.
• This issue is filled out as thoroughly and clearly as possible
  • Pay special attention to the title, results, platforms where the bug occurs, and if the bug happens on staging/production.
• I have reviewed and subscribed to the linked Slack conversation to ensure Slack/Github stay in sync

[melvin-bot]

Triggered auto assignment to @anmurali (External), see https://stackoverflow.com/c/expensify/questions/8582 for more details.

[melvin-bot]

Triggered auto assignment to Contributor-plus team member for initial proposal review - @rushatgabhane (External)

[melvin-bot]

Triggered auto assignment to @stitesExpensify (External), see https://stackoverflow.com/c/expensify/questions/7972 for more details.

[rushatgabhane]

cc: @bernhardoj @tienifr feel free to submit a proposal because I've seen you both work on markdown issues

[dukenv0307]

Proposal

Please re-state the problem that we are trying to solve in this issue.

Mentions are being rendered inside of code and codefence

What is the root cause of that problem?

https://github.com/Expensify/expensify-common/blob/5f4672c84e19122fc77b4bf1ac092d4919472112/lib/ExpensiMark.js#L74-L83

We allow the mention is rendered inside &#x60;|<code>|<pre>

What changes do you think we should make in order to solve the problem?

In this modified regular expression, the negative lookahead assertion

(?!&#x60;|<code>|<pre>)

is added at the beginning. It ensures that the pattern will not match if it starts with any of the specified strings.
Update like this

/^(?!&#x60;|<code>|<pre>)[`.a-zA-Z]?@+${CONST.REG_EXP.EMAIL_PART}/gm

Do same thing at here mention

What alternative solutions did you explore? (Optional)

we can use this regex if don't want to use negative lookahead

/^(?!.*(&#x60;|<code>|<pre>))[`.a-zA-Z]?@+${CONST.REG_EXP.EMAIL_PART}/gm

Result

Both ways work well

[Thanos30]

Proposal

Please re-state the problem that we are trying to solve in this issue.

The problem is that when there is a prefix on mentions inside the codeblocks, or multiple mentions, it will break the logic. While the initial bug states that it breaks when other text is included, you can confirm that this happens when:

• There is prefix text (text before the mention)
• When there are multiple mentions

What is the root cause of that problem?

The root of the problem is the Regex code that handles the mentions, specifically the beginning of it:
((&#x60;|<code>|<pre>)\\s*?)?
The above regex will look for either the Grave Accent symbol, <code> or <pre> element, followed by 0 or more white-spaces. This makes the code break when there is text after these elements, as shown on the example above.

hey @[email protected] what's up? <--- This breaks due to the "hey" before the mention.

Along with the above, the regex also doesn't cover multiple mentions inside the same messages.

What changes do you think we should make in order to solve the problem?

We need to reconfigure the Regex to cover all the possible scenarios, including prefix and multiple mentions.

What alternative solutions did you explore? (Optional)

The possibility of not using regex on this one, but possibly exclude mentions via components when they are inside the CodeRenderer.

[melvin-bot]

📣 @Thanos30! 📣
Hey, it seems we don’t have your contributor details yet! You'll only have to do this once, and this is how we'll hire you on Upwork.
Please follow these steps:

1. Get the email address used to login to your Expensify account. If you don't already have an Expensify account, create one here. If you have multiple accounts (e.g. one for testing), please use your main account email.
2. Get the link to your Upwork profile. It's necessary because we only pay via Upwork. You can access it by logging in, and then clicking on your name. It'll look like this. If you don't already have an account, sign up for one here.
3. Copy the format below and paste it in a comment on this issue. Replace the placeholder text with your actual details.
   
   Format:

Contributor details
Your Expensify account email: <REPLACE EMAIL HERE>
Upwork Profile Link: <REPLACE LINK HERE>

[Thanos30]

Contributor details
Your Expensify account email: [email protected]
Upwork Profile Link: https://www.upwork.com/freelancers/~01db0383e77efb1881

[melvin-bot]

✅ Contributor details stored successfully. Thank you for contributing to Expensify!

[getusha]

Proposal

What is the root cause of that problem?

in order to detect if the mention is inside code or pre tags we're including it to later use it and avoid replacing the mention. but when we include some text before the mention it will match only the mention avoiding the <code> because \s*? only matches space and new lines.

What changes do you think we should make in order to solve the problem?

We should update the regex as follows

// userMentions
new RegExp(`((&#x60;|<code>|<pre>)+(.|\\n)*)?[\`.a-zA-Z]?@+${CONST.REG_EXP.EMAIL_PART}`, 'gm')

//hereMentions
/((&#x60;|<code>|<pre>)+(.|\n)*)?[`.a-zA-Z]?@here\b/gm

Explanation: adding (.|\\n)*)?[\.a-zA-Z]?` to match every character if the mention is inside code or pre block to be used for our logic here

Result

What alternative solutions did you explore? (Optional)

cc @puneetlath

[bernhardoj]

Proposal

Please re-state the problem that we are trying to solve in this issue.

We don't want a mention to be rendered inside a code block or code fence.

What is the root cause of that problem?

Currently, to prevent that, we check if the matches string contains either <code>|<pre>|&#x60;.
https://github.com/Expensify/expensify-common/blob/f8da8e51257d7ea65bf6b27759bd7053f0ec4c7e/lib/str.js#L951-L953

However, if the mention is not at the start of the string, it won't be included into the regex match.
https://github.com/Expensify/expensify-common/blob/f8da8e51257d7ea65bf6b27759bd7053f0ec4c7e/lib/ExpensiMark.js#L75-L76

Notice that we accept <code>|<pre>|&#x60; at the start of the regex.

What changes do you think we should make in order to solve the problem?

The current validation is not good to prevent mention inside a code block/fence. The better way is to not match the mention inside code block/fence in the first place. To do that,

1. we can remove <code>|<pre>|&#x60; from the start of the regex because it won't be needed anymore
2. Add (?![^<]*(<\\/pre>|<\\/code>|<\\/a>)) to the end of the regex (userMentions and hereMentions) to prevent match the mention if it's inside a codeblock/fence and also a link. Why anchor tag too? Later.
3. Remove containsInvalidTag from Str.isValidMention
   Result:

(\\s*?)?[\`.a-zA-Z]?@+${CONST.REG_EXP.EMAIL_PART}(?![^<]*(<\/pre>|<\/code>|<\/a>))`, 'gm')

This new added regex already being used in some rules, for example this one:
https://github.com/Expensify/expensify-common/blob/f8da8e51257d7ea65bf6b27759bd7053f0ec4c7e/lib/ExpensiMark.js#L107-L111

Why also add anchor tag to solve this issue #18857. I added a proposal on that issue with the same solution.

@rushatgabhane thx for tagging, btw

[rushatgabhane]

@puneetlath I like @bernhardoj's proposal
They have the best explanation of the root cause, and the best solution.

C+ reviewed 🎀 👀 🎀

[getusha]

Any comment on my proposal? @rushatgabhane

[rushatgabhane]

@chiragxarora what? howw

[Thanos30]

It's not a duplicate exactly. It's just that the solution of this issue solves both of the issues. Both of them come down to the regex code the solution changes.

[chiragxarora]

issue is about mentions being rendered inside codeblock.
What I reported was "mentions render in multi-line code block but not in in-line codeblock. Consistency should be there, either they render in both or in none". I didn't know the ideal behaviour if they should render or not
@rushatgabhane

[chiragxarora]

It's not a duplicate exactly. It's just that the solution of this issue solves both of the issues. Both of them come down to the regex code the solution changes.

I'd still call it dupe because main BUG is about mentions rendering inside codeblock. I don't see how these appear different

[bernhardoj]

I think @chiragxarora said this one is a dupe because this issue #18893 is created earlier but we handled this one first, which could be confusing for most of us 😄.

[chiragxarora]

I think @chiragxarora said this one is a dupe because this issue #18893 is created earlier but we handled this one first, which could be confusing for most of us 😄.

that is also true but it's dupe for the other reason too which I mentioned above

[puneetlath]

I think that's already been handled @chiragxarora

[parasharrajat]

@puneetlath I think I already reported this in #18167 (comment).

[bernhardoj]

App PR is ready for review

[melvin-bot]

Reviewing label has been removed, please complete the "BugZero Checklist".

[melvin-bot]

The solution for this issue has been 🚀 deployed to production 🚀 in version 1.3.17-5 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

• Don't allow mention inside code block/fence and link #19333

If no regressions arise, payment will be issued on 2023-06-01. 🎊

After the hold period is over and BZ checklist items are completed, please complete any of the applicable payments for this issue, and check them off once done.

• External issue reporter @dhanashree-sawant @parasharrajat
• Contributor that fixed the issue @bernhardoj
• Contributor+ that helped on the issue and/or PR @rushatgabhane

As a reminder, here are the bonuses/penalties that should be applied for any External issue:

• Merged PR within 3 business days of assignment - 50% bonus
• Merged PR more than 9 business days after assignment - 50% penalty

[melvin-bot]

BugZero Checklist: The PR fixing this issue has been merged! The following checklist (instructions) will need to be completed before the issue can be closed:

• [@rushatgabhane] The PR that introduced the bug has been identified. Link to the PR:
• [@rushatgabhane] The offending PR has been commented on, pointing out the bug it caused and why, so the author and reviewers can learn from the mistake. Link to comment:
• [@rushatgabhane] A discussion in #expensify-bugs has been started about whether any other steps should be taken (e.g. updating the PR review checklist) in order to catch this type of bug sooner. Link to discussion:
• [@rushatgabhane] Determine if we should create a regression test for this bug.
• [@rushatgabhane] If we decide to create a regression test for the bug, please propose the regression test steps to ensure the same bug will not reach production again.
• [@puneetlath] Link the GH issue for creating/updating the regression test once above steps have been agreed upon:

[puneetlath]

Upwork contracts sent.

@rushatgabhane friendly reminder about the checklist!

[puneetlath]

@rushatgabhane friendly reminder about the checklist so that I can pay this out tomorrow.

[rushatgabhane]

1. The PR that introduced the bug has been identified. Link to the PR: [No QA] Add rule to recognize user @here mentions expensify-common#532

2. The offending PR has been commented on, pointing out the bug it caused and why, so the author and reviewers can learn from the mistake. Link to comment: https://github.com/Expensify/expensify-common/pull/532/files#r1212698292

3. A discussion in #expensify-bugs has been started about whether any other steps should be taken (e.g. updating the PR review checklist) in order to catch this type of bug sooner. Link to discussion: All cases weren't tested. Fails checklist. We added unit tests for it, so this bug won't happen again.

4. Determine if we should create a regression test for this bug. No because we added unit tests.

5. If we decide to create a regression test for the bug, please propose the regression test steps to ensure the same bug will not reach production again - N.A.

[puneetlath]

All paid. Thanks everyone!