THREESCALE-9492: Bot protection: Remove suspicious only mode

[jlledom]

What this PR does / why we need it:

This comes from #3620. That PR was closed because it's not worth to reinvent the wheel and maintain our own spam protection system. So let's give it another try. In this PR the old Spam protection is removed and replaced by a much simpler bot protection based on reCaptcha v3.

There's no Suspicious only mode anymore. Now the user can only choose between None and reCAPTCHA:

All tenants configured to Suspicious only are now treated as reCAPTCHA.

ReCaptcha v3 doesn't add any prompt or expects the user to perform any action. The script gathers some information and sends it to reCaptcha in order to compute a value between 0 and 1 on how likely is the client human. A value of 0 means reCaptcha is sure the client is a bot, while 1 means the client is human.

When the reCaptcha challenge fails, the received data is rejected and a flash error is returned. This PR also fixes the UI, since the deployed version isn't returning the proper flash message in all screens.

Which issue(s) this PR fixes

THREESCALE-9492

Verification steps

As far as I know, there's no way to force reCaptcha to return a fake result when working on a development environment, so it basically always give me a high score and will probably be the same for reviewers. The only way to make it fail is to set recaptcha_min_bot_score (here) to 0.99 temporary. That way even high scores are considered bots and you can actually see the error message in the UI.

Design

The old spam protection code is removed, and replaced by a much more simpler version of it:

• ThreeScale::BotProtection::LEVELS (code): Available protection levels. Currently, only :none and :captcha.
• ThreeScale::BotProtection::Controller (code): A module to be included from controllers. It implements the method bot_check which returns true when the client is human and false when the client is a bot.
  • This module is included from:
    • DeveloperPortal::SignupController
    • DeveloperPortal::Admin::Account::PasswordsController
    • Provider::SignupsController
• ThreeScale::BotProtection::Form (code): A module to be included from form builders. It implements the method bot_protection_inputs which returns the reCaptcha inputs when bot protection is enabled, or nothing otherwise.
  • This module is included from:
    • ThreeScale::SemanticFormBuilder
    • Liquid::Forms::BotProtected

How to review

Since reviewing 60 files is a nightmare, I created a few commits to split changes into smaller and easier to review change sets:

• Remove old logic (f9a7c4d): This one is pretty straightforward, just removing old code.
• Update recaptcha gem (7b216fa): Easy and self-explanatory one.
• Add new logic (ce22cfa): This one adds the new logic explained above
• UI updates (6f0b42e): I removed old JS code from Developer Portal default templates. No need to update the old ones, though. reCaptcha v3 should work fine for all existing templates.
• Update tests (db0a482): Adapt existing tests to the new bot protection workflow
• Add tests for the new logic (2497f19): Add unit and integration tests for the new modules.

[akostadinov]

Great work! I have many comments but a lot is nitpicks, nothing very big I believe.

[mayorova]

I haven't checked out the changes yet, but looking at the new screenshot, I am wondering:

1. Does it make sense to keep the values as "Never" and "Always"? Without the "in-between" value this looks a bit weird to me 🙃 I would probably have it as "Enabled" or "Disabled", or even convert it into a checkbox.

2. As we are changing the behavior and will need to write release notes anyway, maybe we can take advantage of this and improve the wording on the UI also? e.g. change "Spam protection" to "Bot protection" and also reword the hints? "Spam protection against users that are not signed in" sounds weird, and it seems like we want to protect against any non-logged-in users (including legitimate ones) 😅
   We may ask the docs or the UX team to help us find a better wording.

[jlledom]

@mayorova This issue is very related to THREESCALE-10579 which requires a redesign of the Spam Protection settings screen. For now, I'm calling the options "None" and "reCAPTCHA":

It's not definitive but my intention is to leave it as that for this PR. I'll work next on THREESCALE-10579. I'll discuss with UX to generate a new design for the screen, including bot protection for the login screen and renaming "Spam protection" to "Bot protection" everywhere.

[mayorova]

This looks good. Well done @jlledom 🏆

A couple of things though...

I don't quite understand yet how reCAPTCHA v3 works, but I read somewhere that it should be enabled not only on the pages that need to be protected, but also on others - in order for the system to "learn" about the legitimate interactions.

Also, it seems that the keys would need to be replaced to v3? Although it seems to be working with the old v2, but the docs say explicitly that the keys are not compatible across versions.

[jlledom]

I don't quite understand yet how reCAPTCHA v3 works, but I read somewhere that it should be enabled not only on the pages that need to be protected, but also on others - in order for the system to "learn" about the legitimate interactions.

Yeah, I found that also in the docs. But reCAPTCHA adds a badge in the bottom-right corner, and I think that's too intrusive to place it in all screens. On the other hand, I would've preferred to use non-Google alternatives to reCAPTCHA because I don't like the idea of allowing Google to gather data from our users. I didn't do it because I feel that would require some previous debate and I don't want to delay this feature. And if I'm not wrong, we use analytics anyway.

Also, it seems that the keys would need to be replaced to v3? Although it seems to be working with the old v2, but the docs say explicitly that the keys are not compatible across versions.

I tried with v2 keys locally and it failed. The reCAPTCHA badge doesn't appear and there are some errors in the console. But Recaptcha.captcha_configured? returns true in this case, which in my opinion is wrong. I'll fix this.

[jlledom]

I tried with v2 keys locally and it failed. The reCAPTCHA badge doesn't appear and there are some errors in the console. But Recaptcha.captcha_configured? returns true in this case, which in my opinion is wrong. I'll fix this.

I've been working on this without success. I don't think there's a way to know whether the keys are valid, other than sending an fake query to google with the given key and check the response. And IMO that's and ugly workaround I'd prefer to avoid.

@mayorova could you confirm you get the error reCAPTCHA verification failed, please try again. when you try to send a protected form with invalid keys? That's the expected behavior and from your message I understood you don't see this error.

[mayorova]

@mayorova could you confirm you get the error reCAPTCHA verification failed, please try again. when you try to send a protected form with invalid keys? That's the expected behavior and from your message I understood you don't see this error.

Yeah, I confirm, I get either that or Bot protection failed.. I think I was checking on a wrong page at first 😬

[mayorova]

Good job 👍

[akostadinov]

Look great! I have one nitpick but bot protection is way more sensible now.

[akostadinov]

Thanks!