Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added rate-limiting facility #163

Merged
merged 17 commits into from
Feb 1, 2021
Merged

Added rate-limiting facility #163

merged 17 commits into from
Feb 1, 2021

Conversation

kdhttps
Copy link
Contributor

@kdhttps kdhttps commented Nov 23, 2020

Used express-rate-limit package, currently fetching configurations from production.js file

closes #139

@kdhttps kdhttps self-assigned this Nov 23, 2020
@kdhttps kdhttps force-pushed the 139_feat_rate_limiting branch from 152b4b8 to 51dbe48 Compare November 23, 2020 15:55
@codecov
Copy link

codecov bot commented Nov 23, 2020

Codecov Report

Merging #163 (66b04cd) into master (e6b0031) will increase coverage by 0.54%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #163      +/-   ##
==========================================
+ Coverage   68.82%   69.36%   +0.54%     
==========================================
  Files          31       32       +1     
  Lines         680      692      +12     
==========================================
+ Hits          468      480      +12     
  Misses        212      212              
Impacted Files Coverage Δ
config/default.js 100.00% <ø> (ø)
config/production.js 100.00% <ø> (ø)
config/test.js 100.00% <100.00%> (ø)
server/app-factory.js 94.11% <100.00%> (+0.56%) ⬆️
server/app.js 95.12% <100.00%> (+0.12%) ⬆️
server/utils/rate-limiter.js 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e6b0031...659db5a. Read the comment docs.

Copy link
Contributor

@christian-hawk christian-hawk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice that you created BDD scenario integration tests for new feature 👏🏻
I just need to understand (as commented) how it behaves.

I would also request the following change:

unit test rate-limiter.test.js for rate-limiter.js can be created (actually using TDD the test should be created before developing), for example:
I will also add TDD metodology examples, to illustrate the metodology applied to the current simple file.

  • assert that window exists (TDD: test will break, so now you can create the first line of code in rate-limiter.js: let window, test passes...)
  • assert that window is a const (TDD: test will break, so now to pass you can just const window = '', test passes)
  • assert that window has the same value from default.js, test.js or production.js (TDD: test break... test will pass when you rafactor const window to receive config.get..., test passes)
  • assert that max exists (TDD: test will break, so now you can create let max, test passes...)
  • assert that max is a const (TDD: test will break, so now to pass you can just const max = '', test passes)
  • assert that max has the same value from default.js, test.js or production.js (TDD: test break... test will pass when you rafactor const max to receive config.get...)
  • assert that rateLimit exists (TDD: after this test we create rateLimit (like let rateLimit) in rate-limiter.js file, so test passes)
  • assert that rateLimit is strict equal require('express-rate-limit') (TDD: test will break, cause rateLimit has not this value. So now we refactor existing rateLimit to const rateLimit = require('express-rate-limit') - test passes
  • assert that rateLimiter is equal rateLimit){...}. (TDD: test will break, so you code = rateLimit({...})), test passes

I hope my comments are useful.

server/rate-limiter.js Outdated Show resolved Hide resolved
server/rate-limiter.js Outdated Show resolved Hide resolved
server/rate-limiter.js Outdated Show resolved Hide resolved
test/features/steps/rate-limiting-steps.js Outdated Show resolved Hide resolved
test/features/rate-limiting.feature Show resolved Hide resolved
test/app-factory.test.js Show resolved Hide resolved
@kdhttps kdhttps force-pushed the 139_feat_rate_limiting branch from 51dbe48 to 29669de Compare November 25, 2020 11:38
@kdhttps
Copy link
Contributor Author

kdhttps commented Nov 25, 2020

unit test rate-limiter.test.js for rate-limiter.js can be created (actually using TDD the test should be created before developing), for example:
I will also add TDD metodology examples, to illustrate the metodology applied to the current simple file.

@christian-hawk my friend, Actually my first code was like

const rateLimit = require('express-rate-limit')
const config = require('config')

const rateLimiter = rateLimit({
  windowMs: config.get('rateLimitWindowMs'),
  max: config.get('rateLimitMaxRequestAllow'),
  message: `You have exceeded the max requests`,
  headers: true
})

I covered test-cases about it in app-factory.test.js
but
before push code, I though to change message and due to message, I took extra two variables and forgot to cover test cases about new code refactor, my mistake.

I pushed review changes, please review again and thank you for great review :)

Copy link
Contributor

@christian-hawk christian-hawk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nicely done.
Now we just need to resolver the cucumber's 2 conversations and we are done!

Also please check if https://github.com/GluuFederation/gluu-passport/runs/1453194618 this codacy issue is a false-positive?

test/features/rate-limiting.feature Show resolved Hide resolved
test/features/steps/rate-limiting-steps.js Outdated Show resolved Hide resolved
used express-rate-limit package, currently fetching configurations from production.js file

feat #139
@kdhttps kdhttps force-pushed the 139_feat_rate_limiting branch from d2b320a to 51b6ba3 Compare December 2, 2020 07:28
Copy link
Contributor

@christian-hawk christian-hawk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still have an unresolved conversation from former review to dinamically tests endpoints behaviour.

@kdhttps
Copy link
Contributor Author

kdhttps commented Dec 9, 2020

@christian-hawk what you think now? #163 (comment)

Copy link
Contributor

@christian-hawk christian-hawk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well done made.
We just need to do the correct features and steps dynamically

test/features/steps/enpoint-metric-steps.js Outdated Show resolved Hide resolved
test/rate-limiter.test.js Show resolved Hide resolved
test/app-factory.test.js Show resolved Hide resolved
@christian-hawk christian-hawk marked this pull request as draft December 11, 2020 15:54
@christian-hawk
Copy link
Contributor

I'm changing this to draft as it's still a work in progress.

Copy link
Contributor

@christian-hawk christian-hawk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice structure.
Please check priority comments on rate-limiting-steps.js. I understand after that we are done.

server/app.js Show resolved Hide resolved
test/features/steps/rate-limiting-steps.js Outdated Show resolved Hide resolved
test/features/steps/rate-limiting-steps.js Outdated Show resolved Hide resolved
test/features/steps/rate-limiting-steps.js Outdated Show resolved Hide resolved
test/features/steps/rate-limiting-steps.js Outdated Show resolved Hide resolved
test/helper.js Outdated Show resolved Hide resolved
@kdhttps kdhttps force-pushed the 139_feat_rate_limiting branch 2 times, most recently from adf0371 to 0005a8c Compare December 17, 2020 12:16
@kdhttps kdhttps force-pushed the 139_feat_rate_limiting branch from 0005a8c to 6210dca Compare December 17, 2020 12:20
…configuration file values

As those are not blackbox tests, we should test against the static values from configuration file
@christian-hawk
Copy link
Contributor

here we can see that rate limiting counting is not being reset for each scenario example, I'm working on that.

I also changed to static config values, you got a point here.

After thinking I also understand that changing config file may go to our blackbox stage-testing instead of our whitebox functional tests (here).

@christian-hawk
Copy link
Contributor

run tests

@christian-hawk
Copy link
Contributor

As you requested, I did the changes for accurate tests.
Now we need only to check codacy reports and maybe create a helper or cucumber setup.js to avoid dry.

@christian-hawk
Copy link
Contributor

run tests

1 similar comment
@christian-hawk
Copy link
Contributor

run tests

@kdhttps kdhttps marked this pull request as ready for review January 12, 2021 09:34
@christian-hawk
Copy link
Contributor

run tests

@christian-hawk
Copy link
Contributor

@kdhttps
IDP initiated flow failed:

13:04:45 ============================================================================
13:04:45 
13:04:45 - Provider type: IDP
13:04:45 - Type: passport-saml
13:04:45 - Flow: idp-initiated
13:04:45 - Passport Provider ID: saml-idpinit
13:04:45 TODO: Creating Trust Relation on t3.techno24x7.com automatically
13:04:45 
13:04:45 - User data:
13:04:45 USER_NAME: hansdoe
13:04:45 USER_PASSWORD: test123
13:04:45 USER_MAIL: [email protected]
13:04:45 USER_GIVEN: hans
13:04:45 USER_SUR: doe
13:04:45 
13:04:45 ----------------------------------------------------------------------------
13:04:45 RUNNING BLACKBOX BDT TESTS:
13:04:45 ----------------------------------------------------------------------------
13:04:45 
13:04:45 Feature: accessing protected content # tests/behaver/features/protected-content.feature:1
13:04:50 
13:04:50   Scenario: Not authenticated user access protected content  # tests/behaver/features/protected-content.feature:3
13:04:50     Given that user exists in provider                       # tests/behaver/features/steps/protected-content.py:37
13:04:50     When user tries to access protected content page         # tests/behaver/features/steps/protected-content.py:126
13:04:54     Then user should be redirected to login page             # tests/behaver/features/steps/protected-content.py:140
13:05:08 
13:05:08   Scenario: Authenticated user access protected content  # tests/behaver/features/protected-content.feature:9
13:05:08     Given that user exists in provider                   # tests/behaver/features/steps/protected-content.py:37
13:05:08     And user is authenticated                            # tests/behaver/features/steps/protected-content.py:42
13:05:25 (None, None, None)
13:05:25       Traceback (most recent call last):
13:05:25         File "/var/lib/jenkins/.cache/pypoetry/virtualenvs/gluu-passport-testing-_NjlDPuR-py3.6/lib/python3.6/site-packages/behave/model.py", line 1329, in run
13:05:25           match.run(runner.context)
13:05:25         File "/var/lib/jenkins/.cache/pypoetry/virtualenvs/gluu-passport-testing-_NjlDPuR-py3.6/lib/python3.6/site-packages/behave/matchers.py", line 98, in run
13:05:25           self.func(context, *args, **kwargs)
13:05:25         File "tests/behaver/features/steps/protected-content.py", line 121, in user_authenticates
13:05:25           assert context.web.current_url.startswith(context.base_url)
13:05:25       AssertionError
13:05:25 
13:05:26     When user tries to access protected content page     # None
13:05:26     Then user should access protected content            # None
13:05:26 
13:05:26 
13:05:26 Failing scenarios:
13:05:26   tests/behaver/features/protected-content.feature:9  Authenticated user access protected content
13:05:26 
13:05:26 0 features passed, 1 failed, 0 skipped
13:05:26 1 scenario passed, 1 failed, 0 skipped
13:05:26 4 steps passed, 1 failed, 2 skipped, 0 undefined
13:05:26 Took 0m29.508s
13:05:26 entered teardown...

Basically it says we are not getting in the desired URL at this point.
Would you test the flow manually?

@kdhttps
Copy link
Contributor Author

kdhttps commented Jan 18, 2021

Basically it says we are not getting in the desired URL at this point.
Would you test the flow manually?

Sure, I'll test it

@kdhttps
Copy link
Contributor Author

kdhttps commented Feb 1, 2021

run tests

1 similar comment
@kdhttps
Copy link
Contributor Author

kdhttps commented Feb 1, 2021

run tests

@kdhttps
Copy link
Contributor Author

kdhttps commented Feb 1, 2021

Test are passing on my local machine. so we can merge this PR. @christian-hawk may I merge this PR??

PLEASE DO NOT INTERRUPT
============================================================================
Configuration is loading from test.conf file
============================================================================
Setting up environment....
PASSPORT_HOST=xxxx
PASSPORT_IP=xxxx
PROVIDER_HOST=xxxx
PROVIDER_IP=xxxxx
PROVIDER_SNAPSHOT_ID=xxxxx
CLIENT_HOST=dev.xxxx
API_CLIENT_ID=5aba39e9-xxxx
API_CLIENT_SECRET=xxxx
LATEST_DEV_SNAPSHOT_ID=xxx
LATEST_STABLE_SNAPSHOT_ID=xx
============================================================================
create_drplet=false
run_tests=true
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   299  100   202  100    97    108     52  0:00:01  0:00:01 --:--:--   161
{}
Creating test user on xxxxx...
Token generated:xxxx
Error creating user, please check.
response status: 500
reponse: b''
Setting up test client with pre-selected provider saml-default...
https://xxxx/configuration
{
  "provider_id": "saml-default"
}
============================================================================

STARTING: PASSPORT-SAML-DEFAULT

============================================================================

- Provider type: IDP
- Type: passport-saml
- Flow: default
- Passport Provider ID: saml-default
TODO: Creating Trust Relation on xxxx automatically

- User data:
USER_NAME: johndoe22
USER_PASSWORD: test123
USER_MAIL: [email protected]
USER_GIVEN: john
USER_SUR: doe

----------------------------------------------------------------------------
RUNNING BLACKBOX BDT TESTS:
----------------------------------------------------------------------------

Feature: accessing protected content # tests/behaver/features/protected-content.feature:1

  Scenario: Not authenticated user access protected content  # tests/behaver/features/protected-content.feature:3
    Given that user exists in provider                       # tests/behaver/features/steps/protected-content.py:37 0.000s
    When user tries to access protected content page         # tests/behaver/features/steps/protected-content.py:133 9.524s
    Then user should be redirected to login page             # tests/behaver/features/steps/protected-content.py:148
    Then user should be redirected to login page             # tests/behaver/features/steps/protected-content.py:148 8.016s

  Scenario: Authenticated user access protected content  # tests/behaver/features/protected-content.feature:9
    Given that user exists in provider                   # tests/behaver/features/steps/protected-content.py:37 0.000s
    And user is authenticated                            # tests/behaver/features/steps/protected-content.py:42
    And user is authenticated                            # tests/behaver/features/steps/protected-content.py:42 32.119s
    When user tries to access protected content page     # tests/behaver/features/steps/protected-content.py:133 5.172s
    Then user should access protected content            # tests/behaver/features/steps/protected-content.py:168 0.045s

1 feature passed, 0 failed, 0 skipped
2 scenarios passed, 0 failed, 0 skipped
7 steps passed, 0 failed, 0 skipped, 0 undefined
Took 0m54.877s

----------------------------------------------------------------------------

BLACKBOX BDT TESTS FINISHED FOR TEST CASE: PASSPORT-SAML-DEFAULT
============================================================================
Deleting test users on xxx...
User deleted from idp/oidc provider via API.
Creating test user on xxx...
Token generated: 39deca64-c349-41f3-ba42-06b644ce79f3
User created with success via API...
b'{"inum":"3e7e5a9c-c005-48e4-a232-c0c212ded3d9","surName":"doe","givenName":"john","email":"[email protected]","password":null,"userName":"josephdoe","displayName":"josephdoe","creationDate":1612169752219,"status":"ACTIVE"}'
Setting up test client with pre-selected provider saml-emailreq...
https://dev.techno24x7.com/configuration
{
  "provider_id": "saml-emailreq"
}
============================================================================

STARTING: PASSPORT-SAML-DEFAULT-EMAILREQ

============================================================================

- Provider type: IDP
- Type: passport-saml
- Flow: default emailreq
- Passport Provider ID: saml-emailreq
TODO: Creating Trust Relation on xxx automatically

- User data:
USER_NAME: josephdoe
USER_PASSWORD: test123
USER_MAIL: [email protected]
USER_GIVEN: john
USER_SUR: doe

----------------------------------------------------------------------------
RUNNING BLACKBOX BDT TESTS:
----------------------------------------------------------------------------

Feature: accessing protected content # tests/behaver/features/protected-content.feature:1

  Scenario: Not authenticated user access protected content  # tests/behaver/features/protected-content.feature:3
    Given that user exists in provider                       # tests/behaver/features/steps/protected-content.py:37 0.000s
    When user tries to access protected content page         # tests/behaver/features/steps/protected-content.py:133 6.709s
    Then user should be redirected to login page             # tests/behaver/features/steps/protected-content.py:148
    Then user should be redirected to login page             # tests/behaver/features/steps/protected-content.py:148 8.013s

  Scenario: Authenticated user access protected content  # tests/behaver/features/protected-content.feature:9
    Given that user exists in provider                   # tests/behaver/features/steps/protected-content.py:37 0.000s
    And user is authenticated                            # tests/behaver/features/steps/protected-content.py:42
    And user is authenticated                            # tests/behaver/features/steps/protected-content.py:42 39.887s
    When user tries to access protected content page     # tests/behaver/features/steps/protected-content.py:133 5.160s
    Then user should access protected content            # tests/behaver/features/steps/protected-content.py:168 0.075s

1 feature passed, 0 failed, 0 skipped
2 scenarios passed, 0 failed, 0 skipped
7 steps passed, 0 failed, 0 skipped, 0 undefined
Took 0m59.844s

----------------------------------------------------------------------------

BLACKBOX BDT TESTS FINISHED FOR TEST CASE: PASSPORT-SAML-DEFAULT-EMAILREQ
============================================================================
Deleting test users on xxx...
User deleted from idp/oidc provider via API.
Creating test user on xxx...
Token generated: 417061f8-21f1-4b21-8360-68973db7ce47
User created with success via API...
b'{"inum":"186fc964-5d4b-4cc3-be86-7458b67594e9","surName":"doe","givenName":"john","email":"[email protected]","password":null,"userName":"josephdoe2","displayName":"josephdoe2","creationDate":1612169825885,"status":"ACTIVE"}'
Setting up test client with pre-selected provider saml-emaillink...
https://dev.techno24x7.com/configuration
{
  "provider_id": "saml-emaillink"
}
============================================================================

STARTING: PASSPORT-SAML-DEFAULT-EMAILLINK

============================================================================

- Provider type: IDP
- Type: passport-saml
- Flow: default emaillink
- Passport Provider ID: saml-emaillink
TODO: Creating Trust Relation on xxx automatically

- User data:
USER_NAME: josephdoe2
USER_PASSWORD: test123
USER_MAIL: [email protected]
USER_GIVEN: john
USER_SUR: doe

----------------------------------------------------------------------------
RUNNING BLACKBOX BDT TESTS:
----------------------------------------------------------------------------

Feature: accessing protected content # tests/behaver/features/protected-content.feature:1

  Scenario: Not authenticated user access protected content  # tests/behaver/features/protected-content.feature:3
    Given that user exists in provider                       # tests/behaver/features/steps/protected-content.py:37 0.000s
    When user tries to access protected content page         # tests/behaver/features/steps/protected-content.py:133 6.785s
    Then user should be redirected to login page             # tests/behaver/features/steps/protected-content.py:148
    Then user should be redirected to login page             # tests/behaver/features/steps/protected-content.py:148 8.019s

  Scenario: Authenticated user access protected content  # tests/behaver/features/protected-content.feature:9
    Given that user exists in provider                   # tests/behaver/features/steps/protected-content.py:37 0.000s
    And user is authenticated                            # tests/behaver/features/steps/protected-content.py:42
    And user is authenticated                            # tests/behaver/features/steps/protected-content.py:42 28.696s
    When user tries to access protected content page     # tests/behaver/features/steps/protected-content.py:133 5.177s
    Then user should access protected content            # tests/behaver/features/steps/protected-content.py:168 0.060s

1 feature passed, 0 failed, 0 skipped
2 scenarios passed, 0 failed, 0 skipped
7 steps passed, 0 failed, 0 skipped, 0 undefined
Took 0m48.737s

----------------------------------------------------------------------------

BLACKBOX BDT TESTS FINISHED FOR TEST CASE: PASSPORT-SAML-DEFAULT-EMAILLINK
============================================================================
Deleting test users on xxx...
User deleted from idp/oidc provider via API.
Creating test user on xxx...
Token generated: c9b9bc67-be6b-4d32-a83e-4a5591e7d6db
User created with success via API...
b'{"inum":"7646dd27-4047-49a6-9b86-1e9b464c9af2","surName":"doe","givenName":"hans","email":"[email protected]","password":null,"userName":"hansdoe","displayName":"hansdoe","creationDate":1612169888451,"status":"ACTIVE"}'
Setting up test client with pre-selected provider saml-idpinit...
https://dev.techno24x7.com/configuration
{
  "provider_id": "saml-idpinit"
}
============================================================================

STARTING: PASSPORT-SAML-IDP-INITIATED

============================================================================

- Provider type: IDP
- Type: passport-saml
- Flow: idp-initiated
- Passport Provider ID: saml-idpinit
TODO: Creating Trust Relation on xxx automatically

- User data:
USER_NAME: hansdoe
USER_PASSWORD: test123
USER_MAIL: [email protected]
USER_GIVEN: hans
USER_SUR: doe

----------------------------------------------------------------------------
RUNNING BLACKBOX BDT TESTS:
----------------------------------------------------------------------------

Feature: accessing protected content # tests/behaver/features/protected-content.feature:1

  Scenario: Not authenticated user access protected content  # tests/behaver/features/protected-content.feature:3
    Given that user exists in provider                       # tests/behaver/features/steps/protected-content.py:37 0.000s
    When user tries to access protected content page         # tests/behaver/features/steps/protected-content.py:133 10.193s
    Then user should be redirected to login page             # tests/behaver/features/steps/protected-content.py:148
    Then user should be redirected to login page             # tests/behaver/features/steps/protected-content.py:148 8.015s

  Scenario: Authenticated user access protected content  # tests/behaver/features/protected-content.feature:9
    Given that user exists in provider                   # tests/behaver/features/steps/protected-content.py:37 0.000s
    And user is authenticated                            # tests/behaver/features/steps/protected-content.py:42
    And user is authenticated                            # tests/behaver/features/steps/protected-content.py:42 31.471s
    When user tries to access protected content page     # tests/behaver/features/steps/protected-content.py:133 6.374s
    Then user should access protected content            # tests/behaver/features/steps/protected-content.py:168 0.113s

1 feature passed, 0 failed, 0 skipped
2 scenarios passed, 0 failed, 0 skipped
7 steps passed, 0 failed, 0 skipped, 0 undefined
Took 0m56.167s

----------------------------------------------------------------------------

BLACKBOX BDT TESTS FINISHED FOR TEST CASE: PASSPORT-SAML-IDP-INITIATED
============================================================================
Deleting test users on xxx...
User deleted from idp/oidc provider via API.
ALL TEST FINISHED.
=====================================================================
Deleting droplets...
finished with success!!!
entered teardown...
EXIT detected with exit status 0

@christian-hawk
Copy link
Contributor

Just a note: as jenkins automated tests are in alpha, when we find jenkins problem we run tests locally and merge.

@kdhttps kdhttps merged commit 693da68 into master Feb 1, 2021
@christian-hawk christian-hawk deleted the 139_feat_rate_limiting branch October 6, 2021 06:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Routes has no rate-limiting
2 participants