Optimize PermissionChecker performance && adjust jsonrpc configuration #13034

skabashnyuk · 2019-04-01T12:51:44Z

What does this PR do?

It's an attempt to reduce the time needed to execute a single json-rpc call.
To test that I was using a scenario that is emulating Che6 workspace startup. It has 10 times more minor events aka logs in comparison to major aka statuses.

Figure 1:

this is how che server behaves without changes

what we can see here is

we can see the degradation of major queue. The size of this queue is very big. In this pr, I've changed an upper limit to reduce pressure on the heap.
The maximum duration of json/rpc call between 20-40 and it's started degrading after the point when heap was consumed almost completely.

Figure 2:

this is how che server behaves with queue adjustmens and query cache

what we can see here is :

performance of minor queue increased 8 times, major - ~12 times.
maximum time of message execution reduced about 10 times in avg.
a big part of the heap was consumed by http sessions (TODO create an issue here)

Figure 3

The number of rows fetched with and without query cache.

Time of execution with caching enabled from 13-53 -> 14-57
Time of execution without caching from 15-09 -> 15-13

what we can see here is :
If we are speaking about the test without caching - it's about ten's of millions of rows.
If we are speaking about the test with caching - it's about hundreds of rows.
it's important to constantly monitor DB statistic [Metrics] Database metrics #13223

What issues does this PR fix or reference?

#13221
#13131
redhat-developer/rh-che#1323

che-bot · 2019-09-06T11:58:43Z

E2E Happy path tests of Eclipse Che Single User on K8S (minikube v1.1.1) has been successful:

build details
"che-server" docker image: maxura/che-server:13034

che-bot · 2019-09-06T12:30:40Z

E2E tests of Eclipse Che Multiuser on OCP has been successful:

Build details
Test report
docker image: eclipseche/che-server:13034

Signed-off-by: Sergii Kabashniuk <[email protected]>

che-bot · 2019-09-09T06:29:12Z

E2E Happy path tests of Eclipse Che Single User on K8S (minikube v1.1.1) has been successful:

build details
"che-server" docker image: maxura/che-server:13034

che-bot · 2019-09-09T07:03:23Z

E2E tests of Eclipse Che Multiuser on OCP has failed:

Build details
Test report
docker image: eclipseche/che-server:13034
https://github.com/orgs/eclipse/teams/eclipse-che-qa please check this report.
Use command "ci-test" to rerun the test.

skabashnyuk · 2019-09-09T14:04:22Z

ci-test

...c/main/java/org/eclipse/che/multiuser/permission/workspace/server/model/impl/WorkerImpl.java

…d workspaceid Signed-off-by: Sergii Kabashniuk <[email protected]>

che-bot · 2019-09-10T13:40:37Z

E2E tests of Eclipse Che Multiuser on OCP has failed:

Build details
Test report
docker image: eclipseche/che-server:13034
https://github.com/orgs/eclipse/teams/eclipse-che-qa please check this report.
Use command "ci-test" to rerun the test.

che-bot · 2019-09-10T13:49:49Z

E2E Happy path tests of Eclipse Che Single User on K8S (minikube v1.1.1) has failed:

build details
"che-server" docker image: maxura/che-server:13034
test artifacts
https://github.com/orgs/eclipse/teams/eclipse-che-qa please check this report.
Use command "crw-ci-test" to rerun the test.

…erId and workspaceid Signed-off-by: Sergii Kabashniuk <[email protected]>

che-bot · 2019-09-10T14:10:33Z

E2E Happy path tests of Eclipse Che Single User on K8S (minikube v1.1.1) has failed:

build details
"che-server" docker image: maxura/che-server:13034
test artifacts
https://github.com/orgs/eclipse/teams/eclipse-che-qa please check this report.
Use command "crw-ci-test" to rerun the test.

che-bot · 2019-09-10T14:34:11Z

E2E tests of Eclipse Che Multiuser on OCP has been successful:

Build details
Test report
docker image: eclipseche/che-server:13034

skabashnyuk · 2019-09-10T18:09:12Z

crw-ci-test

che-bot · 2019-09-10T18:23:19Z

E2E Happy path tests of Eclipse Che Single User on K8S (minikube v1.1.1) has failed:

build details
"che-server" docker image: maxura/che-server:13034
test artifacts
https://github.com/orgs/eclipse/teams/eclipse-che-qa please check this report.
Use command "crw-ci-test" to rerun the test.

che-bot · 2019-09-11T04:44:50Z

E2E Happy path tests of Eclipse Che Single User on K8S (minikube v1.1.1) has been successful:

build details
"che-server" docker image: maxura/che-server:13034

che-bot · 2019-09-11T05:19:49Z

E2E tests of Eclipse Che Multiuser on OCP has failed:

Build details
Test report
docker image: eclipseche/che-server:13034
https://github.com/orgs/eclipse/teams/eclipse-che-qa please check this report.
Use command "ci-test" to rerun the test.

skabashnyuk · 2019-09-11T06:08:51Z

crw-ci-test

skabashnyuk · 2019-09-11T06:12:57Z

ci-build

skabashnyuk · 2019-09-11T06:13:12Z

ci-test

che-bot · 2019-09-11T06:19:52Z

E2E Happy path tests of Eclipse Che Single User on K8S (minikube v1.1.1) has been successful:

build details
"che-server" docker image: maxura/che-server:13034

che-bot · 2019-09-11T06:57:05Z

E2E tests of Eclipse Che Multiuser on OCP has been successful:

Build details
Test report
docker image: eclipseche/che-server:13034

sleshchenko

LGTM

...c/test/java/org/eclipse/che/multiuser/permission/workspace/server/spi/tck/WorkerDaoTest.java

Signed-off-by: Sergii Kabashniuk <[email protected]>

ibuziuk · 2019-09-11T08:58:36Z

@ScrewTSW do we override those values on rh-che side ?

che-bot · 2019-09-11T09:05:19Z

E2E Happy path tests of Eclipse Che Single User on K8S (minikube v1.1.1) has been successful:

build details
"che-server" docker image: maxura/che-server:13034

skabashnyuk · 2019-09-11T11:22:26Z

ci-test

che-bot · 2019-09-11T11:48:00Z

E2E tests of Eclipse Che Multiuser on OCP has failed:

Build details
Test report
docker image: eclipseche/che-server:13034
https://github.com/orgs/eclipse/teams/eclipse-che-qa please check this report.
Use command "ci-test" to rerun the test.

skabashnyuk force-pushed the tunejsonrpc branch from 1781cf6 to cde6568 Compare April 2, 2019 12:34

skabashnyuk mentioned this pull request Apr 2, 2019

Need to verify if rh-che is stable with the upstream CHE_CORE_JSONRPC setting and 1 GB of RAM redhat-developer/rh-che#1337

Closed

skabashnyuk force-pushed the tunejsonrpc branch 2 times, most recently from b649430 to c631924 Compare April 6, 2019 20:21

skabashnyuk force-pushed the tunejsonrpc branch 2 times, most recently from 960f8cf to 3b9e278 Compare April 19, 2019 08:17

skabashnyuk force-pushed the tunejsonrpc branch 4 times, most recently from 4c0fefd to ba3b9b3 Compare April 25, 2019 05:58

che-bot added status/code-review This issue has a pull request posted for it and is awaiting code review completion by the community. kind/task Internal things, technical debt, and to-do tasks to be performed. labels Aug 6, 2019

eclipse-che deleted a comment from che-bot Aug 6, 2019

skabashnyuk force-pushed the tunejsonrpc branch from ba3b9b3 to 314eb9f Compare September 6, 2019 11:47

Adjust jsonrpc configuration.

a0a32c7

Signed-off-by: Sergii Kabashniuk <[email protected]>

skabashnyuk force-pushed the tunejsonrpc branch from 314eb9f to a0a32c7 Compare September 9, 2019 06:18

skabashnyuk marked this pull request as ready for review September 9, 2019 14:01

skabashnyuk requested review from ibuziuk, l0rd, nickboldt, rhopp and sleshchenko as code owners September 9, 2019 14:01

skabashnyuk requested a review from metlos September 9, 2019 14:02

skabashnyuk changed the title ~~Adjust jsonrpc configuration.~~ Optimize PermissionChecker performance && adjust jsonrpc configuration Sep 9, 2019

sleshchenko reviewed Sep 9, 2019

View reviewed changes

...c/main/java/org/eclipse/che/multiuser/permission/workspace/server/model/impl/WorkerImpl.java Show resolved Hide resolved

fixup! Make sure that delete also invalidate query cache by userId an…

860b483

…d workspaceid Signed-off-by: Sergii Kabashniuk <[email protected]>

fixup! fixup! Make sure that delete also invalidate query cache by us…

7269299

…erId and workspaceid Signed-off-by: Sergii Kabashniuk <[email protected]>

Merge remote-tracking branch 'origin/master' into tunejsonrpc

3be9e2b

metlos approved these changes Sep 11, 2019

View reviewed changes

sleshchenko approved these changes Sep 11, 2019

View reviewed changes

...c/test/java/org/eclipse/che/multiuser/permission/workspace/server/spi/tck/WorkerDaoTest.java Outdated Show resolved Hide resolved

fixup! Merge remote-tracking branch 'origin/master' into tunejsonrpc

3bad232

Signed-off-by: Sergii Kabashniuk <[email protected]>

eclipse-che deleted a comment from che-bot Sep 11, 2019

skabashnyuk merged commit 00eaa1a into master Sep 11, 2019

skabashnyuk deleted the tunejsonrpc branch September 11, 2019 12:18

che-bot removed the status/code-review This issue has a pull request posted for it and is awaiting code review completion by the community. label Sep 11, 2019

che-bot added this to the 7.2.0 milestone Sep 11, 2019

skabashnyuk mentioned this pull request Sep 19, 2019

Che Server http sessions leak #14599

Closed

23 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optimize PermissionChecker performance && adjust jsonrpc configuration #13034

Optimize PermissionChecker performance && adjust jsonrpc configuration #13034

skabashnyuk commented Apr 1, 2019 •

edited

Loading

che-bot commented Sep 6, 2019

che-bot commented Sep 6, 2019

che-bot commented Sep 9, 2019

che-bot commented Sep 9, 2019

skabashnyuk commented Sep 9, 2019

che-bot commented Sep 10, 2019

che-bot commented Sep 10, 2019

che-bot commented Sep 10, 2019

che-bot commented Sep 10, 2019

skabashnyuk commented Sep 10, 2019

che-bot commented Sep 10, 2019

che-bot commented Sep 11, 2019

che-bot commented Sep 11, 2019

skabashnyuk commented Sep 11, 2019

skabashnyuk commented Sep 11, 2019

skabashnyuk commented Sep 11, 2019

che-bot commented Sep 11, 2019

che-bot commented Sep 11, 2019

sleshchenko left a comment

ibuziuk commented Sep 11, 2019

che-bot commented Sep 11, 2019

skabashnyuk commented Sep 11, 2019

che-bot commented Sep 11, 2019

Optimize PermissionChecker performance && adjust jsonrpc configuration #13034

Optimize PermissionChecker performance && adjust jsonrpc configuration #13034

Conversation

skabashnyuk commented Apr 1, 2019 • edited Loading

What does this PR do?

Figure 1:

Figure 2:

Figure 3

What issues does this PR fix or reference?

che-bot commented Sep 6, 2019

che-bot commented Sep 6, 2019

che-bot commented Sep 9, 2019

che-bot commented Sep 9, 2019

skabashnyuk commented Sep 9, 2019

che-bot commented Sep 10, 2019

che-bot commented Sep 10, 2019

che-bot commented Sep 10, 2019

che-bot commented Sep 10, 2019

skabashnyuk commented Sep 10, 2019

che-bot commented Sep 10, 2019

che-bot commented Sep 11, 2019

che-bot commented Sep 11, 2019

skabashnyuk commented Sep 11, 2019

skabashnyuk commented Sep 11, 2019

skabashnyuk commented Sep 11, 2019

che-bot commented Sep 11, 2019

che-bot commented Sep 11, 2019

sleshchenko left a comment

Choose a reason for hiding this comment

ibuziuk commented Sep 11, 2019

che-bot commented Sep 11, 2019

skabashnyuk commented Sep 11, 2019

che-bot commented Sep 11, 2019

skabashnyuk commented Apr 1, 2019 •

edited

Loading