Improve TestCafe stability and performance in concurrency mode under heavy load

[dbschwartz]

What is your Test Scenario?

Running tests in concurrency in Windows 10 and Node 6.10.3 with TestCafe 0.23.1

What is the Current behavior?

The tests start to run normally, but the status for the first test is the only one that shows in the command prompt. Eventually the browsers either show the TestCafe start page or are on the start pageURL and are just awaiting further commands from hammerhead. They just stay this way until I abort by pressing CTRL + C.

What is the Expected behavior?

Concurrent tests should just as they did in TestCafe 0.22.0

What is your web application and your TestCafe test code?

I can't put the actual tests on here but I could probably make a sample eventually. I'm afraid though that the problem is possibly Node v6.10.3 or maybe Windows 10
Your website URL (or attach your complete example):

Your complete test code (or attach your test files):

Your complete test report:

Screenshots:

Steps to Reproduce:

1. Go to my website ...
2. Execute this command...
3. See the error...

Your Environment details:

• testcafe version: 0.23.1
• node.js version: 6.10.3
• command-line arguments:
• browser name and version: Chrome 70
• platform and version: Windows 10
• other:

[AndreyBelym]

Unfortunately, I can't reproduce this behavior. Could you please provide a HAR report or record a screencast of your test session?

Judging by the description, the problem can be caused by Chrome's background tabs throttling feature. When Chrome suspends a background tab, it can't respond to the TestCafe server and thus can't execute any test command. Try to specify the --disable-background-timer-throttling flag in the TestCafe CLI parameters:

testcafe "chrome --disable-background-timer-throttling" tests.jd

and manually focus each Chrome window when it is opened.

[dbschwartz]

Hi @AndreyBelym ,

Thank you so much for your response. I don't think the Chrome background tabs throttling feature is the issue, because I tried the tests in concurrency with Firefox and was getting the same problems.

What's strange is that concurrency works if testing is simple. I tried to run concurrency with just one test file and it worked fine. This made me think that maybe the problem was happening when I ran concurrency on multiple tests that were multiple folders

This inspired me to reproduce the problem in a demoable repo (there is a link below), which I made by copying and pasting multiple instances of the TestCafe example tests in reptitive files and folders. Everything was still working fine at that point.

At this point I wanted to make these fake tests more like my real ones by adding Roles and testcafe-vue-selectors because I thought the problem might be coming from there.

I added two roles which almost caused the problem, because the test statuses only updated after everything was finished, but the tests still finished and closed at the end.

However once I added a test-cafe-vue-selector it produced the same problem that I've been having. Essentially all the tests run, but the status of the first and second tests are the only ones that appear in the console. After all the tests run, the concurrent instances become infinitely idle and never close. I am running 4 concurrent instances by-the-way

So I think the problem is with the latest version of TestCafe (0.23.1) and testcafe-vue-selectors.

I pushed up the code to my GitHub, https://github.com/dbschwartz/test-cafe-error-demo.

Please clone the repo and run npm test to reproduce.

Let me know if there is anything else I can do to clarify. I love TestCafe and all that you guys do. Thanks for making such an awesome product.

-Dave

[AndreyBelym]

Thank you for your feedback!

I've found that your example works fine with the -c 2 argument on my machine. The relation between resources of your machine and the maximum number of browsers that you can use in a testing session is not linear. We will try to keep TestCafe responsive even when the performance capabilities are not enough to run all requested browsers and use browser resources more efficiently in concurrency mode.

[dbschwartz]

My pleasure!

What's interesting here though is if you run these tests with Test Cafe v 0.22.0, everything finishes. This repo illustrates problems with my current tests because they run concurrently in v 0.22.0 with -c 4 without a hitch, but they become problematic when run concurrently with TestCafe 0.32.1 with -c -4.

I made another repo with everything being same except using Test Cafe v 0.22.0 that runs in a stable state with 4 concurrent instances if you'd like to check it out: https://github.com/dbschwartz/test-cafe-error-demo-stable

Note the test statuses take about 45 seconds before they start updating, but they eventually do and everything finishes and closes as normal.

My guess is that maybe there was some new server code introduced that might have lent itself to this. Perhaps it's related to handling the unhandled promise rejection errors that were happening in concurrency before. Again just a guess.

When I have use concurrency in other applications, its usually recommended to somewhat match instances with the number of cores on the machine, maybe using one less, i.e. If there are 7 cores, use 6 instances, etc.. I have been testing on machines with more than 4 processors. Should Test Cafe be run fairly conservatively with concurrency?

Thank you so much for all your help!

[dbschwartz]

Apologies if you recently cloned the repo, my changes weren't pushed till now. The other abnormal thing I realized is that with v 0.23.1 it seems only 2 of the concurrent instances are actually being used during the test execution. The other 2 just seem to be idle on the Test Cafe example page for most of the test run without any actions, commands, assertions being run. Granted this isn't the ideal scenario, but still strange none-the-less. This does not happen in v 0.22.0

Also correction above, the test statuses when run in TestCafe v0.22.0 take about 45 seconds to a minute to update

[dbschwartz]

Also, after I run concurrency with -c -4 in v 0.22.0 I also get the following:
(node:15240) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 error listeners added. Use emitter.setMaxListeners() to increase limit

This is just a warning, but might concern some users. Maybe those who use concurrency should increase their max listeners in Node.

[dbschwartz]

@AndreyBelym @miherlosev
I believe this is a bug not a feature request. If you run the tests of my demo repo with testcafe v0.22.0 with 4 concurrent instances it works fine as noted above. This is what is in my test-cafe-error-demo-stable repo, a la: https://github.com/dbschwartz/test-cafe-error-demo-stable. Just run npm test, it will run with 4 instances. It takes about a minute, but should run normally, exit and have statuses, etc. Concurrency was a feature of test cafe that I really enjoyed, I hope that it continues to be maintained.

[AndreyBelym]

I tried to reproduce the problem with [email protected] on my machine but it seems to be fixed. Could you please try to update the TestCafe version in your package.json and run tests in your environment?

[dbschwartz]

Yes!! I tried it! Looks great!! Only thing I might want to add is these two tests seem to be failing randomly in some runs and passing in others. They fail on testcafe 0.22.0 too now and weren't failing before so I don't think it's due to a bug in the package. I think perhaps maybe the Test Cafe Example site was updated (https://devexpress.github.io/testcafe/example/). Might just want to update your tests to reflect the change. Minor, but just wanted to let you know. Thanks again for all your help! Love this software.

[AndreyBelym]

@dbschwartz, it seems that the bug is caused by using a role in the test: https://github.com/dbschwartz/test-cafe-error-demo-stable/blob/master/tests/testsCafeExamples1/test3.js#L7. I've created separate ticket #3232 about this issue.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity for a long period. It will be closed and archived if no further activity occurs. However, we may return to this issue in the future. If it still affects you or you have any additional information regarding it, please leave a comment and we will keep it open.

[github-actions]

We're closing this issue after a prolonged period of inactivity. If it still affects you, please create a new issue with up-to-date information. Thank you.