[wip] Re-enable test:browser #1251

giorgiosironi · 2019-01-07T16:03:39Z

No description provided.

giorgiosironi · 2019-01-07T16:44:21Z

https://alfred.elifesciences.org/blue/organizations/jenkins/pull-requests-projects%2Felife-xpub/detail/PR-1251/4/pipeline is the first to reproduce the hanging problem.

giorgiosironi · 2019-01-07T16:52:27Z

https://alfred.elifesciences.org/blue/organizations/jenkins/pull-requests-projects%2Felife-xpub/detail/PR-1251/5/pipeline has a similar output:

test/helpers/setup.js::startServer
test/helpers/setup.js::startServer await start
test/helpers/setup.js::startServer
test/helpers/setup.js::startServer
test/helpers/setup.js::startServer
test/helpers/setup.js::startServer started

Multiple startServer shouldn't really happen.

giorgiosironi · 2019-01-07T16:53:44Z

https://alfred.elifesciences.org/blue/organizations/jenkins/pull-requests-projects%2Felife-xpub/detail/PR-1251/7/pipeline/ and https://alfred.elifesciences.org/blue/organizations/jenkins/pull-requests-projects%2Felife-xpub/detail/PR-1251/8/pipeline/ are identical too.

giorgiosironi · 2019-01-07T16:59:15Z

The actual hanging is happening after

test/helpers/setup.js::setup await startS3Server

so logs are needed after that line.

giorgiosironi · 2019-01-08T09:11:21Z

More recent runs:

 Submission
test/helpers/setup.js::startServer start
test/helpers/setup.js::startServer !server
test/helpers/setup.js::startServer await start
test/helpers/setup.js::startServer start
test/helpers/setup.js::startServer !server
test/helpers/setup.js::startServer start
test/helpers/setup.js::startServer !server
test/helpers/setup.js::startServer start
test/helpers/setup.js::startServer !server
test/helpers/setup.js::startServer started
test/helpers/setup.js::startServer end
test/helpers/setup.js::setup start
test/helpers/setup.js::setup createTables
test/helpers/setup.js::setup replaySetup
test/helpers/setup.js::setup await startS3Server
test/helpers/setup.js::setup end

Yes, testcafe is calling startServer() multiple times after the first attempt is still going on. That attempt eventually hangs after setup() so it may not be related, but it shouldn't happen.

giorgiosironi · 2019-01-08T12:37:52Z

https://alfred.elifesciences.org/blue/organizations/jenkins/pull-requests-projects%2Felife-xpub/detail/PR-1251/41/pipeline got stuck like this

[Tue Jan 08 2019 10:12:50 GMT+0000 (UTC)][4710935] test/helpers/setup.js::startServer start
[Tue Jan 08 2019 10:12:50 GMT+0000 (UTC)][4710935] test/helpers/setup.js::startServer !server
[Tue Jan 08 2019 10:12:50 GMT+0000 (UTC)][4710935] test/helpers/setup.js::startServer await start
[Tue Jan 08 2019 10:14:51 GMT+0000 (UTC)][4710935] test/helpers/setup.js::startServer started
[Tue Jan 08 2019 10:14:51 GMT+0000 (UTC)][4710935] test/helpers/setup.js::startServer end
[Tue Jan 08 2019 10:14:51 GMT+0000 (UTC)][9830361] test/helpers/setup.js::setup start
[Tue Jan 08 2019 10:14:51 GMT+0000 (UTC)][9830361] test/helpers/setup.js::setup createTables
[Tue Jan 08 2019 10:14:51 GMT+0000 (UTC)][9830361] test/helpers/setup.js::setup replaySetup
[Tue Jan 08 2019 10:14:51 GMT+0000 (UTC)][9830361] test/helpers/setup.js::setup await startS3Server
[Tue Jan 08 2019 10:14:51 GMT+0000 (UTC)][9830361] test/helpers/setup.js::setup end

which should be enough to think that even if there are other calls to startServer, they are not necessary for the problem to happen.

giorgiosironi · 2019-01-08T13:19:32Z

https://alfred.elifesciences.org/blue/organizations/jenkins/pull-requests-projects%2Felife-xpub/detail/PR-1251/48/pipeline/39/ seems to get to the start of the test, will need to collect more data in case different runs differ on where they get stuck.

giorgiosironi · 2019-01-08T13:24:05Z

https://alfred.elifesciences.org/blue/organizations/jenkins/pull-requests-projects%2Felife-xpub/detail/PR-1251/50/pipeline seems to agree, adding logs through the test.

giorgiosironi · 2019-01-08T16:44:20Z

Note: at least until 4587185 we are able to reproduce the hanging behavior.

This reverts commit cbfdf23.

This reverts commit d3025a3.

giorgiosironi · 2019-01-08T17:15:08Z

Problem reproduced up to 7a88a36 (https://alfred.elifesciences.org/blue/organizations/jenkins/pull-requests-projects%2Felife-xpub/detail/PR-1251/149/pipeline/39)

giorgiosironi · 2019-01-08T17:24:36Z

Seems hanging is a common problem (issues not necessarily related):

TestCafe hangs if the specified screenshots directory contains forbidden symbols DevExpress/testcafe#681
Test does not resume after pausing/waiting for more than 13 seconds DevExpress/testcafe#3070
Initial t.navigateTo() hangs when it's called after ~15s DevExpress/testcafe#3118
Testcafe freezes when clicking a custom element on Chrome 68 DevExpress/testcafe#2861

giorgiosironi · 2019-01-08T17:33:09Z

Still hanging in
https://alfred.elifesciences.org/blue/organizations/jenkins/pull-requests-projects%2Felife-xpub/detail/PR-1251/159/pipeline
https://alfred.elifesciences.org/blue/organizations/jenkins/pull-requests-projects%2Felife-xpub/detail/PR-1251/160/pipeline
so trying to add debugging options.

giorgiosironi · 2019-01-08T18:41:01Z

Debug options do not tell us much more. Afraid we need to patch testcafe to understand what's going on in there as the control is not in our code.
Any Node.js debugger we could use to understand the flow without having to instrument code with patches?

giorgiosironi · 2019-01-09T10:59:14Z

Theory: too long fixtures can cause the test suite to hang (>2 minutes or similar).
https://alfred.elifesciences.org/blue/organizations/jenkins/pull-requests-projects%2Felife-xpub/detail/PR-1251/130/pipeline/39/ seems to contradict this theory as the fixtures took 16:47:46 - 16:45:35 = 131 and the test completed correctly. However, this was with only 1 test remaining in the test suite so we may need 2 tests to reproduce the bug.

giorgiosironi · 2019-01-09T11:21:59Z

After 30 second sleep addition:

https://alfred.elifesciences.org/blue/organizations/jenkins/pull-requests-projects%2Felife-xpub/detail/PR-1251/169/pipeline/39/ hangs
https://alfred.elifesciences.org/blue/organizations/jenkins/pull-requests-projects%2Felife-xpub/detail/PR-1251/170/pipeline/39/ also hangs (different sleep implementation)

giorgiosironi · 2019-01-09T11:47:34Z

Last 10 builds all hanging, this is now reproducible, I think. I also reproduced it locally by changing the sleep to 121000 (since starting the server takes less time locally), followed by:

ln -sf .env.ci .env # same env as Jenkins
docker-compose -f docker-compose.yml -f docker-compose.ci.yml build
docker-compose -f docker-compose.yml -f docker-compose.ci.yml up -d postgres
NODE_ENV=production NODE_CONFIG_ENV=test PGDATABASE=test_browser docker-compose -f docker-compose.yml -f docker-compose.ci.yml run -p 10081:10081 --rm --name elife-xpub_app_test_browser app bash -c 'socat -d tcp-listen:10081,reuseaddr,fork tcp:localhost:10080 & npm run test:browser -- --screenshots /tmp/screenshots --screenshots-on-fails'

giorgiosironi · 2019-01-09T15:35:10Z

Jenkinsfile

@@ -79,6 +70,14 @@ elifePipeline {
            }
        }

+        stage 'Browser Tests', {
+            actions['test:browser'] = {


actions is a dictionary of functions to be executed by parallel, so if you want to execute the browser tests here you have to remove this assignment and just use the contents of the { withCommitStatus(...) } 0-argument function

giorgiosironi · 2019-01-09T15:36:48Z

Jenkinsfile

+            actions['test:browser'] = {
+                withCommitStatus({
+                    sh "IMAGE_TAG=${commit} NODE_ENV=production NODE_CONFIG_ENV=test PGDATABASE=test_browser docker-compose -f docker-compose.yml -f docker-compose.ci.yml run -p 10081:10081 --rm --name elife-xpub_app_test_browser app bash -c 'socat -d tcp-listen:10081,reuseaddr,fork tcp:localhost:10080 & npm run test:browser -- --screenshots /tmp/screenshots --screenshots-on-fails'"
+                }, 'test:browser', commit)


you also need the setup around it, since this command assumes a database started and ready:

try { sh "IMAGE_TAG=${commit} docker-compose -f docker-compose.yml -f docker-compose.ci.yml up -d postgres" sh "IMAGE_TAG=${commit} docker-compose -f docker-compose.yml -f docker-compose.ci.yml run --rm --name elife-xpub_wait_postgres app bash -c './scripts/wait-for-it.sh postgres:5432'" // do things } finally { sh "IMAGE_TAG=${commit} docker-compose -f docker-compose.yml -f docker-compose.ci.yml down -v" }

also, the archiveArtifacts in finally is missing, which publishes the screenshots on the Jenkins UI

Ah we removed that ... @damnhipster says they don't work anyway :)

hem-brahmbhatt · 2019-01-10T10:08:48Z

Closing as fix has been implemented in #1273

Re-enable test:browser

92fe2ef

giorgiosironi mentioned this pull request Jan 7, 2019

Fix Submission and Supporting File deletion #1252

Merged

5 tasks

giorgiosironi mentioned this pull request Jan 7, 2019

test:browser hanging #1228

Closed

diversemix and others added 3 commits January 7, 2019 17:26

Merge branch 'develop' into reenable_browser_tests

bda8194

Extend logs of setup.js

cb65d7d

Log exception because you cannot trust Testcafe to respect it

696d28e

giorgiosironi added 10 commits January 8, 2019 09:19

Add correlationId to test fixtures

495393a

Round down correlationId

187d63d

Conform to eslint

285a819

Convert other logs of setup.js

3736661

Log a timestamp

ab88456

Differentiate name of fixture as for other tests

53ee7ef

Skip all but one fixture

ec72805

Choose the first test s the one to run

7171e5f

Log stack trace in case of repeated server starts

9171c14

Conform to linter for stack trace dump

9d4be04

giorgiosironi added 2 commits January 8, 2019 13:01

Reformat via commit hook

a6c247a

Log start of only test

05a549d

Log throughout the only test

26bea13

giorgiosironi added 3 commits January 8, 2019 14:10

Reinstate socat for debugging test:browser

ef8a4a7

Remove all uninteresting tests

a403d93

Reinstate submission.browser.js

7daf8b8

hem-brahmbhatt added 2 commits January 8, 2019 16:21

using empty fixture hooks

4587185

deleted submission.browser test

cbfdf23

hem-brahmbhatt added 4 commits January 8, 2019 16:53

Revert "deleted submission.browser test"

be02b8d

This reverts commit cbfdf23.

two empty tests

d3025a3

Revert "two empty tests"

7a88a36

This reverts commit d3025a3.

upgraded testcafe

9dd27f7

Call testcafe with --dev

7b8e994

diversemix added 2 commits January 9, 2019 10:00

Add some timings and memory usage

3047bab

JSON stringify mem usage

c33d80f

giorgiosironi added 2 commits January 9, 2019 11:13

Sleep 30 seconds to try to make tests fail

798480c

Substitute exec with sleep in process

9eb7ab6

diversemix added 3 commits January 9, 2019 15:14

Another test to run browser tests on their own

3a89538

Rejig Jenkinsfile

831a919

remove lint errors

a898fc1

giorgiosironi commented Jan 9, 2019

View reviewed changes

diversemix added 2 commits January 9, 2019 15:39

mods after GS suggestion

a2a553e

Another jenkinsfile

b559b9f

hem-brahmbhatt mentioned this pull request Jan 9, 2019

Enable browser tests #1273

Merged

5 tasks

hem-brahmbhatt closed this Jan 10, 2019

hem-brahmbhatt deleted the reenable_browser_tests branch January 10, 2019 10:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[wip] Re-enable test:browser #1251

[wip] Re-enable test:browser #1251

giorgiosironi commented Jan 7, 2019

giorgiosironi commented Jan 7, 2019

giorgiosironi commented Jan 7, 2019

giorgiosironi commented Jan 7, 2019

giorgiosironi commented Jan 7, 2019

giorgiosironi commented Jan 8, 2019

giorgiosironi commented Jan 8, 2019

giorgiosironi commented Jan 8, 2019

giorgiosironi commented Jan 8, 2019

giorgiosironi commented Jan 8, 2019

giorgiosironi commented Jan 8, 2019

giorgiosironi commented Jan 8, 2019

giorgiosironi commented Jan 8, 2019

giorgiosironi commented Jan 8, 2019

giorgiosironi commented Jan 9, 2019 •

edited

Loading

giorgiosironi commented Jan 9, 2019

giorgiosironi commented Jan 9, 2019 •

edited

Loading

giorgiosironi Jan 9, 2019

giorgiosironi Jan 9, 2019

giorgiosironi Jan 9, 2019

diversemix Jan 9, 2019

hem-brahmbhatt commented Jan 10, 2019

[wip] Re-enable test:browser #1251

[wip] Re-enable test:browser #1251

Conversation

giorgiosironi commented Jan 7, 2019

giorgiosironi commented Jan 7, 2019

giorgiosironi commented Jan 7, 2019

giorgiosironi commented Jan 7, 2019

giorgiosironi commented Jan 7, 2019

giorgiosironi commented Jan 8, 2019

giorgiosironi commented Jan 8, 2019

giorgiosironi commented Jan 8, 2019

giorgiosironi commented Jan 8, 2019

giorgiosironi commented Jan 8, 2019

giorgiosironi commented Jan 8, 2019

giorgiosironi commented Jan 8, 2019

giorgiosironi commented Jan 8, 2019

giorgiosironi commented Jan 8, 2019

giorgiosironi commented Jan 9, 2019 • edited Loading

giorgiosironi commented Jan 9, 2019

giorgiosironi commented Jan 9, 2019 • edited Loading

giorgiosironi Jan 9, 2019

Choose a reason for hiding this comment

giorgiosironi Jan 9, 2019

Choose a reason for hiding this comment

giorgiosironi Jan 9, 2019

Choose a reason for hiding this comment

diversemix Jan 9, 2019

Choose a reason for hiding this comment

hem-brahmbhatt commented Jan 10, 2019

giorgiosironi commented Jan 9, 2019 •

edited

Loading

giorgiosironi commented Jan 9, 2019 •

edited

Loading