Recurring functional test failures

[ErisDS]

We're seeing a couple of recurring test failures, which means there's likely a timing error with these tests.

The main one is:

PASS Users screen is correct (9 tests)
#    type: assertEval
#    file: /home/travis/build/TryGhost/Ghost/core/test/functional/client/team_test.js:68
#    code: }, 'The "Author" role is selected by default when adding a new user');
#    subject: false
#    fn: undefined
#    params: undefined

Here's an example broken build, plus the log link.

[ErisDS]

A second error that we see repeatedly is:

PASS Publish menu - new post status is correct after failed save (2 tests)
#    type: assert
#    file: /home/travis/build/TryGhost/Ghost/core/test/functional/client/editor_test.js:574
#    code: test.assert(false, 'Are you sure you want to leave modal did not appear.');
#    subject: false

With the exception of the classname, the test code hasn't changed here & doesn't look like it has an obvious potential timing issues. Both of these errors are to do with modals. This leads me to believe it may be the result of an issue with the upgrade to ember 1.13? cc @jaswilli

[acburdine]

@ErisDS After spending several hours trying to figure this out, I haven't fully figured out a solution; however, I did figure out where exactly the problems are occurring.

As far as I can tell, neither error actually has to do with the modals themselves.

The 'author role is selected by default' error has to do with and incorrect selection occurring on the gh-select component somewhere around here. I tried a couple of things with this. First I tried changing the $().prop('selected',true) to $().attr('selected',true), which seemed to fix it at first, but in the end didn't. I tried changing a couple of things in the team_test file to get that to fix it, but those didn't work either.

The 'modal not appearing' error is caused by something more interesting. The actual cause is because the editor controller is not returning 'isDirty' when it has actually been modified. (see here). I've tried several things, all of which don't consistently work.

If I have some other time later, I'll keep going at this, but for the moment, this is what I've found out. If anyone else wants to take a look at this, feel free.

[ErisDS]

These errors are new and seemingly related to dependency upgrades rather than code changes, as far as I can tell?

It sounds like the isDirty issue might come from an ember-data upgrade? There have been more releases of ember-data so perhaps we need to upgrade further? It seems isDirty has now been deprecated entirely https://github.com/emberjs/data/blob/master/CHANGELOG.md.

The gh-select component hasn't been updated recently, perhaps there is an underlying change in ember we aren't accounting for?

cc @jaswilli & @rwjblue any ideas on what might have changed to cause these issues ?

[ErisDS]

I'm tempted to comment out those two tests while we work on a fix - they're failing so often - I'm restarting builds 3/4 times before they pass.

[acburdine]

I would agree with temporarily commenting them out – the fact that they pass eventually means that the functionality works at least.

Also, from a functionality standpoint, IMO the Author one doesn't break the entire application if it doesn't work outside of testing. The Editor leave modal I've never seen break outside of the tests either, and it's on master too, so for the moment if it does it might not be a huge issue.

[jxhn]

I would second commenting them out. Perhaps put the issue in the current backlog to ensure its temporary-ness. As it stands it's really taking the continuous out of CI.

[jaswilli]

The editor test that is failing all the time was almost certainly broken by #5399. The test for the role select dropdown is susceptible to a timing failure based on when it runs and when the option is actually set to selected.

The failing editor test can probably be fixed-up pretty easily. The role dropdowns can be moved to something that in theory should be a little more stable and might take care of that failure.

[acburdine]

I'm pretty sure the editor failure was occurring for me before inline-errors was merged. Inline-errors only got merged on Monday, and that failure has been occurring, on multiple branches, for the past couple of weeks. It's blocked me from being able to do grunt validate for a while.

[acburdine]

Nevermind though, you seem to have fixed it. ;)

[ErisDS]

Think there might be one more in the editor tests?

PASS Publish menu - new page (10 tests)
#    type: fail
#    file: /home/travis/build/TryGhost/Ghost/core/test/functional/client/editor_test.js
#    subject: false

[acburdine]

@ErisDS I haven't seen that one in a while, so I think it might be just a slightly more common random Casper error. At any rate, removing the blocker label might be a good idea for those looking for github issues to work on.

[ErisDS]

👍 done.

[ErisDS]

Going to close this in favour of just removing the casper.js tests in their entirety. @kevinansfield is working on a plan for how to do this and what we should replace it with.