Extending check step for auto test assertions (CI/CD)

[kensoh]

In #61, @jtanningbed has a use case where PhantomJS engine does not work with his local web service being tested while Chrome mode works. However, when CasperJS test assertions are used by running TagUI with test option, there is no access to Chrome, due to CasperJS's inherent lack of support for Chrome. (by the way, this is for test automation folks, don't worry if you don't understand)

In between A - trying to code and maintain 30+ CasperJS test assertions to work with Chrome, or B - not supporting test mode at all for use with chrome option, I might be able to extend the existing check step, to become an assertion automatically whenever test option is used. Experimental.

In this way, probably a usable range of assertions can still be made to work with Chrome. As long as it can be put into a condition. For eg using fetch step to get the value of an element into a variable , then use the check step to compare that variable. The end goal is to be able to run assertions tests in Chrome mode, to output the XML file for CI purpose. Checklist -

• use tagui_test_mode variable to dynamically output different code for check step depending whether test option is used to run flow

• in tagui_parse.php, probably map test mode check step to test.assert(condition, message) or test.assert(true, 'text_if_true') / test.assert(false,'text_if_false')

• no change for tagui_header.js since check step not supported in live mode, multi-line condition + statement parsing not applicable

• test and iterate different examples with test option on and off, validate positive_test.signature

• add url(), title(), text(), timer(), count('element'), visible('element') functions in addition to existing present('element') so that web page properties can be easily used in conditions, for printing to output, in check step for simple condition check (or for test assertions when run with test option)

• pending feedback on common test assertions used and see if other supporting changes can be made to increase range of assertions for CI / CD using Chrome (besides checking here, raise issues at CasperJS, Chromy, NightmareJS, Chai, Mocha, Should.js, expect.js to seek inputs)

[kensoh]

So far so good.

sample check_flow and run results

http://tebel.org/index_mobile.php

// element value assertion
fetch footer to footer_text
check footer_text contains 'ABOUT' | 'footer contains ABOUT' | 'footer does not contain ABOUT'
check footer_text not contains 'DUMMY' | 'footer not contains DUMMY' | 'footer contains DUMMY'

// making numerical assertion
random_number = 123
check random_number greater than 456 | 'random_number greater than 456' | 'random number lesser than 456'

default without test option

TA:Desktop kensoh$ tagui check_flow

START - automation started - Sat Oct 21 2017 03:59:24 GMT+0800 (+08)
http://tebel.org/index_mobile.php - Tebel.Automation

fetch footer to footer_text
footer contains ABOUT
footer not contains DUMMY
random number lesser than 456

http://tebel.org/index_mobile.php - Tebel.Automation
FINISH - automation finished - 0.8s

chrome without test option

TA:Desktop kensoh$ tagui check_flow chrome

START - automation started - Sat Oct 21 2017 03:59:45 GMT+0800 (+08)
http://tebel.org/index_mobile.php - Tebel.Automation

fetch footer to footer_text
footer contains ABOUT
footer not contains DUMMY
random number lesser than 456

http://tebel.org/index_mobile.php - Tebel.Automation
FINISH - automation finished - 2.4s

default with test option

TA:Desktop kensoh$ tagui check_flow test
Test file: /Users/kensoh/Desktop/check_flow.js                                  
Sat Oct 21 2017 04:00:04 GMT+0800 (+08)
# /Users/kensoh/Desktop/check_flow
# http://tebel.org/index_mobile.php - Tebel.Automation
# fetch footer to footer_text
PASS footer contains ABOUT
PASS footer not contains DUMMY
FAIL random number lesser than 456
#    type: assert
#    file: /Users/kensoh/Desktop/check_flow.js
#    subject: false
# http://tebel.org/index_mobile.php - Tebel.Automation
PASS /Users/kensoh/Desktop/check_flow (NaN test)
FAIL 3 tests executed in 0.77s, 2 passed, 1 failed, 0 dubious, 0 skipped.       

Details for the 1 failed test:

In /Users/kensoh/Desktop/check_flow.js
  /Users/kensoh/Desktop/check_flow
    assert: random number lesser than 456
Result log stored in /Users/kensoh/Desktop/check_flow.xml                       

chrome with test option

TA:Desktop kensoh$ tagui check_flow test chrome
Test file: /Users/kensoh/Desktop/check_flow.js                                  
Sat Oct 21 2017 04:00:19 GMT+0800 (+08)
# /Users/kensoh/Desktop/check_flow
# http://tebel.org/index_mobile.php - Tebel.Automation
# fetch footer to footer_text
PASS footer contains ABOUT
PASS footer not contains DUMMY
FAIL random number lesser than 456
#    type: assert
#    file: /Users/kensoh/Desktop/check_flow.js
#    subject: false
# http://tebel.org/index_mobile.php - Tebel.Automation
PASS /Users/kensoh/Desktop/check_flow (NaN test)
FAIL 3 tests executed in 2.197s, 2 passed, 1 failed, 0 dubious, 0 skipped.      

Details for the 1 failed test:

In /Users/kensoh/Desktop/check_flow.js
  /Users/kensoh/Desktop/check_flow
    assert: random number lesser than 456
Result log stored in /Users/kensoh/Desktop/check_flow.xml

[kensoh]

@jtanningbed and test automation folks, I'm experimenting with extending check step to let test assertions work with visible / invisible Chrome.

What are the usual test assertions that you use? I can test with some of common use cases and see if other supporting changes can be made to increase the range of assertions possible with TagUI.

This is more for integration with CI/CD (continuous integration / continuous deployment) purpose with the generated XUnit XML file. At the moment test assertions only work with PhantomJS / Firefox mode as CasperJS natively does not support Chrome.

UPDATE -

It looks like using the check step to evaluate a condition for test assertion can already cover a lot of use cases. it is more about reducing user friction towards 0 in getting the stuff for use in the condition, eg page title, url, text on page, element present, counting elements, stop watch timer, element visible (an element can be present on webpage DOM but not rendered as visible). wait for more inputs here and from issues raised in other test automation repos.

[jtanningbed]

hi @kensoh , sorry for the delay in response. thanks for being to diligent in maintaining this! I was actually unaware that the check step was not using CasperJS, but I may have to give it try. As for usual test assertions, I'm generally only looking for basic validations (element existence, elements displayed/visible, text assertions, most of which you have already mentioned above).
At this point in time, the lack of maintenance for PhantomJS and CasparJS has made integrating this into my current testing structure a bit more unreliable, but there is still great use in the ease of writing easy, readable tests for chrome and headless chrome. I will be showing this to some of my colleagues as I believe the work you've done on this is tremendous. Thank you!

[kensoh]

Thanks Jason! Yeah before this, check step does not do test assertions for CI purpose. In this issue, I make TagUI automatically do test assertions for check step whenever test option is used. In this way, a XUnit XML file will be generated for integration with Jenkins or other CI tools. The goal is to make test automation work whether visible Chrome, headless Chrome, Firefox or PhantomJS is used.

Just added visible() function. Looks like supporting commits are more or less completed. Will wait for a while more to see if there are any other inputs from other test automation repos, then cut a Test Automation Release with packaged installations (so users can just unzip and run or npm install tagui).

Yeah PhantomJS has stopped maintenance. It may take years for web technologies to change so drastically that TagUI in its current form will not work, or earlier. Bulk of TagUI's development efforts are optimizations to make the process of creating / deploying / maintaining automation easier. The roadmap is to migrate to Node.js execution environment and Puppeteer when the time is ripe. And release a migration tool to migrate existing TagUI flow scripts to the new format, if required.

It shouldn't be hard to have a new branch and switch the engine. For now, I'll try to hold back moving towards that, as using Node.js execution environment immediately excludes a lot of non-developer users from using TagUI.

UPDATE - removing experimental in issue title. this few commits are so useful that they deserve a release of their own. pending inputs from some other test automation repos for any final changes.

[kensoh]

Don't think there are any other specific inputs from other repos, updated sample flow 7_testing to use check step for test assertions. Next is update readme, create package installations, and cut a test automation release.

This should be good enough for lightweight test automation CI/CD using visible or headless Chrome. Can always improve and update along the way. For Node.js developers, I'll recommended creating test automation frameworks from scratch using Google Puppeteer.

[kensoh]

v2.6 Test Automation Release is out - readme, packaged installations, npm package updated.

Keeping this open for some time for testing related issues or problems related to this release.