Karate RuntimeHooks beforeScenario is not executed for Dynamic scenarios

[ivangsa]

When running dynamic scenarios RuntimeHook#beforeScenario are not executed

This is the output of MyRuntimeHook

beforeScenario is dynamic:true
beforeFeature
afterScenario
afterScenario
afterFeature

myproject.zip

[ptrthomas]

looks introed in #1453 - cc @joelpramos can either of you sort this out, this will take time for me to get to

[ivangsa]

@joelpramos you probably have more context for this already, otherwise let me know I'll look into it..

[joelpramos]

Seems like on beforeRun() the condition shouldn't be background == null (for dynamic scenarios) but rather checking that it's not the background section that's being executed. Probably a combination with the check for dynamic and ensuring the execution is not for the background section, as that's executed separately before each scenario iteration.

@ivangsa I can check this later in the week only, shouldn't be a major change. I haven't opened your project yet but will look into and add as a unit test for Karate. If you can do that beforehand it'd be great (maybe it already is mostly done).

[ivangsa]

yes, it's already set as a junit test

[ptrthomas]

@joelpramos I attempted a fix which I have not tested: aa4ed47

wanted to make sure you were aware of scenario.isOutlineExample()
also I did a micro-optimization for that special case of hooks in debug mode, hope I didn't break anything.

[joelpramos]

Still few tweaks needed based on my unit tests. We are currently triggering the beforeScenario() for the background section (but not after scenario).

For outline scenarios I'd suggest not to trigger beforeScenario() and afterScenario() for the background section. To me it's already a "before" concept to begin with. Those would be executed for each execution of Scenario Outline.

The beforeStep() / afterStep() would still be executed in each step of the background section (and then for each step, for each execution, of scenario outline).

Let me know if you're aligned @ptrthomas .

[ptrthomas]

@joelpramos if there is a background, the beforeScenario() should be executed no matter what scenario it is IMO. and once fired it shouldn't be executed. is that not how it is / should be ?

[joelpramos]

The beforeScenario() is being executed for the background section; then beforeScenario() and afterScenario() are executed as expected for each iteration of Scenario Outline.

Since dynamic scenarios "pre execute" the background section before and just reuse the results, the desired behavior is a decision we'll have to make as it will never be 100% consistent with "normal" scenarios in which background steps.

My thinking is not executing beforeScenario() nor afterScenario() for the background section only. This might create issues for someone trying to get a perfect async scenario reporting to some reporting tool but my opinion before was already this is too complex and might as well deal with it once the root-level feature completes. Also if there's some pre-load of data which is then needed in the background section ... alternatively we could add beforeBackground() and afterBackground(), although not sure how hard it'll make it for "normal" scenarios.

[ptrthomas]

@joelpramos oh ok - if there is a background it should execute only once for a dynamic. agreed. I think you can go ahead, we are aligned. not something I use that much so I'm biased ;)

[ptrthomas]

@joelpramos and for a dynamic - we want the beforeScenario() to fire before every scenario - but unlike "normal" scenarios, we ignore the background. did I get that right ?

[joelpramos]

Correct @ptrthomas . So here's my proposal now that I'm thinking through the different use cases, let me know what you think.

• execute beforeScenario() / afterScenario() per iteration of dynamic scenario outline (so as many times as needed per iteration);
• do not execute beforeScenario() / afterScenario() for background section (again, for dynamic only)
• add beforeBackground() / afterBackground() to allow granular control if there's a need (e.g. load data from a DB, do something in the background which will be reused, release connection)
• beforeBackground() / afterBackground() would NOT be executed in normal Scenario(s) - I think you already have in the documentation that the behavior is different anyways but we can add more verbiage if needed
• if someone wants to execute one beforeScenario() and one afterScenario() holistically for Scenario Outline (and not use the beforeFeature() maybe cause of calling reusable Features) they can do it by checking the exampleIndex (probably add method(s) isFirstExample() and isLastExample() for convenience)

[ptrthomas]

@joelpramos yep this sounds great. IMO beforeBackground() is "good to have" - but go ahead 👍

[joelpramos]

Opened PR #1508. I did not run all the unit tests - will review any test failures tomorrow. @ivangsa check out whether my unit test coverage on HooksTest satisfies all your use cases.
Few notes:

• ended up dropping isFirstExample() and isLastExample() since how would I know in the parallel runner? a future improvement could be exposing the calculated Examples Table in the Scenario Runtime and all the data would be available for any sort of logic related with first/last
• https://github.com/intuit/karate/tree/develop#tags-and-examples this feature was broken - I did the fix but might need someone revisiting (wasn't entirely certain if the default behavior I left is the correct - if there's a tag selector but no matching Examples, use the Examples without tag). The logic was also a bit shoved into a for-loop and could use some cleanup, but we have the unit tests
• did a small change in the placing of the Feature Hook so the background section in scenario outlines is not run if the beforeHook() evalutes to false - don't think it impacts anything else - hopefully no impacts on the generated reports but check the change in FeatureRuntime.java:161 (moved from the processScenario() method)

[ptrthomas]

1.0 released