- Adding possibility of setting env, configDir and disable html repor… #1429

joelpramos · 2021-01-10T18:36:06Z

Description

Thanks for contributing this Pull Request. Make sure that you submit this Pull Request against the develop branch of this repository, add a brief description, and tag the relevant issue(s) and PR(s) below.

Adding parameter that allows debug server to be reused - when debug session terminates VS Code disconnects but debug server continues to run

As I run my own framework on top of Karate one of the things that I always wanted to avoid was the System.exit() that happens after the debug session. I used to have some weird workarounds with the Java Security Manager but introduced a variable (exitAfterDisconnect) which allows that behavior to be controlled. With it you can create a debug that doesn't terminate the JVM (just disconnects the session) server by extending DapServer class and setting that parameter as false.

The behavior from VS Studio Code is just that the debug session will terminate so it's seamless to the user.

Adding possibility of setting env, configDir and disable html report via debugger in VS Code

With Karate v1.0.0 allowing env to be passed in the runner and a configDir I externalized all configurations and use variables in all test cases but the debugger needs to run against a certain environment. This allows the parameters to be passed in the karateOptions of the Karate Runner in VS Code. Here's an example of the field in the Runner:

"karateOptions": "\"${command:karateRunner.getDebugFile}\" \"--configdir=C:\\test_cases\\config\" \"--env=dev01\"",

@ptrthomas let me know your thoughts. Might be good to tag the maintainer of the Karate Runner to check it out.

Relevant Issues : (compulsory)
Relevant PRs : (optional)
Type of change :
- New feature
- Bug fix for existing feature
- Code quality improvement
- Addition or Improvement of tests
- Addition or Improvement of documentation

…t via debugger in VS Code - Adding parameter that allows debug server to be reused - when debug session terminates VS Code disconnects but debug server continues to run

joelpramos · 2021-01-10T22:55:30Z

btw having to wrap each parameter with quotes seems kind of weird... but that's the behavior supported by the function in Karate that parses CLI arguments. was it intended?

ptrthomas · 2021-01-11T04:51:36Z

cc @kirksl and @ivangsa I think you have some good idea about the double-quotes issue. it may take me a couple of days to get to this, any inputs welcome in the meantime

ptrthomas · 2021-01-11T04:55:55Z

karate-core/src/main/java/com/intuit/karate/Main.java

@@ -94,6 +94,9 @@
    @Option(names = {"-o", "--output"}, description = "directory where logs and reports are output (default 'target')")
    String output = FileUtils.getBuildDir();

+    @Option(names = {"-r", "--htmlreport"}, description = "output html report enabled (default 'true')")


prefer --report if that sounds okay. here is where potentially can add more flags in future similar to --format

I was wondering if a system to prefix ~ to disable works, e.g. (not needed now, just seeing if makes sense to you)

-r ~html,~json - which means don't emit html and JSON reports which are both by default

btw having to wrap each parameter with quotes seems kind of weird... but that's the behavior supported by the function in Karate that parses CLI arguments. was it intended?

Hi @joelpramos it's weird but parsing options works well with short options (i.e: -e) but long options (--env) needs quoting, this must be something related to picocli

This works fine, but if using long options version it fails

@Test void testParsingCommandLine7() { Main options = IdeMain.parseIdeCommandLine("-H com.intuit.karate.cli.VSCodeHook -t ~@ignore -e local -g W:\\dev\\sandbox\\karate-1.0.0-tests\\src\\test\\resources w:\\dev\\sandbox\\karate-1.0.0-tests\\src\\test\\resources\\todos.feature:27"); assertEquals(1, options.paths.size()); assertEquals(1, options.tags.size()); assertEquals(1, options.hookFactoryClassNames.size()); assertEquals("local", options.env); assertEquals("W:\\dev\\sandbox\\karate-1.0.0-tests\\src\\test\\resources", options.configDir); }

also this looks very elegant: -r ~html,~json

yeah I like that much more elegant

@ptrthomas what if I change to the --format instead and remove the new property I added but make sure that html is enabled by default unless explicitly negated (~html)?

@joelpramos agreed. I'd try to fit a colon maybe e.g. cucumber:json but fine otherwise also

Makes sense. I was reviewing the usage of the Karate JSON report and I'm not certain whether it can be disabled - the Results object is always instantiated out of the File object where that JSON is stored. I assume it's related with the distributed testing.

We could try to refactor to allow it to be disabled but produce an error if you try using through JobManager with it disabled but I don't really have that set up yet.

It doesn't truly impact my use case as I was just trying not to store anything during a debug session but it's not breaking anything. Maybe I can add a Hook that is executed after the Suite in a debug session to clear any files that are produced?

@joelpramos great point thanks. I propose that if the user disables the karate json it will disable the karate html also or at least the summary HTML and printing the final results tally to the sys.out

I was contemplating an option where the JSON would be kept in memory until the end of the test, but not worth it maybe

just to clarify - the reason to persist the JSON is not just for distributed testing, but to save memory for a large suite. otherwise we would be accumulating lots of objects just for the sake of the summary report. come to think of it there is a feature-summary json you can see used in the Results class. maybe that could suffice. agh so many decisions >_<

I'll leave it for now and maybe try to create a Hook for the debugger later to clear the created files after the tests are done. Can probably be enabled by default to the debugger but might even be useful for CI/CD pipelines if we only care about the logs of the execution or if you integrate with an external system (I use report portal and I also zip the HTML report and ship it there).

… timestamp in folder name to avoid overriding reports

joelpramos · 2021-01-11T23:41:04Z

Update completed and made the default output dir of the debugger end with a timestamp so multiple executions don't override the previous reports.

joelpramos

ready to review again

ptrthomas · 2021-01-13T13:36:59Z

karate-core/src/main/java/com/intuit/karate/debug/DapServerHandler.java

+        // create a subfolder so multiple executions don't override the previous reports
+        String outputDir = options.getOutput();
+        if(outputDir.contentEquals(FileUtils.getBuildDir())) {
+            DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyyMMdd_HHmmss");


I may refactor this to reuse Suite.backup...() https://github.com/intuit/karate/blob/develop/karate-core/src/main/java/com/intuit/karate/Suite.java#L324 - let me know if ok

Yup all good - I searched by date formatter usages and couldn't spot any which is why added it. I think the important part here is being a unique folder for local debugging - different use case than a CI/CD for example where you always want in the same folder for your Jenkins/etc to pick it up from there.

decided to default the behavior of backing up reports folder to true and added runner builder flag some code re-use for parsing cli options

ptrthomas · 2021-01-13T14:04:37Z

@joelpramos can you review this update over your changes: 6436b2c

joelpramos · 2021-01-13T14:11:18Z

@joelpramos can you review this update over your changes: 6436b2c

is the backup dir printed somewhere? wondering whether we can also add it in that System.out.print() report that shows up after Suite is complete where we see the other link.

ptrthomas · 2021-01-13T14:39:45Z

@joelpramos good idea, one extra log statement maybe - go ahead and suggest / PR

#1429 (comment)

ptrthomas · 2021-01-13T14:47:22Z

@joelpramos actually the summary report always has the link to HTML file (full path) may be enough

I just added a log statement, hope that suffices

joelpramos · 2021-01-13T15:38:52Z

@joelpramos actually the summary report always has the link to HTML file (full path) may be enough

I just added a log statement, hope that suffices

great by me, thanks

- Adding possibility of setting env, configDir and disable html repor…

5d0e08c

…t via debugger in VS Code - Adding parameter that allows debug server to be reused - when debug session terminates VS Code disconnects but debug server continues to run

ptrthomas requested changes Jan 11, 2021

View reviewed changes

joelpramos added 2 commits January 11, 2021 17:48

Refactoring as per comments in karatelabs#1429

75ca280

Enabling Html report even when no configuration is passed & including…

2bec96a

… timestamp in folder name to avoid overriding reports

joelpramos commented Jan 12, 2021

View reviewed changes

ptrthomas reviewed Jan 13, 2021

View reviewed changes

ptrthomas merged commit 47638e0 into karatelabs:develop Jan 13, 2021

ptrthomas added a commit that referenced this pull request Jan 13, 2021

refactor over pr #1429

6436b2c

decided to default the behavior of backing up reports folder to true and added runner builder flag some code re-use for parsing cli options

ptrthomas added a commit that referenced this pull request Jan 13, 2021

add log message for report dir backup

65f8ead

#1429 (comment)

This was referenced Jan 15, 2021

DebugHandler: Allow setting KarateOptions along with a single feature #1426

Closed

Clearing debug session properties when disconnecting channel #1441

Merged

dependabot bot mentioned this pull request Mar 15, 2021

build(deps-dev): bump karate-junit5 from 0.9.9.RC4 to 1.0.0 HSRM-Vegetables/myfoodcoop-service#35

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

- Adding possibility of setting env, configDir and disable html repor… #1429

- Adding possibility of setting env, configDir and disable html repor… #1429

joelpramos commented Jan 10, 2021 •

edited

Loading

joelpramos commented Jan 10, 2021

ptrthomas commented Jan 11, 2021

ptrthomas Jan 11, 2021

ivangsa Jan 11, 2021

ivangsa Jan 11, 2021

joelpramos Jan 11, 2021

joelpramos Jan 11, 2021

ptrthomas Jan 11, 2021

joelpramos Jan 11, 2021

ptrthomas Jan 11, 2021

ptrthomas Jan 11, 2021

joelpramos Jan 11, 2021

joelpramos commented Jan 11, 2021 •

edited

Loading

joelpramos left a comment

ptrthomas Jan 13, 2021

joelpramos Jan 13, 2021

ptrthomas commented Jan 13, 2021

joelpramos commented Jan 13, 2021

ptrthomas commented Jan 13, 2021

ptrthomas commented Jan 13, 2021

joelpramos commented Jan 13, 2021

- Adding possibility of setting env, configDir and disable html repor… #1429

- Adding possibility of setting env, configDir and disable html repor… #1429

Conversation

joelpramos commented Jan 10, 2021 • edited Loading

Description

joelpramos commented Jan 10, 2021

ptrthomas commented Jan 11, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joelpramos commented Jan 11, 2021 • edited Loading

joelpramos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ptrthomas commented Jan 13, 2021

joelpramos commented Jan 13, 2021

ptrthomas commented Jan 13, 2021

ptrthomas commented Jan 13, 2021

joelpramos commented Jan 13, 2021

joelpramos commented Jan 10, 2021 •

edited

Loading

joelpramos commented Jan 11, 2021 •

edited

Loading