Failed to generate the serenity report if a test failed in a set (eg: set contains 3 tests (2 tests passed and 1 failed)) - Serenity version 4.2.8

[baskarcsb]

If I executed a set of multiple tests (let's say the set includes 5 tests) and 4 of those tests passed while 1 test failed, then the Serenity report is not being generated correctly.

After conducting the analysis, running the mvn serenity:aggregate command generates the complete report.

Note: If all five tests are successfully completed, then the report will be generated as anticipated.

[wakaleo]

That sounds like an issue with your build configuration. Do you have a project that reproduces this?

[baskarcsb]

Yes, John, this appears to be the configuration for the Serenity Testrunner. I will implement this in my project and keep you updated on the results.

I just wanted to give you a heads-up, John. After a lengthy process, we are finally upgrading Serenity from version 2.0.81 to 4.2.8 (from junit 4 to junit 5). While this isn't a challenge, I would appreciate your assistance if in case I overlook any configuration-related issues. Thank you for your continued support, John!

[baskarcsb]

Hi John,

I have finally upgraded to JUnit 5 with Serenity 4.2.8. However, our Serenity 4.x.x JUnit 5 project is experiencing a slowdown in execution speed when I instantiate more than 10 @steps in a step definition, particularly because each helper method contains over 8000 lines of code. (If I instantiate one or two @steps, the execution speed meets my expectations.)

Moreover, the execution speed is at its best when there are fewer lines in the helper class. But I'm not certain if I've overlooked something or if it's a bug in the newer version?

Note: Sorry John, I cannot create a sample project with 8000+ lines of code to reproduce this issue.

The Source code compiler version is Java 21

Additionally, I do not encounter this issue in Version 2.0.81.

Sample StepDefination class:

public class BankGlobalSteps {

@steps
NavigationHelper navigationHelper;

@steps
BankInvoicesHelper bankInvoicesHelper;

@steps
BankInquiryHelper bankInquiryHelper;

@steps
EmailHelper emailHelper;

@steps
BankTaskListHelper BankTaskListHelper;

@steps
BranchHelper branchHelper;

@steps
LoginHelper loginHelper;

@steps
AppointmentHelper appointmentHelper;

@steps
RegistrationHelper registrationHelper;

@steps
DepositHelper depositHelper;

@steps
LoanChargesHelper loanChargesHelper;

@steps
CustomerInquiryHelper customerInquiryHelper;

@steps
BalanceSetupHelper balanceSetupHelper;

@steps
BranchControlHelper branchControlHelper;

@steps
RuleHelper ruleHelper;

@steps
HtbHelper htbHelper;

@steps
MessageBoxHelper messageBoxHelper;

@steps
PersonalFormHelper personalFormHelper;

@steps
SelectionHelper selectionHelper;

Sample Helper Class

public class NavigationHelper extends ScenarioSteps{

public NavigationHelper(Pages pages) {
super(pages);
}

Sample Page Class

public class LoginPage extends PageObject {
@findby(css = "h2")
private WebElementFacade screenHeader;

}

Serenity Properties:

webdriver.driver = chrome
webdriver.base.url = https://www.google.com

TestRunnerSuite:

@suite
@IncludeEngines("cucumber")
@SelectClasspathResource("/features")
@ConfigurationParameter(key = GLUE_PROPERTY_NAME, value = "com.gehc.cb.bdd.steps")
@ConfigurationParameter(key = PLUGIN_PROPERTY_NAME, value = "io.cucumber.core.plugin.SerenityReporterParallel,pretty,timeline:build/test-results/timeline")
@ConfigurationParameter(key = FILTER_TAGS_PROPERTY_NAME, value="@smoke")
public class TestRunnerSuite {}

My POM.xml:

4.0.0 com.gehc.cb.bdd test 1.0-SNAPSHOT jar

<name>Sample Serenity project using Cucumber and WebDriver</name>
<properties>
	<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
	<serenity.version>4.2.8</serenity.version>
	<junit-vintage-engine.version>5.11.3</junit-vintage-engine.version>
	<junit-platform-suite.version>1.11.3</junit-platform-suite.version>
	<cucumber-junit-platform-engine.version>7.20.1</cucumber-junit-platform-engine.version>
	<gherkin.version>2.12.2</gherkin.version>
	<json.version>20211205</json.version>
	<slf4j-api.version>2.0.0-alpha5</slf4j-api.version>
	<rest-assured.version>5.3.1</rest-assured.version>
	<json.version>20211205</json.version>
	<lambdaj.version>2.3.3</lambdaj.version>
	<log4j.version>2.17.1</log4j.version>
	<httpcore.version>4.4.4</httpcore.version>
	<guava.version>31.0.1-jre</guava.version>
	<commons-net.version>3.3</commons-net.version>
	<jackson-databind.version>2.6.3</jackson-databind.version>
	<HCObjects.version>1.0.0</HCObjects.version>
	<cachejdbc.verison>1.0.0</cachejdbc.verison>
	<commons-csv.version>1.3</commons-csv.version>
	<commons-collections4.version>4.0</commons-collections4.version>
	<groovy.version>3.0.9</groovy.version>
	<json-simple.version>1.1.1</json-simple.version>
	<encoding>UTF-8</encoding>
	<tags></tags>
	<webdriver.base.url></webdriver.base.url>
</properties>
<repositories>
	<repository>
		<id>SmartBearPluginRepository</id>
		<name>SmartBear Repository</name>
	    <url>https://smartbearsoftware.com/repository/maven2</url>
	</repository>
</repositories>
<dependencies>
	<dependency>
		<groupId>net.serenity-bdd</groupId>
		<artifactId>serenity-core</artifactId>
		<version>${serenity.version}</version>
	</dependency>
	<dependency>
		<groupId>net.serenity-bdd</groupId>
		<artifactId>serenity-cucumber</artifactId>
		<version>${serenity.version}</version>
	</dependency>
	<dependency>
		<groupId>net.serenity-bdd</groupId>
		<artifactId>serenity-screenplay</artifactId>
		<version>${serenity.version}</version>
	</dependency>
	<dependency>
		<groupId>net.serenity-bdd</groupId>
		<artifactId>serenity-screenplay-webdriver</artifactId>
		<version>${serenity.version}</version>
	</dependency>
	<dependency>
		<groupId>net.serenity-bdd</groupId>
		<artifactId>serenity-ensure</artifactId>
		<version>${serenity.version}</version>
	</dependency>
	<dependency>
		<groupId>org.junit.platform</groupId>
		<artifactId>junit-platform-launcher</artifactId>
		<version>${junit-platform-suite.version}</version>
	</dependency>
	<dependency>
		<groupId>io.cucumber</groupId>
		<artifactId>cucumber-junit-platform-engine</artifactId>
		<version>${cucumber-junit-platform-engine.version}</version>
	</dependency>
	<dependency>
		<groupId>org.junit.platform</groupId>
		<artifactId>junit-platform-suite</artifactId>
		<version>${junit-platform-suite.version}</version>
	</dependency>
	<dependency>
		<groupId>org.junit.jupiter</groupId>
		<artifactId>junit-jupiter-engine</artifactId>
		<version>${junit-vintage-engine.version}</version>
	</dependency>
	<dependency>
		<groupId>org.junit.vintage</groupId>
		<artifactId>junit-vintage-engine</artifactId>
		<version>${junit-vintage-engine.version}</version>
	</dependency>
	<dependency>
		<groupId>ch.qos.logback</groupId>
		<artifactId>logback-classic</artifactId>
		<version>1.2.10</version>
	</dependency>
	<dependency>
		<groupId>org.assertj</groupId>
		<artifactId>assertj-core</artifactId>
		<version>3.23.1</version>
	</dependency>
	<dependency>
		<groupId>org.json</groupId>
		<artifactId>json</artifactId>
		<version>${json.version}</version>
	</dependency>
	<dependency>
		<groupId>org.slf4j</groupId>
		<artifactId>slf4j-api</artifactId>
		<version>${slf4j-api.version}</version>
	</dependency>
	<!-- https://mvnrepository.com/artifact/io.rest-assured/rest-assured -->
	<dependency>
		<groupId>com.googlecode.lambdaj</groupId>
		<artifactId>lambdaj</artifactId>
		<version>${lambdaj.version}</version>
	</dependency>
	<dependency>
		<groupId>org.apache.logging.log4j</groupId>
		<artifactId>log4j</artifactId>
		<version>${log4j.version}</version>
		<type>pom</type>
	</dependency>
	<dependency>
		<groupId>com.smartbear.soapui</groupId>
		<artifactId>soapui</artifactId>
		<version>5.7.0</version>
		<exclusions>
			<exclusion>
				<groupId>com.smartbear.utils.analytics</groupId>
				<artifactId>*</artifactId>
			</exclusion>
			<exclusion>
				<groupId>log4j</groupId>
				<artifactId>*</artifactId>
			</exclusion>
			<exclusion>
				<groupId>org.codehaus.groovy</groupId>
				<artifactId>groovy-all</artifactId>
			</exclusion>
			<exclusion>
				<groupId>xml-apis</groupId>
				<artifactId>xml-apis</artifactId>
			</exclusion>
			<exclusion>
				<groupId>org.apache.xerces</groupId>
				<artifactId>xml-apis</artifactId>
			</exclusion>
			<exclusion>
				<groupId>xerces</groupId>
				<artifactId>xmlParserAPIs</artifactId>
			</exclusion>
            <exclusion>
                <groupId>xom</groupId>
                <artifactId>xom</artifactId>
            </exclusion>
		</exclusions>
	</dependency>
	<!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-log4j12 -->
	<dependency>
		<groupId>org.apache.httpcomponents</groupId>
		<artifactId>httpcore</artifactId>
		<version>${httpcore.version}</version>
	</dependency>

	<dependency>
	    <groupId>xom</groupId>
	    <artifactId>xom</artifactId>
	    <version>1.3.9</version>
	</dependency>
	<!-- https://mvnrepository.com/artifact/com.google.guava/guava -->
	<dependency>
		<groupId>com.google.guava</groupId>
		<artifactId>guava</artifactId>
		<version>${guava.version}</version>
	</dependency>
	<dependency>
		<groupId>commons-net</groupId>
		<artifactId>commons-net</artifactId>
		<version>${commons-net.version}</version>
	</dependency>
	<dependency>
		<groupId>org.apache.commons</groupId>
		<artifactId>commons-csv</artifactId>
		<version>${commons-csv.version}</version>
	</dependency>
	<dependency>
		<groupId>org.apache.commons</groupId>
		<artifactId>commons-collections4</artifactId>
		<version>${commons-collections4.version}</version>
	</dependency>
	<dependency>
		<groupId>org.codehaus.groovy</groupId>
		<artifactId>groovy</artifactId>
		<version>${groovy.version}</version>
	</dependency>
	<dependency>
		<groupId>com.googlecode.json-simple</groupId>
		<artifactId>json-simple</artifactId>
		<version>${json-simple.version}</version>
	</dependency>
	<!-- https://mvnrepository.com/artifact/javax.jmdns/jmdns -->
	<dependency>
		<groupId>javax.jmdns</groupId>
		<artifactId>jmdns</artifactId>
		<version>3.4.1</version>
	</dependency>
	<dependency>
		<groupId>com.fasterxml.jackson.core</groupId>
		<artifactId>jackson-databind</artifactId>
		<version>${jackson-databind.version}</version>
	</dependency>
	<dependency>
		<groupId>net.serenity-bdd</groupId>
		<artifactId>serenity-rest-assured</artifactId>
		<version>${serenity.version}</version>
	</dependency>
</dependencies>
<build>
	<plugins>
		<plugin>
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-surefire-plugin</artifactId>
			<version>3.1.2</version>
			<configuration>
			<testFailureIgnore>true</testFailureIgnore>
			</configuration>
		</plugin>
		<plugin>
			<artifactId>maven-failsafe-plugin</artifactId>
			<version>3.1.2</version>
			<configuration>
				<includes>
					<include>**/*Runner.java</include>
				</includes>
				<systemPropertyVariables>
					<webdriver.base.url>${webdriver.base.url}</webdriver.base.url>
				</systemPropertyVariables>
				<useUnlimitedThreads>true</useUnlimitedThreads>
			</configuration>
			<executions>
				<execution>
					<goals>
						<goal>integration-test</goal>
						<goal>verify</goal>
					</goals>
				</execution>
			</executions>
		</plugin>
		<plugin>
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-compiler-plugin</artifactId>
			<version>3.8.1</version>
			<configuration>
				**<source>21</source>
				<target>21</target>**
			</configuration>
		</plugin>
		<plugin>
			<groupId>net.serenity-bdd.maven.plugins</groupId>
			<artifactId>serenity-maven-plugin</artifactId>
			<version>${serenity.version}</version>
			<configuration>
				<tags>${tags}</tags>
			</configuration>
			<executions>
				<execution>
					<id>serenity-reports</id>
					<phase>post-integration-test</phase>
					<goals>
						<goal>aggregate</goal>
					</goals>
				</execution>
			</executions>
		</plugin>
	</plugins>
</build>

[wakaleo]

Hmmm, sound's like the byte-buddy byte code instrumentation takes some time with larger classes. Have you tried breaking your step definition class into a number of smaller ones?

[baskarcsb]

John, I haven't had the chance to test that scenario yet, but I will let you know as soon as I do. I also want to take a moment to thank you for your exceptional support over the past 10 years! Thank you, John.