Faster startup of @QuarkusTest (disable web-server?) #14927
Comments
ghost
added
the
|
It would not be the web server, on most machines that is low hundreds of ms. Can you create a flame graph using https://github.com/jvm-profiling-tools/async-profiler and we can have a look at what is taking up the startup time. |
|
I'm not familiar with how profiling works but I have tried to generate the three different profiles I could. |
|
Let's say Quarkus had this functionality, what would your test look like? |
|
FWIW, we have just split up our single module Maven project into a 5 modules project. There is a |
|
So in this case, all you are doing is interacting with the database taking advantage of the fact that Quarkus has wired up all the necessary beans for you? |
|
Yes, and that the same db config is used as for rest assured tests etc. |
|
And is there any indication that the web layer is slowing down the tests significantly? |
|
@geoand I don't think web layer is the main problem for me. I was referring to http mostly because of the surprising need to configure something you don't actually use. What I can give you right away is this: Notes:
|
|
Thanks for the information |
|
The 'Quarkus Stopped' lag should be fixed upstream. So out of the 8s we have: Which leaves 4s. My guess would be this is actually starting hibernate/arc, but without a profile its hard to say. |
I see it's gone already in 2.1.0. 👍
h2 will take its toll as well. With 2.1.0 and some Maven tweaking (
So that is more or less the same. But this is suprising (wasn't there with 2.0.3): Has something changed wrt web layer in tests that don't actually use it? |
Not that I know of. If you have a sample that shows this problem, I'll take a look |
|
WDYT about exposing It's not so much because of performance concerns but DRY config instead: With Quarkus 2.1 I get that warning I posted previously in my What would be even better is that extension could tell PS: I haven't made any progress on profiling, sorry. |
Well, we can likely have a configuration property for that. |
|
I finally managed to produce a flamegraph, captured from a surefire run of an empty QuarkusTest via: Is that of any help? |
|
Unfortunately nothing really stands out as being an easy target. Liqibase is taking up a fair bit of time, as is the hibernate entity enhancer and Jacoco transformation, but I don't know if there is much we can do about it. |
|
BTW how long did startup take in this case? |
|
I took Liquibase out of the equation and added So, until the first "real" log messages from Quarkus (those config warnings), ~4s are spent. The flamegraph for that is here. Btw, what's that PS: It would be nice if PPS: Surefire also takes one second before actually running that specific test, JVM forking overhead I suppose. |
|
|
Open an issue please - this is an easy ask to fulfill :) |
|
|
@mkouba do you have some hints where to look in ArC? |
|
@famod @geoand @stuartwdouglas Well, the output of |
Ah, sorry, that wasn't the right one. Here is a brand new one: profile.zip |
|
Well yes both of ORM and Validator are known to be quite expensive to boot, but I'll need to have a look to see if the specific costs here are expected. |
|
So among costs there is Hibernate ORM's enhancement of entities via ByteBuddy, combined with Jacoco based enhancement? I don't know much about this last one but considering the ORM one is expensive, I'd not be surprised if the other one is as well. It seems the Jacoco enhancer has the same problem as the ORM one: it's bypassing the visitor API to work on the byte array instead; this is not optimal and a known limitation of the Hibernate one as well; perhaps it's time to look at alternative options to optimise both of these. However, both transformations are supposedly using a Quarkus cache that Stuart designed to minimize cost from enhancement, and even if we could improve this by say doubling enhancmenet speed, from the samples it seem they are responsible for less than 20% of total time anyway for enhancement (counting ORM+Jacoco), plu some ~3.5% for ORM's enhanced proxies (in a different section of the flamegraph). If this is taking >7 seconds there is something else at play for sure as well. Perhaps not much memory? Also I don't actually see Hibernate Validator in the graph, besides the ~2,3% cost in the main pillar. It strikes me t hat there is an extremely high and thing flame triggered from |
|
Thanks @Sanne for the detailed answer!
Yeah, I too noticed jacoco. And I've just realized that we are not setting any exclusions like we do for the jacoco-maven-plugin.
I'll check. |
|
Something to bear in mind with the transformation stuff is that even though it is expensive it should be done in a thread pool, so even if it is taking up N ms on the graph, in real time it should only be N/cores. The cache that Sanne mentions is more for dev mode restarts, and won't really help with testing startup time. |
|
Interesting, |
Hm, getting very mixed results with includes/excludes. Maybe I'm using the wrong format. Will have to take a closer look.
Nope, doesn't help. |
|
I am wondering if it makes sense to have something like the following: @QuarkusTest
@ExcludeExtension(groupId="io.quarkus", artifactId="quarkus-flyway")
public class MyCustomTest {
}The result would be that the test would setup Quarkus bootstrap in a way that completely excludes This could obviously lead to problems depending on what extension was excluded, so it would be up to the user the exclude only things that would not be break the application (although we could perhaps also introduce some kind of annotation for excluding classes and/or packages). WDYT? |
Hm, so any exclusion like this would have to result in an app restart, or? Would it make sense to extend the |
Yes, if different exclusions are needed, a restart will be necessary
That's also an option. |
|
A I don't want to overcomplicate this, but a restart is a big price to pay. So I was wondering how "binding" that exclusion should be. |
|
A big problem with this is how you exclude transient dependencies. e.g. if your application depends on quarkus-resteasy, and you exclude it, do you still get the HTTP server brought in by I like the idea in theory, but we need to give some thought to how to handle these cases. |
|
Yeah, my thought was that bootstrap would exclude all the Quarkus dependencies that depended on whatever was excluded. @aloubyansky I assume this is something that could be done relatively easily, correct? |
|
@famod I think that would actually be pretty dangerous behavior. My preference would be that exclusions are taken at face value. But good point about ordering tests, so |
|
I think having something (experimental) as described above would be a nice addition for |
Well, it shouldn't be the default because some tests might actually fail if an extension is still around. So what I'm thinking about is a flag parameter of that new annotation, to opt in into that "lenient" behavior. Without such a mode, I really see the risk that this whole But sure, it's important to get starting somewhere and improve gradually. 👍 |
Yup, that could certainly happen, depending on the use case. |
Not exactly easy but could be done. It's just it would have to be implemented differently for Maven and Gradle, where it will be more tricky. We actually still don't support even forced dependencies in Gradle, although getting closer to that. So, |
|
Ah, I wasn't aware of that. |
|
@aloubyansky, |
|
hey all - I'm a little confused by the direction of the conversation. This discussion seemed it had stalled ~6 months ago with #14927 (comment) - which concluded that @famod was going to have a closer look into jacoco. Then 2 days ago the topic is resurrected but going in a different direction; I really don't mind if you all find a nice solution to not need starting the webserver :) But let's please first make sure this is actually useful? There seems to be no meaningful data supporting this direction. |
No, exclusions are filters that have to be applied by the resolver itself. We can't compensate for it properly after a configuration has been resolved. In Gradle, I suppose it could be done using https://docs.gradle.org/current/javadoc/org/gradle/api/artifacts/Configuration.html#exclude-java.util.Map- |
|
@Sanne what I am proposing is not for the web server per-se, it's a general solution. Not sure what data you would like to see to support the usefulness of the proposal |
|
@geoand just reminding that Stuart had pointed out that starting the server just takes some milliseconds - so let's make sure we're looking at the right problem? Older discussions (above) seem to have been suggesting the need to look further into entity enhancements, and possibly jacoco related instrumentation - to which Stuart also warned that it's performed in parallel so it needs to be interpreted with care. Just not sure why we're back at discussing to skip starting the server - perhaps you have figure out something more? Not meaning to shut you down, I'm just trying to understand why we're back to that initial idea. Regarding Hibernate entity enhancement performance: to be clear I haven't looked into the data of this issue as I had the impression enough smart people were looking into this already, but if that's the culprit then I can certainly have a look as well. |
|
I had a specific use case in mind, which was more about giving users more flexibility when testing, rather than addressing performance. It's been a few days however and it has slipped my mind. |
|
For me the problem with this is not the startup time but the coupling that is introduced that is in contradiction to test units in isolation. |
|
https://quarkusio.zulipchat.com/#narrow/stream/187030-users/topic/Anything.20.22smaller.22.20than.20.40QuarkusTest.3F was another ask that would have been covered by #14927 (comment) |
This is a feature request to speed up tests using the
@QuarkusTestnotation. In our applications we do not use "HTTP based testing" but do "normal" unit testing. It is however nice to use the@QuarkusTestannotation since we can then access to the@Inject,@InjectMock, etc functionality. Essentially, we do what's described here https://quarkus.io/guides/getting-started-testing#injection-into-tests & here https://quarkus.io/guides/getting-started-testing#quarkus_mock, which I think is quite common.The problem is that even though we do not use HTTP based testing a web server will be started before every run which adds several seconds of delay. Starting a Unit test with
@QuarkusTesttakes ~7 seconds on my machine whereas a similar test without it takes less than a second. This adds quite a bit of friction to the developer experience.I'm guessing this is caused by the startup of the web server (but I don't know for sure). If the web server startup could be disabled but still give access to the annotations that are helpful for testing, I think that would be of great value and offer a nicer testing experience.
The text was updated successfully, but these errors were encountered: