Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] SkyWalking Java Agent performance degradation after upgrading to 9.0 #11460

Closed
2 of 3 tasks
lujiajing1126 opened this issue Oct 26, 2023 · 9 comments
Closed
2 of 3 tasks
Labels
agent Language agent related. discussion Discussions
Milestone

Comments

@lujiajing1126
Copy link
Contributor

Search before asking

  • I had searched in the issues and found no similar issues.

Apache SkyWalking Component

Java Agent (apache/skywalking-java)

What happened

After upgrading to SW 9.0, we've observed increasing application start-up in the production cluster.

For the same application, using two different Agent versions,

  • SW Agent 9.0: 2023-10-26 19:12:37.143 [main] INFO com.wosai.middleware.Application.logStarted - Started Application in 10.733 seconds (JVM running for 21.072)
  • SW Agent 8.15: 2023-10-26 19:18:12.700 [main] INFO com.wosai.middleware.Application.logStarted - Started Application in 10.538 seconds (JVM running for 16.828)

What you expected to happen

Roughly the same start-up time and CPU consumption.

How to reproduce

Run a Java application with 8 and 9.

Anything else

No response

Are you willing to submit a pull request to fix on your own?

  • Yes I am willing to submit a pull request on my own!

Code of Conduct

@lujiajing1126 lujiajing1126 added the bug Something isn't working and you are sure it's a bug! label Oct 26, 2023
@lujiajing1126 lujiajing1126 added this to the Java - 9.1.0 milestone Oct 26, 2023
@lujiajing1126 lujiajing1126 added the agent Language agent related. label Oct 26, 2023
@wu-sheng
Copy link
Member

Basically I wouldn't be surprised of this, considering what was changed for 9.0.0 kernel.
It is a tradeoff to make instrumentation classes predictable, it affects booting and weave performance.

@wu-sheng wu-sheng added discussion Discussions and removed bug Something isn't working and you are sure it's a bug! labels Oct 26, 2023
@wu-sheng
Copy link
Member

You could try the version without the change, I believe the latency will be gone.

@lujiajing1126
Copy link
Contributor Author

After some investigation, I found that

.with(AgentBuilder.DescriptionStrategy.Default.POOL_FIRST);

changing this strategy leads to almost double CPU usage during the startup. What is the motivation for the change from apache/skywalking-java#561? @kylixs

@wu-sheng
Copy link
Member

What was the value by default?

@lujiajing1126
Copy link
Contributor Author

What was the value by default?

HYBRID

@kylixs
Copy link
Member

kylixs commented Oct 27, 2023

@lujiajing1126 The POOL_FIRST solution is to solve the retransform failure problem by obtaining the original bytecode and reprocessing it.
What is the approximate number of classes that need to be enhanced in this case? Are multiple java agents mounted ( retransform or not )?

@lujiajing1126
Copy link
Contributor Author

lujiajing1126 commented Oct 27, 2023

@lujiajing1126 The POOL_FIRST solution is to solve the retransform failure problem by obtaining the original bytecode and reprocessing it.

I see.

Are multiple java agents mounted ( retransform or not )?

No. Only SkyWalking agent is mounted

What is the approximate number of classes that need to be enhanced in this case?

As shown in the logs, 79 in total.

@kylixs
Copy link
Member

kylixs commented Oct 27, 2023

@lujiajing1126 Please try this patch: apache/skywalking-java#634

@kylixs
Copy link
Member

kylixs commented Oct 28, 2023

@lujiajing1126 Please try about new solution: apache/skywalking-java#637

wu-sheng pushed a commit to apache/skywalking-java that referenced this issue Nov 3, 2023
### Improve the performance of type description of byte-buddy
The goal is to get the original class description at re-transform, so as to generate consistent results when the Skywalking agent is enhanced again (including implementing the EnhancedInstance interface, auxiliary fields and methods, etc.)

The previous type description used the `AgentBuilder.DescriptionStrategy.Default.POOL_FIRST` policy to get origin type description, which slows down the application startup, due to heavy I/O operations and parsing bytecode. 

New way is to remove dynamic fields, methods and interfaces generated by SkyWalking Agent from `TypeDescription`, and **make it as origin type descripton**.

**Key feature** :  
* No need to cache `TypeDescription` objects, less memory used.
* It only applies to the re-transform class processing flow and does not affect the startup process.

**Process flow:**
1. Find `TypeDescription` from commonly used type cache, such as primitive class.
2. Delegate to `AgentBuilder.DescriptionStrategy.Default.HYBRID`
3. Wrap `TypeDescription` by `SWTypeDescriptionWrapper` , remove fields, methods, interface generated by SkyWalking. 

**Relative Issue:**  apache/skywalking#11460
@wu-sheng wu-sheng closed this as completed Nov 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
agent Language agent related. discussion Discussions
Projects
None yet
Development

No branches or pull requests

3 participants