Not getting all logs from build #861
Comments
|
Hello @BradyShober, Does it make sense? |
|
Hi @cyrille-leclerc, When we choose to not ship logs to Elastic, this is the log output of this pipeline: When we enable log shipping to Elastic, this is the output: If I use the link in the log outputs to Kibana, I see matching output: However, I can find the expected output by using the link on the build page and going to the Logs tab |
|
As Cyrille commented, you only see the logs that come from the Jenkins Controller. That means your build Agents do not have access to the Backend, so they can not send the logs generated by the build Agent. So check that you can reach your log backend from the Build agents, then give them the rules you need in your infrastructure to allow them to reach the Backend. |
|
@kuisathaverat I hadn't explicitly mentioned it in my previous reply, but I did verify connectivity from the agent to the backend. Also if you refer to my last point in my previous reply, with some digging I can locate the logs I expect to see in elastic, something just must not be getting set appropriately to tie them to the build to get them pulled in correctly. |
|
I see, cheking the latest screenshot, the timestamp of the record in backend is in the past compared with the timestamp of the Jenkins record. There are 2 msec of difference but only in the agent. I do not think it is a problem, I just noticed it. I wonder if it could be related to the range of time we set in the link. Your pipeline is simple and it takes only a second to run the whole pipeline. Could you try to add a |
|
@kuisathaverat I should mention that yes while my sample pipeline is very quick, that is just to illustrate the issue with a simple example and this is behavior that was observed on actual builds that do take more time. I did go ahead and add the sleep and run the test again. If I go looking with the other link on the main build page and go to the Logs tab I do see the output I also checked the attributes of the logs in elastic to see if anything stood out to me, the main differences I see that may be making a difference is that the ones that are not getting pulled in do not have span.id or trace.id attributes. |
Does this mean it does not happen in all job builds? I will try to replicate it |
|
I have tested with the demo, and the following environment works as expected: Jenkins Core 2.452.2 [LTS] Only when the agent loses connectivity with the backend, those logs are not sent. All the fields are filled in any case I did not find a way that the span.id or trace.id are not processed. |
|
Can you please test the connectivity from the Jenkins build agent to the OTLP endpoint:
io.jenkins.plugins.opentelemetry.api.util.OpenTelemetryUtils.testLogRecordExporter()
|
|
I attempted this test on an agent and I was able to see the log entry in elastic. |
Thanks, so you saw this test message emitted by the Jenkins Agent but not some of the logs of the pipeline that executes on this agent, correct? |
|
Yes, I was able to see the test message from the agent. The behavior during pipeline executions is that I am able to see logs from the controller but not from the agent in the Jenkins UI, but I am able to find the logs from the agent in elastic. |
|
Do you see the build agent logs on builds that are more than 5 minutes finished? |
|
No, the logs do not show up after any period of time |
Jenkins and plugins versions report
Environment
What Operating System are you using (both controller, and any agents involved in the problem)?
Controller is running in a container based on jenkins/jenkins:2.440.3-lts-alpine
Agents are Windows Server 2022 and Debian 11
Reproduction steps
Expected Results
We would expect to see the same logs within Jenkins as we did before
Actual Results
The logs getting pulled in have essentially no useful content. If you use the "View logs in" link in the console output, you see the same logs as it is pulling in, however I have found that if from the build you use the "View pipeline with" link that I am able to see the logs I expect to see.
An example of what is getting pulled in:
Anything else?
No response
Are you interested in contributing a fix?
No response
The text was updated successfully, but these errors were encountered: