"Unable to read..." exceptions in jenkins server log

[sxa]

We have 184 files causing approximately 200 exceptions each day which is causing a lot of activity in the log.
All of them appear to be from truncated XML files which cannot be parsed properly.

Related: #2774

Example:

2024-03-12 23:00:17.237+0000 [id=645695]        WARNING hudson.model.RunMap#retrieve: could not load /home/jenkins/.jenkins/jobs/build-scripts/jobs/openjdk22-pipeline/builds/91
java.lang.NullPointerException
Caused: com.thoughtworks.xstream.converters.ConversionException: 
---- Debugging information ----
cause-exception     : java.lang.NullPointerException
cause-message       : null
class               : org.jenkinsci.plugins.workflow.job.WorkflowRun
required-type       : org.jenkinsci.plugins.workflow.job.WorkflowRun
converter-type      : hudson.util.RobustReflectionConverter
path                : /flow-build/execution/script
line number         : 995
version             : 2.440.1
-------------------------------
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:81)
        at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:52)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:136)
        at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
        at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1464)
        at hudson.util.XStream2.unmarshal(XStream2.java:230)
        at hudson.util.XStream2.unmarshal(XStream2.java:201)
        at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1441)
        at hudson.XmlFile.unmarshal(XmlFile.java:196)
Caused: java.io.IOException: Unable to read /home/jenkins/.jenkins/jobs/build-scripts/jobs/openjdk22-pipeline/builds/91/build.xml

[sxa]

List of affected jobs - number at the start is the size in KB of each directory:
truncatedxmls.gz

I have taken a backup of those into truncated-xml-builds-issue3469.tar.gz (with the exception of the pr-tester ones which were 1-2GB each) in the jobs directory and plan to remove them from the file system by the end of the day. There should be no risk to any retained release pipelines due since none of them were a significant size so did not have build artifacts attached.

FYI @andrew-m-leonard @smlambert if there is a concern with deleting these individual build/test jobs from jenkins please shout, but it doesn't look like they had any retained artifacts.

[sxa]

Coming back to this and I've deleted all the files mentioned in the comment so these exceptions should no longer be occurring. There were 29420 of them between midnight at 1001 this morning when I perfomed the deletions and no more since. The cleanup continues :-)