junit hangs at `Recording test results`

[matt-undo]

Jenkins and plugins versions report

Environment

Jenkins: 2.470
[Pipeline] echo ([hide](http://jenkins.ci.staging.eu-west-2.undoers.io/view/all/job/junit-test/65/console#))
OS: Linux - 6.1.127-135.201.amzn2023.x86_64
[Pipeline] echo
Java: 17.0.12 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
ace-editor:1.1
analysis-core:1.96
analysis-model-api:12.3.3
ansicolor:1.0.4
ant:497.v94e7d9fffa_b_9
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
apache-httpcomponents-client-5-api:5.4-124.v31e2987e48f4
artifact-manager-s3:871.v72f7f642a_245
asm-api:9.7.1-97.v4cc844130d97
authentication-tokens:1.113.v81215a_241826
aws-credentials:231.v08a_59f17d742
aws-global-configuration:130.v35b_7b_96f53c3
aws-java-sdk:1.12.730-457.v3403b_37d2170
aws-java-sdk-api-gateway:1.12.772-474.v7f79a_2046a_fb_
aws-java-sdk-autoscaling:1.12.772-474.v7f79a_2046a_fb_
aws-java-sdk-cloudformation:1.12.772-474.v7f79a_2046a_fb_
aws-java-sdk-cloudfront:1.12.772-474.v7f79a_2046a_fb_
aws-java-sdk-codebuild:1.12.772-474.v7f79a_2046a_fb_
aws-java-sdk-codedeploy:1.12.772-474.v7f79a_2046a_fb_
aws-java-sdk-ec2:1.12.772-474.v7f79a_2046a_fb_
aws-java-sdk-ecr:1.12.772-474.v7f79a_2046a_fb_
aws-java-sdk-ecs:1.12.772-474.v7f79a_2046a_fb_
aws-java-sdk-efs:1.12.772-474.v7f79a_2046a_fb_
aws-java-sdk-elasticbeanstalk:1.12.772-474.v7f79a_2046a_fb_
aws-java-sdk-elasticloadbalancingv2:1.12.772-474.v7f79a_2046a_fb_
aws-java-sdk-iam:1.12.772-474.v7f79a_2046a_fb_
aws-java-sdk-kinesis:1.12.772-474.v7f79a_2046a_fb_
aws-java-sdk-lambda:1.12.772-474.v7f79a_2046a_fb_
aws-java-sdk-logs:1.12.772-474.v7f79a_2046a_fb_
aws-java-sdk-minimal:1.12.772-474.v7f79a_2046a_fb_
aws-java-sdk-organizations:1.12.772-474.v7f79a_2046a_fb_
aws-java-sdk-secretsmanager:1.12.772-474.v7f79a_2046a_fb_
aws-java-sdk-sns:1.12.772-474.v7f79a_2046a_fb_
aws-java-sdk-sqs:1.12.772-474.v7f79a_2046a_fb_
aws-java-sdk-ssm:1.12.772-474.v7f79a_2046a_fb_
aws-secrets-manager-secret-source:1.72.v61781b_35c542
badge:1.13
basic-branch-build-strategies:81.v05e333931c7d
blueocean:1.27.13
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.13
blueocean-commons:1.27.13
blueocean-config:1.27.13
blueocean-core-js:1.27.13
blueocean-dashboard:1.27.13
blueocean-display-url:2.4.2
blueocean-events:1.27.13
blueocean-executor-info:1.27.13
blueocean-github-pipeline:1.27.13
blueocean-git-pipeline:1.27.13
blueocean-i18n:1.27.13
blueocean-jira:1.27.13
blueocean-jwt:1.27.13
blueocean-personalization:1.27.13
blueocean-pipeline-api-impl:1.27.13
blueocean-pipeline-editor:1.27.13
blueocean-pipeline-scm-api:1.27.13
blueocean-rest:1.27.13
blueocean-rest-impl:1.27.13
blueocean-web:1.27.13
bootstrap4-api:4.6.0-6
bootstrap5-api:5.3.3-1
bouncycastle-api:2.30.1.78.1-248.ve27176eb_46cb_
branch-api:2.1169.va_f810c56e895
build-monitor-plugin:1.14-883.vf620a_44eb_ec1
build-pipeline-plugin:2.0.2
build-timeout:1.32
build-timestamp:1.0.3
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.2.0
cloudbees-bitbucket-branch-source:886.v44cf5e4ecec5
cloudbees-folder:6.942.vb_43318a_156b_2
cloverphp:0.6
cobertura:1.17
code-coverage-api:4.99.0
command-launcher:107.v773860566e2e
commons-httpclient3-api:3.1-3
commons-lang3-api:3.17.0-84.vb_b_938040b_078
commons-text-api:1.12.0-129.v99a_50df237f7
compress-buildlog:1.2
conditional-buildstep:1.4.3
configuration-as-code:1836.vccda_4a_122a_a_e
copyartifact:746.vd2a_674fb_4f6f
coverage:1.16.1
credentials:1337.v60b_d7b_c7b_c9f
credentials-binding:677.vdc9d38cb_254d
dashboard-view:2.508.va_74654f026d1
data-tables-api:2.0.8-1
depgraph-view:1.0.5
display-url-api:2.204.vf6fddd8a_8b_e9
docker-commons:439.va_3cb_0a_6a_fb_29
docker-workflow:580.vc0c340686b_54
downstream-build-cache:1.7
dtkit-api:3.0.2
durable-task:555.v6802fe0f0b_82
ec2:1688.v8c07e01d657f
echarts-api:5.5.0-1
eddsa-api:0.3.0-4.v84c6f0f4969e
email-ext:1814.v404722f34263
embeddable-build-status:487.va_0ef04c898a_2
envinject:2.908.v66a_774b_31d93
envinject-api:1.199.v3ce31253ed13
external-monitor-job:215.v2e88e894db_f8
favorite:2.208.v91d65b_7792a_c
font-awesome-api:6.5.2-1
forensics-api:2.4.0
git:5.2.2
git-client:5.0.0
github:1.39.0
github-api:1.318-461.v7a_c09c9fa_d63
github-branch-source:1789.v5b_0c0cea_18c3
github-checks:554.vb_ee03a_000f65
github-scm-trait-notification-context:40.vfa_7f31a_b_d7f8
git-parameter:0.9.19
git-server:126.v0d945d8d2b_39
global-slack-notifier:1.5
golang:1.4
gradle:2.12
greenballs:1.15.1
gson-api:2.11.0-85.v1f4e87273c33
handlebars:3.0.8
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
htmlpublisher:1.34
http_request:1.18
instance-identity:201.vd2a_b_5a_468a_a_6
ionicons-api:74.v93d5eb_813d5f
jackson2-api:2.17.0-379.v02de8ec9f64c
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javadoc:243.vb_b_503b_b_45537
javax-activation-api:1.2.0-7
javax-mail-api:1.6.2-10
jaxb:2.3.9-1
jdk-tool:73.vddf737284550
jenkins-design-language:1.27.13
jersey2-api:2.44-151.v6df377fff741
jira:3.13
jjwt-api:0.11.5-112.ve82dfb_224b_a_d
jobConfigHistory:1229.v3039470161a_d
joda-time-api:2.13.0-93.v9934da_29b_a_e9
jquery:1.12.4-1
jquery3-api:3.7.1-2
jquery-detached:1.2.1
jsch:0.2.16-86.v42e010d9484b_
json-api:20240303-101.v7a_8666713110
json-path-api:2.9.0-118.v7f23ed82a_8b_8
junit:1312.v1a_235a_b_94a_31
junit-attachments:205.vc0677977deb_0
ldap:725.v3cb_b_711b_1a_ef
lockable-resources:1255.vf48745da_35d0
log-parser:2.3.4
mailer:472.vf7c289a_4b_420
matrix-auth:3.2.2
matrix-project:832.va_66e270d2946
maven-plugin:3.23
mercurial:1260.vdfb_723cdcc81
mina-sshd-api-common:2.14.0-133.vcc091215a_358
mina-sshd-api-core:2.14.0-133.vcc091215a_358
momentjs:1.1.1
monitoring:1.99.0
nexus-artifact-uploader:2.14
node-iterator-api:55.v3b_77d4032326
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
pam-auth:1.11
parameterized-trigger:806.vf6fff3e28c3e
pipeline-aws:1.45
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-githubnotify-step:49.vf37bf92d2bc8
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-groovy-lib:744.v5b_556ee7c253
pipeline-input-step:495.ve9c153f6067b_
pipeline-milestone-step:119.vdfdc43fc3b_9a_
pipeline-model-api:2.2198.v41dd8ef6dd56
pipeline-model-declarative-agent:1.1.1
pipeline-model-definition:2.2198.v41dd8ef6dd56
pipeline-model-extensions:2.2198.v41dd8ef6dd56
pipeline-rest-api:2.34
pipeline-stage-step:312.v8cd10304c27a_
pipeline-stage-tags-metadata:2.2198.v41dd8ef6dd56
pipeline-stage-view:2.34
pipeline-utility-steps:2.16.2
plain-credentials:182.v468b_97b_9dcb_8
plot:2.1.12
plugin-util-api:4.1.0
popper-api:1.16.1-3
prism-api:1.29.0-18
pubsub-light:1.18
pyenv-pipeline:2.1.2
rebuild:332.va_1ee476d8f6d
resource-disposer:0.23
run-condition:1.7
scm-api:690.vfc8b_54395023
script-security:1341.va_2819b_414686
simple-theme-plugin:176.v39740c03a_a_f5
skip-certificate-check:1.1
slack:722.vd07f1ea_7ff40
snakeyaml-api:2.2-111.vc6598e30cc65
sse-gateway:1.27
ssh-credentials:337.v395d2403ccd4
sshd:3.330.vc866a_8389b_58
ssh-slaves:2.968.v6f8823c91de4
structs:337.v1b_04ea_4df7c8
test-results-analyzer:0.4.1
timestamper:1.27
token-macro:400.v35420b_922dcb_
trilead-api:2.147.vb_73cc728a_32e
valgrind:0.29-SNAPSHOT (private-e986e36f-danielp)
variant:60.v7290fc0eb_b_cd
view-job-filters:382.vdf2d5e3f02f0
vncrecorder:1.35
vncviewer:1.9
warnings:5.0.1
warnings-ng:11.3.0
windows-slaves:1.8.1
workflow-aggregator:596.v8c21c963d92d
workflow-api:1316.v33eb_726c50b_a_
workflow-basic-steps:1058.vcb_fc1e3a_21a_9
workflow-cps:3903.v48a_8836749e9
workflow-cps-global-lib:612.v55f2f80781ef
workflow-durable-task-step:1353.v1891a_b_01da_18
workflow-job:1426.v2ecb_a_a_42fd46
workflow-multibranch:791.v28fb_f74dfca_e
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:657.v03b_e8115821b_
workflow-support:907.v6713a_ed8a_573
ws-cleanup:0.46
xunit:3.1.4
xvnc:1.28
yet-another-build-visualizer:1.16

What Operating System are you using (both controller, and any agents involved in the problem)?

Controller Amazon linux 6 and agents are Rocky Linux 8.10

Reproduction steps

1. rsync previous creation of XML results to a new agent node. (these are normally produced as part of a job)
2. copy these files into the job workspace
3. run junit on those files (with no additional arguments)

Jenkins pipeline section contains simply:

stage('JUnit Analysis') {
  junit testResults: "file.xml", keepLongStdio: false
}

Expected Results

Junit processes the XML files

Actual Results

junit hangs until it times out

[Pipeline] junit
Recording test results
Cancelling nested steps due to timeout
null

The agent connection dies as if completely hung although I can still ssh onto it and can see it's not loaded. We provision a single agent per job so there's nothing else happening on the node.

Anything else?

The XML file is 32 Mb in size. If I split it into smaller chunks of XML and loop over each one (500k-1.5mb) it will process 23-26 files, even if I change the order or remove files that were previously stuck, before hanging again.

Note that this is running on a new Jenkins environment and the XML processing is working on an older installation. I am migrating the pipeline to newer installation.

Thread dump of the running agent during the hang
thread.txt

I wondered if it is some kind of resource exhaustion on the agent? I tried increasing memory (-Xmx) in the agent launch command but it didn't help. Have considered moving to the webhook method.

I am checking with our security team on whether we can share the XML contents so might add that later if it's needed.

Are you interested in contributing a fix?

No response

[timja]

If you can provide either the xml files or a sanitised version of them that reproduces the issue that would be needed for this

[matt-undo]

@timja thanks for responding! Here's an example XML with bits redacted.
aws-rocky-810-x64-GmZkSP-pm-default-0.xml.gz

[matt-undo]

@timja did you have any thoughts on this issue? Many thanks for your time.

[timja]

I haven't had time to look at it.

[matt-undo]

I've updated Jenkins and plugins to see if this helps.

Jenkins: 2.492.3

plugins.txt

I also removed a bunch of deprecated plugins to see if this would help.

ace-editor:1.1
analysis-core:1.96
bootstrap4-api:4.6.0-6
cloverphp:0.6
cobertura:1.17
code-coverage-api:4.99.0
greenballs:1.15.1
handlebars:3.0.8
jquery:1.12.4-1
jquery-detached:1.2.1
momentjs:1.1.1
pipeline-model-declarative-agent:1.1.1
popper-api:1.16.1-3
warnings:5.0.1
windows-slaves:1.8.1
workflow-cps-global-lib:612.v55f2f80781ef

These were added
pipeline-groovy-lib:752.vdddedf804e72
pipeline-model-definitionpipeline-groovy-lib:752.vdddedf804e72

I've also managed to get it working with the following groovy that steps over a smaller number of files at a time with a gc in between

        files.toList().collate(5).eachWithIndex { group, batchNum ->
            stage("JUnit Batch ${batchNum + 1}") {
                try {
                    def groupPaths = group.collect { it.path }.join(',')
                    echo "Processing files: ${groupPaths}"
                    
                    junit testResults: groupPaths, keepLongStdio: true
                    
                    echo "Finished batch ${batchNum + 1}"
                } catch (err) {
                    echo "Failed batch ${batchNum + 1}: ${err}"
                }                
                System.gc()  // Optional: still try to reclaim aggressively
            }
        }

I would like to avoid doing this if possible as I'm not really sure why this is happening and might present elsewhere in the system. Perhaps you would know better than I what might be happening.

[timja]

Not off the top of my head no maybe others would

[matt-undo]

I managed to locate the problem. It was due to the fact that we had timestamps with incorrect format. Each time the timestamp was parsed a warning message was sent to the log. We counted over 4000 log events in one run. When we set the correct format the junit processing finished and didn't kill the agent.

The relevant bit of code is https://github.com/jenkinsci/junit-plugin/blob/master/src/main/java/hudson/tasks/junit/SuiteResult.java#L740-L754

The value for timestamp was set to 2025-04-10 09:15:47.210606Z. Looking at the parse function the format should be in ISO format, with a T separator and seconds precision (not milliseconds as we had) something like 2025-04-10T09:15:47.

It's worth noting that we didn't see the same problem with Websocket based agents. Just remoting agents where the agent is connected over SSH.