;; This buffer is for text that is not saved, and for Lisp evaluation. ;; To create a file, visit it with and enter text in its buffer. [ant:junit4] JVM J6: stderr was not empty, see: /var/lib/jenkins/workspace/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA/java11/ES_RUNTIME_JAVA/java11/nodes/immutable&&linux&&docker/server/build/testrun/unitTest/temp/junit4-J6-20190227_041624_4122999029555223831120.syserr HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:19:04, stalled for 40.1s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:19:14, stalled for 50.1s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:19:24, stalled for 60.1s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:19:34, stalled for 70.1s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:19:44, stalled for 80.2s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:19:54, stalled for 90.2s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:20:04, stalled for 100s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:20:14, stalled for 110s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:20:24, stalled for 120s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:20:34, stalled for 130s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:20:44, stalled for 140s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:20:54, stalled for 150s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:21:04, stalled for 160s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:21:14, stalled for 170s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:21:24, stalled for 180s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:21:34, stalled for 190s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:21:44, stalled for 200s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:21:54, stalled for 210s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:22:04, stalled for 220s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:22:14, stalled for 230s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:22:24, stalled for 240s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:22:34, stalled for 250s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:22:44, stalled for 260s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:22:54, stalled for 270s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:23:04, stalled for 280s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:23:14, stalled for 290s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:23:24, stalled for 300s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:23:34, stalled for 310s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:23:44, stalled for 320s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:23:54, stalled for 330s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:24:04, stalled for 340s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:24:14, stalled for 350s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:24:24, stalled for 360s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:24:34, stalled for 370s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:24:44, stalled for 380s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:24:54, stalled for 390s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:25:04, stalled for 400s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:25:14, stalled for 410s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:25:24, stalled for 420s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:25:34, stalled for 430s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:25:44, stalled for 440s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:25:54, stalled for 450s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:26:04, stalled for 460s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:26:14, stalled for 470s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:26:24, stalled for 480s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:26:34, stalled for 490s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:26:44, stalled for 500s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:26:54, stalled for 510s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:27:04, stalled for 520s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:27:14, stalled for 530s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:27:24, stalled for 540s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:27:34, stalled for 550s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:27:44, stalled for 560s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:27:54, stalled for 570s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:28:04, stalled for 580s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:28:14, stalled for 590s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:28:24, stalled for 600s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:28:34, stalled for 610s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:28:44, stalled for 620s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:28:54, stalled for 630s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:29:04, stalled for 640s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:29:14, stalled for 650s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:29:24, stalled for 660s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:29:34, stalled for 670s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:29:44, stalled for 680s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:29:54, stalled for 690s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:30:04, stalled for 700s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:30:14, stalled for 710s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:30:24, stalled for 720s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:30:34, stalled for 730s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:30:44, stalled for 740s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:30:54, stalled for 750s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:31:04, stalled for 760s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:31:14, stalled for 770s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:31:24, stalled for 780s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:31:34, stalled for 790s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:31:44, stalled for 800s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:31:54, stalled for 810s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:32:04, stalled for 820s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:32:14, stalled for 830s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:32:24, stalled for 840s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:32:34, stalled for 850s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:32:44, stalled for 860s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:32:54, stalled for 870s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:33:04, stalled for 880s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:33:14, stalled for 890s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:33:24, stalled for 900s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:33:34, stalled for 910s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:33:44, stalled for 920s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:33:54, stalled for 930s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:34:04, stalled for 940s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:34:14, stalled for 950s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:34:24, stalled for 960s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:34:34, stalled for 970s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:34:44, stalled for 980s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:34:54, stalled for 990s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:35:04, stalled for 1000s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:35:14, stalled for 1010s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:35:24, stalled for 1020s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:35:34, stalled for 1030s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:35:44, stalled for 1040s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:35:54, stalled for 1050s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:36:04, stalled for 1060s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:36:14, stalled for 1070s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:36:24, stalled for 1080s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:36:34, stalled for 1090s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:36:44, stalled for 1100s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:36:54, stalled for 1110s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:37:04, stalled for 1120s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:37:14, stalled for 1130s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:37:24, stalled for 1140s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:37:34, stalled for 1150s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:37:44, stalled for 1160s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:37:54, stalled for 1170s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:38:04, stalled for 1180s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo HEARTBEAT J5 PID(25454@elasticsearch-ci-immutable-centos-7-1551240015924377557): 2019-02-27T04:38:14, stalled for 1190s at: RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo Suite: org.elasticsearch.index.replication.RecoveryDuringReplicationTests 1> [2019-02-27T00:18:23,898][INFO ][o.e.i.r.RecoveryDuringReplicationTests] [testResyncAfterPrimaryPromotion] before test 1> [2019-02-27T00:18:23,919][DEBUG][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] state: [CREATED] 1> [2019-02-27T00:18:23,919][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] [test][0] writing shard state, reason [initial state with allocation id [[id=PAHYAIDaQg-R2puw8edO6Q]]] 1> [2019-02-27T00:18:23,924][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] [test][0] skip writing shard state, has been written before 2> feb. 27, 2019 12:38:23 A.M. com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate 2> WARNING: Suite execution timed out: org.elasticsearch.index.replication.RecoveryDuringReplicationTests 1> [2019-02-27T00:18:23,933][DEBUG][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] state: [CREATED] 2> ==== jstack at approximately timeout time ==== 2> "main" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$2@25186a9f 1> [2019-02-27T00:18:23,933][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] [test][0] writing shard state, reason [initial state with allocation id [[id=d1o89n0eSO2mc0D7BIG-tw]]] 1> [2019-02-27T00:18:23,937][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] [test][0] skip writing shard state, has been written before 2> at java.base@11.0.2/java.lang.Object.wait(Native Method) 1> [2019-02-27T00:18:23,953][DEBUG][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] state: [CREATED] 2> - waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$2@25186a9f 1> [2019-02-27T00:18:23,954][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] [test][0] writing shard state, reason [initial state with allocation id [[id=kIctYWftSzOl22DvzzwCSQ]]] 1> [2019-02-27T00:18:23,957][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] [test][0] skip writing shard state, has been written before 1> [2019-02-27T00:18:23,957][DEBUG][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] state: [CREATED]->[RECOVERING], reason [store] 1> [2019-02-27T00:18:23,958][DEBUG][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] starting recovery from store ... 1> [2019-02-27T00:18:24,034][DEBUG][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] state: [RECOVERING]->[POST_RECOVERY], reason [post recovery from shard_store] 2> at java.base@11.0.2/java.lang.Thread.join(Thread.java:1305) 1> [2019-02-27T00:18:24,035][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] recovery completed from [shard_store], took [76ms] 1> index : files [0] with total_size [0b], took[15ms] 1> : recovered_files [0] with total_size [0b] 1> : reusing_files [0] with total_size [0b] 2> at java.base@11.0.2/java.lang.Thread.join(Thread.java:1379) 1> verify_index : took [0s], check_index [0s] 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:639) 1> translog : number_of_operations [0], took [61ms] 1> [2019-02-27T00:18:24,036][DEBUG][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] state: [POST_RECOVERY]->[STARTED], reason [global state is [STARTED]] 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:496) 1> [2019-02-27T00:18:24,036][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] [test][0] skip writing shard state, has been written before 1> [2019-02-27T00:18:24,036][DEBUG][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] state: [CREATED]->[RECOVERING], reason [remote] 2> at app//com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:269) 2> at app//com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:394) 1> [2019-02-27T00:18:24,049][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] [test][0] skip writing shard state, has been written before 2> at app//com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13) 1> [2019-02-27T00:18:24,137][DEBUG][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] state: [RECOVERING]->[POST_RECOVERY], reason [peer recovery done] 2> "Reference Handler" ID=2 RUNNABLE 2> at java.base@11.0.2/java.lang.ref.Reference.waitForReferencePendingList(Native Method) 1> [2019-02-27T00:18:24,137][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] [test][0] skip writing shard state, has been written before 2> at java.base@11.0.2/java.lang.ref.Reference.processPendingReferences(Reference.java:241) 1> [2019-02-27T00:18:24,138][DEBUG][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] state: [POST_RECOVERY]->[STARTED], reason [global state is [STARTED]] 1> [2019-02-27T00:18:24,138][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] [test][0] skip writing shard state, has been written before 1> [2019-02-27T00:18:24,139][DEBUG][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] state: [CREATED]->[RECOVERING], reason [remote] 2> at java.base@11.0.2/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213) 1> [2019-02-27T00:18:24,141][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] [test][0] skip writing shard state, has been written before 2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@558bc6fc 1> [2019-02-27T00:18:24,188][DEBUG][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] state: [RECOVERING]->[POST_RECOVERY], reason [peer recovery done] 1> [2019-02-27T00:18:24,188][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] [test][0] skip writing shard state, has been written before 1> [2019-02-27T00:18:24,188][DEBUG][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] state: [POST_RECOVERY]->[STARTED], reason [global state is [STARTED]] 1> [2019-02-27T00:18:24,188][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] [test][0] skip writing shard state, has been written before 2> at java.base@11.0.2/java.lang.Object.wait(Native Method) 1> [2019-02-27T00:18:24,193][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][initial_doc_0] seq# [-2] allocation-id [id=PAHYAIDaQg-R2puw8edO6Q] 2> - waiting on java.lang.ref.ReferenceQueue$Lock@558bc6fc 2> at java.base@11.0.2/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155) 1> [2019-02-27T00:18:24,202][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][initial_doc_0] seq# [0] allocation-id [id=d1o89n0eSO2mc0D7BIG-tw] 1> [2019-02-27T00:18:24,220][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][initial_doc_0] seq# [0] allocation-id [id=kIctYWftSzOl22DvzzwCSQ] 1> [2019-02-27T00:18:24,230][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][initial_doc_1] seq# [-2] allocation-id [id=PAHYAIDaQg-R2puw8edO6Q] 2> at java.base@11.0.2/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176) 1> [2019-02-27T00:18:24,232][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][initial_doc_1] seq# [1] allocation-id [id=d1o89n0eSO2mc0D7BIG-tw] 2> at java.base@11.0.2/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:170) 1> [2019-02-27T00:18:24,234][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][initial_doc_1] seq# [1] allocation-id [id=kIctYWftSzOl22DvzzwCSQ] 2> "Signal Dispatcher" ID=4 RUNNABLE 1> [2019-02-27T00:18:24,235][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][initial_doc_2] seq# [-2] allocation-id [id=PAHYAIDaQg-R2puw8edO6Q] 2> "Common-Cleaner" ID=19 TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@4be5e673 2> at java.base@11.0.2/java.lang.Object.wait(Native Method) 1> [2019-02-27T00:18:24,237][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][initial_doc_2] seq# [2] allocation-id [id=d1o89n0eSO2mc0D7BIG-tw] 2> - timed waiting on java.lang.ref.ReferenceQueue$Lock@4be5e673 1> [2019-02-27T00:18:24,238][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][initial_doc_2] seq# [2] allocation-id [id=kIctYWftSzOl22DvzzwCSQ] 2> at java.base@11.0.2/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155) 1> [2019-02-27T00:18:24,239][INFO ][o.e.i.r.RecoveryDuringReplicationTests] [testResyncAfterPrimaryPromotion] --> indexing 4 extra docs 1> [2019-02-27T00:18:24,240][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][extra_doc_0] seq# [-2] allocation-id [id=PAHYAIDaQg-R2puw8edO6Q] 1> [2019-02-27T00:18:24,241][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][extra_doc_0] seq# [3] allocation-id [id=d1o89n0eSO2mc0D7BIG-tw] 1> [2019-02-27T00:18:24,242][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][extra_doc_1] seq# [-2] allocation-id [id=PAHYAIDaQg-R2puw8edO6Q] 1> [2019-02-27T00:18:24,242][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][extra_doc_1] seq# [4] allocation-id [id=d1o89n0eSO2mc0D7BIG-tw] 1> [2019-02-27T00:18:24,243][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][extra_doc_2] seq# [-2] allocation-id [id=PAHYAIDaQg-R2puw8edO6Q] 1> [2019-02-27T00:18:24,244][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][extra_doc_2] seq# [5] allocation-id [id=d1o89n0eSO2mc0D7BIG-tw] 1> [2019-02-27T00:18:24,245][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][extra_doc_3] seq# [-2] allocation-id [id=PAHYAIDaQg-R2puw8edO6Q] 1> [2019-02-27T00:18:24,245][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][extra_doc_3] seq# [6] allocation-id [id=d1o89n0eSO2mc0D7BIG-tw] 1> [2019-02-27T00:18:24,246][INFO ][o.e.i.r.RecoveryDuringReplicationTests] [testResyncAfterPrimaryPromotion] --> indexing 8 extra docs to be trimmed 1> [2019-02-27T00:18:24,246][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][extra_trimmed_0] seq# [-2] allocation-id [id=PAHYAIDaQg-R2puw8edO6Q] 1> [2019-02-27T00:18:24,247][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][extra_trimmed_0] seq# [7] allocation-id [id=kIctYWftSzOl22DvzzwCSQ] 1> [2019-02-27T00:18:24,248][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][extra_trimmed_1] seq# [-2] allocation-id [id=PAHYAIDaQg-R2puw8edO6Q] 1> [2019-02-27T00:18:24,248][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][extra_trimmed_1] seq# [8] allocation-id [id=kIctYWftSzOl22DvzzwCSQ] 1> [2019-02-27T00:18:24,249][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][extra_trimmed_2] seq# [-2] allocation-id [id=PAHYAIDaQg-R2puw8edO6Q] 1> [2019-02-27T00:18:24,250][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][extra_trimmed_2] seq# [9] allocation-id [id=kIctYWftSzOl22DvzzwCSQ] 1> [2019-02-27T00:18:24,250][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][extra_trimmed_3] seq# [-2] allocation-id [id=PAHYAIDaQg-R2puw8edO6Q] 1> [2019-02-27T00:18:24,251][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][extra_trimmed_3] seq# [10] allocation-id [id=kIctYWftSzOl22DvzzwCSQ] 1> [2019-02-27T00:18:24,252][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][extra_trimmed_4] seq# [-2] allocation-id [id=PAHYAIDaQg-R2puw8edO6Q] 1> [2019-02-27T00:18:24,253][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][extra_trimmed_4] seq# [11] allocation-id [id=kIctYWftSzOl22DvzzwCSQ] 1> [2019-02-27T00:18:24,253][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][extra_trimmed_5] seq# [-2] allocation-id [id=PAHYAIDaQg-R2puw8edO6Q] 1> [2019-02-27T00:18:24,254][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][extra_trimmed_5] seq# [12] allocation-id [id=kIctYWftSzOl22DvzzwCSQ] 1> [2019-02-27T00:18:24,255][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][extra_trimmed_6] seq# [-2] allocation-id [id=PAHYAIDaQg-R2puw8edO6Q] 1> [2019-02-27T00:18:24,256][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][extra_trimmed_6] seq# [13] allocation-id [id=kIctYWftSzOl22DvzzwCSQ] 1> [2019-02-27T00:18:24,256][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][extra_trimmed_7] seq# [-2] allocation-id [id=PAHYAIDaQg-R2puw8edO6Q] 2> at java.base@11.0.2/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:148) 1> [2019-02-27T00:18:24,257][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] index [type][extra_trimmed_7] seq# [14] allocation-id [id=kIctYWftSzOl22DvzzwCSQ] 2> at java.base@11.0.2/java.lang.Thread.run(Thread.java:834) 2> at java.base@11.0.2/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:134) 2> "JUnit4-serializer-daemon" ID=20 TIMED_WAITING 1> [2019-02-27T00:18:24,258][INFO ][o.e.i.r.RecoveryDuringReplicationTests] [testResyncAfterPrimaryPromotion] --> resyncing replicas seqno_stats primary SeqNoStats{maxSeqNo=14, localCheckpoint=14, globalCheckpoint=2} replica SeqNoStats{maxSeqNo=6, localCheckpoint=6, globalCheckpoint=2} 2> at java.base@11.0.2/java.lang.Thread.sleep(Native Method) 1> [2019-02-27T00:18:24,279][DEBUG][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] state: [STARTED]->[CLOSED], reason [test] 2> at app//com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:50) 2> "SUITE-RecoveryDuringReplicationTests-seed#[F8E8B6147433686D]" ID=920 RUNNABLE 2> at java.management@11.0.2/sun.management.ThreadImpl.dumpThreads0(Native Method) 2> at java.management@11.0.2/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:502) 1> [2019-02-27T00:18:24,284][TRACE][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] [test][0] writing shard state, reason [routing changed from [test][0], node[s1], [R], s[STARTED], a[id=d1o89n0eSO2mc0D7BIG-tw] to [test][0], node[s1], [P], s[STARTED], a[id=d1o89n0eSO2mc0D7BIG-tw]] 2> at java.management@11.0.2/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:490) 2> at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:681) 1> [2019-02-27T00:18:24,316][TRACE][o.e.i.s.PrimaryReplicaSyncer] [org.elasticsearch.index.replication.RecoveryDuringReplicationTests] [test][0] sending batch of [4][232b] (total sent: [4], skipped: [3]) 2> at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:65) 1> [2019-02-27T00:18:24,329][INFO ][o.e.i.s.IndexShard ] [org.elasticsearch.index.replication.RecoveryDuringReplicationTests] [test][0] detected new primary with primary term [24], global checkpoint [2], max_seq_no [14] 1> [2019-02-27T00:18:24,330][TRACE][o.e.i.s.IndexShard ] [org.elasticsearch.index.replication.RecoveryDuringReplicationTests] [test][0] flush with FlushRequest{waitIfOngoing=true, force=false} 1> [2019-02-27T00:18:24,429][TRACE][o.e.i.s.IndexShard ] [org.elasticsearch.index.replication.RecoveryDuringReplicationTests] [test][0] [translog] recover op Index{id='initial_doc_0', type='type', seqNo=0, primaryTerm=23, version=1, autoGeneratedIdTimestamp=-1} 1> [2019-02-27T00:18:24,429][TRACE][o.e.i.s.IndexShard ] [org.elasticsearch.index.replication.RecoveryDuringReplicationTests] [test][0] index [type][initial_doc_0] seq# [0] allocation-id [id=kIctYWftSzOl22DvzzwCSQ] 2> at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:415) 2> - locked java.lang.Object@40a9feb 1> [2019-02-27T00:18:24,431][TRACE][o.e.i.s.IndexShard ] [org.elasticsearch.index.replication.RecoveryDuringReplicationTests] [test][0] [translog] recover op Index{id='initial_doc_1', type='type', seqNo=1, primaryTerm=23, version=1, autoGeneratedIdTimestamp=-1} 1> [2019-02-27T00:18:24,431][TRACE][o.e.i.s.IndexShard ] [org.elasticsearch.index.replication.RecoveryDuringReplicationTests] [test][0] index [type][initial_doc_1] seq# [1] allocation-id [id=kIctYWftSzOl22DvzzwCSQ] 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:708) 1> [2019-02-27T00:18:24,431][TRACE][o.e.i.s.IndexShard ] [org.elasticsearch.index.replication.RecoveryDuringReplicationTests] [test][0] [translog] recover op Index{id='initial_doc_2', type='type', seqNo=2, primaryTerm=23, version=1, autoGeneratedIdTimestamp=-1} 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138) 1> [2019-02-27T00:18:24,432][TRACE][o.e.i.s.IndexShard ] [org.elasticsearch.index.replication.RecoveryDuringReplicationTests] [test][0] index [type][initial_doc_2] seq# [2] allocation-id [id=kIctYWftSzOl22DvzzwCSQ] 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:629) 1> [2019-02-27T00:18:24,479][TRACE][o.e.i.s.IndexShard ] [org.elasticsearch.index.replication.RecoveryDuringReplicationTests] [test][0] index [type][extra_doc_0] seq# [3] allocation-id [id=kIctYWftSzOl22DvzzwCSQ] 2> "TEST-RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo-seed#[F8E8B6147433686D]" ID=921 WAITING on java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@44e748 1> [2019-02-27T00:18:24,483][TRACE][o.e.i.s.IndexShard ] [org.elasticsearch.index.replication.RecoveryDuringReplicationTests] [test][0] index [type][extra_doc_1] seq# [4] allocation-id [id=kIctYWftSzOl22DvzzwCSQ] 2> at java.base@11.0.2/jdk.internal.misc.Unsafe.park(Native Method) 1> [2019-02-27T00:18:24,483][TRACE][o.e.i.s.IndexShard ] [org.elasticsearch.index.replication.RecoveryDuringReplicationTests] [test][0] index [type][extra_doc_2] seq# [5] allocation-id [id=kIctYWftSzOl22DvzzwCSQ] 2> - waiting on java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@44e748 2> at java.base@11.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) 1> [2019-02-27T00:18:24,483][TRACE][o.e.i.s.IndexShard ] [org.elasticsearch.index.replication.RecoveryDuringReplicationTests] [test][0] index [type][extra_doc_3] seq# [6] allocation-id [id=kIctYWftSzOl22DvzzwCSQ] 2> at java.base@11.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885) 1> [2019-02-27T00:18:24,485][TRACE][o.e.i.s.PrimaryReplicaSyncer] [org.elasticsearch.index.replication.RecoveryDuringReplicationTests] [test][0] resync completed (total sent: [4], skipped: [3]) 1> [2019-02-27T00:18:24,485][INFO ][o.e.i.s.IndexShard ] [org.elasticsearch.index.replication.RecoveryDuringReplicationTests] [test][0] primary-replica resync completed with 4 operations 1> [2019-02-27T00:18:24,530][DEBUG][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] state: [STARTED]->[CLOSED], reason [test] 2> at java.base@11.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:917) 1> [2019-02-27T00:18:24,547][DEBUG][o.e.i.s.IndexShard ] [testResyncAfterPrimaryPromotion] [test][0] state: [STARTED]->[CLOSED], reason [test] 2> at java.base@11.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1240) 2> at java.base@11.0.2/java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:959) 1> [2019-02-27T00:18:24,551][INFO ][o.e.i.r.RecoveryDuringReplicationTests] [testResyncAfterPrimaryPromotion] after test 2> at app//org.elasticsearch.common.util.concurrent.ReleasableLock.acquire(ReleasableLock.java:55) 2> at app//org.elasticsearch.index.engine.Engine.close(Engine.java:1730) 1> [2019-02-27T00:18:24,561][INFO ][o.e.i.r.RecoveryDuringReplicationTests] [testDoNotWaitForPendingSeqNo] before test 2> at app//org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ERROR 1199s J5 | RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo <<< FAILURES! > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached. 2> at app//org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) > at __randomizedtesting.SeedInfo.seed([F8E8B6147433686D]:0) 2> at app//org.elasticsearch.index.shard.IndexShard.close(IndexShard.java:1248) 1> [2019-02-27T00:38:29,001][WARN ][o.e.i.e.Engine ] [testDoNotWaitForPendingSeqNo] [test][0] failed to rollback writer on close 1> java.nio.channels.ClosedByInterruptException: null 2> - locked java.lang.Object@34d361ef 2> at app//org.elasticsearch.index.shard.IndexShardTestCase.lambda$closeShard$8(IndexShardTestCase.java:521) 1> at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:199) ~[?:?] 2> at app//org.elasticsearch.index.shard.IndexShardTestCase$$Lambda$2231/0x0000000100845040.close(Unknown Source) 2> at app//org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) 2> at app//org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) 2> at app//org.elasticsearch.index.shard.IndexShardTestCase.closeShard(IndexShardTestCase.java:521) 1> at sun.nio.ch.FileChannelImpl.endBlocking(FileChannelImpl.java:162) ~[?:?] 2> at app//org.elasticsearch.index.shard.IndexShardTestCase.closeShards(IndexShardTestCase.java:528) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase.access$1100(ESIndexLevelReplicationTestCase.java:108) 1> at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:388) ~[?:?] 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationGroup.close(ESIndexLevelReplicationTestCase.java:491) 2> - locked org.elasticsearch.index.replication.RecoveryDuringReplicationTests$1@50e1cf78 2> at app//org.elasticsearch.index.replication.RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo(RecoveryDuringReplicationTests.java:447) 1> at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:85) ~[lucene-test-framework-8.0.0-snapshot-83f9835.jar:8.0.0-snapshot-83f9835 83f9835a47a00a2ec58a4cf5fc0d492497cf7898 - jpountz - 2019-01-21 13:06:00] 2> at java.base@11.0.2/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 1> at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:85) ~[lucene-test-framework-8.0.0-snapshot-83f9835.jar:8.0.0-snapshot-83f9835 83f9835a47a00a2ec58a4cf5fc0d492497cf7898 - jpountz - 2019-01-21 13:06:00] 1> at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:85) ~[lucene-test-framework-8.0.0-snapshot-83f9835.jar:8.0.0-snapshot-83f9835 83f9835a47a00a2ec58a4cf5fc0d492497cf7898 - jpountz - 2019-01-21 13:06:00] 1> at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:182) ~[lucene-core-8.0.0-snapshot-83f9835.jar:8.0.0-snapshot-83f9835 83f9835a47a00a2ec58a4cf5fc0d492497cf7898 - jpountz - 2019-01-21 13:05:58] 2> at java.base@11.0.2/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 2> at java.base@11.0.2/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 1> at org.apache.lucene.store.LockValidatingDirectoryWrapper.deleteFile(LockValidatingDirectoryWrapper.java:37) ~[lucene-core-8.0.0-snapshot-83f9835.jar:8.0.0-snapshot-83f9835 83f9835a47a00a2ec58a4cf5fc0d492497cf7898 - jpountz - 2019-01-21 13:05:58] 1> at org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:696) ~[lucene-core-8.0.0-snapshot-83f9835.jar:8.0.0-snapshot-83f9835 83f9835a47a00a2ec58a4cf5fc0d492497cf7898 - jpountz - 2019-01-21 13:05:58] 2> at java.base@11.0.2/java.lang.reflect.Method.invoke(Method.java:566) 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750) 1> at org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:690) ~[lucene-core-8.0.0-snapshot-83f9835.jar:8.0.0-snapshot-83f9835 83f9835a47a00a2ec58a4cf5fc0d492497cf7898 - jpountz - 2019-01-21 13:05:58] 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938) 1> at org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:589) ~[lucene-core-8.0.0-snapshot-83f9835.jar:8.0.0-snapshot-83f9835 83f9835a47a00a2ec58a4cf5fc0d492497cf7898 - jpountz - 2019-01-21 13:05:58] 1> at org.apache.lucene.index.IndexFileDeleter.checkpoint(IndexFileDeleter.java:531) ~[lucene-core-8.0.0-snapshot-83f9835.jar:8.0.0-snapshot-83f9835 83f9835a47a00a2ec58a4cf5fc0d492497cf7898 - jpountz - 2019-01-21 13:05:58] 1> at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2323) ~[lucene-core-8.0.0-snapshot-83f9835.jar:8.0.0-snapshot-83f9835 83f9835a47a00a2ec58a4cf5fc0d492497cf7898 - jpountz - 2019-01-21 13:05:58] 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974) 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988) 1> at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2266) ~[lucene-core-8.0.0-snapshot-83f9835.jar:8.0.0-snapshot-83f9835 83f9835a47a00a2ec58a4cf5fc0d492497cf7898 - jpountz - 2019-01-21 13:05:58] 1> at org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:2259) ~[lucene-core-8.0.0-snapshot-83f9835.jar:8.0.0-snapshot-83f9835 83f9835a47a00a2ec58a4cf5fc0d492497cf7898 - jpountz - 2019-01-21 13:05:58] 2> at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) 2> at app//org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) 1> at org.elasticsearch.index.engine.InternalEngine.closeNoLock(InternalEngine.java:2078) [main/:?] 2> at app//org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) 1> at org.elasticsearch.index.engine.Engine.close(Engine.java:1732) [main/:?] 1> at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) [elasticsearch-core-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT] 2> at app//org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) 2> at app//org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) 1> at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) [elasticsearch-core-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT] 2> at app//org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) 2> at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) 1> at org.elasticsearch.index.shard.IndexShard.close(IndexShard.java:1248) [main/:?] 2> at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) 1> at org.elasticsearch.index.shard.IndexShardTestCase.lambda$closeShard$8(IndexShardTestCase.java:521) [framework-8.0.0-SNAPSHOT.jar:?] 2> at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) 1> at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) [elasticsearch-core-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT] 2> at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947) 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832) 1> at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) [elasticsearch-core-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT] 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883) 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894) 1> at org.elasticsearch.index.shard.IndexShardTestCase.closeShard(IndexShardTestCase.java:521) [framework-8.0.0-SNAPSHOT.jar:?] 2> at app//org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) 1> at org.elasticsearch.index.shard.IndexShardTestCase.closeShards(IndexShardTestCase.java:528) [framework-8.0.0-SNAPSHOT.jar:?] 2> at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) 1> at org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase.access$1100(ESIndexLevelReplicationTestCase.java:108) [framework-8.0.0-SNAPSHOT.jar:?] 2> at app//org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) 2> at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) 1> at org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationGroup.close(ESIndexLevelReplicationTestCase.java:491) [framework-8.0.0-SNAPSHOT.jar:?] 1> at org.elasticsearch.index.replication.RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo(RecoveryDuringReplicationTests.java:447) [test/:?] 2> at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) 1> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 2> at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) 1> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 2> at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) 2> at app//org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) 1> at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750) [randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938) [randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974) [randomizedtesting-runner-2.7.1.jar:?] 2> at app//org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988) [randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.7.1.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) [lucene-test-framework-8.0.0-snapshot-83f9835.jar:8.0.0-snapshot-83f9835 83f9835a47a00a2ec58a4cf5fc0d492497cf7898 - jpountz - 2019-01-21 13:06:00] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-8.0.0-snapshot-83f9835.jar:8.0.0-snapshot-83f9835 83f9835a47a00a2ec58a4cf5fc0d492497cf7898 - jpountz - 2019-01-21 13:06:00] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) [lucene-test-framework-8.0.0-snapshot-83f9835.jar:8.0.0-snapshot-83f9835 83f9835a47a00a2ec58a4cf5fc0d492497cf7898 - jpountz - 2019-01-21 13:06:00] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-8.0.0-snapshot-83f9835.jar:8.0.0-snapshot-83f9835 83f9835a47a00a2ec58a4cf5fc0d492497cf7898 - jpountz - 2019-01-21 13:06:00] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-8.0.0-snapshot-83f9835.jar:8.0.0-snapshot-83f9835 83f9835a47a00a2ec58a4cf5fc0d492497cf7898 - jpountz - 2019-01-21 13:06:00] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) [randomizedtesting-runner-2.7.1.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) [randomizedtesting-runner-2.7.1.jar:?] 2> at app//org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) 2> at app//org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) 2> at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) 2> at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947) [randomizedtesting-runner-2.7.1.jar:?] 2> at java.base@11.0.2/java.lang.Thread.run(Thread.java:834) 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832) [randomizedtesting-runner-2.7.1.jar:?] 2> "elasticsearch[org.elasticsearch.index.replication.RecoveryDuringReplicationTests][[timer]]" ID=927 TIMED_WAITING 2> at java.base@11.0.2/java.lang.Thread.sleep(Native Method) 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883) [randomizedtesting-runner-2.7.1.jar:?] 2> at app//org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:585) 2> "elasticsearch[org.elasticsearch.index.replication.RecoveryDuringReplicationTests][generic][T#1]" ID=928 WAITING on java.util.concurrent.CountDownLatch$Sync@37ad9740 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894) [randomizedtesting-runner-2.7.1.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-8.0.0-snapshot-83f9835.jar:8.0.0-snapshot-83f9835 83f9835a47a00a2ec58a4cf5fc0d492497cf7898 - jpountz - 2019-01-21 13:06:00] 2> at java.base@11.0.2/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.CountDownLatch$Sync@37ad9740 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.7.1.jar:?] 2> at java.base@11.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) 2> at java.base@11.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885) 2> at java.base@11.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039) 2> at java.base@11.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345) 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) [lucene-test-framework-8.0.0-snapshot-83f9835.jar:8.0.0-snapshot-83f9835 83f9835a47a00a2ec58a4cf5fc0d492497cf7898 - jpountz - 2019-01-21 13:06:00] 2> at java.base@11.0.2/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232) 2> at app//org.elasticsearch.index.replication.RecoveryDuringReplicationTests$BlockingEngineFactory$1.addDocument(RecoveryDuringReplicationTests.java:922) 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.7.1.jar:?] 2> at app//org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1089) 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.7.1.jar:?] 2> at app//org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1034) 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.7.1.jar:?] 2> at app//org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:861) 2> at app//org.elasticsearch.index.engine.InternalTestEngine.index(InternalTestEngine.java:57) 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.7.1.jar:?] 2> at app//org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:787) 2> at app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:760) 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-8.0.0-snapshot-83f9835.jar:8.0.0-snapshot-83f9835 83f9835a47a00a2ec58a4cf5fc0d492497cf7898 - jpountz - 2019-01-21 13:06:00] 2> at app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:717) 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-8.0.0-snapshot-83f9835.jar:8.0.0-snapshot-83f9835 83f9835a47a00a2ec58a4cf5fc0d492497cf7898 - jpountz - 2019-01-21 13:06:00] 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$3(TransportShardBulkAction.java:452) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda$2675/0x0000000100913040.get(Unknown Source) 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-8.0.0-snapshot-83f9835.jar:8.0.0-snapshot-83f9835 83f9835a47a00a2ec58a4cf5fc0d492497cf7898 - jpountz - 2019-01-21 13:06:00] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) [lucene-test-framework-8.0.0-snapshot-83f9835.jar:8.0.0-snapshot-83f9835 83f9835a47a00a2ec58a4cf5fc0d492497cf7898 - jpountz - 2019-01-21 13:06:00] 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:475) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:450) 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.7.1.jar:?] 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:218) 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.7.1.jar:?] 1> at java.lang.Thread.run(Thread.java:834) [?:?] 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:161) 1> [2019-02-27T00:38:29,006][INFO ][o.e.i.r.RecoveryDuringReplicationTests] [testDoNotWaitForPendingSeqNo] after test 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:153) ERROR 0.00s J5 | RecoveryDuringReplicationTests (suite) <<< FAILURES! > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 1200000 msec). 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase.executeShardBulkOnPrimary(ESIndexLevelReplicationTestCase.java:795) > at __randomizedtesting.SeedInfo.seed([F8E8B6147433686D]:0) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase.access$1900(ESIndexLevelReplicationTestCase.java:108) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$WriteReplicationAction.performOnPrimary(ESIndexLevelReplicationTestCase.java:772) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$WriteReplicationAction.performOnPrimary(ESIndexLevelReplicationTestCase.java:763) Completed [1249/1249] on J5 in 1205.18s, 2 tests, 2 errors <<< FAILURES! 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationAction$PrimaryRef.perform(ESIndexLevelReplicationTestCase.java:653) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationAction$PrimaryRef.perform(ESIndexLevelReplicationTestCase.java:639) 2> at app//org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:102) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationAction.execute(ESIndexLevelReplicationTestCase.java:625) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationGroup.executeWriteRequest(ESIndexLevelReplicationTestCase.java:264) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationGroup.index(ESIndexLevelReplicationTestCase.java:248) 2> at app//org.elasticsearch.index.replication.RecoveryDuringReplicationTests.lambda$testDoNotWaitForPendingSeqNo$2(RecoveryDuringReplicationTests.java:472) 2> at app//org.elasticsearch.index.replication.RecoveryDuringReplicationTests$$Lambda$4334/0x0000000100c8a440.run(Unknown Source) 2> at java.base@11.0.2/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) 2> at java.base@11.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:264) 2> at app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) 2> at java.base@11.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 2> at java.base@11.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 2> at java.base@11.0.2/java.lang.Thread.run(Thread.java:834) 2> Locked synchronizers: 2> - java.util.concurrent.ThreadPoolExecutor$Worker@4e580e38 2> - java.util.concurrent.locks.ReentrantLock$NonfairSync@18de41ad 2> "elasticsearch[org.elasticsearch.index.replication.RecoveryDuringReplicationTests][generic][T#2]" ID=929 WAITING on java.util.concurrent.CountDownLatch$Sync@37ad9740 2> at java.base@11.0.2/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.CountDownLatch$Sync@37ad9740 2> at java.base@11.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) 2> at java.base@11.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885) 2> at java.base@11.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039) 2> at java.base@11.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345) 2> at java.base@11.0.2/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232) 2> at app//org.elasticsearch.index.replication.RecoveryDuringReplicationTests$BlockingEngineFactory$1.addDocument(RecoveryDuringReplicationTests.java:922) 2> at app//org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1089) 2> at app//org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1034) 2> at app//org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:861) 2> at app//org.elasticsearch.index.engine.InternalTestEngine.index(InternalTestEngine.java:57) 2> at app//org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:787) 2> at app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:760) 2> at app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:717) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$3(TransportShardBulkAction.java:452) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda$2675/0x0000000100913040.get(Unknown Source) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:475) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:450) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:218) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:161) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:153) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase.executeShardBulkOnPrimary(ESIndexLevelReplicationTestCase.java:795) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase.access$1900(ESIndexLevelReplicationTestCase.java:108) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$WriteReplicationAction.performOnPrimary(ESIndexLevelReplicationTestCase.java:772) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$WriteReplicationAction.performOnPrimary(ESIndexLevelReplicationTestCase.java:763) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationAction$PrimaryRef.perform(ESIndexLevelReplicationTestCase.java:653) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationAction$PrimaryRef.perform(ESIndexLevelReplicationTestCase.java:639) 2> at app//org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:102) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationAction.execute(ESIndexLevelReplicationTestCase.java:625) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationGroup.executeWriteRequest(ESIndexLevelReplicationTestCase.java:264) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationGroup.index(ESIndexLevelReplicationTestCase.java:248) 2> at app//org.elasticsearch.index.replication.RecoveryDuringReplicationTests.lambda$testDoNotWaitForPendingSeqNo$2(RecoveryDuringReplicationTests.java:472) 2> at app//org.elasticsearch.index.replication.RecoveryDuringReplicationTests$$Lambda$4334/0x0000000100c8a440.run(Unknown Source) 2> at java.base@11.0.2/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) 2> at java.base@11.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:264) 2> at app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) 2> at java.base@11.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 2> at java.base@11.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 2> at java.base@11.0.2/java.lang.Thread.run(Thread.java:834) 2> Locked synchronizers: 2> - java.util.concurrent.ThreadPoolExecutor$Worker@3c6ca70e 2> - java.util.concurrent.locks.ReentrantLock$NonfairSync@69e904c2 2> "elasticsearch[org.elasticsearch.index.replication.RecoveryDuringReplicationTests][generic][T#3]" ID=930 WAITING on java.util.concurrent.CountDownLatch$Sync@37ad9740 2> at java.base@11.0.2/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on java.util.concurrent.CountDownLatch$Sync@37ad9740 2> at java.base@11.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) 2> at java.base@11.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885) 2> at java.base@11.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039) 2> at java.base@11.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345) 2> at java.base@11.0.2/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232) 2> at app//org.elasticsearch.index.replication.RecoveryDuringReplicationTests$BlockingEngineFactory$1.addDocument(RecoveryDuringReplicationTests.java:922) 2> at app//org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1089) 2> at app//org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1034) 2> at app//org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:861) 2> at app//org.elasticsearch.index.engine.InternalTestEngine.index(InternalTestEngine.java:57) 2> at app//org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:787) 2> at app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:760) 2> at app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:717) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$3(TransportShardBulkAction.java:452) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda$2675/0x0000000100913040.get(Unknown Source) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:475) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:450) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:218) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:161) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:153) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase.executeShardBulkOnPrimary(ESIndexLevelReplicationTestCase.java:795) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase.access$1900(ESIndexLevelReplicationTestCase.java:108) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$WriteReplicationAction.performOnPrimary(ESIndexLevelReplicationTestCase.java:772) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$WriteReplicationAction.performOnPrimary(ESIndexLevelReplicationTestCase.java:763) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationAction$PrimaryRef.perform(ESIndexLevelReplicationTestCase.java:653) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationAction$PrimaryRef.perform(ESIndexLevelReplicationTestCase.java:639) 2> at app//org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:102) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationAction.execute(ESIndexLevelReplicationTestCase.java:625) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationGroup.executeWriteRequest(ESIndexLevelReplicationTestCase.java:264) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationGroup.index(ESIndexLevelReplicationTestCase.java:248) 2> at app//org.elasticsearch.index.replication.RecoveryDuringReplicationTests.lambda$testDoNotWaitForPendingSeqNo$2(RecoveryDuringReplicationTests.java:472) 2> at app//org.elasticsearch.index.replication.RecoveryDuringReplicationTests$$Lambda$4334/0x0000000100c8a440.run(Unknown Source) 2> at java.base@11.0.2/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) 2> at java.base@11.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:264) 2> at app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) 2> at java.base@11.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 2> at java.base@11.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 2> at java.base@11.0.2/java.lang.Thread.run(Thread.java:834) 2> Locked synchronizers: 2> - java.util.concurrent.ThreadPoolExecutor$Worker@3b846d55 2> - java.util.concurrent.locks.ReentrantLock$NonfairSync@4e30c010 2> "elasticsearch[org.elasticsearch.index.replication.RecoveryDuringReplicationTests][generic][T#4]" ID=931 WAITING on org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue@24893352 2> at java.base@11.0.2/jdk.internal.misc.Unsafe.park(Native Method) 2> - waiting on org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue@24893352 2> at java.base@11.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) 2> at java.base@11.0.2/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:743) 2> at java.base@11.0.2/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:684) 2> at java.base@11.0.2/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1366) 2> at java.base@11.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) 2> at java.base@11.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) 2> at java.base@11.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 2> at java.base@11.0.2/java.lang.Thread.run(Thread.java:834) 2> ^^============================================== 2> REPRODUCE WITH: ./gradlew :server:unitTest -Dtests.seed=F8E8B6147433686D -Dtests.class=org.elasticsearch.index.replication.RecoveryDuringReplicationTests -Dtests.method="testDoNotWaitForPendingSeqNo" -Dtests.security.manager=true -Dtests.locale=no-NO -Dtests.timezone=America/Antigua -Dcompiler.java=11 -Druntime.java=11 2> feb. 27, 2019 12:38:23 A.M. com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks 2> WARNING: Will linger awaiting termination of 6 leaked thread(s). 2> feb. 27, 2019 12:38:28 A.M. com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks 2> SEVERE: 6 threads leaked from SUITE scope at org.elasticsearch.index.replication.RecoveryDuringReplicationTests: 2> 1) Thread[id=928, name=elasticsearch[org.elasticsearch.index.replication.RecoveryDuringReplicationTests][generic][T#1], state=WAITING, group=TGRP-RecoveryDuringReplicationTests] 2> at java.base@11.0.2/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base@11.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) 2> at java.base@11.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885) 2> at java.base@11.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039) 2> at java.base@11.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345) 2> at java.base@11.0.2/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232) 2> at app//org.elasticsearch.index.replication.RecoveryDuringReplicationTests$BlockingEngineFactory$1.addDocument(RecoveryDuringReplicationTests.java:922) 2> at app//org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1089) 2> at app//org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1034) 2> at app//org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:861) 2> at app//org.elasticsearch.index.engine.InternalTestEngine.index(InternalTestEngine.java:57) 2> at app//org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:787) 2> at app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:760) 2> at app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:717) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$3(TransportShardBulkAction.java:452) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda$2675/0x0000000100913040.get(Unknown Source) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:475) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:450) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:218) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:161) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:153) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase.executeShardBulkOnPrimary(ESIndexLevelReplicationTestCase.java:795) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase.access$1900(ESIndexLevelReplicationTestCase.java:108) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$WriteReplicationAction.performOnPrimary(ESIndexLevelReplicationTestCase.java:772) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$WriteReplicationAction.performOnPrimary(ESIndexLevelReplicationTestCase.java:763) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationAction$PrimaryRef.perform(ESIndexLevelReplicationTestCase.java:653) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationAction$PrimaryRef.perform(ESIndexLevelReplicationTestCase.java:639) 2> at app//org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:102) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationAction.execute(ESIndexLevelReplicationTestCase.java:625) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationGroup.executeWriteRequest(ESIndexLevelReplicationTestCase.java:264) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationGroup.index(ESIndexLevelReplicationTestCase.java:248) 2> at app//org.elasticsearch.index.replication.RecoveryDuringReplicationTests.lambda$testDoNotWaitForPendingSeqNo$2(RecoveryDuringReplicationTests.java:472) 2> at app//org.elasticsearch.index.replication.RecoveryDuringReplicationTests$$Lambda$4334/0x0000000100c8a440.run(Unknown Source) 2> at java.base@11.0.2/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) 2> at java.base@11.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:264) 2> at app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) 2> at java.base@11.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 2> at java.base@11.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 2> at java.base@11.0.2/java.lang.Thread.run(Thread.java:834) 2> 2) Thread[id=929, name=elasticsearch[org.elasticsearch.index.replication.RecoveryDuringReplicationTests][generic][T#2], state=WAITING, group=TGRP-RecoveryDuringReplicationTests] 2> at java.base@11.0.2/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base@11.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) 2> at java.base@11.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885) 2> at java.base@11.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039) 2> at java.base@11.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345) 2> at java.base@11.0.2/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232) 2> at app//org.elasticsearch.index.replication.RecoveryDuringReplicationTests$BlockingEngineFactory$1.addDocument(RecoveryDuringReplicationTests.java:922) 2> at app//org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1089) 2> at app//org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1034) 2> at app//org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:861) 2> at app//org.elasticsearch.index.engine.InternalTestEngine.index(InternalTestEngine.java:57) 2> at app//org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:787) 2> at app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:760) 2> at app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:717) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$3(TransportShardBulkAction.java:452) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda$2675/0x0000000100913040.get(Unknown Source) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:475) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:450) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:218) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:161) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:153) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase.executeShardBulkOnPrimary(ESIndexLevelReplicationTestCase.java:795) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase.access$1900(ESIndexLevelReplicationTestCase.java:108) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$WriteReplicationAction.performOnPrimary(ESIndexLevelReplicationTestCase.java:772) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$WriteReplicationAction.performOnPrimary(ESIndexLevelReplicationTestCase.java:763) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationAction$PrimaryRef.perform(ESIndexLevelReplicationTestCase.java:653) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationAction$PrimaryRef.perform(ESIndexLevelReplicationTestCase.java:639) 2> at app//org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:102) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationAction.execute(ESIndexLevelReplicationTestCase.java:625) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationGroup.executeWriteRequest(ESIndexLevelReplicationTestCase.java:264) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationGroup.index(ESIndexLevelReplicationTestCase.java:248) 2> at app//org.elasticsearch.index.replication.RecoveryDuringReplicationTests.lambda$testDoNotWaitForPendingSeqNo$2(RecoveryDuringReplicationTests.java:472) 2> at app//org.elasticsearch.index.replication.RecoveryDuringReplicationTests$$Lambda$4334/0x0000000100c8a440.run(Unknown Source) 2> at java.base@11.0.2/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) 2> at java.base@11.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:264) 2> at app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) 2> at java.base@11.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 2> at java.base@11.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 2> at java.base@11.0.2/java.lang.Thread.run(Thread.java:834) 2> 3) Thread[id=921, name=TEST-RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo-seed#[F8E8B6147433686D], state=WAITING, group=TGRP-RecoveryDuringReplicationTests] 2> at java.base@11.0.2/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base@11.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) 2> at java.base@11.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885) 2> at java.base@11.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:917) 2> at java.base@11.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1240) 2> at java.base@11.0.2/java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:959) 2> at app//org.elasticsearch.common.util.concurrent.ReleasableLock.acquire(ReleasableLock.java:55) 2> at app//org.elasticsearch.index.engine.Engine.close(Engine.java:1730) 2> at app//org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) 2> at app//org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) 2> at app//org.elasticsearch.index.shard.IndexShard.close(IndexShard.java:1248) 2> at app//org.elasticsearch.index.shard.IndexShardTestCase.lambda$closeShard$8(IndexShardTestCase.java:521) 2> at app//org.elasticsearch.index.shard.IndexShardTestCase$$Lambda$2231/0x0000000100845040.close(Unknown Source) 2> at app//org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) 2> at app//org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) 2> at app//org.elasticsearch.index.shard.IndexShardTestCase.closeShard(IndexShardTestCase.java:521) 2> at app//org.elasticsearch.index.shard.IndexShardTestCase.closeShards(IndexShardTestCase.java:528) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase.access$1100(ESIndexLevelReplicationTestCase.java:108) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationGroup.close(ESIndexLevelReplicationTestCase.java:491) 2> at app//org.elasticsearch.index.replication.RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo(RecoveryDuringReplicationTests.java:447) 2> at java.base@11.0.2/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2> at java.base@11.0.2/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 2> at java.base@11.0.2/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2> at java.base@11.0.2/java.lang.reflect.Method.invoke(Method.java:566) 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750) 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938) 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974) 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988) 2> at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) 2> at app//org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) 2> at app//org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) 2> at app//org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) 2> at app//org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) 2> at app//org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) 2> at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) 2> at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) 2> at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) 2> at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947) 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832) 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883) 2> at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894) 2> at app//org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) 2> at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) 2> at app//org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) 2> at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) 2> at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) 2> at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) 2> at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) 2> at app//org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) 2> at app//org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) 2> at app//org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) 2> at app//org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) 2> at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) 2> at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) 2> at java.base@11.0.2/java.lang.Thread.run(Thread.java:834) 2> 4) Thread[id=927, name=elasticsearch[org.elasticsearch.index.replication.RecoveryDuringReplicationTests][[timer]], state=TIMED_WAITING, group=TGRP-RecoveryDuringReplicationTests] 2> at java.base@11.0.2/java.lang.Thread.sleep(Native Method) 2> at app//org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:585) 2> 5) Thread[id=931, name=elasticsearch[org.elasticsearch.index.replication.RecoveryDuringReplicationTests][generic][T#4], state=WAITING, group=TGRP-RecoveryDuringReplicationTests] 2> at java.base@11.0.2/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base@11.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) 2> at java.base@11.0.2/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:743) 2> at java.base@11.0.2/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:684) 2> at java.base@11.0.2/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1366) 2> at java.base@11.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) 2> at java.base@11.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) 2> at java.base@11.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 2> at java.base@11.0.2/java.lang.Thread.run(Thread.java:834) 2> 6) Thread[id=930, name=elasticsearch[org.elasticsearch.index.replication.RecoveryDuringReplicationTests][generic][T#3], state=WAITING, group=TGRP-RecoveryDuringReplicationTests] 2> at java.base@11.0.2/jdk.internal.misc.Unsafe.park(Native Method) 2> at java.base@11.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) 2> at java.base@11.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885) 2> at java.base@11.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039) 2> at java.base@11.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345) 2> at java.base@11.0.2/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232) 2> at app//org.elasticsearch.index.replication.RecoveryDuringReplicationTests$BlockingEngineFactory$1.addDocument(RecoveryDuringReplicationTests.java:922) 2> at app//org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1089) 2> at app//org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1034) 2> at app//org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:861) 2> at app//org.elasticsearch.index.engine.InternalTestEngine.index(InternalTestEngine.java:57) 2> at app//org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:787) 2> at app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:760) 2> at app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:717) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$3(TransportShardBulkAction.java:452) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda$2675/0x0000000100913040.get(Unknown Source) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:475) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:450) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:218) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:161) 2> at app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:153) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase.executeShardBulkOnPrimary(ESIndexLevelReplicationTestCase.java:795) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase.access$1900(ESIndexLevelReplicationTestCase.java:108) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$WriteReplicationAction.performOnPrimary(ESIndexLevelReplicationTestCase.java:772) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$WriteReplicationAction.performOnPrimary(ESIndexLevelReplicationTestCase.java:763) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationAction$PrimaryRef.perform(ESIndexLevelReplicationTestCase.java:653) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationAction$PrimaryRef.perform(ESIndexLevelReplicationTestCase.java:639) 2> at app//org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:102) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationAction.execute(ESIndexLevelReplicationTestCase.java:625) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationGroup.executeWriteRequest(ESIndexLevelReplicationTestCase.java:264) 2> at app//org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationGroup.index(ESIndexLevelReplicationTestCase.java:248) 2> at app//org.elasticsearch.index.replication.RecoveryDuringReplicationTests.lambda$testDoNotWaitForPendingSeqNo$2(RecoveryDuringReplicationTests.java:472) 2> at app//org.elasticsearch.index.replication.RecoveryDuringReplicationTests$$Lambda$4334/0x0000000100c8a440.run(Unknown Source) 2> at java.base@11.0.2/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) 2> at java.base@11.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:264) 2> at app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) 2> at java.base@11.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 2> at java.base@11.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 2> at java.base@11.0.2/java.lang.Thread.run(Thread.java:834) 2> feb. 27, 2019 12:38:28 A.M. com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll 2> INFO: Starting to interrupt leaked threads: 2> 1) Thread[id=928, name=elasticsearch[org.elasticsearch.index.replication.RecoveryDuringReplicationTests][generic][T#1], state=WAITING, group=TGRP-RecoveryDuringReplicationTests] 2> 2) Thread[id=929, name=elasticsearch[org.elasticsearch.index.replication.RecoveryDuringReplicationTests][generic][T#2], state=WAITING, group=TGRP-RecoveryDuringReplicationTests] 2> 3) Thread[id=921, name=TEST-RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo-seed#[F8E8B6147433686D], state=WAITING, group=TGRP-RecoveryDuringReplicationTests] 2> 4) Thread[id=927, name=elasticsearch[org.elasticsearch.index.replication.RecoveryDuringReplicationTests][[timer]], state=TIMED_WAITING, group=TGRP-RecoveryDuringReplicationTests] 2> 5) Thread[id=931, name=elasticsearch[org.elasticsearch.index.replication.RecoveryDuringReplicationTests][generic][T#4], state=WAITING, group=TGRP-RecoveryDuringReplicationTests] 2> 6) Thread[id=930, name=elasticsearch[org.elasticsearch.index.replication.RecoveryDuringReplicationTests][generic][T#3], state=WAITING, group=TGRP-RecoveryDuringReplicationTests] 2> NOTE: leaving temporary files on disk at: /var/lib/jenkins/workspace/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA/java11/ES_RUNTIME_JAVA/java11/nodes/immutable&&linux&&docker/server/build/testrun/unitTest/J5/temp/org.elasticsearch.index.replication.RecoveryDuringReplicationTests_F8E8B6147433686D-001 2> Feb 27, 2019 4:38:29 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll 2> INFO: All leaked threads terminated. 2> REPRODUCE WITH: ./gradlew :server:unitTest -Dtests.seed=F8E8B6147433686D -Dtests.class=org.elasticsearch.index.replication.RecoveryDuringReplicationTests -Dtests.security.manager=true -Dtests.locale=en-US -Dtests.timezone=UTC -Dcompiler.java=11 -Druntime.java=11 2> NOTE: test params are: codec=Asserting(Lucene80): {}, docValues:{}, maxPointsInLeafNode=1292, maxMBSortInHeap=6.877725780809567, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@7e48e845), locale=no-NO, timezone=America/Antigua 2> NOTE: Linux 3.10.0-957.5.1.el7.x86_64 amd64/Oracle Corporation 11.0.2 (64-bit)/cpus=16,threads=1,free=378614656,total=536870912 2> NOTE: All tests run in this JVM: [TermSuggestionBuilderTests, GetIndexActionTests, SpanNotQueryBuilderTests, InternalValueCountTests, TermsTests, LegacyGeoShapeFieldMapperTests, RemoveCorruptedShardDataCommandTests, TransportBulkActionTests, UnassignedInfoTests, FutureUtilsTests, SmileXContentTests, CompletionSuggestionOptionTests, VersionValueTests, HttpInfoTests, DocumentFieldMapperTests, InternalSimpleValueTests, ByteUtilsTests, SignificantLongTermsTests, GroupShardsIteratorTests, TransportSearchActionSingleNodeTests, NetworkModuleTests, InternalEngineSettingsTests, EnableAllocationTests, BooleanFieldMapperTests, FsDirectoryServiceTests, UpdateMappingTests, BestBucketsDeferringCollectorTests, NestedAggregatorTests, JvmGcMonitorServiceSettingsTests, XMoreLikeThisTests, ScriptServiceTests, UpdateSettingsRequestStreamableTests, InternalSettingsPreparerTests, VersionTests, MembershipActionTests, WeightedAvgAggregatorTests, XContentParserUtilsTests, RestGetActionTests, SnapshotsStatusResponseTests, SourceFieldMapperTests, GlobalCheckpointSyncActionTests, TranslogTests, RolloverRequestTests, YamlFilteringGeneratorTests, RebalanceAfterActiveTests, InternalMinTests, PercentilesBucketTests, FieldCapabilitiesResponseTests, NodeIndicesStatsTests, HistogramAggregatorTests, IngestStatsTests, CoordinationMetaDataTests, StreamTests, SearchRequestTests, ManifestTests, FieldSortBuilderTests, MasterServiceTests, IndexRequestBuilderTests, SumTests, GetTermVectorsTests, MultiCommandTests, StartRecoveryRequestTests, MetaDataStateFormatTests, CategoryQueryContextTests, DateMathExpressionResolverTests, CircuitBreakerUnitTests, WildcardQueryBuilderTests, LuceneTests, SimulatePipelineRequestParsingTests, IndexSearcherWrapperTests, QueryRescoreModeTests, ScriptQueryBuilderTests, RegexTests, InternalRangeTests, NoOrdinalsStringFieldDataTests, GeoShapeQueryBuilderTests, RemoteClusterServiceTests, TransportServiceHandshakeTests, MessageDigestsTests, GetIndexTemplatesResponseTests, RestExplainActionTests, GeoBoundsAggregatorTests, NumbersTests, ListKeyStoreCommandTests, ClearVotingConfigExclusionsRequestTests, BoundedBreakIteratorScannerTests, BulkProcessorTests, FetchSearchPhaseTests, URIPatternTests, ConnectionProfileTests, NestedIdentityTests, DynamicTemplatesTests, ListTasksResponseTests, ShardValidateQueryRequestTests, TermsAggregatorTests, PathMapperTests, RetentionLeaseSyncActionTests, GetSnapshotsResponseTests, MultiFieldTests, HalfFloatFielddataTests, CumulativeSumTests, TemplateUpgradeServiceTests, RepositoryDataTests, PluginInfoTests, TransportNodesActionTests, GeoPointFieldMapperTests, SettingsModuleTests, ScheduleWithFixedDelayTests, CacheBuilderTests, SingleShardOneReplicaRoutingTests, HunspellTokenFilterFactoryTests, MoveDecisionTests, BulkByScrollResponseTests, MergePolicySettingsTests, InternalCardinalityTests, GeoTileGridParserTests, ByteSizeValueTests, UUIDTests, SeedHostsResolverTests, ESFileStoreTests, AsyncShardFetchTests, LuceneChangesSnapshotTests, TextFieldMapperTests, QueryShardContextTests, FunctionScoreEquivalenceTests, RecoveryDuringReplicationTests] [ant:junit4] JVM J5: stderr was not empty, see: /var/lib/jenkins/workspace/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA/java11/ES_RUNTIME_JAVA/java11/nodes/immutable&&linux&&docker/server/build/testrun/unitTest/temp/junit4-J5-20190227_041624_41217461174401174407034.syserr > Task :server:unitTest FAILED Tests with failures: - org.elasticsearch.index.replication.RecoveryDuringReplicationTests.testDoNotWaitForPendingSeqNo - org.elasticsearch.index.replication.RecoveryDuringReplicationTests (suite)