Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky TestIngester_compactBlocksToReduceInMemorySeries_Concurrency #5471

Open
dimitarvdimitrov opened this issue Jul 11, 2023 · 8 comments
Open

Flaky TestIngester_compactBlocksToReduceInMemorySeries_Concurrency #5471

dimitarvdimitrov opened this issue Jul 11, 2023 · 8 comments
Labels
component/ingester type/tests

Comments

@dimitarvdimitrov
Copy link
Contributor

The test failed for a Helm PR in the test-stringlabels step

--- FAIL: TestIngester_compactBlocksToReduceInMemorySeries_Concurrency (48.71s)
    --- FAIL: TestIngester_compactBlocksToReduceInMemorySeries_Concurrency/Run_1 (16.45s)
        ingester_early_compaction_test.go:643: 
            	Error Trace:	/__w/mimir/mimir/pkg/ingester/ingester_early_compaction_test.go:643
            	Error:      	"[0 @[1689066211.439] 1 @[1689066211.44] 2 @[1689066211.441] 3 @[1689066211.4[42](https://github.com/grafana/mimir/actions/runs/5517913553/jobs/10061188447?pr=5469#step:8:43)] 4 @[1689066211.4[43](https://github.com/grafana/mimir/actions/runs/5517913553/jobs/10061188447?pr=5469#step:8:44)] 5 @[1689066211.[44](https://github.com/grafana/mimir/actions/runs/5517913553/jobs/10061188447?pr=5469#step:8:45)4] 6 @[1689066211.4[45](https://github.com/grafana/mimir/actions/runs/5517913553/jobs/10061188447?pr=5469#step:8:46)] 7 @[1689066211.4[46](https://github.com/grafana/mimir/actions/runs/5517913553/jobs/10061188447?pr=5469#step:8:47)] 8 @[1689066211.447] 9 @[1689066211.448] 10 @[1689066211.449] 11 @[1689066211.45] 12 @[1689066211.451] 13 @[1689066211.452] 14 @[1689066211.453] 15 @[1689066211.454] 16 @[1689066211.455] 17 @[1689066211.456] 18 @[1689066211.457] 19 @[1689066211.458] 20 @[1689066211.459] 21 @[1689066211.46] 22 @[1689066211.461] 23 @[1689066211.462] 24 @[1689066211.463] 25 @[1689066211.464] 26 @[1689066211.465] 27 @[1689066211.466] 28 @[1689066211.467] 29 @[1689066211.468] 30 @[1689066211.469] 31 @[1689066211.47] 32 @[1689066211.471] 33 @[1689066211.472] 34 @[1689066211.473] 35 @[1689066211.474] 36 @[1689066211.475] 37 @[1689066211.476] 38 @[1689066211.477] 39 @[1689066211.478] 40 @[1689066211.479] 41 @[1689066211.48] 42 @[1689066211.481] 43 @[1689066211.482] 44 @[1689066211.483] 45 @[1689066211.484] 46 @[1689066211.485] 47 @[1689066211.486] 48 @[1689066211.487] 49 @[1689066211.488] 50 @[1689066211.489] 51 @[1689066211.49] 52 @[1689066211.491] 53 @[1689066211.492] 54 @[1689066211.493] 55 @[1689066211.494] 56 @[1689066211.495] 57 @[1689066211.496] 58 @[1689066211.497] 59 @[1689066211.498] 60 @[1689066211.499] 61 @[1689066211.5] 62 @[1689066211.501] 63 @[1689066211.502] 64 @[1689066211.503] 65 @[1689066211.504] 66 @[1689066211.505] 67 @[1689066211.506] 68 @[1689066211.507] 69 @[1689066211.508] 70 @[1689066211.509] 71 @[1689066211.51] 72 @[1689066211.511] 73 @[1689066211.512] 74 @[1689066211.513] 75 @[1689066211.514] 76 @[1689066211.515] 77 @[1689066211.516] 78 @[1689066211.517] 79 @[1689066211.518] 80 @[1689066211.519] 81 @[1689066211.52] 82 @[1689066211.521] 83 @[1689066211.522] 84 @[1689066211.523] 85 @[1689066211.524] 86 @[1689066211.525] 87 @[1689066211.526] 88 @[1689066211.527] 89 @[1689066211.528] 90 @[1689066211.529] 91 @[1689066211.53] 92 @[1689066211.531] 93 @[1689066211.532] 94 @[1689066211.533] 95 @[1689066211.534] 96 @[1689066211.535] 97 @[1689066211.536] 98 @[1689066211.537] 99 @[1689066211.538] 100 @[1689066211.539] 101 @[1689066211.54] 102 @[1689066211.541] 103 @[1689066211.542] 104 @[1689066211.543] 105 @[1689066211.544] 106 @[1689066211.545] 107 @[1689066211.546] 108 @[1689066211.547] 109 @[1689066211.548] 110 @[1689066211.549] 111 @[1689066211.55] 112 @[1689066211.551] 113 @[1689066211.552] 114 @[1689066211.553] 115 @[1689066211.554] 116 @[1689066211.555] 117 @[1689066211.556] 118 @[1689066211.557] 119 @[1689066211.558] 120 @[1689066211.559] 121 @[1689066211.56] 122 @[1689066211.561] 123 @[1689066211.562] 124 @[1689066211.563] 125 @[1689066211.564] 126 @[1689066211.565] 127 @[1689066211.566] 128 @[1689066211.567] 129 @[1689066211.568] 130 @[1689066211.569] 131 @[1689066211.57] 132 @[1689066211.571] 133 @[1689066211.572] 134 @[1689066211.573] 135 @[1689066211.574] 136 @[1689066211.575] 137 @[1689066211.576] 138 @[1689066211.577] 139 @[1689066211.578] 140 @[1689066211.579] 141 @[1689066211.58] 142 @[1689066211.581] 143 @[1689066211.582] 144 @[1689066211.583] 145 @[1689066211.584] 146 @[1689066211.585] 147 @[1689066211.586] 148 @[1689066211.587] 149 @[1689066211.588] 150 @[1689066211.589] 151 @[1689066211.59] 152 @[1689066211.591] 153 @[1689066211.592] 154 @[1689066211.593] 155 @[1689066211.594] 156 @[1689066211.595] 157 @[1689066211.596] 158 @[1689066211.597] 159 @[1689066211.598] 160 @[1689066211.599] 161 @[1689066211.6] 162 @[1689066211.601] 163 @[1689066211.602] 164 @[1689066211.603] 165 @[1689066211.604] 166 @[1689066211.605] 167 @[1689066211.606] 168 @[1689066211.607] 169 @[1689066211.608] 170 @[1689066211.609] 171 @[1689066211.61] 172 @[1689066211.611] 173 @[1689066211.612] 174 @[1689066211.613] 175 @[1689066211.614] 176 @[1689066211.615] 177 @[1689066211.616] 178 @[1689066211.617] 179 @[1689066211.618] 180 @[1689066211.619] 181 @[1689066211.62] 182 @[1689066211.621] 183 @[1689066211.622] 184 @[1689066211.623] 185 @[1689066211.624] 186 @[1689066211.625] 187 @[1689066211.626] 188 @[1689066211.627] 189 @[1689066211.628] 190 @[1689066211.629] 191 @[1689066211.63] 192 @[1689066211.631] 193 @[1689066211.632] 194 @[1689066211.633] 195 @[1689066211.634] 196 @[1689066211.635] 197 @[1689066211.636] 198 @[1689066211.637] 199 @[1689066211.638] 200 @[1689066211.639] 201 @[1689066211.64] 202 @[1689066211.641] 203 @[1689066211.642] 204 @[1689066211.643] 205 @[1689066211.644] 206 @[1689066211.645] 207 @[1689066211.646] 208 @[1689066211.647] 209 @[1689066211.648] 210 @[1689066211.649] 211 @[1689066211.65] 212 @[1689066211.651] 213 @[1689066211.652] 214 @[1689066211.653] 215 @[1689066211.654] 216 @[1689066211.655] 217 @[1689066211.656] 218 @[1689066211.657] 219 @[1689066211.658] 220 @[1689066211.659] 221 @[1689066211.66] 222 @[1689066211.661] 223 @[1689066211.662] 224 @[1689066211.663] 225 @[1689066211.664] 226 @[1689066211.665] 227 @[1689066211.666] 228 @[1689066211.667] 229 @[1689066211.668] 230 @[1689066211.669] 231 @[1689066211.67] 232 @[1689066211.671] 233 @[1689066211.672] 234 @[1689066211.673] 235 @[1689066211.674] 236 @[1689066211.675] 237 @[1689066211.676] 238 @[1689066211.677] 239 @[1689066211.678] 240 @[1689066211.679] 241 @[1689066211.68] 242 @[1689066211.681] 243 @[1689066211.682] 244 @[1689066211.683] 245 @[1689066211.684] 246 @[1689066211.685] 247 @[1689066211.686] 248 @[1689066211.687] 249 @[1689066211.688] 250 @[1689066211.689] 251 @[1689066211.69] 252 @[1689066211.691] 253 @[1689066211.692] 254 @[1689066211.693] 255 @[1689066211.694] 256 @[1689066211.695] 257 @[1689066211.696] 258 @[1689066211.697] 259 @[1689066211.698] 260 @[1689066211.699] 261 @[1689066211.7] 262 @[1689066211.701] 263 @[1689066211.702] 264 @[1689066211.703] 265 @[1689066211.704] 266 @[1689066211.705] 267 @[1689066211.706] 268 @[1689066211.707] 269 @[1689066211.708] 270 @[1689066211.709] 271 @[1689066211.71] 272 @[1689066211.711] 273 @[1689066211.712] 274 @[1689066211.713] 275 @[1689066211.714] 276 @[1689066211.715] 277 @[1689066211.716] 278 @[1689066211.717] 279 @[1689066211.718] 280 @[1689066211.719] 281 @[1689066211.72] 282 @[1689066211.721] 283 @[1689066211.722] 284 @[1689066211.723] 285 @[1689066211.724] 286 @[1689066211.725] 287 @[1689066211.726] 288 @[1689066211.727] 289 @[1689066211.728] 290 @[1689066211.729] 291 @[1689066211.73] 292 @[1689066211.731] 293 @[1689066211.732] 294 @[1689066211.733] 295 @[1689066211.734] 296 @[1689066211.735] 297 @[1689066211.736] 298 @[1689066211.737] 299 @[1689066211.738] 300 @[1689066211.739] 301 @[1689066211.74] 302 @[1689066211.741] 303 @[1689066211.742] 304 @[1689066211.743] 305 @[1689066211.744] 306 @[1689066211.745] 307 @[1689066211.746] 308 @[1689066211.747] 309 @[1689066211.748] 310 @[1689066211.749] 311 @[1689066211.75] 312 @[1689066211.751] 313 @[1689066211.752] 314 @[1689066211.753] 315 @[1689066211.754] 316 @[1689066211.755] 317 @[1689066211.756] 318 @[1689066211.757] 319 @[1689066211.758] 320 @[1689066211.759] 321 @[1689066211.76] 322 @[1689066211.761] 323 @[1689066211.762] 324 @[1689066211.763] 325 @[1689066211.764] 326 @[1689066211.765] 327 @[1689066211.766] 328 @[1689066211.767] 329 @[1689066211.768] 330 @[1689066211.769] 331 @[1689066211.77] 332 @[1689066211.771] 333 @[1689066211.772] 334 @[1689066211.773] 335 @[1689066211.774] 336 @[1689066211.775] 337 @[1689066211.776] 338 @[1689066211.777] 339 @[1689066211.778] 340 @[1689066211.779] 341 @[1689066211.78] 342 @[1689066211.781] 343 @[1689066211.782] 344 @[1689066211.783] 345 @[1689066211.784] 346 @[1689066211.785] 347 @[1689066211.786] 348 @[1689066211.787] 349 @[1689066211.788] 350 @[1689066211.789] 351 @[1689066211.79] 352 @[1689066211.791] 353 @[1689066211.792] 354 @[1689066211.793] 355 @[1689066211.794] 356 @[1689066211.795] 357 @[1689066211.796] 358 @[1689066211.797] 359 @[1689066211.798] 360 @[1689066211.799] 361 @[1689066211.8] 362 @[1689066211.801] 363 @[1689066211.802] 364 @[1689066211.803] 365 @[1689066211.804] 366 @[1689066211.805] 367 @[1689066211.806] 368 @[1689066211.807] 369 @[1689066211.808] 370 @[1689066211.809] 371 @[1689066211.81] 372 @[1689066211.811] 373 @[1689066211.812] 374 @[1689066211.813] 375 @[1689066211.814] 376 @[1689066211.815] 377 @[1689066211.816] 378 @[1689066211.817] 379 @[1689066211.818] 380 @[1689066211.819] 381 @[1689066211.82] 382 @[1689066211.821] 383 @[1689066211.822] 384 @[1689066211.823] 385 @[1689066211.824] 386 @[1689066211.825] 387 @[1689066211.826] 388 @[1689066211.827] 389 @[1689066211.828] 390 @[1689066211.829] 391 @[1689066211.83] 392 @[1689066211.831] 393 @[1689066211.832] 394 @[1689066211.833] 395 @[1689066211.834] 396 @[1689066211.835] 397 @[1689066211.836] 398 @[1689066211.837] 399 @[1689066211.838] 400 @[1689066211.839] 401 @[1689066211.84] 402 @[1689066211.841] 403 @[1689066211.842] 404 @[1689066211.843] 405 @[1689066211.844] 406 @[1689066211.845] 407 @[1689066211.846] 408 @[1689066211.847] 409 @[1689066211.848] 410 @[1689066211.849] 411 @[1689066211.85] 412 @[1689066211.851] 413 @[1689066211.852] 414 @[1689066211.853] 415 @[1689066211.854] 416 @[1689066211.855] 417 @[1689066211.856] 418 @[1689066211.857] 419 @[1689066211.858] 420 @[1689066211.859] 421 @[1689066211.86] 422 @[1689066211.861] 423 @[1689066211.862] 424 @[1689066211.863] 425 @[1689066211.864] 426 @[1689066211.865] 427 @[1689066211.866] 428 @[1689066211.867] 429 @[1689066211.868] 430 @[1689066211.869] 431 @[1689066211.87] 432 @[1689066211.871] 433 @[1689066211.872] 434 @[1689066211.873] 435 @[1689066211.874] 436 @[1689066211.875] 437 @[1689066211.876] 438 @[1689066211.877] 439 @[1689066211.878] 440 @[1689066211.879] 441 @[1689066211.88] 442 @[1689066211.881] 443 @[1689066211.882] 444 @[1689066211.883] 445 @[1689066211.884] 446 @[1689066211.885] 447 @[1689066211.886] 448 @[1689066211.887] 449 @[1689066211.888] 450 @[1689066211.889] 451 @[1689066211.89] 452 @[1689066211.891] 453 @[1689066211.892] 454 @[1689066211.893] 455 @[1689066211.894] 456 @[1689066211.895] 457 @[1689066211.896] 458 @[1689066211.897] 459 @[1689066211.898] 460 @[1689066211.899] 461 @[1689066211.9] 462 @[1689066211.901] 463 @[1689066211.902] 464 @[1689066211.903] 465 @[1689066211.904] 466 @[1689066211.905] 467 @[1689066211.906] 468 @[1689066211.907] 469 @[1689066211.908] 470 @[1689066211.909] 471 @[1689066211.91] 472 @[1689066211.911] 473 @[1689066211.912] 474 @[1689066211.913] 475 @[1689066211.914] 476 @[1689066211.915] 477 @[1689066211.916] 478 @[1689066211.917] 479 @[1689066211.918] 480 @[1689066211.919] 481 @[1689066211.92] 482 @[1689066211.921] 483 @[1689066211.922] 484 @[1689066211.923] 485 @[1689066211.924] 486 @[1689066211.925] 487 @[1689066211.926] 488 @[1689066211.927] 489 @[1689066211.928] 490 @[1689066211.929] 491 @[1689066211.93] 492 @[1689066211.931] 493 @[1689066211.932] 494 @[1689066211.933] 495 @[1689066211.934] 496 @[1689066211.935] 497 @[1689066211.936] 498 @[1689066211.937] 499 @[1689066211.938] 500 @[1689066211.939] 501 @[1689066211.94] 502 @[1689066211.941] 503 @[1689066211.942] 504 @[1689066211.943] 505 @[1689066211.944] 506 @[1689066211.945] 507 @[1689066211.946] 508 @[1689066211.947] 509 @[1689066211.948] 510 @[1689066211.949] 511 @[1689066211.95] 512 @[1689066211.951] 513 @[1689066211.952] 514 @[1689066211.953] 515 @[1689066211.954] 516 @[1689066211.955] 517 @[1689066211.956] 518 @[1689066211.957] 519 @[1689066211.958] 520 @[1689066211.959] 521 @[1689066211.96] 522 @[1689066211.961] 523 @[1689066211.962] 524 @[1689066211.963] 525 @[1689066211.964] 526 @[1689066211.965] 527 @[1689066211.966] 528 @[1689066211.967] 529 @[1689066211.968] 530 @[1689066211.969] 531 @[1689066211.97] 532 @[1689066211.971] 533 @[1689066211.972] 534 @[1689066211.973] 535 @[1689066211.974] 536 @[1689066211.975] 537 @[1689066211.976] 538 @[1689066211.977] 539 @[1689066211.978] 540 @[1689066211.979] 541 @[1689066211.98] 542 @[1689066211.981] 543 @[1689066211.982] 544 @[1689066211.983] 545 @[1689066211.984] 546 @[1689066211.985] 547 @[1689066211.986] 548 @[1689066211.987] 549 @[1689066211.988] 550 @[1689066211.989] 551 @[1689066211.99] 552 @[1689066211.991] 553 @[1689066211.992] 554 @[1689066211.993] 555 @[1689066211.994] 556 @[1689066211.995] 557 @[1689066211.996] 558 @[1689066211.997] 559 @[1689066211.998] 560 @[1689066211.999] 561 @[1689066212] 562 @[1689066212.001] 563 @[1689066212.002] 564 @[1689066212.003] 565 @[1689066212.004] 566 @[1689066212.005] 567 @[1689066212.006] 568 @[1689066212.007] 569 @[1689066212.008] 570 @[1689066212.009] 571 @[1689066212.01] 572 @[1689066212.011] 573 @[1689066212.012] 574 @[1689066212.013] 575 @[1689066212.014] 576 @[1689066212.015] 577 @[1689066212.016] 578 @[1689066212.017] 579 @[1689066212.018] 580 @[1689066212.019] 581 @[1689066212.02] 582 @[1689066212.021] 583 @[1689066212.022] 584 @[1689066212.023] 585 @[1689066212.024] 586 @[1689066212.025] 587 @[1689066212.026] 588 @[1689066212.027] 589 @[1689066212.028] 590 @[1689066212.029] 591 @[1689066212.03] 592 @[1689066212.031] 593 @[1689066212.032] 594 @[1689066212.033] 595 @[1689066212.034] 596 @[1689066212.035] 597 @[1689066212.036] 598 @[1689066212.037] 599 @[1689066212.038] 600 @[1689066212.039] 601 @[1689066212.04] 602 @[1689066212.041] 603 @[1689066212.042] 604 @[1689066212.043] 605 @[1689066212.044] 606 @[1689066212.045] 607 @[1689066212.046] 608 @[1689066212.047] 609 @[1689066212.048] 610 @[1689066212.049] 611 @[1689066212.05] 612 @[1689066212.051] 613 @[1689066212.052] 614 @[1689066212.053] 615 @[1689066212.054] 616 @[1689066212.055] 617 @[1689066212.056] 618 @[1689066212.057] 619 @[1689066212.058] 620 @[1689066212.059] 621 @[1689066212.06] 622 @[1689066212.061] 623 @[1689066212.062] 624 @[1689066212.063] 625 @[1689066212.064] 626 @[1689066212.065] 627 @[1689066212.066] 628 @[1689066212.067] 629 @[1689066212.068] 630 @[1689066212.069] 631 @[1689066212.07] 632 @[1689066212.071] 633 @[1689066212.072] 634 @[1689066212.073] 635 @[1689066212.074] 636 @[1689066212.075] 637 @[1689066212.076] 638 @[1689066212.077] 639 @[1689066212.078] 640 @[1689066212.079] 641 @[1689066212.08] 642 @[1689066212.081] 643 @[1689066212.082] 644 @[1689066212.083] 645 @[1689066212.084] 646 @[1689066212.085] 647 @[1689066212.086] 648 @[1689066212.087] 649 @[1689066212.088] 650 @[1689066212.089] 651 @[1689066212.09] 652 @[1689066212.091] 653 @[1689066212.092] 654 @[1689066212.093] 655 @[1689066212.094] 656 @[1689066212.095] 657 @[1689066212.096] 658 @[1689066212.097] 659 @[1689066212.098] 660 @[1689066212.099] 661 @[1689066212.1] 662 @[1689066212.101] 663 @[1689066212.102] 664 @[1689066212.103] 665 @[1689066212.104] 666 @[1689066212.105] 667 @[1689066212.106] 668 @[1689066212.107] 669 @[1689066212.108] 670 @[1689066212.109] 671 @[1689066212.11] 672 @[1689066212.111] 673 @[1689066212.112] 674 @[1689066212.113] 675 @[1689066212.114] 676 @[1689066212.115] 677 @[1689066212.116] 678 @[1689066212.117] 679 @[1689066212.118] 680 @[1689066212.119] 681 @[1689066212.12] 682 @[1689066212.121] 683 @[1689066212.122] 684 @[1689066212.123] 685 @[1689066212.124] 686 @[1689066212.125] 687 @[1689066212.126] 688 @[1689066212.127] 689 @[1689066212.128] 690 @[1689066212.129] 691 @[1689066212.13] 692 @[1689066212.131] 693 @[1689066212.132] 694 @[1689066212.133] 695 @[1689066212.134] 696 @[1689066212.135] 697 @[1689066212.136] 698 @[1689066212.137] 699 @[1689066212.138] 700 @[1689066212.139] 701 @[1689066212.14] 702 @[1689066212.141] 703 @[1689066212.142] 704 @[1689066212.143] 705 @[1689066212.144] 706 @[1689066212.145] 707 @[1689066212.146] 708 @[1689066212.147] 709 @[1689066212.148] 710 @[1689066212.149] 711 @[1689066212.15] 712 @[1689066212.151] 713 @[1689066212.152] 714 @[1689066212.153] 715 @[1689066212.154] 716 @[1689066212.155] 717 @[1689066212.156] 718 @[1689066212.157] 719 @[1689066212.158] 720 @[1689066212.159] 721 @[1689066212.16] 722 @[1689066212.161] 723 @[1689066212.162] 724 @[1689066212.163] 725 @[1689066212.164] 726 @[1689066212.165] 727 @[1689066212.166] 728 @[1689066212.167] 729 @[1689066212.168] 730 @[1689066212.169] 731 @[1689066212.17] 732 @[1689066212.171] 733 @[1689066212.172] 734 @[1689066212.173] 735 @[1689066212.174] 736 @[1689066212.175] 737 @[1689066212.176] 738 @[1689066212.177] 739 @[1689066212.178] 740 @[1689066212.179] 741 @[1689066212.18] 742 @[1689066212.181] 743 @[1689066212.182] 744 @[1689066212.183] 745 @[1689066212.184] 746 @[1689066212.185] 747 @[1689066212.186] 748 @[1689066212.187] 749 @[1689066212.188] 750 @[1689066212.189] 751 @[1689066212.19] 752 @[1689066212.191] 753 @[1689066212.192] 754 @[1689066212.193] 755 @[1689066212.194] 756 @[1689066212.195] 757 @[1689066212.196] 758 @[1689066212.197] 759 @[1689066212.198] 760 @[1689066212.199] 761 @[1689066212.2] 762 @[1689066212.201] 763 @[1689066212.202] 764 @[1689066212.203] 765 @[1689066212.204] 766 @[1689066212.205] 767 @[1689066212.206] 768 @[1689066212.207] 769 @[1689066212.208] 770 @[1689066212.209] 771 @[1689066212.21] 772 @[1689066212.211] 773 @[1689066212.212] 774 @[1689066212.213] 775 @[1689066212.214] 776 @[1689066212.215] 777 @[1689066212.216] 778 @[1689066212.217] 779 @[1689066212.218] 780 @[1689066212.219] 781 @[1689066212.22] 782 @[1689066212.221] 783 @[1689066212.222] 784 @[1689066212.223] 785 @[1689066212.224] 786 @[1689066212.225] 787 @[1689066212.226] 788 @[1689066212.227] 789 @[1689066212.228] 790 @[1689066212.229] 791 @[1689066212.23] 792 @[1689066212.231] 793 @[1689066212.232] 794 @[1689066212.233] 795 @[1689066212.234] 796 @[1689066212.235] 797 @[1689066212.236] 798 @[1689066212.237] 799 @[1689066212.238] 800 @[1689066212.239] 801 @[1689066212.24] 802 @[1689066212.241] 803 @[1689066212.242] 804 @[1689066212.243] 805 @[1689066212.244] 806 @[1689066212.245] 807 @[1689066212.246] 808 @[1689066212.247] 809 @[1689066212.248] 810 @[1689066212.249] 811 @[1689066212.25] 812 @[1689066212.251] 813 @[1689066212.252] 814 @[1689066212.253] 815 @[1689066212.254] 816 @[1689066212.255] 817 @[1689066212.256] 818 @[1689066212.257] 819 @[1689066212.258] 820 @[1689066212.259] 821 @[1689066212.26] 822 @[1689066212.261] 823 @[1689066212.262] 824 @[1689066212.263] 825 @[1689066212.264] 826 @[1689066212.265] 827 @[1689066212.266] 828 @[1689066212.267] 829 @[1689066212.268] 830 @[1689066212.269] 831 @[1689066212.27] 832 @[1689066212.271] 833 @[1689066212.272] 834 @[1689066212.273] 835 @[1689066212.274] 836 @[1689066212.275] 837 @[1689066212.276] 838 @[1689066212.277] 839 @[1689066212.278] 840 @[1689066212.279] 841 @[1689066212.28] 842 @[1689066212.281] 843 @[1689066212.282] 844 @[1689066212.283] 845 @[1689066212.284] 846 @[1689066212.285] 847 @[1689066212.286] 848 @[1689066212.287] 849 @[1689066212.288] 850 @[1689066212.289] 851 @[1689066212.29] 852 @[1689066212.291] 853 @[1689066212.292] 854 @[1689066212.293] 855 @[1689066212.294] 856 @[1689066212.295] 857 @[1689066212.296] 858 @[1689066212.297] 859 @[1689066212.298] 860 @[1689066212.299] 861 @[1689066212.3] 862 @[1689066212.301] 863 @[1689066212.302] 864 @[1689066212.303] 865 @[1689066212.304] 866 @[1689066212.305] 867 @[1689066212.306] 868 @[1689066212.307] 869 @[1689066212.308] 870 @[1689066212.309] 871 @[1689066212.31] 872 @[1689066212.311] 873 @[1689066212.312] 874 @[1689066212.313] 875 @[1689066212.314] 876 @[1689066212.315] 877 @[1689066212.316] 878 @[1689066212.317] 879 @[1689066212.318] 880 @[1689066212.319] 881 @[1689066212.32] 882 @[1689066212.321] 883 @[1689066212.322] 884 @[1689066212.323] 885 @[1689066212.324] 886 @[1689066212.325] 887 @[1689066212.326] 888 @[1689066212.327] 889 @[1689066212.328] 890 @[1689066212.329] 891 @[1689066212.33] 892 @[1689066212.331] 893 @[1689066212.332] 894 @[1689066212.333] 895 @[1689066212.334] 896 @[1689066212.335] 897 @[1689066212.336] 898 @[1689066212.337] 899 @[1689066212.338] 900 @[1689066212.339] 901 @[1689066212.34] 902 @[1689066212.341] 903 @[1689066212.342] 904 @[1689066212.343] 905 @[1689066212.344] 906 @[1689066212.345] 907 @[1689066212.346] 908 @[1689066212.347] 909 @[1689066212.348] 910 @[1689066212.349] 911 @[1689066212.35] 912 @[1689066212.351] 913 @[1689066212.352] 914 @[1689066212.353] 915 @[1689066212.354] 916 @[1689066212.355] 917 @[1689066212.356] 918 @[1689066212.357] 919 @[1689066212.358] 920 @[1689066212.359] 921 @[1689066212.36] 922 @[1689066212.361] 923 @[1689066212.362] 924 @[1689066212.363] 925 @[1689066212.364] 926 @[1689066212.365] 927 @[1689066212.366] 928 @[1689066212.367] 929 @[1689066212.368] 930 @[1689066212.369] 931 @[1689066212.37] 932 @[1689066212.371] 933 @[1689066212.372] 934 @[1689066212.373] 935 @[1689066212.374] 936 @[1689066212.375] 937 @[1689066212.376] 938 @[1689066212.377] 939 @[1689066212.378] 940 @[1689066212.379] 941 @[1689066212.38] 942 @[1689066212.381] 943 @[1689066212.382] 944 @[1689066212.383] 945 @[1689066212.384] 946 @[1689066212.385] 947 @[1689066212.386] 948 @[1689066212.387] 949 @[1689066212.388] 950 @[1689066212.389] 951 @[1689066212.39] 952 @[1689066212.391] 953 @[1689066212.392] 954 @[1689066212.393] 955 @[1689066212.394] 956 @[1689066212.395] 957 @[1689066212.396] 958 @[1689066212.397] 959 @[1689066212.398] 960 @[1689066212.399] 961 @[1689066212.4] 962 @[1689066212.401] 963 @[1689066212.402] 964 @[1689066212.403] 965 @[1689066212.404] 966 @[1689066212.405] 967 @[1689066212.406] 968 @[1689066212.407] 969 @[1689066212.408] 970 @[1689066212.409] 971 @[1689066212.41] 972 @[1689066212.411] 973 @[1689066212.412] 974 @[1689066212.413] 975 @[1689066212.414] 976 @[1689066212.415] 977 @[1689066212.416] 978 @[1689066212.417] 979 @[1689066212.418] 980 @[1689066212.419] 981 @[1689066212.42] 982 @[1689066212.421] 983 @[1689066212.422] 984 @[1689066212.423] 985 @[1689066212.424] 986 @[1689066212.425] 987 @[1689066212.426] 988 @[1689066212.427] 989 @[1689066212.428] 990 @[1689066212.429] 991 @[1689066212.43] 992 @[1689066212.431] 993 @[1689066212.432] 994 @[1689066212.433] 995 @[1689066212.434] 996 @[1689066212.435] 997 @[1689066212.436] 998 @[1689066212.437] 999 @[1689066212.438] 1000 @[1689066212.439] 1001 @[1689066212.44] 1002 @[1689066212.441] 1003 @[1689066212.442] 1004 @[1689066212.443] 1005 @[1689066212.444] 1006 @[1689066212.445] 1007 @[1689066212.446] 1008 @[1689066212.447] 1009 @[1689066212.448] 1010 @[1689066212.449] 1011 @[1689066212.45] 1012 @[1689066212.451] 1013 @[1689066212.452] 1014 @[1689066212.453] 1015 @[1689066212.454] 1016 @[1689066212.455] 1017 @[1689066212.456] 1018 @[1689066212.457] 1019 @[1689066212.458] 1020 @[1689066212.459] 1021 @[1689066212.46] 1022 @[1689066212.461] 1023 @[1689066212.462] 1024 @[1689066212.463] 1025 @[1689066212.464] 1026 @[1689066212.465] 1027 @[1689066212.466] 1028 @[1689066212.467] 1029 @[1689066212.468] 1030 @[1689066212.469] 1031 @[1689066212.47] 1032 @[1689066212.471] 1033 @[1689066212.472] 1034 @[1689066212.473] 1035 @[1689066212.474] 1036 @[1689066212.475] 1037 @[1689066212.476] 1038 @[1689066212.477] 1039 @[1689066212.478] 1040 @[1689066212.479] 1041 @[1689066212.48] 1042 @[1689066212.481] 1043 @[1689066212.482] 1044 @[1689066212.483] 1045 @[1689066212.484] 1046 @[1689066212.485] 1047 @[1689066212.486] 1048 @[1689066212.487] 1049 @[1689066212.488] 1050 @[1689066212.489] 1051 @[1689066212.49] 1052 @[1689066212.491] 1053 @[1689066212.492] 1054 @[1689066212.493] 1055 @[1689066212.494] 1056 @[1689066212.495] 1057 @[1689066212.496] 1058 @[1689066212.497] 1059 @[1689066212.498] 1060 @[1689066212.499] 1061 @[1689066212.5] 1062 @[1689066212.501] 1063 @[1689066212.502] 1064 @[1689066212.503] 1065 @[1689066212.504] 1066 @[1689066212.505] 1067 @[1689066212.506] 1068 @[1689066212.507] 1069 @[1689066212.508] 1070 @[1689066212.509] 1071 @[1689066212.51] 1072 @[1689066212.511] 1073 @[1689066212.512] 1074 @[1689066212.513] 1075 @[1689066212.514] 1076 @[1689066212.515] 1077 @[1689066212.516] 1078 @[1689066212.517] 1079 @[1689066212.518] 1080 @[1689066212.519] 1081 @[1689066212.52] 1082 @[1689066212.521] 1083 @[1689066212.522] 1084 @[1689066212.523] 1085 @[1689066212.524] 1086 @[1689066212.525] 1087 @[1689066212.526] 1088 @[1689066212.527] 1089 @[1689066212.528] 1090 @[1689066212.529] 1091 @[1689066212.53] 1092 @[1689066212.531] 1093 @[1689066212.532] 1094 @[1689066212.533] 1095 @[1689066212.534] 1096 @[1689066212.535] 1097 @[1689066212.536] 1098 @[1689066212.537] 1099 @[1689066212.538] 1100 @[1689066212.539] 1101 @[1689066212.54] 1102 @[1689066212.541] 1103 @[1689066212.542] 1104 @[1689066212.543] 1105 @[1689066212.544] 1106 @[1689066212.545] 1107 @[1689066212.546] 1108 @[1689066212.547] 1109 @[1689066212.548] 1110 @[1689066212.549] 1111 @[1689066212.55] 1112 @[1689066212.551] 1113 @[1689066212.552] 1114 @[1689066212.553] 1115 @[1689066212.554] 1116 @[1689066212.555] 1117 @[1689066212.556] 1118 @[1689066212.557] 1119 @[1689066212.558] 1120 @[1689066212.559] 1121 @[1689066212.56] 1122 @[1689066212.561] 1123 @[1689066212.562] 1124 @[1689066212.563] 1125 @[1689066212.564] 1126 @[1689066212.565] 1127 @[1689066212.566] 1128 @[1689066212.567] 1129 @[1689066212.568] 1130 @[1689066212.569] 1131 @[1689066212.57] 1132 @[1689066212.571] 1133 @[1689066212.572] 1134 @[1689066212.573] 1135 @[1689066212.574] 1136 @[1689066212.575] 1137 @[1689066212.576] 1138 @[1689066212.577] 1139 @[1689066212.578] 1140 @[1689066212.579] 1141 @[1689066212.58] 1142 @[1689066212.581] 1143 @[1689066212.582] 1144 @[1689066212.583] 1145 @[1689066212.584] 1146 @[1689066212.585] 1147 @[1689066212.586] 1148 @[1689066212.587] 1149 @[1689066212.588] 1150 @[1689066212.589] 1152 @[1689066212.591] 1153 @[1689066212.592] 1154 @[1689066212.593] 1155 @[1689066212.594] 1156 @[1689066212.595] 1157 @[1689066212.596] 1158 @[1689066212.597] 1159 @[1689066212.598] 1160 @[1689066212.599] 1161 @[1689066212.6] 1162 @[1689066212.601] 1163 @[1689066212.602] 1164 @[1689066212.603] 1165 @[1689066212.604] 1166 @[1689066212.605] 1167 @[1689066212.606] 1168 @[1689066212.607] 1169 @[1689066212.608] 1170 @[1689066212.609] 1171 @[1689066212.61] 1172 @[1689066212.611] 1173 @[1689066212.612] 1174 @[1689066212.613] 1175 @[1689066212.614] 1176 @[1689066212.615] 1177 @[1689066212.616] 1178 @[1689066212.617] 1179 @[1689066212.618] 1180 @[1689066212.619] 1181 @[1689066212.62] 1182 @[1689066212.621] 1183 @[1689066212.622] 1184 @[1689066212.623] 1185 @[1689066212.624] 1186 @[1689066212.625] 1187 @[1689066212.626] 1188 @[1689066212.627] 1189 @[1689066212.628] 1190 @[1689066212.629] 1191 @[1689066212.63] 1192 @[1689066212.631] 1193 @[1689066212.632] 1194 @[1689066212.633] 1195 @[1689066212.634] 1196 @[1689066212.635] 1197 @[1689066212.636] 1198 @[1689066212.637] 1199 @[1689066212.638]]" should have 1200 item(s), but has 1199
            	Test:       	TestIngester_compactBlocksToReduceInMemorySeries_Concurrency/Run_1
            	Messages:   	response entry: 33
level=info msg="uploading new block to long-term storage" block=00000000010000000000000000
level=debug msg="uploaded file" from=/tmp/TestShipper_DeceivingUploadErrors1170[48](https://github.com/grafana/mimir/actions/runs/5517913553/jobs/10061188447?pr=5469#step:8:49)9307/001/00000000010000000000000000/index dst=00000000010000000000000000/index bucket="fs: /tmp/TestShipper_DeceivingUploadErrors1170489307/002"
level=error msg="uploading new block to long-term storage failed" block=00000000010000000000000000 err="upload meta file: base name matches, will fail upload"
FAIL
@charleskorn charleskorn changed the title Falky TestIngester_compactBlocksToReduceInMemorySeries_Concurrency Flaky TestIngester_compactBlocksToReduceInMemorySeries_Concurrency Jul 11, 2023
@dimitarvdimitrov
Copy link
Contributor Author

dimitarvdimitrov commented Jan 25, 2024 •
edited
Loading

another instance. This failure seems different

2024-01-25T11:45:34.3608830Z --- FAIL: TestIngester_compactBlocksToReduceInMemorySeries_Concurrency (26.06s)
2024-01-25T11:45:34.3611000Z     --- FAIL: TestIngester_compactBlocksToReduceInMemorySeries_Concurrency/Run_1 (6.29s)
2024-01-25T11:45:34.3613373Z         ingester_early_compaction_test.go:528: 
2024-01-25T11:45:34.3733108Z             	Error Trace:	/__w/mimir/mimir/pkg/ingester/ingester_early_compaction_test.go:528
2024-01-25T11:45:34.3772545Z             	            				/usr/local/go/src/runtime/asm_amd64.s:1650
2024-01-25T11:45:34.3812668Z             	Error:      	Received unexpected error:
2024-01-25T11:45:34.3841428Z             	            	user=1: the sample has been rejected because its timestamp is too old (err-mimir-sample-timestamp-too-old). The affected sample has timestamp 2024-01-25T11:43:05.717Z and is from series series_00010
2024-01-25T11:45:34.3843724Z             	Test:       	TestIngester_compactBlocksToReduceInMemorySeries_Concurrency/Run_1
2024-01-25T11:45:34.3844826Z         ingester_early_compaction_test.go:627: 
2024-01-25T11:45:34.3846085Z             	Error Trace:	/__w/mimir/mimir/pkg/ingester/ingester_early_compaction_test.go:627
2024-01-25T11:45:34.3847228Z             	Error:      	"0" is not greater than "1"
2024-01-25T11:45:34.3848261Z             	Test:       	TestIngester_compactBlocksToReduceInMemorySeries_Concurrency/Run_1
2024-01-25T11:45:34.3849436Z         ingester_early_compaction_test.go:647: 
2024-01-25T11:45:34.3850535Z             	Error Trace:	/__w/mimir/mimir/pkg/ingester/ingester_early_compaction_test.go:647
2024-01-25T11:45:34.3892312Z             	Error:      	"series_00000 =>
2024-01-25T11:45:34.3893217Z             	            	0 @[1706182985.717]
...
2024-01-25T11:45:35.5137836Z             	            	1199 @[1706182986.916]" should have 50 item(s), but has 40
2024-01-25T11:45:35.5138334Z             	Test:       	TestIngester_compactBlocksToReduceInMemorySeries_Concurrency/Run_1
2024-01-25T11:45:35.5139242Z level=info msg="uploading new block to long-term storage" block=00000000010000000000000000
2024-01-25T11:45:35.5140977Z level=debug msg="uploaded file" from=/tmp/TestShipper_DeceivingUploadErrors1636679149/001/00000000010000000000000000/index dst=00000000010000000000000000/index bucket="fs: /tmp/TestShipper_DeceivingUploadErrors1636679149/002"
2024-01-25T11:45:35.5142258Z level=error msg="uploading new block to long-term storage failed" block=00000000010000000000000000 err="upload meta file: base name matches, will fail upload"
2024-01-25T11:45:35.5142408Z FAIL
2024-01-25T11:45:35.5142702Z FAIL	github.com/grafana/mimir/pkg/ingester	160.028s

ci.log

@dimitarvdimitrov
Copy link
Contributor Author

another instance

logs.txt

@dimitarvdimitrov dimitarvdimitrov mentioned this issue Mar 14, 2024
Merged
4 tasks
@pracucci
Copy link
Collaborator

another instance
logs_25292938557.zip

@zenador
Copy link
Contributor

zenador commented Aug 2, 2024

https://github.com/grafana/mimir/actions/runs/10219303903/job/28278470645#step:8:148

--- FAIL: TestIngester_compactBlocksToReduceInMemorySeries_Concurrency (27.09s)
    --- FAIL: TestIngester_compactBlocksToReduceInMemorySeries_Concurrency/Run_2 (9.03s)
        ingester_early_compaction_test.go:528: 
            	Error Trace:	/__w/mimir/mimir/pkg/ingester/ingester_early_compaction_test.go:528
            	            				/usr/local/go/src/runtime/asm_amd64.s:1695
            	Error:      	Received unexpected error:
            	            	user=1: the sample has been rejected because its timestamp is too old (err-mimir-sample-timestamp-too-old). The affected sample has timestamp 2024-08-02T17:36:01.711Z and is from series series_00010
            	Test:       	TestIngester_compactBlocksToReduceInMemorySeries_Concurrency/Run_2
        ingester_early_compaction_test.go:627: 
            	Error Trace:	/__w/mimir/mimir/pkg/ingester/ingester_early_compaction_test.go:627
            	Error:      	"0" is not greater than "1"
[redacted]

@dimitarvdimitrov
Copy link
Contributor Author

still happening - 11_test (3, 4).txt

@dimitarvdimitrov
Copy link
Contributor Author

another error, which seems a bit scarier: cannot populate chunk 503316480 from block 0000000000XXXXXXXRANGEHEAD: not found

--- FAIL: TestIngester_compactBlocksToReduceInMemorySeries_Concurrency (26.03s)
    --- FAIL: TestIngester_compactBlocksToReduceInMemorySeries_Concurrency/Run_0 (9.01s)
        ingester_early_compaction_test.go:561: 
            	Error Trace:	/__w/mimir/mimir/pkg/ingester/ingester_early_compaction_test.go:561
            	            				/usr/local/go/src/runtime/asm_amd64.s:1700
            	Error:      	Received unexpected error:
            	            	cannot populate chunk 503316480 from block 0000000000XXXXXXXRANGEHEAD: not found
            	Test:       	TestIngester_compactBlocksToReduceInMemorySeries_Concurrency/Run_0
level=info msg="uploading new block to long-term storage" block=00000000010000000000000000
level=debug msg="uploaded file" from=/tmp/TestShipper_DeceivingUploadErrors1680269958/001/00000000010000000000000000/index dst=00000000010000000000000000/index bucket="fs: /tmp/TestShipper_DeceivingUploadErrors1680269958/002"
level=error msg="uploading new block to long-term storage failed" block=00000000010000000000000000 err="upload meta file: base name matches, will fail upload"

@pracucci pracucci mentioned this issue Sep 28, 2024
Merged
4 tasks
@dimitarvdimitrov
Copy link
Contributor Author

another failure; new logs: 8_test (0, 4).txt

@seizethedave
Copy link
Contributor

seizethedave commented Nov 25, 2024 •
edited
Loading

Here's one I received today: (log). Looks like the same cause as most of the ones above.

--- FAIL: TestIngester_compactBlocksToReduceInMemorySeries_Concurrency (42.20s)
    --- FAIL: TestIngester_compactBlocksToReduceInMemorySeries_Concurrency/Run_2 (12.01s)
        ingester_early_compaction_test.go:514: Write worker 0 writing series with IDs between 0 and 9 (both inclusive)
        ingester_early_compaction_test.go:514: Write worker 1 writing series with IDs between 10 and 19 (both inclusive)
        ingester_early_compaction_test.go:514: Write worker 3 writing series with IDs between 30 and 39 (both inclusive)
        ingester_early_compaction_test.go:514: Write worker 4 writing series with IDs between 40 and 49 (both inclusive)
        ingester_early_compaction_test.go:514: Write worker 2 writing series with IDs between 20 and 29 (both inclusive)
        ingester_early_compaction_test.go:529: 
            	Error Trace:	/__w/mimir/mimir/pkg/ingester/ingester_early_compaction_test.go:529
            	            				/usr/local/go/src/runtime/asm_amd64.s:1700
            	Error:      	Received unexpected error:
            	            	user=1: the sample has been rejected because its timestamp is too old (err-mimir-sample-timestamp-too-old). The affected sample has timestamp 2024-11-25T18:22:22.94Z and is from series series_00000
            	Test:       	TestIngester_compactBlocksToReduceInMemorySeries_Concurrency/Run_2
            	Messages:   	worker: 0, sample idx: 0, sample timestamp: 173255[89](https://github.com/grafana/mimir/actions/runs/12016316877/job/33496266244#step:8:90)42940 (2024-11-25 18:22:22.94 +0000 UTC)
        ingester_early_compaction_test.go:630: 
            	Error Trace:	/__w/mimir/mimir/pkg/ingester/ingester_early_compaction_test.go:630
            	Error:      	"0" is not greater than "1"
            	Test:       	TestIngester_compactBlocksToReduceInMemorySeries_Concurrency/Run_2
        ingester_early_compaction_test.go:650: 
            	Error Trace:	/__w/mimir/mimir/pkg/ingester/ingester_early_compaction_test.go:650
            	Error:      	"series_00010 =>
            	            	0 @[1732558[94](https://github.com/grafana/mimir/actions/runs/12016316877/job/33496266244#step:8:95)2.94]
            	            	1 @[1732558942.941]
            	            	2 @[1732558942.942]
            	            	3 @[1732558942.943]
            	            	4 @[1732558942.944]
            	            	5 @[1732558942.945]
            	            	6 @[1732558942.946]
 <snip 1 billion lines>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component/ingester type/tests
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@seizethedave @pracucci @zenador @dimitarvdimitrov