[meta] Pipeline Java Execution important issues

[colinsurprenant]

This is a meta issue about some important Pipeline Java Execution issues.

Priority Issues

• Java execution may start inputs before compilation of filters and outputs has completed Java execution may start inputs before compilation of filters and outputs has completed. #11170 - Fixed in start inputs only when all WorkerLoop are fully initialized #11492
• Java execution reorders events even with 1 pipeline worker Java execution reorders events even with 1 pipeline worker #10938
  • Abandoned tentative fix Preserve input event ordering with single pipeline worker #11020
  • Incomplete POC [POC][DISCUSS] preserve batched events order #11099 (does not solve the [complex aggregate filter scenario]
    (https://gist.github.com/yaauie/82e199687c490a2e04fe1f0c9d9a7fee))
  • New solution PR Add support for pipeline.ordered setting for java execution #11524
• Java execution startup is potentially slower for some configurations
• fix for regression introduced by Move class caching from ComputeStepSyntaxElement to CompiledPipeline #11482 with revert #11482 and fix redundant code generation #11564

Followup Issues

• improve pipeline.ordered setting to rebatch before outputs improve pipeline.ordered setting to rebatch before outputs #11550 - Fixed in Rebatch after filters when using pipeline.ordered #11710
• Document event ordering guaranties or lack thereof with multiple worker versus single worker - [Doc]Add section for conceptual info #11715
• Add integration harness around the expected compilation time which should be similar between single or multiple workers. Add integration test on JEE expected compilation time  #12476

Other issues

• quoted plugin options crashes logstash When using Java Execution Engine, using quoted plugin directives crashes Logstash #11222 - Fixed in support quoted plugin option keys #11688

[colinsurprenant]

I have looked into the "Java execution startup is potentially slower for some configurations #11105" issue and as I commented in that issue, it is not potentially slower, the slowdown is in fact a factor of the number of workers, see #11105 (comment).

I will continue investigating this.

[colinsurprenant]

I have created 2 PRs with alternate implementations to solve the class compilation caching problem leading to recompiling everything per worker; the first #11479 uses the existing global cache but fixes the cache key to correctly reuse compiled classes. There are some potential problems with pipeline reloading and cache invalidation with this implementation and another PR #11482 uses a different strategy by moving the class caching at the pipeline level which solves both the potential invalidation and pipeline reloading problems.

[colinsurprenant]

Good news on the inputs starting before the workers are initialized: it is easier than I thought to fix, now just need to see if we make that configurable. Will push PR soon.

[colinsurprenant]

#11482 was merged and should considerably improve java execution pipeline compilation time as it will not be multiplied by the number of workers anymore. Fix will be included in 7.6.0 and 7.5.2.

[colinsurprenant]

PR to fix worker initialization sequence in #11492. Should we make that behaviour optional?

[colinsurprenant]

#11492 is merged and will be included in 7.6.0 and 7.5.2. We decided that it was indeed a bug fix and that adding an option to eagerly start inputs before workers are fully initialized was a feature to evaluate in #11493

[colinsurprenant]

New solution proposal for the event ordering #11524

[colinsurprenant]

#11482 addressed the multiple workers slowdown but introduced in 7.5.2 a regression for the multiple pipelines use-case. This PR will be reverted and replaced by #11564 which should correctly solve both problems.

[colinsurprenant]

About

Document event ordering guaranties or lack thereof with multiple worker versus single worker

I am wondering where in the docs we should introduce/document this subject. It does not feel right IMO to simply add a snipet un the settings file. /cc @karenzone @jsvd

[karenzone]

We're building out two general sections in Troubleshooting and Tips and Best Practices. It seems like this new info is similar in nature, even though it doesn't fit neatly under either of those current categories. If we renamed Tips and Best Practices or created a new section describing how things work, I'll bet we'd find other stuff to add.

[colinsurprenant]

Yeah, I like the idea of having a more general "concepts" section? we could then reference that in the settings docs for example.

[karenzone]

I like the idea of keeping the more general info (Troubleshooting, Tips and Best Practices, and a new concepts section) together. On the other hand, it seems like a new conceptual section makes more sense in "How Logstash Works." If I expanded that topic, I'd expect to see information such as what you want to add.

Under "How Logstash Works" (Pros and Cons):
+ Higher in book structure
+ Seems like a natural fit as part of "How Logstash Works"
- Level 2 topic nested under a Level 1 topic ("How Logstash Works"), and won't be visible unless somebody clicks to expand the Level 1 topic

A new section grouped with Troubleshooting and Tips and Best Practices (Pros and Cons):
+ Keeps general, conceptual information grouped together
+ New section could be a Level 1 heading, always visible
- The title "How Logstash Works" is already taken, so we'd need to come up with something else

WDYT?

UPDATED:
Here's a DRAFT PR for consideration/discussion: #11581

[colinsurprenant]

The rebatching after filters #11710 which improves the pipeline.ordered performance has been merged and will be available starting in 7.7.0.

[colinsurprenant]

Closing this, all important work was done, one remaining testing issue has been created #12476