JIT: Optimize epilog generation

[JosephTremoulet]

Currently, the JIT's strategy for epilog generation is to generate one epilog per ret in the MSIL, except that:

• it checks for a few constructs needing epilog instrumentation, and generates only one epilog if they apply
• x86 GC info has a limit of 4 epilogs, which is, questionably, enforced for all targets

The analysis and transformation for both cases runs early in the phase list and merges the GT_RETURN nodes when returns are merged.

There's an opportunity here to take a more principled approach weighing the codesize reduction of merged epilogs against the dynamic path length reduction of duplicated epilogs.

One potential approach:

• Keep GT_RETURNs separated through optimization phases to allow a chance for constant propagation and CSE to eliminate redundancies in the return expressions, and allow removing code like we have in loop cloning and loop unrolling that currently shares the burden of respecting the max epilog limit.
• Decide later, ideally when we have an idea both how much codesize each epilog will require and how frequently we expect each return site to execute, how many epilogs to generate and which return sites should share; presumably this could be done at or before the time that the actual LIR for the epilogs is inserted.
• Improve the sharing by looking at the sequences leading up to the returns and the returned values and tail-merge as appropriate; this improves codesize when the same value or variable is returned from multiple return sites, and can remove some branches from the dynamic code paths when the merging is able to swallow up entire blocks (which isn't uncommon for constructs like if (...) { return false; }).

category:cq
theme:basic-cq
skill-level:expert
cost:large

[AndyAyersMS]

Also see #8406