【AutoParallel】Promote fuselinear pass in auto-parallel

[heavyrain-lzy]

PR types

Performance optimization

PR changes

Others

Description

Pcard-76459
Add a pass before the fused_linear pass to promote the performance. This pass mainly solve the following scenario when enabling the MP or TP parallelism:

• The origin linear operator as follows:

matmul --> add

• After enabling MPor TP, some linear operators may become as follows:

matmul --> comm_op --> add

The communication operator prohibits the matmul and add from being fused.

experiment

Take the experiment on the GPT-3 with 6.7B parameters using the single host with 8 V100 GPUs:

strategy	tokens/s	speed change
no-FusedLinear	2166	BaseLine
FusedLinear	2203	1.71%
FusedLinear-Promotion	2215	2.26%

[paddle-bot]

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

[zhiqiu]

Add an unittest

[heavyrain-lzy]

Add an unittest

The unittest 'test_auto_parallel_fused_linear_promotion_pass.py' has been added. But the machine of CI/CE can't run the test, because the 'fused_gemm_epilogue' requires the CUDA11.6 at least. I have run the test by myself:

[zhiqiu]

LGTM

[XiaoguangHu01]

LGTM