[Semi AutoParall] Support Partial Semantic I

[JZ-LIANG]

PR types

New features

PR changes

Others

Description

Pcard-70448

Background
"Partial" is a data distribution type for tensor like "Replicated" and "Sharded".
A tensor is Partial means that: its shape is the same among ranks, but the local element value in each rank is only a partial of the global value, and reduced (sum/max/min/all/any) op over ranks is need to rebuild the global tensor from the locals.

Motivation
Before the PR, dist_tensor in auto parallel only has two type: "Replicated" and "Sharded". And it work quite well for most hybrid parallelism scenario. It is just a design choice to introduce a third distribution type "Partial".

1. "Partial" make it easier to explain why there need an Allreduce in Tensor Parallel (Vocab Parallel Embedding, Forward of Row Parallel Linear, Backward of Col Parallel Linear, etc ) for user and developer.
2. "Partial" make some Optimizations easier in Dynamic Graph mode of Auto Parallel. (Previously, Auto Parallel Only has Static Graph mode, and we could delay any Allreduce by Graph Transformation Pass. But we need "Partial" semantic for these optimizations in Dynamic Graph mode)
   

HOW
The Introducing of Partial is divided into two Stages.

First Stage(This PR):

• Introduce partial but not allow its propagation in computation graph.
• DistTensorAttribute will add new data member to maintain Partial related info.
• Op's inputs will always be non-partial.
• If an Op generate a Partial output, the partial will clean by Reshard(Redistribute) immediately.
• Meet the need of Motivation-I

Second Stage(Future PR):

• Allow partial to propagate in computation graph.
• User is allowed to mark tensor as Partial for Optimization.
• Op with linearity property will propagate the partial status from input to it output.
• Reshard(Partial --> Replicated) would only be called before the last non-linearity Op who consumes that tensor.
• Meet the need of Motivation-II

[paddle-bot]

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

[pkuzyc]

Why use a map structure here? If the "dim_" in Partial indicates the mesh dim, it seems unnecessary to store another mesh dim. In addition, if one tensor has only one reduce type, is it better to use a data structure like:
Partial {
vector<int64_t> mesh_dims;
ReduceType type_;
}

[JZ-LIANG]

Correct ！
firstly I thought we would use 『set』 for partial_status_, so build the Partial struct.
then I found『map』would be better for partial_status_, in most of use cases we use dim as key to retrieve Partial .

[zhiqiu]

LGTM

[zhiqiu]

ALL means?