[multiprocessing] Add paddle.incubate.multiprocessing for sharing tensors between python processes. #37302

ZHUI · 2021-11-17T09:59:37Z

PR types

New features

PR changes

Others

Describe

Add paddle.incubate.multiprocessing for tensor sharing between python process.

This PR aims give an initial version of paddle.incubate.multiprocessing which support both CPU and GPU tensor.
Here are some TODOs for this PR. The full RoadMap is at the end of this PR。

Support FlickerPickle for tensor passing
Support passing CPU tensor on linux
Support passing CUDA tensor using cudaIpcMemHandle on linux
Full test of all tensor behavior

Useage

import paddle
import paddle.multiprocessing as mp

paddle.set_device("cpu")

def fill_tensor(queue, event):
    data = queue.get()
    data[:] = 5
    event.set()

tensor = paddle.zeros([5, 5], dtype="float32")
queue = mp.Queue()
event = mp.Event()
queue.put(tensor)

process = ctx.Process(target=fill_tensor, args=(queue, event))
process.daemon = True
process.start()
event.wait(30)
print(tensor)
# Tensor(shape=[5, 5], dtype=float32, place=CPUPlace, stop_gradient=True,
#        [[5., 5., 5., 5., 5.],
#         [5., 5., 5., 5., 5.],
#         [5., 5., 5., 5., 5.],
#         [5., 5., 5., 5., 5.],
#         [5., 5., 5., 5., 5.]])
process.join(1)

Paddle.multiprocessing RoadMap

Sharing CPU Tensor.

Support global reference counts for IPC CPU tensor

platform

linux
mac
win32

other

support file descriptor mode on linux

Sharing GPU Tensor.

Support cudaIpcMemHandle GPU tensor
Support global reference counts for IPC GPU tensor.

other

Support ROCM
Support cudaIpcEventHandle

Influence on Paddle Framework

Application

Support DataLoader

paddle-bot-old · 2021-11-29T02:35:24Z

Sorry to inform you that 2f5a055's CIs have passed for more than 7 days. To prevent PR conflicts, you need to re-run all CIs manually.

wawltor · 2022-03-04T01:38:52Z

paddle/fluid/memory/allocation/CMakeLists.txt

@@ -131,4 +131,7 @@ cc_library(virtual_memory_auto_growth_best_fit_allocator SRCS virtual_memory_aut
 if(NOT WIN32)
  cc_library(mmap_allocator SRCS mmap_allocator.cc DEPS allocator)
  cc_test(mmap_allocator_test SRCS mmap_allocator_test.cc DEPS mmap_allocator allocator)
+  if (WITH_GPU)


这块需要限制成非WIN32吗？下面cuda_ipc_allocator在define的时候限制了非WIN32

没有尝试，cmake里这么写，也可以减少 WIN32下的编译问题

paddle/fluid/memory/allocation/cuda_ipc_allocator.cc

paddle/fluid/memory/allocation/cuda_ipc_allocator.h

paddle/fluid/pybind/pybind.cc

python/paddle/incubate/multiprocessing/reductions.py

wawltor · 2022-03-04T06:49:03Z

python/paddle/incubate/multiprocessing/reductions.py

+    if tensor.place.is_cpu_place() or tensor.place.is_gpu_place(
+    ) or tensor.place.is_cuda_pinned_place():
+        if type(tensor) == paddle.fluid.framework.ParamBase:
+            metadata = copy.deepcopy(tensor.__dict__)


这里有个疑问，为啥知识对Parameter类型进行deep copy了？或者说这里对persisable类型要进行deep copy

因为 Parameter 的一些属性比较多，对于普通的Tensor我们只单独记录了 stop_gradients 属性，

wawltor

LGTM

XieYunshen

LGTM for set_tests_properties(test_paddle_multiprocessing PROPERTIES TIMEOUT 120)

wanghuancoder

LGTM for skipIf

XiaoguangHu01

LGTM

ZHUI · 2023-02-09T12:31:34Z

相关设计文档 paddle进程间tensor传输设计文档 paddle.multiprocessing

ZHUI force-pushed the multiprocessing branch from 18c3c0b to 2f5a055 Compare November 17, 2021 11:09

paddle-bot-old bot referenced this pull request Nov 17, 2021

init commit for paddle.multiprocessing

2f5a055

PaddlePaddle locked and limited conversation to collaborators Nov 21, 2021

PaddlePaddle unlocked this conversation Nov 21, 2021

paddle-bot-old bot referenced this pull request Dec 22, 2021

handle merge conflict.

51c49ac

paddle-bot-old bot referenced this pull request Jan 6, 2022

add support for CUDAPinnedPlace.

85f807d

paddle-bot-old bot referenced this pull request Jan 20, 2022

delete comments.

b04990b

Add support for paddle.multiprocessing

48747ba

ZHUI force-pushed the multiprocessing branch from b04990b to 48747ba Compare January 20, 2022 06:28

ZHUI added 2 commits January 25, 2022 03:32

Merge remote-tracking branch 'upstream/develop' into multiprocessing

9397b94

fix bugs

90b5937

PaddlePaddle locked and limited conversation to collaborators Jan 25, 2022

PaddlePaddle unlocked this conversation Jan 25, 2022

fix compile bugs

c2b9ccd

PaddlePaddle locked and limited conversation to collaborators Jan 26, 2022

PaddlePaddle unlocked this conversation Jan 26, 2022

ZHUI added 3 commits January 26, 2022 08:38

fix compile of cpu only

1d5c3e1

fix typos

9faecd2

fix cmake.

64ba056

PaddlePaddle deleted a comment from paddle-bot-old bot Jan 27, 2022

not init multiprocessing by default.

88a0008

Shixiaowei02 previously approved these changes Feb 14, 2022

View reviewed changes

fix bugs and improve convergence rate.

c6d6b8f

ZHUI dismissed Shixiaowei02’s stale review via c6d6b8f February 15, 2022 05:31

ZHUI marked this pull request as ready for review February 15, 2022 05:31

ZHUI added 3 commits February 15, 2022 05:41

Merge remote-tracking branch 'upstream/develop' into multiprocessing

c362033

fix merge conflict.

e6cd2f5

Merge remote-tracking branch 'upstream/develop' into multiprocessing

c196c65

ZHUI added 2 commits February 21, 2022 11:21

fix merge issues.

4877bba

move multiprocessing to incubate.

ffb3e21

ZHUI requested a review from wawltor February 28, 2022 13:03

ZHUI changed the title ~~[multiprocessing] Add paddle.multiprocessing for sharing tensors between python processes.~~ [multiprocessing] Add paddle.incubate.multiprocessing for sharing tensors between python processes. Feb 28, 2022

ZHUI removed the request for review from wawltor March 3, 2022 09:18

wawltor reviewed Mar 4, 2022

View reviewed changes

ZHUI added 3 commits March 10, 2022 02:37

Merge remote-tracking branch 'origin/develop' into multiprocessing

1bcb3df

fix as reviews.

63d0638

bugfix

2fcc4bb

wawltor approved these changes Mar 10, 2022

View reviewed changes

ZHUI requested a review from XiaoguangHu01 March 10, 2022 06:06

XieYunshen approved these changes Mar 10, 2022

View reviewed changes

wanghuancoder approved these changes Mar 10, 2022

View reviewed changes

ZHUI requested a review from zhiqiu March 10, 2022 06:44

Shixiaowei02 approved these changes Mar 10, 2022

View reviewed changes

XiaoguangHu01 approved these changes Mar 14, 2022

View reviewed changes

ZHUI merged commit e553f75 into PaddlePaddle:develop Mar 14, 2022

ZHUI mentioned this pull request Jan 10, 2023

support DataLoader with multi-process mode on MacOs and Windows basically #35854

Closed

cloud2009 mentioned this pull request Feb 20, 2023

【PaddlePaddle Hackathon 4】核心框架开源贡献其他任务合集 #50663

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[multiprocessing] Add paddle.incubate.multiprocessing for sharing tensors between python processes. #37302

[multiprocessing] Add paddle.incubate.multiprocessing for sharing tensors between python processes. #37302

ZHUI commented Nov 17, 2021 •

edited

Loading

paddle-bot-old bot commented Nov 29, 2021

wawltor Mar 4, 2022

ZHUI Mar 4, 2022

wawltor Mar 4, 2022

ZHUI Mar 4, 2022

wawltor left a comment

XieYunshen left a comment

wanghuancoder left a comment

XiaoguangHu01 left a comment

ZHUI commented Feb 9, 2023

[multiprocessing] Add paddle.incubate.multiprocessing for sharing tensors between python processes. #37302

[multiprocessing] Add paddle.incubate.multiprocessing for sharing tensors between python processes. #37302

Conversation

ZHUI commented Nov 17, 2021 • edited Loading

PR types

PR changes

Describe

Useage

Paddle.multiprocessing RoadMap

Sharing CPU Tensor.

Sharing GPU Tensor.

Influence on Paddle Framework

Application

paddle-bot-old bot commented Nov 29, 2021

wawltor Mar 4, 2022

Choose a reason for hiding this comment

ZHUI Mar 4, 2022

Choose a reason for hiding this comment

wawltor Mar 4, 2022

Choose a reason for hiding this comment

ZHUI Mar 4, 2022

Choose a reason for hiding this comment

wawltor left a comment

Choose a reason for hiding this comment

XieYunshen left a comment

Choose a reason for hiding this comment

wanghuancoder left a comment

Choose a reason for hiding this comment

XiaoguangHu01 left a comment

Choose a reason for hiding this comment

ZHUI commented Feb 9, 2023

ZHUI commented Nov 17, 2021 •

edited

Loading