Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix the modification of set_expected_place #31177

Merged
merged 3 commits into from
Feb 24, 2021

Conversation

zhiqiu
Copy link
Contributor

@zhiqiu zhiqiu commented Feb 24, 2021

PR types

Bug fixes

PR changes

Others

Describe

revert the modification of set_expected_place

I add cudaSetDevice() in SetExpectedPlace in dygraph mode in PR #30338, and the logic is updated in #30868.

This PR revert the changes since in multi-process dataloader, using CUDA APIs in subprocess may result in error since the
cuda runtime is not initialized.

image

So this PR revert that changes, and the problem in PR #30338 can be fixed by adding DeviceGuard in SetTensorFromPyArrayT, which is already done in #30868.

Reproduce code:

import os
import numpy as np
import paddle
import paddle.nn.functional as F
from paddle.io import Dataset, DataLoader, DistributedBatchSampler

class FakeDataset(Dataset):
    def __init__(self, num_samples=1000):
        self.num_samples = num_samples

    def __getitem__(self, idx):
        data1 = np.random.rand(3, 32, 32).astype("float32")
        data2 = np.random.rand(3, 32, 32).astype("float32")
        return [data1, data2]
        
    def __len__(self):
        return self.num_samples


def collate_fn(batch):
    with paddle.fluid.dygraph.guard(place=paddle.CPUPlace()):
        width_list = np.array(
            [image_set_i[0].shape[2] for image_set_i in batch])
        avg_width = int(width_list.mean() / 8 / 2) * 8
        r_imgs = [[] for _ in batch[0]]
        for img_set in batch:
            for j, img in enumerate(img_set):
                img = paddle.to_tensor(
                    np.expand_dims(img, 0), place=paddle.CPUPlace())
                img = F.upsample(img, size=(32, avg_width), mode='bilinear')
                img = paddle.squeeze(img, axis=0)
                r_imgs[j].append(img.numpy())
    return r_imgs

if __name__ == "__main__":
    place = paddle.set_device('gpu')
    dataset = FakeDataset()
    num_workers = 1

    loader = DataLoader(
            dataset,
            batch_size=8,
            collate_fn=collate_fn,
            places=place,
            return_list=True,
            num_workers=num_workers)

    print("finish init dataloader")
    for batch in loader:
        for data in batch:
            print(data.shape)
        exit()
    

@paddle-bot-old
Copy link

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

@zhiqiu zhiqiu force-pushed the dev/refine_set_expected_place branch from bbe0c32 to 24d56a0 Compare February 24, 2021 03:01
@zhiqiu zhiqiu requested a review from wangxicoding February 24, 2021 03:29
wangxicoding
wangxicoding previously approved these changes Feb 24, 2021
Copy link
Contributor

@wangxicoding wangxicoding left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhiqiu zhiqiu changed the title revert the modification of set_expected_place fix the modification of set_expected_place Feb 24, 2021
@zhiqiu zhiqiu merged commit 0f1fde5 into PaddlePaddle:develop Feb 24, 2021
zhiqiu added a commit to zhiqiu/Paddle that referenced this pull request Feb 24, 2021
* revert the modification of set_expected_place

* set device before op run

* add ut
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants