for compiling with cuda 12.x

[mosout]

No description provided.

[github-actions]

View latest API docs preview at: https://oneflow-staging.oss-cn-beijing.aliyuncs.com/docs/Oneflow-Inc/oneflow/pr/10568/

[github-actions]

Speed stats:

GPU Name: NVIDIA GeForce RTX 3080 Ti 

❌ OneFlow resnet50 time: 43.3ms (= 4328.2ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 57.9ms (= 5793.1ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.34 (= 57.9ms / 43.3ms)

OneFlow resnet50 time: 26.5ms (= 2653.1ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 37.5ms (= 3752.4ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.41 (= 37.5ms / 26.5ms)

OneFlow resnet50 time: 18.6ms (= 3723.0ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 35.3ms (= 7067.4ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.90 (= 35.3ms / 18.6ms)

OneFlow resnet50 time: 17.4ms (= 3475.1ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 30.5ms (= 6095.9ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.75 (= 30.5ms / 17.4ms)

OneFlow resnet50 time: 17.6ms (= 3518.8ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 29.2ms (= 5840.6ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.66 (= 29.2ms / 17.6ms)

OneFlow swin dataloader time: 0.200s (= 39.908s / 200, num_workers=1)
PyTorch swin dataloader time: 0.128s (= 25.669s / 200, num_workers=1)
Relative speed: 0.643 (= 0.128s / 0.200s)

OneFlow swin dataloader time: 0.055s (= 11.087s / 200, num_workers=4)
PyTorch swin dataloader time: 0.033s (= 6.563s / 200, num_workers=4)
Relative speed: 0.592 (= 0.033s / 0.055s)

OneFlow swin dataloader time: 0.030s (= 6.007s / 200, num_workers=8)
PyTorch swin dataloader time: 0.017s (= 3.356s / 200, num_workers=8)
Relative speed: 0.559 (= 0.017s / 0.030s)

❌ OneFlow resnet50 time: 49.5ms (= 4951.4ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 64.5ms (= 6450.4ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.30 (= 64.5ms / 49.5ms)

OneFlow resnet50 time: 36.3ms (= 3629.7ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 46.6ms (= 4661.8ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.28 (= 46.6ms / 36.3ms)

OneFlow resnet50 time: 27.6ms (= 5526.7ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 40.2ms (= 8037.5ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.45 (= 40.2ms / 27.6ms)

OneFlow resnet50 time: 25.1ms (= 5029.0ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 38.7ms (= 7738.1ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.54 (= 38.7ms / 25.1ms)

OneFlow resnet50 time: 24.9ms (= 4983.3ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 38.2ms (= 7649.4ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.54 (= 38.2ms / 24.9ms)

[github-actions]

Speed stats:

GPU Name: NVIDIA GeForce RTX 3080 Ti 

❌ OneFlow resnet50 time: 43.8ms (= 4378.3ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 57.4ms (= 5735.7ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.31 (= 57.4ms / 43.8ms)

OneFlow resnet50 time: 26.2ms (= 2620.1ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 37.5ms (= 3752.4ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.43 (= 37.5ms / 26.2ms)

OneFlow resnet50 time: 18.6ms (= 3713.4ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 35.5ms (= 7103.8ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.91 (= 35.5ms / 18.6ms)

OneFlow resnet50 time: 17.5ms (= 3500.9ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 31.0ms (= 6203.5ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.77 (= 31.0ms / 17.5ms)

OneFlow resnet50 time: 17.4ms (= 3473.1ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 31.0ms (= 6192.0ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.78 (= 31.0ms / 17.4ms)

OneFlow swin dataloader time: 0.200s (= 40.089s / 200, num_workers=1)
PyTorch swin dataloader time: 0.130s (= 26.035s / 200, num_workers=1)
Relative speed: 0.649 (= 0.130s / 0.200s)

OneFlow swin dataloader time: 0.058s (= 11.679s / 200, num_workers=4)
PyTorch swin dataloader time: 0.032s (= 6.464s / 200, num_workers=4)
Relative speed: 0.553 (= 0.032s / 0.058s)

OneFlow swin dataloader time: 0.031s (= 6.105s / 200, num_workers=8)
PyTorch swin dataloader time: 0.017s (= 3.306s / 200, num_workers=8)
Relative speed: 0.542 (= 0.017s / 0.031s)

❌ OneFlow resnet50 time: 49.3ms (= 4928.8ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 65.8ms (= 6577.5ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.33 (= 65.8ms / 49.3ms)

OneFlow resnet50 time: 36.9ms (= 3688.3ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 46.3ms (= 4634.0ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.26 (= 46.3ms / 36.9ms)

OneFlow resnet50 time: 27.7ms (= 5531.9ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 39.8ms (= 7951.0ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.44 (= 39.8ms / 27.7ms)

OneFlow resnet50 time: 25.2ms (= 5033.1ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 39.1ms (= 7823.5ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.55 (= 39.1ms / 25.2ms)

OneFlow resnet50 time: 24.7ms (= 4942.4ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 35.8ms (= 7168.7ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.45 (= 35.8ms / 24.7ms)