Codes submitted initially

.gitignore

@@ -1,3 +1,8 @@
+# ignore directory
+runs/
+.idea/
+
+
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[cod]

README.md

@@ -0,0 +1,54 @@
+## RECCE CVPR 2022
+
+:page_facing_up: End-to-End Reconstruction-Classification Learning for Face Forgery Detection
+
+:boy: Junyi Cao, Chao Ma, Taiping Yao, Shen Chen, Shouhong Ding, Xiaokang Yang
+
+**Please consider citing our paper if you find it interesting or helpful to your research.**
+```
+Bibtex will come up soon ...
+```
+
+----
+
+![RECCE Framework](figure/framework.png)
+
+#### Introduction
+
+This repository is an implementation for *End-to-End Reconstruction-Classification Learning for Face Forgery Detection* presented in CVPR 2022. In the paper, we propose a novel **REC**onstruction-**C**lassification l**E**arning framework called **RECCE** to detect face forgeries. The code is based on Pytorch. Please follow the instructions below to get started.
+
+
+#### Basic Requirements
+Please ensure that you have already installed the following packages.
+- [Pytorch](https://pytorch.org/get-started/previous-versions/) 1.7.1
+- [Torchvision](https://pytorch.org/get-started/previous-versions/) 0.8.2
+- [Albumentations](https://github.com/albumentations-team/albumentations#spatial-level-transforms) 1.0.3
+- [Timm](https://github.com/rwightman/pytorch-image-models) 0.3.4
+- [TensorboardX](https://pypi.org/project/tensorboardX/#history) 2.1
+- [Scipy](https://pypi.org/project/scipy/#history) 1.5.2
+- [PyYaml](https://pypi.org/project/PyYAML/#history) 5.3.1
+
+#### Dataset Preparation
+- We include the dataset loaders for several commonly-used face forgery datasets, *i.e.,* [FaceForensics++](https://github.com/ondyari/FaceForensics), [Celeb-DF](https://www.cs.albany.edu/~lsw/celeb-deepfakeforensics.html), [WildDeepfake](https://github.com/deepfakeinthewild/deepfake-in-the-wild), and [DFDC](https://ai.facebook.com/datasets/dfdc). You can enter the dataset website to download the original data.
+- For FaceForensics++, Celeb-DF, and DFDC, since the original data are in video format, you should first extract the facial images from the sequences and store them. We use [RetinaFace](https://github.com/biubug6/Pytorch_Retinaface) to do this.
+
+#### Config Files
+- We have already provided the config templates in `config/`. You can adjust the parameters in the yaml files to specify a training process. More information is presented in [config/README.md](./config/README.md).
+
+#### Training
+- We use `torch.distributed` package to train the models, for more information, please refer to [PyTorch Distributed Overview](https://pytorch.org/tutorials/beginner/dist_overview.html).
+- To train a model, run the following script in your console. 
+```{bash}
+CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --nproc_per_node=1 --master_port 12345 train.py --config path/to/config.yaml
+```
+- `--config`: Specify the path of the config file. 
+
+#### Testing
+- To test a model, run the following script in your console. 
+```{bash}
+python test.py --config path/to/config.yaml
+```
+- `--config`: Specify the path of the config file.
+
+#### Acknowledgement
+- We thank Qiqi Gu for helping plot the schematic diagram of the proposed method in the manuscript.

config/README.md

@@ -0,0 +1,44 @@
+## Configuration Files
+
+#### Model  Configuration
+- We use a yaml file to specify the hyperparameters of a model. All the training logs will be placed in `${project_root}/runs/${model_name}/${experiment_id}`. An example are shown below.
+
+```yaml
+model:
+  name: Recce  # Model Name
+  num_classes: 1 
+config:
+  lambda_1: 0.1  # balancing weight for L_r
+  lambda_2: 0.1  # balancing weight for L_m
+  distribute:
+    backend: nccl
+  optimizer:
+    name: adam
+    lr: 0.0002
+    weight_decay: 0.00001
+  scheduler:
+    name: StepLR
+    step_size: 22500
+    gamma: 0.5
+  resume: False
+  resume_best: False
+  id: FF++c40  # Specify a unique experiment id.
+  loss: binary_ce  # Loss type, either 'binary_ce' or 'cross_entropy'.
+  metric: Acc  # Main metric, either 'Acc', 'AUC', or 'LogLoss'.
+  debug: False
+  device: "cuda:1"  # NOTE: Used only when testing, annotation this line when training.
+  ckpt: best_model_1000  # NOTE: Used only when testing to specify a checkpoint id, annotating this line when training.
+data:
+  train_batch_size: 32
+  val_batch_size: 64
+  test_batch_size: 64
+  name: FaceForensics
+  file: "./config/dataset/faceforensics.yml"  # config file for a dataset
+  train_branch: "train_cfg"
+  val_branch: "test_cfg"
+  test_branch: "test_cfg"
+```
+
+#### Dataset Configuration
+- We also use a yaml file to specify the dataset to load for the experiment. These files are placed under `config/dataset/` subfold.
+- Briefly, you should change the `root` parameter according to your storage path. 

config/Recce.yml

@@ -0,0 +1,33 @@
+model:
+  name: Recce
+  num_classes: 1
+config:
+  lambda_1: 0.1
+  lambda_2: 0.1
+  distribute:
+    backend: nccl
+  optimizer:
+    name: adam
+    lr: 0.0002
+    weight_decay: 0.00001
+  scheduler:
+    name: StepLR
+    step_size: 22500
+    gamma: 0.5
+  resume: False
+  resume_best: False
+  id: FF++c40
+  loss: binary_ce
+  metric: Acc
+  debug: False
+#  device: "cuda:1"
+#  ckpt: best_model_1000
+data:
+  train_batch_size: 32
+  val_batch_size: 64
+  test_batch_size: 64
+  name: FaceForensics
+  file: "./config/dataset/faceforensics.yml"
+  train_branch: "train_cfg"
+  val_branch: "test_cfg"
+  test_branch: "test_cfg"

config/dataset/celeb_df.yml

@@ -0,0 +1,32 @@
+train_cfg:
+  root: "path/to/data"
+  split: "train"
+  balance: True
+  log_steps: 1000
+  val_steps: 1000
+  num_steps: 92000
+  transforms:
+    - name: "Resize"
+      params:
+        height: 299
+        width: 299
+    - name: "HorizontalFlip"
+      params:
+        p: 0.5
+    - name: "Normalize"
+      params:
+        mean: [0.5, 0.5, 0.5]
+        std: [0.5, 0.5, 0.5]
+test_cfg:
+  root: "path/to/data"
+  split: "test"
+  balance: False
+  transforms:
+    - name: "Resize"
+      params:
+        height: 299
+        width: 299
+    - name: "Normalize"
+      params:
+        mean: [0.5, 0.5, 0.5]
+        std: [0.5, 0.5, 0.5]

config/dataset/dfdc.yml

@@ -0,0 +1,30 @@
+train_cfg:
+  root: "path/to/data"
+  split: "train"
+  log_steps: 1000
+  val_steps: 1000
+  num_steps: 100000
+  transforms:
+    - name: "Resize"
+      params:
+        height: 299
+        width: 299
+    - name: "HorizontalFlip"
+      params:
+        p: 0.5
+    - name: "Normalize"
+      params:
+        mean: [0.5, 0.5, 0.5]
+        std: [0.5, 0.5, 0.5]
+test_cfg:
+  root: "path/to/data"
+  split: "test"
+  transforms:
+    - name: "Resize"
+      params:
+        height: 299
+        width: 299
+    - name: "Normalize"
+      params:
+        mean: [0.5, 0.5, 0.5]
+        std: [0.5, 0.5, 0.5]

config/dataset/faceforensics.yml

@@ -0,0 +1,34 @@
+train_cfg:
+  root: "path/to/data"
+  split: "train"
+  method: "all"
+  compression: "c40"
+  log_steps: 1000
+  val_steps: 1000
+  num_steps: 90000
+  transforms:
+    - name: "Resize"
+      params:
+        height: 299
+        width: 299
+    - name: "HorizontalFlip"
+      params:
+        p: 0.5
+    - name: "Normalize"
+      params:
+        mean: [0.5, 0.5, 0.5]
+        std: [0.5, 0.5, 0.5]
+test_cfg:
+  root: "path/to/data"
+  split: "test"
+  method: "all"
+  compression: "c40"
+  transforms:
+    - name: "Resize"
+      params:
+        height: 299
+        width: 299
+    - name: "Normalize"
+      params:
+        mean: [0.5, 0.5, 0.5]
+        std: [0.5, 0.5, 0.5]

config/dataset/wilddeepfake.yml

@@ -0,0 +1,32 @@
+train_cfg:
+  root: "path/to/data"
+  split: "train"
+  num_image_train:
+  log_steps: 200
+  val_steps: 200
+  num_steps: 40000
+  transforms:
+    - name: "Resize"
+      params:
+        height: 224
+        width: 224
+    - name: "HorizontalFlip"
+      params:
+        p: 0.5
+    - name: "Normalize"
+      params:
+        mean: [0.5, 0.5, 0.5]
+        std: [0.5, 0.5, 0.5]
+test_cfg:
+  root: "path/to/data"
+  split: "test"
+  num_image_test:
+  transforms:
+    - name: "Resize"
+      params:
+        height: 224
+        width: 224
+    - name: "Normalize"
+      params:
+        mean: [0.5, 0.5, 0.5]
+        std: [0.5, 0.5, 0.5]

dataset/__init__.py

@@ -0,0 +1,17 @@
+from .abstract_dataset import AbstractDataset
+from .faceforensics import FaceForensics
+from .wild_deepfake import WildDeepfake
+from .celeb_df import CelebDF
+from .dfdc import DFDC
+
+LOADERS = {
+    "FaceForensics": FaceForensics,
+    "WildDeepfake": WildDeepfake,
+    "CelebDF": CelebDF,
+    "DFDC": DFDC,
+}
+
+
+def load_dataset(name="FaceForensics"):
+    print(f"Loading dataset: '{name}'...")
+    return LOADERS[name]

dataset/abstract_dataset.py

@@ -0,0 +1,41 @@
+import cv2
+import torch
+import numpy as np
+from torchvision.datasets import VisionDataset
+import albumentations
+from albumentations import Compose
+from albumentations.pytorch.transforms import ToTensorV2
+
+
+class AbstractDataset(VisionDataset):
+    def __init__(self, cfg, seed=2022, transforms=None, transform=None, target_transform=None):
+        super(AbstractDataset, self).__init__(cfg['root'], transforms=transforms,
+                                              transform=transform, target_transform=target_transform)
+        # fix for re-production
+        np.random.seed(seed)
+
+        self.images = list()
+        self.targets = list()
+        self.split = cfg['split']
+        if self.transforms is None:
+            self.transforms = Compose(
+                [getattr(albumentations, _['name'])(**_['params']) for _ in cfg['transforms']] +
+                [ToTensorV2()]
+            )
+
+    def __len__(self):
+        return len(self.images)
+
+    def __getitem__(self, index):
+        path = self.images[index]
+        tgt = self.targets[index]
+        return path, tgt
+
+    def load_item(self, items):
+        images = list()
+        for item in items:
+            img = cv2.imread(item)
+            img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
+            image = self.transforms(image=img)['image']
+            images.append(image)
+        return torch.stack(images, dim=0)