Working commit

.gitignore

@@ -0,0 +1,2 @@
+*.pyc 
+

CODE_OF_CONDUCT.md

@@ -0,0 +1,5 @@
+# Code of Conduct
+
+Facebook has adopted a Code of Conduct that we expect project participants to adhere to.
+Please read the [full text](https://code.fb.com/codeofconduct/)
+so that you can understand what actions will and will not be tolerated.

CONTRIBUTING.md

@@ -0,0 +1,34 @@
+# Contributing to Fashion++
+We want to make contributing to this project as easy and transparent as
+possible.
+
+## Our Development Process
+Minor changes and improvements will be released on an ongoing basis.
+
+## Pull Requests
+We actively welcome your pull requests.
+
+1. Fork the repo and create your branch from `master`.
+2. If you've added code that should be tested, add tests.
+3. If you've changed APIs, update the documentation.
+4. Ensure the test suite passes.
+5. Make sure your code lints.
+6. If you haven't already, complete the Contributor License Agreement ("CLA").
+
+## Contributor License Agreement ("CLA")
+In order to accept your pull request, we need you to submit a CLA. You only need
+to do this once to work on any of Facebook's open source projects.
+
+Complete your CLA here: <https://code.facebook.com/cla>
+
+## Issues
+We use GitHub issues to track public bugs. Please ensure your description is
+clear and has sufficient instructions to be able to reproduce the issue.
+
+Facebook has a [bounty program](https://www.facebook.com/whitehat/) for the safe
+disclosure of security bugs. In those cases, please go through the process
+outlined on that page and do not file a public issue.
+
+## License
+By contributing to Fashion++, you agree that your contributions will be licensed
+under the LICENSE file in the root directory of this source tree.

README.md

@@ -0,0 +1,138 @@
+# Fashion++: Minimal Edits for Outfit Improvement
+[Wei-Lin Hsiao](http://www.cs.utexas.edu/~kimhsiao/),
+[Isay Katsman](https://vision.cornell.edu/se3/people/isay-katsman/)\*,
+[Chao-Yuan Wu](https://www.cs.utexas.edu/~cywu/)\*,
+[Devi Parikh](https://www.cc.gatech.edu/~parikh/),
+[Kristen Grauman](http://www.cs.utexas.edu/~grauman/)<br/>
+In ICCV 2019.  [[paper]](https://arxiv.org/abs/1904.09261)
+
+![concept figure](figures/demo_high_res.gif)
+
+## Installation
+This project is tested on Ubuntu 18.04, with python 3.6.4, pytorch 0.4
+* Install pytorch >= 0.4 following instructions from https://pytorch.org/
+* Install python library dominate
+```
+pip install dominate
+```
+* Install python library opencv
+```
+pip install opencv-python
+```
+* Clone this repo:
+```
+git clone https://github.com/facebookresearch/FashionPlus.git
+```
+## Demo
+1. Download pretrained models from:
+
+   | Model | file name   |                           link                                |
+   |-------|-------------|---------------------------------------------------------------|
+   | cGAN  | latest_net_E.pth|[model](https://dl.fbaipublicfiles.com/FashionPlus/generators/latest_net_E.pth)|
+   | cGAN  | latest_net_G.pth|[model](https://dl.fbaipublicfiles.com/FashionPlus/generators/latest_net_G.pth)|
+   | cGAN  | latest_net_D.pth|[model](https://dl.fbaipublicfiles.com/FashionPlus/generators/latest_net_D.pth)|
+   | cGAN  | train_avg_features.p|[features](https://dl.fbaipublicfiles.com/FashionPlus/generators/train_avg_features.p)|
+   | VAE   | latest_Decoder.pth|[model](https://dl.fbaipublicfiles.com/FashionPlus/VAE/latest_Decoder.pth)|
+   | VAE   | latest_Separate_encoder.pth|[model](https://dl.fbaipublicfiles.com/FashionPlus/VAE/latest_Separate_encoder.pth)|
+   | VAE   | latest_Together_encoder.pth|[model](https://dl.fbaipublicfiles.com/FashionPlus/VAE/latest_Together_encoder.pth)|
+   | classifier | 120_classifier.pth|[model](https://dl.fbaipublicfiles.com/FashionPlus/classifier/120_classifier.pth)|
+
+   Place all but `120_classifer.pth` in `checkpoint/humanparsing/`.\
+   Place `120_classifier.pth` in `checkpoint/`.
+
+2. Prepare images in `datasets/images/`.\
+   Prepare corresponding segmentation masks in `datasets/labels/`.\
+   **_Note: Our pretrained models depend on the label taxonomy it was trained on, which is from: [HumanParsing-Dataset](https://github.com/lemondan/HumanParsing-Dataset)_**
+
+```
+# label index:  segment name
+0            :  null
+1            :  hat
+2            :  hair
+3            :  glasses
+4            :  up(per clothing)
+5            :  skirt
+6            :  pants
+7            :  dress
+8            :  belt
+9            :  r-shoe
+10           :  l-shoe
+11           :  face
+12           :  r-leg
+13           :  l-leg
+14           :  r-arm
+15           :  l-arm
+16           :  bag
+17           :  scarf
+```
+   **Segmentation labels need to be converted into the above mapping in order to work with FashionPlus.**
+
+3. Prepare input data for models:\
+   _Change_ `ROOT_DIR` _in script to FashionPlus' absolute path on your system._
+```
+cd preprocess
+./run_prepare_data.sh
+```
+
+4. Encode input images into latent codes:\
+   _Change_ `ROOT_DIR` _in script to FashionPlus' absolute path on your system._
+```
+cd preprocess
+./encode_shape_texture_features.sh
+```
+5. Perform edit and generate the changed output:\
+   _Change_ `ROOT_DIR` _in script to FashionPlus' absolute path on your system._\
+   Specify options:
+
+   * UPDATE_FNAME: image file's name to be edited
+   * UPDATE_TYPE: could choose to update only shape, or only texture, or both
+   * AUTO_SWAP: whether to let the editing module automatically decide which garment to change; if specified, SWAPPED_PARTID will have no effect
+   * SWAPPED_PARTID: explicitly specify which garment to swap out; here our mapping is top=0, skirt=1, pants=2, dress=3
+   * MAX_ITER: if the updating module's stop criterion is among reaching the maximal iteration, then specify the maximal value here
+   * STEP_SZ: amount of edit performed in each iteration; the larger the step size, the more significant the change is
+```
+./scripts/edit_and_visualize_demo.sh <UPDATE_FNAME> <UPDATE_TYPE> <AUTO_SWAP> <SWAPPED_PARTID> <MAX_ITER> <STEP_SZ>
+```
+   Output image results will be saved to `classification/data_dict/shape_and_features/results/demo/images/`.
+
+### Example 1
+<img src="figures/example1.jpg" height="200">
+
+Automatically decide where to change, and change both shape and texture
+```
+cd classification/data_dict/shape_and_feature/
+./scripts/edit_and_visualize_demo.sh 18.jpg shape_and_texture True 0 10 0.25
+```
+### Example 2
+<img src="figures/example2.jpg" height="200">
+
+Specify to change the top, and change both shape and texture
+```
+cd classification/data_dict/shape_and_feature/
+./scripts/edit_and_visualize_demo.sh 3.jpg shape_and_texture False 0 10 0.05
+```
+### Example 3
+<img src="figures/example3.jpg" height="200">
+
+Specify to change the top, and change only the shape
+```
+cd classification/data_dict/shape_and_feature/
+./scripts/edit_and_visualize_demo.sh 4.jpg shape_only False 0 10 0.25
+```
+### License
+FashionPlus is CC-BY-NC 4.0 licensed, as found in the LICENSE file.
+
+## Citation
+If you find this useful for your research, please consider citing:
+```
+@inproceedings{hsiao2019fashionplus,
+  title={Fashion++: Minimal Edits for Outfit Improvement},
+  author={Hsiao, Wei-Lin and Katsman, Isay and Wu, Chao-Yuan and Parikh, Devi and Grauman, Kristen},
+  booktitle={In Proceedings of the IEEE International Conference on Computer Vision (ICCV)},
+  year={2019}
+}
+```
+
+## Acknowledgements
+This code borrows heavily from [pix2pixHD](https://github.com/NVIDIA/pix2pixHD) and [BicycleGAN](https://github.com/junyanz/BicycleGAN).
+# FPLUS

classification/data_dict/shape_and_feature/classifier_options.py

@@ -0,0 +1,61 @@
+# Copyright (c) Facebook, Inc. and its affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+#
+
+import argparse
+
+def option_parser():
+    parser = argparse.ArgumentParser()
+
+    # Input
+    parser.add_argument('--texture_feat_file', type=str, help='pickle file with texture features of pieces')
+    parser.add_argument('--shape_feat_file', type=str, help='pickle file with shape features of pieces')
+    parser.add_argument('--dataset_dir', type=str, help='directory path to read and write files')
+    parser.add_argument('--load_pretrain_clf', type=str, default='', help='load the pretrained classification model from the specified location')
+    parser.add_argument('--update_fname', type=str, help='the filename of the imagee we are updating')
+    parser.add_argument('--clf_epoch', type=int, default = -1, help='load model at epoch; -1 for highest epoch')
+    parser.add_argument('--save_dir', type=str, default='results/fashion/updates/', help='path to save generated image')
+
+    # Network
+    # 1) Classifier
+    parser.add_argument('--network_arch', type=str, default='mlp', help='architecture of the network [mlp|linear]')
+    parser.add_argument('--in_dim', type=int, default = 12, help='input dimension for first fc layer')
+    parser.add_argument('--out_dim', type=int, default = 2, help='output dimension for first fc layer')
+    parser.add_argument('--param_m', type=int, default = 1, help='number of hidden layers in MLP')
+    parser.add_argument('--param_k', type=int, default = 8, help='number of neurons at each hidden layer')
+    parser.add_argument('--fc1_dim', type=int, default = 8, help='dimension for fc layer')
+    parser.add_argument('--fc2_dim', type=int, default = 2, help='dimension for fc layer')
+    parser.add_argument('--use_dropout', action='store_true', help='if specified, use dropout layer')
+    # 2) Generator
+    #    2-1) Texture generator
+    parser.add_argument('--load_pretrain_texture_gen', type=str, default='', help='load the pretrained generator model from the specified location')
+    parser.add_argument('--color_mode', type=str, default='RGB', help='color mode of our color image [Lab|RGB]')
+    parser.add_argument('--model_type', type=str, default='pix2pixHD', help='currently only suppport pix2pixHD')
+    parser.add_argument('--texture_feat_num', type=int, default=3, help='texture generator feature dimension')
+    #    2-2) Shape generator
+    parser.add_argument('--load_pretrain_shape_gen', type=str, default='', help='load the pretrained generator model from the specified location')
+    parser.add_argument('--shape_feat_num', type=int, default=8, help='shapee generator feature dimension')
+
+    # Learning
+    parser.add_argument('--stop_criterion', type=str, default = 'maxiter', help='stop scriterion for optimization process: maxiter | deltaloss | thresholdloss')
+    parser.add_argument('--max_iter_hr', type=int, default = 15, help='how many iterations to run')
+    parser.add_argument('--min_deltaloss', type=float, default = 0.0, help='the amount of change the loss should make before stop')
+    parser.add_argument('--min_thresholdloss', type=float, default = 0.0, help='the thresholded loss that the optimizer needs to reach before stops')
+    parser.add_argument('--lr', type=float, default = 0.05, help='optimizer learning rate; here is the step size for updating module')
+    parser.add_argument('--lambda_smooth', type=float, default = 10, help='weight of the smooth term')
+
+    # Output
+    parser.add_argument('--netG', type=str, default='global', help='generator architecture [global|local]')
+    parser.add_argument('--update_full', action='store_true', help='if specified, update the whole outfit instead of only the swapped')
+    parser.add_argument('--update_type', type=str, default='shape_and_texture', help='when partially update: shape_only | texture_only | shape_and_texture')
+    parser.add_argument('--display_freq', type=int, default = 5, help='how often to compute accuracy')
+    parser.add_argument('--autoswap', action='store_true', help='if specified, automatically decide which part to swap out; should not be used together with swapped_partID')
+    parser.add_argument('--generate_or_save', type=str, default = 'generate', help='generate updated image or save the updated vector: generate | save')
+    parser.add_argument('--iterative_generation', action='store_true', help='if specified, generate each iteration of an image update')
+    parser.add_argument('--classname', type=str, help='segmentation definition from dataset:  humanparsing')
+    parser.add_argument('--swapped_partID', type=int, default = 0, help='predefine which part to swap to; has no effect when autoswap option is specified; for humanparsing classname, 0: top; 1: skirt; 2: pants; 3:dress')
+
+    return parser.parse_args()

classification/data_dict/shape_and_feature/datasets/humanparsing/garment_label_part_map.json

@@ -0,0 +1,22 @@
+{
+	"top": {
+		"label": 4,
+		"partID": 0
+	},
+	"skirt": {
+		"label": 5,
+		"partID": 1
+	},
+	"pants": {
+		"label": 6,
+		"partID": 2
+	},
+	"dress": {
+		"label": 7,
+		"partID": 3
+	},
+	"background": {
+		"label": 0,
+		"partID": 4
+	}
+}

classification/data_dict/shape_and_feature/scripts/batch_edit_and_visualize_demo.sh

@@ -0,0 +1,114 @@
+#!/bin/bash
+
+# Copyright (c) Facebook, Inc. and its affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+#
+
+set -ex
+ROOT_DIR='/content/FashionPlus/' # absolute path for FashionPlus
+CLASS='humanparsing'  # segmentation definition from dataset "humanparsing"
+MODEL='pix2pixHD'
+COLOR_MODE='Lab' # RGB, Lab
+NET_ARCH='mlp' # linear, mlp
+TEXTURE_FEAT_NUM=8
+LAMBDA_KL=0.0001 # hyperparameter for VAE 0.0001
+DIVIDE_K=4 # hyperparameter for VAE 4
+
+# Editing module options
+UPDATE_FNAME='3.jpg' # filename to update
+UPDATE_TYPE='shape_and_texture' # specify whether to edit shape_only, texture_only, or shape_and_texture
+AUTO_SWAP='True' # auto_swap is True if automatically deciding which part to swap out; then swapped_partID will be unused
+SWAPPED_PARTID=0 # swapped_partID specifies which part to update; for class='humanparsing', partID mapping is: 0 top, 1 skirt, 2 pants, 3 dress
+MAXITER=10 # editing module stops at maxiter iterations
+UPDATE_STEP_SZ=0.05 # editing module takes step size at each update iteration
+ITERATIVE_SAVE='False' # iterative_save is True when we generate edited results from each iteration
+
+case ${MODEL} in
+'pix2pixHD')
+  case ${TEXTURE_FEAT_NUM} in
+  8)
+    SAVE_CODES_DIR=${ROOT_DIR}'classification/data_dict/shape_and_feature/results/demo/'
+    SAVE_MASKS_DIR=${ROOT_DIR}'separate_vae/results/'${COLOR_MODE}'/'${CLASS}'/'${UPDATE_TYPE}'/demo'
+    SAVE_IMGS_DIR=${ROOT_DIR}'generation/results/'${COLOR_MODE}'/'${CLASS}'/'${UPDATE_TYPE}'/demo'
+    TEXTURE_GEN_PATH=${ROOT_DIR}'/checkpoint/'
+    SHAPE_GEN_PATH=${ROOT_DIR}'/checkpoint/'
+    ;;
+  *)
+    echo 'WRONG feature_dimension '${TEXTURE_FEAT_NUM}
+    ;;
+  esac
+  ;;
+*)
+  echo 'WRONG category'${MODEL}
+  ;;
+esac
+
+############### UPDATE ###############
+bash scripts/edit_and_save_demo.sh \
+  ROOT_DIR=${ROOT_DIR} \
+  CLASS=${CLASS} \
+  COLOR_MODE=${COLOR_MODE} \
+  NET_ARCH=${NET_ARCH} \
+  MODEL=${MODEL} \
+  UPDATE_TYPE=${UPDATE_TYPE} \
+  TEXTURE_FEAT_NUM=${TEXTURE_FEAT_NUM} \
+  SAVE_CODES_DIR=${SAVE_CODES_DIR} \
+  UPDATE_FNAME=${UPDATE_FNAME} \
+  MAXITER=${MAXITER} \
+  UPDATE_STEP_SZ=${UPDATE_STEP_SZ} \
+  AUTO_SWAP=${AUTO_SWAP} \
+  SWAPPED_PARTID=${SWAPPED_PARTID} \
+  ITERATIVE_SAVE=${ITERATIVE_SAVE}
+
+############## MASKS ###############
+cd ${ROOT_DIR}/separate_vae
+
+bash scripts/batch_decode_masks_from_features.sh \
+  CLASS=${CLASS} \
+  SAVE_CODES_DIR=${SAVE_CODES_DIR} \
+  SAVE_MASKS_DIR=${SAVE_MASKS_DIR} \
+  SHAPE_GEN_PATH=${SHAPE_GEN_PATH}
+
+############## IMAGES ###############
+cd ${ROOT_DIR}/generation
+
+bash scripts/batch_decode_images_from_features.sh \
+  CLASS=${CLASS} \
+  COLOR_MODE=${COLOR_MODE} \
+  MODEL=${MODEL} \
+  TEXTURE_FEAT_NUM=${TEXTURE_FEAT_NUM} \
+  SAVE_IMGS_DIR=${SAVE_IMGS_DIR} \
+  SAVE_CODES_DIR=${SAVE_CODES_DIR} \
+  SAVE_MASKS_DIR=${SAVE_MASKS_DIR} \
+  TEXTURE_GEN_PATH=${TEXTURE_GEN_PATH}
+
+############## POSTPROCESS ###############
+cd ${ROOT_DIR}/postprocess
+if [[ "${ITERATIVE_SAVE}" == 'True'  ]];
+then
+  printf -v ITER_HEADER "%03d" ${MAXITER}
+else
+  ITER_HEADER='final'
+fi
+
+LABEL_DIR=${ROOT_DIR}/datasets/images/
+IMG_DIR=${ROOT_DIR}/datasets/labels/
+python process_face.py \
+        --fname ${ITER_HEADER}_${UPDATE_FNAME} \
+        --orig_img_dir ${LABEL_DIR} \
+	--orig_mask_dir ${IMG_DIR} \
+        --gen_img_dir ${SAVE_IMGS_DIR} \
+	--gen_mask_dir ${SAVE_MASKS_DIR} \
+	--bbox_pickle_file ${ROOT_DIR}/generation/datasets/demo/test.p \
+	--result_dir ${SAVE_CODES_DIR}/images/
+python process_face.py \
+        --fname 001_${UPDATE_FNAME} \
+        --orig_img_dir ${LABEL_DIR} \
+	--orig_mask_dir ${IMG_DIR} \
+        --gen_img_dir ${SAVE_IMGS_DIR} \
+	--gen_mask_dir ${SAVE_MASKS_DIR} \
+	--bbox_pickle_file ${ROOT_DIR}/generation/datasets/demo/test.p \
+	--result_dir ${SAVE_CODES_DIR}/images/