[Partitioner] First Graph Partitioning #2268

beicy · 2019-01-15T22:40:30Z

Description:
Graph partitioning. Use BFS order to get the initial partitions based on memory restrictions. (working on the advanced algorithm, which is not included in this PR)

Testing:
Added an unittest.
The original function is:

And the partitions are:
P1:

P2:

P3:

Documentation: #2045 #2298
[Optional Fixes #issue]

Please see a detailed explanation of how to fill out the fields in the relevant sections in PULL_REQUEST.md.

beicy · 2019-01-15T22:42:52Z

Thanks a lot @bertmaher ! Borrowed some ideas and code from your previous partition work :)

rdzhabarov · 2019-01-16T01:35:12Z

(working on the advanced algorithm, which is not included in this PR)

might be worthwhile outline that design in the Github issue.

rdzhabarov · 2019-01-16T02:13:40Z

lib/Partitioner/Partitioner.cpp

+/// Current only partition the representive function.
+DAGNode *Partitioner::doPartitioning(Function *F, NodeFunctionMap &mapping) {
+  // The dummy node.
+  DAGNode *DAG = new DAGNode();


can you return and use std::unique ptr to avoid any memory leak issues?

these days we should avoid naked new, given all the support we have in std:: various pointers.

@rdzhabarov After discussing with @gcatron, we use shared_ptr since all DAGNodes are in HM.

Sounds good.

@nadavrot we don't use shared_ptr now. This is the current design: #2268 (comment)

jackm321 · 2019-01-16T21:29:10Z

include/glow/Partitioner/Partitioner.h

+
+using namespace runtime;
+
+using ModuleList = std::unique_ptr<Module>;


why is this called list?

also it appears only to be used once so I'm not sure if it's needed

Removed. It was from previous design and not used anymore.

jackm321 · 2019-01-16T21:29:59Z

include/glow/Partitioner/Partitioner.h

+using MemUsageMap = std::unordered_map<Node *, unsigned>;
+
+/// Dependency map type.
+using MapTy = llvm::DenseMap<Module *, ModuleList>;


jackm321 · 2019-01-16T21:33:07Z

include/glow/Partitioner/Partitioner.h

+  Map nodeToFunction_;
+
+public:
+  /// Create a new partition \p F. Add it to the map with initial node \p N.


where is the N param?

what is the "N param" ?

I don't know, it says that in the comment on this line though

I see. Forgot to update the comment :P

jackm321 · 2019-01-16T21:34:09Z

include/glow/Partitioner/Partitioner.h

+/// original function to destination partitions, along with a list of the
+/// newly-created functions.
+class NodeFunctionMap {
+  using Map = llvm::DenseMap<Node *, Function *>;


Maybe a name like NodeToFunctionMap would be better?

why DenseMap here but not in the visited list std::unordered_set<const Node *> visited;.

I think that xxxxxTy would be a better name, because this is not a map, it is a map type.

jackm321 · 2019-01-16T21:34:56Z

include/glow/Partitioner/Partitioner.h

+  const FunctionList &getFunctions() const { return functions_; }
+
+  /// \returns the number of partitions.
+  Map::size_type size() const { return functions_.size(); }


this is using Map's size_type but returning the size of functions_ which is actually a FunctionList.

jackm321 · 2019-01-16T21:54:40Z

lib/Partitioner/Partitioner.cpp

+    size_t size = memSize;
+
+    // The set to keep the placeholders (only for Inputs) whose size is already
+    // calcualted.


jackm321 · 2019-01-16T21:55:54Z

lib/Partitioner/Partitioner.cpp

+    // calcualted.
+    std::set<std::string> pSet;
+
+    for (auto &node : F->getNodes()) {


Maybe a quick overview of the logic here would be helpful like "Sum the sizes of all input Placeholders in the graph"

Also I'm surprised we don't have something for this already that gets all input placeholders, seems useful. I think I remember there being some attempt at this but some placeholders are inputs and outputs due to training, maybe we need to handle that case here also not sure?

Actually, a few utility functions (the input one, the memory cost for one arbitrary sub-graph...) should be helpful. But I would prefer separate PRs.

jackm321 · 2019-01-16T22:00:19Z

lib/Partitioner/Partitioner.cpp

+  for (auto &node : F_->getNodes()) {
+    int n = node.getNumInputs();
+    unsigned size = 0;
+    if (node.getKind() != Kinded::Kind::SaveNodeKind) {


easier to read if we do if (node.getKind() == Kinded::Kind::SaveNodeKind) {continue;}

Actually, there is a statement after this condition. I think it is OK to use "!=" here since I don't want to duplicate that statement .

It took me two reads to get it, how about:
if (node.getKind() == Kinded::Kind::SaveNodeKind) { memUsage_[&node] = 0; continue; }

It's not obvious that the n and size variables are unused in this case.

jackm321 · 2019-01-16T22:02:17Z

lib/Partitioner/Partitioner.cpp

+  GraphPreOrderVisitor visitor(*F);
+  Node *node = nullptr;
+  for (auto &N : visitor.getPreOrder()) {
+    if (isa<Storage>(N))


add brackets here

jackm321 · 2019-01-16T22:03:09Z

lib/Partitioner/Partitioner.cpp

+    if (isa<Storage>(N))
+      continue;
+    node = N;
+    break;


I'm confused, won't this just quit at the first non-storage node? Why can't there be multiple in the first level, is this the save node?

This is a save node. The comment mentioned that now we assume there is only one root now. The current models I've met only have 1 output node. But it could be multiple roots as well later.

jackm321 · 2019-01-16T22:09:52Z

lib/Partitioner/Partitioner.cpp

+  int current = 0;
+  auto newPair = std::make_pair(level, std::vector<Node *>());
+  newPair.second.push_back(node);
+  bfs.levels.push_back(newPair);


this could just be bfs.levels.push_back({0, {node}});

jackm321 · 2019-01-16T22:33:53Z

include/glow/Partitioner/Partitioner.h

+  Map nodeToFunction_;
+
+public:
+  /// Create a new partition \p F. Add it to the map with initial node \p N.


I don't know, it says that in the comment on this line though

nadavrot · 2019-01-17T05:43:31Z

include/glow/Partitioner/Partitioner.h

+#ifndef GLOW_PARTITIONER_PARTITIONER_H
+#define GLOW_PARTITIONER_PARTITIONER_H
+
+#include "glow/Graph/Context.h"


Can we use forward declarations and move some of the includes to the cpp file?

nadavrot · 2019-01-17T05:44:09Z

include/glow/Partitioner/Partitioner.h

+  Function *operator[](Node *n) { return nodeToFunction_[n]; }
+};
+
+/// Partition


/// Partitions a function into multiple functions based on A... B... C...

nadavrot · 2019-01-17T05:44:19Z

include/glow/Partitioner/Partitioner.h

+  /// The result of module partitioning.
+  DAGNodeList partitions_;
+
+  /// Total memory requested by one module.


Size in bytes.

nadavrot · 2019-01-17T05:44:48Z

lib/Partitioner/Partitioner.cpp

+using namespace glow;
+using llvm::isa;
+
+// Margin for code itself -- should be adjusted later.


Stale comment here.

nadavrot · 2019-01-17T05:44:59Z

lib/Partitioner/Partitioner.cpp

+// Margin for code itself -- should be adjusted later.
+//#define MARGIN 1024 * 128
+
+struct BFSLevel {


Doxygen please.

Comment added.

nadavrot · 2019-01-17T05:45:40Z

lib/Partitioner/Partitioner.cpp

+
+  auto funcList = module_->getFunctions();
+  for (Function *F : funcList) {
+    // printf("function name : %s\n", F->getName().data());


rdzhabarov

flush some comments

rdzhabarov · 2019-01-17T21:55:01Z

include/glow/Partitioner/Partitioner.h

+#define GLOW_PARTITIONER_PARTITIONER_H
+
+#include "glow/Graph/Graph.h"
+#include "llvm/ADT/DenseMap.h"


can you separate includes for glow headers vs llvm vs standard ones.

rdzhabarov · 2019-01-17T21:55:31Z

include/glow/Partitioner/Partitioner.h

+
+namespace glow {
+
+using namespace runtime;


should partitioner be in the runtime namespace as well?

The DAGNode is defined in runtime namespace.

I mean,
namespace glow {
namespace runtime {

... // rest of the code here.

}
}

We have include/glow/Runtime/.. and include/glow/Partitioner/...
I think Partitioner is more "compiler" level, not "runtime". The namespace should be glow::Partitioner.

rdzhabarov · 2019-01-17T21:58:01Z

include/glow/Partitioner/Partitioner.h

+
+public:
+  /// Create a new partition \p F.
+  void create(Function *F) { functions_.emplace_back(F); }


createPartition?

rdzhabarov · 2019-01-17T22:00:32Z

include/glow/Partitioner/Partitioner.h

+  void add(Node *N, Function *F) { nodeToFunction_[N] = F; }
+
+  /// Get list of functions contained in this map.
+  const FunctionList &getFunctions() const { return functions_; }


getPartitions

rdzhabarov · 2019-01-17T22:11:54Z

include/glow/Partitioner/Partitioner.h

+};
+
+/// Given a module, partitions each of the its functions into multiple ones
+/// based on memory constraints and minimizes the communication cost.


I'd describe how partitioning is done here as well.

The current one is just a simple algorithm, still evaluating the next step improvement. Will add the algorithm description later.

rdzhabarov · 2019-01-17T22:19:21Z

include/glow/Partitioner/Partitioner.h

+  Partitioner(Module *parent, std::vector<DeviceInfo> &devices);
+
+  /// Decompose each function in a module and return a list of DAGNodes.
+  DAGNodeList &Partition();


why do we allow caller to modify internal state of partitions by returning reference to it?

The DAGNode will be fulfilled by Partitioner, Provisioner, and DeviceManager.

rdzhabarov · 2019-01-17T22:19:58Z

include/glow/Partitioner/Partitioner.h

+  DAGNodeList &Partition();
+};
+} // namespace glow
+#endif


commend on what it ends after #endif

rdzhabarov · 2019-01-17T22:20:22Z

include/glow/Runtime/RuntimeTypes.h

@@ -45,10 +45,10 @@ struct DeviceInfo {
 struct DAGNode {
  /// The children of this node, these are nodes that depend on the current
  /// node.
-  std::vector<DAGNode *> children;
+  std::vector<std::shared_ptr<DAGNode>> children;


can you rebase this on master.

rdzhabarov · 2019-01-17T22:21:08Z

lib/Partitioner/Partitioner.cpp

+
+Partitioner::Partitioner(Module *parent, std::vector<DeviceInfo> &devices)
+    : deviceInfo_(devices) {
+  module_ = parent;


initialize in the initializer list.

rdzhabarov · 2019-01-17T22:22:37Z

include/glow/Partitioner/Partitioner.h

+/// original function to destination partitions, along with a list of the
+/// newly-created functions.
+class NodeFunctionMap {
+  using Map = llvm::DenseMap<Node *, Function *>;


why DenseMap here but not in the visited list std::unordered_set<const Node *> visited;.

rdzhabarov · 2019-01-17T22:25:59Z

lib/Partitioner/Partitioner.cpp

+    : deviceInfo_(devices) {
+  module_ = parent;
+  // Check if all the functions are in the same structure.
+  if (!isSameFunctionFamily()) {


btw, for asserts like that you can do: assert(isSameFunctionFamily() && "not in the same function family")

rdzhabarov · 2019-01-17T22:31:13Z

lib/Partitioner/Partitioner.cpp

+
+/// Get the representative function (the one with the largest input) and update
+/// the memSize_.
+void Partitioner::selectRepFunc() {


this should be static and have module as input, this way you can test it properly.

beicy · 2019-01-18T07:46:53Z

After more discussion with @opti-mix and @bertmaher, now inside of struct DAGNode, we use raw pointer. Then we use a DAGNodeList as a container to keep the unique_ptr of each DAGNode.

opti-mix · 2019-01-18T16:42:43Z

include/glow/Partitioner/Partitioner.h

+  Function *operator[](Node *n) { return nodeToFunction_[n]; }
+};
+
+/// The struct contains all the created DAGNodes.


Nice!

It could be a good idea to add some comments here and/or for DAGNode to explain the ownership model used here and who owns what. One of the consequences of this design is that DAGNode cannot outlive the DAGNodeList owning it. The second consequence is that DAGNodes can only refer to the DAGNodes from the same DAGNodeList and they don't need to use any smart pointers for that as all of DAGNodes are owned by the same DAGNodeList.

nadavrot · 2019-01-18T20:54:41Z

include/glow/Partitioner/Partitioner.h

+
+      // The set to keep the placeholders (only for Inputs) whose size is
+      // already calculated.
+      std::set<std::string> pSet;


I think that we can save stringRef here.

nadavrot · 2019-01-18T20:55:38Z

lib/Partitioner/Partitioner.cpp

+  NodeToFunctionMap mapping;
+  BFSLevel bfs = getBFSLevel(F);
+  unsigned level = bfs.levels.size();
+  std::vector<int>


This is formatted funny.

nickgg · 2019-01-18T20:31:10Z

include/glow/Partitioner/Partitioner.h

+  Function *F_;
+
+  /// The cost model related to device.
+  std::vector<DeviceInfo> &deviceInfo_;


should this be a const ref?

nickgg · 2019-01-18T20:39:59Z

include/glow/Partitioner/Partitioner.h

+  /// identical. The required memory and computation cost for each op can be
+  /// found in Module. The \p devices provides the cost model related to
+  /// devices.
+  Partitioner(Module *parent, std::vector<DeviceInfo> &devices);


const ref then?

nickgg · 2019-01-18T20:53:37Z

lib/Partitioner/Partitioner.cpp

+  for (auto &node : F_->getNodes()) {
+    int n = node.getNumInputs();
+    unsigned size = 0;
+    if (node.getKind() != Kinded::Kind::SaveNodeKind) {


It took me two reads to get it, how about:
if (node.getKind() == Kinded::Kind::SaveNodeKind) { memUsage_[&node] = 0; continue; }

It's not obvious that the n and size variables are unused in this case.

nickgg · 2019-01-18T21:15:10Z

lib/Partitioner/Partitioner.cpp

+           // cut[0]), [cut[0] - 1, cut[1]), ..., [cut[n], -1).
+
+  // Step 1 : get the initial cut based on BFS levels and avaiableMemory.
+  // TODO .. need to remove the duplicated memory useage.


nit: memory usage

nickgg · 2019-01-18T21:16:08Z

lib/Partitioner/Partitioner.cpp

+  BFSLevel bfs = getBFSLevel(F);
+  unsigned level = bfs.levels.size();
+  std::vector<int>
+      cut; // A list of cut. The graph can be partitioned by levels [level - 1,


What is cut? Is this described in a different comment?

The initial partitions are based on the BFS level, and the cut is used for initial partitions.

nickgg · 2019-01-18T21:22:36Z

lib/Partitioner/Partitioner.cpp

+/// Current only partition the representive function.
+void Partitioner::doPartitioning(Function *F, NodeToFunctionMap &mapping) {
+  // The dummy node.
+  std::unique_ptr<DAGNode> DAG(new DAGNode());


std::make_unique<DAGNode>() ?

nickgg · 2019-01-18T21:23:34Z

lib/Partitioner/Partitioner.cpp

+  DAG->logicalDevice = 0;
+  DAG->name = F->getName();
+  DAG->deviceID = 0;
+  DAG->logicalDevice = 0;


Could we have a constructor of DAGNode that takes these initializations as arguments? Does it make sense to have a DAGNode without these set?

The DAGNode is filled by Partitioner, Provisioner and DeviceManager(?). According to my understanding, all the fields should be set before a DAGNode is actually in use. That is why I simply set each field related to Partitioner instead of using a constructor.

nickgg · 2019-01-18T21:56:36Z

lib/Partitioner/Partitioner.cpp

+
+  // Possible minimal k devices for a succesful partitioning
+  // Note: here 2 is for testing;
+  unsigned k = 2; //(memSize_ + MARGIN) / devices[0].availableMemory;


Can k be a member of the Partitioner? This would let us write unit tests to cover multiple cases (k == 1, k == 2, k == n, etc).

For the partitioner, we will try to get the smallest k (which is the return value, not the param). Here 2 is just for testing.

nadavrot · 2019-01-20T22:38:38Z

include/glow/Partitioner/Partitioner.h

+  Function *F_;
+
+  /// The cost model related to device.
+  std::vector<DeviceInfo> &deviceInfo_;


Why is this a vector? Should the comment say that this is per-device info?

The reason we use vector here is maybe later, we are consider heterogeneous devices. Will updated the comment.

nadavrot · 2019-01-20T22:40:09Z

include/glow/Partitioner/Partitioner.h

+
+  /// Get the representative function (the one with the largest input) and
+  /// update the memSize.
+  static Function *selectRepFunc(Module *parent, size_t &memSize) {


We should move this to the CPP file.

nadavrot · 2019-01-20T22:40:30Z

include/glow/Partitioner/Partitioner.h

+
+      // The set to keep the placeholders (only for Inputs) whose size is
+      // already calculated.
+      std::set<llvm::StringRef> pSet;


Do we want to save the names of the pointers to the placeholders?

nadavrot · 2019-01-20T22:40:57Z

include/glow/Partitioner/Partitioner.h

+      for (auto &node : F->getNodes()) {
+        int n = node.getNumInputs();
+        if (node.getKind() == Kinded::Kind::SaveNodeKind) {
+          // Special node, the placeholder should be ignored?


Can you elaborate on this? Why should we ignore this node? What is special?

For "Save" node, it is just create a placeholder for the output. Here, we only consider the input size.

nadavrot · 2019-01-20T22:41:26Z

include/glow/Partitioner/Partitioner.h

+        for (int i = 0; i < n; i++) {
+          Placeholder *in =
+              llvm::dyn_cast<Placeholder>(node.getNthInput(i).getNode());
+          if (in && (pSet.find(in->getName()) == pSet.end())) {


find -> count()

nadavrot · 2019-01-20T22:41:31Z

include/glow/Partitioner/Partitioner.h

+          }
+        }
+      }
+      if (size > memSize) {


nadavrot · 2019-01-20T22:42:41Z

lib/Partitioner/Partitioner.cpp

+Partitioner::Partitioner(Module *parent, std::vector<DeviceInfo> &devices)
+    : module_(parent), deviceInfo_(devices) {
+  // Check if all the functions are in the same structure.
+  assert(isSameFunctionFamily() && "not in the same function family");


I think that we decided not to use the concept of family functions.

@nadavrot We are not using function families for partitioning (this definitely makes sense to me) or at all (I'm not clear yet on how this can done)?

The semantics of "family" is not well defined. You can say that some functions are related if they share constants, or something, but these are all things that you can express in the Graph (with functions that refer to constants).

This will be removed after today's discussion.

nadavrot · 2019-01-20T22:44:06Z

lib/Partitioner/Partitioner.cpp

+  bfs.visited.insert(node);
+  level++;
+  while (current < level) {
+    std::vector<Node *> nList;


nList -> nodes?

nadavrot · 2019-01-20T22:45:28Z

lib/Partitioner/Partitioner.cpp

+/// the number of DAGNodes) is larger than the number of devices. E.g:
+/// node1(6GB) -> node2(14GB) -> node3(6GB). The memory limitation is 16GB, and
+/// there is only 2 devices.
+void Partitioner::adjustLogicalDeviceID(DAGNode *DAG, int num) {}


Should we add something like assert(false && "invalid .. .. ");

Partitioner will return the minimal k partitions. And the provisioner will double check if k fits the devices and give the invalid assert.

nadavrot · 2019-01-20T22:45:45Z

lib/Partitioner/Partitioner.cpp

+      funcDAG[subF] = subDAG.get();
+      partitions_.nodes.push_back(std::move(subDAG));
+    }
+    for (auto &N : subF->getNodes()) {


nadavrot · 2019-01-20T22:45:49Z

lib/Partitioner/Partitioner.cpp

+        if (subF == inputF)
+          continue;
+
+        if (funcDAG.find(inputF) == funcDAG.end()) {


nadavrot · 2019-01-20T22:49:03Z

@beicy This PR is looking good. I have a few small comments but I think that we are close.

nadavrot

Looks good to me. I am excited about seeing graph partitioning landing.

facebook-github-bot added the CLA Signed label Jan 15, 2019

beicy force-pushed the Partition branch 4 times, most recently from 64ab826 to 6580b76 Compare January 16, 2019 00:15

rdzhabarov reviewed Jan 16, 2019

View reviewed changes

beicy force-pushed the Partition branch 2 times, most recently from 1e6b60a to f6a36fe Compare January 16, 2019 22:06

jackm321 reviewed Jan 16, 2019

View reviewed changes

beicy force-pushed the Partition branch 2 times, most recently from 6874ac4 to c9da452 Compare January 16, 2019 23:48

gcatron mentioned this pull request Jan 17, 2019

Add provisioner for new Runtime #2276

Closed

nadavrot reviewed Jan 17, 2019

View reviewed changes

beicy force-pushed the Partition branch from c9da452 to 3862046 Compare January 17, 2019 19:00

beicy mentioned this pull request Jan 17, 2019

[Runtime] Use std::shared_ptr to replace raw pointer. #2279

Closed

rdzhabarov reviewed Jan 17, 2019

View reviewed changes

beicy force-pushed the Partition branch 2 times, most recently from 8e4a859 to 457e4e2 Compare January 18, 2019 07:44

opti-mix reviewed Jan 18, 2019

View reviewed changes

beicy force-pushed the Partition branch from 457e4e2 to 0f07d01 Compare January 18, 2019 19:51

nadavrot reviewed Jan 18, 2019

View reviewed changes

beicy force-pushed the Partition branch from 0f07d01 to 49e5aef Compare January 18, 2019 21:32

nickgg reviewed Jan 18, 2019

View reviewed changes

nadavrot reviewed Jan 20, 2019

View reviewed changes

include/glow/Partitioner/Partitioner.h Outdated

}

}

}

if (size > memSize) {

Copy link

Contributor

nadavrot Jan 20, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comments.

nadavrot reviewed Jan 20, 2019

View reviewed changes

[Partitioner] First Graph Partitioning

14c0485

beicy force-pushed the Partition branch from 49e5aef to 14c0485 Compare January 22, 2019 22:43

nadavrot approved these changes Jan 23, 2019

View reviewed changes

beicy merged commit cfc989f into pytorch:master Jan 23, 2019


		using namespace runtime;

		using ModuleList = std::unique_ptr<Module>;

[Partitioner] First Graph Partitioning #2268

[Partitioner] First Graph Partitioning #2268

Conversation

beicy commented Jan 15, 2019 • edited Loading

beicy commented Jan 15, 2019

rdzhabarov commented Jan 16, 2019

rdzhabarov Jan 16, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rdzhabarov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

beicy commented Jan 18, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

beicy commented Jan 15, 2019 •

edited

Loading

rdzhabarov Jan 16, 2019 •

edited

Loading