PS-3889: Native Partitioning for TokuDB and RocksDB for 5.7 #2493

vlad-lesin · 2018-08-15T14:14:53Z

TokuDB and MyRocks native partitioning is implemented as a copy of partition
storage engine in 5.7 with corresponding changes.

The base class - Partition_base does the most of the work.

The general differences from ha_partition class are the following:

all code which works with .par files was removed;
the code to read partition information from .frm files was added;
the responsibility for creating new handlers for separate partitions is moved
to Partition_base class descendants;
the initialization sequence for native and non-native partitioning handlers
is different, that is why create() and open() functions are changed
delete and rename table functions are changed both for Partition_base and
it's descendants(ha_tokupart and ha_rockspart) because it's necessary to
determine if the table is partitioned before calling the corresponding
functionality.

The changes in the server code are the following:

remove .frm file after a table has been removed to let handler the ability
to read partition info from .frm file during "delete table" execution;
remove "static" keyword from some functions signature to use them outside of
their module.

To see the difference between the original ha_partition and Partition_base classes use the following command: diff -u -w ./storage/partition/ha_partition.cc ./sql/partitioning/partition_base.cc

Testing: https://jenkins.percona.com/view/5.7/job/mysql-5.7-param/1908/

george-lorch

There is a lot to review and understand here. Fortunately, the great bulk of the working code is literally directly copied from the existing generic partition handler. The most interesting thing is how most of the existing partitioning tests pass without any modifications (except for the deprecation warning). Ttat is a very good sign. I like the reuse/factoring out of the common code to be shared by the two engines and how minimal the actual engine changes are. I am a little concerned about the changes needed in the server code, but, they are not terribly invasive and seem needed in order to use the .frm files and not the .par files. Other than the one 'dev' test, nothing really jumps out at me as a potential problem. I will be looking at this again a few times to see if anything else comes out, but, for now I wouldn't hesitate to approve this pending a review from @laurynas-biveinis

george-lorch · 2018-08-16T17:28:10Z

mysql-test/suite/rocksdb/t/native_part_dev.test

@@ -0,0 +1,244 @@
+# This test is for debugging


I'm not understanding the purpose of this test. Is it purely for development purposes and should be removed or does it contain useful test cases that cover cases that are not covered in other tests?

This test is only for debugging, forgot to remove.

george-lorch · 2018-08-17T00:48:21Z

One thing I realized is missing that we talked about a while ago is the ability to enable/disable the use of native parts in 5.7. I don't know how difficult this might be. Ideally we want new options tokudb_enable_native_partition and rocksdb_enable_native_partition with defaults to off. This is specifically because we are releasing this mid-series 5.7 and this changes the default behavior of partitioning. We need try to maintain the existing behavior as well as offer a sort of 'emergency exit' just in case a user runs into some critical issue with the native partitioning. These options do not need to be dynamic, they can be global read-only. @vlad-lesin do you think this would be difficult or take too much time?

laurynas-biveinis

(partial review as discussed)

laurynas-biveinis · 2018-08-16T15:34:39Z

mysql-test/suite/parts/inc/upgrade_parts_from_prev_ver_alter.inc

+--exec echo $MYSQL_TMP_DIR/bootstrap.log
+
+--echo # Restart the server
+--exec echo "restart: --loose-skip-log-bin --skip-log-slave-updates --datadir=$MYSQLD_DATADIR1 $ADDITIONAL_OPTS" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect


Use restart_mysqld.inc

laurynas-biveinis · 2018-08-16T15:35:47Z

mysql-test/suite/parts/inc/upgrade_parts_from_prev_ver_mysql_upgrade.inc

+--file_exists $DATA_ARCH_PATH
+--exec unzip -qo $DATA_ARCH_PATH -d $MYSQL_TMP_DIR
+
+--echo # Restart the server


laurynas-biveinis · 2018-08-17T11:43:32Z

mysql-test/suite/rocksdb/t/native_part_dev.test

@@ -0,0 +1,244 @@
+# This test is for debugging


More descriptive comment please,for exampple "RocksDB native partitioning development tests"

This test is only for debugging, forgot to remove.

laurynas-biveinis · 2018-08-17T11:44:01Z

mysql-test/suite/rocksdb/t/native_part_dev.test

+CREATE TABLE t1 (i INT, j INT, k INT, PRIMARY KEY (i)) ENGINE = ROCKSDB PARTITION BY KEY(i) PARTITIONS 4;
+ALTER TABLE t1 CHECK PARTITION p1;
+DROP TABLE t1;
+--exit


Either enable all of the test, either truncate here

This test is only for debugging, forgot to remove.

laurynas-biveinis · 2018-08-17T11:44:49Z

mysql-test/suite/rocksdb/t/upgrade_parts_from_prev_ver.test

+--source ../include/have_write_committed.inc
+
+# For data directory created in previous major version branch.
+--source include/not_valgrind.inc


Just copied from somewhere. Removed.

laurynas-biveinis · 2018-08-20T09:46:48Z

sql/partitioning/partition_base.h

+  virtual int index_end() { return Partition_helper::ph_index_end(); }
+
+  /**
+    @breif


This is the original comment. I was not so bored to fix the original typos :) Fixed.

laurynas-biveinis · 2018-08-20T09:47:36Z

sql/partitioning/partition_base.h

+ private:
+  static const uint NO_CURRENT_PART_ID;
+  int               loop_extra(enum ha_extra_function operation);
+  void              late_extra_cache(uint partition_id);


TODO remove? Seems MyISAM-related

Yes, removed MyISAM-related stuff.

The testing will be latter.

laurynas-biveinis · 2018-08-20T11:37:31Z

sql/partitioning/partition_base.cc

+  if (!my_stat(path, &frm_stat_info, MYF(0)))
+  {
+    // if .frm file does not exist don't treat it as error
+    if (my_errno() == ENOENT)


if (cond) return true; return false; -> return cond;

laurynas-biveinis · 2018-08-20T11:38:42Z

sql/partitioning/partition_base.cc

+  if (head[0] != (uchar)254 || head[1] != 1 ||
+      !(head[2] == FRM_VER || head[2] == FRM_VER + 1 ||
+        (head[2] >= FRM_VER + 3 && head[2] <= FRM_VER + 4)))
+    // Upon failure, return NULL, but here, we have to close the file first.


Comment differs from code

Removed the comment. The file will be closed on scope exit.

sql/partitioning/partition_base.cc

laurynas-biveinis · 2018-08-21T10:59:19Z

sql/partitioning/partition_base.cc

+    part_name_length+= subpart_name_length;
+  }
+
+  create_partition_name(out_buf, path, part_name, NORMAL_PART_NAME, FALSE);


s/FALSE/false/g

laurynas-biveinis · 2018-08-21T10:59:52Z

sql/partitioning/partition_base.cc

+  create_partition_name(out_buf, path, part_name, NORMAL_PART_NAME, FALSE);
+}
+
+Parts_share_refs::Parts_share_refs() : num_parts(0), ha_shares(NULL) {}


s/NULL/nullptr/g

laurynas-biveinis · 2018-08-21T11:00:23Z

sql/partitioning/partition_base.cc

+
+Parts_share_refs::~Parts_share_refs()
+{
+  uint i;


Declare at use

laurynas-biveinis · 2018-08-21T11:04:59Z

sql/partitioning/partition_base.h

+    for this since the MySQL Server sometimes allocating the handler object
+    without freeing them.
+  */
+  ulong m_low_byte_first;


laurynas-biveinis · 2018-08-21T11:11:04Z

sql/partitioning/partition_base.cc

+    DATA DIRECTORY and INDEX DIRECTORY are never applied to the whole
+    partitioned table, only its parts.
+  */
+  my_bool from_alter= (create_info->data_file_name == (const char *)-1);


s/my_bool/bool/g

sql/partitioning/partition_base.cc

laurynas-biveinis · 2018-08-21T11:18:32Z

sql/partitioning/partition_base.cc

+    file= m_is_clone_of->m_file;
+
+    /*
+      The following code can be refactored. The same pattern of iteration


Remove the comment :)

laurynas-biveinis · 2018-08-21T11:38:10Z

sql/partitioning/partition_base.cc

+  implicit_emptied= m_file[0]->implicit_emptied;
+  /*
+    Add 2 bytes for partition id in position ref length.
+    ref_length=max_in_all_par


laurynas-biveinis · 2018-08-21T11:39:07Z

sql/table.cc

 {
  uchar *pos, *buf;
-  uint names, length;
+  uint   names, length;


Spurious change

Yes, this is autoformatting with clang-format-diff. Removed as discussed.

laurynas-biveinis · 2018-08-21T11:39:40Z

sql/table.h

-bool update_generated_read_fields(uchar *buf, TABLE *table,
-                                  uint active_index= MAX_KEY);
-
+bool  update_generated_write_fields(const MY_BITMAP *bitmap, TABLE *table);


Whitespace-only changes above the added declaration

The same, fixed.

laurynas-biveinis · 2018-08-21T11:40:37Z

storage/rocksdb/ha_rocksdb.cc

  DBUG_ASSERT(!mysqld_embedded);

-  if (rocksdb_db_options->max_open_files > (long) open_files_limit) {
+  if (rocksdb_db_options->max_open_files > (long)open_files_limit) {


Whitespace-only change

The same, fixed.

storage/rocksdb/ha_rocksdb.h

laurynas-biveinis · 2018-08-21T11:56:10Z

storage/rocksdb/ha_rocksdb.h

@@ -865,6 +871,13 @@ class ha_rocksdb : public my_core::handler {
    return m_store_row_debug_checksums && (rand() % 100 < m_checksums_pct);
  }

+  int rename_partitioned_table(const char *const from, const char *const to,
+                               const std::string &partition_string)
+      MY_ATTRIBUTE((__warn_unused_result__));


s/.../MY_NODISCARD/g

It is fixed for the changed code.

laurynas-biveinis · 2018-08-21T11:59:17Z

storage/tokudb/ha_tokudb.h

-    MY_NODISCARD int send_upsert_message(List<Item> &update_fields,
-                                         List<Item> &update_values,
-                                         DB_TXN *txn);
+    MY_NODISCARD int send_upsert_message(List<Item>& update_fields,


Formatting-only change

laurynas-biveinis

Completed a review pass in individual comments

laurynas-biveinis

Can you show how the MTR run log runs with your added combinations?
The MTR combination feature looks good, but I am not sure whether its application here is as good - it adds half a million .result file lines, which are impossible to review manually. The way this ideally should be done that the tests run in combinations, but using a single output file, by disabling warnings etc. @georgelorchpercona what do you think?

laurynas-biveinis · 2018-09-13T12:42:18Z

mysql-test/suite/rocksdb.sys_vars/r/all_vars.result

@@ -9,5 +9,7 @@ There should be *no* long test name listed below:
 select variable_name as `There should be *no* variables listed below:` from t2
 left join t1 on variable_name=test_name where test_name is null ORDER BY variable_name;
 There should be *no* variables listed below:
+rocksdb_enable_native_partition


Please add a sys_vars test

Added both for MyRocks and TokuDB.

With tests added, this all_vars diff should have disappeared

mysql-test/suite/rocksdb.rpl/combinations

laurynas-biveinis · 2018-09-26T11:11:59Z

mysql-test/suite/rocksdb.sys_vars/r/all_vars.result

@@ -9,5 +9,7 @@ There should be *no* long test name listed below:
 select variable_name as `There should be *no* variables listed below:` from t2
 left join t1 on variable_name=test_name where test_name is null ORDER BY variable_name;
 There should be *no* variables listed below:
+rocksdb_enable_native_partition


With tests added, this all_vars diff should have disappeared

mysql-test/suite/rocksdb/t/upgrade_parts_from_prev_ver.test

laurynas-biveinis · 2018-09-26T11:19:53Z

sql/CMakeLists.txt

@@ -475,6 +475,16 @@ ELSE()
  SET(MYSQLD_SOURCE main.cc ${DTRACE_PROBES_ALL})
 ENDIF()

+CHECK_CXX_COMPILER_FLAG(-std=gnu++11 CXX11_OPTION_SUPPORTED)


Comment not addressed

laurynas-biveinis · 2018-09-26T11:22:22Z

sql/handler.cc

+       lock_type == TL_READ_NO_INSERT ||
+       (lock_type != TL_IGNORE && thd->lex->sql_command != SQLCOM_SELECT)) &&
+      thd->lex->sql_command != SQLCOM_ALTER_TABLE &&
+      thd->lex->sql_command != SQLCOM_CHECK)


@vlad-lesin @georgelorchpercona what's the state of this?

laurynas-biveinis · 2018-09-26T11:52:07Z

sql/partitioning/partition_base.cc

+      }))
+    DBUG_RETURN(0);
+
+  auto part_name_i= part_name_collection.begin();


laurynas-biveinis · 2018-09-26T13:09:19Z

storage/rocksdb/ha_rockspart.cc

+
+err:
+  delete new_handler;
+  DBUG_RETURN(NULL);


laurynas-biveinis · 2018-09-26T13:11:44Z

storage/tokudb/ha_tokupart.cc

+    /* If this->table == NULL, then the current handler has been created but not
+    opened. Prohibit cloning such handler. */
+    if (!table)
+        DBUG_RETURN(NULL);


laurynas-biveinis · 2018-09-26T13:11:54Z

storage/tokudb/ha_tokupart.cc

+    new_handler = new (mem_root)
+        ha_tokupart(ht, table_share, m_part_info, this, mem_root);
+    if (!new_handler)
+        DBUG_RETURN(NULL);


laurynas-biveinis · 2018-09-26T13:12:07Z

storage/tokudb/ha_tokupart.cc

+
+err:
+    delete new_handler;
+    DBUG_RETURN(NULL);


laurynas-biveinis · 2018-09-26T13:12:39Z

storage/tokudb/hatoku_hton.cc

+        ha_tokupart* file = new (mem_root) ha_tokupart(hton, table);
+        if (file && file->init_partitioning(mem_root)) {
+            delete file;
+            return (NULL);


vlad-lesin · 2018-09-27T13:43:03Z

The comment about this #2493 (comment) note.

If we remove the change there will be test fails like the following:

--- ./mysql-test/suite/rocksdb/r/partition.result   2018-09-27 15:55:34.739805720 +0300
+++ build/mysql-test/var/44/log/partition.reject      2018-09-27 16:08:47.857325052 +0300
@@ -12,10 +12,14 @@
 test.t1        analyze status  OK
 test.t1        analyze warning The partition engine, used by table 'test.t1', is deprecated and will be removed in a future release. Please use native partitioning instead.
 Table  Op      Msg_type        Msg_text
-test.t1        repair  status  OK
+test.t1        repair  error   Partition p0 returned error
+test.t1        repair  Error   Using Gap Lock without full unique key in multi-table or multi-statement transactions is not allowed. You need to either rewrite queries to use all unique key columns in WHERE equal conditions, or rewrite to single-table, single-statement transaction.  Query: ALTER TABLE t1 REPAIR PARTITION p0,p1
+test.t1        repair  error   Unknown - internal error 149 during operation
 test.t1        repair  warning The partition engine, used by table 'test.t1', is deprecated and will be removed in a future release. Please use native partitioning instead.
 Table  Op      Msg_type        Msg_text
-test.t1        check   status  OK
+test.t1        check   error   Partition p1 returned error
+test.t1        check   Error   Using Gap Lock without full unique key in multi-table or multi-statement transactions is not allowed. You need to either rewrite queries to use all unique key columns in WHERE equal conditions, or rewrite to single-table, single-statement transaction.  Query: ALTER TABLE t1 CHECK PARTITION p1
+test.t1        check   error   Unknown - internal error 149 during operation
 test.t1        check   warning The partition engine, used by table 'test.t1', is deprecated and will be removed in a future release. Please use native partitioning instead.
 Warnings:

See also https://jira.percona.com/browse/PS-4762.

vlad-lesin · 2018-09-27T14:27:08Z

Also comments were added for #2493 (comment) and #2493 (comment), and removed for #2493 (comment).

vlad-lesin · 2018-09-27T15:13:13Z

#2493 (comment) - sys_vars testing: https://jenkins.percona.com/view/5.7/job/mysql-5.7-param/1925/ .

vlad-lesin · 2018-09-27T15:14:10Z

Ready for the next round.

laurynas-biveinis · 2018-09-28T08:14:36Z

I don't see any added comments for e.g. sql/cmakelists.txt comment

vlad-lesin · 2018-09-28T09:39:55Z

sql/CMakeLists.txt is fixed, see build tests here: https://jenkins.percona.com/view/5.7/job/mysql-5.7-param/1926/ .

vlad-lesin · 2018-09-28T10:51:52Z

Ready for the next round.

laurynas-biveinis

Please let me know if again some of the comments appear to be replies to hidden/missing comments
Is @georgelorchpercona OK with https://github.com/percona/percona-server/pull/2493/files#r210890922 ?

sql/CMakeLists.txt

laurynas-biveinis · 2018-10-01T08:13:54Z

sql/partitioning/partition_base.cc

+  implicit_emptied= m_file[0]->implicit_emptied;
+  /*
+    Add 2 bytes for partition id in position ref length.
+    ref_length=max_in_all_par


laurynas-biveinis · 2018-10-01T08:14:25Z

sql/partitioning/partition_base.h

+  };
+  enum_handler_status m_handler_status;
+
+  uint m_num_locks;  // For engines like ha_blackhole, which needs no locks


vlad-lesin · 2018-10-01T14:40:37Z

All the above fixes are done.

laurynas-biveinis

LGTM, please address @georgelorchpercona comment and merge

TokuDB and MyRocks native partitioning is implemented as a copy of partition storage engine in 5.7 with corresponding changes. The base class - Partition_base does the most of the work. The general differences from ha_partition class are the following: 1) all code which works with .par files was removed; 2) the code to read partition information from .frm files was added; 3) the responsibility for creating new handlers for separate partitions is moved to Partition_base class descendants; 4) the initialization sequence for native and non-native partitioning handlers is different, that is why create() and open() functions are changed 5) delete and rename table functions are changed both for Partition_base and it's descendants(ha_tokupart and ha_rockspart) because it's necessary to determine if the table is partitioned before calling the corresponding functionality. The changes in the server code are the following: 1) remove .frm file after a table has been removed to let handler the ability to read partition info from .frm file during "delete table" execution; 2) remove "static" keyword from some functions signature to use them outside of their module. Two new system variables were added: tokudb_enable_native_patition - enable native partitioning for TokuDB, read-only, default value is 'off', rocksdb_enable_native_patition - enable native partitioning for RocksDB, read-only, default value is 'off'. The ability to have custom result directory for the certain options combination was added to MTR. To use it just add option "mtr-result-dir" in the certain options group in 'combinations' file. The ability to add only the certain list of tests for the certain option combination is added to MTR. For this purpose the new "mtr-tests-list" option is introduced. This option contains comma-separated list of tests in the current suite, which will be executed for the current options combination. If the option is missed then all tests from the suite will be executed for the combination. As an example of the new options see changed and newly created "combination" files of this commit. The above features are used to test TokuDB and RocksDB native and non-native partitioning, that is why the corresponding test suites have two directories for result files for one directory of tests.

vlad-lesin requested review from laurynas-biveinis and george-lorch August 15, 2018 14:14

george-lorch reviewed Aug 16, 2018

View reviewed changes

laurynas-biveinis suggested changes Aug 20, 2018

View reviewed changes

laurynas-biveinis reviewed Aug 21, 2018

View reviewed changes

sql/partitioning/partition_base.cc Show resolved Hide resolved

laurynas-biveinis reviewed Aug 21, 2018

View reviewed changes

sql/partitioning/partition_base.cc Show resolved Hide resolved

laurynas-biveinis reviewed Aug 21, 2018

View reviewed changes

storage/rocksdb/ha_rocksdb.h Show resolved Hide resolved

laurynas-biveinis reviewed Aug 21, 2018

View reviewed changes

storage/rocksdb/ha_rocksdb.h Show resolved Hide resolved

laurynas-biveinis reviewed Aug 21, 2018

View reviewed changes

laurynas-biveinis suggested changes Aug 21, 2018

View reviewed changes

vlad-lesin force-pushed the native-partitioning-toku-rocks-PR-5.7 branch 2 times, most recently from 8256425 to 0f89dbe Compare August 28, 2018 13:06

vlad-lesin force-pushed the native-partitioning-toku-rocks-PR-5.7 branch 5 times, most recently from 764df7f to d5edfcf Compare September 13, 2018 11:48

laurynas-biveinis suggested changes Sep 13, 2018

View reviewed changes

laurynas-biveinis suggested changes Sep 26, 2018

View reviewed changes

laurynas-biveinis mentioned this pull request Sep 26, 2018

PS-3890: Native Partitioning for TokuDB and RocksDB for 8.0 #2551

Merged

vlad-lesin force-pushed the native-partitioning-toku-rocks-PR-5.7 branch 2 times, most recently from e6ab001 to f4d409c Compare September 27, 2018 12:01

vlad-lesin force-pushed the native-partitioning-toku-rocks-PR-5.7 branch 3 times, most recently from 558581d to 54ac323 Compare September 27, 2018 14:23

vlad-lesin force-pushed the native-partitioning-toku-rocks-PR-5.7 branch from 54ac323 to 1940d6d Compare September 27, 2018 14:34

vlad-lesin force-pushed the native-partitioning-toku-rocks-PR-5.7 branch from 14b0d42 to 6cf118e Compare September 28, 2018 09:43

laurynas-biveinis suggested changes Oct 1, 2018

View reviewed changes

vlad-lesin force-pushed the native-partitioning-toku-rocks-PR-5.7 branch 2 times, most recently from 308624c to 8f5299c Compare October 1, 2018 14:39

laurynas-biveinis approved these changes Oct 2, 2018

View reviewed changes

vlad-lesin force-pushed the native-partitioning-toku-rocks-PR-5.7 branch 4 times, most recently from aafc858 to a250c78 Compare October 10, 2018 14:51

vlad-lesin force-pushed the native-partitioning-toku-rocks-PR-5.7 branch from a250c78 to 826fa7c Compare October 11, 2018 10:56

george-lorch approved these changes Oct 11, 2018

View reviewed changes

vlad-lesin merged commit 189d25b into percona:5.7 Oct 22, 2018

dutow mentioned this pull request Sep 5, 2019

Native partitioning tokudb draft 5.7 #2313

Closed

PS-3889: Native Partitioning for TokuDB and RocksDB for 5.7 #2493

PS-3889: Native Partitioning for TokuDB and RocksDB for 5.7 #2493

Conversation

vlad-lesin commented Aug 15, 2018

george-lorch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

george-lorch commented Aug 17, 2018

laurynas-biveinis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

laurynas-biveinis left a comment

Choose a reason for hiding this comment

laurynas-biveinis left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vlad-lesin commented Sep 27, 2018 • edited Loading

vlad-lesin commented Sep 27, 2018

vlad-lesin commented Sep 27, 2018

vlad-lesin commented Sep 27, 2018

laurynas-biveinis commented Sep 28, 2018

vlad-lesin commented Sep 28, 2018

vlad-lesin commented Sep 28, 2018

laurynas-biveinis left a comment

Choose a reason for hiding this comment

laurynas-biveinis left a comment •

edited

Loading

vlad-lesin commented Sep 27, 2018 •

edited

Loading