Feature/18 implement hyfd #139

Mstrutov · 2022-11-03T06:58:19Z

HyFD is a hybrid functional dependency discovery algorithm, employing both row-efficient
sample-based agree set generation and column-efficient lattice traversal.

The algorithm has three major components: Sampler, Inductor and Validator --- the former two
representing row-efficient step. Sampler acquires a collection of agree sets using a special
deterministic sampling approach, then Inductor uses this data to enrich an FD prefix tree. This
concludes the first step and the execution moves to Validator, which performs breadth-first
bottom-up traverse of the FDTree, validating the FDs and specializing incorrect candidates.

It is difficult to come up with a criteria or an algorithm for choosing the optimal point to move
from the first step to the second, therefore HyFD uses a notion of efficiency to track how
efficient current step is performing and switches to the other step in case it falls beneath a
configured threshold. Each step's results are useful for the other step: Sampler-Inductor
specialize the prefix tree, which reduces the number or costly PLI checks Validator has to
perform, while Validator yields a collection of row pairs, which invalidated a candidate and can
be used by Sampler to obtain a new agree set.

Thorsten Papenbrock and Felix Naumann. 2016. A Hybrid Approach to Functional Dependency
Discovery. In Proceedings of the 2016 International Conference on Management of Data (SIGMOD
'16). Association for Computing Machinery, New York, NY, USA, 821–833.
https://doi.org/10.1145/2882903.2915203

Mstrutov · 2022-11-03T07:01:59Z

Closes #18, leaving a todo list:

parallelize validator using ParallelFor
implement the Guardian component
investigate performance issues, optimize to beat Metanome
consider implementing a bitset custom iterator to simplify bitset loops, which are used very frequently

src/algorithms/hyfd/hyfd.cpp

src/algorithms/hyfd/validator.cpp

src/algorithms/hyfd/hyfd.h

github-actions

clang-tidy made some suggestions

There were too many comments to post at once. Showing the first 25 out of 38. Check the log or trigger a new build to see more.

github-actions · 2022-11-14T06:52:50Z

src/algorithms/hyfd/efficiency.h

+
+namespace algos::hyfd {
+
+class Sampler::Efficiency {


warning: use of undeclared identifier 'Sampler' [clang-diagnostic-error]

class Sampler::Efficiency { ^

Интересно, а как оно собирается? Перед включением этого хедера всегда (буквально выше) включается хедер, где объявлен Sampler? Не очень какая-то зависимость получается

Это я задумывал как приватный класс у Sampler, в Sampler лежит forward declaration of Efficiency. Таким образом единственный файл, где он должен использоваться, это sampler.cpp, а остальные места его не надо включать.

То же с FDValidations

Возможно, тогда стоит этот код просто целиком перенести в sampler.cpp?

Можно, но хочется по-максимуму вынести выносимое, потому что validator.cpp, например, довольно тяжёлый, не хочется добавлять туда что-либо.

Хотя и много хедеров, которые используются лишь в одном файле, тоже наверное не очень. Хз, по-моему, что так, что так будут свои плюсы и минусы, и бы оставил как есть.

Ну, могу сделать директории validator, sampler и переместить туда всё, им принадлежащее, чтобы как-то виузально хотя бы обозначить, что куда относится.

хм, мне наоборот кажется, что лучше держать локально, если оно нужно только в validator.cpp, чем выносить в отдельный файл. Но можно в целом и в директории объеденить

src/algorithms/hyfd/fd_validations.h

src/algorithms/hyfd/hyfd.cpp

src/algorithms/hyfd/structures/fd_tree_vertex.cpp

src/algorithms/hyfd/structures/non_fd_list.h

src/algorithms/hyfd/validator.cpp

github-actions

clang-tidy made some suggestions

src/algorithms/hyfd/validator.cpp

github-actions

clang-tidy made some suggestions

src/algorithms/hyfd/efficiency.h

src/algorithms/hyfd/fd_validations.h

github-actions · 2022-11-22T05:29:13Z

clang-tidy review says "All clean, LGTM! 👍"

polyntsov

LGTM

polyntsov · 2022-11-25T11:58:39Z

src/model/fd.h

-    // unsigned int Fletcher16() const;
+    [[nodiscard]] RawFD ToRawFD() const {
+        return {lhs_.GetColumnIndices(), rhs_.GetIndex()};
+    }
 };


Перенос строки (:

polyntsov · 2022-11-25T13:42:01Z

src/algorithms/hyfd/fd_validations.h

+    unsigned count_validations_ = 0;
+    unsigned count_intersections_ = 0;
+
+    void add(FDValidations&& other) noexcept {


Может выкинуть bad_alloc

polyntsov · 2022-11-25T13:42:30Z

src/algorithms/hyfd/sampler.cpp

+    assert(first_record_id < compressed_records_->size() &&
+           second_record_id < compressed_records_->size());
+
+    for (size_t i = 0; i < (*compressed_records_)[0].size(); ++i) {


Кажется, все ещё не исправлено

NonFDs essentially stores agree sets, i.e. such a column combination that for some two records their projections are equal, while complements differ. It is filled by the Sampler and read by the Inductor.

Sampler uses a deterministic focused sampling approach to efficiently get most promising agree sets for passing them to the Inductor later.

FDTree is a prefix tree, a core structure of HyFD which represents the FD search space. It is used by the Inductor and the Validator.

Inductor uses the agree set collection received from the Sampler to enrich the FD prefix tree by specializing invalidated candidates.

Validator performs breadth-first bottom-up traverse of the FD prefix tree, validating the FDs and specializing incorrect candidates.

- disable identifier length and recursion checks - allow classes with all variables being public - include libraries as system directories for compiler to run less checks on them

github-actions · 2022-11-26T14:59:41Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2022-11-26T15:02:11Z

clang-tidy review says "All clean, LGTM! 👍"

Mstrutov force-pushed the feature/18-implement-hyfd branch from ba449a2 to 8caffe3 Compare November 3, 2022 07:28

Mstrutov marked this pull request as ready for review November 3, 2022 10:49

Mstrutov requested a review from polyntsov November 3, 2022 10:49

polyntsov added the feature Provides new functionality label Nov 7, 2022

polyntsov requested changes Nov 7, 2022

View reviewed changes

Mstrutov commented Nov 7, 2022

View reviewed changes

src/algorithms/hyfd/hyfd.h Outdated Show resolved Hide resolved

Mstrutov force-pushed the feature/18-implement-hyfd branch 2 times, most recently from 85c4b26 to 2c6fdc7 Compare November 14, 2022 06:42

github-actions bot reviewed Nov 14, 2022

View reviewed changes

Mstrutov force-pushed the feature/18-implement-hyfd branch from 2c6fdc7 to 18bcf12 Compare November 21, 2022 05:39

github-actions bot reviewed Nov 21, 2022

View reviewed changes

src/algorithms/hyfd/validator.cpp Show resolved Hide resolved

src/algorithms/hyfd/validator.cpp Show resolved Hide resolved

src/algorithms/hyfd/validator.cpp Show resolved Hide resolved

Mstrutov force-pushed the feature/18-implement-hyfd branch 3 times, most recently from 41f75ca to 83bc1e4 Compare November 22, 2022 05:07

github-actions bot reviewed Nov 22, 2022

View reviewed changes

src/algorithms/hyfd/efficiency.h Show resolved Hide resolved

src/algorithms/hyfd/fd_validations.h Show resolved Hide resolved

Mstrutov force-pushed the feature/18-implement-hyfd branch from 83bc1e4 to b26452e Compare November 22, 2022 05:23

Mstrutov requested a review from polyntsov November 22, 2022 05:36

polyntsov approved these changes Nov 25, 2022

View reviewed changes

Mstrutov added 8 commits November 26, 2022 20:53

Introduce HyFD algorithm

e6f2503

Implement HyFD main logic using component stubs

44a5312

Implement agree set storage structures

bbfda90

NonFDs essentially stores agree sets, i.e. such a column combination that for some two records their projections are equal, while complements differ. It is filled by the Sampler and read by the Inductor.

Implement Sampler

f34bf13

Sampler uses a deterministic focused sampling approach to efficiently get most promising agree sets for passing them to the Inductor later.

Implement FD prefix tree

57f8e7b

FDTree is a prefix tree, a core structure of HyFD which represents the FD search space. It is used by the Inductor and the Validator.

Implement Inductor

99d557f

Inductor uses the agree set collection received from the Sampler to enrich the FD prefix tree by specializing invalidated candidates.

Implement Validator

abfd5a5

Validator performs breadth-first bottom-up traverse of the FD prefix tree, validating the FDs and specializing incorrect candidates.

Integrate HyFD into main and test infrastructure

3c08e82

Mstrutov force-pushed the feature/18-implement-hyfd branch from b26452e to a47d8ed Compare November 26, 2022 14:54

Refactor HyFD code

75d1bee

Mstrutov added 2 commits November 26, 2022 20:57

Configure clang-tidy tool

7e5c951

- disable identifier length and recursion checks - allow classes with all variables being public - include libraries as system directories for compiler to run less checks on them

Configure clang format header grouping

a5d5dff

Mstrutov force-pushed the feature/18-implement-hyfd branch from a47d8ed to a5d5dff Compare November 26, 2022 14:57

polyntsov merged commit d5f10f3 into main Nov 26, 2022

polyntsov deleted the feature/18-implement-hyfd branch November 26, 2022 15:38

This was referenced Aug 9, 2023

Create clang-tidy config file #142

Closed

Implement HyFD algo #18

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/18 implement hyfd #139

Feature/18 implement hyfd #139

Mstrutov commented Nov 3, 2022

Mstrutov commented Nov 3, 2022

github-actions bot left a comment

github-actions bot Nov 14, 2022

polyntsov Nov 14, 2022

Mstrutov Nov 21, 2022

Mstrutov Nov 21, 2022

polyntsov Nov 21, 2022

Mstrutov Nov 22, 2022

polyntsov Nov 25, 2022

github-actions bot left a comment

github-actions bot left a comment

github-actions bot commented Nov 22, 2022

polyntsov left a comment

polyntsov Nov 25, 2022

polyntsov Nov 25, 2022

polyntsov Nov 25, 2022

github-actions bot commented Nov 26, 2022

github-actions bot commented Nov 26, 2022

Feature/18 implement hyfd #139

Feature/18 implement hyfd #139

Conversation

Mstrutov commented Nov 3, 2022

Mstrutov commented Nov 3, 2022

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Nov 14, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot commented Nov 22, 2022

polyntsov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Nov 26, 2022

github-actions bot commented Nov 26, 2022