ObvDupe filter for ftd

[vikasgupta78]

1. we dont want users to label what they already specify as obvious dupes

2. hypothetically if say all 100% records are filtered due to obv dupe

   what would ftd do?

   It would do nothing. print a message

[vikasgupta78]

Also move obvDupe to common class to be used by matcher and ftd

[vikasgupta78]

Possible Impact on:

1. ZFrame
2. SparkFrame
3. TrainingDataFinder
4. Matcher

[vikasgupta78]

scenarios to be tested for ftd:

1. No obv dupe data [but obv dupe condition there]
2. Everything obv dupe i.e. no ftd left
3. Some obv dupe
4. No obv dupe condition there

[vikasgupta78]

For scenario 2 i.e. Everything obv dupe i.e. no ftd left :

It worked fine, no unmarked data created and following message when label run:

It seems there are no unmarked records at this moment. Please run findTrainingData job to build some pairs to be labelled and then run this labeler.

[sonalgoyal]

it should suggest to the user that it is unable to find any pairs as all pairs it sampled are part of the obvious duplicate condition

[vikasgupta78]

We should do this in ftd stage?

[vikasgupta78]

Added the message in TrainingDataFinder

[vikasgupta78]

There is no impact on linker as linker is just extending from Matcher and all obv dupe related work is being handled by Matcher

[vikasgupta78]

PR #694 Merged