Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

issue #473 and #285 #559

Merged
merged 11 commits into from
Apr 1, 2023
Merged

issue #473 and #285 #559

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
60aa747
now schema is in DDL so json parsing not needed
vikasgupta78 Mar 31, 2023
b609627
model id 100 and 103
vikasgupta78 Mar 31, 2023
43501e0
model id 101
vikasgupta78 Mar 31, 2023
ddcf3c9
model id 102
vikasgupta78 Mar 31, 2023
ac7b814
fixing file location, data type
vikasgupta78 Mar 31, 2023
47e904e
model id 105
vikasgupta78 Mar 31, 2023
fba6dd6
model id 106
vikasgupta78 Mar 31, 2023
30d3ec5
model id 103 with stop words
vikasgupta78 Mar 31, 2023
818e77e
model id 104
vikasgupta78 Mar 31, 2023
6852585
data type correction in py for model id 104
vikasgupta78 Mar 31, 2023
9c7b6ce
regenerated models
vikasgupta78 Mar 31, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion examples/amazon-google/AmazonGoogle.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@

args.setOutput(outputPipe)

options = ClientOptions([ClientOptions.PHASE,"link"])
options = ClientOptions([ClientOptions.PHASE,"match"])

#Zingg execution for the given phase
zingg = Zingg(args, options)
Expand Down
22 changes: 2 additions & 20 deletions examples/amazon-google/configWithStopWords.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,16 +48,7 @@
"delimiter": ",",
"header":true
},
"schema":
"{\"type\" : \"struct\",
\"fields\" : [
{\"name\":\"id\", \"type\":\"string\", \"nullable\":false},
{\"name\":\"title\", \"type\":\"string\", \"nullable\":true},
{\"name\":\"description\", \"type\":\"string\", \"nullable\":true},
{\"name\":\"manufacturer\",\"type\":\"string\",\"nullable\":true} ,
{\"name\":\"price\", \"type\":\"double\", \"nullable\":true}
]
}"
"schema": "id string, title string, description string, manufacturer string, price double"
},
{
"name":"google",
Expand All @@ -67,16 +58,7 @@
"delimiter": ",",
"header":true
},
"schema":
"{\"type\" : \"struct\",
\"fields\" : [
{\"name\":\"id\", \"type\":\"string\", \"nullable\":false},
{\"name\":\"title\", \"type\":\"string\", \"nullable\":true},
{\"name\":\"description\", \"type\":\"string\", \"nullable\":true},
{\"name\":\"manufacturer\",\"type\":\"string\",\"nullable\":true} ,
{\"name\":\"price\", \"type\":\"double\", \"nullable\":true}
]
}"
"schema": "id string, title string, description string, manufacturer string, price double"
}
],
"labelDataSampleSize" : 0.4,
Expand Down
2 changes: 1 addition & 1 deletion examples/febrl/FebrlExample.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@

args.setOutput(outputPipe)

options = ClientOptions([ClientOptions.PHASE,"label"])
options = ClientOptions([ClientOptions.PHASE,"match"])

#Zingg execution for the given phase
zingg = Zingg(args, options)
Expand Down
2 changes: 1 addition & 1 deletion examples/iTunes-amazon/iTunesAmazon.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@
outputPipe = CsvPipe("iTunesAmazonresult", "/tmp/iTunesAmazonOutput")
args.setOutput(outputPipe)

options = ClientOptions([ClientOptions.PHASE,"link"])
options = ClientOptions([ClientOptions.PHASE,"match"])

#Zingg execution for the given phase
zingg = Zingg(args, options)
Expand Down
6 changes: 3 additions & 3 deletions examples/ncVoters5M/ncVoters.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
givenname = FieldDefinition("givenname", "string", MatchType.FUZZY)
surname = FieldDefinition("surname", "string", MatchType.EXACT)
suburb = FieldDefinition("suburb","string", MatchType.FUZZY)
postcode = FieldDefinition("postcode", "double", MatchType.EXACT)
postcode = FieldDefinition("postcode", "string", MatchType.EXACT)

fieldDefs = [recid, givenname, surname, suburb, postcode]
args.setFieldDefinition(fieldDefs)
Expand All @@ -21,7 +21,7 @@
#reading dataset into inputPipe and settint it up in 'args'
#below line should not be required if you are reading from in memory dataset
#in that case, replace df with input df
schema = "recid string, givenname string, surname string, suburb string, postcode double "
schema = "recid string, givenname string, surname string, suburb string, postcode string "
inputPipe = CsvPipe("ncVotersTest", "examples/ncVoters5M/5Party-ocp20/", schema)
args.setData(inputPipe)

Expand All @@ -30,7 +30,7 @@

args.setOutput(outputPipe)

options = ClientOptions([ClientOptions.PHASE,"trainMatch"])
options = ClientOptions([ClientOptions.PHASE,"match"])

#Zingg execution for the given phase
zingg = Zingg(args, options)
Expand Down
Binary file added ...lock/zingg.block/.part-00000-07541197-25f5-49a0-a765-1118752066d5-c000.snappy.parquet.crc
View file
Open in desktop
Binary file not shown.
Binary file removed ...lock/zingg.block/.part-00000-3585cf25-210b-42e5-8f66-fce8d46290f1-c000.snappy.parquet.crc
View file
Open in desktop
Binary file not shown.
Binary file added ...del/block/zingg.block/part-00000-07541197-25f5-49a0-a765-1118752066d5-c000.snappy.parquet
View file
Open in desktop
Binary file not shown.
Binary file removed ...del/block/zingg.block/part-00000-3585cf25-210b-42e5-8f66-fce8d46290f1-c000.snappy.parquet
View file
Open in desktop
Binary file not shown.
Binary file modified models/100/model/classifier/best.model/bestModel/metadata/.part-00000.crc
View file
Open in desktop
Binary file not shown.
2 changes: 1 addition & 1 deletion models/100/model/classifier/best.model/bestModel/metadata/part-00000
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"class":"org.apache.spark.ml.PipelineModel","timestamp":1679653593791,"sparkVersion":"3.1.2","uid":"pipeline_37bd53de1842","paramMap":{"stageUids":["vecAssembler_ab6a72b9ff3a","poly_5441bacea924","logreg_8bb21326b817"]},"defaultParamMap":{}}
{"class":"org.apache.spark.ml.PipelineModel","timestamp":1680262758849,"sparkVersion":"3.1.2","uid":"pipeline_7af7179ee2c9","paramMap":{"stageUids":["vecAssembler_d9c5b06776d6","poly_9e0e03752d9e","logreg_c64f720bac8d"]},"defaultParamMap":{}}
Binary file removed ...assifier/best.model/bestModel/stages/0_vecAssembler_ab6a72b9ff3a/metadata/.part-00000.crc
View file
Open in desktop
Binary file not shown.
0 ...mbler_ab6a72b9ff3a/metadata/._SUCCESS.crc → ...mbler_d9c5b06776d6/metadata/._SUCCESS.crc
View file
Open in desktop
File renamed without changes.
Binary file added ...assifier/best.model/bestModel/stages/0_vecAssembler_d9c5b06776d6/metadata/.part-00000.crc
View file
Open in desktop
Binary file not shown.
0 ...cAssembler_ab6a72b9ff3a/metadata/_SUCCESS → ...cAssembler_d9c5b06776d6/metadata/_SUCCESS
View file
Open in desktop
File renamed without changes.
2 changes: 1 addition & 1 deletion ...ssembler_ab6a72b9ff3a/metadata/part-00000 → ...ssembler_d9c5b06776d6/metadata/part-00000
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"class":"org.apache.spark.ml.feature.VectorAssembler","timestamp":1679653593336,"sparkVersion":"3.1.2","uid":"vecAssembler_ab6a72b9ff3a","paramMap":{"outputCol":"z_featurevector","inputCols":["z_sim0","z_sim1","z_sim2","z_sim3","z_sim4","z_sim5","z_sim6","z_sim7","z_sim8","z_sim9","z_sim10","z_sim11","z_sim12","z_sim13","z_sim14","z_sim15","z_sim16","z_sim17","z_sim18","z_sim19"]},"defaultParamMap":{"outputCol":"vecAssembler_ab6a72b9ff3a__output","handleInvalid":"error"}}
{"class":"org.apache.spark.ml.feature.VectorAssembler","timestamp":1680262758941,"sparkVersion":"3.1.2","uid":"vecAssembler_d9c5b06776d6","paramMap":{"outputCol":"z_featurevector","inputCols":["z_sim0","z_sim1","z_sim2","z_sim3","z_sim4","z_sim5","z_sim6","z_sim7","z_sim8","z_sim9","z_sim10","z_sim11","z_sim12","z_sim13","z_sim14","z_sim15","z_sim16","z_sim17","z_sim18","z_sim19"]},"defaultParamMap":{"outputCol":"vecAssembler_d9c5b06776d6__output","handleInvalid":"error"}}
Binary file removed ...model/classifier/best.model/bestModel/stages/1_poly_5441bacea924/metadata/.part-00000.crc
View file
Open in desktop
Binary file not shown.
1 change: 0 additions & 1 deletion .../100/model/classifier/best.model/bestModel/stages/1_poly_5441bacea924/metadata/part-00000
View file
Open in desktop

This file was deleted.

0 ..._poly_5441bacea924/metadata/._SUCCESS.crc → ..._poly_9e0e03752d9e/metadata/._SUCCESS.crc
View file
Open in desktop
File renamed without changes.
Binary file added ...model/classifier/best.model/bestModel/stages/1_poly_9e0e03752d9e/metadata/.part-00000.crc
View file
Open in desktop
Binary file not shown.
0 ...ges/1_poly_5441bacea924/metadata/_SUCCESS → ...ges/1_poly_9e0e03752d9e/metadata/_SUCCESS
View file
Open in desktop
File renamed without changes.
1 change: 1 addition & 0 deletions .../100/model/classifier/best.model/bestModel/stages/1_poly_9e0e03752d9e/metadata/part-00000
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"class":"org.apache.spark.ml.feature.PolynomialExpansion","timestamp":1680262759042,"sparkVersion":"3.1.2","uid":"poly_9e0e03752d9e","paramMap":{"inputCol":"z_featurevector","degree":3,"outputCol":"z_feature"},"defaultParamMap":{"degree":2,"outputCol":"poly_9e0e03752d9e__output"}}
Binary file removed ...bb21326b817/data/.part-00000-0be162b4-2dc5-4539-b471-1e19b42426e8-c000.snappy.parquet.crc
View file
Open in desktop
Binary file not shown.
Binary file removed ...reg_8bb21326b817/data/part-00000-0be162b4-2dc5-4539-b471-1e19b42426e8-c000.snappy.parquet
View file
Open in desktop
Binary file not shown.
Binary file removed ...del/classifier/best.model/bestModel/stages/2_logreg_8bb21326b817/metadata/.part-00000.crc
View file
Open in desktop
Binary file not shown.
1 change: 0 additions & 1 deletion ...00/model/classifier/best.model/bestModel/stages/2_logreg_8bb21326b817/metadata/part-00000
View file
Open in desktop

This file was deleted.

0 .../2_logreg_8bb21326b817/data/._SUCCESS.crc → .../2_logreg_c64f720bac8d/data/._SUCCESS.crc
View file
Open in desktop
File renamed without changes.
Binary file added ...64f720bac8d/data/.part-00000-0faf943f-3ad3-4fff-b133-467678fdf77d-c000.snappy.parquet.crc
View file
Open in desktop
Binary file not shown.
0 ...tages/2_logreg_8bb21326b817/data/_SUCCESS → ...tages/2_logreg_c64f720bac8d/data/_SUCCESS
View file
Open in desktop
File renamed without changes.
Binary file added ...reg_c64f720bac8d/data/part-00000-0faf943f-3ad3-4fff-b133-467678fdf77d-c000.snappy.parquet
View file
Open in desktop
Binary file not shown.
0 ...ogreg_8bb21326b817/metadata/._SUCCESS.crc → ...ogreg_c64f720bac8d/metadata/._SUCCESS.crc
View file
Open in desktop
File renamed without changes.
Binary file added ...del/classifier/best.model/bestModel/stages/2_logreg_c64f720bac8d/metadata/.part-00000.crc
View file
Open in desktop
Binary file not shown.
0 ...s/2_logreg_8bb21326b817/metadata/_SUCCESS → ...s/2_logreg_c64f720bac8d/metadata/_SUCCESS
View file
Open in desktop
File renamed without changes.
1 change: 1 addition & 0 deletions ...00/model/classifier/best.model/bestModel/stages/2_logreg_c64f720bac8d/metadata/part-00000
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"class":"org.apache.spark.ml.classification.LogisticRegressionModel","timestamp":1680262759138,"sparkVersion":"3.1.2","uid":"logreg_c64f720bac8d","paramMap":{"maxIter":100,"featuresCol":"z_feature","labelCol":"z_isMatch","fitIntercept":true,"regParam":1.0E-4,"probabilityCol":"z_probability","threshold":0.4,"predictionCol":"z_prediction"},"defaultParamMap":{"maxIter":100,"featuresCol":"features","elasticNetParam":0.0,"labelCol":"label","fitIntercept":true,"regParam":0.0,"maxBlockSizeInMB":0.0,"rawPredictionCol":"rawPrediction","probabilityCol":"probability","aggregationDepth":2,"tol":1.0E-6,"family":"auto","threshold":0.5,"predictionCol":"prediction","standardization":true}}
Binary file modified models/100/model/classifier/best.model/estimator/metadata/.part-00000.crc
View file
Open in desktop
Binary file not shown.
2 changes: 1 addition & 1 deletion models/100/model/classifier/best.model/estimator/metadata/part-00000
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"class":"org.apache.spark.ml.Pipeline","timestamp":1679653593142,"sparkVersion":"3.1.2","uid":"pipeline_37bd53de1842","paramMap":{"stageUids":["vecAssembler_ab6a72b9ff3a","poly_5441bacea924","logreg_8bb21326b817"]},"defaultParamMap":{}}
{"class":"org.apache.spark.ml.Pipeline","timestamp":1680262758290,"sparkVersion":"3.1.2","uid":"pipeline_7af7179ee2c9","paramMap":{"stageUids":["vecAssembler_d9c5b06776d6","poly_9e0e03752d9e","logreg_c64f720bac8d"]},"defaultParamMap":{}}
Binary file removed ...assifier/best.model/estimator/stages/0_vecAssembler_ab6a72b9ff3a/metadata/.part-00000.crc
View file
Open in desktop
Binary file not shown.
0 ...mbler_ab6a72b9ff3a/metadata/._SUCCESS.crc → ...mbler_d9c5b06776d6/metadata/._SUCCESS.crc
View file
Open in desktop
File renamed without changes.
Binary file added ...assifier/best.model/estimator/stages/0_vecAssembler_d9c5b06776d6/metadata/.part-00000.crc
View file
Open in desktop
Binary file not shown.
0 ...cAssembler_ab6a72b9ff3a/metadata/_SUCCESS → ...cAssembler_d9c5b06776d6/metadata/_SUCCESS
View file
Open in desktop
File renamed without changes.
2 changes: 1 addition & 1 deletion ...ssembler_ab6a72b9ff3a/metadata/part-00000 → ...ssembler_d9c5b06776d6/metadata/part-00000
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"class":"org.apache.spark.ml.feature.VectorAssembler","timestamp":1679653593881,"sparkVersion":"3.1.2","uid":"vecAssembler_ab6a72b9ff3a","paramMap":{"outputCol":"z_featurevector","inputCols":["z_sim0","z_sim1","z_sim2","z_sim3","z_sim4","z_sim5","z_sim6","z_sim7","z_sim8","z_sim9","z_sim10","z_sim11","z_sim12","z_sim13","z_sim14","z_sim15","z_sim16","z_sim17","z_sim18","z_sim19"]},"defaultParamMap":{"outputCol":"vecAssembler_ab6a72b9ff3a__output","handleInvalid":"error"}}
{"class":"org.apache.spark.ml.feature.VectorAssembler","timestamp":1680262758441,"sparkVersion":"3.1.2","uid":"vecAssembler_d9c5b06776d6","paramMap":{"outputCol":"z_featurevector","inputCols":["z_sim0","z_sim1","z_sim2","z_sim3","z_sim4","z_sim5","z_sim6","z_sim7","z_sim8","z_sim9","z_sim10","z_sim11","z_sim12","z_sim13","z_sim14","z_sim15","z_sim16","z_sim17","z_sim18","z_sim19"]},"defaultParamMap":{"outputCol":"vecAssembler_d9c5b06776d6__output","handleInvalid":"error"}}
Binary file removed ...model/classifier/best.model/estimator/stages/1_poly_5441bacea924/metadata/.part-00000.crc
View file
Open in desktop
Binary file not shown.
1 change: 0 additions & 1 deletion .../100/model/classifier/best.model/estimator/stages/1_poly_5441bacea924/metadata/part-00000
View file
Open in desktop

This file was deleted.

0 ..._poly_5441bacea924/metadata/._SUCCESS.crc → ..._poly_9e0e03752d9e/metadata/._SUCCESS.crc
View file
Open in desktop
File renamed without changes.
Binary file added ...model/classifier/best.model/estimator/stages/1_poly_9e0e03752d9e/metadata/.part-00000.crc
View file
Open in desktop
Binary file not shown.
0 ...ges/1_poly_5441bacea924/metadata/_SUCCESS → ...ges/1_poly_9e0e03752d9e/metadata/_SUCCESS
View file
Open in desktop
File renamed without changes.
1 change: 1 addition & 0 deletions .../100/model/classifier/best.model/estimator/stages/1_poly_9e0e03752d9e/metadata/part-00000
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"class":"org.apache.spark.ml.feature.PolynomialExpansion","timestamp":1680262758623,"sparkVersion":"3.1.2","uid":"poly_9e0e03752d9e","paramMap":{"inputCol":"z_featurevector","degree":3,"outputCol":"z_feature"},"defaultParamMap":{"degree":2,"outputCol":"poly_9e0e03752d9e__output"}}
Binary file removed ...del/classifier/best.model/estimator/stages/2_logreg_8bb21326b817/metadata/.part-00000.crc
View file
Open in desktop
Binary file not shown.
1 change: 0 additions & 1 deletion ...00/model/classifier/best.model/estimator/stages/2_logreg_8bb21326b817/metadata/part-00000
View file
Open in desktop

This file was deleted.

0 ...ogreg_8bb21326b817/metadata/._SUCCESS.crc → ...ogreg_c64f720bac8d/metadata/._SUCCESS.crc
View file
Open in desktop
File renamed without changes.
Binary file added ...del/classifier/best.model/estimator/stages/2_logreg_c64f720bac8d/metadata/.part-00000.crc
View file
Open in desktop
Binary file not shown.
0 ...s/2_logreg_8bb21326b817/metadata/_SUCCESS → ...s/2_logreg_c64f720bac8d/metadata/_SUCCESS
View file
Open in desktop
File renamed without changes.
1 change: 1 addition & 0 deletions ...00/model/classifier/best.model/estimator/stages/2_logreg_c64f720bac8d/metadata/part-00000
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"class":"org.apache.spark.ml.classification.LogisticRegression","timestamp":1680262758752,"sparkVersion":"3.1.2","uid":"logreg_c64f720bac8d","paramMap":{"maxIter":100,"featuresCol":"z_feature","labelCol":"z_isMatch","fitIntercept":true,"probabilityCol":"z_probability","predictionCol":"z_prediction"},"defaultParamMap":{"maxIter":100,"featuresCol":"features","elasticNetParam":0.0,"labelCol":"label","fitIntercept":true,"regParam":0.0,"maxBlockSizeInMB":0.0,"rawPredictionCol":"rawPrediction","probabilityCol":"probability","aggregationDepth":2,"tol":1.0E-6,"family":"auto","threshold":0.5,"predictionCol":"prediction","standardization":true}}
Binary file modified models/100/model/classifier/best.model/evaluator/metadata/.part-00000.crc
View file
Open in desktop
Binary file not shown.
2 changes: 1 addition & 1 deletion models/100/model/classifier/best.model/evaluator/metadata/part-00000
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"class":"org.apache.spark.ml.evaluation.BinaryClassificationEvaluator","timestamp":1679653592846,"sparkVersion":"3.1.2","uid":"binEval_193e68f2fb20","paramMap":{"labelCol":"z_isMatch"},"defaultParamMap":{"metricName":"areaUnderROC","rawPredictionCol":"rawPrediction","labelCol":"label","numBins":1000}}
{"class":"org.apache.spark.ml.evaluation.BinaryClassificationEvaluator","timestamp":1680262758064,"sparkVersion":"3.1.2","uid":"binEval_d85e7822d500","paramMap":{"labelCol":"z_isMatch"},"defaultParamMap":{"metricName":"areaUnderROC","labelCol":"label","numBins":1000,"rawPredictionCol":"rawPrediction"}}
Binary file modified models/100/model/classifier/best.model/metadata/.part-00000.crc
View file
Open in desktop
Binary file not shown.
2 changes: 1 addition & 1 deletion models/100/model/classifier/best.model/metadata/part-00000
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"class":"org.apache.spark.ml.tuning.CrossValidatorModel","timestamp":1679653592157,"sparkVersion":"3.1.2","uid":"cv_9205a151ad83","paramMap":{"seed":-1191137437,"numFolds":2,"foldCol":"","estimatorParamMaps":[[{"parent":"logreg_8bb21326b817","name":"regParam","value":"1.0E-4","isJson":"true"},{"parent":"logreg_8bb21326b817","name":"threshold","value":"0.4","isJson":"true"}],[{"parent":"logreg_8bb21326b817","name":"regParam","value":"0.001","isJson":"true"},{"parent":"logreg_8bb21326b817","name":"threshold","value":"0.4","isJson":"true"}],[{"parent":"logreg_8bb21326b817","name":"regParam","value":"0.01","isJson":"true"},{"parent":"logreg_8bb21326b817","name":"threshold","value":"0.4","isJson":"true"}],[{"parent":"logreg_8bb21326b817","name":"regParam","value":"0.1","isJson":"true"},{"parent":"logreg_8bb21326b817","name":"threshold","value":"0.4","isJson":"true"}],[{"parent":"logreg_8bb21326b817","name":"regParam","value":"1.0","isJson":"true"},{"parent":"logreg_8bb21326b817","name":"threshold","value":"0.4","isJson":"true"}],[{"parent":"logreg_8bb21326b817","name":"regParam","value":"1.0E-4","isJson":"true"},{"parent":"logreg_8bb21326b817","name":"threshold","value":"0.45","isJson":"true"}],[{"parent":"logreg_8bb21326b817","name":"regParam","value":"0.001","isJson":"true"},{"parent":"logreg_8bb21326b817","name":"threshold","value":"0.45","isJson":"true"}],[{"parent":"logreg_8bb21326b817","name":"regParam","value":"0.01","isJson":"true"},{"parent":"logreg_8bb21326b817","name":"threshold","value":"0.45","isJson":"true"}],[{"parent":"logreg_8bb21326b817","name":"regParam","value":"0.1","isJson":"true"},{"parent":"logreg_8bb21326b817","name":"threshold","value":"0.45","isJson":"true"}],[{"parent":"logreg_8bb21326b817","name":"regParam","value":"1.0","isJson":"true"},{"parent":"logreg_8bb21326b817","name":"threshold","value":"0.45","isJson":"true"}],[{"parent":"logreg_8bb21326b817","name":"regParam","value":"1.0E-4","isJson":"true"},{"parent":"logreg_8bb21326b817","name":"threshold","value":"0.5","isJson":"true"}],[{"parent":"logreg_8bb21326b817","name":"regParam","value":"0.001","isJson":"true"},{"parent":"logreg_8bb21326b817","name":"threshold","value":"0.5","isJson":"true"}],[{"parent":"logreg_8bb21326b817","name":"regParam","value":"0.01","isJson":"true"},{"parent":"logreg_8bb21326b817","name":"threshold","value":"0.5","isJson":"true"}],[{"parent":"logreg_8bb21326b817","name":"regParam","value":"0.1","isJson":"true"},{"parent":"logreg_8bb21326b817","name":"threshold","value":"0.5","isJson":"true"}],[{"parent":"logreg_8bb21326b817","name":"regParam","value":"1.0","isJson":"true"},{"parent":"logreg_8bb21326b817","name":"threshold","value":"0.5","isJson":"true"}],[{"parent":"logreg_8bb21326b817","name":"regParam","value":"1.0E-4","isJson":"true"},{"parent":"logreg_8bb21326b817","name":"threshold","value":"0.55","isJson":"true"}],[{"parent":"logreg_8bb21326b817","name":"regParam","value":"0.001","isJson":"true"},{"parent":"logreg_8bb21326b817","name":"threshold","value":"0.55","isJson":"true"}],[{"parent":"logreg_8bb21326b817","name":"regParam","value":"0.01","isJson":"true"},{"parent":"logreg_8bb21326b817","name":"threshold","value":"0.55","isJson":"true"}],[{"parent":"logreg_8bb21326b817","name":"regParam","value":"0.1","isJson":"true"},{"parent":"logreg_8bb21326b817","name":"threshold","value":"0.55","isJson":"true"}],[{"parent":"logreg_8bb21326b817","name":"regParam","value":"1.0","isJson":"true"},{"parent":"logreg_8bb21326b817","name":"threshold","value":"0.55","isJson":"true"}]]},"defaultParamMap":{"seed":-1191137437,"numFolds":3,"foldCol":""},"avgMetrics":[0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571],"persistSubModels":false}
{"class":"org.apache.spark.ml.tuning.CrossValidatorModel","timestamp":1680262757721,"sparkVersion":"3.1.2","uid":"cv_b6b3e46792cc","paramMap":{"numFolds":2,"seed":-1191137437,"foldCol":"","estimatorParamMaps":[[{"parent":"logreg_c64f720bac8d","name":"regParam","value":"1.0E-4","isJson":"true"},{"parent":"logreg_c64f720bac8d","name":"threshold","value":"0.4","isJson":"true"}],[{"parent":"logreg_c64f720bac8d","name":"regParam","value":"0.001","isJson":"true"},{"parent":"logreg_c64f720bac8d","name":"threshold","value":"0.4","isJson":"true"}],[{"parent":"logreg_c64f720bac8d","name":"regParam","value":"0.01","isJson":"true"},{"parent":"logreg_c64f720bac8d","name":"threshold","value":"0.4","isJson":"true"}],[{"parent":"logreg_c64f720bac8d","name":"regParam","value":"0.1","isJson":"true"},{"parent":"logreg_c64f720bac8d","name":"threshold","value":"0.4","isJson":"true"}],[{"parent":"logreg_c64f720bac8d","name":"regParam","value":"1.0","isJson":"true"},{"parent":"logreg_c64f720bac8d","name":"threshold","value":"0.4","isJson":"true"}],[{"parent":"logreg_c64f720bac8d","name":"regParam","value":"1.0E-4","isJson":"true"},{"parent":"logreg_c64f720bac8d","name":"threshold","value":"0.45","isJson":"true"}],[{"parent":"logreg_c64f720bac8d","name":"regParam","value":"0.001","isJson":"true"},{"parent":"logreg_c64f720bac8d","name":"threshold","value":"0.45","isJson":"true"}],[{"parent":"logreg_c64f720bac8d","name":"regParam","value":"0.01","isJson":"true"},{"parent":"logreg_c64f720bac8d","name":"threshold","value":"0.45","isJson":"true"}],[{"parent":"logreg_c64f720bac8d","name":"regParam","value":"0.1","isJson":"true"},{"parent":"logreg_c64f720bac8d","name":"threshold","value":"0.45","isJson":"true"}],[{"parent":"logreg_c64f720bac8d","name":"regParam","value":"1.0","isJson":"true"},{"parent":"logreg_c64f720bac8d","name":"threshold","value":"0.45","isJson":"true"}],[{"parent":"logreg_c64f720bac8d","name":"regParam","value":"1.0E-4","isJson":"true"},{"parent":"logreg_c64f720bac8d","name":"threshold","value":"0.5","isJson":"true"}],[{"parent":"logreg_c64f720bac8d","name":"regParam","value":"0.001","isJson":"true"},{"parent":"logreg_c64f720bac8d","name":"threshold","value":"0.5","isJson":"true"}],[{"parent":"logreg_c64f720bac8d","name":"regParam","value":"0.01","isJson":"true"},{"parent":"logreg_c64f720bac8d","name":"threshold","value":"0.5","isJson":"true"}],[{"parent":"logreg_c64f720bac8d","name":"regParam","value":"0.1","isJson":"true"},{"parent":"logreg_c64f720bac8d","name":"threshold","value":"0.5","isJson":"true"}],[{"parent":"logreg_c64f720bac8d","name":"regParam","value":"1.0","isJson":"true"},{"parent":"logreg_c64f720bac8d","name":"threshold","value":"0.5","isJson":"true"}],[{"parent":"logreg_c64f720bac8d","name":"regParam","value":"1.0E-4","isJson":"true"},{"parent":"logreg_c64f720bac8d","name":"threshold","value":"0.55","isJson":"true"}],[{"parent":"logreg_c64f720bac8d","name":"regParam","value":"0.001","isJson":"true"},{"parent":"logreg_c64f720bac8d","name":"threshold","value":"0.55","isJson":"true"}],[{"parent":"logreg_c64f720bac8d","name":"regParam","value":"0.01","isJson":"true"},{"parent":"logreg_c64f720bac8d","name":"threshold","value":"0.55","isJson":"true"}],[{"parent":"logreg_c64f720bac8d","name":"regParam","value":"0.1","isJson":"true"},{"parent":"logreg_c64f720bac8d","name":"threshold","value":"0.55","isJson":"true"}],[{"parent":"logreg_c64f720bac8d","name":"regParam","value":"1.0","isJson":"true"},{"parent":"logreg_c64f720bac8d","name":"threshold","value":"0.55","isJson":"true"}]]},"defaultParamMap":{"numFolds":3,"seed":-1191137437,"foldCol":""},"avgMetrics":[0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571,0.9821428571428571],"persistSubModels":false}
Binary file added ...lock/zingg.block/.part-00000-9ac38206-8cb9-4357-a37b-7279e46adda7-c000.snappy.parquet.crc
View file
Open in desktop
Binary file not shown.
Binary file added ...del/block/zingg.block/part-00000-9ac38206-8cb9-4357-a37b-7279e46adda7-c000.snappy.parquet
View file
Open in desktop
Binary file not shown.
Binary file modified models/101/model/classifier/best.model/bestModel/metadata/.part-00000.crc
View file
Open in desktop
Binary file not shown.
2 changes: 1 addition & 1 deletion models/101/model/classifier/best.model/bestModel/metadata/part-00000
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"class":"org.apache.spark.ml.PipelineModel","timestamp":1680245294860,"sparkVersion":"3.1.2","uid":"pipeline_f729a6b01032","paramMap":{"stageUids":["vecAssembler_5a4a15106c47","poly_993555b1f9db","logreg_d93ca38a2205"]},"defaultParamMap":{}}
{"class":"org.apache.spark.ml.PipelineModel","timestamp":1680265841416,"sparkVersion":"3.1.2","uid":"pipeline_f47d307ed35b","paramMap":{"stageUids":["vecAssembler_7e2cbf38f293","poly_747cf01a21ce","logreg_549169031548"]},"defaultParamMap":{}}
0 ...mbler_d00f2c39b11a/metadata/._SUCCESS.crc → ...mbler_7e2cbf38f293/metadata/._SUCCESS.crc
View file
Open in desktop
File renamed without changes.
Binary file added ...assifier/best.model/bestModel/stages/0_vecAssembler_7e2cbf38f293/metadata/.part-00000.crc
View file
Open in desktop
Binary file not shown.
0 ...cAssembler_d00f2c39b11a/metadata/_SUCCESS → ...cAssembler_7e2cbf38f293/metadata/_SUCCESS
View file
Open in desktop
File renamed without changes.
1 change: 1 addition & 0 deletions ...el/classifier/best.model/bestModel/stages/0_vecAssembler_7e2cbf38f293/metadata/part-00000
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"class":"org.apache.spark.ml.feature.VectorAssembler","timestamp":1680265841509,"sparkVersion":"3.1.2","uid":"vecAssembler_7e2cbf38f293","paramMap":{"inputCols":["z_sim0","z_sim1","z_sim2","z_sim3","z_sim4","z_sim5","z_sim6","z_sim7","z_sim8","z_sim9","z_sim10","z_sim11","z_sim12","z_sim13","z_sim14","z_sim15","z_sim16","z_sim17"],"outputCol":"z_featurevector"},"defaultParamMap":{"handleInvalid":"error","outputCol":"vecAssembler_7e2cbf38f293__output"}}
0 ..._poly_443766300674/metadata/._SUCCESS.crc → ..._poly_747cf01a21ce/metadata/._SUCCESS.crc
View file
Open in desktop
File renamed without changes.
Binary file added ...model/classifier/best.model/bestModel/stages/1_poly_747cf01a21ce/metadata/.part-00000.crc
View file
Open in desktop
Binary file not shown.
0 ...ges/1_poly_443766300674/metadata/_SUCCESS → ...ges/1_poly_747cf01a21ce/metadata/_SUCCESS
View file
Open in desktop
File renamed without changes.
1 change: 1 addition & 0 deletions .../101/model/classifier/best.model/bestModel/stages/1_poly_747cf01a21ce/metadata/part-00000
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"class":"org.apache.spark.ml.feature.PolynomialExpansion","timestamp":1680265841610,"sparkVersion":"3.1.2","uid":"poly_747cf01a21ce","paramMap":{"outputCol":"z_feature","inputCol":"z_featurevector","degree":3},"defaultParamMap":{"outputCol":"poly_747cf01a21ce__output","degree":2}}
0 .../2_logreg_0079fab6ffc2/data/._SUCCESS.crc → .../2_logreg_549169031548/data/._SUCCESS.crc
View file
Open in desktop
File renamed without changes.
Binary file added ...49169031548/data/.part-00000-4488bb70-c1b3-4678-ba31-5c344b30d8ea-c000.snappy.parquet.crc
View file
Open in desktop
Binary file not shown.
0 ...tages/2_logreg_0079fab6ffc2/data/_SUCCESS → ...tages/2_logreg_549169031548/data/_SUCCESS
View file
Open in desktop
File renamed without changes.
Binary file added ...reg_549169031548/data/part-00000-4488bb70-c1b3-4678-ba31-5c344b30d8ea-c000.snappy.parquet
View file
Open in desktop
Binary file not shown.
0 ...ogreg_0079fab6ffc2/metadata/._SUCCESS.crc → ...ogreg_549169031548/metadata/._SUCCESS.crc
View file
Open in desktop
File renamed without changes.
Binary file added ...del/classifier/best.model/bestModel/stages/2_logreg_549169031548/metadata/.part-00000.crc
View file
Open in desktop
Binary file not shown.
0 ...s/2_logreg_0079fab6ffc2/metadata/_SUCCESS → ...s/2_logreg_549169031548/metadata/_SUCCESS
View file
Open in desktop
File renamed without changes.
Loading