Output Files

The algorithm generates several files as output, in the results/ folder.

object-probabilities.txt

This file contains the estimated category for each object in the dataset, together with estimates about the quality of the labeling for each example. Here is the meaning of the columns. (Note: references to DS and the Dawid-Skene algorithm reference the DS variant implemented by Get-Another-Label. See original workshop paper)

Field	Meaning
Object	Object Id
Correct_Category	Correct Category (subject to evaluation data availability)
DS_MaxLikelihood_Category	Maximum Likelihood Category, according to the Expectation Maximization algorithm
MV_MaxLikelihood_Category	Maximum Likelihood Category, according to the Majority Voting Algorithm
DS_MinCost_Category	Minimum Cost Category, according to the Expectation Maximization algorithm
MV_MinCost_Category	Minimum Cost Category, according to the Majority Voting Algorithm
DS_Pr[...]	The probability that the object belongs to the category, according to the Dawid-Skene algorithm
MV_Pr[...]	The probability that the object belongs to the category, according to naive majority voting
DataCost_Estm_DS_Exp	Estimated classification cost (DS_Exp metric)
DataCost_Estm_MV_Exp	Estimated classification cost (MV_Exp metric)
DataCost_Estm_NoVote_Exp	Baseline classification cost (random spammer)
DataCost_Estm_DS_Min	Estimated classification cost (DS_Min metric)
DataCost_Esm_MV_Min	Estimated classification cost (MV_Min metric)
DataCost_Estm_DS_ML	Estimated classification cost (DS_ML metric)
DataCost_Estm_MV_ML	Estimated classification cost (MV_ML metric)
DataCost_Estm_NoVote_Min	Baseline classification cost (strategic spammer)
DataQuality_Estm_DS_Exp	Estimated data quality, EM algorithm, soft label
DataQuality_Estm_MV_Exp	Estimated data quality, naive soft label
DataQuality_Estm_DS_Min	Estimated data quality, EM algorithm, mincost
DataQuality_Estm_MV_Min	Estimated data quality, naive mincost label
DataQuality_Estm_DS_ML	Estimated data quality, EM algorithm, maximum likelihood
DataQuality_Estm_MV_ML	Estimated data quality, naive majority label
DataCost_Eval_MV_ML	Actual classification cost for majority vote classification
DataCost_Eval_DS_ML	Actual classification cost for EM, maximum likelihood classification
DataCost_Eval_MV_Soft	Actual classification cost for naive soft-label classification
DataCost_Eval_DS_Soft	Actual classification cost for EM, soft-label classification
DataCost_Eval_MV_Min	Actual classification cost for naive min-cost classification
DataCost_Eval_DS_Min	Actual classification cost for EM, min-cost classification
DataQuality_Eval_DS_ML	Actual data quality, EM algorithm, maximum likelihood
DataQuality_Eval_DS_Min	Actual data quality, EM algorithm, mincost
DataQuality_Eval_DS_Soft	Actual data quality, EM algorithm, soft label
DataQuality_Eval_MV_ML	Actual data quality, naive majority label
DataQuality_Eval_MV_Min	Actual data quality, naive mincost label
DataQuality_Eval_MV_Soft	Actual data quality, naive soft label

worker-statistics-summary.txt

Average of the following metrics:

Field	Description
Worker	The id of the worker
WorkerQuality_Estm_DS_Exp_n	Estimated worker quality (non-weighted, DS_Exp metric)
WorkerQuality_Estm_DS_Exp_w	Estimated worker quality (weighted, DS_Exp metric)
WorkerQuality_Estm_DS_ML_n	Estimated worker quality (non-weighted, DS_ML metric)
WorkerQuality_Estm_DS_ML_w	Estimated worker quality (weighted, DS_ML metric)
WorkerQuality_Estm_DS_Min_n	Estimated worker quality (non-weighted, DS_Min metric)
WorkerQuality_Estm_DS_Min_w	Estimated worker quality (weighted, DS_Min metric)
WorkerQuality_Eval_DS_Exp_n	Actual worker quality (non-weighted, DS_Exp metric)
WorkerQuality_Eval_DS_Exp_w	Actual worker quality (weighted, DS_Exp metric)
WorkerQuality_Eval_DS_ML_n	Actual worker quality (non-weighted, DS_ML metric)
WorkerQuality_Eval_DS_ML_w	Actual worker quality (weighted, DS_ML metric)
WorkerQuality_Eval_DS_Min_n	Actual worker quality (non-weighted, DS_Min metric)
WorkerQuality_Eval_DS_Min_w	Actual worker quality (weighted, DS_Min metric)
Number of labels	Labels per worker
Gold Tests	Gold tests per worker

worker-statistics-detail.txt

Same as above, but individually. It also lists the confusion matrix for each worker, both using the estimates of the Dawid-Skene algorithm and showing the confusion matrices based on the evaluation data.

summary.txt

Shows the summary reports (also sent to stderr when finished). Results are averaged from the values outlined in the object-probabilities.txt file (see above).

priors.txt

This file lists the estimated prior values for the different object categories as estimated by the Dawid-Skene algorithm. When the algorithm is run with fixed priors, the file simply lists the fixed prior values.

confusion-matrix.txt

A summary of confusion matrices for each algorithm used.