Adjudication for FLAT-based annotations

[luutuntin]

Hi Maarten,

This relates to #112. In addition to IAA calculation, we need an adjudication interface to compare different annotations and create a gold standard. What are the adjudication practices of FLAT-based annotation projects that you are aware of?

Best,
Alex

[OsmanMutlu]

Hi Alex,

You can use the merging script provided by proycon to merge annotations from different annotators into one file, then make changes on it in FLAT.

[proycon]

We don't really have an interface for adjudication yet in FLAT, this is mostly deferred to application-specific postprocessing scripts, such as the foliaagreement tool (just a computation without interface though) or through foliamerge as @OsmanMutlu suggested. Sometimes you can also leverage the power of FLAT to do explicit corrections on annotations, if appropriate for your project.

[luutuntin]

Thank you, Osman and Maarten. I got the idea now.

I tried foliamerge (folia-2.1.4 and folia-tools-2.1.6) and received the following error:

File "d:\users\luutuntin\anaconda3\envs\py36\lib\site-packages\foliatools\foliamerge.py", line 41, in mergechildren
if isinstance(e, (folia.AbstractTokenAnnotation, folia.AbstractAnnotationLayer)) and parent.id and not hasattr(e, 'merged'):
AttributeError: module 'folia.main' has no attribute 'AbstractTokenAnnotation'

I couldn't find foliaagreement in FoLiA tools, I tried foliaeval (but I was not sure what is the correct way to provide the value for TYPE argument, e.g. POS or pos-annotation?) and received the following error:

File "d:\users\luutuntin\anaconda3\envs\py36\lib\site-packages\foliatools\foliaeval.py", line 380, in main
evaluation = evaluate(docs, Type, foliaset, False, args.corrections, args.confusionmatrix, args.verbose)
UnboundLocalError: local variable 'Type' referenced before assignment

Did I ignore certain important details?

[proycon]

I tried foliamerge (folia-2.1.4 and folia-tools-2.1.6) and received the following error.

Hmm, indeed something broke when the new library came out. I just committed a fix and will do a new release.

I couldn't find foliaagreement in FoLiA tools, I tried foliaeval (but I was not sure what is the correct way to provide the value for TYPE argument, e.g. POS or pos-annotation?) and received the following error:

Oops, I indeed meant foliaeval. The type value is equal to the FoLiA XML tag, so in this case pos.