Skip to content

Example_yellow_world

Jump to bottom
Isabelle Guyon edited this page Oct 30, 2017 · 22 revisions

Hello World competition

This is a very simple example of competition with result submission.

The competition bundle contains the following files:

reference_data.zip	    The solution to the problem
scoring_program.zip	    The program evaluating the solution
logo.jpg                    The logo
competition.yaml            The YAML configuration file
data.html	            HTML documentation pages	
evaluation.html		
overview.html	
terms_and_conditions.html

The YAML file is the configuration file. It consist in a number of attribute: value pairs. For a full list of attributes, see the Codalab competition YAML definition language.

  1. First section: general settings and HTML pages
title: Example Hello World Competition
description: An example competition where submissions should output "Hello World!"
image: logo.jpg
has_registration: True
html:
    overview: overview.html
    evaluation: evaluation.html
    terms: terms_and_conditions.html
    data: data.html
  1. Section section: phases (only 1 phase in this example)
phases:
    1:
        phasenumber: 1
        label: "First phase"
        start_date: 2013-06-30
        max_submissions: 100
        scoring_program: scoring_program.zip
        reference_data: reference_data.zip
  1. Third section: leaderboard configuration
leaderboard:
    leaderboards:
        RESULTS: &RESULTS
            label: Results
            rank: 1
    columns:
        correct:
            leaderboard: *RESULTS
            label: correct
            rank: 1
            numeric_format: 1

In this case, the scoring program writes a file scores.txt containing:

correct:1

for the sample submission.

Building a scoring program

This example uses python.

evaluate.py - is an example that checks that the submission data matches the truth data, which is "Hello World!"

metadata - this is a file that lists the contents of the program.zip bundle for the CodaLab system.

Once these pieces are assembled they are packages as program.zip which CodaLab can then use to evaluate the submissions for a competition.

Building an ingestion program

We give an example of Ingestion Program.

  • Download the example of ingestion program
  • Upload ingestion_program.zip to My Competitions>My datasets>Create dataset (choose type: ingestion program) go to the editor and uncheck "Results Scoring Only", then select in the ingestion program menu the ingestion program you just uloaded.
  • submit the sample code: submission_4_ingestion.zip For another example of code submission, see the Compute Pi competition.
Clone this wiki locally