This repository contains the evaluation system used in the ACM SIGMOD Programming Contest 2015.
It supports automated sandboxed execution of untrusted code in a Docker environment. Besides 
isolation, the user program can also be restricted to use only a certain amount of memory
and limit its total runtime.

The basic principle is to have two independent systems. One to host a dashboard where the users
can submit their programs (SubSys) and a second for evaluation of these programs (EvalSys). For
security purposes only SubSys should be able to connect to EvalSys and not the other way round.
That way if the EvalSys becomes compromised due to bugs in the virtualization layer, it is not
possible to also compromise the SubSys.

On SubSys the submission_agent is responsible for tracking the incoming submissions and sending 
them in timestamp order for execution to the EvalSys and also pushing the result of each submission
into a database. On EvalSys the eval_agent is running and it will watch a folder for submissions that
should be evaluated. These will be running using the sandbox scripts and the results will be put into
a seperated folder. The submission_agent will periodically check this folder and copy the result files
back, update the database and send the next submission for evaluation if one is available.

During all these steps the user submission is only moved and copied. Only inside the container
is it extracted and run. The sandbox bench script currently expects a submission to contain two files:
	- ./compile.sh - Compiles the user program
	- ./run.sh - Executes the user program
Additionally it expects a testdriver program in the sandbox folder which is used to run the tests on 
the user program. Before running the benchmark/test the bench.sh script warms up the input files by
touching them once to enable more stable runtimes. Stdout and Stdin are logged and saved so that in
case of problems, these could be made accessible to the submitters. These logs are truncated to 1kb
each. The sandbox has a shared directory with the host system through which new files can be made 
accessible during the benchmark. Each step, the extraction, compilation and execution of the user 
program can be timed seperately.

When allowing the submission through a web interface, the files should bestored under an artifical name
and not the user name to avoid potential security risks. All agents are basically stateless and can
automatically resume after e.g. a crash.

The only dependencies are:
	- ssh access from SubSys to EvalSys
	- Docker, Java and Bash

All programs are currently tailored for the SIGMOD Programming Contest 2015, but should be easily
changeable to also support other kind of systems.