Skip to content

Final Evaluation

Jump to bottom
Ian Chen edited this page Jul 29, 2022 · 20 revisions

The page provides information on the infrastructure setup will be using for final evaluation. It also includes details on team solution submission.

Simulation infrastructure

Final evaluation will be run on cloud machines on AWS EC2. Here is the set-up:

communication infra

  • The simulator will be run in a container in its own dedicated machine instance.
  • Other separate machines instances will be launched to run the submitted team solutions.
    • Multiple team solutions will be run on one machine instance. For example, UAV1 and UAV2 may run on the same machine instance.
  • The team solution communicates to the simulator over a bridge container which will be co-located in the same machine instance as the team solution container.

We will use g3.4xlarge instance type for all machine instances. Simulation will run headless (no GUI) on the cloud machines.

For practice before final evaluation, you can either reproduce our infrastructure setup locally or on AWS EC2. Take a look at the Simulation setup using Docker Compose page on how to reproduce the simulation setup on a local machine. For testing on AWS EC2, we recommend testing with the same AWS instance type as we can not guarantee that the simulation will run properly on a different cloud provider / instance type.

Submission

Submission should include:

  • One solution docker image
    • Only one solution image should be submitted. The same solution will be used for all vehicles. An argument will be passed to the container to indicate the vehicle it is associated with.
  • One yaml configuration file containing all vehicles and sensor payloads to be spawned into simulation.

Please see the Preparing Solution Docker Image page for information on how to prepare your Docker image for submission.

Instructions

  1. Install the AWS CLI tool and configure it with your teams credentials. Unique AWS credentials will be provided to you. To configure aws to use your keys, you can either:

    1. Run the command:

      aws configure

      Enter the keys and keep other settings the default. Default region should be us-east-1.

    2. Use named profiles. Open ~/.aws/credentials and add a new profile:

      [mbzirc_user]
aws_access_key_id=<your_aws_access_key_id>
aws_secret_access_key=<your_aws_secrete_access_key>

      Then set the env variable:

      export AWS_PROFILE=mbzirc_user

  2. Use the mbzirc_submit.bash script to tag and upload your docker image. Usage:

    bash mbzirc_submit.bash <team_name> <local_docker_image> <local_tag> <upstream_tag> <config_file>
    • team_name: Please use the team name in the credentials provided to you. Must be lowercase letters. This is also the name of your Docker repository.
    • local_docker_image: This is the name of your Docker image. If you type docker image list, you should use the appropriate name under the REPOSITORY column.
    • local_tag: This is the tag associated with your docker image. A common tag is 'latest'.
    • upstream_tag: This is the tag you want associated with your image when uploaded to the repository. This will let you version your docker images. Please specify either, test, dry_run or final_evaluation
    • config_file: Path to your yaml configuration file containing all vehicles and sensor payloads to be spawned into simulation

    For example, if your team name is foo, you have built the example solution docker image called mbzirc_seed:latest, you want to submit this image for dry run, and there is a team.yaml in the current path:

    bash mbzirc_submit.bash foo mbzirc_seed latest dry_run ./team.yaml

    As for submitting your solution for final evaluation:

    bash mbzirc_submit.bash foo mbzirc_seed latest final_evaluation ./team.yaml

    This script will log you into docker, tag your image, push the image to your repository on AWS, and upload your config file to your s3 bucket on AWS.

    Here is an example output of successful submission:

    Logging into docker
WARNING! Your password will be stored unencrypted in /home/developer/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
Tagging image mbzirc_seed:latest with final_evaluation
Pushing to docker repository
The push refers to repository [200670743174.dkr.ecr.us-east-1.amazonaws.com/mbzirc/foo]
d55f14d8ed82: Pushed 
9b61fd5d061f: Pushed 
6832188e7901: Pushed 
a58f9c58c302: Pushed 
5c846eef36c0: Pushed 
83a234b1bc63: Pushed 
0f61b971b629: Pushed 
1d638b9e62f4: Pushed 
d706a4a928cf: Pushed 
3097b3556115: Pushed 
c11f93922563: Pushed 
ad62eb903a80: Pushed 
2ef9f14570a9: Pushed 
c7fa1bd01201: Pushed 
62db52b6d5e9: Pushed 
69560be64149: Pushed 
c5cc56868f26: Pushed 
ccda132558f7: Pushed 
267339c207c6: Pushed 
8e2231cfe414: Pushed 
1962167633ed: Pushed 
047aa3d1de06: Pushed 
df2e8c86e495: Pushed 
20026237d960: Pushed 
c3bc5ba26848: Pushed 
bf8cedc62fb3: Pushed 
final_evaluation: digest: sha256:af98add961de2c311cfa634f041b999b4f26f3e112a7c6435bf584b8950b9971 size: 5779
Docker image URL: 200670743174.dkr.ecr.us-east-1.amazonaws.com/mbzirc/foo:final_evaluation
Uploading config file to s3
upload: ./team.yaml to s3://cloudsim-mbzirc-logs/foo/config/config.yaml
Clone this wiki locally