wiki:DSC/zf_validation

Version 4 (modified by seskar, 11 years ago) ( diff )

—

Image Validation Procedure (Preliminary Torunament)

Table of Contents

  1. DARPA Spectrum Challenge
    1. Hurdles
    2. Wildcard tournament
    3. Preliminary round
    4. Final round
  2. DARPA Spectrum Challenge Q&A
    1. Tournament Q&A
    2. Hurdles Q&A
  3. Tutorial for Hurdle3-like experiment
    1. Objective
    2. Hardware / Software Resources Used
    3. Setting up the nodes
    4. Running the experiment script
  4. Tournament game example !#1
    1. Objective
    2. Hardware / Software Resources Used
    3. Setting up the nodes
    4. Running the experiment script
  5. Match Execution Framework (Post Preliminary)
    1. Objective
    2. Hardware / Software Resources Used
    3. Setting Up the Arena for COMPETITIVE
    4. Setting Up the Arena for COOPERATIVE
    5. Running the Validation/Experiment Script
    6. Running Matches on Sandboxes
  6. Bot Images
  7. Tournament Scripts
    1. Objective
    2. Running Competitive Matches
    3. Running Cooperative Matches
  8. Match Visualization
    1. Competitive Match Visualization
    2. Cooperative Match Visualization
  9. Calibration Reports (2014-03-13)
    1. Primary Competitive
    2. Secondary Competitive
    3. Primary Cooperative
    4. Secondary Cooperative
    5. Full Arena
    6. Calibration Procedure
  10. Image Validation Procedure (Preliminary Torunament)
  11. Image Validation Procedure (Final Tournament)
    1. Validate With Basic Script
    2. Validate With Tournament Scripts

  1. Decide on the image name (e.g. team-image-name.ndz) and store it in /.orbit-image
  2. Run prepare.sh and let the node shut down
  3. Run "omf save" and save the image 
          omf save -n node2-1.grid.obit-lab.org
  4. ssh into repository1 and rename the image 
      ssh repository1 
  cd /export/omf/omf-images
  mv expf-id-based-image-name.ndz team-image-name.ndz
  5. go back to the grid console and load the image into one pair of match nodes i.e.: 
      omf load -i team-image-name.ndz -t system:topo:dsc-teamA
  6. turn the imaged nodes on 
      omf tell -a on -t system:topo:dsc-teamA
  7. Execute competitive match and receive at least one packet successfully 
      omf exec system:exp:dsc-match -- --mode COMP
  8. Execute cooperative match and receive at least one packet successfully 
      omf exec system:exp:dsc-match -- --mode COOP
  9. Not alter the image in any way (including renaming it) for the image you want to be used in the preliminary competition.
  10. If you have multiple images that were scored, tell the ORBIT team which one you want to be used otherwise last one that was successfully verified will be used.

Image Validation Procedure (Final Tournament)

Please note that the submitted image has to be validated for both competitive and cooperative match

  1. Decide on the image name (e.g. team-image-name.ndz) and store it in /.orbit-image
  2. Run prepare.sh and let the node shut down
  3. Run "omf save" and save the image 
          omf save -n node2-1.grid.obit-lab.org
  4. ssh into repository1 and rename the image 
      ssh repository1 
  cd /export/omf/omf-images
  mv expf-id-based-image-name.ndz team-image-name.ndz

Validate With Basic Script

Validate Competitive Match

  1. On the grid console load the saved image into one pair of the competitive match nodes and load one of the bots into the other pair i.e.: 
      omf load -i team-image-name.ndz -t system:topo:dsc-teamXp
  omf load -i dsc-bot1.ndz -t system:topo:dsc-teamYp
  2. Execute competitive match and receive at least one packet successfully 
      omf exec system:exp:dsc-match -- --mode COMP --arena PRI

Validate Cooperative Match

  1. On the grid console load the saved image into one pair of cooperative match nodes and load two other bots into the other two sets of cooperative match nodes i.e.: 
      omf load -i team-image-name.ndz -t system:topo:dsc-teamAp
  omf load -i dsc-bot6.ndz -t system:topo:dsc-teamBp
  omf load -i dsc-bot5.ndz -t system:topo:dsc-teamCp
  2. Execute competitive match and receive at least one packet successfully 
      omf exec system:exp:dsc-match -- --mode COOP --arena PRI
  3. After every successful validation, you should receive confirmation email with the score. Note that you can validate multiple images.
  4. Do not alter the image in any way (including renaming it) once you received validation emails for the image you want to be used in the final competition.
  5. Fill in the submission form with the image name that was successfully validated (you have to choose one image out of the set that passed the validation procedure) and submit it over email.

Validate With Tournament Scripts

  1. To set the extended framework create the three configuration files in current working directory:

teams.yaml

comp_matches.yaml

coop_matches.yaml 

T0:
  name: Team Name
  image: team-image-name.ndz
T1:
  name: ORBIT team 1
  image: dsc-bot1.ndz
T2:
  name: ORBIT team 2
  image: dsc-bot2.ndz 
T3:
  name: ORBIT team 3
  image: dsc-bot3.ndz
T4:
  name: ORBIT team 1
  image: dsc-bot4.ndz
T5:
  name: ORBIT team 2
  image: dsc-bot5.ndz
T6:
  name: ORBIT team 3
  image: dsc-bot6.ndz

G1:
   1: T0
   2: T2

G1:
   1: T0
   2: T6
   3: T5
  1. For competitive validation, in the same directory execute: 
      dsc-comp 1 A
  2. For cooperative validation, in the same directory execute: 
      dsc-coop 1
  3. After every successful validation, you should receive confirmation email with the score. Note that you can validate multiple images.
  4. Do not alter the image in any way (including renaming it) once you received validation emails if you want to use it for the final competition.
  5. Fill in the submission form with the image name that was successfully validated (you have to choose one image out of the set that passed the validation procedure) and submit it over email.
  6. We strongly encourage teams to play at least one match in all 6 possible arenas i.e.: 
      dsc-comp 1 A
  dsc-comp 1 B
  dsc-comp -s 1 A
  dsc-comp -s 1 B
  dsc-coop 1
  dsc-coop -s 1
Note: See TracWiki for help on using the wiki.