Changes between Initial Version and Version 1 of DSC/Backup/zdc_framework


Ignore:
Timestamp:
Nov 12, 2013, 4:49:24 PM (11 years ago)
Author:
Ilya Chigirev
Comment:

Legend:

Unmodified
Added
Removed
Modified
  • DSC/Backup/zdc_framework

    v1 v1  
     1= Match Execution Framework =
     2''(Last Modified: Monday 8/12/2013 11:45PM)''
     3
     4[[TOC(DSC*,depth=2)]]
     5
     6== Objective ==
     7Use an OEDL script to execute either a competitive-style or cooperative-style game for both Preliminary matches of the DARPA Spectrum Challenge. Even though the use of the framework is illustrated by using 2 house bots (or 3 in the case of a cooperative game), in the actual competition, the teams will use their own radio implementations. An additional purpose of the framework is to provide validation to the images that teams submit for the Preliminary matches: teams must use the script in this framework to validate their '''image''' before submission.  Validation is required for both competitive and cooperative style matches.
     8
     9The OEDL script uses three pairs of nodes: dsc-teamA, dsc-teamB, dsc-teamC (the actual topology descriptions are '''system:topo:dsc-team{A,B,C}'''). The competitive matches will use teamA and teamC nodes, while cooperative matches will use teamA, teamB and teamC nodes (contestant radios will be only on teamA nodes for image validation, while house bots will be using teamB and teamC nodes); configurations should be considered to be fixed and will be changed only in the case of hardware failures.
     10
     11== Hardware / Software Resources Used ==
     12 1. Four grid nodes with USRP N210s (two for each team) for competitive match and six nodes with USRP N210s for cooperative match (two for each team).
     13 2. ''dsc-wbot1.ndz,dsc-wbot2.ndz,dsc-wbot3.ndz'': house bot disk images with the prerequisite software to configure the USRPs and execute a house radio bot.
     14 3. ''team-image.ndz'': disk image that contains the team's radio. Please note that it is this image file that will be validated, and that once the image has been loaded for validation, no modification to the radio designs within the image prior to completion of the validation process are allowed. 
     15 4. Scoring packet server: This is the server that resides on the infrastructure machine and is used as the source of data packets as well as the sink for teams to submit received packets for scoring.
     16 5. The OEDL experiment script to execute the matches and perform image validation.
     17
     18Teams must ensure that their images contain '''/root/dsc-code_tx''' and '''/root/dsc-code_rx'''. Only these names will be used for testing and scoring. Further, these executables/scripts must be able to receive as input three parameters:
     19 * '''-f 1700000000''' : option for specifying the center frequency (in this example 1.7GHz)
     20 * '''-s idb2''' : option for specifying the packet server host name (in this example idb2)
     21 * '''--mode COMP''' or '''--mode COOP''' : option that specifies whether the match style is a competitive or cooperative match. 
     22
     23Please note that idb2 and 1700000000 are provided merely for illustration purposes and that the teams must allow for the values passed to be entered by the evaluators. Finally, no modifications to this execution format is allowed, and no additional fields will be used during evaluation.
     24
     25Teams should also implement their radios such that packets are drawn from port 5123, while packets are delivered to port 5125 on the packet server specified with the '''-s''' argument. An illustration of this can be seen in the Python code in the table below:
     26
     27||= Transmitter =||= Receiver =||
     28|--------------------------------------------------------
     29{{{#!td style="border: 0px; font-size: 90%"
     30  {{{
     31class dsc_pkt_src(object):
     32    def __init__(self, server, port=5123 ):
     33        self.pkt_size = 1440 # 1440 bytes of data 
     34        self.pkt_server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
     35        self.pkt_server_socket.connect((server,port))
     36        self.MESSAGE = struct.pack('!l', self.pkt_size)
     37        self.pkt_server_socket.send(self.MESSAGE)
     38
     39    def read(self):
     40           try:
     41              data = self.pkt_server_socket.recv(self.pkt_size)
     42           except socket.error:
     43              print "Connection to packet server closed"
     44              return ''
     45           self.pkt_server_socket.send(self.MESSAGE)
     46        return data
     47  }}}
     48}}}
     49{{{#!td style="border: 0px; font-size: 90%"
     50  {{{
     51class dsc_pkt_sink(object):   
     52      def __init__(self, server, port=5125):       
     53          self.pkt_sink_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
     54          self.pkt_sink_socket.connect((server,port))
     55
     56      def send(self, payload):
     57          try:
     58             self.pkt_sink_socket.recv(4)
     59             self.pkt_sink_socket.send(payload)
     60          except socket.error:
     61             print "Connection to packet sink closed"
     62  }}}
     63}}}
     64
     65Teams are also required to label their image with the image name they intend to use for submission, which should be stored in file named '''/.orbit_image''' on the root of the filesystem before node is saved. For example, if the team intends to submit an image with image name '''orbit-bot.ndz''' the /.orbit_image file should contain '''orbit-bot'''.
     66
     67=== Packet Server ===
     68The packet server application running on the infrastructure machine has
     69* a packet source at port 5123 which serves fixed size packets (1440 bytes). Clients can request packets from the server with the following handshaking process after a TCP socket connection has been established:
     70    1. client sends server the request with packet size as 4 byte word ('''ignored''' for the matches).
     71    2. server sends client a single data packet (fixed at '''1440''' bytes)
     72
     73    The packet source sends out the first packet to each team only after it receives packet requests from all the teams (2 teams for competitive match and 3 for co-operative matches) and the start signal from the experiment script. It closes the transmit socket on the first packet request after the end-of-file is reached or at the 180 second timeout. It will report the score via an email to whoever has a grid reservation (if at least one packet is received correctly).
     74 * a packet sink at port 5125 which receive packets from client and verifies that it matches one of the sent packets. Clients can send data to server after a connection has been established:
     75    1. server sends client a ready packet (4 byte word).
     76    2. client sends server data for scoring.
     77   
     78* a control interface used by the experiment script to prepare the packet server for a match by setting its mode, duration, etc., and to send a start signal so the packet source can send packets to the teams.
     79
     80== Image Validation and Submission ==
     81
     82For the image to be considered for the tournament it must be successfully validated in ''both'' the Competitive and Cooperative modes. Different images for different tournament modes will not be accepted.  Successful execution of the system:exp:dsc-match script (i.e. with at least one error-free packet received by the scoring server) is considered validation of the image for that match type. Please note that the email sent by the scoring server to contestants (as well as the ORBIT team) contains all the information about images that were used for the match. '''It is essential that teams validate their final image without any modifications'''. The best way to ensure that is to simply save the final image, re-image the nodes and execute the scripts (without any further login/access to the nodes). This guarantees that the image checksum will be the same as what gets reported in the validation email. You may run the validation script as often as desired and on multiple images. If the team successfully validates multiple images, the ORBIT team needs to be informed of the actual image to be used, otherwise the last successfully validated image (i.e. passing both types of matches) will be used.
     83
     84Each team is also advised that the file /.orbit_image should contain the final image name (or team name) in order to facilitate proper reporting by the framework.
     85
     86== Setting Up the Arena ==
     87 * To get started, first make a reservation on the [https://www.orbit-lab.org/loginService/ControlPanel Orbit Scheduler] for using the Grid.
     88 * After logging into grid console, make sure all nodes are turned off (in the example below, turn off all the nodes in the grid):
     89{{{
     90username@console.grid:~$ omf tell -a offh -t system:topo:all
     91}}}
     92 * Verify state of node before continuing. Make sure nodes are in the POWEROFF state.
     93{{{
     94username@console.grid:~$ omf stat
     95}}}
     96
     97 * Image the two nodes to be used for the first team's radios (team A nodes):
     98{{{
     99username@console.grid:~$ omf load -i dsc-wbot1.ndz -t system:topo:dsc-teamA
     100}}}
     101
     102In the case of the validation matches these two nodes will receive the competitor's image (and hence replace the house bot image name with the team's image name).
     103
     104 * Image the two nodes to be used for the second team's radios (in this case team C nodes):
     105{{{
     106username@console.grid:~$ omf load -i dsc-wbot2.ndz -t system:topo:dsc-teamC
     107}}}
     108
     109 * FOR COOPERATIVE MATCHES: image the third pair of nodes to be used for the third team's radios (in this case team B nodes):
     110{{{
     111username@console.grid:~$ omf load -i dsc-wbot3.ndz -t system:topo:dsc-teamB
     112}}}
     113
     114 * Turn nodes back on and verify they are in POWERON state
     115{{{
     116username@console.grid:~$ omf tell -a on -t system:topo:dsc-teamA
     117username@console.grid:~$ omf tell -a on -t system:topo:dsc-teamC
     118}}}
     119
     120 * FOR COOPERATIVE MATCHES: turn on the third pair of nodes
     121{{{
     122username@console.grid:~$ omf tell -a on -t system:topo:dsc-teamB
     123}}}
     124
     125== Running the Validation/Experiment Script ==
     126
     127The experiment script will execute '''/root/dsc-code_tx''' and '''/root/dsc-code_rx''' on the source and sink respectively (please note that there are no suffixes), for both (all three) pairs of nodes. Teams must ensure that their images contain '''/root/dsc-code_tx''' and '''/root/dsc-code_rx'''. No modifications to this execution format is allowed. The sender module draws packets from a packet server, while the sink module delivers packets to the same packet server. 
     128
     129Use the '''system:exp:dsc-match''' OEDL script to run the match. The script will start the packet server, start the transmitter and receiver for all teams and release the packet source, wait for 180 seconds and then terminate the match.
     130[[BR]] [[BR]]
     131 * For competitive matches, the command line is:
     132{{{
     133username@console.grid:~$ omf exec system:exp:dsc-match -- --mode COMP
     134}}}
     135     [[CollapsibleStart(Competitive match output)]]
     136     The output of the competitive match should look similar to the following:
     137{{{
     138username@console.grid:~# omf exec system:exp:dsc-match -- --mode COMP
     139
     140 INFO NodeHandler: OMF Experiment Controller 5.4 (git 5385228)
     141 INFO NodeHandler: Slice ID: default_slice (default)
     142 INFO NodeHandler: Experiment ID: default_slice-2013-08-13t11.22.10.060-04.00
     143 INFO NodeHandler: Message authentication is disabled
     144 INFO Experiment: load system:exp:stdlib
     145 INFO property.resetDelay: resetDelay = 220 (Fixnum)
     146 INFO property.resetTries: resetTries = 1 (Fixnum)
     147 INFO Experiment: load system:exp:eventlib
     148 INFO Experiment: load system:exp:winlib
     149 INFO Experiment: load system:exp:dsc-match
     150 INFO property.freq: freq = 600000000 (Fixnum)
     151 INFO property.server: server = "idb2" (String)
     152 INFO property.port: port = 5123 (Fixnum)
     153 INFO property.runtime: runtime = 180 (Fixnum)
     154 INFO property.mode: mode = "COMP" (String)
     155 INFO property.team1: team1 = "dsc-teamA" (String)
     156 INFO property.team2: team2 = "dsc-teamC" (String)
     157 INFO property.team3: team3 = "dsc-teamB" (String)
     158 INFO Topology: Loaded topology 'system:topo:dsc-teamA'.
     159 INFO Topology: Loaded topology 'system:topo:dsc-teamC'.
     160 INFO exp: Connecting to packet server
     161 INFO ALL_UP_AND_INSTALLED: Event triggered. Starting the associated tasks.
     162 INFO exp: Preparing packet server
     163 INFO exp: Request from Experiment Script: Wait for 2s....
     164 INFO exp: Start rx and tx applications
     165 INFO exp: Request from Experiment Script: Wait for 10s....
     166 INFO exp: Sending signal to packet server to start serving packets
     167 INFO exp: Running the experiment
     168 INFO exp: Request from Experiment Script: Wait for 180s....
     169 INFO exp: Getting the score
     170 INFO exp: Team 1: Sent 12817440 Received 2776320 in 180 seconds
     171 INFO exp: Team 2: Sent 21457440 Received 16045920 in 180 seconds
     172 INFO exp: Stopping the applications
     173 INFO exp: Request from Experiment Script: Wait for 2s....
     174 INFO NodeHandler:
     175 INFO NodeHandler: Shutting down experiment, please wait...
     176 INFO NodeHandler:
     177default_slice-2013-08-13t11.22.10.060-04.00 EXP.DONE Running for 206 sec
     178 INFO run: Experiment default_slice-2013-08-13t11.22.10.060-04.00 finished after 3:26
     179
     180username@console.grid:~#
     181}}}
     182     [[CollapsibleEnd]]
     183
     184 * For cooperative matches, the command line is:
     185{{{
     186username@console.grid:~$ omf exec system:exp:dsc-match -- --mode COOP
     187}}}
     188
     189     [[CollapsibleStart(Cooperative match output)]]
     190     The output of the cooperative match should look similar to the following:
     191{{{
     192username@console.grid:~# omf exec system:exp:dsc-match -- --mode COOP
     193
     194 INFO NodeHandler: OMF Experiment Controller 5.4 (git 5385228)
     195 INFO NodeHandler: Slice ID: default_slice (default)
     196 INFO NodeHandler: Experiment ID: default_slice-2013-08-13t11.37.26.574-04.00
     197 INFO NodeHandler: Message authentication is disabled
     198 INFO Experiment: load system:exp:stdlib
     199 INFO property.resetDelay: resetDelay = 220 (Fixnum)
     200 INFO property.resetTries: resetTries = 1 (Fixnum)
     201 INFO Experiment: load system:exp:eventlib
     202 INFO Experiment: load system:exp:winlib
     203 INFO Experiment: load system:exp:dsc-match
     204 INFO property.freq: freq = 600000000 (Fixnum)
     205 INFO property.server: server = "idb2" (String)
     206 INFO property.port: port = 5123 (Fixnum)
     207 INFO property.runtime: runtime = 180 (Fixnum)
     208 INFO property.mode: mode = "COOP" (String)
     209 INFO property.team1: team1 = "dsc-teamA" (String)
     210 INFO property.team2: team2 = "dsc-teamC" (String)
     211 INFO property.team3: team3 = "dsc-teamB" (String)
     212 INFO Topology: Loaded topology 'system:topo:dsc-teamA'.
     213 INFO Topology: Loaded topology 'system:topo:dsc-teamC'.
     214 INFO Topology: Loaded topology 'system:topo:dsc-teamB'.
     215 INFO exp: Connecting to packet server
     216 INFO stdlib: Waiting for nodes (Up/Down/Total): 3/3/6 - (still down: node19-20.grid.orbit-lab.org,node19-19.grid.orbit-lab.org,node2-1.grid.orbit-lab.org) [0 sec.]
     217 INFO ALL_UP_AND_INSTALLED: Event triggered. Starting the associated tasks.
     218 INFO exp: Preparing packet server
     219 INFO exp: Request from Experiment Script: Wait for 2s....
     220 INFO exp: Start rx and tx applications
     221 INFO exp: Request from Experiment Script: Wait for 10s....
     222 INFO exp: Sending signal to packet server to start serving packets
     223 INFO exp: Running the experiment
     224 INFO exp: Request from Experiment Script: Wait for 180s....
     225 INFO exp: Getting the score
     226 INFO exp: Team 1: Sent 12817440 Received 2759040 in 180 seconds
     227 INFO exp: Team 2: Sent 21453120 Received 16178400 in 180 seconds
     228 INFO exp: Team 3: Sent 5850720 Received 1208160 in 180 seconds
     229 INFO exp: Stopping the applications
     230 INFO exp: Request from Experiment Script: Wait for 2s....
     231 INFO NodeHandler:
     232 INFO NodeHandler: Shutting down experiment, please wait...
     233 INFO NodeHandler:
     234default_slice-2013-08-13t11.37.26.574-04.00 EXP.DONE Running for 213 sec
     235 INFO run: Experiment default_slice-2013-08-13t11.37.26.574-04.00 finished after 3:33
     236
     237username@console.grid:~#
     238}}}
     239     [[CollapsibleEnd]]
     240
     241After each run, the team will receive a notification email ONLY if at least one packet was successful received by the packet server. The team image will be considered for the tournament evaluation if it was successfully verified (i.e. notification email was received) for '''both''' competitive and cooperative matches and if image filename and MD5 checksum matches the ones in the email.
     242
     243  [[CollapsibleStart(Competitive match notification email)]]
     244{{{
     245Subject: DSC competitive match score for team: prasanthi
     246
     247Please find the scoring report for the competitive match created at Tue Aug 13 11:26:00 -0400 2013:
     248
     249Team A:  Exp. Run Time = 180000, RX bytes = 2776320, TX bytes = 12817440, Sucess Rate = 21.6605
     250 Image: dsc-wbot1.ndz (Owner: root, Created: Thu Jul 18 12:18:35 -0400 2013, Md5: d783eb15b8e12fb95cd03985fd75c1a6 )
     251 Receiver = node2-1.grid.orbit-lab.org, Imaged by: nilanjan, Imaged on: 2013-08-13 at  11:15:39
     252
     253Team C:  Exp. Run Time = 180000, RX bytes = 16045920, TX bytes = 21457440, Sucess Rate = 74.7802
     254 Image: dsc-wbot2.ndz (Owner: root, Created: Thu Jul 18 12:18:55 -0400 2013, Md5: 5ed3d3b94535d56b6964ad0e58a4e1db )
     255 Receiver = node1-1.grid.orbit-lab.org, Imaged by: nilanjan, Imaged on: 2013-08-13 at  11:09:01
     256}}}
     257  [[CollapsibleEnd]]
     258
     259  [[CollapsibleStart(Cooperative match notification email)]]
     260{{{
     261Subject: DSC cooperative match score for team: prasanthi
     262
     263Please find the scoring report for the cooperative match created at Tue Aug 13 11:45:13 -0400 2013:
     264
     265Team B:  Exp. Run Time = 180000, RX bytes = 1208160, TX bytes = 5850720, Sucess Rate = 20.6498
     266 Image: dsc-wbot3.ndz (Owner: root, Created: Sat Jul 20 01:15:39 -0400 2013, Md5: fbfae44bcbcaec70c0e22759a8e94aa8 )
     267 Receiver = node1-2.grid.orbit-lab.org, Imaged by: nilanjan, Imaged on: 2013-08-13 at  11:10:14
     268
     269Team A:  Exp. Run Time = 180000, RX bytes = 2759040, TX bytes = 12817440, Sucess Rate = 21.5257
     270 Image: dsc-wbot1.ndz (Owner: root, Created: Thu Jul 18 12:18:35 -0400 2013, Md5: d783eb15b8e12fb95cd03985fd75c1a6 )
     271 Receiver = node2-1.grid.orbit-lab.org, Imaged by: nilanjan, Imaged on: 2013-08-13 at  11:15:39
     272
     273Team C:  Exp. Run Time = 180000, RX bytes = 16176960, TX bytes = 21453120, Sucess Rate = 75.4061
     274 Image: dsc-wbot2.ndz (Owner: root, Created: Thu Jul 18 12:18:55 -0400 2013, Md5: 5ed3d3b94535d56b6964ad0e58a4e1db )
     275 Receiver = node1-1.grid.orbit-lab.org, Imaged by: nilanjan, Imaged on: 2013-08-13 at  11:09:01
     276}}}
     277  [[CollapsibleEnd]]
     278
     279Turn the nodes off
     280{{{
     281username@console.grid:~$ omf tell -a offh -t system:topo:all
     282}}}
     283
     284== Running Matches on Sandboxes ==
     285
     286In order to facilitate code development, all three sandboxes (sb2,sb3 and sb7) are available and have a simplified version of the OEDL script and packet server. Please do note that, since they have only one pair of nodes, these environments are really not capable of generating interference, but are rather meant for code testing. Further, sandboxes cannot provide validation of an image. Appropriately, the actual match script is named dsc-test and the sequence of commands to execute it is somewhat simplified and consists of:
     287 
     288  * Loading the image:
     289{{{
     290  omf load -i team-image-name.ndz
     291}}}
     292  * Turning the nodes on:
     293{{{
     294  omf tell -a on
     295}}}
     296  * Executing the ''match'':
     297{{{
     298  omf exec system:exp:dsc-test -- --mode COMP
     299}}}
     300      [[CollapsibleStart(Competitive match output on the sandbox)]]
     301      The output of the 30 second competitive match should look similar to the following:
     302{{{
     303console.sb2:~# omf exec system:exp:dsc-test -- --mode COMP --runtime 30
     304
     305 INFO NodeHandler: OMF Experiment Controller 5.4 (git 5385228)
     306 INFO NodeHandler: Slice ID: default_slice (default)
     307 INFO NodeHandler: Experiment ID: default_slice-2013-08-12t20.55.32.533-04.00
     308 INFO NodeHandler: Message authentication is disabled
     309 INFO Experiment: load system:exp:stdlib
     310 INFO property.resetDelay: resetDelay = 230 (Fixnum)
     311 INFO property.resetTries: resetTries = 1 (Fixnum)
     312 INFO Experiment: load system:exp:eventlib
     313 INFO Experiment: load system:exp:winlib
     314 INFO Experiment: load system:exp:dsc-test
     315 INFO property.freq: freq = 650000000 (Fixnum)
     316 INFO property.server: server = "10.12.0.10" (String)
     317 INFO property.port: port = 5123 (Fixnum)
     318 INFO property.runtime: runtime = 30 (Fixnum)
     319 INFO property.mode: mode = "COMP" (String)
     320 INFO exp: Connecting to packet server
     321 INFO stdlib: Waiting for nodes (Up/Down/Total): 1/1/2 - (still down: node1-1.sb2.orbit-lab.org) [0 sec.]
     322 INFO ALL_UP_AND_INSTALLED: Event triggered. Starting the associated tasks.
     323 INFO exp: Preparing packet server
     324 INFO exp: Request from Experiment Script: Wait for 2s....
     325 INFO exp: Start rx and tx applications
     326 INFO exp: Request from Experiment Script: Wait for 5s....
     327 INFO exp: Running the experiment
     328 INFO exp: Request from Experiment Script: Wait for 30s....
     329 INFO exp: Getting the score
     330 INFO exp: Sent 2161440 Received 2131200 in 30 ms
     331 INFO exp: Stopping the applications
     332 INFO exp: Request from Experiment Script: Wait for 2s....
     333 INFO NodeHandler:
     334 INFO NodeHandler: Shutting down experiment, please wait...
     335 INFO NodeHandler:
     336 INFO run: Experiment default_slice-2013-08-12t20.55.32.533-04.00 finished after 0:52
     337}}}
     338      [[CollapsibleEnd]]
     339  or
     340{{{
     341  omf exec system:exp:dsc-test -- --mode COOP
     342}}}
     343      [[CollapsibleStart(Cooperative match output on the sandbox)]]
     344      The output of the 30 second competitive match should look similar to the following:
     345{{{
     346console.sb2:~# omf exec system:exp:dsc-test -- --mode COOP --runtime 30
     347
     348 INFO NodeHandler: OMF Experiment Controller 5.4 (git 5385228)
     349 INFO NodeHandler: Slice ID: default_slice (default)
     350 INFO NodeHandler: Experiment ID: default_slice-2013-08-12t21.00.56.411-04.00
     351 INFO NodeHandler: Message authentication is disabled
     352 INFO Experiment: load system:exp:stdlib
     353 INFO property.resetDelay: resetDelay = 230 (Fixnum)
     354 INFO property.resetTries: resetTries = 1 (Fixnum)
     355 INFO Experiment: load system:exp:eventlib
     356 INFO Experiment: load system:exp:winlib
     357 INFO Experiment: load system:exp:dsc-test
     358 INFO property.freq: freq = 650000000 (Fixnum)
     359 INFO property.server: server = "10.12.0.10" (String)
     360 INFO property.port: port = 5123 (Fixnum)
     361 INFO property.runtime: runtime = 30 (Fixnum)
     362 INFO property.mode: mode = "COOP" (String)
     363 INFO exp: Connecting to packet server
     364 INFO ALL_UP_AND_INSTALLED: Event triggered. Starting the associated tasks.
     365 INFO exp: Preparing packet server
     366 INFO exp: Request from Experiment Script: Wait for 2s....
     367 INFO exp: Start rx and tx applications
     368 INFO exp: Request from Experiment Script: Wait for 5s....
     369 INFO exp: Running the experiment
     370 INFO exp: Request from Experiment Script: Wait for 30s....
     371 INFO exp: Getting the score
     372 INFO exp: Sent 2161440 Received 1990080 in 30 ms
     373 INFO exp: Stopping the applications
     374 INFO exp: Request from Experiment Script: Wait for 2s....
     375 INFO NodeHandler:
     376 INFO NodeHandler: Shutting down experiment, please wait...
     377 INFO NodeHandler:
     378 INFO run: Experiment default_slice-2013-08-12t21.00.56.411-04.00 finished after 0:52
     379}}}
     380      [[CollapsibleEnd]]
     381
     382Please note that the sandboxes use different frequencies in order to minimize the interference, are not going to produce the scoring email (the score is printed on the screen only) and that your code must support the scoring server hostname option (since the scoring server is running on respective consoles of each of the sandboxes). 
     383
     384NOTES: