Group Experimentation Support

Experiment scheduler

Installing and configuring packages on Ubuntu

This section describes how to set up torque PBS. If it's already set up, skip this section. First, we install the torque PSB system:

apt-get install torque-server torque-scheduler torque-mom torque-client

Then stop all the running torque processes:

/etc/init.d/torque-mom stop
/etc/init.d/torque-scheduler stop
/etc/init.d/torque-server stop

Create the PBS server (say "yes" when prompted):

pbs_server -t create

Then kill the PBS server process:

killall pbs_server

Set up the PBS server:

echo $(hostname -f) > /etc/torque/server_name
echo $(hostname -f) >  /var/spool/torque/server_priv/acl_svr/acl_hosts
echo $(hostname -f) > /var/spool/torque/mom_priv/config

echo root@$(hostname -f) > /var/spool/torque/server_priv/acl_svr/operators
echo root@$(hostname -f) > /var/spool/torque/server_priv/acl_svr/managers

echo "$(hostname -f) np=4" > /var/spool/torque/server_priv/nodes

If you have a line in your /etc/hosts file that resolves your hostname to 127.0.1.1, you have to comment it out, e.g.

#127.0.1.1  console.grid.orbit-lab.org  console

Once you've done that, start everything back up again:

/etc/init.d/torque-server start
/etc/init.d/torque-scheduler start
/etc/init.d/torque-mom start

Now we'll set up some configuration values:

qmgr -c "set server scheduling = True"
qmgr -c "set server acl_host_enable = True"
qmgr -c "set server acl_hosts = $(hostname -f)"
qmgr -c "set server allow_node_submit = True"

You'll have to run the commands above as root, since you've set up the root user as the only PBS operator and manager.

Next we'll set up queues: one for each node.

For various reasons, we've decided to make a queue per node and have the console be the single "compute" node, instead of having the nodes be the "compute" nodes. (Mainly because then we can still use legacy disk images, and don't have to worry about configuring the nodes to work with torque.) It might seem "neater" to use "nodes" instead of queues, because this would make it simpler to run an experiment with multiple nodes. In practice, though, it would still be annoying to run an experiment with multiple nodes because you generally care which nodes in this scenario (e.g. you want nodes that are close.)

First we get a list of nodes, then we'll set up a queue for each one:

list=$(omf stat -t system:topo:all | grep "Node:" | awk -F" " '{print $2}' | cut -f1 -d$'.')

for l in $list
do 
    qmgr -c "create queue $l"
    qmgr -c "set queue $l queue_type = Execution"
    qmgr -c "set queue $l max_running = 1"
    qmgr -c "set queue $l enabled = True"
    qmgr -c "set queue $l started = True"
done

(Again, this must be done as root or as a queue manager.)

Now we'll disable all the queues, because we are going to re-enable them selectively.

list=$(qstat -Q | tail -n+3 | awk -F" " '{print $1}')

for l in $list
do 
    qdisable "$l"
done

Check with

qstat -Q

Scheduler utility script

Enabling/disabling "scheduled" experiments

The easiest way to turn the scheduler on and off is to put the scheduler utility script in /usr/local/bin/omf. When you want it to be turned on, make it executable:

chmod a+x /usr/local/bin/omf

and when you want it to be turned off, turn off the execute bit:

chmod a-x /usr/local/bin/omf

On WITest, as part of the script that runs at the beginning of each reservation, the scheduler script is turned on if it is a group reservation and off if it isn't, and all queues are disabled.

It is then up to the group "leader" (i.e. tutorial instructor, teaching assistant, etc.) to enable individual queues for the experiment that the group will run. The group "leader" also needs to know to use the full path /usr/bin/omf for "load" commands, which are disabled in the utility script (because it is generally not desirable to load images in a group reservation.)

Using the scheduler

Some important commands follow.

To enable a queue named e.g. node13-10 so that it starts accepting jobs (user must be listed as a torque operator or manager):

qenable node13-10

To disable a queue named e.g. node13-10 so that it will finish currently queued jobs but not accept new ones (user must be listed as a torque operator or manager):

qdisable node13-10

To see the current torque server configuration:

qmgr -c 'p s'

To add a user e.g. ffund01 as a torque manager:

qmgr -c "set server managers += ffund@console.grid.orbit-lab.org"

To kill a job with job ID e.g. 13:

qdel 13

To see currently queued and running jobs (all if you're a torque manager, otherwise your own):

qstat

OMF with reporting

Patching OMF EC

Usage instructions

Experiment dashboard

Installing/configuring packages on Ubuntu

These instructions are for installing the dashboard locally as an unprivileged user, not system-wide.

First, install RVM and prepare to use it:

command curl -sSL https://rvm.io/mpapis.asc | gpg --import -
curl -L https://get.rvm.io | bash -s stable
source ~/.rvm/scripts/rvm

Install ruby 1.9.2 with RVM:

rvm install ruby-1.9.2

Get the ​dashing gem:

rvm use 1.9.2
gem install dashing
gem install sqlite3
gem install execjs --version 1.4.0  # see https://github.com/Shopify/dashing/issues/195
gem uninstall execjs --version 2.0.2

In ~/.rvm/gems/ruby-1.9.2-p330/specifications/dashing-1.3.6.gemspec (or equivalent), change the version for the "execjs" dependency to 1.4.0.

Source code for experiment dashboard

Usage