[[TOC(heading=Table of Contents, depth=3)]] = Building our own Testbed = In this section we give a detailed description of our efforts on setting up a mid-size ORBIT-like testbed. In the right of this page you can see a list of all the related actions we had to take from the initial stage until the stage of having a full functional testbed that can be remotely accessed for uploading particular experiments, running the experiments and collecting the results. == Hardware Setup == === Orbit Node === Each ORBIT Radio Node is a PC with a 1 GHz VIA C3 processor, 512 MB of RAM, 40 GB of local disk, two Ethernet ports, two 802.11 a/b/g cards and a Chassis Manager to control the node, see Figure 3. The description about the ethernet ports is following. '''Control port''' - The ethernet port between 4 USB ports, it is a Rtl-8169 Gigabit ethernet port, which is used to load and control the ORBIT node and collect measurements. '''Data port''' - The ethernet port above two USB ports, it is a VT6102 Rhine-II 100/10baseT Ethernet port, which is used for data communication, '''CM port''' - The 10BaseT Ethernet port on Chassis Manager Card, which is used to communicate with gridservice (not gridservice2) [[BR]] [[BR]] [[Image(node-50.png)]] [[Image(orbit_node.JPG)]] === Testbed === The test bed consists of nodes and several servers. Technically, all servers can be put in one machine with at lease two ethernet ports, but it's not recommendatory because of potential security consideration. A typical test bed include three servers discribed below[[BR]] [[Image(testbed.PNG)]] '''Services''' - It is used to host various services including DHCP, DNS, NTP, TFTP, PXE, Frisbee, NFS, mysql, OML and Apache. We have different aliases for the management host to segregate the services that it hosts. This machine or port shall be connected with '''Control port''' of nodes. [[BR]] '''Console''' - It is used to run experiments with nodehandler4. Console is also connected with '''Control port''' of nodes. It may share one Ethernet port with '''Services'''. A better way is setuping a console in one machine exclusively and let it accessible by experimenters with ssh or XDMCP.[[BR]] '''CMC''' - It is the control and monitoring manager for all CM elements of ORBIT nodes. It is connected with '''CM port''' of nodes and can NOT share Ethernet port with '''Services''' and '''Console'''. In our situation, Service and Console share one ethernet port with address 10.10.0.10/16 and CMC is on another ethernet port with address 10.1.200.1/16. You may connect 10.10.0.10 and 10.1.200.1 with an internal route. In my situation. Since both machines connect to outside with their eth1, I explicitly set route on both machine as following.[[BR]] On Console/Services {{{ console:~# route add -host 10.1.200.1 gw 128.238.34.248 dev eth1 console:~# route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface cmc.grid.poly.e cmc.local 255.255.255.255 UGH 0 0 0 eth1 localnet * 255.255.255.0 U 0 0 0 eth1 10.10.0.0 * 255.255.0.0 U 0 0 0 eth0 default 128.238.34.1 0.0.0.0 UG 0 0 0 eth1 }}} On CMC {{{ cmc:~# route add -host 10.10.0.10 gw 128.238.34.247 dev eth1 cmc:~# route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface console.grid.po console.local 255.255.255.255 UGH 0 0 0 eth1 128.238.34.0 * 255.255.255.0 U 0 0 0 eth1 10.1.0.0 * 255.255.0.0 U 0 0 0 eth0 default 128.238.34.1 0.0.0.0 UG 0 0 0 eth1 cmc:~# ping console PING console.grid.poly.edu (10.10.0.10) 56(84) bytes of data. 64 bytes from console.grid.poly.edu (10.10.0.10): icmp_seq=1 ttl=64 time=0.128 ms 64 bytes from services.grid.poly.edu (10.10.0.10): icmp_seq=2 ttl=64 time=0.139 ms }}} == Software and Services == === Linux Installation === You need install Linux on two or more machines. If you want to use only one machine, it must equipped with two or more ethernet interfaces. You can choose whatever Linux distribution you prefer. However, Debian is strongly recommended because some testbed related software are distributed in both source code and Deb package. The installation guide can be found [http://www.debian.org/releases/stable/installmanual here]. For more information about Debian, please visit http://www.debian.org === Configure Apt === If the Linux distribution you choosed is Debian/Ubuntu, please add the following lines in file '''/etc/apt/sources.list''', so let apt-get can find Debian packages provided by [http://www.orbit-lab.org Orbit lab]. {{{ deb http://apt.orbit-lab.org/orbit testing main deb http://apt.orbit-lab.org/orbit unstable main deb http://apt.orbit-lab.org/orbit stable main }}} After each time sources.list is changed, please run command "apt-get update" to resynchronize the package index files from their sources. For more about sources.list and apt-get, please refer with command "man sources.list" and "man apt-get". === Configure host name === If Console/Service and CMC are on different machines, please edit file '''/etc/hostname''' respectively. The host name of Console/Service is "'''console'''", and that of CMC is "'''cmc'''". If the whole three server are on one machine, please set it's host name to "'''console'''". === Configure network interface === Since we place Console/Service and CMC on different machine. Interface configuration file should like following[[BR]] For Console/Service {{{ # This file describes the network interfaces available on your system # and how to activate them. For more information, see interfaces(5). # The loopback network interface auto lo eth0 eth1 iface lo inet loopback # The primary network interface access to outside internet allow-hotplug eth1 iface eth1 inet static address 128.238.34.247 netmask 255.255.255.0 network 128.238.34.0 broadcast 128.238.34.255 gateway 128.238.34.1 #iface eth1 inet dhcp ## dns-* options are implemented by the resolvconf package, if installed dns-nameservers 10.10.0.10 128.238.1.68 dns-search grid.poly.edu dns-domain grid.poly.edu # The internal network interface for console and services(nodehandler4, # dhcp, dns, gridservice2, OML, etc) allow-hotplug eth0 iface eth0 inet static address 10.10.0.10 netmask 255.255.0.0 network 10.10.0.0 broadcast 10.10.255.255 ## dns-* options are implemented by the resolvconf package, if installed dns-nameservers 10.10.0.10 128.238.1.68 dns-search grid.poly.edu dns-domain grid.poly.edu }}} For CMC {{{ # This file describes the network interfaces available on your system # and how to activate them. For more information, see interfaces(5). # The loopback network interface auto lo eth0 eth1 iface lo inet loopback # The primary network interface access to outside internet allow-hotplug eth1 iface eth1 inet static address 128.238.34.248 netmask 255.255.255.0 network 128.238.34.0 broadcast 128.238.34.255 gateway 128.238.34.1 # dns-* options are implemented by the resolvconf package, if installed dns-nameservers 10.10.0.10 128.238.1.68 dns-search grid.poly.edu dns-domain grid.poly.edu #iface eth1 inet dhcp # The internal network interface for CMC(gridservice). allow-hotplug eth0 iface eth0 inet static address 10.1.200.1 netmask 255.255.0.0 network 10.1.0.0 broadcast 10.1.255.255 ## dns-* options are implemented by the resolvconf package, if installed dns-nameservers 10.10.0.10 128.238.1.68 dns-search grid.poly.edu dns-domain grid.poly.edu }}} In each server, interface eth0 is for test bed, and eth1 is for access outside. If all servers is on one machine, the configuration of eth0 on CMC should be moved to /etc/network/interfaces on Console/Services. === Name resolve === Because eth1 might connect with other DHCP/DNS servers from outside, and it used to change file /etc/resolv.conf when servers boot up, we need resolvconf to fix the setting in in resolv.conf. First, install "resolvconf" with command "''apt-get install resolvconf''". Then, run command "dpkg-reconfigure resolvconf" and agree to symlink /etc/resolv.conf to /etc/resolvconf/run/resolv.conf. At last, edit file /etc/resolvconf/interface-order like following {{{ # interface-order(5) eth* }}} Some comment for file /etc/network/interfaces: "'''dns-nameservers 10.10.0.10 128.238.1.68'''" indicates resolvconf to add nameservers 10.10.0.10 and 128.238.1.68 in resolv.conf. 10.10.0.10 is the address of DNS used for testbed, and 128.238.1.68 is the address of DNS for outside, which you can change to others in your situation. Please make sure 10.10.0.10 is the first address appear in the list. "'''dns-search grid.poly.edu'''" and "'''dns-domain grid.poly.edu'''" indicate resolvconf to add "search" and "domain" entries in resolv.conf. "'''grid'''" is considered as the name of test bed, which will be used in later configuration. The final /etc/resolv.conf might looks like below after rebooting the machine. {{{ # Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8) # DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN nameserver 10.10.0.10 nameserver 128.238.1.68 search grid.poly.edu }}} For more information about resolvconf, please refer /usr/share/doc/resolvconf/README.gz and "man resolvconf". === DHCP === '''Purpose''': This software runs a DHCP server that assigns IP addresses to clients on demand. '''Installation and Configuration: ''': * Run command on '''Console/Services''', and select eth0 as interface. {{{ apt-get install dhcp3-server }}} * Make sure that /etc/default/dhcp3-server has eth0 as default interface which DHCP will server for. Just like following {{{ # On what interfaces should the DHCP server (dhcpd) serve DHCP requests? # Separate multiple interfaces with spaces, e.g. "eth0 eth1". INTERFACES="eth0" }}} * Edit /etc/dhcp3/dhcpd.conf. The configuration looks like below. {{{ # The ddns-updates-style parameter controls whether or not the server will # attempt to do a DNS update when a lease is confirmed. We default to the # behavior of the version 2 packages ('none', since DHCP v2 didn't # have support for DDNS.) ddns-update-style interim; use-host-decl-names on; allow booting; allow bootp; # option definitions common to all supported networks... option domain-name "grid.poly.edu"; default-lease-time 259200; max-lease-time 259200; # If this DHCP server is the official DHCP server for the local # network, the authoritative directive should be uncommented. #authoritative; # Use this to send dhcp log messages to a different log file (you also # have to hack syslog.conf to complete the redirection). log-facility local7; subnet 10.10.0.0 netmask 255.255.0.0 { range 10.10.1.1 10.10.255.254; option domain-name "grid.poly.edu"; ddns-updates off; ddns-domainname "grid.poly.edu"; ddns-rev-domainname "in-addr.arpa"; option domain-name-servers 10.10.0.10; next-server 10.10.0.10; host services {hardware ethernet 00:1B:2F:BE:EF:94; fixed-address services.grid.poly.edu;} host console {hardware ethernet 00:1B:2F:BE:DF:6E; fixed-address console.grid.poly.edu;} filename "/tftpboot/pxelinux.bin"; #node 10.10.x.y group { host node1-1 {hardware ethernet 00:0F:EA:8C:AE:39; fixed-address node1-1.grid.poly.edu;} host node1-2 {hardware ethernet 00:03:2D:08:19:fe; fixed-address node1-2.grid.poly.edu;} host node1-3 {hardware ethernet 00:03:2D:07:67:CE; fixed-address node1-3.grid.poly.edu;} } } }}} '''Some comments on the dhcpd.conf'''[[BR]] First line is '''use-host-decl-names''' on which means DNS has actual IP mappings, DHCP just gives out hostnames. This means only IP to name updates need to be done only at DNS.[[br]] '''next-server''' is used to specify the host address of the server from which the initial boot file is to be loaded. In our case, it's the address tftp server.[[BR]] The '''filename''' statement is used to specify the name of the initial boot file which is to be loaded by a client. [[BR]] '''domain-name-servers''' specifies a list of Domain Name System name servers available to the client.[[BR]] The node name in form of node'''x'''-'''y''' determines it's address must be 10.10.'''x'''.'''y'''. For example, the address of node2-3 is 10.10.2.3. The actual address mapping is done by DNS. '''To Run: ''': /etc/init.d/dhcp3-server start -- errors go to /var/log/daemon.log === DNS === '''Purpose''': services.poly.edu hosts the primary DNS service for the zone grid.poly.edu. The DNS server is the standard BIND9 software packaged. '''Installation and Configuration: ''': * ''apt-get install bind9'' * You need edit Named.conf, named.conf.options and named.conf.local under /etc/bind as below * named.conf {{{ include "/etc/bind/named.conf.options"; include "/etc/bind/named.conf.local"; Controls { inet 127.0.0.1 port 953 allow { 127.0.0.1; }; }; // prime the server with knowledge of the root servers zone "." { type hint; file "/etc/bind/db.root"; }; // be authoritative for the localhost forward and reverse zones, and for // broadcast zones as per RFC 1912 zone "localhost" { type master; file "/etc/bind/db.local"; }; zone "127.in-addr.arpa" { type master; file "/etc/bind/db.127"; }; zone "0.in-addr.arpa" { type master; file "/etc/bind/db.0"; }; zone "255.in-addr.arpa" { type master; file "/etc/bind/db.255"; }; }}} * named.conf.options {{{ options { directory "/etc/bind"; auth-nxdomain no; # conform to RFC1035 listen-on-v6 { any; }; }; }}} * named.conf.local {{{ // Consider adding the 1918 zones here, if they are not used in your // organization //include "/etc/bind/zones.rfc1918"; zone "grid.poly.edu" IN { type master; file "/etc/bind/orbit.zone"; }; zone "in-addr.arpa" IN { type master; file "/etc/bind/zone.orbit"; }; }}} * You need create orbit.zone and zone.orbit under /etc/bind as below * orbit.zone: Forward lookup {{{ $TTL 3600 @ IN SOA services.grid.poly.edu. root.services.grid.poly.edu. ( 2008072501 ; serial 3600 ; refresh (1 hour) 600 ; retry (10 min) 10000 ; expire (2 hours) 3600 ); ; @ IN NS services.grid.poly.edu. $ORIGIN grid.poly.edu. $TTL 129600 windows IN A 10.10.1.8 rxwarp IN A 10.10.1.9 node1-1 IN A 10.10.1.1 node1-2 IN A 10.10.1.2 node1-3 IN A 10.10.1.3 console IN A 10.10.0.10 cmc IN A 10.1.200.1 services IN CNAME console.grid.poly.edu. dhcp IN CNAME console.grid.poly.edu. frisbee IN CNAME console.grid.poly.edu. pxe IN CNAME console.grid.poly.edu. oml IN CNAME console.grid.poly.edu. repository IN CNAME console.grid.poly.edu. repository1 IN CNAME console.grid.poly.edu. repository2 IN CNAME console.grid.poly.edu. ntp IN CNAME console.grid.poly.edu. loghost IN CNAME console.grid.poly.edu. idb1 IN CNAME console.grid.poly.edu. idb2 IN CNAME console.grid.poly.edu. }}} * zone.orbit: Reverse lookup {{{ $TTL 3600 @ IN SOA services.grid.poly.edu. root.services.grid.poly.edu. ( 2008021817 ; serial 28800 ; refresh (8hours) 900 ; retry (15 mins) 604800 ; expire (1 week) 86400 ; minimum (1 day) ); @ IN NS services.grid.poly.edu. $ORIGIN in-addr.arpa. $ORIGIN 10.in-addr.arpa. $ORIGIN 0.10.10.in-addr.arpa. $TTL 129600 10 IN PTR services.grid.poly.edu. 10 IN PTR console.grid.poly.edu. $ORIGIN 1.10.10.in-addr.arpa. 9 IN PTR rxwarp.grid.poly.edu. 8 IN PTR windows.grid.poly.edu. 1 IN PTR node1-1.grid.poly.edu. 2 IN PTR node1-2.grid.poly.edu. 3 IN PTR node1-3.grid.poly.edu. $ORIGIN 200.1.10.in-addr.arpa. 1 IN PTR cmc.grid.poly.edu. }}} * Make sure there are dots at the end of the domains. The owner and group of orbit.zone and zone.orbit might be like following. {{{ console:/etc/bind# ls -l orbit.zone zone.orbit -rw-r--r-- 1 root bind 1217 2008-08-07 17:00 orbit.zone -rw-r--r-- 1 root bind 928 2008-08-07 17:01 zone.orbit }}} '''To Run: ''': /etc/init.d/bind9 start -- errors go to /var/log/daemon.log * You may run command "'''host'''" on Console/Services or CMC as below to verify if bind works well. {{{ console:~# host cmc cmc.grid.poly.edu has address 10.1.200.1 console:~# host console console.grid.poly.edu has address 10.10.0.10 console:~# host services services.grid.poly.edu is an alias for console.grid.poly.edu. console:~# host node1-1 node1-1.grid.poly.edu has address 10.10.1.1 console:~# host pxe pxe.grid.poly.edu is an alias for console.grid.poly.edu. console.grid.poly.edu has address 10.10.0.10 console:~# host frisbee frisbee.grid.poly.edu is an alias for console.grid.poly.edu. console.grid.poly.edu has address 10.10.0.10 }}} === Apache Web Server === '''Purpose:''' Apache server is required for maintaining the ORBIT local repository for debian packages and also to view the results of the experiment '''Installation:''' * ''apt-get install apache2'' [[BR]] Note that no additional configuration is needed for apache. Also, make sure that /var/www/cgi-bin points to /usr/lib/cgi-bin (or create a soft link if one does not exist using cd /var/www/; ln –s /usr/lib/cgi-bin cgi-bin). [[BR]] '''To run:'''/etc/init.d/apache2 start -- errors go to /var/log/daemon.log We also need install libgd, which is used to view the results of the experiment. The command below can install it {{{ apt-get install libgd-gd2-perl }}} === NTP === '''Purpose:''' All the machines synchronize their time using the time server as the reference. '''Installation and Configuration:''' * ''apt-get install ntp'' * You may add ntp server "pool.ntp.org" into /etc/ntpd.conf if there is server setting in it. '''To run:''' * /etc/init.d/ntpd start -- errors go to /var/log/daemon.log === TFTP Server === '''Purpose:''' TFTP is needed to install PXE images whenever you need to install an image onto the node (using Frisbee). It is also used to load a memory based image that can be used to fetch the current image of the node into the repository '''Installation and Configuration:''' * ''apt-get install atftpd'' There are two options here: either to run atftpd as a standalone daemon or run it under inetd. For heavy duty tftp services, you can choose to run is as a standalone daemon. For our installation, we choose the standalone daemon. * Edit file /etc/inetd.conf and point the tftp directory to /tftpboot. The configuration may look like following {{{ #:BOOT: TFTP service is provided primarily for booting. Most sites # run this only on machines acting as "boot servers." tftp dgram udp wait root /usr/sbin/atftpd /usr/sbin/in.tftpd /tftpboot }}} * The PXE image can be download from [http://witestlab.poly.edu/attachment/wiki/BOOTB/tftpboot.tar.bz2?format=raw here]. You need to extract it with command[[BR]]tar -xjvf tftpboot.tar.bz2[[BR]] The final content of directory /tftpboot looks like as below. {{{ console:~# ls /tftpboot -R /tftpboot: initramfs-orbit-pxe-2.0.3.gz linux-orbit-pxe-2.6.25.1 pxelinux.bin pxelinux.cfg /tftpboot/pxelinux.cfg: default orbit-2.0.3-omf }}} In case of problems, make sure that lo interface is up. === NFS Service === '''Purpose:''' This service is used to remotely mount directories on the nodes while fetching their image using imagezip utility. Also, Frisbee service makes use of this directory to install images onto nodes. '''Installation and Configuration:''' * ''apt-get install nfs-kernel-server'' * create a path like "/export/orbit/image/tmp" * Add a line in /etc/exports file as follows. "/export/orbit/image/tmp" is the default path nodes use to save frisbee images. {{{ /export/orbit/image/tmp 10.10.0.0/16(rw,sync,no_root_squash) }}} '''To run:''' * /etc/init.d/nfs-kernel-server start -- errors go to /var/log/daemon.log === Mysql Server === '''Purpose:''' This service is used to store the results of the experiments conducted on ORBIT ''' Installation and Configuration:''' * ''apt-get install mysql-server-4.1'' * Edit /etc/mysql/my.cnf and replace bind-address from 127.0.0.1 to 10.10.0.10 type mysql and at the prompt, enter the following * Replace the password by an appropriate one. Basically, this creates a new account called orbit with the password specified and allows access to mysql databases from localhost and any other machine on the network. '''To run:''' * /etc/init.d/mysql start -- errors go to /var/log/mysql.log === ORBIT Server === Orbit specific services include nodehandler, nodeagent, frisbee, gridservices, gridservices2 and OML(Orbit Measurement Library). If you have added configuration in sources.list as described [http://witestlab.poly.edu/wiki/BOOTB#ConfigureApt here], you can follow the commands below to install them. All these services are installed on Console, except gridservices on CMC. For Console * ''apt-get update''[[BR]] * ''apt-get install otg''[[BR]] * ''apt-get install nodehandler4''[[BR]] Since some files in nodehandler4 debian package are obsolete, you need to update it with a [http://witestlab.poly.edu/attachment/wiki/BOOTB/nodehandler.tar.gz?format=raw tar ball]. Extract it and replace fold '''/opt/nodehandler4-4.2.0/'''.[[BR]] The configuration of nodehandler4 is based on [http://www.yaml.org YAML] script file /etc/nodehandler4/nodehandler.yaml. You can create a symbol link named "'''nodehandler.yaml'''" to the actual configuration file. The example and comments below could be helpful to understand it. {{{ 1 # NOTE: use only 'spaces' to indent ! 2 # ('tab' indents are not supported by the ruby yaml parser used to read this file) 3 # 4 # This is the Config file for the NodeHandler4 on the WINLAB platform 5 # 6 --- 7 nodehandler: 8 name_resolv: | 9 name = nil 10 if NodeHandler.JUST_PRINT 11 name = 'debug' 12 else 13 # take first subdomain as grid name (sb0.orbit-lab.org) 14 IO.popen('hostname -d') {|f| name = f.gets.split('.')[0] } 15 end 16 17 testbed: 18 19 # Config Parameter for the "default" Testbed 20 # 21 # In the WINLAB setting the default testbed is the "grid" testbed, using gridservice2 22 default: 23 24 repository: 25 path: [".", "../repository", "/opt/nodehandler4-4.2.0/repository"] 26 27 pxe: 28 # This is the URL where NH can find the PXE GridService 29 url: 'http://pxe:5022/pxe' 30 31 cmc: 32 # This is the URL where NH can find the CMC GridService 33 url: "http://cmc:5012/cmc" # Contact the CMC of GS 1 - Not ported yet for GS 2 34 35 oml: 36 # NodeAgents will use the numerical IP address in 'local_host' to connect 37 # to the machine running the NodeHandler, in order to retrieve the OML defs 38 # (in XML, and generated by NH). These OML defs are used by the NAs' applications 39 # Thus, 'local_host' = Control IP address (reachable by NAs) of the NH's machine 40 local_host: '10.10.0.10' 41 # The parameters below are the contact details for the OML GridService 42 url: "http://oml:5022/oml" 43 port: 5022 44 host: "oml" 45 46 frisbee: 47 # The parameters below are the contact details for the Frisbee GridService 48 default_disk: '/dev/sda' 49 url: 'http://frisbee:5022/frisbee' 50 51 inventory: 52 # This is the URL where NH can find the inventory GridService 53 url: 'http://cmc:5022/inventory' 54 55 # Command used to launch the communication module 56 # The type of comm module to launch depends on the cmd line params 57 # '-c PORT' runs a TCP comm. module that will connect to the node Agent's TCP server on PORT 58 # '-l PORT' runs a TCP comm. Server module that will listen for node Agent's connection on PORT 59 # default: runs a Multicast comm. module 60 # 61 # The following line runs the commServer in TCP Client Mode 62 #commServer: /opt/nodehandler4-4.2.0/sbin/commServer --logfile /tmp/commServer-%ID%.log -d 4 --iface eth1 -c 9026 63 # The following line runs the commServer in Multicast Mode 64 commServer: /opt/nodehandler4-4.2.0/sbin/commServer --logfile /tmp/commServer-%ID%.log -d 4 --iface eth0 65 66 # 67 # Return the IP address of the control interface of 68 # the node a coordinates x:y 69 # 70 # @param x X coordinate of node 71 # @param y Y coordinate of node 72 # 73 controlIp: | 74 |x, y| 75 # This is the Node Agents control IP address used in the WINLAB grid testbed 76 "10.10.#{x}.#{y}" 77 78 # 79 # Return the x:y coordinates of a node signing on with 80 # 'idString'. This string is supposed to be of the type 81 # '/ip/CONTROL_IP'. 82 # 83 # @param idString String of type '/ip/CONTROL_IP' 84 # @return Array of [x, y] 85 # 86 nodeId2coord: | 87 |idString| 88 match = /.*\.(\d+)\.(\d+)$/.match(idString) 89 if (match != nil && match.size == 3) 90 x = match[1].to_i 91 y = match[2].to_i 92 if x > 100 93 # sandbox 94 x = y / 100 95 y = y % 100 96 end 97 return [x, y] 98 end 99 raise "Can't parse #{idString}" 100 101 # Return the control IP address (as string) or DNS name for a node 102 # at a given coordinate. 103 # 104 # Throws an ConfigException if no IP address can be found. 105 # In this testbed, nodes are identified using only 1-dimensional coordinate: X 106 # (At WinLab, nodes are identified using 2D coordinates) 107 # 108 coord2ip: | 109 |x, y| 110 # This is the base name used in the WINLAB testbeds 111 name = "node#{x}-#{y}" 112 begin 113 Socket.gethostbyname(name)[3].unpack('C4').join('.') 114 rescue SocketError 115 raise("Unknown host '#{name}'") 116 end 117 118 load: | 119 | uri, evalRuby | 120 path = [ uri.split(':').join('_') + '.rb'] 121 postfix = '/' + uri.split(':').join('/') + '.rb' 122 REPOSITORY_DEFAULT().each { |dir| 123 path << dir + postfix 124 } 125 #puts "PATH: #{path.join(':')}" 126 file = path.inject(nil) { |found, p| 127 if found == nil && File.readable?(p) 128 found = p 129 end 130 found 131 } 132 if file == nil 133 raise IOError, "Can't find any of '#{path.join(', ')}'" 134 end 135 str = File.new(file).read() 136 if evalRuby 137 #eval(str, nil, path, 1) 138 require file 139 end 140 [str, 'text/ruby'] 141 142 # Config Parameter for the "grid" Testbed 143 # 144 # To use this testbed, call nodeHandler with the option "-d grid" 145 # Any parameter settings within this section will override the settings 146 # done in the "default" section. The nodeHandler fisrt load the "default" 147 # settings, then it uses the "domain" specific settings to override the 148 # relevant parameters 149 # 150 # For more details: see comments in "default" domain section 151 grid: 152 X_MAX: 2 153 Y_MAX: 20 154 oml: 155 local_host: '10.10.0.10' 156 url: "http://oml:5022/oml" 157 port: 5022 158 host: "oml" 159 controlIp: | 160 |x, y| 161 "10.10.#{x}.#{y}" 162 163 # Config Parameter for the "debug" Testbed 164 # 165 # To use this testbed, call nodeHandler with the option PRINT_ONLY or "-d debug" 166 # Any parameter settings within this section will override the settings 167 # done in the "default" section. The nodeHandler fisrt load the "default" 168 # settings, then it uses the "domain" specific settings to override the 169 # relevant parameters 170 debug: 171 repository: 172 path: ['../repository'] 173 commServer: ../c/commServer/commServer -d 4 --iface eth0 174 coord2ip: | 175 |x, y| 176 "10.99.#{x}.#{y}" }}} Comments:[[BR]] Line 29: PXE is one of services from gridservices2, and 5022 is its default port. Please make sure http://pxe could be resolved from '''Console'''.[[BR]] Line 33: CMC is the only service provided by gridservices, and 5012 is its default port. In my situation. The address is http://cmc is 10.1.200.1.[[BR]] Line 48: '/dev/sda' is the paramter frisbee need to image hard drivers. If the hard drivers in nodes are equipped with ATA interface, Please change it to "/dev/hda"[[BR]] Line 49: Similar with PXE, it's also a service by gridsercies2.[[BR]] Line 64: This is command used to launch the communication module of Nodehandler4. The value for parameter "--iface" should be the interface with IP address 10.10.0.10.[[BR]] Line 151: "grid" is the name of testbed which must match with the first word of domain name of '''Console'''. For example, the output of command "hostname -d" is "grid.poly.edu", so "grid" is the name of testbed.[[BR]] Line 152,153: The maximum value of two dimensions of testbed. If the '''Control port''' of each node has IP address 10.10.'''x'''.'''y''', the node's hostname should be like node'''x'''-'''y'''.grid.poly.edu, which is controlled by DNS. '''x''' and '''y''' are intergers less than X_MAX and Y_MAX respectively.[[BR]] * frisbee package is an dependency of gridservices2, which can be downloaded from [http://witestlab.poly.edu/attachment/wiki/BOOTB/frisbee_1.0.3-1_i386.deb?format=raw here] and install it with command[[BR]] ''dpkg -i frisbee_1.0.3-1_i386.deb''[[BR]] You can also get the latest source code of frisbee from [http://www.emulab.net/downloads/emulab-080630.tar.gz here] if you prefer to build it in your own system. You need get at least 2 executable files, '''frisbeed''' and '''frisbee'''. If you want make frisbee images, you must build '''imagezip''' from source code, which is not provided by frisbee debian package. For more information about frisbee, please vist http://www.emulab.net/software.php3 or read "README" attached with the source code.[[BR]][[BR]] * ''apt-get install gridservices2 oml-collection-server''[[BR]] The configuration of gridservices2 is under path /etc/gridservices2, Please go through README.txt under it first. Gridservices2 must enable at least 2 services, frisbee and PXE. The configuration of frisbee, '''frisbee.yaml''', looks like below {{{ 1 # NOTE: use only 'spaces' to indent ! 2 # ('tab' indents are not supported by the ruby yaml parser used to read this file) 3 # 4 # This is the Config file for the Frisbee GridService on the NICTA platform 5 # 6 --- 7 frisbee: 8 9 # Max. number of active daemons allowed 10 maxDaemons: 10 11 12 testbed: 13 default: 14 # Directory images are stored 15 imageDir: /home/node 16 defaultImage: baseline0.4 17 18 # max bandwidth for frisbee server 19 bandwidth: 50000000 20 21 # Multicast address to use for servicing images 22 mcAddress: 224.0.0.2 23 # Using ports starting at ... 24 startPort: 7000 25 26 # Time out frisbee server if nobody requested it within TIMEOUT sec 27 timeout: 3600 28 29 # Directory to find frisbee daemon 30 frisbeeBin: /usr/sbin/frisbeed 31 32 # Local interface to bind to for frisbee traffic 33 multicastIF: 10.10.0.10 34 35 indoor: 36 imageDir: /home/node 37 defaultImage: baseline0.4 38 bandwidth: 50000000 39 mcAddress: 224.0.0.2 40 startPort: 7000 41 timeout: 3600 42 frisbeeBin: /usr/sbin/frisbeed 43 multicastIF: 10.10.0.10 }}} Comments:[[BR]] Line 15: The place where frisbee image files are places.[[BR]] Line 16: The file name of image without "ndz" suffix. For example, if file name is '''baseline0.4.ndz''', it should be set to '''baseline0.4'''. The file will be the default image when no explicit image is given in omf command.[[BR]] Line 19: The maximum bandwidth in bps which frisbee can use to image hard drivers.[[BR]] Line 30: The place where gridservices2 can find frisbeed.[[BR]] The configuration of PXE, '''pxe.yaml''', looks like below. {{{ 1 pxe: 2 # Name of PXE config file 3 defImage: orbit-2.0.3-omf 4 5 # Directory pxe config files are stored 6 cfgDir: /tftpboot/pxelinux.cfg 7 8 # Maximum age of PXE symbolic link [sec] 9 linkLifetime: 900 10 # linkLifetime: 5 # for testing only 11 12 # toIP: mapping from x@y to IP address 13 # listAll: return array of x/y coodinates of all nodes in '[x,y]' form. 14 # 15 testbed: 16 default: 17 toIP: | 18 |x,y| 19 assertRange(x, 1..1, "unknown node #{x}@#{y}") 20 assertRange(y, 1..20, "unknown node #{x}@#{y}") 21 "10.10.#{x}.#{y}" 22 listAll: | 23 defGrid(1,4) }}} If all configuration are done, run following command to start gridservices2 * ''/etc/init.d/gridservices2 start'' For CMC, * ''apt-get install libmysqlclient15'' * ''apt-get install gridservices''[[BR]] Some files of gridservices needs upgrade. Get a [http://witestlab.poly.edu/attachment/wiki/BOOTB/gridservices.tar.gz?format=raw tar ball] and extract it to replace fold /ect/gridservices.[[BR]] Gridservices ONLY provides CMC service. It's configuration is defined in file /ect/gridservices/cmc.yaml, which looks like below.[[BR]] {{{ 1 primaryIF: "128.238.34" 2 communicators: 3 default: &comm_default 4 ip: 10.1.200.1 5 port: 9030 6 7 testbeds: 8 grid: 9 x_max: 2 10 y_max: 20 11 ip_block: lambda {|x,y| "10.1.#{x}.#{y}"} 12 inactive_list: [ ] 13 3vStatus: 0.016 14 5vStatus: 0.032 15 12vStatus: 0.064 }}} Comments:[[BR]] Line 1: This line defines the network address of the interface which connects to outside network. In my situation, CMC and Console are connected within network 128.238.34.*.[[BR]] Line 4: This line define the IP address of the interface which connects to the nodes.[[BR]] Line 9,10: The maximum value of two dimensions of testbed. The '''CM port''' of each node has IP address 10.1.'''x'''.'''y''', which is within the same network with 10.1.200.1.[[BR]] If all configuration are done, run following command to start gridservices * ''/etc/init.d/gridservices start'' If gridservices starts successfully, you can input the address below in web browser. You will find a couple of commands provided by CMC on web page and you can control CMC with the web interface. * ''http://cmc:5012/cmc'' == About the CM == The ORBIT Chassis Manager (CM) is a simple, reliable, platform-independent subsystem for managing and autonomously monitoring the status of each node in the Witestbed. Basically, it is a small PCI card port on the node. Administrators can turn on/off and reboot nodes remotely and monitor the status of nodes throught CM. User can access CM through serial console or telnet. For example. If a node's name is node2-3. the IP address of its '''control port''' is 10.10.2.3 and that of its '''CM port''' is 10.1.2.3. We can telnet to CM with command, * ''telnet 10.1.2.3'' We can also telnet to the node with either of two commands below * ''telnet 10.1.2.3 3025'' * ''telnet 10.10.2.3'' If you want to access CM through serial console, please set the baud rate to '''57600kps'''. == Updating the node BIOS == The new BIOS fixes some bugs and provides better support to network boot and power management. The detail description could be found [wiki:Internal/Infrastructure/SetupTestbed/UpdateTheNodeBios here].