Changes between Version 13 and Version 14 of Internal/BuildingStandaloneOrbit

Timestamp:

Jul 31, 2006, 11:07:22 PM (18 years ago)

Author:

Surya Satyavolu

Comment:

—

Internal/BuildingStandaloneOrbit

@@ -9 +9 @@
 [[Image(9nodesandbox.png)]]
 
-''We need to add some more text here, but right now we are just collecting various bits and pieces and I hope somebody will do some Wiki gardening when it gets unwieldy.''
-
-== Support Services ==
-
- * [wiki:HowTo/Building/PXE PXE] -- Bootstrap Orbit nodes
-
-== Developer Notes ==
-
- * [wiki:HowTo/Building/Notes/PXE Comments on the PXE Makefile]
- * [wiki:Orbit/Design/Notes/cm2cmcCommunication CM <-> CMC communication protocol]
-
-
-
-
+The following naming convention and IP addressing have been configured. 
+
+'''Services''': The machine “services” station is used to host various services such as DHCP, DNS, NTP, NFS, mysql, OML, repository etc. We have different aliases for the management host to segregate the services that it hosts
+
+||'''Station (services.orbit-lab.org)'''||'''IP Address (on eth1)'''||'''IP address (on eth0)'''||
+||dhcp.orbit-lab.org ||10.10.0.9 (fixed) || 10.1.200.2 (fixed)||
+||ntp.orbit-lab.org||||||
+||idb1.orbit-lab.org||||||
+||edb1.orbit-lab.org||||||
+||repository1.orbit-lab.org||||||
+||repository2.orbit-lab.org||||||
+
+'''Console''': The machine ''console'' is used to run the experiments as well as host the CMC service. Hence, console machine has nodeHandler and CMC in order to support the above functionality. The aliases for this machine are
+
+||'''Station (console.orbit-lab.org)'''||'''IP Address (on eth1)'''||'''IP address (on eth0)'''||
+||cmc.orbit-lab.org ||10.10.0.21 (fixed) || 10.1.200.1 (fixed)||
+||console.orbit-lab.org||||||
+||consolec.orbit-lab.org||||||
+
+
+= Installation Common to both machines = 
+
+The following procedure applies to both services and console. On a clean machine,
+ * install debian using CD
+   * Download the net-install ISO image (about 104 MB) from [http://cdimage.debian.org/pub/cdimage-testing/sarge_d-i/i386/rc3/sarge-i386-netinst.iso here]
+   * Burn this onto a CD, make sure that you burn it as an ISO image and not a file; otherwise the CD will not be bootable. You can use the shareware utility UltraISO (http://www.ezbsystems.com/ultraiso/) to burn this image onto a CD
+   * Use this CD to boot the new machine up (make sure that you have the correct boot sequence in the BIOS settings)
+   * Ensure that you have a network connection to outside
+   * At the boot prompt: type linux26 to instruct the installer to use kernel 2.6 for the installation. This will ensure that most of the hardware (including serial ATA harddisks) will be recognized without any problems.
+   * Follow instructions on the installer to proceed with the installation
+
+ * Run apt-get install kernel-image-2.6.10-1-i686
+    * In order to do this make sure you have the debian sources pointing to unstable in the /etc/apt/sources.list file 
+ * Run update-grub
+ * Reboot into 2.6.10
+
+Setting up the networking for these machines can be done as follows
+ * Edit /etc/network/interfaces
+
+For services.orbit-lab.org
+{{{
+auto lo eth0 eth1
+
+iface lo inet loopback
+
+iface eth0 inet static
+        address 10.1.200.2
+        netmask 255.255.0.0
+
+iface eth1 inet static
+address 10.10.0.9
+     netmask 255.255.0.0
+
+}}}
+For Console.orbit-lab.org  
+{{{
+auto lo eth0 eth1
+
+iface lo inet loopback
+
+iface eth0 inet static
+        address 10.1.200.1
+        netmask 255.255.0.0
+
+iface eth1 inet static
+address 10.10.0.21
+     netmask 255.255.0.0
+}}}
+
+== Software and Services ==
+The services and software running on each of the two servers is listed below. Configuration of individual services is explained in detail as well
+||'''(services.orbit-lab.org)'''||'''(console.orbit-lab.org)'''||
+|| DHCP || CMC || 
+|| DNS || NodeHandler || 
+|| NTP ||  || 
+|| TFTP ||  || 
+|| PXE ||  || 
+|| Frisbee ||  || 
+|| NFS ||  || 
+|| Apt ||  || 
+|| Mysql ||  || 
+|| OML collection server ||  || 
+|| Apache ||  || 
+
+=== DHCP ===
+'''Purpose''': This software runs a DHCP server that assigns IP addresses to clients on demand
+
+'''Installation and Configuration: ''': 
+ * apt-get install dhcp3-server : select eth1 as interface
+ * Edit /etc/dhcp3/dhcpd.conf
+ * Make sure that /etc/default/dhcp3-server has eth1 as interface
+ * First line is use-host-decl-names on which means DNS has actual IP mappings, DHCP just gives out hostnames. This means only IP to name updates need to be done only at DNS. 
+
+'''To Run: ''': 
+/etc/init.d/dhcp3-server start  -- errors go to /var/log/daemon.log
+
+'''Configuration ''':
+{{{
+#############dhcpd.conf################
+ddns-update-style interim;
+authoritative;
+use-host-decl-names on;
+
+allow booting;
+allow bootp;
+
+default-lease-time 259200;
+max-lease-time 259200;
+
+filename "/pxelinux.bin";
+option vendor-encapsulated-options 09:0f:80:00:0c:4e:65:74:77:6f:72:6b:20:62:6f:6f:74:0a:07:00:50:72:6f:6d:70:74:06:01:02:08:03:80:00:00:47:04:80:00:00:00:ff;
+option option-150 code 150 = text;
+
+#
+#Grid Control
+subnet 10.10.0.0 netmask 255.255.0.0 {
+        range  10.10.250.1 10.10.250.255;
+        option domain-name "orbit-lab.org";
+        ddns-updates off;
+        ddns-domainname "orbit-lab.org";
+        ddns-rev-domainname "in-addr.arpa";
+        option domain-name-servers 10.10.0.9;
+        next-server 10.10.0.9;
+        option routers 10.10.0.1;
+
+        host services.orbit-lab.org {hardware ethernet 00:40:63:DA:12:0E; fixed-address services.orbit-lab.org ;}
+        #replace the MAC addresses here with those of the new machine
+        host console.orbit-lab.org  {hardware ethernet 00:40:63:C9:F6:D5; fixed-address console.orbit-lab.org ;}
+        
+        #Nodes 10.10.x.y
+        group {
+        host node1-1.orbit-lab.org {hardware ethernet 00:0F:EA:4C:A6:B7; fixed-address node1-1;}
+        host node1-2.orbit-lab.org {hardware ethernet 00:0F:EA:4A:8B:5B; fixed-address node1-2;}
+        host node1-3.orbit-lab.org {hardware ethernet 00:0F:EA:47:6C:87; fixed-address node1-3;}
+        host node2-1.orbit-lab.org {hardware ethernet 00:0F:EA:4C:A6:C1; fixed-address node2-1;}
+        host node2-2.orbit-lab.org {hardware ethernet 00:0F:EA:4C:B7:5D; fixed-address node2-2;}
+        host node2-3.orbit-lab.org {hardware ethernet 00:0F:EA:4B:FE:C7; fixed-address node2-3;}
+        host node3-1.orbit-lab.org {hardware ethernet 00:0F:EA:4A:8A:F7; fixed-address node3-1;}
+        host node3-2.orbit-lab.org {hardware ethernet 00:0F:EA:4C:B7:53; fixed-address node3-2;}
+        host node3-3.orbit-lab.org {hardware ethernet 00:0F:EA:4A:8B:63; fixed-address node3-3;}
+        }
+}
+}}}
+
+ === DNS ===
+'''Purpose''': Services.orbit-lab.org” hosts the primary DNS service for the zone orbit-lab.org. The DNS server is the standard BIND9 software packaged with Linux
+
+
+'''Installation and Configuration: ''': 
+ * apt-get install bind
+ * You need to create the following files under /etc/bind
+   * Named.conf : Configuration for DNS as shown below
+   * Orbit.zone: Forward lookup
+   * Zone.orbit: Reverse lookup
+   * In named.conf, edit destination directory to /etc/bind
+ * Make sure there are dots at the end of the domains
+ * Add the highlighted blue line to /etc/hosts file as follows
+ ||127.0.0.1    || localhost||
+|| 10.10.0.9    || services.orbit-lab.org       ||      services ||
+
+ '''To Run: ''': 
+/etc/init.d/bind start  -- errors go to /var/log/daemon.log
+
+''' Configuration ''' 
+{{{
+########################/etc/bind/named.conf##############################
+options {
+        directory "/etc/bind";
+        /*
+         * If there is a firewall between you and nameservers you want
+         * to talk to, you might need to uncomment the query-source
+         * directive below.  Previous versions of BIND always asked
+         * questions using port 53, but BIND 8.1 uses an unprivileged
+         * port by default.
+         */
+};
+
+Controls {
+       inet 127.0.0.1 port 953
+       allow { 127.0.0.1; };
+};
+
+// a caching only nameserver config
+ 
+zone "." IN {
+        type hint;
+        file "named.ca";
+};
+
+zone "localhost" IN {
+        type master;
+        file "localhost.zone";
+};
+
+
+zone "orbit-lab.org" IN {
+        type master;
+        file "orbit.zone";
+        
+};
+
+zone "in-addr.arpa" IN {
+        type master;
+        file "zone.orbit";
+        
+};
+
+zone "0.0.127.in-addr.arpa" IN {
+        type master;
+        file "named.local";
+};
+#####EOF#########
+
+###############/etc/bind/orbit.zone#################
+$ORIGIN .
+$TTL 3600       ; 1 hour
+orbit-lab.org   IN SOA  orbit-lab.org services.orbit-lab.org (
+                                2005041903 ; serial
+                                3600       ; refresh (1 hour)
+                                600        ; retry (10 minutes)
+                                10000       ; expire (2 hours)
+                                3600       ; minimum (1 hour)
+                                )
+        IN      NS      services.orbit-lab.org.
+                                
+$ORIGIN orbit-lab.org. 
+$TTL 129600     ; 1 day 12 hours
+node1-1         A       10.10.1.1
+node1-2         A       10.10.1.2
+node1-3         A       10.10.1.3
+node2-1         A       10.10.2.1
+node2-2         A       10.10.2.2
+node2-3         A       10.10.2.3
+node3-1         A       10.10.3.1
+node3-2         A       10.10.3.2
+node3-3         A       10.10.3.3
+services                A       10.10.0.9
+console         A       10.10.0.21
+dhcp            CNAME   services.orbit-lab.org.
+cmc             CNAME   console.orbit-lab.org.
+consolec        CNAME   console.orbit-lab.org.
+frisbee CNAME   services.orbit-lab.org.
+pxe             CNAME   services.orbit-lab.org.
+oml             CNAME   services.orbit-lab.org.
+repository      CNAME   services.orbit-lab.org.
+repository1     CNAME   services.orbit-lab.org.
+repository2     CNAME   services.orbit-lab.org.
+ntp             CNAME   services.orbit-lab.org.
+loghost CNAME   console.orbit-lab.org.
+idb1            CNAME   services.orbit-lab.org.
+edb1            CNAME   services.orbit-lab.org.
+#####EOF#########
+
+######################/etc/bind/zone.orbit##################
+
+$ORIGIN .
+$TTL 3600       ; 1 hour
+in-addr.arpa            IN SOA  orbit-lab.org. services.orbit-lab.org. (
+                                33354      ; serial
+                                28800      ; refresh (8 hours)
+                                900        ; retry (15 minutes)
+                                604800     ; expire (1 week)
+                                86400      ; minimum (1 day)
+                                )
+        IN              NS      services.orbit-lab.org.
+$ORIGIN in-addr.arpa.
+$ORIGIN 10.in-addr.arpa.
+$ORIGIN 0.10.10.in-addr.arpa.
+$TTL 129600     ; 1 day 12 hours
+9       IN      PTR             services.orbit-lab.org.
+21      IN      PTR             console.orbit-lab.org.
+
+$ORIGIN 1.10.10.in-addr.arpa.
+1       IN      PTR             node1-1.orbit-lab.org.
+2       IN      PTR             node1-2.orbit-lab.org.
+3       IN      PTR             node1-3.orbit-lab.org.
+
+$ORIGIN 2.10.10.in-addr.arpa.
+1       IN      PTR             node2-1.orbit-lab.org.
+2       IN      PTR             node2-2.orbit-lab.org.
+3       IN      PTR             node2-3.orbit-lab.org.
+
+$ORIGIN 3.10.10.in-addr.arpa.
+1       IN      PTR             node3-1.orbit-lab.org.
+2       IN      PTR             node3-2.orbit-lab.org.
+3       IN      PTR             node3-3.orbit-lab.org.
+
+#####EOF#########
+
+}}}
+=== Apache Web Server ===
+'''Purpose:'''
+Apache server is required for maintaining the ORBIT local repository for debian packages and also to view the results of the experiment
+
+'''Installation:'''
+apt-get install apache2
+Note that no additional configuration is needed for apache
+
+'''To run:'''
+ *  Make sure that /var/www/cgi-bin points to /usr/lib/cgi-bin (or create a soft link if one does not exist using cd /var/www/; ln –s /usr/lib/cgi-bin cgi-bin)
+
+'''To run'''
+/etc/init.d/apache2 start -- errors go to /var/log/daemon.log
+
+Make sure that you have latest version of libgd and the perl modules for GD available. This is needed to view the results of the experiment. To do that:
+
+ * wget http://www.boutell.com/gd/http/gd-2.0.33.tar.gz
+ * tar xfzv gd-2.0.33.tar.gz; cd gd-2.0.33;
+ * ./configure –prefix=/usr ; make ; make install
+Once this is done, download the latest GD perl module that depends on the previously installed libgd library using the following commands
+
+ * wget http://search.cpan.org/CPAN/authors/id/L/LD/LDS/GD-2.19.tar.gz
+ * tar xfzv GD-2.19.tar.gz; cd gd-2.19;
+ * perl Makefile.PL; make; make install
+
+The above two steps should install the GD library and perl modules correctly.
+
+=== Repository ===
+'''Purpose:'''
+This serves as the debian package repository for local ORBIT packages (such as OML, OTG, Libmac, nodeAgent etc). It uses apache services to fetch and distribute debian packages. For this service, it is sufficient to install just the ORBIT packages. There is no need to mirror the entire debian repository which can be easily accessed by syncing with the external debian repository
+
+'''Configuration:'''
+ * mkdir /var/www/winlab 
+ * chmod 755 /var/www/winlab
+ * mkdir /var/www/winlab/binary
+Under the above directory, make sure you have the following packages
+ * Frisbee_1.0.0-1.i386.deb  = This has the Frisbeed, Frisbee client
+ * Gridservices_3.0.0-1_all.deb = This has Frisbee and pxe webservices (http wrapper to initialize frisbeed while frisbeeing)
+ * Oml-client_2.0_0_i386.deb = oml client
+ * Otg_1.1.0-1_i386.deb = traffic generator
+ * OML_server, CMC will be integrated into the gridservices packages soon
+
+Edit the /etc/apt/sources.list file as follows
+{{{
+deb http://repository1.orbit-lab.org/winlab/ binary/
+
+deb http://http.us.debian.org/debian/ testing main
+ deb-src http://http.us.debian.org/debian/ testing main
+deb http://security.debian.org/ stable/updates main
+}}}
+
+=== NTP ===
+
+'''Purpose:'''
+ All the machines synchronize their time using the time server as the reference. 
+
+'''Installation and Configuration:'''
+ * apt-get install ntp-server
+ * Edit /etc/ntpd.conf
+   * Point the server to pool.ntp.org
+
+'''To run:'''
+ * /etc/init.d/ntpd start -- errors go to /var/log/daemon.log
+
+=== TFTP Server ===
+'''Purpose:'''
+TFTP is needed to install PXE images whenever you need to install an image onto the node (using Frisbee). It is also used to load a memory based image that can be used to fetch the current image of the node into the repository
+
+'''Installation and Configuration:'''
+ * apt-get install atftpd
+
+There are two options here: either to run atftpd as a standalone daemon or run it under inetd. For heavy duty tftp services, you can choose to run is as a standalone daemon.
+For our installation, we choose the standalone daemon.
+
+ * Point the tftp directory to /tftpboot
+ * Copy /tftpboot directory from repository2.orbit-lab.org
+
+In case of problems, make sure that lo interface is up
+
+=== NFS Service ===
+
+'''Purpose:'''
+This service is used to remotely mount directories on the nodes while fetching their image using imagezip utility. Also, Frisbee service makes use of this directory to install images onto nodes.
+
+'''Installation and Configuration:'''
+
+ * apt-get install nfs-kernel-server
+
+ * Edit the /etc/exports file as follows
+Probably should be sync here
+
+'''To run:'''
+ * /etc/init.d/nfs-kernel-server start -- errors go to /var/log/daemon.log
+
+=== Mysql Server ===
+
+'''Purpose:'''
+This service is used to store the results of the experiments conducted on ORBIT
+
+''' Installation and Configuration:'''
+ * apt-get install mysql-server-4.1
+ * Edit /etc/mysql/my.cnf and replace bind-address from 127.0.0.1 to 10.10.0.9
+type mysql  and at the prompt, enter the following
+
+ * Replace the password by an appropriate one. Basically, this creates a new account called orbit with the password specified and allows access to mysql databases from localhost and any other machine on the network.
+
+'''To run:'''
+ * /etc/init.d/mysql start -- errors go to /var/log/mysql.log 
+
+=== OML Server (Collection Server) ===
+'''Purpose:'''
+ This service is used to collect the measurements of the experiments at runtime from the nodes and store them into the mysql database.
+
+''' Installation and Configuration:'''
+ * Apt-get install tcl8.4
+ * Apt-get install mysqltcl
+ * Install libghthash-0.5.3 from source and point /etc/ld.so.conf to /usr/local/lib
+ * Copy /etc/init.d/collectionwebService2.0  from idb1
+ * Copy /opt/nodeAgent/bin/ from idb1
+ * Copy /opt/nodeAgent/lib/ from idb1
+ * Copy /opt/oml_server/bin/collection_server from idb1
+ * Copy /usr/local/lib/oml from idb1
+
+'''To run:'''
+ * /etc/init.d/collectionWebService2.0 start -- errors go to /var/log/daemon.log
+
+Note that that will be packaged soon as a complete deb package with all dependency checks
+
+=== Gridservices (PXE and Frisbee) ===
+
+'''Purpose:'''
+ This service is for fetching and installing images from and to the nodes and used the Frisbee utility from University of Utah to distribute the image in the form of chunks using multicast.
+
+'''Installation and Configuration:'''
+ * Apt-get install gridservices
+ * Apt-get install frisbee
+ * /opt/gridservices/services/frisbee.rb has 10.10.0.40 hardcoded for Frisbee server. Make sure you change that to 10.10.0.9 which is our Frisbee server
+ * Frisbeed installs under /usr/sbin when you apt-get it, however /opt/gridservices/services/Frisbee.rb points to /opt/Frisbee/bin .Make sure , you change that or create a soft link from /opt/frisbee/bin/frisbee to /usr/sbin/frisbee
+
+'''To run:'''
+/etc/init.d/gridservices start -- errors go to /var/log/gridservices.log
+
+=== CMC (Runs on console.orbit-lab.org) ===
+'''Purpose:'''
+CMC service talks to the individual Chassis Managers on the nodes and can be used to remotely power on, power off nodes, reset them and also provided access serial console of the nodes to the users. This is done using a http interface (Make sure that you do not run this service and apache on the same machine, since they will both need port 80)
+
+'''Installation and Configuration:'''
+ * apt-get install libmysqlclient12
+ * apt-get install libx11-6
+ * Make sure that the interface eth0  is set to 10.1.200.1 netmask 255.255.0.0
+ * Copy /usr/local/sbin/cmc on to local machine
+ * Copy /etc/init.d/cmc 
+
+'''To run:'''
+ * /etc/init.d/cmc start 
+
+Note that that will be packaged soon as a complete deb package with all dependency checks
+
+=== Node Handler (runs on console.orbit-lab.org) ===
+
+'''Purpose:'''
+NodeHandler is used to run the experiments from the console using reliable multicast to disseminate commands to the nodes. It also initializes measurement collection appropriately. It primarily uses ruby scripts to do the same.  
+
+'''Installation and Configuration:'''
+Make sure that you apt-get install the following packages
+ * liblog4r-ruby  1.0.5-3        
+ * librexml-ruby  3.1.2.1+ruby1. 
+ * libruby1.8     1.8.2-3        
+ * libwebrick-rub 1.3.1+ruby1.8. 
+ * ruby           1.8.2-1        
+ * ruby1.8        1.8.2-3       
+Copy /opt/nodeAgent3 from console.grid.orbit-lab.org to this machine
+
+Note that that will be packaged soon as a complete deb package with all dependency checks
+
+== Switch Configuration ==
+ 
+ * Partition switch into two VLAN’s
+ * 1st 12 ports for control, next 12 for CM
+
+== Useful Tips ==
+ * /etc/init.d/ifupdown should have ifup –a so that all the interfaces are up at system boot
+ * To bring up all the services automatically after a reboot, use sysv-rc-conf utility
+ * To have modules load at boot time (put them in /etc/modules)
+ * Use tzconfig to change your time zone if required
+
+
+
+
+