Changes between Version 45 and Version 46 of Internal/InventoryV3

Timestamp:

Jan 10, 2013, 9:50:32 PM (12 years ago)

Author:

ssugrim

Comment:

—

Internal/InventoryV3

@@ -1 +1 @@
 James is working on a second generation inventory script.
 
-Currently on: Restructuring writer and correcting table assumptions
+Currently on: Rest Integration for new database type and Cpu Bench marks
 
 Versions:
 
- * gather.rb:2.30
- * inventoryNode.rb: (Written by Ivan, Live in the /root directory of the respective console).
-
-It's plan is to be simpler and less ambitious than it's predecessor, but still respect the sql table structure ("as much as possible.") 
-
-There are 4 parts to this process:
- 1. inventoryv2.rb: execs scripts on nodes using orbit framework
- 1. gatherer.rb: collects information using only  operating system based facilities (dmesg, lsusb, lspci, ifconfig, /sys, lshw).  
- 1. writer.rb: checks the mysql repository for changes from the current state. If different changes them.
- 1. logcopy.rb: check the log file for errors, if present copies the Logs to a specfied location
-
-
-The sql structure is a bit of a bit mess, the major tables of interest are:
- 1. motherboards - List of things that can be connected to, has its own id used to tie other tables to it
- 1. devices - List of deviced "connected" to mother boards 
- 1. device_kinds - type identifier for connected devices (an attribute of a device). 
- 1. locations - Converts x,y,testbed_id coordinates to a single integer
- 1. nodes - maps motherboard to locations, also has an id for said mapping
- 1. inventories - records the start and stop time of the inventory pass.
- 1. testbeds - gives a testbed id for the specific domain, thus disambiguating node1-1
-
-A lot of the tables are full of unused colums. I guess we'll just ignore them for now. 
-The basic crux of an update should be the following:
- 1. examine our IP (and hostname) to determine our current location
- 1. We gather information about the mother board:
- 1. '''Gatherer:'''
-  1. Disk Size (dmesg)
-  1. Memory Size (dmesg)
-  1. Cpu number (dmesg)
-  1. motherboard serial number (lshw)
-  1. Gather information about attached devices:
-   1. 2 wired Ethernet addresses (ifconfig, /sys)
-   1. 2 wireless Ethernet addresses (ifconfig, /sys)
-   1. any usb devices (lsusb, /sys)
-  1. export to xml
- 1. '''Writer:'''
-  1. import xml output from gatherer
-  1. collect identifiers from mysql based on gathered infromation (domain => testbed_id ;x,y,testbed_id => loc_id;
-     mb_ser => mb_id ; loc_id => node_id ; )
-  1. update mother board information if different, and stamp with current inventory number
-  1. add kinds if they don't exist already
-  1. update devices if diffrent and stamp with inventory number
-  1. update node mb_id if loc, mb pair don't match
- 1. profit.
-
+ * gather.rb:3.5
+ * rest_db.rb:1.3
+ * log_wrap.rb:0.1
+
+
+It's plan is to be simpler and less ambitious than it's predecessor. Instead of the old sql, we'll now use the new rest client. 
+
+There are 3 parts to this process:
+
+ 1. gatherer.rb: collects information using operating system based facilities and tools(dmesg, lsusb, lspci, ifconfig, /sys, lshw, sysbench).  
+ 1. rest_db.rb: a generic rest-client wrapper library for interfacing the rest based database with and it's flat data structure
+ 1. log_wrap.rb: a unified log class that can be share across many distributed ruby files.
+
+
+The Rest Data base structure is a flat resource / attribute mode. Nodes and their "attached" devices are consiered resouces. The devices attached to a node are children resouces of the node. Both nodes and their devices will have meaning full attributes. Attributes come in 3 heading types INV, CM, INF. Those are described here. gathering is only concerned with INV type attributed: This is what is colleted and how they are related
+ 1. Nodes
+  1. INV_hd_sn
+  1. INV_mb_sn
+  1. INV_cpu_bench
+  1. INV_memory
+  1. INV_cpu_hz
+  1. INV_cpu_type
+  1. INV_hd_size
+  1. INV_check_in
+ 1. Devices
+  1. INV_dev_id
+  1. INV_dev_type
+  1. INV_if_name (optional)
+  1. INV_if_mac (optional)
 ----
 == Require Tools / Libraries ==
 
- 1. lsusb (usbutils.deb)
- 1. lspci (native)
- 1. dmesg (native)
- 1. ifconfig (native)
- 1. libxml-simple-ruby.deb
- 1. libmysql-ruby.deb
- 1. lshw (lshw.deb)
- 1. logger (ruby standard)
+=== Ubuntu Standard ===
+ 1. date 
+ 1. hostname 
+ 1. lsusb
+ 1. lspci
+ 1. dmesg
+ 1. ifconfig 
+ 1. lshw
+
+=== apt-pacakge ===
+ 1. librestclient-ruby
+ 1. sysbench
+
+=== ruby standard ===
+ 1. logger
  1. ftools (ruby standard)
 
 ----
 Gatherer:
-The disk size and memory size are a quick scan from dmesg. The disk size matches, but the memory size is a little off. It probably has to do with the way dmesg reports memory vs /sys reports memeory. It would be nice to find the /sys entry for consistency. 
-
-In /sys/devices/pci0000:00 are the sub directories correlated with the specific Ethernet hardware. In each directory that correlated to an Ethernet device there will be a symbolic link with the operating system name of the device. This will allow us to match up the pci address(name of the subdirectory of /sys/devices/pci0000:00) to the mac address (from ifconfig). lspci can tell us the associated pci address and a hardware identifier string.
-
-lsusb on the otherhand offers a direct correlation to the device kind table, the ordered pair of numbers xxxx:yyyy directly correlated to the tables vendor and device ids.  And the Bus xxx Device yyy number fits into the addres category of the device table.
+
+Gatherer is now the only executable. It preforms both gathering and updating (via the rest client). It's collects information mostly via regular expressions on the output of the various tools and uses them to identify the node (by it's fqdn), and populate the data paramters. 
+
+----
+
+= NOTES =
 
 === 9/29/09 ===
@@ -462 +454 @@
 I may branch off a threaded version to make it a little faster but that might right into blocking issues with concurrent runs of lshw. Also on the Horizon is the idea of using a full Diskless ubuntu server pxe image to house the inventory process. This should resolve some of the missing info/driver issues.
 
+=== 1/10/2013 ===
+
+This project has essentially been completely rewritten from the ground up to accommodate the new data base type described here. Most of the regular expression and tools are the same but the data base interface is completely revamped (separate library). We collect fewer pieces of data, but each piece is more meaningful, and their logical context now easier to ascertain from the db structure. The central identifier is now the fqdn of the node (this establishes it position as well as it's heirarchy), the old "context-less unique identifier" is long gone. 
+
+To launch the process we now use a start up script instead of the old oml construct. This means the process to run inventory is as simple as:
+
+ 1. Image nodes with inventory image
+ 2. restart nodes
+
+While this method is effective, it has the draw back that each time we update gatherer, a new image has to be created. This is not so big a deal because gatherer updates come so infrequently, that the image usually ends up having to be updated any way. The inventory start up script looks like
+{{{
+#!/bin/bash
+echo "Sleeping 30" > /tmp/inventory.log
+sleep 30
+let amt=$RANDOM%10 
+echo "Sleep $amt" >> /tmp/inventory.log
+sleep $amt 
+echo "NTP info" >> /tmp/inventory.log
+ntpq --numeric --peers >> /tmp/inventory.log
+echo "Starting gatherer" >> /tmp/inventory.log
+ruby -I /root/ /root/gatherer.rb -d  -l /tmp/gatherer.log >> /tmp/inventory.log
+echo "Done" >> /tmp/inventory.log
+}}}
+
+Add this file to /etc/init.d and then run
+{{{
+update-rc.d inventory.sh defaults 99
+}}}
+
+The out put should look like:
+{{{
+oot@node1-1:~# update-rc.d inventory.sh defaults 99
+update-rc.d: warning: /etc/init.d/inventory.sh missing LSB information
+update-rc.d: see <http://wiki.debian.org/LSBInitScripts>
+ Adding system startup for /etc/init.d/inventory.sh ...
+   /etc/rc0.d/K99inventory.sh -> ../init.d/inventory.sh
+   /etc/rc1.d/K99inventory.sh -> ../init.d/inventory.sh
+   /etc/rc6.d/K99inventory.sh -> ../init.d/inventory.sh
+   /etc/rc2.d/S99inventory.sh -> ../init.d/inventory.sh
+   /etc/rc3.d/S99inventory.sh -> ../init.d/inventory.sh
+   /etc/rc4.d/S99inventory.sh -> ../init.d/inventory.sh
+   /etc/rc5.d/S99inventory.sh -> ../init.d/inventory.sh
+}}}
+
+Note: For full enumeration of all devices on a baseline image, make sure you remove the modprobe blacklists. If you don't the wifi modules won't get enumerated and the gathering won't find them.