2008-07-20 23:38:04 INFO nodeHandler::init: NodeHandler Version 4.2.0 (1272) 2008-07-20 23:38:04 INFO nodeHandler::init: Experiment ID: sb1_2008_07_20_23_38_04 2008-07-20 23:38:04 DEBUG nodeHandler::Experiment: command line args: --node[[1,1]]--pxe1.2.1-omf--domainsb1. 2008-07-20 23:38:04 DEBUG nodeHandler::ExecApp: Starting application 'commServer': /opt/nodehandler4-4.2.0/sbin/commServer --logfile /tmp/commServer-sb1_2008_07_20_23_38_04.log -d 4 --iface eth1 2008-07-20 23:38:04 DEBUG nodeHandler::Communicator: commServer(STARTED): '' 2008-07-20 23:38:06 DEBUG nodeHandler::ExecApp: writing 'R' to app 'commServer' 2008-07-20 23:38:07 DEBUG nodeHandler::web: Mounting /resource to /opt/nodehandler4-4.2.0/repository/public_html 2008-07-20 23:38:07 INFO nodeHandler::Experiment: load system:exp:stdlib 2008-07-20 23:38:08 INFO nodeHandler::prop.resetDelay: resetDelay = 210:Fixnum 2008-07-20 23:38:08 INFO nodeHandler::prop.resetTries: resetTries = 1:Fixnum 2008-07-20 23:38:08 INFO nodeHandler::Experiment: load system:exp:saveNode 2008-07-20 23:38:09 DEBUG nodeHandler::Experiment: Setting property "node" to "[[1, 1]]". 2008-07-20 23:38:09 INFO nodeHandler::prop.node: node = [[1, 1]]:Array 2008-07-20 23:38:09 DEBUG nodeHandler::Experiment: Setting property "pxe" to ""1.2.1-omf"". 2008-07-20 23:38:09 INFO nodeHandler::prop.pxe: pxe = "1.2.1-omf":String 2008-07-20 23:38:09 DEBUG nodeHandler::Experiment: Setting property "domain" to ""sb1"". 2008-07-20 23:38:09 INFO nodeHandler::prop.domain: domain = "sb1":String 2008-07-20 23:38:09 INFO nodeHandler::prop.started: started = "false":String 2008-07-20 23:38:09 DEBUG nodeHandler::service call: http://cmc:5012/cmc/allStatus?domain= 2008-07-20 23:38:09 DEBUG nodeHandler::ExecApp: writing 'a 10.11.1.1 n_1_1' to app 'commServer' 2008-07-20 23:38:09 DEBUG nodeHandler::node::n_1_1: Created node 1@1 2008-07-20 23:38:09 DEBUG nodeHandler::node::n_1_1: Added to group 'save' 2008-07-20 23:38:09 DEBUG nodeHandler::ExecApp: writing 'A n_1_1 save' to app 'commServer' 2008-07-20 23:38:09 DEBUG nodeHandler::set::save: PXE: http://pxe:5022/pxe/setBootImageNS?domain=sb1&img=orbit-1.2.1-omf&ns=[[1,1]] 2008-07-20 23:38:09 DEBUG nodeHandler::service call: http://pxe:5022/pxe/setBootImageNS?domain=sb1&img=orbit-1.2.1-omf&ns=[[1,1]] 2008-07-20 23:38:09 DEBUG nodeHandler::set::_ALL_: Running each for save 2008-07-20 23:38:09 DEBUG nodeHandler::set::_ALL_: Running 'eachGroup' in RootGroupNodeSet 2008-07-20 23:38:09 DEBUG nodeHandler::set::_ALL_: Call save 2008-07-20 23:38:09 DEBUG nodeHandler::set::_ALL_: Running 'eachGroup' in RootGroupNodeSet 2008-07-20 23:38:09 DEBUG nodeHandler::set::_ALL_: Call save 2008-07-20 23:38:09 DEBUG nodeHandler::CMC: up http://cmc:5012/cmc/nodeSetOn?nodes=[[1,1]] 2008-07-20 23:38:09 DEBUG nodeHandler::service call: http://cmc:5012/cmc/nodeSetOn?nodes=[[1,1]] 2008-07-20 23:38:18 DEBUG nodeHandler::every: *: fires 2008-07-20 23:38:18 DEBUG nodeHandler::set::_ALL_: Running each for save 2008-07-20 23:38:18 INFO nodeHandler::stdlib: Waiting for nodes (Up/Down/Total): 0/1/1 - (still down: n_1_1) 2008-07-20 23:38:19 DEBUG nodeHandler::every: save: fires 2008-07-20 23:38:19 DEBUG nodeHandler::every: save: fires 2008-07-20 23:38:28 DEBUG nodeHandler::every: *: fires 2008-07-20 23:38:28 DEBUG nodeHandler::set::_ALL_: Running each for save 2008-07-20 23:38:28 INFO nodeHandler::stdlib: Waiting for nodes (Up/Down/Total): 0/1/1 - (still down: n_1_1) 2008-07-20 23:38:29 DEBUG nodeHandler::every: save: fires 2008-07-20 23:38:29 DEBUG nodeHandler::every: save: fires 2008-07-20 23:38:38 DEBUG nodeHandler::every: *: fires 2008-07-20 23:38:38 DEBUG nodeHandler::set::_ALL_: Running each for save 2008-07-20 23:38:38 INFO nodeHandler::stdlib: Waiting for nodes (Up/Down/Total): 0/1/1 - (still down: n_1_1) 2008-07-20 23:38:39 DEBUG nodeHandler::every: save: fires 2008-07-20 23:38:39 DEBUG nodeHandler::every: save: fires 2008-07-20 23:38:48 DEBUG nodeHandler::every: *: fires 2008-07-20 23:38:48 DEBUG nodeHandler::set::_ALL_: Running each for save 2008-07-20 23:38:48 INFO nodeHandler::stdlib: Waiting for nodes (Up/Down/Total): 0/1/1 - (still down: n_1_1) 2008-07-20 23:38:49 DEBUG nodeHandler::every: save: fires 2008-07-20 23:38:49 DEBUG nodeHandler::every: save: fires 2008-07-20 23:38:58 DEBUG nodeHandler::every: *: fires 2008-07-20 23:38:58 DEBUG nodeHandler::set::_ALL_: Running each for save 2008-07-20 23:38:58 INFO nodeHandler::stdlib: Waiting for nodes (Up/Down/Total): 0/1/1 - (still down: n_1_1) 2008-07-20 23:38:59 DEBUG nodeHandler::every: save: fires 2008-07-20 23:38:59 DEBUG nodeHandler::every: save: fires 2008-07-20 23:39:08 DEBUG nodeHandler::every: *: fires 2008-07-20 23:39:08 DEBUG nodeHandler::set::_ALL_: Running each for save 2008-07-20 23:39:08 INFO nodeHandler::stdlib: Waiting for nodes (Up/Down/Total): 0/1/1 - (still down: n_1_1) 2008-07-20 23:39:09 DEBUG nodeHandler::every: save: fires 2008-07-20 23:39:09 DEBUG nodeHandler::every: save: fires 2008-07-20 23:39:18 DEBUG nodeHandler::every: *: fires 2008-07-20 23:39:18 DEBUG nodeHandler::set::_ALL_: Running each for save 2008-07-20 23:39:18 INFO nodeHandler::stdlib: Waiting for nodes (Up/Down/Total): 0/1/1 - (still down: n_1_1) 2008-07-20 23:39:19 DEBUG nodeHandler::every: save: fires 2008-07-20 23:39:19 DEBUG nodeHandler::every: save: fires 2008-07-20 23:39:28 DEBUG nodeHandler::every: *: fires 2008-07-20 23:39:28 DEBUG nodeHandler::set::_ALL_: Running each for save 2008-07-20 23:39:28 INFO nodeHandler::stdlib: Waiting for nodes (Up/Down/Total): 0/1/1 - (still down: n_1_1) 2008-07-20 23:39:29 DEBUG nodeHandler::every: save: fires 2008-07-20 23:39:29 DEBUG nodeHandler::every: save: fires 2008-07-20 23:39:38 DEBUG nodeHandler::every: *: fires 2008-07-20 23:39:38 DEBUG nodeHandler::set::_ALL_: Running each for save 2008-07-20 23:39:38 INFO nodeHandler::stdlib: Waiting for nodes (Up/Down/Total): 0/1/1 - (still down: n_1_1) 2008-07-20 23:39:39 DEBUG nodeHandler::every: save: fires 2008-07-20 23:39:39 DEBUG nodeHandler::every: save: fires 2008-07-20 23:39:39 DEBUG nodeHandler::Communicator: commServer(STDOUT): 'n_1_1 ENROLLED' 2008-07-20 23:39:39 DEBUG nodeHandler::Communicator: Process message 'n_1_1 ENROLLED' 2008-07-20 23:39:39 DEBUG nodeHandler::agentCmd::enrolled: n_1_1: n_1_1 senderId: n_1_1: 2008-07-20 23:39:39 INFO nodeHandler::node::n_1_1: - Saving disk image from node n_1_1 in the file 'node-1-1-2008-07-20-23-39-39.ndz' 2008-07-20 23:39:39 INFO nodeHandler::node::n_1_1: (disk images are located at: 'frisbee:/export/orbit/image/tmp') 2008-07-20 23:39:39 DEBUG nodeHandler::Communicator: Send message: S n_1_1 SAVE_IMAGE frisbee:/export/orbit/image/tmp node-1-1-2008-07-20-23-39-39.ndz 2008-07-20 23:39:39 DEBUG nodeHandler::ExecApp: writing 'S n_1_1 SAVE_IMAGE frisbee:/export/orbit/image/tmp node-1-1-2008-07-20-23-39-39.ndz' to app 'commServer' 2008-07-20 23:39:46 DEBUG nodeHandler::Communicator: commServer(STDOUT): 'n_1_1 APP_EVENT STARTED builtin:save_image' 2008-07-20 23:39:46 DEBUG nodeHandler::Communicator: Process message 'n_1_1 APP_EVENT STARTED builtin:save_image' 2008-07-20 23:39:46 DEBUG nodeHandler::agentCmd::APP_EVENT: STARTED' from 'builtin:save_image' executing on n_1_1: '' 2008-07-20 23:39:46 DEBUG nodeHandler::node::n_1_1: Message for app 'builtin:save_image' - '' 2008-07-20 23:39:48 DEBUG nodeHandler::every: *: fires 2008-07-20 23:39:48 DEBUG nodeHandler::set::_ALL_: Running each for save 2008-07-20 23:39:48 DEBUG nodeHandler::every: *: finishes 2008-07-20 23:39:48 DEBUG nodeHandler::Communicator: commServer(STDOUT): 'n_1_1 APP_EVENT STDOUT builtin:save_image " Slice 3 is unused, NOT SAVING."' 2008-07-20 23:39:48 DEBUG nodeHandler::Communicator: Process message 'n_1_1 APP_EVENT STDOUT builtin:save_image Slice 3 is unused, NOT SAVING.' 2008-07-20 23:39:48 DEBUG nodeHandler::agentCmd::APP_EVENT: STDOUT' from 'builtin:save_image' executing on n_1_1: ' Slice 3 is unused, NOT SAVING.' 2008-07-20 23:39:48 DEBUG nodeHandler::node::n_1_1: Message for app 'builtin:save_image' - ' Slice 3 is unused, NOT SAVING.' 2008-07-20 23:39:48 DEBUG nodeHandler::Communicator: commServer(STDOUT): 'n_1_1 APP_EVENT STDOUT builtin:save_image " Slice 4 is unused, NOT SAVING."' 2008-07-20 23:39:48 DEBUG nodeHandler::Communicator: Process message 'n_1_1 APP_EVENT STDOUT builtin:save_image Slice 4 is unused, NOT SAVING.' 2008-07-20 23:39:48 DEBUG nodeHandler::agentCmd::APP_EVENT: STDOUT' from 'builtin:save_image' executing on n_1_1: ' Slice 4 is unused, NOT SAVING.' 2008-07-20 23:39:48 DEBUG nodeHandler::node::n_1_1: Message for app 'builtin:save_image' - ' Slice 4 is unused, NOT SAVING.' 2008-07-20 23:39:49 DEBUG nodeHandler::every: save: fires 2008-07-20 23:39:49 DEBUG nodeHandler::every: save: fires 2008-07-20 23:39:49 INFO nodeHandler::prop.started: started = "true":String 2008-07-20 23:39:49 INFO nodeHandler::exp: - SAVE_IMAGE process started at: Sun Jul 20 23:39:49 -0400 2008 2008-07-20 23:39:49 INFO nodeHandler::exp: (this may take a while, e.g. 5min+, depending of the size of your image) 2008-07-20 23:39:59 DEBUG nodeHandler::every: save: fires 2008-07-20 23:39:59 DEBUG nodeHandler::every: save: fires 2008-07-20 23:40:09 DEBUG nodeHandler::every: save: fires 2008-07-20 23:40:09 DEBUG nodeHandler::every: save: fires 2008-07-20 23:40:19 DEBUG nodeHandler::every: save: fires 2008-07-20 23:40:19 DEBUG nodeHandler::every: save: fires 2008-07-20 23:40:29 DEBUG nodeHandler::every: save: fires 2008-07-20 23:40:29 DEBUG nodeHandler::every: save: fires 2008-07-20 23:40:39 DEBUG nodeHandler::every: save: fires 2008-07-20 23:40:39 DEBUG nodeHandler::every: save: fires 2008-07-20 23:40:49 DEBUG nodeHandler::every: save: fires 2008-07-20 23:40:49 DEBUG nodeHandler::every: save: fires 2008-07-20 23:40:59 DEBUG nodeHandler::every: save: fires 2008-07-20 23:40:59 DEBUG nodeHandler::every: save: fires 2008-07-20 23:41:09 DEBUG nodeHandler::every: save: fires 2008-07-20 23:41:09 DEBUG nodeHandler::every: save: fires 2008-07-20 23:41:19 DEBUG nodeHandler::every: save: fires 2008-07-20 23:41:19 DEBUG nodeHandler::every: save: fires 2008-07-20 23:41:29 DEBUG nodeHandler::every: save: fires 2008-07-20 23:41:29 DEBUG nodeHandler::every: save: fires 2008-07-20 23:41:39 DEBUG nodeHandler::every: save: fires 2008-07-20 23:41:39 DEBUG nodeHandler::every: save: fires 2008-07-20 23:41:49 DEBUG nodeHandler::every: save: fires 2008-07-20 23:41:49 DEBUG nodeHandler::every: save: fires 2008-07-20 23:41:59 DEBUG nodeHandler::every: save: fires 2008-07-20 23:41:59 DEBUG nodeHandler::every: save: fires 2008-07-20 23:42:09 DEBUG nodeHandler::every: save: fires 2008-07-20 23:42:09 DEBUG nodeHandler::every: save: fires 2008-07-20 23:42:19 DEBUG nodeHandler::every: save: fires 2008-07-20 23:42:19 DEBUG nodeHandler::every: save: fires 2008-07-20 23:42:29 DEBUG nodeHandler::every: save: fires 2008-07-20 23:42:29 DEBUG nodeHandler::every: save: fires 2008-07-20 23:42:39 DEBUG nodeHandler::every: save: fires 2008-07-20 23:42:39 DEBUG nodeHandler::every: save: fires 2008-07-20 23:42:49 DEBUG nodeHandler::every: save: fires 2008-07-20 23:42:49 DEBUG nodeHandler::every: save: fires 2008-07-20 23:42:59 DEBUG nodeHandler::every: save: fires 2008-07-20 23:42:59 DEBUG nodeHandler::every: save: fires 2008-07-20 23:43:09 DEBUG nodeHandler::every: save: fires 2008-07-20 23:43:09 DEBUG nodeHandler::every: save: fires 2008-07-20 23:43:19 DEBUG nodeHandler::every: save: fires 2008-07-20 23:43:19 DEBUG nodeHandler::every: save: fires 2008-07-20 23:43:29 DEBUG nodeHandler::every: save: fires 2008-07-20 23:43:29 DEBUG nodeHandler::every: save: fires 2008-07-20 23:43:39 DEBUG nodeHandler::every: save: fires 2008-07-20 23:43:40 DEBUG nodeHandler::every: save: fires 2008-07-20 23:43:50 DEBUG nodeHandler::every: save: fires 2008-07-20 23:43:50 DEBUG nodeHandler::every: save: fires 2008-07-20 23:44:00 DEBUG nodeHandler::every: save: fires 2008-07-20 23:44:00 DEBUG nodeHandler::every: save: fires 2008-07-20 23:44:10 DEBUG nodeHandler::every: save: fires 2008-07-20 23:44:10 DEBUG nodeHandler::every: save: fires 2008-07-20 23:44:20 DEBUG nodeHandler::every: save: fires 2008-07-20 23:44:20 DEBUG nodeHandler::every: save: fires 2008-07-20 23:44:30 DEBUG nodeHandler::every: save: fires 2008-07-20 23:44:30 DEBUG nodeHandler::every: save: fires 2008-07-20 23:44:40 DEBUG nodeHandler::every: save: fires 2008-07-20 23:44:40 DEBUG nodeHandler::every: save: fires 2008-07-20 23:44:50 DEBUG nodeHandler::every: save: fires 2008-07-20 23:44:50 DEBUG nodeHandler::every: save: fires 2008-07-20 23:45:00 DEBUG nodeHandler::every: save: fires 2008-07-20 23:45:00 DEBUG nodeHandler::every: save: fires 2008-07-20 23:45:10 DEBUG nodeHandler::every: save: fires 2008-07-20 23:45:10 DEBUG nodeHandler::every: save: fires 2008-07-20 23:45:20 DEBUG nodeHandler::every: save: fires 2008-07-20 23:45:20 DEBUG nodeHandler::every: save: fires 2008-07-20 23:45:30 DEBUG nodeHandler::every: save: fires 2008-07-20 23:45:30 DEBUG nodeHandler::every: save: fires 2008-07-20 23:45:40 DEBUG nodeHandler::every: save: fires 2008-07-20 23:45:40 DEBUG nodeHandler::every: save: fires 2008-07-20 23:45:50 DEBUG nodeHandler::every: save: fires 2008-07-20 23:45:50 DEBUG nodeHandler::every: save: fires 2008-07-20 23:46:00 DEBUG nodeHandler::every: save: fires 2008-07-20 23:46:00 DEBUG nodeHandler::every: save: fires 2008-07-20 23:46:10 DEBUG nodeHandler::every: save: fires 2008-07-20 23:46:10 DEBUG nodeHandler::every: save: fires 2008-07-20 23:46:20 DEBUG nodeHandler::every: save: fires 2008-07-20 23:46:20 DEBUG nodeHandler::every: save: fires 2008-07-20 23:46:30 DEBUG nodeHandler::every: save: fires 2008-07-20 23:46:30 DEBUG nodeHandler::every: save: fires 2008-07-20 23:46:40 DEBUG nodeHandler::every: save: fires 2008-07-20 23:46:40 DEBUG nodeHandler::every: save: fires 2008-07-20 23:46:50 DEBUG nodeHandler::every: save: fires 2008-07-20 23:46:50 DEBUG nodeHandler::every: save: fires 2008-07-20 23:47:00 DEBUG nodeHandler::every: save: fires 2008-07-20 23:47:00 DEBUG nodeHandler::every: save: fires 2008-07-20 23:47:10 DEBUG nodeHandler::every: save: fires 2008-07-20 23:47:10 DEBUG nodeHandler::every: save: fires 2008-07-20 23:47:11 DEBUG nodeHandler::Communicator: commServer(STDOUT): 'n_1_1 APP_EVENT STDOUT builtin:save_image "29504112128 input (1528301056 compressed) bytes in 442.078 seconds"' 2008-07-20 23:47:11 DEBUG nodeHandler::Communicator: Process message 'n_1_1 APP_EVENT STDOUT builtin:save_image 29504112128 input (1528301056 compressed) bytes in 442.078 seconds' 2008-07-20 23:47:11 DEBUG nodeHandler::agentCmd::APP_EVENT: STDOUT' from 'builtin:save_image' executing on n_1_1: '29504112128 input (1528301056 compressed) bytes in 442.078 seconds' 2008-07-20 23:47:11 DEBUG nodeHandler::node::n_1_1: Message for app 'builtin:save_image' - '29504112128 input (1528301056 compressed) bytes in 442.078 seconds' 2008-07-20 23:47:11 DEBUG nodeHandler::Communicator: commServer(STDOUT): 'n_1_1 APP_EVENT STDOUT builtin:save_image "Image size: 279969792 bytes"' 2008-07-20 23:47:11 DEBUG nodeHandler::Communicator: Process message 'n_1_1 APP_EVENT STDOUT builtin:save_image Image size: 279969792 bytes' 2008-07-20 23:47:11 DEBUG nodeHandler::agentCmd::APP_EVENT: STDOUT' from 'builtin:save_image' executing on n_1_1: 'Image size: 279969792 bytes' 2008-07-20 23:47:11 DEBUG nodeHandler::node::n_1_1: Message for app 'builtin:save_image' - 'Image size: 279969792 bytes' 2008-07-20 23:47:11 DEBUG nodeHandler::Communicator: commServer(STDOUT): 'n_1_1 APP_EVENT STDOUT builtin:save_image "3.297MB/second compressed"' 2008-07-20 23:47:11 DEBUG nodeHandler::Communicator: Process message 'n_1_1 APP_EVENT STDOUT builtin:save_image 3.297MB/second compressed' 2008-07-20 23:47:11 DEBUG nodeHandler::agentCmd::APP_EVENT: STDOUT' from 'builtin:save_image' executing on n_1_1: '3.297MB/second compressed' 2008-07-20 23:47:11 DEBUG nodeHandler::node::n_1_1: Message for app 'builtin:save_image' - '3.297MB/second compressed' 2008-07-20 23:47:11 DEBUG nodeHandler::Communicator: commServer(STDOUT): 'n_1_1 APP_EVENT DONE.OK builtin:save_image "status: 0"' 2008-07-20 23:47:11 DEBUG nodeHandler::Communicator: Process message 'n_1_1 APP_EVENT DONE.OK builtin:save_image status: 0' 2008-07-20 23:47:11 DEBUG nodeHandler::agentCmd::APP_EVENT: DONE.OK' from 'builtin:save_image' executing on n_1_1: 'status: 0' 2008-07-20 23:47:11 DEBUG nodeHandler::node::n_1_1: Message for app 'builtin:save_image' - 'status: 0' 2008-07-20 23:47:11 DEBUG nodeHandler::NodeBuiltin: Application save_image on TraceState finished successfully 2008-07-20 23:47:20 DEBUG nodeHandler::every: save: fires 2008-07-20 23:47:20 DEBUG nodeHandler::every: save: fires 2008-07-20 23:47:20 INFO nodeHandler::exp: - Saving process finished correctly at: Sun Jul 20 23:47:20 -0400 2008 2008-07-20 23:47:20 DEBUG nodeHandler::set::save: PXE: http://pxe:5022/pxe/clearBootImageNS?domain=sb1&img=&ns=[[1,1]] 2008-07-20 23:47:20 DEBUG nodeHandler::service call: http://pxe:5022/pxe/clearBootImageNS?domain=sb1&img=&ns=[[1,1]] 2008-07-20 23:47:20 INFO nodeHandler::Experiment: DONE! 2008-07-20 23:47:20 DEBUG nodeHandler::ExecApp: writing 'q' to app 'commServer' 2008-07-20 23:47:20 DEBUG nodeHandler::every: save: finishes 2008-07-20 23:47:20 INFO nodeHandler::ExecApp: Application 'commServer' finished 2008-07-20 23:47:20 DEBUG nodeHandler::Communicator: commServer(DONE.OK): 'status: 0' 2008-07-20 23:47:22 INFO nodeHandler::run: Experiment sb1_2008_07_20_23_38_04 finished after 9:19