ORBIT-USER: Grid problem

Thu Aug 23 13:35:02 EDT 2007

I tried to image the whole grid, but most of nodes are either ignored or 
given up.
I was only able to turn on 5 of them. "defTopology('topo_grid_active', 
[[5,7],[4,19],[1,9],[6,16],[7,17]])"
I think same thing happened weeks ago and suddenly the grid went to be 
stable two weeks ago, and it happens again.
Is there a way to reset the grid by indivisual users?

- Sangho

sangho at console.grid:~$ imageNodes4 [[1..20,[1..20]] sangho_dfs1.ndz
Imaging nodes: '[[1..20,[1..20]]' with image 'sangho_dfs1.ndz'
(Domain:  default from hostname)
(Timeout:  800 sec.)
 INFO init: NodeHandler Version 4.2.0 (1272)
 INFO init: Experiment ID: grid_2007_08_23_13_00_53
 INFO ExecApp: Starting application 'commServer': 
/opt/nodehandler4-4.2.0/sbin/commServer --logfile 
/tmp/commServer-grid_2007_08_23_13_00_53.log -d 4 --iface eth1 -e
^CERROR NodeHandler:
ERROR '' when starting NH webserver !
ERROR ExecApp: Application 'commServer' failed (code=2)
ERROR NodeHandler: Possible source of this Error: another NH is already 
running on the same tesbed...

ERROR Communicator: ComServer failed: status: 2
 INFO run: Experiment grid_2007_08_23_13_00_53 finished after 0:7
sangho at console.grid:~$ imageNodes4 [[1..20,1..20]] sangho_dfs1.ndz
Imaging nodes: '[[1..20,1..20]]' with image 'sangho_dfs1.ndz'
(Domain:  default from hostname)
(Timeout:  800 sec.)
 INFO init: NodeHandler Version 4.2.0 (1272)
 INFO init: Experiment ID: grid_2007_08_23_13_01_08
 INFO ExecApp: Starting application 'commServer': 
/opt/nodehandler4-4.2.0/sbin/commServer --logfile 
/tmp/commServer-grid_2007_08_23_13_01_08.log -d 4 --iface eth1 -e
 INFO Experiment: load system:exp:stdlib
 INFO prop.resetDelay: resetDelay = 210:Fixnum
 INFO prop.resetTries: resetTries = 1:Fixnum
 INFO Experiment: load system:exp:imageNode
 INFO prop.nodes: nodes = [[[1..20, 1..20]]]:Array
 INFO prop.image: image = "sangho_dfs1.ndz":String
 INFO prop.pxe: pxe = "1.2.1-omf":String
 INFO prop.domain: domain = nil:NilClass
 INFO prop.timeout: timeout = 800:Fixnum
 WARN -:topo:image: Ignoring missing node '1 at 5'
 WARN -:topo:image: Ignoring missing node '1 at 12'
 WARN -:topo:image: Ignoring missing node '1 at 19'
 WARN -:topo:image: Ignoring missing node '2 at 11'
 WARN -:topo:image: Ignoring missing node '2 at 20'
 WARN -:topo:image: Ignoring missing node '3 at 6'
 WARN -:topo:image: Ignoring missing node '4 at 8'
 WARN -:topo:image: Ignoring missing node '4 at 13'
 WARN -:topo:image: Ignoring missing node '4 at 14'
 WARN -:topo:image: Ignoring missing node '4 at 16'
 WARN -:topo:image: Ignoring missing node '4 at 17'
 WARN -:topo:image: Ignoring missing node '4 at 20'
 WARN -:topo:image: Ignoring missing node '5 at 4'
 WARN -:topo:image: Ignoring missing node '5 at 5'
 WARN -:topo:image: Ignoring missing node '5 at 6'
 WARN -:topo:image: Ignoring missing node '5 at 10'
 WARN -:topo:image: Ignoring missing node '5 at 12'
 WARN -:topo:image: Ignoring missing node '5 at 14'
 WARN -:topo:image: Ignoring missing node '5 at 17'
 WARN -:topo:image: Ignoring missing node '6 at 1'
 WARN -:topo:image: Ignoring missing node '6 at 3'
 WARN -:topo:image: Ignoring missing node '6 at 5'
 WARN -:topo:image: Ignoring missing node '6 at 8'
 WARN -:topo:image: Ignoring missing node '6 at 10'
 WARN -:topo:image: Ignoring missing node '6 at 11'
 WARN -:topo:image: Ignoring missing node '6 at 12'
 WARN -:topo:image: Ignoring missing node '6 at 14'
 WARN -:topo:image: Ignoring missing node '6 at 15'
 WARN -:topo:image: Ignoring missing node '6 at 17'
 WARN -:topo:image: Ignoring missing node '6 at 20'
 WARN -:topo:image: Ignoring missing node '7 at 1'
 WARN -:topo:image: Ignoring missing node '7 at 4'
 WARN -:topo:image: Ignoring missing node '7 at 10'
 WARN -:topo:image: Ignoring missing node '7 at 14'
 WARN -:topo:image: Ignoring missing node '7 at 15'
 WARN -:topo:image: Ignoring missing node '7 at 19'
 WARN -:topo:image: Ignoring missing node '8 at 2'
 WARN -:topo:image: Ignoring missing node '8 at 5'
 WARN -:topo:image: Ignoring missing node '8 at 10'
 WARN -:topo:image: Ignoring missing node '8 at 11'
 WARN -:topo:image: Ignoring missing node '8 at 12'
 WARN -:topo:image: Ignoring missing node '8 at 15'
 WARN -:topo:image: Ignoring missing node '8 at 16'
 WARN -:topo:image: Ignoring missing node '8 at 19'
 WARN -:topo:image: Ignoring missing node '8 at 20'
 WARN -:topo:image: Ignoring missing node '9 at 3'
 WARN -:topo:image: Ignoring missing node '9 at 7'
 WARN -:topo:image: Ignoring missing node '9 at 10'
 WARN -:topo:image: Ignoring missing node '9 at 11'
 WARN -:topo:image: Ignoring missing node '9 at 13'
 WARN -:topo:image: Ignoring missing node '9 at 15'
 WARN -:topo:image: Ignoring missing node '9 at 16'
 WARN -:topo:image: Ignoring missing node '9 at 18'
 WARN -:topo:image: Ignoring missing node '9 at 19'
 WARN -:topo:image: Ignoring missing node '9 at 20'
 WARN -:topo:image: Ignoring missing node '10 at 1'
 WARN -:topo:image: Ignoring missing node '10 at 2'
 WARN -:topo:image: Ignoring missing node '10 at 5'
 WARN -:topo:image: Ignoring missing node '10 at 7'
 WARN -:topo:image: Ignoring missing node '10 at 11'
 WARN -:topo:image: Ignoring missing node '10 at 12'
 WARN -:topo:image: Ignoring missing node '10 at 15'
 WARN -:topo:image: Ignoring missing node '11 at 2'
 WARN -:topo:image: Ignoring missing node '11 at 6'
 WARN -:topo:image: Ignoring missing node '11 at 11'
 WARN -:topo:image: Ignoring missing node '11 at 18'
 WARN -:topo:image: Ignoring missing node '12 at 2'
 WARN -:topo:image: Ignoring missing node '12 at 7'
 WARN -:topo:image: Ignoring missing node '12 at 10'
 WARN -:topo:image: Ignoring missing node '12 at 12'
 WARN -:topo:image: Ignoring missing node '12 at 15'
 WARN -:topo:image: Ignoring missing node '12 at 16'
 WARN -:topo:image: Ignoring missing node '12 at 18'
 WARN -:topo:image: Ignoring missing node '13 at 2'
 WARN -:topo:image: Ignoring missing node '13 at 5'
 WARN -:topo:image: Ignoring missing node '13 at 8'
 WARN -:topo:image: Ignoring missing node '13 at 13'
 WARN -:topo:image: Ignoring missing node '14 at 4'
 WARN -:topo:image: Ignoring missing node '14 at 11'
 WARN -:topo:image: Ignoring missing node '14 at 13'
 WARN -:topo:image: Ignoring missing node '14 at 14'
 WARN -:topo:image: Ignoring missing node '15 at 1'
 WARN -:topo:image: Ignoring missing node '15 at 3'
 WARN -:topo:image: Ignoring missing node '15 at 6'
 WARN -:topo:image: Ignoring missing node '15 at 7'
 WARN -:topo:image: Ignoring missing node '15 at 10'
 WARN -:topo:image: Ignoring missing node '15 at 12'
 WARN -:topo:image: Ignoring missing node '15 at 15'
 WARN -:topo:image: Ignoring missing node '16 at 13'
 WARN -:topo:image: Ignoring missing node '16 at 17'
 WARN -:topo:image: Ignoring missing node '17 at 2'
 WARN -:topo:image: Ignoring missing node '17 at 3'
 WARN -:topo:image: Ignoring missing node '17 at 9'
 WARN -:topo:image: Ignoring missing node '17 at 12'
 WARN -:topo:image: Ignoring missing node '17 at 20'
 WARN -:topo:image: Ignoring missing node '18 at 5'
 WARN -:topo:image: Ignoring missing node '18 at 7'
 WARN -:topo:image: Ignoring missing node '18 at 15'
 WARN -:topo:image: Ignoring missing node '18 at 17'
 WARN -:topo:image: Ignoring missing node '18 at 18'
 WARN -:topo:image: Ignoring missing node '18 at 20'
 WARN -:topo:image: Ignoring missing node '19 at 4'
 WARN -:topo:image: Ignoring missing node '19 at 9'
 WARN -:topo:image: Ignoring missing node '19 at 13'
 WARN -:topo:image: Ignoring missing node '19 at 15'
 WARN -:topo:image: Ignoring missing node '19 at 17'
 WARN -:topo:image: Ignoring missing node '20 at 10'
 WARN -:topo:image: Ignoring missing node '20 at 13'
 WARN -:topo:image: Ignoring missing node '20 at 15'
 WARN -:topo:image: Ignoring missing node '20 at 17'
 INFO stdlib: Waiting for nodes (Up/Down/Total): 0/290/290 - (still down: 
n_20_16,n_7_18,n_1_6)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 0/290/290 - (still down: 
n_20_16,n_7_18,n_1_6)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 0/290/290 - (still down: 
n_20_16,n_7_18,n_1_6)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 0/290/290 - (still down: 
n_20_16,n_7_18,n_1_6)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 0/290/290 - (still down: 
n_20_16,n_7_18,n_1_6)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 0/290/290 - (still down: 
n_20_16,n_7_18,n_1_6)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 0/290/290 - (still down: 
n_20_16,n_7_18,n_1_6)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 0/290/290 - (still down: 
n_20_16,n_7_18,n_1_6)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 0/290/290 - (still down: 
n_20_16,n_7_18,n_1_6)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 0/290/290 - (still down: 
n_20_16,n_7_18,n_1_6)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 0/290/290 - (still down: 
n_20_16,n_7_18,n_1_6)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 0/290/290 - (still down: 
n_20_16,n_7_18,n_1_6)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 0/290/290 - (still down: 
n_20_16,n_7_18,n_1_6)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 1/289/290 - (still down: 
n_20_16,n_7_18,n_1_6)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 3/287/290 - (still down: 
n_20_16,n_7_18,n_1_6)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 5/285/290 - (still down: 
n_20_16,n_7_18,n_1_6)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 6/284/290 - (still down: 
n_20_16,n_7_18,n_1_6)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 8/282/290 - (still down: 
n_20_16,n_7_18,n_2_9)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 10/280/290 - (still down: 
n_20_16,n_7_18,n_2_9)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 13/277/290 - (still down: 
n_20_16,n_7_18,n_2_9)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 15/275/290 - (still down: 
n_20_16,n_7_18,n_2_9)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 16/274/290 - (still down: 
n_20_16,n_7_18,n_2_9)
 INFO stdlib: Resetting node n_20_16
 INFO stdlib: Resetting node n_7_18
 INFO stdlib: Resetting node n_2_9
 INFO stdlib: Resetting node n_2_6
 INFO stdlib: Resetting node n_11_17
 INFO stdlib: Resetting node n_14_1
 INFO stdlib: Resetting node n_12_6
 INFO stdlib: Resetting node n_5_7
 INFO stdlib: Resetting node n_19_18
 INFO stdlib: Resetting node n_12_1
 INFO stdlib: Resetting node n_5_1
 INFO stdlib: Resetting node n_18_14
 INFO stdlib: Resetting node n_5_11
 INFO stdlib: Resetting node n_8_9
 INFO stdlib: Resetting node n_8_13
 INFO stdlib: Resetting node n_1_4
 INFO stdlib: Resetting node n_18_12
 INFO stdlib: Resetting node n_1_17
 INFO stdlib: Resetting node n_1_13
 INFO stdlib: Resetting node n_9_5
 INFO stdlib: Resetting node n_16_7
 INFO stdlib: Resetting node n_11_13
 INFO stdlib: Resetting node n_7_12
 INFO stdlib: Resetting node n_9_9
 INFO stdlib: Resetting node n_14_6
 INFO stdlib: Resetting node n_2_19
 INFO stdlib: Resetting node n_17_5
 INFO stdlib: Resetting node n_4_3
 INFO stdlib: Resetting node n_6_6
 INFO stdlib: Resetting node n_13_4
 INFO stdlib: Resetting node n_13_7
 INFO stdlib: Resetting node n_20_11
 INFO stdlib: Resetting node n_11_10
 INFO stdlib: Resetting node n_4_19
 INFO stdlib: Resetting node n_11_16
 INFO stdlib: Resetting node n_11_20
 INFO stdlib: Resetting node n_20_8
 INFO stdlib: Resetting node n_20_7
 INFO stdlib: Resetting node n_20_6
 INFO stdlib: Resetting node n_1_10
 INFO stdlib: Resetting node n_3_16
 INFO stdlib: Resetting node n_15_13
 INFO stdlib: Resetting node n_16_20
 INFO stdlib: Resetting node n_12_20
 INFO stdlib: Resetting node n_13_1
 INFO stdlib: Resetting node n_3_1
 INFO stdlib: Resetting node n_1_15
 INFO stdlib: Resetting node n_15_18
 INFO stdlib: Resetting node n_1_8
 INFO stdlib: Resetting node n_19_3
 INFO stdlib: Resetting node n_13_18
 INFO stdlib: Resetting node n_2_10
 INFO stdlib: Resetting node n_11_12
 INFO stdlib: Resetting node n_10_8
 INFO stdlib: Resetting node n_12_11
 INFO stdlib: Resetting node n_5_15
 INFO stdlib: Resetting node n_2_4
 INFO stdlib: Resetting node n_16_6
 INFO stdlib: Resetting node n_1_18
 INFO stdlib: Resetting node n_1_3
 INFO stdlib: Resetting node n_3_17
 INFO stdlib: Resetting node n_19_10
 INFO stdlib: Resetting node n_19_8
 INFO stdlib: Resetting node n_10_3
 INFO stdlib: Resetting node n_14_2
 INFO stdlib: Resetting node n_16_5
 INFO stdlib: Resetting node n_1_11
 INFO stdlib: Resetting node n_20_12
 INFO stdlib: Resetting node n_14_9
 INFO stdlib: Resetting node n_10_19
 INFO stdlib: Resetting node n_1_9
 INFO stdlib: Resetting node n_13_20
 INFO stdlib: Resetting node n_15_16
 INFO stdlib: Resetting node n_18_16
 INFO stdlib: Resetting node n_17_8
 INFO stdlib: Resetting node n_11_8
 INFO stdlib: Resetting node n_4_15
 INFO stdlib: Resetting node n_4_18
 INFO stdlib: Resetting node n_2_12
 INFO stdlib: Resetting node n_19_19
 INFO stdlib: Resetting node n_17_17
 INFO stdlib: Resetting node n_4_9
 INFO stdlib: Resetting node n_13_19
 INFO stdlib: Resetting node n_14_20
 INFO stdlib: Resetting node n_19_11
 INFO stdlib: Resetting node n_18_13
 INFO stdlib: Resetting node n_17_18
 INFO stdlib: Resetting node n_1_20
 INFO stdlib: Resetting node n_16_14
 INFO stdlib: Resetting node n_4_7
 INFO stdlib: Resetting node n_14_17
 INFO stdlib: Resetting node n_8_7
 INFO stdlib: Resetting node n_2_13
 INFO stdlib: Resetting node n_15_19
 INFO stdlib: Resetting node n_20_18
 INFO stdlib: Resetting node n_19_16
 INFO stdlib: Resetting node n_4_11
 INFO stdlib: Resetting node n_2_18
 INFO stdlib: Resetting node n_15_8
 INFO stdlib: Resetting node n_2_14
 INFO stdlib: Resetting node n_13_14
 INFO stdlib: Resetting node n_20_14
 INFO stdlib: Resetting node n_18_19
 INFO stdlib: Resetting node n_18_9
 INFO stdlib: Resetting node n_16_2
 INFO stdlib: Resetting node n_20_3
 INFO stdlib: Resetting node n_13_3
 INFO stdlib: Resetting node n_20_20
 INFO stdlib: Resetting node n_18_4
 INFO stdlib: Resetting node n_17_13
 INFO stdlib: Resetting node n_16_11
 INFO stdlib: Resetting node n_7_20
 INFO stdlib: Resetting node n_3_14
 INFO stdlib: Resetting node n_16_12
 INFO stdlib: Resetting node n_6_9
 INFO stdlib: Resetting node n_1_16
 INFO stdlib: Resetting node n_13_6
 INFO stdlib: Resetting node n_12_4
 INFO stdlib: Resetting node n_14_8
 INFO stdlib: Resetting node n_16_16
 INFO stdlib: Resetting node n_5_8
 INFO stdlib: Resetting node n_13_10
 INFO stdlib: Resetting node n_14_15
 INFO stdlib: Resetting node n_7_11
 INFO stdlib: Resetting node n_16_1
 INFO stdlib: Resetting node n_2_5
 INFO stdlib: Resetting node n_20_2
 INFO stdlib: Resetting node n_19_5
 INFO stdlib: Resetting node n_12_8
 INFO stdlib: Resetting node n_18_6
 INFO stdlib: Resetting node n_6_16
 INFO stdlib: Resetting node n_16_8
 INFO stdlib: Resetting node n_19_12
 INFO stdlib: Resetting node n_19_7
 INFO stdlib: Resetting node n_4_2
 INFO stdlib: Resetting node n_13_12
 INFO stdlib: Resetting node n_15_14
 INFO stdlib: Resetting node n_2_7
 INFO stdlib: Resetting node n_14_3
 INFO stdlib: Resetting node n_15_9
 INFO stdlib: Resetting node n_8_18
 INFO stdlib: Resetting node n_2_2
 INFO stdlib: Resetting node n_3_12
 INFO stdlib: Resetting node n_7_7
 INFO stdlib: Resetting node n_10_17
 INFO stdlib: Resetting node n_3_7
 INFO stdlib: Resetting node n_12_5
 INFO stdlib: Resetting node n_17_14
 INFO stdlib: Resetting node n_10_9
 INFO stdlib: Resetting node n_11_9
 INFO stdlib: Resetting node n_2_8
 INFO stdlib: Resetting node n_3_13
 INFO stdlib: Resetting node n_3_19
 INFO stdlib: Resetting node n_11_14
 INFO stdlib: Resetting node n_20_5
 INFO stdlib: Resetting node n_20_4
 INFO stdlib: Resetting node n_18_1
 INFO stdlib: Resetting node n_5_19
 INFO stdlib: Resetting node n_2_15
 INFO stdlib: Resetting node n_4_5
 INFO stdlib: Resetting node n_6_13
 INFO stdlib: Resetting node n_14_10
 INFO stdlib: Resetting node n_2_1
 INFO stdlib: Resetting node n_12_14
 INFO stdlib: Resetting node n_9_14
 INFO stdlib: Resetting node n_9_4
 INFO stdlib: Resetting node n_3_3
 INFO stdlib: Resetting node n_10_13
 INFO stdlib: Resetting node n_3_2
 INFO stdlib: Resetting node n_11_5
 INFO stdlib: Resetting node n_16_10
 INFO stdlib: Resetting node n_7_16
 INFO stdlib: Resetting node n_9_17
 INFO stdlib: Resetting node n_9_6
 INFO stdlib: Resetting node n_18_10
 INFO stdlib: Resetting node n_17_6
 INFO stdlib: Resetting node n_3_10
 INFO stdlib: Resetting node n_16_3
 INFO stdlib: Resetting node n_10_10
 INFO stdlib: Resetting node n_11_7
 INFO stdlib: Resetting node n_3_11
 INFO stdlib: Resetting node n_4_10
 INFO stdlib: Resetting node n_14_18
 INFO stdlib: Resetting node n_16_15
 INFO stdlib: Resetting node n_3_5
 INFO stdlib: Resetting node n_12_19
 INFO stdlib: Resetting node n_17_7
 INFO stdlib: Resetting node n_10_16
 INFO stdlib: Resetting node n_15_20
 INFO stdlib: Resetting node n_6_4
 INFO stdlib: Resetting node n_11_3
 INFO stdlib: Resetting node n_15_4
 INFO stdlib: Resetting node n_12_3
 INFO stdlib: Resetting node n_19_1
 INFO stdlib: Resetting node n_9_2
 INFO stdlib: Resetting node n_6_19
 INFO stdlib: Resetting node n_11_4
 INFO stdlib: Resetting node n_10_14
 INFO stdlib: Resetting node n_13_11
 INFO stdlib: Resetting node n_3_4
 INFO stdlib: Resetting node n_3_15
 INFO stdlib: Resetting node n_18_11
 INFO stdlib: Resetting node n_17_1
 INFO stdlib: Resetting node n_4_4
 INFO stdlib: Resetting node n_15_2
 INFO stdlib: Resetting node n_19_14
 INFO stdlib: Resetting node n_18_2
 INFO stdlib: Resetting node n_17_10
 INFO stdlib: Resetting node n_7_13
 INFO stdlib: Resetting node n_12_13
 INFO stdlib: Resetting node n_15_5
 INFO stdlib: Resetting node n_11_15
 INFO stdlib: Resetting node n_6_7
 INFO stdlib: Resetting node n_15_17
 INFO stdlib: Resetting node n_4_6
 INFO stdlib: Resetting node n_20_1
 INFO stdlib: Resetting node n_17_11
 INFO stdlib: Resetting node n_13_17
 INFO stdlib: Resetting node n_10_4
 INFO stdlib: Resetting node n_17_16
 INFO stdlib: Resetting node n_16_18
 INFO stdlib: Resetting node n_10_6
 INFO stdlib: Resetting node n_8_4
 INFO stdlib: Resetting node n_1_2
 INFO stdlib: Resetting node n_3_8
 INFO stdlib: Resetting node n_17_4
 INFO stdlib: Resetting node n_5_3
 INFO stdlib: Resetting node n_13_15
 INFO stdlib: Resetting node n_20_9
 INFO stdlib: Resetting node n_19_6
 INFO stdlib: Resetting node n_17_19
 INFO stdlib: Resetting node n_5_2
 INFO stdlib: Resetting node n_9_8
 INFO stdlib: Resetting node n_12_9
 INFO stdlib: Resetting node n_5_18
 INFO stdlib: Resetting node n_5_13
 INFO stdlib: Resetting node n_7_8
 INFO stdlib: Resetting node n_16_9
 INFO stdlib: Resetting node n_6_18
 INFO stdlib: Resetting node n_14_19
 INFO stdlib: Resetting node n_16_4
 INFO stdlib: Resetting node n_8_14
 INFO stdlib: Resetting node n_3_9
 INFO stdlib: Resetting node n_2_16
 INFO stdlib: Resetting node n_11_1
 INFO stdlib: Resetting node n_14_5
 INFO stdlib: Resetting node n_9_12
 INFO stdlib: Resetting node n_19_2
 INFO stdlib: Resetting node n_17_15
 INFO stdlib: Resetting node n_16_19
 INFO stdlib: Resetting node n_3_18
 INFO stdlib: Resetting node n_2_3
 INFO stdlib: Resetting node n_19_20
 INFO stdlib: Resetting node n_8_6
 INFO stdlib: Resetting node n_14_12
 INFO stdlib: Resetting node n_8_1
 INFO stdlib: Resetting node n_3_20
 INFO stdlib: Resetting node n_14_7
 INFO stdlib: Resetting node n_20_19
 INFO stdlib: Resetting node n_11_19
 INFO stdlib: Resetting node n_4_1
 INFO stdlib: Resetting node n_18_3
 INFO stdlib: Resetting node n_2_17
 INFO stdlib: Resetting node n_8_8
 INFO stdlib: Resetting node n_10_18
 INFO stdlib: Resetting node n_18_8
 INFO stdlib: Resetting node n_15_11
 INFO stdlib: Resetting node n_13_16
 INFO stdlib: Resetting node n_14_16
 INFO stdlib: Resetting node n_13_9
 INFO stdlib: Resetting node n_12_17
 INFO stdlib: Waiting for nodes (Up/Down/Total): 19/271/290 - (still down: 
n_20_16,n_7_18,n_2_9)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 19/271/290 - (still down: 
n_20_16,n_7_18,n_2_9)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 19/271/290 - (still down: 
n_20_16,n_7_18,n_2_9)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 19/271/290 - (still down: 
n_20_16,n_7_18,n_2_9)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 19/271/290 - (still down: 
n_20_16,n_7_18,n_2_9)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 19/271/290 - (still down: 
n_20_16,n_7_18,n_2_9)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 19/271/290 - (still down: 
n_20_16,n_7_18,n_2_9)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 19/271/290 - (still down: 
n_20_16,n_7_18,n_2_9)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 19/271/290 - (still down: 
n_20_16,n_7_18,n_2_9)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 19/271/290 - (still down: 
n_20_16,n_7_18,n_2_9)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 19/271/290 - (still down: 
n_20_16,n_7_18,n_2_9)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 19/271/290 - (still down: 
n_20_16,n_7_18,n_2_9)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 19/271/290 - (still down: 
n_20_16,n_7_18,n_2_9)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 21/269/290 - (still down: 
n_20_16,n_7_18,n_2_9)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 24/266/290 - (still down: 
n_20_16,n_7_18,n_2_9)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 25/265/290 - (still down: 
n_20_16,n_7_18,n_2_9)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 27/263/290 - (still down: 
n_20_16,n_7_18,n_2_9)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 29/261/290 - (still down: 
n_20_16,n_7_18,n_2_9)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 30/260/290 - (still down: 
n_20_16,n_7_18,n_2_9)
 INFO stdlib: Waiting for nodes (Up/Down/Total): 31/259/290 - (still down: 
n_20_16,n_7_18,n_2_9)
 WARN stdlib: Giving up on node n_20_16
 WARN stdlib: Giving up on node n_7_18
 WARN stdlib: Giving up on node n_2_9
 WARN stdlib: Giving up on node n_2_6
 WARN stdlib: Giving up on node n_11_17
 WARN stdlib: Giving up on node n_14_1
 WARN stdlib: Giving up on node n_12_6
 WARN stdlib: Giving up on node n_19_18
 WARN stdlib: Giving up on node n_12_1
 WARN stdlib: Giving up on node n_18_14
 WARN stdlib: Giving up on node n_5_11
 WARN stdlib: Giving up on node n_8_9
 WARN stdlib: Giving up on node n_1_4
 WARN stdlib: Giving up on node n_18_12
 WARN stdlib: Giving up on node n_9_5
 WARN stdlib: Giving up on node n_16_7
 WARN stdlib: Giving up on node n_11_13
 WARN stdlib: Giving up on node n_7_12
 WARN stdlib: Giving up on node n_9_9
 WARN stdlib: Giving up on node n_14_6
 WARN stdlib: Giving up on node n_2_19
 WARN stdlib: Giving up on node n_17_5
 WARN stdlib: Giving up on node n_4_3
 WARN stdlib: Giving up on node n_13_4
 WARN stdlib: Giving up on node n_13_7
 WARN stdlib: Giving up on node n_20_11
 WARN stdlib: Giving up on node n_11_10
 WARN stdlib: Giving up on node n_11_16
 WARN stdlib: Giving up on node n_11_20
 WARN stdlib: Giving up on node n_20_8
 WARN stdlib: Giving up on node n_20_7
 WARN stdlib: Giving up on node n_20_6
 WARN stdlib: Giving up on node n_1_10
 WARN stdlib: Giving up on node n_3_16
 WARN stdlib: Giving up on node n_15_13
 WARN stdlib: Giving up on node n_16_20
 WARN stdlib: Giving up on node n_12_20
 WARN stdlib: Giving up on node n_13_1
 WARN stdlib: Giving up on node n_3_1
 WARN stdlib: Giving up on node n_15_18
 WARN stdlib: Giving up on node n_19_3
 WARN stdlib: Giving up on node n_13_18
 WARN stdlib: Giving up on node n_2_10
 WARN stdlib: Giving up on node n_11_12
 WARN stdlib: Giving up on node n_12_11
 WARN stdlib: Giving up on node n_2_4
 WARN stdlib: Giving up on node n_16_6
 WARN stdlib: Giving up on node n_1_18
 WARN stdlib: Giving up on node n_1_3
 WARN stdlib: Giving up on node n_3_17
 WARN stdlib: Giving up on node n_19_10
 WARN stdlib: Giving up on node n_19_8
 WARN stdlib: Giving up on node n_10_3
 WARN stdlib: Giving up on node n_14_2
 WARN stdlib: Giving up on node n_16_5
 WARN stdlib: Giving up on node n_1_11
 WARN stdlib: Giving up on node n_20_12
 WARN stdlib: Giving up on node n_14_9
 WARN stdlib: Giving up on node n_10_19
 WARN stdlib: Giving up on node n_13_20
 WARN stdlib: Giving up on node n_15_16
 WARN stdlib: Giving up on node n_18_16
 WARN stdlib: Giving up on node n_17_8
 WARN stdlib: Giving up on node n_11_8
 WARN stdlib: Giving up on node n_4_15
 WARN stdlib: Giving up on node n_4_18
 WARN stdlib: Giving up on node n_2_12
 WARN stdlib: Giving up on node n_19_19
 WARN stdlib: Giving up on node n_17_17
 WARN stdlib: Giving up on node n_13_19
 WARN stdlib: Giving up on node n_14_20
 WARN stdlib: Giving up on node n_19_11
 WARN stdlib: Giving up on node n_18_13
 WARN stdlib: Giving up on node n_17_18
 WARN stdlib: Giving up on node n_1_20
 WARN stdlib: Giving up on node n_16_14
 WARN stdlib: Giving up on node n_4_7
 WARN stdlib: Giving up on node n_14_17
 WARN stdlib: Giving up on node n_8_7
 WARN stdlib: Giving up on node n_2_13
 WARN stdlib: Giving up on node n_15_19
 WARN stdlib: Giving up on node n_20_18
 WARN stdlib: Giving up on node n_19_16
 WARN stdlib: Giving up on node n_4_11
 WARN stdlib: Giving up on node n_2_18
 WARN stdlib: Giving up on node n_15_8
 WARN stdlib: Giving up on node n_2_14
 WARN stdlib: Giving up on node n_13_14
 WARN stdlib: Giving up on node n_20_14
 WARN stdlib: Giving up on node n_18_19
 WARN stdlib: Giving up on node n_18_9
 WARN stdlib: Giving up on node n_16_2
 WARN stdlib: Giving up on node n_20_3
 WARN stdlib: Giving up on node n_13_3
 WARN stdlib: Giving up on node n_20_20
 WARN stdlib: Giving up on node n_18_4
 WARN stdlib: Giving up on node n_17_13
 INFO stdlib: Waiting for nodes (Up/Down/Total): 35/255/290 - (still down: 
n_20_16,n_7_18,n_2_9)
 WARN stdlib: Giving up on node n_16_11
 WARN stdlib: Giving up on node n_3_14
 WARN stdlib: Giving up on node n_16_12
 WARN stdlib: Giving up on node n_6_9
 WARN stdlib: Giving up on node n_13_6
 WARN stdlib: Giving up on node n_12_4
 WARN stdlib: Giving up on node n_14_8
 WARN stdlib: Giving up on node n_16_16
 WARN stdlib: Giving up on node n_5_8
 WARN stdlib: Giving up on node n_13_10
 WARN stdlib: Giving up on node n_14_15
 WARN stdlib: Giving up on node n_7_11
 WARN stdlib: Giving up on node n_16_1
 WARN stdlib: Giving up on node n_2_5
 WARN stdlib: Giving up on node n_20_2
 WARN stdlib: Giving up on node n_19_5
 WARN stdlib: Giving up on node n_12_8
 WARN stdlib: Giving up on node n_18_6
 WARN stdlib: Giving up on node n_16_8
 WARN stdlib: Giving up on node n_19_12
 WARN stdlib: Giving up on node n_19_7
 WARN stdlib: Giving up on node n_4_2
 WARN stdlib: Giving up on node n_13_12
 WARN stdlib: Giving up on node n_15_14
 WARN stdlib: Giving up on node n_2_7
 WARN stdlib: Giving up on node n_14_3
 WARN stdlib: Giving up on node n_15_9
 WARN stdlib: Giving up on node n_8_18
 WARN stdlib: Giving up on node n_2_2
 WARN stdlib: Giving up on node n_3_12
 WARN stdlib: Giving up on node n_7_7
 WARN stdlib: Giving up on node n_10_17
 WARN stdlib: Giving up on node n_3_7
 WARN stdlib: Giving up on node n_12_5
 WARN stdlib: Giving up on node n_17_14
 WARN stdlib: Giving up on node n_10_9
 WARN stdlib: Giving up on node n_11_9
 WARN stdlib: Giving up on node n_2_8
 WARN stdlib: Giving up on node n_3_13
 WARN stdlib: Giving up on node n_3_19
 WARN stdlib: Giving up on node n_11_14
 WARN stdlib: Giving up on node n_20_5
 WARN stdlib: Giving up on node n_20_4
 WARN stdlib: Giving up on node n_18_1
 WARN stdlib: Giving up on node n_5_19
 WARN stdlib: Giving up on node n_2_15
 WARN stdlib: Giving up on node n_4_5
 WARN stdlib: Giving up on node n_6_13
 WARN stdlib: Giving up on node n_14_10
 WARN stdlib: Giving up on node n_2_1
 WARN stdlib: Giving up on node n_12_14
 WARN stdlib: Giving up on node n_9_14
 WARN stdlib: Giving up on node n_9_4
 WARN stdlib: Giving up on node n_3_3
 WARN stdlib: Giving up on node n_10_13
 WARN stdlib: Giving up on node n_3_2
 WARN stdlib: Giving up on node n_11_5
 WARN stdlib: Giving up on node n_16_10
 WARN stdlib: Giving up on node n_7_16
 WARN stdlib: Giving up on node n_9_17
 WARN stdlib: Giving up on node n_9_6
 WARN stdlib: Giving up on node n_18_10
 WARN stdlib: Giving up on node n_17_6
 WARN stdlib: Giving up on node n_3_10
 WARN stdlib: Giving up on node n_16_3
 WARN stdlib: Giving up on node n_10_10
 WARN stdlib: Giving up on node n_11_7
 WARN stdlib: Giving up on node n_3_11
 WARN stdlib: Giving up on node n_4_10
 WARN stdlib: Giving up on node n_14_18
 WARN stdlib: Giving up on node n_16_15
 WARN stdlib: Giving up on node n_3_5
 WARN stdlib: Giving up on node n_12_19
 WARN stdlib: Giving up on node n_17_7
 WARN stdlib: Giving up on node n_10_16
 WARN stdlib: Giving up on node n_15_20
 WARN stdlib: Giving up on node n_6_4
 WARN stdlib: Giving up on node n_11_3
 WARN stdlib: Giving up on node n_15_4
 WARN stdlib: Giving up on node n_12_3
 WARN stdlib: Giving up on node n_19_1
 WARN stdlib: Giving up on node n_9_2
 WARN stdlib: Giving up on node n_6_19
 WARN stdlib: Giving up on node n_11_4
 WARN stdlib: Giving up on node n_10_14
 WARN stdlib: Giving up on node n_13_11
 WARN stdlib: Giving up on node n_3_4
 WARN stdlib: Giving up on node n_3_15
 WARN stdlib: Giving up on node n_18_11
 WARN stdlib: Giving up on node n_17_1
 WARN stdlib: Giving up on node n_4_4
 WARN stdlib: Giving up on node n_15_2
 WARN stdlib: Giving up on node n_19_14
 WARN stdlib: Giving up on node n_18_2
 WARN stdlib: Giving up on node n_17_10
 WARN stdlib: Giving up on node n_7_13
 WARN stdlib: Giving up on node n_12_13
 WARN stdlib: Giving up on node n_15_5
 WARN stdlib: Giving up on node n_11_15
 WARN stdlib: Giving up on node n_6_7
 WARN stdlib: Giving up on node n_15_17
 WARN stdlib: Giving up on node n_4_6
 WARN stdlib: Giving up on node n_20_1
 WARN stdlib: Giving up on node n_17_11
 WARN stdlib: Giving up on node n_13_17
 WARN stdlib: Giving up on node n_10_4
 WARN stdlib: Giving up on node n_17_16
 WARN stdlib: Giving up on node n_16_18
 WARN stdlib: Giving up on node n_10_6
 WARN stdlib: Giving up on node n_8_4
 WARN stdlib: Giving up on node n_1_2
 WARN stdlib: Giving up on node n_3_8
 WARN stdlib: Giving up on node n_17_4
 WARN stdlib: Giving up on node n_5_3
 WARN stdlib: Giving up on node n_13_15
 WARN stdlib: Giving up on node n_20_9
 WARN stdlib: Giving up on node n_19_6
 WARN stdlib: Giving up on node n_17_19
 WARN stdlib: Giving up on node n_5_2
 WARN stdlib: Giving up on node n_9_8
 WARN stdlib: Giving up on node n_12_9
 WARN stdlib: Giving up on node n_5_18
 WARN stdlib: Giving up on node n_5_13
 WARN stdlib: Giving up on node n_7_8
 WARN stdlib: Giving up on node n_16_9
 WARN stdlib: Giving up on node n_6_18
 WARN stdlib: Giving up on node n_14_19
 WARN stdlib: Giving up on node n_16_4
 WARN stdlib: Giving up on node n_8_14
 WARN stdlib: Giving up on node n_3_9
 WARN stdlib: Giving up on node n_2_16
 WARN stdlib: Giving up on node n_11_1
 WARN stdlib: Giving up on node n_14_5
 WARN stdlib: Giving up on node n_9_12
 WARN stdlib: Giving up on node n_19_2
 WARN stdlib: Giving up on node n_17_15
 WARN stdlib: Giving up on node n_16_19
 WARN stdlib: Giving up on node n_3_18
 WARN stdlib: Giving up on node n_2_3
 WARN stdlib: Giving up on node n_19_20
 WARN stdlib: Giving up on node n_8_6
 WARN stdlib: Giving up on node n_14_12
 WARN stdlib: Giving up on node n_8_1
 WARN stdlib: Giving up on node n_3_20
 WARN stdlib: Giving up on node n_14_7
 WARN stdlib: Giving up on node n_20_19
 WARN stdlib: Giving up on node n_11_19
 WARN stdlib: Giving up on node n_4_1
 WARN stdlib: Giving up on node n_18_3
 WARN stdlib: Giving up on node n_2_17
 WARN stdlib: Giving up on node n_8_8
 WARN stdlib: Giving up on node n_10_18
 WARN stdlib: Giving up on node n_18_8
 WARN stdlib: Giving up on node n_15_11
 WARN stdlib: Giving up on node n_13_16
 WARN stdlib: Giving up on node n_14_16
 WARN stdlib: Giving up on node n_13_9
 WARN stdlib: Giving up on node n_12_17
 INFO stdlib: Waiting for nodes (Up/Down/Total): 35/158/193 - (still down: 
n_16_11,n_3_14,n_16_12)
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 580 
sec.
 INFO whenAll: *: 'status[@value='UP']' fires
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 570 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 559 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 549 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 539 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 528 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 518 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 508 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 497 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 487 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 477 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 466 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 456 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 446 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 435 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 425 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 415 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 404 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 394 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 383 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 373 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 363 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 352 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 342 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 332 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 321 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 311 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 301 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 290 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 280 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 270 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 259 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 249 
sec.
 INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout: 238 
sec.
 INFO exp: Progress(0/0/35): 0/3/10 min(n_1_6)/avg/max (216) - Timeout: 228 
sec.
 INFO exp: Progress(0/0/35): 0/8/20 min(n_4_12)/avg/max (216) - Timeout: 218 
sec.
 INFO exp: Progress(0/0/35): 0/13/30 min(n_4_12)/avg/max (216) - Timeout: 
207 sec.
 INFO exp: Progress(0/0/35): 0/18/40 min(n_4_12)/avg/max (216) - Timeout: 
197 sec.
 INFO exp: Progress(0/0/35): 0/23/40 min(n_4_12)/avg/max (216) - Timeout: 
186 sec.
 INFO exp: Progress(0/0/35): 10/27/50 min(n_4_12)/avg/max (216) - Timeout: 
176 sec.
 INFO exp: Progress(0/0/35): 10/31/50 min(n_5_16)/avg/max (216) - Timeout: 
166 sec.
 INFO exp: Progress(0/0/35): 10/34/50 min(n_5_20)/avg/max (216) - Timeout: 
155 sec.
 INFO exp: Progress(0/0/35): 10/38/60 min(n_5_20)/avg/max (216) - Timeout: 
145 sec.
 INFO exp: Progress(0/0/35): 10/42/60 min(n_1_14)/avg/max (216) - Timeout: 
134 sec.
 INFO exp: Progress(0/0/35): 10/47/70 min(n_1_14)/avg/max (216) - Timeout: 
124 sec.
 INFO exp: Progress(0/0/35): 10/52/70 min(n_1_14)/avg/max (216) - Timeout: 
113 sec.
 INFO exp: Progress(0/0/35): 10/55/80 min(n_1_14)/avg/max (216) - Timeout: 
103 sec.
 INFO exp: Progress(0/0/35): 10/60/80 min(n_1_14)/avg/max (216) - Timeout: 
93 sec.
 INFO exp: Progress(0/0/35): 10/62/90 min(n_1_14)/avg/max (216) - Timeout: 
82 sec.
 INFO exp: Progress(0/0/35): 10/68/90 min(n_1_14)/avg/max (216) - Timeout: 
72 sec.
 INFO exp: Progress(0/0/35): 10/72/90 min(n_1_14)/avg/max (216) - Timeout: 
62 sec.
 INFO exp: Progress(0/0/35): 10/78/90 min(n_1_14)/avg/max (216) - Timeout: 
51 sec.
 INFO exp: Progress(0/0/35): 10/80/90 min(n_1_14)/avg/max (216) - Timeout: 
41 sec.
 INFO exp: Progress(0/0/35): 10/82/90 min(n_1_14)/avg/max (216) - Timeout: 
30 sec.
 INFO exp: Progress(0/0/35): 10/84/90 min(n_1_14)/avg/max (216) - Timeout: 
20 sec.
 INFO exp: Progress(1/0/35): 10/86/100 min(n_1_14)/avg/max (216) - Timeout: 
10 sec.
 INFO exp: Progress(5/0/35): 10/87/100 min(n_1_14)/avg/max (216) - 
Timeout: -1 sec.
 INFO exp:  ----------------------------- 
 INFO exp:  Imaging Process Done
 INFO exp:  - 30 node(s) timed-out - See the topology file: 
'topo_grid_timedout.rb'
 INFO exp:  - 5 node(s) succesfully imaged - See the topology file: 
'topo_grid_active.rb'
 INFO exp:  ----------------------------- 
 INFO Experiment: DONE!
 INFO ExecApp: Application 'commServer' finished
 INFO run: Experiment grid_2007_08_23_13_01_08 finished after 17:36
sangho at console.grid:~$ ccat topo_grid_active.rb
-bash: ccat: command not found
sangho at console.grid:~$ cat topo_grid_active.rb
# Topology name: topo_grid_active
#
# The following command creates a Topology wih the nodes that have 
successfully been imaged.
#

defTopology('topo_grid_active', [[5,7],[4,19],[1,9],[6,16],[7,17]])

----- Original Message ----- 
From: "Sangho Oh" <sangho at winlab.rutgers.edu>
To: <orbit-user at winlab.rutgers.edu>
Sent: Thursday, August 23, 2007 12:58 PM
Subject: Re: ORBIT-USER: Grid problem

> Right. I just noticed it.
> Most of nodes were given up.
> However, these nodes were usable yesterday, and +60% of nodes are given up 
> today.
> It is quite strange.
>
> - Sangho
>
>
>
> ----- Original Message ----- 
> From: "Haris Kremo" <harisk at winlab.rutgers.edu>
> To: <orbit-user at winlab.rutgers.edu>
> Sent: Thursday, August 23, 2007 12:54 PM
> Subject: Re: ORBIT-USER: Grid problem
>
>
>> Looks like only one was actually imeged?
>>
>> H.
>>
>> On 8/23/07, Sangho Oh <sangho at winlab.rutgers.edu> wrote:
>>>
>>>
>>
>>>  WARN stdlib: Giving up on node n_20_19
>>>  WARN stdlib: Giving up on node n_12_14
>>>  WARN stdlib: Giving up on node n_17_16
>>
>>
>>
>>>  INFO exp:  -----------------------------
>>>  INFO exp:  Imaging Process Done
>>>  INFO exp:  - 1 node(s) succesfully imaged - See the topology file:
>>> 'topo_grid_active.rb'
>>>  INFO exp:  -----------------------------
>>>  INFO Experiment: DONE!
>>>  INFO ExecApp: Application 'commServer' finished
>>>  INFO run: Experiment grid_2007_08_23_11_53_46 finished after 12:49
>>
> 