ORBIT-USER: Grid problem
Ivan Seskar
Seskar at winlab.rutgers.edu
Thu Aug 23 21:04:27 EDT 2007
Hi Thierry,
As far as we can tell, there were two issues:
1.) One of the two DHCP servers was in a weird state effectively
cutting off half of the grid.
2.) Control subnet switches were (at least some of them) having
problems with rate negotiation.
It is still not clear if these two were somehow related; we will keep an
eye on it.
Regards,
Ivan.
-----Original Message-----
From: owner-orbit-user at winlab.rutgers.edu
[mailto:owner-orbit-user at winlab.rutgers.edu] On Behalf Of Thierry
Rakotoarivelo
Sent: Thursday, August 23, 2007 8:05 PM
To: orbit-user at winlab.rutgers.edu
Subject: Re: ORBIT-USER: Grid problem
Dear all,
According to the recent emails, more people have experienced that "nodes
are too slow to respond" problem, which crippled the communication
between the nodeHandler and the multiple nodeAgents (impacting
imageNodes4 and other experiments).
At the moment, it also seems like this problem was fixed. I just
finished an imaging process with 210 nodes correctly imaged
(grid_2007_08_23_19_37_02).
Out of curiosity and for future reference, does anyone know what did fix
the problem we all experienced recently? (e.g. rebooting some devices,
restarting some services,...)
Regards,
Thierry.
Sangho Oh wrote:
> I tried to image the whole grid, but most of nodes are either ignored
> or given up.
> I was only able to turn on 5 of them. "defTopology('topo_grid_active',
> [[5,7],[4,19],[1,9],[6,16],[7,17]])"
> I think same thing happened weeks ago and suddenly the grid went to be
> stable two weeks ago, and it happens again.
> Is there a way to reset the grid by indivisual users?
>
> - Sangho
>
>
> sangho at console.grid:~$ imageNodes4 [[1..20,[1..20]] sangho_dfs1.ndz
> Imaging nodes: '[[1..20,[1..20]]' with image 'sangho_dfs1.ndz'
> (Domain: default from hostname)
> (Timeout: 800 sec.)
> INFO init: NodeHandler Version 4.2.0 (1272) INFO init: Experiment ID:
> grid_2007_08_23_13_00_53 INFO ExecApp: Starting application
> 'commServer':
> /opt/nodehandler4-4.2.0/sbin/commServer --logfile
> /tmp/commServer-grid_2007_08_23_13_00_53.log -d 4 --iface eth1 -e
> ^CERROR NodeHandler:
> ERROR '' when starting NH webserver !
> ERROR ExecApp: Application 'commServer' failed (code=2) ERROR
> NodeHandler: Possible source of this Error: another NH is already
> running on the same tesbed...
>
> ERROR Communicator: ComServer failed: status: 2 INFO run: Experiment
> grid_2007_08_23_13_00_53 finished after 0:7 sangho at console.grid:~$
> imageNodes4 [[1..20,1..20]] sangho_dfs1.ndz Imaging nodes:
> '[[1..20,1..20]]' with image 'sangho_dfs1.ndz'
> (Domain: default from hostname)
> (Timeout: 800 sec.)
> INFO init: NodeHandler Version 4.2.0 (1272) INFO init: Experiment ID:
> grid_2007_08_23_13_01_08 INFO ExecApp: Starting application
> 'commServer':
> /opt/nodehandler4-4.2.0/sbin/commServer --logfile
> /tmp/commServer-grid_2007_08_23_13_01_08.log -d 4 --iface eth1 -e INFO
> Experiment: load system:exp:stdlib INFO prop.resetDelay: resetDelay =
> 210:Fixnum INFO prop.resetTries: resetTries = 1:Fixnum INFO
> Experiment: load system:exp:imageNode INFO prop.nodes: nodes =
> [[[1..20, 1..20]]]:Array INFO prop.image: image =
> "sangho_dfs1.ndz":String INFO prop.pxe: pxe = "1.2.1-omf":String INFO
> prop.domain: domain = nil:NilClass INFO prop.timeout: timeout =
> 800:Fixnum WARN -:topo:image: Ignoring missing node '1 at 5'
> WARN -:topo:image: Ignoring missing node '1 at 12'
> WARN -:topo:image: Ignoring missing node '1 at 19'
> WARN -:topo:image: Ignoring missing node '2 at 11'
> WARN -:topo:image: Ignoring missing node '2 at 20'
> WARN -:topo:image: Ignoring missing node '3 at 6'
> WARN -:topo:image: Ignoring missing node '4 at 8'
> WARN -:topo:image: Ignoring missing node '4 at 13'
> WARN -:topo:image: Ignoring missing node '4 at 14'
> WARN -:topo:image: Ignoring missing node '4 at 16'
> WARN -:topo:image: Ignoring missing node '4 at 17'
> WARN -:topo:image: Ignoring missing node '4 at 20'
> WARN -:topo:image: Ignoring missing node '5 at 4'
> WARN -:topo:image: Ignoring missing node '5 at 5'
> WARN -:topo:image: Ignoring missing node '5 at 6'
> WARN -:topo:image: Ignoring missing node '5 at 10'
> WARN -:topo:image: Ignoring missing node '5 at 12'
> WARN -:topo:image: Ignoring missing node '5 at 14'
> WARN -:topo:image: Ignoring missing node '5 at 17'
> WARN -:topo:image: Ignoring missing node '6 at 1'
> WARN -:topo:image: Ignoring missing node '6 at 3'
> WARN -:topo:image: Ignoring missing node '6 at 5'
> WARN -:topo:image: Ignoring missing node '6 at 8'
> WARN -:topo:image: Ignoring missing node '6 at 10'
> WARN -:topo:image: Ignoring missing node '6 at 11'
> WARN -:topo:image: Ignoring missing node '6 at 12'
> WARN -:topo:image: Ignoring missing node '6 at 14'
> WARN -:topo:image: Ignoring missing node '6 at 15'
> WARN -:topo:image: Ignoring missing node '6 at 17'
> WARN -:topo:image: Ignoring missing node '6 at 20'
> WARN -:topo:image: Ignoring missing node '7 at 1'
> WARN -:topo:image: Ignoring missing node '7 at 4'
> WARN -:topo:image: Ignoring missing node '7 at 10'
> WARN -:topo:image: Ignoring missing node '7 at 14'
> WARN -:topo:image: Ignoring missing node '7 at 15'
> WARN -:topo:image: Ignoring missing node '7 at 19'
> WARN -:topo:image: Ignoring missing node '8 at 2'
> WARN -:topo:image: Ignoring missing node '8 at 5'
> WARN -:topo:image: Ignoring missing node '8 at 10'
> WARN -:topo:image: Ignoring missing node '8 at 11'
> WARN -:topo:image: Ignoring missing node '8 at 12'
> WARN -:topo:image: Ignoring missing node '8 at 15'
> WARN -:topo:image: Ignoring missing node '8 at 16'
> WARN -:topo:image: Ignoring missing node '8 at 19'
> WARN -:topo:image: Ignoring missing node '8 at 20'
> WARN -:topo:image: Ignoring missing node '9 at 3'
> WARN -:topo:image: Ignoring missing node '9 at 7'
> WARN -:topo:image: Ignoring missing node '9 at 10'
> WARN -:topo:image: Ignoring missing node '9 at 11'
> WARN -:topo:image: Ignoring missing node '9 at 13'
> WARN -:topo:image: Ignoring missing node '9 at 15'
> WARN -:topo:image: Ignoring missing node '9 at 16'
> WARN -:topo:image: Ignoring missing node '9 at 18'
> WARN -:topo:image: Ignoring missing node '9 at 19'
> WARN -:topo:image: Ignoring missing node '9 at 20'
> WARN -:topo:image: Ignoring missing node '10 at 1'
> WARN -:topo:image: Ignoring missing node '10 at 2'
> WARN -:topo:image: Ignoring missing node '10 at 5'
> WARN -:topo:image: Ignoring missing node '10 at 7'
> WARN -:topo:image: Ignoring missing node '10 at 11'
> WARN -:topo:image: Ignoring missing node '10 at 12'
> WARN -:topo:image: Ignoring missing node '10 at 15'
> WARN -:topo:image: Ignoring missing node '11 at 2'
> WARN -:topo:image: Ignoring missing node '11 at 6'
> WARN -:topo:image: Ignoring missing node '11 at 11'
> WARN -:topo:image: Ignoring missing node '11 at 18'
> WARN -:topo:image: Ignoring missing node '12 at 2'
> WARN -:topo:image: Ignoring missing node '12 at 7'
> WARN -:topo:image: Ignoring missing node '12 at 10'
> WARN -:topo:image: Ignoring missing node '12 at 12'
> WARN -:topo:image: Ignoring missing node '12 at 15'
> WARN -:topo:image: Ignoring missing node '12 at 16'
> WARN -:topo:image: Ignoring missing node '12 at 18'
> WARN -:topo:image: Ignoring missing node '13 at 2'
> WARN -:topo:image: Ignoring missing node '13 at 5'
> WARN -:topo:image: Ignoring missing node '13 at 8'
> WARN -:topo:image: Ignoring missing node '13 at 13'
> WARN -:topo:image: Ignoring missing node '14 at 4'
> WARN -:topo:image: Ignoring missing node '14 at 11'
> WARN -:topo:image: Ignoring missing node '14 at 13'
> WARN -:topo:image: Ignoring missing node '14 at 14'
> WARN -:topo:image: Ignoring missing node '15 at 1'
> WARN -:topo:image: Ignoring missing node '15 at 3'
> WARN -:topo:image: Ignoring missing node '15 at 6'
> WARN -:topo:image: Ignoring missing node '15 at 7'
> WARN -:topo:image: Ignoring missing node '15 at 10'
> WARN -:topo:image: Ignoring missing node '15 at 12'
> WARN -:topo:image: Ignoring missing node '15 at 15'
> WARN -:topo:image: Ignoring missing node '16 at 13'
> WARN -:topo:image: Ignoring missing node '16 at 17'
> WARN -:topo:image: Ignoring missing node '17 at 2'
> WARN -:topo:image: Ignoring missing node '17 at 3'
> WARN -:topo:image: Ignoring missing node '17 at 9'
> WARN -:topo:image: Ignoring missing node '17 at 12'
> WARN -:topo:image: Ignoring missing node '17 at 20'
> WARN -:topo:image: Ignoring missing node '18 at 5'
> WARN -:topo:image: Ignoring missing node '18 at 7'
> WARN -:topo:image: Ignoring missing node '18 at 15'
> WARN -:topo:image: Ignoring missing node '18 at 17'
> WARN -:topo:image: Ignoring missing node '18 at 18'
> WARN -:topo:image: Ignoring missing node '18 at 20'
> WARN -:topo:image: Ignoring missing node '19 at 4'
> WARN -:topo:image: Ignoring missing node '19 at 9'
> WARN -:topo:image: Ignoring missing node '19 at 13'
> WARN -:topo:image: Ignoring missing node '19 at 15'
> WARN -:topo:image: Ignoring missing node '19 at 17'
> WARN -:topo:image: Ignoring missing node '20 at 10'
> WARN -:topo:image: Ignoring missing node '20 at 13'
> WARN -:topo:image: Ignoring missing node '20 at 15'
> WARN -:topo:image: Ignoring missing node '20 at 17'
> INFO stdlib: Waiting for nodes (Up/Down/Total): 0/290/290 - (still
down:
> n_20_16,n_7_18,n_1_6)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 0/290/290 - (still
down:
> n_20_16,n_7_18,n_1_6)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 0/290/290 - (still
down:
> n_20_16,n_7_18,n_1_6)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 0/290/290 - (still
down:
> n_20_16,n_7_18,n_1_6)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 0/290/290 - (still
down:
> n_20_16,n_7_18,n_1_6)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 0/290/290 - (still
down:
> n_20_16,n_7_18,n_1_6)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 0/290/290 - (still
down:
> n_20_16,n_7_18,n_1_6)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 0/290/290 - (still
down:
> n_20_16,n_7_18,n_1_6)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 0/290/290 - (still
down:
> n_20_16,n_7_18,n_1_6)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 0/290/290 - (still
down:
> n_20_16,n_7_18,n_1_6)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 0/290/290 - (still
down:
> n_20_16,n_7_18,n_1_6)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 0/290/290 - (still
down:
> n_20_16,n_7_18,n_1_6)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 0/290/290 - (still
down:
> n_20_16,n_7_18,n_1_6)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 1/289/290 - (still
down:
> n_20_16,n_7_18,n_1_6)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 3/287/290 - (still
down:
> n_20_16,n_7_18,n_1_6)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 5/285/290 - (still
down:
> n_20_16,n_7_18,n_1_6)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 6/284/290 - (still
down:
> n_20_16,n_7_18,n_1_6)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 8/282/290 - (still
down:
> n_20_16,n_7_18,n_2_9)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 10/280/290 - (still
> down: n_20_16,n_7_18,n_2_9)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 13/277/290 - (still
> down: n_20_16,n_7_18,n_2_9)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 15/275/290 - (still
> down: n_20_16,n_7_18,n_2_9)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 16/274/290 - (still
> down: n_20_16,n_7_18,n_2_9)
> INFO stdlib: Resetting node n_20_16
> INFO stdlib: Resetting node n_7_18
> INFO stdlib: Resetting node n_2_9
> INFO stdlib: Resetting node n_2_6
> INFO stdlib: Resetting node n_11_17
> INFO stdlib: Resetting node n_14_1
> INFO stdlib: Resetting node n_12_6
> INFO stdlib: Resetting node n_5_7
> INFO stdlib: Resetting node n_19_18
> INFO stdlib: Resetting node n_12_1
> INFO stdlib: Resetting node n_5_1
> INFO stdlib: Resetting node n_18_14
> INFO stdlib: Resetting node n_5_11
> INFO stdlib: Resetting node n_8_9
> INFO stdlib: Resetting node n_8_13
> INFO stdlib: Resetting node n_1_4
> INFO stdlib: Resetting node n_18_12
> INFO stdlib: Resetting node n_1_17
> INFO stdlib: Resetting node n_1_13
> INFO stdlib: Resetting node n_9_5
> INFO stdlib: Resetting node n_16_7
> INFO stdlib: Resetting node n_11_13
> INFO stdlib: Resetting node n_7_12
> INFO stdlib: Resetting node n_9_9
> INFO stdlib: Resetting node n_14_6
> INFO stdlib: Resetting node n_2_19
> INFO stdlib: Resetting node n_17_5
> INFO stdlib: Resetting node n_4_3
> INFO stdlib: Resetting node n_6_6
> INFO stdlib: Resetting node n_13_4
> INFO stdlib: Resetting node n_13_7
> INFO stdlib: Resetting node n_20_11
> INFO stdlib: Resetting node n_11_10
> INFO stdlib: Resetting node n_4_19
> INFO stdlib: Resetting node n_11_16
> INFO stdlib: Resetting node n_11_20
> INFO stdlib: Resetting node n_20_8
> INFO stdlib: Resetting node n_20_7
> INFO stdlib: Resetting node n_20_6
> INFO stdlib: Resetting node n_1_10
> INFO stdlib: Resetting node n_3_16
> INFO stdlib: Resetting node n_15_13
> INFO stdlib: Resetting node n_16_20
> INFO stdlib: Resetting node n_12_20
> INFO stdlib: Resetting node n_13_1
> INFO stdlib: Resetting node n_3_1
> INFO stdlib: Resetting node n_1_15
> INFO stdlib: Resetting node n_15_18
> INFO stdlib: Resetting node n_1_8
> INFO stdlib: Resetting node n_19_3
> INFO stdlib: Resetting node n_13_18
> INFO stdlib: Resetting node n_2_10
> INFO stdlib: Resetting node n_11_12
> INFO stdlib: Resetting node n_10_8
> INFO stdlib: Resetting node n_12_11
> INFO stdlib: Resetting node n_5_15
> INFO stdlib: Resetting node n_2_4
> INFO stdlib: Resetting node n_16_6
> INFO stdlib: Resetting node n_1_18
> INFO stdlib: Resetting node n_1_3
> INFO stdlib: Resetting node n_3_17
> INFO stdlib: Resetting node n_19_10
> INFO stdlib: Resetting node n_19_8
> INFO stdlib: Resetting node n_10_3
> INFO stdlib: Resetting node n_14_2
> INFO stdlib: Resetting node n_16_5
> INFO stdlib: Resetting node n_1_11
> INFO stdlib: Resetting node n_20_12
> INFO stdlib: Resetting node n_14_9
> INFO stdlib: Resetting node n_10_19
> INFO stdlib: Resetting node n_1_9
> INFO stdlib: Resetting node n_13_20
> INFO stdlib: Resetting node n_15_16
> INFO stdlib: Resetting node n_18_16
> INFO stdlib: Resetting node n_17_8
> INFO stdlib: Resetting node n_11_8
> INFO stdlib: Resetting node n_4_15
> INFO stdlib: Resetting node n_4_18
> INFO stdlib: Resetting node n_2_12
> INFO stdlib: Resetting node n_19_19
> INFO stdlib: Resetting node n_17_17
> INFO stdlib: Resetting node n_4_9
> INFO stdlib: Resetting node n_13_19
> INFO stdlib: Resetting node n_14_20
> INFO stdlib: Resetting node n_19_11
> INFO stdlib: Resetting node n_18_13
> INFO stdlib: Resetting node n_17_18
> INFO stdlib: Resetting node n_1_20
> INFO stdlib: Resetting node n_16_14
> INFO stdlib: Resetting node n_4_7
> INFO stdlib: Resetting node n_14_17
> INFO stdlib: Resetting node n_8_7
> INFO stdlib: Resetting node n_2_13
> INFO stdlib: Resetting node n_15_19
> INFO stdlib: Resetting node n_20_18
> INFO stdlib: Resetting node n_19_16
> INFO stdlib: Resetting node n_4_11
> INFO stdlib: Resetting node n_2_18
> INFO stdlib: Resetting node n_15_8
> INFO stdlib: Resetting node n_2_14
> INFO stdlib: Resetting node n_13_14
> INFO stdlib: Resetting node n_20_14
> INFO stdlib: Resetting node n_18_19
> INFO stdlib: Resetting node n_18_9
> INFO stdlib: Resetting node n_16_2
> INFO stdlib: Resetting node n_20_3
> INFO stdlib: Resetting node n_13_3
> INFO stdlib: Resetting node n_20_20
> INFO stdlib: Resetting node n_18_4
> INFO stdlib: Resetting node n_17_13
> INFO stdlib: Resetting node n_16_11
> INFO stdlib: Resetting node n_7_20
> INFO stdlib: Resetting node n_3_14
> INFO stdlib: Resetting node n_16_12
> INFO stdlib: Resetting node n_6_9
> INFO stdlib: Resetting node n_1_16
> INFO stdlib: Resetting node n_13_6
> INFO stdlib: Resetting node n_12_4
> INFO stdlib: Resetting node n_14_8
> INFO stdlib: Resetting node n_16_16
> INFO stdlib: Resetting node n_5_8
> INFO stdlib: Resetting node n_13_10
> INFO stdlib: Resetting node n_14_15
> INFO stdlib: Resetting node n_7_11
> INFO stdlib: Resetting node n_16_1
> INFO stdlib: Resetting node n_2_5
> INFO stdlib: Resetting node n_20_2
> INFO stdlib: Resetting node n_19_5
> INFO stdlib: Resetting node n_12_8
> INFO stdlib: Resetting node n_18_6
> INFO stdlib: Resetting node n_6_16
> INFO stdlib: Resetting node n_16_8
> INFO stdlib: Resetting node n_19_12
> INFO stdlib: Resetting node n_19_7
> INFO stdlib: Resetting node n_4_2
> INFO stdlib: Resetting node n_13_12
> INFO stdlib: Resetting node n_15_14
> INFO stdlib: Resetting node n_2_7
> INFO stdlib: Resetting node n_14_3
> INFO stdlib: Resetting node n_15_9
> INFO stdlib: Resetting node n_8_18
> INFO stdlib: Resetting node n_2_2
> INFO stdlib: Resetting node n_3_12
> INFO stdlib: Resetting node n_7_7
> INFO stdlib: Resetting node n_10_17
> INFO stdlib: Resetting node n_3_7
> INFO stdlib: Resetting node n_12_5
> INFO stdlib: Resetting node n_17_14
> INFO stdlib: Resetting node n_10_9
> INFO stdlib: Resetting node n_11_9
> INFO stdlib: Resetting node n_2_8
> INFO stdlib: Resetting node n_3_13
> INFO stdlib: Resetting node n_3_19
> INFO stdlib: Resetting node n_11_14
> INFO stdlib: Resetting node n_20_5
> INFO stdlib: Resetting node n_20_4
> INFO stdlib: Resetting node n_18_1
> INFO stdlib: Resetting node n_5_19
> INFO stdlib: Resetting node n_2_15
> INFO stdlib: Resetting node n_4_5
> INFO stdlib: Resetting node n_6_13
> INFO stdlib: Resetting node n_14_10
> INFO stdlib: Resetting node n_2_1
> INFO stdlib: Resetting node n_12_14
> INFO stdlib: Resetting node n_9_14
> INFO stdlib: Resetting node n_9_4
> INFO stdlib: Resetting node n_3_3
> INFO stdlib: Resetting node n_10_13
> INFO stdlib: Resetting node n_3_2
> INFO stdlib: Resetting node n_11_5
> INFO stdlib: Resetting node n_16_10
> INFO stdlib: Resetting node n_7_16
> INFO stdlib: Resetting node n_9_17
> INFO stdlib: Resetting node n_9_6
> INFO stdlib: Resetting node n_18_10
> INFO stdlib: Resetting node n_17_6
> INFO stdlib: Resetting node n_3_10
> INFO stdlib: Resetting node n_16_3
> INFO stdlib: Resetting node n_10_10
> INFO stdlib: Resetting node n_11_7
> INFO stdlib: Resetting node n_3_11
> INFO stdlib: Resetting node n_4_10
> INFO stdlib: Resetting node n_14_18
> INFO stdlib: Resetting node n_16_15
> INFO stdlib: Resetting node n_3_5
> INFO stdlib: Resetting node n_12_19
> INFO stdlib: Resetting node n_17_7
> INFO stdlib: Resetting node n_10_16
> INFO stdlib: Resetting node n_15_20
> INFO stdlib: Resetting node n_6_4
> INFO stdlib: Resetting node n_11_3
> INFO stdlib: Resetting node n_15_4
> INFO stdlib: Resetting node n_12_3
> INFO stdlib: Resetting node n_19_1
> INFO stdlib: Resetting node n_9_2
> INFO stdlib: Resetting node n_6_19
> INFO stdlib: Resetting node n_11_4
> INFO stdlib: Resetting node n_10_14
> INFO stdlib: Resetting node n_13_11
> INFO stdlib: Resetting node n_3_4
> INFO stdlib: Resetting node n_3_15
> INFO stdlib: Resetting node n_18_11
> INFO stdlib: Resetting node n_17_1
> INFO stdlib: Resetting node n_4_4
> INFO stdlib: Resetting node n_15_2
> INFO stdlib: Resetting node n_19_14
> INFO stdlib: Resetting node n_18_2
> INFO stdlib: Resetting node n_17_10
> INFO stdlib: Resetting node n_7_13
> INFO stdlib: Resetting node n_12_13
> INFO stdlib: Resetting node n_15_5
> INFO stdlib: Resetting node n_11_15
> INFO stdlib: Resetting node n_6_7
> INFO stdlib: Resetting node n_15_17
> INFO stdlib: Resetting node n_4_6
> INFO stdlib: Resetting node n_20_1
> INFO stdlib: Resetting node n_17_11
> INFO stdlib: Resetting node n_13_17
> INFO stdlib: Resetting node n_10_4
> INFO stdlib: Resetting node n_17_16
> INFO stdlib: Resetting node n_16_18
> INFO stdlib: Resetting node n_10_6
> INFO stdlib: Resetting node n_8_4
> INFO stdlib: Resetting node n_1_2
> INFO stdlib: Resetting node n_3_8
> INFO stdlib: Resetting node n_17_4
> INFO stdlib: Resetting node n_5_3
> INFO stdlib: Resetting node n_13_15
> INFO stdlib: Resetting node n_20_9
> INFO stdlib: Resetting node n_19_6
> INFO stdlib: Resetting node n_17_19
> INFO stdlib: Resetting node n_5_2
> INFO stdlib: Resetting node n_9_8
> INFO stdlib: Resetting node n_12_9
> INFO stdlib: Resetting node n_5_18
> INFO stdlib: Resetting node n_5_13
> INFO stdlib: Resetting node n_7_8
> INFO stdlib: Resetting node n_16_9
> INFO stdlib: Resetting node n_6_18
> INFO stdlib: Resetting node n_14_19
> INFO stdlib: Resetting node n_16_4
> INFO stdlib: Resetting node n_8_14
> INFO stdlib: Resetting node n_3_9
> INFO stdlib: Resetting node n_2_16
> INFO stdlib: Resetting node n_11_1
> INFO stdlib: Resetting node n_14_5
> INFO stdlib: Resetting node n_9_12
> INFO stdlib: Resetting node n_19_2
> INFO stdlib: Resetting node n_17_15
> INFO stdlib: Resetting node n_16_19
> INFO stdlib: Resetting node n_3_18
> INFO stdlib: Resetting node n_2_3
> INFO stdlib: Resetting node n_19_20
> INFO stdlib: Resetting node n_8_6
> INFO stdlib: Resetting node n_14_12
> INFO stdlib: Resetting node n_8_1
> INFO stdlib: Resetting node n_3_20
> INFO stdlib: Resetting node n_14_7
> INFO stdlib: Resetting node n_20_19
> INFO stdlib: Resetting node n_11_19
> INFO stdlib: Resetting node n_4_1
> INFO stdlib: Resetting node n_18_3
> INFO stdlib: Resetting node n_2_17
> INFO stdlib: Resetting node n_8_8
> INFO stdlib: Resetting node n_10_18
> INFO stdlib: Resetting node n_18_8
> INFO stdlib: Resetting node n_15_11
> INFO stdlib: Resetting node n_13_16
> INFO stdlib: Resetting node n_14_16
> INFO stdlib: Resetting node n_13_9
> INFO stdlib: Resetting node n_12_17
> INFO stdlib: Waiting for nodes (Up/Down/Total): 19/271/290 - (still
> down: n_20_16,n_7_18,n_2_9)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 19/271/290 - (still
> down: n_20_16,n_7_18,n_2_9)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 19/271/290 - (still
> down: n_20_16,n_7_18,n_2_9)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 19/271/290 - (still
> down: n_20_16,n_7_18,n_2_9)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 19/271/290 - (still
> down: n_20_16,n_7_18,n_2_9)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 19/271/290 - (still
> down: n_20_16,n_7_18,n_2_9)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 19/271/290 - (still
> down: n_20_16,n_7_18,n_2_9)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 19/271/290 - (still
> down: n_20_16,n_7_18,n_2_9)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 19/271/290 - (still
> down: n_20_16,n_7_18,n_2_9)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 19/271/290 - (still
> down: n_20_16,n_7_18,n_2_9)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 19/271/290 - (still
> down: n_20_16,n_7_18,n_2_9)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 19/271/290 - (still
> down: n_20_16,n_7_18,n_2_9)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 19/271/290 - (still
> down: n_20_16,n_7_18,n_2_9)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 21/269/290 - (still
> down: n_20_16,n_7_18,n_2_9)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 24/266/290 - (still
> down: n_20_16,n_7_18,n_2_9)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 25/265/290 - (still
> down: n_20_16,n_7_18,n_2_9)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 27/263/290 - (still
> down: n_20_16,n_7_18,n_2_9)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 29/261/290 - (still
> down: n_20_16,n_7_18,n_2_9)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 30/260/290 - (still
> down: n_20_16,n_7_18,n_2_9)
> INFO stdlib: Waiting for nodes (Up/Down/Total): 31/259/290 - (still
> down: n_20_16,n_7_18,n_2_9)
> WARN stdlib: Giving up on node n_20_16 WARN stdlib: Giving up on node
> n_7_18 WARN stdlib: Giving up on node n_2_9 WARN stdlib: Giving up on
> node n_2_6 WARN stdlib: Giving up on node n_11_17 WARN stdlib: Giving
> up on node n_14_1 WARN stdlib: Giving up on node n_12_6 WARN stdlib:
> Giving up on node n_19_18 WARN stdlib: Giving up on node n_12_1 WARN
> stdlib: Giving up on node n_18_14 WARN stdlib: Giving up on node
> n_5_11 WARN stdlib: Giving up on node n_8_9 WARN stdlib: Giving up on
> node n_1_4 WARN stdlib: Giving up on node n_18_12 WARN stdlib: Giving
> up on node n_9_5 WARN stdlib: Giving up on node n_16_7 WARN stdlib:
> Giving up on node n_11_13 WARN stdlib: Giving up on node n_7_12 WARN
> stdlib: Giving up on node n_9_9 WARN stdlib: Giving up on node n_14_6
> WARN stdlib: Giving up on node n_2_19 WARN stdlib: Giving up on node
> n_17_5 WARN stdlib: Giving up on node n_4_3 WARN stdlib: Giving up on
> node n_13_4 WARN stdlib: Giving up on node n_13_7 WARN stdlib: Giving
> up on node n_20_11 WARN stdlib: Giving up on node n_11_10 WARN stdlib:
> Giving up on node n_11_16 WARN stdlib: Giving up on node n_11_20 WARN
> stdlib: Giving up on node n_20_8 WARN stdlib: Giving up on node n_20_7
> WARN stdlib: Giving up on node n_20_6 WARN stdlib: Giving up on node
> n_1_10 WARN stdlib: Giving up on node n_3_16 WARN stdlib: Giving up on
> node n_15_13 WARN stdlib: Giving up on node n_16_20 WARN stdlib:
> Giving up on node n_12_20 WARN stdlib: Giving up on node n_13_1 WARN
> stdlib: Giving up on node n_3_1 WARN stdlib: Giving up on node n_15_18
> WARN stdlib: Giving up on node n_19_3 WARN stdlib: Giving up on node
> n_13_18 WARN stdlib: Giving up on node n_2_10 WARN stdlib: Giving up
> on node n_11_12 WARN stdlib: Giving up on node n_12_11 WARN stdlib:
> Giving up on node n_2_4 WARN stdlib: Giving up on node n_16_6 WARN
> stdlib: Giving up on node n_1_18 WARN stdlib: Giving up on node n_1_3
> WARN stdlib: Giving up on node n_3_17 WARN stdlib: Giving up on node
> n_19_10 WARN stdlib: Giving up on node n_19_8 WARN stdlib: Giving up
> on node n_10_3 WARN stdlib: Giving up on node n_14_2 WARN stdlib:
> Giving up on node n_16_5 WARN stdlib: Giving up on node n_1_11 WARN
> stdlib: Giving up on node n_20_12 WARN stdlib: Giving up on node
> n_14_9 WARN stdlib: Giving up on node n_10_19 WARN stdlib: Giving up
> on node n_13_20 WARN stdlib: Giving up on node n_15_16 WARN stdlib:
> Giving up on node n_18_16 WARN stdlib: Giving up on node n_17_8 WARN
> stdlib: Giving up on node n_11_8 WARN stdlib: Giving up on node n_4_15
> WARN stdlib: Giving up on node n_4_18 WARN stdlib: Giving up on node
> n_2_12 WARN stdlib: Giving up on node n_19_19 WARN stdlib: Giving up
> on node n_17_17 WARN stdlib: Giving up on node n_13_19 WARN stdlib:
> Giving up on node n_14_20 WARN stdlib: Giving up on node n_19_11 WARN
> stdlib: Giving up on node n_18_13 WARN stdlib: Giving up on node
> n_17_18 WARN stdlib: Giving up on node n_1_20 WARN stdlib: Giving up
> on node n_16_14 WARN stdlib: Giving up on node n_4_7 WARN stdlib:
> Giving up on node n_14_17 WARN stdlib: Giving up on node n_8_7 WARN
> stdlib: Giving up on node n_2_13 WARN stdlib: Giving up on node
> n_15_19 WARN stdlib: Giving up on node n_20_18 WARN stdlib: Giving up
> on node n_19_16 WARN stdlib: Giving up on node n_4_11 WARN stdlib:
> Giving up on node n_2_18 WARN stdlib: Giving up on node n_15_8 WARN
> stdlib: Giving up on node n_2_14 WARN stdlib: Giving up on node
> n_13_14 WARN stdlib: Giving up on node n_20_14 WARN stdlib: Giving up
> on node n_18_19 WARN stdlib: Giving up on node n_18_9 WARN stdlib:
> Giving up on node n_16_2 WARN stdlib: Giving up on node n_20_3 WARN
> stdlib: Giving up on node n_13_3 WARN stdlib: Giving up on node
> n_20_20 WARN stdlib: Giving up on node n_18_4 WARN stdlib: Giving up
> on node n_17_13 INFO stdlib: Waiting for nodes (Up/Down/Total):
> 35/255/290 - (still
> down: n_20_16,n_7_18,n_2_9)
> WARN stdlib: Giving up on node n_16_11 WARN stdlib: Giving up on node
> n_3_14 WARN stdlib: Giving up on node n_16_12 WARN stdlib: Giving up
> on node n_6_9 WARN stdlib: Giving up on node n_13_6 WARN stdlib:
> Giving up on node n_12_4 WARN stdlib: Giving up on node n_14_8 WARN
> stdlib: Giving up on node n_16_16 WARN stdlib: Giving up on node n_5_8
> WARN stdlib: Giving up on node n_13_10 WARN stdlib: Giving up on node
> n_14_15 WARN stdlib: Giving up on node n_7_11 WARN stdlib: Giving up
> on node n_16_1 WARN stdlib: Giving up on node n_2_5 WARN stdlib:
> Giving up on node n_20_2 WARN stdlib: Giving up on node n_19_5 WARN
> stdlib: Giving up on node n_12_8 WARN stdlib: Giving up on node n_18_6
> WARN stdlib: Giving up on node n_16_8 WARN stdlib: Giving up on node
> n_19_12 WARN stdlib: Giving up on node n_19_7 WARN stdlib: Giving up
> on node n_4_2 WARN stdlib: Giving up on node n_13_12 WARN stdlib:
> Giving up on node n_15_14 WARN stdlib: Giving up on node n_2_7 WARN
> stdlib: Giving up on node n_14_3 WARN stdlib: Giving up on node n_15_9
> WARN stdlib: Giving up on node n_8_18 WARN stdlib: Giving up on node
> n_2_2 WARN stdlib: Giving up on node n_3_12 WARN stdlib: Giving up on
> node n_7_7 WARN stdlib: Giving up on node n_10_17 WARN stdlib: Giving
> up on node n_3_7 WARN stdlib: Giving up on node n_12_5 WARN stdlib:
> Giving up on node n_17_14 WARN stdlib: Giving up on node n_10_9 WARN
> stdlib: Giving up on node n_11_9 WARN stdlib: Giving up on node n_2_8
> WARN stdlib: Giving up on node n_3_13 WARN stdlib: Giving up on node
> n_3_19 WARN stdlib: Giving up on node n_11_14 WARN stdlib: Giving up
> on node n_20_5 WARN stdlib: Giving up on node n_20_4 WARN stdlib:
> Giving up on node n_18_1 WARN stdlib: Giving up on node n_5_19 WARN
> stdlib: Giving up on node n_2_15 WARN stdlib: Giving up on node n_4_5
> WARN stdlib: Giving up on node n_6_13 WARN stdlib: Giving up on node
> n_14_10 WARN stdlib: Giving up on node n_2_1 WARN stdlib: Giving up on
> node n_12_14 WARN stdlib: Giving up on node n_9_14 WARN stdlib: Giving
> up on node n_9_4 WARN stdlib: Giving up on node n_3_3 WARN stdlib:
> Giving up on node n_10_13 WARN stdlib: Giving up on node n_3_2 WARN
> stdlib: Giving up on node n_11_5 WARN stdlib: Giving up on node
> n_16_10 WARN stdlib: Giving up on node n_7_16 WARN stdlib: Giving up
> on node n_9_17 WARN stdlib: Giving up on node n_9_6 WARN stdlib:
> Giving up on node n_18_10 WARN stdlib: Giving up on node n_17_6 WARN
> stdlib: Giving up on node n_3_10 WARN stdlib: Giving up on node n_16_3
> WARN stdlib: Giving up on node n_10_10 WARN stdlib: Giving up on node
> n_11_7 WARN stdlib: Giving up on node n_3_11 WARN stdlib: Giving up on
> node n_4_10 WARN stdlib: Giving up on node n_14_18 WARN stdlib: Giving
> up on node n_16_15 WARN stdlib: Giving up on node n_3_5 WARN stdlib:
> Giving up on node n_12_19 WARN stdlib: Giving up on node n_17_7 WARN
> stdlib: Giving up on node n_10_16 WARN stdlib: Giving up on node
> n_15_20 WARN stdlib: Giving up on node n_6_4 WARN stdlib: Giving up on
> node n_11_3 WARN stdlib: Giving up on node n_15_4 WARN stdlib: Giving
> up on node n_12_3 WARN stdlib: Giving up on node n_19_1 WARN stdlib:
> Giving up on node n_9_2 WARN stdlib: Giving up on node n_6_19 WARN
> stdlib: Giving up on node n_11_4 WARN stdlib: Giving up on node
> n_10_14 WARN stdlib: Giving up on node n_13_11 WARN stdlib: Giving up
> on node n_3_4 WARN stdlib: Giving up on node n_3_15 WARN stdlib:
> Giving up on node n_18_11 WARN stdlib: Giving up on node n_17_1 WARN
> stdlib: Giving up on node n_4_4 WARN stdlib: Giving up on node n_15_2
> WARN stdlib: Giving up on node n_19_14 WARN stdlib: Giving up on node
> n_18_2 WARN stdlib: Giving up on node n_17_10 WARN stdlib: Giving up
> on node n_7_13 WARN stdlib: Giving up on node n_12_13 WARN stdlib:
> Giving up on node n_15_5 WARN stdlib: Giving up on node n_11_15 WARN
> stdlib: Giving up on node n_6_7 WARN stdlib: Giving up on node n_15_17
> WARN stdlib: Giving up on node n_4_6 WARN stdlib: Giving up on node
> n_20_1 WARN stdlib: Giving up on node n_17_11 WARN stdlib: Giving up
> on node n_13_17 WARN stdlib: Giving up on node n_10_4 WARN stdlib:
> Giving up on node n_17_16 WARN stdlib: Giving up on node n_16_18 WARN
> stdlib: Giving up on node n_10_6 WARN stdlib: Giving up on node n_8_4
> WARN stdlib: Giving up on node n_1_2 WARN stdlib: Giving up on node
> n_3_8 WARN stdlib: Giving up on node n_17_4 WARN stdlib: Giving up on
> node n_5_3 WARN stdlib: Giving up on node n_13_15 WARN stdlib: Giving
> up on node n_20_9 WARN stdlib: Giving up on node n_19_6 WARN stdlib:
> Giving up on node n_17_19 WARN stdlib: Giving up on node n_5_2 WARN
> stdlib: Giving up on node n_9_8 WARN stdlib: Giving up on node n_12_9
> WARN stdlib: Giving up on node n_5_18 WARN stdlib: Giving up on node
> n_5_13 WARN stdlib: Giving up on node n_7_8 WARN stdlib: Giving up on
> node n_16_9 WARN stdlib: Giving up on node n_6_18 WARN stdlib: Giving
> up on node n_14_19 WARN stdlib: Giving up on node n_16_4 WARN stdlib:
> Giving up on node n_8_14 WARN stdlib: Giving up on node n_3_9 WARN
> stdlib: Giving up on node n_2_16 WARN stdlib: Giving up on node n_11_1
> WARN stdlib: Giving up on node n_14_5 WARN stdlib: Giving up on node
> n_9_12 WARN stdlib: Giving up on node n_19_2 WARN stdlib: Giving up on
> node n_17_15 WARN stdlib: Giving up on node n_16_19 WARN stdlib:
> Giving up on node n_3_18 WARN stdlib: Giving up on node n_2_3 WARN
> stdlib: Giving up on node n_19_20 WARN stdlib: Giving up on node n_8_6
> WARN stdlib: Giving up on node n_14_12 WARN stdlib: Giving up on node
> n_8_1 WARN stdlib: Giving up on node n_3_20 WARN stdlib: Giving up on
> node n_14_7 WARN stdlib: Giving up on node n_20_19 WARN stdlib: Giving
> up on node n_11_19 WARN stdlib: Giving up on node n_4_1 WARN stdlib:
> Giving up on node n_18_3 WARN stdlib: Giving up on node n_2_17 WARN
> stdlib: Giving up on node n_8_8 WARN stdlib: Giving up on node n_10_18
> WARN stdlib: Giving up on node n_18_8 WARN stdlib: Giving up on node
> n_15_11 WARN stdlib: Giving up on node n_13_16 WARN stdlib: Giving up
> on node n_14_16 WARN stdlib: Giving up on node n_13_9 WARN stdlib:
> Giving up on node n_12_17 INFO stdlib: Waiting for nodes
> (Up/Down/Total): 35/158/193 - (still
> down: n_16_11,n_3_14,n_16_12)
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 580 sec.
> INFO whenAll: *: 'status[@value='UP']' fires INFO exp:
> Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 570 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 559 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 549 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 539 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 528 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 518 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 508 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 497 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 487 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 477 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 466 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 456 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 446 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 435 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 425 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 415 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 404 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 394 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 383 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 373 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 363 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 352 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 342 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 332 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 321 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 311 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 301 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 290 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 280 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 270 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 259 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 249 sec.
> INFO exp: Progress(0/0/35): 0/0/0 min(n_1_6)/avg/max (216) - Timeout:
> 238 sec.
> INFO exp: Progress(0/0/35): 0/3/10 min(n_1_6)/avg/max (216) - Timeout:
> 228 sec.
> INFO exp: Progress(0/0/35): 0/8/20 min(n_4_12)/avg/max (216) -
Timeout:
> 218 sec.
> INFO exp: Progress(0/0/35): 0/13/30 min(n_4_12)/avg/max (216) -
Timeout:
> 207 sec.
> INFO exp: Progress(0/0/35): 0/18/40 min(n_4_12)/avg/max (216) -
Timeout:
> 197 sec.
> INFO exp: Progress(0/0/35): 0/23/40 min(n_4_12)/avg/max (216) -
Timeout:
> 186 sec.
> INFO exp: Progress(0/0/35): 10/27/50 min(n_4_12)/avg/max (216) -
> Timeout: 176 sec.
> INFO exp: Progress(0/0/35): 10/31/50 min(n_5_16)/avg/max (216) -
> Timeout: 166 sec.
> INFO exp: Progress(0/0/35): 10/34/50 min(n_5_20)/avg/max (216) -
> Timeout: 155 sec.
> INFO exp: Progress(0/0/35): 10/38/60 min(n_5_20)/avg/max (216) -
> Timeout: 145 sec.
> INFO exp: Progress(0/0/35): 10/42/60 min(n_1_14)/avg/max (216) -
> Timeout: 134 sec.
> INFO exp: Progress(0/0/35): 10/47/70 min(n_1_14)/avg/max (216) -
> Timeout: 124 sec.
> INFO exp: Progress(0/0/35): 10/52/70 min(n_1_14)/avg/max (216) -
> Timeout: 113 sec.
> INFO exp: Progress(0/0/35): 10/55/80 min(n_1_14)/avg/max (216) -
> Timeout: 103 sec.
> INFO exp: Progress(0/0/35): 10/60/80 min(n_1_14)/avg/max (216) -
> Timeout: 93 sec.
> INFO exp: Progress(0/0/35): 10/62/90 min(n_1_14)/avg/max (216) -
> Timeout: 82 sec.
> INFO exp: Progress(0/0/35): 10/68/90 min(n_1_14)/avg/max (216) -
> Timeout: 72 sec.
> INFO exp: Progress(0/0/35): 10/72/90 min(n_1_14)/avg/max (216) -
> Timeout: 62 sec.
> INFO exp: Progress(0/0/35): 10/78/90 min(n_1_14)/avg/max (216) -
> Timeout: 51 sec.
> INFO exp: Progress(0/0/35): 10/80/90 min(n_1_14)/avg/max (216) -
> Timeout: 41 sec.
> INFO exp: Progress(0/0/35): 10/82/90 min(n_1_14)/avg/max (216) -
> Timeout: 30 sec.
> INFO exp: Progress(0/0/35): 10/84/90 min(n_1_14)/avg/max (216) -
> Timeout: 20 sec.
> INFO exp: Progress(1/0/35): 10/86/100 min(n_1_14)/avg/max (216) -
> Timeout: 10 sec.
> INFO exp: Progress(5/0/35): 10/87/100 min(n_1_14)/avg/max (216) -
> Timeout: -1 sec.
> INFO exp: ----------------------------- INFO exp: Imaging Process
> Done INFO exp: - 30 node(s) timed-out - See the topology file:
> 'topo_grid_timedout.rb'
> INFO exp: - 5 node(s) succesfully imaged - See the topology file:
> 'topo_grid_active.rb'
> INFO exp: ----------------------------- INFO Experiment: DONE!
> INFO ExecApp: Application 'commServer' finished INFO run: Experiment
> grid_2007_08_23_13_01_08 finished after 17:36 sangho at console.grid:~$
> ccat topo_grid_active.rb
> -bash: ccat: command not found
> sangho at console.grid:~$ cat topo_grid_active.rb # Topology name:
> topo_grid_active # # The following command creates a Topology wih the
> nodes that have successfully been imaged.
> #
>
> defTopology('topo_grid_active', [[5,7],[4,19],[1,9],[6,16],[7,17]])
>
>
> ----- Original Message ----- From: "Sangho Oh"
> <sangho at winlab.rutgers.edu>
> To: <orbit-user at winlab.rutgers.edu>
> Sent: Thursday, August 23, 2007 12:58 PM
> Subject: Re: ORBIT-USER: Grid problem
>
>
>> Right. I just noticed it.
>> Most of nodes were given up.
>> However, these nodes were usable yesterday, and +60% of nodes are
>> given up today.
>> It is quite strange.
>>
>> - Sangho
>>
>>
>>
>> ----- Original Message ----- From: "Haris Kremo"
>> <harisk at winlab.rutgers.edu>
>> To: <orbit-user at winlab.rutgers.edu>
>> Sent: Thursday, August 23, 2007 12:54 PM
>> Subject: Re: ORBIT-USER: Grid problem
>>
>>
>>> Looks like only one was actually imeged?
>>>
>>> H.
>>>
>>> On 8/23/07, Sangho Oh <sangho at winlab.rutgers.edu> wrote:
>>>>
>>>>
>>>
>>>> WARN stdlib: Giving up on node n_20_19 WARN stdlib: Giving up on
>>>> node n_12_14 WARN stdlib: Giving up on node n_17_16
>>>
>>>
>>>
>>>> INFO exp: -----------------------------
>>>> INFO exp: Imaging Process Done
>>>> INFO exp: - 1 node(s) succesfully imaged - See the topology file:
>>>> 'topo_grid_active.rb'
>>>> INFO exp: -----------------------------
>>>> INFO Experiment: DONE!
>>>> INFO ExecApp: Application 'commServer' finished
>>>> INFO run: Experiment grid_2007_08_23_11_53_46 finished after 12:49
>>>
>>
>
More information about the orbit-user
mailing list