wiki:Tutorials/m0SDN/cTutorial2

Tutorial 2: Multiple Controllers

In the OpenFlow model, traffic can be separated along many different boundaries and flows can be controlled by multiple controllers. In this particular example we will run two separate controllers. The first one will handle the production traffic, and run a commodity controller, SNAC. The second controller will manage the "experimental" bit-torrent traffic, and run a NOX controller. The data interfaces (eth0) of 4 nodes are connected to the OpenFlow switch (in this case NEC IP8800). The OpenFlow switch is pre-configured to speak to a controller at console.sb9.orbit-lab.org:6633.

The logical setup should look like:

Controllers

Splitter: Flowvisor - IP port 6633

The flowvisor tool serves as an intermediary between controllers. It splits traffic based on predefined rules. For this example we're running flowvisor on the SB9 console. While it is completely possible to compile your own flowvisor from scratch and install it in your home directory, a precompiled installation is located in /opt. In order to properly steer the flowvisor tool, you must populate a flovisor-config.d directory with files that configure the flowvisor behavior. This directory must be a subdirectory of the current working directory, as flow visor will create statistics files and logs in the current working directory. For the purposes of this demo we'll be using these files:

bittorent.guest
default.switch
routing.guest

The default file primes the flowvisor and contains only the following lines:

 # For switches that do not have a their own, specific config file
 Default: 1
 # Start numbering "default" switches at 10000
 Id: 10000

The routing.guest file directs production traffic to SNAC running on port 6634, it denies high numbered IP traffic from getting to port 6634:

Name: routing
ID: 1

Host: tcp:localhost:6634

# By default, with now "FlowSpace" param, this slice has
# permissions to affect all traffic
FlowSpace: deny: tp_src: 10001
FlowSpace: deny: tp_dst: 10001
FlowSpace: deny: tp_src: 10002
FlowSpace: deny: tp_dst: 10002
FlowSpace: deny: tp_src: 10003
FlowSpace: deny: tp_dst: 10003
FlowSpace: deny: tp_src: 10004
FlowSpace: deny: tp_dst: 10004
FlowSpace: deny: tp_src: 10005
FlowSpace: deny: tp_dst: 10005

# limit 10000 == at most 10000 packet_in's per second == infinity
FlowSpace: allow:   limit: 10000

Finally the bittorent.guest directs high numbered IP port traffic to a controller on 6635:

Id: 1001
Host: tcp:localhost:6635

FlowSpace: allow: tp_src: 10001 limit: 10000
FlowSpace: allow: tp_dst: 10001 limit: 10000
FlowSpace: allow: tp_src: 10002 limit: 10000
FlowSpace: allow: tp_dst: 10002 limit: 10000
FlowSpace: allow: tp_src: 10003 limit: 10000
FlowSpace: allow: tp_dst: 10003 limit: 10000
FlowSpace: allow: tp_src: 10004 limit: 10000
FlowSpace: allow: tp_dst: 10004 limit: 10000
FlowSpace: allow: tp_src: 10005 limit: 10000
FlowSpace: allow: tp_dst: 10005 limit: 10000

This setup of flowvisor will split along IP port boundaries. If we want to direct more traffic to the experiment, we simply manipulate the flow rules. To start the actual controller in user:

user@console.sb9:~$ /opt/flowvisor-0.4/flowvisor ptcp:6633
Mar 14 22:57:08|00001|rconn|WARN|tcp:localhost:6635: connecting...
Mar 14 22:57:08|00002|rconn|WARN|tcp:localhost:6635: connection failed (Connection refused)
Mar 14 22:57:08|00003|rconn|WARN|tcp:localhost:6635: connection dropped (Connection refused)

You might notice some connection failures for port 6635, since the controller isn't running yet.

Production: SNAC - IP port 6634

On console.sb9 SNAC is preconfigured to listen to port 6634. For the purposes of this example, SNAC will handle the production traffic and as such is not configured to do anything beyond the scope of a simple learning switch. The web interface for SNAC is accessible externally at https://sb9.orbit-lab.org (default credentials).

Experimental: NOX - IP port 6635

For demonstration purposes we will have this controller simply pass traffic. We could manipulate this traffic by configuring NOX to behave differently, or by running a completely different controller that listens on port 6635. Since our model is that "production" traffic is handled by a controller at port 6634 and experimental traffic is handled by a controller at port 6635, it is expected that the states of the port 6635 controller be transient.

Using my compiled nox I ran

./nox_core -i ptcp:6635 switch packetdump

The experimental Traffic

FIXMEThis process will probably be Scripted.

Our experiment setup is as follows (refer to diagram above):

  • Node1-1: Tracker 192.168.1.1:6969
  • Node1-2: Seeder 192.168.1.2:10001-10005
  • Node1-3: First downloader 192.168.1.3:10001-10005
  • Node1-4: Second downloader (starts 1 Minute later to pull from multiple sources) 192.168.1.4:10001-10005

A torrent file and data file were created on node1-1 and the respective components moved to each node using the control interface.

root@node1-1:~# dd if=/dev/zero of=./data.mp3 bs=100MB count=1
root@node1-1:~# btmakemetafile data.mp3 http://192.168.1.1:6969/announce
root@node1-1:~# scp data.mp3 root@node1-2:
root@node1-1:~# scp data.mp3.torrent root@node1-2:
root@node1-1:~# scp data.mp3.torrent root@node1-3:
root@node1-1:~# scp data.mp3.torrent root@node1-4:

We then start the tracker on node1-1 and when the process is running we see:

root@node1-1:~# bttrack --port 6969 --dfile tracker.log
192.168.1.4 - - [14/Mar/2010:19:44:46] "GET /announce?info_hash=%7Es%D0%B1%03%E5%02%BA%B6c_%7D%D6%DB%E1%A9%F5%CD%BC%DE&peer_id=M3-4-2--
7f9e2d5669d8&port=10001&key=8a1ab764&uploaded=0&downloaded=0&left=1000000000&compact=1&event=started HTTP/1.1" 200 33 "-" "Python-urllib/2.6"
192.168.1.4 - - [14/Mar/2010:19:44:47] "GET /announce?info_hash=%7Es%D0%B1%03%E5%02%BA%B6c_%7D%D6%DB%E1%A9%F5%CD%BC%DE&peer_id=M3-4-2--
7f9e2d5669d8&port=10001&key=8a1ab764&uploaded=0&downloaded=0&left=1000000000&compact=1&event=stopped HTTP/1.1" 200 33 "-" "Python-urllib/2.6"
...

The seeders and downloaders on the respective nodes output:

root@node1-2:~# btdownloadheadless --minport 10001 --maxport 10005 data.mp3.torrent

saving:         data.mp3 (953.7 M)
percent done:   0.0
time left:      checking existing file
download to:    /root/data.mp3

saving:         data.mp3 (953.7 M)
percent done:   0.2
time left:      checking existing file
download to:    /root/data.mp3
...
root@node1-4:~# btdownloadheadless --minport 10001 --maxport 10005 data.mp3.torrent
saving:         data.mp3 (953.7 M)
percent done:   0.0
time left:
download to:    /root/data.mp3
download rate:  0.00 K/s
upload rate:    0.00 K/s
download total: 0.0 M
upload total:   0.0 M

saving:         data.mp3 (953.7 M)
percent done:   9.5
time left:      5 min 55 sec
download to:    /root/data.mp3
download rate:  2497.60 K/s
upload rate:    665.95 K/s
download total: 91.5 M
upload total:   19.8 M
...
root@node1-3:~# btdownloadheadless --minport 10001 --maxport 10005 data.mp3.torrent 
saving:         data.mp3 (953.7 M)
percent done:   1.9
time left:      6 min 57 sec
download to:    /root/data.mp3
download rate:  2556.24 K/s
upload rate:    223.78 K/s
download total: 18.8 M
upload total:   2.5 M
...

Finally on the NOX controller we should see events like this:

NOX 0.8.0~full~beta (nox_core), compiled Mar 14 2010 16:21:17
Compiled with OpenFlow 0x97 (exp)
[(Giga-Byte Technology Co.,LTD.):84:9b:30>(Giga-Byte Technology Co.,LTD.):83:a0:5c:IP]([v:4hl:5l:60t:64]TCP cs:8a65[192.168.1.1>192.168.1.2]){44653>10001} seq:3830918239 ack:0 f:0x2
[(Giga-Byte Technology Co.,LTD.):83:a0:5c>(Giga-Byte Technology Co.,LTD.):84:9b:30:IP]([v:4hl:5l:40t:64]TCP cs:b76c[192.168.1.2>192.168.1.1]){10001>44653} seq:0 ack:3830918240 f:0x14
[(Giga-Byte Technology Co.,LTD.):84:9b:30>(Giga-Byte Technology Co.,LTD.):85:9b:64:IP]([v:4hl:5l:60t:64]TCP cs:8d0d[192.168.1.1>192.168.1.4]){45508>10001} seq:1710024440 ack:0 f:0x2
...

ASIDE: TESTS for connectivity

In case your experiment is failing, here are a few sanity checks.

The SNAC controller Web interface

Assuming your flowvisor is passing along high priority traffic to snac, the summary should look like the this:

Inter-node traffic

Once the data interfaces are brought up on each of the 4 nodes, they should be discovered by SNAC. Assuming we place them all in the same IP subnet, it should be possible to ping each node from every other node. A simple test can be done as follows:

for node1-i:

ssh root@node1-i
ifconfig eth0 up
ifconfig eth0 192.168.1.i
ping 192.168.1.(i-1)

where i runs from 1 to 4.

This should be done with independent sessions, as each session will start to display traffic information. Once completed SNAC should report an increase in production traffic, and 4 discovered hosts with the specified IPs.

High Numbered port Test

Start a fresh NOX controller instance on sb9 and enable packet dumps

root@console:/root/nox/build/src# ./nox_core -i ptcp:6635 switch packetdump

Attempt to connect to a high numbered port on a different node (using the Ip of the data interface):

root@node1-1:~# telnet 192.168.1.2 10001
Trying 192.168.1.2...
telnet: Unable to connect to remote host: Connection refused

If the traffic is split properly, your controller should register events similar to this:

[(Giga-Byte Technology Co.,LTD.):84:9b:30>(Giga-Byte Technology Co.,LTD.):83:a0:5c:IP]([v:4hl:5l:60t:64]TCP cs:8a65[192.168.1.1>192.168.1.2]){44653>10001} seq:3830918239 ack:0 f:0x2
[(Giga-Byte Technology Co.,LTD.):83:a0:5c>(Giga-Byte Technology Co.,LTD.):84:9b:30:IP]([v:4hl:5l:40t:64]TCP cs:b76c[192.168.1.2>192.168.1.1]){10001>44653} seq:0 ack:3830918240 f:0x14
Last modified 2 years ago Last modified on 09/11/14 16:58:03

Attachments (2)

Download all attachments as: .zip