[[TOC(Documentation/f*, depth=3)]] = Software Defined Networking = == Experimentation in ORBIT == Sandbox 9 is a great option for !OpenFlow/SDN experimentation. As shown in figure below, SB9 is built around an !OpenFlow capable switch, a Pronto 3290, with 11 experimentation nodes connected to it. Seven of these nodes (node1-1 through node1-7) have [http://netfpga.org/1G_specs.html NetFPGA 1G] cards, two (node1-8 and node1-9) have [http://netfpga.org/1G_specs.html NetFPGA 10G] cards, while the other two are general purpose ORBIT nodes that could be used as controllers or monitors. The switch provides the 'DATA' backplane for SB9 with the 'eth0' interfaces on each node and the 4 data ports on each of the NetFPGAs connected to it at the specific ports shown in the figure below. The control interfaces on the nodes (i.e., 'eth1' ) are connected to an external control switch not accessible to the experimenter (primarily used for node imaging and node access from console). || [[Image(http://www.orbit-lab.org/raw-attachment/wiki/Hardware/bDomains/cSandboxes/iSB9/SB9.jpg, 600px, align=center)]] || ||Figure 1: Overview of the SB9 setup. || === OpenFlow Operation Modes for P-3290 === The P-3290 can operate in 2 basic modes: as a L2 switching/L3 routing element using Pica8's PicOS protocol stack, or as a multi-layer virtual switch enabled by porting open vSwitch (OVS), where the control path is open to experimentation. In addition, since the U-Boot bootloader on the switch can load custom images via NFS/TFTP, one can load any compatible switch OS solutions (e.g., Indigo), presenting a third path for !OpenFlow experimentation. === A. Openflow in L2/L3 Mode === As of PicOS 2.0, the switch can mix data traffic between the !OpenFlow and L2/L3 networks. They call this enhancement “crossflow” mode. Individual ports can be designated either as legacy or crossflow port, where the control on the legacy ports is handled by PicOS L2/L3 stack. The control for crossflow ports can be left to one of local handling (by PicOS stack), by flow rules entered through CLI, or by establishing path to an !OpenFlow controller. || [[Image(P3290-Crossflow-HW-Structures.jpg, 400px, align=center)]]|| ||Figure 2: Hardware structures used to store flow rules in the TCAM and L2/L3 crossflow modes in P-3290 || There are two further distinct modes within the crossflow mode, L2/L3 mode and TCAM mode, that use different hardware lookup structures during forwarding. If you enable L2/L3 mode, all flows will be installed in the routing table and the FDB table in the hardware. If you enable TCAM mode, all flows will be installed in the TCAM table. In each of these sub-modes, there are limitations on what flow patterns can be specified - refer to the attached configuration guide for these constraints. Preparing the switch for !OpenFlow mode includes: assigning h/w resources for !OpenFlow flows, enabling !OpenFlow mode and enabling specific versions (e.g., 1.1, 1.2 or 1.3), setting ports as crossflow, and lastly adding the 'match' and 'action' rules for expected flow. As an alternative to locally introduced flow rules using the CLI, a remote !OpenFlow controller can be configured which will then receive !OpenFlow notifications (!PacketIns) for flows not handled locally. The following CLI session excerpt details the commands required to accomplish the above steps: {{{ -- 0. Load a 'clean' configuration to nullify changes made by previous experimenters -- -- 1. In configure mode, set tcam/routing table resources XorPlus# set interface max-acl-rule-limit ingress 400 [edit] XorPlus# set interface max-route-limit 6000 [edit] -- 2. Commit configuration and restart switch for resource allocation to take hold -- XorPlus# save running-to-startup Save done. [edit] XorPlus# exit XorPlus> request system reboot -- 3. In config mode, enable OpenFlow operation and set the working mode to L2/L3 or TCAM mode -- XorPlus# set open-flow [edit] XorPlus# set open-flow working-mode tcam-mode [edit] -- 4. Enable specific OF version, e.g., v 1.3 XorPlus# set open-flow allowed-versions openflow-v1.3 disable false [edit] -- 5. Set required ports in crossflow mode, with or without local control XorPlus# set interface gigabit-ethernet ge-1/1/1 crossflow enable true [edit] XorPlus# set interface gigabit-ethernet ge-1/1/2 crossflow enable true [edit] XorPlus# set interface gigabit-ethernet ge-1/1/2 crossflow local-control false [edit] -- 6. Add flow rules directly from CLI for ports with local control enabled XorPlus# set open-flow flow f1 match-field ethernet-destination-address 22:22:22:22:22:22 [edit] XorPlus# set open-flow flow f1 match-field vlan-id 100 [edit] XorPlus# set open-flow flow f1 match-field ip-destination-address 192.168.1.0/24 [edit] XorPlus# set open-flow flow f1 match-field ethernet-type 2048 [edit] XorPlus# set open-flow flow f1 action output interface ge-1/1/1 -- 7. Configure controller IP/port/transport for non-local control XorPlus# set open-flow controller contr-serv address 10.19.0.11 [edit] XorPlus# set open-flow controller contr-serv port 6633 [edit] XorPlus# set open-flow controller contr-serv protocol tcp [edit] XorPlus# commit Waiting for merging configuration. Commit OK. Save done. [edit] XorPlus# exit XorPlus> }}} === B. OpenFlow in OVS Mode === === C. OpenFlow through Network Boot (of custom image) === === Switching Modes using Web Service === Note: The web service is being upgraded to conform to the latest ORBIT Management Framework software, hence the switch will remain in the L2/L3 mode at all times. This means the OVS and NFS modes are presently unavailable. === Access to Switch CLI === You can access switch CLI through a ssh session, if the switch is in either of L2/L3 or OVS modes: {{{ ssh hostname/IP: sw-sb-09.orbit-lab.org/10.19.0.253 username: sb9user passwd: sb9passwd }}} Note: ssh access is restricted to from within the sb9 sandbox (incl. the sb9 console) and would therefore require the experimenter to have reserved SB9 to access the switch CLI. === Loading and Saving Switch Configurations === While in L2/L3 mode, the !XorPlus CLI under the above account grants complete control to configure the switch and also to load/store configuration files on the root file system. However, it is suggested that experimenters preserve their configs by copying out their configurations (using tftp, etc.) to external locations, and to load them back when required. Below are the relevant CLI commands with further details in the attached PicOS configuration guide from Pica8. Please exercise prudence and caution when saving or moving configuration files in/out of the root file system. {{{ -- retrieve config file from external location; scp can be used in place of tftp XorPlus> file tftp get remote-file basic.conf local-file /pica/config/sb9user/iamxyz-mm-dd-yyyy.conf XorPlus> configure -- load locally stored config file XorPlus# load iamxyz-mm-dd-yyyy.conf -- make any number of changes and commit ... -- save current configuration changes to specific file XorPlus# save iamxyz-latest.conf Save done. [edit] -- transfer out config file to safe location for later use XorPlus# exit XorPlus> file tftp put local-file /pica/config/sb9user/iamxyz-latest.conf remote-file updated.conf }}} === External References: === * [http://www.openflow.org/ OpenFlow] * [http://www.pica8.org Pica8's Pronto 3290 (now simply P-3290)] | [http://pica8.com/support/resources.php Installation/Configuration Guides] * [http://www.netfpga.org NetFPGA 1G and 10G cards] .