An attempt at Floodlight Development.

This page documents the (attempted) process of developing a version of Floodlight with FlowVisor-like slicing capabilities.

This is an on-going process - therefore this page will be updated frequently.

Quick links

Overview
Setup
Background
Approach - a diary-style log of the implementation. Following are some 'highlights' relevant to the architecture of the software being ported in general:

• Floodlight Module System
• Flowvisor FlowMap Overview
• Flowvisor Internals, Continued
• Floodlight packet processing chain
• Comparisons of the two

jump to bottom.

Overview.

Slicing is the SDN (in our case referring to OpenFlow) term for network virtualization. Specifically, A slice is a SDN controller and the network resources allocated to it by a hypervisor-like entity such as FlowVisor. With proper resource isolation and allocation, multiple controllers can coexist on a single physical network.

In the usual case, FlowVisor behaves as a proxy, intercepting and modifying the OpenFlow control messages between the switches and the multiple controllers so that each slice only sees a subset of the network.

Floodlight is a modularized OpenFlow controller, in that its functional components can be split into two major parts:

1. The core event dispatcher, and
2. The various event handlers.

The event handlers (Floodlight modules) are what define the behavior of swicthes (hub, learning switch, router, etc.). A more in-depth discussion of these components can be found ​here.

The key point here is that Floodlight's modular structure allows for multiple different handlers to coexist, with various (groups of) modules imposing different capabilities on the switches. If these modules controlled different groups of switches e.g. had their own bit of resource, each <module:resource> set can be viewed as a slice. Floodlight does not currently have this capability, so the objective of the process described here is to try to realize this feature.

Setup

Development is done on two VMs, one for Flowvisor and Floodlight source, and the other, Mininet for testing our code.

• VM1(10.50.250.2): 3-switch, 3-host Mininet topology pointed to external controller in VM2:

Topology:

h4   h5   h6
 |    |    | 
s1---s2---s3 

Mininet config:

# mn --topo=linear,3 --controller=remote --ip=10.50.250.17 --port=6633

• VM2(10.50.250.17): Flowvisor on port 6633, with a slice to point two switches to Floodlight instance on 6634

FlowVisor configuration:

# fvctl createSlice fl-1 tcp:localhost:6634 foo@sampledomain.org
# fvctl addFlowSpace 00:00:00:00:00:00:00:02 1000 any "Slice:fl-1=7"
# fvctl addFlowSpace 00:00:00:00:00:00:00:01 1000 any "Slice:fl-1=7"

Floodlight config alteration (in src/main/resources/config.properties):

net.floodlightcontroller.restserver.RestApiServer.port = 8088
net.floodlightcontroller.core.FloodlightProvider.openflowport = 6634

The REST API port is changed from 8080 to prevent conflict with Flowvisor's services.

Background.

The general flow of operation with respect to Flowvisor (in terms of components) is the following:

1. The Acceptor (OFSwitchAcceptor) listens for all OpenFlow message types (OFTypes) and all switch join/leave events.
2. When a new switch joins, the Acceptor hands the switch's connection to a Classifier (FVClassifier).
3. The Classifier fetches all slices associated with the switch and launches a Slicer (FVSlicer) per module.
   • mappings of DPID to slices are found through configurations (the Flowmap)
   • modules associated with the control module (the primary module defining the behavior of a switch for that slice) may be grouped together in a slice.
4. Each Slicer sets up event dispatching for the modules associated with a slice.

Where the messages go once Flowvisor receives a message are determined by the FlowMap. the origin of a message can be determined several ways, one being checking the OFType (as some messages are sent by switches but never controllers, and vice versa).

Flowvisor implements all of the needed controller functions, so a good chunk of it is life-support (event handling mechanism, timers, OpenFlow signaling, etc.). Therefore, we need to be aware of what Floodlight already provides, and what can/needs-to be pulled in from Flowvisor.

A Floodlight module, in general:

• can subscribe to switch join/leave events. They are notified of joins after the switch sends a features reply.
• can subscribe to all OFTypes, and once processed, can choose to pass it down the processing chain to other modules or drop it.
• the modules themselves cannot control which modules receive the messages if it choses to pass them downstream.

Approach (and implementation).

Since Floodlight source is available on github, we fork it and work from there.

(6/12):

We generally believe that the following parts will need to be implemented:

1. A broker module to register with the core module for all events - the core module can notify this module of OpenFlow messages and new switches.
2. A component that provides the IFloodlightProviderService interface to the rest of the modules. To the rest of the modules, the broker module will appear to be the core controller module.
3. Flow-mapping mechanism - it is possible that the components needed for a working FlowMap can be taken directly from Flowvisor, since it is relatively independent from the rest of the code base (according to Ali).

Therefore, the module is both a module in Floodlight's usual sense as a subscriber to the controller core, but also a controller that services the rest of the modules. The rough parallels are:

• the main broker module's behavior as a listener roughly corresponds to the OFSwitchAcceptor
• the new IFloodlightProviderService interface (call it "FVController"?) and receive() functions correspond to the FVSlicers and FVClassifiers
• the modules correspond to the multiple controllers

The FVController interface should be responsible for keeping track of which modules are associated with which slice, and which messages reach what (with help from the FlowMap). This means that the modules must register with it, as opposed to the core controller defined in Controller.java.

So, for the meanwhile, the next steps include:

• isolating out the FlowMap code in Flowvisor
• mapping out the module registration mechanisms in Floodlight
• finding a "nice" way to make the modules register with FVController.

(6/13):[Floodlight Module System]

A (not-so) quick revisit of the module loading system in Floodlight was done to figure out how exactly a module learns to subscribe to the main controller module.

Starting at the top, in the init() function of each module is almost always this line:

floodlightProvider = context.getServiceImpl(IFloodlightProviderService.class);

Where context is an instance of FloodlightModuleContext, which contains a mapping between the interfaces and the services providing them. getServiceImpl(), in the case above, returns the service providing the IFloodlightProviderService interface, which is the core controller.

FloodlightModuleLoader is the actual class responsible for populating FloodlightModuleContext's mapping; Recalling, from Main.java:

FloodlightModuleLoader fml = new FloodlightModuleLoader();
IFloodlightModuleContext moduleContext = fml.loadModulesFromConfig(settings.getModuleFile());

Internally, for each module exporting a service, in initModules() FloodlightModuleLoader calls getServiceImpls(), an IFloodlightModule function that lets modules define the services that they export. It then uses FloodlightModuleContext's addService() in order to fill its mapping.

Returning to the controller mapped to IFloodlightProviderService.class, we see that it is a service exported by the module FloodlightProvider. It is the first module to be loaded by the module loading system, and therefore the mapping to the controller is there for the rest of the modules to register with.

This leads us to a potential way to place a module between the controller and the rest of the modules:

• Re-name the real IFloodlightProviderService to something else
• Implement a module that subscribes to the re-named controller service
• Implement a FVController from IFloodlightProviderService
• export this as a service with key IFloodlightProviderService.class

This shouldn't require any change on the other modules (except Controller.java).

(6/14):

A preliminary attempt at trying the above (do not follow if you ever come across this, it is an experiment):

create copies of core files:

• FVController, proxy version of Controller - for now a blank IFloodlightProxy implementation
• IFloodlightProxy, a copy of IFloodlightProviderService to be used by FVController

re-wired files:

• FVController implements IFloodlightProxy, with FVAcceptor as its version of FloodlightProvider

For the following files, 'rewiring' refers to swapping out IFloodlightProviderService with IFloodlightProxy.

• ForwardingBase, Forwarding (the generic forwarding components loaded by default)

additional changes:

• Add FVAcceptor to /floodlight/src/main/resources/META-INF/services/net.floodlightcontroller.core.module.IFloodlightModule
• Have FVAcceptor subscribe to IFloodlightProviderService as its floodlightprovider
• Incorporate FloodlightProvider's functions into FVAcceptor (for now, brutely drop in, just exporting IFloodlightProxy service)

This process forces the forwarding component to register with FVController, as opposed to the core module. The core module still takes care of OpenFlow signaling e.g. echo keepalives and the initial handshake, so any switches will still connect properly and keep the connection alive.

(6/18):

After much tweaking and code-tracing with log.debug outputs, a (seemingly) working proxy was implemented.

• FVController: implements its own bare-bones versions of:
  • updateSwitch: track list of active switches
  • add/removeOFMessageListener
  • add/removeOFSwitchListener
  • getSwitches: return immutable list of active switches
  • handleOutgoingMessage
  • injectOfMessage
  • handleMessage
  • init: initialize class variables

• OFSwitchImpl:
  • a setFloodlightProxy setter function
  • modified write() functions that call either FVController or Controller's handleOutgoingMessage() depending on OFType:

    switch (m.getType()) {
        case PACKET_OUT:
        case FLOW_MOD:
        case PORT_MOD:	
            this.floodlightProxy.handleOutgoingMessage(this, m, bc);
            break;
        default: 	
            this.floodlightProvider.handleOutgoingMessage(this, m, bc);
    }
    

• Controller: add functions to set the floodlightProxy in OFSwitchImpl
• FloodlightProvider: fetch and set Controller's FVProxy in init()

  controller.setFVProxy(
      context.getServiceImpl(IFloodlightProxy.class));
  

(6/20):[Flowvisor FlowMap Overview]

The FlowMap is one of Flowvisor's central components that must be ported. The first step, as always, is to understand how its components fit together. Hopefully there will be time for a clear re-write of the banter to follow…

First up, some terms:

• Flow rule : Mappings of Packet header pattern to actions (DROP, FLOOD, FORWARD, etc.)
• FlowSpace : A collection of flow rules
• FlowMap : A uniform interface and structure for manipulating and holding a FlowSpace. A FlowMap also maps a FlowSpace to the proper slices.

A flow rule is defined by a matching criteria (FVMatch.java → OFMatch.java) and an action (OFAction.java derivative). The "traditional" OFMatch construct doesn't do matching on DPID, so it is extended by FVMatch to do so. Flowvisor v0.8.3 supports two types of FlowMaps, Linear and Federated. FlowMaps are implementations of the Java Interface FlowMap.java. The enum MatchType defines how each header pattern of a (Flowvisor-defined) rule and the received rule patterns relate, in semi-set theoretic terms:

public enum MatchType {
	UNKNOWN, // not yet calculated
	NONE, // no overlap between A and B
	SUBSET, // match A is a subset of B
	SUPERSET, // match A is a superset of match B
	INTERSECT, // match A and B overlap, but neither subsumes the other
	EQUAL; // match A and B are exactly the same
}

The !Matchtype is used to describe Intersections, descriptions of how two FlowEntries relate. An intersection takes the form of a FlowEntry built by choosing the more heavily constrained pattern for each header field and DPID(s). FlowIntersect.java implements the representation of the intersection, and FlowEntry defines matches(), which compares two FlowEntries to produce the intersection. Comparisons are implemented in FlowTestOp.java.

(6/22):

Some thoughts from reading Flowvisor code.

• FVConfig interfaces database access operations pertaining to Flowspaces, slices, switches, and Flowvisor itself. The DB-related classes are all named *Impl.
• The handling of messages received by Flowvisor are defined within the messages themselves, as part of the implementation of Classifiable and Sliceable interfaces.
• The static flow pusher and related components may be eventually useful for these configuration aspects.

(6/25):[Flowvisor Internals, cont'd]

Today's conference call elucidated quite a bit about the workings of the FlowMap and how it fits in with the rest of the system.

Item 1: the usage of config databases.
In Flowvisor, the FlowMap is coupled to the database, but only loosely - the specific implementations are, indeed, hidden behind FVConfig. Furthermore, the database is only accessed when:

• an FVClassifier initializes connections to controllers (see FVClassifier.java:751)
• the flowspace changes.

in short, the database is a convenience item for storing configurations, and for initial purposes of this project, something that can likely be dealt with later.

Item 2: FlowMap internals.
The two implementations of interface FlowMap process the flowspace very differently.

The Linear FlowMap takes a literal copy of the flowspace; to save space, the flowspace is carted up by DPID, and the piece representing the 'correct' DPID e.g. matching that of the incoming message, is processed.

The Federated FlowMap manipulates references. If multiple FlowMaps existed, they would all point to the same flowspace. This FlowMap relies on a global bitset, indexed by FlowEntry ID, to keep track of entries that are part of a intersection (1=member, 0=otherwise). Starting with DPID, each field of a OFMatch is taken and compared to those of the flow entries. Each field is stored in its own hashmap, keyed by value. Each field comparison should narrow the number of 1's in the global bitset. At the end, the entry with the highest priority is chosen.

The goal of the Federated FlowMap is speed - many of its operations are kept to bitwise ops, and storage, hashmaps. Slower operations that involve loops e.g. IP address matching, are kept until later when the set of entries to work with is smaller.

Item 3: Next steps.
Now that we are pretty clear on the inner workings of Flowvisor, some next steps to do are:

• get flowspace working, with config file (ignoring databases for now)

to get this working, we need:

• JSON parsing in Floodlight
• ability to load configs to a module
• exception handling for Flowvisor-related errors.

(6/28)

Not all information used by Flowvisor will be applicable to the Floodlight port. Looking through the Flowvisor configuration file (can be saved to file running fvctl dumpConfig [filename] against a running Flowvisor), our initial guesses say that the following do not apply for each component:

• "Slice": controller hostname, port (we are working with modules, not external controllers)
• "flowvisor": most of the information kept for flowvisor can be taken from Floodlight e.g. API and listen port, versioning, and host.
• "FlowSpaceRule": this is probably the only unchanged component.

Config file parsing in Flowvisor happens in FVConfig:readFromFile(). Java reflection is used to then configure each component e.g manipulate database contents relevant to each.

(6/30)

With the database components removed, we need to re-build the components that parse the Flowvisor configs and initialize everything (FlowMap definitions, slices, etc.). FVConfSetup.java is intended for this purpose. So far, it uses Jackson (used in Floodlight for the Static Flow Pusher) to parse the Flowvisor configuration file to initialize the FlowMap (linear for now, since it is simple than the federated).

(7/2)

• A Classifier knows about the Slicers associated with the DPID it is associated with. This is kept in a Java HashMap.
• A Slicer knows the name, controller/TCP port, and switch ports associated with a slice.
• Each FlowSpaceRule entry (flow entry) stores which DPID and slice (by name and permissions, via SliceAction.java) it is associated with.

Unlike the FlowMap, a Slice does not seem to be a concrete object, but a collection of information the Slicers and Classifiers access using FlowSpaceUtil. We create a new configuration option, "module_name," to replace the controller information stored in a "slice". For now we keep the old parameters for each Slice, but modified:

         "controller_hostname": "localhost",
         "controller_port": 0,
         "module_name": "forwarding",                 

Another way to do this would be to re-use the existing fields (e.g. "controller_hostname") to hold the module's name, but that might become rather confusing despite keeping the syntax compatible with that of Flowvisor's configuration file. In any case, we need an easily manipulated mapping of DPIDs to slices, and slices to their modules, where modules are specified by the names returned by getName().

(7/3)

Since we don't rely on storage (for now), the same class handling the initialization of the Flowvisor components may also provide the functions of FlowSpaceUtil e.g. supply the means for the proxy module to access slice configurations.

• in FVAcceptor, switch DPIDs may be lifted when the switch joins.
• the origin of the message can be (mostly) determined by OFP message type (OFType).

For now, a sane goal is probably to have the proxy identify which slice a message belongs to based on the contents of the configuration file.

(7/7)

To tie back to the proxy module:

• receive() gets all IO Events.
• Added/RemovedSwitch() gets all switch joinleave events.

The actions that must occur at reception of an IO event include, as per Flowvisor documentations:

• identification of origin - switch or a slice?
• if from a switch, which slice(s) it belongs to
• if from a slice, which switch(es) they apply to

As of now, we assume that there is one module per slice (forwarding, switch, hub…).

(7/10):[Floodlight packet processing chain]

The following function calls are made when Floodlight processes a OF Message from a switch:

Where the colors indicate the class membership of the function:

• green: Controller.java
• lavender: IOFMessageListener.java
• blue: IOFSwitch.java
• yellow: Netty.channel

injectOFMessage() is a sanity check function that modules may call to keep their changes to any messages sane. receive() may be called multiple times in this process chain, but this does not result in a logical loop.

The flow of event handling for messages, looks like the following in the modified version of Floodlight:

   FVAcceptor        FVController        Modules        OFSwitchImpl       OFSwitchImpl       Netty.channel.Channel    
   receive()  ->   handleMesssage() ->  receive()  ->     write()    ->    write(List<>) ->         write() 

(7/11)

The following shows the modifications (shown in pink) added to the message processing chain shown above.

The grey dotted box containing receive() and write() represents a single module. We avoid making changes to the existing modules. The box containing the pink elements represents the part of the proxy module sitting between the main controller module and the rest; the other pink boxes show the proxy module's components that sit between the modules and the switches (involving modifications to OFSwitchImpl).

(7/19)

1. A major cleanup was done to the base proxy code. We essentially build analogous classes to the core components:

core		proxy
module	FloodlightProvider	FVProxyProvider
service	IFloodlightProviderService	IFloodlightProviderService
controller	Controller	FVProxyController

FVProxyController is a derived class of Controller - meaning it has all of the functions of the core controller class, plus any features to be added in. We simply replace the original core controller module with the proxy module. This approach has two (big) benefits:

1. We no longer have two separate services (IFPS and IFPP) that the core classes would have to consider. This means the changes that were made in the original code base to shuffle between the proxy service and the FloodlightProvider are not needed.
2. We can swap between normal and proxy mode by merely changing the entries in the config file.

The overall structure of Floodlight, then, returns to the original form e.g. like in (7/10).

In addition to these changes, we have the following:

• net.floodlightcontroller.flowvisor : FlowMap and Flowspace-related code
• net.floodlightcontroller.proxy : classes interfacing proxy and Flowvisor-related components

2. Organization of branches.

To keep things organized, we work on two main branches.

• proxy-edge-0.82 : main branch where changes are made
• proxy-stable-0.82 : once the changes in proxy-edge look ok, they go here (major save points).

(7/21)

Not much progress in the past few, which needs to be rectified.

The processing chain for incoming (from switch) messages seems to be simpler; we only need what we already have to get the incoming components working (finding matches out of flowspace entries and sending to the proper module).

Next steps:

• find way to map slice to module (slice name string → module name → module reference).
• begin porting flowvisor message classes.
• figure out how the message.actions classes fit into everything else.

(7/23)

Since we decided that the messgae processing scheme used in Flowvisor is the way to go (e.g. each message is its own class, and implements sliceFromController() and classifyFromSwitch() for modification according to the Flowspace), the plan is to implement the FVClassifier/Slicer interfaces available to these message classes.

Other points from discussion:

• Floodlight's native topology discovery should be used for Flowvisor's topology controller functions.
• The FVClassifier's slicerMap can be used as a quick-fetch mechanism for the slices associated with a DPID, but with the faster Federated FlowMap this should not be an issue.
• The modifications of the messaged actually happen in message.actions.*

(7/28)

There were several e-mail exchanges for clarifying how the project should go on.

• IFloodlightProvider versus module virtualization. Module virtualization implies usage of a same module by multiple slices, whereas the former allows usage of the IFloodlightProvider (Controller.java) by multiple users. As a method to do this, Controller.java would have access to all modules that the module loader finds necessary (according to the configurations file). Each slice is a abstraction that restricts a user to a subset of this list of modules.

(7/30) [Comparisons]

It seems to be a good time to step back and look at the forest, as things were getting stalled due to confusion.

• Flowvisor keeps track of switches based on incoming TCP source port. This is a pretty obvious but important point¹, which allows Flowvisor to differentiate message sources and have them go to the right classifier/slicer early on, even before having to check the OpenFlow information (e.g. DPID).

• Floodlight's channel handler (OFChannelHandler), a subclass of Controller, accepts all incoming connections, much like Flowvisor's OFSwitchAcceptor.

• Controller also handles (as its name suggests) many controller-related functions, such as taking care of the initial handshake and keep-alives (ECHO_REQUEST/REPLY). In Flowvisor, each FVClassifier and Slicer handle both of these tasks individually, with the former doing so for switches, and the latter, for each controller.

• Flowvisor does not allow the virtualization of controllers e.g. allow two slices to associate with the same controller (See DuplicateControllerException, thrown by SliceImpl when creating slices).

Gathering these points together, we can try to come up with processing scheme that should be used.

• As connections come into OFChannelHandler, a "classifier" can be spawned to track each incoming TCP connection.

• a check can be done at the slice creation stage (when configurations are being read) to avoid slices associating with the same modules.

• Each slicer in Floodlight would is a dispatch queue containing user-chosen services.

• Keep-alives and handshakes can be tracked by Controller (possibly).

This gives us two main choices of architecture:

1. A bare-bones IFloodlightProvider (almost just OFCHannelHandler) catches incoming connections, and passes it on to instances of more fully implemented IFloodlightProviders (Classifiers). Each have their own group of slicers and messaging to manage.

2. A relatively fully-featured IFloodlightProvider decides which messages go to which group of slicers, based on incoming DPID. The classifiers are just abstractions within the IFloodlightProvider.

The first case allows each IFloodlightProvider to be simpler in slice management, but guarantees that if n switches are present, n IFloodlightProviders are needed, which may get messy. The second case guarantees a complicated IFloodlightProvider, but probably less overall modifications. Both probably suffer scalability issues if done carelessly: for 1 this is the number of IFloodlightProviders and for 2 the large hashMap of DPID to slicers.

(7/31-8/1)

Further comparisons suggest that classifiers are essentially analogous to Floodlight's OFSwitchImpl. Treating each OFSwitchImpl as a Classifier seems to be a good way to seamlessly incorporate the slicing portion of Flowvisor.

(8/7)

The errors in the org.flowvisor.message.* packages that were moved from Flowvisor are resolved from a compilation standpoint, though this does not mean (at all) that there are no errors in the logic. A new FVlog class was improvised using slf4j.Logger, and a big guess was taken that the LLDP and FlowDB components won't really be needed. To maintain logical parallels, The IFloodlightProvider implementation was renamed FlowVisor, with appropriate interfaces needed by the message packages. Now these components need to be 'wired in', e.g. called in the right locations.

—-

home

—-

^{1. It wasn't for me until I realized it.}