Multi-Genenrator with real-time data logging

Multi-Generator (mgen) is an open source network utility capable of performing performance tests and measurements by generating and receiving UDP/TCP packets. This tool is already well documented under the ​Networks and Communication Systems Branch at US Navy Research.

An integrated version of mgen (version 5.02c) and Orbit Measurement Framework & Library (OML) has been built and tested in the Orbit environment. The following steps describe how to build and run this version of mgen.

Set up

1) Install the OML library and develpoment files.

apt-get update
apt-get install liboml2

While building mgen other dependencies may be required to complete the compilation. Simply search and install the missing libraries and continue compiling.

2) Download the source file for mgen and apply oml patch

wget https://downloads.pf.itd.nrl.navy.mil/mgen/src-mgen-5.02c.tgz
tar -zxvf src-mgen-5.02c.tgz
mv src-mgen-5.02c mgen
cd mgen

wget http://www.orbit-lab.org/raw-attachment/wiki/Tutorials/i1Net/mgen/mgen_oml.patch
patch -tb -p1 < mgen_oml.patch

3) Navigate to 'makefiles' directory and build.

cd mgen/makefile
make -f Makefile.linux mgen

4) use help for a listing of options

./mgen help
GPSSubscribe(): fopen() error: No such file or directory
GPSSubscribe(): fopen() error: No such file or directory
mgen: version 5.02c
mgen [ipv4][ipv6][input <scriptFile>][save <saveFile>]
     [output <logFile>][log <logFile>][hostAddr {on|off}
     [logData {on|off}][logGpsData {on|off}]
     [binary][txlog][nolog][flush]
     [event "<mgen event>"][port <recvPortList>]
     [instance <name>][command <cmdInput>]
     [sink <sinkFile>][block][source <sourceFile>]
     [interface <interfaceName>][ttl <multicastTimeToLive>]
     [unicast_ttl <unicastTimeToLive>]
     [df <on|off>]
     [tos <typeOfService>][label <value>]
     [txbuffer <txSocketBufferSize>][rxbuffer <rxSocketBufferSize>]
     [start <hr:min:sec>[GMT]][offset <sec>]
     [precise {on|off}][ifinfo <ifName>]
     [txcheck][rxcheck][check]
     [queue <queueSize>][broadcast {on|off}]
     [convert <binaryLog>][debug <debugLevel>]
     [gpskey <gpsSharedMemoryLocation>]
     [boost] [reuse {on|off}]
     [oml <server:port[,omlFile]>]
MgenApp::ProcessCommands() OnCommand(help) error
mgen: error while processing startup commands

The errors shown above are just informatory and can be ignored.

All the options above (with the exception of 'oml') are explained in details in the ​reference guide.

The 'oml' option specifies the oml server and port number to send and record events into an sql file. A specific filename can be given as well otherwise if the field is left blank the filename will default to 'oml_mgen_test.sq3'. Example

./mgen event "listen udp 5124" oml oml:3003,test_file

The table below describes the parameters that are recorded

Parameter	Type	Description
oml_sender_id	INTEGER	identify which node the recorded data came from
oml_seq	INTEGER	oml sequence number
oml_ts_client	REAL	oml client time stamp
oml_ts_server	REAL	oml server time stamp
"proto"	TEXT	type of traffic → UDP, TCP
"direction"	TEXT	direction of traffic → sender, receiver
"flowid"	INTEGER	mgen flow id
"seq_num"	INTEGER	sequence number for each flow id
"src_addr"	TEXT	source IP address
"src_port"	INTEGER	source port
"dst_addr"	TEXT	destination IP
"dst_port"	INTEGER	destination port
"data_len"	INTEGER	length of data

Simple OML recording

1) Refer to the reference guide for complete details on using formatted scripts and options to run mgen. In this section we'll only describe

2) To demonstrate the OML recording feature, setup mgen on two nodes and run a script with the input option. For this example node6-6 and node15-6 on the grid were used and the corresponding script is showing the table.

Node Id	Traffic Direction	Script contents
node6-6	sender	0.0 ON 0 UDP SRC 5000 DST 10.10.15.6/5000 POISSON [10 1024:1200] 2.0 ON 1 UDP SRC 5001 DST 10.10.15.6/5001 POISSON [8 1024:1200] 4.0 ON 2 UDP SRC 5002 DST 10.10.15.6/5002 POISSON [6 1024:1200] 6.0 ON 3 UDP SRC 5003 DST 10.10.15.6/5003 POISSON [4 1024:1200] 8.0 ON 4 UDP SRC 5004 DST 10.10.15.6/5004 PERIODIC [1 256]
node15-6	receiver	0.0 LISTEN UDP 5000-5004

The sender script file (udp_send_script.mgn) configures the node to transmit UDP traffic to the receiver node which is configured to listen for UDP packet on port ranging from 5000 to 5004.

The format of the sender script is shown below separated into fields.

Event time	Event	Flow id	Protocol	Source port	Destination IP and port	Traffic pattern
0.0	ON	0	UDP	SRC 5000	DST 10.10.15.6/5000	POISSON [10 1024:1200]

Likewise the receiver script (udp_recv_script.mgn) is condensed into a single liner

Event time	Event	Protocol	Port list
0.0	LISTEN	UDP	5000-5004

Check ​MGEN SCRIPT Format section of the reference page for an explanation of all the fields.

3) Run each script using the 'input' option of mgen and specify the oml options for recording.

node6-6> ./mgen input udp_send_script.mgn       oml oml:3003,udp_send_mgen

node15-6> ./mgen input udp_recv_script.mgn oml oml:3003,udp_recv_mgen

Each instance of mgen will start producing the text output. In the background it will also send packet infomation to the specified oml server and database. To quickly view the contents of the data files, run the following command from the console:

console> sqlite3 /var/lib/oml2/udp_recv_mgen.sq3 .dump
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE _senders (name TEXT PRIMARY KEY, id INTEGER UNIQUE);
INSERT INTO "_senders" VALUES('et',1);
CREATE TABLE "_experiment_metadata" (oml_tuple_id INTEGER PRIMARY KEY, oml_sender_id INTEGER, oml_seq INTEGER, oml_ts_client REAL, oml_ts_server REAL, "subject" TEXT, "key" TEXT, "value" TEXT);
INSERT INTO "_experiment_metadata" VALUES(1,NULL,NULL,NULL,NULL,NULL,'table__experiment_metadata','0 _experiment_metadata subject:string key:string value:string');
INSERT INTO "_experiment_metadata" VALUES(2,NULL,NULL,NULL,NULL,NULL,'start_time','1528726472');
INSERT INTO "_experiment_metadata" VALUES(3,NULL,NULL,NULL,NULL,NULL,'table__mp__udp_recv_mgen','1 _mp__udp_recv_mgen proto:string flowid:int32 seq_num:int32 src_addr:string src_port:int32 dst_addr:string dst_port:int32 data_len:int32');
CREATE TABLE "_mp__udp_recv_mgen" (oml_tuple_id INTEGER PRIMARY KEY, oml_sender_id INTEGER, oml_seq INTEGER, oml_ts_client REAL, oml_ts_server REAL, "proto" TEXT, "flowid" INTEGER, "seq_num" INTEGER, "src_addr" TEXT, "src_port" INTEGER, "dst_addr" TEXT, "dst_port" INTEGER, "data_len" INTEGER);
INSERT INTO "_mp__udp_recv_mgen" VALUES(1,1,1,3.50581999868154525756e-01,2.95809999999999961861e-01,'UDP',0,4,'10.10.6.6',5000,'10.10.15.6',5000,1044);
INSERT INTO "_mp__udp_recv_mgen" VALUES(2,1,2,4.55220999661833047866e-01,0.3979,'UDP',0,5,'10.10.6.6',5000,'10.10.15.6',5000,1172);
INSERT INTO "_mp__udp_recv_mgen" VALUES(3,1,3,0.585224999114871,5.2790399999999992886e-01,'UDP',0,6,'10.10.6.6',5000,'10.10.15.6',5000,1192);
INSERT INTO "_mp__udp_recv_mgen" VALUES(4,1,4,1.40533899888396263122e+00,1.489031,'UDP',0,7,'10.10.6.6',5000,'10.10.15.6',5000,1041);
INSERT INTO "_mp__udp_recv_mgen" VALUES(5,1,5,1.59353199973702430725e+00,1.614587,'UDP',0,8,'10.10.6.6',5000,'10.10.15.6',5000,1110);
INSERT INTO "_mp__udp_recv_mgen" VALUES(6,1,6,1.68116699904203414916e+00,1.62387099999999984234e+00,'UDP',0,9,'10.10.6.6',5000,'10.10.15.6',5000,1198);
INSERT INTO "_mp__udp_recv_mgen" VALUES(7,1,7,1.72220599837601184844e+00,1.66488099999999983324e+00,'UDP',0,10,'10.10.6.6',5000,'10.10.15.6',5000,1176);
INSERT INTO "_mp__udp_recv_mgen" VALUES(8,1,8,1.89545799978077411651e+00,1.838213,'UDP',1,0,'10.10.6.6',5001,'10.10.15.6',5001,1049);
INSERT INTO "_mp__udp_recv_mgen" VALUES(9,1,9,1.91608599945902824401e+00,1.858813,'UDP',1,1,'10.10.6.6',5001,'10.10.15.6',5001,1127);
INSERT INTO "_mp__udp_recv_mgen" VALUES(10,1,10,2.05955099686980247497e+00,2.144107,'UDP',1,2,'10.10.6.6',5001,'10.10.15.6',5001,1169);
INSERT INTO "_mp__udp_recv_mgen" VALUES(11,1,11,2.0766779966652393341e+00,2.269656,'UDP',1,3,'10.10.6.6',5001,'10.10.15.6',5001,1080);
INSERT INTO "_mp__udp_recv_mgen" VALUES(12,1,12,2.10336599871516227722e+00,2.269673,'UDP',1,4,'10.10.6.6',5001,'10.10.15.6',5001,1064);
INSERT INTO "_mp__udp_recv_mgen" VALUES(13,1,13,2.15250899642705917358e+00,2.269681,'UDP',1,5,'10.10.6.6',5001,'10.10.15.6',5001,1075);
INSERT INTO "_mp__udp_recv_mgen" VALUES(14,1,14,2.19350999966263771057e+00,2.269688,'UDP',0,11,'10.10.6.6',5000,'10.10.15.6',5000,1180);
INSERT INTO "_mp__udp_recv_mgen" VALUES(15,1,15,2.24396899715065956115e+00,2.269796,'UDP',1,6,'10.10.6.6',5001,'10.10.15.6',5001,1073);
INSERT INTO "_mp__udp_recv_mgen" VALUES(16,1,16,2.24748399853706359863e+00,2.269805,'UDP',0,12,'10.10.6.6',5000,'10.10.15.6',5000,1161);
INSERT INTO "_mp__udp_recv_mgen" VALUES(17,1,17,2.26098299771547317504e+00,2.269826,'UDP',1,7,'10.10.6.6',5001,'10.10.15.6',5001,1148);
INSERT INTO "_mp__udp_recv_mgen" VALUES(18,1,18,2.2951769977808,2.269834,'UDP',0,13,'10.10.6.6',5000,'10.10.15.6',5000,1164);
INSERT INTO "_mp__udp_recv_mgen" VALUES(19,1,19,2.30865999683737754821e+00,2.26984,'UDP',1,8,'10.10.6.6',5001,'10.10.15.6',5001,1128);
INSERT INTO "_mp__udp_recv_mgen" VALUES(20,1,20,2.32708599790930747985e+00,2.269847,'UDP',1,9,'10.10.6.6',5001,'10.10.15.6',5001,1171);
INSERT INTO "_mp__udp_recv_mgen" VALUES(21,1,21,2.35045599937438964843e+00,2.293127,'UDP',0,14,'10.10.6.6',5000,'10.10.15.6',5000,1153);
INSERT INTO "_mp__udp_recv_mgen" VALUES(22,1,22,2.44623499736189842224e+00,2.388898,'UDP',0,15,'10.10.6.6',5000,'10.10.15.6',5000,1166);
INSERT INTO "_mp__udp_recv_mgen" VALUES(23,1,23,2.58714799955487251281e+00,2.52982499999999976836e+00,'UDP',1,10,'10.10.6.6',5001,'10.10.15.6',5001,1173);