Here are thr steps to test Kamailio under load.

First of all lets describe our network setup:

The a user from extension 300X registered to Asterisk 1 initiates a call to an extension 400X registered at Asterisk 2. Kamailio is registered as a trunk to both Asterisk 1 & 2; which intercepts the call which load balances it to either Asterisk X or Y where they do some fancy pre-processing to current call before its received by the callee.

Now for our testing purposes, we needed to remove the effect on performance by Asterisk 1 & 2 so we installed SIPp on another host which generates calls and receives them.

Installation and Execution Steps

  1. Download and Modify SIPp to auto respond always and include OPTIONS packet as well (-aa broken?), edit src/call.cpp:
call::T_AutoMode call::checkAutomaticResponseMode(char * P_recv)
{
    if (strcmp(P_recv, "BYE")==0) {
        return E_AM_UNEXP_BYE;
    } else if (strcmp(P_recv, "CANCEL") == 0) {
        return E_AM_UNEXP_CANCEL;
    } else if (strcmp(P_recv, "PING") == 0) {
        return E_AM_PING;
    } else if ((strcmp(P_recv, "INFO") == 0) || (strcmp(P_recv, "NOTIFY") == 0) || (strcmp(P_recv, "UPDATE") == 0) || (strcmp(P_recv, "OPTIONS") == 0)
               ) {
        return E_AM_AA;
    } else {
        return E_AM_DEFAULT;
    }
}

Compile sipp-3.3.990 with RTP Support.

To run the test, from SIPp Box:

# sipp 10.254.1.30 -i 10.254.1.40 -sf uac.xml -aa -inf accounts.csv -l 10000 -r 1 -rp 1000 -trace_msg -trace_err -trace_stat
Parameter Description
10.254.1.30 target Kamailio's IP on the LAN A side (see network diagram)
-i 10.254.1.40 make sure to bind SIPp on this IP especially if we are using IP Authentication on Kamailio
-sf uac.xml use this scenario file that generates calls.
-inf accounts.csv use this input CSV file, this is where the [field0],[field1],[field2] and [field3] values are derived in uac.xml.
Edit this file accordingly in format: CallID;Kamailio LAN A IP;[authentication];Extension on Asterisk 2;Asterisk 2 LAN B IP;
-l 10000 run 1000 calls.
-r 1 -rp 1000 make one call per 1000ms (1 secs)
-trace_msg log all messages to a file (filename auto generated)
-trace_err log all errors to a separate file (filename auto generated)
-trace_stat generate a CSV file with statistics which is good for making graphs (default 1 minute interval)

Make sure to edit the accounts.csv, change 10.254.1.30 and 10.254.7.31 accordingly.

Make sure to edit the uac.xml, change Route:

<sip:10.254.1.30;r2=on;lr=on;nat=yes>,<sip:10.254.3.30;r2=on;lr=on;nat=yes>```
accordingly since sipp-3.3.990 can't reliably generate this header so we had to hard code this for now. 

You may run a SIPp on Asterisk 2 box to test higher concurrent calls (eg: testing more than 200 concurrent calls).

Lets shutdown Asterisk 1 & 2: 
```bash
# asterisk -rx "core stop now"

To run a server listening to incoming calls (server mode), run:

# sipp 10.254.7.30 -i 10.254.7.31 -sf uas.xml -aa -trace_msg -trace_err -trace_stat

Makes sure to edit the uas.xml to include the IP routes.

Now lets see how effective is Kamailio in this setup, here are the results I had:

Test Name Concurrent Calls Success Failed Dead Calls Retransmissions Average Response Time Average Call Rate Per Seconds
Test1 200 1000 0 0 3 2.52747 03.615000
Test2 300 998 2 5 252 3.15839 04.550000
Test3 400 993 7 12 1355 3.61512 13.049000
Test4 600 831 169 127 3554 4.05337 13.04900

We stop at Test 4 seeing Failed Calls spiked up at 169 calls, this was significant from our base capacity of 50 concurrent calls already.

Many thanks to Gohar Ahmed for helping me figuring most of the bugs.