Wednesday, August 17, 2011

HAProxy Load Balancer with SSL Offloading

In my previous blog, where I mentioned how to use HA Proxy( – Open source Load balancing Solution) based load balancer for Exchange 2010 (Link to Post), I had a few requests on how to do the SSL offloading as well. If you have followed that guide, you have your SSL certificates hosted on the Exchange Hub CAS servers.
Microsoft does suggest to use the SSL on the boxes, but you can offload it as well, please remember that I am writing this for a generic https server and we will be using 2 open source products.
We will use HA Proxy for the Load balancing and the Pound for the SSL offloading. Now, for this purposes, I have used a single box, but for production machines, you may want to make a HA Pair of Linux box, which I will cover in a different post.
I have been running HA Proxy in the exchange environment and it is great, the only thing, I could ask for is source port based persistence Smile.
Anycase, back to the topic. Here is how the architecture will look like.

As you can see the Pound will do the SSL offload and send the traffic to HA Proxy and HA Proxy will do the load balancing.
Don’t get me wrong guys, pound is capable of sending the traffic directly to the servers, but then you will not be able to use the cool features of ha proxy like cookie persistence, so on and so forth.

In our test environment, here is what we will do.
Client Facing IP:
Server IP Address:,
The Linux Box is

First, install Ubuntu (  on the box.
Add the as a Virtual IP on the box (If you are using a Active standby setup, this will be the floating IP)
Edit the interfaces script in Ubuntu)
vi /etc/network/interfaces

auto eth0
iface eth0 inet static
auto eth0:1
iface eth0:1 inet static

Restart the Networking subsystem
/etc/init.d/networking restart

Once that is done, the IP addresses will be shown when you execute “ifconfig –a” command.

Then install the following packages

   1: sudo apt-get install haproxy pound

This will install the haproxy and pound,

Now, lets roll

Configuring pound:

In order to configure pound, you will need to edit the config file using vi (or editor of your choice)

vi /etc/pound/pound.cfg

The file should look like the following. Please note, that I have installed the server certificate as PEM on the system.

## global options:
User		"www-data"
Group		"www-data"
#RootJail	"/chroot/pound"
## Logging: (goes to syslog by default)
##	0	no logging
##	1	normal
##	2	extended
##	3	Apache-style (common log format)
LogLevel	1
## check backend every X secs:
Alive		30
# Creating Listener
Port    443
Cert    "/etc/pound/testdom.pem"
Client  20
HeadRequire "Host:.**"
Port 80

So, as you can see, it’s a generic configuration and then it is using the 127.x.x.x subnet to talk to the HA Proxy, the HAProxy will be listening on the particular IP and port.

After the configuration is done, go ahead and restart pound.

/etc/init.d/pound restart

The HA Proxy configuration will look like this

   1: global
   2:         #uid 99
   3:         #gid 99
   4:         daemon
   5:         stats socket /var/run/haproxy.stat mode 600 level admin
   6:         maxconn 40000
   7:         ulimit-n 81000
   8:         pidfile /var/run/
   9: defaults
  10:         mode    http
  11:         contimeout      4000
  12:         clitimeout      3600000
  13:         srvtimeout      3600000
  14:         balance roundrobin
  15: listen  ServerSSL
  16:         mode    tcp
  17:         option  persist
  18:         balance leastconn
  19:         stick-table type ip size 10240k expire 15m
  20:         stick on src
  21:         server Server1 weight 1 check port 80 inter 5000 rise 2 fall 3
  22:         server Server2 weight 1 check port 80 inter 5000 rise 2 fall 3
  23:         option redispatch
  24:         option abortonclose
  25:         maxconn 40000
  26: listen  stats :7000
  27:         stats   enable
  28:         stats   uri /
  29:         option  httpclose
  30:         stats   auth username:password

This might not seem like worth it, as this is for a single HTTPs client, but let me go ahead and do the same thing for exchange.

Here is how the Pound configuration and HA Proxy configuration will look for Exchange 2010, with SSL offload.

Pound and HAProxy for Exchange 2010

Pound Configuration:

   1: ## Pound Configuration
   2: User        "www-data"
   3: Group        "www-data"
   4: ## Logging: (goes to syslog by default)
   5: ##    0    no logging
   6: ##    1    normal
   7: ##    2    extended
   8: ##    3    Apache-style (common log format)
   9: LogLevel    1
  10: ## check backend every X secs:
  11: Alive        30
  12:               ListenHTTP
  13:                   Address
  14:                   Port    80
  15:                   Client  10
  16:               End
  17:               ListenHTTPS
  18:                   Address
  19:                   Port    443
  20:                   Cert    "/etc/pound/mailserver.pem"
  21:                   Client  20
  22:               End
  23:               Service
  24:                   HeadRequire "Host:.**"
  25:                   BackEnd
  26:                       Address
  27:                       Port    80
  28:                   End

HA Proxy Configuration:

   1: global
   2:         #uid 99
   3:         #gid 99
   4:         daemon
   5:         stats socket /var/run/haproxy.stat mode 600 level admin
   6:         maxconn 40000
   7:         ulimit-n 81000
   8:         pidfile /var/run/
   9: defaults
  10:         mode    http
  11:         contimeout      4000
  12:         clitimeout      3600000
  13:         srvtimeout      3600000
  14:         balance roundrobin
  15: listen  Exchange2010-HTTP-HTTPS
  16:         mode    tcp
  17:         option  persist
  18:         balance roundrobin
  19:         stick-table type ip size 10240k expire 30m
  20:         stick on src
  21:         server HC-CAS1 weight 1 check port 80 inter 5000 rise 2 fall 3
  22:         server HC-CAS2 weight 1 check port 80 inter 5000 rise 2 fall 3
  23:         option redispatch
  24:         option abortonclose
  25:         maxconn 40000
  26: listen  Exchange2010
  27:         bind,
  28:         bind,
  29:         bind,
  30:         bind
  31:         bind
  32:         mode    tcp
  33:         option  persist
  34:         balance roundrobin
  35:         stick-table type ip size 10240k expire 30m
  36:         stick on src
  37:         server HC-CAS1 weight 1 check port 80 inter 5000 rise 2 fall 3
  38:         server HC-CAS2 weight 1 check port 80 inter 5000 rise 2 fall 3
  39:         option redispatch
  40:         option abortonclose
  41:         maxconn 40000
  42: listen  stats :7000
  43:         stats   enable
  44:         stats   uri /
  45:         option  httpclose
  46:         stats   auth username:password

Please note, that you will still have to make those changes on the Exchange server as noted in my earlier blog post.

If you need explanation on any of the configuration components, do ask (for the HA proxy they are all explained in my previous blog post)

I hope you find this informative and would like to thank you for reading.

Juniper Aggregate Interfaces (LACP/No LACP)

Last week, I was implementing link aggregation on Juniper EX Series switches (EX4200). I was moving them off HP switches and putting them into Juniper.
We were moving VMWare ESX boxes, NetApp and some physical boxes, though the configuration is very simple, I would like to mention how to do those with these boxes.
Points to remember:
  • You will explicitly need to create “Aggregated Ethernet” interfaces on juniper
  • You cannot have more than 64 AE interfaces on a single Virtual Chassis
  • You cannot change the algorithm for load balancing between the multiple links. (If you are interested in the Junipers LAG algorithm read this )
Now having said this, lets look at the common steps, you need to create the Aggregated Ethernet interfaces on the Juniper.
root@EX4200#set chassis aggregated-devices ethernet device-count 4
The above command will create 4 “ae” interfaces, after you commit, you should see ae0 – ae 3 created. You can have any value between 1 and 64 in the place of the 4.
Please also remember that, the individual interfaces should have absolutely no configuration before they can be added to the aggregation. In our example, we have taken interfaces ge-0/0/0 and ge-0/0/1
root@EX4200#delete interfaces ge-0/0/0 unit 0
root@EX4200#delete interfaces ge-0/0/1 unit 0
You may chose to completely delete the interface itself, so all the configuration under the interfaces (like mtu) will be cleared.
At this point, you will have some thing similar
chassis {
aggregated-devices {
ethernet {
device-count 4;
ge-0/0/0 {
ge-0/0/1 {

So, now that we have created the place holders for the aggregated interfaces, we will look into LACP in a little bit more detail.

Please remember that LACP is a negotiation protocol. It’s a standard protocol comparable to Cisco PAgP, which was later standardized into LACP. Also, remember that both ends must support LACP for the bundling to work using LACP.

ESX and juniper EX:

When creating the Aggregated interface, remember that the ESX (version 4) doesn’t support LACP, so we will be doing a generic aggregation. In this example, the ESX ports are vmnic2 and vmnic3 and the Switch ports are ge-0/0/0 and ge-0/0/1

  • Add the multiple networks in a vSwitch
  • Set Load Balancing as “Route based on IP Hash”
  • Network Failover Detection as “Link Status Only”
  • Notify Switch as Yes


That’s all the change you need on VMWare.

 On Juniper EX

  • Connect vmnic2 => ge-0/0/0
  • Connect vmnic3 => ge-0/0/1

1: set interface ge-0/0/0 ether-options 802.3ad ae0
2: set interface ge-0/0/1 ether-options 802.3ad ae0
3: set ae5 unit 0 description "ESX - LAN Bundle"
4: set ae5 unit 0 family ethernet-switching port-mode access vlan members vlan1

So, with this, you see, we havent set any LACP properties, this is the way ESX likes is Smile. We should have the interfaces up.

You can make sure by

1: root@EX4200> show interfaces terse | match ae0
2: ge-0/0/0.0 up up aenet --> ae0.0
3: ge-0/0/1.0 up up aenet --> ae0.0
4: ae0 up up
5: ae0.0 up up eth-switch

If the ae interface status is down, which means some thing went wrong and you need to reconfigure.

There is one thing I forgot to mention, some times you will lose connectivity while doing the change. In that case, you would have configured the Management Network differently, but the source Hash will not be working...

Please use the following to correct it ...

"# vim-cmd hostsvc/net/vswitch_setpolicy --nicteaming-policy=loadbalance_ip vSwitch0
# vim-cmd hostsvc/net/portgroup_set --nicteaming-policy=loadbalance_ip vSwitch0 

I was working with the ESX and ESXi and I was in for a shocking revelation... When I connected the ESXi ports which hosted the Managment network (even after making the Mgmt network to route based on IP Hash), the LACP bundle came up but there were 60% packet loss. On trying to find the reason, I found a weird work around.

Now, in my case, there is just one single subnet, so obviously, on the Juniper end, I had set to port mode access. Turns out ESX doesn't like it, though it is an access port, so to make the LACP bundle work, I converted the ae interface to trunk and set the access vlan as a native-vlan-id ... wallah !! it started working :)

Juniper EX configuration for the ESXi with Management Network

unit 0 {
    description "ESX1 - LAN";
    family ethernet-switching {
        port-mode trunk;
        native-vlan-id server_vlan1;

Now, if you are using multiple vlans, then you may not have faced the problem in the first place as the port mode would have been trunk. 

Another thing, I would like to mention is that, in Juniper, if you have noticed, I haven't put anything under vlan members, that is because, if the port mode is trunk, anything that you put under vlan members becomes tagged ... Wierd I know ...

I wouldn't leave this to chance that it is working, so i did some more digging around, I again read this link. (Which I had read some 5 times before), but here is what i read ... 

"HP Switches Sample Configuration

The following configuration is specific to HP switches:
  • HP switches support only two modes of LACP, ACTIVE, and PASSIVE while ESX/ESXi does not support either LACP mode currently.
  • Set HP Switch port mode to TRUNK to accomplish static link aggregation with ESX/ESXi.
  • TRUNK Mode of HP switch ports are the only supported aggregation method compatible with ESX 3.X NIC teaming mode IP hash. "

Now, if you will notice, the Juniper also supports only Active and Passive LACP modes ... I had ignored it as it was for HP switches and for ESX3.x (I am running 4.x), but I guess this is the only explanation as to why it works this way. 

Hopefully, you save some time digging around the info :)

NetApp and Juniper EX

NetApp supports LACP, so here we will see how to create a LACP vif on the NetApp and the corresponding configuration on Juniper EX.

In this case, on the netapp, we have 2 interfaces e0a and e0b (vif1), and on the Juniper side we have ge-0/0/2 and ge-0/0/3

On Net App:

If the vif is created, we will remove that and recreate.

the commands for NetApp are

1: ifconfig vif1 down
2: vif destroy vif1
3: vif create lacp vif1 -b ip e0a e0b

So, this is really very clear, we turned down an interface, then destroyed its configuration and then we said, we will create a virtual interface called vif1 with lacp and the load balancing method of “ip” , that’s what the “-b” switch stood for, and then we added the 2 interfaces.

The LACP is only available from version 7 and up, if you have a lower version, please go ahead and create a multi-mode vif and the Juniper switch configuration will like the ESX.

On Juniper EX

  • Create the ae interfaces as mentioned in the beginning (if you already have configured them, then you don’t need to do it)
  • Clean the configuration of the 2 interfaces.
The configuration on the EX switches will look like this

1: ae1 {
2:     aggregated-ether-options {
3:         link-speed 1g;
4:         lacp {
5:             active;
6:             periodic slow;
7:         }
8:     }
9:     unit 0 {
10:         description "NetApp1 - Controller 2 - LAN";
11:         family ethernet-switching {
12:             port-mode access;
13:             vlan {
14:                 members vlan1;
15:             }
16:         }
17:     }
18: }
19: interfaces
20: {
21:      ge-0/0/2 {
22:         ether-options {
23:         802.3ad ae1;
24:     }
25: }
26:     ge-0/0/3 {
27:     ether-options {
28:         802.3ad ae1;
29:     }
30: }

Once this is created, you can see the only difference is the LACP configuration that we put on the ae interface (this was not done for ESX/ESXi)
You can check if the interfaces are up, the very same way, but in addition, you also will have a LACP command to check.

1: root@PDX-SWIT-SVC02> show interfaces terse | match ae1.0
2: ge-0/0/2.0 up up aenet --> ae1.0
3: ge-0/0/3.0 up up aenet --> ae1.0
4: ae1.0 up up eth-switch

6: {master:0}
7: root@EX4200> show lacp interfaces ae1
8: Aggregated interface: ae1
9: LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
10: ge-0/0/2 Actor No No Yes Yes Yes Yes Slow Active
11: ge-0/0/2 Partner No No Yes Yes Yes Yes Slow Active
12: ge-0/0/3 Actor No No Yes Yes Yes Yes Slow Active
13: ge-0/0/3 Partner No No Yes Yes Yes Yes Slow Active
14: LACP protocol: Receive State Transmit State Mux State
15: ge-0/0/3 Current Slow periodic Collecting distributing
16: ge-0/0/3 Current Slow periodic Collecting distributing

You should be able to see if the LACP should be active for both partner and the Actor, if so, then the configuration works.

Hope you have found the guide useful. Please do comment if some thing needs to be corrected / altered.