HA Proxy for Exchange 2010 Deployment & SMTP Restriction

Hi ... Sorry I have been away for some time doing some stuff, so I hadn't got a chance to update my blog. I thought I would share my recent experience of deploying an Open Source "Linux Load Balancer" (HA Proxy - http://haproxy.1wt.eu/ ). This Load Balancer is amazing ... I have been able to do so much with it in the past, here is one of the very simple and effective solutions (Cost wise & Performance) that we gave to the client using this ... I worked with a friend M L N Rao, who was helping me out with the Exchange front of things. Please read on to see .... I have tried to explain how to do it as well as why ... let me know what you think in the comments ...

So, here is what happened, Our Exchange Team deployed Microsoft Exchange Server 2010 at a clients location and decided to go with MS-NLB. The MS NLB (Network Load Balancer) is a Load Balancer for different Microsoft products like IIS Servers/ISA/TMG, etc but the truth be told more pain than good for network admins. Since I happen to be one, I was called in as the LAN was choked. Obviously I did a sniff on the LAN and saw there there was a severe Multicast Storm and that was eating  up all the LAN bandwidth. 

We saw that the NLB was deployed in the Multicast Mode (There are 3 modes Unicast, Multicast, IGMP Multicast). Since the Exchange Hub Transport /Client Access Server (HT/CAS) was in VMWare, we would have had to do a lot of workaround to get Unicast working, and there is no confirmation that the Unicast flooding wont happen (The way NLB works is that it uses masks and doesn't allow switches to learn the MAC - I wont get into details of that). Well, NLB we had to do a lot of work around for reducing the traffic and I knew it was not worth the effort, so we decided to dump NLB and put in a Load balancer. 

Getting a hardware load balancer like F5-LTM or Cisco CSS, was the easiest option, but it was the costliest as well. We dint want to pay through the nose for just one application, so I decided we will go with HAProxy and you will be amazed to see how well it worked. 

Now since it was a Flat environment, we couldn't put the load balancer in line with the exchange servers (we will talk about this later in the post) so here is the gist of the setup.

Fig 1: Simple diagram explaining the placement of the Load Balancer

As you can see this is the easiest implementation strategy. We deployed this is multiple sites on 2 major platforms CentOS and Ubuntu. (All on VMware as we had the VM for the CAS and HT Servers, we used the same ESX hosts)

Here I will show you the installation in Ubuntu (This can be done in any system by almost the same method) on VMware.

From here on, the post is divided into 3 Sections 

1. Setup of Load balancer 

2. Configuration of Exchange server

3. Configuration of the HA Proxy

Basic Info

Here the following example will be used 

IP of the Load balancer (For Management) : 10.10.10.10

Virtual IP : 10.10.10.11

IP of HT/CAS1 : 10.10.10.20

IP of HT/CAS2 : 10.10.10.21

Default Gateway : 10.10.10.1

Subnet Mask : 255.255.255.0 (/24)

Setup of Load balancer

In this I will show you a sample of how to setup a load balancer. The requirements are a fresh installed ubuntu server, get the ISO from here. Once you install the Ubuntu with most of the default configuration, you will end up with a base installation. 

Please do the following 

1. Set up Networking

We will need to configure the network card. We will add Static IP on Ubuntu and also add the VIP, so that the Ubuntu server can do a proxy ARP. (Other Linux, you can use the ifconfig command and also the individual network scripts)

Edit the Interface script in Ubuntu using vi, (You can use your favorite editor, vi/vim/pico, etc.)

vi /etc/network/interfaces

Make sure you modify the eth0 properties and add the properties of eth0:1. Here the eth0 will act for management and eth0:1 will be the VIP. (If you have to add another VIP, you can just add eth0:2, but more on that later) 

auto eth0

iface eth0 inet static

           address 10.10.10.10

           netmask 255.255.255.0

           network 10.10.10.0

           broadcast 10.10.10.255

           gateway 10.10.10.1


auto eth0:1

iface eth0:1 inet static

           address 10.10.10.11

           netmask 255.255.255.0

           network 10.10.10.0

           broadcast 10.10.10.255

           gateway 10.10.10.1

As you can see I have configured the interfaces for the IP adresses, Now save the file and come back to the shell and execute the following command

/etc/init.d/networking restart

This will now set those IP addresses on the box. You can test this by pinging both the IP address from another machine, you will be able to get the response.

2. Install OpenSSH and Apache (Optional)

sudo apt-get install openssh-server

The above command will install the OpenSSH server which will allow us to SSH in the box so we can get off the console. (In Ubuntu it is not installed by default). You will not need this step in CentOS or RHEL as they install the SSH Server automatically

Once the server is installed, Just SSH to the Management IP (10.10.10.10) using Putty / Tera term or something , cause then you will be able to copy paste (I personally like the SSH than sitting on console)

You may continue to use the console if you like ... doesn't really matter. 

OPTIONAL : (Please don't do it , if you don't have to use the Web Server for some function - In my case, I wrote PHP for management of HA Proxy, but I will keep that for another day)

You may want to install apache and PHP, though this is not required, it might be easier if you wanted to test some stuff 

sudo apt-get install php5

This will install apache. Please note that this will use the port 80, so we have to modify the port to some thing else (like 8080) before we can go further, that is because even exchange will bind on 80, 443 and other ports.  If we try to bind them , the HA Proxy will fail with "Unable to Bind" error. 

In order to change the binding, edit the file 

/etc/apache2/ports.conf

It might be in

/etc/httpd/ports.conf

please change it to 8080 in the listener, so it doesn't conflict

---- END OPTIONAL -----

3. Install GCC and HAProxy

So here we will install the HAProxy and the GCC compiler. We need the GCC as we will install from source. During writing of this, the HAProxy Version  1.4.13 was released, but the apt-get command would install 1.3 (Which dint have source persistence, which is much needed by Exchange). I personally recommend you compile from the source. 

Install GCC 

sudo apt-get install gcc build-essential

Now you are ready to rock and roll (Actually install HAProxy) . I have compiled in the simplest of manner for Linux 2.6 and 1386 Architecture. Please read the README file in the directory.

cd /var/tmp

wget http://haproxy.1wt.eu/download/1.4/src/haproxy-1.4.13.tar.gz

tar -xvzf haproxy-1.4.13.tar.gz

cd haproxy-1.4.13

make TARGET=linux26 ARCH=i386

make install

This will install the haproxy in the /usr/local/sbin/ directory

You can check that by using the command 

which haproxy

It should be in the path, you can choose to copy it to the /usr/sbin directory. 

You officially have installed HAProxy 

{EDIT}
You can skip the above, you can install it by the following command

apt-get install haproxy

This will install all the files, you need to just do the haproxy.cfg file and the autostarting of haproxy

Configuration of Exchange Server

You need to do some change on the Exchange Hub Cast Servers as well. Let me explain why... 

Normally, the way MAPI works is the fact that it connects on port 135 and then the Exchange server assigns a Random Port, on which the client connects to the Exchange for the Email and Address directory. If this has to be allowed, we have to load balance across all 65K ports, which is bit of a over kill for the Load balancer and if we have to allow the MAPI through a firewall, then a security threat as well. 

In Exchange Server, we will just restrict the dynamic ports to a known value and the load balancer can load balance across those ports. This is done by making a change in registry and rebooting the server. (Believe me doesn't take much). 

Follow the following Article

http://social.technet.microsoft.com/wiki/contents/articles/configuring-static-rpc-ports-on-an-exchange-2010-client-access-server.aspx

This will give you a step by step approach on how to do that, basically it is creating 2 registry entries 

Summarizing

Step 1: For Mail (We will use the port 60000)

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\MSExchangeRPC

Under this create a new Key ParametersSystem (If doesnt exist) 

Under this create the following 

Name : TCP/IP Port

Type : REG_DWORD

Value: 60000

Step 2: For Address Book (We will use the port 60001) 

Here it depends on the version, in my case it was the SP1, so it was another Registry Key

For Exchange 2010 RTM

Open the file : microsoft.exchange.addressbook.service.exe.config configuration file located in C:\Program Files\Microsoft\Exchange Server\V14\Bin and you have to make the modification there (Details in the Microsoft Article mentioned above)

For Exchange 2010 SP1

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\MSExchangeAB\Parameters

Create the following 

Name: RpcTcpPort

Type: REZ_SZ

Value: 60001

Once we add this, you need to restart the server for the changes to take effect. You can confirm if the changes took effect or not by the following command (on the cmd terminal of Exchange)

netstat -nap tcp | find "LISTENING" 

You should see the two ports like this 

TCP    0.0.0.0:60000          0.0.0.0:0              LISTENING

TCP    0.0.0.0:60001          0.0.0.0:0              LISTENING

Now we are ready to configure the HA Proxy finally and test our results 

Configuring the HA Proxy

The HA Proxy in our case is going to be very easy to configure. All HA Proxy Configuration is kept in a single file. We will be configuring HA Proxy in a Layer 4 Mode and the SSL Termination is directly on the servers. We will need the Sticky source persistence else this will break OWA and Active Sync and other such stuff. 

The Configuration file will be at 

/etc/haproxy/haproxy.cfg

If the file doesn't exist, just create it 

The content of the file is shown (Will explain the lines after this) 

global

        #uid 99

        #gid 99

        daemon

        stats socket /var/run/haproxy.stat mode 600 level admin

        maxconn 40000

        ulimit-n 81000

        pidfile /var/run/haproxy.pid

defaults

        mode    http

        contimeout      4000

        clitimeout      3600000

        srvtimeout      3600000

        balance roundrobin

listen  Exchange2010 10.10.10.11:80

        bind 10.10.10.11:25

        bind 10.10.10.11:110,10.10.10.11:135

        bind 10.10.10.11:139,10.10.10.11:443

        bind 10.10.10.11:60000,10.10.10.11:60001

        bind 10.10.10.11:6001-6004

        bind 10.10.10.11:993-995

        mode    tcp

        option  persist

        balance roundrobin

        stick-table type ip size 10240k expire 30m

        stick on src

        server HC-CAS1 10.10.10.20 weight 1 check port 80 inter 5000 rise 2 fall 3

        server HC-CAS2 10.10.10.21 weight 1 check port 80 inter 5000 rise 2 fall 3

        option redispatch

        option abortonclose

        maxconn 40000

listen  stats :7000

        stats   enable

        stats   uri /

        option  httpclose

        stats   auth username:password

Ok, that might seem long, but it is very simple. The global section mentions the UID and GID (Which are commented), this will make the HA Proxy run as root. There are a few issues, if it doesn't run as root. The 'maxconn' is the maximum number of connections, we set that to 40000, which is a bit of overkill but then will allow all people to access the exchange server, you can keep it at a low value. 

The defaults section have the connection default values for Client Timeout, Server Timeout, Connection timeout. For details on this, refer the HAProxy Documentation, you can use the values given above as is, and it should work. 

The Listen section is most important, this is what creates the life in the load balancer. As you can see we have given it a name Exchange2010 and you can see the ports they are binded to 

135 - RPC 

80/443 - OWA/ActiveSync/OutlookAnywhere

110 : POP

139 : Netbios

60000/60001 - These are the ones we just set on exchange server (For Mail and Address book) 

993-995: SSL POP/IMAP

6001 - 6004 : Exchange Information Store 

Please note that all the ports are TCP. 

After the binding, I tell it the IP's of the actual server and mention that it has to monitor the port 80 to verify the server is up (its much easier that way). The balance option tells that it has to do round robin and I have set the persistence for about 30 minutes. 

The 'option redispatch' tells it to look for another server when an existing one goes down. You can see the explanation of all of those values in the HA Proxy Documentation. 

The next important thing is the stats. HA Proxy can give you stats on its load balancing work. We create a stats listener and we tell it to use the credentials Username : username & Password: password :) . Please change it here and then point your browsers at http://10.10.10.11:7000 and you will see it. 

Thats it ... we are done... we have a load balancer that load balances 2 exchange servers for all different stuff (Time to kick NLB out of the window :) ) 

Ok, in order to start the haproxy you can just 

haproxy -f /etc/haproxy/haproxy.cfg

If it doesn't give you any errors, then you are good. Use the 'netstat -ln' to confirm it is listening on those ports. 

This pretty much helps us in the installation, but after a reboot, the HAProxy needs to be manually started. I have pasted here an example of the startup script for HA Proxy. Please put this in the following location (if you have another start script, you can use that or write one as per your wish :) . 

/etc/init.d/haproxy

PLEASE IGNORE THIS IF YOU HAVE INSTALLED HA PROXY USING THE APT-GET command

The script was a part of old HA Proxy bundle

#!/bin/sh

### BEGIN INIT INFO

# Provides:          haproxy

# Required-Start:    $local_fs $network

# Required-Stop:     $local_fs

# Default-Start:     2 3 4 5

# Default-Stop:      0 1 6

# Short-Description: fast and reliable load balancing reverse proxy

# Description:       This file should be used to start and stop haproxy.

### END INIT INFO



# Author: Arnaud Cornet 



PATH=/sbin:/usr/sbin:/bin:/usr/bin

PIDFILE=/var/run/haproxy.pid

CONFIG=/etc/haproxy/haproxy.cfg

HAPROXY=/usr/local/sbin/haproxy

EXTRAOPTS=

ENABLED=0



test -x $HAPROXY || exit 0

test -f "$CONFIG" || exit 0



if [ -e /etc/default/haproxy ]; then

        . /etc/default/haproxy

fi



test "$ENABLED" != "0" || exit 0



[ -f /etc/default/rcS ] && . /etc/default/rcS

. /lib/lsb/init-functions





haproxy_start()

{

        start-stop-daemon --start --pidfile "$PIDFILE" \

                --exec $HAPROXY -- -f "$CONFIG" -D -p "$PIDFILE" \

                $EXTRAOPTS || return 2

        return 0

}



haproxy_stop()

{

        if [ ! -f $PIDFILE ] ; then

                # This is a success according to LSB

                return 0

        fi

        for pid in $(cat $PIDFILE) ; do

                /bin/kill $pid || return 4

        done

        rm -f $PIDFILE

        return 0

}



haproxy_reload()

{

        $HAPROXY -f "$CONFIG" -p $PIDFILE -D $EXTRAOPTS -sf $(cat $PIDFILE) \

                || return 2

        return 0

}



haproxy_status()

{

        if [ ! -f $PIDFILE ] ; then

                # program not running

                return 3

        fi



        for pid in $(cat $PIDFILE) ; do

                if ! ps --no-headers p "$pid" | grep haproxy > /dev/null ; then

                        # program running, bogus pidfile

                        return 1

                fi

        done



        return 0

}





case "$1" in

start)

        log_daemon_msg "Starting haproxy" "haproxy"

        haproxy_start

        ret=$?

        case "$ret" in

        0)

                log_end_msg 0

                ;;

        1)

                log_end_msg 1

                echo "pid file '$PIDFILE' found, haproxy not started."

                ;;

        2)

                log_end_msg 1

                ;;

        esac

        exit $ret

        ;;

stop)

        log_daemon_msg "Stopping haproxy" "haproxy"

        haproxy_stop

        ret=$?

        case "$ret" in

        0|1)

                log_end_msg 0

                ;;

        2)

                log_end_msg 1

                ;;

        esac

        exit $ret

        ;;

reload|force-reload)

        log_daemon_msg "Reloading haproxy" "haproxy"

        haproxy_reload

        case "$?" in

        0|1)

                log_end_msg 0

                ;;

        2)

                log_end_msg 1

                ;;

        esac

        ;;

restart)

        log_daemon_msg "Restarting haproxy" "haproxy"

        haproxy_stop

        haproxy_start

        case "$?" in

        0)

                log_end_msg 0

                ;;

        1)

                log_end_msg 1

                ;;

        2)

                log_end_msg 1

                ;;

        esac

        ;;

status)

        haproxy_status

        ret=$?

        case "$ret" in

        0)

                echo "haproxy is running."

                ;;

        1)

                echo "haproxy dead, but $PIDFILE exists."

                ;;

        *)

                echo "haproxy not running."

                ;;

        esac

        exit $ret

        ;;

*)

        echo "Usage: /etc/init.d/haproxy {start|stop|reload|restart|status}"

        exit 2

        ;;

esac



:

The use the following to add it in the startup 

update-rc.d haproxy start 36 2 3 4 5 . stop 20 0 1 6 .

Also check that the file 

/etc/default/haproxy

and make sure it has a line that says

ENABLED=1

Now you are ready to grab a cup of coffee and relax. Reload the box and make sure the HAProxy starts fine (Automatically). 

Just point your DNS to this new IP and the load balancer will show its magic.

Advancements

Now lets talk about a few enhancements, shall we? Here you can see that the port 25 is also load balanced. Since the load balancer is in the same LAN as the Exchange, it is NATting all requests to its own IP. 

Having said that, say you want to use the Server for relay, and use the load balanced IP, then how will we restrict the servers, I mean in the exchange, you have to allow the Load balancer IP as thats the only thing the exchange sees. So where will be block the actual IP's ? On the load balancer off course :) 

We can use IP Tables and do it, but I much rather have it all done in a single place (I mean in the HAProxy configuration), so behold, we are doing iRules (F5 style :) ) IP restriction. 

In order to do it, we need to bring the SMTP on its own virtual (No the IP will remain same), look at the config you will understand. 

listen  Exchange2010 10.10.10.11:80

        bind 10.10.10.11:110,10.10.10.11:135

        bind 10.10.10.11:139,10.10.10.11:443

        bind 10.10.10.11:60000,10.10.10.11:60001

        bind 10.10.10.11:6001-6004

        bind 10.10.10.11:993-995

        mode    tcp

        option  persist

        balance roundrobin

        stick-table type ip size 10240k expire 30m

        stick on src

        server HT-CAS2 10.10.10.21 weight 1 check port 80 inter 5000 rise 2 fall 3

        server HT-CAS1 10.10.10.20 weight 1 check port 80 inter 5000 rise 2 fall 3

        option redispatch

        option abortonclose

        maxconn 40000

listen  ExchangeSMTP 10.10.10.11:25

        mode    tcp

        option  persist

        balance leastconn

        stick-table type ip size 10240k expire 30m

        stick on src

        acl allowed_relay1 src 10.10.10.3

        acl allowed_relay2 src 10.10.10.7

        tcp-request content accept if allowed_relay1

        tcp-request content accept if allowed_relay2

        tcp-request content reject

        server HT-CAS1-SMTP 10.10.10.20:25 weight 1 check port 80 inter 5000 rise 2 fall 3

        server HT-CAS2-SMTP 10.10.10.21:25 weight 1 check port 80 inter 5000 rise 2 fall 3

        option redispatch

        option abortonclose

        maxconn 40000

As you can see, we have divided the Exchange 2010 Virtual server into 2, we kept the first one as it is (just removed the binding of port 25 from the first one. In the second virtual server, we only bind port 25. Also note, when there are lot of ports, the actual servers should not have the port specified. If there is just one port then you can specify it. 

The lines to look for are

acl allowed_relay1 src 10.10.10.3

and the line

tcp-request content accept if allowed_relay1

You can see that we are matching the allowed IP address as with an ACL and if the ACL matches, we accept the connection, else, we reject it (which is shown by

tcp-request content reject

)

As you can see, I have allowed 2 servers to relay the mails through the exchange, and blocked every one else, you can write as many acls as you want, you can also use the mask notation to match multiple IPs / subnets in a single ACL. 

This way, we achieved the Load balancing and access list for just the port 25 and can use it as a relay. 

Please note that any change in the configuration, requires a restart of HAProxy (Don't worry that is not impacting)

Its been a long blog, hope it was informative and useful.