Pages

Tuesday, March 15, 2011

HA Proxy for Exchange 2010 Deployment & SMTP Restriction

Hi ... Sorry I have been away for some time doing some stuff, so I hadn't got a chance to update my blog. I thought I would share my recent experience of deploying an Open Source "Linux Load Balancer" (HA Proxy - http://haproxy.1wt.eu/ ). This Load Balancer is amazing ... I have been able to do so much with it in the past, here is one of the very simple and effective solutions (Cost wise & Performance) that we gave to the client using this ... I worked with a friend M L N Rao, who was helping me out with the Exchange front of things. Please read on to see .... I have tried to explain how to do it as well as why ... let me know what you think in the comments ...

So, here is what happened, Our Exchange Team deployed Microsoft Exchange Server 2010 at a clients location and decided to go with MS-NLB. The MS NLB (Network Load Balancer) is a Load Balancer for different Microsoft products like IIS Servers/ISA/TMG, etc but the truth be told more pain than good for network admins. Since I happen to be one, I was called in as the LAN was choked. Obviously I did a sniff on the LAN and saw there there was a severe Multicast Storm and that was eating  up all the LAN bandwidth. 

We saw that the NLB was deployed in the Multicast Mode (There are 3 modes Unicast, Multicast, IGMP Multicast). Since the Exchange Hub Transport /Client Access Server (HT/CAS) was in VMWare, we would have had to do a lot of workaround to get Unicast working, and there is no confirmation that the Unicast flooding wont happen (The way NLB works is that it uses masks and doesn't allow switches to learn the MAC - I wont get into details of that). Well, NLB we had to do a lot of work around for reducing the traffic and I knew it was not worth the effort, so we decided to dump NLB and put in a Load balancer. 

Getting a hardware load balancer like F5-LTM or Cisco CSS, was the easiest option, but it was the costliest as well. We dint want to pay through the nose for just one application, so I decided we will go with HAProxy and you will be amazed to see how well it worked. 

Now since it was a Flat environment, we couldn't put the load balancer in line with the exchange servers (we will talk about this later in the post) so here is the gist of the setup.



Fig 1: Simple diagram explaining the placement of the Load Balancer

As you can see this is the easiest implementation strategy. We deployed this is multiple sites on 2 major platforms CentOS and Ubuntu. (All on VMware as we had the VM for the CAS and HT Servers, we used the same ESX hosts)


Here I will show you the installation in Ubuntu (This can be done in any system by almost the same method) on VMware.

From here on, the post is divided into 3 Sections 

1. Setup of Load balancer 
2. Configuration of Exchange server
3. Configuration of the HA Proxy

Basic Info
Here the following example will be used 
IP of the Load balancer (For Management) : 10.10.10.10
Virtual IP : 10.10.10.11
IP of HT/CAS1 : 10.10.10.20
IP of HT/CAS2 : 10.10.10.21
Default Gateway : 10.10.10.1
Subnet Mask : 255.255.255.0 (/24)

Setup of Load balancer

In this I will show you a sample of how to setup a load balancer. The requirements are a fresh installed ubuntu server, get the ISO from here. Once you install the Ubuntu with most of the default configuration, you will end up with a base installation. 

Please do the following 

1. Set up Networking
We will need to configure the network card. We will add Static IP on Ubuntu and also add the VIP, so that the Ubuntu server can do a proxy ARP. (Other Linux, you can use the ifconfig command and also the individual network scripts)

Edit the Interface script in Ubuntu using vi, (You can use your favorite editor, vi/vim/pico, etc.)

vi /etc/network/interfaces


Make sure you modify the eth0 properties and add the properties of eth0:1. Here the eth0 will act for management and eth0:1 will be the VIP. (If you have to add another VIP, you can just add eth0:2, but more on that later) 

auto eth0
iface eth0 inet static
           address 10.10.10.10
           netmask 255.255.255.0
           network 10.10.10.0
           broadcast 10.10.10.255
           gateway 10.10.10.1
auto eth0:1
iface eth0:1 inet static
           address 10.10.10.11
           netmask 255.255.255.0
           network 10.10.10.0
           broadcast 10.10.10.255
           gateway 10.10.10.1


As you can see I have configured the interfaces for the IP adresses, Now save the file and come back to the shell and execute the following command

/etc/init.d/networking restart

This will now set those IP addresses on the box. You can test this by pinging both the IP address from another machine, you will be able to get the response.


2. Install OpenSSH and Apache (Optional)

sudo apt-get install openssh-server


The above command will install the OpenSSH server which will allow us to SSH in the box so we can get off the console. (In Ubuntu it is not installed by default). You will not need this step in CentOS or RHEL as they install the SSH Server automatically

Once the server is installed, Just SSH to the Management IP (10.10.10.10) using Putty / Tera term or something , cause then you will be able to copy paste (I personally like the SSH than sitting on console)

You may continue to use the console if you like ... doesn't really matter. 

OPTIONAL : (Please don't do it , if you don't have to use the Web Server for some function - In my case, I wrote PHP for management of HA Proxy, but I will keep that for another day)

You may want to install apache and PHP, though this is not required, it might be easier if you wanted to test some stuff 

sudo apt-get install php5


This will install apache. Please note that this will use the port 80, so we have to modify the port to some thing else (like 8080) before we can go further, that is because even exchange will bind on 80, 443 and other ports.  If we try to bind them , the HA Proxy will fail with "Unable to Bind" error. 

In order to change the binding, edit the file 

/etc/apache2/ports.conf


It might be in
/etc/httpd/ports.conf


please change it to 8080 in the listener, so it doesn't conflict

---- END OPTIONAL -----


3. Install GCC and HAProxy

So here we will install the HAProxy and the GCC compiler. We need the GCC as we will install from source. During writing of this, the HAProxy Version  1.4.13 was released, but the apt-get command would install 1.3 (Which dint have source persistence, which is much needed by Exchange). I personally recommend you compile from the source. 

Install GCC 

sudo apt-get install gcc build-essential


Now you are ready to rock and roll (Actually install HAProxy) . I have compiled in the simplest of manner for Linux 2.6 and 1386 Architecture. Please read the README file in the directory.



cd /var/tmp
wget http://haproxy.1wt.eu/download/1.4/src/haproxy-1.4.13.tar.gz
tar -xvzf haproxy-1.4.13.tar.gz
cd haproxy-1.4.13
make TARGET=linux26 ARCH=i386
make install


This will install the haproxy in the /usr/local/sbin/ directory

You can check that by using the command 

which haproxy


It should be in the path, you can choose to copy it to the /usr/sbin directory. 

You officially have installed HAProxy 


{EDIT}
You can skip the above, you can install it by the following command

apt-get install haproxy

This will install all the files, you need to just do the haproxy.cfg file and the autostarting of haproxy

Configuration of Exchange Server

You need to do some change on the Exchange Hub Cast Servers as well. Let me explain why... 
Normally, the way MAPI works is the fact that it connects on port 135 and then the Exchange server assigns a Random Port, on which the client connects to the Exchange for the Email and Address directory. If this has to be allowed, we have to load balance across all 65K ports, which is bit of a over kill for the Load balancer and if we have to allow the MAPI through a firewall, then a security threat as well. 

In Exchange Server, we will just restrict the dynamic ports to a known value and the load balancer can load balance across those ports. This is done by making a change in registry and rebooting the server. (Believe me doesn't take much). 

Follow the following Article


This will give you a step by step approach on how to do that, basically it is creating 2 registry entries 

Summarizing

Step 1: For Mail (We will use the port 60000)

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\MSExchangeRPC

Under this create a new Key ParametersSystem (If doesnt exist) 

Under this create the following 

Name : TCP/IP Port
Type : REG_DWORD
Value: 60000


Step 2: For Address Book (We will use the port 60001) 


Here it depends on the version, in my case it was the SP1, so it was another Registry Key

For Exchange 2010 RTM

Open the file : microsoft.exchange.addressbook.service.exe.config configuration file located in C:\Program Files\Microsoft\Exchange Server\V14\Bin and you have to make the modification there (Details in the Microsoft Article mentioned above)

For Exchange 2010 SP1

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\MSExchangeAB\Parameters

Create the following 

Name: RpcTcpPort
Type: REZ_SZ
Value: 60001



Once we add this, you need to restart the server for the changes to take effect. You can confirm if the changes took effect or not by the following command (on the cmd terminal of Exchange)

netstat -nap tcp | find "LISTENING" 


You should see the two ports like this 

TCP    0.0.0.0:60000          0.0.0.0:0              LISTENING
TCP    0.0.0.0:60001          0.0.0.0:0              LISTENING



Now we are ready to configure the HA Proxy finally and test our results 

Configuring the HA Proxy

The HA Proxy in our case is going to be very easy to configure. All HA Proxy Configuration is kept in a single file. We will be configuring HA Proxy in a Layer 4 Mode and the SSL Termination is directly on the servers. We will need the Sticky source persistence else this will break OWA and Active Sync and other such stuff. 

The Configuration file will be at 

/etc/haproxy/haproxy.cfg


If the file doesn't exist, just create it 

The content of the file is shown (Will explain the lines after this) 

global
        #uid 99
        #gid 99
        daemon
        stats socket /var/run/haproxy.stat mode 600 level admin
        maxconn 40000
        ulimit-n 81000
        pidfile /var/run/haproxy.pid
defaults
        mode    http
        contimeout      4000
        clitimeout      3600000
        srvtimeout      3600000
        balance roundrobin
listen  Exchange2010 10.10.10.11:80
        bind 10.10.10.11:25
        bind 10.10.10.11:110,10.10.10.11:135
        bind 10.10.10.11:139,10.10.10.11:443
        bind 10.10.10.11:60000,10.10.10.11:60001
        bind 10.10.10.11:6001-6004
        bind 10.10.10.11:993-995
        mode    tcp
        option  persist
        balance roundrobin
        stick-table type ip size 10240k expire 30m
        stick on src
        server HC-CAS1 10.10.10.20 weight 1 check port 80 inter 5000 rise 2 fall 3
        server HC-CAS2 10.10.10.21 weight 1 check port 80 inter 5000 rise 2 fall 3
        option redispatch
        option abortonclose
        maxconn 40000
listen  stats :7000
        stats   enable
        stats   uri /
        option  httpclose
        stats   auth username:password



Ok, that might seem long, but it is very simple. The global section mentions the UID and GID (Which are commented), this will make the HA Proxy run as root. There are a few issues, if it doesn't run as root. The 'maxconn' is the maximum number of connections, we set that to 40000, which is a bit of overkill but then will allow all people to access the exchange server, you can keep it at a low value. 

The defaults section have the connection default values for Client Timeout, Server Timeout, Connection timeout. For details on this, refer the HAProxy Documentation, you can use the values given above as is, and it should work. 

The Listen section is most important, this is what creates the life in the load balancer. As you can see we have given it a name Exchange2010 and you can see the ports they are binded to 

135 - RPC 
80/443 - OWA/ActiveSync/OutlookAnywhere
110 : POP
139 : Netbios
60000/60001 - These are the ones we just set on exchange server (For Mail and Address book) 
993-995: SSL POP/IMAP
6001 - 6004 : Exchange Information Store 

Please note that all the ports are TCP. 

After the binding, I tell it the IP's of the actual server and mention that it has to monitor the port 80 to verify the server is up (its much easier that way). The balance option tells that it has to do round robin and I have set the persistence for about 30 minutes. 

The 'option redispatch' tells it to look for another server when an existing one goes down. You can see the explanation of all of those values in the HA Proxy Documentation. 

The next important thing is the stats. HA Proxy can give you stats on its load balancing work. We create a stats listener and we tell it to use the credentials Username : username & Password: password :) . Please change it here and then point your browsers at http://10.10.10.11:7000 and you will see it. 


Thats it ... we are done... we have a load balancer that load balances 2 exchange servers for all different stuff (Time to kick NLB out of the window :) ) 

Ok, in order to start the haproxy you can just 

haproxy -f /etc/haproxy/haproxy.cfg


If it doesn't give you any errors, then you are good. Use the 'netstat -ln' to confirm it is listening on those ports. 


This pretty much helps us in the installation, but after a reboot, the HAProxy needs to be manually started. I have pasted here an example of the startup script for HA Proxy. Please put this in the following location (if you have another start script, you can use that or write one as per your wish :) . 

/etc/init.d/haproxy

PLEASE IGNORE THIS IF YOU HAVE INSTALLED HA PROXY USING THE APT-GET command

The script was a part of old HA Proxy bundle

#!/bin/sh
### BEGIN INIT INFO
# Provides:          haproxy
# Required-Start:    $local_fs $network
# Required-Stop:     $local_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: fast and reliable load balancing reverse proxy
# Description:       This file should be used to start and stop haproxy.
### END INIT INFO
# Author: Arnaud Cornet
PATH=/sbin:/usr/sbin:/bin:/usr/bin
PIDFILE=/var/run/haproxy.pid
CONFIG=/etc/haproxy/haproxy.cfg
HAPROXY=/usr/local/sbin/haproxy
EXTRAOPTS=
ENABLED=0
test -x $HAPROXY || exit 0
test -f "$CONFIG" || exit 0
if [ -e /etc/default/haproxy ]; then
        . /etc/default/haproxy
fi
test "$ENABLED" != "0" || exit 0
[ -f /etc/default/rcS ] && . /etc/default/rcS
. /lib/lsb/init-functions
haproxy_start()
{
        start-stop-daemon --start --pidfile "$PIDFILE" \
                --exec $HAPROXY -- -f "$CONFIG" -D -p "$PIDFILE" \
                $EXTRAOPTS || return 2
        return 0
}
haproxy_stop()
{
        if [ ! -f $PIDFILE ] ; then
                # This is a success according to LSB
                return 0
        fi
        for pid in $(cat $PIDFILE) ; do
                /bin/kill $pid || return 4
        done
        rm -f $PIDFILE
        return 0
}
haproxy_reload()
{
        $HAPROXY -f "$CONFIG" -p $PIDFILE -D $EXTRAOPTS -sf $(cat $PIDFILE) \
                || return 2
        return 0
}
haproxy_status()
{
        if [ ! -f $PIDFILE ] ; then
                # program not running
                return 3
        fi
        for pid in $(cat $PIDFILE) ; do
                if ! ps --no-headers p "$pid" | grep haproxy > /dev/null ; then
                        # program running, bogus pidfile
                        return 1
                fi
        done
        return 0
}
case "$1" in
start)
        log_daemon_msg "Starting haproxy" "haproxy"
        haproxy_start
        ret=$?
        case "$ret" in
        0)
                log_end_msg 0
                ;;
        1)
                log_end_msg 1
                echo "pid file '$PIDFILE' found, haproxy not started."
                ;;
        2)
                log_end_msg 1
                ;;
        esac
        exit $ret
        ;;
stop)
        log_daemon_msg "Stopping haproxy" "haproxy"
        haproxy_stop
        ret=$?
        case "$ret" in
        0|1)
                log_end_msg 0
                ;;
        2)
                log_end_msg 1
                ;;
        esac
        exit $ret
        ;;
reload|force-reload)
        log_daemon_msg "Reloading haproxy" "haproxy"
        haproxy_reload
        case "$?" in
        0|1)
                log_end_msg 0
                ;;
        2)
                log_end_msg 1
                ;;
        esac
        ;;
restart)
        log_daemon_msg "Restarting haproxy" "haproxy"
        haproxy_stop
        haproxy_start
        case "$?" in
        0)
                log_end_msg 0
                ;;
        1)
                log_end_msg 1
                ;;
        2)
                log_end_msg 1
                ;;
        esac
        ;;
status)
        haproxy_status
        ret=$?
        case "$ret" in
        0)
                echo "haproxy is running."
                ;;
        1)
                echo "haproxy dead, but $PIDFILE exists."
                ;;
        *)
                echo "haproxy not running."
                ;;
        esac
        exit $ret
        ;;
*)
        echo "Usage: /etc/init.d/haproxy {start|stop|reload|restart|status}"
        exit 2
        ;;
esac
:



The use the following to add it in the startup 

update-rc.d haproxy start 36 2 3 4 5 . stop 20 0 1 6 .


Also check that the file 
/etc/default/haproxy
and make sure it has a line that says
ENABLED=1



Now you are ready to grab a cup of coffee and relax. Reload the box and make sure the HAProxy starts fine (Automatically). 

Just point your DNS to this new IP and the load balancer will show its magic.

Advancements

Now lets talk about a few enhancements, shall we? Here you can see that the port 25 is also load balanced. Since the load balancer is in the same LAN as the Exchange, it is NATting all requests to its own IP. 

Having said that, say you want to use the Server for relay, and use the load balanced IP, then how will we restrict the servers, I mean in the exchange, you have to allow the Load balancer IP as thats the only thing the exchange sees. So where will be block the actual IP's ? On the load balancer off course :) 

We can use IP Tables and do it, but I much rather have it all done in a single place (I mean in the HAProxy configuration), so behold, we are doing iRules (F5 style :) ) IP restriction. 

In order to do it, we need to bring the SMTP on its own virtual (No the IP will remain same), look at the config you will understand. 


listen  Exchange2010 10.10.10.11:80
        bind 10.10.10.11:110,10.10.10.11:135
        bind 10.10.10.11:139,10.10.10.11:443
        bind 10.10.10.11:60000,10.10.10.11:60001
        bind 10.10.10.11:6001-6004
        bind 10.10.10.11:993-995
        mode    tcp
        option  persist
        balance roundrobin
        stick-table type ip size 10240k expire 30m
        stick on src
        server HT-CAS2 10.10.10.21 weight 1 check port 80 inter 5000 rise 2 fall 3
        server HT-CAS1 10.10.10.20 weight 1 check port 80 inter 5000 rise 2 fall 3
        option redispatch
        option abortonclose
        maxconn 40000
listen  ExchangeSMTP 10.10.10.11:25
        mode    tcp
        option  persist
        balance leastconn
        stick-table type ip size 10240k expire 30m
        stick on src
        acl allowed_relay1 src 10.10.10.3
        acl allowed_relay2 src 10.10.10.7
        tcp-request content accept if allowed_relay1
        tcp-request content accept if allowed_relay2
        tcp-request content reject
        server HT-CAS1-SMTP 10.10.10.20:25 weight 1 check port 80 inter 5000 rise 2 fall 3
        server HT-CAS2-SMTP 10.10.10.21:25 weight 1 check port 80 inter 5000 rise 2 fall 3
        option redispatch
        option abortonclose
        maxconn 40000


As you can see, we have divided the Exchange 2010 Virtual server into 2, we kept the first one as it is (just removed the binding of port 25 from the first one. In the second virtual server, we only bind port 25. Also note, when there are lot of ports, the actual servers should not have the port specified. If there is just one port then you can specify it. 

The lines to look for are
acl allowed_relay1 src 10.10.10.3
and the line
tcp-request content accept if allowed_relay1


You can see that we are matching the allowed IP address as with an ACL and if the ACL matches, we accept the connection, else, we reject it (which is shown by
tcp-request content reject
)

As you can see, I have allowed 2 servers to relay the mails through the exchange, and blocked every one else, you can write as many acls as you want, you can also use the mask notation to match multiple IPs / subnets in a single ACL. 

This way, we achieved the Load balancing and access list for just the port 25 and can use it as a relay. 

Please note that any change in the configuration, requires a restart of HAProxy (Don't worry that is not impacting)
Its been a long blog, hope it was informative and useful.









34 comments:

  1. hi! I want to install haproxy on my ubuntu server buy the haproxy version from the 10.04 repo is 1.3.x version.

    I wanted the latest build 1.4.13, since it has improvements and bugfixes.

    What i did: 1. installed the haproxy from the official repo: apt-get install haproxy. 2. Download source file from the website, and compile it. 3. swapped the new haproxy with the old.

    so far it works, but i as if i should be aware of something?

    Why i did the following? since apt-get creates a init.d startup script, /etc/haproxy.cfg file, and user and group.

    ReplyDelete
  2. @ElGreco ... What you did is totally fine. It will work without any issues. The beauty of HA Proxy is that it all gets compiled in a single executable. So when you built it, using the source and swapped, you saved yourself a bunch of trouble in the init scripts.

    I would recommend that you test the following

    1. Reboot the box and see the HA Proxy Starts Automatically

    Because if you haven't modified the initialization parameters (update-rc.d haproxy start 36 2 3 4 5 . stop 20 0 1 6 .)

    It might try and start itself before networking and fail to bind ports. So thats the only test you need to do and you will be good to go.

    ReplyDelete
  3. I have been following your instructions with great ease (new to Linux too) and have a few questions.
    I did not have any issue installing HAProxy to the /usr/local/sbin directory and moving it to the /usr/sbin directory. However there is no trace of the associated files in any of the etc/ directories did I miss a step?

    Since the default haproxy.cfg file is missing or not created can you post or refer me to a copy of the default config?

    ReplyDelete
  4. @KraftyKick ... Hi, I am sorry, they will not be created automatically ... You will have to create them manually or you can do this ...

    Before Installing it from source, you can use the pre-compiled binary

    Step 1:

    apt-get install haproxy

    (Use sudo if you are not root)

    This will install all the realted files, but it will install the old HA Proxy version 1.3 or some thing

    You can then follow my instructions, compile the new version and replace it ...

    Here are the instructions

    sudo su
    (This will turn you into root)

    apt-get install haproxy

    This will install the HA proxy

    Execute the command

    which haproxy



    It will show you the path where ha proxy has been installed (It should be in /usr/sbin or some thing ...

    Execute haproxy -v , this will show you the version, since it is the old version we will not have many features (Like src persistance)


    Now we will remove the executable alone and use the other files :)

    which haproxy | xargs rm -rf

    That will delete the executable of HA Proxy (You can just delete the file directly)

    Now we will compile the new version (Please make sure you follow the above and install the gcc and other stuff)

    cd /var/tmp
    wget http://haproxy.1wt.eu/download/1.4/src/haproxy-1.4.13.tar.gz
    tar -xvzf haproxy-1.4.13.tar.gz
    cd haproxy-1.4.13
    make TARGET=linux26 ARCH=i386
    make install


    This will install the haproxy in /usr/local/sbin/.

    Now execute the

    which haproxy

    You will get the new path. If not use find / -name haproxy

    that will find the files for you ...

    Now you should have all the files created and the new installer.

    You can copy the haproxy executable to the old directory, or you can modify the init.d script to point it to the new path. (Which ever feels easier)

    Please make sure that you use the update rc.d command (mentioned in the blog) , restart the box making sure the haproxy starts automatically.




    Another point there is nothing like a default CFG file, what i have posted is an example that I implemented, you can use this as base file, if you need other config examples, i will send it to you ... just let me know


    Hope this helps :)

    ReplyDelete
  5. Guys, Just to make it clear, the above given example is used on Live servers and it works. If you are hesitant do try and contact me and I will answer your questions as best as I can.

    ReplyDelete
  6. Thanks for the very nice article.

    But there is no mention of SSL offloading which according to this article ( page 6) http://www.barracudanetworks.com/ns/downloads/Other/Exchange%202010%20and%20the%20Barracuda%20Load%20Balancer.pdf without SSL offloading , persistence /affinity will not work

    ReplyDelete
  7. Hey Francis, we are using an IP based persistance here, so we dont need to offload the SSL on the load balancer. If we had to use cookie or some thing, then we would have to use it, if You are intrested, I can go ahead and put in another howto of how to do SSL offloading in conjunction.

    ReplyDelete
  8. I would be very happy for an article with offloading in conjunction.

    ReplyDelete
  9. Just wrote it ... http://3-4-5-6.blogspot.com/2011/08/haproxy-load-balancer-with-ssl.html

    Check it out :)

    ReplyDelete
  10. helo,
    thanks for this article.

    I have a question. you use haproxy to replace nlb function. but using only one haproxy you create a single point of failure.

    My question is :
    Is it possible to use ha proxy with NLB ?

    regards,

    Michael.

    ReplyDelete
  11. Michael,

    The HA proxy is to replace the NLB, The NLB causes too much network chaos, and you can actually have a redundant pair so that you dont have a single point of failure ... I will write a blog tonight on how to do that ...

    ReplyDelete
  12. Michael, I have written the blog http://3-4-5-6.blogspot.com/2011/11/haproxy-for-exchange-2010part-2.html to have a redundant load balancer solution

    ReplyDelete
  13. Hi Alok,

    We had just the same problem that led you to setup and share this configuration with HAProxy.

    Thank you for this article as, based on it, I setup a Debian Squeeze with HAProxy running just fine.

    But I've a question that I don't know if you can help me or redirect me anywhere.

    Whenever we take one of our 2 CAS+HUB servers down, for maintenance as example, the clients that where using this CAS+HUB get disconnected and they have only 2 choices: quit Outlook and launch it again or wait for the server for coming back up again. Is this normal behavior or should HAProxy redirect the client to the available server?

    Thanks in advance.

    Cheers,

    Joao

    ReplyDelete
  14. Actually the HAProxy doesnt have a re-select option for the existing connections. The only way to actually force this is to reduce the server timeout 5 mins ... so if the server is not responding for 5 mins it will be considered down and the client connection will be aborted

    ReplyDelete
  15. Hi All,
    Thank you for this post and I am currently running this on a Centos 5 and it is running perfectly in production. The only issue I have is that some users get the pop that says "The Administrator has made changes to the server Please restart outlook" without any of the servers failing over....

    What could be the issue?

    Thank you

    ReplyDelete
  16. This comment has been removed by the author.

    ReplyDelete
  17. I have followed this blog and got everything working, seems to be functioning great except for one issue the we are having terrible Calendar performance, opening events, accepting invitations etc. takes forever to complete, and the GAL loads slowly. I doubt that the HAProxy setup is the cause of this but wanted to check to see if this has ever been seen before?

    ReplyDelete
    Replies
    1. Hi,

      This is not seen before .. but here is what i fear may have happened. Remember that we have done a static RPC port mapping on the exchange. We do 2 ports

      One for the MAPI and one for OAB .... The OAB takes care of the address book. I guess that the OAB port configuration was not done / dint take effect on may be one server. This may cause this kind of an issue


      I would implore you to check that ...

      In order to do that


      log on to the CAS servers and run on the cmd


      netstat -ano | findstr "" (replace it with mapi and address ook


      if you have followed the bog as is


      netstat -ano | findstr 60000
      netstat -ano | findstr 60001

      Ensure that you get both the ports as listening

      Chances are that 60001 is not listening on one of the servers ... Check that ... and if it is not listening follow the technet article (mentioned in the blog) and reboot the CAS servers

      I hope this fixes the issue

      Delete
  18. Thanks for putting this together - it's the most useful Exchange/HAProxy step by step I've found sofar.

    I've just started looking into NLB and noticed that it doesn't do any transparent authentication -- my test machines all get prompted for basic auth credentials.

    So - my question is about what the best way to would be to do NLB for Exchange 2010 and retain use of Windows Credentials. We currently have this working with Windows LB, but as we know, it's a network mess, and now our network admins are complaining about it, but my help desk doesn't want to lose the transparent auth/NTLM feature. Is this doable with HAProxy or other free solutions, I'm hesitant to spend a ton on Citrix/F5/etc just to do this feature.

    Thoughts? Thanks in advance.

    ReplyDelete
    Replies
    1. Hi Chris

      Glad you found it useful. Actually this does transparent authentication. If you have the static RPC ports open correctly on the CAS.

      I have deployed this in a few customers and all are using integrated windows authenticaction

      Delete
    2. Thanks for your reply, it does look like I had my statics misconfigured. Upon correcting that it looks like its a little bit of a rocky transition - I forced a PW change on myself, and Outlook didn't seem to pick up on that until after a reboot - but, lo and behold, it didn't ask for my credentials anymore. I guess it could have worked if I had waited around longer.

      Thanks again!

      Delete
  19. Also, I've noticed that when I do a haproxy restart my Outlook clients immediately prompt for credentials. I can either enter them basic auth style, or restart Outlook to get the transparent auth working. So - what could be wrong, since it's supposed to be non-impacting?

    ReplyDelete
    Replies
    1. Hi Chris,

      Sorry about the delay in response, I was away on some assignment, What version of HA Proxy are you using, I would like to test it and see how best we can stop this from happening

      Delete
    2. I'm using 1.5_17. Thanks!

      Delete
    3. I think I found an answer to this here, where I've modified the naming convention to jive with what you setup above - it appears to work well enough in my lab, haven't tried it on the production systems yet though, wondered your thoughts on it.

      #!/bin/bash
      #A hot rehaproxy.cfguration script would look like this :
      #Extract from http://haproxy.1wt.eu/download/1.3/doc/architecture.txt
      #Modified by Burns

      # save previous state
      mv /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.old
      mv /var/run/haproxy.pid /var/run/haproxy.pid.old

      mv /etc/haproxy/haproxy.cfg.new /etc/haproxy/haproxy.cfg
      kill -TTOU $(cat /var/run/haproxy.pid.old)
      if haproxy -p /var/run/haproxy.pid -f /etc/haproxy/haproxy.cfg; then
      echo "New instance successfully loaded, stopping previous one."
      kill -USR1 $(cat /var/run/haproxy.pid.old)
      rm -f /var/run/haproxy.pid.old
      exit 1
      else
      echo "New instance failed to start, resuming previous one."
      kill -TTIN $(cat /var/run/haproxy.pid.old)
      rm -f /var/run/haproxy.pid
      mv /var/run/haproxy.pid.old /var/run/haproxy.pid
      mv /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.new
      mv /etc/haproxy/haproxy.cfg.old /etc/haproxy/haproxy.cfg
      exit 0
      fi

      Delete
  20. This comment has been removed by the author.

    ReplyDelete
  21. Hi,

    wondering how you dealt with relaying emails using this method?

    fyi, our 2010 exchange environment comprises of 2x cas/hub servers (casarray) + 2x mailbox servers (DAG).
    I followed your guide (thank you!) and put in this NLB solution.

    however we have a number of non-exchange servers that need to relay emails externally eg a reporting sql server that sends to external clients, printers which scan and send to external clients... when I pointed these devices to the ubuntu NLB, and added the IP off the NLB to the relay-receive connector in Exchange, this worked, BUT ... because the NLB is internet facing, it fails the open relay tests at http://www.mailradar.com/openrelay/

    for the interim, i've just pointed these devices to one of the cas/hub servers. although this will work, it's obviously not making use of the casarray.

    can you suggest a solution? or is there something i'm not doing correctly with my exchange receive connectors.

    FYI i have kept the Default and Client receive-connectors as part of the exchange installation
    > with the Default connector,
    >> for Permissions i've ticked: "Anonymous users" , "Exchange users", "Exchange Servers", "Legacy servers".
    >> for Authentication i've ticked: "TLS" , both "Basic Authentication" checkboxes, "Exchange Server Authentication", Integrated Window Authentication"

    i created a new relay based on this guide and set permissions as written: http://exchangeserverpro.com/how-to-configure-a-relay-connector-for-exchange-server-2010/
    This is the 'Relay' connector that i've permissioned the internal sevrers/devices to relay out against.

    ReplyDelete
    Replies
    1. Hello Ben,

      I had the same issue, hence the Restriction

      Since the Exchange is seeing that all mails are coming from the Load balancer, the relay configuration there will not work, hence the way I created it was by creating the relay block list in the config itself

      In my SMTP VIP config above you will see some thing like this

      acl allowed_relay1 src 10.10.10.3
      acl allowed_relay2 src 10.10.10.7

      This creates a IP based block on the port 25

      I know it is not a very elagant solution but it worked for the time.

      I hope this helps

      Delete
    2. ahh of course. sorry i skipped the advanced section above and was satisfied with the basic implementation.
      thanks for the reply!

      Delete
  22. I salute to the admin of this diary. i actually like and that i can sure as shooting advocate this diary to my friends.
    Your nice work is very attractive and impressive.
    I have to survey many sites now I m telling you about blocked sites to access a great method just one click here you will be surprised.
    The information on this web log is extremely helpful and extremely attention-grabbing.
    These are all very helpful proxy list I would like to thanks to admin of this blog.
    I really appreciate this blog and I will sure promote this blog to others in my circle.
    Whenever I need some problem relating this issue I come to your site and get very informative stuff thanks a lot for your article . Keep it up.
    Great to know about but i can open in seconds just one click and you can open any blocked site just a click on this.
    The information on this web log is extremely helpful and extremely attention-grabbing.
    I salute to the admin of this diary. i actually like and that i can sure as shooting advocate this diary to my friends.
    Bee MP3 UK proxy

    ReplyDelete
  23. We have setup this exact setup however have one major issue, when the access is from a machine on a remote site (90% of the users are remote) The connection to exchange is across a WAN and into a Cisco ASA firewall, monitoring the firewall indicates that there is no issue, nothing is blocked or dropped. No remote user can successfully access or setup the outlook client however a ping / traceroute to the HAProxy address is successful as it a connection to all the resuired ports.

    If we test with one of the local clients that is routed across the LAN and does not pass through the firewall then Outlook cliens can access exchage successfully.

    Is there any TCP timeout that could be causing this perhaps ? So far we have drawn a blank as nothing is indicating a failure at any point.

    ReplyDelete
  24. Upon further investigation it seems that clients accessing HAProxy / Exchange through the firewall are unable to negotiate to use port 60001 for address book lookup and they try to use randon high ports.

    Local clients negotiate the port successfully and switch to use it hence they work.

    Still don't understand why this would be the case.

    ReplyDelete
    Replies
    1. You may not have Protocol inspection on your firewall for MS RPC. Please see if this helps

      http://support.microsoft.com/kb/154596

      Delete
  25. Just want to say your article is astounding. The clarity in your post is simply spectacular
    Mp3Juices UK proxy

    ReplyDelete