Exchange Server 2013/2016 native Load Balancing & Failover options

Although NLB is a supported method for load balancing Exchange Server 2013/2016, it does have some limitations, especially for small to medium environments. If you are trying to architect a highly available Exchange solution you will almost certainly be using Database Availability Groups (DAG’s). DAG’s were introduced in Exchange 2010, and are used to replicate Mailbox Databases between Exchange Servers. The DAG technology is built on Windows Failover Clustering.

Exchange Server 2013 is broken down into two server roles the Client Access Server (CAS) and Mailbox Servers (MBX). The DAG technology only protects failure at the Mailbox Server level, however it does not failover incoming connections to Exchange.  In small networks you want to keep the number of servers to a minimum, it is possible to have both Exchange roles installed on the same server, therefore if you wanted to deploy a highly available Exchange setup you can get away with two servers in total.

Network Load Balancing and Failover Clustering cannot be installed on the same server. Therefore, you cannot use WNLB to load balance the CAS server if you are using dual roles (CAS + MBX) servers. In order for WNLB to be a compatible solution with DAG’s you must separate the CAS and MBX roles onto separate servers.

DNS Round Robin

Although DNS RR can be used to load balance traffic across multiple CAS servers it does not offer any kind of failover. This is mainly because the native Windows DNS Server does not offer any kind of DNS record weight or priority. This means you can have multiple records pointing to your Exchange namespace (i.e mail.domain.com) using the IP’s of the CAS servers.

For this to even offer load balancing you have to reconfigure the TTL of the DNS records at both the DNS Server level, and the end clients.

To reduce the TTL on the DNS A records enable the Advanced Features inside DNS, when you create an (A) record you will then have the option to set a TTL on the record.

In this example I have set it very low to only 15 seconds, this maybe a bit low for most production environments. Especially if you only have a couple of DC/DNS servers.

So I have DNS RR setup to load balance across 3 x CAS servers.

·         mail.ryanbetts.co.uk > 192.168.1.104

·         mail.ryanbetts.co.uk > 192.168.1.105

·         mail.ryanbetts.co.uk > 192.168.1.106

Although the TTL has been lowered at the server side it must also be done on the clients. By default Windows caches DNS lookups for 1 hour.  To configure the DNS TTL cache on a localling in Windows, open the Registry at Local Machine\System CurrentControlSet\Services\DnsCache\Parameters

Create a 32 bit DWORD called "MaxCacheTtl" , set this to a value in seconds (Decimal).

For the changes to take a affect you must restart the DNS Client service. Although making these configuration changes load balances the traffic across CAS servers, it does not offer a very good user experience.

DNS RR in action on a Windows 8.1 client, when the DNS cache expires and the client does another query to it's primary DNS server the Outlook client remains connected to Exchange.

However I have noticed (I'm guessing) when the cache expires the client is sometimes prompted to enter credentials again. Which is annoying for any users, especially if you set the TTL to somethin extremely low like 15 seconds.

DNS Round Robin no "failover"

For a test to prove this to myself, so that I really understand it. I failed over my only DAG to the 2nd Exchange server (192.168.1.105). 

I then turned the 1st server off (192.168.1.104) however, as I did not change the DNS RR configuration the client was still resolving mail.ryanbetts.co.uk to the IP address of the 1st Exchange server. When the TTL expired it started resolving to 192.168.1.105 which made a successful connection.

The only way to make this a "highly available" configuration is to effectively disable DNS RR in the event one of the CAS servers fails. In this scenario you would simple delete the mail.ryanbetts.co.uk record that was pointing to the failed server, therefore all clients would be directed to the surviving server. This might be acceptable for some businesses but as it's not active/active it won't fit all requirements.

I am going to look at the free Kemp Layer 4-7 Load Balancer to load balance Exchange Server, this is free but has a limit of 20 mpb/s throughput and a maximum of 50 concurrent sessions. 

Base Configuring an F5 Big-IP with LTM Module Device

You can get a trial of the F5 Big-IP device from their website, it provides a full Big-IP device (with the LTM module) for up to 90 days from the date of activation. I am looking to load balance Exchange 2016 and VMware View through a pair of Big-IP's with the LTM module so I thought I would spin it up and get it working in my lab prior to doing it for real. The Big-IP is much like Citrix NetScaler in the sense that it's an Application Delivery Controller, therefore many of the concepts are the same. I am using VMware Workstation, the Virtual Appliance is available for VMware, Citrix and Hyper-V. I have found the OVA hardware spec is a little low, the box seems to be pretty slow if you leave it at 2Gb of memory, I would recommend upping it to 4Gb.

Import the OVA into either vSphere or VMware Workstation.

Configure the Network Interfaces

·        vmnet0 - bridged (mgmt)

·        vmnet1 - host only (int)

·        vmnet2 - host only (ext)

·        vmnet 3 - host only (ha)

Your network topology could of course be different, I am choosing to build a topology close to the one I will be deploying the devices into in live. Obviously if you are deploying physical appliances instead of putting interfaces on to logical networks in VMware, you would have each of your interfaces in the corresponding VLAN's.

The default credentials for the vAppliance are root/default.

Type "config" at the initial prompt to launch the initial configuration utility. You can use native Linux commands such as ifconfig to set the mgmt address etc, but this way is easier.

In VMware I have found that the F5 sometimes screws up the order of the vnic's that are attached to the VM. Therefore I would recommend attaching a single vnic to the appliance, configuring your mgmt address then attaching the other vnics when you know what interface the F5 is interpreting as it's mgmt interface.

If you are having issues with the order of your network interfaces use the netstat -i to display all the physical interfaces that F5 has. In F5 TMOS mgmt IP's are also known as Self IP's, these are the much like NSIP's on the NetScaler platform.

The web interface has a different set of credentials admin/admin out of the box.

Before you can do anything you have to license the F5. Under Setup Utility and click Next.

Select the method of activation that suits you best. My F5 did not have a route to the Internet at this stage so I opted for the manual method. The Registration Key is the code F5 provide to you within an e-mail at the time you download a trial.

After a minute of so the verification will complete and you are free to start configuring some of the F5 features. The trial license comes with the Local Traffic (LTM) and Application Visibility and Reporting modules.

Problem

After bouncing my F5 a couple of times I started getting this when I logged in via the web interface "The Big-IP system has encountered a configuration problem that may prevent the configuration utility from functioning properly". I never managed to work out why, other that the F5 was shut down incorrectly. I just re provisioned another F5 device.

Azure External Load Balancer "Failed to join virtual machine to load balanced set. The operation failed Port 443 is already in use by one of the endpoints in this deployment."

When you try to create an external/public Azure Load Balancer for HTTPS traffic, you recieve "Failed to join virtual machine to load balanced set. The operation failed Port 443 is already in use by one of the endpoints in this deployment. Ensure that the port numbers are unique across endpoints within a deployment."

The key thing to understand here is that "deployment" actually means the Cloud Service, therefore this error is saying another application or service is currently using TCP port 443 within your current Cloud Service.

Problem spotted, if you click on your Cloud Service and review what is listed under Input Endpoints you will probably find there is a VM instance using TCP 443 behind your Cloud Service public IP address.

The best way to fix this (and only from a GUI) is to use the Preview Portal. Click on the VM that is listed under the Cloud Service as using port 443 and click All Settings then Load Balanced Sets.

As you will see I currently have two Load Balanced Sets, one for Internal traffic and one for External traffic. The public load balanced set is causing the issue here for me, the internal LB was created only a few minutes ago to load balance the internal "tier" of ADFS. The external LB is going to point to WAP endpoints on separate VM's.

So I clicked on the Public LB and it was currently using the Cloud Services IP on TCP 443.

After selecting the Public LB you must click Leave to remove the old endpoints from the LB, if this is the last endpoint within an LB the LB will be automatically deleted. As this was a stale LB instance that I had created weeks ago for testing I went ahead and removed it.

Hopefully deleting the load balanced set works first time, I believe some people have experienced problems doing this from the preview portal, and have had to revert to Powershell to complete the operation.

Now lets try re-creating our Public/External Load Balanced Set with the Cloud Service public IP and TCP port 443. Click the VM's you want to provision into the LB set and click All Settings/Load Balanced Sets, in my case this was the first of my WAP servers that were going to be internet facing. 

Click Join.

If you had the same problem as me and you have successfully removed everything that was conflicting the operation should complete successfully.

Configuring Citrix NetScaler v10.5 VPX High Availability to Load Balance HTTP Traffic

Citrix Netscaler is an Application Delivery Controller (ADC), by Citrix Systems. Netscaler is a widely deployed appliance that is available in three forms, the MPX (physical appliance), the VPX (virtual appliance) and the SPX, the physical appliance running XenServer that can host multiple virtual instances of Netscaler. If am using Netscaler to load balance ordinary HTTP traffic between two Windows Server 2008 R2 servers, with the IIS 7.5 role installed.

The topology that is being adopted is the “Two-armed mode, multi-subnet” model as show below, this is a Citrix recommended design when deploying Netscaler.

You can download a trial of the Citrix NetScaler 10.5 VPX from Citrix. It is available for XenServer, Hyper-V and VMware vSphere. In this example I am using vSphere, when you download the vSphere version of the VPX it comes as an OVF file that should be imported into vSphere. This can be done from the local machine you are using the vSphere Console from, so there is no need to upload the OVF to a vSphere datastore.

In Citrix Netscaler there is a significant difference between Clustering and High Availability, for one Clustering requires a special "clustering" license, where as traditional High Availability is provided as part of all the Netscaler editions.

In my example I am configuring two Netscaler VPX's in a HA pair, the following facts should be noted with HA and Citrix Netscaler;

·       Setup in Pairs (max 2 nodes)

·       Primary Node owns the VIP, SNIP (only one per pair)

·       Heartbeat every 200ms over UDP/3003 (3 second threshold for failover to initiate)

·       TCP port 3010, 3008 is used for node sync, file sync TCP 22

·       Configuration made on the primary are replicated over TCP 3011, 3009

As this is only a test environment I have created two new vSphere Standard Switches, with no adapter uplinks connected. The External vSwitch represents a DMZ, and the Internal my local area network.

My TCP/IP configuration(s) are as follows;

• RB_Test_Internal (LAN Subnet) – 192.168.0.0/24
• RB_Test_External (DMZ Subnet) – 172.16.0.0/24
• NS01 (NSIP) is 192.168.0.20/24
• NS02 (NSIP) is 192.168.0.21/24
• HA Pair (SNIP) is 192.168.0.23/24
• Web Server 1 is 192.168.0.50/24
• Web Server 2 is 192.168.0.51/24
• NS HA Pair VIP is 172.16.0.100/24

If you have reviewed the Citrix eDocs on Netscaler, the physical topology and logical subnet configuration I am doing in this example is referred to as a “mutli-armed, multi-subnet” deployment.

In a production environment you would probably have several dedicated uplinks from each of these vSwitches to provide connectivity to the physical networks. These uplinks would be either access ports or trunk ports depending where you are doing EST, or VGT for VLAN tagging.

Once the OVF appliance is imported, open a Console Connection to the VPX to set the initial configuration at this stage this will be the address that is used to manage the Netscaler VPX from your web browser.

Once the initial management IP is set, you can use a browser to connect to the Netscaler. It would suggest using Google Chrome as it seems to have the least amount of issues with Java when you are making administration changes.

When you login the first screen you will be presented with will have four options, the Netscaler (NSIP) should already be configured and show a green tick indicating this. 

The next part to configure is the Subnet IP Address (SNIP), this is an interface that is used to communicate with servers on the backend. Click on the Subnet IP Address option to begin configuring it. 

The SNIP address should be on the same subnet and VLAN that your internal servers that you are trying to load balance are. The wizard also provides a simplified break down of how the SNIP is used to communicate with the backend servers.

Step 3 is to configure a hostname for the device along with a DNS server, call this whatever you want a point it to your local DNS server, which will typically be a Domain Controller. You should also remember to manually create an (A) record for the Netscaler pointing to the correct IP in your DNS Forward Lookup Zones. This is usually forgotten as Microsoft devices use Dynamic DNS to do this automatically.

You will be prompted to restart the VPX appliance once your click on done. Step 4 is where you configure the license for the VPX appliance, you can get a 90 trial from Citrix that should be ample for testing. The following blog post here http://blog.ryanbetts.co.uk/2014/09/downloading-licensing-and-basic.html covers licensing the Netscaler VPX in detail.

Once the reboot is completed you should be able to log back into the VPX, and you will be taken to the Configuration window. To ensure your license file has been imported correct click on Licenses, the trial license should allow Load Balancing, Content Switching and SSL Offloading. 

The next step is to configure the High Availability between the two Netscaler VPX's, to do this click System, High Availability, from there you should see the first node in the state UP. Click the Add button. 

You should now enter the NSIP of the secondary node into the Node IP field. The username and password to login to the Netscaler should be the same on both these devices, I have left these as there default nsroot/nsroot.

When you click Create the Netscaler will prompt you to restart the running configuration and reboot the device.

Once the restart has completed, under the High Availability section you should see both nodes. As heart beating should be operating between the devices the first Netscaler VPX should still be operating as the Primary. 

The Actions menu can be used to show details, Force Synchronization and Force Failover between the two devices. 

The next step is to define the Services (or Servers, that you want to load balance between), to do this expand Traffic Management, then Load Balancing and click on Services. Click on Add to launch the wizard.

Configure the settings to be in line with your environment, I have two Web Servers (192.168.0.50 & 51) that are inside the local area network. You must create a Service for each of these servers. 

The servers are still offline for me at the moment therefore they appear as DOWN. This will automatically change when the Netscaler can communicate using the SNIP over ICMP.

Once I brought the servers online and there was connectivity between the Netscaler and the Web Servers the State changed to UP, and the lights went green. It would be a good time to save the running configuration.

Also from the Load Balancing menu, click on Virtual Servers, a Virtual Server in NetScaler is a Netscaler entity that external clients can use to access applications hosted on the servers. A Virtual Server is represented by a hostname, Virtual IP (VIP), port and protocol. Click Add to begin creating a new Virtual Server.

The name of the Netscaler Virtual Server is only locally relevant, therefore it does not make much difference what this is called here. I have configured my Virtual Server with the IP address of 172.16.0.100, which is the subnet that is in use on my DMZ side of the network. The Netscaler VPX's have two NIC's, one on each side of the two networks, LAN and DMZ.

Once you click OK, you will be prompted to enable the feature "LB", click Yes to this.

After this completes you will see under Services and Service Groups, no Virtual Server Service Bindings, click on the arrow to begin configuring this.

This is where we bind the services (or servers to be load balanced) to the Netscaler Virtual Server, click the Plus button to open the console.

Select both of the services that you created in the previous steps, in my example I have named both of my web servers "iisx". Click OK once this has been done.

Click Bind.

Click Done.

You must now click on the Method button from under the Advanced menu, this will expand the configuration screen and allow you to choose a High Availability method. Netscaler supports a number of different load balancing algorithms, the most common ones being;

• LEASTCONNECTION (Which service currently has the fewest client connections. This is the default load balancing algorithm.)
• ROUNDROBIN (Which service is at the top of a list of services. After that service is selected for a connection, it moves to the bottom of the list.)
• LEASTRESPONSETIME (Which load balanced server currently has the quickest response time.)
• URLHASH (A hash of the destination URL.)

A full list of the supported algorithms can be found at the following Citrix eDocs article http://support.citrix.com/proddocs/topic/netscaler-load-balancing-93/ns-lb-customizing-lbalgorithms-wrapper-con.html

I am going to configure LEAST CONNECTION at this stage, once done click OK. You should review the eDocs page to determine which algorithm will suit your needs the best.

The Virtual Server still appears to be “DOWN”, this will come online when the configuration is applied and saved to the memory. Click Done.

Once a refresh has occurred, click the Save icon.

Click Yes to confirm.

Now if you browse to the external VIP IP address, you should be connected to one of the web servers, I changed the default IIS landing page to ensure it was working correctly.