Configuring Citrix NetScaler v10.5 VPX High Availability to Load Balance HTTP Traffic

Citrix Netscaler is an Application Delivery Controller (ADC), by Citrix Systems. Netscaler is a widely deployed appliance that is available in three forms, the MPX (physical appliance), the VPX (virtual appliance) and the SPX, the physical appliance running XenServer that can host multiple virtual instances of Netscaler. If am using Netscaler to load balance ordinary HTTP traffic between two Windows Server 2008 R2 servers, with the IIS 7.5 role installed.

The topology that is being adopted is the “Two-armed mode, multi-subnet” model as show below, this is a Citrix recommended design when deploying Netscaler.

You can download a trial of the Citrix NetScaler 10.5 VPX from Citrix. It is available for XenServer, Hyper-V and VMware vSphere. In this example I am using vSphere, when you download the vSphere version of the VPX it comes as an OVF file that should be imported into vSphere. This can be done from the local machine you are using the vSphere Console from, so there is no need to upload the OVF to a vSphere datastore.

In Citrix Netscaler there is a significant difference between Clustering and High Availability, for one Clustering requires a special "clustering" license, where as traditional High Availability is provided as part of all the Netscaler editions.

In my example I am configuring two Netscaler VPX's in a HA pair, the following facts should be noted with HA and Citrix Netscaler;

·       Setup in Pairs (max 2 nodes)

·       Primary Node owns the VIP, SNIP (only one per pair)

·       Heartbeat every 200ms over UDP/3003 (3 second threshold for failover to initiate)

·       TCP port 3010, 3008 is used for node sync, file sync TCP 22

·       Configuration made on the primary are replicated over TCP 3011, 3009

As this is only a test environment I have created two new vSphere Standard Switches, with no adapter uplinks connected. The External vSwitch represents a DMZ, and the Internal my local area network.

My TCP/IP configuration(s) are as follows;

• RB_Test_Internal (LAN Subnet) – 192.168.0.0/24
• RB_Test_External (DMZ Subnet) – 172.16.0.0/24
• NS01 (NSIP) is 192.168.0.20/24
• NS02 (NSIP) is 192.168.0.21/24
• HA Pair (SNIP) is 192.168.0.23/24
• Web Server 1 is 192.168.0.50/24
• Web Server 2 is 192.168.0.51/24
• NS HA Pair VIP is 172.16.0.100/24

If you have reviewed the Citrix eDocs on Netscaler, the physical topology and logical subnet configuration I am doing in this example is referred to as a “mutli-armed, multi-subnet” deployment.

In a production environment you would probably have several dedicated uplinks from each of these vSwitches to provide connectivity to the physical networks. These uplinks would be either access ports or trunk ports depending where you are doing EST, or VGT for VLAN tagging.

Once the OVF appliance is imported, open a Console Connection to the VPX to set the initial configuration at this stage this will be the address that is used to manage the Netscaler VPX from your web browser.

Once the initial management IP is set, you can use a browser to connect to the Netscaler. It would suggest using Google Chrome as it seems to have the least amount of issues with Java when you are making administration changes.

When you login the first screen you will be presented with will have four options, the Netscaler (NSIP) should already be configured and show a green tick indicating this. 

The next part to configure is the Subnet IP Address (SNIP), this is an interface that is used to communicate with servers on the backend. Click on the Subnet IP Address option to begin configuring it. 

The SNIP address should be on the same subnet and VLAN that your internal servers that you are trying to load balance are. The wizard also provides a simplified break down of how the SNIP is used to communicate with the backend servers.

Step 3 is to configure a hostname for the device along with a DNS server, call this whatever you want a point it to your local DNS server, which will typically be a Domain Controller. You should also remember to manually create an (A) record for the Netscaler pointing to the correct IP in your DNS Forward Lookup Zones. This is usually forgotten as Microsoft devices use Dynamic DNS to do this automatically.

You will be prompted to restart the VPX appliance once your click on done. Step 4 is where you configure the license for the VPX appliance, you can get a 90 trial from Citrix that should be ample for testing. The following blog post here http://blog.ryanbetts.co.uk/2014/09/downloading-licensing-and-basic.html covers licensing the Netscaler VPX in detail.

Once the reboot is completed you should be able to log back into the VPX, and you will be taken to the Configuration window. To ensure your license file has been imported correct click on Licenses, the trial license should allow Load Balancing, Content Switching and SSL Offloading. 

The next step is to configure the High Availability between the two Netscaler VPX's, to do this click System, High Availability, from there you should see the first node in the state UP. Click the Add button. 

You should now enter the NSIP of the secondary node into the Node IP field. The username and password to login to the Netscaler should be the same on both these devices, I have left these as there default nsroot/nsroot.

When you click Create the Netscaler will prompt you to restart the running configuration and reboot the device.

Once the restart has completed, under the High Availability section you should see both nodes. As heart beating should be operating between the devices the first Netscaler VPX should still be operating as the Primary. 

The Actions menu can be used to show details, Force Synchronization and Force Failover between the two devices. 

The next step is to define the Services (or Servers, that you want to load balance between), to do this expand Traffic Management, then Load Balancing and click on Services. Click on Add to launch the wizard.

Configure the settings to be in line with your environment, I have two Web Servers (192.168.0.50 & 51) that are inside the local area network. You must create a Service for each of these servers. 

The servers are still offline for me at the moment therefore they appear as DOWN. This will automatically change when the Netscaler can communicate using the SNIP over ICMP.

Once I brought the servers online and there was connectivity between the Netscaler and the Web Servers the State changed to UP, and the lights went green. It would be a good time to save the running configuration.

Also from the Load Balancing menu, click on Virtual Servers, a Virtual Server in NetScaler is a Netscaler entity that external clients can use to access applications hosted on the servers. A Virtual Server is represented by a hostname, Virtual IP (VIP), port and protocol. Click Add to begin creating a new Virtual Server.

The name of the Netscaler Virtual Server is only locally relevant, therefore it does not make much difference what this is called here. I have configured my Virtual Server with the IP address of 172.16.0.100, which is the subnet that is in use on my DMZ side of the network. The Netscaler VPX's have two NIC's, one on each side of the two networks, LAN and DMZ.

Once you click OK, you will be prompted to enable the feature "LB", click Yes to this.

After this completes you will see under Services and Service Groups, no Virtual Server Service Bindings, click on the arrow to begin configuring this.

This is where we bind the services (or servers to be load balanced) to the Netscaler Virtual Server, click the Plus button to open the console.

Select both of the services that you created in the previous steps, in my example I have named both of my web servers "iisx". Click OK once this has been done.

Click Bind.

Click Done.

You must now click on the Method button from under the Advanced menu, this will expand the configuration screen and allow you to choose a High Availability method. Netscaler supports a number of different load balancing algorithms, the most common ones being;

• LEASTCONNECTION (Which service currently has the fewest client connections. This is the default load balancing algorithm.)
• ROUNDROBIN (Which service is at the top of a list of services. After that service is selected for a connection, it moves to the bottom of the list.)
• LEASTRESPONSETIME (Which load balanced server currently has the quickest response time.)
• URLHASH (A hash of the destination URL.)

A full list of the supported algorithms can be found at the following Citrix eDocs article http://support.citrix.com/proddocs/topic/netscaler-load-balancing-93/ns-lb-customizing-lbalgorithms-wrapper-con.html

I am going to configure LEAST CONNECTION at this stage, once done click OK. You should review the eDocs page to determine which algorithm will suit your needs the best.

The Virtual Server still appears to be “DOWN”, this will come online when the configuration is applied and saved to the memory. Click Done.

Once a refresh has occurred, click the Save icon.

Click Yes to confirm.

Now if you browse to the external VIP IP address, you should be connected to one of the web servers, I changed the default IIS landing page to ensure it was working correctly. 

Active Directory Federation Services (AD FS) 3.0 Configuration Wizard "An error occurred while trying to configure this machine". "A certificate with the thumbprint xxxx was not found in the LocalMachine certificate store."

I was recently installing Active Directory Federation Services (AD FS) 3.0 which is shipped with Windows Server 2012 R2. I had provisioned and configured the first AD FS server which was utilizing the WID database as this was only in a test lab. I had requested a certificate from my internal CA with the Common Name fs.domain.com from the first AD FS server. It was issued and imported without problem into the Personal store on the primary AD FS server.

Once I had installed the AD FS role on the second server I was following the configuration wizard to get it added to the AD FS farm, I received the error "An error occurred while trying to configure this machine". "A certificate with the thumbprint xxxx was not found in the LocalMachine certificate store." this was because I had forgot that all of the AD FS components in a farm must use the exact same certificate.

I moved back to the primary AD FS server and launched the Certificates (Computer) MMC, from there I opened the fs.domain.com certificate from the Personal store, I checked that the thumbprint of this certificate matched the once required by the AD FS wizard on the secondary server.

The certificate must be Exported from the Primary and Imported into the Personal store on the secondary AD FS server (or any other FS servers such as ADFS Proxy/WAP's).

Rerun the AD FS configuration wizard and ensure you tick the box Overwrite Existing AD FS Configuration Database Data.

AD CS: The Revocation Function was Unable to Check Revocation Because the Revocation Server was Offline 0x800092013 (-2146885613 CRYPT_E_REVOCATION_OFFLINE) The Request ID is 57

When you try to request a new certificate you receive the following error "The Revocation Function was Unable to Check Revocation Because the Revocation Server was Offline 0x800092013 (-2146885613 CRYPT_E_REVOCATION_OFFLINE) The Request ID is 57" and the certificate fails to enrol.

Solution 

 

If you check the Certificate Authority console you will see the CA is not operational as the icon has turned black.

 

 

After some research this error is caused by one of the following things; 

• The CRL location (CDP) is unreachable
• The CRL location (CDP) is unpublished
• The CRL has expired and a new one was not published
• The CRL is reachable but the cient is picking up from the old cache

I knew the issuing CA was online so the first one was ruled out, I knew the CRL had been published to the CDP location as it was operational last week. It then began to look like the CRL had expired. I opened up the CRL file which was hosted on my Issuing CA, and checked the Next Update field, as you can see it was out of date by more than two weeks.

 

 

On the offline CA, where the CRL is generated I checked to see if the CRL dates matched, and they did not meaning that the new CRL had not been copied across to the issuing CA.

 

 

The next step was to copy the new CRL file across to the issuing CA into the location that is configured to host the CRL, in my environment the CRL is published to an IIS website.

 

 

Once this complete the Certificate Authority came back online and certs could be issues.

 

 

Workaround

If for whatever reason you did not want the CRL to be updated, you can configure the CA to ignore any errors related to the CRL life, it is done using the following commands;

 

 

 

 

 

If the CA is set to ignore CRL errors you are at risk your AD CS infrastructure is not functioning correctly.

AD FS 3.0 and Hardware Load Balancers Compatibility with TLS SSL SNI

For organisations considering the move to Office 365, you will probably have explored the requirement for Active Directory Federation Services (AD FS) to provide Single Sign On (SSO) to your domain users when connecting to Office 365 services. Without AD FS in place to federate and project users domain credentials to Office 365, users would have to maintain a separate set of logins details for both the domain and Office 365.

If you have looked at AD FS and are planning on deploying it on Windows Server 2012 R2, with hardware-load balancers I suggest you read and understand this post in full. Active Directory Federation Services in Windows Server 2012 R2 is currently at version 3.0. This is different to even Windows Server 2012, which was shipped with AD FS 2.1.  You can off course use Windows Network Load Balancing to load balance AD FS but I am sure you will agree for a component as important as AD FS a dedicated appliance would be better. If you decide to deploy AD FS, it is vital the service is highly available, if for whatever reason your AD FS infrastructure goes down on-site users will not be able to access O365 services.

One of the improvements in AD FS 3.0 is built-in compatibility for Server Name Indication (SNI), which is an extension of the TLS element of the SSL protocol. Although AD FS 3.0 supports SNI not all networking equipment and end devices do. SNI allows multiple SSL certificates to be bound to the same IP Address and Port Number, which allows support for multiple SSL (or HTTPS) connections to the same address.

SNI works by providing application server host names during the SSL handshaking phases, thus allowing multiple HTTPS applications to reside behind a single IP Address.

Although AD FS 3.0 supports the SNI feature, not all hardware load balancers do. It is important to consider this when making a purchasing decision for your hardware load balancers.

At the time of writing the following servers do offer support for SNI:

Citrix NetScaler 9.2 or higher

F5 Networks Local Traffic Manager, version 11.1 or higher

KEMP Load Balancers (as of October 2014)

Apache 2.2.12 or higher

Apache Traffic Server 3.2.0 or higher

Apache Tomcat on Java 7 or higher

Microsoft Internet Information Server (IIS) 8

Web browsers also require support for SNI; all of the recent versions of Internet Explorer from 7 onwards support SNI along with Google Chrome and Mozilla Firefox.

Windows Server 2012 "Features on Demand": Configuring Static Path to “WinSxS” Folder via Group Policy

In my lab I tend to spin up loads of Windows Server 2012 R2 servers to test various things. One thing that always causes me frustration is having to manually point the \sources\sxs folder when installing a new Server Role, via Server Manager or PowerShell.  Its frustrating because I normally forget to do it, and the installation fails and I need to start again.

To configure a network based copy of the SXS folder and have it automatically mapped via GPO, first copy the \sources\sxs folder from the DVD media to somewhere you can share the folder from. Then configure the share and NTFS permissions, I have allowed Full Control to Everyone on the share permissions, and Full Control to Domain Computers on the NTFS permissions.

As I want this setting to be global across the domain I am going to put the setting in the Default Domain Policy open the GPMC and edit the GPO you want to host the settings.

Expand Computer Configuration/Policies/Administrative Templates/System and double click on the policy Specify settings for optional component installation and component repair.

Click the Enable button and enter the UNC path to the SXS folder you copied over, if the access permissions are not configured correctly this obviously will not work. Click Apply and OK.

Now when you try to install Server Roles or Features via Server Manager or PowerShell the machine will have a known location for the required binaries.

Active Directory Certificate Services (AD CS): The Revocation Function was Unable to Check Revocation Because the Server was Offline 0x80092013 (-2146885613 CRYTP_E_REVOCATION_OFFLINE)

You are configuring an Active Directory Certificate Services (AD CS) infrastructure, and you are at the point where you are importing the certificate that was generated by your certificate request from the Root CA. When you try to use the Import CA Certificate from the Certificate Authority MMC you receive the following error "The Revocation Function was Unable to Check Revocation Because the Server was Offline 0x80092013 (-2146885613 CRYTP_E_REVOCATION_OFFLINE)."

This was because when I configured the CDP and AIA information when I was configuring the Root CA I added HTTP locations for the Issuing CA.

When I look at the certificate that I am trying to import, under CRL Distribution Points there are two listed. The Offline CA currently has no network connectivity, therefore it cannot resolve that one, even if it could it will be taken offline after the initial configuration of the AD CS infrastructure. Notice how there is a URL pointing to the Issuing CA.

The problem here was I had forgotten to copy the CRL file generated from the Root CA into the IIS Web Server folder on the Issuing CA. 

Once I copied the CRL file over to the Web Server folder that was accessible in the CRL Distribution Points URL I tried starting the AD CS service again.