Configuring Citrix NetScaler to Load Balance ADFS v3.0

Configuring Citrix NetScaler to Load Balance ADFS v3.0

The link below is broken, I also do not have a copy of the full document I created in July 2015. Do not e-mail me directly asking for a copy.

To ensure ADFS and the WAP servers are highly available a hardware load balance is recommended. In this example I have outlined how to configure Citrix NetScalers to do this. If you are deploying ADFS for Office 365, it is important that the service is highly available otherwise users will not be able to authenticate to ADFS. 

Microsoft supports NetScaler as a hardware load balance but there is lack of documentation around how to deploy the two solutions together.

Some of the configuration settings such as MAC based forwarding, SSL bridging, session persistence and Client IP header are some of the settings that are not obviously required.

Although the full guide I created no longer exists, I have summarised how to configure the NetScaler to load balance ADFS 3.0 below. This is not a full and conclusive guide. I won't be re-doing the guide until I am asked to do it for a customer. 

Although this diagram is for MS Lync, it's similar for ADFS. Imagine you have 2 x WAP in the DMZ and 2 x ADFS in your LAN. 

Most places won't have two separate physical NetScaler pairs. The above topology can be achieved using a single pair of Netscalers, with network interfaces on each of the DMZ/LAN networks. Netscaler vServers can then be used to load balance each component. If your security requirements state you must have a physical break then it's different.

High level steps (from memory)

• Base configure the Netscalers (NSIP, SIP, routes, default gateways etc)
• at this point it's important to have an interface on each network. If you are deploying the ADFS WAP servers, these should be in your DMZ. Therefore an interface from the NS is required on each of the separate Layer 2 networks. 
• if you have problems research "Mac based forwarding"

• Configure high availability of the Netscalers
• Easy to do, Google it. I normally set my HA configuration to Active/Passive.

• Define the server's both ADFS and WAP
• Traffic Management/Load Balancing/Server
• Create a new server object for each of the ADFS/WAP's. Set the protocol to SSL Bridge
• Create a new service one for the WAP's and one for the ADFS's. Traffic Management/Load Balancing/Services
• service 1 = "adfs_https" protocol SSL Bridge
• service 2 = "wap_https" protocol SSL Bridge
• It's important the protocol type matches on the servers/services.

At this point, if you servers are listening on TCP 443 (https) and you have configured the networking correctly the server/services will show as "up" and the light will be green. If not there is some kind of networking problem (remember your WAP servers don't listen on 443 until you have them successfully paired with the ADFS servers)

• You should configure SSL session persistence (I cant remember the exactly menu, you either do this in the service configuration or on the vServer, check eDocs)
• Now create a new vServer for each the WAP's and the ADFS's
• vserver1 = "adfs" give the vserver an IP on the network the backend servers sit on
• vserver2 = "wap" same as above
• When you create the vserver, bind them to the corresponding service (adfs - adfs etc), and set the load balancing method to round robin.

DNS 

Now you have the NS configured, on the WAP servers create a hosts file entry to point your ADFS service name i.e sso.domain.com to the NS vServer for the ADFS servers. Try to run the WAP-ADFS pairing. If you get an error check out the bindings.

http://blog.ryanbetts.co.uk/2015/12/adfs-wap-unable-to-retrieve-proxy.html

Citrix NetScaler High Availability Configuration "Remote Node x.x.x.x PARTIAL_FAIL" & "Remote Node x.x.x.x COMPLETE_FAIL" HA Relationship Fails to Form

You have two identical NetScaler MPX 5560's, and they both have Management interfaces (NSIP's) using the 0/1 interface on each appliance which are configured on VLAN 8. When you try to configure High Availability between the two appliances, you recieve the following "Node State - Not Up" on the secondary appliance.

Firstly I clicked on the Dashboard, to see from the System Log if there were any clues to why the HA relationship was failing to form correctly, the System Log was stamped with "remote node x.x.x.x PARTIAL_FAIL" and "remote node x.x.x.x COMPLETE_FAIL".

The next step was to take a closer look at the /var/nslog/newnslog log file which can be accessed through the Web GUI from Configuration>Diagnostics>View Events.

Not much more information was listed in this log but my attention soon drifted towards the number of interfaces that were reporting "No heartbeats since HA startup".

More of the same referencing the "PARTIAL_FAIL" error message that was present in the System Log.

The first thing I tried was to disabled all of the NetScalers interfaces with the exception of the 0/1 interface which was the NSIP address on each appliance. You will receive an error if you try to disabled the Loopback interface.

Once all of the interfaces were in the "disabled" state, I tried to run the High Availability wizard again and this time it worked. It was obvious the issue was around a setting on the enabled interfaces.

For good practice I only enabled the two additional interfaces that were currently in use on the appliances anyway, which were 1/1 and 1/2.

I left most of the settings at defaults with the exception of "HA Monitoring", which I changed to the OFF state. This appeared to fix the original issue.

The next stage was to re-configure the NetScaler High Availability pair, to do this click System>High Availability. In my deployment I want the first appliance to always remain the Primary node unless a failure occurs, therefore I highlighted the appliance and clicked Edit.

From the Configure HA Node menu, the High Availability Status can be configured for each of the NetScaler nodes, I set the first appliance to STAY PRIMARY.

And the second appliance to STAY SECONDARY.

Once I have completed all of the above configuration I saved the NetScaler running config. I then run through the HA pairing wizard again, which completed successfully.

Citrix NetScaler Firmware Upgrade - 10.1 Build 127.11 to 10.5 Build 56.15 with the Web GUI

To upgrade the firmware of a Citrix NetScaler is a very easy thing to do, I have documented the steps for anyone who has never done it before, or has limited exposure to the NetScaler platform. Please do not contact me asking for the NetScaler firmware, as I cannot provide this for you. Login to your Citrix account for the latest firmwares. 

The easiest way to do it, is to download the new firmware to your local device. The wizard then allows you to upload it to the device, it can be sourced from the flash of the device.

Open the Web GUI, and login. From the Configuration screen, click Upgrade Wizard...

Click Next, on the Introduction screen.

Select Local Computer, and click Browse, then search for the firmware file that you have downloaded from Citrix.

If you have a license file currently configured on your it should show here, I am actually not sure if it lets your upgrade the firmware if you don't have a license file imported.

I prefer to disable the Automatically move file to create space in flash, as this will basically delete the old firmware. For safety I would rather have that firmware image available in case the upgrade does not work for some reason, or if it needs rolled back. Select Reboot after successful installation and click Next.

Click Finish.

The wizard will upload the new firmware image from your local device to the NetScaler appliance.

You will be prompted if you want to enable Citrix Call Home, this allows the appliance to report issues directly to Citrix, this is entirely up to you if you enable this.

Once the device (or vAppliance) reboot, the Web GUI interface changes to darker colours. 

You should note that if you are going to configure services such as High Availability, both the NetScalers have to be exactly the same. This includes the platform, so you cannot mix VPX's with MPX's for example, the firmware and network interfaces used has to also match on neighboring devices.

Citrix NetScaler 10.1 MPX, throws the following error when you try to run certain operations from the Web GUI "Cannot load Applet, Java Applet could not be loaded Details Possible reasons: JRE(Java Runtime Environment) not installed. JRE is installed but not running."

Disclaimer: Many people have blogged about this issue with NetScaler and Java Runtime, the reason I have done it again is because although there are blogs documenting similar fixes, none of these fixes resolved the problem for me alone. Therefore I have consolidated a single post, documenting all of the steps it took me to fix the problem in my environment. I have referenced the articles I used below to give the original contributors the credit. 

Citrix NetScaler, throws the following error when you try to run certain operations from the Web GUI "Cannot load Applet, Java Applet could not be loaded Details Possible reasons: JRE(Java Runtime Environment) not installed. JRE is installed but not running." I came across the issue trying to upgrade the firmware and enabling HA on two physical appliances.

In this environment I have the latest Java Runtime installed (April 2015) which is version 8 update 45. Open Control Panel, and then open the Java control panel. Click on the General Tab, and ensure the Keep Temporary Files on my Computer is disabled.

Then click Security and then Edit Site List..., and put an entry in for your problem NetScaler device here. 

Click on the Advanced tab, and set Mixed Code (sandboxed vs. trusted) security verification to Disable verification (not recommended) and Perform signed code certificate revocation checks on and Do not check (not recommneded).

References:

http://terenceluk.blogspot.co.uk/2015/01/citrix-netscaler-upgrade-wizard-fails.html

http://discussions.citrix.com/topic/338689-java-upgrade-to-latest-version-causes-gui-issues-netscaler-vpx200-100/

http://danielruiz.org/2014/02/04/netscaler-10-1-unable-to-load-applet/

Citrix NetScaler 10.5 VPX Configuring Active Directory Authentication

It is possible to integrate the Netscaler with Active Directory for Administrator authentication, this means that administrators can use their dedicated admin account to login to the Netscaler and make configuration changes. This prevents organizations from having to manage a local user database on the Netscaler appliances. You should however always have at least one backup local "super user" account to hand in case the LDAP integration breaks for some reason.

Login to the Netscaler using local credentials, expand System and then Authentication, from there click on the LDAP option.

In the Name field enter something descriptive for the policy you are about to create, I have chosen "AD Connector" in my environment. Click the + button next to the Server field.

When the Create Authentication LDAP Server window opens you must populate it with details specific for your network. As this is only a test lab I do not have DNS setup correctly so I have used the IP of a Domain Controller. In addition to this I have chosen to use ordinary LDAP, not LDAP:S which you really should be using in a production network. By default LDAP uses TCP port 389 and LDAP:S uses TCP 636.

Scroll down within the same window until you get to Connection Settings. These settings again must be populated to match your infrastructure. The Base DN should be taken from the root of the Domain. This can be viewed if you enable the Advanced Features view from the View menu from ADCU. You click on Attribute Editor.

You can find the Administrators Bind DN in exactly the same way, the screenshots below show exactly where these details can be viewed. Click on the BindDN Password tick box and enter the password for the Administrator account you have used.

Scroll down further until you see the following properties, you should populate these to be the same as mine except the Group Attribute field, this should be set to the Active Directory group you are going to use to manage users that are authorized to access the Netscaler. Append "memberof=" before the CN path. Click OK.

When you are back on the Configure Authentication page you must add an expression, all this has to be set to is ns_true, click OK once this is in place.

From the Policies screen, you will notice that the policy is not bound to anything, for this to apply across the entire Netscaler it must be Globally Bound. Click the Policy once to highlight it and select Global Bindings.

Select the policy from the Policy Binding drop down, and leave the priority at 100. Click Bind.

The icon under Globally Bound? should not change to a green tick.

The next step is to create a Netscaler Security Group to map to the Active Directory Security group, from System expand User Administration and click Groups.

You must create a security group with EXACTLY THE SAME NAME AS THE ACTIVE DIRECTORY GROUP otherwise it will not work.

Scroll down to Policy Name and give the group "superuser" permissions. Click Create.

Save the settings. 

Now users who are in the Active Directory Security Group should be able to login to the Netscaler using DOMAIN\User with their Active Directory password.

Configuring Citrix NetScaler v10.5 VPX High Availability to Load Balance HTTP Traffic

Citrix Netscaler is an Application Delivery Controller (ADC), by Citrix Systems. Netscaler is a widely deployed appliance that is available in three forms, the MPX (physical appliance), the VPX (virtual appliance) and the SPX, the physical appliance running XenServer that can host multiple virtual instances of Netscaler. If am using Netscaler to load balance ordinary HTTP traffic between two Windows Server 2008 R2 servers, with the IIS 7.5 role installed.

The topology that is being adopted is the “Two-armed mode, multi-subnet” model as show below, this is a Citrix recommended design when deploying Netscaler.

You can download a trial of the Citrix NetScaler 10.5 VPX from Citrix. It is available for XenServer, Hyper-V and VMware vSphere. In this example I am using vSphere, when you download the vSphere version of the VPX it comes as an OVF file that should be imported into vSphere. This can be done from the local machine you are using the vSphere Console from, so there is no need to upload the OVF to a vSphere datastore.

In Citrix Netscaler there is a significant difference between Clustering and High Availability, for one Clustering requires a special "clustering" license, where as traditional High Availability is provided as part of all the Netscaler editions.

In my example I am configuring two Netscaler VPX's in a HA pair, the following facts should be noted with HA and Citrix Netscaler;

·       Setup in Pairs (max 2 nodes)

·       Primary Node owns the VIP, SNIP (only one per pair)

·       Heartbeat every 200ms over UDP/3003 (3 second threshold for failover to initiate)

·       TCP port 3010, 3008 is used for node sync, file sync TCP 22

·       Configuration made on the primary are replicated over TCP 3011, 3009

As this is only a test environment I have created two new vSphere Standard Switches, with no adapter uplinks connected. The External vSwitch represents a DMZ, and the Internal my local area network.

My TCP/IP configuration(s) are as follows;

• RB_Test_Internal (LAN Subnet) – 192.168.0.0/24
• RB_Test_External (DMZ Subnet) – 172.16.0.0/24
• NS01 (NSIP) is 192.168.0.20/24
• NS02 (NSIP) is 192.168.0.21/24
• HA Pair (SNIP) is 192.168.0.23/24
• Web Server 1 is 192.168.0.50/24
• Web Server 2 is 192.168.0.51/24
• NS HA Pair VIP is 172.16.0.100/24

If you have reviewed the Citrix eDocs on Netscaler, the physical topology and logical subnet configuration I am doing in this example is referred to as a “mutli-armed, multi-subnet” deployment.

In a production environment you would probably have several dedicated uplinks from each of these vSwitches to provide connectivity to the physical networks. These uplinks would be either access ports or trunk ports depending where you are doing EST, or VGT for VLAN tagging.

Once the OVF appliance is imported, open a Console Connection to the VPX to set the initial configuration at this stage this will be the address that is used to manage the Netscaler VPX from your web browser.

Once the initial management IP is set, you can use a browser to connect to the Netscaler. It would suggest using Google Chrome as it seems to have the least amount of issues with Java when you are making administration changes.

When you login the first screen you will be presented with will have four options, the Netscaler (NSIP) should already be configured and show a green tick indicating this. 

The next part to configure is the Subnet IP Address (SNIP), this is an interface that is used to communicate with servers on the backend. Click on the Subnet IP Address option to begin configuring it. 

The SNIP address should be on the same subnet and VLAN that your internal servers that you are trying to load balance are. The wizard also provides a simplified break down of how the SNIP is used to communicate with the backend servers.

Step 3 is to configure a hostname for the device along with a DNS server, call this whatever you want a point it to your local DNS server, which will typically be a Domain Controller. You should also remember to manually create an (A) record for the Netscaler pointing to the correct IP in your DNS Forward Lookup Zones. This is usually forgotten as Microsoft devices use Dynamic DNS to do this automatically.

You will be prompted to restart the VPX appliance once your click on done. Step 4 is where you configure the license for the VPX appliance, you can get a 90 trial from Citrix that should be ample for testing. The following blog post here http://blog.ryanbetts.co.uk/2014/09/downloading-licensing-and-basic.html covers licensing the Netscaler VPX in detail.

Once the reboot is completed you should be able to log back into the VPX, and you will be taken to the Configuration window. To ensure your license file has been imported correct click on Licenses, the trial license should allow Load Balancing, Content Switching and SSL Offloading. 

The next step is to configure the High Availability between the two Netscaler VPX's, to do this click System, High Availability, from there you should see the first node in the state UP. Click the Add button. 

You should now enter the NSIP of the secondary node into the Node IP field. The username and password to login to the Netscaler should be the same on both these devices, I have left these as there default nsroot/nsroot.

When you click Create the Netscaler will prompt you to restart the running configuration and reboot the device.

Once the restart has completed, under the High Availability section you should see both nodes. As heart beating should be operating between the devices the first Netscaler VPX should still be operating as the Primary. 

The Actions menu can be used to show details, Force Synchronization and Force Failover between the two devices. 

The next step is to define the Services (or Servers, that you want to load balance between), to do this expand Traffic Management, then Load Balancing and click on Services. Click on Add to launch the wizard.

Configure the settings to be in line with your environment, I have two Web Servers (192.168.0.50 & 51) that are inside the local area network. You must create a Service for each of these servers. 

The servers are still offline for me at the moment therefore they appear as DOWN. This will automatically change when the Netscaler can communicate using the SNIP over ICMP.

Once I brought the servers online and there was connectivity between the Netscaler and the Web Servers the State changed to UP, and the lights went green. It would be a good time to save the running configuration.

Also from the Load Balancing menu, click on Virtual Servers, a Virtual Server in NetScaler is a Netscaler entity that external clients can use to access applications hosted on the servers. A Virtual Server is represented by a hostname, Virtual IP (VIP), port and protocol. Click Add to begin creating a new Virtual Server.

The name of the Netscaler Virtual Server is only locally relevant, therefore it does not make much difference what this is called here. I have configured my Virtual Server with the IP address of 172.16.0.100, which is the subnet that is in use on my DMZ side of the network. The Netscaler VPX's have two NIC's, one on each side of the two networks, LAN and DMZ.

Once you click OK, you will be prompted to enable the feature "LB", click Yes to this.

After this completes you will see under Services and Service Groups, no Virtual Server Service Bindings, click on the arrow to begin configuring this.

This is where we bind the services (or servers to be load balanced) to the Netscaler Virtual Server, click the Plus button to open the console.

Select both of the services that you created in the previous steps, in my example I have named both of my web servers "iisx". Click OK once this has been done.

Click Bind.

Click Done.

You must now click on the Method button from under the Advanced menu, this will expand the configuration screen and allow you to choose a High Availability method. Netscaler supports a number of different load balancing algorithms, the most common ones being;

• LEASTCONNECTION (Which service currently has the fewest client connections. This is the default load balancing algorithm.)
• ROUNDROBIN (Which service is at the top of a list of services. After that service is selected for a connection, it moves to the bottom of the list.)
• LEASTRESPONSETIME (Which load balanced server currently has the quickest response time.)
• URLHASH (A hash of the destination URL.)

A full list of the supported algorithms can be found at the following Citrix eDocs article http://support.citrix.com/proddocs/topic/netscaler-load-balancing-93/ns-lb-customizing-lbalgorithms-wrapper-con.html

I am going to configure LEAST CONNECTION at this stage, once done click OK. You should review the eDocs page to determine which algorithm will suit your needs the best.

The Virtual Server still appears to be “DOWN”, this will come online when the configuration is applied and saved to the memory. Click Done.

Once a refresh has occurred, click the Save icon.

Click Yes to confirm.

Now if you browse to the external VIP IP address, you should be connected to one of the web servers, I changed the default IIS landing page to ensure it was working correctly.