The server vc01.company.com could not interpret the client's request. (The remote server returned an error: (503) Server Unavailable. Call "Service.RetrieveContent" for object "ServiceInstance" on Server "dreamline".

When you try to login to the VMware vSphere 6.0 vCenter using the C# client you recieve the following error "The server vc01.company.com could not interpret the client's request. (The remote server returned an error: (503) Server Unavailable. Call "Service.RetrieveContent" for object "ServiceInstance" on Server "dreamline".", the Web Client is also not responding.

To fix, firstly reboot the vCenter server. Either RDP to it, or login to the ESXi host that is hosting the VM and restart the guest gracefully.

Let the vCenter come back online and ensure you have left enough time for all of the services to start if you are using vCenter on a Windows Server. Once noticable improvement in vSphere 6 is the speed all of the servers come online.

Secondly, SSH to the problem ESXi hosts and issue the following command;

/sbin/services.sh restart  

You should then be able to login to vCenter.

Configuring Citrix NetScaler to Load Balance ADFS v3.0

Configuring Citrix NetScaler to Load Balance ADFS v3.0

The link below is broken, I also do not have a copy of the full document I created in July 2015. Do not e-mail me directly asking for a copy.

To ensure ADFS and the WAP servers are highly available a hardware load balance is recommended. In this example I have outlined how to configure Citrix NetScalers to do this. If you are deploying ADFS for Office 365, it is important that the service is highly available otherwise users will not be able to authenticate to ADFS. 

Microsoft supports NetScaler as a hardware load balance but there is lack of documentation around how to deploy the two solutions together.

Some of the configuration settings such as MAC based forwarding, SSL bridging, session persistence and Client IP header are some of the settings that are not obviously required.

Although the full guide I created no longer exists, I have summarised how to configure the NetScaler to load balance ADFS 3.0 below. This is not a full and conclusive guide. I won't be re-doing the guide until I am asked to do it for a customer. 

Although this diagram is for MS Lync, it's similar for ADFS. Imagine you have 2 x WAP in the DMZ and 2 x ADFS in your LAN. 

Most places won't have two separate physical NetScaler pairs. The above topology can be achieved using a single pair of Netscalers, with network interfaces on each of the DMZ/LAN networks. Netscaler vServers can then be used to load balance each component. If your security requirements state you must have a physical break then it's different.

High level steps (from memory)

• Base configure the Netscalers (NSIP, SIP, routes, default gateways etc)
• at this point it's important to have an interface on each network. If you are deploying the ADFS WAP servers, these should be in your DMZ. Therefore an interface from the NS is required on each of the separate Layer 2 networks. 
• if you have problems research "Mac based forwarding"

• Configure high availability of the Netscalers
• Easy to do, Google it. I normally set my HA configuration to Active/Passive.

• Define the server's both ADFS and WAP
• Traffic Management/Load Balancing/Server
• Create a new server object for each of the ADFS/WAP's. Set the protocol to SSL Bridge
• Create a new service one for the WAP's and one for the ADFS's. Traffic Management/Load Balancing/Services
• service 1 = "adfs_https" protocol SSL Bridge
• service 2 = "wap_https" protocol SSL Bridge
• It's important the protocol type matches on the servers/services.

At this point, if you servers are listening on TCP 443 (https) and you have configured the networking correctly the server/services will show as "up" and the light will be green. If not there is some kind of networking problem (remember your WAP servers don't listen on 443 until you have them successfully paired with the ADFS servers)

• You should configure SSL session persistence (I cant remember the exactly menu, you either do this in the service configuration or on the vServer, check eDocs)
• Now create a new vServer for each the WAP's and the ADFS's
• vserver1 = "adfs" give the vserver an IP on the network the backend servers sit on
• vserver2 = "wap" same as above
• When you create the vserver, bind them to the corresponding service (adfs - adfs etc), and set the load balancing method to round robin.

DNS 

Now you have the NS configured, on the WAP servers create a hosts file entry to point your ADFS service name i.e sso.domain.com to the NS vServer for the ADFS servers. Try to run the WAP-ADFS pairing. If you get an error check out the bindings.

http://blog.ryanbetts.co.uk/2015/12/adfs-wap-unable-to-retrieve-proxy.html

Resetting an F5 Networks Big-IP System's Root Password from the Console

I was recently asked to configure a pair of F5 Big-IP's for a customer, the Big-IP system is something I have only limited exposure to but as I know Citrix NetScaler (which is also an ADC) I thought the concepts would be pretty similar, which they are!

I did a configuration using the Big-IP virtual appliance in ESXi so that I could test the functionality using emulated networks etc. So the plan was to export the config and then replace the config on the new box, thus saving time going through all the steps again. I made a stupid mistake in that I tried to restore the config (which had services such as SNAT enabled) to a device with nothing other than the base license. So in short when the device rebooted it hung at this stage.

I opened a console session which is where I realized my problem.

My next plan was to factory restore the device and start again. After I pasted on the config something weird happened with the logins, both the administrators and the root passwords would not work, either from the new config or the old one.

Quick job turns into an episode, so I had to reset the root password to factory reset the device. To do this open a console session (I had to set my BAUD rate to 19200 for it to work) and then hard reset the device. Break the boot sequence and highlight your operating system image and press E.

Resetting an F5 Networks Big-IP System's Root Password from the Console

From the sub menu highlight the line that start kernel /boot/l/vmlinuz and press E again.

Type the word single onto the end of the string on the page, and hit enter, this will take you back to the main page.

Press Esc to get back to the main boot screen and hit enter while you are on the operating system image, this will boot the F5 into single user mode in which you can reset the password.

You will be prompted from logname: no login name.

Type passwd to launch the password reset script, set the password and then type reboot to cycle the device. When it comes back up you should be able to get in with your root account and the new password.

Active Directory Federation Services (AD FS) Web Application Proxy "The operation stopped due to an unknown general error. Error Code 0x8007520C" and "Unable to retrieve proxy configuration data from the Federation Service."

The AD FS WAP throws the error "The operation stopped due to an unknown general error. Error Code 0x8007520C" and it is reporting Critical Errors under the Operation Status. The AD FS infrastructure is out of action when this is in this state.

From the Event Logs there are multiple errors relating to the issue the first is Event 422 "Unable to retrieve proxy configuration data from the Federation Service." It also lists a Certificate Thumbprint that is nowhere to be found from the certificate stores. I would expect this to be the thumbprint of the shared SSL certificate that is also installed on the AD FS servers, but it is not.

Event 394 is also present "The federation server proxy could not renew its trust with the Federation Service. Additional Data, Exception Details: The proxy trust certificate specified by thumbprint {0} has expired."

The following command can be used to try and reset the AD FS WAP configuration. I found this from the following blog post http://www.concurrency.com/infrastructure/web-application-proxy-fails-error-code-0x8007520c/ unfortunately Shannon's fix did not work for me in this instance.

Install-WebApplicationProxy –CertificateThumbprint “thumbprint” –FederationServiceName “sso.domain.com”

It returned an error "The certificate that is specified by the CertificateThumbprint parameter could not be found in the Local Computer Personal Certificate Store", this was strange as I confirmed the certificate was present, valid and had the matching thumbprint.

The next step was to remove the Remote Access server role entirely and then reinstall and configure it. I used the following PowerShell command, I also deleted the certificate that was originally configured to work with the WAP so that I could reinstall it fresh. This can be done from the CLI or from the Certificates MMC snap-in.

Remove-WindowsFeature RemoteAccess, Web-Application-Proxy -Restart

Once the servers have rebooted, reinstall the Remote Access (WAP) server role. Ensure the certificate has been imported back in and it has the matching private key.

Install-WindowsFeature RemoteAccess, Web-Application-Proxy -Restart

When you launch the Remote Access Console you should be able to run the wizard again, once I did this the WAP server worked as expected again. It was a strange problem which I think was related to the certificate. If you have details on a fix without blowing the config away please share below.