How to run the HCW to configure hybrid from Exchange Server to Exchange Online

I’ve put together a comprehensive list of my notes on how to implement an Exchange Hybrid to Office 365 with Exchange 2016. This is something I have done countless times now, but I always forget some of the finer details every time I do it, so this is for my own archive more than anything.

The setup in this example is simple, there is a single Exchange Server on premise which is used for:

• Email administration (both on premise and Office 365)
• Local SMTP relay for devices with no Internet route

The topology looks something like this, a single Exchange 2016 sits on premise, there is a firewall between it and the Internet. All external MX Records point to Office 365, the firewall is configured to all the Exchange Server out to the EXO IP’s on SMTP TCP 25. It also allows inbound SMTP TCP 25 which is NAT’d to the Exchange Server. In addition to this the NAT rule also include HTTPS TCP 443 from anywhere.

I’m going to assume you have the following things already configured:

1. Install Exchange Server 2016 (latest CU)
2. Configure Exchange Server
1. Virtual Directories with External URL’s (e.g mail.domain.com)
3. Configure External DNS
4. Configure Exchange Certificates
1. A wildcard certificate is best, but if you do not have a wildcard certificate ensure your certificate has multiple Subject Alternative Names (SAN’s) to cover things like mail.domain.com, autodiscover.domain.com to prevent any certificate warnings from Outlook.

In this post I’m going to cover the following with some lessons learned.

5. Hybrid Configuration Wizard
6. Verify and Test Hybrid
7. Administering a Hybrid

Firewall Rules	Exchange Server should be presented to the Internet on TCP ports 25 and 443. The firewall rule should only accept incoming SMTP (TCP 25) from the Exchange Online IP’s. HTTPS (TCP 443) however should be open to all to ensure services such as Autodiscover work correctly. These rules should be done using NAT on your perimeter firewall. The Exchange Server should be allowed out to the Exchange Online IP’s for SMTP (TCP 25). It’s advisable to only allow the server itself out and not the other nodes in the subnet.
MX Records	It’s possible to have your MX Records point to either Office 365 or the on premise server for a hybrid deployment. It’s my preference to always have these points to the Office 365 tenant. When your setup your custom domain for Office 365 you will be given the target address for your MX records to point.
Centralized Mail Flow	Centralized Mail Flow is when all your outbound mail is forced to go through the on premise Exchange Server. Personally I prefer all mail to be routed out via Office 365, ensure this option is not selected if you want Office 365 to route outbound mail.
Full or Minimal Hybrid	Full Hybrid gives the richest coexistence and will be required if your organization is going to retain hybrid for an extended period of time.
Edge Transport	The Edge Transport server is an SMTP Relay which sits between your Exchange Server and the Internet, most commonly in a DMZ. You must decide if this is going to be in your topology before you run through the hybrid wizard.
AD Connect	AD Connect will be required to sync you on premise identities to Office 365. This should be configured with Password Sync and Exchange Hybrid Deployment. Please note if the Exchange Hybrid Deployment option is not selected AD Connect will not write-back the Exchange-specific attributes to AD which will cause internal mail routing issues.

If AD Connect is already in place, you must change the settings so that you can reconfigure it to use Exchange Hybrid Configuration. To do this open the AD Connect admin screen and edit the configuration, on the Optional Features you will see the option.

Login to the ECP on the Exchange Server and click the Hybrid tab, then click Configure. This will download the HCW, install and run it.

It should auto detect the Exchange Server, ensure you select the correct Office 365 (more than likely Worldwide).

You will be prompted for credentials for on premise AD and Office 365, the AAD creds must be Global Admin.

The HCW will check over each environment.

Select which features you want as part of this hybrid, Full Hybrid gives the richest integration between on premise and Office 365.

Now you need to publish a TXT record to the root of your domain so that the HCW can validate you own the domain. Create the record, wait for it to replicate and verify the record. This part of the HCW is prone to break. Top tip is to rerun the HCW if it continously fail, it will eventually pass the validation test if the record has been created correctly. 

Enable the federation trust, this will ensure free/busy information etc. can be share across the environments.

I don't have an Edge Transport server in my environment so I've chosen the CAS option. If you click Advanced you will be presented with an option to use Centralized Mail Flow, this will route all mail through your on premise Exchange, which is probabaly not wise.

The HCW will auto create Receive connector so that the Exchange Server can accept incoming mail from Office 365, choose from the drop down the server you want this to be created on.

Same goes for a Send Connector, we will examine the Send Connector further down and create another one to ensure mail flow to external recipients works correctly. 

A globally trusted certificate must be installed on the Exchange Server, the same certificate you use to secure the servers Virtual Directories can be used by the HCW. A wildcard is generally best as it can cover multiple service names behind a single domain.

You then need to input the on premise service name I tend to use mail.domain.com, this is how Exchange Online will route mail to mail-enabled objects that reside in your domain. Obviously a DNS A record must be created to direct this service name to the outside IP address of your Exchange Server. Please do some DNS testing to ensure this is working before you attempt this part.

Click Update and the HCW will run a bunch of scripts to implement the changes, based on the information you have provided to the HCW.

Understanding AD Attributes in an Exchange Hybrid

Scenrio: You have Active Directory which stores users with AD Connect doing password sync, all mailboxes with the exception of a few are in Office 365. Hybrid mail flow is required between the on premise mailboxes and Office 365. All the AD users were created using AD Users and Computers and not the Exchange Server.

Problem: If you have on premise user accounts with Office 365 mailboxes. If the user accounts have been created using Active Directory Users and Computers and not Exchange they will not have all the required attributes to route mail internally.

It's worth having an Exchange Server purely to do email administration, unless you know what AD attributes to manipulate manually.

Office 365 mailboxes should be created using the on premise Exchange Server, this will create the new AD object with the correct attitrbutes set.

When a new mailbox in Office 365 is created with an on premise user account Exchange populates the "targetAddress" attribute which is used to route mail from on premise to Office 365. Please note this attribute is not synced to AAD using AD Connect.

It's added in the format SMTP:username@domain.mail.onmicrosoft.com

It also populates the "proxyAddresses" attribute with a primary email set to the custom domain and another set to @domain.mail.onmicrosoft.com 

If you force AD Connect to sync the user will appear as unlicensed in the portal. 

Assign a license to it then force another AD sync.

If you return to the "proxyAddresses" attribute you will see another x500 field has been create. All of these attributes are required to ensure mail can be routed between on premise and Office 365.

If you have created a batch of users in the traditional way, synced them to Office 365, assigned licenses then allowed users to use the mailboxes. You will not be able to send mail from on premise mailboxes to them, this is because the "targetAddress" attribute is not populated by ADUC.

Exchange Hybrid Mail Flow error "A matching connector cannot be found to route the external recipient"

I was doing some troubleshooting with external recipient mail flow from an Exchange hybrid setup.

When I tried to send mail to an external domain I got "A matching connector cannot be found to route the external recipient" from the Queue Viewer on the Exchange Server.

This was caused by an incorrectly configured Send Connector on the Exchange Server, the Exchange Hybrid Wizard creates a new Send Connector which pushes all mail to your onmicrosoft.com. This will cause a problem if you are trying to send mail to external receipents from an on premise mailbox.

To resolve the issue, create a new Send Connector, using the SMTP space * (all domains) and scope it to your Exchange hybrid server, after this external mail flow will start working.

All Exchange 2016 services appear as disabled after Windows updates

A useful script to reconfigure the Exchange services to automatically restart after a botched update.

Get-Service | Where-Object { $_.DisplayName –like “Microsoft Exchange *”} | ft Name,Status

Get-Service | Where-Object { $_.DisplayName –like “Microsoft Exchange *” } | Set-Service –StartupType Automatic

Get-Service | Where-Object { $_.DisplayName –like “Microsoft Exchange *” } | Start-Service 

Get-Service | Where-Object { $_.DisplayName –eq “World Wide Web Publishing Service” } | Set-Service –StartupType Automatic

Get-Service | Where-Object { $_.DisplayName –eq “World Wide Web Publishing Service” } | Start-Service 

Get-Service | Where-Object { $_.DisplayName –eq “IIS Admin Service” } | Set-Service –StartupType Automatic

Get-Service | Where-Object { $_.DisplayName –eq “IIS Admin Service” } | Start-Service 

Office 365 OME setup mail encryption with a transport rule in Exchange Online

Office 365 offers multiple options for providing an encrypted mail service to users. Office 365 Message Encryption (OME) offers the best flexibility as Office 365 automatically trusts many of the main stream mail system providers for message decryption. S/MIME still requires a complex certificate exchange to be done between sender and recipient to ensure messages can be decrypted.

Mail providers Outlook.com, Yahoo and Gmail are automatically trusted by Office 365 and can automatically decrypt messages.

Other mail services which cannot automatically decrypt OME messages present a different set of challenges. One Time Passwords (OTP) are issued by Microsoft when an encrypted message is received by a mail server that cannot automatically decrypt messages. The OTP is sent directly from Microsoft to a user’s inbox (using the email address which was set as the recipient of the encrypted message). This way Microsoft can authenticate that only the intended recipient has received the OTP to decrypt the secured message.

Office 365 OME is a feature which is available in the following SKU’s Office 365 E3/E5, Microsoft E3/E5, A1, A3, A5 and G3/5. No licenses are required on the receiving side to accept encrypted mail. In this example we are going to make use of the Microsoft managed keys, however understand that it’s possible to BYOK to Azure Information Protection.

Go to the Azure Portal when you are logged in with your Office 365 credentials. Find Azure Information Protection from the portal. 

Click on Protection Activation, if this is not active, go ahead and activate it. 

Open the Exchange Admin Center and go to Rules. We are going to create a new Transport Rule to apply the encryption policy to emails that contain certain words. Click the + icon and select Apply Office 365 Message Encryption and Rights Protection to Messages.

Label the Transport Rule something sensible and click the Apply this rule if...and select The subject or body...then subject includes any of these words. 

Enter all the key words in which you want to be used to encrypt mail. I think it's generally good to use "encrypted", if you do this ensure you put in all the obvious spelling mistakes for "encrypted" so that users don't send clear-text messages because of typo's.

It's very easy to send encrypted email from Outlook with a Transport Rule in place to apply the encryption rule. Simply ensure the work "encrypted" is in the subject line, this words that is looks for before encrypting the message are user-defined in the steps above. 

When a recipient is also on Office 365 the message is automatically decrypted. You will notice the red marker to outline that this message is encrypted. Please note that when a single message (or reply) is encrypted, the entire conversation/message chain is encrypted.

If an encrypted message is viewable to a recipient, it's automatically opened into OWA. For some reason the message does not open directly in Outlook. This does ensure that the encrypted message cannot be forwarded. 

What happens if your send an encrypted email to a mail service which cannot automatically decrypted the message?

The encrypted message is opened in the browser.

The OTP is sent to the inbox.

The recipient can respond to the message securely in the browser.

AD CS reinstall causes templates to show "Template information could not be loaded. Element not found."

After a reinstall of AD CS when you try to expand the Certificate Templates pane of the Certificate Authority MMC it throws the error "Template information could not be loaded. Element not found.".

To resolve this you need to use ADSI Edit from one of your Domain Controllers. Open ADSI Edit and connect to the Configuration partition, expand CN=Services, CN=Public Key Services and find CN=Certificate Templates. Right click on CN=Certificate Templates and select Properties. 

The error is caused because the Computer Object of the newly install AD CS server does not have Read permissions on the Certificate Templates top level folder. Add the Computer Object and grant it Read permissions. 

It's a good idea at this point to force an AD replication cycle.

Relaunch the Certificate Authority MMC and refresh Certificate Templates, please not you will have to recreate any of your Certificate Templates before the reinstall, these will also need republished to AD before they can be used for enrollment. 

AD CS custom templates not shown in AD CS Web Enrollment portal

When you try to enroll a new certificate using the AD CS Web Enrollment point you are unable to select the custom AD CS template you would like to use. 

When the custom template is being created click on the Subject Name tab and ensure Supply in the Request is selected. This is a must if you are creating certificate for non domain-joined devices like network appliances. 

Click on the Security tab and add the Web Enrollment server computer object to the ACL and ensure it has Full Control permissions. 

Ensure you issue the certificate template to AD.

Go back to the web enrollment point and you should have the option to use this new template. Please note you might have to force AD replication.

Planning and Configuring Active Directory Certificate Services (ADCS) with OCSP Revocation

Active Directory Certificate Services is Microsoft implementation of a Public Key Infrastructure, which allows enterprise organisation to deploy digital certificates to secure internal corporate services.

This is an update version of my post from 2015

http://blog.ryanbetts.co.uk/2015/01/implementing-two-tier-active-directory.html

In this example I'm going to deploy two AD CS servers in a two-tier fashion with a Root CA and an Online CA, the Root CA will only be used to issue a single certificate to the Online CA. The Root CA is then taken offline once this certificate has been issued, this increases the security posture of the PKI. The idea being, that if the Root CA is offline it cannot be comprised, meaning the PKI trust chain cannot be compromised. 

• CA1 is a non-domain joined server which is going to become an offline root certificate authority.

• CA2 is a domain joined server which is going to be the Certificate Authority for my AD domain along with the CRL point for both CA servers.

• CA2 is also going to host the OCSP component of AD CS to streamline certificate revocation requests.

The following bullet points cover, at a high level the work flow on how this AD CS infrastructure is going to work. The most important thing to understand is how certificate revocation works, without proper certificate revocation the validity of your digital certificates cannot be checked. 

• CA1 is configured with CRL paths to both for C:\ and HTTP

• CA1 is configured to use CRL paths which reside on CA1 served via HTTP

• CA2 generates a CSR for a new root certificate

• CA1 issues the certificate based on the CSR, cert is installed

• CA1's root certificate is installed on CA2

• CA2 checks the validity of the certificate using the CRL paths published in the certificate

At this point in the process, there is a trusted certificate installed on CA2 which has been issued by CA1. However, certificate revocation would likely fail unless the CRL for CA1 was somehow accessible to CA2. I cover how to do this further down.

• CA2 is configured with CRL paths for both HTTP and LDAP, including the OCSP path for certificate it issues.

• CA2 is configured to use CRL paths which reside on CA2 served via HTTP

• CA2 issues certificates with CRL and OCSP paths which point to http://revocation.ryanbetts.co.uk/crl and http://revocation.ryanbetts.co.uk/ocsp

• CA2 issued certificates also have CRL paths published to AD over LDAP.

Configuring the Certificate Authority Servers

Install and Configure the Certificate Authority Role on CA1.

Use the GUI to install the Certification Authority role on CA1.

Select Standalone CA.

Select Root CA.

Choose to Create a New Private Key.

Choose to set your key length to a minimum of 2048.

 Retain the default CA names and continue.

You can extend the root CA length if you like 5 years is long enough.

After you complete the wizard you are left with a functioning standalone CA.

We have to jump over to CA2 before we can continue with the config on CA1, install the following roles on CA2

·         Certification Authority

·         Online Responder

Choose Enterprise CA, this option will only be available if this CA server is domain joined, we want CA2 to be domain joined.

It's not going to be the top of the trust chain so select Subordinate CA. Subordinate CA only means that this CA is not the highest in the trust chain, in otherwords there is another CA above it in the trust chain.

Create a New Private Key.

Again ensure the key length is set to a minimum of 2048.

Do not alter any of the CA attributes below.

Take now of the path below, this is the CSR that CA2 is creating. This will be needed to generate a new certificate on CA1.

Configuring the Revocation Websites in IIS

Once the wizard of installing and configuring the CA is complete open up IIS which will have been installed for you. Expand the Default Website and create a new Application.

I like to call this /crl and point it to the physical path on your server I have used C:\inetpub\wwwroot\crl - please note I had to create the /crl folder.

Create your CRL application and your IIS installation should look like this, for testing it’s a good idea to enable Directory Browsing on the CRL application.

Logon to one of your DNS servers and create a new A record in your domain forward lookup zone. This DNS service name is going to be used in the CRL and OCSP paths when we configure CA2.

I use the service name revocation.ryanbetts.co.uk for my certificate revocation. This should point to CA2 in this example as this server will host the CRL and the OCSP.

Now return to CA1, the step above is important so make sure you do it correctly.

Configuring CA1's Certificate Revocation CRL Paths

Right click on the Properties of CA1 and go to the Extensions tab.

Delete all the default entries until you are left with only C:\Windows\system32....

Click Add

Add        http://revocation.ryanbetts.co.uk/crl/<CAName>.crl

Select Include in CRL's, Include in the CDP Extension, Include in the IDP etc.

Click Apply and the AD CS service will restart.

From the same pane use the drop down and select Authority Information Access (AIA)

Click Add

Remove all the paths except C:\Windows and ldap://

Add        http://revocation.ryanbetts.co.uk/crl/<CAName>.crt

Ensure you select Include in the AIA extension of issued certificates

Again click Apply and AD CS will restart.

At this point we have CA1 configured to issue certificates, in our example CA1 will only ever issue a single certificate to CA2. The details above are setting the CRL paths which will be published into any certificates issued by CA1. Without these paths CA2 would not be able to validate certificates issued by CA1.

Remain on CA1, now we need to complete the CSR generated on CA2.

Go to All Tasks and Submit New Request.

You might need to copy the CSR from the C:\ on CA2 so that it's available on CA1.

Once done, go to Pending Certificates, find the certificate, right click and select Issue.

Go to Issued Certificates and open the Properties and Details of the certificate.

Click Copy to File, run through the wizard and export the certificate to the Desktop.

Now we need to copy the following files from CA1 to CA2 to the respective locations.

• CA1's CRL and CRT files which are at C:\Windows\System32\CertSrv\CertEnroll 
• copy these to CA2 C:\inetpub\wwwroot\crl 

• CA1's root certificate copy and install this on CA2. - we export in a couple of steps!

• CA1's certificate issued to CA2 based on the CSR.

Once the CRL and CRL files are copied to CA2 C:\inetpub\wwwroot\crl they will be accessible over HTTP using our IIS website.

The CRT file must be renamed to so that both files have the same name. In other words the CRT file must be named <CAName> as shown below, this is because we configured the path to use <CAName>.crt.

It should look something like this, the root of your IIS application should have a .CRT and .CRL file both from CA1, they must be labelled correctly or validation will fail.

To test these files are accessible over HTTP open a browser and go to http://revocation.ryanbetts.co.uk/crl

Exporting the Root CA (CA1) Certificate

Now we must export the CA1 root certificate and install it on CA2, open the Properties and double click the Certificate #0 go to Details and Copy to File.

Dump this out to the Desktop along with the CA2 certificate.

The two certificate files now need copied over to CA2.

Return to CA2 and install the Root Certificate first, ensure this is installed under the Local Computer under the Trusted Root Certification Authorities store.

 Now open Certificate Authority on CA2 and go to All Tasks and Install CA Certificate.

Browse for the other certificate copied from CA1.

Refresh and the CA should become active. If you have made a mistake at this point it's probably due to the location of your CRL/CRT files. If the CA cannot access them in the paths you provided validation will fail.

To double check validation we can open the root issued CA cert on CA2 and look under Details and you will see CRL Distribution Point, please note that http://revocation.ryanbetts.co.uk/crl is listed as the path.

Another good way to check the overall health of the AD CS is to use pkiview.msc

Configuring Certificate Revocation (CRL) and OCSP Paths for CA2

It will report any issues it finds with the deployment.

On CA2 open up the Properties from the Certificate Authority MMC. Select AIA from the drop down and add the following

http://revocation.ryanbetts.co.uk/ocsp and ensure to tick Include in the Online Certificate OCSP extension.

Add another which points to

http://revocation.ryanbetts.co.uk/crl/CAName.crt

Click Apply to retain they changes.

Select CDP from the drop down and use Add.

Add a new point which points to http://revocation.ryanbetts.co.uk/crl/CAName.crl

Ensure all the option to Include are selected.

From CA2 open C:\Windows\System32\CertSrv\CertEnroll, these files need copied to the IIS virtual directory folder so they can be accessed over HTTP. Copy these files to C:\inetpub\wwwroot\crl

The wwwroot\crl folder should look like this with two sets of CRL/CRT’s. One pair from both CA1 and CA2.

Now if you refresh pkiview.msc it will be reporting an issue with the OCSP location, this is because we have not configured the OCSP yet.

Configuring the OCSP Service

 On CA2 go to Certificate Authority and Certificate Templates right click and select Manage

We need to copy and apply permissions to the OCSP Repsonse Signing certificate template before we can configure the OCSP. Right click on it and select Duplicate Template.

Click on the General tab and rename the template something descriptive. 

Click on Security and add the computer object to the ACL, this should be the computer object of the server which is going to host the OCSP role, in this case that is CA2.

Ensure the object has the Read and Enroll permissions. Click Ok.

Close the Certificate Template management window and return to the CA, right click on the Certificate Template folder and select New > Certificate Template to Issue.

Now open the OCSP MMC (if it’s not there you probably never installed it), right click and select Add Revocation Configuration.

Name it something descriptive.

Select to use an Existing Enteprise CA.

Select Browse CA Certificate Published in AD, click Browse, then Next.

If the OCSP template has been configured correctly the wizard should detect it. 

If you return to pkiview.msc and it still reports a OCSP Error #1 you might need to renew the CA Exchange certificate, to do this open Issued Certificates on CA2 and find the latest CA Exchange certificate, you can search using the Certificate Template column.

Right click it and select Revoke.

Open an admin CMD and type certutil -cainfo xchg

Return to pkiview.msc and refresh the report and the issue with the OCSP should disappear. 

Configuring Certificate Distribution with GPO's

We now need to distribute the root certificates to the domain devices, open Group Policy Management, create a new GPO and scope it to the correct place.

Edit the GPO and go to Computer > Polices > Windows Settings > Security Settings > Public Key Policies > Trusted Root Certification Authorities

Right click and select Import, select both the root certificate for CA1 and CA2 so that the entire chain of certificate is trusted. 

We can also put an additional setting in to auto enroll computers with device authentication certificates. 

Perform a GPUPDATE /force and you will notice the root certificates from CA1 and CA2 have been automatically installed on devices which are scoped to the GPO.

If you have followed all the steps correctly you now have a fully functioning PKI infrastructure.