Office 365 Hybrid with Exchange 2013 – Migration Batch Remove Failed User

When you run a migration batch job with multiple user accounts it completes for some users and fails for others. The migration batch job has successfully migrated 3/5 of the user mailbox to Office 365, but I still have to get the other mailboxes up to Office 365. When I tried re-running the same migration job the same mailboxes failed again. I then tried to create a new migration batch job for each of the failed mailboxes, however that failed as you cannot have the same mailbox referenced in the same batch job at the same time.

From what I could tell there is no way from inside the GUI to remove migration users from a batch job, and if you are like me and have successful syncs within the same job you probably don’t want to delete the job, and if you are not at the point in which you want to “finalise” the migration job you must somehow remove the failed users from this batch job.

This can only be done using the Exchange Online PowerShell cmdlets. Firstly, create a session to Office 365 and then Exchange Online. From there you can run the following command to view and remove migration users.

Get-MigrationUser –Identity user@domain.com | Remove-MigrationUser

When you return to the GUI the users will have been removed from the migration batch and you will be free to create new migration batches to get the mailboxes ingested into Office 365.

It should be noted that you cannot actually complete (or finalise) a migration batch job if there are failed mailboxes referenced in the job. The option does not appear inside the GUI until all mailboxes in that job are “synced” successfully. 

Exchange Server 2013 CU13 Can't mount mailbox database "Unable to mount database. (hr=0x80004005, ec=1108"

Exchange Mailbox Database won't mount with the error "Unable to mount database. (hr=0x80004005, ec=1108" it turns out the transaction logs in this case are also corrupt, so the only option is to perform a hard database recovery. 

The /mh will show you the status of the EDB file, in my environment it was "dirty shutdown" which basically means not all the transaction logs were committed properly. 

• ESEUTIL /mh ".edb path" - with quotes.
• cd .edb location (the next command does not work unless you do this)
• ESEUTIL /p ".edb file"

This perform a hard recovery on the Mailbox Database, in other words it does not attempt to copy in the old transaction logs it simply disregards that data and concentrates on repairing the EDB file. 

Before you try and mount the database it's important you remove the old transaction logs, I recommend moving them to a folder called "old" in their original location. If you do not do this the database will not mount.

• Mount-Database "database name"

After this the db mounted without any issues. 

Exchange 2013 SP1: Client Access Servers (CAS) with Network Load Balancing (NLB) Connecting to the ECP Displays Error "HTTP 500 Internal Server Error"

Exchange 2013 SP1: Client Access Servers (CAS) with Network Load Balancing (NLB) Connecting to the ECP Displays Error "HTTP 500 Internal Server Error"

You have configured Exchange 2013 Client Access Servers (CAS) in a Highly Available cluster using Windows Network Load Balancing (NLB) when you try and connect up to the single namespace for the ECP (https://mail.domain.com/ecp) you are prompted to enter your credentials but when you login it times out and displays "HTTP 500 Internal Server Error", for me this also happened when I tried to connect to my CAS Servers via https://localhost/ecp therefore it caused quite a problem!

 

It seemed to happen straight after I configured the Network Load Balanced (NLB) cluster, so my first thought was to disable the cluster and then try connecting via https://localhost/ecp on each of the CAS servers, this worked!

To disable an NLB Cluster from PowerShell use the following command;

Stop-NLBCluster

 

I then decided to change the Cluster Operation Mode from the default of Unicast to Multicast, this can be done using the nlbmgr.exe GUI, by right clicking on the cluster and selecting Cluster Properties.

VMware Article on Windows Network Load Balancing Cluster Operation Modes http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006580

From the Cluster Parameters you can change it from Unicast to Multicast.

 

Then use the following PowerShell command to restart the cluster;

Start-NLBCluster

 

 

It's a good idea to check inside the GUI that the cluster has started successfully and both nodes are showing green and are in the Converged state.

 

Now if you connect back to the single namespace /ECP, enter your credentials and it should now work as expected.

 

Outlook 2010 Clients connecting to Exchange 2013 "Information you exchange with this site cannot be viewed or changed by others. However, there is a problem with the site's security certificate. The name on the security certificate is invalid or does not match the name of the site."

When you launch Outlook it throws the following error "Information you exchange with this site cannot be viewed or changed by others. However, there is a problem with the site's security certificate. The name on the security certificate is invalid or does not match the name of the site." I noticed from the error message that the Outlook client was trying to connect to mail.domain.local which I was surprised to see as all of the Virtual Directories should have been configured to use the mail.domain.com address. The Exchange 2013 certificate currently being used was sourced from a Public Certificate Authority and therefore did not have the .local suffix within it, therefore the error made perfect sense.

 

The first step was to reconfigure the Virtual Directories for AutoDiscover and Outlook Anywhere to confirm they were using the mail.domain.com FQDN. The following PowerShell command can be used to set the Internal AutoDiscover URL;

Set-ClientAccessServer -Identity CAS01 –AutoDiscoverServiceInternalUri https://mail.domain.com/Autodiscover/Autodiscover.xml

The Outlook Anywhere directories can be configured using the ECP, click on Servers and then Servers again click the Client Access (CAS) servers and use the Pencil tool to open the configuration window. Click Outlook Anywhere and ensure the directories are set to mail.domain.com.

 

I then created an SVR DNS Records for AutoDiscover, this is done using DNS Manager. Right click on the Forward Lookup Zone for your domain and select Other Resource Record, scroll down and select Service Location (SRV)  and click Create Record...

 

Fill in the New Resource Record details as shown below.

 

Expand you domain and then _tcp and you should see an SVR Record for _autodiscover.

 

Now if you re-launch Outlook there are no certificate errors. Although this worked for me in this instance due to the broad nature of this error this may not off course resolve the issue in your environment.

Exchange 2013 Client Access Servers (CAS) using Single Namespace and Windows Network Load Balancing (NLB) "An error occurred while using SSL configuration for endpoint 0.0.0.0:443. The error status code is contained within the returned data".

Exchange 2013 Client Access Servers (CAS) using Single Namespace and Windows Network Load Balancing (NLB) "An error occurred while using SSL configuration for endpoint 0.0.0.0:443. The error status code is contained within the returned data".

You have recently just completed a new Certificate Request in Exchange 2013, the certificate seems to have bound correctly to the Client Access Server (CAS) in which you raised the request, although some users cannot connect to Outlook Web App (OWA) or Outlook. There are two Client Access Servers (CAS) in the environment, load balanced using Network Load Balancing (NLB). On investigating the certificates on the second CAS (not the server that the request was raised on), the Event Logs are filled with the following error "An error occurred while using SSL configuration for endpoint 0.0.0.0:443. The error status code is contained within the returned data" to get users working and to take the pressure of me to fix the issue I used nlbmgr.exe to Stop the problematic host, thus passing all Client Access traffic through only a single CAS.

 

The error "An error occurred while using SSL configuration for endpoint 0.0.0.0:443. The error status code is contained within the returned data" is caused by a conflicting binding in IIS. Therefore I opened an MMC to view the Certificates, it appeared the newly installed public certificate had not been installed correctly on the secondary CAS, the private key associated to the certificate was not found. I removed this certificate and used the MMC on the working CAS server to Export the working Certificate.

 

The wizard is self-explanatory but ensure the Yes, Export the Private Key radio button is selected, click Next.

 

I then used the MMC on the problematic server to import the Certificate with it's Private Key, you can do this by right clicking on the Personal/Certificates store and selecting Import.

 

Now that both of the Client Access Servers have the correct certificate with corresponding private key, it should be viewable by it's Friendly Name from the ECP. You will notice the IIS service is currently assigned to this certificate, and it was working correctly on the original CAS server.

 

 

The next stage was to manually reconfigure the Bindings in IIS on the problematic server, from the IIS Manager GUI click on Default Web Site and click Bindings... from the right hand menu. On the working CAS the bindings were are follows, please note 10.10.7.34 is the VIP of the NLB cluster. I therefore reconfigured the problematic server to match this configuration.

 

HTTP to 10.10.7.34 on Port 80 with No Hostname.

 

HTTPS binding to IP Address 10.10.7.34 on Port 443, with the Hostname mail.domain.com and the SSL Certificate bound should match the Friendly Name of your Certificate request viewable from the ECP.

 

HTTPS with No IP on Port 443, again using the same Friendly Named SSL Certificate.

 

Once this was completed I restarted the IIS Service, and the "An error occurred while using SSL configuration for endpoint 0.0.0.0:443. The error status code is contained within the returned data". was no longer filling the Event Logs. I also went back into NLB Manager and started the problematic CAS again, to test thoroughly I open Internet Explorer and browsed to OWA and checked the Certificates status.

 

Mailbox New-MoveRequest StatusDetail "StalledDuetoCI" Exchange Server 2013 SP1

During mailbox move requests in Exchange 2013 the StatusDetail field changes to StalledDuetoCI, it sits here for hours on end!

http://support.microsoft.com/kb/2807668

To resolve the issue, do the following:

1. Create a new Active Directory group that is named "ContentSubmitters" and then grant Admistrators and NetworkService full access to the group using the Security tab on the AD object. This is a dummy group and should be used as a placeholder only. You might want to add a description so that the group is not removed.
2. Force or wait for Active Directory replication.
3. Restart the following services:
• Microsoft Exchange Search
• Microsoft Exchange Search Host Controller

This worked a treat for me! Almost immediately the StalledDuetoCI changed to either CopyingMessages or InitalSeeding on the problem mailboxes. The migration continues!

Exchange 2007 to Exchange 2013 SP1: Get-MoveRequest script StatusDetail "FailedStuck" progress halted

Warning - this may not work for you if your move request is in the 90% complete region. I noticed when a move request get to between 95-99% the status changes to CleanUp which I can only imagine it means it's deleting the original mailbox from Exchange 2007. Therefore I would be careful if you are this far into a mailbox move before you try this

I was migrating mailboxes from Exchange 2007 to 2013, and over night the batch job ran into an issue that changed some of the mailbox statuses to FailedStuck, this was effectively stopping the migration. I used the following command to check the progress;

Get-Content C:\ListofUsers.txt | Get-MoveRequestStatistics

It was concerning to see a number of the mailboxes status was FailedStuck. 

I decided to cancel and then restart a move request to see if that would resolve the issue;

Remove-MoveRequest –Identity “username”

Then when I run the Get-MoveRequestStatistics command again, it could not find a move request job for that user. Therefore it had effectively cancelled the job.

Get-Content C:\ListofUsers.txt | Get-MoveRequestStatistics

I then restarted the move request manually using

New-MoveRequest –Identity “username” –TargetDatabase “db name”

Now if I run the Get-MoveRequestStatistics again, it has restarted the job and the StatusDetail has changed to CopyingMessages.

Since the testing on a single mailbox resolved the issue and got the migration going again I decided to cancel all of the FailedStuck job, I did this using the following command;

Get-Content C:\FailedUsers.txt | Remove-MoveRequest

I could off course used the -Confirm:$false switch to automate these prompts but I wanted to ensure all of the failed users started again.

Once I have run through and cancelled all of the job, I run the Get-MoveRequestStatistics again to ensure all the move request had actually been cancelled.

The next stage was to restart the move requests from scratch, I did this by copying the list of failed users text file and using the following command;

Get-Content C:\FailedUsers.txt | New-MoveRequest –TargetDatabase “db name”

Exchange 2013 SP1: Failed to mount database "db name". Error: An Active Manager operation failed. Error: The database action failed. Error: Operation failed with Message: MapiExceptionDatabaseError

I have recently been working on an Exchange 2013 migration, so we are at the stage of migrating the mailboxes. I run a script to migrate all of the mailboxes contained in a text file. 

Get-Content C:\ListofUsers.txt | New-MoveRequest –TargetDatabase “db name” Confirm:$false

I was then using the Get-MoveRequest to check on the progress.

Get-Content C:\ListofUsers.txt | Get-MoveRequest

As the migration had been running over night I checked up on it and it had appeared to halt as not much had changed since I looked at it the night before. I opened up the ECP and headed over to Server\Databases and all of the databases were in the dismounted stage. The lesson here was that I forgot to enable Circular Logging on the databases, my colleague Terence Luk http://terenceluk.blogspot.com contributed with the wisdom on enabling Circular Logging on the database instances!

When I tried to use the ECP to mount the databases I got the following error;

Failed to mount database "db name". Error: An Active Manager operation failed. Error: The database action failed. Error: Operation failed with Message: MapiExceptionDatabaseError: Unable to mount database.

I then used the following command to test the status of the DAG replication;

Test-ReplicationHealth | fl check,error

The error was Error: There is not enough space on the disk. I then opened up Explorer on one of the Mailbox servers and the entire Transaction Log partition had been consumed within 3 days. It was set to 20 GB's. As the entire Exchange infrastructure was virtual running on VMware vSphere 5.1 I simply extended the VMDK used by Windows to store the Transaction Logs. I also rebooted both of the MBX server, to refresh and effectively reset all of the Exchange services.

I then tried to mount the databases again from the ECP, and they mounted successfully this time. You could off course do this using the Mount-Database command from the Exchange Management Shell also.

Exchange 2007 to 2013 Migration: Get-Content C:\userlist | New-MoveRequest -TargetDatabase "The operation couldn't be performed because object ' user' couldn't be found on 'dc.domain.local"

This post is not particularly a complex or interesting fix, I am posting it here for my own reference should I ever need to do something similar again. Who knows hopefully you find it useful too!

I was recently working on an Exchange 2007 to 2013 migration for a client and part of the project was to implement Database Availability Groups (DAG's) and segment mailboxes into Mailbox Databases by each department in the business. To do this I requested a list of users in each department from the client. The list had been generated by Active Directory and unfortunately all of the user names had a two space characters at the beginning of the string.

When I use the following command with the -Whatif switch to test my syntax and text file I receive an error.

Get-Content C:\Error.txt | New-MoveRequest –TargetDatabase “Mailbox Database” –Confirm:$false -Whatif

The operation couldn't be performed because object '  user' couldn't be found on 'dc.domain.local' this was because of the two space characters before each of the usernames in the C:\Error.txt file. 

I copied the lists of users over into an Excel spreadsheet, and used the following formula to remove the first two characters from the user name strings;

=RIGHT(A1, LEN (A1)-2)

This was in the column on the right of the user names, you can then drag the box down to populate all of the cells and the magic of Excel changes A1 to A2, A3,A4 etc. I then simply copy and pasted this back into a text file.

Now if I run the command again, the Exchange Shell does What if: Creating move request for each of the user accounts. If I were to remove the -Whatif­ switch, the Exchange Shell would begin migrating all of the users in the text file over to the new database.

Get-Content C:\Error.txt | New-MoveRequest –TargetDatabase “Mailbox Database” –Confirm:$false -Whatif

Configuring Database Availability Group (DAG) in Exchange 2013 SP1

In this example I have two Mailbox Servers, and two Client Access Servers. I am going to configure a Database Availability Group. The official Microsoft description of a DAG, borrowed from TechNet.

 

 

DAG's work by replicating Mailbox Databases to neighbouring Mailbox Servers, deployed correctly this can create a highly resilient Exchange environment. Although Replication traffic can share the traditional MAPI network, it is considered best practise to separate the Replication traffic do it own network, either a subnet and/or VLAN.

 

I have provisioned my Mailbox Severs with two network adapters to support this configuration. It is important to name the adapters descriptively, as by default Exchange 2013 automatically configures the separate Replication network.

 

My adapters have been renamed Domain and Replication. The Domain interface is on the 192.168.1.0/24 subnet which is my production subnet that contains Domain Controllers, SCCM etc. The Replication interface is configured on the 196.100.10.0/24 subnet.

 

 

The Domain interface should be configured like it would on any other server, although as a Windows computer can only have a single default route (Default Gateway) the Replication interface should not be configured with a Gateway.

 

 

Click on the Advanced... button from the Internet Protocol Version 4 (TCP/IPv4) Properties page, and select the DNS tab. Untick the Register this connection's addresses in DNS.

 

 

As the Replication interface does not have a default gateway, a static route is required to provide a route to the rest of the subnet. The route add subnet mask subnetmask interface -p command to add this route. The -p switch ensures the route is persistent.

 

 

You can perform a route print to display the routing table on the server.

 

 

You will run into issues when creating the DAG, and adding DAG members if the bind order is not configured correctly. It is important the Domain interface is the primary connection.

 

 

The DAG itself must be represented in Active Directory, by the way of a Cluster Name Object. This is done by creating a disabled computer object. To do this open Active Directory Users and Computers, right click and select New, Computer name this object according. Please note when creating the DAG it must be named the same as the object.

 

 

As mentioned previously it's important the DAG computer object is disabled.

 

 

The security principal Exchange Trusted Subsystem is added to the ACL of the Cluster Name Object. It must also have Full Control access to this object. You can do this by using the Security tab on the properties of the object.

 

 

Now open the Exchange ECP and click Servers, Database Availability Groups. Click the + symbol to create a new DAG.

 

 

Now populate the fields with the corresponding information. Notice my mistake here, that will cause this to fail. My DAG is not named exactly the same as the Cluster Name Object, please ensure this is done properly. The File Share Witness is used as a tie breaker if servers suffer failure, if you do not specify a Witness server the wizard will automatically create it on one of the Client Access Servers. If you choose to configure your own File Share Witness, the Exchange Trusted Subsystem security principal must be in the local Administrators group of the server that is going to host the File Share. You will notice a Domain Controller cannot be a File Share Witness because DC's do not have local users and groups.

 

A DAG must also have an IP address, it is my recommendation to make this a static address even though the option is there to have DHCP assign an address dynamically.

 

 

Click Save and the DAG is created.

 

 

Now you must add DAG members to the DAG, this is done using the Add DAG Member button.

 

 

Click the + to add servers.

 

Select the Mailbox Servers you want to be part of the DAG, click Add and OK.

 

 

Click Save to commit the configuration.

 

 

 

By default Exchange automatically detects and configures the MAPI and Replication networks, there is not an obvious place in the GUI to show the MAP and Replication networks configuration. The PowerShell command Get-DatabaseAvailabilityGroupNetworks this shows how Exchange has configured the networking.

 

 

Now the DAG is created, with DAG members you must now configure Database Copies on the individual Mailbox Databases you want to be replicated by the DAG.

 

Click the ... icon and select Add Database Copy.

 

 

 

 

Specify the Mailbox Server you would like the Database replicated to. The Preference number is used to define a preference of where the database should be mounted while the DAG is operating normally. Click Save.

 

 

The wizard will now replicate the initial copy to the new DAG member server.

 

 

The Active button can be used when a server is selected to gracefully manage the failing over of what server is currently hosting the Mailbox Database. The Activate button will mount the Database on another DAG member and automatically redirect all connection, and it will also manage the reverse of the replication traffic. This can be useful if a server hosting a Mailbox Database requires updating or patching.