Thursday 28 August 2014

Exchange 2010 SP3: After Unexpected Power Outage "Active Manager failed to mount database MailDB1 on server exchsrv.domain.local. Error: An Active Manager operation failed. Error The database action failed. Error: Operation failed with message: MapiExceptionJetErrorRequiredLogFilesMissing: When You Try and Mount the Mailbox Database

After an unexpected power outage you are unable to mount your Mailbox Database in Exchange 2010.  When you try to manually mount the Mailbox Database you receive the error "Active Manager failed to mount database MailDB1 on server exchcas.domain.local. Error: An Active Manager operation failed. Error The database action failed. Error: Operation failed with message: MapiExceptionJetErrorRequiredLogFilesMissing: Unable to mount database. (hr=0x80004005, ec=-543)". Please note I copied the original database file to another location first, and performed all these actions before I did it on my live database, I advise you do the same!


I used the following ESEUTIL command to check the State of the Mailbox Database file. The /mh switch tells ESEUTIL to check the state of the database.
eseutil /mh “database path”

As you can see from the screen shot the Database State is reported as Dirty Shutdown, this makes perfect sense as the Exchange server lost connectivity unexpectedly.


I then attempted to use the following ESEUTIL command to repair the Mailbox Database file.
eseutil /p “database path”

It failed with "Operation Terminated with Error -1032 (Jet_errFileAccessDenied, Cannt Access File, the File is Locked or in Use) after 10.47 Seconds" I checked the Event Logs for more details.



The Event Logs state that ESEUTIL is trying to create a temporary database file at C:\ProgramData\Microsoft\Windows\StateMenu\Programs\Microsoft Exchange Server 2010\temp.edb. As you can see it was reporting an "Access Denied" error.


As it was trying to write to the C:\ drive I had to expand my C:\ so that it had enough space for the temp database


I then tried to run ESEUTIL /P again and it also failed, with the same error. The fix here was to change the ACL on the C:\ProgramData\Microsoft\Windows\StateMenu\Programs\Microsoft Exchange Server 2010\ directory to allow Everyone Full Control.

When I ran ESEUTIL /P again it began to work. It returned the following warning "You should only run Repair on damaged or corrupted databases. Repair will not apply information in the transaction log files to the database and may cause information to be lost. Do you wish to proceed?" as my database was offline and unaccessible I had no option. Please note I copied the original database file before doing any of this.

The ESEUTIL tool then started scanning the database, my database was 60GB's and this took approximately 30 minutes to complete.



At one stage it looked like the process had hung, so I checked the Event Logs. A Warning "eseutil (3616) The database enginer lost one or more bad columns of data in one record. It is highly recommended that an application-level integrity check of the database be run to ensure application-level data integrity". I decided to let ESEUTIL run it's course and it did complete.

When it completed it confirmed it by displaying "Repair completed. Database corruption has been repaired".

For good practice I then decided to run a defrag on the Mailbox Database. I used the following ESEUTIL command to do this;
eseutil /d “database path”




When the degrag completes you will get the following prompt to confirm it has happened successfully.

Now, as I was performing all of this on a copy of the production database I then copied the file to the original location. Not forgetting to change the name of the original database.



I then opened services.msc and started all of the Exchange Server, services that I had stopped at the beginning of the process.


I enabled Circular Logging temporarily. You can do this by right clicking on the Mailbox Database, clicking Properties and ticking the Enable Circular Logging option. http://technet.microsoft.com/en-us/library/bb331958(v=exchg.141).aspx



I then tried to mount the database and it returned the following error, this was different from before. The error had changed to ErrorAttachedDatabaseMismatch this was to be expected as I had copied the original file.


The fix was to create a new folder called old in the original locations for both the Mailbox Database and the Transaction Logs, then move all of the file into the old folder except the new copy of the mailbox database.



Now when I tried to mount the database it worked successfully.


When you are satisfied that the database is back to a healthy stage, I would disable Circular Logging.