Broken LDAP Database on Mac OS X Server

I had a client with a broken LDAP database on their OS X Server today. The power had failed and the UPS didn’t do it’s thing, so the power was yanked from the server.

Upon rebooting, things weren’t looking too good – no-one could authenticate over the network to it. Logging in as the localadmin on the console worked, but I couldn’t authenticate to it as anyone via the VPN (which the server was hosting, not a firewall). Fortunately I could ssh in to it, and that’s where it all started from.

The logs were showing a heap of errors with slapd, and my first guess was that the database had hosed itself when the power went out. It’s supposed to run an automatic recovery, but this wasn’t working either.

After checking the logs, I tried running slapd manually in tool mode to test the database:

[[email protected] /var/log]$ sudo /usr/libexec/slapd -Tt
overlay_config(): warning, overlay "dynid" already in list
overlay_config(): warning, overlay "dynid" already in list
overlay_config(): warning, overlay "dynid" already in list
overlay_config(): warning, overlay "dynid" already in list
overlay_config(): warning, overlay "dynid" already in list
bdb_db_open: unclean shutdown detected; attempting recovery.
bdb_db_open: Recovery skipped in read-only mode. Run manual recovery if errors are encountered.
config file testing succeeded
[[email protected] /var/log]$ 

Well, that sure didn’t look good. A lot of the guides I saw online dealt with OpenLDAP running on Linux, and they generally recommended running slapd_dp_recover – unfortunately this command doesn’t exist on Mac OS X. It turns out that it’s simply named do_recover so I tried running it but that had a whole heap of errors happening:

[[email protected] /var/log]$ sudo db_recover -h /var/db/openldap/openldap-data
db_recover: Log sequence error: page LSN 6 5493846; previous LSN 6 5840038
db_recover: Recovery function for LSN 6 5840196 failed on forward pass
db_recover: PANIC: Invalid argument
db_recover: PANIC: fatal region error detected; run recovery
... snip...
db_recover: PANIC: fatal region error detected; run recovery
db_recover: DB_ENV->open: DB_RUNRECOVERY: Fatal error, run database recovery
[[email protected] /var/log]$ 

This was also not looking very good. After trying to interpret the error messages and reading a few other pages regarding openldap on the web, it sounded like the logs may have become corrupt, so I backed them up and deleted them.

I then ran db_recover once more, and this time it didn’t report any errors. Finally, running slapd in tool mode again didn’t throw up any errors, just a couple of warnings so I rebooted the box and everything is looking a lot better now. Except for the VPN service still not letting me in, but that’s another problem for another day…

This article was posted by Kai Howells. If you liked this content and have any technical work in the Melbourne area, say hello via my contact form or give me a call on 0419 361 653 - I cover most of the greater Melbourne area and my rates are competitive.

4 Responses to Broken LDAP Database on Mac OS X Server

  1. Hi, thank you very much for sharing this troubleshooting details, it enabled me to get openldap to work again in our xserve box !

    I’ve created a backup of it, and now I’m moving the OpenLDAP service to another Xserve – but this one is running Ubuntu.
    Hopefully in case of another problem, then it’d be easier to troubleshoot.

    Thanks again. Much appreciated.

  2. thanks, this worked great – my LDAP service went down after a borked system update, took the wiki pages with it.

  3. In some cases it’s the openldap-data database that gets borked. Sometimes it’s the authdata.

    Run db_recover against either database like so:

    sudo db_recover -h /var/db/openldap/authdata
    sudo db_recover -h /var/db/openldap/openldap-data

Leave a Reply

Your email address will not be published. Required fields are marked *