Thursday, March 20, 2014

Handling reboots in your valuestream

Reboots - I loath the concept, but with Windows, you have no choice.  Since our valuestream manages our chef-client runs, it needs to handle the reboot cycle. Until the chef-client is happy, we can not put that server back into rotation - especially if it is in the middle of rebooting!

I hacked together some Powershell that did a few things:

  1. Parses the chef-client output to look for the reboot strings that occur in the reboot handler using a regex.
  2. Sleeps for a minute
  3. Waits in a loop for the server to come back and be responsive - I decided on a simple remote invoke-command $True.
  4. Does a chef-client again, and goes back to #1 above in a loop
It is far from perfect, but seems pretty ok.

The only time we ran into issues with the reboot looping is when the cookbook failed, and the reboot never triggered. This is because the handler is registered on for Report.  Registering the reboot handler for Exceptions, in addition to Report, should fix that issue so the reboot happens on error or on success.