Friday, October 19, 2012

Why aren't you embracing automated configuration management?

Automated configuration management, why aren't all professionals embracing this technology?

I spend most of my time in what I consider my professional community. I consider this group my friends and peers I have met through the USENIX LISA conference and LOPSA organization. These people are pretty firmly in the open source camp, they tend to push technology to new levels, and they are the ones who see the benefits of automation and standardization.  When I'm hanging with my professional circle, there's friendly banter about configuration management; it tends to be a popular topic from year-to-year. Whether your views lean toward Cfengine, Bcfg, Puppet, or Chef... at least you're using these tools to automate your environments. When you hang with the same group from year-to-year, you tend to think that they represent the international sysadmin community at large.

I cannot imagine any medium-to-large sysadmin shop that isn't using something akin to one of the tools above to manage their environment in today's complex IT world. Yet, the more I talk to sysadmins outside of my USENIX/LOPSA circles, from fairly sizeable organizations, I find that they're not using anything to manage their configurations. Some of them have heard of these tools and are starting to look at them, some claim there's no management support for such things. I'm amazed at the number who are using home grown scripts or rely on complex databases of configuration differences (excel spreadsheets anyone?).  I want to yell "there's a better way!"

So what are the benefits of automation and standardization? Well, for years I've seen my job as trying to automate myself out of what I currently do so I can use that time to do new stuff. If I spend all of my time keeping track of server and client configurations, performing upgrades, and monitoring individual logs then I don't have time to figure out the next thing that will save my customers time/money or make their lives better. I think of professional sysadmins as always trying to make the IT world better by reducing complexity, increasing consistency and system resiliency, and helping customers be the most productive with their IT. These lead to better IT security, reduced downtime and support costs, and increased overall value of IT to the organization. For my group, it means we work on new and fun projects for our customers instead of spending most of our time slogging through manual methods to perform basic day-to-day sysadmin functions.

Someone recently told me that only about 10% of medium-to-large organizations are using any kind of automated configuration management. If that number is remotely close to reality, it amazes me.

We started using configuration management ~10 years ago with Cfengine. We quickly saw the benefit of standardizing the way we configured Unix machines. Even machines that had configuration differences, such as servers, were easier to manage with Cfengine. Here we are 10 years later and there are more Open Source tools available to choose from, but the concept is still the same: use something to automate and manage your machines. We've expanded from Unix/Linux servers to clients including all of our Macintosh computers (automated configuration management, it's not just for Linux/Unix anymore).

We keep our Macs configured according to our organization's IT security requirements. Building a new Mac is so easy. Load OS, load our Cfengine package, reboot, done. All of the configs are magically loaded and the machine is ready for use. It has made deploying and managing hundreds of Macs simple. Mac under the hood is not your typical "Unix/Linux" though it may seem so on the surface. Getting MacOS X management under the same system as your Linux machines makes them seem much more manageable (and that's saying a lot for OS X).

With automation, we have the ability to audit our configs and to check whether a certain state applies to every machine. We have the ability to see if all Macs have a critical patch or if the antivirus is up-to-date. All of this is reportable to a central location. We also decentralize authority so people who aren't sysadmins can monitor machines within their own areas to ensure compliance and IT security. All of this is possible through automation. I love that even machines that don't connect every day can phone home when they're on-line. It means home computers can benefit from this technology, connecting when they VPN into the internal network. I can also see by our management console which machines are connecting now and which haven't connected in a while.

We're even using this system for some of our scientific devices such as robots that run the Cfengine configuration to ensure IT security compliance. I cannot image a life without automated configuration management.

I admit there is a learning curve, but isn't there always learning curve when you're a sysadmin? I got into this profession because there was always something new to learn and a better way to solve a problem. Go out and do some research, figure out what is important in a tool, and set up a test network. There are a lot of resources for all of these tools to help you with recipes and getting started. I won't list them all here because I'm sure your Google foo can produce help on any of the tools listed at the beginning of this rant... er, post.

Automated configuration management isn't bleeding edge, it isn't new, it should be a part of any mature enterprise.


  1. I agree, it's easy to get inside the bubble and think everyone is embracing tools. Even in places that are using tools it's not uncommon for people to want to short-circuit the tool, do manual configuration, and port it in to the tool as a last step.

    People are significantly more difficult to configure than machines.

  2. I agree, people are more difficult to configure. :)