joeware - never stop exploring... :)

I was reading through some of the blogs I monitor and Larry Osterman’s blog referred to a blog entry from Dare Obasanjo (and another) talking about why Developers SHOULD NOT be doing your ops.

I agree, dev should not be doing ops. As Larry says the skill sets are slightly different. More importantly, the goals and vision are QUITE different. As a general rule I have seen most developers who are cutting edge let’s do the coolest latest things where most ops people (or good ops people if I may) are about putting stable stuff out there that runs well and doesn’t keep them out of bed at 3AM (unless that is our normal shift). While ops people think the new cool stuff is… well cool… they also prefer not to be screamed at when they are blowing SLAs because some cool new thing really isn’t ready. An ops person isn’t seen as good based on how fast they get new tech out there, they are seen as good based on how well the stuff they have out there is running and how much they AREN’T impacting the users. A good ops person who rushes stuff into production is, well, that is an incorrect statement.

As I have said in the past, Dev is the gas pedal and Ops is the brake pedal. Dev wants to go forward as far and as fast as possible. Ops wants things running smoothly and so that it is supportable when it does break. They cannot completely prevent all failures but they can certainly make sure things with higher percentages of failure have a harder time getting put into place. There is more than one project that I, as one of the senior Ops people at the time, completely slammed to a stop because it was being done half ass. While costs may increase doing things properly, it is far cheaper than catching the costs later in supporting the app.

I do think that Developers and Ops should run very closely together. A meeting every two weeks minimum seems like a good idea so that Ops can feed back info to Dev on the issues and problems they have encountered and Dev can toss ideas past Ops on what they are thinking for new ideas. If there are major initiatives being implemented, that schedule could be moved up to once per week or even short morning meetings every morning to keep everyone in the loop.

I also have thought that some sort of Dev/Ops exchange program would be good too, sort of like the foreign exchange program where you send little Bobby to Europe and bring little Guido to the US so they can both experience something different.  So you do that with Dev and Ops folks only it is more formal and recurring. Every Dev person has to work in ops for 1-2 months every 2 years and vice versa. I think that it would help show Dev and Ops each the different issues they have to deal with and help better share insights because each has something to learn from the other.

 

I am one of the odd birds who was grew liking and ended up training for Dev but as I got into it it bored the hell out of me and I wanted the “excitement” and ownership of Ops. I still like to develop though generally smaller projects that many call tools or utilities. If you read this blog in fact, it is likely, you are aware of me through my tool/utility work. This mix puts me in a unique position because I am well suited to understand what ops would want out of tools I write because I am writing the tools for myself doing the same ops work. It also helps me greatly in understanding what Dev people are trying to do and really really helps me when someone in Dev says something isn’t possible or something doesn’t work in the way that I tell them they need to work towards. My unlock tool was written directly to prove to the MTEC Developers that delegated unlock worked in Active Directory. I sat in a meeting where they told me they needed Account Operator or Domain Admin to unlock accounts and I said they were completely incorrect and their lead developer told me I was wrong and it was impossible… I didn’t argue, the next day I came in with a tool that did the impossible (in his eyes). Six months later they had corrected their app. They did however have another issue and stated that they needed Certs on the DCs to do the password sets… Of course I knew this was incorrect as well and told them but this time didn’t have to prove anything. Six months later they came back with a version of PSYNCH that worked pretty well and from then on we worked quite well together any time I pointed at something and said that was wrong they worked on correcting it. If you use PSYNCH, you can thank me for how well it handles delegation and AD as I was the guy saying that thing isn’t coming anywhere near my AD until it works right, I don’t care who wants it up and running. In reality I didn’t have that much power, if someone pushed the fact I would have been forced to implement. Standing up strongly for blocking its implementation made people really think about it and made it so no one was willing to say I should be overridden.

Anyway, Ops and Dev definitely should be kept separate as the skills are different and the goals are generally very different but they should work closely together and an Ops/Dev Exchange program is a great idea if you can get it implemented.

 

Now to answer the first question I feel coming on… But joe… if no one implements the new stuff, how will it ever get proven out? Good question, actually great question. This is a big topic that comes up with new OSes and Service Packs and QFEs. Every one expecting someone else to prove it out for them. You get this in the various computer “superstitions” like don’t ever deploy an OS when first launched or don’t deploy even numbered service packs. The answer here is test environments. This is why you should have them. Then as the test environments prove things out you slowly work it into production. Now let me explain what I mean by test environment. I mean an environment that you actually do comprehensive testing of your Line of Business applications. It isn’t a place where you have two domain controllers a member server and 3 clients loaded with World of Warcraft where you load a patch or SP or OS and say, yep, looks good, the machine didn’t blue screen… If you don’t have dedicated people testing by running through formal documented test matrixes you probably can’t be considered as having a test environment. You have a lab to play in when you are bored doing real work.

Depending on what the company does, the more proven out things should be. For instance, pure financial and pure manufacturing companies generally should be on the conservative side. Their business isn’t technology. Moving just prior to end of life on the previous products is fine in my book unless there is something staggeringly amazing about the new product which could substantially impact how well they work.

Technology companies, on the other hand, need to be a little more cutting edge. Companies that do technology consulting and services for other companies should be bleeding cutting edge… Companies, for instance, like HP, IBM, UNISYS, Microsoft, and Avanade should be running the latest things in production to help iron out the issues; that is what they do as businesses.

I can personally say from talking directly to friends involved that HP, for instance, is running Longhorn x64 domain controllers in production right now as well as gearing up to add Read Only DCs. They were running x64 Windows Server 2003 well before that OS was launched as well, a couple of years ago. Before HP “merged” with COMPAQ, COMPAQ was the big leading edge company with the global test domain called QTEST (the largest in existence from what I recall hearing) that most everyone who has been on the AD scene for a long time has heard of. These are the folks who have seen some amazingly bad failures and issues and have some very odd stories. You want to have fun hearing about spectacular screwups talk to the likes of Wook Lee and Jesse Sutela and others who have or have been responsible for running HP’s (and prior, COMPAQ’s)  internal production AD environment. While Wook makes up a lot of weird names and stories to spice up what happened there is some seriously interesting stuff under all of it that really doesn’t need the extra spice. As a dev type person that stuff is cool as anything possibly could be to me, as an Ops person it all makes me shudder.

Would I recommend that for a company like Walmart or Chrysler or Toyota or Bank of America, hell no. Companies like that should be trying to run the most stable and controlled environments they possibly can. Cutting edge technology is not their business and trying to be cutting edge for IT could almost be irresponsible. The goal isn’t to have the coolest and latest tech, it is to have a stable environment so the folks responsible for making money for the company can do so undisturbed.

So are you in the right job? Do you push to get the latest and greatest into place or do you push for a solid, secure, stable environment? If the former, you are more cut out for Dev work or Ops work in more cutting edge types of companies. If the latter, you probably shouldn’t be in Dev and doing Ops in a cutting edge type company will probably give you ulcers but you will probably be a rock star doing Ops in non-technology companies.

  joe