joeware - never stop exploring... :)

The last blog post generated a considerable amount of offline feedback to me. This always cracks me up as so many people seem to be against posting comments. Comment away folks, but if you want to email me directly that is fine as well…

So the feedback seemed to fall into three main categories

1. There is no way you can run without doinging the ad hocs and the one offs. Some people just plain get special attention and the support folks who realize that and take care of that really are the good ones…
2. Around here, groups have to be whipped up on the spot when someone needs them, you can’t tell them they have to wait a week…
3. How do you prevent people from creating server accounts, is there some hack that prevents folks from creating servers but allows them to create workstations?

Ok let me take these in order…

1. One offs and ad hoc requests to a support organization are one of the most inefficient ways you can do things. THIS is one of the primary reasons that IT orgs aren’t efficient and can’t run well and people are bitching about escalating IT costs. It definitely IS the job of IT to support the business but it has to be realistic. If the business wants special one off and very flexible support, they need to allow the budget for it. Usually instead what happens is that business says IT’s budget needs to be near nothing AND they want ad hoc specialty support. Any management that allows that is insane because they are pretty much mutually exclusive items. Something will drop because I have not and I don’t think anyone has found a way to pull off great ad hoc support for free. You have a couple of options, standardize the processes and try not to deviate so that support is understood and easy (this is the most efficient, least flexible, and least expensive), go completely ad hoc and have people on standby to handle the requests so that they can be done as needed without impacting SLAs (this is the least efficient, most flexible, and most expensive), follow some mixture of this.

Most companies try to do the latter though don’t really realize it, therefore they do it poorly and aren’t staffed to properly handle a bunch of ad hoc or even really a little bit of ad hoc so they run around from high priority to high priority based on who is screaming the loudest and who is the most important and other things that take time to sort out instead of just getting work done. If a company is doing this latter piece properly, they have different teams or multiple levels of teams.

For an example of the first idea and using multiple different teames, consider a company that has a standard help desk and support process for a majority of the company and a special help desk to handle executives or some other critical care group. This critical care support group tends to be “better” support engineers and could even overall cost more than the standard help desk even though they handle a small minority of the overall work and people. One large finance company I worked for actually broke support up into 3 main groups, the majority of everyone, an executive support team, and a support team for treasury operations which was the most critical aspect of the finance company.

As an example of the second idea and using multiple levels of teams, all tickets come through the main help desk and the first level folks try to ascertain if a ticket fits into standard molds they have available and if they don’t, they dump into a higher level special handling queue which is processed by the next level of the team. Sort of an escalation process but the escalation occurs simply because the request isn’t standard. I actually don’t like this method though it is at least acknowledging that you are officially doing it so it means you are probably budgeting for it. The reason I don’t like it… if people know they can order a la carte off the menu flexible support instead of standard inflexible support they will generally, IMO, go for the first because it will be easier for them. This CAN work if you have a bill back mechanism however IT traditionally has HORRENDOUS bill back capability to the business. Usually if they try to bill something back the business tells them to suck it up and just do the work.

So, the most efficient mechanism to run IT large scale and actually hit SLAs without running yourself ragged is to run in a standardized model and to not deviate from it. The deviations puts you into some bastardized version of the ad hoc / standardized model and again, unless you properly plan for it and budget it, it will screw you, ESPECIALLY if you are trying to reduce staffing needs in IT, etc. A team that runs in that model but has some cowboy on the team who doesn’t mind helping people around the rules is killing the entire support model and taking any ability of the team to be efficient away.

Yes yes yes, there is always someone who thinks what they are doing is soooo important that they deserve special handling and shouldn’t have to follow the rules, etc… That is fine, cough up the cash. If you are as important as you say, that shouldn’t be an issue. If you can’t get that special budget approval, then why are you trying to steal the resources from IT and support?

 

2. If someone says that groups or actually any objects such as OUs or computer accounts or whatnot have to be whipped up at the drop of the I always have to ask how they are doing their planning. For instance, the fastest I have seen someone order and get a server into place from a manufacturer is about 3 days and that was with tremendous focus on that and EVERYONE knew it was coming because the project was that important… So given 3 whole days, what is the point of waiting until the hardware is in the building, racked, cabled, hardware configured, and software loaded (figure at least 5 hours rushing) to submit a ticket saying you need a new computer account in the next 30 minutes… All that says to me is that someone needs to be slapped. Oh maybe a computer is being renamed… So rename it, that isn’t a new account creation. About the only place I can see a new for an emergency computer account creation is if you are repurposing hardware and even that usually isn’t something that is done in 30 minutes, there is a leadtime and putting in a ticket and having it processed within 12 hours is usually sufficient. If not, ok, high priority ticket and how often does that happen, once a year? Once a quarter? Any more than that and you again have to start figuring out why you can’t plan things properly. As for groups, a lot of the same planning questions come up but in addition there is usually even better planning that should be in place for groups because the Microsoft solution for figuring out who has access to what sucks so bad. If you don’t have a good solid plan for group management and good planning around it all then you are screwed in that area, ad hoc deployment and use of groups usually isn’t a good thing. Especially if you are throwing out new permissioning at various levels in a directory structure.

Now I have said all of that and someone is adamant… no no no you are wrong joe, that stuff can still be an emergency and we deal with it all of the time and we are GREAT planners… ;)  Well then you have a simple solution, provisioning system, sit down for a couple of hours or a couple of days and whip up a web page to do the work on the behalf of others. Then they can use that web page any time they want to get those emergencies handled. Actually in the place that I worked prior to getting ripped into working on Exchange 2000 I had just about finished a web provisioning system that allowed folks to create their own groups and servers and what not. It wasn’t because we had emergencies to deal with, it is was because I was looking to getting out of the request business entirely. I think in a really well run environment the DAs are

1. Making sure the monitoring and provisioning tools are working
2. Adding/removing domain controllers as needed for new sites or sites being removed
3. Thinking up better ways to run things
4. Working with application owners that want to use the directory
5. Watching over security (auditing, patching, etc)
6. Dealing with the occasional hardware failure

Anyway, just as the Exchange environment got completed and I was getting ready to go back and finish the web system, my contract was terminated so that was never finished. To bad, so sad. 🙁

 

3. How do you prevent people from adding servers to a domain when they have the ability to add workstations… How is that done? You can’t. Microsoft was a trifle silly and while they had the chance to allow us to do that with server computer objects and workstation computer objects (both derived from computer objects) they didn’t do it. In part, it is a legacy issue, if an NT4 machine joins a domain, is a server or a workstation, the lines weren’t as clear with NT4 and certainly NT4 had no concept of server versus workstation accounts though Windows 2000 and better machines could have…

So… What did we do to prevent people from adding servers? We actually didn’t directly prevent them from doing it. We made people realize that we would figure out that they did and break it for them if they did. If you go into that company and mention JAIL any admin who works on workstations or whatnot will know exactly what you are talking about. There is actually a sort of fear of JAIL. What is JAIL? JAIL is an OU that exists in every domain and only has read access for localsystem and Enterprise/Domain Admins. If a machine is found that for some reason doesn’t measure up the corporate standards for naming or operating system or being a server in a workstation OU or a workstation in a server OU it went to JAIL. The machine account would be moved into the JAIL OU, disabled, and all permissions stripped from the object. Why do that you ask? Why not delete it? Because when you do this, the computer comes up with a nice message about the computer account not being able to be found and then the admin tries to rejoin the computer and gets the message that the computer account already exists… That machine name is now completely unavailable. So they try a different name and that gets in, next thing they know, same thing happens there. Now if they could keep readding with the same name, that might be an ok thing, but changing the name over and over just makes it too much of a hassle and really unusable. The key to all of this is a script that runs hourly and looks for the bad things and jails them when found. In the beginning there were quite a few people who figured the rules didn’t apply to them or that they could sneak something by, they all learned that the rules did apply and they couldn’t sneak anything through. So while we couldn’t prevent them from adding a server, it was worthless to them to add servers. The bonus was that if someone kept trying to do it, we would just take away their rights to do any admin. Being an admin isn’t a right, it is a privilege and the DAs are ultimately responsible for the security and stability of AD. If someone is abusing it or not following the rules or standards, they lose their privilege.

 

So those are the responses to the major questions. Of course there are still going to be folks who say, you just can’t run things that way, it won’t work. Don’t worry, I used to hear that EVERY time I made a change to lock things down. Not once did I run into an occasion where a well trained admin couldn’t do their work. The lesser capability admins tend to require the most rights to do things because they really don’t understand what they are doing. This isn’t entirely their fault, a lot of the tools MSFT puts out, especially for things like Exchange enforce this… For instance what kind of rights do you need to mailbox enable a user? Most people will say full control on the user object (or preferably at least account operator) and some level of Exchange rights, probably Exchange Admin or may be Exchange View if they are one of the more informed Exchange admins. Incorrect, all you need to mailbox enable a user is one of the following

1. write access to homemdb and mailnickname
2. create a user
3. create an OU or container
4. change the SD on a container, OU, or user.

However, the tools Exchange supplies requires more permissions including at least Exchange view.

 

Don’t get me wrong, don’t just start taking everyone’s rights away to do everything. I don’t even do that, you need to look at what work is being done and who does it and possibly move some responsibilities around. This moving of responsibilities may be done before or after the lockdown but is likely to happen. For instance, previously, local site admins could create groups for their site in their NT4 resource domains, when they came to AD, they lost that right, they could only modify membership of groups. Did they like, what do you think? Not at all and just about every one of them said that everything would fail now but in no case out of the 500 or so migrated in resource domains did that ever happen. Everyone could always do their jobs and overall everything ran much much much better.

  joe