A comment came in today about my admod post from yesterday that prompted a response that I thought was good enough to pull up to a full blog entry in its own right instead of responding in the comments…
The question is from Fred and goes like this…
So, Joe, which tools would you say you use in your general work with AD everyday? What would you say your troubleshooting steps and utilities for AD are? I know you probably do this stuff in your sleep, but would you be willing to… help a brotha out? 🙂And yes, I do definitely agree: adfind kicks mucho butt!
That is a pretty good question that I actually receive on a fairly regular basis, especially when someone hears I used to be the technical lead of a team of three people (including me) that managed a fairly large AD environment  – ~350-400 DCs in nearly every timezone across the world, 250K users, 180K Exchange enabled users, everybody and their third cousin on their mom’s side wanting something from us, everyone submitting their tickets to us because they couldn’t figure out what was wrong, etc… Well my #1 tool is going to sound like an advertisement but is the dead honest truth. The tool I use the absolute most without any doubt in my mind is adfind. I use it each and every single day that I touch a computer, not only for work but at home because I tend to do a lot of AD stuff at home to figure things out that people email me. I couldn’t even guess the number of times a day I use the tool but at a minimum I probably do 20-30 schema attribute lookups alone which is why I put in the -sc s: option
e.g. adfind -sc s:gatewayProxy
Then there are just any number of other queries to look at the general info stored in a given object… Here are some queries that are in my history of my home machine of just one command prompt I opened today… All of the schema def lookups that are just lookups are excluded.
adfind -config -f name=”default policy”
adfind -h . -config -f name=”directory service” msDS-Other-Settings
adfind -h . -b -pr -f objectclass=user objectsid
adfind -default -t 0 -f joewaretestnumeric=* -dn
adfind -sc s:joewaretestnumeric -dsq |admod attributesyntax::2.5.5.12 -exterr
adfind -sc sl:joeware*
adfind -default -f mail=joe@joeware2.net
adfind -default -f mail=joe@joeware.net
adfind -default -f objectsid=S-1-5-21-1862701446-4008382571-2198042679-21236 -dn -extname
adfind -default -f name=someuser -extname
adfind -h 2k3dc01 -gc -b -binenc -f msExchMasterAccountSid={{SID:S-1-5-21-1862701446-4008382571-2198042679-21236}} -dn
adfind -gc -b -f msExchMasterAccountSid=* msExchMasterAccountSid
adfind -gc -b -f name=2k3dc10 serviceprincipalname
Â
So AdFind without a doubt is the tool I use the most often. Heck I use it to look up my employeeID in my work directory anytime I need it for anything, I can do that faster than looking it up on the piece of paper I have it written on. Also I would say pretty much who has even worked with me extensively probably uses AdFind a lot as well including my old manager who isn’t even supposed to be touching AD.
After adfind I would say probably dsacls is pretty heavily used, mostly for viewing but even that is slowly moving towards adfind now that I added the -sddl+ and -resolvesids options. Most troubleshooting work is about looking at things, not changing things, that is why I constantly tell people that most admins really shouldn’t be using their admin IDs a whole lot unless they are strictly doing implementation. Troubleshooting should be a lot of looking with very little changing and that changing only occurring after you really are sure that that targeted strike is going to fix the problem. Too many people get themselves in far too much trouble just changing things to see if that fixes things. This is also why you don’t generally need very many admins. I constantly hear stories of directories with upwards of 100 admins and there is no reason in the world for that other than people who want to put themselves in the possible position to feel a lot of pain and do a lot of unnecessary work.
After those two  I would say I have a variety of very custom perl scripts for doing things like getting the full membership of a user across an entire forest, getting the full expanded membership of a group, dumping ACLs in special ways or only dumping specific ACEs of the ACLs across an entire forest, or dumping all of the Exchange Data in a forest and scanning through looking for issues and dumping them out in a report. I have seen companies that have reported tens of thousands of issues with their Exchange data, anything ranging from duplicate values that shouldn’t be duplicated to bad email addresses to who knows what. This script I have that checks that stuff I started working on a long time ago and brought it from my old job to my new job where it has grown even more and become immensely useful. Usually I pull it out when people are saying that Exchange is constantly acting “hokey” and I tend to find just oodles of configuration problems and just plain crap data. All of these scripts btw all leverage adfind as well, so though I am not running adfind directly, it is being used in the backend to do the work for me. Much easier than using ADO for the most part.
Once I get past those items I would say it then goes into admod, ldp, cpau, oldcmp, ADSIEDIT, ADAM-ADSITEDIT, getuserinfo, dsa.msc, dssites.msc, ESM…
Not AD related tools would be Perfmon, Ethereal, EditPlus, Notepad, Outlook, MS Word, OneNote, VPC, Virtual Server, VMWare Workstation,
Â
As for troubleshooting steps… That is a tougher one. It depends on what I am working on. In general I will either try to duplicate the problem or look at the objects involved with the problem (adfind/dsacls) or maybe get a network trace and absorb what is occurring. Again it really depends on the problem and I have found from people trying to shadow me to “learn my ways” that I don’t always do things in a logical step by step manner. I don’t have a problem with people shadowing me but I think there are very few who would really get much out of it because of the way I troubleshoot. I have had junior managers chew me out because I didn’t properly “train people” how to do things and that I was being stingy and not sharing info. Quite the contrary, I tend to flood people with info so they can understand all of the backend and why and hows because that is the info I am using to solving problems. Anyone who thinks that all problems can be whittled down to “if you see this it means this” are living in a dream and junior managers who whine when I don’t show people things in that way can just go off and suck it up.
Anyway, how do I troubleshoot? I usually make a lot of intuitive leaps which tend to take me to the source of the problem faster than some others can get there; I may not logically know up front why I am standing in front of a given server but I can usually back trace how I got there in logical steps, my mind just chops out the drudgery that you generally want to outsource. 😉
I chalk being able to do that up simply to having done support a long time and having done development a long time and getting a “feel” for where problems occur; there are natural pinch or break points in every system and all things being equal, that is where the problems tend to occur. This method if troubleshooting is especially true if the environment is one I have been in before and the problem reminds me of something I was thinking when I was previously looking at the environment and making mental notes on where I expected things to break. Nine times out of ten when I get called back into a place to work on something I am going to be saying “I told you so” to someone because usually I have already pointed out the location where a failure is going to occur and someone didn’t want to listen to me previously. It isn’t unusual where someone will be fixing something months or even years after I pointed it out because whatever I said was going to eventually happen did in fact happen and now they have no choice but to fix it.
My most recent intuitive jump troubleshooting I can think of involved a client having trouble getting email and since the user was a manager who happened to be a previous MCS guy he thought he could troubleshoot it and that his ideas were more accurate that mine since I was several states away and “obviously” guessing because I hadn’t heard everything. Anyway the problem was described and I immediately said it was local network issues, nothing else made sense to me. But of course this former MCS person did not agree and would not let us bring in network services to find the issue so I had to spend a week looking at network traces and pointing out blindingly obvious examples of network failures such as packets out of order, dupe acks, resends, etc which he was positive had to be caused by the Exchange server side while I knew it absolutely wasn’t. We called network services, he canceled the ticket, it went on like that for over a week until I finally said, this is it, I don’t care that you don’t like that answer, get network services and they can fix the problem, I am not working on it anymore. What was the issue in the end? Local switch hard set to 100/Half, corporate load for PCs set to 100/Full, Sygate personal firewall software on client also dorking with packets. Had the person been a normal user instead of a “technical” manager the entire troubleshooting and correction phase would have been done in probably less than an hour due to intuition. Instead it dragged on for over a week while I had to prove over and over again it wasn’t an Exchange issue.
Does that answer the question?Â
   joe
Joe,
Thanks for the insight into your toolset. I was wondering, have you ever used ldapbrowser/ldapsdministrator from Softerra? Since I am more of a GUI guy, tend to look things up using this tool on a single object to then plug into ADFind to get a report for the domain. I used to use logparser to run reports from AD, but I switched to adfind when i notice a 1000% performance improvement.
I have a friend that has help me frontend adfind using excel to help me run standard reports from AD. I am lazy about typing and almost know nothing about coding. I have even used adfind output to run dsmod scripts using the stdin input. I am always trying to find the best/easiest way to do something and you have made my job a lot easier. Thanks Joe
Nope, never used anything from softerra, never had a reason too. The CLI stuff has pretty much served me as it is much faster. If I need the GUI usually LDP or ADSIEDIT will do it.
I have had a couple of MSFT friends who have been after me for about three years to write a GUI around adfind… maybe someday. 🙂
Great article Joe, how early into your troubleshooting do you analyze the event logs. Your tools are certainly some of the best free tools in the world.
Yes it does. I’m printing this out for further “digestion”. Thanks.
Mike: It entirely depends on the situation, possibly I will go to the event logs immediately. Say I need to track down a lockout that is occurring, I would run one script that tells me what DCs are processing bad auths and then run another script that yanks the logs, parses out bad password attempts and generates a report for me.
Most of the stuff I get called to work on anymore though isn’t of that caliber and anything that could be found in the event logs was already found. I tend to get called when everyone else, often times including MSFT, is stumped or there is a possible perceived issue in the overall architecture/design or process.
Man, I can’t imagine that types of things you get to see and work on. I guess that is why you are so good on the boards. You have seen or dealt with a lot of the issues people ask about. You are also very passionate and dedicated to tech and helping people out.
It would be cool to shadow you for a few weeks. What about the Joe Richards adopt an admin program haha
Mike: While it is true I see a lot of odd items, I attribute my success less to that and more to several other things; I will try to voice them.
1. Desire for deep understanding of how things work versus how to fix things. Fixing flows naturally from understanding. It is, in my opinion, worthless to try and memorize symptoms and apply them to a run book of solutions. This does nothing to help with understanding and the symptoms and solutions will change as the OS and applications evolve.
2. Looking at things when they are “working” as well as when they are broken. I am just as quick to pull out ethereal to look at traffic on a network for an app people says is fine as I am one where people say it is broken. Why? I want to see what it looks like when fine. The dirty secret though is that even when things seem fine, some apps can be having all sorts of issues, I often find this with Exchange. The issues just haven’t gotten to the point of failure yet. I guess it is an issue where everyone has grown used to poor perf from Exchange so as long as it works at all, no one questions it. I guess this is just another aspect of #1 so I guess this is 1B instead of 2.
3. Mistrust. I trust very few people and the trust I do have is very segmented to areas that I have found them trustworthy in. It pisses people off but more times than not I have found the issue is something that someone else has told me it couldn’t be because they already checked it or they knew for sure it was something else.
4. Trust my gut. I have found that if I get a feeling I should look in a certain direction, I really should. I think trusting your gut or your instinct is more about listening to your subconscious which is noticing things you aren’t consciously noticing than anything else. That is why it should be listened to. The segmented trust applies here though as well, I don’t always do this (maybe I should) for everything, mostly only for things that don’t matter a great deal to me personally. For instance if I have a gut feeling I could walk around on the roof of the house just fine without a stabilizing rope I will dismiss that quite quickly, the repercussion of being incorrect is a little adverse for my taste. But when working on systems with no chance of bloodshed, broken bones, or loss of anything _truly_ important, gut feeling all day baby! So this is probably 3B or if really keeping track, 2B.
5. I try to be confident and vocal. If I am relatively sure I know what the problem is, I step up and say it and say it with some force. I have found that people will not listen to other people who sound like they are guessing or being wishy washy. This is actually a sales technique, I have several years of big ticket item sales experience under my belt and went through quite a few of the sales classes, etc. This I can mostly apply to working on computers and actual selling. It is like #4 in that I don’t consistently follow this across all aspects of my life so it isn’t just that I think this is easy. It is probably one of the hardest aspects of everything I do. Figuring out technical issues is easy in comparison.
6. I play around and test and try things. So much stuff isn’t documented or is documented incorrectly. You won’t ever know though if you aren’t pushing the limits and trying to do things that you “aren’t supposed to do”. You should question most everything unless you have seen it with your own eyes, even then realize that it could still work differently under different conditions. Right now I am replacing my main lab domain, joe.com, with test.local which is a ground up R2 forest which is quite painful as I am just tossing the old domain out. It shouldn’t be much different than my SP1 domain was but who knows what I will find. Doing this kind of stuff is one of the reasons why I have found and submitted so many bugs and documentation errors to MSFT for various things. This is tough to do because there is only so much time in a day though.
7. I have fun. If I am not having fun with something, I try to find a way to make it fun again because the moment I stop enjoying what I am doing everything gets tougher and it becomes a “job” instead of something that I want to do. This is one of the reasons why you will see my posting to activedir org slow down sometimes or postings to the newsgroups will slow down. I am not having fun and if I force myself to continue with it as I am it will be something I learn to hate. So usually I will just slow down for a bit and that solves it, at some level I do enjoy helping people. Plus it leads into my next and final item.
8. Keep your eyes open for SEP’s. If you have read the Adams series (as well you should) you know that an SEP is a “somebody else’s problem”. In the book, Adams mentions that certain races of aliens use an SEP field to hide their spaceship because people don’t want to see other people’s problems and with it they can say set down the ship in the middle of a cricket pitch and no one will see them (A spaceship is obviously not my problem, I should focus on something else that is). I think purposely going after those problems is a great way to learn and see what can happen to yourself. If I see a problem that I don’t recognize right away I will often find myself setting it up or trying to work through it logically to see where it could come from. Obviously I can’t do that for everything as I am already jam packed for most days but I have my favorites types of issues and those are the ones that you will see AD Org posts from me on or newsgroup posts. My newsgroup posts in general cover a far wider range of things than my AD Org posts from C++ questions to kernel API questions to security questions to service programming questions and even some AD type questions.
More on SEPs here – http://en.wikipedia.org/wiki/Somebody_Else's_Problem_field
The Hitchhiker “trilogy” is great and if you haven’t read it, you should. Especially if it is sitting around handy somewhere. I tend to use a lot of Hitchhiker references because it is great fun and helps me with #7. If someone gives you the trilogy, they love your smile or laugh and want to see you happy. It is that good. I think it is quite tough to read it and not find yourself laughing out loud in parts.
“Heck I use it to look up my employeeID in my work directory anytime I need it for anything, I can do that faster than looking it up on the piece of paper I have it written on. ”
LOL. So I was not the only one doing that !
But do you query the authoritative directory (ED) or trust the AD ? 😉
In my heart, AD is always authoritative. 🙂