joeware - never stop exploring... :)

I was planning to release AdFind and AdMod today as per the normal yearly cycle but I am not quite happy with them yet so I am holding off for a little bit. I expect I should be releasing them within two weeks at the latest assuming nothing bad comes up.

   joe

[Thu 02/09/2012 23:44:13.22]
C:\>adfind -schema -f attributeid=1.2.840.113556.1.4.2066 attributeid

AdFind V01.46.00cpp **BETA** Joe Richards (joe@joeware.net) January 2012

Using server: WIN8Dom-DC1.win8dom.loc:389
Directory: Windows Server 8 Developer Preview
Base DN: CN=Schema,CN=Configuration,DC=win8dom,DC=loc

dn:CN=ms-DS-Required-Domain-Behavior-Version,CN=Schema,CN=Configuration,DC=win8dom,DC=loc
>attributeID: 1.2.840.113556.1.4.2066

1 Objects returned

[Thu 02/09/2012 23:44:40.89]
C:\>adfind -rootdse | grep 1.2.840.113556.1.4.2066

AdFind V01.46.00cpp **BETA** Joe Richards (joe@joeware.net) January 2012

File STDIN:
>supportedControl: 1.2.840.113556.1.4.2066 [LDAP_SERVER_POLICY_HINTS_OID]

I am thinking someone who is the gatekeeper for the corporate OID stash (or at least the DS portion of the OID stash) made a booboo.

   joe

As more and more companies move forward with automated provisioning and automated password management and workflow an issue was found with tools like FIM that proxy the password change. I.E. You identify yourself to FIM and tell it to change the password for you, FIM uses the SET Password operation instead of the CHANGE Password operation which bypasses the password history rules.

What does that mean? In other words, say your company has a password policy that includes complex passwords that have to be changed every month and can’t be repeated for 25 passwords. Normally that means if you want to use JoeRocks1! as your password every month you would have to change it to 24 other values first and then re-change it back to JoeRocks1![1].  Well this "bug", or really, implementation flaw with using SET instead of CHANGE means that you could just tell FIM to re-use JoeRocks1! every month. FIM doesn’t perform any validation, it just depends on the OS and again… It is telling the OS to ignore the history policy via the use of SET.

To fix this, Microsoft released a patch for Active Directory which introduces a new LDAP control that helps alleviate this issue. This control is called… tada – LDAP_SERVER_POLICY_HINTS_OID and is used with password SET operations.

The KB article for the curious is http://support.microsoft.com/kb/2443871

 

For the folks that use AdMod, I added functionality this evening to allow AdMod to take advantage of the new control. There is a new switch called -policyhints which enables the control.

[Thu 02/09/2012 23:03:49.78]
C:\>admod -default -rb cn=testuser1,cn=testusers unicodepwd::Password2! -optenc -exterr -policyhints

AdMod V01.18.00cpp BETA Joe Richards (joe@joeware.net) February 2012

DN Count: 1
Using server: WIN8Dom-DC1.win8dom.loc:389
Directory: Windows Server 8 Developer Preview
Base DN: cn=testuser1,cn=testusers,DC=win8dom,DC=loc

Modifying specified objects…
   DN: cn=testuser1,cn=testusers,DC=win8dom,DC=loc…

The command completed successfully

[Thu 02/09/2012 23:04:01.46]
C:\>admod -default -rb cn=testuser1,cn=testusers unicodepwd::Password2! -optenc -exterr -policyhints

AdMod V01.18.00cpp BETA Joe Richards (joe@joeware.net) February 2012

DN Count: 1
Using server: WIN8Dom-DC1.win8dom.loc:389
Directory: Windows Server 8 Developer Preview
Base DN: cn=testuser1,cn=testusers,DC=win8dom,DC=loc

Modifying specified objects…
   DN: cn=testuser1,cn=testusers,DC=win8dom,DC=loc…: [WIN8Dom-DC1.win8dom.loc] Error 0x35 (53) – Unwilling To Perform

   Extended Error: 0000052D: SvcErr: DSID-031A126A, problem 5003 (WILL_NOT_PERFORM), data 0

ERROR: Too many errors encountered, terminating…

The command did not complete successfully

[Thu 02/09/2012 23:04:04.85]
C:\>admod -default -rb cn=testuser1,cn=testusers unicodepwd::Password2! -optenc -exterr

AdMod V01.18.00cpp BETA Joe Richards (joe@joeware.net) February 2012

DN Count: 1
Using server: WIN8Dom-DC1.win8dom.loc:389
Directory: Windows Server 8 Developer Preview
Base DN: cn=testuser1,cn=testusers,DC=win8dom,DC=loc

Modifying specified objects…
   DN: cn=testuser1,cn=testusers,DC=win8dom,DC=loc…

The command completed successfully

 

[Thu 02/09/2012 23:29:28.30]

C:\>err 52d
# for hex 0x52d / decimal 1325 :
  ERROR_PASSWORD_RESTRICTION                                    winerror.h
# Unable to update the password. The value provided for the
# new password does not meet the length, complexity, or
# history requirement of the domain.
# 1 matches found for "52d"

 

   joe

 

[1] Don’t laugh, a few years ago I ran into some folks who had set up a batch file to do something exactly like this… Well they weren’t using JoeRocks1! but they used the whole idea of looping through enough passwords to be able to reuse the same password every month.

Previously we had a discussion on ActiveDir Org about using Tree Delete on an OU or other container with a large number of objects, specifically more than 16,000 or so objects (16384 to be even more specific).

http://www.activedir.org/ListArchives/tabid/55/view/topic/postid/44636/Default.aspx

MSDN Article about the error

http://msdn.microsoft.com/en-us/library/cc223490(v=prot.10).aspx

And an old blog post from me about the error

http://blog.joeware.net/2007/06/01/905/

 

Anyway, on the ActiveDir Org thread I mentioned that I should add a treenuke option sometime. Well I have now down that, it will be available for public destruction when V01.18.00 is released which will hopefully occur in the next week or so if I can fully sort out a bug that our friend Brian Desmond sent me recently when trying to "add" conflict objects from one AD to another AD.  The fix for that is to print out an error that says "Seriously Brian????", no, I am not sure what the fix will be in the end yet but I know where part of the issue is. 😉

More on the tree nuke. This is simply a little piece of code that I added which simply loops on the treedelete LDAP deletion operation when you specify the -treenuke switch. Initially that was all I was going to do but then when I ran it trying to delete a container with over 100,000 objects AdMod just seemed to sit there looking stupid so now I have it perform a little extra work. For every treedelete LDAP operation it submits, it adds another dot onto the output on the line… So for example, instead of just seeing:

DN: cn=nuke,cn=adamrocks…

You will see

DN: cn=nuke,cn=adamrocks…

and then

DN: cn=nuke,cn=adamrocks….

and then

DN: cn=nuke,cn=adamrocks…..

and then

DN: cn=nuke,cn=adamrocks……

etc.

Not really awesome in terms of some of the stuff they do in the GUI world but if you are looking for awesome stuff from the GUI world you have made a serious mistake coming here…

So for the real example or as I like to call it, how to create and then destroy 100,000 users in 30 minutes:

1. Create the container

[Tue 02/07/2012 22:20:01.00]
F:\>f:\dev\cpp\admod\release\admod -hh . -b cn=nuke,CN=ADAMRocks -add objectclass::container

AdMod V01.18.00cpp BETA Joe Richards (joe@joeware.net) February 2012

DN Count: 1
Using server: DellLT17:389
Directory: Windows Server 2008 R2 Active Directory Application Mode

Adding specified objects…
   DN: cn=nuke,CN=ADAMRocks…

The command completed successfully

 

2. Create 100,000 users

[Tue 02/07/2012 22:20:18.89]
F:\>f:\dev\cpp\admod\release\admod -hh .  -sc adamau:100000;password;cn=testuser,cn=nuke,cn=adamrocks

AdMod V01.18.00cpp BETA Joe Richards (joe@joeware.net) February 2012

DN Count: 100000
Using server: DellLT17:389
Directory: Windows Server 2008 R2 Active Directory Application Mode

Adding specified objects…
   DN: cn=testuser_0,cn=nuke,cn=adamrocks…
   DN: cn=testuser_1,cn=nuke,cn=adamrocks…
   DN: cn=testuser_2,cn=nuke,cn=adamrocks…
   DN: cn=testuser_3,cn=nuke,cn=adamrocks…
   DN: cn=testuser_4,cn=nuke,cn=adamrocks…
   DN: cn=testuser_5,cn=nuke,cn=adamrocks…
   DN: cn=testuser_6,cn=nuke,cn=adamrocks…
   DN: cn=testuser_7,cn=nuke,cn=adamrocks…
   DN: cn=testuser_8,cn=nuke,cn=adamrocks…
   DN: cn=testuser_9,cn=nuke,cn=adamrocks…
   DN: cn=testuser_10,cn=nuke,cn=adamrocks…
   DN: cn=testuser_11,cn=nuke,cn=adamrocks…
   DN: cn=testuser_12,cn=nuke,cn=adamrocks…
   DN: cn=testuser_13,cn=nuke,cn=adamrocks…
   DN: cn=testuser_14,cn=nuke,cn=adamrocks…
   DN: cn=testuser_15,cn=nuke,cn=adamrocks…
   DN: cn=testuser_16,cn=nuke,cn=adamrocks…
   DN: cn=testuser_17,cn=nuke,cn=adamrocks…
<SNIP>
   DN: cn=testuser_99979,cn=nuke,cn=adamrocks…
   DN: cn=testuser_99980,cn=nuke,cn=adamrocks…
   DN: cn=testuser_99981,cn=nuke,cn=adamrocks…
   DN: cn=testuser_99982,cn=nuke,cn=adamrocks…
   DN: cn=testuser_99983,cn=nuke,cn=adamrocks…
   DN: cn=testuser_99984,cn=nuke,cn=adamrocks…
   DN: cn=testuser_99985,cn=nuke,cn=adamrocks…
   DN: cn=testuser_99986,cn=nuke,cn=adamrocks…
   DN: cn=testuser_99987,cn=nuke,cn=adamrocks…
   DN: cn=testuser_99988,cn=nuke,cn=adamrocks…
   DN: cn=testuser_99989,cn=nuke,cn=adamrocks…
   DN: cn=testuser_99990,cn=nuke,cn=adamrocks…
   DN: cn=testuser_99991,cn=nuke,cn=adamrocks…
   DN: cn=testuser_99992,cn=nuke,cn=adamrocks…
   DN: cn=testuser_99993,cn=nuke,cn=adamrocks…
   DN: cn=testuser_99994,cn=nuke,cn=adamrocks…
   DN: cn=testuser_99995,cn=nuke,cn=adamrocks…
   DN: cn=testuser_99996,cn=nuke,cn=adamrocks…
   DN: cn=testuser_99997,cn=nuke,cn=adamrocks…
   DN: cn=testuser_99998,cn=nuke,cn=adamrocks…
   DN: cn=testuser_99999,cn=nuke,cn=adamrocks…

The command completed successfully

3. Issue standard treedelete command and watch it partially complete and then fail.

[Tue 02/07/2012 22:41:56.54]
F:\>f:\dev\cpp\admod\release\admod -hh .  -b cn=nuke,cn=adamrocks -treedelete -del

AdMod V01.18.00cpp BETA Joe Richards (joe@joeware.net) February 2012

DN Count: 1
Using server: DellLT17:389
Directory: Windows Server 2008 R2 Active Directory Application Mode

Deleting specified objects…
   DN: cn=nuke,cn=adamrocks…: [DellLT17] Error 0xb (11) – Administration Limit Exceeded

ERROR: Too many errors encountered, terminating…

The command did not complete successfully

4. Count the objects that are left.

[Tue 02/07/2012 22:43:26.09]
F:\>f:\dev\cpp\adfind\release\adfind -hh . -b cn=nuke,cn=adamrocks -c

AdFind V01.46.00cpp **BETA** Joe Richards (joe@joeware.net) January 2012

Using server: DellLT17:389
Directory: Windows Server 2008 R2 Active Directory Application Mode

83617 Objects returned

 

5. Nuke it!

[Tue 02/07/2012 22:47:38.82]
F:\>f:\dev\cpp\admod\release\admod -hh .  -b cn=nuke,cn=adamrocks -treenuke -del

AdMod V01.18.00cpp BETA Joe Richards (joe@joeware.net) February 2012

DN Count: 1
Using server: DellLT17:389
Directory: Windows Server 2008 R2 Active Directory Application Mode

Deleting specified objects…
   DN: cn=nuke,cn=adamrocks………

The command completed successfully

 

6. Count the remaining objects.

[Tue 02/07/2012 22:51:09.27]
F:\>f:\dev\cpp\adfind\release\adfind -hh . -b cn=nuke,cn=adamrocks -c

AdFind V01.46.00cpp **BETA** Joe Richards (joe@joeware.net) January 2012

Using server: DellLT17:389
Directory: Windows Server 2008 R2 Active Directory Application Mode

ldap_get_next_page_s: [DellLT17] Error 0x20 (32) – No Such Object

Best Match of: ‘CN=ADAMRocks’

0 Objects returned

[Tue 02/07/2012 22:51:15.30]

We, as IT people in general and AD people in particular, often get dinged with the ubiquitous “quick question”. You all know what I mean, the “Hey I have a quick question” or “Hey do you have a second for a quick question” or most often just “Hey, got a sec?” as the person sits down and searches your desk for cookies, candy, or other things that they have no right to but will instantly latch onto as they settle in for decidedly more than “a second”.

Or if you are lucky enough to work at home the IM window pops up with “Got a sec? ” and you look to make sure you previously set your status to unavailable or away so if you want, you can just ignore the implied lie behind the seemingly harmless text with the disarming smiley face.

Either way, in our minds we are screaming “NOOOOOOOO, not for you. My life is composed in its entirety of ‘these seconds’ you take so cavalierly and I would rather not waste them on whatever you think will only take ‘just a second’ plus I have the letters Q, J, K, D, W, B, and O in Words with Friends and I have no clue where I am going to place any of them and am already losing by 160 points.” But… in the end… we know that saying no is pretty much pointless and that “second” could turn into three times as long as it would have been anyway if we waste the time trying to fight it… so we only get to respond with… “Sure, what’s up?” and may even feign some level of enthusiasm for dramatic effect.

Anyway…

How big is the AD DIT?

“Got a sec?”

“Sure. What’s up?”

“What is the size of the AD DIT?”

<LONG PAUSE with deep breath>

The only thing “got a sec” about that question is the amount of time to utter the syllables. The only person that single datum is valuable to is the person worried about disk space on a domain controller so unless you are looking to figure out how big of a disk you need to order for your next DC or perhaps you are in a “who has the biggest DIT contest???” asking that specific question is simply the act of pushing the first in a long chain of dominos.

So, instead of looking at your favorite DC and quickly spouting whatever the value is you instead[1] say “Why?”. You then get the response you likely were dreading… “Because we are having problems with Exchange and the Microsoft support guy wants to know how big the DIT is.”……………….. Sigh.

Some of you may be asking? But joe? What’s wrong with asking that question? The problem is that the answer to that question doesn’t really tell you anything without the appropriate contextual information to go around it. Say the answer is 3GB. What does that mean? Do we jump for joy? Do we skulk in shame? Do we yip in pain? I don’t know. It could be good, it could be bad, it may not matter at all – who am I to know with the information in front of me?

The answer starts to make some amount of sense once you know the OS level, Windows 2000 versus Windows Server 2003 versus Windows Server 2008 R2 . It makes more sense when you have some clue as to what other functions are running on the domain controller and what memory load those functions utilize. And finally it makes a heck of a lot more sense when you know where on the scale between 256MB of RAM and 64GB of RAM that your domain controller is at.  The fact that you have a 6GB DIT means something entirely different on machine with Windows 2000 and 512MB of RAM with SQL Server running in the background than it does with a Windows Server 2008 R2 with 16 processors and 64GB of RAM and only running DNS and AD functions. So simply asking “How big is the DIT?” is like asking how much oxygen is in the room. Without understanding context around it, it is pointless.  

SIDEBAR: That being said, how nice would it be to have a fancy RootDSE operational attribute that you could query on all of your DCs for some value that gives you a clue about DIT size versus RAM utilization so if someone was say, troubleshooting Exchange or something else, they could query the DC for that attribute and it would give them an idea on whether or not they should follow up with the DAs, or perhaps the DA’s could even monitor[2] the attribute across all of their DCs and be alerted that perhaps they need to be a little more aggressive in checking things out. Sure sure there are a ton of performance counters available that could be used but in all reality, most admins look at them and their eyes glaze over. Heck my eyes don’t much like them either. It would be nice if they broke those out by role and feature like they have been doing with the Server Manager functions[3]. Anyway Microsoft Exchange Support Engineers, imagine if you could ask the Exchange folks you are working with if they could do a quick LDAP query of the RootDSE of a DC to get the answer you really want versus asking them to ask someone else what the size of the DIT is? Heck it could be put into the ExRAP tool as well as the Baseline Analyzer tools.

We are seeing delays in replication…

“Got a sec?”

“Sure. What’s up?”

“We are seeing delays in replication, why?”

<PAUSE>

My response, to get a feel of what direction the questioner is driving and what kind of vehicle they are using is usually of the type “Why do you think there is a delay?” That often, but not always results in a response of the type “It just doesn’t seem to be moving as fast as we would expect.” Which I translate in my head to “We have no clue how long it is supposed to take and our stuff isn’t working correctly and we need a wall to throw the problem over…” and when I get the feeling someone is looking for a wall to toss things over I usually come out with the old standby “You need to get a network trace of the problem” which tends to make them go away for awhile if not permanently when they find some other group to accept the task of troubleshooting their problem.

But in this case of replication delay there is a better response… “What is your expected theoretical max replication latency from the source DC to the destination DC?” If they say they don’t know then I respond with “How do you know you are seeing delays? You don’t even know how long it is supposed to take in the first place.” The fact that it “feels slow” or isn’t what you expect doesn’t mean it is delayed. The entire issue could be and very often is that they have an incorrect expectation. To be able to make an objective claim of “it is delayed” means you have a thorough understanding of what it is designed to be and is during normal functioning. You should be able to say it is delayed by x minutes or hours and be able to point at the expected latency based on the design and point at what it is really taking.

SIDEBAR: And again… That being said, it doesn’t seem like it would be terribly hard for the AD site and subnet tool or for some tool supplied by MSFT that could tell you the expected max theoretical convergence time when selecting a source and destination DC. I actually have, and have had for some time, a tool listed on my “tools to build someday” list that could do this. Unfortunately, my time isn’t as free as it once was and you may notice that joeware updates and tools don’t flow quite as freely as previously. This is being worked on but MSFT definitely has quite a few more available man hours for producing things like this. Again, how nice would it be for the PSS guys to tell the admin that is having problems, fire up this tool, click on the DC that you put the change on, click on the DC you want the change to get to, and the tool will tell you a theoretical minimum and maximum time frame we have for convergence assuming a properly running replication environment.

Why are my LDAP queries going slow???

“Got a sec?”

“Sure. What’s up?”

“Why are my queries going slow?” or alternately “The PSS ExRAP or the Exchange PSS guy says the LDAP Queries are going slow. Why?”[4]

<PAUSE>

My response to this is always, “What exactly is the query that is going slow? Specifically I want the Host you are querying, the search base, the search scope, and the filter and what attributes you are asking for.” This one is really quite annoying to me because the Exchange people through the years have really irked me by looking at some DSACCESS counters and it says things aren’t good but no one can tell me specifically what it is that isn’t good…. Just something. Sorry, that isn’t good enough. Find out the queries, try them manually and show me that they are not performing properly. Otherwise I am more likely to believe based on personal experience through the years that Exchange is screwed up in its configuration somewhere versus the DCs not functioning properly. A problem isn’t a problem to me unless you can show me specifically what isn’t working properly as it applies to me, showing me some generic counter from your application isn’t proof. It has literally been dozens if not more times that someone has come to me with those DSACCESS counter complaints and I start performing LDAP Query tests on the DCs and the DCs are operating just fine and I tell the Exchange folks that and they go off and find something else to blame.

If you come to me with specific queries, I can *usually* determine why they are going slow and it is 98.9% of the time because of a poorly formulated query or a real poor choice for search scope or complete lack of anything resembling an indexed attribute. Have I had DCs that were underperforming, yes, but that is the rounding error compared to the other issues that resided outside of the domain controller.

SIDEBAR: And finally… Debugging LDAP queries on Active Directory and ADAM, IMO, is more painful than it should be. Most LDAP directories I have seen have a simple LDAP query debugging capability that dumps LDAP queries and debugging info into a simple text log file; Active Directory doesn’t have this. I know there is the whole Tracing thing but I have had zero time to dig into it and if it requires me to dig in and study it to figure it out, it is too difficult to enable and use.

Anyway, that is my rant for the day.  Have a good week and Happy Lunar New Year / Chinese New Year – Year of the Dragon.

 

 

   joe

[1] Because you naively think you can nip the whole chain of events you know is about to start in the bud.

[2] Monitor – to proactively and automatically check the service quality, availability, and functionality of your service in substantial regular intervals and alert on system faults and non-optimal performance. I only define this because lately I seem to be finding a lot of people who think the best “monitors” for AD are called “Users” and “The Help Desk”. When your users contact you to tell you the service isn’t working, that isn’t called monitoring, that is called failing. 

[3] And perhaps they have been in the most recent versions of the OS. I, unfortunately, seem to be spending a lot of time on Windows Server 2003 lately which is a step up from the Windows 2000 I had to keep dealing with previously.

[4] Yes yes I am picking on Exchange. But as I said years ago completely off the cuff in a humorous (but serious) manner in a Dean and joe Show session at one of the Directory Experts Conferences, <finger air quotes>Exchange is Special</finger air quotes>. To be honest, they aren’t the only ones I have had issues with this over the last 12 or so years, but they certainly win the award for the most consistent and excessive volume. :D  I also had some nice fun with issues around poorly written LDAP queries with IBM’s WebSphere Portal application software. That one was pretty bad, IBM consultants onsite testing WebSphere functionality against a test DC sitting on the same switch as their app server… A DC with an AD they built “out” with 5 users and 3 groups on hardware that was 50 times better than anything anyone has ever used anywhere in the world for a DC and then getting pissed when they try to run the same queries against an environment with hundreds of thousands of users and hundreds of thousands of groups across 6 routers shared with thousands of people.

So today I needed to test a script and as part of the test I needed to update a multi-value attribute on my own ID without admin rights. I wasn’t sure off the top of my head which attributes I could modify were multi-value so just told Active Directory to tell me…

for /f %i in (‘adfind -default -f "name=joe is freezing" allowedattributeseffective -list’) do @adfind -sc s:%i -af issinglevalued=FALSE -nodn attr:%i issinglevalued -csv –nocsvheader

 

That gave output like

"otherPager","FALSE"
"otherHomePhone","FALSE"
"otherTelephone","FALSE"
"otherFacsimileTelephoneNumber","FALSE"
"otherMobile","FALSE"
"otherIpPhone","FALSE"
"url","FALSE"
"userCertificate","FALSE"
"userSharedFolderOther","FALSE"
"preferredDeliveryMethod","FALSE"
"mSMQDigests","FALSE"
"registeredAddress","FALSE"
"internationalISDNNumber","FALSE"
"x121Address","FALSE"
"teletexTerminalIdentifier","FALSE"
"telexNumber","FALSE"
"postOfficeBox","FALSE"
"postalAddress","FALSE"
"msPKIDPAPIMasterKeys","FALSE"
"msPKIAccountCredentials","FALSE"
"msPKI-CredentialRoamingTokens","FALSE"
"userSMIMECertificate","FALSE"

 

In this case, the FALSE is a reference to the value of isSingleValued and of course a multivalued attribute would have a FALSE value for that property for the attribute in the schema.

   joe

So another new file system supposedly coming out of Redmond…

http://www.engadget.com/2012/01/17/microsoft-introducing-refs-file-system-with-windows-server-8/

 

or if you prefer a variety of articles

http://bit.ly/wS2rhK

 

Hopefully it will fix this type of issue…