joeware - never stop exploring... :)

I recently had a conversation with an Active Directory engineer who was working on some token bloat issues. As AD has pushed past the decade mark I seem to be seeing/hearing more and more of this bloat problem as many companies continue consolidating groups into and creating new groups in AD without cleanup or normalization.

To be more clear, say a company took whole hog the groups they had before, likely in a couple, but possibly up to hundreds of NT4 domains and collapsed it all into a single forest or they took all of the machine local groups on member servers and sucked them up into AD for “easier management and reporting”. On top of that, all of it has remained with no cleanup and companies moved forward adding more permission and DL groups and merging/consolidating more and more domains (or companies), etc. And then on top of THAT, you get companies who used sIDHistory but never went back and cleaned up what the sIDHistory was initially needed for. What should take place is group/permission normalization, ACL cleanup, etc.

Err ok, I just sidetracked myself with a soapbox speech… back to the AD Engineer patiently waiting for me to shut up and listen to what he wants to accomplish…

So part of the problem this engineer is experiencing is that the Active Directory he is concerned about had been through a double hop AD Consolidation after multiple mergers/acquisitions. I.E. OldOldDomain was migrated/collapsed into OldDomain and later OldDomain was migrated/collapsed into ShinyNewDomain[1]. The migrations involved pulling the objects over whole hog and populating sIDHistory in the new accounts. So what you ended up with was (SIDs shortened for readability):

OldOldDomain – Domain SID S-1-5-12345
OldDomain – Domain SID S-1-5-54321
ShinyNewDomain – Domain SID S-1-5-24680

So a migrated object in ShinyNewDomain has objectSid of S-1-5-24680-2500, and has sIDHistory values of S-1-5-54321-78 and S-1-5-12345-19.

Now let’s say some cleanup had actually occurred and any ACLs with references to SIDs from OldOldDomain were believed to be all gone but there definitely were resources using OldDomain SIDs still. Simple right? Target and remove the OldOldDomain SIDs and leave the OldDomain SIDs in place.

Well unfortunately, the engineer had spoken to another engineer about it and was told it wasn’t possible to perform granular sIDHistory cleanup. He was further told that the process he would have to use to perform this cleanup was to go back to OldDomain, clear the sIDHistory attribute of all objects there, then clear all of the sIDHistory attribute of all matching objects in NewDomain, then re-migrate the SIDs from OldDomain to NewDomain…

I was a bit shocked by this and responded that you could indeed remove individual SIDs from the sIDHistory. And while I don’t know how long it would have taken to perform the other process (assuming OldDomain still existed!), using targeted cleaning of individual SIDs could be done relatively quickly. I would shoot from the hip and say that original process would take a couple of days to get everything all lined up and completed where a granular cleanup could be completed in minutes or hours at most.

I then asked around to some other friends that I know to be good admins, people that aren’t dev guys but generally a big step up from “My name is Peggy”, and a majority of them thought the same thing, it was an all or nothing thing when clearing sIDHistory with one person, who shall remain nameless, actually responding, “Why would you ever want to remove sIDHistory values????” Moving on…

One of the guys sent me a couple of links to MSFT web pages that had VBScript and PowerShell examples and they both removed all values with no indication that you could, if you so desired, remove only a subset of the values. Even a blog entry I had written back in 2006 (http://blog.joeware.net/2006/09/16/621/) only mentioned clearing all values from the attributes on all objects. I did have a later blog post in 2009 (http://blog.joeware.net/2009/06/10/1655/) that explained how to clean up SIDs for a specific domain, looking it over I have to admit it is a bit terse. Plus if you are googling for “sIDHistory cleanup”, the 2009 post doesn’t come up (at least not in the first three pages[2]).

Now I don’t think there is any great conspiracy here, I just think that MSFT didn’t really visualize stacking migrations and thereby stacking up sIDHistory values and I don’t think Google is out to make life hard for AD Admins. That being said, I have now personally seen cases where a single object has 3,4 or even up to 6 sIDHistory values. This is likely unusual for most companies but it is occurring out there and I would hate to think admins in those positions thought they were stuck with keeping all of those entries because they need just one or two of them and can’t find guidance on how to do it.

Granted, if you know enough about the vbscript or the powershell commands or even the AdFind/AdMod commands I used in the 2006 blog you could divine that removal of a single SID from sIDHistory is possible because they all clear the attribute by removing individual values. But low and behold, looking around, I didn’t find it stated anywhere except in my 2009 post and again, that post isn’t popping as early as it could for people. Additionally some people just aren’t all that willing to experiment or don’t have a place to experiment or perhaps even the time to experiment.

So based on the idea that if several smart people I have spoken to don’t know/understand something or, as in this case, have an incorrect understanding of it, I assume that the chances are good that a good number of others don’t know either. This led me to sit down and write this new blog post which will hopefully score higher when you search for “sIDHistory cleanup”. So, if you are still with me, let’s work through the process of “sIDHistory cleanup” in a non-terse manner …

Q: Can you remove a single SID from the sIDHistory attribute of an object? I.E. Can I perform targeted sIDHistory cleanup?

A: ABSOLUTELY! You can indeed remove a single SID or even some subset of SIDs from sIDHistory on an object and in fact complete a targeted sIDHistory cleanup.

Here is a real life example of targeted sidHistory cleanup with AdMod:

C:\temp>adfind -default -f name=$joe sidhistory

AdFind V01.45.00cpp Joe Richards (joe@joeware.net) March 2011

Using server: TEST-DC1.test.loc:389
Directory: Windows Server 2003
Base DN: DC=test,DC=loc

dn:CN=$joe,OU=Users,OU=My,DC=test,DC=loc
>sIDHistory: S-1-5-21-3641047700-3957557241-2644433309-6569
>sIDHistory: S-1-5-21-3641047700-3957557241-2644433309-6570
>sIDHistory: S-1-5-21-3641047700-3957557241-2644433309-6571
>sIDHistory: S-1-5-21-3641047700-3957557241-2644433309-6572

1 Objects returned

C:\temp>admod -b CN=$joe,OU=Users,OU=My,DC=test,DC=loc SID##sidhistory:-:S-1-5-21-3641047700-3957557241-2644433309-6571

AdMod V01.13.00cpp Joe Richards (joe@joeware.net) April 2010

DN Count: 1
Using server: TEST-DC1.test.loc:389
Directory: Windows Server 2003

Modifying specified objects…
   DN: CN=$joe,OU=Users,OU=My,DC=test,DC=loc…

The command completed successfully

C:\temp>adfind -default -f name=$joe sidhistory

AdFind V01.45.00cpp Joe Richards (joe@joeware.net) March 2011

Using server: TEST-DC1.test.loc:389
Directory: Windows Server 2003
Base DN: DC=test,DC=loc

dn:CN=$joe,OU=Users,OU=My,DC=test,DC=loc
>sIDHistory: S-1-5-21-3641047700-3957557241-2644433309-6569
>sIDHistory: S-1-5-21-3641047700-3957557241-2644433309-6570
>sIDHistory: S-1-5-21-3641047700-3957557241-2644433309-6572

1 Objects returned

The key to a targeted sIDHistory cleanup is to specify the actual SID you want removed in the LDAP command. The tricky part of that though is that the SID needs to be specified as a binary SID structure, not the nice friendly string you see above. AdMod is nice in that it allows you to specify the SID## prefix on an attribute to alert AdMod to convert a string SID to a SID structure on your behalf. PowerShell should have something similar (I really don’t know for sure) and vbscript… well you will have to do some digging around on that one. However, to my knowledge, binary structures were never VBScript’s strong point.

Now I did get a follow-up question, “So you can use a wildcard for the SIDs?” I.E. Specify removing any SIDs in sIDHistory that start with the Domain SID of the Domain that is gone? That certainly would be cool but nope, sorry, no such luck. The wildcarding of the SIDs is not possible. If you wanted to clear all of the SIDs from sIDHistory for a specific domain, you would want to dump all sIDHistory values and then filter the strings for the values. This generally just takes a basic string parsing script to perform that filtering for you. Below I walk through some basic steps involved and show how you can quite easily do this with AdFind and AdMod. It is likely that after I post this blog, some PowerShell person somewhere (http://bsonposh.com/) will post a blog on how to do it with PowerShell.

Anyway… now for the interesting stuff…

First… Query the GC with a base of “” for any objects with any value in sIDHistory. Then once that data comes back filter through it for the specific sIDHistory value. Finally mute any records that don’t have any attributes being output (i.e. any objects that don’t have the specified SID listed in sIDHistory).

adfind -gcb -f sidhistory=* sidhistory -mvfilter sidhistory=S-1-5-21-1173133699-2878410131-473346761 -recmute

This generates output like:

AdFind V01.45.00cpp Joe Richards (joe@joeware.net) March 2011

Using server: dc1.sidtest.loc:3268
Directory: Windows Server 2008 R2

dn:CN=group1,OU=TestGroups,DC=sidtest,DC=loc
>sIDHistory: S-1-5-21-1173133699-2878410131-473346761-7912

dn:CN=group2,OU=TestGroups,DC=sidtest,DC=loc
>sIDHistory: S-1-5-21-1173133699-2878410131-473346761-14644

dn:CN=group3,OU=TestGroups,DC=sidtest,DC=loc
>sIDHistory: S-1-5-21-1173133699-2878410131-473346761-15651

dn:CN=group4,OU=TestGroups,DC=sidtest,DC=loc
>sIDHistory: S-1-5-21-1173133699-2878410131-473346761-15662

dn:CN=group5,OU=TestGroups,DC=sidtest,DC=loc
>sIDHistory: S-1-5-21-1173133699-2878410131-473346761-16443

Now visualize hundreds or thousands or tens of thousands of those… You absolutely can parse that by hand if you want but I am lazy, even for five objects so I would go a step further and have AdFind do more of the work by running the following command which is the same as the last but sticks the info into a delimited format using the colon character as the delimiter, oh yeah, don’t output a header…

adfind -gcb -f sidhistory=* sidhistory -mvfilter sidhistory=S-1-5-21-1173133699-2878410131-473346761 -recmute -csv -csvdelim : -nocsvheader

which produces this output

"CN=group1,OU=TestGroups,DC=sidtest,DC=loc":"S-1-5-21-1173133699-2878410131-473346761-7912"
"CN=group2,OU=TestGroups,DC=sidtest,DC=loc":"S-1-5-21-1173133699-2878410131-473346761-14644"
"CN=group3,OU=TestGroups,DC=sidtest,DC=loc":"S-1-5-21-1173133699-2878410131-473346761-15651"
"CN=group4,OU=TestGroups,DC=sidtest,DC=loc":"S-1-5-21-1173133699-2878410131-473346761-15662"
"CN=group5,OU=TestGroups,DC=sidtest,DC=loc":"S-1-5-21-1173133699-2878410131-473346761-16443"

Next you could push that command through a FOR /F loop and generate and write AdMod commands to a text file for your perusal.

C:\qnd\sidtest>for /f "tokens=1,2 delims=:" %i in (‘adfind -gcb -f "sidhistory=*" sidhistory -mvfilter "sidhistory=S-1-5-21-1173133699-2878410131-473346761" -recmute -csv -csvdelim : -nocsvheader’) do @echo admod -b %i SID##sidhistory:-:%j >> SIDHISTORY_CLEANUP.TXT

C:\qnd\sidtest>type SIDHISTORY_CLEANUP.TXT
admod -b "CN=group1,OU=TestGroups,DC=sidtest,DC=loc" SID##sidhistory:-:"S-1-5-21-1173133699-2878410131-473346761-7912"
admod -b "CN=group2,OU=TestGroups,DC=sidtest,DC=loc" SID##sidhistory:-:"S-1-5-21-1173133699-2878410131-473346761-14644"
admod -b "CN=group3,OU=TestGroups,DC=sidtest,DC=loc" SID##sidhistory:-:"S-1-5-21-1173133699-2878410131-473346761-15651"
admod -b "CN=group4,OU=TestGroups,DC=sidtest,DC=loc" SID##sidhistory:-:"S-1-5-21-1173133699-2878410131-473346761-15662"
admod -b "CN=group5,OU=TestGroups,DC=sidtest,DC=loc" SID##sidhistory:-:"S-1-5-21-1173133699-2878410131-473346761-16443"

You can then validate the commands are what you wanted and expected and then batch them to run in phases or through a pilot or run them all at once, just copy the commands to a .CMD file or rename the .TXT file and you are good to go. However… If you just want the commands to directly run you could remove the @echo after the DO and it will execute the commands for you. But in that case… it is simpler to output the information with -ADCSV format and push straight into AdMod like the example in http://blog.joeware.net/2006/09/16/621/

The search command for this case would be adfind -gcb -f "sidhistory=*" sidhistory -mvfilter "sidhistory=S-1-5-21-1173133699-2878410131-473346761" -recmute -adcsv

Which would generate the following output:

~~~ADCSV~~~~~~ADCSV~~~~~~ADCSV~~~~~~ADCSV~~~~~~ADCSV~~~~~~ADCSV~~~~~~ADCSV~~~~~~ADCSV~~~~~~ADCSV~~~~~~ADCSV~~~
"dn","sidhistory"
"CN=group1,OU=TestGroups,DC=sidtest,DC=loc","S-1-5-21-1173133699-2878410131-473346761-7912"
"CN=group2,OU=TestGroups,DC=sidtest,DC=loc","S-1-5-21-1173133699-2878410131-473346761-14644"
"CN=group3,OU=TestGroups,DC=sidtest,DC=loc","S-1-5-21-1173133699-2878410131-473346761-15651"
"CN=group4,OU=TestGroups,DC=sidtest,DC=loc","S-1-5-21-1173133699-2878410131-473346761-15662"
"CN=group5,OU=TestGroups,DC=sidtest,DC=loc","S-1-5-21-1173133699-2878410131-473346761-16443"

The whole command to cleanup would then be

adfind -gcb -f "sidhistory=*" sidhistory -mvfilter "sidhistory=S-1-5-21-1173133699-2878410131-473346761" -recmute -adcsv | admod sidhistory:-:{{sidhistory}} -unsafe

Not bad for 30 seconds worth of “scripting” eh?

joe

[1] Shiny here refers to the Firefly definition and if you don’t know what Firefly is… You have gotten your nerd/geek card revoked. 😉

[2] I have a three page rule with search engines I tend to follow, if it isn’t on the first three pages, I am not likely to see it because I stop looking.