joeware - never stop exploring... :)

I recently had a conversation with an Active Directory engineer who was working on some token bloat issues. As AD has pushed past the decade mark I seem to be seeing/hearing more and more of this bloat problem as many companies continue consolidating groups into and creating new groups in AD without cleanup or normalization.

To be more clear, say a company took whole hog the groups they had before, likely in a couple, but possibly up to hundreds of NT4 domains and collapsed it all into a single forest or they took all of the machine local groups on member servers and sucked them up into AD for “easier management and reporting”. On top of that, all of it has remained with no cleanup and companies moved forward adding more permission and DL groups and merging/consolidating more and more domains (or companies), etc. And then on top of THAT, you get companies who used sIDHistory but never went back and cleaned up what the sIDHistory was initially needed for. What should take place is group/permission normalization, ACL cleanup, etc.

Err ok, I just sidetracked myself with a soapbox speech… back to the AD Engineer patiently waiting for me to shut up and listen to what he wants to accomplish…

So part of the problem this engineer is experiencing is that the Active Directory he is concerned about had been through a double hop AD Consolidation after multiple mergers/acquisitions. I.E. OldOldDomain was migrated/collapsed into OldDomain and later OldDomain was migrated/collapsed into ShinyNewDomain[1]. The migrations involved pulling the objects over whole hog and populating sIDHistory in the new accounts. So what you ended up with was (SIDs shortened for readability):

OldOldDomain – Domain SID S-1-5-12345
OldDomain – Domain SID S-1-5-54321
ShinyNewDomain – Domain SID S-1-5-24680

So a migrated object in ShinyNewDomain has objectSid of S-1-5-24680-2500, and has sIDHistory values of S-1-5-54321-78 and S-1-5-12345-19.

Now let’s say some cleanup had actually occurred and any ACLs with references to SIDs from OldOldDomain were believed to be all gone but there definitely were resources using OldDomain SIDs still. Simple right? Target and remove the OldOldDomain SIDs and leave the OldDomain SIDs in place.

Well unfortunately, the engineer had spoken to another engineer about it and was told it wasn’t possible to perform granular sIDHistory cleanup. He was further told that the process he would have to use to perform this cleanup was to go back to OldDomain, clear the sIDHistory attribute of all objects there, then clear all of the sIDHistory attribute of all matching objects in NewDomain, then re-migrate the SIDs from OldDomain to NewDomain…

I was a bit shocked by this and responded that you could indeed remove individual SIDs from the sIDHistory. And while I don’t know how long it would have taken to perform the other process (assuming OldDomain still existed!), using targeted cleaning of individual SIDs could be done relatively quickly. I would shoot from the hip and say that original process would take a couple of days to get everything all lined up and completed where a granular cleanup could be completed in minutes or hours at most.

I then asked around to some other friends that I know to be good admins, people that aren’t dev guys but generally a big step up from “My name is Peggy”, and a majority of them thought the same thing, it was an all or nothing thing when clearing sIDHistory with one person, who shall remain nameless, actually responding, “Why would you ever want to remove sIDHistory values????” Moving on…

One of the guys sent me a couple of links to MSFT web pages that had VBScript and PowerShell examples and they both removed all values with no indication that you could, if you so desired, remove only a subset of the values. Even a blog entry I had written back in 2006 (http://blog.joeware.net/2006/09/16/621/) only mentioned clearing all values from the attributes on all objects. I did have a later blog post in 2009 (http://blog.joeware.net/2009/06/10/1655/) that explained how to clean up SIDs for a specific domain, looking it over I have to admit it is a bit terse. Plus if you are googling for “sIDHistory cleanup”, the 2009 post doesn’t come up (at least not in the first three pages[2]).

Now I don’t think there is any great conspiracy here, I just think that MSFT didn’t really visualize stacking migrations and thereby stacking up sIDHistory values and I don’t think Google is out to make life hard for AD Admins. That being said, I have now personally seen cases where a single object has 3,4 or even up to 6 sIDHistory values. This is likely unusual for most companies but it is occurring out there and I would hate to think admins in those positions thought they were stuck with keeping all of those entries because they need just one or two of them and can’t find guidance on how to do it.

Granted, if you know enough about the vbscript or the powershell commands or even the AdFind/AdMod commands I used in the 2006 blog you could divine that removal of a single SID from sIDHistory is possible because they all clear the attribute by removing individual values. But low and behold, looking around, I didn’t find it stated anywhere except in my 2009 post and again, that post isn’t popping as early as it could for people. Additionally some people just aren’t all that willing to experiment or don’t have a place to experiment or perhaps even the time to experiment.

So based on the idea that if several smart people I have spoken to don’t know/understand something or, as in this case, have an incorrect understanding of it, I assume that the chances are good that a good number of others don’t know either. This led me to sit down and write this new blog post which will hopefully score higher when you search for “sIDHistory cleanup”. So, if you are still with me, let’s work through the process of “sIDHistory cleanup” in a non-terse manner …

Q: Can you remove a single SID from the sIDHistory attribute of an object? I.E. Can I perform targeted sIDHistory cleanup?

A: ABSOLUTELY! You can indeed remove a single SID or even some subset of SIDs from sIDHistory on an object and in fact complete a targeted sIDHistory cleanup.

Here is a real life example of targeted sidHistory cleanup with AdMod:

C:\temp>adfind -default -f name=$joe sidhistory

AdFind V01.45.00cpp Joe Richards (joe@joeware.net) March 2011

Using server: TEST-DC1.test.loc:389
Directory: Windows Server 2003
Base DN: DC=test,DC=loc

dn:CN=$joe,OU=Users,OU=My,DC=test,DC=loc
>sIDHistory: S-1-5-21-3641047700-3957557241-2644433309-6569
>sIDHistory: S-1-5-21-3641047700-3957557241-2644433309-6570
>sIDHistory: S-1-5-21-3641047700-3957557241-2644433309-6571
>sIDHistory: S-1-5-21-3641047700-3957557241-2644433309-6572

1 Objects returned

C:\temp>admod -b CN=$joe,OU=Users,OU=My,DC=test,DC=loc SID##sidhistory:-:S-1-5-21-3641047700-3957557241-2644433309-6571

AdMod V01.13.00cpp Joe Richards (joe@joeware.net) April 2010

DN Count: 1
Using server: TEST-DC1.test.loc:389
Directory: Windows Server 2003

Modifying specified objects…
   DN: CN=$joe,OU=Users,OU=My,DC=test,DC=loc…

The command completed successfully

C:\temp>adfind -default -f name=$joe sidhistory

AdFind V01.45.00cpp Joe Richards (joe@joeware.net) March 2011

Using server: TEST-DC1.test.loc:389
Directory: Windows Server 2003
Base DN: DC=test,DC=loc

dn:CN=$joe,OU=Users,OU=My,DC=test,DC=loc
>sIDHistory: S-1-5-21-3641047700-3957557241-2644433309-6569
>sIDHistory: S-1-5-21-3641047700-3957557241-2644433309-6570
>sIDHistory: S-1-5-21-3641047700-3957557241-2644433309-6572

1 Objects returned

The key to a targeted sIDHistory cleanup is to specify the actual SID you want removed in the LDAP command. The tricky part of that though is that the SID needs to be specified as a binary SID structure, not the nice friendly string you see above. AdMod is nice in that it allows you to specify the SID## prefix on an attribute to alert AdMod to convert a string SID to a SID structure on your behalf. PowerShell should have something similar (I really don’t know for sure) and vbscript… well you will have to do some digging around on that one. However, to my knowledge, binary structures were never VBScript’s strong point.

Now I did get a follow-up question, “So you can use a wildcard for the SIDs?” I.E. Specify removing any SIDs in sIDHistory that start with the Domain SID of the Domain that is gone? That certainly would be cool but nope, sorry, no such luck. The wildcarding of the SIDs is not possible. If you wanted to clear all of the SIDs from sIDHistory for a specific domain, you would want to dump all sIDHistory values and then filter the strings for the values. This generally just takes a basic string parsing script to perform that filtering for you. Below I walk through some basic steps involved and show how you can quite easily do this with AdFind and AdMod. It is likely that after I post this blog, some PowerShell person somewhere (http://bsonposh.com/) will post a blog on how to do it with PowerShell.

Anyway… now for the interesting stuff…

First… Query the GC with a base of “” for any objects with any value in sIDHistory. Then once that data comes back filter through it for the specific sIDHistory value. Finally mute any records that don’t have any attributes being output (i.e. any objects that don’t have the specified SID listed in sIDHistory).

adfind -gcb -f sidhistory=* sidhistory -mvfilter sidhistory=S-1-5-21-1173133699-2878410131-473346761 -recmute

This generates output like:

AdFind V01.45.00cpp Joe Richards (joe@joeware.net) March 2011

Using server: dc1.sidtest.loc:3268
Directory: Windows Server 2008 R2

dn:CN=group1,OU=TestGroups,DC=sidtest,DC=loc
>sIDHistory: S-1-5-21-1173133699-2878410131-473346761-7912

dn:CN=group2,OU=TestGroups,DC=sidtest,DC=loc
>sIDHistory: S-1-5-21-1173133699-2878410131-473346761-14644

dn:CN=group3,OU=TestGroups,DC=sidtest,DC=loc
>sIDHistory: S-1-5-21-1173133699-2878410131-473346761-15651

dn:CN=group4,OU=TestGroups,DC=sidtest,DC=loc
>sIDHistory: S-1-5-21-1173133699-2878410131-473346761-15662

dn:CN=group5,OU=TestGroups,DC=sidtest,DC=loc
>sIDHistory: S-1-5-21-1173133699-2878410131-473346761-16443

Now visualize hundreds or thousands or tens of thousands of those… You absolutely can parse that by hand if you want but I am lazy, even for five objects so I would go a step further and have AdFind do more of the work by running the following command which is the same as the last but sticks the info into a delimited format using the colon character as the delimiter, oh yeah, don’t output a header…

adfind -gcb -f sidhistory=* sidhistory -mvfilter sidhistory=S-1-5-21-1173133699-2878410131-473346761 -recmute -csv -csvdelim : -nocsvheader

which produces this output

"CN=group1,OU=TestGroups,DC=sidtest,DC=loc":"S-1-5-21-1173133699-2878410131-473346761-7912"
"CN=group2,OU=TestGroups,DC=sidtest,DC=loc":"S-1-5-21-1173133699-2878410131-473346761-14644"
"CN=group3,OU=TestGroups,DC=sidtest,DC=loc":"S-1-5-21-1173133699-2878410131-473346761-15651"
"CN=group4,OU=TestGroups,DC=sidtest,DC=loc":"S-1-5-21-1173133699-2878410131-473346761-15662"
"CN=group5,OU=TestGroups,DC=sidtest,DC=loc":"S-1-5-21-1173133699-2878410131-473346761-16443"

Next you could push that command through a FOR /F loop and generate and write AdMod commands to a text file for your perusal.

C:\qnd\sidtest>for /f "tokens=1,2 delims=:" %i in (‘adfind -gcb -f "sidhistory=*" sidhistory -mvfilter "sidhistory=S-1-5-21-1173133699-2878410131-473346761" -recmute -csv -csvdelim : -nocsvheader’) do @echo admod -b %i SID##sidhistory:-:%j >> SIDHISTORY_CLEANUP.TXT

C:\qnd\sidtest>type SIDHISTORY_CLEANUP.TXT
admod -b "CN=group1,OU=TestGroups,DC=sidtest,DC=loc" SID##sidhistory:-:"S-1-5-21-1173133699-2878410131-473346761-7912"
admod -b "CN=group2,OU=TestGroups,DC=sidtest,DC=loc" SID##sidhistory:-:"S-1-5-21-1173133699-2878410131-473346761-14644"
admod -b "CN=group3,OU=TestGroups,DC=sidtest,DC=loc" SID##sidhistory:-:"S-1-5-21-1173133699-2878410131-473346761-15651"
admod -b "CN=group4,OU=TestGroups,DC=sidtest,DC=loc" SID##sidhistory:-:"S-1-5-21-1173133699-2878410131-473346761-15662"
admod -b "CN=group5,OU=TestGroups,DC=sidtest,DC=loc" SID##sidhistory:-:"S-1-5-21-1173133699-2878410131-473346761-16443"

You can then validate the commands are what you wanted and expected and then batch them to run in phases or through a pilot or run them all at once, just copy the commands to a .CMD file or rename the .TXT file and you are good to go. However… If you just want the commands to directly run you could remove the @echo after the DO and it will execute the commands for you. But in that case… it is simpler to output the information with -ADCSV format and push straight into AdMod like the example in http://blog.joeware.net/2006/09/16/621/

The search command for this case would be adfind -gcb -f "sidhistory=*" sidhistory -mvfilter "sidhistory=S-1-5-21-1173133699-2878410131-473346761" -recmute -adcsv

Which would generate the following output:

~~~ADCSV~~~~~~ADCSV~~~~~~ADCSV~~~~~~ADCSV~~~~~~ADCSV~~~~~~ADCSV~~~~~~ADCSV~~~~~~ADCSV~~~~~~ADCSV~~~~~~ADCSV~~~
"dn","sidhistory"
"CN=group1,OU=TestGroups,DC=sidtest,DC=loc","S-1-5-21-1173133699-2878410131-473346761-7912"
"CN=group2,OU=TestGroups,DC=sidtest,DC=loc","S-1-5-21-1173133699-2878410131-473346761-14644"
"CN=group3,OU=TestGroups,DC=sidtest,DC=loc","S-1-5-21-1173133699-2878410131-473346761-15651"
"CN=group4,OU=TestGroups,DC=sidtest,DC=loc","S-1-5-21-1173133699-2878410131-473346761-15662"
"CN=group5,OU=TestGroups,DC=sidtest,DC=loc","S-1-5-21-1173133699-2878410131-473346761-16443"

The whole command to cleanup would then be

adfind -gcb -f "sidhistory=*" sidhistory -mvfilter "sidhistory=S-1-5-21-1173133699-2878410131-473346761" -recmute -adcsv | admod sidhistory:-:{{sidhistory}} -unsafe

Not bad for 30 seconds worth of “scripting” eh?

joe

[1] Shiny here refers to the Firefly definition and if you don’t know what Firefly is… You have gotten your nerd/geek card revoked. 😉

[2] I have a three page rule with search engines I tend to follow, if it isn’t on the first three pages, I am not likely to see it because I stop looking.

As some of you may have noticed over the last few weeks as I found time I was posting test posts to the blog after several of the most recent "real"[1] posts got FUBAR’ed after posting and I wasn’t bright enough to go back and look at the post right away to validate that it had indeed posted properly. This resulted in completely confusing posts that required people to email me to say, hey dude, your blog is all dorked up, you might want to check into that.

Anyway I worked out the problem… finally. Certainly it wasn’t as easy as I thought it should be. Well the fix itself was simple, finding it was another matter entirely. I just didn’t seem to hit on the right combination of search words that didn’t result in post where someone was describing the issue and some bright person was telling them they didn’t know what they were doing.

The issue I was seeing was that HTML tags in Windows Live Writer (WLW) were getting half way sort of stripped. Specifically the opening and closing, ummm brackets? braces? The less than symbol (<) and the greater than symbol (>). The most common issue was the paragraph tag was getting changed from <p> to p and the end paragraph was changing from </p> to /p. Obviously that plays a bit of havoc with the formatting when a browser is trying to render the page.

I wanted to post this so hopefully all of the other people I found out asking the same questions I was asking could get the solution, which is remarkably easy.

So without further adieu… The issue with the partially stripped HTML tags is a bug in libxml2. Specifically using anything less than PHP 5.2.9+ with libxml2 2.7.3+ is susceptible.  The fix is to install a plugin into WordPress called LibXML2 Fix which you can get from your plugins menu or look here -> http://wordpress.org/extend/plugins/libxml2-fix/. 

 

For more detailed info see http://core.trac.wordpress.org/ticket/7771

 

Oh… How do you know what version of PHP and LibXML2 you are running you ask? Well you can ask your admins or you can look for an info page on your provider’s web site. But, as I found with my provider, they neglected to update their page so I found you can add the following PHP script to your website and just run it by calling it from a browser and it will tell you all about your PHP Installation.

<html>
<head>
<title> PHP Info</title>
</head>
<body>
<?php
phpinfo( );
?>
</body>
</html>

That will give you something that looks like:

which will go on and on with info for you. The main things you need to look at are the version of PHP

and the version of libxml2.

 

As soon as I applied the libxml2 fix plugin I was able to upload posts from Windows Live Writer with no issue.

I hope this helps someone else. 🙂

 

    joe

 

[1] The use of the word real does not in any way shape or form imply quality, suitability, accuracy, or safety to use in any environment whether test, production or otherwise.

This is kind of interesting if you are into that sort of thing.

http://fabiensanglard.net/quake2/index.php

So I had to write another C# program recently for work and my thoughts on C# is that it is growing on me, it reminds me very much of PERL actually. However I wish C# could be compiled directly to native machine code. If it could be compiled directly to native machine code, I would be very close to jumping to it from C++. Perhaps though I am only saying this as I am in the process of switching from C++ Builder to Visual Studio and I haven’t yet ported all of my libraries and utility code which do a lot for me that is done in C#, especially around string handling etc.

Anyway, what did I write? Unfortunately not something I can share on my website as it was for work. It is a command line tool for performing some list functions in MOSS 2007. Specifically it will enumerate the sites on a MOSS Server, then it will enumerate the#160; lists for a given site, then it will list the views of a given list and finally it will retrieve the data for a given list/view to a TAB delimited file.

I used to get that data from a list/view via the XML FEED format of the list web page, however I found a case where MOSS breaks and won’t output *any* information at all based on a certain field type and if you add that field to a view, bam you go from getting data to getting nothing. I had a PERL script that would pull that page and parse the XML and generate the TAB file in less than a minute. The new C# utility takes several minutes. Mostly I think that time is sucked up in the MOSS 2007 Web Services that have to be used to get the data. Thankfully my use of the tool is just to run it once a day to check for updates.

joe