I have been working on AdFind and AdMod again. I wanted to get a few things fixed that have bothered me and heard some interesting ideas for some new features that I have encountered. The more I go back to it the more I wish I had the time to perform a complete rewrite. It is something I have been thinking about for a long time, but alas, I am still not rich enough to spend that time. 🙁
So anyway, one bug that I was able to get out of AdFind that has been painful for at least one company that I expect very few people have encountered is around largish groups. I don’t mean a group with 10,000 users in it, AdFind won’t even sneeze at that. I mean large groups, groups with hundreds of thousands of users in it. The issue is that when AdFind is outputting CSV and you have a very large group, it can appear to go out to lunch and in fact, actually may never return from that lunch. Now this isn’t something that most companies deal with because most companies don’t have hundreds of thousands of users to even put into a group, so it wasn’t something I thought about when working on the CSV functionality initially. I have only had one documented complaint that this was the cause of an issue and one case where I thought it might have been related but it turns out it wasn’t (at least we don’t think so to this point) in I don’t know how many years so I feel pretty justified in my decision at the time I put the CSV formatting in to do it like I did it.
Regardless of the number of people possibly affected, I might someday be affected by it because I work with a lot of organizations that could easily have that many members and quite frankly, from the standpoint of elegance, the issue pissed me off and I wanted to fix it. 🙂 So I did. I haven’t figured out the scale to which it will work now but so far it has worked with groups exceeding 725,000 members. I am currently adding another 300,000 objects to my test domain to add them to the test group so I will officially test it to just over 1,000,000 members. Anything above that and my friend Dean says my response to people should be, for a small fee you can have me build you an Enterprise version that works for that scale. Of course that small fee will be enough to pay for me to quit my job and start having nice Hawaiian vacations for the next few years because I write much better code in Hawaii. I just decided that. 🙂
So the next version of AdFind to come out in the next month’ish will have that fix. It will also have some other fixes including a fix for when you cut and paste a command from Outlook or Word and AdFind/AdMod have no clue what you sent in for the command because dashes and quotes get converted to a different but similar character in Outlook and Word. There is also at least one cool new feature though depending on how confident I am in that feature I may call it *beta* which means it could change in the next version. I am not ready to talk about the cool new feature yet but I did want to let the folks know that may have hit the CSV output blowing up once you get into the hundred thousand members range that relief is coming.
So I have a little time right now to chat so I will answer the question I was asked a couple of times as I have described this bug to several of my friends and that question is… well why does it break like that? My sarcastic answer is because resources are limited and you write code by the 80/20 rule…. But the real issue is because AdFind was never intended to output CSV when I first wrote it. The CSV functionality is out and out a hack I put into it. :) Better?
Basically, and this is a dirty state secret and I will absolutely deny it if stated outside of our little group here, but in order to implement the CSV functionality I changed AdFind from writing its output straight to the console to writing it to a variable. And that variable is basically, well it actually is just a giant string. Maybe you see where I am going with this… It was the quickest fastest way to get it in there and at the time, the only way I could conceive to get it in there without rewriting the whole app because of how it was structured. Now once the data is in that string variable, I could manipulate it… Including restructuring the output into CSV formatted output. So once all of the object is loaded from AD it either shoots that string to the console or it sends it into a CSV rework routine which waves its hand over the string and does some string manipulation and then outputs the CSV. The problem here is when you have a lot of data in a single attribute say like hundreds of thousands of DNs, the cost in memory alone to duplicate the string so you can safely manipulate it is high, not to mention the cost on CPU to race through the string changing something like
>attribute: value1
>attribute: value2
>attribute: value3
>attribute: value4
>attribute: value5
>attribute: value6
>attribute: value7
>attribute: valueN
to
“value1;value2;value3;value4;value5;value6;value7;valueN”
is pretty darn high as well.
In the debugging that I have done, while it looks like AdFind is completely out to lunch, it is actually trying to process the string. Whether or not that would ever finish likely depends on your machine and how long you have available to wait though I am not guaranteeing it won’t just eventually blow up. The positive side of this, again, is that 99.99% of the folks out there never would have hit this and if you did, you very likely knew it because AdFind would just be sitting there blinking the cursor at you.
The fix was about a 5 on the complexity scale. Mostly because I have been away from the code base for so long. Had I been working on it like when I used to have more time to work on it, it likely would have been more along the lines of a 3. The basic fix was to tie into when attribute ranging is in progress which was already a special case anyway. Attribute ranging occurs, by default, when an attribute has more than 1500 values (1000 in Windows 2000 IIRC). It is a way for AD to return the data in a more efficient manner versus one gigantic single blob. You get the first 1500 values, then you have to ask for the next bunch, etc etc etc. Since the issue has only presented itself for very large groups, this seems like it will be the best way to handle it outside of a rewrite of the core underlying engine itself. It greatly reduced the surface area of what could be broken now based on the changes versus a rewrite.
That’s all for now. 
joe
