…and boy did I find a heck of a memory leak in AdMod…
I admit, this is probably on the unusual side for tasks people would normally be executing but just the same I am glad that I found it so I can stamp it out. So much for CodeGuard because it never once warned me about that in the testing I was doing before I had to disable it.
Oh so WHY were you resetting 300,000 accounts you ask??? Well it wasn’t quite 300,000, it was just under that but it was just to test DIT expansion for a larger directory. Nothing exciting like a security breach or anything.
My memory leak was from three main factors.
- I have a smart pointer class that keeps track of pointers and releases memory when not needed which is either when I specifically tell the class to release the memory e.g. Pointers.Release() or when the class goes out of scope. Generally I have two instances of the class, a global and a local instance. The global instance handles all memory I allocate that can be given back before the program exits (which it would do anyway but I like to be complete). The local instance is used in functions/routines/code segments where there is a lot of memory allocation either in sheer size that isn’t needed long term or I am in some sort of loop allocating more and more memory and it isn’t needed outside of that. Well some of the pointers which should have been thrown on the local instance were in the global instance so it wasn’t released until almost program termination. This was strictly due to my stupidity when I did some cut and paste from the non-CSV versions of the ADD/MODIFY/RENAME functions to the CSV versions. The looping setup is different on the CSV versions so previously where the memory allocation fell outside of the tight loop it wasn’t an issue, in the CSV versions that portion of the code got pulled into the loop and hence “leaked”.
- I did something stupid with the asynch versions of LDAP calls where I checked the results but didn’t give back the memory allocated by LDAP. This has been in their ever since I add the ability to output extended errors but never saw it cause pain but then usually I wasn’t doing mass updates to hundreds of thousands of objects over and over again. To give you an idea of how much testing I have been doing, I have several million objects in my deleted objects container. I knocked my tombstoneLifetime down to 1 day to clean it up. That was handy however for testing the undelete processes in admod. 😉
- Either my older compiler (Borland Builder 6 Pro) or the older STL implementation (Rogue Wave) has an issue with memory leaking on strings. I didn’t dig into it too deeply other than to come up with a workaround. I am only using the older compiler because I haven’t “ported” admod over to the new compiler yet. I won’t do that until I make a clean break from US ASCII to hopefully full UNICODE support so it will be an involved port. Basically the workaround involved creating a char[] (an array of characters) to pass to the LDAP API calls instead of using the c_str() function of the string class. There was also something goofy with the string class when returning a string from one (and only that one) function. Those are the kind of funky things you get to deal with when you depend on someone else’s implementations of classes to do things. I try to do that as little as possible which is YAR I avoid ADSI and NET and other APIs that do it all for you.
So what kind of differences are we seeing now? Was it worth spending several days of cleaning up these memory leaks? Well from my standpoint, any memory leak is worth me spending time cleaning it up. Memory leaks bother the hell out of me, it is sloppy so of course it was worth it to me regardless, but going back to the original example, I reset the passwords of the 300k accounts with the old and new versions and the differences are pretty substantial. So if you ever find yourself resetting 300,000 passwords with AdMod you can thank me. Both for making AdMod in the first place and for cleaning up after myself and making it less memory intensive due to stupid leaks. 😉
300,000 Password Resets with Beta 1, Private Bytes in Use at end of code execution: 545870912 – yes, just over 512MB.
300,000 Password Resets with Beta 2, Private Bytes in Use at end of code execution: 133619712 – about 127MB.
So we are talking about 385 less memory being used during the execution of the app. Depending on the system this could result in a speed increase as well as you will have less or possibly no paging of data out to disk as you exceed physical RAM capacity amongst the various things you are running.
Note that of both of the Private Bytes usage, AdMod only needs about 11MB or so. The rest of that memory is the DN’s sitting in memory that were piped over from AdFind and in Beta 1 the leaks…
joe
