I was also surprised there were not more responses overall as this is a very important topic.

Just wanted to say that this tool gives far better results then any other solution I have found so far, with 100% success rate(so far)! I thought I would add that this seems to give results where the wizard to rejoin the domain works but only for 24 hours or so. I wasn’t able to get the powershell cmdlet Test-ComputerSecureChannel [-reset] to get results either so this is a god send! Perhaps more importantly it has highlighted that there is a time discrepancy which gives me something to look at for a more preventative solution.

Thanks!

Andy

On the cross domain functions issues stuff, that is interesting. I was just working with my co-author on an issue he was hitting in a customer’s virtual AD lab recently and cross domain trust and replication issues was one of the symptoms. We were digging through source code and trying to figure out various hacks and in the end, IIRC, he declared that lab a complete loss and told them rebuild it. For some odd and interesting reason the customer seemed to think the lab environment worked fine…

I am unsure on the next AdFind update. I have been seriously busy with my real job and life so hard to dedicate enough time for dev work. 10 minutes here or there is not enough to do it. So no, it also isn’t converted to Visual C++ yet.

Puddle… LOL. Hahahaha. That cracked me up, that should be the name, it is funnier than the real name. The utility is called Ripple. It basically emulated tossing a pebble into the AD “Puddle” and then watching for the ripples to hit the edges. I actually first started working on it back in about 2000 or so but never got it to scale well once the number of DCs hit several hundred since it spawned a single thread for every DC. I need to rework the whole threading model in it. I should get back to it as it was pretty cool. Would tell you live what was going on with replication.

ActiveDir.org… I do miss that list, my main PC that I used to work on my personal joeware email on including the List work has been down. I need to completely rebuild it. I could go through the email in the gmail web interface (joeware is hosted out of google apps now) but it isn’t conducive to lists really. It is hard enough just to respond to regular email on it.

I am glad the tools are useful to you.

joe

Ironically the environments that truly seem to understand what needs to be done to properly support it in a minimal risk manner and have the capability to do so are often very unlikely to do so. Some very large serious tech companies with scary smart tech resources are willing to assist customers to virtualize but won’t virtualize their own internal corp forest. That though is simply reflecting the difference between the goals of IT and Sales.

I think “enterprise sized companies” start around multiple tens of thousands of users with global sites. The smaller environments would be small to mid-sized and are likely more consolidated and aren’t measuring their site outages in hundreds of thousands of dollars per hour and corporate outages in the millions or tens of millions per hour. A company like AT&T or Toyota Motors for example would be in really bad shape if they lost corporate authentication for a day or worse days. But a company like Tesla Motors, perhaps not so much. I imagine that if a serious issue occurred with authentication at AT&T, the CIO wouldn’t be in trouble, he would be put in front of a firing squad so the consideration of risk isn’t just to the company assets but personally as well.

We aren’t anywhere near AT&T big but we run our production single domain forest environment with hardware. We have thought about virtual but have refrained for a lot of the reasons listed plus the concern that despite how well you follow process you still have enhanced risk and less cushion for a mistakes and there are always mistakes; no one is perfect – if that were the case you wouldn’t need change control. So even with a little risk, the results of an issue could be bad. As my manager says, his brother is a police officer and he isn’t likely to be shot in the course of his job even though he is out in a cruiser arresting people nearly every working day but he wears a bullet proof vest – not because he expects to get shot, but because he wants to lessen the danger in case it does happen.

We have lost two different lab environments that were all virtual. Specifically we hit really weird issues with cross domain functions and Exchange acting hokey and no one could figure out why and said it wasn’t worth spending weeks trying. It really sucked because we were put behind on projects but they were labs and not production line critical. We weren’t ever sure what we did wrong as we thought we followed the rules properly. We now have three separate single domain forest virtual labs which more closely matches our production forest and things have worked out well so far but if we lose one, it will hopefully be one and not the whole environment. However now we are adding several physical machines to replicate production hardware as we recently performed a firmware/driver update on some DCs and something went hokey and they got really slow. We rolled back and everything is good again but it took us a while to figure it out which angered the application people. Now we are required to test all firmware/driver updates in the saftey of the lab and to the satisfaction of the line of business teams. It will slow things down but we did break them so we have to deal with the consequences of that.

I wanted to say AdFind rocks. I use it every day. When is the next update coming out? Did you convert it to Visual Studio yet like you blogged about? I know you mentioned that should reduce the size of the tool. Also you probably don’t remember me, but I talked to you at TEC after the great Joe and Dean Show presentation and you mentioned you had wrote a tool that was multithreaded and could give “up to the second” replication times to all of the DCs in a domain or forest, I think you called it Puddle? Did you ever release that under a different name or something, I have looked a couple of times but haven’t found it yet.

Thanks for everything you share. We appreciate it in the trenches. Also come back to ActiveDir.org, you don’t ever seem to answer questions there anymore.

I need to re-phrase this a bit. You are working for the truly large enterprises of this world, and the advise and opinions that you have reflect this. What I was thinking is that 90% of your readers manage smaller environments than this. Where I live, our customer base has maybe 10 customers at 50.000 seats and over. My team of engineers typically encounter customers between 5000 and 20.00 seats. That accounts for the majority of the work.

For these customers, the impact of a global AD disaster is less than for a 150.000 seat giant — exceptions noted. Many of them adapt the “virtualize everything” attitude.

> Putting one DC on physical isn’t saving anything unless you have all of your other DCs on a single SAN and then someone just plain needs to be slapped.

That is exactly what I see happening. Worse, they think they are safe when they have twin datacenters and have the SAN replicate everything between them.

What we have seen from the VM-related AD disasters that we have encountered is that having one physical DC would have allowed them to get back up to speed much quicker. DNS works, you can logon on, the ESX layer starts up and you can actually manage it (vCenter), you have a working source for dcpromo, etc. Is it a 100% solution? No, of course not, as you convincingly argued. In the end, you need a proven backup as last resort. And as you say, if all your AD is physical you are simply less vulnerable. No argument.

So yes, I see your point. I really do. I’m just wondering if it is the most realistic advice for most companies.

Before anything else. How many issues have you experienced directly or indirectly with virtual DCs? I have found that folks who haven’t had issues tend to fall on the side of “not much to worry about” where the folks who have encountered issues are a bit more tentative and want to understand environments and support limitations before recommending direction. Also the folks who have been burned by the “rarely happens” side of the coin are also on the tentative side because they have learned that rarely happens or very unlikely is not the same as truly impossible and then once they make that realization they have to weigh out the worst impact and how they would handle it and how the company would handle it. This is standard IT stuff though redundancy is waste and it is always a balancing act on how much waste you are willing to accept to mitigate risk or on the flip side how much risk you are willing to accept to reduce waste.

5000 seats is what I would consider mid-sized. My personal admin experience is in environments of 75k up through a couple of hundred thousand. My technical lead / escalation / SWAT experience is wider but usually about 10k seats is on the smaller end of things that I have ended up working with. Mostly visualize environments that have follow the sun type operations with people sitting all over the world at different parts of the 24 hour day handling work. North America, South America, KL, China, India, Europe, etc. Think of any country in the world that has supplied a low-cost support center and I have probably worked at some point with a company that has worked with that center. Aside from that though a lot of it comes down to costs and penalties. You don’t have to be huge in order to have an environment where damage to AD or some other critical infrastructure will be measured in hundreds of thousands or perhaps millions of dollars per hour. I’ve seen them. I have also seen companies, large companies in the 150k user range, that could at one point fairly recently have lost their AD and been fine for a week or more.

Anyway, it is currently looking like the smaller implementations are doing heavier amounts of virtualization and again, I kind of expected that.

The larger the org, the larger the range in number, size, and quality of the support groups involved which possibly for anyone who isn’t used to large orgs (100k+) may seem counter-intuitive. Some would, and some often do, think that the larger the org, the faster and more flexible because of more money being available or whatever. It truly doesn’t work out that way in most large orgs I have seen. Some of them are almost paralyzed by process (and change control) that is designed to prevent massive screw-ups from occurring too often because of resources who likely shouldn’t even be doing the work they are doing and also cost savings concerns because you need to squeeze every penny that comes along to make stock holders happy. For them to try and implement new tech or a new processes can be excessively costly in time and human resources if they even have enough solid resources to accomplish the tasks in the first place.

It is tougher and tougher now a days, the IT Talent pool is much more shallow than it was even 10 years ago due to the various financial collapses. Not many are willing to stay in a field where salaries have been cut 30-70% and layoff is a regular real concern after one rough quarter or market downturn.

I am curious about the “when I encounter that configuration I always get them to put the PDCe on a class-A physical machine with local storage.”… Why? I am not trying to be leading or facetious but if you trust the virtual environment enough to use for every other DC in the environment except FSMO roles, why wouldn’t you do them all? Corruption can replicate, I have seen it first hand. Putting one DC on physical isn’t saving anything unless you have all of your other DCs on a single SAN and then someone just plain needs to be slapped.

The new guidance from MSFT says that you don’t have to keep anything physical, it is ok for them all to be virtual. And that isn’t just for 2012, as mentioned previously, there is no real change in 2012 safety-wise other than to lock off one piece that could hurt you that has already been broadcast for a long time that it could hurt you. The true major enhancement is the cloning piece. So effectively any safety guidance for 2012 is valid for 2003 SP1 other than the “if you accidently click snapshot rollback you shouldn’t be screwed”. But again, that has been a known no no for a long time.

On your last paragraph I think we agree more than disagree. I think perhaps I may have had opportunity to see more issues and more environments with processes and personnel that make me think that companies need to slow down and really work through it before jumping in head first. And again, you would think most companies would do that but when you get a bunch of IT Managers and CIO’s who are a long ways away from the tech and have some consultants buzzing in their ears or going off to fun conferences, it can be difficult to try and explain things out and what is really involved because the folks they first spoke to have no understanding at all of the environment they are talking about.

Overall I am absolutely fine with virtualizing DCs, it just needs to be done with full knowledge and support from all necessary involved groups and in environments that live or die based on documentation and standard process all of that needs to be A+ quality because those environments are often trying to live on process and documentation because they know they don’t have “I.T. S.W.A.T.”-like capable support staff across the board – can’t afford to. Again, smaller support orgs and consulting firms are more likely to have a relatively high level of capability consistently across the board. It much easier to maintain quality if you have 50 support guys versus a couple of hundred or couple of thousand or more. The larger you get, the more you have to depend on process and tools and if you can’t depend on those near 100% of the time something could slip through and you have to understand what the implications of that are and whether or not you are willing to accept the impact. You want to know whether or not a given company can safely virtualize DCs, go and talk to the AD tech lead that is directly working with the guys running the systems; that is the main person I would trust assuming they were a solid technical resource. Next thing is to look at the sev-1 issue log for the last couple of years and see what kind of issues have occurred and how long it took to fix them and the quality of the root cause analysis docs. What? The company doesn’t track the sev-1 issues? That is a strike against doing anything complex right from the word go.

joe

Yes, I agree that you need to know what you are doing when handling virtual DC’s. I also agree with you that vm-gen ID is just a partial solution. Still… it seems to me that you are overreacting. Just my 2 cts.

First and foremost, thank you for your great website, tools, and blogs that have helped my MS career over the many years. I work mostly in DoD environments and all are looking to run, if not already running their own private virtual cloud (VMware or Hyper-V). Domain controllers are a big part of this virtualization. The general rule I have seen is at least one physical DC per domain as a recovery from a virtual network\storage fail but, other than that, to utilize the virtual environment as much as possible to save on hardware costs and quickly provision new DCs (if needed)

Mark

I am not aware of any DC compare tools from MSFT and haven’t thought about it myself really. I guess you could use my GCChk tool (http://www.joeware.net/freetools/tools/gcchk/index.htm) to give it a shot. I should look closer at that as I think it is going to become more important. It was initially written to try and help find lingering objects a little faster and in situations where the MSFT methods didn’t work.

There are several different types of possible corruption from corruption down in the DIT that causes weird issues (like MAPI and LDAP returning different results) up to data corruption up in the top levels of the directory that simply impact applications and AD has no issue with the data and doesn’t respond differently based on the data – say like putting in a fake value for a homeMDB value which could possibly crash Exchange. There is nothing that I am aware of that MSFT has to perform a corruption check at any level. Probably the “best” tool currently for the lowest level stuff is promoting a new DC (not an IFM, but a real promo) because it checks a lot of the data. That being said, I have seen incidents where bad info at the database level still replicated and in fact the only fix was to wipe the DCs involved. In one of them, thankfully the corruption was in the GC partition so a whole domain wasn’t lost because the corruption pre-dated the TSL so none of the backups would have been able to fix it either. That issue had been in place for a long time before someone noticed something breaking in Exchange in an unusual way.

I had another issue that I eventually ended up having to hack the replication and server objects to force replication. Something (I never got to an RCA) had occurred within a virtual DC that caused it to split the replication topology for an environment and had we just dumped the DCs that had been cut off we would have lost months of changes that had occurred on the segregated DCs and likely have impacted an entire region of the world to the point that most every machine would have required a rejoin and every user would have had to have gotten a password reset and the password somehow relayed to them (without email).

BTW, are there any tools for comparing domain controllers to see if they are really in sync with each other and haven’t had rollback issues that slipped through the various “nets”?

On twitter you mention data corruption, but don’t mention any tools to look for it. What tools should be used?

(btw, guess you don’t remember me, but we were in contact during your time at the blue oval employer ;o) and I’ve been using your tools which were deployed there almost everywhere during my daily helpdesk business ;o))

I am having this trust issue with a users computer account, would this tool reset the users password in AD? Do I run this under the local admin account on her computer? What do the other switches do? such as /forcescreset /fix

Big Thanks

Carsten

Scan type: Full scan (C:\|D:\|E:\|F:\|G:\|H:\|)
Scan options enabled: Memory | Startup | Registry | File System | Heuristics/Extra | Heuristics/Shuriken | PUP | PUM
Scan options disabled: P2P
Objects scanned: 530325
Time elapsed: 1 hour(s), 14 minute(s), 37 second(s)

Memory Processes Detected: 0
(No malicious items detected)

Memory Modules Detected: 0
(No malicious items detected)

Registry Keys Detected: 1
HKLM\SYSTEM\CurrentControlSet\Services\Adobe Licensing Console (Trojan.Clicker.CT) -> Quarantined and deleted successfully.

Registry Values Detected: 2
HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\Explorer\Run|Policies (Trojan.Agent) -> Data: c:\directory\CyberGate\install\Svchost.exe -> Quarantined and deleted successfully.
HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run|HKLM (Trojan.Agent) -> Data: c:\directory\CyberGate\install\Svchost.exe -> Quarantined and deleted successfully.

Registry Data Items Detected: 0
(No malicious items detected)

Folders Detected: 0
(No malicious items detected)

Files Detected: 10
C:\Windows\System32\msvfd32.exe (Trojan.Clicker.CT) -> Quarantined and deleted successfully.
C:\Games\Left 4 Dead\Left 4 Dead 2\left4dead2\addons\Name_Enabler.dll (Malware.UPX.Mod) -> Quarantined and deleted successfully.
C:\Users\Derp\AppData\Local\Temp\CouponDropDown.exe (PUP.CrossRider.CDD) -> Quarantined and deleted successfully.
C:\Users\Derp\AppData\Local\Temp\setup_coupondropdown.exe (PUP.CrossRider.CDD) -> Quarantined and deleted successfully.
C:\Users\Derp\AppData\Local\Temp\IXP001.TMP\flaudit.exe (Trojan.Clicker.CT) -> Quarantined and deleted successfully.
C:\Users\Derp\AppData\Local\Temp\mrt806A.tmp\stdrt.exe (Trojan.Clicker.CT) -> Quarantined and deleted successfully.
C:\Users\Derp\AppData\Roaming\8 0\svchost.exe (PUP.BitMiner) -> Quarantined and deleted successfully.
C:\Users\Derp\Documents\Adobe After Effects CS6\Patch\32bit\amtlib.dll (PUP.RiskwareTool.CK) -> Quarantined and deleted successfully.
C:\Users\Derp\Documents\Adobe After Effects CS6\Patch\64bit\amtlib.dll (PUP.RiskwareTool.CK) -> Quarantined and deleted successfully.
C:\Users\Derp\AppData\Local\Temp\.exe (Trojan.Agent) -> Quarantined and deleted successfully.

(end)

Currently running AVG and scanning it now. Also, here is a screen cap of what pops up after I removed everything that Malwarebytes picked up: http://i.imgur.com/ZU71H.jpg

http://jacksonshaw.blogspot.com/2012/12/samba-40-released-first-free-software.html

I have a fairly extensive lab environment that I just migrated from ESXi to Hyper-V 2012 and in doing so, took the time to do some performance testing on the storage. In the end, I ended up with a RAID-10 of four 128 GB SSD drives and then a couple RAID-10 arrays of 1 TB and 2 TB drives. I use the SSD for more IOPS dependent and the rest goes on the capacity arrays. I tried the “Storage Spaces” option in Server 2012 and found the performance to be horrible. Contact me if you want any of the results from my tests.

My hosts are Intel S5520HC boards with the Xeon E5520 processors and lots of RAM; that side of it is a few years old now though and I’m sure there’s better options.

Happy Holidays,

Byron Pearce
bpearce@interthinx.com
byronwp@yahoo.com

If the contribution doesn’t grow then I will consider opening it up to anon but I really do think people should get some recognition for helping and several of the books on Wiki’s I read spoke to vandalism and I don’t have time to monitor for wide spread vandalism. It could still happen with fake IDs etc but I would hope that it will slow it down at least a little.

There’s a PowerShell cmdlet that you can use to supposedly fix the secure channel, Test-ComputerSecureChannel [-reset]

Haven’t had the opportunity to test it yet, but yeah.

Great stuff, this article. Though this issue is rather “obscure”, I’d definitely love to know why it happens. I know of a brand new computer (+-1 month) that has had this happen many times already, like, how is that even possible if it’s only related to the machine password (changing)?

Any thoughts or solutions?

I have just finished a little app for a very similar purpose for a client. It is C#/ASP.NET and SQL based. It’s AD-centric I guess, so it asks for:

* location of app server(s)
* location of users
* what auth mechanisms they use (NTLMv1/v2/kerberos etc)
* how they connect (DC name/domain/GSLB/DC Locator etc).
* how many/what type of authentications
* how many LDAP searches
* whether they download data from AD (e.g. for caching user/group names in identity management tools)
* a bunch of other stuff

They have like 4000 applications across *NIX/samba and windows, and as well as trying to build up a use profile of AD they are trying to do specific things like weed out NTLMv1 and stop people writing to certain attributes.

The idea is that the application admin enters the info into the site, then the app works out if they are ‘compliant’ with the AD team’s requirements (must use kerberos/NTLMv2 etc) and lets them know. If not, someone from the ops team calls them and tells them to get compliant ASAP The app also has an admin page where AD admins can add notes, and classify the apps as compliant/non-compliant/work in progress etc.

I have also been doing some stuff to collect and present LDAP search stats (from 1644 and 1643 events) to further extend knowledge of who is actually querying and using AD.

I am sure I could make this available somewhere if people were interested. All you’d need would be IIS and SQL express.

Dan

LDAP_CAP_ACTIVE_DIRECTORY_W8_OID – 1.2.840.113556.1.4.2237

in the RTM version.

The currently released version of AdFind does not decode it as they didn’t tell me what the OID would end up being, just that they would have one. The next released version of AdFind will decode it. Right now it will just look like:

1.2.840.113556.1.4.2237 []

That being said I am supportive of my g/f who does enjoy doing that stuff. It is 3.25 miles with I think 14 obstacles.

I have not changed in my general approach to information disclosure. I will not allow any irresponsible detailed escalation steps/processes.

I think I would be open to general topics indicating that it is possible though.

Overall I will try to moderate very very little. Irresponsible security stuff, outright advertising for products (versus pages discussing utilities and if someone desires detailed scenarios/instructions about using products), spam, personal attacks I would feel would be the items most likely to be cut. But I don’t even want to have to police that so much as hope that people self-police themselves. I really want this to be a good solid resource that anyone in any place using any products can come and figure things out and share.

Question: How will you stop it from becoming another powershell script repository? (not that that will necessarily be a bad thing). It seems that’s what it will fill up with most.

Have you given thought to handling security issues? In the past, you’ve been reticent to discuss methods of escalating DA’s to EA’s since there’s no mitigation. Will these types of discussions be disallowed or just frowned upon? I don’t have a real strong opinion one way or the other; I’m just curious how you plan to approach it.

I just hope people aren’t intimidated to post after they see posts from you and others and think they have to be on that level.