I wanted to share an analogy I wrote that was part of a response to a document effectively describing the cloning of a production AD environment that would live on the same physical network without a firewall.

Visualize you have a locked and secured gun cabinet with loaded shotguns with hair triggers. But instead of a cabinet you really have a corner of the room with some masking tape drawing lines on the floor around the guns of where the cabinet would be if it actually existed. Visualize your cute but perhaps mentally challenged 5 year old quadruplets playing in that room. Can you visualize anything bad happening as a result of having the tape on the floor instead of the actual cabinet even after you have explained in detail to the 5 year olds that the tape means stay away? A firewall can’t even be considered a cabinet; it is a 3 foot baby gate that can be bypassed by the 5 year olds given enough unattended time. A cabinet would be disconnected networks.

I was in a discussion and someone said to me that greater than 60% of Enterprise class Microsoft customers are already virtualizing writeable Domain Controllers in their production corporate environments.  !!!B??u?!l!l!?s??h!!i!??t?!!!

I started chuckling when I heard this. I don’t believe even for a second that the numbers are ANYWHERE near that level of penetration. Certainly there is a lot of chatter in this space but my personal experience is that the numbers are down in the single digit penetration of companies using writeable virtual domain controllers in Enterprise class corporate environments where AD failure could impact thousands, tens of thousands, or hundreds of thousands of people. People are concerned and despite Windows Server 2012 there are reasons behind it. Certainly it can be done, I have seen it done magnificently with no loss of redundancy by using a single external disk storage system or other single points of failure. I have also seen it done completely half ass and know of a production environment that completely blew up and had to be rebuilt from scratch, not from backup, but from scratch. If you are lacking complete of solid process in one area, chances are that isn’t the only area…

So I guess you could play around with the definition of Enterprise class and different people considering different size environments as "Enterprise Class". I think the smaller the environment, the more likely people may be likely to take the risk but also have the tight operational control they need to successfully virtualize DCs.

Now lab environments… I see virtual lab environments all the time. Bravo, I would rather see a virtual lab environment than NO lab environment and really I am quite ok with lab environments being virtual assuming they are true no-SLA no-SLO no-production-expectations-of-any-sort sandbox environments.  Environments that if they went a bit wonky on you you could fairly painlessly or better, absolutely painlessly blow them away and start over. If you have an SLA/SLO or an expectation of it being available and expect people to be running to put it back together if it blows… That isn’t a lab environment, that is another production environment.

I have been running virtual DCs in a lab / sandbox manner since the first beta I received of Windows 2000 and loaded it in a guest on an NT4 box running a beta or POC of the first VMware Workstation product. And yes, I have seen issues when I have made a mistake or the backend storage wasn’t as solid as it needed to be or took a power hit at just the wrong time, etc. But again, it is a lab environment, when it screws up, I delete it or sometimes see just how bad things can get in AD before I start to cry or get a headache. I have one lab AD that won’t allow me to promote another DC no matter what. I have it off to the side because I want to troubleshoot it until I figure out why. Everyday I think I am more likely to see that somewhere else out in the real world. (This isn’t a request for someone to help me sort it out, I will get to it when I get to it)

Anyway, I am guessing based on what I have seen out in the world, this other person is, IMO, wildly guessing based on what they have seen out in the world. I figured I would give the joeware followers a chance to respond as I think they will comprise a good number of the big Enterprise class companies (and militaries and governments) out there and I truly am curious. The poll, if I set it up correctly, will run until March 31 and results should be out first week of April. Please please respond and get your friends in other companies to respond too. I would really like to see where we truly are at.

If you like, you can always email me as well. If you don’t virtualize DCs, are you being pressured to do so? Do you have written policy against it? If you do virtualize DCs, I would like to hear those stories as well. How big? How many issues? How is the redundancy handled? Internal pass-thru disks on the physical host per MSFTs recommendations or external or ????

      joe

POLLS START HERE

[yop_poll id=”2″] [yop_poll id=”3″] [yop_poll id=”4″] [yop_poll id=”6″] [yop_poll id=”7″] [yop_poll id=”8″] [yop_poll id=”9″]

THANKYOU!!!

Semi-imaginary conversation…

Techie1: Here is our private cloud solution, it costs X.

Techie2 (speaking for customer): Ummm, they were thinking more of a budget "just-like-a-private-cloud" solution I guess.

Techie1: Ah, so they really want the fake movie fog, not a Private Cloud.

Techie1: On the outside it looks like a real private cloud solution but you save money. It should work great until something breaks and all of the redundancy and "costly" stuff that they don’t want to pay for is missed.

…going around telling companies that virtualizing DCs is perfectly safe and there are no concerns without having even the slightest bit of information about the delivery model and environment in question.

Morons.

If you are talking to someone from one of those two companies and really any consulting company and they say something like virtualizing DCs is perfectly safe before actually diving in and checking with the delivery team and validating that they are comfortable with idea of it and the extra troubleshooting that is likely required when you have issues (like performance issues) and looking at the environment and how it is working and configured as a whole just haul off and blast them square in the mouth because they really shouldn’t be opening their mouth without having a full understanding of the environment in question.

It is very easy to be able to find enough info to say no, this isn’t a good idea; it is much tougher to be sure you can say yes to an environment. Most consultants don’t look at it that way because they don’t have to support it, they say go do it and then they go off to the next company to give them bad advice.

Oh and another thing, the new VMGENID capability of Windows Server 2012 AD and HV is __NOT__ USN Rollback Protection. It just helps reduce the possible spread of stupid ways in which you can encounter it. It is not, nor was it designed to be, a comprehensive end all be all there is no way to cause a problem solution. Maybe it will get there someday but we are not there yet. If you were the type of individual that was bright enough to figure out how to virtualize your DCs but stupid enough to click on the SNAPSHOT buttons then hurray, you are now sort of protected. Otherwise, not much change here folks.

IMO, my overall thoughts I tell people when they ask is that we are not much safer now with the VMGENID capability than we were before assuming you would have been following proper processes and procedures with your virtualized DCs in the first place… In other words, if you determined it wasn’t safe for you to do it under Windows 2000, 2003, 2008, 2008R2 I don’t see enough difference in the products to make it safe now. And keep in mind, someone telling you you are going to be perfectly safe needs to be right all the time, someone telling you it could screw up only needs to be right once. If you don’t have absolutely awesome disaster recovery processes for AD that are regularly tested, you are in no position to consider putting your AD in a position of further risk.

     joe

P.S. You can tell them that I said it needed to be done and if they want to bitch they can contact me.

P.P.S. I think you can virtualize DCs, but if you are thinking, this is how I can save a ton of money, perhaps you should be reviewing your purposes. I am not a strong proponent of removing redundancy, introducing insecurity, and making an environment more complex on the idea that I might save a little money when the system we are talking about is AD and companies that don’t have a proper functioning AD may cease to exist.

P.P.P.S. Yes I have seen and heard of DCs, Domains, and even forests wiped out due to problems with virtualized DCs. Just because you may not have heard of it, doesn’t mean it doesn’t happen. Most people and companies aren’t all that quick to share info about security breaches and identity system failures.

…what items trigger VMWare to update the vmgenid? I don’t know, so I asked them at

http://blogs.vmware.com/apps/2013/01/windows-server-2012-vm-generation-id-support-in-vsphere.html#comment-1252

This is in relation to yesterday’s post at http://blog.joeware.net/2013/02/20/2675/

   joe

…an alias for Active Directory anti-USN Rollback functionality. I heard that today and I wanted to spit on my monitor.

…a statement from Microsoft that you can’t hurt yourself when virtualizing DCs. I have heard this implied multiple times in the last few months; primarily from Microsoft Consulting folks who aren’t actually supporting any systems, just selling solutions.

Yes people, USN rollback is STILL absolutely possible with VM-Generation ID (vmgenid) functionality fully engaged and properly configured. You are only protected in very limited set of very certain very specific circumstances.. Specifically reverting a snapshot on a vmgenid aware virtualization platform or when you use the export settings feature of a vmgenid aware virtualization platform. Any other type of activity with the VHD files and you better be dead sure that the functionality works such as file copies, file restores, SAN/NAS functions, etc. I will make it simple, it probably doesn’t work like you think because Microsoft didn’t try to account for every possible stupid thing people might consider doing or accidently do when in the heat of battle.

So outside of the two things they protect you from, there are other actions that can put you into a USN Rollback situation and are not protected against with Windows Server 2012 AD.  I can list several actions off the top of my head that are not protected and they aren’t a stretch that someone would try to use them.

Repeat after me… Windows Server 2012 AD makes it "safer" to virtualize, that is a far cry from a cart blanche statement that virtualizing Domain Controllers is "safe"[1] under Windows Server 2012… And yes I have heard of people, including Microsoft Consultants make that very mistaken statement… "Virtualizing Domain Controllers is now safe under Windows Server 2012" when what they really meant if someone was smart enough to press them is the actual statement that "Virtualizing Domain Controllers is now safeR under Windows Server 2012.".

So to make this simple.

Yes, you absolutely can shoot yourself in the foot with AD on physical DCs. It can also be insecure.

Virtualization made the dangerous and insecure scenarios possible on physical DCs easier and, IMO, way more feasible to occur in the real world.

Windows Server 2012 AD makes it a little less easy and less likely to occur if you are the type that likes hitting EXPORT SETTINGS or REVERT SNAPSHOT. Otherwise it is the same level of danger as every other version of AD.

So if you have a bug under your bonnet to virtualize a Domain Controller, you still need to think very long and hard about it and making sure you are willing to spend the extra money to build the proper fully redundant infrastructure that you automatically get with having multiple physical DCs and that you are willing to support it in such a way that you don’t hurt yourself in any of the many ways that become more feasible with virtualized DCs. With Windows Server 2012 AD Microsoft,thankfully, moved one of the knives a little further out of reach, they didn’t make your skin invincible.

Note I am not saying that virtualizing DCs can’t be done properly. It absolutely can. I have seen and heard of companies who have been doing it for many years. I even actually recommended it once for a very specific use case… Once. As a general rule though, most every design I have seen has had significant shortcomings in the area of redundancy. One of the most common being all virtual guest VHDs living on a single NAS/SAN that everyone seems to think can’t fail. Listen people, I have been in Enterprise Level Data Center situations where a SAN blew out for a couple of days and no virtual machines were able to run because no one could get to their virtual disk files – happened twice in a single week at one company in fact. How well would you do if all of the DCs in one of your core corporate Data Centers were hard down and people still needed to authenticate?

Overall, in my experience over the last decade+, most companies are more worried about costs than doing things properly and those companies should stick with physical DCs because cost cutting doesn’t fit in with the idea of virtualizing Domain Controllers. I would love to say it would always be safe and good, it would make life simpler for most Domain Admins.

    joe

P.S. I have driven my Mustang GT well over 130MPH on several occasions with no ill effects. It doesn’t mean the next time I won’t splatter myself all over the highway no matter how careful I am about it. I won’t do it with someone else in the car or in an area where I can endanger someone else, I only have the right to endanger myself. Similarly, I will often, well usually, recommend against virtualization of Domain Controllers for most companies.

UPDATE: Note if it wasn’t clear or you weren’t aware, the vmgenid "triggers" are dependent upon the virtualization platform.  Different platforms could have different triggers meaning you could have various levels of protection from different actions on various platforms, this is key information to understand when architecting your solutions. You need to know when you are protected and when you aren’t protected and how.

[1] "safe" as defined as if you fall it doesn’t matter, we have a nice bed of feathers for you to fall in and can’t possibly hurt yourself.

…unless there is no aspect of AD testing for the Dev/Test stuff. I.E. No testing of AD authentication, no testing of AD IDs, no testing of management of AD, either on purpose or as a side effect of the testing you really are doing, etc. For example, if you are testing a new version of an application, how do you know it won’t flood AD with auth requests for an AD ID? How do you know you won’t need to troubleshoot at the AD level like with network sniffing or something? You don’t.

However, if you are testing an application on a member server and that application has nothing to do with AD (local IDs and groups, etc), go for it, but add that dev or test server to the production forest in a DEV/TEST OU. You don’t need a domain. If you think you need a domain, then you are possibly thinking that there exists some sort of boundaries between domains in a forest that do not really exist.

To put it another way. If I were a domain admin for an environment, I would be the domain admin for every domain in the forest in that environment. I would not allow anyone not on my team to have access to any DC in the forest, regardless of whether someone thought it was a Dev or Test Domain because it would still be a production forest domain and hence, production.

If you do not have a formal Dev/Test environment, meaning an entirely separate forest or forests, then in actuality, you have no production environment regardless of what you want to call it – you only have a lab environment and well, don’t expect production availability and stability out of a test/lab environment.

For those in the know, they realize I am paraphrasing something said by one of the father’s of Active Directory – Mr. AD – Don Hacherl on the ActiveDir Org list (Friday, February 20, 2009 4:08 PM) that I previously quoted on this blog (http://blog.joeware.net/2009/03/11/1623/).

I have to make a comment here, as I’ve heard this too many times.  You do, in fact, have a lab environment.  What you do not have is a production environment.

DonH

   joe

http://blogs.vmware.com/apps/2013/01/windows-server-2012-vm-generation-id-support-in-vsphere.html

Update: Also read this –> http://blog.joeware.net/2013/02/20/2675/