joeware - never stop exploring... :)

Pre-Creating Computer Accounts

When you create a machine account in Active Directory via ADUC (DSA.MSC), by default Windows generates a random (possibly, perhaps even probably non-printable) password for the computer account. If you select the “Assign this computer account as a pre-Windows 2000 computer” it takes the sAMAccountName, removes the trailing $ and then takes the first 14 characters, converts them to all lowercase and that becomes the computer password – this we will call the NT4 default computer password as it was the default prior to Windows 2000. Note that using that older default will not cause an issue for clients newer than NT4, they don’t care what the initial password is. When you create a computer account via script, it is most likely that you will use the NT4 default computer password as well because all the docs say to do it that way. NET COMPUTER is the old legacy API from NT4 so it also uses the NT4 default computer password. In fact, pretty much every tool you use to create a machine account, will generally use the NT4 default computer password unless there is a specific option to allow you to specify whether or not you want to use that default. This is generally a good thing or else people would have had no end of issues when we had NT4 machines still joining our domains. Oh one additional thing about Computer accounts created with ADUC… The userAccountControl will include the flag for PWD_NOT_REQD (bit 5, value 32) after the account is created. There is no need for this flag and can be removed. I don’t know of any known exploits or security issues around it being set but that doesn’t mean there aren’t any. At the very least it tends to freak out security people when they stumble over it.

Expiring Computer Accounts

Contrary to very wide belief, computer accounts DO NOT EXPIRE. Ever. The domain’s password age policy does not affect computer accounts. Ever. So if you talk to someone and they say, oh AD must have expired my computer account password, you can say, no, it most certainly did not, they don’t expire. Computer accounts will, by default, try to change their passwords every 30 days (it used to be 7 days back in NT times). You can modify this to be a different frequency. You can even modify it so that a computer will NEVER changes its password. I generally wouldn’t recommend this as computer passwords are one of the ways in which you can try to ascertain whether computer accounts are in use or not. It isn’t authoritative, but it can possibly narrow down the list of accounts you are looking at for possible lifecycle management. If an account has a password that is greater than 0 and less than 30 days old, you can generally ignore it in terms of lifecycle management because if something is changing the password, it is likely the computer and it could be a good indicator that it was still in use.

Computer Account Password Storage

Computers store their domain password in their “secrets” storage portion of the registry. This special storage is encrypted and stores the current and previous passwords along with the time stamps of when they were set. Computers maintain both passwords to assist with authentication. How you ask? When a computer attempts to logon to the domain, it will use the new password, if that password doesn’t work it will try the previous password. In this way, if a computer changes its password (excluding domain join) at DC1 and them attempts to authenticate at DC2 prior to replication of that new password, the computer should still be able to logon with the old password.

Computer Joining a Domain

When a computer joins a domain, it generates a new password and writes that value into the computer’s “secrets” storage as the machine’s new password in the domain. When a computer joins the domain and sets the password, the domain controllers DO NOT forward the password to the domain’s PDC (aka PDCe for any pendants out there[1]). This is likely contrary to what you may have learned or heard or believed previously. When I polled several different groups of admins, none of them were aware of any special process for computer account passwords and all assumed it was the same as how user accounts were handled. For the sake of clarity, I personally validated how the process works on every version of Windows from Windows Server 2003 AD through to the hot off the presses Windows Server 2012 AD.

Another point of interest, there is nothing special about a computer account to the computer. I.E. If you join a computer to a domain which results in the password change and setting some other attributes and that info hasn’t replicated to another DC that the computer touches, the computer will fail to establish a secure channel because of password mismatch, not any other reason such as missing SPNs or other special data. In fact, you can pre-create a machine account, replicate it out to all DCs. Then have the computer join at some random DC and reboot. Now, delete the machine account on all Domain Controllers and recreate the computer account with the NT4 default password and let that replicate out. When the PC comes up and the new computer object has replicated around, run my MachinePwd utility with the -fix switch or use -setpwd and set the password to the NT4 default password and then reset the secure channel with nltest /sc_reset and the machine will happily connect to the domain. I am not saying that I guarantee that every aspect of the machine will work properly, I don’t have the facilities to test and make that claim, but none of the quick normal stuff had any clue that something was amiss.

Computer Password Change

At some defined point, by default every 30 days or other specified time frame (reg key = HKLM\SYSTEM\CurrentControlSet\Services\NetLogon\Parameters reg value = MaximumPasswordAge) or when an admin forces a password change (nltest /sc_change_pwd:domain) a computer will attempt to change its domain password. The first step is that the computer checks with the DC it has a secure channel with to see if it is alive and responding. If it isn’t available the process will bail out and try again later (I believe every 15 minutes by default, whatever is set for the ScavengeInterval for the Workstation Scavenger Thread). If the DC is available, the computer will generate a new secure password, store it locally as described immediately below and then change the password on the DC. If the DC is configured to not allow a password change for the computer and responds as such, the computer will revert the local password back to the previous password. If the DC becomes unavailable between the start and the update portion of the process, the password will *not* be reverted. However this should not be an issue due to how the local password storage is configured and used as previously described.

When a computer changes its local version of the computer account password, it writes that password into the current password slot of the computer’s “secrets” storage. If the previous password is the NT4 default computer password (described previously) it will write the new password over the current and previous slots for security reasons. If the previous password was not the NT4 default computer password, then that password will be moved into the previous password slot. On Windows 2000 and Windows Server 2003, when the computer changes its password at the DC, just like with users[2], that password is forwarded to the PDC via a special out of band RPC call (that is not replication based[3]) just like with users (and unlike the join process). As of Windows Server 2008, this functionality changed. The password is no longer forwarded to the PDC from other DCs. By default, the only way the password is getting from the DCs back to the PDC is through standard replication. The main impact here is that depending on your replication topology, you may find that it takes longer to get a computer password replicated around your environment once it is changed with Windows 2008 or newer Domain Controllers.

PDC Chaining

The last paragraph may make one think of some cool fairly well known functionality that exists for users called PDC Chaining. When a user attempts to connect to a random DC with a new password and that new password hasn’t yet replicated to the DC, the DC will “chain” the request to the PDC. The idea being that the PDC should have the latest and most authoritative password for a given user. If the PDC says the password is ok, the user is allowed to logon. If the PDC says no, the user is denied because they supplied an incorrect password. Computer accounts are not handled by the domain controllers in the same way. Let me repeat *computer account authentication attempts DO NOT get chained like user account authentication does*.

If a computer presents an incorrect password, the domain controller responds with Access Denied, bad credentials! As mentioned previously, the computer has the old password and assuming the DC has the old password (and that old password isn’t the NT4 default computer password), all should be good; the computer will try again with the old password so is capable of establishing the secure channel. Eventually the new password should replicate to the DC and the next time the computer starts or has to otherwise re-establish the secure channel it will try that new password and it should work, and if it doesn’t, it still has the old password.

Issues Around Domain Join…

So given all of that information, it seems the biggest cause of issue would be around a domain join process being performed against one DC and the PC rebooting and realizing it should be using a different DC and eventually trying to establish a secure channel with that other DC prior to the updated password info replicating to the newly selected DC.

So for example, say you have three sites. Site1, Site2, and Site 3 and a DC in each one of those sites DC1, DC2, and DC3. The site topology is defined as…

            Site1

             / \

           /     \

    (12 hours) (15 minutes)

        /           \

      /              \

   Site2           Site3

And for whatever reason a machine joins a domain via DC2 in Site2; could be a script or tool or perhaps the DNS server was giving out hokey info or say the user got his/her machine joined when at building 2 and then left and went to building 3 or whatever. The PC then probably starts up and logs into DC2 in Site2 (because that is what it initially has cached and “remembers” from the join) but then later realizes that it really should be using Site3. Since it has a valid secure channel, the machine can flip to saying DC3 in Site3 is the preferred DC and things should work fine even though the password isn’t at Site 2 yet. Then for whatever reason (again) at some random time, the secure connection to Site2 breaks or the user reboots and so now the computer tries to establish its connection to DC3… Uh oh, it has only been seconds, minutes, or a few hours and the change hasn’t replicated to Site1 let alone all the way to Site3. You get a message about access denied when trying to establish the secure channel or an error saying there isn’t an account for the computer in the domain. You wait 12 hours and 15 minutes and the password from Site2 for the computer has now replicated to Site3 and the computer will be able to function normally.

…And What You Can Do About It

So know we know the problem with joins, what can we do about it?

Best Solution…

Well the first and most obvious answer would be to make sure that a computer joins the domain at a DC that is in the site that the computer is really supposed to be part of and will remain part of until replication convergence occurs. I.E. if you know your machine is part of Site3, make sure the domain join process occurs on DC3. The reason I consider this the best solution is because you don’t have to do anything except make sure you know what is going on up front and plan properly for it. However, it isn’t the best if the preferred site could change for some valid reason prior to replication convergence. Actually in that case the solution sucks. 🙂

So-so-might-work solution depending on environment with minimal impact…

If your DCs (particularly the DCs in the sites you will be performing the join against) are Windows 2003 R2 or older, then you can perform a little trick that may help depending on the specifics of the situation. First you need to pre-create the machine account so it exists across at least the PDC and the DC you will join at. Then, after the join but before the reboot run nltest /sc_change_pwd:domainname which will force a password change and push the password to the PDC. In many environments the PDC is in a central location and things are set up to receive changes from the PDC in a fairly quick manner and in the specific example above would reduce time to get the password to Site 3 from 12 hours 15 minutes to 15 minutes. Admittedly this will give you limited relief and only in specific cases. If Windows actually used PDC Chaining for computer accounts, this would be a perfect solution for pre-Windows 2008 DCs. Pssst Microsoft, did you read that?

Should-definitely-work complex solution with small impact…

Pre-create the computer account and let it replicate to all DCs. Join the computer to the domain and reboot. Logon to the computer as an administrator and using MachinePwd with the –setpwd switch set the password to some known value that isn’t the NT4 default computer value say like joerocks. Use a script or for /f batch loop with AdMod

Ex:

for /f %i in (‘adfind -sc dclist’) do adfind -hh %i –default -f name=computername -dsq | admod -hh %i unicodepwd::joerocks -optenc

to change the password on the computer account on all DCs in the domain to the known value. Re-establish the secure channel with nltest /sc_reset:domain. Finally execute nltest /sc_change_pwd:domain to update the password to something secure. You will now have both the new secure password and the insecure known value stored in the computer’s “secret” storage and the DCs will have either the secure value or the insecure known value as the secure password replicates across the domain. You should be able to move the computer to any site and it should work ok. The nice thing on this solution is that the level of permissions is limited to people who have rights to change passwords on the computer accounts and log into the workstations as an admin. I am not saying this is a good workaround but it is a workaround, smile when you curse me, I at least took time to figure it out and write it down here for you… 😉

Guaranteed-to-work solution with maximum impact and requiring serious creds…

Join computer to domain. There is no need to pre-create the computer account but doing so in domains with many DCs may decrease impact after the join a little as less data will be need to be sent at that time. Once the computer has successfully joined, force a replicate single object operation for the computer to every DC in the domain which will immediately send the updated computer object. This solution requires you to have the rights to force AD replication so likely requires a Domain Admin’s involvement or some sort of proxy tool that you set up specifically for this functionality.

Hybrid solution with small impact…

Follow the guaranteed-to-work solution except instead of syncing the computer object to every DC in the domain, link to specific key DCs or to the DC that you think (or know) the client is using. This solution requires you to have the rights to force AD replication so, again, likely requires a Domain Admin’s involvement.

But you ask, how do you find the DC with the latest password info?

Ex:

for /f %i in (‘adfind -sc dclist’) do @adfind -hh %i -f name=computername server:%i pwdlastset -tdcs -csv -nodn -csvnoheader

or if you want to do it via metadata info (and version)…

for /f %i in (‘adfind -sc dclist’) do @adfind -hh %i -f name=computername server:%i -ameta unicodepwd -tdcs -csv -nodn -csvnoheader

or if you want both pieces of info…

for /f %i in (‘adfind -sc dclist’) do @adfind -hh %i -f name=computername server:%i pwdlastset -ameta unicodepwd -tdcs -csv -nodn -csvnoheader

That is about all I have to say at the moment about the join workarounds…

And in summary…

But joe, what about machines that are joined already but still “fall out of the domain”? Well since I haven’t seen this first hand myself, we need to start collecting actual data about the situation. So when it happens do the following:

1. On the machine with an issue, run nltest /sc_query:domainname. Record the info.
2. If the machine’s secure channel really is broke, try to reset it…
   1. nltest /sc_reset:domainname.
      1. Record the results.
3. If the sc_reset didn’t work, try it again with a known DC name…
   1. nltest /sc_reset:domainname\dcname.
      1. Record the results.
4. If still not working, get the timestamp info on the computer password at the computer…
   1. Machinepwd.
      1. Record the results.
5. Now get the info from all domain controllers.
   1. for /f %i in (‘adfind -sc dclist’) do adfind -hh %i –default -f name=computername -ameta unicodepwd -tdcs
      1. Record the results
6. Send me the info so I can look at it.
7. Fix the computer
   1. You have three possible options
      1. Reset Machine account and use machinepwd /fix
      2. Rejoin the machine
      3. If one of the DCs has a newer password stamp than the other DCs, use the replicate single object functionality (via repadmin) to pull that info to the DC the workstation is using and then repeat above steps again.

 

I hope this helps…

 

    joe

 

 

[1] Just stop. 🙂

[2] Assuming you do not have AvoidPDConWan set and the PDC is reachable, this is entirely best effort, not guaranteed!

[3] It isn’t replication based *probably* because when this mechanism was put into place to account for the new functionality on Windows 2000 that allowed a password to be changed somewhere other than the PDC the Replicate Single Object functionality didn’t exist yet and replication could only occur between two DCs with a replication agreement and of course not every DC has a replication agreement with the PDC. You can see this for yourself by having the computer change its password on some non-PDC DC and then looking at the metadata for unicodePwd on the DC the change was made and the PDC. You will see that the metadata is very different and that the PDC will show as the originating DSA, not the other DC as it would show if it was replicated.