joeware - never stop exploring... :)

Ever see that message before?

How about on a machine that was previously working just perfectly?

It’s anything but [OK]. You can usually think of some different verbiage that should be in the button under that message… Especially when the machine has special software on it that takes offense when you remove a machine from the domain and then try to rejoin the domain…

So what is the fix when you encounter this problem? You, as the AD fixit person, tell the people complaining that you have no clue why that happened but it must be something they did because if it was AD, there would be a lot of people with the issue. 🙂

And most likely, you are probably right.

To date, with over 12 years playing around with Active Directory and 16 years of playing with Windows Domains total I have yet to positively determine a case where AD broke and caused this issue. I am able to track it down to bad processes like cloning desktops after they were joined to a domain or someone resetting the machine account thinking that “reset account” meant something else or some goofy software that tries to make it so your machine can be a member of multiple domains and screws it up instead or finally when I can’t find anything but I also can see that AD is just fine and the password hasn’t been changed in AD for a week or more meaning if AD was the issue, you should have been hurting long before now…

Certainly part of the issue is troubleshooting. It is difficult to get information for this problem; you get that crappy error message and you can look at the date of the last password change in AD on the computer object (and even see it in human readable formats thanks to AdFind) and that is pretty much about it. So in the end, the fix for the problem is to tell the people to disjoin and rejoin the machine to the domain. That will solve it… Again, this gets painful with some apps that don’t like the idea of being dropped out of the domain and then rejoined to the domain. You can probably think of one or two (Microsoft designed and written even) apps like that…

Sidebar: So what is actually wrong? Why can’t the machine talk to the domain properly?

So as you may or may not know, every computer that is joined to a domain has an object in the domain that is called a “computer account”, this computer account is pretty much the same thing as a “user account” as AD looks at computers as users. There are some special cases like AD doesn’t force computers to change its password like it can do for a normal user, but realistically, there isn’t much more difference than that and the fact that various attributes that can be stacked onto the different object types. So like users, these computer accounts have passwords and the computer has to know the password for its account in order to “log into” the domain just like a normal user must log into the domain.

When this broken trust message is displayed, it is either because the computer object has been deleted OR the computer doesn’t agree with Active Directory on what it’s password is (and it can’t find the post-it note that it wrote the correct password down on). So like any user that forgets their password, AD tells them to go pound sand. Only in this case, instead of seeing an error message of “Hey numpty, that password isn’t correct, try again”, the message is far more dire… The trust relationship has FAILED! DAH DA DUMMMMMMMM! And the men swoon and the women scream… And what is the fix to the failed trust? To divorce the machine from AD and then to remarry them… At least effectively… You have to tell the machine that it is no longer with that pretty Domain X and is now hitting the single’s bar WORKGROUP. But then, phew, an admin tells Domain X you want her back and she accepts and the machine gets to join back up with pretty Domain X.

How is that accomplished? Well you reset the machine account in AD (or if you are silly you actually delete the machine account, then recreate it) which results in a machine account with a password that is known to the domain and to the machine and then you tell the machine to disjoin from the domain and then rejoin the domain. When the machine rejoins it uses this already well known password and voila, it can log on again. <And everyone cheers and throws bird seed>.

And now you are saying, great, so what, I have learned absolutely nothing…

Some time ago when digging through the new protocol docs that Microsoft was required to document (http://msdn.microsoft.com/en-us/library/hh128055(v=prot.13).aspx) I found some interesting discussion about computer passwords and that got me to digging into the API more and also into the Windows source and learned where the member computers store the current password they have saved for their domain account (i.e. the machine’s form of post-it note). I wrote some basic code and was able to display the information stored in the password slots but even cooler, I was able to display when the password (and previous password) were last set… Ah finally, additional debugging info. Now I also would really have liked to have gotten a good clear text password so I could say, hey, if we just change the password in AD to this same password, we will be good again as the machine and Active Directory will agree with each other on what the password is. Unfortunately, something else came up and I stopped looking at it and got dragged into some other issues and promptly forgot all about that line of research for some time…

Then recently a friend of mine and former Microsoft Directory Services MVP (which is how I met him years and years ago) who will go nameless to protect the guilty was complaining about some issues he was seeing in an environment he works in around machines that were failing their trusts. This guy is a bright guy, good photographer, and also knows a fair bit about Espresso I have learned; we shall call him Rich for lack of a better name. Well Rich starts pointing out to me that some folks in his account are having this issue and he has given them the standard AD response but “Gosh darn it joe… isn’t there a better answer?” I asked how many machines? He responded it was some smallish amount, like tens of machines out of tens of thousands machines… I said, that isn’t even a rounding error, no their isn’t a better answer; it’s their fault. 🙂

But then I thought about it and recalled my previous experiments and said to Rich, “Hey, I was working on something to extract the machine account passwords from machines previously, I don’t recall where I was on it, but let me check it out, we can probably do something with it.” and he thought that would be pretty cool.

So I dug into the my source code and saw where I got stuck and spent some nights walking through the Microsoft source code and completely failed to determine how to get nice clean clear text passwords out. But then I thought… who cares? Does it even matter? The old proverb… If you can’t solve the problem, change the problem.

So I changed my direction a little and started working on a tool that could be used both to get some troubleshooting information as well as actually set the password on the machine to something that AD and the machine could agree upon. Specifically, the default of the tool is to set the password to the default used by ADUC when you tell it to reset a computer account (though I let you specify a different password if you want). Once the computer account has been reset in AD and on the machine, you can simply force a secure channel reset (via NLTEST /SC_RESET:DOMAIN) and the machine should reconnect fine. In fact, I have also that secure channel reset capability within the tool so you don’t have to add an extra step. Voila, the machine and AD are trusting each other again. Sort of like what a great marriage counselor could do I guess only there are no lingering doubts and silent internal screams of “I HATE YOU!” afterwards. 😉

The utility isn’t completely done, I am slowing adding new features to it in the spare time I may or may not have in the evenings after work (doesn’t that make some/any company want to pay me millions to have me around to just dream crap up like this???). The most recent piece I added this last weekend is for those security conscious people that say… “Wait a minute, so you set that password to a known password and now the machine and AD are talking happily… but I don’t like the idea that the machine account now has a well-known password….” . And to that I say, ah good thinking, that is a bad idea isn’t it? Good thing I thought of it first. 😉 So there is an updatepwd option that tells the machine to kick LSASS into performing its normal password update routine. That way the weak password that originally got it talking to AD is no longer there and instead it is now has the default strength computer password.

Once I put that last piece in I decided I should simply the switches and allow a single switch to set the password to the ADUC default, reset the secure channel, then update the password to something secure. And for that I used a very simple, and hopefully intuitive, switch… /fix.

 

Here is a sample run of the whole process…

Pretend you saw the dialog box above at the top of the post and you have finished swearing. You have moved forward and logged into a machine that can still talk to the domain with an ID that has the ability to reset the password of the untrusted machine account and into the machine itself with a local administrator ID[1].

You open dsa.msc (ADUC aka Active Directory Users and Computers) on the machine that can talk to the domain and you find the computer account in question and then you right click and select actions and then reset account.

You then open an admin command prompt on the machine that is hurt and this is what it will look like…

Yep this trust isn’t happy… NLTEST with both QUERY and RESET say so…

C:\temp>nltest /sc_query:test
Flags: 0
Trusted DC Name
Trusted DC Connection Status Status = 5 0x5 ERROR_ACCESS_DENIED
The command completed successfully

C:\temp>nltest /sc_reset:test
I_NetLogonControl failed: Status = 5 0x5 ERROR_ACCESS_DENIED

 

So what does my new utility, MachinePwd, say about the password age… See below… Note that AD shows you a different time stamp. Depending on how well your domain is maintaining time, you will have a good idea how far out of sync they are. My example times below are forced and arbitrary. In real life I expect you will see one or the other timestamp has a much older value. If AD has the older time, something happened on the machine. If AD has the younger time, someone probably “accidently” reset the computer account.

One thing I learned is that the standard processes update both the current and previous passwords every time they are told to update the current password. I don’t know why they do that but I saw it clearly in the source code. One guess is that someone didn’t realize that the Secrets portion of the registry would maintain the previous secret string for you automatically, you don’t have to mess with it. If you do, and they do, well then the time stamp gets dorked. Alternately, there is some other reason that I couldn’t divine.

Notice the string “LocalTime”… that is there because I am writing this code in Visual Studio 2010 and I haven’t ported all of my previous libs over to it so the cool stuff I had for building time strings including time zone info isn’t available yet and I didn’t want to go through and whip up a temp TZ processing routine. You know what your local time is, I don’t have to tell you. I expect future versions will get it as I get more modules ported.

Anyway, what the machine says…

C:\temp>machinepwd

MachinePwd V01.00.00cpp Joe Richards (joe@joeware.net)  May 2012

Determining Machine Name…
MachineName: MEMB1-K8
Determining Domain Membership…
Domain Name: TEST
Opening policy…
Determining primary domain DNS info from machine policy…
DNS Domain Name: test.loc
DNS Forest Name: test.loc
Retrieving Machine Password Information…
Current  Password TimeStamp: 2012/06/04-20:40:47 LocalTime
Previous Password TimeStamp: 2012/06/04-20:40:47 LocalTime
Command completed successfully.

 

What AD says… I can tell you that I purposely changed the password in AD to break the trust between the machine and the domain.

C:\temp>adfind -default -f samaccountname=memb1-k8$ pwdlastset -tdcs

AdFind V01.45.00cpp Joe Richards (joe@joeware.net) March 2011

Using server: TEST-DC1.test.loc:389
Directory: Windows Server 2003
Base DN: DC=test,DC=loc

dn:CN=MEMB1-K8,OU=LockoutOU,DC=test,DC=loc
>pwdLastSet: 2012/06/04-21:42:03 Eastern Daylight Time

1 Objects returned

 

So we have the current info, let’s fix the secure channel (trust)…

C:\temp>machinepwd /fix

MachinePwd V01.00.00cpp Joe Richards (joe@joeware.net)  May 2012

Determining Machine Name…
MachineName: MEMB1-K8
Determining Domain Membership…
Domain Name: TEST
Opening policy…
Determining primary domain DNS info from machine policy…
DNS Domain Name: test.loc
DNS Forest Name: test.loc
Retrieving Machine Password Information…
Current  Password TimeStamp: 2012/06/04-20:40:47 LocalTime
Previous Password TimeStamp: 2012/06/04-20:40:47 LocalTime
Setting Machine Password to memb1-k8…
Set Machine Password.
Checking Secure Channel for test.loc…
Secure Channel is not established, error 0…
Resetting Secure Channel for test.loc…
Secure Channel is established with \\TEST-DC1.test.loc.
Request LSASS to securely update local machine password…
Machine password successfully updated.
Retrieving Machine Password Information…
Current  Password TimeStamp: 2012/06/04-21:46:18 LocalTime
Previous Password TimeStamp: 2012/06/04-21:46:18 LocalTime
Command completed successfully.

 

And let’s take a look at what NLTEST says now…

C:\temp>nltest /sc_query:test
Flags: 30 HAS_IP  HAS_TIMESERV
Trusted DC Name \\TEST-DC1.test.loc
Trusted DC Connection Status Status = 0 0x0 NERR_Success
The command completed successfully

If you so chose, you don’t have to use /fix. You can instead set a default (or other password) and then manually use nltest to reset the secure channel or you can, if you so choose, use the tool to tell the machine to change its current password (assuming there is a good secure channel in place already).

I expect to release the tool within the next few days. If you have any thoughts, leave a comment or send me an email. 🙂

[UPDATE]: You can find info on the new tool and download info here –> http://blog.joeware.net/2012/06/07/2513/

 

joe

 

[1] You can actually use the broken trust member to do all of the work if you use something other than ADUC that knows how to talk to AD without an established  trust (say like AdMod).