Keeping correct time is understandably pretty important. Without correct time logs are unreliable, for both troubleshooting and investigations, domain logins don't work since Kerberos tickets have time based expiration, SSL/TLS won't work for the same reason. In short, invalid time is all kinds of bad for numerous reasons and the failures may not be obviously due to time skew.

At ${dayjob} we have a rather large number of appliances deployed across the country and one production run ended up with a batch of bad batteries. While the systems are up and running NTP does a good job of periodically asking the time server what the current time is and setting it back if the time ends up being off. Once the OS stops handling the clock the hardware falls back to the motherboard to keep time and with the faulty battery the time gets reset to some preset value, sometimes the oldest time the BIOS can store but more frequently when the BIOS was originally compiled. Whatever the time is it will most likely be a long time ago, frequently years.

When the device boots up it will ask the time server for the correct time...problem solved right? As Patience would say, there's a kind of hitch. NTP client services get a little cranky if the time from the NTP server is too different from their own and refuse to change the time. In the case of ntpd on Linux will log a message and exit if the difference is more than 1000 seconds (16 minutes 40 seconds), Windows W32Time will keep running but not change the time if it differs by more than 54,000 seconds (15 hours). Most Linuxes account for a bad hardware clock by allowing you to perform a one-time hard set of the time on boot, this brings the clock into shape such that ntpd can do it's job.

Unfortunately for us our appliances run Windows which doesn't have such a feature. After some internal discussions the best option we had was a startup task to hard set the time which we all felt was pretty terrible. Some careful and exceptionally boring reading of the Windows Time Service Tools and Settings TechNet article solved the problem for me. There are two registry keys,

MaxNegPhaseCorrection
Registry path
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Config
Version
Windows XP, Windows Vista, Windows 7, Windows Server 2003, Windows Server 2003 R2, Windows Server 2008, and Windows Server 2008 R2

This entry specifies the largest negative time correction in seconds that the service 
makes. If the service determines that a change larger than this is required, it logs 
an event instead. Special case: 0xFFFFFFFF means always make time correction. The 
default value for domain members is 0xFFFFFFFF. The default value for stand-alone 
clients and servers is 54,000 (15 hrs).

and

MaxPosPhaseCorrection
Registry path
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Config
Version
Windows XP, Windows Vista, Windows 7, Windows Server 2003, Windows Server 2003 R2, Windows Server 2008, and Windows Server 2008 R2.

This entry specifies the largest positive time correction in seconds that the service 
makes. If the service determines that a change larger than this is required, it logs 
an event instead. Special case: 0xFFFFFFFF means always make time correction. The 
default value for domain members is 0xFFFFFFFF. The default value for stand-alone 
clients and servers is 54,000 (15 hrs).

By setting both keys to 0xFFFFFFFF the W32Time service will always correct the time on every time sync. It doesn't fix the root problem of faulty hardware but it does keep affected systems running while we go through the repair process.