Subscribe

TIMES UP, MICROSOFT —

Windows feature that resets system clocks based on random data is wreaking havoc

Windows Secure Time Seeding resets clocks months or years off the correct time.

-

Windows feature that resets system clocks based on random data is wreaking havoc
Enlarge

A few months ago, an engineer in a data center in Norway encountered some perplexing errors that caused a Windows server to suddenly reset its system clock to 55 days in the future. The engineer relied on the server to maintain a routing table that tracked cell phone numbers in real time as they moved from one carrier to the other. A jump of eight weeks had dire consequences because it caused numbers that had yet to be transferred to be listed as having already been moved and numbers that had already been transferred to be reported as pending.

“With these updated routing tables, a lot of people were unable to make calls, as we didn't have a correct state!” the engineer, who asked to be identified only by his first name, Simen, wrote in an email. “We would route incoming and outgoing calls to the wrong operators! This meant, e.g., children could not reach their parents and vice versa.”

A show-stopping issue

Simen had experienced a similar error last August when a machine running Windows Server 2019 reset its clock to January 2023 and then changed it back a short time later. Troubleshooting the cause of that mysterious reset was hampered because the engineers didn’t discover it until after event logs had been purged. The newer jump of 55 days, on a machine running Windows Server 2016, prompted him to once again search for a cause, and this time, he found it.

The culprit was a little-known feature in Windows known as Secure Time Seeding. Microsoft introduced the time-keeping feature in 2016 as a way to ensure that system clocks were accurate. Windows systems with clocks set to the wrong time can cause disastrous errors when they can’t properly parse timestamps in digital certificates or they execute jobs too early, too late, or out of the prescribed order. Secure Time Seeding, Microsoft said, was a hedge against failures in the battery-powered onboard devices designed to keep accurate time even when the machine is powered down.

“You may ask—why doesn’t the device ask the nearest time server for the current time over the network?” Microsoft engineers wrote. “Since the device is not in a state to communicate securely over the network, it cannot obtain time securely over the network as well, unless you choose to ignore network security or at least punch some holes into it by making exceptions.”

To avoid making security exceptions, Secure Time Seeding sets the time based on data inside an SSL handshake the machine makes with remote servers. These handshakes occur whenever two devices connect using the Secure Sockets Layer protocol, the mechanism that provides encrypted HTTPS sessions (it is also known as Transport Layer Security). Because Secure Time Seeding (abbreviated as STS for the rest of this article) used SSL certificates Windows already stored locally, it could ensure that the machine was securely connected to the remote server. The mechanism, Microsoft engineers wrote, “helped us to break the cyclical dependency between client system time and security keys, including SSL certificates.”

Simen wasn’t the only person encountering wild and spontaneous fluctuations in Windows system clocks used in mission-critical environments. Sometime last year, a separate engineer named Ken began seeing similar time drifts. They were limited to two or three servers and occurred every few months. Sometimes, the clock times jumped by a matter of weeks. Other times, the times changed to as late as the year 2159.

“It has exponentially grown to be more and more servers that are affected by this,” Ken wrote in an email. “In total, we have around 20 servers (VMs) that have experienced this, out of 5,000. So it's not a huge amount, but it is considerable, especially considering the damage this does. It usually happens to database servers. When a database server jumps in time, it wreaks havoc, and the backup won’t run, either, as long as the server has such a huge offset in time. For our customers, this is crucial.”

Simen and Ken, who both asked to be identified only by their first names because they weren’t authorized by their employers to speak on the record, soon found that engineers and administrators had been reporting the same time resets since 2016.

Dan Goodin Dan Goodin is Senior Security Editor at Ars Technica, where he oversees coverage of malware, computer espionage, botnets, hardware hacking, encryption, and passwords. In his spare time, he enjoys gardening, cooking, and following the independent music scene.

Promoted Comments

NTP has worked for decades under Unix. An organization of any real size (more than, say, a dozen systems) would do well to have their own trio (or more) of local NTP hosts that peer with each other while looking at distinct external time sources (or even a local GPS clock), then have all other systems sync from the peers. The peering helps to ensure that one broken clock doesn't throw the others completely off, and having all other systems look at the cluster of peers ensures the organization as a whole stays in sync with itself.

Large organizations would do well to just have their own set of stratum 1 time servers (minimum of three) anyway. Some NTP appliances struggle with heavy client loads, so having a set of stratum 2 hosts can help buffer the load.

NTP is an open protocol that's been around long enough that it's seen use in a fairly wide variety of environments. It is probably way more reliable than something Microsoft cooked up and acknowledges is based on trusting SSL handshakes occur in "expected" ways.
So, for those that are unaware radio clock sync is a thing when you need a very reliable reference clock but either can't afford your own atomic clock (100s of thousands of dollars) or remote NTP stratum clocks aren't feasible, radio interface reference receivers accessed by USB or RS-232 are a thing. I have no idea if this would work around STS, because no one seems to know what its problem is, but you can turn it off and rely on your radio receiver.

You can run an external antenna, pick up your choice of positioning satellite data or the clock sync radio beacons your country or WWV broadcast from the US NIST on various SW frequencies. The computer then takes that high quality time data and uses it to set the internal system clock and calculate skew. Then you can use that computer to sync the rest of the clocks on your isolated networks if you so desire over a secure version of ntpd. Also, please note, not just any GPS receiver will work. You have to have one that includes a reference signal on the access line so the computer can determine response lag and jitter across the interface cable. Most consumer grade GPS receivers don't have that.
WWV signals can be difficult to tune outside of certain hours and weather conditions. It's also fairly difficult to find WWV recievers that can easily plug into computers that aren't expensive bespoke legacy devices. They were basically extinct when I went searching a few years ago, but maybe someone sells one again.

GPS recievers are more reliable and easier to source (uBlox makes a lot of them), and you only need to care about the speed of light through the cable in special circumstances. For the majority of the population being off by a few microseconds is not an issue. You can also often add manual delay calculations by doing math on the propagation speed through the cables. Not quite as good as testing them, but good enough for most work. Your computer hardware and OS delays will cause far larger errors than that anyway.

Channel Ars Technica

Related Stories

    Today on Ars