Chronos Keeps Time: Building an Enterprise-Grade NTP Service with chrony and Integrating it with FortiGate

Time is foundational — even in a lab

Every network device, every server, every log event in a well-run environment shares one thing: a consistent view of time. When clocks are out of sync, log correlation falls apart, certificate validation silently fails, and troubleshooting becomes an archaeology project trying to figure out which timestamps to believe. In a production environment this is obvious enough that it gets solved early. In a lab it tends to get kicked down the road.

We kicked it down the road once too often. When we started seeing our FortiGate and our Linux hosts disagreeing on the time — sometimes by enough to cause certificate warnings and confuse syslog — it was clearly time to do this properly. Not just “point everything at pool.ntp.org and call it done”, but build a proper internal time source, harden it correctly, and use it as a foundation the rest of the lab depends on. We called our new NTP server Chronos.

This post covers the whole build: why we chose chrony, why an enterprise approach makes sense even for a home lab, what the relevant security frameworks actually say, the full chrony configuration including a mistake we made that will feel immediately familiar to anyone who’s written a network ACL, how to generate NTP keys that survive a trip into FortiOS, and finally the FortiGate configuration itself. There’s also a section at the end on stratum levels — what they mean, how to manipulate them, and what the hierarchy looks like in production when you have access to a GPS reference.


Why chrony and not the alternatives

Before picking up a package manager there is a genuine choice to make here. Three NTP implementations come up regularly on Linux and they’re not equivalent.

ntpd (from ntp.org) is the original. It has been around for decades, it works, and its configuration format is widely documented. The downside is that it was designed for systems that are permanently connected to good upstream sources with stable, low-jitter paths. It reacts slowly to large initial clock offsets and uses a gradual slewing approach that can take a long time to bring a system with a badly wrong clock into line. It also carries a longer historical security record — not all of it reassuring.

systemd-timesyncd is a lightweight SNTP-only client built into systemd. It is perfectly fine for a desktop workstation that just needs to know roughly what time it is. It is not appropriate for a system that is going to serve time to other devices. It implements a simplified subset of NTP, does not support authentication, and has no server mode. It is there for convenience, not for infrastructure.

chrony is the modern choice. It was designed specifically for systems that aren’t always connected, that may have variable network conditions, and that need to converge quickly from large offsets. It handles clock drift well, supports NTP authentication (both MD5 and SHA), runs as a low-privilege daemon by default, has both client and server modes, and is actively maintained. It’s what the CIS Linux Benchmarks recommend, what Red Hat and Debian both ship as their default NTP implementation on server builds, and what our lab needed.

Installation on Debian/Ubuntu:

sudo apt update && sudo apt install chrony

On RHEL/Rocky/AlmaLinux:

sudo dnf install chrony

If systemd-timesyncd is running on the system you’re building on, disable and mask it before chrony takes over:

sudo systemctl disable --now systemd-timesyncd
sudo systemctl mask systemd-timesyncd

Masking it prevents something else from pulling it back up as a dependency later.


Why build a proper NTP service for a lab at all

This is the question worth answering before spending an evening on it. The short version: if you’re building a lab to develop and maintain enterprise skills, the discipline matters as much as the technology.

Log correlation is only as good as your time source. When something goes wrong — or you’re building a scenario to practice incident response — you need to sequence events across devices. A FortiGate log, a Linux auth log, and a syslog message all need to agree on what happened first. With each device independently polling a different upstream server, you’re accepting unknown and variable offsets between them. A single internal authoritative source eliminates that variable.

Certificate validity depends on time. TLS certificates have notBefore and notAfter fields. OCSP responses are time-sensitive. If your FortiGate’s clock is several minutes ahead of a server it’s inspecting traffic to, SSL inspection will behave oddly in ways that are hard to diagnose. Get the time right and this class of problem disappears entirely.

Authentication matters even internally. An unauthenticated NTP server is a trivially exploited attack path. A rogue device advertising itself as an NTP source can shift clocks across the network, which in turn can invalidate certificates, confuse Kerberos, and shift log timestamps to obscure an attack. Building NTP with authentication from day one means you’re learning the right pattern.

Fortinet’s own tooling depends on it. We’ve already seen from the FortiSwitch pairing series what happens when NTP isn’t solid before you start building — FortiLink management tunnels that silently fail, certificates that won’t validate, FortiGuard lookups that behave oddly. A trusted internal time source with authentication takes all of this off the table.


Security frameworks and what they say about time

Before writing a single line of configuration, it’s worth knowing what the relevant frameworks expect.

CIS Linux Benchmark

The CIS Benchmark for Linux (we’re working against the Ubuntu 22.04 LTS and Debian 12 versions) has explicit time synchronisation requirements under section 1.5:

  • Ensure a time synchronisation service is installed and chrony is the implementation of choice
  • Ensure ntpd and timesyncd are not running alongside chrony — only one time daemon should be active
  • The chrony daemon should run as the _chrony user (not root), which modern packages handle by default
  • The server or pool directives should reference known-good upstream sources
  • Access control should be configured to restrict which clients can query the server
  • Authentication should be enabled where the daemon is acting as a server to clients

For audit purposes, CIS expects you to be able to confirm:

systemctl is-enabled chronyd
# expected: enabled

chronyc tracking | grep "Leap status"
# expected: Normal

grep -E "^server|^pool" /etc/chrony/chrony.conf
# Should show your upstream sources

NIST SP 800-53

NIST’s AU-8 (Time Stamps) requires that systems use internal clocks or an authoritative time source to generate timestamps for audit records, mapped to UTC or GMT. The practical implication for us: the NTP server in the lab is the authoritative internal clock, and everything else syncs to it. Lab hosts and the FortiGate should not independently reach out to internet time sources — all NTP traffic flows to Chronos.

NIST SC-45 (System Time Synchronisation) extends this to require that time synchronisation prevents and detects modification. That’s the authentication requirement: both the server and client use a shared key to validate that NTP responses are coming from the right place and haven’t been tampered with in transit.

Fortinet Security Hardening Guide

Fortinet’s own hardening documentation for FortiOS covers NTP specifically. The key points:

  • Enable NTP synchronisation with ntpsync enable
  • Set the source type to custom and point to an internal, trusted server rather than Fortinet’s default internet sources
  • Enable authentication on the NTP server configuration with a matching key on both sides
  • Set a reasonable syncinterval — the hardening guide suggests 60 minutes as a baseline; tightening to 15–30 minutes is reasonable for a lab where you want quick recovery after an outage

There’s also a broader point in Fortinet’s documentation worth noting: the FortiGate should have NTP working and verified before you start building Security Fabric members or FortiLink pairs. We learned this one the hard way in an earlier post.


Building Chronos: the chrony configuration

The base configuration

The main chrony configuration file lives at /etc/chrony/chrony.conf on Debian/Ubuntu or /etc/chrony.conf on RHEL-family systems. Here’s the configuration we landed on for Chronos:

# Upstream time sources — UK pool servers
# iburst gets us to an initial sync quickly on startup
pool 0.uk.pool.ntp.org iburst
pool 1.uk.pool.ntp.org iburst
pool 2.uk.pool.ntp.org iburst
pool 3.uk.pool.ntp.org iburst

# Record the rate at which the system clock gains/loses time
# so chrony can compensate for hardware clock drift
driftfile /var/lib/chrony/drift

# Allow the system clock to be stepped (rather than slewed) in
# the first three updates if the offset is larger than 1 second.
# After three updates, only slewing is used — gradual adjustment.
makestep 1.0 3

# Sync the hardware real-time clock to the system clock
rtcsync

# Use a keyfile for NTP authentication when serving clients
keyfile /etc/chrony.keys

# Log directory
logdir /var/log/chrony

# Log statistics and measurements
log measurements statistics tracking

# Advertise as stratum 2 when serving time from local clock
# (if upstream sources are temporarily unavailable)
local stratum 2

# Access control — the ordering here matters, see the next section
deny all
allow 10.0.100.0/24

That deny all before allow ordering is deliberate and there is a story behind it.

Starting and enabling the service

sudo systemctl enable --now chronyd

Verify it’s running and has found upstream sources:

chronyc sources -v

You should see output like:

MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^+ ntp1.wtnet.de                 1   6   377    29  -2117us[-2117us] +/-   15ms
^* pool.ntp.example.net          1   6   377    30   +312us[ +475us] +/-   15ms
^+ ntp2.example.pool.ntp         1   6   377    29   +890us[ +890us] +/-   11ms

The * marks the currently selected primary source. + means contributing to the combined estimate.

chronyc tracking

This gives the full synchronisation picture:

Reference ID    : 4F866A14 (ntp.example.com)
Stratum         : 2
System time     : 0.000001234 seconds fast of NTP time
Last offset     : +0.000001200 seconds
Frequency       : -5.234 ppm slow
Root delay      : 0.015243412 seconds
Leap status     : Normal

Leap status: Normal is what you want. To see which clients are querying Chronos once the FortiGate is configured, see the chronyc clients note in the verification section below.


The lesson we learned about chrony.conf and access control

This is the part of the build that will feel familiar to anyone who has written a network access control list — and that’s exactly why it caught us out.

When you’re configuring ACLs on a router or firewall, the pattern is: add a deny any any at the end as an explicit catch-all. It makes the deny behaviour visible rather than relying on an implicit drop. It’s good practice, and it’s how most network engineers instinctively think about ordered rule sets.

We applied the same thinking to chrony.conf and wrote this:

# What we wrote first — and it didn't work the way we expected
allow 10.0.100.0/24
deny all

The logic seemed sound: allow our lab subnet, then deny everything else. The problem is that chrony doesn’t work like a firewall ACL. Chrony’s access control evaluates rules using a last-matching-entry-wins model, not first match. deny all matches every address — including addresses in 10.0.100.0/24 — and it appears after the allow rule, so it wins for every address. The result was Chronos refusing to serve time to any NTP client, including the FortiGate.

The fix is straightforward once you understand the model, but it’s the inverse of what a network engineer will reach for instinctively. deny all goes at the top, with specific allow statements following it:

# Correct — deny all first, specific allows afterwards
deny all
allow 10.0.100.0/24

Now when a client in 10.0.100.0/24 connects, the evaluation finds deny all (matches), then finds allow 10.0.100.0/24 (also matches). The allow rule is the last matching entry, so it wins. Clients in the allowed range get through. Everything else hits deny all as its only or final matching entry.

The correct behaviour is documented in the chrony configuration reference under the allow and deny directives:

“The address matching is tested in the order in which the commands appear in the configuration file, with the last matching entry winning.”

One sentence. Easy to read past. We didn’t read it carefully enough the first time, and the symptom — Chronos simply not responding to NTP clients — wasn’t immediately obvious as an access control problem rather than a connectivity or firewall issue. Once we found that line in the documentation and corrected the ordering, everything came up immediately.

If you’ve come from writing network ACLs and you’re configuring chrony access control: put deny all first, allow statements after. It’s the inverse of what you know.


NTP keys: creating credentials that FortiOS will actually accept

NTP authentication uses a shared key between the server and the client. Both sides configure the same key ID and key value; the server signs its responses with the key, and the client rejects anything that doesn’t match. It’s not a substitute for network-level access control, but it’s a meaningful second layer — a client needs the correct key to accept time from Chronos, which prevents rogue NTP servers from poisoning the clock.

Generating a key

Chrony includes a key generation command:

chronyc keygen 1 SHA256 256

This generates a key with ID 1, using SHA256, with a 256-bit key length. The output looks something like:

1 SHA256 HpGq5J+nDzBvY2sK9mT7cX8oLwFj3Qr/eAiU6NbM0kZhV4tPdCyWu1RlOE=

That line goes directly into /etc/chrony.keys. The format is <key-id> <algorithm> <key>.

The special characters problem

The first key we generated contained a /, a +, and an = — standard base64 output. When we entered that key into the FortiGate’s NTP configuration, the CLI accepted it without complaint. NTP authentication then silently failed. No error, no warning — diagnose sys ntp status showed the FortiGate reaching Chronos but authenticate: no in the output.

The issue is that FortiOS’s NTP key field doesn’t handle certain special characters reliably. The /, +, and = that appear in base64 output can be interpreted or encoded differently depending on whether you’re using the GUI or CLI, causing the key that lands in the running config to differ from what you intended. The client and server end up with different key strings, authentication fails, and you get no useful diagnostic output pointing at the cause.

The fix is to generate a key that contains only characters FortiOS handles safely — alphanumeric only. The right tool is tr:

cat /dev/urandom | tr -dc 'a-zA-Z0-9' | head -c 32

This reads from /dev/urandom, discards (-d) everything not in (-c = complement) the character class a-zA-Z0-9, and takes the first 32 characters. The result is a 32-character alphanumeric string with no special characters that could be mishandled.

If you want to clean up output from chronyc keygen rather than generating directly:

chronyc keygen 1 SHA256 256 | awk '{print $3}' | tr -d '+/=\n' | head -c 32

This generates the key, extracts the key value, strips the problematic characters, and takes 32 characters. Note that stripping characters from base64 does reduce the effective entropy slightly — generating directly from /dev/urandom via tr is the cleaner approach.

Loading the key into chrony

Edit /etc/chrony.keys:

1 SHA256 YourAlphanumericKeyHere12345678

The key ID (1) is the number the FortiGate will reference in its configuration. You can have multiple entries with different IDs if you want separate keys for different clients or client groups.

Set the file permissions correctly:

sudo chown root:chrony /etc/chrony.keys
sudo chmod 640 /etc/chrony.keys

Restart chronyd to pick up the key:

sudo systemctl restart chronyd

Stratum levels: what they mean and how to think about them

The stratum system is how NTP represents the distance from an authoritative time source. Understanding it helps you make sensible decisions about your hierarchy and avoid either overclaiming accuracy or misrepresenting your clock quality to downstream clients.

Stratum 0 is not an NTP server — it’s a reference clock. GPS receivers, caesium atomic clocks, rubidium oscillators, and WWVB/DCF77 radio receivers are stratum 0 devices. They connect to stratum 1 servers directly via serial port or PPS (pulse-per-second) signal and are never directly reachable over a network.

Stratum 1 servers are directly disciplined by a stratum 0 reference. They are the root of the NTP hierarchy. The servers in pool.ntp.org are generally stratum 1 or stratum 2.

Stratum 2 is what Chronos becomes. It syncs from stratum 1 servers in the pool, making it one hop removed from a primary reference. When a client queries Chronos, Chronos reports its own stratum as 2 in the NTP response.

Stratum 3 is what our FortiGate and lab hosts become once they’re pointing at Chronos.

[Pool NTP servers — Stratum 1]
         |
    [Chronos — Stratum 2]
         |
  [FortiGate — Stratum 3]
         |
   [Lab hosts — Stratum 3/4]

Manipulating stratum in chrony

The local stratum directive controls what Chronos advertises when operating from its local clock rather than from synchronised upstream sources:

local stratum 2

This tells chrony: when serving time from the local clock (for example, during a brief internet outage when upstream pool servers are unreachable), advertise yourself as stratum 2. It does not override the stratum reported during normal synchronised operation — that’s derived automatically from the upstream stratum. The local stratum directive keeps lab devices receiving some time reference from Chronos rather than going unsynchronised entirely, with the stratum number signalling to clients that accuracy has degraded.

We settled on local stratum 2 for Chronos. When synced from the pool, it naturally reports as stratum 2. If it loses upstream temporarily, it continues at stratum 2 from local clock. The lab’s NTP hierarchy stays stable.

Why not advertise as stratum 1?

We could configure local stratum 1 and have Chronos claim to be a primary reference. That would be inaccurate. Stratum 1 carries a specific meaning: directly connected to a hardware reference clock. Falsely advertising stratum 1 is poor practice — it’s the kind of misrepresentation that causes confusion when someone interrogates the NTP hierarchy and finds a claimed stratum 1 source with the jitter and root delay of a pool-synced server.

Stratum 1 in production: GPS-based NTP

In a production environment you often have access to better time sources than a public NTP pool. The most common is a GPS-based NTP appliance — Meinberg, Spectracom, and Trimble all produce rack-mountable units that pair a GPS receiver with an NTP server in a single box. They provide stratum 1 timestamps with accuracy in the microsecond range, and they continue working when internet access is unavailable.

For a more accessible option, a Raspberry Pi with a GPS HAT and PPS signal produces a working stratum 1 source. The gpsd daemon feeds the GPS data to chrony via the refclock directive:

refclock SHM 0 offset 0.5 delay 0.2 refid GPS noselect
refclock PPS /dev/pps0 lock GPS refid PPS prefer

Cloud platforms also expose their own NTP infrastructure. AWS provides 169.254.169.123, Google offers time.google.com, and Azure has time.windows.com. These are typically stratum 2 or 3 depending on the region, but with very low latency from within those platforms. If you’re running production workloads in the cloud, using the platform’s NTP endpoint as an upstream source alongside or instead of the public pool is a sensible choice.

If Chronos were syncing from a GPS-based stratum 1 source rather than pool.ntp.org, nothing would change about the configuration downstream. Chronos would still be stratum 2 (one hop from the GPS reference), the FortiGate would still be stratum 3. The accuracy would improve significantly; the configuration and hierarchy are identical. The stratum chain stays correct regardless of what’s upstream of Chronos.


Connecting Chronos to the FortiGate

Chronos connects to a dedicated port on the FortiGate — we’re treating it as a protected management resource rather than putting it on the general lab network. The addressing used in these examples is illustrative; use whatever fits your own design.

Interface configuration

config system interface
    edit "port5"
        set vdom "root"
        set ip 10.0.100.1 255.255.255.252
        set allowaccess ping
        set alias "NTP-Chronos"
        set role lan
    next
end

Chronos itself is configured with 10.0.100.2/30 and a default gateway of 10.0.100.1. The chrony.conf allow directive needs to cover the FortiGate’s source address — the /30 is sufficient for this direct link:

deny all
allow 10.0.100.0/30

If other lab subnets need to query Chronos, add them:

deny all
allow 10.0.100.0/30
allow 10.0.200.0/24

NTP configuration on the FortiGate

config system ntp
    set ntpsync enable
    set type custom
    set syncinterval 60
    config ntpserver
        edit 1
            set server 10.0.100.2
            set authentication enable
            set key-id 1
            set key "YourAlphanumericKeyHere12345678"
        next
    end
end

set type custom switches the FortiGate away from Fortinet’s default internet NTP pool. set authentication enable requires both key-id and key to be set, and the key value must exactly match what’s in /etc/chrony.keys on Chronos, including case.

Firewall policy for lab hosts reaching Chronos

The FortiGate itself reaches Chronos directly via the interface above. Lab hosts on other interfaces need a policy:

config firewall address
    edit "Host-Chronos-NTP"
        set type ipmask
        set subnet 10.0.100.2 255.255.255.255
    next
end

config firewall policy
    edit 0
        set name "Lab-to-NTP-Chronos"
        set srcintf "lan"
        set dstintf "port5"
        set srcaddr "all"
        set dstaddr "Host-Chronos-NTP"
        set action accept
        set schedule "always"
        set service "NTP"
        set logtraffic disable
    next
end

NTP is UDP/123. The built-in NTP service object on FortiOS covers this. set logtraffic disable keeps the log clean — NTP polling every 60 seconds from every lab host generates a lot of low-value log entries.

Verifying NTP on the FortiGate

diagnose sys ntp status

Expected output when working correctly:

synchronized: yes
ntpsync: enabled
server: 10.0.100.2
  reachable: yes
  unreachable count: 0
  offset: 0.8ms
  delay: 1.2ms
  authenticate: yes
  refid: 10.0.100.2
  stratum: 3

synchronized: yes and authenticate: yes are the two lines to look for. stratum: 3 confirms Chronos (stratum 2) as the upstream source.

execute date

If this matches what chronyc tracking reports on Chronos, and both are within a second or two of actual UTC, the setup is working correctly.


Verification commands to keep handy

On Chronos:

# Check upstream source quality and reachability
chronyc sources -v

# Detailed synchronisation state: offset, frequency error, root delay
chronyc tracking

# Per-source statistics — useful for spotting a flapping upstream
chronyc sourcestats

# See which NTP clients are actively querying Chronos
# Important: this command requires root privileges. Running it as a standard
# user returns "501 Not authorised" — which looks alarming but simply means
# chronyc needs elevated access for this particular query. It does NOT mean
# your NTP clients are unauthorised or that authentication has failed.
sudo chronyc clients

On the FortiGate:

# Full NTP status including authentication state
diagnose sys ntp status

# Quick system time sanity check
execute date

# NTP daemon debug output when something isn't working
diagnose debug application ntpd -1
diagnose debug enable
# Wait a minute, then:
diagnose debug disable
diagnose debug reset

What this gives the lab

With Chronos running and the FortiGate pointed at it, every device we add to the lab gets the same treatment: point it at Chronos for NTP, confirm with sudo chronyc clients that it shows up, and the timestamp problem is solved at the infrastructure level.

The deny-all lesson is one of those things that looks obvious in hindsight but is easy to get wrong if you’re carrying mental models from a different protocol family. The chrony documentation is clear about it once you find the right sentence. The symptom — NTP clients getting no response from Chronos — doesn’t immediately suggest an access control ordering problem, which is why it cost us some time to diagnose.

The key generation issue is similar: FortiOS accepts the key in the CLI without complaining, which makes the subsequent silent authentication failure a particularly irritating thing to track down. Generating alphanumeric-only keys with tr from the start avoids the problem entirely and takes seconds longer than doing it the first way.

Next up in the lab build series: building an internal certificate authority, issuing certificates for FortiGate admin access, and exploring what certificate-based authentication gives you over password accounts.

  • SSH hardening: certificate authorities and bastion patterns — the natural next step after Chronos: building the internal CA model for SSH today sets up the same pattern for issuing FortiGate admin certificates
  • nftables — the same deny-all/allow last-match-wins ordering mistake that bit the chrony access control rules shows up identically in firewall chain policy design