How to rotate X.509 certificates in Percona Server for MongoDB without stopping the service

An expired TLS certificate is one of the most predictable incidents, and yet one of the most painful in production. When the X.509 is only valid for client connections, its expiration blocks the entry of new sessions. When it is also used for internal authentication between the nodes of a replica set or a sharded cluster, the problem is more serious: the members themselves stop authenticating with one another. This is the scenario addressed in a recent article by Ivan Groenewold on the Percona blog, showing how to renew certificates with minimal disruption using the rotateCertificates command.

The central point of the article is a distinction the DBA needs to have clear before touching any file: there is the simple renewal (same CA) and there is the structural change (new CA, different DN, altered cluster-membership attributes). Only the former is eligible for hot reload. Confusing the two is the shortest path to a broken cluster.

What a same-CA renewal is

The procedure described is what Percona calls a same-CA renewal: server, member, and client certificates are reissued by the same already-trusted CA, and the X.509 attributes used for cluster membership do not change. In practice, this means the trust chain remains identical, and the values of O (Organization), OU (Organizational Unit), and DC (Domain Component) used in internal membership matching stay the same.

If none of that changes, mongod and mongos can simply reread the certificate files and start presenting them in new connections, without a process restart and without a forced replica set election. That's where the operational gain lies.

Do not apply this hot-reload procedure when replacing the issuing CA, changing a certificate subject DN, or changing cluster-membership attributes. Those are not ordinary renewals.

>

-- Ivan Groenewold, Percona

What gets reloaded and what doesn't

Percona Server for MongoDB rereads the files pointed to by three TLS configuration directives:

net:
  tls:
    mode: requireTLS
    certificateKeyFile: /etc/mongod/tls/server.pem
    CAFile: /etc/mongod/tls/ca.pem
    clusterFile: /etc/mongod/tls/cluster.pem

certificateKeyFile holds the certificate and private key presented to regular clients. clusterFile holds the certificate and key that a mongod or mongos process presents when connecting to other cluster members. A detail that avoids headaches: if clusterFile is not configured, certificateKeyFile is also used for authentication between members, meaning a single file carries both roles.

The behavior of rotateCertificates is deliberately conservative. It only affects new TLS connections. Already-established client sessions are not terminated, and the command does not trigger a replica set election. This is precisely the property that makes the operation compatible with the zero-downtime requirement so common in regulated environments in Brazil, where stopping the service to swap a certificate is often unacceptable.

Preparation before the maintenance window

The source's recommendation is to start well before expiration and never make the first attempt in production. The preparation checklist involves a few precautions worth repeating:

• Inventory every process and client certificate. This includes every mongod member, every mongos router, the application drivers, hosts running mongosh, backup jobs, monitoring, and automation tools. A client certificate forgotten in a backup job becomes a 3 a.m. incident.
• Confirm that it is a same-CA renewal. The trusted issuer chain needs to remain the same, and the O, OU, and DC attributes used in internal matching cannot change.
• Assemble a new PEM file for each server and client. The file referenced by certificateKeyFile or clusterFile needs to contain both the certificate and the corresponding private key, in this order: the key first, followed by the certificate, with the encapsulation delimiters preserved.
• Validate the certificate against the CA and check the details before copying anything to the production TLS directory.

Restrictions on the online reload

There are two limitations that must be respected when hot-rotating:

| Restriction | Detail | |---|---| | File name and path | Each new certificate must have the same file name and the same path as the certificate it replaces | | Password | If the certificate is password-protected, the new one's password must be identical to the old one's |

If CAFile, a CRL, or any other configured TLS entry is also being renewed in the same operation, it needs to be replaced before invoking the reload. The command rereads the TLS entries as a set: a missing or invalid entry makes the entire reload fail.

Here's a reassuring point highlighted in the source: incorrect certificate files make the rotation fail, but do not invalidate the existing configuration or produce side effects. The process keeps serving with the previous material. From the operator's point of view, this turns rotateCertificates into a fail-safe operation, which greatly reduces the risk of the window.

Running the rotation

The execution itself is straightforward. Connect directly to the specific mongod or mongos with an administrative user and run:

db.getSiblingDB("admin").runCommand({rotateCertificates: 1, message: "Renewed TLS certificate"})

The message field is logged and serves as an audit trail for the operation, something that helps in compliance environments. Right after running it, the author himself recommends immediately validating a new TLS connection to that process using a renewed client certificate, as well as inspecting the log for the successful rotation message and any TLS errors. For replica sets and sharded clusters, Percona points to the documentation with the specific step-by-step, since the order in which it's applied across members matters.

Final validation and cleanup

After completing it, the checklist suggests double-checking the expiration date and the SANs of the certificate presented by each mongod and mongos. Old certificates should only be kept during the approved overlap period, and then removed or revoked. And, closing the mature operation cycle: record the new expiration dates and set up alerts with sufficient lead time. A certificate rotation that doesn't generate an alert for the next rotation is technical debt that only comes due at expiration.

When this procedure does NOT apply

The boundary is explicit. Hot reload does not apply when any of the items below changes:

• the issuing CA or the trusted CA chain;
• the subject DN used by a MONGODB-X509 client user;
• the O, OU, or DC values used in the default intra-cluster membership matching;
• net.tls.clusterAuthX509.attributes or net.tls.clusterAuthX509.extensionValue.

In these cases, the member's identity or the root of trust is changing, and a simple reload cannot handle it; it requires its own CA-coexistence procedure, which the source leaves for a future article. For the DBA, the practical message is clear: before scheduling the window, correctly classify the type of renewal. Treating a CA change as if it were a routine renewal is the mistake that turns a five-minute task into a full cluster-outage incident.