Database credential rotation after an ISPConfig breach

I rotate credentials the moment I suspect a breach. Not after I’ve confirmed one. Database credential rotation on an ISPConfig server sounds like routine maintenance. It stops being routine the day you find a web shell on a hosted site. Or a configuration file you can’t explain, sitting in a public backup. At that point, every password tied to that database stack needs to be new. Verified. Documented. Not just assumed to be fine.

database credential rotation lock and key rotating above an ISPConfig database

When does rotation stop being optional?

A few scenarios bring this up often enough that I keep a rotation plan ready for them. I’d rather not improvise one under pressure.

A plugin or CMS on a hosted site gets compromised. I wrote about one real case on this kind of server in PHP2Shell: how I protected my servers while waiting for the patch, a PHP vulnerability that let an attacker drop a web shell before a patch existed. Once a web shell runs under a hosting user, it can often read that site’s own config.inc.php. On a shared ISPConfig box, that file sits next to credentials for the control panel’s own database. Not only the one credential the attacker was actually after.

A configuration file ends up somewhere it shouldn’t. A backup archive left in a public folder. A repository pushed with a stray file in it. A log that’s more readable than anyone intended. config.inc.php and mysql_clientdb.conf have no business in any of those places. Mistakes happen anyway, including mine.

A system administrator leaves the team. However amicable the departure, shared database credentials they once held are credentials I no longer fully trust. Trust doesn’t transfer automatically when someone walks out the door.

A suspected intrusion, even an unconfirmed one. This case is hardest to explain to anyone who hasn’t run a server under suspicion of attack. You don’t wait for proof before acting. You assume compromise and rotate. An unnecessary rotation costs an afternoon. A skipped one, after a real breach, can cost every site the server hosts.

Why manual database credential rotation is so error-prone

The first time I rotated ISPConfig’s database credentials by hand, I underestimated how many files were involved. They all have to change together, or something quietly breaks a few hours later.

There’s config.inc.php on the server side, and a second copy on the interface side. Both need the same new values. There’s mysql_clientdb.conf, which ISPConfig reads to manage every hosted site’s own database, not just its own housekeeping. Dovecot, Pure-FTPd and Postfix each keep their own copy of some of the same credentials. Mail and FTP can stop authenticating silently if you update the database user and forget just one of those three.

Then there’s the Master/Slave topology some of my setups use. A Slave node authenticates to its Master’s database over the network, under a separate password. That password has to be rotated on the Master itself, not only locally. Get the sequence wrong, and a Slave can lock itself out of the very Master it’s supposed to be replicating from.

None of that is conceptually difficult. It’s a lot of files. Several services to restart, in a sensible order. Almost no tolerance for a typo, especially while you’re doing it under pressure, right after finding something you didn’t want to find. That combination is exactly why database credential rotation kept feeling riskier by hand than the task itself should be. It’s why I wrote reset_DB_password.sh, after living through the problem myself more than once.

What reset_DB_password.sh actually does

reset_DB_password.sh detects whether the node is Standalone, Master or Slave. It then runs only the phases that role needs.

Phase A rotates the local MySQL root password. That’s the password ISPConfig itself uses to manage every hosted site’s database. The script updates mysql_clientdb.conf to match the new value.

Phase B rotates the ispconfig control user’s password. It updates config.inc.php on both the server and interface side. Then it updates the matching credentials inside Dovecot, Pure-FTPd and Postfix, so mail and FTP keep authenticating without a separate manual pass through three more config files.

Phase C runs only on a Slave. It connects to the Master’s database remotely. It rotates the Slave user’s password there, then updates dbmaster_password locally, so replication keeps authenticating correctly against the new value.

After Phase B, the script offers to restart Dovecot, Pure-FTPd and Postfix for you. I almost always say yes. A credential rotation that leaves a mail service running on the old password isn’t finished, it’s just quiet about being broken. Restarting in the same run, right after the config files change, keeps the whole thing inside one short maintenance window instead of three separate ones discovered later.

Backups before anything changes

Before touching a single file, the script takes a full rsync copy of /usr/local/ispconfig/ into a timestamped folder. It also attempts a non-blocking mysqldump of the mysql and dbispconfig databases. Non-blocking on purpose: I’d rather finish a credential rotation than sit waiting on a dump that’s stuck for some unrelated reason. Every configuration file it edits gets its own .bak copy first, before any string inside it gets replaced.

None of that replaces an independent backup strategy. I still take a separate snapshot, or a full backup, before running this on any server that actually matters. The script’s own disclaimer says the same thing, more bluntly than I just did.

Dry run first, every time

Every phase asks for confirmation before changing anything. –dry-run simulates an entire run without touching a single file. I use dry-run first on every server, every time. Even on ones I’ve rotated credentials on before, and think I know well. –skip-local-root skips Phase A, for when only the control user’s password needs rotating. –role=master or –role=slave overrides the automatic detection, for the rare case where it doesn’t read a server’s topology correctly on its own.

The passwords it generates are 32 characters, letters and digits only. Long enough to resist guessing. Plain enough that MySQL’s and PHP’s own parsers never trip over a stray symbol in a connection string. I’ve tested the script on ISPConfig 3.3.1p1. I haven’t run it against older or newer branches. I wouldn’t assume identical behaviour there, without checking first myself.

A node’s role isn’t always obvious at a glance, especially on a server you inherited rather than built yourself. That’s why –role exists, and why the dry-run output is worth reading line by line the first time, not skimmed. I’d rather spend two minutes confirming a Slave is really a Slave than find out the hard way mid-rotation.

What I’d tell you before you run it

Disclaimer: the script is provided as-is, for informational purposes. You use it entirely at your own risk: I accept no responsibility for data loss, downtime, or any error caused by incorrect use or by running it in a context different from the one I use it in myself. Always keep an independent backup or VM snapshot before running it in production, regardless of what the script’s own backup step already captures for you.

It’s free software. You can redistribute and modify it, with attribution to me as the original author. The source for reset_DB_password.sh, and the rest of the ispconfig-fast-tools collection, is on GitHub at github.com/matteocisilino/ispconfig-fast-tools.

Database credential rotation is a habit, not a cleanup job

Database credential rotation after a confirmed breach is just competent incident response. Most administrators would do it without being told twice. The harder habit is doing it after a merely suspected one, before anyone has proof. That’s the only version of the plan that actually limits the damage, while it’s still limitable. A script that runs the right phases for a server’s role, takes its own backups first, and lets me dry-run the whole thing before touching anything real. That’s what makes the habit practical on a Tuesday night, instead of theoretical in a blog post.

ISPConfig, MySQL, PHP, Dovecot, Pure-FTPd, Postfix and the other names mentioned are trademarks of their respective owners. They are used only to identify the products.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

WordPress Cookie Notice by Real Cookie Banner