When does rotation stop being optional?

A few scenarios bring this up often enough that I keep a rotation plan ready for them. I’d rather not improvise one under pressure.

A plugin or CMS on a hosted site gets compromised. I wrote about one real case on this kind of server in PHP2Shell: how I protected my servers while waiting for the patch, a PHP vulnerability that let an attacker drop a web shell before a patch existed. Once a web shell runs under a hosting user, it can often read that site’s own config.inc.php. On a shared ISPConfig box, that file sits next to credentials for the control panel’s own database. Not only the one credential the attacker was actually after.

A configuration file ends up somewhere it shouldn’t. A backup archive left in a public folder. A repository pushed with a stray file in it. A log that’s more readable than anyone intended. config.inc.php and mysql_clientdb.conf have no business in any of those places. Mistakes happen anyway, including mine.

A system administrator leaves the team. However amicable the departure, shared database credentials they once held are credentials I no longer fully trust. Trust doesn’t transfer automatically when someone walks out the door.

A suspected intrusion, even an unconfirmed one. This case is hardest to explain to anyone who hasn’t run a server under suspicion of attack. You don’t wait for proof before acting. You assume compromise and rotate. An unnecessary rotation costs an afternoon. A skipped one, after a real breach, can cost every site the server hosts.

Why manual database credential rotation is so error-prone

The first time I rotated ISPConfig’s database credentials by hand, I underestimated how many files were involved. They all have to change together, or something quietly breaks a few hours later.

There’s config.inc.php on the server side, and a second copy on the interface side. Both need the same new values. There’s mysql_clientdb.conf, which ISPConfig reads to manage every hosted site’s own database, not just its own housekeeping. Dovecot, Pure-FTPd and Postfix each keep their own copy of some of the same credentials. Mail and FTP can stop authenticating silently if you update the database user and forget just one of those three.

Then there’s the Master/Slave topology some of my setups use. A Slave node authenticates to its Master’s database over the network, under a separate password. That password has to be rotated on the Master itself, not only locally. Get the sequence wrong, and a Slave can lock itself out of the very Master it’s supposed to be replicating from.

None of that is conceptually difficult. It’s a lot of files. Several services to restart, in a sensible order. Almost no tolerance for a typo, especially while you’re doing it under pressure, right after finding something you didn’t want to find. That combination is exactly why database credential rotation kept feeling riskier by hand than the task itself should be. It’s why I wrote reset_DB_password.sh, after living through the problem myself more than once.

What reset_DB_password.sh actually does

reset_DB_password.sh detects whether the node is Standalone, Master or Slave. It then runs only the phases that role needs.

Phase A rotates the local MySQL root password. That’s the password ISPConfig itself uses to manage every hosted site’s database. The script updates mysql_clientdb.conf to match the new value.

Phase B rotates the ispconfig control user’s password. It updates config.inc.php on both the server and interface side. Then it updates the matching credentials inside Dovecot, Pure-FTPd and Postfix, so mail and FTP keep authenticating without a separate manual pass through three more config files.

Phase C runs only on a Slave. It connects to the Master’s database remotely. It rotates the Slave user’s password there, then updates dbmaster_password locally, so replication keeps authenticating correctly against the new value.

After Phase B, the script offers to restart Dovecot, Pure-FTPd and Postfix for you. I almost always say yes. A credential rotation that leaves a mail service running on the old password isn’t finished, it’s just quiet about being broken. Restarting in the same run, right after the config files change, keeps the whole thing inside one short maintenance window instead of three separate ones discovered later.

Backups before anything changes

Before touching a single file, the script takes a full rsync copy of /usr/local/ispconfig/ into a timestamped folder. It also attempts a non-blocking mysqldump of the mysql and dbispconfig databases. Non-blocking on purpose: I’d rather finish a credential rotation than sit waiting on a dump that’s stuck for some unrelated reason. Every configuration file it edits gets its own .bak copy first, before any string inside it gets replaced.

None of that replaces an independent backup strategy. I still take a separate snapshot, or a full backup, before running this on any server that actually matters. The script’s own disclaimer says the same thing, more bluntly than I just did.

Dry run first, every time

Every phase asks for confirmation before changing anything. –dry-run simulates an entire run without touching a single file. I use dry-run first on every server, every time. Even on ones I’ve rotated credentials on before, and think I know well. –skip-local-root skips Phase A, for when only the control user’s password needs rotating. –role=master or –role=slave overrides the automatic detection, for the rare case where it doesn’t read a server’s topology correctly on its own.

The passwords it generates are 32 characters, letters and digits only. Long enough to resist guessing. Plain enough that MySQL’s and PHP’s own parsers never trip over a stray symbol in a connection string. I’ve tested the script on ISPConfig 3.3.1p1. I haven’t run it against older or newer branches. I wouldn’t assume identical behaviour there, without checking first myself.

A node’s role isn’t always obvious at a glance, especially on a server you inherited rather than built yourself. That’s why –role exists, and why the dry-run output is worth reading line by line the first time, not skimmed. I’d rather spend two minutes confirming a Slave is really a Slave than find out the hard way mid-rotation.

What I’d tell you before you run it

Disclaimer: the script is provided as-is, for informational purposes. You use it entirely at your own risk: I accept no responsibility for data loss, downtime, or any error caused by incorrect use or by running it in a context different from the one I use it in myself. Always keep an independent backup or VM snapshot before running it in production, regardless of what the script’s own backup step already captures for you.

It’s free software. You can redistribute and modify it, with attribution to me as the original author. The source for reset_DB_password.sh, and the rest of the ispconfig-fast-tools collection, is on GitHub at github.com/matteocisilino/ispconfig-fast-tools.

Database credential rotation is a habit, not a cleanup job

Database credential rotation after a confirmed breach is just competent incident response. Most administrators would do it without being told twice. The harder habit is doing it after a merely suspected one, before anyone has proof. That’s the only version of the plan that actually limits the damage, while it’s still limitable. A script that runs the right phases for a server’s role, takes its own backups first, and lets me dry-run the whole thing before touching anything real. That’s what makes the habit practical on a Tuesday night, instead of theoretical in a blog post.

ISPConfig, MySQL, PHP, Dovecot, Pure-FTPd, Postfix and the other names mentioned are trademarks of their respective owners. They are used only to identify the products.

The post Database credential rotation after an ISPConfig breach first appeared on Matteo Cisilino.

I ran into it after an optimisation. I had run OPTIMIZE TABLE hoping to reclaim space, but the filesystem didn’t shrink as expected. Without a “before” to compare with the “after”, it was impossible to tell whether the optimisation had worked. Nor could I tell which database was really responsible, or whether the missing space was somewhere else entirely.

The tools I had answered only half of the problem:

• du shows the files on disk, but doesn’t say which table they belong to;
• information_schema (or pg_class on PostgreSQL) gives the logical sizes, but doesn’t see binlogs, ibdata1, undo logs or WAL;
• both photograph a single instant: comparing over time remains manual work, done with copy and paste and a spreadsheet.

I needed something simple: take a snapshot of the state of all databases at a given moment, take another later and have it tell me what changed, sorting the differences from the largest to the smallest. That way the culprit jumps out immediately.

The result is a single Bash script. It has no dependencies beyond the database client and the standard Linux tools (awk, du, df). It works with MySQL/MariaDB and with PostgreSQL. I’m sharing it here in the hope it saves someone the same afternoon of frustration.

What the database disk space script does

The script has four commands:

• snapshot saves the current state (databases, tables, files in the data directory, filesystem space);
• compare compares two snapshots and shows the differences, the largest on top;
• report shows the largest databases and tables in a single snapshot;
• list lists the saved snapshots.

The comparison is split into four sections, so as not to leave any blind spots:

• size of each database (data + indexes);
• free or reclaimable space (data_free on MySQL, an estimate of dead tuples on PostgreSQL);
• size of each table;
• actual files in the data directory: database folders, ibdata1, binlogs, undo, pg_wal.

The last section is often the one that solves the mystery. Many times the “anomalous” space doesn’t belong to any database, but to the binary logs or to InnoDB’s shared files.

Requirements

• Linux with Bash 4 or later, awk, du, df (all standard);
• the mysql or psql client, with a user who can read the metadata of all databases;
• root permissions (or those of the mysql/postgres user) if you also want to measure the files on disk; without them, that section is skipped with a warning.

Installation

Save the script (you will find it at the end of the article) to a file and make it executable:

nano db_space_compare.sh
chmod +x db_space_compare.sh

Credentials

For MySQL/MariaDB the cleanest way is a ~/.my.cnf file with the credentials, so they don’t appear in the command history:

[client]
user=root
password=LA_TUA_PASSWORD

Alternatively you can pass the options through an environment variable:

export MYSQL_OPTS="-u root -pPASSWORD -h 127.0.0.1"

For PostgreSQL, ~/.pgpass, the PG* variables or PG_OPTS work:

export PG_OPTS="-U postgres -h localhost"

How to use it

The typical flow is before, optimisation, after, comparison:

# 1. fotografia prima dell'ottimizzazione
sudo ./db_space_compare.sh snapshot prima

# 2. esegui l'ottimizzazione (OPTIMIZE TABLE, VACUUM FULL, ...)

# 3. fotografia dopo
sudo ./db_space_compare.sh snapshot dopo

# 4. confronto
sudo ./db_space_compare.sh compare prima dopo

For PostgreSQL just add the -e postgres option:

sudo ./db_space_compare.sh -e postgres snapshot prima
sudo ./db_space_compare.sh -e postgres snapshot dopo
sudo ./db_space_compare.sh -e postgres compare prima dopo

The available options are:

• -e mysql|postgres chooses the engine (default: mysql);
• -d DIR changes the folder where snapshots are saved (default: ./db_snapshots);
• -n N sets how many rows to show per section (default: 20).

If you don’t give a label, the snapshot is named after the date and time. This is handy for monitoring growth over time.

Example output

Here is a comparison on test data, built on purpose to show a typical case. The shop database shrinks after the optimisation, but the log database grows and the binlogs double.

Confronto 'prima' (2026-10-11 20:00:00)  ->  'dopo' (2026-10-11 22:00:00)
Filesystem /var: usato 46.6 GiB -> 48.4 GiB (+1.9 GiB)

=== Database - dimensione totale ===
  NOME                                                      PRIMA         DOPO    DIFFERENZA  NOTE
  log                                                    21.4 GiB     27.5 GiB      +6.1 GiB  +28.3%
  shop                                                    5.6 GiB      3.6 GiB      -2.0 GiB  -35.0%

=== Tabelle - dimensione (dati + indici) ===
  NOME                                                      PRIMA         DOPO    DIFFERENZA  NOTE
  log.eventi                                             21.4 GiB     27.5 GiB      +6.1 GiB  +28.3%
  shop.ordini                                             5.6 GiB      3.6 GiB      -2.0 GiB  -35.0%
  log.nuova                                                   0 B      1.0 KiB      +1.0 KiB  NUOVO

=== File nella data directory (cartelle DB, ibdata1, binlog, undo, WAL) ===
  NOME                                                      PRIMA         DOPO    DIFFERENZA  NOTE
  log                                                    21.4 GiB     27.5 GiB      +6.1 GiB  +28.3%
  mysql-bin.*                                             3.7 GiB      8.4 GiB      +4.7 GiB  +125.0%
  shop                                                    5.7 GiB      3.6 GiB      -2.0 GiB  -36.1%
  ibdata1                                                11.2 GiB     11.2 GiB           0 B

Here the culprit is obvious: the log.eventi table has grown by more than 6 GiB. The mysql-bin.* files went from 3.7 to 8.4 GiB. The optimisation on shop worked, but the gain was eaten up by the rest.

How to read the database disk space results

• At the top of each section are the items that changed the most. A database or table that doesn’t shrink (or even grows) after the optimisation is the first suspect.
• If the tables shrink but the filesystem doesn’t, the space is elsewhere. Look at the section on files in the data directory (binlogs, ibdata1, undo, pg_wal).
• MySQL/MariaDB: ibdata1 never shrinks, not even with OPTIMIZE TABLE. With innodb_file_per_table=OFF the value of data_free is shared and not very informative.
• PostgreSQL: a plain VACUUM does not give space back to the operating system, you need VACUUM FULL (or pg_repack). The “reclaimable” space shown is an estimate based on dead tuples. The PostgreSQL documentation on monitoring disk usage covers the underlying functions.

Monitoring database disk space over time

If the problem isn’t tied to a single optimisation but to gradual growth, it is worth automating snapshots with cron, one an hour, and then comparing any two moments:

# crontab -e (come root)
0 * * * * /usr/local/bin/db_space_compare.sh -d /var/lib/db_snapshots snapshot >/dev/null 2>&1

Without a label, every snapshot is named after the date and time (for example 20261011-140000). With list you see the available ones and with compare you compare two. Within a few days it becomes clear which database is growing abnormally.

./db_space_compare.sh -d /var/lib/db_snapshots list
./db_space_compare.sh -d /var/lib/db_snapshots compare 20261011-140000 20261012-140000

The complete script

Here is the code. Copy it into a file called db_space_compare.sh, make it executable and you are ready.

#!/usr/bin/env bash
#
# db_space_compare.sh
# Fotografa l'occupazione di spazio di tutti i database (MySQL/MariaDB o
# PostgreSQL) e confronta due fotografie (es. prima/dopo l'ottimizzazione)
# per capire quale database, tabella o file occupa spazio in modo anomalo.
#
# Uso:
#   db_space_compare.sh [opzioni] snapshot [etichetta]   (default: data-ora)
#   db_space_compare.sh [opzioni] compare  <prima> <dopo>
#   db_space_compare.sh [opzioni] report   <etichetta>
#   db_space_compare.sh [opzioni] list
#
# Opzioni:
#   -e mysql|postgres   motore del database (default: mysql)
#   -d DIR              cartella delle fotografie (default: ./db_snapshots)
#   -n N                righe mostrate per sezione (default: 20)
#
# Credenziali:
#   MySQL/MariaDB: ~/.my.cnf oppure MYSQL_OPTS="-u root -pPASSWORD -h 127.0.0.1"
#   PostgreSQL:    ~/.pgpass, variabili PG* oppure PG_OPTS="-U postgres -h localhost"
#                  (non mettere -d in PG_OPTS: lo script si collega a ogni database)
#
# Per misurare anche i file su disco (cartelle dei DB, ibdata1, binlog, undo,
# pg_wal) va eseguito come root o come utente mysql/postgres; altrimenti
# quella parte viene saltata con un avviso.
#
# Esempio:
#   sudo ./db_space_compare.sh snapshot prima
#   ... ottimizzazione (OPTIMIZE TABLE / VACUUM FULL / ...) ...
#   sudo ./db_space_compare.sh snapshot dopo
#   sudo ./db_space_compare.sh compare prima dopo

set -euo pipefail
export LC_ALL=C

ENGINE=mysql
OUTDIR=./db_snapshots
TOPN=20
SNAP_IN_PROGRESS=""

die()  { echo "ERRORE: $*" >&2; exit 1; }
warn() { echo "ATTENZIONE: $*" >&2; }
usage() { sed -n '2,/^$/{s/^# \{0,1\}//;p}' "$0"; exit "${1:-0}"; }

cleanup() {
  if [[ -n $SNAP_IN_PROGRESS ]]; then
    rm -rf -- "$SNAP_IN_PROGRESS"
    echo "Fotografia incompleta rimossa: $SNAP_IN_PROGRESS" >&2
  fi
}
trap cleanup EXIT

while getopts ":e:d:n:h" opt; do
  case $opt in
    e) ENGINE=$OPTARG ;;
    d) OUTDIR=$OPTARG ;;
    n) TOPN=$OPTARG ;;
    h) usage 0 ;;
    *) usage 1 ;;
  esac
done
shift $((OPTIND - 1))

[[ $ENGINE == mysql || $ENGINE == postgres ]] || die "motore non valido: '$ENGINE' (usa mysql o postgres)"
[[ $TOPN =~ ^[0-9]+$ ]] || die "-n deve essere un numero"

read -r -a MYSQL_ARGS <<< "${MYSQL_OPTS:-}"
read -r -a PG_ARGS    <<< "${PG_OPTS:-}"

my() { mysql "${MYSQL_ARGS[@]+"${MYSQL_ARGS[@]}"}" --batch --skip-column-names "$@"; }
pg() { psql  "${PG_ARGS[@]+"${PG_ARGS[@]}"}" -X -q -A -t -F $'\t' -v ON_ERROR_STOP=1 "$@"; }

# Funzioni awk comuni: formattazione in KiB/MiB/GiB, chiave e valore di riga.
#   nk = numero di colonne che formano il nome (1 = database, 2 = db.tabella)
#   vc = colonne da sommare per il valore (es. "4,5" = dati + indici)
AWK_LIB='
function h(b,   a, i, u) {
  split("B KiB MiB GiB TiB PiB", u, " ")
  a = (b < 0) ? -b : b; i = 1
  while (a >= 1024 && i < 6) { a /= 1024; i++ }
  if (i == 1) return sprintf("%s%d B", (b < 0 ? "-" : ""), a)
  return sprintf("%s%.1f %s", (b < 0 ? "-" : ""), a, u[i])
}
function hs(b) { return (b > 0 ? "+" : "") h(b) }
function key(   k, i) { k = $1; for (i = 2; i <= nk; i++) k = k "." $i; return k }
function val(   n, c, i, s) { n = split(vc, c, ","); s = 0; for (i = 1; i <= n; i++) s += $(c[i]); return s }
'

meta_get() { awk -F= -v k="$2" '$1 == k { sub(/^[^=]*=/, ""); print; exit }' "$1/meta.txt"; }

# ---------------------------------------------------------------- raccolta dati

# Occupazione reale su disco delle voci di primo livello della data directory:
# cartelle dei database, ibdata1, binlog, undo, redo log, pg_wal, base, ...
# I file numerati (mysql-bin.000123, relay-log.000045, ...) sono sommati in una voce.
collect_disk() {
  local d=$1 out=$2
  : > "$out"
  if [[ -z $d || ! -r $d || ! -x $d ]]; then
    warn "data directory '${d:-?}' non leggibile: salto i file su disco (esegui come root per includerli)"
    return 0
  fi
  { du -sb -- "$d"/* 2>/dev/null || true; } |
    awk -F'\t' -v p="$d/" '
      { n = $2
        if (index(n, p) == 1) n = substr(n, length(p) + 1)
        if (n ~ /\.[0-9][0-9][0-9][0-9][0-9][0-9]$/) sub(/[0-9]+$/, "*", n)
        s[n] += $1 }
      END { for (n in s) printf "%s\t%.0f\n", n, s[n] }' > "$out"
}

write_meta() {
  local dir=$1 datadir=$2 version=$3
  {
    echo "timestamp=$(date '+%Y-%m-%d %H:%M:%S')"
    echo "engine=$ENGINE"
    echo "host=$(hostname)"
    echo "version=$version"
    echo "datadir=$datadir"
    if [[ -n $datadir && -e $datadir ]]; then
      { df -P -B1 -- "$datadir" 2>/dev/null || true; } |
        awk 'NR == 2 { print "fs_mount=" $6; print "fs_used=" $3; print "fs_avail=" $4 }'
    fi
  } > "$dir/meta.txt"
}

# File prodotti da ogni fotografia (separati da TAB):
#   tables.tsv     db, tabella, engine, dati, indici, libero/recuperabile, righe
#   databases.tsv  db, dati, indici, libero/recuperabile, totale
#   disk.tsv       voce della data directory, byte su disco
#   meta.txt       data, host, versione, filesystem

snapshot_mysql() {
  local dir=$1 pre="" datadir version
  command -v mysql >/dev/null 2>&1 || die "client 'mysql' non trovato"
  my -e "SELECT 1" >/dev/null || die "connessione a MySQL/MariaDB fallita (controlla ~/.my.cnf o MYSQL_OPTS)"

  # MySQL 8 tiene in cache le statistiche di information_schema fino a 24 ore:
  # senza questo, i valori "dopo" l'ottimizzazione potrebbero essere quelli vecchi.
  if [[ -n $(my -e "SHOW VARIABLES LIKE 'information_schema_stats_expiry'") ]]; then
    pre="SET SESSION information_schema_stats_expiry = 0;"
  fi

  my -e "$pre
    SELECT table_schema, table_name, IFNULL(engine, '-'),
           IFNULL(data_length, 0), IFNULL(index_length, 0), IFNULL(data_free, 0), IFNULL(table_rows, 0)
      FROM information_schema.TABLES
     WHERE table_type IN ('BASE TABLE', 'SYSTEM VERSIONED')
       AND table_schema NOT IN ('information_schema', 'performance_schema', 'sys')" > "$dir/tables.tsv"

  my -e "$pre
    SELECT s.schema_name,
           IFNULL(SUM(t.data_length), 0), IFNULL(SUM(t.index_length), 0), IFNULL(SUM(t.data_free), 0),
           IFNULL(SUM(t.data_length), 0) + IFNULL(SUM(t.index_length), 0)
      FROM information_schema.SCHEMATA s
      LEFT JOIN information_schema.TABLES t
             ON t.table_schema = s.schema_name
            AND t.table_type IN ('BASE TABLE', 'SYSTEM VERSIONED')
     WHERE s.schema_name NOT IN ('information_schema', 'performance_schema', 'sys')
     GROUP BY s.schema_name" > "$dir/databases.tsv"

  datadir=$(my -e "SELECT @@datadir"); datadir=${datadir%/}
  version=$(my -e "SELECT VERSION()")
  collect_disk "$datadir" "$dir/disk.tsv"
  write_meta "$dir" "$datadir" "$version"
}

snapshot_postgres() {
  local dir=$1 maindb=${PGDATABASE:-postgres} datadir version db
  command -v psql >/dev/null 2>&1 || die "client 'psql' non trovato"
  pg -d "$maindb" -c "SELECT 1" >/dev/null || die "connessione a PostgreSQL fallita (controlla PG_OPTS, PGUSER, ~/.pgpass)"

  pg -d "$maindb" -c "SELECT datname FROM pg_database WHERE datallowconn AND NOT datistemplate ORDER BY 1" > "$dir/.dblist"

  # Per ogni database: dimensione di ogni tabella (catalogo incluso, es. pg_largeobject)
  # e stima dello spazio occupato da tuple morte (bloat recuperabile con VACUUM FULL).
  : > "$dir/tables.tsv"
  while IFS= read -r db; do
    [[ -n $db ]] || continue
    pg -d "$db" -c "
      SELECT current_database(), s.schemaname || '.' || s.relname, 'table',
             pg_table_size(s.relid), pg_indexes_size(s.relid),
             CASE WHEN s.n_live_tup + s.n_dead_tup > 0
                  THEN (pg_table_size(s.relid)::numeric * s.n_dead_tup / (s.n_live_tup + s.n_dead_tup))::bigint
                  ELSE 0 END,
             s.n_live_tup
        FROM pg_stat_all_tables s
       WHERE s.schemaname <> 'information_schema'
         AND s.schemaname NOT LIKE 'pg_toast%'" < /dev/null >> "$dir/tables.tsv" \
      || warn "impossibile leggere le tabelle del database '$db'"
  done < "$dir/.dblist"

  pg -d "$maindb" -c "SELECT datname, pg_database_size(oid) FROM pg_database WHERE datallowconn AND NOT datistemplate" > "$dir/.dbsize"
  awk -F'\t' '
    FILENAME == ARGV[1] { d[$1] += $4; i[$1] += $5; f[$1] += $6; next }
    { printf "%s\t%.0f\t%.0f\t%.0f\t%s\n", $1, d[$1], i[$1], f[$1], $2 }' \
    "$dir/tables.tsv" "$dir/.dbsize" > "$dir/databases.tsv"

  datadir=$(pg -d "$maindb" -c "SHOW data_directory" 2>/dev/null || true)
  version=$(pg -d "$maindb" -c "SHOW server_version")
  collect_disk "$datadir" "$dir/disk.tsv"

  # Dettaglio di base/<oid> con il nome del database al posto dell'oid
  if [[ -n $datadir && -r $datadir/base && -x $datadir/base ]]; then
    pg -d "$maindb" -c "SELECT oid, datname FROM pg_database" > "$dir/.oidmap"
    { du -sb -- "$datadir"/base/* 2>/dev/null || true; } |
      awk -F'\t' '
        FILENAME == ARGV[1] { name[$1] = $2; next }
        { o = $2; sub(/.*\//, "", o); printf "base/%s\t%s\n", ((o in name) ? name[o] : o), $1 }' \
        "$dir/.oidmap" - >> "$dir/disk.tsv"
  fi
  write_meta "$dir" "$datadir" "$version"
}

# --------------------------------------------------------------- presentazione

print_fs() {
  local d=$1 used avail mount
  used=$(meta_get "$d" fs_used); avail=$(meta_get "$d" fs_avail); mount=$(meta_get "$d" fs_mount)
  [[ -n $used ]] || return 0
  awk -v u="$used" -v a="$avail" -v m="$mount" "$AWK_LIB"'
    BEGIN { printf "Filesystem %s: usato %s, libero %s\n", m, h(u), h(a) }'
}

top_section() {
  local f=$1 nk=$2 vc=$3 title=$4
  echo
  echo "=== $title ==="
  if [[ ! -s $f ]]; then echo "  (nessun dato)"; return 0; fi
  awk -F'\t' -v nk="$nk" -v vc="$vc" "$AWK_LIB"'
    { v = val(); printf "%.0f\t%s\t%s\n", v, key(), h(v) }' "$f" |
    sort -t $'\t' -k1,1nr -k2,2 |
    awk -F'\t' -v top="$TOPN" 'NR <= top && $1 > 0 { printf "  %-60s %12s\n", $2, $3 }'
}

compare_section() {
  local fa=$1 fb=$2 nk=$3 vc=$4 title=$5
  [[ -f $fa ]] || fa=/dev/null
  [[ -f $fb ]] || fb=/dev/null
  echo
  echo "=== $title ==="
  if [[ ! -s $fa && ! -s $fb ]]; then echo "  (nessun dato)"; return 0; fi
  printf '  %-50s %12s %12s %13s  %s\n' "NOME" "PRIMA" "DOPO" "DIFFERENZA" "NOTE"
  awk -F'\t' -v nk="$nk" -v vc="$vc" "$AWK_LIB"'
    FILENAME == ARGV[1] { k = key(); A[k] = val(); S[k] = 1; next }
                        { k = key(); B[k] = val(); S[k] = 1 }
    END {
      for (k in S) {
        inA = (k in A); inB = (k in B)
        a = inA ? A[k] : 0; b = inB ? B[k] : 0; d = b - a
        if (!inA)                 note = "NUOVO"
        else if (!inB)            note = "RIMOSSO"
        else if (a > 0 && d != 0) note = sprintf("%+.1f%%", 100 * d / a)
        else                      note = ""
        printf "%.0f\t%s\t%s\t%s\t%s\t%s\n", (d < 0 ? -d : d), k, h(a), h(b), hs(d), note
      }
    }' "$fa" "$fb" |
    sort -t $'\t' -k1,1nr -k2,2 |
    awk -F'\t' -v top="$TOPN" 'NR <= top { printf "  %-50s %12s %12s %13s  %s\n", $2, $3, $4, $5, $6 }'
}

# -------------------------------------------------------------------- comandi

cmd_snapshot() {
  local label=${1:-$(date '+%Y%m%d-%H%M%S')}
  [[ $label =~ ^[A-Za-z0-9][A-Za-z0-9._-]*$ ]] || die "etichetta non valida: usa solo lettere, numeri, . _ -"
  local dir=$OUTDIR/$label
  [[ ! -e $dir ]] || die "la fotografia '$label' esiste già ($dir)"
  mkdir -p -- "$dir"
  SNAP_IN_PROGRESS=$dir
  echo "Fotografia '$label' ($ENGINE) in corso..."
  "snapshot_$ENGINE" "$dir"
  rm -f -- "$dir/.dblist" "$dir/.dbsize" "$dir/.oidmap"
  SNAP_IN_PROGRESS=""
  echo "Salvata in $dir ($(wc -l < "$dir/tables.tsv") tabelle)"
  print_fs "$dir"
  top_section "$dir/databases.tsv" 1 5 "Database più grandi"
}

cmd_report() {
  local label=${1:-}
  [[ -n $label ]] || die "uso: report <etichetta>"
  local d=$OUTDIR/$label
  [[ -f $d/meta.txt ]] || die "fotografia '$label' non trovata in $OUTDIR"
  echo "Fotografia '$label' del $(meta_get "$d" timestamp) - $(meta_get "$d" engine) $(meta_get "$d" version) su $(meta_get "$d" host)"
  print_fs "$d"
  top_section "$d/databases.tsv" 1 5   "Database più grandi (dati + indici)"
  top_section "$d/tables.tsv"    2 4,5 "Tabelle più grandi (dati + indici)"
  top_section "$d/tables.tsv"    2 6   "Tabelle con più spazio recuperabile (data_free / tuple morte stimate)"
  top_section "$d/disk.tsv"      1 2   "Voci più grandi nella data directory"
}

cmd_compare() {
  local a=${1:-} b=${2:-}
  [[ -n $a && -n $b ]] || die "uso: compare <etichetta_prima> <etichetta_dopo>"
  [[ $a != "$b" ]] || die "le due etichette sono uguali"
  local da=$OUTDIR/$a db=$OUTDIR/$b
  [[ -f $da/meta.txt ]] || die "fotografia '$a' non trovata in $OUTDIR"
  [[ -f $db/meta.txt ]] || die "fotografia '$b' non trovata in $OUTDIR"
  [[ $(meta_get "$da" engine) == "$(meta_get "$db" engine)" ]] || warn "le due fotografie sono di motori diversi"

  echo "Confronto '$a' ($(meta_get "$da" timestamp))  ->  '$b' ($(meta_get "$db" timestamp))"
  local ua ub m
  ua=$(meta_get "$da" fs_used); ub=$(meta_get "$db" fs_used); m=$(meta_get "$db" fs_mount)
  if [[ -n $ua && -n $ub ]]; then
    awk -v a="$ua" -v b="$ub" -v m="$m" "$AWK_LIB"'
      BEGIN { printf "Filesystem %s: usato %s -> %s (%s)\n", m, h(a), h(b), hs(b - a) }'
  fi

  compare_section "$da/databases.tsv" "$db/databases.tsv" 1 5   "Database - dimensione totale"
  compare_section "$da/databases.tsv" "$db/databases.tsv" 1 4   "Database - spazio libero/recuperabile"
  compare_section "$da/tables.tsv"    "$db/tables.tsv"    2 4,5 "Tabelle - dimensione (dati + indici)"
  compare_section "$da/disk.tsv"      "$db/disk.tsv"      1 2   "File nella data directory (cartelle DB, ibdata1, binlog, undo, WAL)"

  cat <<'EOF'

Come leggere il risultato:
  - In cima a ogni sezione ci sono le voci cambiate di più: un database o una
    tabella che dopo l'ottimizzazione non cala, o addirittura cresce, è il
    primo sospetto.
  - Se le tabelle calano ma il filesystem no, lo spazio sta altrove: guarda la
    sezione "File nella data directory" (binlog, ibdata1, undo, pg_wal).
  - MySQL/MariaDB: ibdata1 non si restringe mai, nemmeno con OPTIMIZE; con
    innodb_file_per_table=OFF il data_free è condiviso e poco indicativo.
  - PostgreSQL: VACUUM semplice non restituisce spazio al disco, serve
    VACUUM FULL (o pg_repack); lo spazio "recuperabile" è una stima.
EOF
}

cmd_list() {
  if [[ ! -d $OUTDIR ]]; then echo "Nessuna fotografia in $OUTDIR"; return 0; fi
  local d found=0
  for d in "$OUTDIR"/*/; do
    [[ -f $d/meta.txt ]] || continue
    found=1
    printf '%-24s %s  %-8s  %s\n' "$(basename "$d")" "$(meta_get "$d" timestamp)" \
      "$(meta_get "$d" engine)" "$(meta_get "$d" host)"
  done
  [[ $found -eq 1 ]] || echo "Nessuna fotografia in $OUTDIR"
}

cmd=${1:-}
shift || true
case $cmd in
  snapshot) cmd_snapshot "$@" ;;
  compare)  cmd_compare "$@" ;;
  report)   cmd_report "$@" ;;
  list)     cmd_list ;;
  ""|help)  usage 0 ;;
  *)        die "comando sconosciuto: '$cmd' (usa snapshot, compare, report, list)" ;;
esac

Final notes on database disk space

• The script is read-only on the databases: it only runs SELECT on the metadata. It writes only to the snapshots folder.
• On MySQL 8 it sets information_schema_stats_expiry = 0 in the session. Otherwise the cached statistics (up to 24 hours) could show “old” values even after the optimisation.
• If a snapshot fails halfway, it is removed automatically so as not to leave incomplete data.
• Before using it in production, try it in a test environment and check the database user’s permissions.

If you use it and find a case it doesn’t cover, write to me in the comments. The same goes for ideas to extend it (other engines, CSV export, email alerts). If the problem is a MySQL that won’t start rather than a full disk, see MySQL won’t start after a Debian 12 upgrade.

The post Which database is using up your disk space? A script for MySQL and PostgreSQL first appeared on Matteo Cisilino.

We needed MySQL 5.7 in Docker, a legacy database on a server that otherwise runs a recent distribution. Some legacy applications are not compatible with MySQL 8. The reasons: a change of default authentication plugin, stricter SQL modes, new reserved words. Migrating them quickly was not an option.

Installing such an old version directly on the host brings several problems. The official repositories of current distributions no longer offer it. Third-party packages risk conflicting with the rest of the system. An OS upgrade could break it.

So we decided to isolate it in a Docker container: the host stays clean and up to date. The database version is fixed and reproducible. Data and configuration live in a folder that is easy to back up and move.

This article tells how we set it up and, above all, the obstacles we met along the way. First the unintentional exposure of port 3306, then an iptables error at reboot. Last, writing a firewall with nftables that lets through only authorised IPs.

Note: MySQL 5.7 reached end of life (EOL) in October 2023 and no longer receives security patches. It is a stopgap: the goal remains to migrate to MySQL 8.x or MariaDB. That is exactly why it is even more important not to leave it exposed.

1. Folder structure

/opt/mysql57/
├── docker-compose.yml
├── .env
├── conf.d/        # configurazioni personalizzate (my.cnf aggiuntivi)
├── data/          # dati del database (volume persistente)
└── logs/

The .env file holds the credentials, so they don’t end up in the compose file:

TZ=Europe/Rome
MYSQL_ROOT_PASSWORD=una_password_molto_lunga_e_casuale
MYSQL_DATABASE=nome_db
MYSQL_USER=nome_utente
MYSQL_PASSWORD=altra_password_robusta

Protect it with chmod 600 .env.

2. MySQL 5.7 in Docker: the docker-compose.yml

services:
  mysql57:
    image: mysql:5.7
    container_name: mysql57
    restart: unless-stopped
    env_file:
      - .env
    ports:
      - "3306:3306"
    environment:
      TZ: "${TZ:-Europe/Rome}"
      MYSQL_ROOT_PASSWORD: "${MYSQL_ROOT_PASSWORD}"
      MYSQL_DATABASE: "${MYSQL_DATABASE}"
      MYSQL_USER: "${MYSQL_USER}"
      MYSQL_PASSWORD: "${MYSQL_PASSWORD}"
    command:
      - --default-authentication-plugin=mysql_native_password
    volumes:
      - ./data:/var/lib/mysql
      - ./conf.d:/etc/mysql/conf.d:ro
      - ./logs:/var/log/mysql
    healthcheck:
      test: ["CMD-SHELL", "mysqladmin ping -h 127.0.0.1 -uroot -p$$MYSQL_ROOT_PASSWORD --silent"]
      interval: 20s
      timeout: 5s
      retries: 10
      start_period: 45s
    ulimits:
      nofile:
        soft: 65535
        hard: 65535

A few choices worth noting:

• restart: unless-stopped: the database restarts by itself after a server reboot.
• Bind-mount volumes: data, configuration and logs are on the host, so backups and migrations are simple folder copies.
• Healthcheck: lets other services wait until MySQL is really ready ($$ makes the variable expand inside the container and not in Compose).
• ulimits nofile: avoids the “too many open files” error under load.

3. First problem: the port is open to the whole world

The line - "3306:3306" is equivalent to 0.0.0.0:3306:3306. Docker publishes the port on all interfaces of the host, including the public one.

The most insidious part is that Docker writes its own rules straight into the kernel firewall. These are evaluated before those of UFW or firewalld. Having a firewall running is not enough: the port stays reachable. Docker’s own documentation on packet filtering and firewalls describes this behaviour.

The possible approaches are:

1. Localhost only: "127.0.0.1:3306:3306", if the database is only used by the host or through an SSH tunnel.
2. No published port: if the clients are other containers, a shared Docker network is enough.
3. Specific IP: publish only on an address of a private network/VPN.
4. Public port with IP filtering: our case, because an external client (a phpMyAdmin instance on another server) has to be able to connect.

For case 4 you need a firewall that correctly filters the traffic of the ports published by Docker.

4. Second problem: an iptables error at startup

After touching the firewall, docker compose up returned:

failed to create network mysql57_default: Error response from daemon:
Failed to Setup IP tables: Unable to enable ACCEPT OUTGOING rule:
(iptables failed: iptables --wait -t filter -A DOCKER-FORWARD -i br-... -j ACCEPT:
iptables: No chain/target/match by that name.

The error doesn’t depend on the compose file: the DOCKER-FORWARD chain (introduced in recent Docker versions) no longer existed. It had been deleted by something that cleaned up the rules after the daemon started. Restarting Docker (systemctl restart docker) recreates it, but the problem comes back until the real culprit is removed.

The culprit: flush ruleset

Our /etc/nftables.conf began with:

flush ruleset

This command deletes all nftables tables, including those created by Docker. On top of that the file contained a static copy of Docker’s rules. The bridge name (br-278550235c3f) and the container IP (172.18.0.2) were written by hand. But the bridge name changes every time the Compose network is recreated, so the copy immediately became obsolete. The ip nat table also clashed with the one managed by Docker.

Golden rule: with Docker, in the nftables file you don’t use flush ruleset and you don’t copy its rules. You manage only your own firewall, in dedicated tables.

5. The nftables firewall for MySQL 5.7 in Docker

The principle is this: the ports published by Docker are rewritten with DNAT and then forwarded by the host to the container, so they go through the forward chain, not input. A drop rule in a chain of ours, forward, with a slightly lower priority (so evaluated first). It drops the packet regardless of Docker’s accept rules.

To avoid touching other people’s tables we use the table ... + delete table ... pattern: it makes the file re-runnable without errors and without flush.

/etc/nftables.conf:

#!/usr/sbin/nft -f

# NIENTE "flush ruleset": cancellerebbe anche le tabelle di Docker

# ---------------- firewall host ----------------
table inet filter
delete table inet filter

table inet filter {

    chain input {
        type filter hook input priority filter; policy drop;
        ct state established,related accept
        iifname "lo" accept
        ip6 nexthdr icmpv6 accept
        iifname "enp7s0" accept                       # rete privata
        ip saddr 203.0.113.20 tcp dport 22 accept     # SSH da host fidato
        ip saddr 203.0.113.30 tcp dport 22 accept     # SSH da VPN
        ip saddr 203.0.113.20 icmp type echo-request accept
        tcp dport { 80, 443 } accept
    }

    chain output {
        type filter hook output priority filter; policy accept;
    }
}

# ---------------- guardia MySQL (porta pubblicata da Docker) ----------------
table ip mysql_guard
delete table ip mysql_guard

table ip mysql_guard {
    set allowed {
        type ipv4_addr
        flags interval
        elements = {
            203.0.113.10        # phpMyAdmin
        }
    }

    chain forward {
        type filter hook forward priority filter - 5; policy accept;
        iifname "enp7s0" accept
        ct status dnat tcp dport 3306 ip saddr @allowed accept
        ct status dnat tcp dport 3306 counter drop
    }
}

How the guard works:

• ct status dnat selects only packets that went through DNAT, that is those headed to the ports published by Docker.
• The first rule accepts whoever is in the allowed set; the second drops everyone else and counts them (counter).
• Local traffic (127.0.0.1) and traffic between containers on the same network doesn’t go through the router’s forward. So it isn’t touched.

Compatibility: in a first version we used ct original proto-dst 3306, but some versions of nft answer with “Can’t parse symbolic invalid expressions”. In the forward hook the DNAT has already happened and the container port matches the published one (3306:3306). So tcp dport 3306 combined with ct status dnat is equivalent and works everywhere. If you change the mapping (e.g. 3307:3306), the rule must be adapted to the internal port.

Applying the changes without locking yourself out

Keep a spare SSH session open, then:

nft -c -f /etc/nftables.conf     # solo verifica della sintassi
nft delete table ip nat          # una tantum: elimina la vecchia copia statica
nft -f /etc/nftables.conf
systemctl restart docker         # Docker ricrea le sue tabelle pulite
cd /opt/mysql57 && docker compose up -d

For loading at boot, make sure the service is enabled (systemctl enable --now nftables) and that /etc/nftables.conf does not contain flush ruleset.

Managing IPs at runtime

nft add element ip mysql_guard allowed { 203.0.113.50 }
nft delete element ip mysql_guard allowed { 203.0.113.50 }

To make it permanent, also update the set in the file.

6. Verification

From the authorised server:

nc -zv IP_PUBBLICO_DB 3306        # deve rispondere "succeeded"

From any other IP:

nmap -Pn -p 3306 IP_PUBBLICO_DB   # deve risultare "filtered" / timeout

On the server, the drop rule’s counter must grow with every unauthorised attempt:

nft list chain ip mysql_guard forward
ss -tlnp | grep 3306

7. Second barrier: the MySQL users

The firewall doesn’t replace database permissions. Create the user with the host limited to the client’s IP:

CREATE USER 'pma_user'@'203.0.113.10' IDENTIFIED BY 'password_robusta';
GRANT ALL ON nome_db.* TO 'pma_user'@'203.0.113.10';
FLUSH PRIVILEGES;

If the TCP connection succeeds but MySQL answers Access denied, the problem is here and not in the firewall.

8. Things to remember about MySQL 5.7 in Docker

• Docker and the host firewall don’t ignore each other. Understanding which chain the traffic goes through (input or forward) is the key to everything.
• Never flush ruleset on a host with Docker; use dedicated tables.
• Don’t copy the rules generated by Docker into your own files: bridge names and IPs change.
• If you have already exposed the port, check the logs in ./logs and consider changing the passwords, especially root’s.
• Where possible, prefer a VPN (WireGuard) or SSH tunnel to exposing a database publicly.
• Plan the migration from MySQL 5.7 to a supported version.

Do you have questions or a different solution to the same problem? Write it in the comments. If you run MySQL on Debian, see also MySQL won’t start after a Debian 12 upgrade.

The post MySQL 5.7 in Docker protected with nftables first appeared on Matteo Cisilino.

MySQL won’t start: the service shows as failed and the log reports the error Can't read dir of '/etc/mysql/mariadb.conf.d/':

# service mysql status
× mysql.service - MySQL Community Server
     Loaded: loaded (/lib/systemd/system/mysql.service; enabled; preset: enabled)
     Active: failed (Result: exit-code)
     Status: "Server startup in progress"

mysqld: Can't read dir of '/etc/mysql/mariadb.conf.d/' (OS errno 2 - No such file or directory)
mysqld: [ERROR] Stopped processing the 'includedir' directive in file /etc/mysql/my.cnf at line 29.
mysqld: [ERROR] Fatal error in defaults handling. Program aborted!
systemd[1]: Failed to start mysql.service - MySQL Community Server.

Checking the installed packages, MySQL is there as expected (mysql-community-server 8.0 packages from the official Oracle repository):

dpkg -l | grep mysql

The cause: the my.cnf alternative points to mariadb.cnf

On Debian the file /etc/mysql/my.cnf is not a real file but a link managed by update-alternatives. Several packages can register their own version and, in automatic mode, the one with the highest priority wins.

During the upgrade the mariadb-common package was installed (and later removed), which registers /etc/mysql/mariadb.cnf with priority 500, higher than /etc/mysql/mysql.cnf (priority 300). The package was removed but not fully purged (state rc), so its configuration stayed active. On line 29 of that file there is an include of /etc/mysql/mariadb.conf.d/, a directory that does not exist. MySQL reads the configuration, cannot find the directory and stops, which is why MySQL won’t start.

Good news: the data in /var/lib/mysql has not been touched. It is only a configuration problem.

Diagnosis: check where my.cnf points

ls -l /etc/mysql/my.cnf /etc/alternatives/my.cnf
update-alternatives --display my.cnf
sed -n '25,32p' /etc/mysql/my.cnf
dpkg -l | grep -i mariadb

If this is the problem, the output will look like this:

/etc/alternatives/my.cnf -> /etc/mysql/mariadb.cnf
/etc/mysql/my.cnf -> /etc/alternatives/my.cnf

my.cnf - modalità automatica
  la versione migliore del collegamento è /etc/mysql/mariadb.cnf
  il collegamento attualmente punta a /etc/mysql/mariadb.cnf
/etc/mysql/mariadb.cnf - priorità 500
/etc/mysql/my.cnf.fallback - priorità 100
/etc/mysql/mysql.cnf - priorità 300

!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mariadb.conf.d/

rc  mariadb-common   1:10.11.18-0+deb12u1   all   MariaDB common configuration files

Step-by-step fix

1. Back up the configuration

cp -a /etc/mysql /root/etc-mysql-backup-$(date +%F)

2. Remove the mariadb-common leftovers

The purge removes mariadb.cnf and its entry in update-alternatives:

apt purge mariadb-common

3. Point my.cnf back to mysql.cnf

update-alternatives --remove my.cnf /etc/mysql/mariadb.cnf 2>/dev/null
update-alternatives --set my.cnf /etc/mysql/mysql.cnf

4. Check the configuration

update-alternatives --display my.cnf
cat /etc/mysql/my.cnf
my_print_defaults mysqld

Now my.cnf must point to /etc/mysql/mysql.cnf and include /etc/mysql/conf.d/ and /etc/mysql/mysql.conf.d/. If my_print_defaults mysqld prints the parameters without errors, the configuration is correct.

5. Start MySQL

systemctl start mysql
systemctl status mysql
journalctl -u mysql -n 50 --no-pager

If MySQL won’t start and my.cnf is not a symbolic link

If /etc/mysql/my.cnf is a regular file or /etc/mysql/mysql.cnf does not exist, you can recreate the configuration by hand:

mv /etc/mysql/my.cnf /etc/mysql/my.cnf.mariadb.bak
cat > /etc/mysql/my.cnf <<'EOF'
!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mysql.conf.d/
EOF
mkdir -p /etc/mysql/conf.d
systemctl start mysql

Quick temporary fix when MySQL won’t start

If you need to bring the database back online right away, just create the missing directory. It is only a stopgap, though. MySQL would keep reading the configuration meant for MariaDB. Better to apply the complete fix as soon as possible.

mkdir /etc/mysql/mariadb.conf.d && systemctl start mysql

How to stop it happening again

It is worth finding out which upgrade installed MariaDB, by looking at the apt history:

grep -i -B3 -A3 mariadb /var/log/apt/history.log*

To stop mariadb-common being reinstalled in the future you can block it with an APT pin at a negative priority:

cat > /etc/apt/preferences.d/no-mariadb <<'EOF'
Package: mariadb-*
Pin: release *
Pin-Priority: -1
EOF

After every major upgrade it is also good practice to check the state of the alternative with update-alternatives --display my.cnf. Before the next upgrade, read the Debian 12 release notes for known issues. A related case is MySQL 5.7 in Docker protected with nftables.

Frequently asked questions

Will I lose my database data?

No. The problem only concerns the configuration files in /etc/mysql; the data in /var/lib/mysql stays intact.

Why did Debian choose mariadb.cnf?

Because the my.cnf alternative was in automatic mode and mariadb.cnf has priority 500, higher than the 300 of mysql.cnf. With update-alternatives --set the choice becomes manual and no longer changes by itself.

Does it apply to Ubuntu too?

Yes, the update-alternatives mechanism for my.cnf is the same on Ubuntu and on the other Debian-derived distributions.

Conclusion

The error “Can’t read dir of /etc/mysql/mariadb.conf.d/” after a Debian upgrade does not mean MySQL is damaged. It means a leftover MariaDB configuration has stayed active through update-alternatives. By removing mariadb-common with apt purge and setting my.cnf back to mysql.cnf, MySQL works normally again. If MySQL won’t start after an upgrade, check update-alternatives first.

The post MySQL won’t start after a Debian 12 upgrade: cause and fix first appeared on Matteo Cisilino.