PostgreSQL Point-in-Time-Recovery bei Ups-Queries
Bei unseren Arbeiten zum Thema “PostgreSQL für Delfine und Seelöwen” bin ich darauf gestossen, dass im PostgreSQL Universum um das Thema Point-in-Time-Recovery auf ein spezifisches Statement/eine spezifische Transaktion ein Bogen gemacht wird. Und wenn man Beispiele findet, werden vor allem DDL und nicht DML Statements betrachtet.
If you want to recover to some previous point in time (say, right before the junior DBA dropped your main transaction table), just specify the required stopping point. You can specify the stop point, known as the “recovery target”, either by date/time, named restore point or by completion of a specific transaction ID. As of this writing only the date/time and named restore point options are very usable, since there are no tools to help you identify with any accuracy which transaction ID to use. [ 1 ]
In dem Universum, aus dem ich komme, kennt man das anders. Hier ist es absolut kein Problem ein Point-in-Time-Recovery auf ein Statement oder eine Transaktion genau zu machen. Sofern man sein Handwerk beherrscht.
Jetzt sticht mich aber der Hafer, ob das in dieser, für mich neuen Welt, nicht auch geht…?
Die Testumgebung
Wie aus unseren Schulungen bekannt, machen wir solche Experimente auf unserer test Tabelle immer unter Last (insert_test.sh):
test=# DELETE FROM test WHERE id BETWEEN 10 AND 20;
DELETE 11
Der Alarm
Jetzt kommt die Meldung an den DBA: “Ups! Es ist etwas kaputt gegangen! Kannst Du das bitte wieder reparieren?”
Welche Informationen brauchen wir also, um hier helfen zu können?
- Wann ist es, möglichst genau, kaputt gegangen?
- Am 22. Juli, ca. 10:55/CEST ➜ Achtung: PostgreSQL scheint hier immer in UTC zu arbeiten, also: 08:55/UTC.
- Wie, mit welchem Statement und auf welche Datenbank ist es kaputt gegangen?
delete from test where id between 10 and 20- Datenbank:
test
Neben den üblichen Diskussionen über Datenbank stilllegen, Applikation stilllegen, etc. damit nicht noch mehr Unheil angerichtet wird, geht es jetzt darum, wie man die Datenbank wieder in den Zustand von vor dem Ups-Query bringt.
Vorbereitungen
Um herauszufinden, wo GENAU wir beim Point-in-Time-Recovery stoppen wollen, brauchen wir die archivierten WAL Files. Und mit dem Tool pg_waldump gucken wir dann in die WAL Files rein.
Auf noch “aktive” WAL Files sollten wir nicht zugreifen,
Can give wrong results when the server is running. [ 2 ]
das gibt potentiell falsche Resultate und Fehler:
$ pg_waldump 000000010000000000000033 >/dev/null
pg_waldump: error: error in WAL record at 0/330B6CF8: invalid record length at 0/330B6D20: expected at least 24, got 0
$ pg_waldump 000000010000000000000034 >/dev/null
pg_waldump: error: could not find a valid record after 0/34000000
Um das sicher zu stellen rotieren wir das aktuelle WAL File weg, damit es archiviert wird:
postgres=# SELECT pg_switch_wal();
pg_switch_wal
---------------
0/3228AB50
Zuerst wollen wir wissen, welche WAL Files in dem betreffendem Zeitraum in Betracht kommen (Achtung: Server läuft ebenfalls in UTC!):
$ PGDATA='/var/lib/postgresql/18/main'
$ WALARCHIVE='/mnt/backup/wal_archive'
$ PATH="${PATH}:/usr/lib/postgresql/18/bin"
$ ll ${WALARCHIVE}/????????????????????????
-rw------- 1 postgres postgres 16777216 Jul 22 08:32 /mnt/backup/wal_archive/00000001000000000000002F
-rw------- 1 postgres postgres 16777216 Jul 22 08:47 /mnt/backup/wal_archive/000000010000000000000030
-rw------- 1 postgres postgres 16777216 Jul 22 09:02 /mnt/backup/wal_archive/000000010000000000000031
-rw------- 1 postgres postgres 16777216 Jul 22 09:11 /mnt/backup/wal_archive/000000010000000000000032
Etwas genauer sehen wir es, wen wir in die WAL Files reinschauen (wahrscheinlich geht das eleganter?):
$ for file in ${WALARCHIVE}/???????????????????????? ; do
echo ${file}
pg_waldump ${file} | head | grep COMMIT
done
/mnt/backup/wal_archive/000000010000000000000030
rmgr: Transaction len (rec/tot): 34/ 34, tx: 109112, lsn: 0/300000E0, prev 0/300000A0, desc: COMMIT 2026-07-22 08:32:31.317115 UTC
rmgr: Transaction len (rec/tot): 34/ 34, tx: 109113, lsn: 0/300001C0, prev 0/30000180, desc: COMMIT 2026-07-22 08:32:31.368472 UTC
rmgr: Transaction len (rec/tot): 34/ 34, tx: 109114, lsn: 0/300002A0, prev 0/30000260, desc: COMMIT 2026-07-22 08:32:31.427410 UTC
/mnt/backup/wal_archive/000000010000000000000031
rmgr: Transaction len (rec/tot): 34/ 34, tx: 127114, lsn: 0/31001D78, prev 0/31000FF8, desc: COMMIT 2026-07-22 08:47:31.912547 UTC
rmgr: Transaction len (rec/tot): 34/ 34, tx: 127115, lsn: 0/31001E58, prev 0/31001E18, desc: COMMIT 2026-07-22 08:47:31.971700 UTC
rmgr: Transaction len (rec/tot): 34/ 34, tx: 127116, lsn: 0/31001F38, prev 0/31001EF8, desc: COMMIT 2026-07-22 08:47:32.026471 UTC
/mnt/backup/wal_archive/000000010000000000000032
rmgr: Transaction len (rec/tot): 34/ 34, tx: 145081, lsn: 0/320000E0, prev 0/320000A0, desc: COMMIT 2026-07-22 09:02:31.648356 UTC
rmgr: Transaction len (rec/tot): 34/ 34, tx: 145082, lsn: 0/320001C0, prev 0/32000180, desc: COMMIT 2026-07-22 09:02:31.689970 UTC
rmgr: Transaction len (rec/tot): 34/ 34, tx: 145083, lsn: 0/320002A0, prev 0/32000260, desc: COMMIT 2026-07-22 09:02:31.749528 UTC
Wenn die Zeitangabe zur Katastrophe stimmt (08:55/CEST), dann müsste das Statement in WAL 000000010000000000000031 liegen…
Zur Sicherheit machen wir einen kurzen Check:
$ pg_waldump ${WALARCHIVE}/000000010000000000000031 | grep DELETE
rmgr: Heap len (rec/tot): 59/ 8179, tx: 135586, lsn: 0/312DF748, prev 0/312DF720, desc: DELETE xmax: 135586, off: 10, infobits: [KEYS_UPDATED], flags: 0x01, blkref #0: rel 1663/21506/21508 blk 0 FPW
rmgr: Heap len (rec/tot): 54/ 54, tx: 135586, lsn: 0/312E1758, prev 0/312DF748, desc: DELETE xmax: 135586, off: 11, infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/21506/21508 blk 0
rmgr: Heap len (rec/tot): 54/ 54, tx: 135586, lsn: 0/312E1790, prev 0/312E1758, desc: DELETE xmax: 135586, off: 12, infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/21506/21508 blk 0
rmgr: Heap len (rec/tot): 54/ 54, tx: 135586, lsn: 0/312E17C8, prev 0/312E1790, desc: DELETE xmax: 135586, off: 13, infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/21506/21508 blk 0
rmgr: Heap len (rec/tot): 54/ 54, tx: 135586, lsn: 0/312E1800, prev 0/312E17C8, desc: DELETE xmax: 135586, off: 14, infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/21506/21508 blk 0
rmgr: Heap len (rec/tot): 54/ 54, tx: 135586, lsn: 0/312E1838, prev 0/312E1800, desc: DELETE xmax: 135586, off: 15, infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/21506/21508 blk 0
rmgr: Heap len (rec/tot): 54/ 54, tx: 135586, lsn: 0/312E1870, prev 0/312E1838, desc: DELETE xmax: 135586, off: 16, infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/21506/21508 blk 0
rmgr: Heap len (rec/tot): 54/ 54, tx: 135586, lsn: 0/312E18A8, prev 0/312E1870, desc: DELETE xmax: 135586, off: 17, infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/21506/21508 blk 0
rmgr: Heap len (rec/tot): 54/ 54, tx: 135586, lsn: 0/312E18E0, prev 0/312E18A8, desc: DELETE xmax: 135586, off: 18, infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/21506/21508 blk 0
rmgr: Heap len (rec/tot): 54/ 54, tx: 135586, lsn: 0/312E1918, prev 0/312E18E0, desc: DELETE xmax: 135586, off: 19, infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/21506/21508 blk 0
rmgr: Heap len (rec/tot): 54/ 54, tx: 135586, lsn: 0/312E1950, prev 0/312E1918, desc: DELETE xmax: 135586, off: 20, infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/21506/21508 blk 0
Glück gehabt! Da gibt es DELETEs drin und zwar genau 11! Wir könnten also schon nahe dran sein…
Um genauer filtern zu können brauchen wir jetzt noch:
- die OID des Tablespaces
- die OID der Datenbank
- die OID der Tabelle
postgres=# SELECT oid, spcname AS tablespace, spcowner::regrole AS owner
FROM pg_tablespace
;
oid | tablespace | owner
------+------------+----------
1663 | pg_default | postgres
1664 | pg_global | postgres
postgres=# SELECT oid, datname AS database
FROM pg_database
WHERE oid = 21506
;
oid | database
-------+-----------
5 | postgres
16388 | enswitch
1 | template1
4 | template0
21498 | oli
21506 | test
\connect test
test=# SELECT c.oid, ns.nspname AS schema, c.relname AS object_name, c.relkind
FROM pg_class AS c
JOIN pg_namespace AS ns ON ns.oid = c.relnamespace
WHERE c.relkind = 'r' -- Ordinary Table
AND c.relname = 'test'
;
oid | schema | object_name | relkind
-------+--------+-------------+---------
21508 | public | test | r
Jetzt können wir auf Tabelle genau filtern:
$ pg_waldump --relation=1663/21506/21508 --rmgr=Heap ${WALARCHIVE}/000000010000000000000031 | grep DELETE
rmgr: Heap len (rec/tot): 59/ 8179, tx: 135586, lsn: 0/312DF748, prev 0/312DF720, desc: DELETE xmax: 135586, off: 10, infobits: [KEYS_UPDATED], flags: 0x01, blkref #0: rel 1663/21506/21508 blk 0 FPW
rmgr: Heap len (rec/tot): 54/ 54, tx: 135586, lsn: 0/312E1758, prev 0/312DF748, desc: DELETE xmax: 135586, off: 11, infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/21506/21508 blk 0
rmgr: Heap len (rec/tot): 54/ 54, tx: 135586, lsn: 0/312E1790, prev 0/312E1758, desc: DELETE xmax: 135586, off: 12, infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/21506/21508 blk 0
rmgr: Heap len (rec/tot): 54/ 54, tx: 135586, lsn: 0/312E17C8, prev 0/312E1790, desc: DELETE xmax: 135586, off: 13, infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/21506/21508 blk 0
rmgr: Heap len (rec/tot): 54/ 54, tx: 135586, lsn: 0/312E1800, prev 0/312E17C8, desc: DELETE xmax: 135586, off: 14, infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/21506/21508 blk 0
rmgr: Heap len (rec/tot): 54/ 54, tx: 135586, lsn: 0/312E1838, prev 0/312E1800, desc: DELETE xmax: 135586, off: 15, infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/21506/21508 blk 0
rmgr: Heap len (rec/tot): 54/ 54, tx: 135586, lsn: 0/312E1870, prev 0/312E1838, desc: DELETE xmax: 135586, off: 16, infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/21506/21508 blk 0
rmgr: Heap len (rec/tot): 54/ 54, tx: 135586, lsn: 0/312E18A8, prev 0/312E1870, desc: DELETE xmax: 135586, off: 17, infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/21506/21508 blk 0
rmgr: Heap len (rec/tot): 54/ 54, tx: 135586, lsn: 0/312E18E0, prev 0/312E18A8, desc: DELETE xmax: 135586, off: 18, infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/21506/21508 blk 0
rmgr: Heap len (rec/tot): 54/ 54, tx: 135586, lsn: 0/312E1918, prev 0/312E18E0, desc: DELETE xmax: 135586, off: 19, infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/21506/21508 blk 0
rmgr: Heap len (rec/tot): 54/ 54, tx: 135586, lsn: 0/312E1950, prev 0/312E1918, desc: DELETE xmax: 135586, off: 20, infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/21506/21508 blk 0
Das gibt jetzt genau die 11 Rows auf der Tabelle test.
Jetzt schauen wir das Ganze mit dem Betrachter unserer Wahl noch etwas genauer an (am besten Filtern auf LSN). Wir sehen, die Transaktion vorher und unsere Transaktion bis zum COMMIT und das ganze als rückwärts-verkettete Liste (prev ➜ lsn)?
rmgr: Transaction len (rec/tot): 34/ 34, tx: 135585, lsn: 0/312DF720, prev 0/312DF6E0, desc: COMMIT 2026-07-22 08:54:34.033961 UTC
rmgr: Heap len (rec/tot): 59/ 8179, tx: 135586, lsn: 0/312DF748, prev 0/312DF720, desc: DELETE xmax: 135586, off: 10, infobits: [KEYS_UPDATED], flags: 0x01, blkref #0: rel 1663/21506/21508 blk 0 FPW
rmgr: Heap len (rec/tot): 54/ 54, tx: 135586, lsn: 0/312E1758, prev 0/312DF748, desc: DELETE xmax: 135586, off: 11, infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/21506/21508 blk 0
rmgr: Heap len (rec/tot): 54/ 54, tx: 135586, lsn: 0/312E1790, prev 0/312E1758, desc: DELETE xmax: 135586, off: 12, infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/21506/21508 blk 0
rmgr: Heap len (rec/tot): 54/ 54, tx: 135586, lsn: 0/312E17C8, prev 0/312E1790, desc: DELETE xmax: 135586, off: 13, infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/21506/21508 blk 0
rmgr: Heap len (rec/tot): 54/ 54, tx: 135586, lsn: 0/312E1800, prev 0/312E17C8, desc: DELETE xmax: 135586, off: 14, infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/21506/21508 blk 0
rmgr: Heap len (rec/tot): 54/ 54, tx: 135586, lsn: 0/312E1838, prev 0/312E1800, desc: DELETE xmax: 135586, off: 15, infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/21506/21508 blk 0
rmgr: Heap len (rec/tot): 54/ 54, tx: 135586, lsn: 0/312E1870, prev 0/312E1838, desc: DELETE xmax: 135586, off: 16, infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/21506/21508 blk 0
rmgr: Heap len (rec/tot): 54/ 54, tx: 135586, lsn: 0/312E18A8, prev 0/312E1870, desc: DELETE xmax: 135586, off: 17, infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/21506/21508 blk 0
rmgr: Heap len (rec/tot): 54/ 54, tx: 135586, lsn: 0/312E18E0, prev 0/312E18A8, desc: DELETE xmax: 135586, off: 18, infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/21506/21508 blk 0
rmgr: Heap len (rec/tot): 54/ 54, tx: 135586, lsn: 0/312E1918, prev 0/312E18E0, desc: DELETE xmax: 135586, off: 19, infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/21506/21508 blk 0
rmgr: Heap len (rec/tot): 54/ 54, tx: 135586, lsn: 0/312E1950, prev 0/312E1918, desc: DELETE xmax: 135586, off: 20, infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/21506/21508 blk 0
rmgr: Transaction len (rec/tot): 34/ 34, tx: 135586, lsn: 0/312E1988, prev 0/312E1950, desc: COMMIT 2026-07-22 08:54:34.052628 UTC
Unsere Ups-Query Transaktion hat also die XID 135586 und fängt bei der LSN 0/312DF748 an. Da wir ja die Transaktion beim Recovery NICHT mehr haben wollen, ist das Recovery dann exklusiv!
Somit hätten wir also bereits die recovery_target Informationen beisammen:
#
# postgres.conf
#
recovery_target_xid = '135586'
recovery_target_lsn = '0/312DF748'
recovery_target_inclusive = off
Recovery zur XID
Jetzt können wir das Point-in-Time-Recovery mit XID als Target ausführen. Das sieht dann im Log wie folgt aus:
LOG: starting PostgreSQL 18.4 (Debian 18.4-1.pgdg13+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 14.2.0-19) 14.2.0, 64-bit
LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
LOG: database system was interrupted; last known up at 2026-07-21 14:21:59 UTC
cp: cannot stat '/mnt/backup/wal_archive/00000002.history': No such file or directory
LOG: starting backup recovery with redo LSN 0/21000108, checkpoint LSN 0/2100BAF0, on timeline ID 1
LOG: restored log file "000000010000000000000021" from archive
LOG: starting point-in-time recovery to XID 135586
LOG: redo starts at 0/21000108
LOG: restored log file "000000010000000000000022" from archive
LOG: completed backup recovery with redo LSN 0/21000108 and end LSN 0/2100C070
LOG: consistent recovery state reached at 0/2100C070
LOG: database system is ready to accept read-only connections
LOG: restored log file "000000010000000000000023" from archive
LOG: restored log file "000000010000000000000024" from archive
LOG: restored log file "000000010000000000000025" from archive
LOG: restored log file "000000010000000000000026" from archive
LOG: restored log file "000000010000000000000027" from archive
LOG: restored log file "000000010000000000000028" from archive
LOG: restored log file "000000010000000000000029" from archive
LOG: restored log file "00000001000000000000002A" from archive
LOG: restored log file "00000001000000000000002B" from archive
LOG: restored log file "00000001000000000000002C" from archive
LOG: restored log file "00000001000000000000002D" from archive
LOG: restored log file "00000001000000000000002E" from archive
LOG: restored log file "00000001000000000000002F" from archive
LOG: restored log file "000000010000000000000030" from archive
LOG: restored log file "000000010000000000000031" from archive
LOG: recovery stopping before commit of transaction 135586, time 2026-07-22 08:54:34.052628+00
LOG: pausing at the end of recovery
HINT: Execute pg_wal_replay_resume() to promote.
Dann prüfen wir zuerst, ob das Ups-Query Rückgängig gemacht wurde und allenfalls, bis wohin die Daten noch/wieder da sind. Wenn das Resultat zufriedenstellend ist kann die Datenbank wieder aus dem Recovery Mode in Primary Mode promoted werden:
test=# SELECT pg_is_in_recovery();
pg_is_in_recovery
-------------------
t
test=# select pg_promote();
pg_promote
------------
t
Ein Blick ins Log zeigt uns, dass alles geklappt hat:
LOG: received promote request
LOG: redo done at 0/312E1988 system usage: CPU: user: 0.12 s, system: 0.07 s, elapsed: 169.82 s
LOG: last completed transaction was at log time 2026-07-22 08:54:34.033961+00
cp: cannot stat '/mnt/backup/wal_archive/00000002.history': No such file or directory
LOG: selected new timeline ID: 2
cp: cannot stat '/mnt/backup/wal_archive/00000001.history': No such file or directory
LOG: archive recovery complete
LOG: checkpoint starting: force
LOG: database system is ready to accept connections
Recovery zur LSN
Ob das Ganz mit LSN auch funktioniert, wollten wir natürlich ebenfalls ausprobieren. Das sieht dann im Log wie folgt aus:
LOG: starting PostgreSQL 18.4 (Debian 18.4-1.pgdg13+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 14.2.0-19) 14.2.0, 64-bit
LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
LOG: database system was interrupted; last known up at 2026-07-21 14:21:59 UTC
cp: cannot stat '/mnt/backup/wal_archive/00000002.history': No such file or directory
LOG: starting backup recovery with redo LSN 0/21000108, checkpoint LSN 0/2100BAF0, on timeline ID 1
LOG: restored log file "000000010000000000000021" from archive
LOG: starting point-in-time recovery to WAL location (LSN) "0/312DF748"
LOG: redo starts at 0/21000108
LOG: restored log file "000000010000000000000022" from archive
LOG: completed backup recovery with redo LSN 0/21000108 and end LSN 0/2100C070
LOG: consistent recovery state reached at 0/2100C070
LOG: database system is ready to accept read-only connections
LOG: restored log file "000000010000000000000023" from archive
LOG: restored log file "000000010000000000000024" from archive
LOG: restored log file "000000010000000000000025" from archive
LOG: restored log file "000000010000000000000026" from archive
LOG: restored log file "000000010000000000000027" from archive
LOG: restored log file "000000010000000000000028" from archive
LOG: restored log file "000000010000000000000029" from archive
LOG: restored log file "00000001000000000000002A" from archive
LOG: restored log file "00000001000000000000002B" from archive
LOG: restored log file "00000001000000000000002C" from archive
LOG: restored log file "00000001000000000000002D" from archive
LOG: restored log file "00000001000000000000002E" from archive
LOG: restored log file "00000001000000000000002F" from archive
LOG: restored log file "000000010000000000000030" from archive
LOG: restored log file "000000010000000000000031" from archive
LOG: recovery stopping before WAL location (LSN) "0/312DF748"
LOG: pausing at the end of recovery
HINT: Execute pg_wal_replay_resume() to promote.
Und dann, nach Prüfen und Promoten geht es wie folgt weiter:
LOG: received promote request
LOG: redo done at 0/312DF748 system usage: CPU: user: 0.13 s, system: 0.05 s, elapsed: 90.70 s
LOG: last completed transaction was at log time 2026-07-22 08:54:34.033961+00
cp: cannot stat '/mnt/backup/wal_archive/00000002.history': No such file or directory
LOG: selected new timeline ID: 2
cp: cannot stat '/mnt/backup/wal_archive/00000001.history': No such file or directory
LOG: archive recovery complete
LOG: checkpoint starting: force
LOG: database system is ready to accept connections
Nacharbeiten
Ob und wie man die Daten die NACH dem Ups-Query angefallen sind, retten kann und will, ist dann noch mal eine ganz andere Geschichte…
Zur Sicherheit: Die betroffene Datenbank vor dem Restore/Recovery noch mal wegsichern, vielleicht können oder müssen wir da noch Daten rausfieseln?

