borgmatic-collective/borgmatic
borgmatic-collective
/
borgmatic
14
57
Fork 41
Code Issues 54 Pull Requests 2 Actions Packages Releases 122 Activity

After upgrading Synology and Debian Server, my borgmatic no longer terminates with systemd #801

New Issue
Closed
opened 2023-12-26 14:21:19 +00:00 by Auroch · 5 comments
Auroch commented 2023-12-26 14:21:19 +00:00
Copy Link

What I'm trying to do and why

Launch my borgbackup throught borgmatic with systemd service and timer

My /etc/borgmatic/config.yaml

source_directories:
    - /home
    - /etc
    - /srv
    - /opt
    - /root
    - /lib/systemd/system/
    - /var/www
    - /var/backups/ldap
    - /var/log/syslog*
    - /var/spool/cron/crontabs/
repositories:
    - path: ssh://<user>@<server>/path/borg01
      label: <label>
encryption_passphrase: "..."
ssh_command: ssh -i /etc/borg/ssh/<keyname> -p <port> -o BatchMode=yes -o VerifyHostKeyDNS=yes
keep_daily: 7
keep_weekly: 5
keep_monthly: 12
keep_yearly: 5
checks:
    - name: repository
    - name: archives
      frequency: 2 week
    - name: data
      frequency: 1 month
check_last: 3
before_actions:
    - echo Starting actions
    - echo LDAP Git Backup
    - /usr/sbin/ldap-git-backup --commit-msg "Sauvegarde quotidienne avant Borgmatic"
postgresql_databases:
    - name: all
      username: <user>
      password: <pass>
mariadb_databases:
    - name: all
      hostname: <host>
      port: <port>
      username: <user>
      password: <pass>
      options: --single-transaction --quick
healthchecks:
    ping_url: https://hc-ping.com/<personalID>

My /etc/systemd/system/borgmatic.service

[Unit]
Description=borgmatic backup
Wants=network-online.target
After=network-online.target
ConditionACPower=true

[Service]
Type=oneshot

LockPersonality=true

MemoryDenyWriteExecute=no
NoNewPrivileges=yes
PrivateDevices=yes
PrivateTmp=yes
ProtectClock=yes
ProtectControlGroups=yes
ProtectHostname=yes
ProtectKernelLogs=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK
RestrictNamespaces=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
SystemCallArchitectures=native
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM
ProtectSystem=full
CapabilityBoundingSet=CAP_DAC_READ_SEARCH CAP_NET_RAW

Nice=19
CPUSchedulingPolicy=batch
IOSchedulingClass=best-effort
IOSchedulingPriority=7
IOWeight=100

Restart=no

LogRateLimitIntervalSec=0

ExecStartPre=sleep 1m
ExecStart=systemd-inhibit --who="borgmatic" --what="sleep:shutdown" --why="Prevent interrupting scheduled backup" /usr/bin/borgmatic --syslog-verbosity 2

On the server side, it's a Synology with

  • borg: 1.2.6
  • borgmatic: 1.8.2

Steps to reproduce

No response

Actual behavior

Run the command manually as root, create the backup correctly, perform the checks and finish successfully.

/usr/bin/borgmatic --verbosity 1

/etc/borgmatic/config.yaml: Pinging Healthchecks start
/etc/borgmatic/config.yaml: Running 3 commands for pre-actions hook
Starting actions
LDAP Git Backup
sechlama-b: Creating archive
ssh://borg-backup@borg.mes.li/volume1/borg-backup/start01: Dumping MariaDB databases
ssh://borg-backup@borg.mes.li/volume1/borg-backup/start01: Dumping PostgreSQL databases
ssh://borg-backup@borg.mes.li/volume1/borg-backup/start01: Excluding special files to prevent Borg from hanging: /etc/systemd/system/php7.4-fpm.service, /etc/systemd/system/matrix-synapse.service, /etc/systemd/system/php8.0-fpm.service, /etc/systemd/system/php8.1-fpm.service, /etc/systemd/system/coturn.service, /etc/systemd/system/prosody.service, /lib/systemd/system/mdadm-waitidle.service, /lib/systemd/system/mdadm.service, /lib/systemd/system/screen-cleanup.service, /lib/systemd/system/sudo.service, /lib/systemd/system/cryptdisks-early.service, /lib/systemd/system/cryptdisks.service, /lib/systemd/system/hwclock.service, /lib/systemd/system/rc.service, /lib/systemd/system/rcS.service, /lib/systemd/system/x11-common.service, /root/recovery/dev/stderr, /root/recovery/dev/stdin, /root/recovery/dev/stdout, /root/recovery/etc/systemd/system/containerd.service, /root/recovery/etc/systemd/system/nginx.service, /root/recovery/etc/systemd/system/rtkit-daemon.service, /root/recovery/lib/systemd/system/bootlogd.service, /root/recovery/lib/systemd/system/bootlogs.service, /root/recovery/lib/systemd/system/bootmisc.service, /root/recovery/lib/systemd/system/checkfs.service, /root/recovery/lib/systemd/system/checkroot-bootclean.service, /root/recovery/lib/systemd/system/checkroot.service, /root/recovery/lib/systemd/system/cryptdisks-early.service, /root/recovery/lib/systemd/system/cryptdisks.service, /root/recovery/lib/systemd/system/fuse.service, /root/recovery/lib/systemd/system/halt.service, /root/recovery/lib/systemd/system/hostname.service, /root/recovery/lib/systemd/system/hwclock.service, /root/recovery/lib/systemd/system/killprocs.service, /root/recovery/lib/systemd/system/mdadm-waitidle.service, /root/recovery/lib/systemd/system/mdadm.service, /root/recovery/lib/systemd/system/motd.service, /root/recovery/lib/systemd/system/mountall-bootclean.service, /root/recovery/lib/systemd/system/mountall.service, /root/recovery/lib/systemd/system/mountdevsubfs.service, /root/recovery/lib/systemd/system/mountkernfs.service, /root/recovery/lib/systemd/system/mountnfs-bootclean.service, /root/recovery/lib/systemd/system/mountnfs.service, /root/recovery/lib/systemd/system/rc.service, /root/recovery/lib/systemd/system/rcS.service, /root/recovery/lib/systemd/system/reboot.service, /root/recovery/lib/systemd/system/rmnologin.service, /root/recovery/lib/systemd/system/screen-cleanup.service, /root/recovery/lib/systemd/system/sendsigs.service, /root/recovery/lib/systemd/system/single.service, /root/recovery/lib/systemd/system/stop-bootlogd-single.service, /root/recovery/lib/systemd/system/stop-bootlogd.service, /root/recovery/lib/systemd/system/umountfs.service, /root/recovery/lib/systemd/system/umountnfs.service, /root/recovery/lib/systemd/system/umountroot.service, /root/recovery/lib/systemd/system/x11-common.service, /root/recovery/tmp/borgmatic/postgresql_databases/localhost/all
Creating archive at "ssh://<user>@<server>/path/borg01::borg01-2023-12-26T14:12:32.224960"
<label>: Pruning archives
<label>: Compacting segments
Remote: compaction freed about 5.63 MB repository space.
<label>: Running consistency checks
Remote: Starting repository check
Remote: finished segment check at segment 9917
Remote: Starting repository index check
Remote: Index object count match.
Remote: Finished full repository check, no problems found.
Starting archive consistency check...
Remote: Verified integrity of /path/borg01/index.9930
TAM-verified manifest
Analyzing archive borg01-2023-12-23T18:04:44.905470 (1/3)
TAM-verified archive
Analyzing archive borg01-2023-12-25T06:46:16.925447 (2/3)
TAM-verified archive
Analyzing archive borg01-2023-12-26T15:01:19.092972 (3/3)
TAM-verified archive
RepositoryCache: current items 1140, size 44.12 MB / 2.15 GB, 2098 hits, 1140 misses, 0 slow misses (+0.0s), 0 evictions, 0 ENOSPC hit
Orphaned objects check skipped (needs all archives checked).
Archive consistency check complete, no problems found.
RemoteRepository: 61.05 kB bytes sent, 62.44 MB bytes received, 1159 messages sent
terminating with success status, rc 0
Writing check time at /root/.borgmatic/checks/<randomID>/repository
Writing check time at /root/.borgmatic/checks/<randomID>/archives/<randomID>
/etc/borgmatic/config.yaml: No commands to run for post-check hook
/etc/borgmatic/config.yaml: No commands to run for post-actions hook
/etc/borgmatic/config.yaml: Calling healthchecks hook function ping_monitor
/etc/borgmatic/config.yaml: Pinging Healthchecks log
/etc/borgmatic/config.yaml: Using Healthchecks ping URL https://hc-ping.com/<personalID>
/etc/borgmatic/config.yaml: Calling healthchecks hook function ping_monitor
/etc/borgmatic/config.yaml: Pinging Healthchecks finish
/etc/borgmatic/config.yaml: Using Healthchecks ping URL https://hc-ping.com/<personalID>
/etc/borgmatic/config.yaml: Calling healthchecks hook function destroy_monitor
/etc/borgmatic/config.yaml: No commands to run for post-everything hook

summary:
/etc/borgmatic/config.yaml: Loading configuration file
/etc/borgmatic/config.yaml: Successfully ran configuration file

BUT

Starting the service manually never ends and remains inactive indefinitely

systemctl restart borgmatic.service &

Dec 26 12:56:32 start01 systemd[1]: Starting borgmatic.service - borgmatic backup...
Dec 26 12:58:32 start01 systemd-inhibit[2216135]: /etc/borgmatic/config.yaml: No commands to run for pre-everything hook
Dec 26 12:58:32 start01 borgmatic[2216135]: DEBUG /etc/borgmatic/config.yaml: No commands to run for pre-everything hook
Dec 26 12:58:32 start01 systemd-inhibit[2216135]: borg --version --debug --show-rc
Dec 26 12:58:32 start01 borgmatic[2216135]: DEBUG borg --version --debug --show-rc
Dec 26 12:58:33 start01 systemd-inhibit[2216135]: /etc/borgmatic/config.yaml: Calling healthchecks hook function initialize_monitor
Dec 26 12:58:33 start01 systemd-inhibit[2216135]: /etc/borgmatic/config.yaml: Calling healthchecks hook function ping_monitor
Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG /etc/borgmatic/config.yaml: Calling healthchecks hook function initialize_monitor
Dec 26 12:58:33 start01 systemd-inhibit[2216135]: /etc/borgmatic/config.yaml: Pinging Healthchecks start
Dec 26 12:58:33 start01 systemd-inhibit[2216135]: /etc/borgmatic/config.yaml: Using Healthchecks ping URL https://hc-ping.com/<personalID>/start
Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG /etc/borgmatic/config.yaml: Calling healthchecks hook function ping_monitor
Dec 26 12:58:33 start01 borgmatic[2216135]: INFO /etc/borgmatic/config.yaml: Pinging Healthchecks start
Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG /etc/borgmatic/config.yaml: Using Healthchecks ping URL https://hc-ping.com/<personalID>/start
Dec 26 12:58:33 start01 systemd-inhibit[2216135]: sechlama-b: Running actions for repository
Dec 26 12:58:33 start01 systemd-inhibit[2216135]: /etc/borgmatic/config.yaml: No commands to run for pre-actions hook
Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG sechlama-b: Running actions for repository
Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG /etc/borgmatic/config.yaml: No commands to run for pre-actions hook
Dec 26 12:58:33 start01 systemd-inhibit[2216135]: /etc/borgmatic/config.yaml: No commands to run for pre-backup hook
Dec 26 12:58:33 start01 systemd-inhibit[2216135]: sechlama-b: Creating archive
Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG /etc/borgmatic/config.yaml: No commands to run for pre-backup hook
Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Calling mariadb_databases hook function remove_data_source_dumps
Dec 26 12:58:33 start01 borgmatic[2216135]: INFO sechlama-b: Creating archive
Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Removing MariaDB data source dumps
Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Calling mariadb_databases hook function remove_data_source_dumps
Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Removing MariaDB data source dumps
Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Calling mysql_databases hook function remove_data_source_dumps
Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Calling mysql_databases hook function remove_data_source_dumps
Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Removing MySQL data source dumps
Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Removing MySQL data source dumps
Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Calling mongodb_databases hook function remove_data_source_dumps
Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Calling mongodb_databases hook function remove_data_source_dumps
Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Removing MongoDB data source dumps
Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Removing MongoDB data source dumps
Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Calling postgresql_databases hook function remove_data_source_dumps
Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Calling postgresql_databases hook function remove_data_source_dumps
Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Removing PostgreSQL data source dumps
Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Removing PostgreSQL data source dumps
Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Calling sqlite_databases hook function remove_data_source_dumps
Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Calling sqlite_databases hook function remove_data_source_dumps
Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Removing SQLite data source dumps
Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Removing SQLite data source dumps
Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Calling mariadb_databases hook function dump_data_sources
Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Calling mariadb_databases hook function dump_data_sources
Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Dumping MariaDB databases
Dec 26 12:58:33 start01 borgmatic[2216135]: INFO ssh://<user>@<server>/path/borg01: Dumping MariaDB databases
Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Querying for "all" MariaDB databases to dump
Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Querying for "all" MariaDB databases to dump
Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG mariadb --host <host> --port <port> --protocol tcp --user root --skip-column-names --batch --execute show schemas
Dec 26 12:58:33 start01 systemd-inhibit[2216135]: mariadb --host <host> --port <port> --protocol tcp --user root --skip-column-names --batch --execute show schemas
Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Dumping MariaDB database "all" to /root/.borgmatic/mariadb_databases/<host>/all
Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Dumping MariaDB database "all" to /root/.borgmatic/mariadb_databases/<host>/all
Dec 26 12:58:33 start01 systemd-inhibit[2216135]: mariadb-dump --single-transaction --quick --add-drop-database --host <host> --port <port> --protocol tcp --user root --databases BDCN cachet_db dolibarr dolibarr_flammelle dolibarr_labart freescout hubzilla_db i3d_wiki vimbadmin wp_flammelle --result-file /root/.borgmatic/mariadb_databases/<host>/all
Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG mariadb-dump --single-transaction --quick --add-drop-database --host <host> --port <port> --protocol tcp --user root --databases BDCN cachet_db dolibarr dolibarr_flammelle dolibarr_labart freescout hubzilla_db i3d_wiki vimbadmin wp_flammelle --result-file /root/.borgmatic/mariadb_databases/<host>/all
Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Calling postgresql_databases hook function dump_data_sources
Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Calling postgresql_databases hook function dump_data_sources
Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Dumping PostgreSQL databases
Dec 26 12:58:33 start01 borgmatic[2216135]: INFO ssh://<user>@<server>/path/borg01: Dumping PostgreSQL databases
Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Dumping PostgreSQL database "all" to /root/.borgmatic/postgresql_databases/<host>/all
Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Dumping PostgreSQL database "all" to /root/.borgmatic/postgresql_databases/<host>/all
Dec 26 12:58:33 start01 systemd-inhibit[2216135]: pg_dumpall --no-password --clean --if-exists --username postgres > /root/.borgmatic/postgresql_databases/<host>/all
Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG pg_dumpall --no-password --clean --if-exists --username postgres > /root/.borgmatic/postgresql_databases/<host>/all
Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Collecting special file paths
Dec 26 12:58:33 start01 systemd-inhibit[2216135]: borg create --one-file-system --read-special --list --filter AMEx- ssh://<user>@<server>/path/borg01::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /etc /home /lib/systemd/system/ /opt /root /srv /var/backups/ldap /var/log/syslog /var/log/syslog.1 /var/log/syslog.2.gz /var/log/syslog.3.gz /var/log/syslog.4.gz /var/>
Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Collecting special file paths
Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG borg create --one-file-system --read-special --list --filter AMEx- ssh://<user>@<server>/path/borg01::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /etc /home /lib/systemd/system/ /opt /root /srv /var/backups/ldap /var/log/syslog /var/log/syslog.1 /var/log/syslog.2.gz /var/log/syslog.3.gz /var/log/syslog.4.gz /var/>
Dec 26 12:59:11 start01 borgmatic[2216135]: WARNING ssh://<user>@<server>/path/borg01: Excluding special files to prevent Borg from hanging: /etc/systemd/system/php7.4-fpm.service, /etc/systemd/system/matrix-synapse.service, /etc/systemd/system/php8.0-fpm.service, /etc/systemd/system/php8.1-fpm.service, /etc/systemd/system/coturn.service, /etc/systemd/syst>
Dec 26 12:59:11 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Excluding special files to prevent Borg from hanging: /etc/systemd/system/php7.4-fpm.service, /etc/systemd/system/matrix-synapse.service, /etc/systemd/system/php8.0-fpm.service, /etc/systemd/system/php8.1-fpm.service, /etc/systemd/system/coturn.service, /etc/systemd/system>
Dec 26 12:59:11 start01 systemd-inhibit[2216135]: borg create --one-file-system --read-special --list --filter AMEx- --exclude-from /tmp/tmp2io923g_ --stats --debug --show-rc ssh://<user>@<server>/path/borg01::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /etc /home /lib/systemd/system/ /opt /root /srv /var/backups/ldap /var/log/syslog /var/log/syslog.1 /var/log/s>
Dec 26 12:59:11 start01 borgmatic[2216135]: DEBUG borg create --one-file-system --read-special --list --filter AMEx- --exclude-from /tmp/tmp2io923g_ --stats --debug --show-rc ssh://<user>@<server>/path/borg01::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /etc /home /lib/systemd/system/ /opt /root /srv /var/backups/ldap /var/log/syslog /var/log/syslog.1 /var/log/s>
Dec 26 12:59:11 start01 systemd-inhibit[2216135]: using builtin fallback logging configuration
Dec 26 12:59:11 start01 borgmatic[2216135]: ANSWER using builtin fallback logging configuration
Dec 26 12:59:11 start01 systemd-inhibit[2216135]: 33 self tests completed in 0.08 seconds
Dec 26 12:59:11 start01 borgmatic[2216135]: ANSWER 33 self tests completed in 0.08 seconds
Dec 26 12:59:11 start01 systemd-inhibit[2216135]: SSH command line: ['ssh', '-i', '/etc/borg/ssh/<keyName>', '-p', '<port>', '-o', 'BatchMode=yes', '-o', 'VerifyHostKeyDNS=yes', '<borgUser>@<FQDN>', 'borg', 'serve', '--debug']
Dec 26 12:59:11 start01 borgmatic[2216135]: ANSWER SSH command line: ['ssh', '-i', '/etc/borg/ssh/<keyName>', '-p', '<port>', '-o', 'BatchMode=yes', '-o', 'VerifyHostKeyDNS=yes', '<borgUser>@<FQDN>', 'borg', 'serve', '--debug']
Dec 26 12:59:14 start01 systemd-inhibit[2216135]: Remote: using builtin fallback logging configuration
Dec 26 12:59:14 start01 borgmatic[2216135]: ANSWER Remote: using builtin fallback logging configuration
Dec 26 12:59:15 start01 systemd-inhibit[2216135]: Remote: 33 self tests completed in 1.16 seconds
Dec 26 12:59:15 start01 borgmatic[2216135]: ANSWER Remote: 33 self tests completed in 1.16 seconds
Dec 26 12:59:15 start01 systemd-inhibit[2216135]: Remote: using builtin fallback logging configuration
Dec 26 12:59:15 start01 borgmatic[2216135]: ANSWER Remote: using builtin fallback logging configuration
Dec 26 12:59:15 start01 borgmatic[2216135]: ANSWER Remote: Initialized logging system for JSON-based protocol
Dec 26 12:59:15 start01 systemd-inhibit[2216135]: Remote: Initialized logging system for JSON-based protocol
Dec 26 12:59:15 start01 systemd-inhibit[2216135]: Remote: Resolving repository path b'/<path>/borg01'
Dec 26 12:59:15 start01 borgmatic[2216135]: ANSWER Remote: Resolving repository path b'/<path>/borg01'
Dec 26 12:59:15 start01 borgmatic[2216135]: ANSWER Remote: Resolved repository path to '/<path>/borg01'
Dec 26 12:59:15 start01 systemd-inhibit[2216135]: Remote: Resolved repository path to '/<path>/borg01'
Dec 26 12:59:16 start01 systemd-inhibit[2216135]: Remote: Verified integrity of /<path>/borg01/index.9891
Dec 26 12:59:16 start01 borgmatic[2216135]: ANSWER Remote: Verified integrity of /<path>/borg01/index.9891
Dec 26 12:59:16 start01 systemd-inhibit[2216135]: TAM-verified manifest
Dec 26 12:59:16 start01 borgmatic[2216135]: ANSWER TAM-verified manifest
Dec 26 12:59:16 start01 systemd-inhibit[2216135]: security: read previous location 'ssh://<user>@<server>/path/borg01'
Dec 26 12:59:16 start01 borgmatic[2216135]: ANSWER security: read previous location 'ssh://<user>@<server>/path/borg01'
Dec 26 12:59:16 start01 systemd-inhibit[2216135]: security: read manifest timestamp '2023-12-26T09:43:15.962630'
Dec 26 12:59:16 start01 borgmatic[2216135]: ANSWER security: read manifest timestamp '2023-12-26T09:43:15.962630'
Dec 26 12:59:16 start01 systemd-inhibit[2216135]: security: determined newest manifest timestamp as 2023-12-26T09:43:15.962630
Dec 26 12:59:16 start01 borgmatic[2216135]: ANSWER security: determined newest manifest timestamp as 2023-12-26T09:43:15.962630
Dec 26 12:59:16 start01 systemd-inhibit[2216135]: security: repository checks ok, allowing access
Dec 26 12:59:16 start01 borgmatic[2216135]: ANSWER security: repository checks ok, allowing access
Dec 26 12:59:16 start01 borgmatic[2216135]: ANSWER Creating archive at "ssh://<user>@<server>/path/borg01::start01-2023-12-26T12:59:11.493256"
Dec 26 12:59:16 start01 systemd-inhibit[2216135]: Creating archive at "ssh://<user>@<server>/path/borg01::start01-2023-12-26T12:59:11.493256"
Dec 26 12:59:16 start01 systemd-inhibit[2216135]: Verified integrity of /root/.cache/borg/<RandomID>/chunks
Dec 26 12:59:16 start01 systemd-inhibit[2216135]: Reading files cache ...
Dec 26 12:59:16 start01 borgmatic[2216135]: ANSWER Verified integrity of /root/.cache/borg/<RandomID>/chunks
Dec 26 12:59:16 start01 borgmatic[2216135]: ANSWER Reading files cache ...
Dec 26 12:59:18 start01 systemd-inhibit[2216135]: Verified integrity of /root/.cache/borg/<RandomID>/files
Dec 26 12:59:18 start01 systemd-inhibit[2216135]: security: read previous location 'ssh://<user>@<server>/path/borg01'
Dec 26 12:59:18 start01 systemd-inhibit[2216135]: security: read manifest timestamp '2023-12-26T09:43:15.962630'
Dec 26 12:59:18 start01 systemd-inhibit[2216135]: security: determined newest manifest timestamp as 2023-12-26T09:43:15.962630
Dec 26 12:59:18 start01 systemd-inhibit[2216135]: security: repository checks ok, allowing access
Dec 26 12:59:18 start01 borgmatic[2216135]: ANSWER Verified integrity of /root/.cache/borg/<RandomID>/files
Dec 26 12:59:18 start01 systemd-inhibit[2216135]: Processing files ...
Dec 26 12:59:18 start01 borgmatic[2216135]: ANSWER security: read previous location 'ssh://<user>@<server>/path/borg01'
Dec 26 12:59:18 start01 borgmatic[2216135]: ANSWER security: read manifest timestamp '2023-12-26T09:43:15.962630'
Dec 26 12:59:18 start01 borgmatic[2216135]: ANSWER security: determined newest manifest timestamp as 2023-12-26T09:43:15.962630
Dec 26 12:59:18 start01 borgmatic[2216135]: ANSWER security: repository checks ok, allowing access
Dec 26 12:59:18 start01 borgmatic[2216135]: ANSWER Processing files ...
Dec 26 12:59:19 start01 systemd-inhibit[2216135]: M /etc/systemd/system/borgmatic.service
Dec 26 12:59:19 start01 borgmatic[2216135]: ANSWER M /etc/systemd/system/borgmatic.service
Dec 26 12:59:19 start01 systemd-inhibit[2216135]: x /etc/systemd/system/php7.4-fpm.service
Dec 26 12:59:19 start01 borgmatic[2216135]: ANSWER x /etc/systemd/system/php7.4-fpm.service
Dec 26 12:59:19 start01 systemd-inhibit[2216135]: x /etc/systemd/system/matrix-synapse.service
Dec 26 12:59:19 start01 borgmatic[2216135]: ANSWER x /etc/systemd/system/matrix-synapse.service
Dec 26 12:59:19 start01 borgmatic[2216135]: ANSWER x /etc/systemd/system/php8.0-fpm.service
Dec 26 12:59:19 start01 systemd-inhibit[2216135]: x /etc/systemd/system/php8.0-fpm.service
Dec 26 12:59:19 start01 systemd-inhibit[2216135]: x /etc/systemd/system/php8.1-fpm.service
Dec 26 12:59:19 start01 borgmatic[2216135]: ANSWER x /etc/systemd/system/php8.1-fpm.service
Dec 26 12:59:19 start01 borgmatic[2216135]: ANSWER x /etc/systemd/system/coturn.service
Dec 26 12:59:19 start01 systemd-inhibit[2216135]: x /etc/systemd/system/coturn.service
Dec 26 12:59:19 start01 systemd-inhibit[2216135]: x /etc/systemd/system/prosody.service
Dec 26 12:59:19 start01 borgmatic[2216135]: ANSWER x /etc/systemd/system/prosody.service
Dec 26 12:59:20 start01 systemd-inhibit[2216135]: x /lib/systemd/system/mdadm-waitidle.service
Dec 26 12:59:20 start01 systemd-inhibit[2216135]: x /lib/systemd/system/mdadm.service
Dec 26 12:59:20 start01 borgmatic[2216135]: ANSWER x /lib/systemd/system/mdadm-waitidle.service
Dec 26 12:59:20 start01 borgmatic[2216135]: ANSWER x /lib/systemd/system/mdadm.service
Dec 26 12:59:20 start01 systemd-inhibit[2216135]: x /lib/systemd/system/screen-cleanup.service
Dec 26 12:59:20 start01 borgmatic[2216135]: ANSWER x /lib/systemd/system/screen-cleanup.service
Dec 26 12:59:20 start01 systemd-inhibit[2216135]: x /lib/systemd/system/sudo.service
Dec 26 12:59:20 start01 borgmatic[2216135]: ANSWER x /lib/systemd/system/sudo.service
Dec 26 12:59:20 start01 systemd-inhibit[2216135]: x /lib/systemd/system/cryptdisks-early.service
Dec 26 12:59:20 start01 borgmatic[2216135]: ANSWER x /lib/systemd/system/cryptdisks-early.service
Dec 26 12:59:20 start01 borgmatic[2216135]: ANSWER x /lib/systemd/system/cryptdisks.service
Dec 26 12:59:20 start01 systemd-inhibit[2216135]: x /lib/systemd/system/cryptdisks.service
Dec 26 12:59:20 start01 systemd-inhibit[2216135]: x /lib/systemd/system/hwclock.service
Dec 26 12:59:20 start01 borgmatic[2216135]: ANSWER x /lib/systemd/system/hwclock.service
Dec 26 12:59:20 start01 systemd-inhibit[2216135]: x /lib/systemd/system/rc.service
Dec 26 12:59:20 start01 borgmatic[2216135]: ANSWER x /lib/systemd/system/rc.service
Dec 26 12:59:20 start01 systemd-inhibit[2216135]: x /lib/systemd/system/rcS.service
Dec 26 12:59:20 start01 borgmatic[2216135]: ANSWER x /lib/systemd/system/rcS.service
Dec 26 12:59:20 start01 systemd-inhibit[2216135]: x /lib/systemd/system/x11-common.service
Dec 26 12:59:20 start01 borgmatic[2216135]: ANSWER x /lib/systemd/system/x11-common.service
Dec 26 12:59:22 start01 systemd-inhibit[2216135]: M /root/.bash_history
Dec 26 12:59:22 start01 borgmatic[2216135]: ANSWER M /root/.bash_history
Dec 26 12:59:22 start01 systemd-inhibit[2216135]: Remote: Cleaned up 1 uncommitted segment files (== everything after segment 9891).
Dec 26 12:59:22 start01 borgmatic[2216135]: ANSWER Remote: Cleaned up 1 uncommitted segment files (== everything after segment 9891).
Dec 26 12:59:22 start01 borgmatic[2216135]: ANSWER Remote: Verified integrity of /<path>/borg01/hints.9891
Dec 26 12:59:22 start01 systemd-inhibit[2216135]: Remote: Verified integrity of /<path>/borg01/hints.9891
Dec 26 12:59:22 start01 systemd-inhibit[2216135]: M /root/.config/borg/security/<RandomID>/nonce
Dec 26 12:59:22 start01 borgmatic[2216135]: ANSWER M /root/.config/borg/security/<RandomID>/nonce
Dec 26 12:59:22 start01 systemd-inhibit[2216135]: M /root/.config/borg/security/<RandomID>/location
Dec 26 12:59:22 start01 borgmatic[2216135]: ANSWER M /root/.config/borg/security/<RandomID>/location
Dec 26 12:59:22 start01 systemd-inhibit[2216135]: M /root/.config/borg/security/<RandomID>/key-type
Dec 26 12:59:22 start01 borgmatic[2216135]: ANSWER M /root/.config/borg/security/<RandomID>/key-type
Dec 26 12:59:22 start01 systemd-inhibit[2216135]: M /root/.config/borg/security/<RandomID>/manifest-timestamp
Dec 26 12:59:22 start01 borgmatic[2216135]: ANSWER M /root/.config/borg/security/<RandomID>/manifest-timestamp
Dec 26 14:28:40 start01 systemd-inhibit[2216135]: Exiting due to TERM signal

Expected behavior

I don't understand the difference between the two and where I need to look to get my backup to work as it did before my upgrades on Synology and Server.
I think it's more a SystemD trouble but not sure ...

Other notes / implementation ideas

No response

borgmatic version

1.8.3

borgmatic installation method

Debian package

Borg version

1.2.7

Python version

Python 3.11.2

Database version (if applicable)

No response

Operating system and version

Debian GNU/Linux 12 (bookworm) + testing package for borg & borgmatic

### What I'm trying to do and why Launch my borgbackup throught borgmatic with systemd service and timer ### My /etc/borgmatic/config.yaml ``` source_directories: - /home - /etc - /srv - /opt - /root - /lib/systemd/system/ - /var/www - /var/backups/ldap - /var/log/syslog* - /var/spool/cron/crontabs/ repositories: - path: ssh://<user>@<server>/path/borg01 label: <label> encryption_passphrase: "..." ssh_command: ssh -i /etc/borg/ssh/<keyname> -p <port> -o BatchMode=yes -o VerifyHostKeyDNS=yes keep_daily: 7 keep_weekly: 5 keep_monthly: 12 keep_yearly: 5 checks: - name: repository - name: archives frequency: 2 week - name: data frequency: 1 month check_last: 3 before_actions: - echo Starting actions - echo LDAP Git Backup - /usr/sbin/ldap-git-backup --commit-msg "Sauvegarde quotidienne avant Borgmatic" postgresql_databases: - name: all username: <user> password: <pass> mariadb_databases: - name: all hostname: <host> port: <port> username: <user> password: <pass> options: --single-transaction --quick healthchecks: ping_url: https://hc-ping.com/<personalID> ``` ### My /etc/systemd/system/borgmatic.service ``` [Unit] Description=borgmatic backup Wants=network-online.target After=network-online.target ConditionACPower=true [Service] Type=oneshot LockPersonality=true MemoryDenyWriteExecute=no NoNewPrivileges=yes PrivateDevices=yes PrivateTmp=yes ProtectClock=yes ProtectControlGroups=yes ProtectHostname=yes ProtectKernelLogs=yes ProtectKernelModules=yes ProtectKernelTunables=yes RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK RestrictNamespaces=yes RestrictRealtime=yes RestrictSUIDSGID=yes SystemCallArchitectures=native SystemCallFilter=@system-service SystemCallErrorNumber=EPERM ProtectSystem=full CapabilityBoundingSet=CAP_DAC_READ_SEARCH CAP_NET_RAW Nice=19 CPUSchedulingPolicy=batch IOSchedulingClass=best-effort IOSchedulingPriority=7 IOWeight=100 Restart=no LogRateLimitIntervalSec=0 ExecStartPre=sleep 1m ExecStart=systemd-inhibit --who="borgmatic" --what="sleep:shutdown" --why="Prevent interrupting scheduled backup" /usr/bin/borgmatic --syslog-verbosity 2 ``` ## On the server side, it's a Synology with - borg: 1.2.6 - borgmatic: 1.8.2 ### Steps to reproduce _No response_ ### Actual behavior Run the command manually as root, create the backup correctly, perform the checks and finish successfully. `/usr/bin/borgmatic --verbosity 1` ``` /etc/borgmatic/config.yaml: Pinging Healthchecks start /etc/borgmatic/config.yaml: Running 3 commands for pre-actions hook Starting actions LDAP Git Backup sechlama-b: Creating archive ssh://borg-backup@borg.mes.li/volume1/borg-backup/start01: Dumping MariaDB databases ssh://borg-backup@borg.mes.li/volume1/borg-backup/start01: Dumping PostgreSQL databases ssh://borg-backup@borg.mes.li/volume1/borg-backup/start01: Excluding special files to prevent Borg from hanging: /etc/systemd/system/php7.4-fpm.service, /etc/systemd/system/matrix-synapse.service, /etc/systemd/system/php8.0-fpm.service, /etc/systemd/system/php8.1-fpm.service, /etc/systemd/system/coturn.service, /etc/systemd/system/prosody.service, /lib/systemd/system/mdadm-waitidle.service, /lib/systemd/system/mdadm.service, /lib/systemd/system/screen-cleanup.service, /lib/systemd/system/sudo.service, /lib/systemd/system/cryptdisks-early.service, /lib/systemd/system/cryptdisks.service, /lib/systemd/system/hwclock.service, /lib/systemd/system/rc.service, /lib/systemd/system/rcS.service, /lib/systemd/system/x11-common.service, /root/recovery/dev/stderr, /root/recovery/dev/stdin, /root/recovery/dev/stdout, /root/recovery/etc/systemd/system/containerd.service, /root/recovery/etc/systemd/system/nginx.service, /root/recovery/etc/systemd/system/rtkit-daemon.service, /root/recovery/lib/systemd/system/bootlogd.service, /root/recovery/lib/systemd/system/bootlogs.service, /root/recovery/lib/systemd/system/bootmisc.service, /root/recovery/lib/systemd/system/checkfs.service, /root/recovery/lib/systemd/system/checkroot-bootclean.service, /root/recovery/lib/systemd/system/checkroot.service, /root/recovery/lib/systemd/system/cryptdisks-early.service, /root/recovery/lib/systemd/system/cryptdisks.service, /root/recovery/lib/systemd/system/fuse.service, /root/recovery/lib/systemd/system/halt.service, /root/recovery/lib/systemd/system/hostname.service, /root/recovery/lib/systemd/system/hwclock.service, /root/recovery/lib/systemd/system/killprocs.service, /root/recovery/lib/systemd/system/mdadm-waitidle.service, /root/recovery/lib/systemd/system/mdadm.service, /root/recovery/lib/systemd/system/motd.service, /root/recovery/lib/systemd/system/mountall-bootclean.service, /root/recovery/lib/systemd/system/mountall.service, /root/recovery/lib/systemd/system/mountdevsubfs.service, /root/recovery/lib/systemd/system/mountkernfs.service, /root/recovery/lib/systemd/system/mountnfs-bootclean.service, /root/recovery/lib/systemd/system/mountnfs.service, /root/recovery/lib/systemd/system/rc.service, /root/recovery/lib/systemd/system/rcS.service, /root/recovery/lib/systemd/system/reboot.service, /root/recovery/lib/systemd/system/rmnologin.service, /root/recovery/lib/systemd/system/screen-cleanup.service, /root/recovery/lib/systemd/system/sendsigs.service, /root/recovery/lib/systemd/system/single.service, /root/recovery/lib/systemd/system/stop-bootlogd-single.service, /root/recovery/lib/systemd/system/stop-bootlogd.service, /root/recovery/lib/systemd/system/umountfs.service, /root/recovery/lib/systemd/system/umountnfs.service, /root/recovery/lib/systemd/system/umountroot.service, /root/recovery/lib/systemd/system/x11-common.service, /root/recovery/tmp/borgmatic/postgresql_databases/localhost/all Creating archive at "ssh://<user>@<server>/path/borg01::borg01-2023-12-26T14:12:32.224960" <label>: Pruning archives <label>: Compacting segments Remote: compaction freed about 5.63 MB repository space. <label>: Running consistency checks Remote: Starting repository check Remote: finished segment check at segment 9917 Remote: Starting repository index check Remote: Index object count match. Remote: Finished full repository check, no problems found. Starting archive consistency check... Remote: Verified integrity of /path/borg01/index.9930 TAM-verified manifest Analyzing archive borg01-2023-12-23T18:04:44.905470 (1/3) TAM-verified archive Analyzing archive borg01-2023-12-25T06:46:16.925447 (2/3) TAM-verified archive Analyzing archive borg01-2023-12-26T15:01:19.092972 (3/3) TAM-verified archive RepositoryCache: current items 1140, size 44.12 MB / 2.15 GB, 2098 hits, 1140 misses, 0 slow misses (+0.0s), 0 evictions, 0 ENOSPC hit Orphaned objects check skipped (needs all archives checked). Archive consistency check complete, no problems found. RemoteRepository: 61.05 kB bytes sent, 62.44 MB bytes received, 1159 messages sent terminating with success status, rc 0 Writing check time at /root/.borgmatic/checks/<randomID>/repository Writing check time at /root/.borgmatic/checks/<randomID>/archives/<randomID> /etc/borgmatic/config.yaml: No commands to run for post-check hook /etc/borgmatic/config.yaml: No commands to run for post-actions hook /etc/borgmatic/config.yaml: Calling healthchecks hook function ping_monitor /etc/borgmatic/config.yaml: Pinging Healthchecks log /etc/borgmatic/config.yaml: Using Healthchecks ping URL https://hc-ping.com/<personalID> /etc/borgmatic/config.yaml: Calling healthchecks hook function ping_monitor /etc/borgmatic/config.yaml: Pinging Healthchecks finish /etc/borgmatic/config.yaml: Using Healthchecks ping URL https://hc-ping.com/<personalID> /etc/borgmatic/config.yaml: Calling healthchecks hook function destroy_monitor /etc/borgmatic/config.yaml: No commands to run for post-everything hook summary: /etc/borgmatic/config.yaml: Loading configuration file /etc/borgmatic/config.yaml: Successfully ran configuration file ``` ## BUT Starting the service manually never ends and remains inactive indefinitely `systemctl restart borgmatic.service &` ``` Dec 26 12:56:32 start01 systemd[1]: Starting borgmatic.service - borgmatic backup... Dec 26 12:58:32 start01 systemd-inhibit[2216135]: /etc/borgmatic/config.yaml: No commands to run for pre-everything hook Dec 26 12:58:32 start01 borgmatic[2216135]: DEBUG /etc/borgmatic/config.yaml: No commands to run for pre-everything hook Dec 26 12:58:32 start01 systemd-inhibit[2216135]: borg --version --debug --show-rc Dec 26 12:58:32 start01 borgmatic[2216135]: DEBUG borg --version --debug --show-rc Dec 26 12:58:33 start01 systemd-inhibit[2216135]: /etc/borgmatic/config.yaml: Calling healthchecks hook function initialize_monitor Dec 26 12:58:33 start01 systemd-inhibit[2216135]: /etc/borgmatic/config.yaml: Calling healthchecks hook function ping_monitor Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG /etc/borgmatic/config.yaml: Calling healthchecks hook function initialize_monitor Dec 26 12:58:33 start01 systemd-inhibit[2216135]: /etc/borgmatic/config.yaml: Pinging Healthchecks start Dec 26 12:58:33 start01 systemd-inhibit[2216135]: /etc/borgmatic/config.yaml: Using Healthchecks ping URL https://hc-ping.com/<personalID>/start Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG /etc/borgmatic/config.yaml: Calling healthchecks hook function ping_monitor Dec 26 12:58:33 start01 borgmatic[2216135]: INFO /etc/borgmatic/config.yaml: Pinging Healthchecks start Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG /etc/borgmatic/config.yaml: Using Healthchecks ping URL https://hc-ping.com/<personalID>/start Dec 26 12:58:33 start01 systemd-inhibit[2216135]: sechlama-b: Running actions for repository Dec 26 12:58:33 start01 systemd-inhibit[2216135]: /etc/borgmatic/config.yaml: No commands to run for pre-actions hook Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG sechlama-b: Running actions for repository Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG /etc/borgmatic/config.yaml: No commands to run for pre-actions hook Dec 26 12:58:33 start01 systemd-inhibit[2216135]: /etc/borgmatic/config.yaml: No commands to run for pre-backup hook Dec 26 12:58:33 start01 systemd-inhibit[2216135]: sechlama-b: Creating archive Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG /etc/borgmatic/config.yaml: No commands to run for pre-backup hook Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Calling mariadb_databases hook function remove_data_source_dumps Dec 26 12:58:33 start01 borgmatic[2216135]: INFO sechlama-b: Creating archive Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Removing MariaDB data source dumps Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Calling mariadb_databases hook function remove_data_source_dumps Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Removing MariaDB data source dumps Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Calling mysql_databases hook function remove_data_source_dumps Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Calling mysql_databases hook function remove_data_source_dumps Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Removing MySQL data source dumps Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Removing MySQL data source dumps Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Calling mongodb_databases hook function remove_data_source_dumps Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Calling mongodb_databases hook function remove_data_source_dumps Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Removing MongoDB data source dumps Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Removing MongoDB data source dumps Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Calling postgresql_databases hook function remove_data_source_dumps Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Calling postgresql_databases hook function remove_data_source_dumps Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Removing PostgreSQL data source dumps Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Removing PostgreSQL data source dumps Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Calling sqlite_databases hook function remove_data_source_dumps Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Calling sqlite_databases hook function remove_data_source_dumps Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Removing SQLite data source dumps Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Removing SQLite data source dumps Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Calling mariadb_databases hook function dump_data_sources Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Calling mariadb_databases hook function dump_data_sources Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Dumping MariaDB databases Dec 26 12:58:33 start01 borgmatic[2216135]: INFO ssh://<user>@<server>/path/borg01: Dumping MariaDB databases Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Querying for "all" MariaDB databases to dump Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Querying for "all" MariaDB databases to dump Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG mariadb --host <host> --port <port> --protocol tcp --user root --skip-column-names --batch --execute show schemas Dec 26 12:58:33 start01 systemd-inhibit[2216135]: mariadb --host <host> --port <port> --protocol tcp --user root --skip-column-names --batch --execute show schemas Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Dumping MariaDB database "all" to /root/.borgmatic/mariadb_databases/<host>/all Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Dumping MariaDB database "all" to /root/.borgmatic/mariadb_databases/<host>/all Dec 26 12:58:33 start01 systemd-inhibit[2216135]: mariadb-dump --single-transaction --quick --add-drop-database --host <host> --port <port> --protocol tcp --user root --databases BDCN cachet_db dolibarr dolibarr_flammelle dolibarr_labart freescout hubzilla_db i3d_wiki vimbadmin wp_flammelle --result-file /root/.borgmatic/mariadb_databases/<host>/all Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG mariadb-dump --single-transaction --quick --add-drop-database --host <host> --port <port> --protocol tcp --user root --databases BDCN cachet_db dolibarr dolibarr_flammelle dolibarr_labart freescout hubzilla_db i3d_wiki vimbadmin wp_flammelle --result-file /root/.borgmatic/mariadb_databases/<host>/all Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Calling postgresql_databases hook function dump_data_sources Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Calling postgresql_databases hook function dump_data_sources Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Dumping PostgreSQL databases Dec 26 12:58:33 start01 borgmatic[2216135]: INFO ssh://<user>@<server>/path/borg01: Dumping PostgreSQL databases Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Dumping PostgreSQL database "all" to /root/.borgmatic/postgresql_databases/<host>/all Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Dumping PostgreSQL database "all" to /root/.borgmatic/postgresql_databases/<host>/all Dec 26 12:58:33 start01 systemd-inhibit[2216135]: pg_dumpall --no-password --clean --if-exists --username postgres > /root/.borgmatic/postgresql_databases/<host>/all Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG pg_dumpall --no-password --clean --if-exists --username postgres > /root/.borgmatic/postgresql_databases/<host>/all Dec 26 12:58:33 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Collecting special file paths Dec 26 12:58:33 start01 systemd-inhibit[2216135]: borg create --one-file-system --read-special --list --filter AMEx- ssh://<user>@<server>/path/borg01::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /etc /home /lib/systemd/system/ /opt /root /srv /var/backups/ldap /var/log/syslog /var/log/syslog.1 /var/log/syslog.2.gz /var/log/syslog.3.gz /var/log/syslog.4.gz /var/> Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG ssh://<user>@<server>/path/borg01: Collecting special file paths Dec 26 12:58:33 start01 borgmatic[2216135]: DEBUG borg create --one-file-system --read-special --list --filter AMEx- ssh://<user>@<server>/path/borg01::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /etc /home /lib/systemd/system/ /opt /root /srv /var/backups/ldap /var/log/syslog /var/log/syslog.1 /var/log/syslog.2.gz /var/log/syslog.3.gz /var/log/syslog.4.gz /var/> Dec 26 12:59:11 start01 borgmatic[2216135]: WARNING ssh://<user>@<server>/path/borg01: Excluding special files to prevent Borg from hanging: /etc/systemd/system/php7.4-fpm.service, /etc/systemd/system/matrix-synapse.service, /etc/systemd/system/php8.0-fpm.service, /etc/systemd/system/php8.1-fpm.service, /etc/systemd/system/coturn.service, /etc/systemd/syst> Dec 26 12:59:11 start01 systemd-inhibit[2216135]: ssh://<user>@<server>/path/borg01: Excluding special files to prevent Borg from hanging: /etc/systemd/system/php7.4-fpm.service, /etc/systemd/system/matrix-synapse.service, /etc/systemd/system/php8.0-fpm.service, /etc/systemd/system/php8.1-fpm.service, /etc/systemd/system/coturn.service, /etc/systemd/system> Dec 26 12:59:11 start01 systemd-inhibit[2216135]: borg create --one-file-system --read-special --list --filter AMEx- --exclude-from /tmp/tmp2io923g_ --stats --debug --show-rc ssh://<user>@<server>/path/borg01::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /etc /home /lib/systemd/system/ /opt /root /srv /var/backups/ldap /var/log/syslog /var/log/syslog.1 /var/log/s> Dec 26 12:59:11 start01 borgmatic[2216135]: DEBUG borg create --one-file-system --read-special --list --filter AMEx- --exclude-from /tmp/tmp2io923g_ --stats --debug --show-rc ssh://<user>@<server>/path/borg01::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /etc /home /lib/systemd/system/ /opt /root /srv /var/backups/ldap /var/log/syslog /var/log/syslog.1 /var/log/s> Dec 26 12:59:11 start01 systemd-inhibit[2216135]: using builtin fallback logging configuration Dec 26 12:59:11 start01 borgmatic[2216135]: ANSWER using builtin fallback logging configuration Dec 26 12:59:11 start01 systemd-inhibit[2216135]: 33 self tests completed in 0.08 seconds Dec 26 12:59:11 start01 borgmatic[2216135]: ANSWER 33 self tests completed in 0.08 seconds Dec 26 12:59:11 start01 systemd-inhibit[2216135]: SSH command line: ['ssh', '-i', '/etc/borg/ssh/<keyName>', '-p', '<port>', '-o', 'BatchMode=yes', '-o', 'VerifyHostKeyDNS=yes', '<borgUser>@<FQDN>', 'borg', 'serve', '--debug'] Dec 26 12:59:11 start01 borgmatic[2216135]: ANSWER SSH command line: ['ssh', '-i', '/etc/borg/ssh/<keyName>', '-p', '<port>', '-o', 'BatchMode=yes', '-o', 'VerifyHostKeyDNS=yes', '<borgUser>@<FQDN>', 'borg', 'serve', '--debug'] Dec 26 12:59:14 start01 systemd-inhibit[2216135]: Remote: using builtin fallback logging configuration Dec 26 12:59:14 start01 borgmatic[2216135]: ANSWER Remote: using builtin fallback logging configuration Dec 26 12:59:15 start01 systemd-inhibit[2216135]: Remote: 33 self tests completed in 1.16 seconds Dec 26 12:59:15 start01 borgmatic[2216135]: ANSWER Remote: 33 self tests completed in 1.16 seconds Dec 26 12:59:15 start01 systemd-inhibit[2216135]: Remote: using builtin fallback logging configuration Dec 26 12:59:15 start01 borgmatic[2216135]: ANSWER Remote: using builtin fallback logging configuration Dec 26 12:59:15 start01 borgmatic[2216135]: ANSWER Remote: Initialized logging system for JSON-based protocol Dec 26 12:59:15 start01 systemd-inhibit[2216135]: Remote: Initialized logging system for JSON-based protocol Dec 26 12:59:15 start01 systemd-inhibit[2216135]: Remote: Resolving repository path b'/<path>/borg01' Dec 26 12:59:15 start01 borgmatic[2216135]: ANSWER Remote: Resolving repository path b'/<path>/borg01' Dec 26 12:59:15 start01 borgmatic[2216135]: ANSWER Remote: Resolved repository path to '/<path>/borg01' Dec 26 12:59:15 start01 systemd-inhibit[2216135]: Remote: Resolved repository path to '/<path>/borg01' Dec 26 12:59:16 start01 systemd-inhibit[2216135]: Remote: Verified integrity of /<path>/borg01/index.9891 Dec 26 12:59:16 start01 borgmatic[2216135]: ANSWER Remote: Verified integrity of /<path>/borg01/index.9891 Dec 26 12:59:16 start01 systemd-inhibit[2216135]: TAM-verified manifest Dec 26 12:59:16 start01 borgmatic[2216135]: ANSWER TAM-verified manifest Dec 26 12:59:16 start01 systemd-inhibit[2216135]: security: read previous location 'ssh://<user>@<server>/path/borg01' Dec 26 12:59:16 start01 borgmatic[2216135]: ANSWER security: read previous location 'ssh://<user>@<server>/path/borg01' Dec 26 12:59:16 start01 systemd-inhibit[2216135]: security: read manifest timestamp '2023-12-26T09:43:15.962630' Dec 26 12:59:16 start01 borgmatic[2216135]: ANSWER security: read manifest timestamp '2023-12-26T09:43:15.962630' Dec 26 12:59:16 start01 systemd-inhibit[2216135]: security: determined newest manifest timestamp as 2023-12-26T09:43:15.962630 Dec 26 12:59:16 start01 borgmatic[2216135]: ANSWER security: determined newest manifest timestamp as 2023-12-26T09:43:15.962630 Dec 26 12:59:16 start01 systemd-inhibit[2216135]: security: repository checks ok, allowing access Dec 26 12:59:16 start01 borgmatic[2216135]: ANSWER security: repository checks ok, allowing access Dec 26 12:59:16 start01 borgmatic[2216135]: ANSWER Creating archive at "ssh://<user>@<server>/path/borg01::start01-2023-12-26T12:59:11.493256" Dec 26 12:59:16 start01 systemd-inhibit[2216135]: Creating archive at "ssh://<user>@<server>/path/borg01::start01-2023-12-26T12:59:11.493256" Dec 26 12:59:16 start01 systemd-inhibit[2216135]: Verified integrity of /root/.cache/borg/<RandomID>/chunks Dec 26 12:59:16 start01 systemd-inhibit[2216135]: Reading files cache ... Dec 26 12:59:16 start01 borgmatic[2216135]: ANSWER Verified integrity of /root/.cache/borg/<RandomID>/chunks Dec 26 12:59:16 start01 borgmatic[2216135]: ANSWER Reading files cache ... Dec 26 12:59:18 start01 systemd-inhibit[2216135]: Verified integrity of /root/.cache/borg/<RandomID>/files Dec 26 12:59:18 start01 systemd-inhibit[2216135]: security: read previous location 'ssh://<user>@<server>/path/borg01' Dec 26 12:59:18 start01 systemd-inhibit[2216135]: security: read manifest timestamp '2023-12-26T09:43:15.962630' Dec 26 12:59:18 start01 systemd-inhibit[2216135]: security: determined newest manifest timestamp as 2023-12-26T09:43:15.962630 Dec 26 12:59:18 start01 systemd-inhibit[2216135]: security: repository checks ok, allowing access Dec 26 12:59:18 start01 borgmatic[2216135]: ANSWER Verified integrity of /root/.cache/borg/<RandomID>/files Dec 26 12:59:18 start01 systemd-inhibit[2216135]: Processing files ... Dec 26 12:59:18 start01 borgmatic[2216135]: ANSWER security: read previous location 'ssh://<user>@<server>/path/borg01' Dec 26 12:59:18 start01 borgmatic[2216135]: ANSWER security: read manifest timestamp '2023-12-26T09:43:15.962630' Dec 26 12:59:18 start01 borgmatic[2216135]: ANSWER security: determined newest manifest timestamp as 2023-12-26T09:43:15.962630 Dec 26 12:59:18 start01 borgmatic[2216135]: ANSWER security: repository checks ok, allowing access Dec 26 12:59:18 start01 borgmatic[2216135]: ANSWER Processing files ... Dec 26 12:59:19 start01 systemd-inhibit[2216135]: M /etc/systemd/system/borgmatic.service Dec 26 12:59:19 start01 borgmatic[2216135]: ANSWER M /etc/systemd/system/borgmatic.service Dec 26 12:59:19 start01 systemd-inhibit[2216135]: x /etc/systemd/system/php7.4-fpm.service Dec 26 12:59:19 start01 borgmatic[2216135]: ANSWER x /etc/systemd/system/php7.4-fpm.service Dec 26 12:59:19 start01 systemd-inhibit[2216135]: x /etc/systemd/system/matrix-synapse.service Dec 26 12:59:19 start01 borgmatic[2216135]: ANSWER x /etc/systemd/system/matrix-synapse.service Dec 26 12:59:19 start01 borgmatic[2216135]: ANSWER x /etc/systemd/system/php8.0-fpm.service Dec 26 12:59:19 start01 systemd-inhibit[2216135]: x /etc/systemd/system/php8.0-fpm.service Dec 26 12:59:19 start01 systemd-inhibit[2216135]: x /etc/systemd/system/php8.1-fpm.service Dec 26 12:59:19 start01 borgmatic[2216135]: ANSWER x /etc/systemd/system/php8.1-fpm.service Dec 26 12:59:19 start01 borgmatic[2216135]: ANSWER x /etc/systemd/system/coturn.service Dec 26 12:59:19 start01 systemd-inhibit[2216135]: x /etc/systemd/system/coturn.service Dec 26 12:59:19 start01 systemd-inhibit[2216135]: x /etc/systemd/system/prosody.service Dec 26 12:59:19 start01 borgmatic[2216135]: ANSWER x /etc/systemd/system/prosody.service Dec 26 12:59:20 start01 systemd-inhibit[2216135]: x /lib/systemd/system/mdadm-waitidle.service Dec 26 12:59:20 start01 systemd-inhibit[2216135]: x /lib/systemd/system/mdadm.service Dec 26 12:59:20 start01 borgmatic[2216135]: ANSWER x /lib/systemd/system/mdadm-waitidle.service Dec 26 12:59:20 start01 borgmatic[2216135]: ANSWER x /lib/systemd/system/mdadm.service Dec 26 12:59:20 start01 systemd-inhibit[2216135]: x /lib/systemd/system/screen-cleanup.service Dec 26 12:59:20 start01 borgmatic[2216135]: ANSWER x /lib/systemd/system/screen-cleanup.service Dec 26 12:59:20 start01 systemd-inhibit[2216135]: x /lib/systemd/system/sudo.service Dec 26 12:59:20 start01 borgmatic[2216135]: ANSWER x /lib/systemd/system/sudo.service Dec 26 12:59:20 start01 systemd-inhibit[2216135]: x /lib/systemd/system/cryptdisks-early.service Dec 26 12:59:20 start01 borgmatic[2216135]: ANSWER x /lib/systemd/system/cryptdisks-early.service Dec 26 12:59:20 start01 borgmatic[2216135]: ANSWER x /lib/systemd/system/cryptdisks.service Dec 26 12:59:20 start01 systemd-inhibit[2216135]: x /lib/systemd/system/cryptdisks.service Dec 26 12:59:20 start01 systemd-inhibit[2216135]: x /lib/systemd/system/hwclock.service Dec 26 12:59:20 start01 borgmatic[2216135]: ANSWER x /lib/systemd/system/hwclock.service Dec 26 12:59:20 start01 systemd-inhibit[2216135]: x /lib/systemd/system/rc.service Dec 26 12:59:20 start01 borgmatic[2216135]: ANSWER x /lib/systemd/system/rc.service Dec 26 12:59:20 start01 systemd-inhibit[2216135]: x /lib/systemd/system/rcS.service Dec 26 12:59:20 start01 borgmatic[2216135]: ANSWER x /lib/systemd/system/rcS.service Dec 26 12:59:20 start01 systemd-inhibit[2216135]: x /lib/systemd/system/x11-common.service Dec 26 12:59:20 start01 borgmatic[2216135]: ANSWER x /lib/systemd/system/x11-common.service Dec 26 12:59:22 start01 systemd-inhibit[2216135]: M /root/.bash_history Dec 26 12:59:22 start01 borgmatic[2216135]: ANSWER M /root/.bash_history Dec 26 12:59:22 start01 systemd-inhibit[2216135]: Remote: Cleaned up 1 uncommitted segment files (== everything after segment 9891). Dec 26 12:59:22 start01 borgmatic[2216135]: ANSWER Remote: Cleaned up 1 uncommitted segment files (== everything after segment 9891). Dec 26 12:59:22 start01 borgmatic[2216135]: ANSWER Remote: Verified integrity of /<path>/borg01/hints.9891 Dec 26 12:59:22 start01 systemd-inhibit[2216135]: Remote: Verified integrity of /<path>/borg01/hints.9891 Dec 26 12:59:22 start01 systemd-inhibit[2216135]: M /root/.config/borg/security/<RandomID>/nonce Dec 26 12:59:22 start01 borgmatic[2216135]: ANSWER M /root/.config/borg/security/<RandomID>/nonce Dec 26 12:59:22 start01 systemd-inhibit[2216135]: M /root/.config/borg/security/<RandomID>/location Dec 26 12:59:22 start01 borgmatic[2216135]: ANSWER M /root/.config/borg/security/<RandomID>/location Dec 26 12:59:22 start01 systemd-inhibit[2216135]: M /root/.config/borg/security/<RandomID>/key-type Dec 26 12:59:22 start01 borgmatic[2216135]: ANSWER M /root/.config/borg/security/<RandomID>/key-type Dec 26 12:59:22 start01 systemd-inhibit[2216135]: M /root/.config/borg/security/<RandomID>/manifest-timestamp Dec 26 12:59:22 start01 borgmatic[2216135]: ANSWER M /root/.config/borg/security/<RandomID>/manifest-timestamp Dec 26 14:28:40 start01 systemd-inhibit[2216135]: Exiting due to TERM signal ``` ### Expected behavior I don't understand the difference between the two and where I need to look to get my backup to work as it did before my upgrades on Synology and Server. I think it's more a SystemD trouble but not sure ... ### Other notes / implementation ideas _No response_ ### borgmatic version 1.8.3 ### borgmatic installation method Debian package ### Borg version 1.2.7 ### Python version Python 3.11.2 ### Database version (if applicable) _No response_ ### Operating system and version Debian GNU/Linux 12 (bookworm) + testing package for borg & borgmatic
Auroch changed title from My borgmatic does not terminate with systemd to My borgmatic no more terminate with systemd after Synology and Debian Server upgrade 2023-12-26 14:22:03 +00:00
Auroch changed title from My borgmatic no more terminate with systemd after Synology and Debian Server upgrade to After upgrading Synology and Debian Server, my borgmatic no longer terminates with systemd 2023-12-26 14:24:06 +00:00
witten commented 2023-12-26 17:33:20 +00:00
Owner
Copy Link

I'm not exactly sure what's going on here, but I do have some ideas:

  1. You could be experiencing a hang due to the database hook. This shouldn't happen, but it's possible there's an edge case where it does. See the Limitations section for ideas on how to troubleshoot. And/or just comment out your database hooks to see if that makes the problem go away.
  2. Or there could be an issue with the systemd security options. To test that theory, try commenting out everything from LockPersonality to CapabilityBoundingSet, reload the service, and then try to reproduce the problem.

I see that when you run borgmatic with systemd, the log ends with Exiting due to TERM signal. Is that you manually killing the process? Because if the process remains active indefinitely, I would not expect to see it exit.

I'm not exactly sure what's going on here, but I do have some ideas: 1. You could be experiencing [a hang due to the database hook](https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#borgmatic-hangs-during-backup). This _shouldn't_ happen, but it's possible there's an edge case where it does. See the [Limitations](https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#limitations) section for ideas on how to troubleshoot. And/or just comment out your database hooks to see if that makes the problem go away. 2. Or there could be an issue with the systemd security options. To test that theory, try commenting out everything from `LockPersonality` to `CapabilityBoundingSet`, reload the service, and then try to reproduce the problem. I see that when you run borgmatic with systemd, the log ends with `Exiting due to TERM signal`. Is that you manually killing the process? Because if the process remains active indefinitely, I would not expect to see it exit.
witten added the
question / support
 label 2023-12-26 17:33:32 +00:00
Auroch commented 2023-12-26 23:34:06 +00:00
Author
Copy Link

I'm not exactly sure what's going on here, but I do have some ideas:

  1. You could be experiencing a hang due to the database hook. This shouldn't happen, but it's possible there's an edge case where it does. See the Limitations section for ideas on how to troubleshoot. And/or just comment out your database hooks to see if that makes the problem go away.

Indeed, when I comment out the postgres and mariadb lines in config.yaml, the systemd service terminates successfully
I've tried with each one separately, but as soon as there's a DB dump, cel skates around in a vacuum.

However, I don't understand why it works when I run the command manually.

I'll run it with a cron tonight and post the result

Thank you very much for this step forward.

Any other ideas for solving this problem?

  1. Or there could be an issue with the systemd security options. To test that theory, try commenting out everything from LockPersonality to CapabilityBoundingSet, reload the service, and then try to reproduce the problem.

Nothing change with this test

I see that when you run borgmatic with systemd, the log ends with Exiting due to TERM signal. Is that you manually killing the process? Because if the process remains active indefinitely, I would not expect to see it exit.

Yes, I killed -HUP the process

> I'm not exactly sure what's going on here, but I do have some ideas: > > 1. You could be experiencing [a hang due to the database hook](https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#borgmatic-hangs-during-backup). This _shouldn't_ happen, but it's possible there's an edge case where it does. See the [Limitations](https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#limitations) section for ideas on how to troubleshoot. And/or just comment out your database hooks to see if that makes the problem go away. Indeed, when I comment out the postgres and mariadb lines in config.yaml, the systemd service terminates successfully I've tried with each one separately, but as soon as there's a DB dump, cel skates around in a vacuum. However, I don't understand why it works when I run the command manually. I'll run it with a cron tonight and post the result Thank you very much for this step forward. Any other ideas for solving this problem? > 2. Or there could be an issue with the systemd security options. To test that theory, try commenting out everything from `LockPersonality` to `CapabilityBoundingSet`, reload the service, and then try to reproduce the problem. > Nothing change with this test > I see that when you run borgmatic with systemd, the log ends with `Exiting due to TERM signal`. Is that you manually killing the process? Because if the process remains active indefinitely, I would not expect to see it exit. Yes, I killed -HUP the process
witten commented 2023-12-26 23:54:12 +00:00
Owner
Copy Link

One difference I can see in the output is that the log entry for Excluding special files to prevent Borg from hanging shows many more files when you're running borgmatic manually compared to when you're running with systemd. That might indicate that some files that should be excluded to prevent hanging aren't being excluded. This file in particular looks pretty interesting: /root/recovery/tmp/borgmatic/postgresql_databases/localhost/all. If it's an old named pipe that for whatever reason isn't getting excluded with systemd, that could cause Borg to hang.

One difference I can see in the output is that the log entry for `Excluding special files to prevent Borg from hanging` shows many more files when you're running borgmatic manually compared to when you're running with systemd. That might indicate that some files that should be excluded to prevent hanging aren't being excluded. This file in particular looks pretty interesting: `/root/recovery/tmp/borgmatic/postgresql_databases/localhost/all`. If it's an old named pipe that for whatever reason isn't getting excluded with systemd, that could cause Borg to hang.
Auroch commented 2023-12-27 10:44:38 +00:00
Author
Copy Link

You were right !
I've deleted these broken elements and taken the opportunity to archive the unnecessary content.
Since then, the systemd service has been working perfectly.

(I won't close the ticket in case you need any further information, otherwise feel free to close this ticket)

Thank you very very much !

You were right ! I've deleted these broken elements and taken the opportunity to archive the unnecessary content. Since then, the systemd service has been working perfectly. (I won't close the ticket in case you need any further information, otherwise feel free to close this ticket) Thank you very very much !
witten commented 2023-12-27 18:07:56 +00:00
Owner
Copy Link

Glad to hear that did it for you! The one thing I don't understand though is why those problem files got auto-excluded when borgmatic was run manually but not when run under systemd. I'm fine closing this for now, but if this reoccurs again, please let me know and we can dive into how those various systemd service options may be impacting this. Thanks!

Glad to hear that did it for you! The one thing I don't understand though is why those problem files got auto-excluded when borgmatic was run manually but not when run under systemd. I'm fine closing this for now, but if this reoccurs again, please let me know and we can dive into how those various systemd service options may be impacting this. Thanks!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
main
spot-check
1.8.11
1.8.10
1.8.9
1.8.8
1.8.7
1.8.6
1.8.5
1.8.4
1.8.3
1.8.2
1.8.1
1.8.0
1.7.15
1.7.14
1.7.13
1.7.12
1.7.11
1.7.10
1.7.9
1.7.8
1.7.7
1.7.6
1.7.5
1.7.4
1.7.3
1.7.2
1.7.1
1.7.0
1.6.6
1.6.5
1.6.4
1.6.3
1.6.2
1.6.1
1.6.0
1.5.24
1.5.23
1.5.22
1.5.21
1.5.20
1.5.19
1.5.18
1.5.17
1.5.16
1.5.15
1.5.14
1.5.13
1.5.12
1.5.11
1.5.10
1.5.9
1.5.8
1.5.7
1.5.7.dev0
1.5.6
1.5.5
1.5.4
1.5.3
1.5.2
1.5.1
1.5.0
1.4.22
1.4.21
1.4.20
1.4.19
1.4.18
1.4.17
1.4.16
1.4.15
1.4.14
1.4.13
1.4.12
1.4.11
1.4.10
1.4.9
1.4.8
1.4.7
1.4.6
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0
1.3.26
1.3.25
1.3.24
1.3.23
1.3.22
1.3.21
1.3.20
1.3.19
1.3.18
1.3.17
1.3.16
1.3.15
1.3.14
1.3.13
1.3.12
1.3.11
1.3.10
1.3.9
1.3.8
1.3.7
1.3.6
1.3.5
1.3.4
1.3.3
1.3.2
1.3.1
1.3.0
1.2.18
1.2.17
1.2.16
1.2.15
1.2.14
1.2.13
1.2.12
1.2.11
1.2.10
1.2.9
1.2.8
1.2.7
1.2.6
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.15
1.1.14
1.1.13
1.1.12
1.1.11
1.1.10
1.1.9
1.1.7
1.1.6
1.1.5
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.3
1.0.2
1.0.1
1.0.0
0.1.8
0.1.7
0.1.6
0.1.5
0.1.4
0.1.3
0.1.2
0.1.1
0.1.0
0.0.7
0.0.6
0.0.5
0.0.4
0.0.3
0.0.2
Labels
Clear labels   
bug

   
data loss

   
design finalized

   
good first issue

   
new feature area

   
question / support

   
security

   
waiting for response
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#801
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?