[admin@frontserver ~]$ sudo borgmatic --verbosity 2 [sudo] password for admin: Ensuring legacy configuration is upgraded /etc/borgmatic/config.yaml: No commands to run for pre-everything hook /etc/borgmatic/config.yaml: No commands to run for pre-prune hook /etc/borgmatic/config.yaml: No commands to run for pre-backup hook /etc/borgmatic/config.yaml: No commands to run for pre-check hook /mnt/RAID1-699GiB/backup/borg/frontserver: Pruning archives borg prune --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --keep-yearly 2 --prefix {hostname}- --debug --show-rc /mnt/RAID1-699GiB/backup/borg/frontserver using builtin fallback logging configuration 35 self tests completed in 0.23 seconds Verified integrity of /mnt/RAID1-699GiB/backup/borg/frontserver/index.307 TAM-verified manifest security: read previous location '/mnt/RAID1-699GiB/backup/borg/frontserver' security: read manifest timestamp '2021-11-16T09:32:03.051941' security: determined newest manifest timestamp as 2021-11-16T09:32:03.051941 security: repository checks ok, allowing access Verified integrity of /root/.cache/borg/e5e3763b02c1b0f03c268964ab39342b11b2bfbfb877f9a3a1e02ca945c525a3/chunks security: read previous location '/mnt/RAID1-699GiB/backup/borg/frontserver' security: read manifest timestamp '2021-11-16T09:32:03.051941' security: determined newest manifest timestamp as 2021-11-16T09:32:03.051941 security: repository checks ok, allowing access terminating with success status, rc 0 /mnt/RAID1-699GiB/backup/borg/frontserver: Creating archive borg create --compression zstd --debug --show-rc /mnt/RAID1-699GiB/backup/borg/frontserver::{hostname}-{now:%Y-%m-%d-%H%M%S} /etc /home /opt /root /srv /usr/local /var/account /var/adm /var/db /var/empty /var/ftp /var/games /var/gopher /var/kerberos /var/lib /var/local /var/log /var/nis /var/opt /var/preserve /var/spool/cron /var/www /var/yp using builtin fallback logging configuration 35 self tests completed in 0.23 seconds Verified integrity of /mnt/RAID1-699GiB/backup/borg/frontserver/index.307 TAM-verified manifest security: read previous location '/mnt/RAID1-699GiB/backup/borg/frontserver' security: read manifest timestamp '2021-11-16T09:32:03.051941' security: determined newest manifest timestamp as 2021-11-16T09:32:03.051941 security: repository checks ok, allowing access Creating archive at "/mnt/RAID1-699GiB/backup/borg/frontserver::{hostname}-{now:%Y-%m-%d-%H%M%S}" Verified integrity of /root/.cache/borg/e5e3763b02c1b0f03c268964ab39342b11b2bfbfb877f9a3a1e02ca945c525a3/chunks Reading files cache ... Verified integrity of /root/.cache/borg/e5e3763b02c1b0f03c268964ab39342b11b2bfbfb877f9a3a1e02ca945c525a3/files security: read previous location '/mnt/RAID1-699GiB/backup/borg/frontserver' security: read manifest timestamp '2021-11-16T09:32:03.051941' security: determined newest manifest timestamp as 2021-11-16T09:32:03.051941 security: repository checks ok, allowing access Processing files ... Cleaned up 0 uncommitted segment files (== everything after segment 307). Verified integrity of /mnt/RAID1-699GiB/backup/borg/frontserver/hints.307 check_free_space: few segments, not requiring a full free segment check_free_space: calculated working space for compact as 545259520 bytes check_free_space: required bytes 587207478, free bytes 257200144384 security: saving state for e5e3763b02c1b0f03c268964ab39342b11b2bfbfb877f9a3a1e02ca945c525a3 to /root/.config/borg/security/e5e3763b02c1b0f03c268964ab39342b11b2bfbfb877f9a3a1e02ca945c525a3 security: current location /mnt/RAID1-699GiB/backup/borg/frontserver security: key type 5 security: manifest timestamp 2021-11-16T18:00:15.529652 terminating with success status, rc 0 /mnt/RAID1-699GiB/backup/borg/frontserver: Running consistency checks borg check --prefix {hostname}- --debug --show-rc /mnt/RAID1-699GiB/backup/borg/frontserver using builtin fallback logging configuration 35 self tests completed in 0.32 seconds Starting repository check Verified integrity of /mnt/RAID1-699GiB/backup/borg/frontserver/index.311 Read committed index of transaction 311 Segment transaction is 311 Determined transaction is 311 Found 170 segments checking segment file /mnt/RAID1-699GiB/backup/borg/frontserver/data/0/1... checking segment file /mnt/RAID1-699GiB/backup/borg/frontserver/data/0/2... ... checking segment file /mnt/RAID1-699GiB/backup/borg/frontserver/data/0/310... checking segment file /mnt/RAID1-699GiB/backup/borg/frontserver/data/0/311... Starting repository index check Index object count match. Completed repository check, no problems found. Starting archive consistency check... Verified integrity of /mnt/RAID1-699GiB/backup/borg/frontserver/index.311 TAM-verified manifest Analyzing archive frontserver-2021-11-16-092834 (1/2) Analyzing archive frontserver-2021-11-16-175729 (2/2) Orphaned objects check skipped (needs all archives checked). Archive consistency check complete, no problems found. terminating with success status, rc 0 /etc/borgmatic/config.yaml: No commands to run for post-prune hook /etc/borgmatic/config.yaml: No commands to run for post-backup hook /etc/borgmatic/config.yaml: No commands to run for post-check hook /etc/borgmatic/config.yaml: No commands to run for post-everything hook summary: /etc/borgmatic/config.yaml: Successfully ran configuration file [admin@frontserver ~]$ sudo systemctl start borgmatic.service [sudo] password for admin: Job for borgmatic.service failed because the control process exited with error code. See "systemctl status borgmatic.service" and "journalctl -xe" for details. [admin@frontserver ~]$ sudo journalctl -xe [sudo] password for admin: ... Nov 16 19:09:56 frontserver.lan borgmatic[2552172]: /mnt/RAID1-699GiB/backup/borg/frontserver: Error running actions for repository Nov 16 19:09:56 frontserver.lan borgmatic[2552172]: CRITICAL /mnt/RAID1-699GiB/backup/borg/frontserver: Error running actions for repository Nov 16 19:09:56 frontserver.lan borgmatic[2552172]: Command 'borg prune --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --keep-yearly 2 --prefix {hostname}- /mnt/RAID1-699GiB/backup/borg/frontserver' returned non-zero exit status 2. Nov 16 19:09:56 frontserver.lan borgmatic[2552172]: CRITICAL Command 'borg prune --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --keep-yearly 2 --prefix {hostname}- /mnt/RAID1-699GiB/backup/borg/frontserver' returned non-zero exit status 2. Nov 16 19:09:56 frontserver.lan borgmatic[2552172]: /etc/borgmatic/config.yaml: Error running configuration file Nov 16 19:09:56 frontserver.lan borgmatic[2552172]: CRITICAL /etc/borgmatic/config.yaml: Error running configuration file Nov 16 19:09:56 frontserver.lan borgmatic[2552172]: CRITICAL Nov 16 19:09:56 frontserver.lan borgmatic[2552172]: summary: Nov 16 19:09:56 frontserver.lan borgmatic[2552172]: CRITICAL summary: Nov 16 19:09:56 frontserver.lan borgmatic[2552172]: /etc/borgmatic/config.yaml: Error running configuration file Nov 16 19:09:56 frontserver.lan borgmatic[2552172]: CRITICAL /etc/borgmatic/config.yaml: Error running configuration file Nov 16 19:09:56 frontserver.lan borgmatic[2552172]: /mnt/RAID1-699GiB/backup/borg/frontserver: Error running actions for repository Nov 16 19:09:56 frontserver.lan borgmatic[2552172]: CRITICAL /mnt/RAID1-699GiB/backup/borg/frontserver: Error running actions for repository Nov 16 19:09:56 frontserver.lan borgmatic[2552172]: Failed to create/acquire the lock /mnt/RAID1-699GiB/backup/borg/frontserver/lock.exclusive ([Errno 13] Permission denied: '/mnt/RAID1-699GiB/backup/borg/frontserver/lock.exclusive'). Nov 16 19:09:56 frontserver.lan borgmatic[2552172]: Traceback (most recent call last): Nov 16 19:09:56 frontserver.lan borgmatic[2552172]: File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line 4703, in main Nov 16 19:09:56 frontserver.lan borgmatic[2552172]: exit_code = archiver.run(args) Nov 16 19:09:56 frontserver.lan borgmatic[2552172]: File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line 4635, in run Nov 16 19:09:56 frontserver.lan borgmatic[2552172]: Command 'borg prune --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --keep-yearly 2 --prefix {hostname}- /mnt/RAID1-699GiB/backup/borg/frontserver' returned non-zero exit status 2. Nov 16 19:09:56 frontserver.lan borgmatic[2552172]: CRITICAL Failed to create/acquire the lock /mnt/RAID1-699GiB/backup/borg/frontserver/lock.exclusive ([Errno 13] Permission denied: '/mnt/RAID1-699GiB/backup/borg/frontserver/lock.exclusive'). Traceback (most recent call last): File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line 4703, in main exit_code = archiver.run(args) File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line 4635, in run Nov 16 19:09:56 frontserver.lan borgmatic[2552172]: CRITICAL Command 'borg prune --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --keep-yearly 2 --prefix {hostname}- /mnt/RAID1-699GiB/backup/borg/frontserver' returned non-zero exit status 2. Nov 16 19:09:56 frontserver.lan borgmatic[2552172]: CRITICAL Nov 16 19:09:56 frontserver.lan borgmatic[2552172]: Need some help? https://torsion.org/borgmatic/#issues Nov 16 19:09:56 frontserver.lan borgmatic[2552172]: CRITICAL Need some help? https://torsion.org/borgmatic/#issues Nov 16 19:09:56 frontserver.lan systemd[1]: systemd-journald.service: Received EPOLLHUP on stored fd 38 (stored), closing. Nov 16 19:09:56 frontserver.lan systemd[1]: Received SIGCHLD from PID 2552172 (borgmatic). Nov 16 19:09:56 frontserver.lan systemd[1]: Child 2552172 (borgmatic) died (code=exited, status=1/FAILURE) Nov 16 19:09:56 frontserver.lan systemd[1]: borgmatic.service: Child 2552172 belongs to borgmatic.service. Nov 16 19:09:56 frontserver.lan systemd[1]: borgmatic.service: Main process exited, code=exited, status=1/FAILURE Nov 16 19:09:56 frontserver.lan systemd[1]: borgmatic.service: Failed with result 'exit-code'. -- Subject: Unit failed -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- The unit borgmatic.service has entered the 'failed' state with result 'exit-code'. Nov 16 19:09:56 frontserver.lan systemd[1]: borgmatic.service: Changed start -> failed Nov 16 19:09:56 frontserver.lan systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=21 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a Nov 16 19:09:56 frontserver.lan systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=231362 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a Nov 16 19:09:56 frontserver.lan systemd[1]: borgmatic.service: Job borgmatic.service/start finished, result=failed Nov 16 19:09:56 frontserver.lan systemd[1]: Failed to start borgmatic backup. -- Subject: Unit borgmatic.service has failed -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit borgmatic.service has failed. -- -- The result is failed. Nov 16 19:09:56 frontserver.lan systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=JobRemoved cookie=22 reply_cookie=0 signature=uoss error-name=n/a error-message=n/a Nov 16 19:09:56 frontserver.lan systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=JobRemoved cookie=231363 reply_cookie=0 signature=uoss error-name=n/a error-message=n/a Nov 16 19:09:56 frontserver.lan systemd[1]: borgmatic.timer: Adding 1.296899s random time. Nov 16 19:09:56 frontserver.lan sudo[2552030]: pam_unix(sudo:session): session closed for user root Nov 16 19:09:56 frontserver.lan systemd[1]: borgmatic.timer: Realtime timer elapses at Wed 2021-11-17 03:00:01 GMT. Nov 16 19:09:56 frontserver.lan systemd[1]: borgmatic.service: Unit entered failed state. Nov 16 19:09:56 frontserver.lan systemd[1]: borgmatic.timer: Adding 21.853468s random time. Nov 16 19:09:56 frontserver.lan systemd[1]: borgmatic.timer: Realtime timer elapses at Wed 2021-11-17 03:00:21 GMT. Nov 16 19:09:56 frontserver.lan systemd[1]: Spawning thread to nuke /tmp/systemd-private-13a9d6c34d38468dac7ce2692ccd524f-borgmatic.service-MNBjwg Nov 16 19:09:56 frontserver.lan systemd[1]: Spawning thread to nuke /var/tmp/systemd-private-13a9d6c34d38468dac7ce2692ccd524f-borgmatic.service-PCDz3e Nov 16 19:09:56 frontserver.lan systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/unit/borgmatic_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=23 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a Nov 16 19:09:56 frontserver.lan systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/unit/borgmatic_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=24 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a Nov 16 19:09:56 frontserver.lan systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/borgmatic_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=231364 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a Nov 16 19:09:56 frontserver.lan systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/borgmatic_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=231365 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a Nov 16 19:09:56 frontserver.lan systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/unit/borgmatic_2etimer interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=25 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a Nov 16 19:09:56 frontserver.lan systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/unit/borgmatic_2etimer interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=26 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a Nov 16 19:09:56 frontserver.lan systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/borgmatic_2etimer interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=231366 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a Nov 16 19:09:56 frontserver.lan systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/borgmatic_2etimer interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=231367 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a Nov 16 19:09:56 frontserver.lan systemd[1]: Got message type=method_call sender=n/a destination=org.freedesktop.systemd1 path=/org/freedesktop/systemd1/unit/borgmatic_2eservice interface=org.freedesktop.DBus.Properties member=Get cookie=4 reply_cookie=0 signature=ss error-name=n/a error-message=n/a Nov 16 19:09:56 frontserver.lan systemd[1]: SELinux access check scon=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcon=system_u:object_r:systemd_unit_file_t:s0 tclass=service perm=status path=/usr/lib/systemd/system/borgmatic.service cmdline=: 0 Nov 16 19:09:56 frontserver.lan systemd[1]: Sent message type=method_return sender=org.freedesktop.systemd1 destination=n/a path=n/a interface=n/a member=n/a cookie=27 reply_cookie=4 signature=v error-name=n/a error-message=n/a Nov 16 19:09:56 frontserver.lan systemd[1]: Bus private-bus-connection: changing state RUNNING → CLOSING Nov 16 19:09:56 frontserver.lan systemd[1]: Bus private-bus-connection: changing state CLOSING → CLOSED Nov 16 19:09:56 frontserver.lan systemd[1]: Got disconnect on private connection. Nov 16 19:09:56 frontserver.lan systemd[1]: Got cgroup empty notification for: /system.slice/borgmatic.service ... [admin@frontserver ~]$