borgmatic-collective/borgmatic
borgmatic-collective
/
borgmatic
15
56
Fork 39
Code Issues 52 Pull Requests 1 Actions Packages Releases 121 Activity

Behavior change in 1.5.7 with missing locations #387

New Issue
Closed
opened 2021-01-27 16:23:24 +00:00 by sysedit · 4 comments
sysedit commented 2021-01-27 16:23:24 +00:00
Copy Link

Not sure if this is a bug or if the change was intented (I assume it's linked to issue #333 fixed in 1.5.7): if you have optional files to backup, missing files was not an issue until borgmatic 1.5.7.

Now it fails with:

xxx/backups/repository: Error running actions for repository
[Errno 2] No such file or directory: '/xxx/supervisor/extras'
/xxx/borgmatic.conf: Error running configuration file

I can likely workaround by using patterns in the configuration file, but I was curious if this was intented or if this is a bug / side effect of another fix.

Not sure if this is a bug or if the change was intented (I assume it's linked to issue #333 fixed in 1.5.7): if you have optional files to backup, missing files was not an issue until borgmatic 1.5.7. Now it fails with: xxx/backups/repository: Error running actions for repository [Errno 2] No such file or directory: '/xxx/supervisor/extras' /xxx/borgmatic.conf: Error running configuration file I can likely workaround by using patterns in the configuration file, but I was curious if this was intented or if this is a bug / side effect of another fix.
witten commented 2021-01-28 23:51:08 +00:00
Owner
Copy Link

Thanks for filing this!

Could a get a look at your configuration file, specifically your source_directories and one_file_system value? And could you also let me know whether your various source_directories are on different filesystems? The partial output of running mount might help with that.

What I'm trying to figure out with all this: Would your "missing" file have been de-deduplicated prior to #333 and now is no longer deduplicated? That in theory could result in the error you're seeing, since the missing directory would now be passed to Borg... And Borg would of course get angry about it being missing.

Thanks for filing this! Could a get a look at your configuration file, specifically your `source_directories` and `one_file_system` value? And could you also let me know whether your various `source_directories` are on different filesystems? The partial output of running `mount` might help with that. What I'm trying to figure out with all this: Would your "missing" file have been de-deduplicated prior to #333 and now is no longer deduplicated? That *in theory* could result in the error you're seeing, since the missing directory would now be passed to Borg... And Borg would of course get angry about it being missing.
witten added the
question / support
 label 2021-01-28 23:51:18 +00:00
sysedit commented 2021-02-03 13:11:42 +00:00
Author
Copy Link

Here's our configuration file:

location:
    source_directories:
        - /usr/local/filewave/certs
        - /usr/local/filewave/apache/conf/httpd.conf
        - /usr/local/filewave/apache/conf/httpd_custom.conf
        - /usr/local/filewave/apache/conf/httpd_webadmin.conf
        - /usr/local/filewave/apache/conf/mdm_auth.conf
        - /usr/local/filewave/django/filewave/settings_custom.py
        - /usr/local/filewave/apache/passwd
        - /usr/local/filewave/conf/user_postgresql.conf
        - /usr/local/etc/filewave/supervisor/extras
        - /usr/local/etc/filewave/mtail/progs
        - /usr/local/etc/filewave/prometheus/conf.d/jobs/http
        - /usr/local/etc/filewave/prometheus/conf.d/jobs/https
        - /usr/local/filewave/instrumentation_data
    exclude_patterns:
        - /usr/local/filewave/instrumentation_data/grafana/plugins
        - /usr/local/filewave/instrumentation_data/prometheus
    repositories:
        - /usr/local/filewave/fwxserver/Data Folder/backups/repository
    local_path: /usr/local/filewave/python/bin/borg
storage:
    compression: zlib
    archive_name_format: 'pre-upgrade-backup-{now}'
    borg_cache_directory: /usr/local/filewave/fwxserver/Data Folder/backups/cache
    borg_security_directory: /usr/local/filewave/fwxserver/Data Folder/backups/security
retention:
    keep_within: 1y
    prefix: 'pre-upgrade-backup'
consistency:
    checks:
         - repository
         - data
    check_last: 3
    prefix: 'pre-upgrade-backup'
hooks:
    postgresql_databases:
        - name: mdm
          port: 9432
          username: django
          options: --encoding=UTF8 -N committed_* -c

And it should always be the same FS:

/dev/mapper/centos_centos7-root on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)

with:

# /usr/local/filewave/python/bin/borgmatic --version
1.5.5
# /usr/local/filewave/python/bin/borgmatic -c /usr/local/etc/filewave/backups/on-premise-pre-upgrade.conf --log-file /usr/local/filewave/log/pre-upgrade-backup.log --log-file-verbosity=1

this works without any issue.

Now I upgrade to 1.5.7:

 /usr/local/filewave/python/bin/borgmatic --version
1.5.7
 /usr/local/filewave/python/bin/borgmatic -c /usr/local/etc/filewave/backups/on-premise-pre-upgrade.conf --log-file /usr/local/filewave/log/pre-upgrade-backup-1.5.7.log --log-file-verbosity=1
/usr/local/filewave/fwxserver/Data Folder/backups/repository: Error running actions for repository
[Errno 2] No such file or directory: '/usr/local/filewave/apache/conf/httpd_webadmin.conf'
/usr/local/etc/filewave/backups/on-premise-pre-upgrade.conf: Error running configuration file

summary:
/usr/local/etc/filewave/backups/on-premise-pre-upgrade.conf: Error running configuration file
/usr/local/filewave/fwxserver/Data Folder/backups/repository: Error running actions for repository
[Errno 2] No such file or directory: '/usr/local/filewave/apache/conf/httpd_webadmin.conf'

Need some help? https://torsion.org/borgmatic/#issues

Indeed, the file httpd_webadmin.conf is not there ; we provide a .conf.example file which can be renamed .conf if needed, and we want to backup this file if it exists.

ls -al /usr/local/filewave/apache/conf/
total 228
drwxr-xr-x.  4 apache apache  4096 Feb  3 07:36 .
drwxr-xr-x. 16 root   root     194 Feb  3 07:36 ..
-rwxr-xr-x.  1 root   root   21420 Feb  3 05:17 default_httpd.conf
-rwxr-xr-x.  1 root   root   21423 Feb  3 05:17 default_httpd.conf.orig
-rwxr-xr-x.  1 root   root    1159 Feb  3 05:17 default_mdm_auth.conf
drwxr-xr-x.  2 root   root    4096 Feb  3 07:35 extra
-rwxr-xr-x.  1 root   root   21420 Feb  3 07:35 httpd.conf
-rwxr-xr-x.  1 root   root   19118 Jan 29 05:21 httpd.conf.bak
-rwxr-xr-x.  1 root   root   21423 Feb  3 04:46 httpd.conf.orig
-rw-r--r--.  1 root   root       0 Feb  3 07:36 httpd_custom.conf
-rwxr-xr-x.  1 root   root    2058 Feb  3 05:17 httpd_webadmin.conf.example
-rwxr-xr-x.  1 root   root   13064 Jan 29 05:21 magic
-rwxr-xr-x.  1 apache apache  1159 Feb  3 07:35 mdm_auth.conf
-rwxr-xr-x.  1 apache apache  1159 Feb  3 07:35 mdm_auth.conf.bak
-rwxr-xr-x.  1 root   root    1172 Feb  3 04:46 mdm_auth.conf.example_basic_auth
-rwxr-xr-x.  1 root   root    2587 Feb  3 04:46 mdm_auth.conf.example_ldap_auth
-rwxr-xr-x.  1 root   root     282 Feb  3 04:46 mdm_auth.conf.example_no_auth
-rwxr-xr-x.  1 root   root   60847 Jan 29 05:21 mime.types
drwxr-xr-x.  3 root   root      19 Feb  3 07:35 original
[root@centos7 ~]#

I could not find any explicit reason for this change in change log. The new behavior seems legit (even if it would be nice to have optional file list), but I wanted to make sure this has been done on purpose (or to check if we're using borgmatic properly).

Thanks !

Here's our configuration file: ``` location: source_directories: - /usr/local/filewave/certs - /usr/local/filewave/apache/conf/httpd.conf - /usr/local/filewave/apache/conf/httpd_custom.conf - /usr/local/filewave/apache/conf/httpd_webadmin.conf - /usr/local/filewave/apache/conf/mdm_auth.conf - /usr/local/filewave/django/filewave/settings_custom.py - /usr/local/filewave/apache/passwd - /usr/local/filewave/conf/user_postgresql.conf - /usr/local/etc/filewave/supervisor/extras - /usr/local/etc/filewave/mtail/progs - /usr/local/etc/filewave/prometheus/conf.d/jobs/http - /usr/local/etc/filewave/prometheus/conf.d/jobs/https - /usr/local/filewave/instrumentation_data exclude_patterns: - /usr/local/filewave/instrumentation_data/grafana/plugins - /usr/local/filewave/instrumentation_data/prometheus repositories: - /usr/local/filewave/fwxserver/Data Folder/backups/repository local_path: /usr/local/filewave/python/bin/borg storage: compression: zlib archive_name_format: 'pre-upgrade-backup-{now}' borg_cache_directory: /usr/local/filewave/fwxserver/Data Folder/backups/cache borg_security_directory: /usr/local/filewave/fwxserver/Data Folder/backups/security retention: keep_within: 1y prefix: 'pre-upgrade-backup' consistency: checks: - repository - data check_last: 3 prefix: 'pre-upgrade-backup' hooks: postgresql_databases: - name: mdm port: 9432 username: django options: --encoding=UTF8 -N committed_* -c ``` And it should always be the same FS: ``` /dev/mapper/centos_centos7-root on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota) ``` with: ``` # /usr/local/filewave/python/bin/borgmatic --version 1.5.5 # /usr/local/filewave/python/bin/borgmatic -c /usr/local/etc/filewave/backups/on-premise-pre-upgrade.conf --log-file /usr/local/filewave/log/pre-upgrade-backup.log --log-file-verbosity=1 ``` this works without any issue. Now I upgrade to 1.5.7: ``` /usr/local/filewave/python/bin/borgmatic --version 1.5.7 /usr/local/filewave/python/bin/borgmatic -c /usr/local/etc/filewave/backups/on-premise-pre-upgrade.conf --log-file /usr/local/filewave/log/pre-upgrade-backup-1.5.7.log --log-file-verbosity=1 /usr/local/filewave/fwxserver/Data Folder/backups/repository: Error running actions for repository [Errno 2] No such file or directory: '/usr/local/filewave/apache/conf/httpd_webadmin.conf' /usr/local/etc/filewave/backups/on-premise-pre-upgrade.conf: Error running configuration file summary: /usr/local/etc/filewave/backups/on-premise-pre-upgrade.conf: Error running configuration file /usr/local/filewave/fwxserver/Data Folder/backups/repository: Error running actions for repository [Errno 2] No such file or directory: '/usr/local/filewave/apache/conf/httpd_webadmin.conf' Need some help? https://torsion.org/borgmatic/#issues ``` Indeed, the file httpd_webadmin.conf is not there ; we provide a .conf.example file which can be renamed .conf if needed, and we want to backup this file if it exists. ``` ls -al /usr/local/filewave/apache/conf/ total 228 drwxr-xr-x. 4 apache apache 4096 Feb 3 07:36 . drwxr-xr-x. 16 root root 194 Feb 3 07:36 .. -rwxr-xr-x. 1 root root 21420 Feb 3 05:17 default_httpd.conf -rwxr-xr-x. 1 root root 21423 Feb 3 05:17 default_httpd.conf.orig -rwxr-xr-x. 1 root root 1159 Feb 3 05:17 default_mdm_auth.conf drwxr-xr-x. 2 root root 4096 Feb 3 07:35 extra -rwxr-xr-x. 1 root root 21420 Feb 3 07:35 httpd.conf -rwxr-xr-x. 1 root root 19118 Jan 29 05:21 httpd.conf.bak -rwxr-xr-x. 1 root root 21423 Feb 3 04:46 httpd.conf.orig -rw-r--r--. 1 root root 0 Feb 3 07:36 httpd_custom.conf -rwxr-xr-x. 1 root root 2058 Feb 3 05:17 httpd_webadmin.conf.example -rwxr-xr-x. 1 root root 13064 Jan 29 05:21 magic -rwxr-xr-x. 1 apache apache 1159 Feb 3 07:35 mdm_auth.conf -rwxr-xr-x. 1 apache apache 1159 Feb 3 07:35 mdm_auth.conf.bak -rwxr-xr-x. 1 root root 1172 Feb 3 04:46 mdm_auth.conf.example_basic_auth -rwxr-xr-x. 1 root root 2587 Feb 3 04:46 mdm_auth.conf.example_ldap_auth -rwxr-xr-x. 1 root root 282 Feb 3 04:46 mdm_auth.conf.example_no_auth -rwxr-xr-x. 1 root root 60847 Jan 29 05:21 mime.types drwxr-xr-x. 3 root root 19 Feb 3 07:35 original [root@centos7 ~]# ``` I could not find any explicit reason for this change in change log. The new behavior seems legit (even if it would be nice to have optional file list), but I wanted to make sure this has been done on purpose (or to check if we're using borgmatic properly). Thanks !
witten commented 2021-10-11 17:39:35 +00:00
Owner
Copy Link

First of all, I sincerely apologize for letting this ticket languish for so long. I've dug into the issue, and I believe you are right—this is a regression! What happened is that in #333, introduced in borgmatic 1.5.7, the code to deduplicate source directories started calling Python's os.stat() on each path. Which meant that if a source directory wasn't present on the filesystem, borgmatic errored.

So the fix I made is to only call os.stat() if the source directory is actually present. Now, all source directories get passed to Borg. And like with the pre-1.5.7 borgmatic behavior, Borg complains about missing directories but continues making the backup anyway without erroring.

Hope this ends up working for you! Should be part of the next release.

First of all, I sincerely apologize for letting this ticket languish for so long. I've dug into the issue, and I believe you are right—this is a regression! What happened is that in #333, introduced in borgmatic 1.5.7, the code to deduplicate source directories started calling Python's `os.stat()` on each path. Which meant that if a source directory wasn't present on the filesystem, borgmatic errored. So the fix I made is to only call `os.stat()` if the source directory is actually present. Now, all source directories get passed to Borg. And like with the pre-1.5.7 borgmatic behavior, Borg complains about missing directories but continues making the backup anyway without erroring. Hope this ends up working for you! Should be part of the next release.
witten added
bug
 and removed
question / support
 labels 2021-10-11 17:39:45 +00:00
witten commented 2021-10-11 18:14:15 +00:00
Owner
Copy Link

The fix has been released with borgmatic 1.5.19! Thanks for your patience here.

The fix has been released with borgmatic 1.5.19! Thanks for your patience here.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
main
spot-check
1.8.10
1.8.9
1.8.8
1.8.7
1.8.6
1.8.5
1.8.4
1.8.3
1.8.2
1.8.1
1.8.0
1.7.15
1.7.14
1.7.13
1.7.12
1.7.11
1.7.10
1.7.9
1.7.8
1.7.7
1.7.6
1.7.5
1.7.4
1.7.3
1.7.2
1.7.1
1.7.0
1.6.6
1.6.5
1.6.4
1.6.3
1.6.2
1.6.1
1.6.0
1.5.24
1.5.23
1.5.22
1.5.21
1.5.20
1.5.19
1.5.18
1.5.17
1.5.16
1.5.15
1.5.14
1.5.13
1.5.12
1.5.11
1.5.10
1.5.9
1.5.8
1.5.7
1.5.7.dev0
1.5.6
1.5.5
1.5.4
1.5.3
1.5.2
1.5.1
1.5.0
1.4.22
1.4.21
1.4.20
1.4.19
1.4.18
1.4.17
1.4.16
1.4.15
1.4.14
1.4.13
1.4.12
1.4.11
1.4.10
1.4.9
1.4.8
1.4.7
1.4.6
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0
1.3.26
1.3.25
1.3.24
1.3.23
1.3.22
1.3.21
1.3.20
1.3.19
1.3.18
1.3.17
1.3.16
1.3.15
1.3.14
1.3.13
1.3.12
1.3.11
1.3.10
1.3.9
1.3.8
1.3.7
1.3.6
1.3.5
1.3.4
1.3.3
1.3.2
1.3.1
1.3.0
1.2.18
1.2.17
1.2.16
1.2.15
1.2.14
1.2.13
1.2.12
1.2.11
1.2.10
1.2.9
1.2.8
1.2.7
1.2.6
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.15
1.1.14
1.1.13
1.1.12
1.1.11
1.1.10
1.1.9
1.1.7
1.1.6
1.1.5
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.3
1.0.2
1.0.1
1.0.0
0.1.8
0.1.7
0.1.6
0.1.5
0.1.4
0.1.3
0.1.2
0.1.1
0.1.0
0.0.7
0.0.6
0.0.5
0.0.4
0.0.3
0.0.2
Labels
Clear labels   
bug

   
data loss

   
design finalized

   
good first issue

   
new feature area

   
question / support

   
security

   
waiting for response
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#387
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?