borgmatic-collective/borgmatic
borgmatic-collective
/
borgmatic
15
57
Fork 41
Code Issues 54 Pull Requests 1 Actions Packages Releases 121 Activity

Permission denied does not result in failure (Continuation of #486) #709

New Issue
Closed
opened 2023-06-05 11:25:17 +00:00 by julian · 8 comments
julian commented 2023-06-05 11:25:17 +00:00
Copy Link

What I'm trying to do and why

Trying to setup a gitlab backup process (though this issue is not gitlab-specific)

Steps to reproduce (if a bug)

gitlab.yml

location:
    source_directories:
        - /home/julian/gitlab/config
        - /home/julian/gitlab/data
        - /home/julian/gitlab/logs

    repositories:
        - path: /home/julian/dev/backup/gitlab
          label: local

    exclude_patterns:
        - '~/.borgmatic/checks/*'

    source_directories_must_exist: true

storage:
    encryption_passphrase: ${BORG_PASSWD}

retention:
    keep_daily: 3
    keep_weekly: 6
    keep_monthly: 12


consistency:
    checks:
        - name: "repository"
        - name: "archives"
        - name: "data"
        - name: "extract"

    check_last: 7

output:
    color: true
    
hooks:
     before_backup:
         - echo "Starting a backup." && (docker stop gitlab || exit 1)

     before_prune:
         - echo "Starting pruning."

     after_backup:
         - echo "Finished a backup." && (docker start gitlab || exit 1)

     on_error:
         - echo "Error during create/prune/compact/check."

Dockerfile:

version: '3'
services:
  gitlab:
    image: gitlab/gitlab-ce:latest
    container_name: gitlab
    ports:
      - "127.0.0.1:8081:80"
      - "127.0.0.1:44301:443"
    volumes:
      - /some/path/config:/etc/gitlab
      - /some/path/logs:/var/log/gitlab
      - /some/path/data:/var/opt/gitlab
    restart: always

Actual behavior (if a bug)

  • borg setup: borgmatic init --encryption repokey --verbosity 2 --config gitlab.yml

  • borgmatic: borgmatic create --verbosity 1 --list --stats --config gitlab.yml results in

gitlab.yml: Running command for pre-backup hook
Starting a backup.
gitlab
local: Creating archive
Creating archive at "/home/julian/dev/backup/gitlab::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}"
/home/julian/gitlab/config/gitlab.rb: open: [Errno 13] Permission denied: '/home/julian/gitlab/config/gitlab.rb'
E /home/julian/gitlab/config/gitlab.rb
....
....
E /home/julian/gitlab/logs/prometheus
/home/julian/gitlab/logs/alertmanager: scandir: [Errno 13] Permission denied: '/home/julian/gitlab/logs/alertmanager'
E /home/julian/gitlab/logs/alertmanager
/home/julian/gitlab/logs/postgres-exporter: scandir: [Errno 13] Permission denied: '/home/julian/gitlab/logs/postgres-exporter'
E /home/julian/gitlab/logs/postgres-exporter
------------------------------------------------------------------------------
Archive name: julian-Precision-3541-2023-06-05T13:09:08.026900
Archive fingerprint: b4dcac05e4b0f903a6242ebb541869aa975d609f0ad860b9761798360e40ee76
Time (start): Mon, 2023-06-05 13:09:08
Time (end):   Mon, 2023-06-05 13:09:08
Duration: 0.03 seconds
Number of files: 16
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:              295.05 kB             76.33 kB             76.22 kB
All archives:              295.05 kB             76.33 kB             76.22 kB
                       Unique chunks         Total chunks
Chunk index:                      17                   18
------------------------------------------------------------------------------
gitlab.yml: Running command for post-backup hook
Finished a backup.
gitlab

summary:
gitlab.yml: Successfully ran configuration file

Expected behavior (if a bug)

Permission denied error should result in hard exit, imo.

I realize that I could run borgmatic as sudo. Yet, I'd rather fix my permissions between docker and gitlab. I understand the tutorial always suggests using sudo, however this issue may arise in other instances as well.

Other notes / implementation ideas

Seems like the patch that fixed #486 should be applicable here.

Environment

borgmatic version: 1.7.14

borgmatic installation method: pip

Borg version: 1.15

Python version: 3.8.10

operating system and version: Ubuntu 20.04

#### What I'm trying to do and why Trying to setup a gitlab backup process (though this issue is not gitlab-specific) #### Steps to reproduce (if a bug) `gitlab.yml` ``` location: source_directories: - /home/julian/gitlab/config - /home/julian/gitlab/data - /home/julian/gitlab/logs repositories: - path: /home/julian/dev/backup/gitlab label: local exclude_patterns: - '~/.borgmatic/checks/*' source_directories_must_exist: true storage: encryption_passphrase: ${BORG_PASSWD} retention: keep_daily: 3 keep_weekly: 6 keep_monthly: 12 consistency: checks: - name: "repository" - name: "archives" - name: "data" - name: "extract" check_last: 7 output: color: true hooks: before_backup: - echo "Starting a backup." && (docker stop gitlab || exit 1) before_prune: - echo "Starting pruning." after_backup: - echo "Finished a backup." && (docker start gitlab || exit 1) on_error: - echo "Error during create/prune/compact/check." ``` Dockerfile: ``` version: '3' services: gitlab: image: gitlab/gitlab-ce:latest container_name: gitlab ports: - "127.0.0.1:8081:80" - "127.0.0.1:44301:443" volumes: - /some/path/config:/etc/gitlab - /some/path/logs:/var/log/gitlab - /some/path/data:/var/opt/gitlab restart: always ``` #### Actual behavior (if a bug) * borg setup: `borgmatic init --encryption repokey --verbosity 2 --config gitlab.yml` * borgmatic: `borgmatic create --verbosity 1 --list --stats --config gitlab.yml` results in ``` gitlab.yml: Running command for pre-backup hook Starting a backup. gitlab local: Creating archive Creating archive at "/home/julian/dev/backup/gitlab::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}" /home/julian/gitlab/config/gitlab.rb: open: [Errno 13] Permission denied: '/home/julian/gitlab/config/gitlab.rb' E /home/julian/gitlab/config/gitlab.rb .... .... E /home/julian/gitlab/logs/prometheus /home/julian/gitlab/logs/alertmanager: scandir: [Errno 13] Permission denied: '/home/julian/gitlab/logs/alertmanager' E /home/julian/gitlab/logs/alertmanager /home/julian/gitlab/logs/postgres-exporter: scandir: [Errno 13] Permission denied: '/home/julian/gitlab/logs/postgres-exporter' E /home/julian/gitlab/logs/postgres-exporter ------------------------------------------------------------------------------ Archive name: julian-Precision-3541-2023-06-05T13:09:08.026900 Archive fingerprint: b4dcac05e4b0f903a6242ebb541869aa975d609f0ad860b9761798360e40ee76 Time (start): Mon, 2023-06-05 13:09:08 Time (end): Mon, 2023-06-05 13:09:08 Duration: 0.03 seconds Number of files: 16 Utilization of max. archive size: 0% ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size This archive: 295.05 kB 76.33 kB 76.22 kB All archives: 295.05 kB 76.33 kB 76.22 kB Unique chunks Total chunks Chunk index: 17 18 ------------------------------------------------------------------------------ gitlab.yml: Running command for post-backup hook Finished a backup. gitlab summary: gitlab.yml: Successfully ran configuration file ``` #### Expected behavior (if a bug) Permission denied error should result in hard exit, imo. I realize that I could run borgmatic as sudo. Yet, I'd rather fix my permissions between docker and gitlab. I understand the tutorial always suggests using sudo, however this issue may arise in other instances as well. #### Other notes / implementation ideas Seems like the patch that fixed #486 should be applicable here. #### Environment **borgmatic version:** 1.7.14 **borgmatic installation method:** pip **Borg version:** 1.15 **Python version:** 3.8.10 **operating system and version:** Ubuntu 20.04
witten commented 2023-06-05 23:30:57 +00:00
Owner
Copy Link

Thanks for taking the time to file this and provide a detailed description. I think what's going on here, and the reason that #486 doesn't solve this, is because the changes in #486 only look at the top-level source directory paths; they don't recurse into subdirectories. And in your particular example, you've got /home/julian/gitlab/logs in your source directories, but Borg is erroring on /home/julian/gitlab/logs/alertmanager and adjacent files.

So I'll have to look into how feasible recursing would be to solve this. I have a vague memory that we might already do it for some unrelated use cases.

Thanks for taking the time to file this and provide a detailed description. I think what's going on here, and the reason that #486 doesn't solve this, is because the changes in #486 only look at the top-level source directory paths; they don't recurse into subdirectories. And in your particular example, you've got `/home/julian/gitlab/logs` in your source directories, but Borg is erroring on `/home/julian/gitlab/logs/alertmanager` and adjacent files. So I'll have to look into how feasible recursing would be to solve this. I have a vague memory that we might already do it for some unrelated use cases.
julian commented 2023-06-06 08:11:52 +00:00
Author
Copy Link

I tested the same with a barebones borg command: borg create --verbose --stats test::archive-{hostname}-{now} /home/julian/gitlab/, and, as expected, I didn't really get a clear (written) indication from borg that the backup failed. However, the exit code was 1.

Thinking about it further...Why not check for the exit code instead? Sure, borgmatic could check permissions on every single file, but that just adds complexity in the end.

I tested the same with a barebones borg command: `borg create --verbose --stats test::archive-{hostname}-{now} /home/julian/gitlab/`, and, as expected, I didn't really get a clear (written) indication from borg that the backup failed. However, the exit code was 1. Thinking about it further...Why not check for the exit code instead? Sure, borgmatic could check permissions on *every single file*, but that just adds complexity in the end.
witten commented 2023-06-06 17:20:18 +00:00
Owner
Copy Link

Why not check for the exit code instead? Sure, borgmatic could check permissions on every single file, but that just adds complexity in the end.

From Borg's perspective, exit code 1 is a warning and 2+ is an error. If borgmatic decides to error whenever Borg exits with a code of 1, that will cause many warnings to get elevated to errors. So unfortunately the complexity in borgmatic may be the best option here if we really need some functionality Borg doesn't provide.

> Why not check for the exit code instead? Sure, borgmatic could check permissions on every single file, but that just adds complexity in the end. From Borg's perspective, exit code 1 is a warning and 2+ is an error. If borgmatic decides to error whenever Borg exits with a code of 1, that will cause many warnings to get elevated to errors. So unfortunately the complexity in borgmatic may be the best option here if we really need some functionality Borg doesn't provide.
julian commented 2023-06-07 12:31:00 +00:00
Author
Copy Link

Understood.

I may open an issue at the borg repo as well. I could suggest elevating permission errors to 2+.

This should be the relevant section, if I'm not mistaken. Do you think opening each file in huge backups could create some issues down the line?

Understood. I may open an issue at the borg repo as well. I could suggest elevating permission errors to 2+. [This](https://projects.torsion.org/borgmatic-collective/borgmatic/src/commit/28fef3264bff9a15e07d4a39ce11ca44ddf148e8/borgmatic/borg/create.py#L116) should be the relevant section, if I'm not mistaken. Do you think opening each file in huge backups could create some issues down the line?
witten commented 2023-06-07 16:14:46 +00:00
Owner
Copy Link

I may open an issue at the borg repo as well. I could suggest elevating permission errors to 2+.

That would be great. If you do so, feel free to link the issue from here. Borg taking care of this would be a cleaner approach IMO.

This should be the relevant section, if I'm not mistaken. Do you think opening each file in huge backups could create some issues down the line?

Yes, that's the relevant code I was thinking of. I do think there is some performance risk of opening each file recursively, although I could see potentially hiding that behind an option instead of making the recursive opens the default behavior.

> I may open an issue at the borg repo as well. I could suggest elevating permission errors to 2+. That would be great. If you do so, feel free to link the issue from here. Borg taking care of this would be a cleaner approach IMO. > This should be the relevant section, if I'm not mistaken. Do you think opening each file in huge backups could create some issues down the line? Yes, that's the relevant code I was thinking of. I do think there is some performance risk of opening each file recursively, although I could see potentially hiding that behind an option instead of making the recursive opens the default behavior.
shukon commented 2023-07-29 11:12:51 +00:00
Copy Link

Hi. A relevant borg issue on this regarding potential breaking changes to return codes.

I think it would be really helpful to add an option to borgmatic regarding this. Something along the lines of

# Borg exits with return code 1 for warnings. Borgmatic only treats return codes >= 2 as errors.
# Warnings include files that are not found and permission errors. This can result in files not being backed up.
# Set this value to true to treat warnings as errors. Defaults to false.
# treat_warnings_as_errors: true

This could translate directly to this variable, which would probably make implementation easier while not changing default behaviour.
There has to be a way to notify me when a backup did not actually successfully include all files, right?

Hi. A relevant borg issue on this regarding potential [breaking changes to return codes](https://github.com/borgbackup/borg/issues/6756). I think it would be really helpful to add an option to borgmatic regarding this. Something along the lines of ``` # Borg exits with return code 1 for warnings. Borgmatic only treats return codes >= 2 as errors. # Warnings include files that are not found and permission errors. This can result in files not being backed up. # Set this value to true to treat warnings as errors. Defaults to false. # treat_warnings_as_errors: true ``` This could translate directly to [this variable](https://github.com/borgmatic-collective/borgmatic/blob/c11dcdef0addd852c030bc713689802d88dd6cae/borgmatic/execute.py#L11), which would probably make implementation easier while not changing default behaviour. There has to be a way to notify me when a backup did not actually successfully include all files, right?
witten commented 2023-08-03 03:38:12 +00:00
Owner
Copy Link

I'm not sure that would be sufficient for many users given that Borg issues warnings (exit code 1) for many other situations than permission denied. For instance, on borg create, Borg returns exit code 1 whenever a file changes while it's being read (assuming that retries that don't take care of that). This can happen on log files or other frequently changing files. And that's just one particular example of Borg's warnings.

So the two remaining classes of solutions I can see are:

  • Optionally (or by default) do a dry run with Borg prior to the actual borg create to enumerate all source files. Then make sure each one is actually readable. If any are not, error. This has the downside of time-of-check / time-of-use discrepancies. And performance, I guess, but we already pay that cost with a similar approach when looking for special files to auto-exclude.
  • Or ... just run borg create optimistically, but then optionally (or by default) parse Borg's output to look for permissions denied warnings. If any are found, error. This has the downside of having to parse Borg output, which borgmatic for the most part isn't in the business of doing.
I'm not sure that would be sufficient for many users given that Borg issues warnings (exit code 1) for many other situations than permission denied. For instance, on `borg create`, Borg returns exit code 1 whenever [a file changes while it's being read](https://github.com/borgbackup/borg/issues/6379) (assuming that retries that don't take care of that). This can happen on log files or other frequently changing files. And that's just one particular example of Borg's warnings. So the two remaining classes of solutions I can see are: * Optionally (or by default) do a dry run with Borg prior to the actual `borg create` to enumerate all source files. Then make sure each one is actually readable. If any are not, error. This has the downside of time-of-check / time-of-use discrepancies. And performance, I guess, but we already pay that cost with a similar approach when looking for special files to auto-exclude. * Or ... just run `borg create` optimistically, but then optionally (or by default) parse Borg's output to look for permissions denied warnings. If any are found, error. This has the downside of having to parse Borg output, which borgmatic for the most part isn't in the business of doing.
witten added the
waiting for response
 label 2023-08-15 03:38:02 +00:00
witten commented 2024-01-12 00:44:38 +00:00
Owner
Copy Link

I think that #798 is probably the right solution to this issue, so I'll close this ticket (#709) accordingly. The downside is it'll require Borg 1.4.0+ (not yet a stable release at the time of this comment).

I think that #798 is probably the right solution to this issue, so I'll close this ticket (#709) accordingly. The downside is it'll require Borg 1.4.0+ (not yet a stable release at the time of this comment).
👍 1
witten removed the
waiting for response
 label 2024-01-12 00:44:48 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
main
spot-check
1.8.10
1.8.9
1.8.8
1.8.7
1.8.6
1.8.5
1.8.4
1.8.3
1.8.2
1.8.1
1.8.0
1.7.15
1.7.14
1.7.13
1.7.12
1.7.11
1.7.10
1.7.9
1.7.8
1.7.7
1.7.6
1.7.5
1.7.4
1.7.3
1.7.2
1.7.1
1.7.0
1.6.6
1.6.5
1.6.4
1.6.3
1.6.2
1.6.1
1.6.0
1.5.24
1.5.23
1.5.22
1.5.21
1.5.20
1.5.19
1.5.18
1.5.17
1.5.16
1.5.15
1.5.14
1.5.13
1.5.12
1.5.11
1.5.10
1.5.9
1.5.8
1.5.7
1.5.7.dev0
1.5.6
1.5.5
1.5.4
1.5.3
1.5.2
1.5.1
1.5.0
1.4.22
1.4.21
1.4.20
1.4.19
1.4.18
1.4.17
1.4.16
1.4.15
1.4.14
1.4.13
1.4.12
1.4.11
1.4.10
1.4.9
1.4.8
1.4.7
1.4.6
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0
1.3.26
1.3.25
1.3.24
1.3.23
1.3.22
1.3.21
1.3.20
1.3.19
1.3.18
1.3.17
1.3.16
1.3.15
1.3.14
1.3.13
1.3.12
1.3.11
1.3.10
1.3.9
1.3.8
1.3.7
1.3.6
1.3.5
1.3.4
1.3.3
1.3.2
1.3.1
1.3.0
1.2.18
1.2.17
1.2.16
1.2.15
1.2.14
1.2.13
1.2.12
1.2.11
1.2.10
1.2.9
1.2.8
1.2.7
1.2.6
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.15
1.1.14
1.1.13
1.1.12
1.1.11
1.1.10
1.1.9
1.1.7
1.1.6
1.1.5
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.3
1.0.2
1.0.1
1.0.0
0.1.8
0.1.7
0.1.6
0.1.5
0.1.4
0.1.3
0.1.2
0.1.1
0.1.0
0.0.7
0.0.6
0.0.5
0.0.4
0.0.3
0.0.2
Labels
Clear labels   
bug

   
data loss

   
design finalized

   
good first issue

   
new feature area

   
question / support

   
security

   
waiting for response
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#709
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?