Question: Why is /root/.borgmatic being backed up? #638

New Issue

Closed

opened 2023-02-12 04:11:14 +00:00 by baj0k · 4 comments

baj0k commented 2023-02-12 04:11:14 +00:00

Copy Link

What I'm trying to do and why

I have a working borg backup setup but I noticed recently during manual extraction of archive that files in the

• /root/.borgmatic
  are included in it.

I was not aware of this and could not find information about it on borg / borgmatic wikis so I end up posting an issue about it. I feel like the reason behind it should be obvious since I think this is related to checks being enabled on the repository. The files that are being backed up are in /root/.borgmatic/checks directory, but it doesn't match my inclusion/exclusion patterns and I wanted to be sure that this behavior is intended.

Steps to reproduce (if a bug)

config.yaml:

location:
    repositories:
        - ${REPO_EU}
        - ${REPO_US}

    patterns:
        - R /backup
        - P fm
        - '+ */docker-compose.yml'
        - '+ */.env'
        # borgmatic
        - '+ */borgmatic/*'
        # postgres
        - '+ */postgres/entrypoint/*'
        # vaultwarden
        - '+ */vaultwarden/attachments/*'
        - '+ */vaultwarden/sends/*'
        - '+ */vaultwarden/config.json'
        # nginx
        - '+ */nginx/*'
        # exclude catch-all
        - '- /backup*'


storage:
    compression: lz4
    archive_name_format: 'selfhost-{now:%Y-%m-%dT%H}'

retention:
    keep_daily: 3
    keep_weekly: 2
    keep_monthly: 1
    prefix: selfhost

consistency:
    checks:
        - name: repository
          frequency: 1 day
        - name: archives
          frequency: 3 days

    check_last: 3
    prefix: selfhost

output:
    color: true

hooks:
    before_everything:
        - echo "Stopping containers..."
        - 'name=wiki; curl -s --unix-socket /var/run/docker.sock -X POST http://localhost/v1.41/containers/$name/stop  || echo "Failed to stop container $name"'
        - 'name=vault; curl -s --unix-socket /var/run/docker.sock -X POST http://localhost/v1.41/containers/$name/stop || echo "Failed to stop container $name"'
        - sleep 10
        - echo "Finished stopping containers"

    after_everything:
        - echo "Starting containers..."
        - 'name=wiki; curl -s --unix-socket /var/run/docker.sock -X POST http://localhost/v1.41/containers/$name/start || echo "Failed to start container $name"'
        - 'name=vault; curl -s --unix-socket /var/run/docker.sock -X POST http://localhost/v1.41/containers/$name/start || echo "Failed to start container $name"'
        - echo "Finished starting containers"

    before_prune:
        - echo "Starting pruning..."
    after_prune:
        - echo "Finished pruning"

    before_backup:
        - echo "Starting backup..."
    after_backup:
        - echo "Finished backup"

    before_check:
        - echo "Starting checks..."
    after_check:
        - echo "Finished checks"

    before_extract:
        - echo "Starting extracting..."
    after_extract:
        - echo "Finished extracting"

    on_error:
        - echo "Error durring borgmatic operation or while running one of the hooks!"

    postgresql_databases:
      - name: all
        hostname: db
        username: ${POSTGRES_USERNAME}
        password: ${POSTGRES_PASSWORD}

Actual behavior (if a bug)

When running borgmatic --dry-run --list with the above config I see following lines in the output

...
x /backup/letsencrypt/log/letsencrypt.log.3
x /backup/letsencrypt/log/letsencrypt.log.2
x /backup/letsencrypt/log/letsencrypt.log.1
x /backup/letsencrypt/log/letsencrypt.log
- /root/.borgmatic/checks/61baece2a5326eccdea987b1b15430d748f1b0ff3c51453c22f8cc8df87cf5c2/repository
- /root/.borgmatic/checks/61baece2a5326eccdea987b1b15430d748f1b0ff3c51453c22f8cc8df87cf5c2/archives
- /root/.borgmatic/checks/61baece2a5326eccdea987b1b15430d748f1b0ff3c51453c22f8cc8df87cf5c2
- /root/.borgmatic/checks
- /root/.borgmatic
x /backup

Environment

borgmatic version: 1.7.6

Borg version: 1.2.3

Python version: 3.11.1

#### What I'm trying to do and why I have a working borg backup setup but I noticed recently during manual extraction of archive that files in the - /root/.borgmatic are included in it. I was not aware of this and could not find information about it on borg / borgmatic wikis so I end up posting an issue about it. I feel like the reason behind it should be obvious since I think this is related to checks being enabled on the repository. The files that are being backed up are in /root/.borgmatic/checks directory, but it doesn't match my inclusion/exclusion patterns and I wanted to be sure that this behavior is intended. #### Steps to reproduce (if a bug) config.yaml: ``` location: repositories: - ${REPO_EU} - ${REPO_US} patterns: - R /backup - P fm - '+ */docker-compose.yml' - '+ */.env' # borgmatic - '+ */borgmatic/*' # postgres - '+ */postgres/entrypoint/*' # vaultwarden - '+ */vaultwarden/attachments/*' - '+ */vaultwarden/sends/*' - '+ */vaultwarden/config.json' # nginx - '+ */nginx/*' # exclude catch-all - '- /backup*' storage: compression: lz4 archive_name_format: 'selfhost-{now:%Y-%m-%dT%H}' retention: keep_daily: 3 keep_weekly: 2 keep_monthly: 1 prefix: selfhost consistency: checks: - name: repository frequency: 1 day - name: archives frequency: 3 days check_last: 3 prefix: selfhost output: color: true hooks: before_everything: - echo "Stopping containers..." - 'name=wiki; curl -s --unix-socket /var/run/docker.sock -X POST http://localhost/v1.41/containers/$name/stop || echo "Failed to stop container $name"' - 'name=vault; curl -s --unix-socket /var/run/docker.sock -X POST http://localhost/v1.41/containers/$name/stop || echo "Failed to stop container $name"' - sleep 10 - echo "Finished stopping containers" after_everything: - echo "Starting containers..." - 'name=wiki; curl -s --unix-socket /var/run/docker.sock -X POST http://localhost/v1.41/containers/$name/start || echo "Failed to start container $name"' - 'name=vault; curl -s --unix-socket /var/run/docker.sock -X POST http://localhost/v1.41/containers/$name/start || echo "Failed to start container $name"' - echo "Finished starting containers" before_prune: - echo "Starting pruning..." after_prune: - echo "Finished pruning" before_backup: - echo "Starting backup..." after_backup: - echo "Finished backup" before_check: - echo "Starting checks..." after_check: - echo "Finished checks" before_extract: - echo "Starting extracting..." after_extract: - echo "Finished extracting" on_error: - echo "Error durring borgmatic operation or while running one of the hooks!" postgresql_databases: - name: all hostname: db username: ${POSTGRES_USERNAME} password: ${POSTGRES_PASSWORD} ``` #### Actual behavior (if a bug) When running borgmatic --dry-run --list with the above config I see following lines in the output ``` ... x /backup/letsencrypt/log/letsencrypt.log.3 x /backup/letsencrypt/log/letsencrypt.log.2 x /backup/letsencrypt/log/letsencrypt.log.1 x /backup/letsencrypt/log/letsencrypt.log - /root/.borgmatic/checks/61baece2a5326eccdea987b1b15430d748f1b0ff3c51453c22f8cc8df87cf5c2/repository - /root/.borgmatic/checks/61baece2a5326eccdea987b1b15430d748f1b0ff3c51453c22f8cc8df87cf5c2/archives - /root/.borgmatic/checks/61baece2a5326eccdea987b1b15430d748f1b0ff3c51453c22f8cc8df87cf5c2 - /root/.borgmatic/checks - /root/.borgmatic x /backup ``` #### Environment **borgmatic version:** 1.7.6 **Borg version:** 1.2.3 **Python version:** 3.11.1

👍 1

witten commented 2023-02-12 05:51:34 +00:00

Owner

Copy Link

Thanks for taking the time to file this including all the details about your setup! The short answer as to why ~/.borgmatic is being backed up is that doing so directly supports database dump backups. Each database dump is streamed from your database to a special named pipe within ~/.borgmatic and from there it goes directly to Borg without taking up any temporary space on disk.

This "feature" is documented on the database backup page, but maybe it needs to be made more prominent. And as a side effect, it backs up ~/.borgmatic/checks as well. Which isn't the worst thing, because it means your check state gets backed up as well. (But it is an implicit side effect, which maybe isn't so great.)

So, to answer your question: It is intended behavior. But I'm still open to making improvements to it!

Thanks for taking the time to file this including all the details about your setup! The short answer as to why `~/.borgmatic` is being backed up is that doing so directly supports database dump backups. Each database dump is streamed from your database to a special named pipe within `~/.borgmatic` and from there it goes directly to Borg without taking up any temporary space on disk. This "feature" *is* documented on the [database backup page](https://torsion.org/borgmatic/docs/how-to/backup-your-databases/), but maybe it needs to be made more prominent. And as a side effect, it backs up `~/.borgmatic/checks` as well. Which isn't the worst thing, because it means your check state gets backed up as well. (But it is an implicit side effect, which maybe isn't so great.) So, to answer your question: It is intended behavior. But I'm still open to making improvements to it!

👍 1

witten added the

question / support

label 2023-02-12 05:51:41 +00:00

baj0k commented 2023-02-12 11:55:50 +00:00

Author

Copy Link

So I think now the question is - is it a good thing to include ~/.borgmatic/checks just because they happen to share a common directory with the named pipe for the databases.

I can think of scenarios where backing up the check states is desired but I think it shouldn't be done automatically without the user knowledge. At least to avoid confusion like in my case.

If you agree then I think the solution would be to simply include ~/.borgmatic/*/db/* instead of ~/.borgmatic/*. What do you think?

Edit: There is a really simple workaround for this if you think the behavior is not worth changing. If you don't want to have check states backed up you can exclude ~/.borgmatic/checks/* in the config but I still think that the "check states being backed up when database hooks are enabled" behavior should be documented somewhere.

So I think now the question is - is it a good thing to include `~/.borgmatic/checks` just because they happen to share a common directory with the named pipe for the databases. I can think of scenarios where backing up the check states is desired but I think it shouldn't be done automatically without the user knowledge. At least to avoid confusion like in my case. If you agree then I think the solution would be to simply include `~/.borgmatic/*/db/*` instead of `~/.borgmatic/*`. What do you think? Edit: There is a really simple workaround for this if you think the behavior is not worth changing. If you don't want to have check states backed up you can exclude `~/.borgmatic/checks/*` in the config but I still think that the "check states being backed up when database hooks are enabled" behavior should be documented somewhere.

👍 1

witten commented 2023-02-14 17:35:09 +00:00

Owner

Copy Link

Changing this to avoid implicitly backing up the check state makes sense to me.

Changing this to avoid implicitly backing up the check state makes sense to me.

👍 1

witten removed the

question / support

label 2023-04-07 04:31:02 +00:00

witten referenced this issue 2024-03-12 04:06:31 +00:00

XDG directories #562

witten referenced this issue from a commit 2024-10-31 05:38:23 +00:00

Move the default check state directory (#562, #638). Deprecate the "borgmatic_source_directory" option in favor of "borgmatic_runtime_directory" and "borgmatic_state_directory" (#562).

witten commented 2024-10-31 05:42:48 +00:00

Owner

Copy Link

This is implemented in main and will be part of the next release! ~/.borgmatic/checks is moving to ~/.local/state/borgmatic/checks by default, but you can override it by setting a new borgmatic_state_directory option or even via the $XDG_STATE_HOME environment variable. See #562 for some of the rationale for this particular location.

This is implemented in main and will be part of the next release! `~/.borgmatic/checks` is moving to `~/.local/state/borgmatic/checks` by default, but you can override it by setting a new `borgmatic_state_directory` option or even via the [$XDG_STATE_HOME environment variable](https://wiki.archlinux.org/title/XDG_Base_Directory). See #562 for some of the rationale for this particular location.

witten closed this issue 2024-10-31 05:42:48 +00:00

witten referenced this issue 2024-11-03 21:31:56 +00:00

XDG directories #562

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

No results found.

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.14

1.8.13

1.8.12

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels

blocked

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

No Label

Milestone

No items

No Milestone

Assignees

Clear assignees

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#638

No description provided.