Question: Why is /root/.borgmatic being backed up? #638

New Issue

baj0k · 2023-02-12T04:11:14Z

baj0k commented

2023-02-12 04:11:14 +00:00

What I'm trying to do and why

I have a working borg backup setup but I noticed recently during manual extraction of archive that files in the

/root/.borgmatic
are included in it.

I was not aware of this and could not find information about it on borg / borgmatic wikis so I end up posting an issue about it. I feel like the reason behind it should be obvious since I think this is related to checks being enabled on the repository. The files that are being backed up are in /root/.borgmatic/checks directory, but it doesn't match my inclusion/exclusion patterns and I wanted to be sure that this behavior is intended.

Steps to reproduce (if a bug)

config.yaml:

location:
    repositories:
        - ${REPO_EU}
        - ${REPO_US}

    patterns:
        - R /backup
        - P fm
        - '+ */docker-compose.yml'
        - '+ */.env'
        # borgmatic
        - '+ */borgmatic/*'
        # postgres
        - '+ */postgres/entrypoint/*'
        # vaultwarden
        - '+ */vaultwarden/attachments/*'
        - '+ */vaultwarden/sends/*'
        - '+ */vaultwarden/config.json'
        # nginx
        - '+ */nginx/*'
        # exclude catch-all
        - '- /backup*'


storage:
    compression: lz4
    archive_name_format: 'selfhost-{now:%Y-%m-%dT%H}'

retention:
    keep_daily: 3
    keep_weekly: 2
    keep_monthly: 1
    prefix: selfhost

consistency:
    checks:
        - name: repository
          frequency: 1 day
        - name: archives
          frequency: 3 days

    check_last: 3
    prefix: selfhost

output:
    color: true

hooks:
    before_everything:
        - echo "Stopping containers..."
        - 'name=wiki; curl -s --unix-socket /var/run/docker.sock -X POST http://localhost/v1.41/containers/$name/stop  || echo "Failed to stop container $name"'
        - 'name=vault; curl -s --unix-socket /var/run/docker.sock -X POST http://localhost/v1.41/containers/$name/stop || echo "Failed to stop container $name"'
        - sleep 10
        - echo "Finished stopping containers"

    after_everything:
        - echo "Starting containers..."
        - 'name=wiki; curl -s --unix-socket /var/run/docker.sock -X POST http://localhost/v1.41/containers/$name/start || echo "Failed to start container $name"'
        - 'name=vault; curl -s --unix-socket /var/run/docker.sock -X POST http://localhost/v1.41/containers/$name/start || echo "Failed to start container $name"'
        - echo "Finished starting containers"

    before_prune:
        - echo "Starting pruning..."
    after_prune:
        - echo "Finished pruning"

    before_backup:
        - echo "Starting backup..."
    after_backup:
        - echo "Finished backup"

    before_check:
        - echo "Starting checks..."
    after_check:
        - echo "Finished checks"

    before_extract:
        - echo "Starting extracting..."
    after_extract:
        - echo "Finished extracting"

    on_error:
        - echo "Error durring borgmatic operation or while running one of the hooks!"

    postgresql_databases:
      - name: all
        hostname: db
        username: ${POSTGRES_USERNAME}
        password: ${POSTGRES_PASSWORD}

Actual behavior (if a bug)

When running borgmatic --dry-run --list with the above config I see following lines in the output

...
x /backup/letsencrypt/log/letsencrypt.log.3
x /backup/letsencrypt/log/letsencrypt.log.2
x /backup/letsencrypt/log/letsencrypt.log.1
x /backup/letsencrypt/log/letsencrypt.log
- /root/.borgmatic/checks/61baece2a5326eccdea987b1b15430d748f1b0ff3c51453c22f8cc8df87cf5c2/repository
- /root/.borgmatic/checks/61baece2a5326eccdea987b1b15430d748f1b0ff3c51453c22f8cc8df87cf5c2/archives
- /root/.borgmatic/checks/61baece2a5326eccdea987b1b15430d748f1b0ff3c51453c22f8cc8df87cf5c2
- /root/.borgmatic/checks
- /root/.borgmatic
x /backup

Environment

borgmatic version: 1.7.6

Borg version: 1.2.3

Python version: 3.11.1

#### What I'm trying to do and why I have a working borg backup setup but I noticed recently during manual extraction of archive that files in the - /root/.borgmatic are included in it. I was not aware of this and could not find information about it on borg / borgmatic wikis so I end up posting an issue about it. I feel like the reason behind it should be obvious since I think this is related to checks being enabled on the repository. The files that are being backed up are in /root/.borgmatic/checks directory, but it doesn't match my inclusion/exclusion patterns and I wanted to be sure that this behavior is intended. #### Steps to reproduce (if a bug) config.yaml: ``` location: repositories: - ${REPO_EU} - ${REPO_US} patterns: - R /backup - P fm - '+ */docker-compose.yml' - '+ */.env' # borgmatic - '+ */borgmatic/*' # postgres - '+ */postgres/entrypoint/*' # vaultwarden - '+ */vaultwarden/attachments/*' - '+ */vaultwarden/sends/*' - '+ */vaultwarden/config.json' # nginx - '+ */nginx/*' # exclude catch-all - '- /backup*' storage: compression: lz4 archive_name_format: 'selfhost-{now:%Y-%m-%dT%H}' retention: keep_daily: 3 keep_weekly: 2 keep_monthly: 1 prefix: selfhost consistency: checks: - name: repository frequency: 1 day - name: archives frequency: 3 days check_last: 3 prefix: selfhost output: color: true hooks: before_everything: - echo "Stopping containers..." - 'name=wiki; curl -s --unix-socket /var/run/docker.sock -X POST http://localhost/v1.41/containers/$name/stop || echo "Failed to stop container $name"' - 'name=vault; curl -s --unix-socket /var/run/docker.sock -X POST http://localhost/v1.41/containers/$name/stop || echo "Failed to stop container $name"' - sleep 10 - echo "Finished stopping containers" after_everything: - echo "Starting containers..." - 'name=wiki; curl -s --unix-socket /var/run/docker.sock -X POST http://localhost/v1.41/containers/$name/start || echo "Failed to start container $name"' - 'name=vault; curl -s --unix-socket /var/run/docker.sock -X POST http://localhost/v1.41/containers/$name/start || echo "Failed to start container $name"' - echo "Finished starting containers" before_prune: - echo "Starting pruning..." after_prune: - echo "Finished pruning" before_backup: - echo "Starting backup..." after_backup: - echo "Finished backup" before_check: - echo "Starting checks..." after_check: - echo "Finished checks" before_extract: - echo "Starting extracting..." after_extract: - echo "Finished extracting" on_error: - echo "Error durring borgmatic operation or while running one of the hooks!" postgresql_databases: - name: all hostname: db username: ${POSTGRES_USERNAME} password: ${POSTGRES_PASSWORD} ``` #### Actual behavior (if a bug) When running borgmatic --dry-run --list with the above config I see following lines in the output ``` ... x /backup/letsencrypt/log/letsencrypt.log.3 x /backup/letsencrypt/log/letsencrypt.log.2 x /backup/letsencrypt/log/letsencrypt.log.1 x /backup/letsencrypt/log/letsencrypt.log - /root/.borgmatic/checks/61baece2a5326eccdea987b1b15430d748f1b0ff3c51453c22f8cc8df87cf5c2/repository - /root/.borgmatic/checks/61baece2a5326eccdea987b1b15430d748f1b0ff3c51453c22f8cc8df87cf5c2/archives - /root/.borgmatic/checks/61baece2a5326eccdea987b1b15430d748f1b0ff3c51453c22f8cc8df87cf5c2 - /root/.borgmatic/checks - /root/.borgmatic x /backup ``` #### Environment **borgmatic version:** 1.7.6 **Borg version:** 1.2.3 **Python version:** 3.11.1

👍 1

witten commented

2023-02-12 05:51:34 +00:00

Thanks for taking the time to file this including all the details about your setup! The short answer as to why ~/.borgmatic is being backed up is that doing so directly supports database dump backups. Each database dump is streamed from your database to a special named pipe within ~/.borgmatic and from there it goes directly to Borg without taking up any temporary space on disk.

This "feature" is documented on the database backup page, but maybe it needs to be made more prominent. And as a side effect, it backs up ~/.borgmatic/checks as well. Which isn't the worst thing, because it means your check state gets backed up as well. (But it is an implicit side effect, which maybe isn't so great.)

So, to answer your question: It is intended behavior. But I'm still open to making improvements to it!

Thanks for taking the time to file this including all the details about your setup! The short answer as to why `~/.borgmatic` is being backed up is that doing so directly supports database dump backups. Each database dump is streamed from your database to a special named pipe within `~/.borgmatic` and from there it goes directly to Borg without taking up any temporary space on disk. This "feature" *is* documented on the [database backup page](https://torsion.org/borgmatic/docs/how-to/backup-your-databases/), but maybe it needs to be made more prominent. And as a side effect, it backs up `~/.borgmatic/checks` as well. Which isn't the worst thing, because it means your check state gets backed up as well. (But it is an implicit side effect, which maybe isn't so great.) So, to answer your question: It is intended behavior. But I'm still open to making improvements to it!

👍 1

witten added the

question / support

label 2023-02-12 05:51:41 +00:00

baj0k commented

2023-02-12 11:55:50 +00:00

So I think now the question is - is it a good thing to include ~/.borgmatic/checks just because they happen to share a common directory with the named pipe for the databases.

I can think of scenarios where backing up the check states is desired but I think it shouldn't be done automatically without the user knowledge. At least to avoid confusion like in my case.

If you agree then I think the solution would be to simply include ~/.borgmatic/*/db/* instead of ~/.borgmatic/*. What do you think?

Edit: There is a really simple workaround for this if you think the behavior is not worth changing. If you don't want to have check states backed up you can exclude ~/.borgmatic/checks/* in the config but I still think that the "check states being backed up when database hooks are enabled" behavior should be documented somewhere.

So I think now the question is - is it a good thing to include `~/.borgmatic/checks` just because they happen to share a common directory with the named pipe for the databases. I can think of scenarios where backing up the check states is desired but I think it shouldn't be done automatically without the user knowledge. At least to avoid confusion like in my case. If you agree then I think the solution would be to simply include `~/.borgmatic/*/db/*` instead of `~/.borgmatic/*`. What do you think? Edit: There is a really simple workaround for this if you think the behavior is not worth changing. If you don't want to have check states backed up you can exclude `~/.borgmatic/checks/*` in the config but I still think that the "check states being backed up when database hooks are enabled" behavior should be documented somewhere.

👍 1

witten commented

2023-02-14 17:35:09 +00:00

Changing this to avoid implicitly backing up the check state makes sense to me.

👍 1

witten removed the

question / support

label 2023-04-07 04:31:02 +00:00

witten referenced this issue

2024-03-12 04:06:31 +00:00

XDG directories #562

Sign in to join this conversation.