#201 Using Docker-compose to make backups but fails and gives me errorlog

Open
opened 2 months ago by suyiwunoye · 16 comments

What I’m trying to do and why

Using docker-compose with b3vis/borgmatic docker image to make backups of my current computer. For now testing if it works. But it didnt seem to work. It’s probably my mistake as its my first time trying this up.

Steps to reproduce (if a bug)

Docker-compose.yml contains

version: ‘3’ services: borgmatic:

image: b3vis/borgmatic
container_name: borgmatic
volumes:
  - '/home:/mnt/source:ro'                 # backup source
  - '$PWD/repository:/mnt/repository'    # backup target
  - '$PWD/borgmatic.d:/etc/borgmatic.d/' # borgmatic config file(s) + crontab.txt
  - '$PWD/data/.cache:/root/.cache/borg'      # checksums used for deduplication
environment:
  - TZ=Europe/Helsinki

config.yaml contains

location:

# List of source directories to backup. Globs are expanded.
source_directories:
    - /home/Docker
repositories:
    - /mnt/repository/
one_file_system: true

storage:

compression: lz4
archive_name_format: 'backup-{now}'

retention:

keep_hourly: 2
keep_daily: 7
keep_weekly: 4
keep_monthly: 12
keep_yearly: 10
prefix: 'backup-'

consistency:

# List of consistency checks to run: "repository", "archives", or both.
checks:
    - repository
    - archives
check_last: 3
prefix: 'backup-'

hooks:

before_backup:
    - echo "Starting a backup job."
after_backup:
    - echo "Backup created."
on_error:
    - echo "Error while creating a backup."

So I logged into the image by this docker exec -it borgmatic sh

And then created the passwordless repo borg init -e none /mnt/repository

I thought that this should be enough to make it automatically make backups cos it does read the cronfile.

Include (sanitized) --verbosity 2 output if applicable.

From docker-compose logs

borgmatic | crond: crond (busybox 1.29.3) started, log level 8 borgmatic | crond: USER root pid 8 cmd PATH=$PATH:/usr/bin /usr/bin/borgmatic --stats -v 0 2>&1 borgmatic | Starting a backup job. borgmatic | Exception ignored in: > borgmatic | Traceback (most recent call last): borgmatic | File “/usr/lib/python3.6/borg/repository.py”, line 179, in del borgmatic | assert False, “cleanup happened in Repository.del“ borgmatic | AssertionError: cleanup happened in Repository.del borgmatic | Local Exception borgmatic | Traceback (most recent call last): borgmatic | File “/usr/lib/python3.6/borg/archiver.py”, line 4455, in main borgmatic | exit_code = archiver.run(args) borgmatic | File “/usr/lib/python3.6/borg/archiver.py”, line 4387, in run borgmatic | return set_ec(func(args)) borgmatic | File “/usr/lib/python3.6/borg/archiver.py”, line 139, in wrapper borgmatic | with repository: borgmatic | File “/usr/lib/python3.6/borg/repository.py”, line 189, in enter borgmatic | self.open(self.path, bool(self.exclusive), lock_wait=self.lock_wait, lock=self.do_lock) borgmatic | File “/usr/lib/python3.6/borg/repository.py”, line 396, in open borgmatic | with open(os.path.join(self.path, ‘config’)) as fd: borgmatic | FileNotFoundError: [Errno 2] No such file or directory: ‘/mnt/repository/config’ borgmatic | borgmatic | Platform: Linux 95d40d52d63b 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u3 (2019-06-16) x86_64 borgmatic | Linux: borgmatic | Borg: 1.1.9 Python: CPython 3.6.8 borgmatic | PID: 10 CWD: /root borgmatic | sys.argv: [‘/usr/bin/borg’, ‘prune’, ‘/mnt/repository/’, ‘--keep-hourly’, ‘2’, ‘--keep-daily’, ‘7’, ‘--keep-weekly’, ‘4’, ‘--keep-monthly’, ‘12’, ‘--keep-yearly’, ‘10’, ‘--prefix’, ‘backup-’, ‘--stats’] borgmatic | SSH_ORIGINAL_COMMAND: None borgmatic | borgmatic | Error while creating a backup. borgmatic | /etc/borgmatic.d/config.yaml: Error running configuration file borgmatic | Command ‘(‘borg’, ‘prune’, ‘/mnt/repository/’, ‘--keep-hourly’, ‘2’, ‘--keep-daily’, ‘7’, ‘--keep-weekly’, ‘4’, ‘--keep-monthly’, ‘12’, ‘--keep-yearly’, ‘10’, ‘--prefix’, ‘backup-’, ‘--stats’)’ returned non-zero exit status 2. borgmatic | borgmatic | Need some help? https://torsion.org/borgmatic/#issues

Other notes / implementation ideas

I thought it would automatically just backup my /home/Docker location to /mnt/repository/ without passwords.

Environment

borgmatic version: [version here]

1.3.0

borgmatic installation method: [e.g., Debian package, Docker container, etc.]

Debian server using docker.

#### What I'm trying to do and why Using docker-compose with b3vis/borgmatic docker image to make backups of my current computer. For now testing if it works. But it didnt seem to work. It's probably my mistake as its my first time trying this up. #### Steps to reproduce (if a bug) Docker-compose.yml contains version: '3' services: borgmatic: image: b3vis/borgmatic container_name: borgmatic volumes: - '/home:/mnt/source:ro' # backup source - '$PWD/repository:/mnt/repository' # backup target - '$PWD/borgmatic.d:/etc/borgmatic.d/' # borgmatic config file(s) + crontab.txt - '$PWD/data/.cache:/root/.cache/borg' # checksums used for deduplication environment: - TZ=Europe/Helsinki config.yaml contains location: # List of source directories to backup. Globs are expanded. source_directories: - /home/Docker repositories: - /mnt/repository/ one_file_system: true storage: compression: lz4 archive_name_format: 'backup-{now}' retention: keep_hourly: 2 keep_daily: 7 keep_weekly: 4 keep_monthly: 12 keep_yearly: 10 prefix: 'backup-' consistency: # List of consistency checks to run: "repository", "archives", or both. checks: - repository - archives check_last: 3 prefix: 'backup-' hooks: before_backup: - echo "Starting a backup job." after_backup: - echo "Backup created." on_error: - echo "Error while creating a backup." So I logged into the image by this docker exec -it borgmatic sh And then created the passwordless repo borg init -e none /mnt/repository I thought that this should be enough to make it automatically make backups cos it does read the cronfile. Include (sanitized) `--verbosity 2` output if applicable. From docker-compose logs borgmatic | crond: crond (busybox 1.29.3) started, log level 8 borgmatic | crond: USER root pid 8 cmd PATH=$PATH:/usr/bin /usr/bin/borgmatic --stats -v 0 2>&1 borgmatic | Starting a backup job. borgmatic | Exception ignored in: <bound method Repository.__del__ of <Repository /mnt/repository>> borgmatic | Traceback (most recent call last): borgmatic | File "/usr/lib/python3.6/borg/repository.py", line 179, in __del__ borgmatic | assert False, "cleanup happened in Repository.__del__" borgmatic | AssertionError: cleanup happened in Repository.__del__ borgmatic | Local Exception borgmatic | Traceback (most recent call last): borgmatic | File "/usr/lib/python3.6/borg/archiver.py", line 4455, in main borgmatic | exit_code = archiver.run(args) borgmatic | File "/usr/lib/python3.6/borg/archiver.py", line 4387, in run borgmatic | return set_ec(func(args)) borgmatic | File "/usr/lib/python3.6/borg/archiver.py", line 139, in wrapper borgmatic | with repository: borgmatic | File "/usr/lib/python3.6/borg/repository.py", line 189, in __enter__ borgmatic | self.open(self.path, bool(self.exclusive), lock_wait=self.lock_wait, lock=self.do_lock) borgmatic | File "/usr/lib/python3.6/borg/repository.py", line 396, in open borgmatic | with open(os.path.join(self.path, 'config')) as fd: borgmatic | FileNotFoundError: [Errno 2] No such file or directory: '/mnt/repository/config' borgmatic | borgmatic | Platform: Linux 95d40d52d63b 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u3 (2019-06-16) x86_64 borgmatic | Linux: borgmatic | Borg: 1.1.9 Python: CPython 3.6.8 borgmatic | PID: 10 CWD: /root borgmatic | sys.argv: ['/usr/bin/borg', 'prune', '/mnt/repository/', '--keep-hourly', '2', '--keep-daily', '7', '--keep-weekly', '4', '--keep-monthly', '12', '--keep-yearly', '10', '--prefix', 'backup-', '--stats'] borgmatic | SSH_ORIGINAL_COMMAND: None borgmatic | borgmatic | Error while creating a backup. borgmatic | /etc/borgmatic.d/config.yaml: Error running configuration file borgmatic | Command '('borg', 'prune', '/mnt/repository/', '--keep-hourly', '2', '--keep-daily', '7', '--keep-weekly', '4', '--keep-monthly', '12', '--keep-yearly', '10', '--prefix', 'backup-', '--stats')' returned non-zero exit status 2. borgmatic | borgmatic | Need some help? https://torsion.org/borgmatic/#issues #### Other notes / implementation ideas I thought it would automatically just backup my /home/Docker location to /mnt/repository/ without passwords. #### Environment **borgmatic version:** [version here] 1.3.0 **borgmatic installation method:** [e.g., Debian package, Docker container, etc.] Debian server using docker.
witten commented 2 months ago
Owner

Thanks for the detailed bug report. Based on the error message you included, it looks like Borg isn’t finding a file it expects to find in any initialized Borg repository, /mnt/repository/config in this case.

When you list /mnt/repository from within the container after, do you see a config directory, a data directory, etc? And do you see those same files show up outside of the container at $PWD/repository?

Note that I don’t maintain the b3vis/borgmatic Docker image myself, so I’m making a guess here!

Thanks for the detailed bug report. Based on the error message you included, it looks like Borg isn't finding a file it expects to find in any initialized Borg repository, `/mnt/repository/config` in this case. When you list `/mnt/repository` from within the container after, do you see a `config` directory, a `data` directory, etc? And do you see those same files show up outside of the container at `$PWD/repository`? Note that I don't maintain the b3vis/borgmatic Docker image myself, so I'm making a guess here!
witten added the
question / support
label 2 months ago
suyiwunoye commented 2 months ago
Poster

Hi!

Yeah you were right! That was the problem. Seems like I messed that one up as I made too many changes afterward.

Sorry I should’ve have probably add t he issue on to the b3vis/borgmatic.

If I am not wrong it doesnt allow using unencrypted repositories?

borgmatic | Starting a backup job. borgmatic | Warning: Attempting to access a previously unknown unencrypted re pository! borgmatic | Do you want to continue? [yN] Aborting. borgmatic | Cache initialization aborted borgmatic | Error while creating a backup. borgmatic | /etc/borgmatic.d/config.yaml: Error running configuration file borgmatic | Command ‘(‘borg’, ‘prune’, ‘/mnt/repository/’, ‘--keep-hourly’, ‘ 2’, ‘--keep-daily’, ‘7’, ‘--keep-weekly’, ‘4’, ‘--keep-monthly’, ‘12’, ‘--keep-y early’, ‘10’, ‘--prefix’, ‘backup-’, ‘--stats’)’ returned non-zero exit status 2 . borgmatic | borgmatic | Need some help? https://torsion.org/borgmatic/#issues

PS. Seems like Gitea doesnt allow formating.

Hi! Yeah you were right! That was the problem. Seems like I messed that one up as I made too many changes afterward. Sorry I should've have probably add t he issue on to the b3vis/borgmatic. If I am not wrong it doesnt allow using unencrypted repositories? borgmatic | Starting a backup job. borgmatic | Warning: Attempting to access a previously unknown unencrypted re pository! borgmatic | Do you want to continue? [yN] Aborting. borgmatic | Cache initialization aborted borgmatic | Error while creating a backup. borgmatic | /etc/borgmatic.d/config.yaml: Error running configuration file borgmatic | Command '('borg', 'prune', '/mnt/repository/', '--keep-hourly', ' 2', '--keep-daily', '7', '--keep-weekly', '4', '--keep-monthly', '12', '--keep-y early', '10', '--prefix', 'backup-', '--stats')' returned non-zero exit status 2 . borgmatic | borgmatic | Need some help? https://torsion.org/borgmatic/#issues PS. Seems like Gitea doesnt allow formating.
witten commented 2 months ago
Owner

I’m not sure that using an unencrypted repository would be a problem, but it does look like it’s giving you a warning and trying to prompt for input (“Do you want to continue?”) So you could try running the borg prune command manually from within the container and see what happens if you answer “y” to the prompt:

borg prune /mnt/repository/ --keep-hourly  2 --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --keep-y early 10 --prefix backup- --stats

Maybe after doing that, the next run will work without prompting?

Alternatively, you could consider using an authenticated encryption mode instead of none if you’re using Borg 1.1+. More at https://borgbackup.readthedocs.io/en/stable/usage/init.html#encryption-modes

I'm not sure that using an unencrypted repository would be a problem, but it does look like it's giving you a warning and trying to prompt for input ("Do you want to continue?") So you could try running the `borg prune` command manually from within the container and see what happens if you answer "y" to the prompt: ``` borg prune /mnt/repository/ --keep-hourly 2 --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --keep-y early 10 --prefix backup- --stats ``` Maybe after doing that, the next run will work without prompting? Alternatively, you could consider using an `authenticated` encryption mode instead of `none` if you're using Borg 1.1+. More at https://borgbackup.readthedocs.io/en/stable/usage/init.html#encryption-modes
suyiwunoye commented 2 months ago
Poster

Hi!

Thanks for your persistence and help.

For now when I try

/mnt # borg prune repository/

I get

Warning: Attempting to access a previously unknown unencrypted repository! Do you want to continue? [yN] y At least one of the “keep-within”, “keep-last”, “keep-secondly”, “keep-minutely”, “keep-hourly”, “keep-daily”, “keep-weekly”, “keep-monthly” or “keep-yearly” settings must be specified.

So I guess its need to be specified on some way? I thought it was meant to read from the config.yaml

Hi! Thanks for your persistence and help. For now when I try /mnt # borg prune repository/ I get Warning: Attempting to access a previously unknown unencrypted repository! Do you want to continue? [yN] y At least one of the "keep-within", "keep-last", "keep-secondly", "keep-minutely", "keep-hourly", "keep-daily", "keep-weekly", "keep-monthly" or "keep-yearly" settings must be specified. So I guess its need to be specified on some way? I thought it was meant to read from the config.yaml
witten commented 2 months ago
Owner

If you run the borg command by itself, it doesn’t know anything about borgmatic’s config file, so it won’t use any of the settings from it. So you can try the whole borg prune command above, which effectively provides those “keep” settings to Borg directly on the command-line.

If you run the `borg` command by itself, it doesn't know anything about borgmatic's config file, so it won't use any of the settings from it. So you can try the whole `borg prune` command above, which effectively provides those "keep" settings to Borg directly on the command-line.
suyiwunoye commented 2 months ago
Poster

What are the possible problems when using authenticated version instead of none? Not encrypted, but authenticated and based on Hash/MAC. Does it mean that it needs the same computer MAC to unencrypt it? So the problems will come when I try to open it on my another laptop.

What are the possible problems when using authenticated version instead of none? Not encrypted, but authenticated and based on Hash/MAC. Does it mean that it needs the same computer MAC to unencrypt it? So the problems will come when I try to open it on my another laptop.
witten commented 2 months ago
Owner

I don’t think it has anything to do with a computer hardware MAC. I think it’s referring to a different concept: A cryptographic Message Authentication Code. My understanding is that it allows Borg to validate that each piece of your data is correctly stored and hasn’t become corrupted. I can’t think of any problems/downsides of that, except perhaps a small performance hit versus the “none” encryption mode.

I don't think it has anything to do with a computer hardware MAC. I think it's referring to a different concept: A cryptographic [Message Authentication Code](https://en.wikipedia.org/wiki/HMAC). My understanding is that it allows Borg to validate that each piece of your data is correctly stored and hasn't become corrupted. I can't think of any problems/downsides of that, except perhaps a small performance hit versus the "none" encryption mode.
pr0totyp commented 3 days ago

Had the same problem while using the b3vis/docker-borgmatic docker container.

The config directory at /root/.config/borg is not persisted as docker volume. Even when using the none encryption option borg is still storing some information in this directory.

Just add a bind mount for /root/.config/borg to your compose file.

Had the same problem while using the [b3vis/docker-borgmatic](https://github.com/b3vis/docker-borgmatic) docker container. The config directory at `/root/.config/borg` is not persisted as docker volume. Even when using the none encryption option borg is still storing some information in this directory. Just add a bind mount for `/root/.config/borg` to your compose file.
pr0totyp commented 3 days ago

But I am also having the first issue where borgmatic is loocking at a config file..

While creating the borg repository inside the container:

/ # /usr/bin/borgmatic -c /etc/borgmatic.d/config.yml init -e none
Exception ignored in: <function Repository.__del__ at 0x7f2b1cae5268>
Traceback (most recent call last):
  File "/usr/lib/python3.7/borg/repository.py", line 179, in __del__
    assert False, "cleanup happened in Repository.__del__"
AssertionError: cleanup happened in Repository.__del__
Local Exception
Traceback (most recent call last):
  File "/usr/lib/python3.7/borg/archiver.py", line 4501, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3.7/borg/archiver.py", line 4433, in run
    return set_ec(func(args))
  File "/usr/lib/python3.7/borg/archiver.py", line 151, in wrapper
    with repository:
  File "/usr/lib/python3.7/borg/repository.py", line 189, in __enter__
    self.open(self.path, bool(self.exclusive), lock_wait=self.lock_wait, lock=self.do_lock)
  File "/usr/lib/python3.7/borg/repository.py", line 396, in open
    with open(os.path.join(self.path, 'config')) as fd:
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/borg-repository/config'

Platform: Linux 52ae536bd212 4.14.85 #1 SMP Wed Dec 5 12:54:02 UTC 2018 x86_64
Linux:   
Borg: 1.1.10  Python: CPython 3.7.3 msgpack: 0.5.6
PID: 8  CWD: /
sys.argv: ['/usr/bin/borg', 'info', '/mnt/borg-repository']
SSH_ORIGINAL_COMMAND: None

Borgmatic config:

location:                                                                                                                                                                                                                                                                                                                               
    source_directories:                                                                                                                                                                               
        - /mnt/source/system                                                                                                                                                    
    repositories:
        - /mnt/borg-repository
storage:
    borg_cache_directory: /root/.cache/borg/
    archive_name_format: 'web-{now}'

retention:
    keep_daily: 14
    keep_weekly: 4
    keep_monthly: 12
    keep_yearly: 1
    prefix: 'web-'

consistency:
    checks:
        - repository
        - archives

    prefix: 'web-'
hooks:
    before_backup:
        - echo "`date` - Starting a backup job."
    after_backup:
        - echo "`date` - Backup created."
   on_error:
        - echo "`date` - Error while creating a backup.

docker-compose:

  web_file_backup:
    image: b3vis/borgmatic
    env_file: .env
    volumes: 
      - ./borgbackup/config.yml:/etc/borgmatic.d/config.yml
      - ./borgbackup/crontab.txt:/etc/borgmatic.d/crontab.txt
      - /srv/web/public/:/mnt/source/:ro
      - /srv/web/log/:/log/
      - /srv/web/backup/borg_repo:/mnt/borg-repository
      - /srv/web/backup/borg_cache:/root/.cache/borg
      - /srv/web/backup/borg_config:/root/.config/borg
But I am also having the first issue where borgmatic is loocking at a config file.. While creating the borg repository inside the container: ``` / # /usr/bin/borgmatic -c /etc/borgmatic.d/config.yml init -e none Exception ignored in: <function Repository.__del__ at 0x7f2b1cae5268> Traceback (most recent call last): File "/usr/lib/python3.7/borg/repository.py", line 179, in __del__ assert False, "cleanup happened in Repository.__del__" AssertionError: cleanup happened in Repository.__del__ Local Exception Traceback (most recent call last): File "/usr/lib/python3.7/borg/archiver.py", line 4501, in main exit_code = archiver.run(args) File "/usr/lib/python3.7/borg/archiver.py", line 4433, in run return set_ec(func(args)) File "/usr/lib/python3.7/borg/archiver.py", line 151, in wrapper with repository: File "/usr/lib/python3.7/borg/repository.py", line 189, in __enter__ self.open(self.path, bool(self.exclusive), lock_wait=self.lock_wait, lock=self.do_lock) File "/usr/lib/python3.7/borg/repository.py", line 396, in open with open(os.path.join(self.path, 'config')) as fd: FileNotFoundError: [Errno 2] No such file or directory: '/mnt/borg-repository/config' Platform: Linux 52ae536bd212 4.14.85 #1 SMP Wed Dec 5 12:54:02 UTC 2018 x86_64 Linux: Borg: 1.1.10 Python: CPython 3.7.3 msgpack: 0.5.6 PID: 8 CWD: / sys.argv: ['/usr/bin/borg', 'info', '/mnt/borg-repository'] SSH_ORIGINAL_COMMAND: None ``` Borgmatic config: ``` location: source_directories: - /mnt/source/system repositories: - /mnt/borg-repository storage: borg_cache_directory: /root/.cache/borg/ archive_name_format: 'web-{now}' retention: keep_daily: 14 keep_weekly: 4 keep_monthly: 12 keep_yearly: 1 prefix: 'web-' consistency: checks: - repository - archives prefix: 'web-' hooks: before_backup: - echo "`date` - Starting a backup job." after_backup: - echo "`date` - Backup created." on_error: - echo "`date` - Error while creating a backup. ``` docker-compose: ``` web_file_backup: image: b3vis/borgmatic env_file: .env volumes: - ./borgbackup/config.yml:/etc/borgmatic.d/config.yml - ./borgbackup/crontab.txt:/etc/borgmatic.d/crontab.txt - /srv/web/public/:/mnt/source/:ro - /srv/web/log/:/log/ - /srv/web/backup/borg_repo:/mnt/borg-repository - /srv/web/backup/borg_cache:/root/.cache/borg - /srv/web/backup/borg_config:/root/.config/borg ```
witten commented 3 days ago
Owner

If you ls /srv/web/backup/borg_repo on the host, do you see a config directory in there? And what about if you exec into the backup container and run ls /mnt/borg-repository?

I think the problem here is that the code that tries to probe for an existing repository works in most cases, but may not work across a Docker mount. FYI the code is in borgmatic/borg/init.py:initialize_repository().

If you `ls /srv/web/backup/borg_repo` on the host, do you see a `config` directory in there? And what about if you `exec` into the backup container and run `ls /mnt/borg-repository`? I *think* the problem here is that the code that tries to probe for an existing repository works in most cases, but may not work across a Docker mount. FYI the code is in `borgmatic/borg/init.py:initialize_repository()`.
witten added the
bug
label 3 days ago
witten commented 3 days ago
Owner

Also, is borgmatic actually erroring upon repository creation, or is it just spewing the Borg traceback to the console and then continuing along successfully?

Also, is borgmatic actually erroring upon repository creation, or is it just spewing the Borg traceback to the console and then continuing along successfully?
pr0totyp commented 2 days ago

The folder is empty when checking inside and outside the container.

The error is only raised while borg repo initialization.

Unfortunately I am not so much experienced with python to debug this problem.

The folder is empty when checking inside and outside the container. The error is only raised while borg repo initialization. Unfortunately I am not so much experienced with python to debug this problem.
pr0totyp commented 2 days ago

Borgmatic is finishing the repository creation successfully. The error is just from the Borg traceback.

Borgmatic is finishing the repository creation successfully. The error is just from the Borg traceback.
witten commented 2 days ago
Owner

Great. I’ve got a local repro here, and a potential solution.

Great. I've got a local repro here, and a potential solution.
witten commented 2 days ago
Owner

Okay, this is now fixed in master. It’ll be part of the next release. Thanks for the bug report!

Okay, this is now fixed in master. It'll be part of the next release. Thanks for the bug report!
witten commented 1 day ago
Owner

Reopening this, because the fix broke some tests, and I had to back it out.

Reopening this, because the fix broke some tests, and I had to back it out.
witten reopened 1 day ago
Sign in to join this conversation.
No Milestone
No Assignees
3 Participants
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Loading…
Cancel
Save
There is no content yet.