Using Docker-compose to make backups but fails and gives me errorlog #201
Labels
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No Milestone
No Assignees
4 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: borgmatic-collective/borgmatic#201
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What I'm trying to do and why
Using docker-compose with b3vis/borgmatic docker image to make backups of my current computer. For now testing if it works. But it didnt seem to work. It's probably my mistake as its my first time trying this up.
Steps to reproduce (if a bug)
Docker-compose.yml contains
version: '3'
services:
borgmatic:
image: b3vis/borgmatic
container_name: borgmatic
volumes:
- '/home:/mnt/source:ro' # backup source
- '$PWD/repository:/mnt/repository' # backup target
- '$PWD/borgmatic.d:/etc/borgmatic.d/' # borgmatic config file(s) + crontab.txt
- '$PWD/data/.cache:/root/.cache/borg' # checksums used for deduplication
environment:
- TZ=Europe/Helsinki
config.yaml contains
location:
# List of source directories to backup. Globs are expanded.
source_directories:
- /home/Docker
repositories:
- /mnt/repository/
one_file_system: true
storage:
compression: lz4
archive_name_format: 'backup-{now}'
retention:
keep_hourly: 2
keep_daily: 7
keep_weekly: 4
keep_monthly: 12
keep_yearly: 10
prefix: 'backup-'
consistency:
# List of consistency checks to run: "repository", "archives", or both.
checks:
- repository
- archives
check_last: 3
prefix: 'backup-'
hooks:
before_backup:
- echo "Starting a backup job."
after_backup:
- echo "Backup created."
on_error:
- echo "Error while creating a backup."
So I logged into the image by this
docker exec -it borgmatic sh
And then created the passwordless repo
borg init -e none /mnt/repository
I thought that this should be enough to make it automatically make backups cos it does read the cronfile.
Include (sanitized)
--verbosity 2
output if applicable.From docker-compose logs
borgmatic | crond: crond (busybox 1.29.3) started, log level 8
borgmatic | crond: USER root pid 8 cmd PATH=$PATH:/usr/bin /usr/bin/borgmatic --stats -v 0 2>&1
borgmatic | Starting a backup job.
borgmatic | Exception ignored in: <bound method Repository.del of <Repository /mnt/repository>>
borgmatic | Traceback (most recent call last):
borgmatic | File "/usr/lib/python3.6/borg/repository.py", line 179, in del
borgmatic | assert False, "cleanup happened in Repository.del"
borgmatic | AssertionError: cleanup happened in Repository.del
borgmatic | Local Exception
borgmatic | Traceback (most recent call last):
borgmatic | File "/usr/lib/python3.6/borg/archiver.py", line 4455, in main
borgmatic | exit_code = archiver.run(args)
borgmatic | File "/usr/lib/python3.6/borg/archiver.py", line 4387, in run
borgmatic | return set_ec(func(args))
borgmatic | File "/usr/lib/python3.6/borg/archiver.py", line 139, in wrapper
borgmatic | with repository:
borgmatic | File "/usr/lib/python3.6/borg/repository.py", line 189, in enter
borgmatic | self.open(self.path, bool(self.exclusive), lock_wait=self.lock_wait, lock=self.do_lock)
borgmatic | File "/usr/lib/python3.6/borg/repository.py", line 396, in open
borgmatic | with open(os.path.join(self.path, 'config')) as fd:
borgmatic | FileNotFoundError: [Errno 2] No such file or directory: '/mnt/repository/config'
borgmatic |
borgmatic | Platform: Linux 95d40d52d63b 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u3 (2019-06-16) x86_64
borgmatic | Linux:
borgmatic | Borg: 1.1.9 Python: CPython 3.6.8
borgmatic | PID: 10 CWD: /root
borgmatic | sys.argv: ['/usr/bin/borg', 'prune', '/mnt/repository/', '--keep-hourly', '2', '--keep-daily', '7', '--keep-weekly', '4', '--keep-monthly', '12', '--keep-yearly', '10', '--prefix', 'backup-', '--stats']
borgmatic | SSH_ORIGINAL_COMMAND: None
borgmatic |
borgmatic | Error while creating a backup.
borgmatic | /etc/borgmatic.d/config.yaml: Error running configuration file
borgmatic | Command '('borg', 'prune', '/mnt/repository/', '--keep-hourly', '2', '--keep-daily', '7', '--keep-weekly', '4', '--keep-monthly', '12', '--keep-yearly', '10', '--prefix', 'backup-', '--stats')' returned non-zero exit status 2.
borgmatic |
borgmatic | Need some help? https://torsion.org/borgmatic/#issues
Other notes / implementation ideas
I thought it would automatically just backup my /home/Docker location to /mnt/repository/ without passwords.
Environment
borgmatic version: [version here]
1.3.0
borgmatic installation method: [e.g., Debian package, Docker container, etc.]
Debian server using docker.
Thanks for the detailed bug report. Based on the error message you included, it looks like Borg isn't finding a file it expects to find in any initialized Borg repository,
/mnt/repository/config
in this case.When you list
/mnt/repository
from within the container after, do you see aconfig
directory, adata
directory, etc? And do you see those same files show up outside of the container at$PWD/repository
?Note that I don't maintain the b3vis/borgmatic Docker image myself, so I'm making a guess here!
Hi!
Yeah you were right! That was the problem. Seems like I messed that one up as I made too many changes afterward.
Sorry I should've have probably add t he issue on to the b3vis/borgmatic.
If I am not wrong it doesnt allow using unencrypted repositories?
borgmatic | Starting a backup job.
borgmatic | Warning: Attempting to access a previously unknown unencrypted re pository!
borgmatic | Do you want to continue? [yN] Aborting.
borgmatic | Cache initialization aborted
borgmatic | Error while creating a backup.
borgmatic | /etc/borgmatic.d/config.yaml: Error running configuration file
borgmatic | Command '('borg', 'prune', '/mnt/repository/', '--keep-hourly', ' 2', '--keep-daily', '7', '--keep-weekly', '4', '--keep-monthly', '12', '--keep-y early', '10', '--prefix', 'backup-', '--stats')' returned non-zero exit status 2 .
borgmatic |
borgmatic | Need some help? https://torsion.org/borgmatic/#issues
PS. Seems like Gitea doesnt allow formating.
I'm not sure that using an unencrypted repository would be a problem, but it does look like it's giving you a warning and trying to prompt for input ("Do you want to continue?") So you could try running the
borg prune
command manually from within the container and see what happens if you answer "y" to the prompt:Maybe after doing that, the next run will work without prompting?
Alternatively, you could consider using an
authenticated
encryption mode instead ofnone
if you're using Borg 1.1+. More at https://borgbackup.readthedocs.io/en/stable/usage/init.html#encryption-modesHi!
Thanks for your persistence and help.
For now when I try
/mnt # borg prune repository/
I get
Warning: Attempting to access a previously unknown unencrypted repository!
Do you want to continue? [yN] y
At least one of the "keep-within", "keep-last", "keep-secondly", "keep-minutely", "keep-hourly", "keep-daily", "keep-weekly", "keep-monthly" or "keep-yearly" settings must be specified.
So I guess its need to be specified on some way? I thought it was meant to read from the config.yaml
If you run the
borg
command by itself, it doesn't know anything about borgmatic's config file, so it won't use any of the settings from it. So you can try the wholeborg prune
command above, which effectively provides those "keep" settings to Borg directly on the command-line.What are the possible problems when using authenticated version instead of none? Not encrypted, but authenticated and based on Hash/MAC. Does it mean that it needs the same computer MAC to unencrypt it? So the problems will come when I try to open it on my another laptop.
I don't think it has anything to do with a computer hardware MAC. I think it's referring to a different concept: A cryptographic Message Authentication Code. My understanding is that it allows Borg to validate that each piece of your data is correctly stored and hasn't become corrupted. I can't think of any problems/downsides of that, except perhaps a small performance hit versus the "none" encryption mode.
Had the same problem while using the b3vis/docker-borgmatic docker container.
The config directory at
/root/.config/borg
is not persisted as docker volume. Even when using the none encryption option borg is still storing some information in this directory.Just add a bind mount for
/root/.config/borg
to your compose file.But I am also having the first issue where borgmatic is loocking at a config file..
While creating the borg repository inside the container:
Borgmatic config:
docker-compose:
If you
ls /srv/web/backup/borg_repo
on the host, do you see aconfig
directory in there? And what about if youexec
into the backup container and runls /mnt/borg-repository
?I think the problem here is that the code that tries to probe for an existing repository works in most cases, but may not work across a Docker mount. FYI the code is in
borgmatic/borg/init.py:initialize_repository()
.Also, is borgmatic actually erroring upon repository creation, or is it just spewing the Borg traceback to the console and then continuing along successfully?
The folder is empty when checking inside and outside the container.
The error is only raised while borg repo initialization.
Unfortunately I am not so much experienced with python to debug this problem.
Borgmatic is finishing the repository creation successfully. The error is just from the Borg traceback.
Great. I've got a local repro here, and a potential solution.
Okay, this is now fixed in master. It'll be part of the next release. Thanks for the bug report!
Reopening this, because the fix broke some tests, and I had to back it out.
Well, I haven't found a good way to probe for Borg repository existence in an empty Docker volume mount directory without resulting in that spurious Borg traceback. So I kicked the issue upstream to the Borg project itself. We'll see what comes of it!
An update on the upstream Borg issue: There's now an open pull request with a fix. Once it lands (assuming it does), I'll close this borgmatic issue.
witten/borgmatic#201 (comment)
Hi!
This directory is mapped out of the container;
.env.template
docker-compose.yml
Okay, the Borg ticket https://github.com/borgbackup/borg/issues/4764 has been fixed and will be part of the next stable Borg release, so I'm closing this ticket! Thanks again for reporting it.