Silent REGRESSION in 1.8.10: backing up either files or database #903
Loading…
x
Reference in New Issue
Block a user
No description provided.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What I'm trying to do and why
A Borgmatic container version (
borgmatic >1.8.5 <= 1.8.11
) stopped backing up files while no configuration file was changed (i.e. the container update from 1.8.5 to 1.8.11 alone has this effect).This only concerns Borgmatic configurations which do include
mysql_databases:
orsqlite_databases:
(and potentially all/other database hooks, but not tested).When commenting that block out, files do show up with
--list
& do back up. Using--dry-dun
makes files show up as well.Steps to reproduce
Actual behavior
Notice that these folders are present but empty:
Expected behavior
Run again, after having commented out the
mysql_databases:
block inconfig.yml
.Notice that files are being backed up, but not database:
Other notes / implementation ideas
Temporary solution for me: revert back to
image: b3vis/borgmatic:1.8.5
.borgmatic version
1.8.11, 1.8.12, 1.8.13
borgmatic installation method
image: b3vis/borgmatic
Borg version
borg 1.2.8
Python version
Python 3.12.3
Database version (if applicable)
mysql Ver 15.1 Distrib 10.11.6-MariaDB, for Linux (x86_64) using readline 5.1
Operating system and version
Alpine Linux v3.19
Regression after 1.8.5: backing up either files or databaseto Silent regression after 1.8.5: backing up either files or databaseSilent regression after 1.8.5: backing up either files or databaseto Silent REGRESSION after 1.8.5: backing up either files or databaseSo I have an idea about why you're seeing this behavior (files not getting backed up when databases are enabled), but I have no idea how this previously ever worked for you with older versions of borgmatic.
When databases are enabled, borgmatic implicitly enables the
one_file_system
configuration option. This tells Borg not to cross filesystem boundaries, which has the side effect of preventing Borg from reading the contents of your source directories. You can confirm or disprove this theory by commenting out the database configuration and then addingone_file_system: true
to your configuration. If that also causes your source files to get omitted, then you have your answer.The workaround for this limitation is to be explicit about the directory paths to include instead of expecting Borg to cross those filesystem boundaries. Example:
The real mystery here is how this ever worked with borgmatic 1.8.5. My wild guess is something must have changed with the container construction that impacted how filesystem paths got mounted. And if you'd like to get to the bottom of that, my recommendation would be to ask the developers of that container image. EDIT: Although see the comment below...
I hope that helps!
Okay, so one thing that did change in borgmatic 1.8.6 is that the SQLite hook switched from dumping to an intermediate file on the filesystem to instead streaming dumps directly to Borg like all the other database hooks do (for performance / disk space reasons). This also means that as of borgmatic 1.8.6, configurations with SQLite started implicitly enabling
one_file_system
like the other database hooks. And that could in theory explain the behavior you're seeing with Borg not crossing filesystem boundaries to grab source files.Major aside, mostly for myself: I'll also note the idea (as of yet unexplored) that borgmatic could maybe stop implicitly enabling
one_file_system
in database hooks. IIRC, the original reason for that was to prevent the (also implicitly enabled)read_special
option from trying to read all sorts of special files on separate filesystems and thereby causing Borg to hang. But somewhere along the way, borgmatic got the ability to detect and automatically omit special files from the source directories even whenread_special
is enabled. So it's possible that implicitly enablingone_file_system
isn't necessary anymore to prevent hangs.Thanks for your quick reply!
I had indeed seen that that remark, but figured the same way as you that it then should not have worked in 1.8.5 (and numerous versions before) either.
I do confirm that no database configured +
one_file_system: true
prevents files from being included in the backup.That does indeed work as well.
It fails with
mysql_databases:
as well though.I thought I'd test all versions to narrow down, all with the config shown in first post, only change is
image: b3vis/borgmatic:1.8.5
:manifest for b3vis/borgmatic:1.8.6 not found: manifest unknown: manifest unknown
(not on Docker Hub anymore)If the container versions are perfect mirrors of releases of this repo, then it must be somewhere in 1.8.10's commits.
If not, it could be in borgmatic-collective/docker-borgmatic as you mentioned, which includes the bump from
borgbackup==1.2.7
toborgbackup==1.2.8
, but at that point I'm lost in code without a clue.It seems that the "streaming" part you mentioned is actually in 1.8.10?
Silent REGRESSION after 1.8.5: backing up either files or databaseto Silent REGRESSION in 1.8.10: backing up either files or databaseAs far as I recall, the borgmatic MySQL hook has always enabled
one_file_system
implicitly.Interesting! Thanks for taking the time to narrow this down. But I don't immediately see anything in borgmatic's own 1.8.10 changelog that would readily explain the behavior you're seeing. It's possible that some of the refactoring that was part of the
spot
check (in 1.8.10) could've impacted something, but that's just speculation at this point.Actually, no, that was just making the existing streaming status (enabled/disabled) available to other parts of borgmatic.
The container versions AFAIK do use unmodified versions of borgmatic, but they also layer additional software on top of it (e.g. for scheduling, environment variables, etc.). I don't know how inconvenient it would be for you, but have you tried installing borgmatic temporarily without a container to see if you get the same behavior difference between 1.8.9 and 1.8.10? That could at least rule out the container as a factor (or rule it in).
Running borgmatic on host and still accessing data in volumes without a container, yet not modifying the config to accurately test versions, seemed too complicated, if not impossible.
As such, I rather kept
and updated borgmatic inside the container:
Is that okay?
Result: does work (database and files in backup).
The bug or regression must be something in the container, I sent the issue downstream: https://github.com/borgmatic-collective/docker-borgmatic/issues/355
I think updating borgmatic inside the container with pip should be sufficient for temporary testing like this. You should be able to confirm that the new version of borgmatic is actually getting used by changing your
crontab.txt
to runborgmatic --version
instead of justborgmatic
. Newer versions of the container might print the borgmatic version in use, but that doesn't help you here.Anyway, I'll close this ticket for now in favor of the docker-borgmatic ticket you've filed. Thanks for your patience and persistence here.
This is very likely fixed by borgmatic 1.9.0 via #918.