Archive gets created despite failed database hook #1032

New issue

Closed

opened 2025-03-18 14:27:52 +00:00 by gzm0 · 29 comments

gzm0 commented 2025-03-18 14:27:52 +00:00

Contributor

Copy link

What I'm trying to do and why

When a database hook fails (see reproduction for full example), the borgmatic create command fails (as expected). However, an archive gets created nevertheless (unexpected).

As a result, there is a correct looking archive with incomplete data.

IMHO, this is problematic for at least the following reasons:

• Pruning might now delete a perfectly fine archive in favor of a broken one like this.
• It this violates the principle of least surprise.

Steps to reproduce

location:
  repositories: [/root/backup-test]
storage:
  encryption_passphrase: the-secret

hooks:
  postgresql_databases:
    - name: test
      hostname: test
      pg_dump_command: "false" # fail for test

root@408bff8b9fcf:~# mkdir backup-test
root@408bff8b9fcf:~# borgmatic init -e keyfile
root@408bff8b9fcf:~# borgmatic create --verbosity 2
// fails as expected, logs below

borgmatic create log

Ensuring legacy configuration is upgraded
/etc/borgmatic/config.yaml: No commands to run for pre-everything hook
borg --version --debug --show-rc
/etc/borgmatic/config.yaml: No commands to run for pre-actions hook
/etc/borgmatic/config.yaml: No commands to run for pre-backup hook
/root/backup-test: Creating archive
/root/backup-test: Calling postgresql_databases hook function remove_database_dumps
/root/backup-test: Removing PostgreSQL database dumps
/root/backup-test: Calling mysql_databases hook function remove_database_dumps
/root/backup-test: Removing MySQL database dumps
/root/backup-test: Calling mongodb_databases hook function remove_database_dumps
/root/backup-test: Removing MongoDB database dumps
/root/backup-test: Calling postgresql_databases hook function dump_databases
/root/backup-test: Dumping PostgreSQL databases
/root/backup-test: Dumping PostgreSQL database "test" to /root/.borgmatic/postgresql_databases/test/test
false --no-password --clean --if-exists --host test --format custom test > /root/.borgmatic/postgresql_databases/test/test
/root/backup-test: Collecting special file paths
borg create --one-file-system --read-special /root/backup-test::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /root/.borgmatic --dry-run --list
borg create --one-file-system --read-special /root/backup-test::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /root/.borgmatic --debug --show-rc
using builtin fallback logging configuration
33 self tests completed in 0.05 seconds
Verified integrity of /root/backup-test/index.1
TAM-verified manifest
security: read previous location '/root/backup-test'
security: read manifest timestamp '2025-03-18T14:14:38.995866'
security: determined newest manifest timestamp as 2025-03-18T14:14:38.995866
security: repository checks ok, allowing access
Creating archive at "/root/backup-test::408bff8b9fcf-2025-03-18T14:15:21.073212"
Verified integrity of /root/.cache/borg/d400d0af2f0630a08f5eb7fc1d8e8b455530da10b02d00460493b136820058de/chunks
Reading files cache ...
security: read previous location '/root/backup-test'
security: read manifest timestamp '2025-03-18T14:14:38.995866'
security: determined newest manifest timestamp as 2025-03-18T14:14:38.995866
security: repository checks ok, allowing access
Processing files ...
Cleaned up 0 uncommitted segment files (== everything after segment 1).
Verified integrity of /root/backup-test/hints.1
check_free_space: few segments, not requiring a full free segment
check_free_space: calculated working space for compact as 24 bytes
check_free_space: required bytes 169232, free bytes 32613781504
security: saving state for d400d0af2f0630a08f5eb7fc1d8e8b455530da10b02d00460493b136820058de to /root/.config/borg/security/d400d0af2f0630a08f5eb7fc1d8e8b455530da10b02d00460493b136820058de
security: current location   /root/backup-test
security: key type           0
security: manifest timestamp 2025-03-18T14:15:21.155406
terminating with success status, rc 0
/root/backup-test: Error running actions for repository
Command 'false --no-password --clean --if-exists --host test --format custom test > /root/.borgmatic/postgresql_databases/test/test' returned non-zero exit status 1.
/etc/borgmatic/config.yaml: No commands to run for on-error hook
/etc/borgmatic/config.yaml: Error running configuration file
/etc/borgmatic/config.yaml: No commands to run for post-everything hook

summary:
/etc/borgmatic/config.yaml: Error running configuration file
/root/backup-test: Error running actions for repository
Command 'false --no-password --clean --if-exists --host test --format custom test > /root/.borgmatic/postgresql_databases/test/test' returned non-zero exit status 1.

Need some help? https://torsion.org/borgmatic/#issues

Actual behavior

root@408bff8b9fcf:~# borgmatic list 
/root/backup-test: Listing archives
408bff8b9fcf-2025-03-18T14:15:21.073212 Tue, 2025-03-18 14:15:21 [ad611ac6569a69cff908b6a019020a0cf614601f717edc361b8c5db7713af9db]

Expected behavior

root@408bff8b9fcf:~# borgmatic list 
/root/backup-test: Listing archives

Other notes / implementation ideas

I will keep investigating if I can find a fix/workaround for this.

borgmatic version

1.7.7 and 1.9.14

borgmatic installation method

Debian package on bookworm for 1.7.7 / pip install borgmatic for 1.9.14

Borg version

borg 1.2.4

Python version

Python 3.11.2

Database version (if applicable)

No response

Operating system and version

Debian 12 (bookworm)

### What I'm trying to do and why When a database hook fails (see reproduction for full example), the `borgmatic create` command fails (as expected). However, an archive gets created nevertheless (unexpected). As a result, there is a correct looking archive with incomplete data. IMHO, this is problematic for at least the following reasons: - Pruning might now delete a perfectly fine archive in favor of a broken one like this. - It this violates the principle of least surprise. ### Steps to reproduce ```yaml location: repositories: [/root/backup-test] storage: encryption_passphrase: the-secret hooks: postgresql_databases: - name: test hostname: test pg_dump_command: "false" # fail for test ``` ``` root@408bff8b9fcf:~# mkdir backup-test root@408bff8b9fcf:~# borgmatic init -e keyfile root@408bff8b9fcf:~# borgmatic create --verbosity 2 // fails as expected, logs below ``` <details> <summary>borgmatic create log</summary> ``` Ensuring legacy configuration is upgraded /etc/borgmatic/config.yaml: No commands to run for pre-everything hook borg --version --debug --show-rc /etc/borgmatic/config.yaml: No commands to run for pre-actions hook /etc/borgmatic/config.yaml: No commands to run for pre-backup hook /root/backup-test: Creating archive /root/backup-test: Calling postgresql_databases hook function remove_database_dumps /root/backup-test: Removing PostgreSQL database dumps /root/backup-test: Calling mysql_databases hook function remove_database_dumps /root/backup-test: Removing MySQL database dumps /root/backup-test: Calling mongodb_databases hook function remove_database_dumps /root/backup-test: Removing MongoDB database dumps /root/backup-test: Calling postgresql_databases hook function dump_databases /root/backup-test: Dumping PostgreSQL databases /root/backup-test: Dumping PostgreSQL database "test" to /root/.borgmatic/postgresql_databases/test/test false --no-password --clean --if-exists --host test --format custom test > /root/.borgmatic/postgresql_databases/test/test /root/backup-test: Collecting special file paths borg create --one-file-system --read-special /root/backup-test::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /root/.borgmatic --dry-run --list borg create --one-file-system --read-special /root/backup-test::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /root/.borgmatic --debug --show-rc using builtin fallback logging configuration 33 self tests completed in 0.05 seconds Verified integrity of /root/backup-test/index.1 TAM-verified manifest security: read previous location '/root/backup-test' security: read manifest timestamp '2025-03-18T14:14:38.995866' security: determined newest manifest timestamp as 2025-03-18T14:14:38.995866 security: repository checks ok, allowing access Creating archive at "/root/backup-test::408bff8b9fcf-2025-03-18T14:15:21.073212" Verified integrity of /root/.cache/borg/d400d0af2f0630a08f5eb7fc1d8e8b455530da10b02d00460493b136820058de/chunks Reading files cache ... security: read previous location '/root/backup-test' security: read manifest timestamp '2025-03-18T14:14:38.995866' security: determined newest manifest timestamp as 2025-03-18T14:14:38.995866 security: repository checks ok, allowing access Processing files ... Cleaned up 0 uncommitted segment files (== everything after segment 1). Verified integrity of /root/backup-test/hints.1 check_free_space: few segments, not requiring a full free segment check_free_space: calculated working space for compact as 24 bytes check_free_space: required bytes 169232, free bytes 32613781504 security: saving state for d400d0af2f0630a08f5eb7fc1d8e8b455530da10b02d00460493b136820058de to /root/.config/borg/security/d400d0af2f0630a08f5eb7fc1d8e8b455530da10b02d00460493b136820058de security: current location /root/backup-test security: key type 0 security: manifest timestamp 2025-03-18T14:15:21.155406 terminating with success status, rc 0 /root/backup-test: Error running actions for repository Command 'false --no-password --clean --if-exists --host test --format custom test > /root/.borgmatic/postgresql_databases/test/test' returned non-zero exit status 1. /etc/borgmatic/config.yaml: No commands to run for on-error hook /etc/borgmatic/config.yaml: Error running configuration file /etc/borgmatic/config.yaml: No commands to run for post-everything hook summary: /etc/borgmatic/config.yaml: Error running configuration file /root/backup-test: Error running actions for repository Command 'false --no-password --clean --if-exists --host test --format custom test > /root/.borgmatic/postgresql_databases/test/test' returned non-zero exit status 1. Need some help? https://torsion.org/borgmatic/#issues ``` </details> ### Actual behavior ``` root@408bff8b9fcf:~# borgmatic list /root/backup-test: Listing archives 408bff8b9fcf-2025-03-18T14:15:21.073212 Tue, 2025-03-18 14:15:21 [ad611ac6569a69cff908b6a019020a0cf614601f717edc361b8c5db7713af9db] ``` ### Expected behavior ``` root@408bff8b9fcf:~# borgmatic list /root/backup-test: Listing archives ``` ### Other notes / implementation ideas I will keep investigating if I can find a fix/workaround for this. ### borgmatic version 1.7.7 and 1.9.14 ### borgmatic installation method Debian package on bookworm for 1.7.7 / pip install borgmatic for 1.9.14 ### Borg version borg 1.2.4 ### Python version Python 3.11.2 ### Database version (if applicable) _No response_ ### Operating system and version Debian 12 (bookworm)

gzm0 commented 2025-03-18 14:43:55 +00:00

Author

Contributor

Copy link

I have looked at the current implementation and existing borg options, and I'm afraid it looks like this is not fixable without additional support from borg.

Note that there is the --content-from-command which pretty much fulfills this need:
https://borgbackup.readthedocs.io/en/stable/usage/create.html#reading-backup-data-from-stdin

Unfortunately, it only supports a single command.

I have looked at the current implementation and existing borg options, and I'm afraid it looks like this is not fixable without additional support from borg. Note that there is the `--content-from-command` which pretty much fulfills this need: https://borgbackup.readthedocs.io/en/stable/usage/create.html#reading-backup-data-from-stdin Unfortunately, it only supports a single command.

witten commented 2025-03-18 17:39:41 +00:00

Owner

Copy link

Thanks for filing this! I swear this limitation was discussed in a previous ticket, but I can't seem to find it. If I recall correctly though, the consensus was basically what you found—there's not really a good solution right now given: 1. Borg's lack of support for anything like this, and: 2. borgmatic's current approach of streaming data sources to Borg via multiple named pipes. And yeah, --content-from-command won't work for the general case because of that borgmatic support for shoving multiple data sources into a single archive.

Just brainstorming though, here are a few different ideas:

• borgmatic could maybe delete the Borg archive on database error so as to avoid some of the downsides you've called out. I don't feel great about this though, because: 1. It might be difficult to determine when to delete (on a database error) vs. when not to delete (on another error), and 2. A partial archive is useful in some ways to serve as a checkpoint archive so all the data doesn't have to be re-uploaded. I'm not saying it's great to leave it there, but it's also not great to auto-delete it.
• Instead of auto-deleting, maybe borgmatic could convert the archive to a Borg "checkpoint" archive so its data still exist but the archive isn't considered a real archive for purposes of pruning or anything else.
• Most database errors occur at the very start of the database dump running (e.g. an authentication error). If there were some way to detect that before running Borg, we could avoid creating the archive. The difficulty though is that I'm not sure this is possible with the named pipe setup, because the database process might wait to run until another process (in this case Borg) actually consumes from the named pipe.

Thanks for filing this! I swear this limitation was discussed in a previous ticket, but I can't seem to find it. If I recall correctly though, the consensus was basically what you found—there's not really a good solution right now given: 1. Borg's lack of support for anything like this, and: 2. borgmatic's current approach of streaming data sources to Borg via multiple named pipes. And yeah, `--content-from-command` won't work for the general case because of that borgmatic support for shoving multiple data sources into a single archive. Just brainstorming though, here are a few different ideas: - borgmatic could maybe delete the Borg archive on database error so as to avoid some of the downsides you've called out. I don't feel great about this though, because: 1. It might be difficult to determine when to delete (on a database error) vs. when not to delete (on another error), and 2. A partial archive *is* useful in some ways to serve as a checkpoint archive so all the data doesn't have to be re-uploaded. I'm not saying it's great to leave it there, but it's also not great to auto-delete it. - Instead of auto-deleting, maybe borgmatic could convert the archive to a Borg ["checkpoint" archive](https://borgbackup.readthedocs.io/en/stable/usage/create.html#description) so its data still exist but the archive isn't considered a real archive for purposes of pruning or anything else. - Most database errors occur at the very start of the database dump running (e.g. an authentication error). If there were some way to detect that before running Borg, we could avoid creating the archive. The difficulty though is that I'm not sure this is possible with the named pipe setup, because the database process might wait to run until another process (in this case Borg) actually consumes from the named pipe.

witten commented 2025-03-18 18:08:28 +00:00

Owner

Copy link

Another idea:

• Maybe borgmatic could forcibly kill the Borg process in the case of database error, so that only a checkpoint archive gets created. I'd actually already expect this to happen, now that I think about it, so maybe there's a bug in borgmatic that prevents this. Or maybe it's just a race between borgmatic killing the Borg process and Borg exiting "successfully" on its own. EDIT: Ah, yeah, as suspected, it looks like Borg has already exited "successfully" at the point we might be ready to kill it.

EDIT: Yet another idea:

• Okay, this one may be a little out there. borgmatic could create a separate sentinel named pipe, named such that Borg only reaches it after all other database named pipes have been successfully consumed. Until that point, borgmatic leaves this pipe open but doesn't write anything to it so that Borg hangs if it reaches it. This prevents Borg from exiting "successfully" before we have a chance to kill it. Then, if a database error does occur, we forcibly kill Borg, which is just sitting around waiting for that sentinel named pipe to fill. But if all databases execute successfully, we close the named pipe and allow Borg to continue running (and exit successfully). This would results in a zero-byte file getting written into the archive, but that's a small price to pay for hopefully solving the ask in this ticket.

Another idea: - Maybe borgmatic could forcibly kill the Borg process in the case of database error, so that only a checkpoint archive gets created. I'd actually already expect this to happen, now that I think about it, so maybe there's a bug in borgmatic that prevents this. Or maybe it's just a race between borgmatic killing the Borg process and Borg exiting "successfully" on its own. EDIT: Ah, yeah, as suspected, it looks like Borg has already exited "successfully" at the point we might be ready to kill it. EDIT: Yet another idea: - Okay, this one may be a little out there. borgmatic could create a separate sentinel named pipe, named such that Borg only reaches it after all other database named pipes have been successfully consumed. Until that point, borgmatic leaves this pipe open but doesn't write anything to it so that Borg hangs if it reaches it. This prevents Borg from exiting "successfully" before we have a chance to kill it. Then, if a database error does occur, we forcibly kill Borg, which is just sitting around waiting for that sentinel named pipe to fill. But if all databases execute successfully, we close the named pipe and allow Borg to continue running (and exit successfully). This would results in a zero-byte file getting written into the archive, but that's a small price to pay for hopefully solving the ask in this ticket.

gzm0 commented 2025-03-19 08:03:46 +00:00

Author

Contributor

Copy link

Heya, thanks a lot for the quick and comprehensive reply.

My thoughts:

keeping a partial archive instead of deleting

100% agree with you there. a partial archive is better than no archive, as long as it is clear it is partial.

deleting / adjusting / ... the archive on database error

I would caution against "going back and fixing things", as it is not crash safe (arguably, it's a tiny fraction of time, but it can still happen that a full archive gets created when it shouldn't and the removal / conversion to checkpoint fails).

Instead, have you considered creating a checkpoint(-like) archive first and renaming it on success? It feels this could be quite easily added:

1. borg create <archive-name>.checkpoint (borg 1.2.4 let's you do this)
2. Check that all db jobs have completed successfully.
3. borg rename <archive-name>.checkpoint <archive-name> (borg 1.2.4 let's you do this)

IIUC this would pretty much give us the behavior we'd want:

• If any db job fails, we do not get a full archive.
• We still have a partial archive in case a db job fails.

WDYT? (we should probably talk to the borg maintainer's to make sure they are onboard with borgmatic using this behavior).

killing the borg process with & pipe

I have been toying with this idea as well; I have even considered building it in user space. If necessary, it feels to me this would be better than having this bug.

IMHO the additional zero byte file is absolutely acceptable. However, I'm a bit concerned that the additional pipe magic will add quite some complexity to borgmatic itself (you're the better judge of this of course).

FWIW: I think the "create a checkpoint" option is really neat. Let me know what you think about it.

Heya, thanks a lot for the quick and comprehensive reply. My thoughts: **keeping a partial archive instead of deleting** 100% agree with you there. a partial archive is better than no archive, as long as it is clear it is partial. **deleting / adjusting / ... the archive on database error** I would caution against "going back and fixing things", as it is not crash safe (arguably, it's a tiny fraction of time, but it can still happen that a full archive gets created when it shouldn't and the removal / conversion to checkpoint fails). Instead, have you considered creating a checkpoint(-like) archive first and renaming it on success? It feels this could be quite easily added: 1. `borg create <archive-name>.checkpoint` (borg 1.2.4 let's you do this) 2. Check that all db jobs have completed successfully. 3. `borg rename <archive-name>.checkpoint <archive-name>` (borg 1.2.4 let's you do this) IIUC this would pretty much give us the behavior we'd want: - If any db job fails, we do not get a full archive. - We still have a partial archive in case a db job fails. WDYT? (we should probably talk to the borg maintainer's to make sure they are onboard with borgmatic using this behavior). **killing the borg process with & pipe** I have been toying with this idea as well; I have even considered building it in user space. If necessary, it feels to me this would be better than having this bug. IMHO the additional zero byte file is absolutely acceptable. However, I'm a bit concerned that the additional pipe magic will add quite some complexity to borgmatic itself (you're the better judge of this of course). FWIW: I think the "create a checkpoint" option is really neat. Let me know what you think about it.

witten commented 2025-03-19 17:38:10 +00:00

Owner

Copy link

I like your "start with a checkpoint" idea as well, and I agree that the sentinel pipe approach feels really brittle. I've asked the Borg dev about the checkpoint approach on IRC; we'll see what he says. The only downside I can see is that we'd lose Borg 1.0.x compatibility, but that's a small price to pay.

I like your "start with a checkpoint" idea as well, and I agree that the sentinel pipe approach feels really brittle. I've asked the Borg dev about the checkpoint approach on IRC; we'll see what he says. The only downside I can see is that we'd lose Borg 1.0.x compatibility, but that's a small price to pay.

witten commented 2025-03-20 02:26:18 +00:00

Owner

Copy link

witten: try it, not sure if borg allows *.checkpoint as archive name.
also, what will happen if borg create then creates checkpoints after a while of running?
so guess you'll rather need a normal archive name and then rename to .checkpoint after borg finished AND you know it was not successful.

> witten: try it, not sure if borg allows *.checkpoint as archive name. > also, what will happen if borg create then creates checkpoints after a while of running? > so guess you'll rather need a normal archive name and then rename to .checkpoint after borg finished AND you know it was not successful.

gzm0 commented 2025-03-24 16:12:35 +00:00

Author

Contributor

Copy link

I have looked at borg code for this, and from code inspection alone, it seems like borg would write checkpoints for foo.checkpoint to foo.checkpoint.checkpoint:

github.com/ThomasWaldmann/borg@839434891c/src/borg/archive.py (L466)

I will try and see if I can verify this experimentally.

The upshot of using this hack is that borg would identify the "pipe-checkpoint" archive as checkpoint as well (for listing and more importantly pruning):

github.com/ThomasWaldmann/borg@839434891c/src/borg/archiver.py (L1545-L1551)

Otherwise we need another way to make sure these archives get pruned and hidden from lists (or marked differently in lists).

I have looked at borg code for this, and from code inspection alone, it seems like borg would write checkpoints for `foo.checkpoint` to `foo.checkpoint.checkpoint`: https://github.com/ThomasWaldmann/borg/blob/839434891c790fa2104d052251039665d5a047cd/src/borg/archive.py#L466 I will try and see if I can verify this experimentally. The upshot of using this hack is that borg would identify the "pipe-checkpoint" archive as checkpoint as well (for listing and more importantly pruning): https://github.com/ThomasWaldmann/borg/blob/839434891c790fa2104d052251039665d5a047cd/src/borg/archiver.py#L1545-L1551 Otherwise we need another way to make sure these archives get pruned and hidden from lists (or marked differently in lists).

witten commented 2025-03-24 20:04:02 +00:00

Owner

Copy link

Thanks for looking into that. Yeah, foo.checkpoint.checkpoint would probably be fine as long as Borg still considers it a checkpoint archive. And based on that regular expression you linked, it looks like it would.

Thanks for looking into that. Yeah, `foo.checkpoint.checkpoint` would probably be fine as long as Borg still considers it a checkpoint archive. And based on that regular expression you linked, it looks like it would.

gzm0 commented 2025-04-09 14:57:35 +00:00

Author

Contributor

Copy link

I have been trying to reproduce this, but I cannot get borg to actually write checkpoints (at all).

I've tried the following (on Debian so borgmatic 1.7.7 / borg 1.2.4).

Config:

location:
  repositories: [/root/backup-test]
  source_directories: [/root/backup-input]
storage:
  encryption_passphrase: the-secret
  checkpoint_interval: 1

hooks:
  postgresql_databases:
    - name: test
      hostname: test
      pg_dump_command: "sleep 5 && false"

root@3b358ebe3b7a:~# mkdir backup-test
root@3b358ebe3b7a:~# mkdir backup-input
root@3b358ebe3b7a:~# truncate -s 100 backup-input/file
root@3b358ebe3b7a:~# borgmatic init -e keyfile
root@3b358ebe3b7a:~# borgmatic create --verbosity 2
// ctrl-c the process after a couple of seconds.
root@3b358ebe3b7a:~# borg break-lock /root/backup-test
root@3b358ebe3b7a:~# borg list  --consider-checkpoints /root/backup-test
Enter passphrase for key /root/.config/borg/keys/root_backup_test.2: 

I would have expected a checkpoint to be there, but there isn't.

My plan was to then set archive_name_format: '{hostname}-test-{now}.checkpoint' and see what happens, but I didn't even get there.

Any help of how to actually get a checkpoint (w/o running a backup for a long time) would be very much appreciated.

I have been trying to reproduce this, but I cannot get borg to actually write checkpoints (at all). I've tried the following (on Debian so borgmatic 1.7.7 / borg 1.2.4). Config: ```yaml location: repositories: [/root/backup-test] source_directories: [/root/backup-input] storage: encryption_passphrase: the-secret checkpoint_interval: 1 hooks: postgresql_databases: - name: test hostname: test pg_dump_command: "sleep 5 && false" ``` ``` root@3b358ebe3b7a:~# mkdir backup-test root@3b358ebe3b7a:~# mkdir backup-input root@3b358ebe3b7a:~# truncate -s 100 backup-input/file root@3b358ebe3b7a:~# borgmatic init -e keyfile root@3b358ebe3b7a:~# borgmatic create --verbosity 2 // ctrl-c the process after a couple of seconds. root@3b358ebe3b7a:~# borg break-lock /root/backup-test root@3b358ebe3b7a:~# borg list --consider-checkpoints /root/backup-test Enter passphrase for key /root/.config/borg/keys/root_backup_test.2: ``` I would have expected a checkpoint to be there, but there isn't. My plan was to then set `archive_name_format: '{hostname}-test-{now}.checkpoint'` and see what happens, but I didn't even get there. Any help of how to actually get a checkpoint (w/o running a backup for a long time) would be very much appreciated.

witten commented 2025-04-09 16:42:32 +00:00

Owner

Copy link

Set the checkpoint interval to a low value! Checkpoints are every 30 minutes by default.

# Number of seconds between each checkpoint during a long-running
# backup. See https://borgbackup.readthedocs.io/en/stable/faq.html for
# details. Defaults to checkpoints every 1800 seconds (30 minutes).
# checkpoint_interval: 1800

Set the checkpoint interval to a low value! Checkpoints are every 30 minutes by default. ``` # Number of seconds between each checkpoint during a long-running # backup. See https://borgbackup.readthedocs.io/en/stable/faq.html for # details. Defaults to checkpoints every 1800 seconds (30 minutes). # checkpoint_interval: 1800 ```

gzm0 commented 2025-04-09 19:35:28 +00:00

Author

Contributor

Copy link

I have tried setting it to 1, but there was no checkpoint created (after 2-3 secs). Do you know if there are conditions (like non zero amount of data? In my experiment I've only used a sparse file).

I have tried setting it to 1, but there was no checkpoint created (after 2-3 secs). Do you know if there are conditions (like non zero amount of data? In my experiment I've only used a sparse file).

witten commented 2025-04-09 20:22:28 +00:00

Owner

Copy link

Oh, I didn't see that you were already using checkpoint_interval! I don't know the answer about whether there are other conditions by default, but related to that, another thing you can try is checkpoint_volume—if you (temporarily) switch to Borg 2:

# Number of backed up bytes between each checkpoint during a
# long-running backup. Only supported with Borg 2+. See
# https://borgbackup.readthedocs.io/en/stable/faq.html for details.
# Defaults to only time-based checkpointing (see
# "checkpoint_interval") instead of volume-based checkpointing.
# checkpoint_volume: 1048576

Oh, I didn't see that you were already using `checkpoint_interval`! I don't know the answer about whether there are other conditions by default, but related to that, another thing you can try is `checkpoint_volume`—if you (temporarily) switch to Borg 2: ``` # Number of backed up bytes between each checkpoint during a # long-running backup. Only supported with Borg 2+. See # https://borgbackup.readthedocs.io/en/stable/faq.html for details. # Defaults to only time-based checkpointing (see # "checkpoint_interval") instead of volume-based checkpointing. # checkpoint_volume: 1048576 ```

gzm0 commented 2025-04-10 07:36:53 +00:00

Author

Contributor

Copy link

Ok, I managed to get borg to checkpoint. Two adjustments I did:

• Put real content in the file dd if=/dev/random of=file bs=512k count=10
• Change pg_dump_command: "/root/drip.sh"

If we set archive_name_format: '{hostname}-test-{now}.checkpoint', we'll get .checkpoint.checkpoint:

root@3b358ebe3b7a:~#  borg list  --consider-checkpoints /root/backup-test
Enter passphrase for key /root/.config/borg/keys/root_backup_test.3: 
3b358ebe3b7a-test-2025-04-10T07:32:40.checkpoint.checkpoint Thu, 2025-04-10 07:32:40 [884c42b7a711c02999670a04df27a51883edde5a70fd03bda468abbb1fb6d38e]

drip.sh

#! /bin/sh

while true; do
    sleep 1
    echo "a"
done

This means we're on a good path here. My next steps are to test with different versions of borgmatic / borg. Notably newest borg 1.x and borg 2.x.

Ok, I managed to get borg to checkpoint. Two adjustments I did: - Put real content in the file `dd if=/dev/random of=file bs=512k count=10` - Change `pg_dump_command: "/root/drip.sh"` If we set `archive_name_format: '{hostname}-test-{now}.checkpoint'`, we'll get `.checkpoint.checkpoint`: ``` root@3b358ebe3b7a:~# borg list --consider-checkpoints /root/backup-test Enter passphrase for key /root/.config/borg/keys/root_backup_test.3: 3b358ebe3b7a-test-2025-04-10T07:32:40.checkpoint.checkpoint Thu, 2025-04-10 07:32:40 [884c42b7a711c02999670a04df27a51883edde5a70fd03bda468abbb1fb6d38e] ``` <details> <summary>drip.sh</summary> ```sh #! /bin/sh while true; do sleep 1 echo "a" done ``` </details> This means we're on a good path here. My next steps are to test with different versions of borgmatic / borg. Notably newest borg 1.x and borg 2.x.

witten commented 2025-04-10 16:49:46 +00:00

Owner

Copy link

Nice work! Yeah, this is looking like a promising approach. If your investigations pan out, then I could see creating all borgmatic archives with a .checkpoint suffix and then renaming them to remove the suffix upon successful completion.

Nice work! Yeah, this is looking like a promising approach. If your investigations pan out, then I could see creating *all* borgmatic archives with a `.checkpoint` suffix and then renaming them to remove the suffix upon successful completion.

gzm0 commented 2025-04-11 10:03:42 +00:00

Author

Contributor

Copy link

I have spent some time analyzing this.

Overall findings:

• Borg 2.x does not write .checkpoint archives (anymore). It allows creating archives ending in .checkpoint.
• Borg up to 1.4.x seems to happily create .checkpoint.checkpoint archives (see "Test Results" and following for analysis).

Proposal:

• Change borgmatic as discussed:
  1. Write an archive ending in .checkpoint.
  2. If streaming dumps complete successfully, rename the archive to remove the .checkpoint suffix.
• Once borg 2.0 stabilizes, backport relevant pruning logic to borgmatic to
  ensure .checkpoints (now only from streaming) get collected correctly.

I'm happy to take a stab at the first bullet. @witten WDYT?

Test Results

Overall, getting borg to actually write checkpoints was the hardest here. I have stopped testing with later borgmatic versions, since it seemed that I'm mostly chasing changes in borgmatic.

✅ = .checkpoint.checkpoint archive created.
⚠️ = Unable to produce any checkpoint in my setup.

borgmatic	borg	test
1.7.7	1.2.4	✅
1.7.7	1.4.0	✅
1.8.13	1.2.4	✅️
1.8.13	1.4.0	✅️
1.9.0	1.2.4	✅️️
1.9.0	1.4.0	✅️️
1.9.1	1.2.4	✅️️️
1.9.1	1.4.0	✅️️️
1.9.2	1.4.0	✅️️️
1.9.3	1.4.0	✅️️️
1.9.5	1.2.4	✅️️️
1.9.5	1.4.0	✅️️️
1.9.6	1.2.4	⚠️
1.9.6	1.4.0	⚠️
1.9.7	1.2.4	⚠️
1.9.7	1.4.0	⚠️️️️
1.9.14	1.2.4	⚠️
2.0.2	1.2.4	⚠️

Test Setup

FROM debian

RUN apt-get update
RUN apt-get install -y emacs pipx wget

ADD --chmod=755 https://github.com/borgbackup/borg/releases/download/1.4.0/borg-linux-glibc231 /usr/local/bin/borg
#ADD --chmod=755 https://github.com/borgbackup/borg/releases/download/1.2.8/borg-linux64 /usr/local/bin/borg
#ADD --chmod=755 https://github.com/borgbackup/borg/releases/download/2.0.0b14/borg-linux-glibc236 /usr/local/bin/borg

Then run:

cd
pipx install borgmatic==1.9.6
export PATH="/root/.local/bin:$PATH"
mkdir /etc/borgmatic
cat > /etc/borgmatic/config.yaml <<EOF
repositories:
  - path: /root/backup-test
source_directories: [/root/backup-input]
encryption_passphrase: the-secret
archive_name_format: '{hostname}-test-{now}.checkpoint'
user_runtime_directory: /borgmatic
checkpoint_interval: 1
postgresql_databases:
  - name: test
    hostname: test
    pg_dump_command: "/root/drip.sh"
EOF
cat > drip.sh <<EOF
#! /bin/sh

while true; do
    sleep 1
    echo "a"
done
EOF
chmod +x drip.sh
mkdir backup-input
dd if=/dev/random of=/root/backup-input/file bs=512k count=10

Test Commands

Borg 1.x

rm -rf backup-test
borgmatic init -e keyfile
borgmatic create --verbosity 2
# kill with ctrl-c
borg break-lock /root/backup-test # necessary only in borgmatic 1.7.7
borg list  --consider-checkpoints /root/backup-test

Borg 2.x

rm -rf backup-test
borgmatic init -e none
borgmatic create --verbosity 2
# don't kill!
borg repo-list -r /root/backup-test

Test Configs

Borgmatic 1.7.x

location:
  repositories: [/root/backup-test]
  source_directories: [/root/backup-input]
storage:
  encryption_passphrase: the-secret
  checkpoint_interval: 1
  archive_name_format: '{hostname}-test-{now}.checkpoint'

hooks:
  postgresql_databases:
    - name: test
      hostname: test
      pg_dump_command: "/root/drip.sh"

Borgmatic > 1.8.x

repositories:
  - path: /root/backup-test
source_directories: [/root/backup-input]
encryption_passphrase: the-secret
archive_name_format: '{hostname}-test-{now}.checkpoint'
checkpoint_interval: 1
postgresql_databases:
  - name: test
    hostname: test
    pg_dump_command: "/root/drip.sh"

Borgmatic >= 1.9.2

+user_runtime_directory: /borgmatic

Borg 2.0.0b14:

-checkpoint_interval: 1
-    pg_dump_command: "/root/drip.sh"
+    pg_dump_command: "true" 

Error logs

Borg 2.0.0b14 / Borgmatic 1.7.7, 1.8.13

root@3b358ebe3b7a:~# borgmatic init -e keyfile
usage: borg [-V] [-h] [--critical] [--error] [--warning] [--info] [--debug]
            [--debug-topic TOPIC] [-p] [--iec] [--log-json]
            [--lock-wait SECONDS] [--show-version] [--show-rc] [--umask M]
            [--remote-path PATH] [--upload-ratelimit RATE]
            [--upload-buffer UPLOAD_BUFFER] [--debug-profile FILE] [--rsh RSH]
            [--socket [PATH]] [-r REPO]
            <command> ...
borg: error: argument <command>: invalid choice: 'rinfo' (choose from 'analyze', 'benchmark', 'check', 'compact', 'create', 'debug', 'delete', 'diff', 'extract', 'help', 'info', 'key', 'list', 'break-lock', 'with-lock', 'mount', 'umount', 'prune', 'repo-compress', 'repo-create', 'repo-delete', 'repo-info', 'repo-list', 'recreate', 'rename', 'repo-space', 'serve', 'tag', 'export-tar', 'import-tar', 'transfer', 'undelete', 'version')
usage: borg [-V] [-h] [--critical] [--error] [--warning] [--info] [--debug] [--debug-topic TOPIC] [-p] [--iec] [--log-json] [--lock-wait SECONDS] [--show-version] [--show-rc] [--umask M]
            [--remote-path PATH] [--upload-ratelimit RATE] [--upload-buffer UPLOAD_BUFFER] [--debug-profile FILE] [--rsh RSH] [--socket [PATH]] [-r REPO]
            <command> ...
borg: error: argument <command>: invalid choice: 'rcreate' (choose from 'analyze', 'benchmark', 'check', 'compact', 'create', 'debug', 'delete', 'diff', 'extract', 'help', 'info', 'key', 'list', 'break-lock', 'with-lock', 'mount', 'umount', 'prune', 'repo-compress', 'repo-create', 'repo-delete', 'repo-info', 'repo-list', 'recreate', 'rename', 'repo-space', 'serve', 'tag', 'export-tar', 'import-tar', 'transfer', 'undelete', 'version')
/root/backup-test: Error running actions for repository
Command 'borg rcreate --encryption keyfile --repo /root/backup-test' returned non-zero exit status 2.
/etc/borgmatic/config.yaml: Error running configuration file

summary:
/etc/borgmatic/config.yaml: Error running configuration file
/root/backup-test: Error running actions for repository
Command 'borg rcreate --encryption keyfile --repo /root/backup-test' returned non-zero exit status 2.

Need some help? https://torsion.org/borgmatic/#issues

Borg 2.0.0b14 / Borgmatic >= 1.9.x

--checkpoint-interval seems not accepted

root@3b358ebe3b7a:~# borgmatic create --verbosity 2
/etc/borgmatic/config.yaml: No commands to run for pre-everything hook
BORG_PASSPHRASE=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** borg --version --debug --show-rc
/etc/borgmatic/config.yaml: Borg 2.0.0b14
/root/backup-test: Running actions for repository
/etc/borgmatic/config.yaml: No commands to run for pre-actions hook
/etc/borgmatic/config.yaml: No commands to run for pre-backup hook
/root/backup-test: Creating archive
/root/backup-test: Calling mariadb_databases hook function remove_data_source_dumps
/root/backup-test: Removing MariaDB data source dumps
/root/backup-test: Calling mysql_databases hook function remove_data_source_dumps
/root/backup-test: Removing MySQL data source dumps
/root/backup-test: Calling mongodb_databases hook function remove_data_source_dumps
/root/backup-test: Removing MongoDB data source dumps
/root/backup-test: Calling postgresql_databases hook function remove_data_source_dumps
/root/backup-test: Removing PostgreSQL data source dumps
/root/backup-test: Calling sqlite_databases hook function remove_data_source_dumps
/root/backup-test: Removing SQLite data source dumps
/root/backup-test: Calling postgresql_databases hook function dump_data_sources
/root/backup-test: Dumping PostgreSQL databases
/root/backup-test: Dumping PostgreSQL database "test" to /run/user/0/./borgmatic/postgresql_databases/test/test
/root/drip.sh --no-password --clean --if-exists --host test --format custom test > /run/user/0/./borgmatic/postgresql_databases/test/test
/root/backup-test: Ignoring configured "read_special" value of false, as true is needed for database hooks.
/root/backup-test: Collecting special file paths
BORG_PASSPHRASE=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** borg create --checkpoint-interval 1 --read-special --repo /root/backup-test {hostname}-test-{now}.checkpoint /etc/borgmatic/config.yaml /root/backup-input /run/user/0/./borgmatic --dry-run --list
/root/backup-test: Error running actions for repository
Command '('borg', 'create', '--checkpoint-interval', '1', '--read-special', '--repo', '/root/backup-test', '{hostname}-test-{now}.checkpoint', '/etc/borgmatic/config.yaml', '/root/backup-input', '/run/user/0/./borgmatic', '--dry-run', '--list')' returned non-zero exit status 2.
/etc/borgmatic/config.yaml: No commands to run for on-error hook
/etc/borgmatic/config.yaml: An error occurred
/etc/borgmatic/config.yaml: No commands to run for post-everything hook

summary:
/etc/borgmatic/config.yaml: Loading configuration file
/etc/borgmatic/config.yaml: An error occurred
/root/backup-test: Error running actions for repository
usage: borg [-V] [-h] [--critical] [--error] [--warning] [--info] [--debug]
            [--debug-topic TOPIC] [-p] [--iec] [--log-json]
            [--lock-wait SECONDS] [--show-version] [--show-rc] [--umask M]
            [--remote-path PATH] [--upload-ratelimit RATE]
            [--upload-buffer UPLOAD_BUFFER] [--debug-profile FILE] [--rsh RSH]
            [--socket [PATH]] [-r REPO]
            <command> ...
borg: error: unrecognized arguments: --checkpoint-interval {hostname}-test-{now}.checkpoint /etc/borgmatic/config.yaml /root/backup-input /run/user/0/./borgmatic
Command '('borg', 'create', '--checkpoint-interval', '1', '--read-special', '--repo', '/root/backup-test', '{hostname}-test-{now}.checkpoint', '/etc/borgmatic/config.yaml', '/root/backup-input', '/run/user/0/./borgmatic', '--dry-run', '--list')' returned non-zero exit status 2.

Need some help? https://torsion.org/borgmatic/#issues

--checkpoint-volume seems not accepted either:

root@3b358ebe3b7a:~# borgmatic create --verbosity 2
/etc/borgmatic/config.yaml: No commands to run for pre-everything hook
BORG_PASSPHRASE=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** borg --version --debug --show-rc
/etc/borgmatic/config.yaml: Borg 2.0.0b14
/root/backup-test: Running actions for repository
/etc/borgmatic/config.yaml: No commands to run for pre-actions hook
/etc/borgmatic/config.yaml: No commands to run for pre-backup hook
/root/backup-test: Creating archive
/root/backup-test: Calling mariadb_databases hook function remove_data_source_dumps
/root/backup-test: Removing MariaDB data source dumps
/root/backup-test: Calling mysql_databases hook function remove_data_source_dumps
/root/backup-test: Removing MySQL data source dumps
/root/backup-test: Calling mongodb_databases hook function remove_data_source_dumps
/root/backup-test: Removing MongoDB data source dumps
/root/backup-test: Calling postgresql_databases hook function remove_data_source_dumps
/root/backup-test: Removing PostgreSQL data source dumps
/root/backup-test: Calling sqlite_databases hook function remove_data_source_dumps
/root/backup-test: Removing SQLite data source dumps
/root/backup-test: Calling postgresql_databases hook function dump_data_sources
/root/backup-test: Dumping PostgreSQL databases
/root/backup-test: Dumping PostgreSQL database "test" to /run/user/0/./borgmatic/postgresql_databases/test/test
/root/drip.sh --no-password --clean --if-exists --host test --format custom test > /run/user/0/./borgmatic/postgresql_databases/test/test
/root/backup-test: Ignoring configured "read_special" value of false, as true is needed for database hooks.
/root/backup-test: Collecting special file paths
BORG_PASSPHRASE=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** borg create --checkpoint-volume 1 --read-special --repo /root/backup-test {hostname}-test-{now}.checkpoint /etc/borgmatic/config.yaml /root/backup-input /run/user/0/./borgmatic --dry-run --list
/root/backup-test: Error running actions for repository
Command '('borg', 'create', '--checkpoint-volume', '1', '--read-special', '--repo', '/root/backup-test', '{hostname}-test-{now}.checkpoint', '/etc/borgmatic/config.yaml', '/root/backup-input', '/run/user/0/./borgmatic', '--dry-run', '--list')' returned non-zero exit status 2.
/etc/borgmatic/config.yaml: No commands to run for on-error hook
/etc/borgmatic/config.yaml: An error occurred
/etc/borgmatic/config.yaml: No commands to run for post-everything hook

summary:
/etc/borgmatic/config.yaml: Loading configuration file
/etc/borgmatic/config.yaml: An error occurred
/root/backup-test: Error running actions for repository
usage: borg [-V] [-h] [--critical] [--error] [--warning] [--info] [--debug]
            [--debug-topic TOPIC] [-p] [--iec] [--log-json]
            [--lock-wait SECONDS] [--show-version] [--show-rc] [--umask M]
            [--remote-path PATH] [--upload-ratelimit RATE]
            [--upload-buffer UPLOAD_BUFFER] [--debug-profile FILE] [--rsh RSH]
            [--socket [PATH]] [-r REPO]
            <command> ...
borg: error: unrecognized arguments: --checkpoint-volume {hostname}-test-{now}.checkpoint /etc/borgmatic/config.yaml /root/backup-input /run/user/0/./borgmatic
Command '('borg', 'create', '--checkpoint-volume', '1', '--read-special', '--repo', '/root/backup-test', '{hostname}-test-{now}.checkpoint', '/etc/borgmatic/config.yaml', '/root/backup-input', '/run/user/0/./borgmatic', '--dry-run', '--list')' returned non-zero exit status 2.

Need some help? https://torsion.org/borgmatic/#issues

I have spent some time analyzing this. Overall findings: - Borg 2.x [does not write `.checkpoint` archives (anymore)]( https://github.com/borgbackup/borg/commit/5e3f2c04d578cba7291417c798fda863e8ad36c8). It allows creating archives ending in `.checkpoint`. - Borg up to 1.4.x seems to happily create `.checkpoint.checkpoint` archives (see "Test Results" and following for analysis). Proposal: - Change borgmatic as discussed: 1. Write an archive ending in `.checkpoint`. 2. If streaming dumps complete successfully, rename the archive to remove the `.checkpoint` suffix. - Once borg 2.0 stabilizes, backport relevant pruning logic to borgmatic to ensure `.checkpoints` (now only from streaming) get collected correctly. I'm happy to take a stab at the first bullet. @witten WDYT? <details> <summary>Test Results</summary> Overall, getting borg to actually write checkpoints was the hardest here. I have stopped testing with later borgmatic versions, since it seemed that I'm mostly chasing changes in borgmatic. ✅ = `.checkpoint.checkpoint` archive created. ⚠️ = Unable to produce any checkpoint in my setup. | borgmatic | borg | test | |-----------|----------|------| | 1.7.7 | 1.2.4 | ✅ | | 1.7.7 | 1.4.0 | ✅ | | 1.8.13 | 1.2.4 | ✅️ | | 1.8.13 | 1.4.0 | ✅️ | | 1.9.0 | 1.2.4 | ✅️️ | | 1.9.0 | 1.4.0 | ✅️️ | | 1.9.1 | 1.2.4 | ✅️️️ | | 1.9.1 | 1.4.0 | ✅️️️ | | 1.9.2 | 1.4.0 | ✅️️️ | | 1.9.3 | 1.4.0 | ✅️️️ | | 1.9.5 | 1.2.4 | ✅️️️ | | 1.9.5 | 1.4.0 | ✅️️️ | | 1.9.6 | 1.2.4 | ⚠️ | | 1.9.6 | 1.4.0 | ⚠️ | | 1.9.7 | 1.2.4 | ⚠️ | | 1.9.7 | 1.4.0 | ⚠️️️️ | | 1.9.14 | 1.2.4 | ⚠️ | | 2.0.2 | 1.2.4 | ⚠️ | </details> <details> <summary>Test Setup</summary> ```Dockerfile FROM debian RUN apt-get update RUN apt-get install -y emacs pipx wget ADD --chmod=755 https://github.com/borgbackup/borg/releases/download/1.4.0/borg-linux-glibc231 /usr/local/bin/borg #ADD --chmod=755 https://github.com/borgbackup/borg/releases/download/1.2.8/borg-linux64 /usr/local/bin/borg #ADD --chmod=755 https://github.com/borgbackup/borg/releases/download/2.0.0b14/borg-linux-glibc236 /usr/local/bin/borg ``` Then run: ``` cd pipx install borgmatic==1.9.6 export PATH="/root/.local/bin:$PATH" mkdir /etc/borgmatic cat > /etc/borgmatic/config.yaml <<EOF repositories: - path: /root/backup-test source_directories: [/root/backup-input] encryption_passphrase: the-secret archive_name_format: '{hostname}-test-{now}.checkpoint' user_runtime_directory: /borgmatic checkpoint_interval: 1 postgresql_databases: - name: test hostname: test pg_dump_command: "/root/drip.sh" EOF cat > drip.sh <<EOF #! /bin/sh while true; do sleep 1 echo "a" done EOF chmod +x drip.sh mkdir backup-input dd if=/dev/random of=/root/backup-input/file bs=512k count=10 ``` </details> <details> <summary>Test Commands</summary> # Borg 1.x ```sh rm -rf backup-test borgmatic init -e keyfile borgmatic create --verbosity 2 # kill with ctrl-c borg break-lock /root/backup-test # necessary only in borgmatic 1.7.7 borg list --consider-checkpoints /root/backup-test ``` # Borg 2.x ```sh rm -rf backup-test borgmatic init -e none borgmatic create --verbosity 2 # don't kill! borg repo-list -r /root/backup-test ``` </details> <details> <summary>Test Configs</summary> # Borgmatic 1.7.x ```yaml location: repositories: [/root/backup-test] source_directories: [/root/backup-input] storage: encryption_passphrase: the-secret checkpoint_interval: 1 archive_name_format: '{hostname}-test-{now}.checkpoint' hooks: postgresql_databases: - name: test hostname: test pg_dump_command: "/root/drip.sh" ``` # Borgmatic > 1.8.x ```yaml repositories: - path: /root/backup-test source_directories: [/root/backup-input] encryption_passphrase: the-secret archive_name_format: '{hostname}-test-{now}.checkpoint' checkpoint_interval: 1 postgresql_databases: - name: test hostname: test pg_dump_command: "/root/drip.sh" ``` Borgmatic >= 1.9.2 ```diff +user_runtime_directory: /borgmatic ``` Borg 2.0.0b14: ```diff -checkpoint_interval: 1 - pg_dump_command: "/root/drip.sh" + pg_dump_command: "true" ``` </details> <details> <summary>Error logs</summary> # Borg 2.0.0b14 / Borgmatic 1.7.7, 1.8.13 ``` root@3b358ebe3b7a:~# borgmatic init -e keyfile usage: borg [-V] [-h] [--critical] [--error] [--warning] [--info] [--debug] [--debug-topic TOPIC] [-p] [--iec] [--log-json] [--lock-wait SECONDS] [--show-version] [--show-rc] [--umask M] [--remote-path PATH] [--upload-ratelimit RATE] [--upload-buffer UPLOAD_BUFFER] [--debug-profile FILE] [--rsh RSH] [--socket [PATH]] [-r REPO] <command> ... borg: error: argument <command>: invalid choice: 'rinfo' (choose from 'analyze', 'benchmark', 'check', 'compact', 'create', 'debug', 'delete', 'diff', 'extract', 'help', 'info', 'key', 'list', 'break-lock', 'with-lock', 'mount', 'umount', 'prune', 'repo-compress', 'repo-create', 'repo-delete', 'repo-info', 'repo-list', 'recreate', 'rename', 'repo-space', 'serve', 'tag', 'export-tar', 'import-tar', 'transfer', 'undelete', 'version') usage: borg [-V] [-h] [--critical] [--error] [--warning] [--info] [--debug] [--debug-topic TOPIC] [-p] [--iec] [--log-json] [--lock-wait SECONDS] [--show-version] [--show-rc] [--umask M] [--remote-path PATH] [--upload-ratelimit RATE] [--upload-buffer UPLOAD_BUFFER] [--debug-profile FILE] [--rsh RSH] [--socket [PATH]] [-r REPO] <command> ... borg: error: argument <command>: invalid choice: 'rcreate' (choose from 'analyze', 'benchmark', 'check', 'compact', 'create', 'debug', 'delete', 'diff', 'extract', 'help', 'info', 'key', 'list', 'break-lock', 'with-lock', 'mount', 'umount', 'prune', 'repo-compress', 'repo-create', 'repo-delete', 'repo-info', 'repo-list', 'recreate', 'rename', 'repo-space', 'serve', 'tag', 'export-tar', 'import-tar', 'transfer', 'undelete', 'version') /root/backup-test: Error running actions for repository Command 'borg rcreate --encryption keyfile --repo /root/backup-test' returned non-zero exit status 2. /etc/borgmatic/config.yaml: Error running configuration file summary: /etc/borgmatic/config.yaml: Error running configuration file /root/backup-test: Error running actions for repository Command 'borg rcreate --encryption keyfile --repo /root/backup-test' returned non-zero exit status 2. Need some help? https://torsion.org/borgmatic/#issues ``` # Borg 2.0.0b14 / Borgmatic >= 1.9.x `--checkpoint-interval` seems not accepted ``` root@3b358ebe3b7a:~# borgmatic create --verbosity 2 /etc/borgmatic/config.yaml: No commands to run for pre-everything hook BORG_PASSPHRASE=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** borg --version --debug --show-rc /etc/borgmatic/config.yaml: Borg 2.0.0b14 /root/backup-test: Running actions for repository /etc/borgmatic/config.yaml: No commands to run for pre-actions hook /etc/borgmatic/config.yaml: No commands to run for pre-backup hook /root/backup-test: Creating archive /root/backup-test: Calling mariadb_databases hook function remove_data_source_dumps /root/backup-test: Removing MariaDB data source dumps /root/backup-test: Calling mysql_databases hook function remove_data_source_dumps /root/backup-test: Removing MySQL data source dumps /root/backup-test: Calling mongodb_databases hook function remove_data_source_dumps /root/backup-test: Removing MongoDB data source dumps /root/backup-test: Calling postgresql_databases hook function remove_data_source_dumps /root/backup-test: Removing PostgreSQL data source dumps /root/backup-test: Calling sqlite_databases hook function remove_data_source_dumps /root/backup-test: Removing SQLite data source dumps /root/backup-test: Calling postgresql_databases hook function dump_data_sources /root/backup-test: Dumping PostgreSQL databases /root/backup-test: Dumping PostgreSQL database "test" to /run/user/0/./borgmatic/postgresql_databases/test/test /root/drip.sh --no-password --clean --if-exists --host test --format custom test > /run/user/0/./borgmatic/postgresql_databases/test/test /root/backup-test: Ignoring configured "read_special" value of false, as true is needed for database hooks. /root/backup-test: Collecting special file paths BORG_PASSPHRASE=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** borg create --checkpoint-interval 1 --read-special --repo /root/backup-test {hostname}-test-{now}.checkpoint /etc/borgmatic/config.yaml /root/backup-input /run/user/0/./borgmatic --dry-run --list /root/backup-test: Error running actions for repository Command '('borg', 'create', '--checkpoint-interval', '1', '--read-special', '--repo', '/root/backup-test', '{hostname}-test-{now}.checkpoint', '/etc/borgmatic/config.yaml', '/root/backup-input', '/run/user/0/./borgmatic', '--dry-run', '--list')' returned non-zero exit status 2. /etc/borgmatic/config.yaml: No commands to run for on-error hook /etc/borgmatic/config.yaml: An error occurred /etc/borgmatic/config.yaml: No commands to run for post-everything hook summary: /etc/borgmatic/config.yaml: Loading configuration file /etc/borgmatic/config.yaml: An error occurred /root/backup-test: Error running actions for repository usage: borg [-V] [-h] [--critical] [--error] [--warning] [--info] [--debug] [--debug-topic TOPIC] [-p] [--iec] [--log-json] [--lock-wait SECONDS] [--show-version] [--show-rc] [--umask M] [--remote-path PATH] [--upload-ratelimit RATE] [--upload-buffer UPLOAD_BUFFER] [--debug-profile FILE] [--rsh RSH] [--socket [PATH]] [-r REPO] <command> ... borg: error: unrecognized arguments: --checkpoint-interval {hostname}-test-{now}.checkpoint /etc/borgmatic/config.yaml /root/backup-input /run/user/0/./borgmatic Command '('borg', 'create', '--checkpoint-interval', '1', '--read-special', '--repo', '/root/backup-test', '{hostname}-test-{now}.checkpoint', '/etc/borgmatic/config.yaml', '/root/backup-input', '/run/user/0/./borgmatic', '--dry-run', '--list')' returned non-zero exit status 2. Need some help? https://torsion.org/borgmatic/#issues ``` `--checkpoint-volume` seems not accepted either: ``` root@3b358ebe3b7a:~# borgmatic create --verbosity 2 /etc/borgmatic/config.yaml: No commands to run for pre-everything hook BORG_PASSPHRASE=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** borg --version --debug --show-rc /etc/borgmatic/config.yaml: Borg 2.0.0b14 /root/backup-test: Running actions for repository /etc/borgmatic/config.yaml: No commands to run for pre-actions hook /etc/borgmatic/config.yaml: No commands to run for pre-backup hook /root/backup-test: Creating archive /root/backup-test: Calling mariadb_databases hook function remove_data_source_dumps /root/backup-test: Removing MariaDB data source dumps /root/backup-test: Calling mysql_databases hook function remove_data_source_dumps /root/backup-test: Removing MySQL data source dumps /root/backup-test: Calling mongodb_databases hook function remove_data_source_dumps /root/backup-test: Removing MongoDB data source dumps /root/backup-test: Calling postgresql_databases hook function remove_data_source_dumps /root/backup-test: Removing PostgreSQL data source dumps /root/backup-test: Calling sqlite_databases hook function remove_data_source_dumps /root/backup-test: Removing SQLite data source dumps /root/backup-test: Calling postgresql_databases hook function dump_data_sources /root/backup-test: Dumping PostgreSQL databases /root/backup-test: Dumping PostgreSQL database "test" to /run/user/0/./borgmatic/postgresql_databases/test/test /root/drip.sh --no-password --clean --if-exists --host test --format custom test > /run/user/0/./borgmatic/postgresql_databases/test/test /root/backup-test: Ignoring configured "read_special" value of false, as true is needed for database hooks. /root/backup-test: Collecting special file paths BORG_PASSPHRASE=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** borg create --checkpoint-volume 1 --read-special --repo /root/backup-test {hostname}-test-{now}.checkpoint /etc/borgmatic/config.yaml /root/backup-input /run/user/0/./borgmatic --dry-run --list /root/backup-test: Error running actions for repository Command '('borg', 'create', '--checkpoint-volume', '1', '--read-special', '--repo', '/root/backup-test', '{hostname}-test-{now}.checkpoint', '/etc/borgmatic/config.yaml', '/root/backup-input', '/run/user/0/./borgmatic', '--dry-run', '--list')' returned non-zero exit status 2. /etc/borgmatic/config.yaml: No commands to run for on-error hook /etc/borgmatic/config.yaml: An error occurred /etc/borgmatic/config.yaml: No commands to run for post-everything hook summary: /etc/borgmatic/config.yaml: Loading configuration file /etc/borgmatic/config.yaml: An error occurred /root/backup-test: Error running actions for repository usage: borg [-V] [-h] [--critical] [--error] [--warning] [--info] [--debug] [--debug-topic TOPIC] [-p] [--iec] [--log-json] [--lock-wait SECONDS] [--show-version] [--show-rc] [--umask M] [--remote-path PATH] [--upload-ratelimit RATE] [--upload-buffer UPLOAD_BUFFER] [--debug-profile FILE] [--rsh RSH] [--socket [PATH]] [-r REPO] <command> ... borg: error: unrecognized arguments: --checkpoint-volume {hostname}-test-{now}.checkpoint /etc/borgmatic/config.yaml /root/backup-input /run/user/0/./borgmatic Command '('borg', 'create', '--checkpoint-volume', '1', '--read-special', '--repo', '/root/backup-test', '{hostname}-test-{now}.checkpoint', '/etc/borgmatic/config.yaml', '/root/backup-input', '/run/user/0/./borgmatic', '--dry-run', '--list')' returned non-zero exit status 2. Need some help? https://torsion.org/borgmatic/#issues ``` </details>

gzm0 commented 2025-04-11 12:45:01 +00:00

Author

Contributor

Copy link

I've started looking at code to make this happen. Two things we need to figure for the proposed design to work:

• It seems that borgmatic has full support to retrieve the json output back from borg (example pasted below). If we change the commands we run, this might result in a breaking change. An option to fix this would be something similar than what is done for repository labels:
  • Replace the name parameter in archive with the renamed archive
  • Add a borgmatic_checkpoint_name field with the actual archive name we requested from borg.
• It seems that borgmatic is currently relying on borg to create the archive name (from archive_name_format):
  
  borgmatic/borg/create.py

  Lines 213 to 215 in 81db67c

  	create_positional_arguments = flags.make_repository_archive_flags(
  	repository_path, archive_name_format, local_borg_version
  	)

  
  As a result, we wouldn't know which archive to rename (and what to rename to).
  A way to fix this would be to force --json output and retrieve the created archive name.

borgmatic create --verbosity 2 --json

[
  {
    archive: {
      command_line: [
        "borg",
        "create",
        "--patterns-from",
        "/tmp/borgmatic-lfvy5f2a/borgmatic/tmpaap7dm64",
        "--json",
        "/root/backup-test::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}",
      ],
      duration: 0.028948,
      end: "2025-04-11T12:37:49.000000",
      id: "b184079a5eabc1ffd620779b3bbf8b084438f4dd520dd2fdb53a62a34d463e2a",
      limits: { max_archive_size: 2.9039439707614328e-5 },
      name: "75216f00dc38-2025-04-11T12:37:49.393591",
      start: "2025-04-11T12:37:49.000000",
      stats: {
        compressed_size: 5243795,
        deduplicated_size: 5243795,
        nfiles: 3,
        original_size: 5243709,
      },
    },
    cache: {
      path: "/root/.cache/borg/9a9a170cd359280a7d12b89100616941fa7d605de2564a34e1b0144379c21194",
      stats: {
        total_chunks: 5,
        total_csize: 5243186,
        total_size: 5243075,
        total_unique_chunks: 5,
        unique_csize: 5244259,
        unique_size: 5244427,
      },
    },
    encryption: {
      keyfile: "/root/.config/borg/keys/root_backup_test",
      mode: "keyfile",
    },
    repository: {
      id: "9a9a170cd359280a7d12b89100616941fa7d605de2564a34e1b0144379c21194",
      last_modified: "2025-04-11T12:37:49.000000",
      location: "/root/backup-test",
      label: "",
    },
  },
];

I've started looking at code to make this happen. Two things we need to figure for the proposed design to work: - It seems that borgmatic has full support to retrieve the json output back from borg (example pasted below). If we change the commands we run, this might result in a breaking change. An option to fix this would be something similar than what is done for repository labels: - Replace the `name` parameter in `archive` with the renamed archive - Add a `borgmatic_checkpoint_name` field with the actual archive name we requested from borg. - It seems that borgmatic is currently relying on borg to create the archive name (from `archive_name_format`): https://projects.torsion.org/borgmatic-collective/borgmatic/src/commit/81db67c759feebde311ccd8e9e7516cd4970e5e0/borgmatic/borg/create.py#L213-L215 As a result, we wouldn't know which archive to rename (and what to rename to). A way to fix this would be to force `--json` output and retrieve the created archive name. <details> <summary> `borgmatic create --verbosity 2 --json` </summary> ```json [ { archive: { command_line: [ "borg", "create", "--patterns-from", "/tmp/borgmatic-lfvy5f2a/borgmatic/tmpaap7dm64", "--json", "/root/backup-test::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}", ], duration: 0.028948, end: "2025-04-11T12:37:49.000000", id: "b184079a5eabc1ffd620779b3bbf8b084438f4dd520dd2fdb53a62a34d463e2a", limits: { max_archive_size: 2.9039439707614328e-5 }, name: "75216f00dc38-2025-04-11T12:37:49.393591", start: "2025-04-11T12:37:49.000000", stats: { compressed_size: 5243795, deduplicated_size: 5243795, nfiles: 3, original_size: 5243709, }, }, cache: { path: "/root/.cache/borg/9a9a170cd359280a7d12b89100616941fa7d605de2564a34e1b0144379c21194", stats: { total_chunks: 5, total_csize: 5243186, total_size: 5243075, total_unique_chunks: 5, unique_csize: 5244259, unique_size: 5244427, }, }, encryption: { keyfile: "/root/.config/borg/keys/root_backup_test", mode: "keyfile", }, repository: { id: "9a9a170cd359280a7d12b89100616941fa7d605de2564a34e1b0144379c21194", last_modified: "2025-04-11T12:37:49.000000", location: "/root/backup-test", label: "", }, }, ]; ``` </details>

witten commented 2025-04-11 18:39:58 +00:00

Owner

Copy link

Thanks for doing all the investigatory work! Some thoughts:

• Unfortunately, I don't think we can switch to create --json, as convenient as it would be for getting the created archive name. That's because we're currently relying on Borg to present its create output to the user. But if we switch to capturing all of its output so we can consume the JSON output, then all of the user-facing output will get swallowed.
• As an alternative, what do you think of the following: After the backup succeeds, borgmatic could lookup the latest archive (there's already code to do that, and incidentally it uses --json) and then rename it accordingly. It's not nearly as nice as getting it straight from create output, but it has the nice property of not swallowing that create output.

Once borg 2.0 stabilizes, backport relevant pruning logic to borgmatic to ensure .checkpoints (now only from streaming) get collected correctly.

If running with Borg 2, borgmatic could just always delete all .checkpoint archives upon success, as presumably borgmatic will be the only one creating .checkpoint archives. (Although maybe it would have to restricted to archives matching the current archive_name_format, which would make it a little more complicated.)

Thanks for doing all the investigatory work! Some thoughts: - Unfortunately, I don't think we can switch to `create --json`, as convenient as it would be for getting the created archive name. That's because we're currently relying on Borg to present its `create` output to the user. But if we switch to capturing all of its output so we can consume the JSON output, then all of the user-facing output will get swallowed. - As an alternative, what do you think of the following: After the backup succeeds, borgmatic could lookup the latest archive (there's already code to do that, and incidentally it uses `--json`) and then rename it accordingly. It's not nearly as nice as getting it straight from `create` output, but it has the nice property of not swallowing that `create` output. > Once borg 2.0 stabilizes, backport relevant pruning logic to borgmatic to ensure .checkpoints (now only from streaming) get collected correctly. If running with Borg 2, borgmatic could just always delete all `.checkpoint` archives upon success, as presumably borgmatic will be the only one creating `.checkpoint` archives. (Although maybe it would have to restricted to archives matching the current `archive_name_format`, which would make it a little more complicated.)

gzm0 commented 2025-04-16 15:10:16 +00:00

Author

Contributor

Copy link

As an alternative, what do you think of the following: After the backup succeeds, borgmatic could lookup the latest archive (there's already code to do that, and incidentally it uses --json) and then rename it accordingly. It's not nearly as nice as getting it straight from create output, but it has the nice property of not swallowing that create output.

I think that could work. There is technically a race condition, but it would surprise me to find a use case that is susceptible to that race condition and not to other (intrinsic) race conditions (e.g. two borgmatic processes).

I am wondering if it tips the balance in terms of complexity w.r.t. a "magic" pipe solution. When considering it, I did not consider the output of the borgmatic command itself. From that POV, a "magic" pipe might be much simpler.

I will keep thinking about it fiddle with some code to see what it would take.

If running with Borg 2, borgmatic could just always delete all .checkpoint archives upon success, as presumably borgmatic will be the only one creating .checkpoint archives.

Ah yes, that's a quite neat solution indeed.

> As an alternative, what do you think of the following: After the backup succeeds, borgmatic could lookup the latest archive (there's already code to do that, and incidentally it uses --json) and then rename it accordingly. It's not nearly as nice as getting it straight from create output, but it has the nice property of not swallowing that create output. I think that could work. There is technically a race condition, but it would surprise me to find a use case that is susceptible to that race condition and not to other (intrinsic) race conditions (e.g. two borgmatic processes). I am wondering if it tips the balance in terms of complexity w.r.t. a "magic" pipe solution. When considering it, I did not consider the output of the borgmatic command itself. From that POV, a "magic" pipe might be much simpler. I will keep thinking about it fiddle with some code to see what it would take. > If running with Borg 2, borgmatic could just always delete all .checkpoint archives upon success, as presumably borgmatic will be the only one creating .checkpoint archives. Ah yes, that's a quite neat solution indeed.

witten commented 2025-04-20 03:50:23 +00:00

Owner

Copy link

I think that could work. There is technically a race condition, but it would surprise me to find a use case that is susceptible to that race condition and not to other (intrinsic) race conditions (e.g. two borgmatic processes).

If you really wanted to prevent the race, you could probably just create the archive with a name only known to this borgmatic process, e.g. f'{randomhexdigest}.checkpoint' and then after the create is done, rename that very particular temporary archive name using the configured archive name format. So, to the name it would normally be without this ticket implemented.

I am wondering if it tips the balance in terms of complexity w.r.t. a "magic" pipe solution. When considering it, I did not consider the output of the borgmatic command itself. From that POV, a "magic" pipe might be much simpler.

Say more?

I will keep thinking about it fiddle with some code to see what it would take.

I appreciate your persistence here!

> I think that could work. There is technically a race condition, but it would surprise me to find a use case that is susceptible to that race condition and not to other (intrinsic) race conditions (e.g. two borgmatic processes). If you really wanted to prevent the race, you could probably just create the archive with a name only known to this borgmatic process, e.g. `f'{randomhexdigest}.checkpoint'` and then after the `create` is done, `rename` that very particular temporary archive name using the configured archive name format. So, to the name it would normally be without this ticket implemented. > I am wondering if it tips the balance in terms of complexity w.r.t. a "magic" pipe solution. When considering it, I did not consider the output of the borgmatic command itself. From that POV, a "magic" pipe might be much simpler. Say more? > I will keep thinking about it fiddle with some code to see what it would take. I appreciate your persistence here!

gzm0 commented 2025-05-09 08:34:12 +00:00

Author

Contributor

Copy link

Sorry for the late reply here, I was struggling to find focus time.

With the "magic pipe" I mean a solution where we create an additional named pipe for the sole purpose of ensuring the borg process cannot complete before borgmatic has checked all exit codes of all streaming processes.

This gives borgmatic the chance to kill the borg process in case a streaming process fails.

IIUC we have briefly discussed this, but dismissed the idea because checkpointing seems cleaner.
However, after looking at the borgmatic cli interface in more detail, I feel it might be a simpler solution overall and also make the borg 2 story much simpler.

I'll try to spend some cycles on seeing what the necessary changes would need to be (seems like the trickiest bit will be borgmatic.execute.log_outputs: we need some form of callback when individual processes complete).j

Just to set expectations: I'll be out for 2 weeks starting today, so nothing is going to happen for sure in this time :)

Sorry for the late reply here, I was struggling to find focus time. With the "magic pipe" I mean a solution where we create an additional named pipe for the sole purpose of ensuring the borg process cannot complete before borgmatic has checked all exit codes of all streaming processes. This gives borgmatic the chance to kill the borg process in case a streaming process fails. IIUC we have briefly discussed this, but dismissed the idea because checkpointing seems cleaner. However, after looking at the borgmatic cli interface in more detail, I feel it might be a simpler solution overall and also make the borg 2 story much simpler. I'll try to spend some cycles on seeing what the necessary changes would need to be (seems like the trickiest bit will be `borgmatic.execute.log_outputs`: we need some form of callback when individual processes complete).j Just to set expectations: I'll be out for 2 weeks starting today, so nothing is going to happen for sure in this time :)

witten commented 2025-05-10 01:57:22 +00:00

Owner

Copy link

Sorry for the late reply here, I was struggling to find focus time.

No worries.. I know exactly how it is.

With the "magic pipe" I mean a solution where we create an additional named pipe for the sole purpose of ensuring the borg process cannot complete before borgmatic has checked all exit codes of all streaming processes.

Ah gotcha!

This gives borgmatic the chance to kill the borg process in case a streaming process fails.

IIUC we have briefly discussed this, but dismissed the idea because checkpointing seems cleaner.
However, after looking at the borgmatic cli interface in more detail, I feel it might be a simpler solution overall and also make the borg 2 story much simpler.

I'll try to spend some cycles on seeing what the necessary changes would need to be (seems like the trickiest bit will be borgmatic.execute.log_outputs: we need some form of callback when individual processes complete).j

Yeah, that seems a little tricky. But on second thought, it might actually not be needed...? For instance, once the magic pipe that's blocking Borg from exiting is put in place, then as soon the database process exits with error, that might be enough for the existing log_outputs() code to kill all other open processes (so, Borg) and return. Then you could do any cleanup work (removing the pipe, etc.) afterwards.

I might be missing something though!

Just to set expectations: I'll be out for 2 weeks starting today, so nothing is going to happen for sure in this time :)

No worries. Enjoy your presumed time off!

> Sorry for the late reply here, I was struggling to find focus time. No worries.. I know exactly how it is. > With the "magic pipe" I mean a solution where we create an additional named pipe for the sole purpose of ensuring the borg process cannot complete before borgmatic has checked all exit codes of all streaming processes. Ah gotcha! > This gives borgmatic the chance to kill the borg process in case a streaming process fails. > > IIUC we have briefly discussed this, but dismissed the idea because checkpointing seems cleaner. > However, after looking at the borgmatic cli interface in more detail, I feel it might be a simpler solution overall and also make the borg 2 story much simpler. > > I'll try to spend some cycles on seeing what the necessary changes would need to be (seems like the trickiest bit will be borgmatic.execute.log_outputs: we need some form of callback when individual processes complete).j Yeah, that seems a little tricky. But on second thought, it might actually not be needed...? For instance, once the magic pipe that's blocking Borg from exiting is put in place, then as soon the database process exits with error, that might be enough for the existing `log_outputs()` code to kill all other open processes (so, Borg) and return. Then you could do any cleanup work (removing the pipe, etc.) afterwards. I might be missing something though! > Just to set expectations: I'll be out for 2 weeks starting today, so nothing is going to happen for sure in this time :) No worries. Enjoy your presumed time off!

gzm0 commented 2025-05-26 12:14:36 +00:00

Author

Contributor

Copy link

But on second thought, it might actually not be needed...? For instance, once the magic pipe that's blocking Borg from exiting is put in place, then as soon the database process exits with error, that might be enough for the existing log_outputs() code to kill all other open processes (so, Borg) and return. Then you could do any cleanup work (removing the pipe, etc.) afterwards.

Hah, yes! Good point. There is one thing we need to do though: If all database processes exit successfully, we need to close the pipe. Otherwise the system will deadlock itself.

However, that simplifies things quite a bit I think. I'll try and come up with a draft for this this week.

> But on second thought, it might actually not be needed...? For instance, once the magic pipe that's blocking Borg from exiting is put in place, then as soon the database process exits with error, that might be enough for the existing log_outputs() code to kill all other open processes (so, Borg) and return. Then you could do any cleanup work (removing the pipe, etc.) afterwards. Hah, yes! Good point. There is one thing we need to do though: If all database processes exit successfully, we need to close the pipe. Otherwise the system will deadlock itself. However, that simplifies things quite a bit I think. I'll try and come up with a draft for this this week.

witten commented 2025-05-26 16:25:04 +00:00

Owner

Copy link

Sounds good! Let me know how it goes.

Sounds good! Let me know how it goes.

gzm0 commented 2025-05-27 06:30:07 +00:00

Author

Contributor

Copy link

I have a WIP version using a named pipe / FIFO here:
https://projects.torsion.org/borgmatic-collective/borgmatic/compare/main...gzm0/borgmatic:dump-safe-pipe

However, it currently hangs. I suspect the approach with a named pipe / FIFO is doomed to fail: it assumes that borg will make independent progress on all files it needs to back up. Now that I think about it, that's not a reasonable assumption.

Or in other words: for the "FIFO blocks completion of the archive" approach to work, we'd need borg to backup the special FIFO last. At least without additional config (if even possible), we cannot expect that to just happen.

@witten I'd appreciate if you could give this some thought. But as it stands now, I don't think the FIFO approach is actually workable :(

I have a WIP version using a named pipe / FIFO here: https://projects.torsion.org/borgmatic-collective/borgmatic/compare/main...gzm0/borgmatic:dump-safe-pipe However, it currently hangs. I suspect the approach with a named pipe / FIFO is doomed to fail: it assumes that borg will make independent progress on all files it needs to back up. Now that I think about it, that's not a reasonable assumption. Or in other words: for the "FIFO blocks completion of the archive" approach to work, we'd need borg to backup the special FIFO last. At least without additional config (if even possible), we cannot expect that to just happen. @witten I'd appreciate if you could give this some thought. But as it stands now, I don't think the FIFO approach is actually workable :(

gzm0 referenced this issue from a pull request that will close it, 2025-05-28 14:25:42 +00:00

Create a checkpoint archive first with streaming processes #1102

gzm0 commented 2025-05-28 14:26:20 +00:00

Author

Contributor

Copy link

#1102 contains an implementation of the checkpoint + rename approach.

#1102 contains an implementation of the checkpoint + rename approach.

witten commented 2025-06-06 05:47:59 +00:00

Owner

Copy link

For the FIFO approach: The only thing I can think of is that the borgmatic_runtime_directory is already in the source patterns, and therefore your FIFO lock is getting included twice—and hanging. Also, the first time it's included in source patterns, it's probably not last like you need it to be.

For the FIFO approach: The only thing I can think of is that the `borgmatic_runtime_directory` is *already* in the source patterns, and therefore your FIFO lock is getting included twice—and hanging. Also, the first time it's included in source patterns, it's probably not last like you need it to be.

witten closed this issue 2025-06-25 16:24:08 +00:00

witten referenced this issue from a commit 2025-06-25 16:24:09 +00:00

Create a checkpoint archive first with streaming processes (#1032).

witten added the

bug

label 2025-06-25 16:26:13 +00:00

witten referenced this issue from a commit 2025-06-25 16:28:23 +00:00

Add fix for failing database hook creating archive to NEWS (#1032).

witten commented 2025-06-25 16:29:58 +00:00

Owner

Copy link

The checkpoint + rename approach is now implemented in main by @gzm0 and will be part of the next release!

The checkpoint + rename approach is now implemented in main by @gzm0 and will be part of the next release!

witten commented 2025-07-01 01:07:04 +00:00

Owner

Copy link

Released in borgmatic 2.0.7!

Released in borgmatic 2.0.7!

witten referenced this issue 2025-09-15 06:22:01 +00:00

Archive renaming and "--archive latest" broken with Borg 2 #1146

witten referenced this issue 2025-10-08 20:24:09 +00:00

Add Borg 2.0.0b17 features #1098

witten commented 2025-10-08 21:02:28 +00:00

Owner

Copy link

FYI here's an edge case where the checkpoint + rename approach causes problems: Borg 2's no-delete permission, which when enabled apparently breaks renames but not creates. See #1098 for more info.

FYI here's an edge case where the checkpoint + rename approach causes problems: Borg 2's `no-delete` permission, which when enabled apparently breaks `rename`s but not `create`s. See #1098 for more info.

witten referenced this issue 2025-10-09 01:55:06 +00:00

Add Borg 2.0.0b17 features #1098

wrexor referenced this issue 2025-10-09 02:15:00 +00:00

Add Borg 2.0.0b17 features #1098

witten referenced this issue 2025-10-09 05:07:00 +00:00

Add Borg 2.0.0b17 features #1098

gzm0 referenced this issue 2025-10-11 12:09:26 +00:00

Add Borg 2.0.0b17 features #1098

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

parallel-borgmatic

command-hook-steps

2.1.7

2.1.6

2.1.5

2.1.4

2.1.3

2.1.2

2.1.1

2.1.0

2.0.13

2.0.12

2.0.11

2.0.10

2.0.9

2.0.8

2.0.7

2.0.6

2.0.5

2.0.4

2.0.3

2.0.2

2.0.1

2.0.0

1.9.14

1.9.13

1.9.12

1.9.11

1.9.10

1.9.9

1.9.8

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.14

1.8.13

1.8.12

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels   

blocked

  

breaking

  

bug

  

data loss

  

design finalized

  

good first issue

  

new feature area

  

question / support

  

security

  

waiting for response

No labels

blocked

breaking

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

Milestone

Clear milestone

No items

No milestone

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

borgmatic-collective/borgmatic#1032

No description provided.