Misc new user feedback #554

New Issue

pikeas · 2022-06-28T21:19:44Z

pikeas commented

2022-06-28 21:19:44 +00:00

What I'm trying to do and why

Become familiar with Borgmatic and use it regularly.

borgmatic run without arguments is underdocumented. There's a note under actions for borgmatic -h, but it's easy to miss. Perhaps include some "running default commands: <a,b,c>" output at higher verbosity levels?
Healthcheck fails silently. I have hooks: healthchecks: ping_url: https://hc-ping.com/your-uuid-here, with the literal placeholder text. By default, this should ping on start, which shouuld fail due to using an invalid endpoint.
After creating a new repo with borgmatic init, I wanted to back up my repo key. I had to use borg key export <repo> to do this. It would be nice if borgmatic wrapped this with a sub-command (borgmatic export-all-keys?).
--files is recommended in the docs but not documented under borgmatic -h.
It's not clear what Borgmatic and/or Borg are doing while running.

For the last item, here's my most recent output on my first, currently running, backup:

A <file>

Remote: check_free_space: required bytes 545919898, free bytes 1096484959232
security: saving state for <id> to /Users/<user>/.config/borg/security/<id>
security: current location   ssh://<host>/./backups/<repo>
security: key type           5
security: manifest timestamp 2022-06-28T20:08:23.975795
Remote: Verified integrity of /home/backups/<repo>/index.48
Remote: Cleaned up 0 uncommitted segment files (== everything after segment 48).
Remote: Verified integrity of /home/backups/<repo>/hints.48
Remote: check_free_space: required bytes 545919998, free bytes 1095468191744
security: saving state for <id> to /Users/<user>/.config/borg/security/<id>
security: current location   ssh://<host>/./backups/<repo>
security: key type           5
security: manifest timestamp 2022-06-28T20:41:24.359237
Remote: Verified integrity of /home/backups/<repo>/index.53

The process was run with -v 2 --files and hasn't output additional lines for at least several minutes. Borgmatic and Borg are both running, but it's not clear what operation is in progress - or how recently - following integrity verification.

#### What I'm trying to do and why Become familiar with Borgmatic and use it regularly. - `borgmatic` run without arguments is underdocumented. There's a note under actions for `borgmatic -h`, but it's easy to miss. Perhaps include some "running default commands: <a,b,c>" output at higher verbosity levels? - Healthcheck fails silently. I have `hooks: healthchecks: ping_url: https://hc-ping.com/your-uuid-here`, with the literal placeholder text. By default, this should ping on start, which shouuld fail due to using an invalid endpoint. - After creating a new repo with `borgmatic init`, I wanted to back up my repo key. I had to use `borg key export <repo>` to do this. It would be nice if borgmatic wrapped this with a sub-command (`borgmatic export-all-keys`?). - `--files` is recommended in the docs but not documented under `borgmatic -h`. - It's not clear what Borgmatic and/or Borg are doing while running. For the last item, here's my most recent output on my first, currently running, backup: ``` A <file> Remote: check_free_space: required bytes 545919898, free bytes 1096484959232 security: saving state for <id> to /Users/<user>/.config/borg/security/<id> security: current location ssh://<host>/./backups/<repo> security: key type 5 security: manifest timestamp 2022-06-28T20:08:23.975795 Remote: Verified integrity of /home/backups/<repo>/index.48 Remote: Cleaned up 0 uncommitted segment files (== everything after segment 48). Remote: Verified integrity of /home/backups/<repo>/hints.48 Remote: check_free_space: required bytes 545919998, free bytes 1095468191744 security: saving state for <id> to /Users/<user>/.config/borg/security/<id> security: current location ssh://<host>/./backups/<repo> security: key type 5 security: manifest timestamp 2022-06-28T20:41:24.359237 Remote: Verified integrity of /home/backups/<repo>/index.53 ``` The process was run with `-v 2 --files` and hasn't output additional lines for at least several minutes. Borgmatic and Borg are both running, but it's not clear what operation is in progress - or how recently - following integrity verification.

witten referenced this issue from a commit

2022-06-29 16:20:39 +00:00

Clarify --files flag action in documentation (#554).

witten commented

2022-06-29 16:40:45 +00:00

This is awesome feedback! It's really good to get a list of all the speedbumps a new user hits, coming at things with a fresh set of eyes. Responses to each point below.

borgmatic run without arguments is underdocumented. There's a note under actions for borgmatic -h, but it's easy to miss. Perhaps include some "running default commands: <a,b,c>" output at higher verbosity levels?

Good feedback. I'll clarify in the documentation.

Healthcheck fails silently. I have hooks: healthchecks: ping_url: https://hc-ping.com/your-uuid-here, with the literal placeholder text. By default, this should ping on start, which shouuld fail due to using an invalid endpoint.

I'll look into this and get back to you. A Healthchecks failure doesn't cause borgmatic to error (any longer) by design. The idea is that a mere monitoring failure shouldn't block backups ... and Healthchecks will in theory notify you if it doesn't hear from backups after a configured period of time. But borgmatic should at least be warning upon a Healthchecks failure, so that might need fixing.

After creating a new repo with borgmatic init, I wanted to back up my repo key. I had to use borg key export to do this. It would be nice if borgmatic wrapped this with a sub-command (borgmatic export-all-keys?).

There's a ticket for that! #345. In the meantime, you may be able to use the borgmatic borg action to run an arbitrary/unwrapped Borg command like key.

--files is recommended in the docs but not documented under borgmatic -h.

It's actually documented under borgmatic create --help since it's a create-specific action. borgmatic -h only shows global actions. I'll clarify that in the documentation here: https://torsion.org/borgmatic/docs/how-to/set-up-backups/#backups

It's not clear what Borgmatic and/or Borg are doing while running. The process was run with -v 2 --files and hasn't output additional lines for at least several minutes. Borgmatic and Borg are both running, but it's not clear what operation is in progress - or how recently - following integrity verification.

I think this is a Borg issue. You could verify that by running borgmatic with --verbosity 2 (if you're not already) and grabbing the Borg command that it's running. Then, run that Borg command directly without borgmatic. If you get that same behavior, then it's not caused by borgmatic interference. If the behavior differs (e.g. if missing log lines show up), then it might be a borgmatic bug. I believe what I'd expect from Borg after verifying integrity is check_free_space.

This is awesome feedback! It's really good to get a list of all the speedbumps a new user hits, coming at things with a fresh set of eyes. Responses to each point below. > borgmatic run without arguments is underdocumented. There's a note under actions for borgmatic -h, but it's easy to miss. Perhaps include some "running default commands: <a,b,c>" output at higher verbosity levels? Good feedback. I'll clarify in the documentation. > Healthcheck fails silently. I have hooks: healthchecks: ping_url: https://hc-ping.com/your-uuid-here, with the literal placeholder text. By default, this should ping on start, which shouuld fail due to using an invalid endpoint. I'll look into this and get back to you. A Healthchecks failure doesn't cause borgmatic to error (any longer) by design. The idea is that a mere monitoring failure shouldn't block backups ... and Healthchecks will in theory notify you if it doesn't hear from backups after a configured period of time. But borgmatic should at least be *warning* upon a Healthchecks failure, so that might need fixing. > After creating a new repo with borgmatic init, I wanted to back up my repo key. I had to use borg key export <repo> to do this. It would be nice if borgmatic wrapped this with a sub-command (borgmatic export-all-keys?). There's a ticket for that! #345. In the meantime, you may be able to use the [`borgmatic borg` action](https://torsion.org/borgmatic/docs/how-to/run-arbitrary-borg-commands/) to run an arbitrary/unwrapped Borg command like `key`. > --files is recommended in the docs but not documented under borgmatic -h. It's actually documented under `borgmatic create --help` since it's a create-specific action. `borgmatic -h` only shows global actions. I'll clarify that in the documentation here: https://torsion.org/borgmatic/docs/how-to/set-up-backups/#backups > It's not clear what Borgmatic and/or Borg are doing while running. The process was run with -v 2 --files and hasn't output additional lines for at least several minutes. Borgmatic and Borg are both running, but it's not clear what operation is in progress - or how recently - following integrity verification. I *think* this is a Borg issue. You could verify that by running `borgmatic` with `--verbosity 2` (if you're not already) and grabbing the Borg command that it's running. Then, run that Borg command directly without borgmatic. If you get that same behavior, then it's not caused by borgmatic interference. If the behavior differs (e.g. if missing log lines show up), then it might be a borgmatic bug. I believe what I'd expect from Borg after verifying integrity is `check_free_space`.

witten added the

question / support

label 2022-06-29 16:40:52 +00:00

witten referenced this issue from a commit

2022-06-30 04:19:58 +00:00

Fix all monitoring hooks to warn if the server returns an HTTP 4xx error (#554).

witten commented

2022-06-30 04:25:44 +00:00

I'll look into this and get back to you. A Healthchecks failure doesn't cause borgmatic to error (any longer) by design. The idea is that a mere monitoring failure shouldn't block backups ... and Healthchecks will in theory notify you if it doesn't hear from backups after a configured period of time. But borgmatic should at least be warning upon a Healthchecks failure, so that might need fixing.

Okay, this should be fixed in master now. borgmatic now warns loudly for an invalid ping URL—but still does not error. The problem was that the Healtchecks server returns an HTTP 400 when there's an invalid Ping URL, and the Python requests library does not raise by default on an HTTP 400. I've worked around that with a more explicit status check. Thanks again for bringing this to my attention.

I'll leave this ticket open for now in case you have any other follow-up.

> I'll look into this and get back to you. A Healthchecks failure doesn't cause borgmatic to error (any longer) by design. The idea is that a mere monitoring failure shouldn't block backups ... and Healthchecks will in theory notify you if it doesn't hear from backups after a configured period of time. But borgmatic should at least be warning upon a Healthchecks failure, so that might need fixing. Okay, this should be fixed in master now. borgmatic now warns loudly for an invalid ping URL—but still does not error. The problem was that the Healtchecks server returns an HTTP 400 when there's an invalid Ping URL, and the Python requests library does not raise by default on an HTTP 400. I've worked around that with a more explicit status check. Thanks again for bringing this to my attention. I'll leave this ticket open for now in case you have any other follow-up.

witten referenced this issue from a commit

2022-06-30 04:32:26 +00:00

In documentation, be more explicit about default actions (#554).

pikeas commented

2022-06-30 21:10:43 +00:00

Okay, this should be fixed in master now. borgmatic now warns loudly for an invalid ping URL—but still does not error. The problem was that the Healtchecks server returns an HTTP 400 when there's an invalid Ping URL, and the Python requests library does not raise by default on an HTTP 400. I've worked around that with a more explicit status check. Thanks again for bringing this to my attention.

Sounds good!

--files is recommended in the docs but not documented under borgmatic -h.

It's actually documented under borgmatic create --help since it's a create-specific action. borgmatic -h only shows global actions. I'll clarify that in the documentation here: https://torsion.org/borgmatic/docs/how-to/set-up-backups/#backups

It's not clear what Borgmatic and/or Borg are doing while running. The process was run with -v 2 --files and hasn't output additional lines for at least several minutes. Borgmatic and Borg are both running, but it's not clear what operation is in progress - or how recently - following integrity verification.

I think this is a Borg issue. You could verify that by running borgmatic with --verbosity 2 (if you're not already) and grabbing the Borg command that it's running. Then, run that Borg command directly without borgmatic. If you get that same behavior, then it's not caused by borgmatic interference. If the behavior differs (e.g. if missing log lines show up), then it might be a borgmatic bug. I believe what I'd expect from Borg after verifying integrity is check_free_space.

Thinking on this a bit more, I think there are two parts here: what Borg does and what Borgmatic does.

Borg's output is a bit cryptic. In the core loop, it's adding files individually and occasionally doing a verification pass. There's no additional feedback with the default arguments.

Borgmatic recommends running borgmatic create --verbosity 1 --files. I think that the --progress and --stats flags may be more useful to users who are new to both Borg and Borgmatic.

I have run into another sharp edge now that my first backup has completed, regarding listing files in my archive:

$ borg list <repo>::<archive>
(works)

$ borgmatic list <archive>
Error parsing arguments: ...

$ borgmatic list <repo>::<archive>
Error parsing arguments: ...

$ borgmatic borg list <repo>::<archive>
Error parsing arguments: ...

It would be nice if #2 and #3 worked. #4 looks like it may be an argument parsing bug.

> Okay, this should be fixed in master now. borgmatic now warns loudly for an invalid ping URL—but still does not error. The problem was that the Healtchecks server returns an HTTP 400 when there's an invalid Ping URL, and the Python requests library does not raise by default on an HTTP 400. I've worked around that with a more explicit status check. Thanks again for bringing this to my attention. Sounds good! > > --files is recommended in the docs but not documented under borgmatic -h. > > It's actually documented under `borgmatic create --help` since it's a create-specific action. `borgmatic -h` only shows global actions. I'll clarify that in the documentation here: https://torsion.org/borgmatic/docs/how-to/set-up-backups/#backups > > > It's not clear what Borgmatic and/or Borg are doing while running. The process was run with -v 2 --files and hasn't output additional lines for at least several minutes. Borgmatic and Borg are both running, but it's not clear what operation is in progress - or how recently - following integrity verification. > > I *think* this is a Borg issue. You could verify that by running `borgmatic` with `--verbosity 2` (if you're not already) and grabbing the Borg command that it's running. Then, run that Borg command directly without borgmatic. If you get that same behavior, then it's not caused by borgmatic interference. If the behavior differs (e.g. if missing log lines show up), then it might be a borgmatic bug. I believe what I'd expect from Borg after verifying integrity is `check_free_space`. Thinking on this a bit more, I think there are two parts here: what Borg does and what Borgmatic does. Borg's output is a bit cryptic. In the core loop, it's adding files individually and occasionally doing a verification pass. There's no additional feedback with the default arguments. Borgmatic recommends running `borgmatic create --verbosity 1 --files`. I think that the ` --progress` and `--stats` flags may be more useful to users who are new to both Borg and Borgmatic. I have run into another sharp edge now that my first backup has completed, regarding listing files in my archive: ``` $ borg list <repo>::<archive> (works) $ borgmatic list <archive> Error parsing arguments: ... $ borgmatic list <repo>::<archive> Error parsing arguments: ... $ borgmatic borg list <repo>::<archive> Error parsing arguments: ... ``` It would be nice if #2 and #3 worked. #4 looks like it may be an argument parsing bug.

witten commented

2022-06-30 21:23:40 +00:00

Borgmatic recommends running borgmatic create --verbosity 1 --files. I think that the --progress and --stats flags may be more useful to users who are new to both Borg and Borgmatic.

Gotcha. I'll make those more prominent in the docs.

I have run into another sharp edge now that my first backup has completed, regarding listing files in my archive:

borgmatic actually doesn't support Borg-style repository/archive specification (e.g. repo::archive) at all. borgmatic gets the repository implicitly from your configuration (or you can select a single configured repository on some commands with --repository), and gets the archive via --archive.

Perhaps interestingly, Borg 2 will likely drop support for repo::archive-style specification and switch to separate options more similar to borgmatic's command-line!

> Borgmatic recommends running borgmatic create --verbosity 1 --files. I think that the --progress and --stats flags may be more useful to users who are new to both Borg and Borgmatic. Gotcha. I'll make those more prominent in the docs. > I have run into another sharp edge now that my first backup has completed, regarding listing files in my archive: borgmatic actually doesn't support Borg-style repository/archive specification (e.g. `repo::archive`) at all. borgmatic gets the repository implicitly from your configuration (or you can select a single configured repository on some commands with `--repository`), and gets the archive via `--archive`. Perhaps interestingly, Borg 2 will likely drop support for `repo::archive`-style specification and switch to separate options more similar to borgmatic's command-line!

witten commented

2022-06-30 23:35:10 +00:00

Oh, and for number 4 above (borgmatic borg), the behavior is similar: borgmatic passes in your configured repositories (one at a time) by default, and you can override that behavior with --repository or --archive. See the examples here for more info: https://torsion.org/borgmatic/docs/how-to/run-arbitrary-borg-commands/

Oh, and for number 4 above (`borgmatic borg`), the behavior is similar: borgmatic passes in your configured repositories (one at a time) by default, and you can override that behavior with `--repository` or `--archive`. See the examples here for more info: https://torsion.org/borgmatic/docs/how-to/run-arbitrary-borg-commands/

witten referenced this issue from a commit

2022-06-30 23:54:54 +00:00

Add another recommended flag to the backup documentation (#554).

witten commented

2022-07-01 05:17:13 +00:00

Healthchecks fix released in borgmatic 1.6.5!

witten closed this issue

2022-07-04 05:29:48 +00:00

Sign in to join this conversation.