How do I test the exclusions in the Borgmatic config? #680

New Issue

iainelder · 2023-04-18T09:56:45Z

iainelder commented

2023-04-18 09:56:45 +00:00

What I'm trying to do and why

I want to test the exclusion filters in my Borgmatic config without having to create a real backup.

Here is how I test a single exclusion using plain Borg.

borg create --list --log-json --dry-run \
--pattern='-home/isme/.cache' \
-- repo1::arch1 ~ \
2>/tmp/list.jsonl

In the output "-" means "include" and "x" means "exclude".

{"type": "file_status", "status": "-", "path": "/home/isme/.parallel/runs-without-willing-to-cite"}
{"type": "file_status", "status": "-", "path": "/home/isme/.parallel/will-cite"}
...
{"type": "file_status", "status": "x", "path": "/home/isme/.cache"}
{"type": "file_status", "status": "x", "path": "/home/isme/.cache/opera"}
...
{"type": "file_status", "status": "-", "path": "/home/isme/.gnupg/private-keys-v1.d"}
{"type": "file_status", "status": "-", "path": "/home/isme/.gnupg/trustdb.gpg"}
...

My Borgmatic config looks like this:

location:
  source_directories:
    - /home/isme
  repositories:
    - path: ssh://...@....repo.borgbase.com/./repo
      label: BorgBase
  patterns:
    - '- /home/isme/.cache'
storage:
  encryption_passphrase: "..."
retention:
  keep_within: 24H
  keep_daily: 7
  keep_monthly: 6
  keep_yearly: 1

I invoke Borgmatic like this:

borgmatic create --dry-run --list --json --verbosity 2

The output is completely different. No list of files. The only maybe useful thing I see is an invocation of Borg (borg create --dry-run ssh://...@....repo.borgbase.com/./repo::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /home/isme --json).

Ensuring legacy configuration is upgraded
/home/isme/.config/borgmatic/config.yaml: No commands to run for pre-everything hook
borg --version --debug --show-rc
ssh://...@....repo.borgbase.com/./repo: Running actions for repository
/home/isme/.config/borgmatic/config.yaml: No commands to run for pre-actions hook
/home/isme/.config/borgmatic/config.yaml: No commands to run for pre-backup hook
ssh://...@....repo.borgbase.com/./repo: Creating archive (dry run; not making any changes)
ssh://...@....repo.borgbase.com/./repo: Calling postgresql_databases hook function remove_database_dumps
ssh://...@....repo.borgbase.com/./repo: Removing PostgreSQL database dumps (dry run; not actually removing anything)
ssh://...@....repo.borgbase.com/./repo: Calling mysql_databases hook function remove_database_dumps
ssh://...@....repo.borgbase.com/./repo: Removing MySQL database dumps (dry run; not actually removing anything)
ssh://...@....repo.borgbase.com/./repo: Calling mongodb_databases hook function remove_database_dumps
ssh://...@....repo.borgbase.com/./repo: Removing MongoDB database dumps (dry run; not actually removing anything)
ssh://...@....repo.borgbase.com/./repo: Calling sqlite_databases hook function remove_database_dumps
ssh://...@....repo.borgbase.com/./repo: Removing SQLite database dumps (dry run; not actually removing anything)
borg create --dry-run ssh://...@....repo.borgbase.com/./repo::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /home/isme --json
/home/isme/.config/borgmatic/config.yaml: Calling postgresql_databases hook function remove_database_dumps
/home/isme/.config/borgmatic/config.yaml: Removing PostgreSQL database dumps (dry run; not actually removing anything)
/home/isme/.config/borgmatic/config.yaml: Calling mysql_databases hook function remove_database_dumps
/home/isme/.config/borgmatic/config.yaml: Removing MySQL database dumps (dry run; not actually removing anything)
/home/isme/.config/borgmatic/config.yaml: Calling mongodb_databases hook function remove_database_dumps
/home/isme/.config/borgmatic/config.yaml: Removing MongoDB database dumps (dry run; not actually removing anything)
/home/isme/.config/borgmatic/config.yaml: Calling sqlite_databases hook function remove_database_dumps
/home/isme/.config/borgmatic/config.yaml: Removing SQLite database dumps (dry run; not actually removing anything)
/home/isme/.config/borgmatic/config.yaml: No commands to run for post-backup hook
/home/isme/.config/borgmatic/config.yaml: No commands to run for post-actions hook
/home/isme/.config/borgmatic/config.yaml: No commands to run for post-everything hook

summary:
/home/isme/.config/borgmatic/config.yaml: Successfully ran configuration file

What am I missing?

And why am I see so many hook outputs? I haven't configured any.

Environment

borgmatic version: 1.7.10

borgmatic installation method: pipx

Borg version: 1.2.4

Python version: 3.8.10

operating system and version: Ubuntu 20.04.5 LTS

#### What I'm trying to do and why I want to test the exclusion filters in my Borgmatic config without having to create a real backup. Here is how I test a single exclusion using plain Borg. ```bash borg create --list --log-json --dry-run \ --pattern='-home/isme/.cache' \ -- repo1::arch1 ~ \ 2>/tmp/list.jsonl ``` In the output `"-"` means "include" and `"x"` means "exclude". ```json {"type": "file_status", "status": "-", "path": "/home/isme/.parallel/runs-without-willing-to-cite"} {"type": "file_status", "status": "-", "path": "/home/isme/.parallel/will-cite"} ... {"type": "file_status", "status": "x", "path": "/home/isme/.cache"} {"type": "file_status", "status": "x", "path": "/home/isme/.cache/opera"} ... {"type": "file_status", "status": "-", "path": "/home/isme/.gnupg/private-keys-v1.d"} {"type": "file_status", "status": "-", "path": "/home/isme/.gnupg/trustdb.gpg"} ... ``` My Borgmatic config looks like this: ```yaml location: source_directories: - /home/isme repositories: - path: ssh://...@....repo.borgbase.com/./repo label: BorgBase patterns: - '- /home/isme/.cache' storage: encryption_passphrase: "..." retention: keep_within: 24H keep_daily: 7 keep_monthly: 6 keep_yearly: 1 ``` I invoke Borgmatic like this: ```bash borgmatic create --dry-run --list --json --verbosity 2 ``` The output is completely different. No list of files. The only maybe useful thing I see is an invocation of Borg (`borg create --dry-run ssh://...@....repo.borgbase.com/./repo::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /home/isme --json`). ```text Ensuring legacy configuration is upgraded /home/isme/.config/borgmatic/config.yaml: No commands to run for pre-everything hook borg --version --debug --show-rc ssh://...@....repo.borgbase.com/./repo: Running actions for repository /home/isme/.config/borgmatic/config.yaml: No commands to run for pre-actions hook /home/isme/.config/borgmatic/config.yaml: No commands to run for pre-backup hook ssh://...@....repo.borgbase.com/./repo: Creating archive (dry run; not making any changes) ssh://...@....repo.borgbase.com/./repo: Calling postgresql_databases hook function remove_database_dumps ssh://...@....repo.borgbase.com/./repo: Removing PostgreSQL database dumps (dry run; not actually removing anything) ssh://...@....repo.borgbase.com/./repo: Calling mysql_databases hook function remove_database_dumps ssh://...@....repo.borgbase.com/./repo: Removing MySQL database dumps (dry run; not actually removing anything) ssh://...@....repo.borgbase.com/./repo: Calling mongodb_databases hook function remove_database_dumps ssh://...@....repo.borgbase.com/./repo: Removing MongoDB database dumps (dry run; not actually removing anything) ssh://...@....repo.borgbase.com/./repo: Calling sqlite_databases hook function remove_database_dumps ssh://...@....repo.borgbase.com/./repo: Removing SQLite database dumps (dry run; not actually removing anything) borg create --dry-run ssh://...@....repo.borgbase.com/./repo::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /home/isme --json /home/isme/.config/borgmatic/config.yaml: Calling postgresql_databases hook function remove_database_dumps /home/isme/.config/borgmatic/config.yaml: Removing PostgreSQL database dumps (dry run; not actually removing anything) /home/isme/.config/borgmatic/config.yaml: Calling mysql_databases hook function remove_database_dumps /home/isme/.config/borgmatic/config.yaml: Removing MySQL database dumps (dry run; not actually removing anything) /home/isme/.config/borgmatic/config.yaml: Calling mongodb_databases hook function remove_database_dumps /home/isme/.config/borgmatic/config.yaml: Removing MongoDB database dumps (dry run; not actually removing anything) /home/isme/.config/borgmatic/config.yaml: Calling sqlite_databases hook function remove_database_dumps /home/isme/.config/borgmatic/config.yaml: Removing SQLite database dumps (dry run; not actually removing anything) /home/isme/.config/borgmatic/config.yaml: No commands to run for post-backup hook /home/isme/.config/borgmatic/config.yaml: No commands to run for post-actions hook /home/isme/.config/borgmatic/config.yaml: No commands to run for post-everything hook summary: /home/isme/.config/borgmatic/config.yaml: Successfully ran configuration file ``` What am I missing? And why am I see so many hook outputs? I haven't configured any. #### Environment **borgmatic version:** 1.7.10 **borgmatic installation method:** pipx **Borg version:** 1.2.4 **Python version:** 3.8.10 **operating system and version:** Ubuntu 20.04.5 LTS

witten commented

2023-04-18 17:32:13 +00:00

What am I missing?

I think what's going on is that in borgmatic, if --json and --list are used together, the --list flag gets ignored so as not to interfere with the JSON. That's because --list output is not valid JSON and any tools consuming the output will choke on it. Maybe though borgmatic should error when both flags are used together?

In terms of solving your particular need though, you could drop the --json flag and just use --list, and then borgmatic should output all the added and excluded files. Or, if you really do need JSON output, maybe we need to add borgmatic support for --log-json?

And why am I see so many hook outputs? I haven't configured any.

At a high verbosity level (2), borgmatic is showing you absolutely everything it's doing, including the fact that it's removing stale database dumps—which it does regardless of hook configuration to prevent Borg hangs.

> What am I missing? I think what's going on is that in borgmatic, if `--json` and `--list` are used together, the `--list` flag gets ignored so as not to interfere with the JSON. That's because `--list` output is not valid JSON and any tools consuming the output will choke on it. Maybe though borgmatic should error when both flags are used together? In terms of solving your particular need though, you could drop the `--json` flag and just use `--list`, and then borgmatic should output all the added and excluded files. Or, if you really do need JSON output, maybe we need to add borgmatic support for `--log-json`? > And why am I see so many hook outputs? I haven't configured any. At a high verbosity level (2), borgmatic is showing you absolutely everything it's doing, including the fact that it's removing stale database dumps—which it does regardless of hook configuration to prevent Borg hangs.

iainelder commented

2023-04-19 09:39:24 +00:00

maybe we need to add borgmatic support for --log-json

That would be great. The --log-json option was what I was trying to activate.

The problem with the basic list format is that file names can contain newlines.

This command generates the basic list using plain Borg.

borg create --dry-run --list \
--pattern='-home/isme/.cache' \
-- repo1::arch1 ~ \
2>/tmp/borg-list.txt

I use the next command to count the number of inclusions - and exclusions x. It reads the first character on each output line and counts the unique characters. But because of a rogue filename my data contains an extra C character! (The C follows the newline in the filename.)

$ cat /tmp/borg-list.txt | cut --characters=1 | sort | uniq --count
1021532 -
      1 C
 136174 x

To avoid that this command generates a JSON list using plain Borg.

borg create --dry-run --list --log-json \
--pattern='-home/isme/.cache' \
-- repo1::arch1 ~ \
2>/tmp/borg-list.jsonl

The output is JSON lines. Each newline terminates a JSON-formatted record. Any newlines in the file names are now encoded in a JSON string. So I can count the number of inclusions and exclusions like this.

$ cat /tmp/borg-list.jsonl | jq --raw-output '.status' | sort | uniq --count
1021532 -
 136173 x

And I can count how many file names contain a newline.

$ cat /tmp/borg-list.jsonl | grep '\\n' | wc --lines
1

you could drop the --json flag and just use --list

Thanks. This does produce something useful when I add --no-color.

borgmatic create --dry-run --list --no-color > /tmp/matic-list.txt

I use the same command to count the inclusions and exclusions.

$ cat /tmp/matic-list.txt | cut --characters=1 | sort | uniq --count
1021575 -
      1 C

Borgmatic doesn't print any exclusions. So even with this output, it's still hard to see exactly what is being excluded.

Do I need to set something else to print the exlusions?

> maybe we need to add borgmatic support for `--log-json` That would be great. The `--log-json` option was what I was trying to activate. The problem with the basic list format is that file names can contain newlines. This command generates the basic list using plain Borg. ```bash borg create --dry-run --list \ --pattern='-home/isme/.cache' \ -- repo1::arch1 ~ \ 2>/tmp/borg-list.txt ``` I use the next command to count the number of inclusions `-` and exclusions `x`. It reads the first character on each output line and counts the unique characters. But because of a rogue filename my data contains an extra `C` character! (The `C` follows the newline in the filename.) ```console $ cat /tmp/borg-list.txt | cut --characters=1 | sort | uniq --count 1021532 - 1 C 136174 x ``` To avoid that this command generates a JSON list using plain Borg. ```bash borg create --dry-run --list --log-json \ --pattern='-home/isme/.cache' \ -- repo1::arch1 ~ \ 2>/tmp/borg-list.jsonl ``` The output is [JSON lines](https://jsonlines.org/). Each newline terminates a JSON-formatted record. Any newlines in the file names are now encoded in a JSON string. So I can count the number of inclusions and exclusions like this. ```console $ cat /tmp/borg-list.jsonl | jq --raw-output '.status' | sort | uniq --count 1021532 - 136173 x ``` And I can count how many file names contain a newline. ```console $ cat /tmp/borg-list.jsonl | grep '\\n' | wc --lines 1 ``` --- > you could drop the `--json` flag and just use `--list` Thanks. This does produce something useful when I add `--no-color`. ```bash borgmatic create --dry-run --list --no-color > /tmp/matic-list.txt ``` I use the same command to count the inclusions and exclusions. ```console $ cat /tmp/matic-list.txt | cut --characters=1 | sort | uniq --count 1021575 - 1 C ``` Borgmatic doesn't print any exclusions. So even with this output, it's still hard to see exactly what is being excluded. Do I need to set something else to print the exlusions?

witten referenced this issue from a commit

2023-05-04 00:17:17 +00:00

Error if --list is used with --json for create action (#680).

witten referenced this issue from a commit

2023-05-09 06:02:23 +00:00

Add support for logging each log line as a JSON object via global "--log-json" flag (#680).

witten commented

2023-05-09 06:05:07 +00:00

--log-json is implemented now in main and will be part of the next release!

Do I need to set something else to print the exclusions?

As of borgmatic 1.7.8, excluded files are only shown with --list when at verbosity level 2 (e.g. --verbosity 2).

`--log-json` is implemented now in main and will be part of the next release! > Do I need to set something else to print the exclusions? As of borgmatic 1.7.8, excluded files are only shown with `--list` when at verbosity level 2 (e.g. `--verbosity 2`).

witten closed this issue

2023-05-09 06:05:07 +00:00

witten commented

2023-05-16 17:04:50 +00:00

This was just released in borgmatic 1.7.13!

iainelder commented

2023-05-16 19:31:04 +00:00

excluded files are only shown with --list when at verbosity level 2

Got it. That's a bit surprising because --verbosity=2 also adds a lot of output I don't need. Nonetheless, it does give me a solution.

Now with Borgmatic 1.7.13 I can count the number of inclusions and exclusions like this:

borgmatic create --dry-run --list --log-json --no-color --verbosity=2 \
> /tmp/matic-list.jsonl

cat /tmp/matic-list.jsonl \
| sed --quiet '/^{/p' \
| jq --raw-output 'select(.type == "file_status") | .status' \
| sort \
| uniq --count

The sed command prints only the JSON lines, dropping all the other stuff generated by the verbosity setting.

Thanks a lot for addressing this. Now I have a robust way to test the configured Borgmatic exclusions.

> excluded files are only shown with `--list` when at verbosity level 2 Got it. That's a bit surprising because `--verbosity=2` also adds a lot of output I don't need. Nonetheless, it does give me a solution. Now with Borgmatic 1.7.13 I can count the number of inclusions and exclusions like this: ```bash borgmatic create --dry-run --list --log-json --no-color --verbosity=2 \ > /tmp/matic-list.jsonl cat /tmp/matic-list.jsonl \ | sed --quiet '/^{/p' \ | jq --raw-output 'select(.type == "file_status") | .status' \ | sort \ | uniq --count ``` The `sed` command prints only the JSON lines, dropping all the other stuff generated by the verbosity setting. Thanks a lot for addressing this. Now I have a robust way to test the configured Borgmatic exclusions.

witten commented

2023-05-16 19:34:10 +00:00

I'm glad to hear it's working for you!

Sign in to join this conversation.