Fix typo in NEWS (#1044).

Reviewed-on: #1041

NEWS

@@ -1,3 +1,162 @@
+2.0.0.dev0
+ * #345: Add a "key import" action to import a repository key from backup.
+ * #422: Add home directory expansion to file-based and KeePassXC credential hooks.
+ * #790, #821: Deprecate all "before_*", "after_*" and "on_error" command hooks in favor of more
+   flexible "commands:". See the documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/
+ * #790: BREAKING: For both new and deprecated command hooks, run a configured "after" hook even if
+   an error occurs first. This allows you to perform cleanup steps that correspond to "before"
+   preparation commands—even when something goes wrong.
+ * #790: BREAKING: Run all command hooks (both new and deprecated) respecting the
+   "working_directory" option if configured, meaning that hook commands are run in that directory.
+ * #836: Add a custom command option for the SQLite hook.
+ * #837: Add custom command options for the MongoDB hook.
+ * #1010: When using Borg 2, don't pass the "--stats" flag to "borg prune".
+ * #1020: Document a database use case involving a temporary database client container:
+   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#containers
+ * #1037: Fix an error with the "extract" action when both a remote repository and a
+   "working_directory" are used.
+ * #1044: Fix an error in the systemd credential hook when the credential name contains a "."
+   character.
+
+1.9.14
+ * #409: With the PagerDuty monitoring hook, send borgmatic logs to PagerDuty so they show up in the
+   incident UI. See the documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#pagerduty-hook
+ * #936: Clarify Zabbix monitoring hook documentation about creating items:
+   https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#zabbix-hook
+ * #1017: Fix a regression in which some MariaDB/MySQL passwords were not escaped correctly.
+ * #1021: Fix a regression in which the "exclude_patterns" option didn't expand "~" (the user's
+   home directory). This fix means that all "patterns" and "patterns_from" also now expand "~".
+ * #1023: Fix an error in the Btrfs hook when attempting to snapshot a read-only subvolume. Now,
+   read-only subvolumes are ignored since Btrfs can't actually snapshot them.
+
+1.9.13
+ * #975: Add a "compression" option to the PostgreSQL database hook.
+ * #1001: Fix a ZFS error during snapshot cleanup.
+ * #1003: In the Zabbix monitoring hook, support Zabbix 7.2's authentication changes.
+ * #1009: Send database passwords to MariaDB and MySQL via anonymous pipe, which is more secure than
+   using an environment variable.
+ * #1013: Send database passwords to MongoDB via anonymous pipe, which is more secure than using
+   "--password" on the command-line!
+ * #1015: When ctrl-C is pressed, more strongly encourage Borg to actually exit.
+ * Add a "verify_tls" option to the Uptime Kuma monitoring hook for disabling TLS verification.
+ * Add "tls" options to the MariaDB and MySQL database hooks to enable or disable TLS encryption
+   between client and server.
+
+1.9.12
+ * #1005: Fix the credential hooks to avoid using Python 3.12+ string features. Now borgmatic will
+   work with Python 3.9, 3.10, and 3.11 again.
+
+1.9.11
+ * #795: Add credential loading from file, KeePassXC, and Docker/Podman secrets. See the
+   documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/provide-your-passwords/
+ * #996: Fix the "create" action to omit the repository label prefix from Borg's output when
+   databases are enabled.
+ * #998: Send the "encryption_passphrase" option to Borg via an anonymous pipe, which is more secure
+   than using an environment variable.
+ * #999: Fix a runtime directory error from a conflict between "extra_borg_options" and special file
+   detection.
+ * #1001: For the ZFS, Btrfs, and LVM hooks, only make snapshots for root patterns that come from
+   a borgmatic configuration option (e.g. "source_directories")—not from other hooks within
+   borgmatic.
+ * #1001: Fix a ZFS/LVM error due to colliding snapshot mount points for nested datasets or logical
+   volumes.
+ * #1001: Don't try to snapshot ZFS datasets that have the "canmount=off" property.
+ * Fix another error in the Btrfs hook when a subvolume mounted at "/" is configured in borgmatic's
+   source directories.
+
+1.9.10
+ * #966: Add a "{credential ...}" syntax for loading systemd credentials into borgmatic
+   configuration files. See the documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/provide-your-passwords/
+ * #987: Fix a "list" action error when the "encryption_passcommand" option is set.
+ * #987: When both "encryption_passcommand" and "encryption_passphrase" are configured, prefer
+   "encryption_passphrase" even if it's an empty value.
+ * #988: With the "max_duration" option or the "--max-duration" flag, run the archives and
+   repository checks separately so they don't interfere with one another. Previously, borgmatic
+   refused to run checks in this situation.
+ * #989: Fix the log message code to avoid using Python 3.10+ logging features. Now borgmatic will
+   work with Python 3.9 again.
+ * Capture and delay any log records produced before logging is fully configured, so early log
+   records don't get lost.
+ * Add support for Python 3.13.
+
+1.9.9
+ * #635: Log the repository path or label on every relevant log message, not just some logs.
+ * #961: When the "encryption_passcommand" option is set, call the command once from borgmatic to
+   collect the encryption passphrase and then pass it to Borg multiple times. See the documentation
+   for more information: https://torsion.org/borgmatic/docs/how-to/provide-your-passwords/
+ * #981: Fix a "spot" check file count delta error.
+ * #982: Fix for borgmatic "exclude_patterns" and "exclude_from" recursing into excluded
+   subdirectories.
+ * #983: Fix the Btrfs hook to support subvolumes with names like "@home" different from their
+   mount points.
+ * #985: Change the default value for the "--original-hostname" flag from "localhost" to no host
+   specified. This way, the "restore" action works without a hostname if there's a single matching
+   database dump.
+
+1.9.8
+ * #979: Fix root patterns so they don't have an invalid "sh:" prefix before getting passed to Borg.
+ * Expand the recent contributors documentation section to include ticket submitters—not just code
+   contributors—because there are multiple ways to contribute to the project! See:
+   https://torsion.org/borgmatic/#recent-contributors
+
+1.9.7
+ * #855: Add a Sentry monitoring hook. See the documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#sentry-hook
+ * #968: Fix for a "spot" check error when a filename in the most recent archive contains a newline.
+ * #970: Fix for an error when there's a blank line in the configured patterns or excludes.
+ * #971: Fix for "exclude_from" files being completely ignored.
+ * #977: Fix for "exclude_patterns" and "exclude_from" not supporting explicit pattern styles (e.g.,
+   "sh:" or "re:").
+
+1.9.6
+ * #959: Fix an error in the Btrfs hook when a subvolume mounted at "/" is configured in borgmatic's
+   source directories.
+ * #960: Fix for archives storing relative source directory paths such that they contain the working
+   directory.
+ * #960: Fix the "spot" check to support relative source directory paths.
+ * #962: For the ZFS, Btrfs, and LVM hooks, perform path rewriting for excludes and patterns in
+   addition to the existing source directories rewriting.
+ * #962: Under the hood, merge all configured source directories, excludes, and patterns into a
+   unified temporary patterns file for passing to Borg. The borgmatic configuration options remain
+   unchanged.
+ * #962: For the LVM hook, add support for nested logical volumes.
+ * #965: Fix a borgmatic runtime directory error when running the "spot" check with a database hook
+   enabled.
+ * #969: Fix the "restore" action to work on database dumps without a port when a default port is
+   present in configuration.
+ * Fix the "spot" check to no longer consider pipe files within an archive for file comparisons.
+ * Fix the "spot" check to have a nicer error when there are no source paths to compare.
+ * Fix auto-excluding of special files (when databases are configured) to support relative source
+   directory paths.
+ * Drop support for Python 3.8, which has been end-of-lifed.
+
+1.9.5
+ * #418: Backup and restore databases that have the same name but with different ports, hostnames,
+   or hooks.
+ * #947: To avoid a hang in the database hooks, error and exit when the borgmatic runtime
+   directory overlaps with the configured excludes.
+ * #954: Fix a findmnt command error in the Btrfs hook by switching to parsing JSON output.
+ * #956: Fix the printing of a color reset code even when color is disabled.
+ * #958: Drop colorama as a library dependency.
+ * When the ZFS, Btrfs, or LVM hooks aren't configured, don't try to cleanup snapshots for them.
+
+1.9.4
+ * #80 (beta): Add an LVM hook for snapshotting and backing up LVM logical volumes. See the
+   documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/snapshot-your-filesystems/
+ * #251 (beta): Add a Btrfs hook for snapshotting and backing up Btrfs subvolumes. See the
+   documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/snapshot-your-filesystems/
+ * #926: Fix a library error when running within a PyInstaller bundle.
+ * #950: Fix a snapshot unmount error in the ZFS hook when using nested datasets.
+ * Update the ZFS hook to discover and snapshot ZFS datasets even if they are parent/grandparent
+   directories of your source directories.
+ * Reorganize data source and monitoring hooks to make developing new hooks easier.
+
 1.9.3
  * #261 (beta): Add a ZFS hook for snapshotting and backing up ZFS datasets. See the documentation
    for more information: https://torsion.org/borgmatic/docs/how-to/snapshot-your-filesystems/

README.md

@@ -56,13 +56,22 @@ borgmatic is powered by [Borg Backup](https://www.borgbackup.org/).
 
 ## Integrations
 
+### Data
+
 <a href="https://www.postgresql.org/"><img src="docs/static/postgresql.png" alt="PostgreSQL" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://www.mysql.com/"><img src="docs/static/mysql.png" alt="MySQL" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://mariadb.com/"><img src="docs/static/mariadb.png" alt="MariaDB" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://www.mongodb.com/"><img src="docs/static/mongodb.png" alt="MongoDB" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://sqlite.org/"><img src="docs/static/sqlite.png" alt="SQLite" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://openzfs.org/"><img src="docs/static/openzfs.png" alt="OpenZFS" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://btrfs.readthedocs.io/"><img src="docs/static/btrfs.png" alt="Btrfs" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://sourceware.org/lvm2/"><img src="docs/static/lvm.png" alt="LVM" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://rclone.org"><img src="docs/static/rclone.png" alt="rclone" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.borgbase.com/?utm_source=borgmatic"><img src="docs/static/borgbase.png" alt="BorgBase" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+
+
+### Monitoring
+
 <a href="https://healthchecks.io/"><img src="docs/static/healthchecks.png" alt="Healthchecks" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://uptime.kuma.pet/"><img src="docs/static/uptimekuma.png" alt="Uptime Kuma" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://cronitor.io/"><img src="docs/static/cronitor.png" alt="Cronitor" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
@@ -73,7 +82,15 @@ borgmatic is powered by [Borg Backup](https://www.borgbackup.org/).
 <a href="https://grafana.com/oss/loki/"><img src="docs/static/loki.png" alt="Loki" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://github.com/caronc/apprise/wiki"><img src="docs/static/apprise.png" alt="Apprise" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://www.zabbix.com/"><img src="docs/static/zabbix.png" alt="Zabbix" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://www.borgbase.com/?utm_source=borgmatic"><img src="docs/static/borgbase.png" alt="BorgBase" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://sentry.io/"><img src="docs/static/sentry.png" alt="Sentry" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+
+
+### Credentials
+
+<a href="https://systemd.io/"><img src="docs/static/systemd.png" alt="Sentry" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.docker.com/"><img src="docs/static/docker.png" alt="Docker" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://podman.io/"><img src="docs/static/podman.png" alt="Podman" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://keepassxc.org/"><img src="docs/static/keepassxc.png" alt="Podman" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
 
 
 ## Getting started
@@ -162,4 +179,8 @@ info on cloning source code, running tests, etc.
 
 ### Recent contributors
 
+Thanks to all borgmatic contributors! There are multiple ways to contribute to
+this project, so the following includes those who have fixed bugs, contributed
+features, *or* filed tickets.
+
 {% include borgmatic/contributors.html %}

borgmatic/actions/borg.py

@@ -22,9 +22,7 @@ def run_borg(
     if borg_arguments.repository is None or borgmatic.config.validate.repositories_match(
         repository, borg_arguments.repository
     ):
-        logger.info(
-            f'{repository.get("label", repository["path"])}: Running arbitrary Borg command'
-        )
+        logger.info('Running arbitrary Borg command')
         archive_name = borgmatic.borg.repo_list.resolve_archive_name(
             repository['path'],
             borg_arguments.archive,

borgmatic/actions/break_lock.py

@@ -21,9 +21,7 @@ def run_break_lock(
     if break_lock_arguments.repository is None or borgmatic.config.validate.repositories_match(
         repository, break_lock_arguments.repository
     ):
-        logger.info(
-            f'{repository.get("label", repository["path"])}: Breaking repository and cache locks'
-        )
+        logger.info('Breaking repository and cache locks')
         borgmatic.borg.break_lock.break_lock(
             repository['path'],
             config,

borgmatic/actions/change_passphrase.py

@@ -16,7 +16,7 @@ def run_change_passphrase(
     remote_path,
 ):
     '''
-    Run the "key change-passprhase" action for the given repository.
+    Run the "key change-passphrase" action for the given repository.
     '''
     if (
         change_passphrase_arguments.repository is None
@@ -24,9 +24,7 @@ def run_change_passphrase(
             repository, change_passphrase_arguments.repository
         )
     ):
-        logger.info(
-            f'{repository.get("label", repository["path"])}: Changing repository passphrase'
-        )
+        logger.info('Changing repository passphrase')
         borgmatic.borg.change_passphrase.change_passphrase(
             repository['path'],
             config,

borgmatic/actions/check.py

@@ -363,18 +363,19 @@ def collect_spot_check_source_paths(
         borgmatic.hooks.dispatch.call_hooks(
             'use_streaming',
             config,
-            repository['path'],
-            borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES,
+            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
         ).values()
     )
+    working_directory = borgmatic.config.paths.get_working_directory(config)
 
-    (create_flags, create_positional_arguments, pattern_file, exclude_file) = (
+    (create_flags, create_positional_arguments, pattern_file) = (
         borgmatic.borg.create.make_base_create_command(
             dry_run=True,
             repository_path=repository['path'],
             config=config,
-            source_directories=borgmatic.actions.create.process_source_directories(
-                config,
+            patterns=borgmatic.actions.create.process_patterns(
+                borgmatic.actions.create.collect_patterns(config),
+                working_directory,
             ),
             local_borg_version=local_borg_version,
             global_arguments=global_arguments,
@@ -385,13 +386,12 @@ def collect_spot_check_source_paths(
             stream_processes=stream_processes,
         )
     )
-    borg_environment = borgmatic.borg.environment.make_environment(config)
     working_directory = borgmatic.config.paths.get_working_directory(config)
 
     paths_output = borgmatic.execute.execute_command_and_capture_output(
         create_flags + create_positional_arguments,
         capture_stderr=True,
-        extra_environment=borg_environment,
+        environment=borgmatic.borg.environment.make_environment(config),
         working_directory=working_directory,
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),
@@ -399,7 +399,7 @@ def collect_spot_check_source_paths(
 
     paths = tuple(
         path_line.split(' ', 1)[1]
-        for path_line in paths_output.split('\n')
+        for path_line in paths_output.splitlines()
         if path_line and path_line.startswith('- ') or path_line.startswith('+ ')
     )
 
@@ -409,6 +409,7 @@ def collect_spot_check_source_paths(
 
 
 BORG_DIRECTORY_FILE_TYPE = 'd'
+BORG_PIPE_FILE_TYPE = 'p'
 
 
 def collect_spot_check_archive_paths(
@@ -426,6 +427,9 @@ def collect_spot_check_archive_paths(
     local Borg version, global arguments as an argparse.Namespace instance, the local Borg path, the
     remote Borg path, and the borgmatic runtime directory, collect the paths from the given archive
     (but only include files and symlinks and exclude borgmatic runtime directories).
+
+    These paths do not have a leading slash, as that's how Borg stores them. As a result, we don't
+    know whether they came from absolute or relative source directories.
     '''
     borgmatic_source_directory = borgmatic.config.paths.get_borgmatic_source_directory(config)
 
@@ -437,15 +441,17 @@ def collect_spot_check_archive_paths(
             config,
             local_borg_version,
             global_arguments,
-            path_format='{type} /{path}{NL}',  # noqa: FS003
+            path_format='{type} {path}{NUL}',  # noqa: FS003
             local_path=local_path,
             remote_path=remote_path,
         )
         for (file_type, path) in (line.split(' ', 1),)
-        if file_type != BORG_DIRECTORY_FILE_TYPE
-        if pathlib.Path('/borgmatic') not in pathlib.Path(path).parents
-        if pathlib.Path(borgmatic_source_directory) not in pathlib.Path(path).parents
-        if pathlib.Path(borgmatic_runtime_directory) not in pathlib.Path(path).parents
+        if file_type not in (BORG_DIRECTORY_FILE_TYPE, BORG_PIPE_FILE_TYPE)
+        if pathlib.Path('borgmatic') not in pathlib.Path(path).parents
+        if pathlib.Path(borgmatic_source_directory.lstrip(os.path.sep))
+        not in pathlib.Path(path).parents
+        if pathlib.Path(borgmatic_runtime_directory.lstrip(os.path.sep))
+        not in pathlib.Path(path).parents
     )
 
 
@@ -460,15 +466,14 @@ def compare_spot_check_hashes(
     global_arguments,
     local_path,
     remote_path,
-    log_prefix,
     source_paths,
 ):
     '''
     Given a repository configuration dict, the name of the latest archive, a configuration dict, the
     local Borg version, global arguments as an argparse.Namespace instance, the local Borg path, the
-    remote Borg path, a log label, and spot check source paths, compare the hashes for a sampling of
-    the source paths with hashes from corresponding paths in the given archive. Return a sequence of
-    the paths that fail that hash comparison.
+    remote Borg path, and spot check source paths, compare the hashes for a sampling of the source
+    paths with hashes from corresponding paths in the given archive. Return a sequence of the paths
+    that fail that hash comparison.
     '''
     # Based on the configured sample percentage, come up with a list of random sample files from the
     # source directories.
@@ -484,7 +489,7 @@ def compare_spot_check_hashes(
         if os.path.exists(os.path.join(working_directory or '', source_path))
     }
     logger.debug(
-        f'{log_prefix}: Sampling {sample_count} source paths (~{spot_check_config["data_sample_percentage"]}%) for spot check'
+        f'Sampling {sample_count} source paths (~{spot_check_config["data_sample_percentage"]}%) for spot check'
     )
 
     source_sample_paths_iterator = iter(source_sample_paths)
@@ -532,7 +537,7 @@ def compare_spot_check_hashes(
                     local_borg_version,
                     global_arguments,
                     list_paths=source_sample_paths_subset,
-                    path_format='{xxh64} /{path}{NL}',  # noqa: FS003
+                    path_format='{xxh64} {path}{NUL}',  # noqa: FS003
                     local_path=local_path,
                     remote_path=remote_path,
                 )
@@ -544,7 +549,7 @@ def compare_spot_check_hashes(
     failing_paths = []
 
     for path, source_hash in source_hashes.items():
-        archive_hash = archive_hashes.get(path)
+        archive_hash = archive_hashes.get(path.lstrip(os.path.sep))
 
         if archive_hash is not None and archive_hash == source_hash:
             continue
@@ -572,8 +577,7 @@ def spot_check(
     disk to those stored in the latest archive. If any differences are beyond configured tolerances,
     then the check fails.
     '''
-    log_prefix = f'{repository.get("label", repository["path"])}'
-    logger.debug(f'{log_prefix}: Running spot check')
+    logger.debug('Running spot check')
 
     try:
         spot_check_config = next(
@@ -596,7 +600,7 @@ def spot_check(
         remote_path,
         borgmatic_runtime_directory,
     )
-    logger.debug(f'{log_prefix}: {len(source_paths)} total source paths for spot check')
+    logger.debug(f'{len(source_paths)} total source paths for spot check')
 
     archive = borgmatic.borg.repo_list.resolve_archive_name(
         repository['path'],
@@ -607,7 +611,7 @@ def spot_check(
         local_path,
         remote_path,
     )
-    logger.debug(f'{log_prefix}: Using archive {archive} for spot check')
+    logger.debug(f'Using archive {archive} for spot check')
 
     archive_paths = collect_spot_check_archive_paths(
         repository,
@@ -619,18 +623,27 @@ def spot_check(
         remote_path,
         borgmatic_runtime_directory,
     )
-    logger.debug(f'{log_prefix}: {len(archive_paths)} total archive paths for spot check')
+    logger.debug(f'{len(archive_paths)} total archive paths for spot check')
+
+    if len(source_paths) == 0:
+        logger.debug(
+            f'Paths in latest archive but not source paths: {", ".join(set(archive_paths)) or "none"}'
+        )
+        raise ValueError(
+            'Spot check failed: There are no source paths to compare against the archive'
+        )
 
     # Calculate the percentage delta between the source paths count and the archive paths count, and
     # compare that delta to the configured count tolerance percentage.
     count_delta_percentage = abs(len(source_paths) - len(archive_paths)) / len(source_paths) * 100
 
     if count_delta_percentage > spot_check_config['count_tolerance_percentage']:
+        rootless_source_paths = set(path.lstrip(os.path.sep) for path in source_paths)
         logger.debug(
-            f'{log_prefix}: Paths in source paths but not latest archive: {", ".join(set(source_paths) - set(archive_paths)) or "none"}'
+            f'Paths in source paths but not latest archive: {", ".join(rootless_source_paths - set(archive_paths)) or "none"}'
         )
         logger.debug(
-            f'{log_prefix}: Paths in latest archive but not source paths: {", ".join(set(archive_paths) - set(source_paths)) or "none"}'
+            f'Paths in latest archive but not source paths: {", ".join(set(archive_paths) - rootless_source_paths) or "none"}'
         )
         raise ValueError(
             f'Spot check failed: {count_delta_percentage:.2f}% file count delta between source paths and latest archive (tolerance is {spot_check_config["count_tolerance_percentage"]}%)'
@@ -644,25 +657,24 @@ def spot_check(
         global_arguments,
         local_path,
         remote_path,
-        log_prefix,
         source_paths,
     )
 
     # Error if the percentage of failing hashes exceeds the configured tolerance percentage.
-    logger.debug(f'{log_prefix}: {len(failing_paths)} non-matching spot check hashes')
+    logger.debug(f'{len(failing_paths)} non-matching spot check hashes')
     data_tolerance_percentage = spot_check_config['data_tolerance_percentage']
     failing_percentage = (len(failing_paths) / len(source_paths)) * 100
 
     if failing_percentage > data_tolerance_percentage:
         logger.debug(
-            f'{log_prefix}: Source paths with data not matching the latest archive: {", ".join(failing_paths)}'
+            f'Source paths with data not matching the latest archive: {", ".join(failing_paths)}'
         )
         raise ValueError(
             f'Spot check failed: {failing_percentage:.2f}% of source paths with data not matching the latest archive (tolerance is {data_tolerance_percentage}%)'
         )
 
     logger.info(
-        f'{log_prefix}: Spot check passed with a {count_delta_percentage:.2f}% file count delta and a {failing_percentage:.2f}% file data delta'
+        f'Spot check passed with a {count_delta_percentage:.2f}% file count delta and a {failing_percentage:.2f}% file data delta'
     )
 
 
@@ -670,7 +682,6 @@ def run_check(
     config_filename,
     repository,
     config,
-    hook_context,
     local_borg_version,
     check_arguments,
     global_arguments,
@@ -687,17 +698,7 @@ def run_check(
     ):
         return
 
-    borgmatic.hooks.command.execute_hook(
-        config.get('before_check'),
-        config.get('umask'),
-        config_filename,
-        'pre-check',
-        global_arguments.dry_run,
-        **hook_context,
-    )
-
-    log_prefix = repository.get('label', repository['path'])
-    logger.info(f'{log_prefix}: Running consistency checks')
+    logger.info('Running consistency checks')
 
     repository_id = borgmatic.borg.check.get_repository_id(
         repository['path'],
@@ -750,9 +751,7 @@ def run_check(
         write_check_time(make_check_time_path(config, repository_id, 'extract'))
 
     if 'spot' in checks:
-        with borgmatic.config.paths.Runtime_directory(
-            config, log_prefix
-        ) as borgmatic_runtime_directory:
+        with borgmatic.config.paths.Runtime_directory(config) as borgmatic_runtime_directory:
             spot_check(
                 repository,
                 config,
@@ -763,12 +762,3 @@ def run_check(
                 borgmatic_runtime_directory,
             )
         write_check_time(make_check_time_path(config, repository_id, 'spot'))
-
-    borgmatic.hooks.command.execute_hook(
-        config.get('after_check'),
-        config.get('umask'),
-        config_filename,
-        'post-check',
-        global_arguments.dry_run,
-        **hook_context,
-    )

borgmatic/actions/compact.py

@@ -12,7 +12,6 @@ def run_compact(
     config_filename,
     repository,
     config,
-    hook_context,
     local_borg_version,
     compact_arguments,
     global_arguments,
@@ -28,18 +27,8 @@ def run_compact(
     ):
         return
 
-    borgmatic.hooks.command.execute_hook(
-        config.get('before_compact'),
-        config.get('umask'),
-        config_filename,
-        'pre-compact',
-        global_arguments.dry_run,
-        **hook_context,
-    )
     if borgmatic.borg.feature.available(borgmatic.borg.feature.Feature.COMPACT, local_borg_version):
-        logger.info(
-            f'{repository.get("label", repository["path"])}: Compacting segments{dry_run_label}'
-        )
+        logger.info(f'Compacting segments{dry_run_label}')
         borgmatic.borg.compact.compact_segments(
             global_arguments.dry_run,
             repository['path'],
@@ -53,14 +42,4 @@ def run_compact(
             threshold=compact_arguments.threshold,
         )
     else:  # pragma: nocover
-        logger.info(
-            f'{repository.get("label", repository["path"])}: Skipping compact (only available/needed in Borg 1.2+)'
-        )
-    borgmatic.hooks.command.execute_hook(
-        config.get('after_compact'),
-        config.get('umask'),
-        config_filename,
-        'post-compact',
-        global_arguments.dry_run,
-        **hook_context,
-    )
+        logger.info('Skipping compact (only available/needed in Borg 1.2+)')

borgmatic/actions/config/bootstrap.py

@@ -41,11 +41,10 @@ def get_config_paths(archive_name, bootstrap_arguments, global_arguments, local_
     config = make_bootstrap_config(bootstrap_arguments)
 
     # Probe for the manifest file in multiple locations, as the default location has moved to the
-    # borgmatic runtime directory (which get stored as just "/borgmatic" with Borg 1.4+). But we
+    # borgmatic runtime directory (which gets stored as just "/borgmatic" with Borg 1.4+). But we
     # still want to support reading the manifest from previously created archives as well.
     with borgmatic.config.paths.Runtime_directory(
         {'user_runtime_directory': bootstrap_arguments.user_runtime_directory},
-        bootstrap_arguments.repository,
     ) as borgmatic_runtime_directory:
         for base_directory in (
             'borgmatic',

borgmatic/actions/create.py

@@ -6,141 +6,264 @@ import pathlib
 
 import borgmatic.actions.json
 import borgmatic.borg.create
+import borgmatic.borg.pattern
 import borgmatic.config.paths
 import borgmatic.config.validate
 import borgmatic.hooks.command
 import borgmatic.hooks.dispatch
-import borgmatic.hooks.dump
 
 logger = logging.getLogger(__name__)
 
 
+def parse_pattern(pattern_line, default_style=borgmatic.borg.pattern.Pattern_style.NONE):
+    '''
+    Given a Borg pattern as a string, parse it into a borgmatic.borg.pattern.Pattern instance and
+    return it.
+    '''
+    try:
+        (pattern_type, remainder) = pattern_line.split(' ', maxsplit=1)
+    except ValueError:
+        raise ValueError(f'Invalid pattern: {pattern_line}')
+
+    try:
+        (parsed_pattern_style, path) = remainder.split(':', maxsplit=1)
+        pattern_style = borgmatic.borg.pattern.Pattern_style(parsed_pattern_style)
+    except ValueError:
+        pattern_style = default_style
+        path = remainder
+
+    return borgmatic.borg.pattern.Pattern(
+        path,
+        borgmatic.borg.pattern.Pattern_type(pattern_type),
+        borgmatic.borg.pattern.Pattern_style(pattern_style),
+        source=borgmatic.borg.pattern.Pattern_source.CONFIG,
+    )
+
+
+def collect_patterns(config):
+    '''
+    Given a configuration dict, produce a single sequence of patterns comprised of the configured
+    source directories, patterns, excludes, pattern files, and exclude files.
+
+    The idea is that Borg has all these different ways of specifying includes, excludes, source
+    directories, etc., but we'd like to collapse them all down to one common format (patterns) for
+    ease of manipulation within borgmatic.
+    '''
+    try:
+        return (
+            tuple(
+                borgmatic.borg.pattern.Pattern(
+                    source_directory, source=borgmatic.borg.pattern.Pattern_source.CONFIG
+                )
+                for source_directory in config.get('source_directories', ())
+            )
+            + tuple(
+                parse_pattern(pattern_line.strip())
+                for pattern_line in config.get('patterns', ())
+                if not pattern_line.lstrip().startswith('#')
+                if pattern_line.strip()
+            )
+            + tuple(
+                parse_pattern(
+                    f'{borgmatic.borg.pattern.Pattern_type.NO_RECURSE.value} {exclude_line.strip()}',
+                    borgmatic.borg.pattern.Pattern_style.FNMATCH,
+                )
+                for exclude_line in config.get('exclude_patterns', ())
+            )
+            + tuple(
+                parse_pattern(pattern_line.strip())
+                for filename in config.get('patterns_from', ())
+                for pattern_line in open(filename).readlines()
+                if not pattern_line.lstrip().startswith('#')
+                if pattern_line.strip()
+            )
+            + tuple(
+                parse_pattern(
+                    f'{borgmatic.borg.pattern.Pattern_type.NO_RECURSE.value} {exclude_line.strip()}',
+                    borgmatic.borg.pattern.Pattern_style.FNMATCH,
+                )
+                for filename in config.get('exclude_from', ())
+                for exclude_line in open(filename).readlines()
+                if not exclude_line.lstrip().startswith('#')
+                if exclude_line.strip()
+            )
+        )
+    except (FileNotFoundError, OSError) as error:
+        logger.debug(error)
+
+        raise ValueError(f'Cannot read patterns_from/exclude_from file: {error.filename}')
+
+
 def expand_directory(directory, working_directory):
     '''
     Given a directory path, expand any tilde (representing a user's home directory) and any globs
     therein. Return a list of one or more resulting paths.
-    '''
-    expanded_directory = os.path.join(working_directory or '', os.path.expanduser(directory))
 
-    return glob.glob(expanded_directory) or [expanded_directory]
+    Take into account the given working directory so that relative paths are supported.
+    '''
+    expanded_directory = os.path.expanduser(directory)
+
+    # This would be a lot easier to do with glob(..., root_dir=working_directory), but root_dir is
+    # only available in Python 3.10+.
+    normalized_directory = os.path.join(working_directory or '', expanded_directory)
+    glob_paths = glob.glob(normalized_directory)
+
+    if not glob_paths:
+        return [expanded_directory]
+
+    working_directory_prefix = os.path.join(working_directory or '', '')
+
+    return [
+        (
+            glob_path
+            # If these are equal, that means we didn't add any working directory prefix above.
+            if normalized_directory == expanded_directory
+            # Remove the working directory prefix that we added above in order to make glob() work.
+            # We can't use os.path.relpath() here because it collapses any use of Borg's slashdot
+            # hack.
+            else glob_path.removeprefix(working_directory_prefix)
+        )
+        for glob_path in glob_paths
+    ]
 
 
-def expand_directories(directories, working_directory=None):
+def expand_patterns(patterns, working_directory=None, skip_paths=None):
     '''
-    Given a sequence of directory paths and an optional working directory, expand tildes and globs
-    in each one. Return all the resulting directories as a single flattened tuple.
+    Given a sequence of borgmatic.borg.pattern.Pattern instances and an optional working directory,
+    expand tildes and globs in each root pattern and expand just tildes in each non-root pattern.
+    The idea is that non-root patterns may be regular expressions or other pattern styles containing
+    "*" that borgmatic should not expand as a shell glob.
+
+    Return all the resulting patterns as a tuple.
+
+    If a set of paths are given to skip, then don't expand any patterns matching them.
     '''
-    if directories is None:
+    if patterns is None:
         return ()
 
     return tuple(
         itertools.chain.from_iterable(
-            expand_directory(directory, working_directory) for directory in directories
+            (
+                (
+                    borgmatic.borg.pattern.Pattern(
+                        expanded_path,
+                        pattern.type,
+                        pattern.style,
+                        pattern.device,
+                        pattern.source,
+                    )
+                    for expanded_path in expand_directory(pattern.path, working_directory)
+                )
+                if pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT
+                and pattern.path not in (skip_paths or ())
+                else (
+                    borgmatic.borg.pattern.Pattern(
+                        os.path.expanduser(pattern.path),
+                        pattern.type,
+                        pattern.style,
+                        pattern.device,
+                        pattern.source,
+                    ),
+                )
+            )
+            for pattern in patterns
         )
     )
 
 
-def map_directories_to_devices(directories, working_directory=None):
+def device_map_patterns(patterns, working_directory=None):
     '''
-    Given a sequence of directories and an optional working directory, return a map from directory
-    to an identifier for the device on which that directory resides or None if the path doesn't
-    exist.
+    Given a sequence of borgmatic.borg.pattern.Pattern instances and an optional working directory,
+    determine the identifier for the device on which the pattern's path resides—or None if the path
+    doesn't exist or is from a non-root pattern. Return an updated sequence of patterns with the
+    device field populated. But if the device field is already set, don't bother setting it again.
 
-    This is handy for determining whether two different directories are on the same filesystem (have
-    the same device identifier).
+    This is handy for determining whether two different pattern paths are on the same filesystem
+    (have the same device identifier).
     '''
-    return {
-        directory: os.stat(full_directory).st_dev if os.path.exists(full_directory) else None
-        for directory in directories
-        for full_directory in (os.path.join(working_directory or '', directory),)
-    }
+    return tuple(
+        borgmatic.borg.pattern.Pattern(
+            pattern.path,
+            pattern.type,
+            pattern.style,
+            device=pattern.device
+            or (
+                os.stat(full_path).st_dev
+                if pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT
+                and os.path.exists(full_path)
+                else None
+            ),
+            source=pattern.source,
+        )
+        for pattern in patterns
+        for full_path in (os.path.join(working_directory or '', pattern.path),)
+    )
 
 
-def deduplicate_directories(directory_devices, additional_directory_devices):
+def deduplicate_patterns(patterns):
     '''
-    Given a map from directory to the identifier for the device on which that directory resides,
-    return the directories as a sorted sequence with all duplicate child directories removed. For
-    instance, if paths is ['/foo', '/foo/bar'], return just: ['/foo']
+    Given a sequence of borgmatic.borg.pattern.Pattern instances, return them with all duplicate
+    root child patterns removed. For instance, if two root patterns are given with paths "/foo" and
+    "/foo/bar", return just the one with "/foo". Non-root patterns are passed through without
+    modification.
 
-    The one exception to this rule is if two paths are on different filesystems (devices). In that
-    case, they won't get de-duplicated in case they both need to be passed to Borg (e.g. the
-    location.one_file_system option is true).
+    The one exception to deduplication is two paths are on different filesystems (devices). In that
+    case, they won't get deduplicated, in case they both need to be passed to Borg (e.g. the
+    one_file_system option is true).
 
-    The idea is that if Borg is given a parent directory, then it doesn't also need to be given
-    child directories, because it will naturally spider the contents of the parent directory. And
+    The idea is that if Borg is given a root parent pattern, then it doesn't also need to be given
+    child patterns, because it will naturally spider the contents of the parent pattern's path. And
     there are cases where Borg coming across the same file twice will result in duplicate reads and
     even hangs, e.g. when a database hook is using a named pipe for streaming database dumps to
     Borg.
-
-    If any additional directory devices are given, also deduplicate against them, but don't include
-    them in the returned directories.
     '''
-    deduplicated = set()
-    directories = sorted(directory_devices.keys())
-    additional_directories = sorted(additional_directory_devices.keys())
-    all_devices = {**directory_devices, **additional_directory_devices}
+    deduplicated = {}  # Use just the keys as an ordered set.
 
-    for directory in directories:
-        deduplicated.add(directory)
-        parents = pathlib.PurePath(directory).parents
+    for pattern in patterns:
+        if pattern.type != borgmatic.borg.pattern.Pattern_type.ROOT:
+            deduplicated[pattern] = True
+            continue
 
-        # If another directory in the given list (or the additional list) is a parent of current
-        # directory (even n levels up) and both are on the same filesystem, then the current
-        # directory is a duplicate.
-        for other_directory in directories + additional_directories:
-            for parent in parents:
-                if (
-                    pathlib.PurePath(other_directory) == parent
-                    and all_devices[directory] is not None
-                    and all_devices[other_directory] == all_devices[directory]
-                ):
-                    if directory in deduplicated:
-                        deduplicated.remove(directory)
-                    break
+        parents = pathlib.PurePath(pattern.path).parents
 
-    return sorted(deduplicated)
+        # If another directory in the given list is a parent of current directory (even n levels up)
+        # and both are on the same filesystem, then the current directory is a duplicate.
+        for other_pattern in patterns:
+            if other_pattern.type != borgmatic.borg.pattern.Pattern_type.ROOT:
+                continue
+
+            if any(
+                pathlib.PurePath(other_pattern.path) == parent
+                and pattern.device is not None
+                and other_pattern.device == pattern.device
+                for parent in parents
+            ):
+                break
+        else:
+            deduplicated[pattern] = True
+
+    return tuple(deduplicated.keys())
 
 
-ROOT_PATTERN_PREFIX = 'R '
-
-
-def pattern_root_directories(patterns=None):
+def process_patterns(patterns, working_directory, skip_expand_paths=None):
     '''
-    Given a sequence of patterns, parse out and return just the root directories.
+    Given a sequence of Borg patterns and a configured working directory, expand and deduplicate any
+    "root" patterns, returning the resulting root and non-root patterns as a list.
+
+    If any paths are given to skip, don't expand them.
     '''
-    if not patterns:
-        return []
+    skip_paths = set(skip_expand_paths or ())
 
-    return [
-        pattern.split(ROOT_PATTERN_PREFIX, maxsplit=1)[1]
-        for pattern in patterns
-        if pattern.startswith(ROOT_PATTERN_PREFIX)
-    ]
-
-
-def process_source_directories(config, source_directories=None):
-    '''
-    Given a sequence of source directories (either in the source_directories argument or, lacking
-    that, from config), expand and deduplicate the source directories, returning the result.
-    '''
-    working_directory = borgmatic.config.paths.get_working_directory(config)
-
-    if source_directories is None:
-        source_directories = tuple(config.get('source_directories', ()))
-
-    return deduplicate_directories(
-        map_directories_to_devices(
-            expand_directories(
-                tuple(source_directories),
-                working_directory=working_directory,
+    return list(
+        deduplicate_patterns(
+            device_map_patterns(
+                expand_patterns(
+                    patterns,
+                    working_directory=working_directory,
+                    skip_paths=skip_paths,
+                )
             )
-        ),
-        additional_directory_devices=map_directories_to_devices(
-            expand_directories(
-                pattern_root_directories(config.get('patterns')),
-                working_directory=working_directory,
-            )
-        ),
+        )
     )
 
 
@@ -149,7 +272,6 @@ def run_create(
     repository,
     config,
     config_paths,
-    hook_context,
     local_borg_version,
     create_arguments,
     global_arguments,
@@ -167,52 +289,39 @@ def run_create(
     ):
         return
 
-    borgmatic.hooks.command.execute_hook(
-        config.get('before_backup'),
-        config.get('umask'),
-        config_filename,
-        'pre-backup',
-        global_arguments.dry_run,
-        **hook_context,
-    )
+    logger.info(f'Creating archive{dry_run_label}')
+    working_directory = borgmatic.config.paths.get_working_directory(config)
 
-    log_prefix = repository.get('label', repository['path'])
-    logger.info(f'{log_prefix}: Creating archive{dry_run_label}')
-
-    with borgmatic.config.paths.Runtime_directory(
-        config, log_prefix
-    ) as borgmatic_runtime_directory:
+    with borgmatic.config.paths.Runtime_directory(config) as borgmatic_runtime_directory:
         borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
             'remove_data_source_dumps',
             config,
-            repository['path'],
-            borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES,
+            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
             borgmatic_runtime_directory,
             global_arguments.dry_run,
         )
-        source_directories = process_source_directories(config)
+        patterns = process_patterns(collect_patterns(config), working_directory)
         active_dumps = borgmatic.hooks.dispatch.call_hooks(
             'dump_data_sources',
             config,
-            repository['path'],
-            borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES,
+            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
             config_paths,
             borgmatic_runtime_directory,
-            source_directories,
+            patterns,
             global_arguments.dry_run,
         )
 
-        # Process source directories again in case any data source hooks updated them. Without this
-        # step, we could end up with duplicate paths that cause Borg to hang when it tries to read
-        # from the same named pipe twice.
-        source_directories = process_source_directories(config, source_directories)
+        # Process the patterns again in case any data source hooks updated them. Without this step,
+        # we could end up with duplicate paths that cause Borg to hang when it tries to read from
+        # the same named pipe twice.
+        patterns = process_patterns(patterns, working_directory, skip_expand_paths=config_paths)
         stream_processes = [process for processes in active_dumps.values() for process in processes]
 
         json_output = borgmatic.borg.create.create_archive(
             global_arguments.dry_run,
             repository['path'],
             config,
-            source_directories,
+            patterns,
             local_borg_version,
             global_arguments,
             borgmatic_runtime_directory,
@@ -231,17 +340,7 @@ def run_create(
         borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
             'remove_data_source_dumps',
             config,
-            config_filename,
-            borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES,
+            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
             borgmatic_runtime_directory,
             global_arguments.dry_run,
         )
-
-    borgmatic.hooks.command.execute_hook(
-        config.get('after_backup'),
-        config.get('umask'),
-        config_filename,
-        'post-backup',
-        global_arguments.dry_run,
-        **hook_context,
-    )

borgmatic/actions/delete.py

@@ -23,7 +23,7 @@ def run_delete(
     if delete_arguments.repository is None or borgmatic.config.validate.repositories_match(
         repository, delete_arguments.repository
     ):
-        logger.answer(f'{repository.get("label", repository["path"])}: Deleting archives')
+        logger.answer('Deleting archives')
 
         archive_name = (
             borgmatic.borg.repo_list.resolve_archive_name(

borgmatic/actions/export_key.py

@@ -21,7 +21,7 @@ def run_export_key(
     if export_arguments.repository is None or borgmatic.config.validate.repositories_match(
         repository, export_arguments.repository
     ):
-        logger.info(f'{repository.get("label", repository["path"])}: Exporting repository key')
+        logger.info('Exporting repository key')
         borgmatic.borg.export_key.export_key(
             repository['path'],
             config,

borgmatic/actions/export_tar.py

@@ -22,9 +22,7 @@ def run_export_tar(
     if export_tar_arguments.repository is None or borgmatic.config.validate.repositories_match(
         repository, export_tar_arguments.repository
     ):
-        logger.info(
-            f'{repository["path"]}: Exporting archive {export_tar_arguments.archive} as tar file'
-        )
+        logger.info(f'Exporting archive {export_tar_arguments.archive} as tar file')
         borgmatic.borg.export_tar.export_tar_archive(
             global_arguments.dry_run,
             repository['path'],

borgmatic/actions/extract.py

@@ -12,7 +12,6 @@ def run_extract(
     config_filename,
     repository,
     config,
-    hook_context,
     local_borg_version,
     extract_arguments,
     global_arguments,
@@ -22,20 +21,10 @@ def run_extract(
     '''
     Run the "extract" action for the given repository.
     '''
-    borgmatic.hooks.command.execute_hook(
-        config.get('before_extract'),
-        config.get('umask'),
-        config_filename,
-        'pre-extract',
-        global_arguments.dry_run,
-        **hook_context,
-    )
     if extract_arguments.repository is None or borgmatic.config.validate.repositories_match(
         repository, extract_arguments.repository
     ):
-        logger.info(
-            f'{repository.get("label", repository["path"])}: Extracting archive {extract_arguments.archive}'
-        )
+        logger.info(f'Extracting archive {extract_arguments.archive}')
         borgmatic.borg.extract.extract_archive(
             global_arguments.dry_run,
             repository['path'],
@@ -58,11 +47,3 @@ def run_extract(
             strip_components=extract_arguments.strip_components,
             progress=extract_arguments.progress,
         )
-    borgmatic.hooks.command.execute_hook(
-        config.get('after_extract'),
-        config.get('umask'),
-        config_filename,
-        'post-extract',
-        global_arguments.dry_run,
-        **hook_context,
-    )

borgmatic/actions/import_key.py

@@ -0,0 +1,33 @@
+import logging
+
+import borgmatic.borg.import_key
+import borgmatic.config.validate
+
+logger = logging.getLogger(__name__)
+
+
+def run_import_key(
+    repository,
+    config,
+    local_borg_version,
+    import_arguments,
+    global_arguments,
+    local_path,
+    remote_path,
+):
+    '''
+    Run the "key import" action for the given repository.
+    '''
+    if import_arguments.repository is None or borgmatic.config.validate.repositories_match(
+        repository, import_arguments.repository
+    ):
+        logger.info('Importing repository key')
+        borgmatic.borg.import_key.import_key(
+            repository['path'],
+            config,
+            local_borg_version,
+            import_arguments,
+            global_arguments,
+            local_path=local_path,
+            remote_path=remote_path,
+        )

borgmatic/actions/info.py

@@ -27,9 +27,7 @@ def run_info(
         repository, info_arguments.repository
     ):
         if not info_arguments.json:
-            logger.answer(
-                f'{repository.get("label", repository["path"])}: Displaying archive summary information'
-            )
+            logger.answer('Displaying archive summary information')
         archive_name = borgmatic.borg.repo_list.resolve_archive_name(
             repository['path'],
             info_arguments.archive,

borgmatic/actions/list.py

@@ -27,9 +27,9 @@ def run_list(
     ):
         if not list_arguments.json:
             if list_arguments.find_paths:  # pragma: no cover
-                logger.answer(f'{repository.get("label", repository["path"])}: Searching archives')
+                logger.answer('Searching archives')
             elif not list_arguments.archive:  # pragma: no cover
-                logger.answer(f'{repository.get("label", repository["path"])}: Listing archives')
+                logger.answer('Listing archives')
 
         archive_name = borgmatic.borg.repo_list.resolve_archive_name(
             repository['path'],

borgmatic/actions/mount.py

@@ -23,11 +23,9 @@ def run_mount(
         repository, mount_arguments.repository
     ):
         if mount_arguments.archive:
-            logger.info(
-                f'{repository.get("label", repository["path"])}: Mounting archive {mount_arguments.archive}'
-            )
+            logger.info(f'Mounting archive {mount_arguments.archive}')
         else:  # pragma: nocover
-            logger.info(f'{repository.get("label", repository["path"])}: Mounting repository')
+            logger.info('Mounting repository')
 
         borgmatic.borg.mount.mount_archive(
             repository['path'],

borgmatic/actions/prune.py

@@ -11,7 +11,6 @@ def run_prune(
     config_filename,
     repository,
     config,
-    hook_context,
     local_borg_version,
     prune_arguments,
     global_arguments,
@@ -27,15 +26,7 @@ def run_prune(
     ):
         return
 
-    borgmatic.hooks.command.execute_hook(
-        config.get('before_prune'),
-        config.get('umask'),
-        config_filename,
-        'pre-prune',
-        global_arguments.dry_run,
-        **hook_context,
-    )
-    logger.info(f'{repository.get("label", repository["path"])}: Pruning archives{dry_run_label}')
+    logger.info(f'Pruning archives{dry_run_label}')
     borgmatic.borg.prune.prune_archives(
         global_arguments.dry_run,
         repository['path'],
@@ -46,11 +37,3 @@ def run_prune(
         local_path=local_path,
         remote_path=remote_path,
     )
-    borgmatic.hooks.command.execute_hook(
-        config.get('after_prune'),
-        config.get('umask'),
-        config_filename,
-        'post-prune',
-        global_arguments.dry_run,
-        **hook_context,
-    )

borgmatic/actions/repo_create.py

@@ -23,7 +23,7 @@ def run_repo_create(
     ):
         return
 
-    logger.info(f'{repository.get("label", repository["path"])}: Creating repository')
+    logger.info('Creating repository')
     borgmatic.borg.repo_create.create_repository(
         global_arguments.dry_run,
         repository['path'],

borgmatic/actions/repo_delete.py

@@ -21,8 +21,7 @@ def run_repo_delete(
         repository, repo_delete_arguments.repository
     ):
         logger.answer(
-            f'{repository.get("label", repository["path"])}: Deleting repository'
-            + (' cache' if repo_delete_arguments.cache_only else '')
+            'Deleting repository' + (' cache' if repo_delete_arguments.cache_only else '')
         )
 
         borgmatic.borg.repo_delete.delete_repository(

borgmatic/actions/repo_info.py

@@ -25,9 +25,7 @@ def run_repo_info(
         repository, repo_info_arguments.repository
     ):
         if not repo_info_arguments.json:
-            logger.answer(
-                f'{repository.get("label", repository["path"])}: Displaying repository summary information'
-            )
+            logger.answer('Displaying repository summary information')
 
         json_output = borgmatic.borg.repo_info.display_repository_info(
             repository['path'],

borgmatic/actions/repo_list.py

@@ -25,7 +25,7 @@ def run_repo_list(
         repository, repo_list_arguments.repository
     ):
         if not repo_list_arguments.json:
-            logger.answer(f'{repository.get("label", repository["path"])}: Listing repository')
+            logger.answer('Listing repository')
 
         json_output = borgmatic.borg.repo_list.list_repository(
             repository['path'],

borgmatic/actions/restore.py

@@ -1,4 +1,4 @@
-import copy
+import collections
 import logging
 import os
 import pathlib
@@ -11,58 +11,111 @@ import borgmatic.borg.mount
 import borgmatic.borg.repo_list
 import borgmatic.config.paths
 import borgmatic.config.validate
+import borgmatic.hooks.data_source.dump
 import borgmatic.hooks.dispatch
-import borgmatic.hooks.dump
 
 logger = logging.getLogger(__name__)
 
 
-UNSPECIFIED_HOOK = object()
+UNSPECIFIED = object()
 
 
-def get_configured_data_source(
-    config,
-    archive_data_source_names,
-    hook_name,
-    data_source_name,
-    configuration_data_source_name=None,
-):
+Dump = collections.namedtuple(
+    'Dump',
+    ('hook_name', 'data_source_name', 'hostname', 'port'),
+    defaults=('localhost', None),
+)
+
+
+def dumps_match(first, second, default_port=None):
     '''
-    Find the first data source with the given hook name and data source name in the configuration
-    dict and the given archive data source names dict (from hook name to data source names contained
-    in a particular backup archive). If UNSPECIFIED_HOOK is given as the hook name, search all data
-    source hooks for the named data source. If a configuration data source name is given, use that
-    instead of the data source name to lookup the data source in the given hooks configuration.
-
-    Return the found data source as a tuple of (found hook name, data source configuration dict) or
-    (None, None) if not found.
+    Compare two Dump instances for equality while supporting a field value of UNSPECIFIED, which
+    indicates that the field should match any value. If a default port is given, then consider any
+    dump having that port to match with a dump having a None port.
     '''
-    if not configuration_data_source_name:
-        configuration_data_source_name = data_source_name
+    for field_name in first._fields:
+        first_value = getattr(first, field_name)
+        second_value = getattr(second, field_name)
 
-    if hook_name == UNSPECIFIED_HOOK:
-        hooks_to_search = {
-            hook_name: value
-            for (hook_name, value) in config.items()
-            if hook_name in borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES
-        }
+        if default_port is not None and field_name == 'port':
+            if first_value == default_port and second_value is None:
+                continue
+            if second_value == default_port and first_value is None:
+                continue
+
+        if first_value == UNSPECIFIED or second_value == UNSPECIFIED:
+            continue
+
+        if first_value != second_value:
+            return False
+
+    return True
+
+
+def render_dump_metadata(dump):
+    '''
+    Given a Dump instance, make a display string describing it for use in log messages.
+    '''
+    name = 'unspecified' if dump.data_source_name is UNSPECIFIED else dump.data_source_name
+    hostname = dump.hostname or UNSPECIFIED
+    port = None if dump.port is UNSPECIFIED else dump.port
+
+    if port:
+        metadata = f'{name}@:{port}' if hostname is UNSPECIFIED else f'{name}@{hostname}:{port}'
     else:
-        try:
-            hooks_to_search = {hook_name: config[hook_name]}
-        except KeyError:
-            return (None, None)
+        metadata = f'{name}' if hostname is UNSPECIFIED else f'{name}@{hostname}'
 
-    return next(
-        (
-            (name, hook_data_source)
-            for (name, hook) in hooks_to_search.items()
-            for hook_data_source in hook
-            if hook_data_source['name'] == configuration_data_source_name
-            and data_source_name in archive_data_source_names.get(name, [])
-        ),
-        (None, None),
+    if dump.hook_name not in (None, UNSPECIFIED):
+        return f'{metadata} ({dump.hook_name})'
+
+    return metadata
+
+
+def get_configured_data_source(config, restore_dump):
+    '''
+    Search in the given configuration dict for dumps corresponding to the given dump to restore. If
+    there are multiple matches, error.
+
+    Return the found data source as a data source configuration dict or None if not found.
+    '''
+    try:
+        hooks_to_search = {restore_dump.hook_name: config[restore_dump.hook_name]}
+    except KeyError:
+        return None
+
+    matching_dumps = tuple(
+        hook_data_source
+        for (hook_name, hook_config) in hooks_to_search.items()
+        for hook_data_source in hook_config
+        for default_port in (
+            borgmatic.hooks.dispatch.call_hook(
+                function_name='get_default_port',
+                config=config,
+                hook_name=hook_name,
+            ),
+        )
+        if dumps_match(
+            Dump(
+                hook_name,
+                hook_data_source.get('name'),
+                hook_data_source.get('hostname', 'localhost'),
+                hook_data_source.get('port'),
+            ),
+            restore_dump,
+            default_port,
+        )
     )
 
+    if not matching_dumps:
+        return None
+
+    if len(matching_dumps) > 1:
+        raise ValueError(
+            f'Cannot restore data source {render_dump_metadata(restore_dump)} because there are multiple matching data sources configured'
+        )
+
+    return matching_dumps[0]
+
 
 def strip_path_prefix_from_extracted_dump_destination(
     destination_path, borgmatic_runtime_directory
@@ -97,7 +150,7 @@ def strip_path_prefix_from_extracted_dump_destination(
         break
 
 
-def restore_single_data_source(
+def restore_single_dump(
     repository,
     config,
     local_borg_version,
@@ -115,18 +168,19 @@ def restore_single_data_source(
     username/password as connection params, and a configured data source configuration dict, restore
     that data source from the archive.
     '''
-    logger.info(
-        f'{repository.get("label", repository["path"])}: Restoring data source {data_source["name"]}'
+    dump_metadata = render_dump_metadata(
+        Dump(hook_name, data_source['name'], data_source.get('hostname'), data_source.get('port'))
     )
 
+    logger.info(f'Restoring data source {dump_metadata}')
+
     dump_patterns = borgmatic.hooks.dispatch.call_hooks(
         'make_data_source_dump_patterns',
         config,
-        repository['path'],
-        borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES,
+        borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
         borgmatic_runtime_directory,
         data_source['name'],
-    )[hook_name]
+    )[hook_name.split('_databases', 1)[0]]
 
     destination_path = (
         tempfile.mkdtemp(dir=borgmatic_runtime_directory)
@@ -141,7 +195,11 @@ def restore_single_data_source(
             dry_run=global_arguments.dry_run,
             repository=repository['path'],
             archive=archive_name,
-            paths=[borgmatic.hooks.dump.convert_glob_patterns_to_borg_pattern(dump_patterns)],
+            paths=[
+                borgmatic.hooks.data_source.dump.convert_glob_patterns_to_borg_pattern(
+                    dump_patterns
+                )
+            ],
             config=config,
             local_borg_version=local_borg_version,
             global_arguments=global_arguments,
@@ -162,11 +220,10 @@ def restore_single_data_source(
             shutil.rmtree(destination_path, ignore_errors=True)
 
     # Run a single data source restore, consuming the extract stdout (if any).
-    borgmatic.hooks.dispatch.call_hooks(
+    borgmatic.hooks.dispatch.call_hook(
         function_name='restore_data_source_dump',
         config=config,
-        log_prefix=repository['path'],
-        hook_names=[hook_name],
+        hook_name=hook_name,
         data_source=data_source,
         dry_run=global_arguments.dry_run,
         extract_process=extract_process,
@@ -175,7 +232,7 @@ def restore_single_data_source(
     )
 
 
-def collect_archive_data_source_names(
+def collect_dumps_from_archive(
     repository,
     archive,
     config,
@@ -187,17 +244,17 @@ def collect_archive_data_source_names(
 ):
     '''
     Given a local or remote repository path, a resolved archive name, a configuration dict, the
-    local Borg version, global_arguments an argparse.Namespace, local and remote Borg paths, and the
-    borgmatic runtime directory, query the archive for the names of data sources it contains as
-    dumps and return them as a dict from hook name to a sequence of data source names.
+    local Borg version, global arguments an argparse.Namespace, local and remote Borg paths, and the
+    borgmatic runtime directory, query the archive for the names of data sources dumps it contains
+    and return them as a set of Dump instances.
     '''
     borgmatic_source_directory = str(
         pathlib.Path(borgmatic.config.paths.get_borgmatic_source_directory(config))
     )
 
     # Probe for the data source dumps in multiple locations, as the default location has moved to
-    # the borgmatic runtime directory (which get stored as just "/borgmatic" with Borg 1.4+). But we
-    # still want to support reading dumps from previously created archives as well.
+    # the borgmatic runtime directory (which gets stored as just "/borgmatic" with Borg 1.4+). But
+    # we still want to support reading dumps from previously created archives as well.
     dump_paths = borgmatic.borg.list.capture_archive_listing(
         repository,
         archive,
@@ -206,7 +263,9 @@ def collect_archive_data_source_names(
         global_arguments,
         list_paths=[
             'sh:'
-            + borgmatic.hooks.dump.make_data_source_dump_path(base_directory, '*_databases/*/*')
+            + borgmatic.hooks.data_source.dump.make_data_source_dump_path(
+                base_directory, '*_databases/*/*'
+            )
             for base_directory in (
                 'borgmatic',
                 borgmatic.config.paths.make_runtime_directory_glob(borgmatic_runtime_directory),
@@ -217,110 +276,148 @@ def collect_archive_data_source_names(
         remote_path=remote_path,
     )
 
-    # Determine the data source names corresponding to the dumps found in the archive and
-    # add them to restore_names.
-    archive_data_source_names = {}
+    # Parse the paths of dumps found in the archive to get their respective dump metadata.
+    dumps_from_archive = set()
 
     for dump_path in dump_paths:
         if not dump_path:
             continue
 
+        # Probe to find the base directory that's at the start of the dump path.
         for base_directory in (
             'borgmatic',
             borgmatic_runtime_directory,
             borgmatic_source_directory,
         ):
             try:
-                (hook_name, _, data_source_name) = dump_path.split(base_directory + os.path.sep, 1)[
-                    1
-                ].split(os.path.sep)[0:3]
+                (hook_name, host_and_port, data_source_name) = dump_path.split(
+                    base_directory + os.path.sep, 1
+                )[1].split(os.path.sep)[0:3]
             except (ValueError, IndexError):
-                pass
-            else:
-                if data_source_name not in archive_data_source_names.get(hook_name, []):
-                    archive_data_source_names.setdefault(hook_name, []).extend([data_source_name])
-                    break
+                continue
+
+            parts = host_and_port.split(':', 1)
+
+            if len(parts) == 1:
+                parts += (None,)
+
+            (hostname, port) = parts
+
+            try:
+                port = int(port)
+            except (ValueError, TypeError):
+                port = None
+
+            dumps_from_archive.add(Dump(hook_name, data_source_name, hostname, port))
+
+            # We've successfully parsed the dump path, so need to probe any further.
+            break
         else:
             logger.warning(
-                f'{repository}: Ignoring invalid data source dump path "{dump_path}" in archive {archive}'
+                f'Ignoring invalid data source dump path "{dump_path}" in archive {archive}'
             )
 
-    return archive_data_source_names
+    return dumps_from_archive
 
 
-def find_data_sources_to_restore(requested_data_source_names, archive_data_source_names):
+def get_dumps_to_restore(restore_arguments, dumps_from_archive):
     '''
-    Given a sequence of requested data source names to restore and a dict of hook name to the names
-    of data sources found in an archive, return an expanded sequence of data source names to
-    restore, replacing "all" with actual data source names as appropriate.
+    Given restore arguments as an argparse.Namespace instance indicating which dumps to restore and
+    a set of Dump instances representing the dumps found in an archive, return a set of specific
+    Dump instances from the archive to restore. As part of this, replace any Dump having a data
+    source name of "all" with multiple named Dump instances as appropriate.
 
-    Raise ValueError if any of the requested data source names cannot be found in the archive.
+    Raise ValueError if any of the requested data source names cannot be found in the archive or if
+    there are multiple archive dump matches for a given requested dump.
     '''
-    # A map from data source hook name to the data source names to restore for that hook.
-    restore_names = (
-        {UNSPECIFIED_HOOK: requested_data_source_names}
-        if requested_data_source_names
-        else {UNSPECIFIED_HOOK: ['all']}
+    requested_dumps = (
+        {
+            Dump(
+                hook_name=(
+                    (
+                        restore_arguments.hook
+                        if restore_arguments.hook.endswith('_databases')
+                        else f'{restore_arguments.hook}_databases'
+                    )
+                    if restore_arguments.hook
+                    else UNSPECIFIED
+                ),
+                data_source_name=name,
+                hostname=restore_arguments.original_hostname or UNSPECIFIED,
+                port=restore_arguments.original_port,
+            )
+            for name in restore_arguments.data_sources or (UNSPECIFIED,)
+        }
+        if restore_arguments.hook
+        or restore_arguments.data_sources
+        or restore_arguments.original_hostname
+        or restore_arguments.original_port
+        else {
+            Dump(
+                hook_name=UNSPECIFIED,
+                data_source_name='all',
+                hostname=UNSPECIFIED,
+                port=UNSPECIFIED,
+            )
+        }
     )
+    missing_dumps = set()
+    dumps_to_restore = set()
 
-    # If "all" is in restore_names, then replace it with the names of dumps found within the
-    # archive.
-    if 'all' in restore_names[UNSPECIFIED_HOOK]:
-        restore_names[UNSPECIFIED_HOOK].remove('all')
+    # If there's a requested "all" dump, add every dump from the archive to the dumps to restore.
+    if any(dump for dump in requested_dumps if dump.data_source_name == 'all'):
+        dumps_to_restore.update(dumps_from_archive)
 
-        for hook_name, data_source_names in archive_data_source_names.items():
-            restore_names.setdefault(hook_name, []).extend(data_source_names)
+    # If any archive dump matches a requested dump, add the archive dump to the dumps to restore.
+    for requested_dump in requested_dumps:
+        if requested_dump.data_source_name == 'all':
+            continue
 
-            # If a data source is to be restored as part of "all", then remove it from restore names
-            # so it doesn't get restored twice.
-            for data_source_name in data_source_names:
-                if data_source_name in restore_names[UNSPECIFIED_HOOK]:
-                    restore_names[UNSPECIFIED_HOOK].remove(data_source_name)
-
-    if not restore_names[UNSPECIFIED_HOOK]:
-        restore_names.pop(UNSPECIFIED_HOOK)
-
-    combined_restore_names = set(
-        name for data_source_names in restore_names.values() for name in data_source_names
-    )
-    combined_archive_data_source_names = set(
-        name
-        for data_source_names in archive_data_source_names.values()
-        for name in data_source_names
-    )
-
-    missing_names = sorted(set(combined_restore_names) - combined_archive_data_source_names)
-    if missing_names:
-        joined_names = ', '.join(f'"{name}"' for name in missing_names)
-        raise ValueError(
-            f"Cannot restore data source{'s' if len(missing_names) > 1 else ''} {joined_names} missing from archive"
+        matching_dumps = tuple(
+            archive_dump
+            for archive_dump in dumps_from_archive
+            if dumps_match(requested_dump, archive_dump)
         )
 
-    return restore_names
+        if len(matching_dumps) == 0:
+            missing_dumps.add(requested_dump)
+        elif len(matching_dumps) == 1:
+            dumps_to_restore.add(matching_dumps[0])
+        else:
+            raise ValueError(
+                f'Cannot restore data source {render_dump_metadata(requested_dump)} because there are multiple matching dumps in the archive. Try adding flags to disambiguate.'
+            )
+
+    if missing_dumps:
+        rendered_dumps = ', '.join(
+            f'{render_dump_metadata(dump)}' for dump in sorted(missing_dumps)
+        )
+
+        raise ValueError(
+            f"Cannot restore data source dump{'s' if len(missing_dumps) > 1 else ''} {rendered_dumps} missing from archive"
+        )
+
+    return dumps_to_restore
 
 
-def ensure_data_sources_found(restore_names, remaining_restore_names, found_names):
+def ensure_requested_dumps_restored(dumps_to_restore, dumps_actually_restored):
     '''
-    Given a dict from hook name to data source names to restore, a dict from hook name to remaining
-    data source names to restore, and a sequence of found (actually restored) data source names,
-    raise ValueError if requested data source to restore were missing from the archive and/or
+    Given a set of requested dumps to restore and a set of dumps actually restored, raise ValueError
+    if any requested dumps to restore weren't restored, indicating that they were missing from the
     configuration.
     '''
-    combined_restore_names = set(
-        name
-        for data_source_names in tuple(restore_names.values())
-        + tuple(remaining_restore_names.values())
-        for name in data_source_names
-    )
-
-    if not combined_restore_names and not found_names:
+    if not dumps_actually_restored:
         raise ValueError('No data source dumps were found to restore')
 
-    missing_names = sorted(set(combined_restore_names) - set(found_names))
-    if missing_names:
-        joined_names = ', '.join(f'"{name}"' for name in missing_names)
+    missing_dumps = sorted(
+        dumps_to_restore - dumps_actually_restored, key=lambda dump: dump.data_source_name
+    )
+
+    if missing_dumps:
+        rendered_dumps = ', '.join(f'{render_dump_metadata(dump)}' for dump in missing_dumps)
+
         raise ValueError(
-            f"Cannot restore data source{'s' if len(missing_names) > 1 else ''} {joined_names} missing from borgmatic's configuration"
+            f"Cannot restore data source{'s' if len(missing_dumps) > 1 else ''} {rendered_dumps} missing from borgmatic's configuration"
         )
 
 
@@ -337,24 +434,21 @@ def run_restore(
     Run the "restore" action for the given repository, but only if the repository matches the
     requested repository in restore arguments.
 
-    Raise ValueError if a configured data source could not be found to restore.
+    Raise ValueError if a configured data source could not be found to restore or there's no
+    matching dump in the archive.
     '''
     if restore_arguments.repository and not borgmatic.config.validate.repositories_match(
         repository, restore_arguments.repository
     ):
         return
 
-    log_prefix = repository.get('label', repository['path'])
-    logger.info(f'{log_prefix}: Restoring data sources from archive {restore_arguments.archive}')
+    logger.info(f'Restoring data sources from archive {restore_arguments.archive}')
 
-    with borgmatic.config.paths.Runtime_directory(
-        config, log_prefix
-    ) as borgmatic_runtime_directory:
+    with borgmatic.config.paths.Runtime_directory(config) as borgmatic_runtime_directory:
         borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
             'remove_data_source_dumps',
             config,
-            repository['path'],
-            borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES,
+            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
             borgmatic_runtime_directory,
             global_arguments.dry_run,
         )
@@ -368,7 +462,7 @@ def run_restore(
             local_path,
             remote_path,
         )
-        archive_data_source_names = collect_archive_data_source_names(
+        dumps_from_archive = collect_dumps_from_archive(
             repository['path'],
             archive_name,
             config,
@@ -378,11 +472,9 @@ def run_restore(
             remote_path,
             borgmatic_runtime_directory,
         )
-        restore_names = find_data_sources_to_restore(
-            restore_arguments.data_sources, archive_data_source_names
-        )
-        found_names = set()
-        remaining_restore_names = {}
+        dumps_to_restore = get_dumps_to_restore(restore_arguments, dumps_from_archive)
+
+        dumps_actually_restored = set()
         connection_params = {
             'hostname': restore_arguments.hostname,
             'port': restore_arguments.port,
@@ -391,69 +483,49 @@ def run_restore(
             'restore_path': restore_arguments.restore_path,
         }
 
-        for hook_name, data_source_names in restore_names.items():
-            for data_source_name in data_source_names:
-                found_hook_name, found_data_source = get_configured_data_source(
-                    config, archive_data_source_names, hook_name, data_source_name
-                )
+        # Restore each dump.
+        for restore_dump in dumps_to_restore:
+            found_data_source = get_configured_data_source(
+                config,
+                restore_dump,
+            )
 
-                if not found_data_source:
-                    remaining_restore_names.setdefault(found_hook_name or hook_name, []).append(
-                        data_source_name
-                    )
-                    continue
-
-                found_names.add(data_source_name)
-                restore_single_data_source(
-                    repository,
+            # For a dump that wasn't found via an exact match in the configuration, try to fallback
+            # to an "all" data source.
+            if not found_data_source:
+                found_data_source = get_configured_data_source(
                     config,
-                    local_borg_version,
-                    global_arguments,
-                    local_path,
-                    remote_path,
-                    archive_name,
-                    found_hook_name or hook_name,
-                    dict(found_data_source, **{'schemas': restore_arguments.schemas}),
-                    connection_params,
-                    borgmatic_runtime_directory,
-                )
-
-        # For any data sources that weren't found via exact matches in the configuration, try to
-        # fallback to "all" entries.
-        for hook_name, data_source_names in remaining_restore_names.items():
-            for data_source_name in data_source_names:
-                found_hook_name, found_data_source = get_configured_data_source(
-                    config, archive_data_source_names, hook_name, data_source_name, 'all'
+                    Dump(restore_dump.hook_name, 'all', restore_dump.hostname, restore_dump.port),
                 )
 
                 if not found_data_source:
                     continue
 
-                found_names.add(data_source_name)
-                data_source = copy.copy(found_data_source)
-                data_source['name'] = data_source_name
+                found_data_source = dict(found_data_source)
+                found_data_source['name'] = restore_dump.data_source_name
 
-                restore_single_data_source(
-                    repository,
-                    config,
-                    local_borg_version,
-                    global_arguments,
-                    local_path,
-                    remote_path,
-                    archive_name,
-                    found_hook_name or hook_name,
-                    dict(data_source, **{'schemas': restore_arguments.schemas}),
-                    connection_params,
-                    borgmatic_runtime_directory,
-                )
+            dumps_actually_restored.add(restore_dump)
+
+            restore_single_dump(
+                repository,
+                config,
+                local_borg_version,
+                global_arguments,
+                local_path,
+                remote_path,
+                archive_name,
+                restore_dump.hook_name,
+                dict(found_data_source, **{'schemas': restore_arguments.schemas}),
+                connection_params,
+                borgmatic_runtime_directory,
+            )
 
         borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
             'remove_data_source_dumps',
             config,
-            repository['path'],
-            borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES,
+            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
             borgmatic_runtime_directory,
             global_arguments.dry_run,
         )
 
-    ensure_data_sources_found(restore_names, remaining_restore_names, found_names)
+    ensure_requested_dumps_restored(dumps_to_restore, dumps_actually_restored)

borgmatic/actions/transfer.py

@@ -17,9 +17,7 @@ def run_transfer(
     '''
     Run the "transfer" action for the given repository.
     '''
-    logger.info(
-        f'{repository.get("label", repository["path"])}: Transferring archives to repository'
-    )
+    logger.info('Transferring archives to repository')
     borgmatic.borg.transfer.transfer_archives(
         global_arguments.dry_run,
         repository['path'],

borgmatic/borg/borg.py

@@ -61,7 +61,7 @@ def run_arbitrary_borg(
         tuple(shlex.quote(part) for part in full_command),
         output_file=DO_NOT_CAPTURE,
         shell=True,
-        extra_environment=dict(
+        environment=dict(
             (environment.make_environment(config) or {}),
             **{
                 'BORG_REPO': repository_path,

borgmatic/borg/break_lock.py

@@ -34,10 +34,9 @@ def break_lock(
         + flags.make_repository_flags(repository_path, local_borg_version)
     )
 
-    borg_environment = environment.make_environment(config)
     execute_command(
         full_command,
-        extra_environment=borg_environment,
+        environment=environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/change_passphrase.py

@@ -41,7 +41,7 @@ def change_passphrase(
     )
 
     if global_arguments.dry_run:
-        logger.info(f'{repository_path}: Skipping change password (dry run)')
+        logger.info('Skipping change password (dry run)')
         return
 
     # If the original passphrase is set programmatically, then Borg won't prompt for a new one! So
@@ -56,7 +56,7 @@ def change_passphrase(
         full_command,
         output_file=borgmatic.execute.DO_NOT_CAPTURE,
         output_log_level=logging.ANSWER,
-        extra_environment=environment.make_environment(config_without_passphrase),
+        environment=environment.make_environment(config_without_passphrase),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/check.py

@@ -64,15 +64,11 @@ def make_check_name_flags(checks, archive_filter_flags):
 
         ('--repository-only',)
 
-    However, if both "repository" and "archives" are in checks, then omit them from the returned
-    flags because Borg does both checks by default. If "data" is in checks, that implies "archives".
+    However, if both "repository" and "archives" are in checks, then omit the "only" flags from the
+    returned flags because Borg does both checks by default. Note that a "data" check only works
+    along with an "archives" check.
     '''
-    if 'data' in checks:
-        data_flags = ('--verify-data',)
-        checks.update({'archives'})
-    else:
-        data_flags = ()
-
+    data_flags = ('--verify-data',) if 'data' in checks else ()
     common_flags = (archive_filter_flags if 'archives' in checks else ()) + data_flags
 
     if {'repository', 'archives'}.issubset(checks):
@@ -142,51 +138,51 @@ def check_archives(
     except StopIteration:
         repository_check_config = {}
 
-    if check_arguments.max_duration and 'archives' in checks:
-        raise ValueError('The archives check cannot run when the --max-duration flag is used')
-    if repository_check_config.get('max_duration') and 'archives' in checks:
-        raise ValueError(
-            'The archives check cannot run when the repository check has the max_duration option set'
-        )
-
     max_duration = check_arguments.max_duration or repository_check_config.get('max_duration')
+
     umask = config.get('umask')
-
-    borg_environment = environment.make_environment(config)
     borg_exit_codes = config.get('borg_exit_codes')
-
-    full_command = (
-        (local_path, 'check')
-        + (('--repair',) if check_arguments.repair else ())
-        + (('--max-duration', str(max_duration)) if max_duration else ())
-        + make_check_name_flags(checks, archive_filter_flags)
-        + (('--remote-path', remote_path) if remote_path else ())
-        + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if global_arguments.log_json else ())
-        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
-        + verbosity_flags
-        + (('--progress',) if check_arguments.progress else ())
-        + (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
-        + flags.make_repository_flags(repository_path, local_borg_version)
-    )
-
     working_directory = borgmatic.config.paths.get_working_directory(config)
 
-    # The Borg repair option triggers an interactive prompt, which won't work when output is
-    # captured. And progress messes with the terminal directly.
-    if check_arguments.repair or check_arguments.progress:
+    if 'data' in checks:
+        checks.add('archives')
+
+    grouped_checks = (checks,)
+
+    # If max_duration is set, then archives and repository checks need to be run separately, as Borg
+    # doesn't support --max-duration along with an archives checks.
+    if max_duration and 'archives' in checks and 'repository' in checks:
+        checks.remove('repository')
+        grouped_checks = (checks, {'repository'})
+
+    for checks_subset in grouped_checks:
+        full_command = (
+            (local_path, 'check')
+            + (('--repair',) if check_arguments.repair else ())
+            + (
+                ('--max-duration', str(max_duration))
+                if max_duration and 'repository' in checks_subset
+                else ()
+            )
+            + make_check_name_flags(checks_subset, archive_filter_flags)
+            + (('--remote-path', remote_path) if remote_path else ())
+            + (('--umask', str(umask)) if umask else ())
+            + (('--log-json',) if global_arguments.log_json else ())
+            + (('--lock-wait', str(lock_wait)) if lock_wait else ())
+            + verbosity_flags
+            + (('--progress',) if check_arguments.progress else ())
+            + (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+            + flags.make_repository_flags(repository_path, local_borg_version)
+        )
+
         execute_command(
             full_command,
-            output_file=DO_NOT_CAPTURE,
-            extra_environment=borg_environment,
-            working_directory=working_directory,
-            borg_local_path=local_path,
-            borg_exit_codes=borg_exit_codes,
-        )
-    else:
-        execute_command(
-            full_command,
-            extra_environment=borg_environment,
+            # The Borg repair option triggers an interactive prompt, which won't work when output is
+            # captured. And progress messes with the terminal directly.
+            output_file=(
+                DO_NOT_CAPTURE if check_arguments.repair or check_arguments.progress else None
+            ),
+            environment=environment.make_environment(config),
             working_directory=working_directory,
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,

borgmatic/borg/compact.py

@@ -43,13 +43,13 @@ def compact_segments(
     )
 
     if dry_run:
-        logging.info(f'{repository_path}: Skipping compact (dry run)')
+        logging.info('Skipping compact (dry run)')
         return
 
     execute_command(
         full_command,
         output_log_level=logging.INFO,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/create.py

@@ -6,6 +6,7 @@ import stat
 import tempfile
 import textwrap
 
+import borgmatic.borg.pattern
 import borgmatic.config.paths
 import borgmatic.logger
 from borgmatic.borg import environment, feature, flags
@@ -19,81 +20,42 @@ from borgmatic.execute import (
 logger = logging.getLogger(__name__)
 
 
-def expand_home_directories(directories):
+def write_patterns_file(patterns, borgmatic_runtime_directory, patterns_file=None):
     '''
-    Given a sequence of directory paths, expand tildes in each one. Do not perform any globbing.
-    Return the results as a tuple.
-    '''
-    if directories is None:
-        return ()
+    Given a sequence of patterns as borgmatic.borg.pattern.Pattern instances, write them to a named
+    temporary file in the given borgmatic runtime directory and return the file object so it can
+    continue to exist on disk as long as the caller needs it.
 
-    return tuple(os.path.expanduser(directory) for directory in directories)
-
-
-def write_pattern_file(patterns=None, sources=None, pattern_file=None):
-    '''
-    Given a sequence of patterns and an optional sequence of source directories, write them to a
-    named temporary file (with the source directories as additional roots) and return the file.
-    If an optional open pattern file is given, overwrite it instead of making a new temporary file.
+    If an optional open pattern file is given, append to it instead of making a new temporary file.
     Return None if no patterns are provided.
     '''
-    if not patterns and not sources:
+    if not patterns:
         return None
 
-    if pattern_file is None:
-        pattern_file = tempfile.NamedTemporaryFile('w')
+    if patterns_file is None:
+        patterns_file = tempfile.NamedTemporaryFile('w', dir=borgmatic_runtime_directory)
+        operation_name = 'Writing'
     else:
-        pattern_file.seek(0)
+        patterns_file.write('\n')
+        operation_name = 'Appending'
 
-    pattern_file.write(
-        '\n'.join(tuple(patterns or ()) + tuple(f'R {source}' for source in (sources or [])))
+    patterns_output = '\n'.join(
+        f'{pattern.type.value} {pattern.style.value}{":" if pattern.style.value else ""}{pattern.path}'
+        for pattern in patterns
     )
-    pattern_file.flush()
+    logger.debug(f'{operation_name} patterns to {patterns_file.name}:\n{patterns_output}')
 
-    return pattern_file
+    patterns_file.write(patterns_output)
+    patterns_file.flush()
+
+    return patterns_file
 
 
-def ensure_files_readable(*filename_lists):
+def make_exclude_flags(config):
     '''
-    Given a sequence of filename sequences, ensure that each filename is openable. This prevents
-    unreadable files from being passed to Borg, which in certain situations only warns instead of
-    erroring.
+    Given a configuration dict with various exclude options, return the corresponding Borg flags as
+    a tuple.
     '''
-    for file_object in itertools.chain.from_iterable(
-        filename_list for filename_list in filename_lists if filename_list
-    ):
-        open(file_object).close()
-
-
-def make_pattern_flags(config, pattern_filename=None):
-    '''
-    Given a configuration dict with a potential patterns_from option, and a filename containing any
-    additional patterns, return the corresponding Borg flags for those files as a tuple.
-    '''
-    pattern_filenames = tuple(config.get('patterns_from') or ()) + (
-        (pattern_filename,) if pattern_filename else ()
-    )
-
-    return tuple(
-        itertools.chain.from_iterable(
-            ('--patterns-from', pattern_filename) for pattern_filename in pattern_filenames
-        )
-    )
-
-
-def make_exclude_flags(config, exclude_filename=None):
-    '''
-    Given a configuration dict with various exclude options, and a filename containing any exclude
-    patterns, return the corresponding Borg flags as a tuple.
-    '''
-    exclude_filenames = tuple(config.get('exclude_from') or ()) + (
-        (exclude_filename,) if exclude_filename else ()
-    )
-    exclude_from_flags = tuple(
-        itertools.chain.from_iterable(
-            ('--exclude-from', exclude_filename) for exclude_filename in exclude_filenames
-        )
-    )
     caches_flag = ('--exclude-caches',) if config.get('exclude_caches') else ()
     if_present_flags = tuple(
         itertools.chain.from_iterable(
@@ -104,13 +66,7 @@ def make_exclude_flags(config, exclude_filename=None):
     keep_exclude_tags_flags = ('--keep-exclude-tags',) if config.get('keep_exclude_tags') else ()
     exclude_nodump_flags = ('--exclude-nodump',) if config.get('exclude_nodump') else ()
 
-    return (
-        exclude_from_flags
-        + caches_flag
-        + if_present_flags
-        + keep_exclude_tags_flags
-        + exclude_nodump_flags
-    )
+    return caches_flag + if_present_flags + keep_exclude_tags_flags + exclude_nodump_flags
 
 
 def make_list_filter_flags(local_borg_version, dry_run):
@@ -134,13 +90,14 @@ def make_list_filter_flags(local_borg_version, dry_run):
         return f'{base_flags}-'
 
 
-def special_file(path):
+def special_file(path, working_directory=None):
     '''
     Return whether the given path is a special file (character device, block device, or named pipe
-    / FIFO).
+    / FIFO). If a working directory is given, take it into account when making the full path to
+    check.
     '''
     try:
-        mode = os.stat(path).st_mode
+        mode = os.stat(os.path.join(working_directory or '', path)).st_mode
     except (FileNotFoundError, OSError):
         return False
 
@@ -160,52 +117,87 @@ def any_parent_directories(path, candidate_parents):
 
 
 def collect_special_file_paths(
-    create_command, config, local_path, working_directory, borg_environment, skip_directories
+    dry_run,
+    create_command,
+    config,
+    local_path,
+    working_directory,
+    borgmatic_runtime_directory,
 ):
     '''
-    Given a Borg create command as a tuple, a configuration dict, a local Borg path, a working
-    directory, a dict of environment variables to pass to Borg, and a sequence of parent directories
-    to skip, collect the paths for any special files (character devices, block devices, and named
-    pipes / FIFOs) that Borg would encounter during a create. These are all paths that could cause
-    Borg to hang if its --read-special flag is used.
+    Given a dry-run flag, a Borg create command as a tuple, a configuration dict, a local Borg path,
+    a working directory, and the borgmatic runtime directory, collect the paths for any special
+    files (character devices, block devices, and named pipes / FIFOs) that Borg would encounter
+    during a create. These are all paths that could cause Borg to hang if its --read-special flag is
+    used.
+
+    Skip looking for special files in the given borgmatic runtime directory, as borgmatic creates
+    its own special files there for database dumps and we don't want those omitted.
+
+    Additionally, if the borgmatic runtime directory is not contained somewhere in the files Borg
+    plans to backup, that means the user must have excluded the runtime directory (e.g. via
+    "exclude_patterns" or similar). Therefore, raise, because this means Borg won't be able to
+    consume any database dumps and therefore borgmatic will hang when it tries to do so.
     '''
     # Omit "--exclude-nodump" from the Borg dry run command, because that flag causes Borg to open
-    # files including any named pipe we've created.
+    # files including any named pipe we've created. And omit "--filter" because that can break the
+    # paths output parsing below such that path lines no longer start with th expected "- ".
     paths_output = execute_command_and_capture_output(
-        tuple(argument for argument in create_command if argument != '--exclude-nodump')
+        flags.omit_flag_and_value(flags.omit_flag(create_command, '--exclude-nodump'), '--filter')
         + ('--dry-run', '--list'),
         capture_stderr=True,
         working_directory=working_directory,
-        extra_environment=borg_environment,
+        environment=environment.make_environment(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),
     )
 
+    # These are all the individual files that Borg is planning to backup as determined by the Borg
+    # create dry run above.
     paths = tuple(
         path_line.split(' ', 1)[1]
         for path_line in paths_output.split('\n')
         if path_line and path_line.startswith('- ') or path_line.startswith('+ ')
     )
 
+    # These are the subset of those files that contain the borgmatic runtime directory.
+    paths_containing_runtime_directory = {}
+
+    if os.path.exists(borgmatic_runtime_directory):
+        paths_containing_runtime_directory = {
+            path for path in paths if any_parent_directories(path, (borgmatic_runtime_directory,))
+        }
+
+        # If no paths to backup contain the runtime directory, it must've been excluded.
+        if not paths_containing_runtime_directory and not dry_run:
+            raise ValueError(
+                f'The runtime directory {os.path.normpath(borgmatic_runtime_directory)} overlaps with the configured excludes or patterns with excludes. Please ensure the runtime directory is not excluded.'
+            )
+
     return tuple(
         path
         for path in paths
-        if special_file(path) and not any_parent_directories(path, skip_directories)
+        if special_file(path, working_directory)
+        if path not in paths_containing_runtime_directory
     )
 
 
-def check_all_source_directories_exist(source_directories):
+def check_all_root_patterns_exist(patterns):
     '''
-    Given a sequence of source directories, check that the source directories all exist. If any do
-    not, raise an exception.
+    Given a sequence of borgmatic.borg.pattern.Pattern instances, check that all root pattern
+    paths exist. If any don't, raise an exception.
     '''
-    missing_directories = [
-        source_directory
-        for source_directory in source_directories
-        if not os.path.exists(source_directory)
+    missing_paths = [
+        pattern.path
+        for pattern in patterns
+        if pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT
+        if not os.path.exists(pattern.path)
     ]
-    if missing_directories:
-        raise ValueError(f"Source directories do not exist: {', '.join(missing_directories)}")
+
+    if missing_paths:
+        raise ValueError(
+            f"Source directories / root pattern paths do not exist: {', '.join(missing_paths)}"
+        )
 
 
 MAX_SPECIAL_FILE_PATHS_LENGTH = 1000
@@ -215,7 +207,7 @@ def make_base_create_command(
     dry_run,
     repository_path,
     config,
-    source_directories,
+    patterns,
     local_borg_version,
     global_arguments,
     borgmatic_runtime_directory,
@@ -227,23 +219,16 @@ def make_base_create_command(
     stream_processes=None,
 ):
     '''
-    Given vebosity/dry-run flags, a local or remote repository path, a configuration dict, a
-    sequence of loaded configuration paths, the local Borg version, global arguments as an
-    argparse.Namespace instance, and a sequence of borgmatic source directories, return a tuple of
-    (base Borg create command flags, Borg create command positional arguments, open pattern file
-    handle, open exclude file handle).
+    Given verbosity/dry-run flags, a local or remote repository path, a configuration dict, a
+    sequence of patterns as borgmatic.borg.pattern.Pattern instances, the local Borg version,
+    global arguments as an argparse.Namespace instance, and a sequence of borgmatic source
+    directories, return a tuple of (base Borg create command flags, Borg create command positional
+    arguments, open pattern file handle).
     '''
     if config.get('source_directories_must_exist', False):
-        check_all_source_directories_exist(source_directories)
+        check_all_root_patterns_exist(patterns)
 
-    ensure_files_readable(config.get('patterns_from'), config.get('exclude_from'))
-
-    pattern_file = (
-        write_pattern_file(config.get('patterns'), source_directories)
-        if config.get('patterns') or config.get('patterns_from')
-        else None
-    )
-    exclude_file = write_pattern_file(expand_home_directories(config.get('exclude_patterns')))
+    patterns_file = write_patterns_file(patterns, borgmatic_runtime_directory)
     checkpoint_interval = config.get('checkpoint_interval', None)
     checkpoint_volume = config.get('checkpoint_volume', None)
     chunker_params = config.get('chunker_params', None)
@@ -286,8 +271,8 @@ def make_base_create_command(
     create_flags = (
         tuple(local_path.split(' '))
         + ('create',)
-        + make_pattern_flags(config, pattern_file.name if pattern_file else None)
-        + make_exclude_flags(config, exclude_file.name if exclude_file else None)
+        + (('--patterns-from', patterns_file.name) if patterns_file else ())
+        + make_exclude_flags(config)
         + (('--checkpoint-interval', str(checkpoint_interval)) if checkpoint_interval else ())
         + (('--checkpoint-volume', str(checkpoint_volume)) if checkpoint_volume else ())
         + (('--chunker-params', chunker_params) if chunker_params else ())
@@ -317,27 +302,24 @@ def make_base_create_command(
 
     create_positional_arguments = flags.make_repository_archive_flags(
         repository_path, archive_name_format, local_borg_version
-    ) + (tuple(source_directories) if not pattern_file else ())
+    )
 
     # If database hooks are enabled (as indicated by streaming processes), exclude files that might
     # cause Borg to hang. But skip this if the user has explicitly set the "read_special" to True.
     if stream_processes and not config.get('read_special'):
         logger.warning(
-            f'{repository_path}: Ignoring configured "read_special" value of false, as true is needed for database hooks.'
+            'Ignoring configured "read_special" value of false, as true is needed for database hooks.'
         )
-        borg_environment = environment.make_environment(config)
         working_directory = borgmatic.config.paths.get_working_directory(config)
 
-        logger.debug(f'{repository_path}: Collecting special file paths')
+        logger.debug('Collecting special file paths')
         special_file_paths = collect_special_file_paths(
+            dry_run,
             create_flags + create_positional_arguments,
             config,
             local_path,
             working_directory,
-            borg_environment,
-            skip_directories=(
-                [borgmatic_runtime_directory] if os.path.exists(borgmatic_runtime_directory) else []
-            ),
+            borgmatic_runtime_directory=borgmatic_runtime_directory,
         )
 
         if special_file_paths:
@@ -347,24 +329,33 @@ def make_base_create_command(
                 placeholder=' ...',
             )
             logger.warning(
-                f'{repository_path}: Excluding special files to prevent Borg from hanging: {truncated_special_file_paths}'
+                f'Excluding special files to prevent Borg from hanging: {truncated_special_file_paths}'
             )
-            exclude_file = write_pattern_file(
-                expand_home_directories(
-                    tuple(config.get('exclude_patterns') or ()) + special_file_paths
+            patterns_file = write_patterns_file(
+                tuple(
+                    borgmatic.borg.pattern.Pattern(
+                        special_file_path,
+                        borgmatic.borg.pattern.Pattern_type.NO_RECURSE,
+                        borgmatic.borg.pattern.Pattern_style.FNMATCH,
+                        source=borgmatic.borg.pattern.Pattern_source.INTERNAL,
+                    )
+                    for special_file_path in special_file_paths
                 ),
-                pattern_file=exclude_file,
+                borgmatic_runtime_directory,
+                patterns_file=patterns_file,
             )
-            create_flags += make_exclude_flags(config, exclude_file.name)
 
-    return (create_flags, create_positional_arguments, pattern_file, exclude_file)
+            if '--patterns-from' not in create_flags:
+                create_flags += ('--patterns-from', patterns_file.name)
+
+    return (create_flags, create_positional_arguments, patterns_file)
 
 
 def create_archive(
     dry_run,
     repository_path,
     config,
-    source_directories,
+    patterns,
     local_borg_version,
     global_arguments,
     borgmatic_runtime_directory,
@@ -377,7 +368,7 @@ def create_archive(
     stream_processes=None,
 ):
     '''
-    Given vebosity/dry-run flags, a local or remote repository path, a configuration dict, a
+    Given verbosity/dry-run flags, a local or remote repository path, a configuration dict, a
     sequence of loaded configuration paths, the local Borg version, and global arguments as an
     argparse.Namespace instance, create a Borg archive and return Borg's JSON output (if any).
 
@@ -388,22 +379,20 @@ def create_archive(
 
     working_directory = borgmatic.config.paths.get_working_directory(config)
 
-    (create_flags, create_positional_arguments, pattern_file, exclude_file) = (
-        make_base_create_command(
-            dry_run,
-            repository_path,
-            config,
-            source_directories,
-            local_borg_version,
-            global_arguments,
-            borgmatic_runtime_directory,
-            local_path,
-            remote_path,
-            progress,
-            json,
-            list_files,
-            stream_processes,
-        )
+    (create_flags, create_positional_arguments, patterns_file) = make_base_create_command(
+        dry_run,
+        repository_path,
+        config,
+        patterns,
+        local_borg_version,
+        global_arguments,
+        borgmatic_runtime_directory,
+        local_path,
+        remote_path,
+        progress,
+        json,
+        list_files,
+        stream_processes,
     )
 
     if json:
@@ -417,8 +406,6 @@ def create_archive(
     # the terminal directly.
     output_file = DO_NOT_CAPTURE if progress else None
 
-    borg_environment = environment.make_environment(config)
-
     create_flags += (
         (('--info',) if logger.getEffectiveLevel() == logging.INFO and not json else ())
         + (('--stats',) if stats and not json and not dry_run else ())
@@ -435,7 +422,7 @@ def create_archive(
             output_log_level,
             output_file,
             working_directory=working_directory,
-            extra_environment=borg_environment,
+            environment=environment.make_environment(config),
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,
         )
@@ -443,7 +430,7 @@ def create_archive(
         return execute_command_and_capture_output(
             create_flags + create_positional_arguments,
             working_directory=working_directory,
-            extra_environment=borg_environment,
+            environment=environment.make_environment(config),
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,
         )
@@ -453,7 +440,7 @@ def create_archive(
             output_log_level,
             output_file,
             working_directory=working_directory,
-            extra_environment=borg_environment,
+            environment=environment.make_environment(config),
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,
         )

borgmatic/borg/delete.py

@@ -128,7 +128,7 @@ def delete_archives(
     borgmatic.execute.execute_command(
         command,
         output_log_level=logging.ANSWER,
-        extra_environment=borgmatic.borg.environment.make_environment(config),
+        environment=borgmatic.borg.environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/environment.py

@@ -1,5 +1,8 @@
 import os
 
+import borgmatic.borg.passcommand
+import borgmatic.hooks.credential.parse
+
 OPTION_TO_ENVIRONMENT_VARIABLE = {
     'borg_base_directory': 'BORG_BASE_DIR',
     'borg_config_directory': 'BORG_CONFIG_DIR',
@@ -7,8 +10,6 @@ OPTION_TO_ENVIRONMENT_VARIABLE = {
     'borg_files_cache_ttl': 'BORG_FILES_CACHE_TTL',
     'borg_security_directory': 'BORG_SECURITY_DIR',
     'borg_keys_directory': 'BORG_KEYS_DIR',
-    'encryption_passcommand': 'BORG_PASSCOMMAND',
-    'encryption_passphrase': 'BORG_PASSPHRASE',
     'ssh_command': 'BORG_RSH',
     'temporary_directory': 'TMPDIR',
 }
@@ -25,17 +26,59 @@ DEFAULT_BOOL_OPTION_TO_UPPERCASE_ENVIRONMENT_VARIABLE = {
 
 def make_environment(config):
     '''
-    Given a borgmatic configuration dict, return its options converted to a Borg environment
-    variable dict.
+    Given a borgmatic configuration dict, convert it to a Borg environment variable dict, merge it
+    with a copy of the current environment variables, and return the result.
+
+    Do not reuse this environment across multiple Borg invocations, because it can include
+    references to resources like anonymous pipes for passphrases—which can only be consumed once.
+
+    Here's how native Borg precedence works for a few of the environment variables:
+
+      1. BORG_PASSPHRASE, if set, is used first.
+      2. BORG_PASSCOMMAND is used only if BORG_PASSPHRASE isn't set.
+      3. BORG_PASSPHRASE_FD is used only if neither of the above are set.
+
+    In borgmatic, we want to simulate this precedence order, but there are some additional
+    complications. First, values can come from either configuration or from environment variables
+    set outside borgmatic; configured options should take precedence. Second, when borgmatic gets a
+    passphrase—directly from configuration or indirectly via a credential hook or a passcommand—we
+    want to pass that passphrase to Borg via an anonymous pipe (+ BORG_PASSPHRASE_FD), since that's
+    more secure than using an environment variable (BORG_PASSPHRASE).
     '''
-    environment = {}
+    environment = dict(os.environ)
 
     for option_name, environment_variable_name in OPTION_TO_ENVIRONMENT_VARIABLE.items():
         value = config.get(option_name)
 
-        if value:
+        if value is not None:
             environment[environment_variable_name] = str(value)
 
+    if 'encryption_passphrase' in config:
+        environment.pop('BORG_PASSPHRASE', None)
+        environment.pop('BORG_PASSCOMMAND', None)
+
+    if 'encryption_passcommand' in config:
+        environment.pop('BORG_PASSCOMMAND', None)
+
+    passphrase = borgmatic.hooks.credential.parse.resolve_credential(
+        config.get('encryption_passphrase'), config
+    )
+
+    if passphrase is None:
+        passphrase = borgmatic.borg.passcommand.get_passphrase_from_passcommand(config)
+
+    # If there's a passphrase (from configuration, from a configured credential, or from a
+    # configured passcommand), send it to Borg via an anonymous pipe.
+    if passphrase is not None:
+        read_file_descriptor, write_file_descriptor = os.pipe()
+        os.write(write_file_descriptor, passphrase.encode('utf-8'))
+        os.close(write_file_descriptor)
+
+        # This plus subprocess.Popen(..., close_fds=False) in execute.py is necessary for the Borg
+        # child process to inherit the file descriptor.
+        os.set_inheritable(read_file_descriptor, True)
+        environment['BORG_PASSPHRASE_FD'] = str(read_file_descriptor)
+
     for (
         option_name,
         environment_variable_name,

borgmatic/borg/export_key.py

@@ -60,14 +60,14 @@ def export_key(
     )
 
     if global_arguments.dry_run:
-        logger.info(f'{repository_path}: Skipping key export (dry run)')
+        logger.info('Skipping key export (dry run)')
         return
 
     execute_command(
         full_command,
         output_file=output_file,
         output_log_level=logging.ANSWER,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=working_directory,
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/export_tar.py

@@ -63,14 +63,14 @@ def export_tar_archive(
         output_log_level = logging.INFO
 
     if dry_run:
-        logging.info(f'{repository_path}: Skipping export to tar file (dry run)')
+        logging.info('Skipping export to tar file (dry run)')
         return
 
     execute_command(
         full_command,
         output_file=DO_NOT_CAPTURE if destination_path == '-' else None,
         output_log_level=output_log_level,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/extract.py

@@ -44,7 +44,6 @@ def extract_last_archive_dry_run(
         return
 
     list_flag = ('--list',) if logger.isEnabledFor(logging.DEBUG) else ()
-    borg_environment = environment.make_environment(config)
     full_extract_command = (
         (local_path, 'extract', '--dry-run')
         + (('--remote-path', remote_path) if remote_path else ())
@@ -59,7 +58,7 @@ def extract_last_archive_dry_run(
 
     execute_command(
         full_extract_command,
-        extra_environment=borg_environment,
+        environment=environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),
@@ -135,16 +134,13 @@ def extract_archive(
             # Make the repository path absolute so the destination directory used below via changing
             # the working directory doesn't prevent Borg from finding the repo. But also apply the
             # user's configured working directory (if any) to the repo path.
-            borgmatic.config.validate.normalize_repository_path(
-                os.path.join(working_directory or '', repository)
-            ),
+            borgmatic.config.validate.normalize_repository_path(repository, working_directory),
             archive,
             local_borg_version,
         )
         + (tuple(paths) if paths else ())
     )
 
-    borg_environment = environment.make_environment(config)
     borg_exit_codes = config.get('borg_exit_codes')
     full_destination_path = (
         os.path.join(working_directory or '', destination_path) if destination_path else None
@@ -156,7 +152,7 @@ def extract_archive(
         return execute_command(
             full_command,
             output_file=DO_NOT_CAPTURE,
-            extra_environment=borg_environment,
+            environment=environment.make_environment(config),
             working_directory=full_destination_path,
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,
@@ -168,7 +164,7 @@ def extract_archive(
             full_command,
             output_file=subprocess.PIPE,
             run_to_completion=False,
-            extra_environment=borg_environment,
+            environment=environment.make_environment(config),
             working_directory=full_destination_path,
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,
@@ -178,7 +174,7 @@ def extract_archive(
     # if the restore paths don't exist in the archive.
     execute_command(
         full_command,
-        extra_environment=borg_environment,
+        environment=environment.make_environment(config),
         working_directory=full_destination_path,
         borg_local_path=local_path,
         borg_exit_codes=borg_exit_codes,

borgmatic/borg/feature.py

@@ -17,6 +17,7 @@ class Feature(Enum):
     MATCH_ARCHIVES = 11
     EXCLUDED_FILES_MINUS = 12
     ARCHIVE_SERIES = 13
+    NO_PRUNE_STATS = 14
 
 
 FEATURE_TO_MINIMUM_BORG_VERSION = {
@@ -33,6 +34,7 @@ FEATURE_TO_MINIMUM_BORG_VERSION = {
     Feature.MATCH_ARCHIVES: parse('2.0.0b3'),  # borg --match-archives
     Feature.EXCLUDED_FILES_MINUS: parse('2.0.0b5'),  # --list --filter uses "-" for excludes
     Feature.ARCHIVE_SERIES: parse('2.0.0b11'),  # identically named archives form a series
+    Feature.NO_PRUNE_STATS: parse('2.0.0b10'),  # prune --stats is not available
 }
 
 

borgmatic/borg/flags.py

@@ -156,3 +156,44 @@ def warn_for_aggressive_archive_flags(json_command, json_output):
         logger.debug(f'Cannot parse JSON output from archive command: {error}')
     except (TypeError, KeyError):
         logger.debug('Cannot parse JSON output from archive command: No "archives" key found')
+
+
+def omit_flag(arguments, flag):
+    '''
+    Given a sequence of Borg command-line arguments, return them with the given (valueless) flag
+    omitted. For instance, if the flag is "--flag" and arguments is:
+
+        ('borg', 'create', '--flag', '--other-flag')
+
+    ... then return:
+
+        ('borg', 'create', '--other-flag')
+    '''
+    return tuple(argument for argument in arguments if argument != flag)
+
+
+def omit_flag_and_value(arguments, flag):
+    '''
+    Given a sequence of Borg command-line arguments, return them with the given flag and its
+    corresponding value omitted. For instance, if the flag is "--flag" and arguments is:
+
+        ('borg', 'create', '--flag', 'value', '--other-flag')
+
+    ... or:
+
+        ('borg', 'create', '--flag=value', '--other-flag')
+
+    ... then return:
+
+        ('borg', 'create', '--other-flag')
+    '''
+    # This works by zipping together a list of overlapping pairwise arguments. E.g., ('one', 'two',
+    # 'three', 'four') becomes ((None, 'one'), ('one, 'two'), ('two', 'three'), ('three', 'four')).
+    # This makes it easy to "look back" at the previous arguments so we can exclude both a flag and
+    # its value.
+    return tuple(
+        argument
+        for (previous_argument, argument) in zip((None,) + arguments, arguments)
+        if flag not in (previous_argument, argument)
+        if not argument.startswith(f'{flag}=')
+    )

borgmatic/borg/import_key.py

@@ -0,0 +1,70 @@
+import logging
+import os
+
+import borgmatic.config.paths
+import borgmatic.logger
+from borgmatic.borg import environment, flags
+from borgmatic.execute import DO_NOT_CAPTURE, execute_command
+
+logger = logging.getLogger(__name__)
+
+
+def import_key(
+    repository_path,
+    config,
+    local_borg_version,
+    import_arguments,
+    global_arguments,
+    local_path='borg',
+    remote_path=None,
+):
+    '''
+    Given a local or remote repository path, a configuration dict, the local Borg version, import
+    arguments, and optional local and remote Borg paths, import the repository key from the
+    path indicated in the import arguments.
+
+    If the path is empty or "-", then read the key from stdin.
+
+    Raise ValueError if the path is given and it does not exist.
+    '''
+    umask = config.get('umask', None)
+    lock_wait = config.get('lock_wait', None)
+    working_directory = borgmatic.config.paths.get_working_directory(config)
+
+    if import_arguments.path and import_arguments.path != '-':
+        if not os.path.exists(os.path.join(working_directory or '', import_arguments.path)):
+            raise ValueError(f'Path {import_arguments.path} does not exist. Aborting.')
+
+        input_file = None
+    else:
+        input_file = DO_NOT_CAPTURE
+
+    full_command = (
+        (local_path, 'key', 'import')
+        + (('--remote-path', remote_path) if remote_path else ())
+        + (('--umask', str(umask)) if umask else ())
+        + (('--log-json',) if global_arguments.log_json else ())
+        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
+        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+        + flags.make_flags('paper', import_arguments.paper)
+        + flags.make_repository_flags(
+            repository_path,
+            local_borg_version,
+        )
+        + ((import_arguments.path,) if input_file is None else ())
+    )
+
+    if global_arguments.dry_run:
+        logger.info('Skipping key import (dry run)')
+        return
+
+    execute_command(
+        full_command,
+        input_file=input_file,
+        output_log_level=logging.INFO,
+        environment=environment.make_environment(config),
+        working_directory=working_directory,
+        borg_local_path=local_path,
+        borg_exit_codes=config.get('borg_exit_codes'),
+    )

borgmatic/borg/info.py

@@ -102,7 +102,7 @@ def display_archives_info(
 
     json_info = execute_command_and_capture_output(
         json_command,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=working_directory,
         borg_local_path=local_path,
         borg_exit_codes=borg_exit_codes,
@@ -116,7 +116,7 @@ def display_archives_info(
     execute_command(
         main_command,
         output_log_level=logging.ANSWER,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=working_directory,
         borg_local_path=local_path,
         borg_exit_codes=borg_exit_codes,

borgmatic/borg/list.py

@@ -106,8 +106,6 @@ def capture_archive_listing(
     format to use for the output, and local and remote Borg paths, capture the
     output of listing that archive and return it as a list of file paths.
     '''
-    borg_environment = environment.make_environment(config)
-
     return tuple(
         execute_command_and_capture_output(
             make_list_command(
@@ -120,19 +118,19 @@ def capture_archive_listing(
                     paths=[path for path in list_paths] if list_paths else None,
                     find_paths=None,
                     json=None,
-                    format=path_format or '{path}{NL}',  # noqa: FS003
+                    format=path_format or '{path}{NUL}',  # noqa: FS003
                 ),
                 global_arguments,
                 local_path,
                 remote_path,
             ),
-            extra_environment=borg_environment,
+            environment=environment.make_environment(config),
             working_directory=borgmatic.config.paths.get_working_directory(config),
             borg_local_path=local_path,
             borg_exit_codes=config.get('borg_exit_codes'),
         )
-        .strip('\n')
-        .split('\n')
+        .strip('\0')
+        .split('\0')
     )
 
 
@@ -194,7 +192,6 @@ def list_archive(
             'The --json flag on the list action is not supported when using the --archive/--find flags.'
         )
 
-    borg_environment = environment.make_environment(config)
     borg_exit_codes = config.get('borg_exit_codes')
 
     # If there are any paths to find (and there's not a single archive already selected), start by
@@ -224,20 +221,20 @@ def list_archive(
                     local_path,
                     remote_path,
                 ),
-                extra_environment=borg_environment,
+                environment=environment.make_environment(config),
                 working_directory=borgmatic.config.paths.get_working_directory(config),
                 borg_local_path=local_path,
                 borg_exit_codes=borg_exit_codes,
             )
             .strip('\n')
-            .split('\n')
+            .splitlines()
         )
     else:
         archive_lines = (list_arguments.archive,)
 
     # For each archive listed by Borg, run list on the contents of that archive.
     for archive in archive_lines:
-        logger.answer(f'{repository_path}: Listing archive {archive}')
+        logger.answer(f'Listing archive {archive}')
 
         archive_arguments = copy.copy(list_arguments)
         archive_arguments.archive = archive
@@ -260,7 +257,7 @@ def list_archive(
         execute_command(
             main_command,
             output_log_level=logging.ANSWER,
-            extra_environment=borg_environment,
+            environment=environment.make_environment(config),
             working_directory=borgmatic.config.paths.get_working_directory(config),
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,

borgmatic/borg/mount.py

@@ -59,7 +59,6 @@ def mount_archive(
         + (tuple(mount_arguments.paths) if mount_arguments.paths else ())
     )
 
-    borg_environment = environment.make_environment(config)
     working_directory = borgmatic.config.paths.get_working_directory(config)
 
     # Don't capture the output when foreground mode is used so that ctrl-C can work properly.
@@ -67,7 +66,7 @@ def mount_archive(
         execute_command(
             full_command,
             output_file=DO_NOT_CAPTURE,
-            extra_environment=borg_environment,
+            environment=environment.make_environment(config),
             working_directory=working_directory,
             borg_local_path=local_path,
             borg_exit_codes=config.get('borg_exit_codes'),
@@ -76,7 +75,7 @@ def mount_archive(
 
     execute_command(
         full_command,
-        extra_environment=borg_environment,
+        environment=environment.make_environment(config),
         working_directory=working_directory,
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/passcommand.py

@@ -0,0 +1,40 @@
+import functools
+import logging
+import shlex
+
+import borgmatic.config.paths
+import borgmatic.execute
+
+logger = logging.getLogger(__name__)
+
+
+@functools.cache
+def run_passcommand(passcommand, working_directory):
+    '''
+    Run the given passcommand using the given working directory and return the passphrase produced
+    by the command.
+
+    Cache the results so that the passcommand only needs to run—and potentially prompt the user—once
+    per borgmatic invocation.
+    '''
+    return borgmatic.execute.execute_command_and_capture_output(
+        shlex.split(passcommand),
+        working_directory=working_directory,
+    )
+
+
+def get_passphrase_from_passcommand(config):
+    '''
+    Given the configuration dict, call the configured passcommand to produce and return an
+    encryption passphrase. In effect, we're doing an end-run around Borg by invoking its passcommand
+    ourselves. This allows us to pass the resulting passphrase to multiple different Borg
+    invocations without the user having to be prompted multiple times.
+
+    If no passcommand is configured, then return None.
+    '''
+    passcommand = config.get('encryption_passcommand')
+
+    if not passcommand:
+        return None
+
+    return run_passcommand(passcommand, borgmatic.config.paths.get_working_directory(config))

borgmatic/borg/pattern.py

@@ -0,0 +1,50 @@
+import collections
+import enum
+
+
+# See https://borgbackup.readthedocs.io/en/stable/usage/help.html#borg-help-patterns
+class Pattern_type(enum.Enum):
+    ROOT = 'R'  # A ROOT pattern always has a NONE pattern style.
+    PATTERN_STYLE = 'P'
+    EXCLUDE = '-'
+    NO_RECURSE = '!'
+    INCLUDE = '+'
+
+
+class Pattern_style(enum.Enum):
+    NONE = ''
+    FNMATCH = 'fm'
+    SHELL = 'sh'
+    REGULAR_EXPRESSION = 're'
+    PATH_PREFIX = 'pp'
+    PATH_FULL_MATCH = 'pf'
+
+
+class Pattern_source(enum.Enum):
+    '''
+    Where the pattern came from within borgmatic. This is important because certain use cases (like
+    filesystem snapshotting) only want to consider patterns that the user actually put in a
+    configuration file and not patterns from other sources.
+    '''
+
+    # The pattern is from a borgmatic configuration option, e.g. listed in "source_directories".
+    CONFIG = 'config'
+
+    # The pattern is generated internally within borgmatic, e.g. for special file excludes.
+    INTERNAL = 'internal'
+
+    # The pattern originates from within a borgmatic hook, e.g. a database hook that adds its dump
+    # directory.
+    HOOK = 'hook'
+
+
+Pattern = collections.namedtuple(
+    'Pattern',
+    ('path', 'type', 'style', 'device', 'source'),
+    defaults=(
+        Pattern_type.ROOT,
+        Pattern_style.NONE,
+        None,
+        Pattern_source.HOOK,
+    ),
+)

borgmatic/borg/prune.py

@@ -75,7 +75,13 @@ def prune_archives(
         + (('--umask', str(umask)) if umask else ())
         + (('--log-json',) if global_arguments.log_json else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
-        + (('--stats',) if prune_arguments.stats and not dry_run else ())
+        + (
+            ('--stats',)
+            if prune_arguments.stats
+            and not dry_run
+            and not feature.available(feature.Feature.NO_PRUNE_STATS, local_borg_version)
+            else ()
+        )
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + flags.make_flags_from_arguments(
             prune_arguments,
@@ -96,7 +102,7 @@ def prune_archives(
     execute_command(
         full_command,
         output_log_level=output_log_level,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/repo_create.py

@@ -57,7 +57,7 @@ def create_repository(
                 f'Requested encryption mode "{encryption_mode}" does not match existing repository encryption mode "{repository_encryption_mode}"'
             )
 
-        logger.info(f'{repository_path}: Repository already exists. Skipping creation.')
+        logger.info('Repository already exists. Skipping creation.')
         return
     except subprocess.CalledProcessError as error:
         if error.returncode not in REPO_INFO_REPOSITORY_NOT_FOUND_EXIT_CODES:
@@ -91,14 +91,14 @@ def create_repository(
     )
 
     if dry_run:
-        logging.info(f'{repository_path}: Skipping repository creation (dry run)')
+        logging.info('Skipping repository creation (dry run)')
         return
 
     # Do not capture output here, so as to support interactive prompts.
     execute_command(
         repo_create_command,
         output_file=DO_NOT_CAPTURE,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/repo_delete.py

@@ -88,7 +88,7 @@ def delete_repository(
             if repo_delete_arguments.force or repo_delete_arguments.cache_only
             else borgmatic.execute.DO_NOT_CAPTURE
         ),
-        extra_environment=borgmatic.borg.environment.make_environment(config),
+        environment=borgmatic.borg.environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/repo_info.py

@@ -50,14 +50,13 @@ def display_repository_info(
         + flags.make_repository_flags(repository_path, local_borg_version)
     )
 
-    extra_environment = environment.make_environment(config)
     working_directory = borgmatic.config.paths.get_working_directory(config)
     borg_exit_codes = config.get('borg_exit_codes')
 
     if repo_info_arguments.json:
         return execute_command_and_capture_output(
             full_command,
-            extra_environment=extra_environment,
+            environment=environment.make_environment(config),
             working_directory=working_directory,
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,
@@ -66,7 +65,7 @@ def display_repository_info(
         execute_command(
             full_command,
             output_log_level=logging.ANSWER,
-            extra_environment=extra_environment,
+            environment=environment.make_environment(config),
             working_directory=working_directory,
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,

borgmatic/borg/repo_list.py

@@ -49,7 +49,7 @@ def resolve_archive_name(
 
     output = execute_command_and_capture_output(
         full_command,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),
@@ -59,7 +59,7 @@ def resolve_archive_name(
     except IndexError:
         raise ValueError('No archives found in the repository')
 
-    logger.debug(f'{repository_path}: Latest archive is {latest_archive}')
+    logger.debug(f'Latest archive is {latest_archive}')
 
     return latest_archive
 
@@ -140,7 +140,6 @@ def list_repository(
     return JSON output).
     '''
     borgmatic.logger.add_custom_log_levels()
-    borg_environment = environment.make_environment(config)
 
     main_command = make_repo_list_command(
         repository_path,
@@ -165,7 +164,7 @@ def list_repository(
 
     json_listing = execute_command_and_capture_output(
         json_command,
-        extra_environment=borg_environment,
+        environment=environment.make_environment(config),
         working_directory=working_directory,
         borg_local_path=local_path,
         borg_exit_codes=borg_exit_codes,
@@ -179,7 +178,7 @@ def list_repository(
     execute_command(
         main_command,
         output_log_level=logging.ANSWER,
-        extra_environment=borg_environment,
+        environment=environment.make_environment(config),
         working_directory=working_directory,
         borg_local_path=local_path,
         borg_exit_codes=borg_exit_codes,

borgmatic/borg/transfer.py

@@ -57,7 +57,7 @@ def transfer_archives(
         full_command,
         output_log_level=logging.ANSWER,
         output_file=DO_NOT_CAPTURE if transfer_arguments.progress else None,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/version.py

@@ -21,7 +21,7 @@ def local_borg_version(config, local_path='borg'):
     )
     output = execute_command_and_capture_output(
         full_command,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/commands/arguments.py

@@ -349,12 +349,12 @@ def make_parsers():
     global_group.add_argument(
         '--log-file-format',
         type=str,
-        help='Log format string used for log messages written to the log file',
+        help='Python format string used for log messages written to the log file',
     )
     global_group.add_argument(
         '--log-json',
         action='store_true',
-        help='Write log messages and console output as one JSON object per log line instead of formatted text',
+        help='Write Borg log messages and console output as one JSON object per log line instead of formatted text',
     )
     global_group.add_argument(
         '--override',
@@ -547,7 +547,7 @@ def make_parsers():
         dest='stats',
         default=False,
         action='store_true',
-        help='Display statistics of the pruned archive',
+        help='Display statistics of the pruned archive [Borg 1 only]',
     )
     prune_group.add_argument(
         '--list', dest='list_archives', action='store_true', help='List archives kept/pruned'
@@ -1153,7 +1153,7 @@ def make_parsers():
         metavar='NAME',
         dest='data_sources',
         action='append',
-        help="Name of data source (e.g. database) to restore from archive, must be defined in borgmatic's configuration, can specify flag multiple times, defaults to all data sources in the archive",
+        help="Name of data source (e.g. database) to restore from the archive, must be defined in borgmatic's configuration, can specify the flag multiple times, defaults to all data sources in the archive",
     )
     restore_group.add_argument(
         '--schema',
@@ -1182,6 +1182,19 @@ def make_parsers():
         '--restore-path',
         help='Path to restore SQLite database dumps to. Defaults to the "restore_path" option in borgmatic\'s configuration',
     )
+    restore_group.add_argument(
+        '--original-hostname',
+        help='The hostname where the dump to restore came from, only necessary if you need to disambiguate dumps',
+    )
+    restore_group.add_argument(
+        '--original-port',
+        type=int,
+        help="The port where the dump to restore came from (if that port is in borgmatic's configuration), only necessary if you need to disambiguate dumps",
+    )
+    restore_group.add_argument(
+        '--hook',
+        help='The name of the data source hook for the dump to restore, only necessary if you need to disambiguate dumps',
+    )
     restore_group.add_argument(
         '-h', '--help', action='help', help='Show this help message and exit'
     )
@@ -1466,6 +1479,31 @@ def make_parsers():
         '-h', '--help', action='help', help='Show this help message and exit'
     )
 
+    key_import_parser = key_parsers.add_parser(
+        'import',
+        help='Import a copy of the repository key from backup',
+        description='Import a copy of the repository key from backup',
+        add_help=False,
+    )
+    key_import_group = key_import_parser.add_argument_group('key import arguments')
+    key_import_group.add_argument(
+        '--paper',
+        action='store_true',
+        help='Import interactively from a backup done with --paper',
+    )
+    key_import_group.add_argument(
+        '--repository',
+        help='Path of repository to import the key from, defaults to the configured repository if there is only one, quoted globs supported',
+    )
+    key_import_group.add_argument(
+        '--path',
+        metavar='PATH',
+        help='Path to import the key from backup, defaults to stdin',
+    )
+    key_import_group.add_argument(
+        '-h', '--help', action='help', help='Show this help message and exit'
+    )
+
     key_change_passphrase_parser = key_parsers.add_parser(
         'change-passphrase',
         help='Change the passphrase protecting the repository key',

borgmatic/config/generate.py

@@ -1,5 +1,6 @@
 import collections
 import io
+import itertools
 import os
 import re
 
@@ -24,41 +25,65 @@ def insert_newline_before_comment(config, field_name):
 def get_properties(schema):
     '''
     Given a schema dict, return its properties. But if it's got sub-schemas with multiple different
-    potential properties, returned their merged properties instead.
+    potential properties, returned their merged properties instead (interleaved so the first
+    properties of each sub-schema come first). The idea is that the user should see all possible
+    options even if they're not all possible together.
     '''
     if 'oneOf' in schema:
         return dict(
-            collections.ChainMap(*[sub_schema['properties'] for sub_schema in schema['oneOf']])
+            item
+            for item in itertools.chain(
+                *itertools.zip_longest(
+                    *[sub_schema['properties'].items() for sub_schema in schema['oneOf']]
+                )
+            )
+            if item is not None
         )
 
     return schema['properties']
 
 
-def schema_to_sample_configuration(schema, level=0, parent_is_sequence=False):
+def schema_to_sample_configuration(schema, source_config=None, level=0, parent_is_sequence=False):
     '''
-    Given a loaded configuration schema, generate and return sample config for it. Include comments
-    for each option based on the schema "description".
+    Given a loaded configuration schema and a source configuration, generate and return sample
+    config for the schema. Include comments for each option based on the schema "description".
+
+    If a source config is given, walk it alongside the given schema so that both can be taken into
+    account when commenting out particular options in add_comments_to_configuration_object().
     '''
     schema_type = schema.get('type')
     example = schema.get('example')
+
     if example is not None:
         return example
 
     if schema_type == 'array' or (isinstance(schema_type, list) and 'array' in schema_type):
         config = ruamel.yaml.comments.CommentedSeq(
-            [schema_to_sample_configuration(schema['items'], level, parent_is_sequence=True)]
+            [
+                schema_to_sample_configuration(
+                    schema['items'], source_config, level, parent_is_sequence=True
+                )
+            ]
         )
         add_comments_to_configuration_sequence(config, schema, indent=(level * INDENT))
     elif schema_type == 'object' or (isinstance(schema_type, list) and 'object' in schema_type):
+        if source_config and isinstance(source_config, list) and isinstance(source_config[0], dict):
+            source_config = dict(collections.ChainMap(*source_config))
+
         config = ruamel.yaml.comments.CommentedMap(
             [
-                (field_name, schema_to_sample_configuration(sub_schema, level + 1))
+                (
+                    field_name,
+                    schema_to_sample_configuration(
+                        sub_schema, (source_config or {}).get(field_name, {}), level + 1
+                    ),
+                )
                 for field_name, sub_schema in get_properties(schema).items()
             ]
         )
         indent = (level * INDENT) + (SEQUENCE_INDENT if parent_is_sequence else 0)
         add_comments_to_configuration_object(
-            config, schema, indent=indent, skip_first=parent_is_sequence
+            config, schema, source_config, indent=indent, skip_first=parent_is_sequence
         )
     else:
         raise ValueError(f'Schema at level {level} is unsupported: {schema}')
@@ -178,14 +203,21 @@ def add_comments_to_configuration_sequence(config, schema, indent=0):
         return
 
 
-REQUIRED_KEYS = {'source_directories', 'repositories', 'keep_daily'}
+DEFAULT_KEYS = {'source_directories', 'repositories', 'keep_daily'}
 COMMENTED_OUT_SENTINEL = 'COMMENT_OUT'
 
 
-def add_comments_to_configuration_object(config, schema, indent=0, skip_first=False):
+def add_comments_to_configuration_object(
+    config, schema, source_config=None, indent=0, skip_first=False
+):
     '''
     Using descriptions from a schema as a source, add those descriptions as comments to the given
-    config mapping, before each field. Indent the comment the given number of characters.
+    configuration dict, putting them before each field. Indent the comment the given number of
+    characters.
+
+    And a sentinel for commenting out options that are neither in DEFAULT_KEYS nor the the given
+    source configuration dict. The idea is that any options used in the source configuration should
+    stay active in the generated configuration.
     '''
     for index, field_name in enumerate(config.keys()):
         if skip_first and index == 0:
@@ -194,10 +226,12 @@ def add_comments_to_configuration_object(config, schema, indent=0, skip_first=Fa
         field_schema = get_properties(schema).get(field_name, {})
         description = field_schema.get('description', '').strip()
 
-        # If this is an optional key, add an indicator to the comment flagging it to be commented
+        # If this isn't a default key, add an indicator to the comment flagging it to be commented
         # out from the sample configuration. This sentinel is consumed by downstream processing that
         # does the actual commenting out.
-        if field_name not in REQUIRED_KEYS:
+        if field_name not in DEFAULT_KEYS and (
+            source_config is None or field_name not in source_config
+        ):
             description = (
                 '\n'.join((description, COMMENTED_OUT_SENTINEL))
                 if description
@@ -217,21 +251,6 @@ def add_comments_to_configuration_object(config, schema, indent=0, skip_first=Fa
 RUAMEL_YAML_COMMENTS_INDEX = 1
 
 
-def remove_commented_out_sentinel(config, field_name):
-    '''
-    Given a configuration CommentedMap and a top-level field name in it, remove any "commented out"
-    sentinel found at the end of its YAML comments. This prevents the given field name from getting
-    commented out by downstream processing that consumes the sentinel.
-    '''
-    try:
-        last_comment_value = config.ca.items[field_name][RUAMEL_YAML_COMMENTS_INDEX][-1].value
-    except KeyError:
-        return
-
-    if last_comment_value == f'# {COMMENTED_OUT_SENTINEL}\n':
-        config.ca.items[field_name][RUAMEL_YAML_COMMENTS_INDEX].pop()
-
-
 def merge_source_configuration_into_destination(destination_config, source_config):
     '''
     Deep merge the given source configuration dict into the destination configuration CommentedMap,
@@ -246,12 +265,6 @@ def merge_source_configuration_into_destination(destination_config, source_confi
         return source_config
 
     for field_name, source_value in source_config.items():
-        # Since this key/value is from the source configuration, leave it uncommented and remove any
-        # sentinel that would cause it to get commented out.
-        remove_commented_out_sentinel(
-            ruamel.yaml.comments.CommentedMap(destination_config), field_name
-        )
-
         # This is a mapping. Recurse for this key/value.
         if isinstance(source_value, collections.abc.Mapping):
             destination_config[field_name] = merge_source_configuration_into_destination(
@@ -297,7 +310,7 @@ def generate_sample_configuration(
         normalize.normalize(source_filename, source_config)
 
     destination_config = merge_source_configuration_into_destination(
-        schema_to_sample_configuration(schema), source_config
+        schema_to_sample_configuration(schema, source_config), source_config
     )
 
     if dry_run:

borgmatic/config/load.py

@@ -69,7 +69,7 @@ def include_configuration(loader, filename_node, include_directory, config_paths
         ]
 
     raise ValueError(
-        '!include value is not supported; use a single filename or a list of filenames'
+        'The value given for the !include tag is invalid; use a single filename or a list of filenames instead'
     )
 
 

borgmatic/config/normalize.py

@@ -58,6 +58,90 @@ def normalize_sections(config_filename, config):
     return []
 
 
+def make_command_hook_deprecation_log(config_filename, option_name):  # pragma: no cover
+    '''
+    Given a configuration filename and the name of a configuration option, return a deprecation
+    warning log for it.
+    '''
+    return logging.makeLogRecord(
+        dict(
+            levelno=logging.WARNING,
+            levelname='WARNING',
+            msg=f'{config_filename}: {option_name} is deprecated and support will be removed from a future release. Use commands: instead.',
+        )
+    )
+
+
+def normalize_commands(config_filename, config):
+    '''
+    Given a configuration filename and a configuration dict, transform any "before_*"- and
+    "after_*"-style command hooks into "commands:".
+    '''
+    logs = []
+
+    # Normalize "before_actions" and "after_actions".
+    for preposition in ('before', 'after'):
+        option_name = f'{preposition}_actions'
+        commands = config.pop(option_name, None)
+
+        if commands:
+            logs.append(make_command_hook_deprecation_log(config_filename, option_name))
+            config.setdefault('commands', []).append(
+                {
+                    preposition: 'repository',
+                    'run': commands,
+                }
+            )
+
+    # Normalize "before_backup", "before_prune", "after_backup", "after_prune", etc.
+    for action_name in ('create', 'prune', 'compact', 'check', 'extract'):
+        for preposition in ('before', 'after'):
+            option_name = f'{preposition}_{"backup" if action_name == "create" else action_name}'
+            commands = config.pop(option_name, None)
+
+            if not commands:
+                continue
+
+            logs.append(make_command_hook_deprecation_log(config_filename, option_name))
+            config.setdefault('commands', []).append(
+                {
+                    preposition: 'action',
+                    'when': [action_name],
+                    'run': commands,
+                }
+            )
+
+    # Normalize "on_error".
+    commands = config.pop('on_error', None)
+
+    if commands:
+        logs.append(make_command_hook_deprecation_log(config_filename, 'on_error'))
+        config.setdefault('commands', []).append(
+            {
+                'after': 'error',
+                'when': ['create', 'prune', 'compact', 'check'],
+                'run': commands,
+            }
+        )
+
+    # Normalize "before_everything" and "after_everything".
+    for preposition in ('before', 'after'):
+        option_name = f'{preposition}_everything'
+        commands = config.pop(option_name, None)
+
+        if commands:
+            logs.append(make_command_hook_deprecation_log(config_filename, option_name))
+            config.setdefault('commands', []).append(
+                {
+                    preposition: 'everything',
+                    'when': ['create'],
+                    'run': commands,
+                }
+            )
+
+    return logs
+
+
 def normalize(config_filename, config):
     '''
     Given a configuration filename and a configuration dict of its loaded contents, apply particular
@@ -67,6 +151,7 @@ def normalize(config_filename, config):
     Raise ValueError the configuration cannot be normalized.
     '''
     logs = normalize_sections(config_filename, config)
+    logs += normalize_commands(config_filename, config)
 
     if config.get('borgmatic_source_directory'):
         logs.append(

borgmatic/config/paths.py

@@ -33,10 +33,13 @@ def get_borgmatic_source_directory(config):
 TEMPORARY_DIRECTORY_PREFIX = 'borgmatic-'
 
 
-def replace_temporary_subdirectory_with_glob(path):
+def replace_temporary_subdirectory_with_glob(
+    path, temporary_directory_prefix=TEMPORARY_DIRECTORY_PREFIX
+):
     '''
-    Given an absolute temporary directory path, look for a subdirectory within it starting with the
-    temporary directory prefix and replace it with an appropriate glob. For instance, given:
+    Given an absolute temporary directory path and an optional temporary directory prefix, look for
+    a subdirectory within it starting with the temporary directory prefix (or a default) and replace
+    it with an appropriate glob. For instance, given:
 
         /tmp/borgmatic-aet8kn93/borgmatic
 
@@ -50,8 +53,8 @@ def replace_temporary_subdirectory_with_glob(path):
         '/',
         *(
             (
-                f'{TEMPORARY_DIRECTORY_PREFIX}*'
-                if subdirectory.startswith(TEMPORARY_DIRECTORY_PREFIX)
+                f'{temporary_directory_prefix}*'
+                if subdirectory.startswith(temporary_directory_prefix)
                 else subdirectory
             )
             for subdirectory in path.split(os.path.sep)
@@ -73,14 +76,13 @@ class Runtime_directory:
     automatically gets cleaned up as necessary.
     '''
 
-    def __init__(self, config, log_prefix):
+    def __init__(self, config):
         '''
-        Given a configuration dict and a log prefix, determine the borgmatic runtime directory,
-        creating a secure, temporary directory within it if necessary. Defaults to
-        $XDG_RUNTIME_DIR/./borgmatic or $RUNTIME_DIRECTORY/./borgmatic or
-        $TMPDIR/borgmatic-[random]/./borgmatic or $TEMP/borgmatic-[random]/./borgmatic or
-        /tmp/borgmatic-[random]/./borgmatic where "[random]" is a randomly generated string intended
-        to avoid path collisions.
+        Given a configuration dict determine the borgmatic runtime directory, creating a secure,
+        temporary directory within it if necessary. Defaults to $XDG_RUNTIME_DIR/./borgmatic or
+        $RUNTIME_DIRECTORY/./borgmatic or $TMPDIR/borgmatic-[random]/./borgmatic or
+        $TEMP/borgmatic-[random]/./borgmatic or /tmp/borgmatic-[random]/./borgmatic where "[random]"
+        is a randomly generated string intended to avoid path collisions.
 
         If XDG_RUNTIME_DIR or RUNTIME_DIRECTORY is set and already ends in "/borgmatic", then don't
         tack on a second "/borgmatic" path component.
@@ -124,7 +126,7 @@ class Runtime_directory:
         )
         os.makedirs(self.runtime_path, mode=0o700, exist_ok=True)
 
-        logger.debug(f'{log_prefix}: Using runtime directory {os.path.normpath(self.runtime_path)}')
+        logger.debug(f'Using runtime directory {os.path.normpath(self.runtime_path)}')
 
     def __enter__(self):
         '''
@@ -132,7 +134,7 @@ class Runtime_directory:
         '''
         return self.runtime_path
 
-    def __exit__(self, exception, value, traceback):
+    def __exit__(self, exception_type, exception, traceback):
         '''
         Delete any temporary directory that was created as part of initialization.
         '''

borgmatic/config/schema.yaml

@@ -68,9 +68,7 @@ properties:
         type: boolean
         description: |
             Stay in same file system; do not cross mount points beyond the given
-            source directories. Defaults to false. But when a database hook is
-            used, the setting here is ignored and one_file_system is considered
-            true.
+            source directories. Defaults to false.
         example: true
     numeric_ids:
         type: boolean
@@ -133,8 +131,7 @@ properties:
             Any paths matching these patterns are included/excluded from
             backups. Globs are expanded. (Tildes are not.) See the output of
             "borg help patterns" for more details. Quote any value if it
-            contains leading punctuation, so it parses correctly. Note that only
-            one of "patterns" and "source_directories" may be used.
+            contains leading punctuation, so it parses correctly.
         example:
             - 'R /'
             - '- /home/*/.cache'
@@ -146,9 +143,8 @@ properties:
             type: string
         description: |
             Read include/exclude patterns from one or more separate named files,
-            one pattern per line. Note that Borg considers this option
-            experimental. See the output of "borg help patterns" for more
-            details.
+            one pattern per line. See the output of "borg help patterns" for
+            more details.
         example:
             - /etc/borgmatic/patterns
     exclude_patterns:
@@ -209,8 +205,8 @@ properties:
         description: |
             Deprecated. Only used for locating database dumps and bootstrap
             metadata within backup archives created prior to deprecation.
-            Replaced by borgmatic_runtime_directory and
-            borgmatic_state_directory. Defaults to ~/.borgmatic
+            Replaced by user_runtime_directory and user_state_directory.
+            Defaults to ~/.borgmatic
         example: /tmp/borgmatic
     user_runtime_directory:
         type: string
@@ -232,8 +228,8 @@ properties:
     source_directories_must_exist:
         type: boolean
         description: |
-            If true, then source directories must exist, otherwise an error is
-            raised. Defaults to false.
+            If true, then source directories (and root pattern paths) must
+            exist. If they don't, an error is raised. Defaults to false.
         example: true
     encryption_passcommand:
         type: string
@@ -254,7 +250,7 @@ properties:
             repositories that were initialized with passphrase/repokey/keyfile
             encryption. Quote the value if it contains punctuation, so it parses
             correctly. And backslash any quote or backslash literals as well.
-            Defaults to not set.
+            Defaults to not set. Supports the "{credential ...}" syntax.
         example: "!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~"
     checkpoint_interval:
         type: integer
@@ -636,8 +632,8 @@ properties:
                               long-running repository check into multiple
                               partial checks. Defaults to no interruption. Only
                               applies to the "repository" check, does not check
-                              the repository index, and is not compatible with a
-                              simultaneous "archives" check or "--repair" flag.
+                              the repository index and is not compatible with
+                              the "--repair" flag.
                           example: 3600
                 - required:
                     - name
@@ -800,8 +796,9 @@ properties:
         items:
             type: string
         description: |
-            List of one or more shell commands or scripts to execute before all
-            the actions for each repository.
+            Deprecated. Use "commands:" instead. List of one or more shell
+            commands or scripts to execute before all the actions for each
+            repository.
         example:
             - "echo Starting actions."
     before_backup:
@@ -809,8 +806,9 @@ properties:
         items:
             type: string
         description: |
-            List of one or more shell commands or scripts to execute before
-            creating a backup, run once per repository.
+            Deprecated. Use "commands:" instead. List of one or more shell
+            commands or scripts to execute before creating a backup, run once
+            per repository.
         example:
             - "echo Starting a backup."
     before_prune:
@@ -818,8 +816,9 @@ properties:
         items:
             type: string
         description: |
-            List of one or more shell commands or scripts to execute before
-            pruning, run once per repository.
+            Deprecated. Use "commands:" instead. List of one or more shell
+            commands or scripts to execute before pruning, run once per
+            repository.
         example:
             - "echo Starting pruning."
     before_compact:
@@ -827,8 +826,9 @@ properties:
         items:
             type: string
         description: |
-            List of one or more shell commands or scripts to execute before
-            compaction, run once per repository.
+            Deprecated. Use "commands:" instead. List of one or more shell
+            commands or scripts to execute before compaction, run once per
+            repository.
         example:
             - "echo Starting compaction."
     before_check:
@@ -836,8 +836,9 @@ properties:
         items:
             type: string
         description: |
-            List of one or more shell commands or scripts to execute before
-            consistency checks, run once per repository.
+            Deprecated. Use "commands:" instead. List of one or more shell
+            commands or scripts to execute before consistency checks, run once
+            per repository.
         example:
             - "echo Starting checks."
     before_extract:
@@ -845,8 +846,9 @@ properties:
         items:
             type: string
         description: |
-            List of one or more shell commands or scripts to execute before
-            extracting a backup, run once per repository.
+            Deprecated. Use "commands:" instead. List of one or more shell
+            commands or scripts to execute before extracting a backup, run once
+            per repository.
         example:
             - "echo Starting extracting."
     after_backup:
@@ -854,8 +856,9 @@ properties:
         items:
             type: string
         description: |
-            List of one or more shell commands or scripts to execute after
-            creating a backup, run once per repository.
+            Deprecated. Use "commands:" instead. List of one or more shell
+            commands or scripts to execute after creating a backup, run once per
+            repository.
         example:
             - "echo Finished a backup."
     after_compact:
@@ -863,8 +866,9 @@ properties:
         items:
             type: string
         description: |
-            List of one or more shell commands or scripts to execute after
-            compaction, run once per repository.
+            Deprecated. Use "commands:" instead. List of one or more shell
+            commands or scripts to execute after compaction, run once per
+            repository.
         example:
             - "echo Finished compaction."
     after_prune:
@@ -872,8 +876,9 @@ properties:
         items:
             type: string
         description: |
-            List of one or more shell commands or scripts to execute after
-            pruning, run once per repository.
+            Deprecated. Use "commands:" instead. List of one or more shell
+            commands or scripts to execute after pruning, run once per
+            repository.
         example:
             - "echo Finished pruning."
     after_check:
@@ -881,8 +886,9 @@ properties:
         items:
             type: string
         description: |
-            List of one or more shell commands or scripts to execute after
-            consistency checks, run once per repository.
+            Deprecated. Use "commands:" instead. List of one or more shell
+            commands or scripts to execute after consistency checks, run once
+            per repository.
         example:
             - "echo Finished checks."
     after_extract:
@@ -890,8 +896,9 @@ properties:
         items:
             type: string
         description: |
-            List of one or more shell commands or scripts to execute after
-            extracting a backup, run once per repository.
+            Deprecated. Use "commands:" instead. List of one or more shell
+            commands or scripts to execute after extracting a backup, run once
+            per repository.
         example:
             - "echo Finished extracting."
     after_actions:
@@ -899,8 +906,9 @@ properties:
         items:
             type: string
         description: |
-            List of one or more shell commands or scripts to execute after all
-            actions for each repository.
+            Deprecated. Use "commands:" instead. List of one or more shell
+            commands or scripts to execute after all actions for each
+            repository.
         example:
             - "echo Finished actions."
     on_error:
@@ -908,9 +916,10 @@ properties:
         items:
             type: string
         description: |
-            List of one or more shell commands or scripts to execute when an
-            exception occurs during a "create", "prune", "compact", or "check"
-            action or an associated before/after hook.
+            Deprecated. Use "commands:" instead. List of one or more shell
+            commands or scripts to execute when an exception occurs during a
+            "create", "prune", "compact", or "check" action or an associated
+            before/after hook.
         example:
             - "echo Error during create/prune/compact/check."
     before_everything:
@@ -918,10 +927,10 @@ properties:
         items:
             type: string
         description: |
-            List of one or more shell commands or scripts to execute before
-            running all actions (if one of them is "create"). These are
-            collected from all configuration files and then run once before all
-            of them (prior to all actions).
+            Deprecated. Use "commands:" instead. List of one or more shell
+            commands or scripts to execute before running all actions (if one of
+            them is "create"). These are collected from all configuration files
+            and then run once before all of them (prior to all actions).
         example:
             - "echo Starting actions."
     after_everything:
@@ -929,12 +938,148 @@ properties:
         items:
             type: string
         description: |
-            List of one or more shell commands or scripts to execute after
-            running all actions (if one of them is "create"). These are
-            collected from all configuration files and then run once after all
-            of them (after any action).
+            Deprecated. Use "commands:" instead. List of one or more shell
+            commands or scripts to execute after running all actions (if one of
+            them is "create"). These are collected from all configuration files
+            and then run once after all of them (after any action).
         example:
             - "echo Completed actions."
+    commands:
+        type: array
+        items:
+            type: object
+            oneOf:
+                - required: [before, run]
+                  additionalProperties: false
+                  properties:
+                      before:
+                          type: string
+                          enum:
+                              - action
+                              - repository
+                              - configuration
+                              - everything
+                          description: |
+                              Name for the point in borgmatic's execution that
+                              the commands should be run before (required if
+                              "after" isn't set):
+                               * "action" runs before each action for each
+                              repository.
+                               * "repository" runs before all actions for each
+                              repository.
+                               * "configuration" runs before all actions and
+                              repositories in the current configuration file.
+                               * "everything" runs before all configuration
+                              files.
+                          example: action
+                      when:
+                          type: array
+                          items:
+                              type: string
+                              enum:
+                                  - repo-create
+                                  - transfer
+                                  - prune
+                                  - compact
+                                  - create
+                                  - check
+                                  - delete
+                                  - extract
+                                  - config
+                                  - export-tar
+                                  - mount
+                                  - umount
+                                  - repo-delete
+                                  - restore
+                                  - repo-list
+                                  - list
+                                  - repo-info
+                                  - info
+                                  - break-lock
+                                  - key
+                                  - borg
+                          description: |
+                              List of actions for which the commands will be
+                              run. Defaults to running for all actions.
+                          example: [create, prune, compact, check]
+                      run:
+                          type: array
+                          items:
+                              type: string
+                          description: |
+                              List of one or more shell commands or scripts to
+                              run when this command hook is triggered. Required.
+                          example:
+                              - "echo Doing stuff."
+                - required: [after, run]
+                  additionalProperties: false
+                  properties:
+                      after:
+                          type: string
+                          enum:
+                              - action
+                              - repository
+                              - configuration
+                              - everything
+                              - error
+                          description: |
+                              Name for the point in borgmatic's execution that
+                              the commands should be run after (required if
+                              "before" isn't set):
+                               * "action" runs after each action for each
+                              repository.
+                               * "repository" runs after all actions for each
+                              repository.
+                               * "configuration" runs after all actions and
+                              repositories in the current configuration file.
+                               * "everything" runs after all configuration
+                              files.
+                               * "error" runs after an error occurs.
+                          example: action
+                      when:
+                          type: array
+                          items:
+                              type: string
+                              enum:
+                                  - repo-create
+                                  - transfer
+                                  - prune
+                                  - compact
+                                  - create
+                                  - check
+                                  - delete
+                                  - extract
+                                  - config
+                                  - export-tar
+                                  - mount
+                                  - umount
+                                  - repo-delete
+                                  - restore
+                                  - repo-list
+                                  - list
+                                  - repo-info
+                                  - info
+                                  - break-lock
+                                  - key
+                                  - borg
+                          description: |
+                              Only trigger the hook when borgmatic is run with
+                              particular actions listed here. Defaults to
+                              running for all actions.
+                          example: [create, prune, compact, check]
+                      run:
+                          type: array
+                          items:
+                              type: string
+                          description: |
+                              List of one or more shell commands or scripts to
+                              run when this command hook is triggered. Required.
+                          example:
+                              - "echo Doing stuff."
+        description: |
+            List of one or more command hooks to execute, triggered at
+            particular points during borgmatic's execution. For each command
+            hook, specify one of "before" or "after", not both.
     bootstrap:
         type: object
         properties:
@@ -963,8 +1108,8 @@ properties:
                         dump all databases on the host. (Also set the "format"
                         to dump each database to a separate file instead of one
                         combined file.) Note that using this database hook
-                        implicitly enables both read_special and one_file_system
-                        (see above) to support dump and restore streaming.
+                        implicitly enables read_special (see above) to support
+                        dump and restore streaming.
                     example: users
                 hostname:
                     type: string
@@ -993,13 +1138,15 @@ properties:
                         Username with which to connect to the database. Defaults
                         to the username of the current user. You probably want
                         to specify the "postgres" superuser here when the
-                        database name is "all".
+                        database name is "all". Supports the "{credential ...}"
+                        syntax.
                     example: dbuser
                 restore_username:
                     type: string
                     description: |
                         Username with which to restore the database. Defaults to
-                        the "username" option.
+                        the "username" option. Supports the "{credential ...}"
+                        syntax.
                     example: dbuser
                 password:
                     type: string
@@ -1007,13 +1154,15 @@ properties:
                         Password with which to connect to the database. Omitting
                         a password will only work if PostgreSQL is configured to
                         trust the configured username without a password or you
-                        create a ~/.pgpass file.
+                        create a ~/.pgpass file. Supports the "{credential ...}"
+                        syntax.
                     example: trustsome1
                 restore_password:
                     type: string
                     description: |
                         Password with which to connect to the restore database.
-                        Defaults to the "password" option.
+                        Defaults to the "password" option. Supports the
+                        "{credential ...}" syntax.
                     example: trustsome1
                 no_owner:
                     type: boolean
@@ -1040,6 +1189,18 @@ properties:
                         individual databases. See the pg_dump documentation for
                         more about formats.
                     example: directory
+                compression:
+                    type: ["string", "integer"]
+                    description: |
+                        Database dump compression level (integer) or method
+                        ("gzip", "lz4", "zstd", or "none") and optional
+                        colon-separated detail. Defaults to moderate "gzip" for
+                        "custom" and "directory" formats and no compression for
+                        the "plain" format. Compression is not supported for the
+                        "tar" format. Be aware that Borg does its own
+                        compression as well, so you may not need it in both
+                        places.
+                    example: none
                 ssl_mode:
                     type: string
                     enum: ['disable', 'allow', 'prefer',
@@ -1076,11 +1237,11 @@ properties:
                         Command to use instead of "pg_dump" or "pg_dumpall".
                         This can be used to run a specific pg_dump version
                         (e.g., one inside a running container). If you run it
-                        from within a container, make sure to mount your
-                        host's ".borgmatic" folder into the container using
-                        the same directory structure. Defaults to "pg_dump"
-                        for single database dump or "pg_dumpall" to dump all
-                        databases.
+                        from within a container, make sure to mount the path in
+                        the "user_runtime_directory" option from the host into
+                        the container at the same location. Defaults to
+                        "pg_dump" for single database dump or "pg_dumpall" to
+                        dump all databases.
                     example: docker exec my_pg_container pg_dump
                 pg_restore_command:
                     type: string
@@ -1145,9 +1306,8 @@ properties:
                     description: |
                         Database name (required if using this hook). Or "all" to
                         dump all databases on the host. Note that using this
-                        database hook implicitly enables both read_special and
-                        one_file_system (see above) to support dump and restore
-                        streaming.
+                        database hook implicitly enables read_special (see
+                        above) to support dump and restore streaming.
                     example: users
                 hostname:
                     type: string
@@ -1174,13 +1334,15 @@ properties:
                     type: string
                     description: |
                         Username with which to connect to the database. Defaults
-                        to the username of the current user.
+                        to the username of the current user. Supports the
+                        "{credential ...}" syntax.
                     example: dbuser
                 restore_username:
                     type: string
                     description: |
                         Username with which to restore the database. Defaults to
-                        the "username" option.
+                        the "username" option. Supports the "{credential ...}"
+                        syntax.
                     example: dbuser
                 password:
                     type: string
@@ -1188,16 +1350,39 @@ properties:
                         Password with which to connect to the database. Omitting
                         a password will only work if MariaDB is configured to
                         trust the configured username without a password.
+                        Supports the "{credential ...}" syntax.
                     example: trustsome1
+                restore_password:
+                    type: string
+                    description: |
+                        Password with which to connect to the restore database.
+                        Defaults to the "password" option. Supports the
+                        "{credential ...}" syntax.
+                    example: trustsome1
+                tls:
+                    type: boolean
+                    description: |
+                        Whether to TLS-encrypt data transmitted between the
+                        client and server. The default varies based on the
+                        MariaDB version.
+                    example: false
+                restore_tls:
+                    type: boolean
+                    description: |
+                        Whether to TLS-encrypt data transmitted between the
+                        client and restore server. The default varies based on
+                        the MariaDB version.
+                    example: false
                 mariadb_dump_command:
                     type: string
                     description: |
                         Command to use instead of "mariadb-dump". This can be
                         used to run a specific mariadb_dump version (e.g., one
-                        inside a running container). If you run it from within
-                        a container, make sure to mount your host's
-                        ".borgmatic" folder into the container using the same
-                        directory structure. Defaults to "mariadb-dump".
+                        inside a running container). If you run it from within a
+                        container, make sure to mount the path in the
+                        "user_runtime_directory" option from the host into the
+                        container at the same location. Defaults to
+                        "mariadb-dump".
                     example: docker exec mariadb_container mariadb-dump
                 mariadb_command:
                     type: string
@@ -1206,12 +1391,6 @@ properties:
                         run a specific mariadb version (e.g., one inside a
                         running container). Defaults to "mariadb".
                     example: docker exec mariadb_container mariadb           
-                restore_password:
-                    type: string
-                    description: |
-                        Password with which to connect to the restore database.
-                        Defaults to the "password" option.
-                    example: trustsome1
                 format:
                     type: string
                     enum: ['sql']
@@ -1272,9 +1451,8 @@ properties:
                     description: |
                         Database name (required if using this hook). Or "all" to
                         dump all databases on the host. Note that using this
-                        database hook implicitly enables both read_special and
-                        one_file_system (see above) to support dump and restore
-                        streaming.
+                        database hook implicitly enables read_special (see
+                        above) to support dump and restore streaming.
                     example: users
                 hostname:
                     type: string
@@ -1301,13 +1479,15 @@ properties:
                     type: string
                     description: |
                         Username with which to connect to the database. Defaults
-                        to the username of the current user.
+                        to the username of the current user. Supports the
+                        "{credential ...}" syntax.
                     example: dbuser
                 restore_username:
                     type: string
                     description: |
                         Username with which to restore the database. Defaults to
-                        the "username" option.
+                        the "username" option. Supports the "{credential ...}"
+                        syntax.
                     example: dbuser
                 password:
                     type: string
@@ -1315,22 +1495,38 @@ properties:
                         Password with which to connect to the database. Omitting
                         a password will only work if MySQL is configured to
                         trust the configured username without a password.
+                        Supports the "{credential ...}" syntax.
                     example: trustsome1
                 restore_password:
                     type: string
                     description: |
                         Password with which to connect to the restore database.
-                        Defaults to the "password" option.
+                        Defaults to the "password" option. Supports the
+                        "{credential ...}" syntax.
                     example: trustsome1
+                tls:
+                    type: boolean
+                    description: |
+                        Whether to TLS-encrypt data transmitted between the
+                        client and server. The default varies based on the
+                        MySQL installation.
+                    example: false
+                restore_tls:
+                    type: boolean
+                    description: |
+                        Whether to TLS-encrypt data transmitted between the
+                        client and restore server. The default varies based on
+                        the MySQL installation.
+                    example: false
                 mysql_dump_command:
                     type: string
                     description: |
-                        Command to use instead of "mysqldump". This can be
-                        used to run a specific mysql_dump version (e.g., one
-                        inside a running container). If you run it from within
-                        a container, make sure to mount your host's
-                        ".borgmatic" folder into the container using the same
-                        directory structure. Defaults to "mysqldump".
+                        Command to use instead of "mysqldump". This can be used
+                        to run a specific mysql_dump version (e.g., one inside a
+                        running container). If you run it from within a
+                        container, make sure to mount the path in the
+                        "user_runtime_directory" option from the host into the
+                        container at the same location. Defaults to "mysqldump".
                     example: docker exec mysql_container mysqldump
                 mysql_command:
                     type: string
@@ -1407,9 +1603,9 @@ properties:
                     description: |
                         Path to the SQLite database file to dump. If relative,
                         it is relative to the current working directory. Note
-                        that using this database hook implicitly enables both
-                        read_special and one_file_system (see above) to support
-                        dump and restore streaming.
+                        that using this database hook implicitly enables
+                        read_special (see above) to support dump and restore
+                        streaming.
                     example: /var/lib/sqlite/users.db
                 restore_path:
                     type: string
@@ -1417,6 +1613,24 @@ properties:
                         Path to the SQLite database file to restore to. Defaults
                         to the "path" option.
                     example: /var/lib/sqlite/users.db
+                sqlite_command:
+                    type: string
+                    description: |
+                        Command to use instead of "sqlite3". This can be used to
+                        run a specific sqlite3 version (e.g., one inside a
+                        running container). If you run it from within a
+                        container, make sure to mount the path in the
+                        "user_runtime_directory" option from the host into the
+                        container at the same location. Defaults to "sqlite3".
+                    example: docker exec sqlite_container sqlite3
+                sqlite_restore_command:
+                    type: string
+                    description: |
+                        Command to run when restoring a database instead
+                        of "sqlite3". This can be used to run a specific 
+                        sqlite3 version (e.g., one inside a running container). 
+                        Defaults to "sqlite3".
+                    example: docker exec sqlite_container sqlite3
     mongodb_databases:
         type: array
         items:
@@ -1429,9 +1643,8 @@ properties:
                     description: |
                         Database name (required if using this hook). Or "all" to
                         dump all databases on the host. Note that using this
-                        database hook implicitly enables both read_special and
-                        one_file_system (see above) to support dump and restore
-                        streaming.
+                        database hook implicitly enables read_special (see
+                        above) to support dump and restore streaming.
                     example: users
                 hostname:
                     type: string
@@ -1458,25 +1671,29 @@ properties:
                     type: string
                     description: |
                         Username with which to connect to the database. Skip it
-                        if no authentication is needed.
+                        if no authentication is needed. Supports the
+                        "{credential ...}" syntax.
                     example: dbuser
                 restore_username:
                     type: string
                     description: |
                         Username with which to restore the database. Defaults to
-                        the "username" option.
+                        the "username" option. Supports the "{credential ...}"
+                        syntax.
                     example: dbuser
                 password:
                     type: string
                     description: |
                         Password with which to connect to the database. Skip it
-                        if no authentication is needed.
+                        if no authentication is needed. Supports the
+                        "{credential ...}" syntax.
                     example: trustsome1
                 restore_password:
                     type: string
                     description: |
                         Password with which to connect to the restore database.
-                        Defaults to the "password" option.
+                        Defaults to the "password" option. Supports the
+                        "{credential ...}" syntax.
                     example: trustsome1
                 authentication_database:
                     type: string
@@ -1509,6 +1726,25 @@ properties:
                         dump command, without performing any validation on them.
                         See mongorestore documentation for details.
                     example: --restoreDbUsersAndRoles
+                mongodump_command:
+                    type: string
+                    description: |
+                        Command to use instead of "mongodump". This can be used
+                        to run a specific mongodump version (e.g., one inside a
+                        running container). If you run it from within a
+                        container, make sure to mount the path in the
+                        "user_runtime_directory" option from the host into the
+                        container at the same location.  Defaults to
+                        "mongodump".
+                    example: docker exec mongodb_container mongodump
+                mongorestore_command:
+                    type: string
+                    description: |
+                        Command to run when restoring a database instead of
+                        "mongorestore". This can be used to run a specific
+                        mongorestore version (e.g., one inside a running
+                        container). Defaults to "mongorestore".
+                    example: docker exec mongodb_container mongorestore
         description: |
             List of one or more MongoDB databases to dump before creating a
             backup, run once per configuration file. The database dumps are
@@ -1535,18 +1771,20 @@ properties:
             username:
                 type: string
                 description: |
-                    The username used for authentication.
+                    The username used for authentication. Supports the
+                    "{credential ...}" syntax.
                 example: testuser
             password:
                 type: string
                 description: |
-                    The password used for authentication.
+                    The password used for authentication. Supports the
+                    "{credential ...}" syntax.
                 example: fakepassword
             access_token:
                 type: string
                 description: |
                     An ntfy access token to authenticate with instead of
-                    username/password.
+                    username/password. Supports the "{credential ...}" syntax.
                 example: tk_AgQdq7mVBoFD37zQVN29RhuMzNIz2
             start:
                 type: object
@@ -1641,14 +1879,16 @@ properties:
             token:
                 type: string
                 description: |
-                    Your application's API token.
+                    Your application's API token. Supports the "{credential
+                    ...}" syntax.
                 example: 7ms6TXHpTokTou2P6x4SodDeentHRa
             user:
                 type: string
                 description: |
-                    Your user/group key (or that of your target user), viewable 
-                    when logged into your dashboard: often referred to as 
+                    Your user/group key (or that of your target user), viewable
+                    when logged into your dashboard: often referred to as
                     USER_KEY in Pushover documentation and code examples.
+                    Supports the "{credential ...}" syntax.
                 example: hwRwoWsXMBWwgrSecfa9EfPey55WSN
             start:
                 type: object
@@ -1894,6 +2134,8 @@ properties:
     zabbix:
         type: object
         additionalProperties: false
+        required:
+            - server
         properties:
             itemid:
                 type: integer
@@ -1916,25 +2158,26 @@ properties:
             server:
                 type: string
                 description: |
-                    The address of your Zabbix instance.
+                    The API endpoint URL of your Zabbix instance, usually ending
+                    with "/api_jsonrpc.php". Required.
                 example: https://zabbix.your-domain.com
             username:
                 type: string
                 description: |
                     The username used for authentication. Not needed if using
-                    an API key.
+                    an API key. Supports the "{credential ...}" syntax.
                 example: testuser
             password:
                 type: string
                 description: |
                     The password used for authentication. Not needed if using
-                    an API key.
+                    an API key. Supports the "{credential ...}" syntax.
                 example: fakepassword
             api_key:
                 type: string
                 description: |
-                    The API key used for authentication. Not needed if using
-                    an username/password.
+                    The API key used for authentication. Not needed if using an
+                    username/password. Supports the "{credential ...}" syntax.
                 example: fakekey
             start:
                 type: object
@@ -2187,6 +2430,12 @@ properties:
                     - start
                     - finish
                     - fail
+            verify_tls:
+                type: boolean
+                description: |
+                    Verify the TLS certificate of the push URL host. Defaults to
+                    true.
+                example: false
         description: |
             Configuration for a monitoring integration with Uptime Kuma using
             the Push monitor type.
@@ -2214,9 +2463,15 @@ properties:
             integration_key:
                 type: string
                 description: |
-                    PagerDuty integration key used to notify PagerDuty
-                    when a backup errors.
+                    PagerDuty integration key used to notify PagerDuty when a
+                    backup errors. Supports the "{credential ...}" syntax.
                 example: a177cad45bd374409f78906a810a3074
+            send_logs:
+                type: boolean
+                description: |
+                    Send borgmatic logs to PagerDuty when a backup errors.
+                    Defaults to true.
+                example: false
         description: |
             Configuration for a monitoring integration with PagerDuty. Create an
             account at https://www.pagerduty.com if you'd like to use this
@@ -2266,7 +2521,45 @@ properties:
             can send the logs to a self-hosted instance or create an account at
             https://grafana.com/auth/sign-up/create-user. See borgmatic
             monitoring documentation for details.
-
+    sentry:
+        type: object
+        required: ['data_source_name_url', 'monitor_slug']
+        additionalProperties: false
+        properties:
+            data_source_name_url:
+                type: string
+                description: |
+                    Sentry Data Source Name (DSN) URL, associated with a
+                    particular Sentry project. Used to construct a cron URL,
+                    notified when a backup begins, ends, or errors.
+                example: https://5f80ec@o294220.ingest.us.sentry.io/203069
+            monitor_slug:
+                type: string
+                description: |
+                    Sentry monitor slug, associated with a particular Sentry
+                    project monitor. Used along with the data source name URL to
+                    construct a cron URL.
+                example: mymonitor
+            states:
+                type: array
+                items:
+                    type: string
+                    enum:
+                        - start
+                        - finish
+                        - fail
+                    uniqueItems: true
+                description: |
+                    List of one or more monitoring states to ping for: "start",
+                    "finish", and/or "fail". Defaults to pinging for all states.
+                example:
+                    - start
+                    - finish
+        description: |
+            Configuration for a monitoring integration with Sentry. You can use
+            a self-hosted instance via https://develop.sentry.dev/self-hosted/
+            or create a cloud-hosted account at https://sentry.io. See borgmatic
+            monitoring documentation for details.
     zfs:
         type: ["object", "null"]
         additionalProperties: false
@@ -2288,3 +2581,88 @@ properties:
                 example: /usr/local/bin/umount
         description: |
             Configuration for integration with the ZFS filesystem.
+    btrfs:
+        type: ["object", "null"]
+        additionalProperties: false
+        properties:
+            btrfs_command:
+                type: string
+                description: |
+                    Command to use instead of "btrfs".
+                example: /usr/local/bin/btrfs
+            findmnt_command:
+                type: string
+                description: |
+                    Command to use instead of "findmnt".
+                example: /usr/local/bin/findmnt
+        description: |
+            Configuration for integration with the Btrfs filesystem.
+    lvm:
+        type: ["object", "null"]
+        additionalProperties: false
+        properties:
+            snapshot_size:
+                type: string
+                description: |
+                    Size to allocate for each snapshot taken, including the
+                    units to use for that size. Defaults to "10%ORIGIN" (10%
+                    of the size of logical volume being snapshotted). See the
+                    lvcreate "--size" and "--extents" documentation for more
+                    information:
+                    https://www.man7.org/linux/man-pages/man8/lvcreate.8.html
+                example: 5GB
+            lvcreate_command:
+                type: string
+                description: |
+                    Command to use instead of "lvcreate".
+                example: /usr/local/bin/lvcreate
+            lvremove_command:
+                type: string
+                description: |
+                    Command to use instead of "lvremove".
+                example: /usr/local/bin/lvremove
+            lvs_command:
+                type: string
+                description: |
+                    Command to use instead of "lvs".
+                example: /usr/local/bin/lvs
+            lsblk_command:
+                type: string
+                description: |
+                    Command to use instead of "lsblk".
+                example: /usr/local/bin/lsblk
+            mount_command:
+                type: string
+                description: |
+                    Command to use instead of "mount".
+                example: /usr/local/bin/mount
+            umount_command:
+                type: string
+                description: |
+                    Command to use instead of "umount".
+                example: /usr/local/bin/umount
+        description: |
+            Configuration for integration with Linux LVM (Logical Volume
+            Manager).
+    container:
+        type: object
+        additionalProperties: false
+        properties:
+            secrets_directory:
+                type: string
+                description: |
+                    Secrets directory to use instead of "/run/secrets".
+                example: /path/to/secrets
+        description: |
+            Configuration for integration with Docker or Podman secrets.
+    keepassxc:
+        type: object
+        additionalProperties: false
+        properties:
+            keepassxc_cli_command:
+                type: string
+                description: |
+                    Command to use instead of "keepassxc-cli".
+                example: /usr/local/bin/keepassxc-cli
+        description: |
+            Configuration for integration with the KeePassXC password manager.

borgmatic/config/validate.py

@@ -88,8 +88,9 @@ def parse_configuration(config_filename, schema_filename, overrides=None, resolv
     '''
     Given the path to a config filename in YAML format, the path to a schema filename in a YAML
     rendition of JSON Schema format, a sequence of configuration file override strings in the form
-    of "option.suboption=value", return the parsed configuration as a data structure of nested dicts
-    and lists corresponding to the schema. Example return value:
+    of "option.suboption=value", and whether to resolve environment variables, return the parsed
+    configuration as a data structure of nested dicts and lists corresponding to the schema. Example
+    return value:
 
         {
             'source_directories': ['/home', '/etc'],
@@ -124,6 +125,7 @@ def parse_configuration(config_filename, schema_filename, overrides=None, resolv
         validator = jsonschema.Draft7Validator(schema)
     except AttributeError:  # pragma: no cover
         validator = jsonschema.Draft4Validator(schema)
+
     validation_errors = tuple(validator.iter_errors(config))
 
     if validation_errors:
@@ -136,16 +138,22 @@ def parse_configuration(config_filename, schema_filename, overrides=None, resolv
     return config, config_paths, logs
 
 
-def normalize_repository_path(repository):
+def normalize_repository_path(repository, base=None):
     '''
     Given a repository path, return the absolute path of it (for local repositories).
+    Optionally, use a base path for resolving relative paths, e.g. to the configured working directory.
     '''
     # A colon in the repository could mean that it's either a file:// URL or a remote repository.
     # If it's a remote repository, we don't want to normalize it. If it's a file:// URL, we do.
     if ':' not in repository:
-        return os.path.abspath(repository)
+        return (
+            os.path.abspath(os.path.join(base, repository)) if base else os.path.abspath(repository)
+        )
     elif repository.startswith('file://'):
-        return os.path.abspath(repository.partition('file://')[-1])
+        local_path = repository.partition('file://')[-1]
+        return (
+            os.path.abspath(os.path.join(base, local_path)) if base else os.path.abspath(local_path)
+        )
     else:
         return repository
 

borgmatic/execute.py

@@ -1,11 +1,12 @@
 import collections
 import enum
 import logging
-import os
 import select
 import subprocess
 import textwrap
 
+import borgmatic.logger
+
 logger = logging.getLogger(__name__)
 
 
@@ -241,6 +242,9 @@ def mask_command_secrets(full_command):
 MAX_LOGGED_COMMAND_LENGTH = 1000
 
 
+PREFIXES_OF_ENVIRONMENT_VARIABLES_TO_LOG = ('BORG_', 'PG', 'MARIADB_', 'MYSQL_')
+
+
 def log_command(full_command, input_file=None, output_file=None, environment=None):
     '''
     Log the given command (a sequence of command/argument strings), along with its input/output file
@@ -249,14 +253,21 @@ def log_command(full_command, input_file=None, output_file=None, environment=Non
     logger.debug(
         textwrap.shorten(
             ' '.join(