Fix line wrapping / code style (#837).

Reviewed-on: #1041

NEWS

@@ -1,3 +1,70 @@
+2.0.0.dev0
+ * #345: Add a "key import" action to import a repository key from backup.
+ * #422: Add home directory expansion to file-based and KeePassXC credential hooks.
+ * #790, #821: Deprecate all "before_*", "after_*" and "on_error" command hooks in favor of more
+   flexible "commands:". See the documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/
+ * #790: BREAKING: For both new and deprecated command hooks, run a configured "after" hook even if
+   an error occurs first. This allows you to perform cleanup steps that correspond to "before"
+   preparation commands—even when something goes wrong.
+ * #790: BREAKING: Run all command hooks (both new and deprecated) respecting the
+   "working_directory" option if configured, meaning that hook commands are run in that directory.
+ * #836: Add a custom command option for the SQLite hook.
+ * #837: Add custom command options for the MongoDB hook.
+ * #1010: When using Borg 2, don't pass the "--stats" flag to "borg prune".
+ * #1020: Document a database use case involving a temporary database client container:
+   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#containers
+ * #1037: Fix an error with the "extract" action when both a remote repository and a
+   "working_directory" are used.
+
+1.9.14
+ * #409: With the PagerDuty monitoring hook, send borgmatic logs to PagerDuty so they show up in the
+   incident UI. See the documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#pagerduty-hook
+ * #936: Clarify Zabbix monitoring hook documentation about creating items:
+   https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#zabbix-hook
+ * #1017: Fix a regression in which some MariaDB/MySQL passwords were not escaped correctly.
+ * #1021: Fix a regression in which the "exclude_patterns" option didn't expand "~" (the user's
+   home directory). This fix means that all "patterns" and "patterns_from" also now expand "~".
+ * #1023: Fix an error in the Btrfs hook when attempting to snapshot a read-only subvolume. Now,
+   read-only subvolumes are ignored since Btrfs can't actually snapshot them.
+
+1.9.13
+ * #975: Add a "compression" option to the PostgreSQL database hook.
+ * #1001: Fix a ZFS error during snapshot cleanup.
+ * #1003: In the Zabbix monitoring hook, support Zabbix 7.2's authentication changes.
+ * #1009: Send database passwords to MariaDB and MySQL via anonymous pipe, which is more secure than
+   using an environment variable.
+ * #1013: Send database passwords to MongoDB via anonymous pipe, which is more secure than using
+   "--password" on the command-line!
+ * #1015: When ctrl-C is pressed, more strongly encourage Borg to actually exit.
+ * Add a "verify_tls" option to the Uptime Kuma monitoring hook for disabling TLS verification.
+ * Add "tls" options to the MariaDB and MySQL database hooks to enable or disable TLS encryption
+   between client and server.
+
+1.9.12
+ * #1005: Fix the credential hooks to avoid using Python 3.12+ string features. Now borgmatic will
+   work with Python 3.9, 3.10, and 3.11 again.
+
+1.9.11
+ * #795: Add credential loading from file, KeePassXC, and Docker/Podman secrets. See the
+   documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/provide-your-passwords/
+ * #996: Fix the "create" action to omit the repository label prefix from Borg's output when
+   databases are enabled.
+ * #998: Send the "encryption_passphrase" option to Borg via an anonymous pipe, which is more secure
+   than using an environment variable.
+ * #999: Fix a runtime directory error from a conflict between "extra_borg_options" and special file
+   detection.
+ * #1001: For the ZFS, Btrfs, and LVM hooks, only make snapshots for root patterns that come from
+   a borgmatic configuration option (e.g. "source_directories")—not from other hooks within
+   borgmatic.
+ * #1001: Fix a ZFS/LVM error due to colliding snapshot mount points for nested datasets or logical
+   volumes.
+ * #1001: Don't try to snapshot ZFS datasets that have the "canmount=off" property.
+ * Fix another error in the Btrfs hook when a subvolume mounted at "/" is configured in borgmatic's
+   source directories.
+
 1.9.10
  * #966: Add a "{credential ...}" syntax for loading systemd credentials into borgmatic
    configuration files. See the documentation for more information:

README.md

@@ -88,6 +88,9 @@ borgmatic is powered by [Borg Backup](https://www.borgbackup.org/).
 ### Credentials
 
 <a href="https://systemd.io/"><img src="docs/static/systemd.png" alt="Sentry" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.docker.com/"><img src="docs/static/docker.png" alt="Docker" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://podman.io/"><img src="docs/static/podman.png" alt="Podman" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://keepassxc.org/"><img src="docs/static/keepassxc.png" alt="Podman" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
 
 
 ## Getting started

borgmatic/actions/change_passphrase.py

@@ -16,7 +16,7 @@ def run_change_passphrase(
     remote_path,
 ):
     '''
-    Run the "key change-passprhase" action for the given repository.
+    Run the "key change-passphrase" action for the given repository.
     '''
     if (
         change_passphrase_arguments.repository is None

borgmatic/actions/check.py

@@ -391,7 +391,7 @@ def collect_spot_check_source_paths(
     paths_output = borgmatic.execute.execute_command_and_capture_output(
         create_flags + create_positional_arguments,
         capture_stderr=True,
-        extra_environment=borgmatic.borg.environment.make_environment(config),
+        environment=borgmatic.borg.environment.make_environment(config),
         working_directory=working_directory,
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),
@@ -682,7 +682,6 @@ def run_check(
     config_filename,
     repository,
     config,
-    hook_context,
     local_borg_version,
     check_arguments,
     global_arguments,
@@ -699,15 +698,6 @@ def run_check(
     ):
         return
 
-    borgmatic.hooks.command.execute_hook(
-        config.get('before_check'),
-        config.get('umask'),
-        config_filename,
-        'pre-check',
-        global_arguments.dry_run,
-        **hook_context,
-    )
-
     logger.info('Running consistency checks')
 
     repository_id = borgmatic.borg.check.get_repository_id(
@@ -772,12 +762,3 @@ def run_check(
                 borgmatic_runtime_directory,
             )
         write_check_time(make_check_time_path(config, repository_id, 'spot'))
-
-    borgmatic.hooks.command.execute_hook(
-        config.get('after_check'),
-        config.get('umask'),
-        config_filename,
-        'post-check',
-        global_arguments.dry_run,
-        **hook_context,
-    )

borgmatic/actions/compact.py

@@ -12,7 +12,6 @@ def run_compact(
     config_filename,
     repository,
     config,
-    hook_context,
     local_borg_version,
     compact_arguments,
     global_arguments,
@@ -28,14 +27,6 @@ def run_compact(
     ):
         return
 
-    borgmatic.hooks.command.execute_hook(
-        config.get('before_compact'),
-        config.get('umask'),
-        config_filename,
-        'pre-compact',
-        global_arguments.dry_run,
-        **hook_context,
-    )
     if borgmatic.borg.feature.available(borgmatic.borg.feature.Feature.COMPACT, local_borg_version):
         logger.info(f'Compacting segments{dry_run_label}')
         borgmatic.borg.compact.compact_segments(
@@ -52,12 +43,3 @@ def run_compact(
         )
     else:  # pragma: nocover
         logger.info('Skipping compact (only available/needed in Borg 1.2+)')
-
-    borgmatic.hooks.command.execute_hook(
-        config.get('after_compact'),
-        config.get('umask'),
-        config_filename,
-        'post-compact',
-        global_arguments.dry_run,
-        **hook_context,
-    )

borgmatic/actions/create.py

@@ -36,6 +36,7 @@ def parse_pattern(pattern_line, default_style=borgmatic.borg.pattern.Pattern_sty
         path,
         borgmatic.borg.pattern.Pattern_type(pattern_type),
         borgmatic.borg.pattern.Pattern_style(pattern_style),
+        source=borgmatic.borg.pattern.Pattern_source.CONFIG,
     )
 
 
@@ -51,7 +52,9 @@ def collect_patterns(config):
     try:
         return (
             tuple(
-                borgmatic.borg.pattern.Pattern(source_directory)
+                borgmatic.borg.pattern.Pattern(
+                    source_directory, source=borgmatic.borg.pattern.Pattern_source.CONFIG
+                )
                 for source_directory in config.get('source_directories', ())
             )
             + tuple(
@@ -127,8 +130,11 @@ def expand_directory(directory, working_directory):
 def expand_patterns(patterns, working_directory=None, skip_paths=None):
     '''
     Given a sequence of borgmatic.borg.pattern.Pattern instances and an optional working directory,
-    expand tildes and globs in each root pattern. Return all the resulting patterns (not just the
-    root patterns) as a tuple.
+    expand tildes and globs in each root pattern and expand just tildes in each non-root pattern.
+    The idea is that non-root patterns may be regular expressions or other pattern styles containing
+    "*" that borgmatic should not expand as a shell glob.
+
+    Return all the resulting patterns as a tuple.
 
     If a set of paths are given to skip, then don't expand any patterns matching them.
     '''
@@ -144,12 +150,21 @@ def expand_patterns(patterns, working_directory=None, skip_paths=None):
                         pattern.type,
                         pattern.style,
                         pattern.device,
+                        pattern.source,
                     )
                     for expanded_path in expand_directory(pattern.path, working_directory)
                 )
                 if pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT
                 and pattern.path not in (skip_paths or ())
-                else (pattern,)
+                else (
+                    borgmatic.borg.pattern.Pattern(
+                        os.path.expanduser(pattern.path),
+                        pattern.type,
+                        pattern.style,
+                        pattern.device,
+                        pattern.source,
+                    ),
+                )
             )
             for pattern in patterns
         )
@@ -178,6 +193,7 @@ def device_map_patterns(patterns, working_directory=None):
                 and os.path.exists(full_path)
                 else None
             ),
+            source=pattern.source,
         )
         for pattern in patterns
         for full_path in (os.path.join(working_directory or '', pattern.path),)
@@ -256,7 +272,6 @@ def run_create(
     repository,
     config,
     config_paths,
-    hook_context,
     local_borg_version,
     create_arguments,
     global_arguments,
@@ -274,15 +289,6 @@ def run_create(
     ):
         return
 
-    borgmatic.hooks.command.execute_hook(
-        config.get('before_backup'),
-        config.get('umask'),
-        config_filename,
-        'pre-backup',
-        global_arguments.dry_run,
-        **hook_context,
-    )
-
     logger.info(f'Creating archive{dry_run_label}')
     working_directory = borgmatic.config.paths.get_working_directory(config)
 
@@ -338,12 +344,3 @@ def run_create(
             borgmatic_runtime_directory,
             global_arguments.dry_run,
         )
-
-    borgmatic.hooks.command.execute_hook(
-        config.get('after_backup'),
-        config.get('umask'),
-        config_filename,
-        'post-backup',
-        global_arguments.dry_run,
-        **hook_context,
-    )

borgmatic/actions/extract.py

@@ -12,7 +12,6 @@ def run_extract(
     config_filename,
     repository,
     config,
-    hook_context,
     local_borg_version,
     extract_arguments,
     global_arguments,
@@ -22,14 +21,6 @@ def run_extract(
     '''
     Run the "extract" action for the given repository.
     '''
-    borgmatic.hooks.command.execute_hook(
-        config.get('before_extract'),
-        config.get('umask'),
-        config_filename,
-        'pre-extract',
-        global_arguments.dry_run,
-        **hook_context,
-    )
     if extract_arguments.repository is None or borgmatic.config.validate.repositories_match(
         repository, extract_arguments.repository
     ):
@@ -56,11 +47,3 @@ def run_extract(
             strip_components=extract_arguments.strip_components,
             progress=extract_arguments.progress,
         )
-    borgmatic.hooks.command.execute_hook(
-        config.get('after_extract'),
-        config.get('umask'),
-        config_filename,
-        'post-extract',
-        global_arguments.dry_run,
-        **hook_context,
-    )

borgmatic/actions/import_key.py

@@ -0,0 +1,33 @@
+import logging
+
+import borgmatic.borg.import_key
+import borgmatic.config.validate
+
+logger = logging.getLogger(__name__)
+
+
+def run_import_key(
+    repository,
+    config,
+    local_borg_version,
+    import_arguments,
+    global_arguments,
+    local_path,
+    remote_path,
+):
+    '''
+    Run the "key import" action for the given repository.
+    '''
+    if import_arguments.repository is None or borgmatic.config.validate.repositories_match(
+        repository, import_arguments.repository
+    ):
+        logger.info('Importing repository key')
+        borgmatic.borg.import_key.import_key(
+            repository['path'],
+            config,
+            local_borg_version,
+            import_arguments,
+            global_arguments,
+            local_path=local_path,
+            remote_path=remote_path,
+        )

borgmatic/actions/prune.py

@@ -11,7 +11,6 @@ def run_prune(
     config_filename,
     repository,
     config,
-    hook_context,
     local_borg_version,
     prune_arguments,
     global_arguments,
@@ -27,14 +26,6 @@ def run_prune(
     ):
         return
 
-    borgmatic.hooks.command.execute_hook(
-        config.get('before_prune'),
-        config.get('umask'),
-        config_filename,
-        'pre-prune',
-        global_arguments.dry_run,
-        **hook_context,
-    )
     logger.info(f'Pruning archives{dry_run_label}')
     borgmatic.borg.prune.prune_archives(
         global_arguments.dry_run,
@@ -46,11 +37,3 @@ def run_prune(
         local_path=local_path,
         remote_path=remote_path,
     )
-    borgmatic.hooks.command.execute_hook(
-        config.get('after_prune'),
-        config.get('umask'),
-        config_filename,
-        'post-prune',
-        global_arguments.dry_run,
-        **hook_context,
-    )

borgmatic/borg/borg.py

@@ -61,7 +61,7 @@ def run_arbitrary_borg(
         tuple(shlex.quote(part) for part in full_command),
         output_file=DO_NOT_CAPTURE,
         shell=True,
-        extra_environment=dict(
+        environment=dict(
             (environment.make_environment(config) or {}),
             **{
                 'BORG_REPO': repository_path,

borgmatic/borg/break_lock.py

@@ -36,7 +36,7 @@ def break_lock(
 
     execute_command(
         full_command,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/change_passphrase.py

@@ -56,7 +56,7 @@ def change_passphrase(
         full_command,
         output_file=borgmatic.execute.DO_NOT_CAPTURE,
         output_log_level=logging.ANSWER,
-        extra_environment=environment.make_environment(config_without_passphrase),
+        environment=environment.make_environment(config_without_passphrase),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/check.py

@@ -182,7 +182,7 @@ def check_archives(
             output_file=(
                 DO_NOT_CAPTURE if check_arguments.repair or check_arguments.progress else None
             ),
-            extra_environment=environment.make_environment(config),
+            environment=environment.make_environment(config),
             working_directory=working_directory,
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,

borgmatic/borg/compact.py

@@ -49,7 +49,7 @@ def compact_segments(
     execute_command(
         full_command,
         output_log_level=logging.INFO,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/create.py

@@ -132,41 +132,53 @@ def collect_special_file_paths(
     used.
 
     Skip looking for special files in the given borgmatic runtime directory, as borgmatic creates
-    its own special files there for database dumps. And if the borgmatic runtime directory is
-    configured to be excluded from the files Borg backs up, error, because this means Borg won't be
-    able to consume any database dumps and therefore borgmatic will hang.
+    its own special files there for database dumps and we don't want those omitted.
+
+    Additionally, if the borgmatic runtime directory is not contained somewhere in the files Borg
+    plans to backup, that means the user must have excluded the runtime directory (e.g. via
+    "exclude_patterns" or similar). Therefore, raise, because this means Borg won't be able to
+    consume any database dumps and therefore borgmatic will hang when it tries to do so.
     '''
     # Omit "--exclude-nodump" from the Borg dry run command, because that flag causes Borg to open
-    # files including any named pipe we've created.
+    # files including any named pipe we've created. And omit "--filter" because that can break the
+    # paths output parsing below such that path lines no longer start with th expected "- ".
     paths_output = execute_command_and_capture_output(
-        tuple(argument for argument in create_command if argument != '--exclude-nodump')
+        flags.omit_flag_and_value(flags.omit_flag(create_command, '--exclude-nodump'), '--filter')
         + ('--dry-run', '--list'),
         capture_stderr=True,
         working_directory=working_directory,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),
     )
 
+    # These are all the individual files that Borg is planning to backup as determined by the Borg
+    # create dry run above.
     paths = tuple(
         path_line.split(' ', 1)[1]
         for path_line in paths_output.split('\n')
         if path_line and path_line.startswith('- ') or path_line.startswith('+ ')
     )
-    skip_paths = {}
+
+    # These are the subset of those files that contain the borgmatic runtime directory.
+    paths_containing_runtime_directory = {}
 
     if os.path.exists(borgmatic_runtime_directory):
-        skip_paths = {
+        paths_containing_runtime_directory = {
             path for path in paths if any_parent_directories(path, (borgmatic_runtime_directory,))
         }
 
-        if not skip_paths and not dry_run:
+        # If no paths to backup contain the runtime directory, it must've been excluded.
+        if not paths_containing_runtime_directory and not dry_run:
             raise ValueError(
                 f'The runtime directory {os.path.normpath(borgmatic_runtime_directory)} overlaps with the configured excludes or patterns with excludes. Please ensure the runtime directory is not excluded.'
             )
 
     return tuple(
-        path for path in paths if special_file(path, working_directory) if path not in skip_paths
+        path
+        for path in paths
+        if special_file(path, working_directory)
+        if path not in paths_containing_runtime_directory
     )
 
 
@@ -325,6 +337,7 @@ def make_base_create_command(
                         special_file_path,
                         borgmatic.borg.pattern.Pattern_type.NO_RECURSE,
                         borgmatic.borg.pattern.Pattern_style.FNMATCH,
+                        source=borgmatic.borg.pattern.Pattern_source.INTERNAL,
                     )
                     for special_file_path in special_file_paths
                 ),
@@ -409,7 +422,7 @@ def create_archive(
             output_log_level,
             output_file,
             working_directory=working_directory,
-            extra_environment=environment.make_environment(config),
+            environment=environment.make_environment(config),
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,
         )
@@ -417,7 +430,7 @@ def create_archive(
         return execute_command_and_capture_output(
             create_flags + create_positional_arguments,
             working_directory=working_directory,
-            extra_environment=environment.make_environment(config),
+            environment=environment.make_environment(config),
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,
         )
@@ -427,7 +440,7 @@ def create_archive(
             output_log_level,
             output_file,
             working_directory=working_directory,
-            extra_environment=environment.make_environment(config),
+            environment=environment.make_environment(config),
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,
         )

borgmatic/borg/delete.py

@@ -128,7 +128,7 @@ def delete_archives(
     borgmatic.execute.execute_command(
         command,
         output_log_level=logging.ANSWER,
-        extra_environment=borgmatic.borg.environment.make_environment(config),
+        environment=borgmatic.borg.environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/environment.py

@@ -10,13 +10,10 @@ OPTION_TO_ENVIRONMENT_VARIABLE = {
     'borg_files_cache_ttl': 'BORG_FILES_CACHE_TTL',
     'borg_security_directory': 'BORG_SECURITY_DIR',
     'borg_keys_directory': 'BORG_KEYS_DIR',
-    'encryption_passphrase': 'BORG_PASSPHRASE',
     'ssh_command': 'BORG_RSH',
     'temporary_directory': 'TMPDIR',
 }
 
-CREDENTIAL_OPTIONS = {'encryption_passphrase'}
-
 DEFAULT_BOOL_OPTION_TO_DOWNCASE_ENVIRONMENT_VARIABLE = {
     'relocated_repo_access_is_ok': 'BORG_RELOCATED_REPO_ACCESS_IS_OK',
     'unknown_unencrypted_repo_access_is_ok': 'BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK',
@@ -29,32 +26,55 @@ DEFAULT_BOOL_OPTION_TO_UPPERCASE_ENVIRONMENT_VARIABLE = {
 
 def make_environment(config):
     '''
-    Given a borgmatic configuration dict, return its options converted to a Borg environment
-    variable dict.
+    Given a borgmatic configuration dict, convert it to a Borg environment variable dict, merge it
+    with a copy of the current environment variables, and return the result.
 
     Do not reuse this environment across multiple Borg invocations, because it can include
     references to resources like anonymous pipes for passphrases—which can only be consumed once.
+
+    Here's how native Borg precedence works for a few of the environment variables:
+
+      1. BORG_PASSPHRASE, if set, is used first.
+      2. BORG_PASSCOMMAND is used only if BORG_PASSPHRASE isn't set.
+      3. BORG_PASSPHRASE_FD is used only if neither of the above are set.
+
+    In borgmatic, we want to simulate this precedence order, but there are some additional
+    complications. First, values can come from either configuration or from environment variables
+    set outside borgmatic; configured options should take precedence. Second, when borgmatic gets a
+    passphrase—directly from configuration or indirectly via a credential hook or a passcommand—we
+    want to pass that passphrase to Borg via an anonymous pipe (+ BORG_PASSPHRASE_FD), since that's
+    more secure than using an environment variable (BORG_PASSPHRASE).
     '''
-    environment = {}
+    environment = dict(os.environ)
 
     for option_name, environment_variable_name in OPTION_TO_ENVIRONMENT_VARIABLE.items():
         value = config.get(option_name)
 
-        if option_name in CREDENTIAL_OPTIONS and value is not None:
-            value = borgmatic.hooks.credential.parse.resolve_credential(value)
-
         if value is not None:
             environment[environment_variable_name] = str(value)
 
-    passphrase = borgmatic.borg.passcommand.get_passphrase_from_passcommand(config)
+    if 'encryption_passphrase' in config:
+        environment.pop('BORG_PASSPHRASE', None)
+        environment.pop('BORG_PASSCOMMAND', None)
 
-    # If the passcommand produced a passphrase, send it to Borg via an anonymous pipe.
-    if passphrase:
+    if 'encryption_passcommand' in config:
+        environment.pop('BORG_PASSCOMMAND', None)
+
+    passphrase = borgmatic.hooks.credential.parse.resolve_credential(
+        config.get('encryption_passphrase'), config
+    )
+
+    if passphrase is None:
+        passphrase = borgmatic.borg.passcommand.get_passphrase_from_passcommand(config)
+
+    # If there's a passphrase (from configuration, from a configured credential, or from a
+    # configured passcommand), send it to Borg via an anonymous pipe.
+    if passphrase is not None:
         read_file_descriptor, write_file_descriptor = os.pipe()
         os.write(write_file_descriptor, passphrase.encode('utf-8'))
         os.close(write_file_descriptor)
 
-        # This, plus subprocess.Popen(..., close_fds=False) in execute.py, is necessary for the Borg
+        # This plus subprocess.Popen(..., close_fds=False) in execute.py is necessary for the Borg
         # child process to inherit the file descriptor.
         os.set_inheritable(read_file_descriptor, True)
         environment['BORG_PASSPHRASE_FD'] = str(read_file_descriptor)

borgmatic/borg/export_key.py

@@ -67,7 +67,7 @@ def export_key(
         full_command,
         output_file=output_file,
         output_log_level=logging.ANSWER,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=working_directory,
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/export_tar.py

@@ -70,7 +70,7 @@ def export_tar_archive(
         full_command,
         output_file=DO_NOT_CAPTURE if destination_path == '-' else None,
         output_log_level=output_log_level,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/extract.py

@@ -58,7 +58,7 @@ def extract_last_archive_dry_run(
 
     execute_command(
         full_extract_command,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),
@@ -134,9 +134,7 @@ def extract_archive(
             # Make the repository path absolute so the destination directory used below via changing
             # the working directory doesn't prevent Borg from finding the repo. But also apply the
             # user's configured working directory (if any) to the repo path.
-            borgmatic.config.validate.normalize_repository_path(
-                os.path.join(working_directory or '', repository)
-            ),
+            borgmatic.config.validate.normalize_repository_path(repository, working_directory),
             archive,
             local_borg_version,
         )
@@ -154,7 +152,7 @@ def extract_archive(
         return execute_command(
             full_command,
             output_file=DO_NOT_CAPTURE,
-            extra_environment=environment.make_environment(config),
+            environment=environment.make_environment(config),
             working_directory=full_destination_path,
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,
@@ -166,7 +164,7 @@ def extract_archive(
             full_command,
             output_file=subprocess.PIPE,
             run_to_completion=False,
-            extra_environment=environment.make_environment(config),
+            environment=environment.make_environment(config),
             working_directory=full_destination_path,
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,
@@ -176,7 +174,7 @@ def extract_archive(
     # if the restore paths don't exist in the archive.
     execute_command(
         full_command,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=full_destination_path,
         borg_local_path=local_path,
         borg_exit_codes=borg_exit_codes,

borgmatic/borg/feature.py

@@ -17,6 +17,7 @@ class Feature(Enum):
     MATCH_ARCHIVES = 11
     EXCLUDED_FILES_MINUS = 12
     ARCHIVE_SERIES = 13
+    NO_PRUNE_STATS = 14
 
 
 FEATURE_TO_MINIMUM_BORG_VERSION = {
@@ -33,6 +34,7 @@ FEATURE_TO_MINIMUM_BORG_VERSION = {
     Feature.MATCH_ARCHIVES: parse('2.0.0b3'),  # borg --match-archives
     Feature.EXCLUDED_FILES_MINUS: parse('2.0.0b5'),  # --list --filter uses "-" for excludes
     Feature.ARCHIVE_SERIES: parse('2.0.0b11'),  # identically named archives form a series
+    Feature.NO_PRUNE_STATS: parse('2.0.0b10'),  # prune --stats is not available
 }
 
 

borgmatic/borg/flags.py

@@ -156,3 +156,44 @@ def warn_for_aggressive_archive_flags(json_command, json_output):
         logger.debug(f'Cannot parse JSON output from archive command: {error}')
     except (TypeError, KeyError):
         logger.debug('Cannot parse JSON output from archive command: No "archives" key found')
+
+
+def omit_flag(arguments, flag):
+    '''
+    Given a sequence of Borg command-line arguments, return them with the given (valueless) flag
+    omitted. For instance, if the flag is "--flag" and arguments is:
+
+        ('borg', 'create', '--flag', '--other-flag')
+
+    ... then return:
+
+        ('borg', 'create', '--other-flag')
+    '''
+    return tuple(argument for argument in arguments if argument != flag)
+
+
+def omit_flag_and_value(arguments, flag):
+    '''
+    Given a sequence of Borg command-line arguments, return them with the given flag and its
+    corresponding value omitted. For instance, if the flag is "--flag" and arguments is:
+
+        ('borg', 'create', '--flag', 'value', '--other-flag')
+
+    ... or:
+
+        ('borg', 'create', '--flag=value', '--other-flag')
+
+    ... then return:
+
+        ('borg', 'create', '--other-flag')
+    '''
+    # This works by zipping together a list of overlapping pairwise arguments. E.g., ('one', 'two',
+    # 'three', 'four') becomes ((None, 'one'), ('one, 'two'), ('two', 'three'), ('three', 'four')).
+    # This makes it easy to "look back" at the previous arguments so we can exclude both a flag and
+    # its value.
+    return tuple(
+        argument
+        for (previous_argument, argument) in zip((None,) + arguments, arguments)
+        if flag not in (previous_argument, argument)
+        if not argument.startswith(f'{flag}=')
+    )

borgmatic/borg/import_key.py

@@ -0,0 +1,70 @@
+import logging
+import os
+
+import borgmatic.config.paths
+import borgmatic.logger
+from borgmatic.borg import environment, flags
+from borgmatic.execute import DO_NOT_CAPTURE, execute_command
+
+logger = logging.getLogger(__name__)
+
+
+def import_key(
+    repository_path,
+    config,
+    local_borg_version,
+    import_arguments,
+    global_arguments,
+    local_path='borg',
+    remote_path=None,
+):
+    '''
+    Given a local or remote repository path, a configuration dict, the local Borg version, import
+    arguments, and optional local and remote Borg paths, import the repository key from the
+    path indicated in the import arguments.
+
+    If the path is empty or "-", then read the key from stdin.
+
+    Raise ValueError if the path is given and it does not exist.
+    '''
+    umask = config.get('umask', None)
+    lock_wait = config.get('lock_wait', None)
+    working_directory = borgmatic.config.paths.get_working_directory(config)
+
+    if import_arguments.path and import_arguments.path != '-':
+        if not os.path.exists(os.path.join(working_directory or '', import_arguments.path)):
+            raise ValueError(f'Path {import_arguments.path} does not exist. Aborting.')
+
+        input_file = None
+    else:
+        input_file = DO_NOT_CAPTURE
+
+    full_command = (
+        (local_path, 'key', 'import')
+        + (('--remote-path', remote_path) if remote_path else ())
+        + (('--umask', str(umask)) if umask else ())
+        + (('--log-json',) if global_arguments.log_json else ())
+        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
+        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+        + flags.make_flags('paper', import_arguments.paper)
+        + flags.make_repository_flags(
+            repository_path,
+            local_borg_version,
+        )
+        + ((import_arguments.path,) if input_file is None else ())
+    )
+
+    if global_arguments.dry_run:
+        logger.info('Skipping key import (dry run)')
+        return
+
+    execute_command(
+        full_command,
+        input_file=input_file,
+        output_log_level=logging.INFO,
+        environment=environment.make_environment(config),
+        working_directory=working_directory,
+        borg_local_path=local_path,
+        borg_exit_codes=config.get('borg_exit_codes'),
+    )

borgmatic/borg/info.py

@@ -102,7 +102,7 @@ def display_archives_info(
 
     json_info = execute_command_and_capture_output(
         json_command,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=working_directory,
         borg_local_path=local_path,
         borg_exit_codes=borg_exit_codes,
@@ -116,7 +116,7 @@ def display_archives_info(
     execute_command(
         main_command,
         output_log_level=logging.ANSWER,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=working_directory,
         borg_local_path=local_path,
         borg_exit_codes=borg_exit_codes,

borgmatic/borg/list.py

@@ -124,7 +124,7 @@ def capture_archive_listing(
                 local_path,
                 remote_path,
             ),
-            extra_environment=environment.make_environment(config),
+            environment=environment.make_environment(config),
             working_directory=borgmatic.config.paths.get_working_directory(config),
             borg_local_path=local_path,
             borg_exit_codes=config.get('borg_exit_codes'),
@@ -221,7 +221,7 @@ def list_archive(
                     local_path,
                     remote_path,
                 ),
-                extra_environment=environment.make_environment(config),
+                environment=environment.make_environment(config),
                 working_directory=borgmatic.config.paths.get_working_directory(config),
                 borg_local_path=local_path,
                 borg_exit_codes=borg_exit_codes,
@@ -257,7 +257,7 @@ def list_archive(
         execute_command(
             main_command,
             output_log_level=logging.ANSWER,
-            extra_environment=environment.make_environment(config),
+            environment=environment.make_environment(config),
             working_directory=borgmatic.config.paths.get_working_directory(config),
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,

borgmatic/borg/mount.py

@@ -66,7 +66,7 @@ def mount_archive(
         execute_command(
             full_command,
             output_file=DO_NOT_CAPTURE,
-            extra_environment=environment.make_environment(config),
+            environment=environment.make_environment(config),
             working_directory=working_directory,
             borg_local_path=local_path,
             borg_exit_codes=config.get('borg_exit_codes'),
@@ -75,7 +75,7 @@ def mount_archive(
 
     execute_command(
         full_command,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=working_directory,
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/passcommand.py

@@ -9,21 +9,14 @@ logger = logging.getLogger(__name__)
 
 
 @functools.cache
-def run_passcommand(passcommand, passphrase_configured, working_directory):
+def run_passcommand(passcommand, working_directory):
     '''
     Run the given passcommand using the given working directory and return the passphrase produced
-    by the command. But bail first if a passphrase is already configured; this mimics Borg's
-    behavior.
+    by the command.
 
     Cache the results so that the passcommand only needs to run—and potentially prompt the user—once
     per borgmatic invocation.
     '''
-    if passcommand and passphrase_configured:
-        logger.warning(
-            'Ignoring the "encryption_passcommand" option because "encryption_passphrase" is set'
-        )
-        return None
-
     return borgmatic.execute.execute_command_and_capture_output(
         shlex.split(passcommand),
         working_directory=working_directory,
@@ -44,7 +37,4 @@ def get_passphrase_from_passcommand(config):
     if not passcommand:
         return None
 
-    passphrase = config.get('encryption_passphrase')
-    working_directory = borgmatic.config.paths.get_working_directory(config)
-
-    return run_passcommand(passcommand, bool(passphrase is not None), working_directory)
+    return run_passcommand(passcommand, borgmatic.config.paths.get_working_directory(config))

borgmatic/borg/pattern.py

@@ -20,12 +20,31 @@ class Pattern_style(enum.Enum):
     PATH_FULL_MATCH = 'pf'
 
 
+class Pattern_source(enum.Enum):
+    '''
+    Where the pattern came from within borgmatic. This is important because certain use cases (like
+    filesystem snapshotting) only want to consider patterns that the user actually put in a
+    configuration file and not patterns from other sources.
+    '''
+
+    # The pattern is from a borgmatic configuration option, e.g. listed in "source_directories".
+    CONFIG = 'config'
+
+    # The pattern is generated internally within borgmatic, e.g. for special file excludes.
+    INTERNAL = 'internal'
+
+    # The pattern originates from within a borgmatic hook, e.g. a database hook that adds its dump
+    # directory.
+    HOOK = 'hook'
+
+
 Pattern = collections.namedtuple(
     'Pattern',
-    ('path', 'type', 'style', 'device'),
+    ('path', 'type', 'style', 'device', 'source'),
     defaults=(
         Pattern_type.ROOT,
         Pattern_style.NONE,
         None,
+        Pattern_source.HOOK,
     ),
 )

borgmatic/borg/prune.py

@@ -75,7 +75,13 @@ def prune_archives(
         + (('--umask', str(umask)) if umask else ())
         + (('--log-json',) if global_arguments.log_json else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
-        + (('--stats',) if prune_arguments.stats and not dry_run else ())
+        + (
+            ('--stats',)
+            if prune_arguments.stats
+            and not dry_run
+            and not feature.available(feature.Feature.NO_PRUNE_STATS, local_borg_version)
+            else ()
+        )
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + flags.make_flags_from_arguments(
             prune_arguments,
@@ -96,7 +102,7 @@ def prune_archives(
     execute_command(
         full_command,
         output_log_level=output_log_level,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/repo_create.py

@@ -98,7 +98,7 @@ def create_repository(
     execute_command(
         repo_create_command,
         output_file=DO_NOT_CAPTURE,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/repo_delete.py

@@ -88,7 +88,7 @@ def delete_repository(
             if repo_delete_arguments.force or repo_delete_arguments.cache_only
             else borgmatic.execute.DO_NOT_CAPTURE
         ),
-        extra_environment=borgmatic.borg.environment.make_environment(config),
+        environment=borgmatic.borg.environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/repo_info.py

@@ -56,7 +56,7 @@ def display_repository_info(
     if repo_info_arguments.json:
         return execute_command_and_capture_output(
             full_command,
-            extra_environment=environment.make_environment(config),
+            environment=environment.make_environment(config),
             working_directory=working_directory,
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,
@@ -65,7 +65,7 @@ def display_repository_info(
         execute_command(
             full_command,
             output_log_level=logging.ANSWER,
-            extra_environment=environment.make_environment(config),
+            environment=environment.make_environment(config),
             working_directory=working_directory,
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,

borgmatic/borg/repo_list.py

@@ -49,7 +49,7 @@ def resolve_archive_name(
 
     output = execute_command_and_capture_output(
         full_command,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),
@@ -164,7 +164,7 @@ def list_repository(
 
     json_listing = execute_command_and_capture_output(
         json_command,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=working_directory,
         borg_local_path=local_path,
         borg_exit_codes=borg_exit_codes,
@@ -178,7 +178,7 @@ def list_repository(
     execute_command(
         main_command,
         output_log_level=logging.ANSWER,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=working_directory,
         borg_local_path=local_path,
         borg_exit_codes=borg_exit_codes,

borgmatic/borg/transfer.py

@@ -57,7 +57,7 @@ def transfer_archives(
         full_command,
         output_log_level=logging.ANSWER,
         output_file=DO_NOT_CAPTURE if transfer_arguments.progress else None,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/version.py

@@ -21,7 +21,7 @@ def local_borg_version(config, local_path='borg'):
     )
     output = execute_command_and_capture_output(
         full_command,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/commands/arguments.py

@@ -547,7 +547,7 @@ def make_parsers():
         dest='stats',
         default=False,
         action='store_true',
-        help='Display statistics of the pruned archive',
+        help='Display statistics of the pruned archive [Borg 1 only]',
     )
     prune_group.add_argument(
         '--list', dest='list_archives', action='store_true', help='List archives kept/pruned'
@@ -1479,6 +1479,31 @@ def make_parsers():
         '-h', '--help', action='help', help='Show this help message and exit'
     )
 
+    key_import_parser = key_parsers.add_parser(
+        'import',
+        help='Import a copy of the repository key from backup',
+        description='Import a copy of the repository key from backup',
+        add_help=False,
+    )
+    key_import_group = key_import_parser.add_argument_group('key import arguments')
+    key_import_group.add_argument(
+        '--paper',
+        action='store_true',
+        help='Import interactively from a backup done with --paper',
+    )
+    key_import_group.add_argument(
+        '--repository',
+        help='Path of repository to import the key from, defaults to the configured repository if there is only one, quoted globs supported',
+    )
+    key_import_group.add_argument(
+        '--path',
+        metavar='PATH',
+        help='Path to import the key from backup, defaults to stdin',
+    )
+    key_import_group.add_argument(
+        '-h', '--help', action='help', help='Show this help message and exit'
+    )
+
     key_change_passphrase_parser = key_parsers.add_parser(
         'change-passphrase',
         help='Change the passphrase protecting the repository key',

borgmatic/commands/borgmatic.py

@@ -21,6 +21,7 @@ import borgmatic.actions.delete
 import borgmatic.actions.export_key
 import borgmatic.actions.export_tar
 import borgmatic.actions.extract
+import borgmatic.actions.import_key
 import borgmatic.actions.info
 import borgmatic.actions.list
 import borgmatic.actions.mount
@@ -33,6 +34,7 @@ import borgmatic.actions.restore
 import borgmatic.actions.transfer
 import borgmatic.commands.completion.bash
 import borgmatic.commands.completion.fish
+import borgmatic.config.paths
 from borgmatic.borg import umount as borg_umount
 from borgmatic.borg import version as borg_version
 from borgmatic.commands.arguments import parse_arguments
@@ -67,6 +69,113 @@ def get_skip_actions(config, arguments):
     return skip_actions
 
 
+class Monitoring_hooks:
+    '''
+    A Python context manager for pinging monitoring hooks for the start state before the wrapped
+    code and log and finish (or failure) after the wrapped code. Also responsible for
+    initializing/destroying the monitoring hooks.
+
+    Example use as a context manager:
+
+       with Monitoring_hooks(config_filename, config, arguments, global_arguments):
+           do_stuff()
+    '''
+
+    def __init__(self, config_filename, config, arguments, global_arguments):
+        '''
+        Given a configuration filename, a configuration dict, command-line arguments as an
+        argparse.Namespace, and global arguments as an argparse.Namespace, save relevant data points
+        for use below.
+        '''
+        using_primary_action = {'create', 'prune', 'compact', 'check'}.intersection(arguments)
+        self.config_filename = config_filename
+        self.config = config
+        self.dry_run = global_arguments.dry_run
+        self.monitoring_log_level = verbosity_to_log_level(global_arguments.monitoring_verbosity)
+        self.monitoring_hooks_are_activated = (
+            using_primary_action and self.monitoring_log_level != DISABLED
+        )
+
+    def __enter__(self):
+        '''
+        If monitoring hooks are enabled and a primary action is in use, initialize monitoring hooks
+        and ping them for the "start" state.
+        '''
+        if not self.monitoring_hooks_are_activated:
+            return
+
+        dispatch.call_hooks(
+            'initialize_monitor',
+            self.config,
+            dispatch.Hook_type.MONITORING,
+            self.config_filename,
+            self.monitoring_log_level,
+            self.dry_run,
+        )
+
+        try:
+            dispatch.call_hooks(
+                'ping_monitor',
+                self.config,
+                dispatch.Hook_type.MONITORING,
+                self.config_filename,
+                monitor.State.START,
+                self.monitoring_log_level,
+                self.dry_run,
+            )
+        except (OSError, CalledProcessError) as error:
+            raise ValueError(f'Error pinging monitor: {error}')
+
+    def __exit__(self, exception_type, exception, traceback):
+        '''
+        If monitoring hooks are enabled and a primary action is in use, ping monitoring hooks for
+        the "log" state and also the "finish" or "fail" states (depending on whether there's an
+        exception). Lastly, destroy monitoring hooks.
+        '''
+        if not self.monitoring_hooks_are_activated:
+            return
+
+        # Send logs irrespective of error.
+        try:
+            dispatch.call_hooks(
+                'ping_monitor',
+                self.config,
+                dispatch.Hook_type.MONITORING,
+                self.config_filename,
+                monitor.State.LOG,
+                self.monitoring_log_level,
+                self.dry_run,
+            )
+        except (OSError, CalledProcessError) as error:
+            raise ValueError(f'Error pinging monitor: {error}')
+
+        try:
+            dispatch.call_hooks(
+                'ping_monitor',
+                self.config,
+                dispatch.Hook_type.MONITORING,
+                self.config_filename,
+                monitor.State.FAIL if exception else monitor.State.FINISH,
+                self.monitoring_log_level,
+                self.dry_run,
+            )
+        except (OSError, CalledProcessError) as error:
+            # If the wrapped code errored, prefer raising that exception, as it's probably more
+            # important than a monitor failing to ping.
+            if exception:
+                return
+
+            raise ValueError(f'Error pinging monitor: {error}')
+
+        dispatch.call_hooks(
+            'destroy_monitor',
+            self.config,
+            dispatch.Hook_type.MONITORING,
+            self.monitoring_log_level,
+            self.dry_run,
+        )
+
+
 def run_configuration(config_filename, config, config_paths, arguments):
     '''
     Given a config filename, the corresponding parsed config dict, a sequence of loaded
@@ -84,11 +193,9 @@ def run_configuration(config_filename, config, config_paths, arguments):
     remote_path = config.get('remote_path')
     retries = config.get('retries', 0)
     retry_wait = config.get('retry_wait', 0)
+    repo_queue = Queue()
     encountered_error = None
-    error_repository = ''
-    using_primary_action = {'create', 'prune', 'compact', 'check'}.intersection(arguments)
-    monitoring_log_level = verbosity_to_log_level(global_arguments.monitoring_verbosity)
-    monitoring_hooks_are_activated = using_primary_action and monitoring_log_level != DISABLED
+    error_repository = None
     skip_actions = get_skip_actions(config, arguments)
 
     if skip_actions:
@@ -97,168 +204,105 @@ def run_configuration(config_filename, config, config_paths, arguments):
         )
 
     try:
-        local_borg_version = borg_version.local_borg_version(config, local_path)
-        logger.debug(f'Borg {local_borg_version}')
+        with Monitoring_hooks(config_filename, config, arguments, global_arguments):
+            with borgmatic.hooks.command.Before_after_hooks(
+                command_hooks=config.get('commands'),
+                before_after='configuration',
+                umask=config.get('umask'),
+                working_directory=borgmatic.config.paths.get_working_directory(config),
+                dry_run=global_arguments.dry_run,
+                action_names=arguments.keys(),
+                configuration_filename=config_filename,
+                log_file=arguments['global'].log_file or '',
+            ):
+                try:
+                    local_borg_version = borg_version.local_borg_version(config, local_path)
+                    logger.debug(f'Borg {local_borg_version}')
+                except (OSError, CalledProcessError, ValueError) as error:
+                    yield from log_error_records(
+                        f'{config_filename}: Error getting local Borg version', error
+                    )
+                    return
+
+                for repo in config['repositories']:
+                    repo_queue.put(
+                        (repo, 0),
+                    )
+
+                while not repo_queue.empty():
+                    repository, retry_num = repo_queue.get()
+
+                    with Log_prefix(repository.get('label', repository['path'])):
+                        logger.debug('Running actions for repository')
+                        timeout = retry_num * retry_wait
+                        if timeout:
+                            logger.warning(f'Sleeping {timeout}s before next retry')
+                            time.sleep(timeout)
+                        try:
+                            yield from run_actions(
+                                arguments=arguments,
+                                config_filename=config_filename,
+                                config=config,
+                                config_paths=config_paths,
+                                local_path=local_path,
+                                remote_path=remote_path,
+                                local_borg_version=local_borg_version,
+                                repository=repository,
+                            )
+                        except (OSError, CalledProcessError, ValueError) as error:
+                            if retry_num < retries:
+                                repo_queue.put(
+                                    (repository, retry_num + 1),
+                                )
+                                tuple(  # Consume the generator so as to trigger logging.
+                                    log_error_records(
+                                        'Error running actions for repository',
+                                        error,
+                                        levelno=logging.WARNING,
+                                        log_command_error_output=True,
+                                    )
+                                )
+                                logger.warning(f'Retrying... attempt {retry_num + 1}/{retries}')
+                                continue
+
+                            if command.considered_soft_failure(error):
+                                continue
+
+                            yield from log_error_records(
+                                'Error running actions for repository',
+                                error,
+                            )
+                            encountered_error = error
+                            error_repository = repository
+
     except (OSError, CalledProcessError, ValueError) as error:
-        yield from log_error_records(f'{config_filename}: Error getting local Borg version', error)
+        yield from log_error_records('Error running configuration', error)
+
+        encountered_error = error
+
+    if not encountered_error:
         return
 
     try:
-        if monitoring_hooks_are_activated:
-            dispatch.call_hooks(
-                'initialize_monitor',
-                config,
-                dispatch.Hook_type.MONITORING,
-                config_filename,
-                monitoring_log_level,
-                global_arguments.dry_run,
-            )
-
-            dispatch.call_hooks(
-                'ping_monitor',
-                config,
-                dispatch.Hook_type.MONITORING,
-                config_filename,
-                monitor.State.START,
-                monitoring_log_level,
-                global_arguments.dry_run,
-            )
+        command.execute_hooks(
+            command.filter_hooks(
+                config.get('commands'), after='error', action_names=arguments.keys()
+            ),
+            config.get('umask'),
+            borgmatic.config.paths.get_working_directory(config),
+            global_arguments.dry_run,
+            configuration_filename=config_filename,
+            log_file=arguments['global'].log_file or '',
+            repository=error_repository.get('path', '') if error_repository else '',
+            repository_label=error_repository.get('label', '') if error_repository else '',
+            error=encountered_error,
+            output=getattr(encountered_error, 'output', ''),
+        )
     except (OSError, CalledProcessError) as error:
         if command.considered_soft_failure(error):
             return
 
-        encountered_error = error
-        yield from log_error_records(f'{config_filename}: Error pinging monitor', error)
-
-    if not encountered_error:
-        repo_queue = Queue()
-        for repo in config['repositories']:
-            repo_queue.put(
-                (repo, 0),
-            )
-
-        while not repo_queue.empty():
-            repository, retry_num = repo_queue.get()
-
-            with Log_prefix(repository.get('label', repository['path'])):
-                logger.debug('Running actions for repository')
-                timeout = retry_num * retry_wait
-                if timeout:
-                    logger.warning(f'Sleeping {timeout}s before next retry')
-                    time.sleep(timeout)
-                try:
-                    yield from run_actions(
-                        arguments=arguments,
-                        config_filename=config_filename,
-                        config=config,
-                        config_paths=config_paths,
-                        local_path=local_path,
-                        remote_path=remote_path,
-                        local_borg_version=local_borg_version,
-                        repository=repository,
-                    )
-                except (OSError, CalledProcessError, ValueError) as error:
-                    if retry_num < retries:
-                        repo_queue.put(
-                            (repository, retry_num + 1),
-                        )
-                        tuple(  # Consume the generator so as to trigger logging.
-                            log_error_records(
-                                'Error running actions for repository',
-                                error,
-                                levelno=logging.WARNING,
-                                log_command_error_output=True,
-                            )
-                        )
-                        logger.warning(f'Retrying... attempt {retry_num + 1}/{retries}')
-                        continue
-
-                    if command.considered_soft_failure(error):
-                        continue
-
-                    yield from log_error_records(
-                        'Error running actions for repository',
-                        error,
-                    )
-                    encountered_error = error
-                    error_repository = repository['path']
-
-    try:
-        if monitoring_hooks_are_activated:
-            # Send logs irrespective of error.
-            dispatch.call_hooks(
-                'ping_monitor',
-                config,
-                dispatch.Hook_type.MONITORING,
-                config_filename,
-                monitor.State.LOG,
-                monitoring_log_level,
-                global_arguments.dry_run,
-            )
-    except (OSError, CalledProcessError) as error:
-        if not command.considered_soft_failure(error):
-            encountered_error = error
-            yield from log_error_records('Error pinging monitor', error)
-
-    if not encountered_error:
-        try:
-            if monitoring_hooks_are_activated:
-                dispatch.call_hooks(
-                    'ping_monitor',
-                    config,
-                    dispatch.Hook_type.MONITORING,
-                    config_filename,
-                    monitor.State.FINISH,
-                    monitoring_log_level,
-                    global_arguments.dry_run,
-                )
-                dispatch.call_hooks(
-                    'destroy_monitor',
-                    config,
-                    dispatch.Hook_type.MONITORING,
-                    monitoring_log_level,
-                    global_arguments.dry_run,
-                )
-        except (OSError, CalledProcessError) as error:
-            if command.considered_soft_failure(error):
-                return
-
-            encountered_error = error
-            yield from log_error_records(f'{config_filename}: Error pinging monitor', error)
-
-    if encountered_error and using_primary_action:
-        try:
-            command.execute_hook(
-                config.get('on_error'),
-                config.get('umask'),
-                config_filename,
-                'on-error',
-                global_arguments.dry_run,
-                repository=error_repository,
-                error=encountered_error,
-                output=getattr(encountered_error, 'output', ''),
-            )
-            dispatch.call_hooks(
-                'ping_monitor',
-                config,
-                dispatch.Hook_type.MONITORING,
-                config_filename,
-                monitor.State.FAIL,
-                monitoring_log_level,
-                global_arguments.dry_run,
-            )
-            dispatch.call_hooks(
-                'destroy_monitor',
-                config,
-                dispatch.Hook_type.MONITORING,
-                monitoring_log_level,
-                global_arguments.dry_run,
-            )
-        except (OSError, CalledProcessError) as error:
-            if command.considered_soft_failure(error):
-                return
-
-            yield from log_error_records(f'{config_filename}: Error running on-error hook', error)
+        yield from log_error_records(f'{config_filename}: Error running after error hook', error)
 
 
 def run_actions(
@@ -289,6 +333,7 @@ def run_actions(
     global_arguments = arguments['global']
     dry_run_label = ' (dry run; not making any changes)' if global_arguments.dry_run else ''
     hook_context = {
+        'configuration_filename': config_filename,
         'repository_label': repository.get('label', ''),
         'log_file': global_arguments.log_file if global_arguments.log_file else '',
         # Deprecated: For backwards compatibility with borgmatic < 1.6.0.
@@ -297,240 +342,248 @@ def run_actions(
     }
     skip_actions = set(get_skip_actions(config, arguments))
 
-    command.execute_hook(
-        config.get('before_actions'),
-        config.get('umask'),
-        config_filename,
-        'pre-actions',
-        global_arguments.dry_run,
+    with borgmatic.hooks.command.Before_after_hooks(
+        command_hooks=config.get('commands'),
+        before_after='repository',
+        umask=config.get('umask'),
+        working_directory=borgmatic.config.paths.get_working_directory(config),
+        dry_run=global_arguments.dry_run,
+        action_names=arguments.keys(),
         **hook_context,
-    )
+    ):
+        for action_name, action_arguments in arguments.items():
+            if action_name == 'global':
+                continue
 
-    for action_name, action_arguments in arguments.items():
-        if action_name == 'repo-create' and action_name not in skip_actions:
-            borgmatic.actions.repo_create.run_repo_create(
-                repository,
-                config,
-                local_borg_version,
-                action_arguments,
-                global_arguments,
-                local_path,
-                remote_path,
-            )
-        elif action_name == 'transfer' and action_name not in skip_actions:
-            borgmatic.actions.transfer.run_transfer(
-                repository,
-                config,
-                local_borg_version,
-                action_arguments,
-                global_arguments,
-                local_path,
-                remote_path,
-            )
-        elif action_name == 'create' and action_name not in skip_actions:
-            yield from borgmatic.actions.create.run_create(
-                config_filename,
-                repository,
-                config,
-                config_paths,
-                hook_context,
-                local_borg_version,
-                action_arguments,
-                global_arguments,
-                dry_run_label,
-                local_path,
-                remote_path,
-            )
-        elif action_name == 'prune' and action_name not in skip_actions:
-            borgmatic.actions.prune.run_prune(
-                config_filename,
-                repository,
-                config,
-                hook_context,
-                local_borg_version,
-                action_arguments,
-                global_arguments,
-                dry_run_label,
-                local_path,
-                remote_path,
-            )
-        elif action_name == 'compact' and action_name not in skip_actions:
-            borgmatic.actions.compact.run_compact(
-                config_filename,
-                repository,
-                config,
-                hook_context,
-                local_borg_version,
-                action_arguments,
-                global_arguments,
-                dry_run_label,
-                local_path,
-                remote_path,
-            )
-        elif action_name == 'check' and action_name not in skip_actions:
-            if checks.repository_enabled_for_checks(repository, config):
-                borgmatic.actions.check.run_check(
-                    config_filename,
-                    repository,
-                    config,
-                    hook_context,
-                    local_borg_version,
-                    action_arguments,
-                    global_arguments,
-                    local_path,
-                    remote_path,
-                )
-        elif action_name == 'extract' and action_name not in skip_actions:
-            borgmatic.actions.extract.run_extract(
-                config_filename,
-                repository,
-                config,
-                hook_context,
-                local_borg_version,
-                action_arguments,
-                global_arguments,
-                local_path,
-                remote_path,
-            )
-        elif action_name == 'export-tar' and action_name not in skip_actions:
-            borgmatic.actions.export_tar.run_export_tar(
-                repository,
-                config,
-                local_borg_version,
-                action_arguments,
-                global_arguments,
-                local_path,
-                remote_path,
-            )
-        elif action_name == 'mount' and action_name not in skip_actions:
-            borgmatic.actions.mount.run_mount(
-                repository,
-                config,
-                local_borg_version,
-                action_arguments,
-                global_arguments,
-                local_path,
-                remote_path,
-            )
-        elif action_name == 'restore' and action_name not in skip_actions:
-            borgmatic.actions.restore.run_restore(
-                repository,
-                config,
-                local_borg_version,
-                action_arguments,
-                global_arguments,
-                local_path,
-                remote_path,
-            )
-        elif action_name == 'repo-list' and action_name not in skip_actions:
-            yield from borgmatic.actions.repo_list.run_repo_list(
-                repository,
-                config,
-                local_borg_version,
-                action_arguments,
-                global_arguments,
-                local_path,
-                remote_path,
-            )
-        elif action_name == 'list' and action_name not in skip_actions:
-            yield from borgmatic.actions.list.run_list(
-                repository,
-                config,
-                local_borg_version,
-                action_arguments,
-                global_arguments,
-                local_path,
-                remote_path,
-            )
-        elif action_name == 'repo-info' and action_name not in skip_actions:
-            yield from borgmatic.actions.repo_info.run_repo_info(
-                repository,
-                config,
-                local_borg_version,
-                action_arguments,
-                global_arguments,
-                local_path,
-                remote_path,
-            )
-        elif action_name == 'info' and action_name not in skip_actions:
-            yield from borgmatic.actions.info.run_info(
-                repository,
-                config,
-                local_borg_version,
-                action_arguments,
-                global_arguments,
-                local_path,
-                remote_path,
-            )
-        elif action_name == 'break-lock' and action_name not in skip_actions:
-            borgmatic.actions.break_lock.run_break_lock(
-                repository,
-                config,
-                local_borg_version,
-                action_arguments,
-                global_arguments,
-                local_path,
-                remote_path,
-            )
-        elif action_name == 'export' and action_name not in skip_actions:
-            borgmatic.actions.export_key.run_export_key(
-                repository,
-                config,
-                local_borg_version,
-                action_arguments,
-                global_arguments,
-                local_path,
-                remote_path,
-            )
-        elif action_name == 'change-passphrase' and action_name not in skip_actions:
-            borgmatic.actions.change_passphrase.run_change_passphrase(
-                repository,
-                config,
-                local_borg_version,
-                action_arguments,
-                global_arguments,
-                local_path,
-                remote_path,
-            )
-        elif action_name == 'delete' and action_name not in skip_actions:
-            borgmatic.actions.delete.run_delete(
-                repository,
-                config,
-                local_borg_version,
-                action_arguments,
-                global_arguments,
-                local_path,
-                remote_path,
-            )
-        elif action_name == 'repo-delete' and action_name not in skip_actions:
-            borgmatic.actions.repo_delete.run_repo_delete(
-                repository,
-                config,
-                local_borg_version,
-                action_arguments,
-                global_arguments,
-                local_path,
-                remote_path,
-            )
-        elif action_name == 'borg' and action_name not in skip_actions:
-            borgmatic.actions.borg.run_borg(
-                repository,
-                config,
-                local_borg_version,
-                action_arguments,
-                global_arguments,
-                local_path,
-                remote_path,
-            )
-
-    command.execute_hook(
-        config.get('after_actions'),
-        config.get('umask'),
-        config_filename,
-        'post-actions',
-        global_arguments.dry_run,
-        **hook_context,
-    )
+            with borgmatic.hooks.command.Before_after_hooks(
+                command_hooks=config.get('commands'),
+                before_after='action',
+                umask=config.get('umask'),
+                working_directory=borgmatic.config.paths.get_working_directory(config),
+                dry_run=global_arguments.dry_run,
+                action_names=arguments.keys(),
+                **hook_context,
+            ):
+                if action_name == 'repo-create' and action_name not in skip_actions:
+                    borgmatic.actions.repo_create.run_repo_create(
+                        repository,
+                        config,
+                        local_borg_version,
+                        action_arguments,
+                        global_arguments,
+                        local_path,
+                        remote_path,
+                    )
+                elif action_name == 'transfer' and action_name not in skip_actions:
+                    borgmatic.actions.transfer.run_transfer(
+                        repository,
+                        config,
+                        local_borg_version,
+                        action_arguments,
+                        global_arguments,
+                        local_path,
+                        remote_path,
+                    )
+                elif action_name == 'create' and action_name not in skip_actions:
+                    yield from borgmatic.actions.create.run_create(
+                        config_filename,
+                        repository,
+                        config,
+                        config_paths,
+                        local_borg_version,
+                        action_arguments,
+                        global_arguments,
+                        dry_run_label,
+                        local_path,
+                        remote_path,
+                    )
+                elif action_name == 'prune' and action_name not in skip_actions:
+                    borgmatic.actions.prune.run_prune(
+                        config_filename,
+                        repository,
+                        config,
+                        local_borg_version,
+                        action_arguments,
+                        global_arguments,
+                        dry_run_label,
+                        local_path,
+                        remote_path,
+                    )
+                elif action_name == 'compact' and action_name not in skip_actions:
+                    borgmatic.actions.compact.run_compact(
+                        config_filename,
+                        repository,
+                        config,
+                        local_borg_version,
+                        action_arguments,
+                        global_arguments,
+                        dry_run_label,
+                        local_path,
+                        remote_path,
+                    )
+                elif action_name == 'check' and action_name not in skip_actions:
+                    if checks.repository_enabled_for_checks(repository, config):
+                        borgmatic.actions.check.run_check(
+                            config_filename,
+                            repository,
+                            config,
+                            local_borg_version,
+                            action_arguments,
+                            global_arguments,
+                            local_path,
+                            remote_path,
+                        )
+                elif action_name == 'extract' and action_name not in skip_actions:
+                    borgmatic.actions.extract.run_extract(
+                        config_filename,
+                        repository,
+                        config,
+                        local_borg_version,
+                        action_arguments,
+                        global_arguments,
+                        local_path,
+                        remote_path,
+                    )
+                elif action_name == 'export-tar' and action_name not in skip_actions:
+                    borgmatic.actions.export_tar.run_export_tar(
+                        repository,
+                        config,
+                        local_borg_version,
+                        action_arguments,
+                        global_arguments,
+                        local_path,
+                        remote_path,
+                    )
+                elif action_name == 'mount' and action_name not in skip_actions:
+                    borgmatic.actions.mount.run_mount(
+                        repository,
+                        config,
+                        local_borg_version,
+                        action_arguments,
+                        global_arguments,
+                        local_path,
+                        remote_path,
+                    )
+                elif action_name == 'restore' and action_name not in skip_actions:
+                    borgmatic.actions.restore.run_restore(
+                        repository,
+                        config,
+                        local_borg_version,
+                        action_arguments,
+                        global_arguments,
+                        local_path,
+                        remote_path,
+                    )
+                elif action_name == 'repo-list' and action_name not in skip_actions:
+                    yield from borgmatic.actions.repo_list.run_repo_list(
+                        repository,
+                        config,
+                        local_borg_version,
+                        action_arguments,
+                        global_arguments,
+                        local_path,
+                        remote_path,
+                    )
+                elif action_name == 'list' and action_name not in skip_actions:
+                    yield from borgmatic.actions.list.run_list(
+                        repository,
+                        config,
+                        local_borg_version,
+                        action_arguments,
+                        global_arguments,
+                        local_path,
+                        remote_path,
+                    )
+                elif action_name == 'repo-info' and action_name not in skip_actions:
+                    yield from borgmatic.actions.repo_info.run_repo_info(
+                        repository,
+                        config,
+                        local_borg_version,
+                        action_arguments,
+                        global_arguments,
+                        local_path,
+                        remote_path,
+                    )
+                elif action_name == 'info' and action_name not in skip_actions:
+                    yield from borgmatic.actions.info.run_info(
+                        repository,
+                        config,
+                        local_borg_version,
+                        action_arguments,
+                        global_arguments,
+                        local_path,
+                        remote_path,
+                    )
+                elif action_name == 'break-lock' and action_name not in skip_actions:
+                    borgmatic.actions.break_lock.run_break_lock(
+                        repository,
+                        config,
+                        local_borg_version,
+                        action_arguments,
+                        global_arguments,
+                        local_path,
+                        remote_path,
+                    )
+                elif action_name == 'export' and action_name not in skip_actions:
+                    borgmatic.actions.export_key.run_export_key(
+                        repository,
+                        config,
+                        local_borg_version,
+                        action_arguments,
+                        global_arguments,
+                        local_path,
+                        remote_path,
+                    )
+                elif action_name == 'import' and action_name not in skip_actions:
+                    borgmatic.actions.import_key.run_import_key(
+                        repository,
+                        config,
+                        local_borg_version,
+                        action_arguments,
+                        global_arguments,
+                        local_path,
+                        remote_path,
+                    )
+                elif action_name == 'change-passphrase' and action_name not in skip_actions:
+                    borgmatic.actions.change_passphrase.run_change_passphrase(
+                        repository,
+                        config,
+                        local_borg_version,
+                        action_arguments,
+                        global_arguments,
+                        local_path,
+                        remote_path,
+                    )
+                elif action_name == 'delete' and action_name not in skip_actions:
+                    borgmatic.actions.delete.run_delete(
+                        repository,
+                        config,
+                        local_borg_version,
+                        action_arguments,
+                        global_arguments,
+                        local_path,
+                        remote_path,
+                    )
+                elif action_name == 'repo-delete' and action_name not in skip_actions:
+                    borgmatic.actions.repo_delete.run_repo_delete(
+                        repository,
+                        config,
+                        local_borg_version,
+                        action_arguments,
+                        global_arguments,
+                        local_path,
+                        remote_path,
+                    )
+                elif action_name == 'borg' and action_name not in skip_actions:
+                    borgmatic.actions.borg.run_borg(
+                        repository,
+                        config,
+                        local_borg_version,
+                        action_arguments,
+                        global_arguments,
+                        local_path,
+                        remote_path,
+                    )
 
 
 def load_configurations(config_filenames, overrides=None, resolve_env=True):
@@ -810,19 +863,21 @@ def collect_configuration_run_summary_logs(configs, config_paths, arguments):
         )
         return
 
-    if 'create' in arguments:
-        try:
-            for config_filename, config in configs.items():
-                command.execute_hook(
-                    config.get('before_everything'),
-                    config.get('umask'),
-                    config_filename,
-                    'pre-everything',
-                    arguments['global'].dry_run,
-                )
-        except (CalledProcessError, ValueError, OSError) as error:
-            yield from log_error_records('Error running pre-everything hook', error)
-            return
+    try:
+        for config_filename, config in configs.items():
+            command.execute_hooks(
+                command.filter_hooks(
+                    config.get('commands'), before='everything', action_names=arguments.keys()
+                ),
+                config.get('umask'),
+                borgmatic.config.paths.get_working_directory(config),
+                arguments['global'].dry_run,
+                configuration_filename=config_filename,
+                log_file=arguments['global'].log_file or '',
+            )
+    except (CalledProcessError, ValueError, OSError) as error:
+        yield from log_error_records('Error running before everything hook', error)
+        return
 
     # Execute the actions corresponding to each configuration file.
     json_results = []
@@ -830,6 +885,7 @@ def collect_configuration_run_summary_logs(configs, config_paths, arguments):
     for config_filename, config in configs.items():
         with Log_prefix(config_filename):
             results = list(run_configuration(config_filename, config, config_paths, arguments))
+
             error_logs = tuple(
                 result for result in results if isinstance(result, logging.LogRecord)
             )
@@ -862,18 +918,20 @@ def collect_configuration_run_summary_logs(configs, config_paths, arguments):
     if json_results:
         sys.stdout.write(json.dumps(json_results))
 
-    if 'create' in arguments:
-        try:
-            for config_filename, config in configs.items():
-                command.execute_hook(
-                    config.get('after_everything'),
-                    config.get('umask'),
-                    config_filename,
-                    'post-everything',
-                    arguments['global'].dry_run,
-                )
-        except (CalledProcessError, ValueError, OSError) as error:
-            yield from log_error_records('Error running post-everything hook', error)
+    try:
+        for config_filename, config in configs.items():
+            command.execute_hooks(
+                command.filter_hooks(
+                    config.get('commands'), after='everything', action_names=arguments.keys()
+                ),
+                config.get('umask'),
+                borgmatic.config.paths.get_working_directory(config),
+                arguments['global'].dry_run,
+                configuration_filename=config_filename,
+                log_file=arguments['global'].log_file or '',
+            )
+    except (CalledProcessError, ValueError, OSError) as error:
+        yield from log_error_records('Error running after everything hook', error)
 
 
 def exit_with_help_link():  # pragma: no cover

borgmatic/config/generate.py

@@ -1,5 +1,6 @@
 import collections
 import io
+import itertools
 import os
 import re
 
@@ -24,41 +25,65 @@ def insert_newline_before_comment(config, field_name):
 def get_properties(schema):
     '''
     Given a schema dict, return its properties. But if it's got sub-schemas with multiple different
-    potential properties, returned their merged properties instead.
+    potential properties, returned their merged properties instead (interleaved so the first
+    properties of each sub-schema come first). The idea is that the user should see all possible
+    options even if they're not all possible together.
     '''
     if 'oneOf' in schema:
         return dict(
-            collections.ChainMap(*[sub_schema['properties'] for sub_schema in schema['oneOf']])
+            item
+            for item in itertools.chain(
+                *itertools.zip_longest(
+                    *[sub_schema['properties'].items() for sub_schema in schema['oneOf']]
+                )
+            )
+            if item is not None
         )
 
     return schema['properties']
 
 
-def schema_to_sample_configuration(schema, level=0, parent_is_sequence=False):
+def schema_to_sample_configuration(schema, source_config=None, level=0, parent_is_sequence=False):
     '''
-    Given a loaded configuration schema, generate and return sample config for it. Include comments
-    for each option based on the schema "description".
+    Given a loaded configuration schema and a source configuration, generate and return sample
+    config for the schema. Include comments for each option based on the schema "description".
+
+    If a source config is given, walk it alongside the given schema so that both can be taken into
+    account when commenting out particular options in add_comments_to_configuration_object().
     '''
     schema_type = schema.get('type')
     example = schema.get('example')
+
     if example is not None:
         return example
 
     if schema_type == 'array' or (isinstance(schema_type, list) and 'array' in schema_type):
         config = ruamel.yaml.comments.CommentedSeq(
-            [schema_to_sample_configuration(schema['items'], level, parent_is_sequence=True)]
+            [
+                schema_to_sample_configuration(
+                    schema['items'], source_config, level, parent_is_sequence=True
+                )
+            ]
         )
         add_comments_to_configuration_sequence(config, schema, indent=(level * INDENT))
     elif schema_type == 'object' or (isinstance(schema_type, list) and 'object' in schema_type):
+        if source_config and isinstance(source_config, list) and isinstance(source_config[0], dict):
+            source_config = dict(collections.ChainMap(*source_config))
+
         config = ruamel.yaml.comments.CommentedMap(
             [
-                (field_name, schema_to_sample_configuration(sub_schema, level + 1))
+                (
+                    field_name,
+                    schema_to_sample_configuration(
+                        sub_schema, (source_config or {}).get(field_name, {}), level + 1
+                    ),
+                )
                 for field_name, sub_schema in get_properties(schema).items()
             ]
         )
         indent = (level * INDENT) + (SEQUENCE_INDENT if parent_is_sequence else 0)
         add_comments_to_configuration_object(
-            config, schema, indent=indent, skip_first=parent_is_sequence
+            config, schema, source_config, indent=indent, skip_first=parent_is_sequence
         )
     else:
         raise ValueError(f'Schema at level {level} is unsupported: {schema}')
@@ -178,14 +203,21 @@ def add_comments_to_configuration_sequence(config, schema, indent=0):
         return
 
 
-REQUIRED_KEYS = {'source_directories', 'repositories', 'keep_daily'}
+DEFAULT_KEYS = {'source_directories', 'repositories', 'keep_daily'}
 COMMENTED_OUT_SENTINEL = 'COMMENT_OUT'
 
 
-def add_comments_to_configuration_object(config, schema, indent=0, skip_first=False):
+def add_comments_to_configuration_object(
+    config, schema, source_config=None, indent=0, skip_first=False
+):
     '''
     Using descriptions from a schema as a source, add those descriptions as comments to the given
-    config mapping, before each field. Indent the comment the given number of characters.
+    configuration dict, putting them before each field. Indent the comment the given number of
+    characters.
+
+    And a sentinel for commenting out options that are neither in DEFAULT_KEYS nor the the given
+    source configuration dict. The idea is that any options used in the source configuration should
+    stay active in the generated configuration.
     '''
     for index, field_name in enumerate(config.keys()):
         if skip_first and index == 0:
@@ -194,10 +226,12 @@ def add_comments_to_configuration_object(config, schema, indent=0, skip_first=Fa
         field_schema = get_properties(schema).get(field_name, {})
         description = field_schema.get('description', '').strip()
 
-        # If this is an optional key, add an indicator to the comment flagging it to be commented
+        # If this isn't a default key, add an indicator to the comment flagging it to be commented
         # out from the sample configuration. This sentinel is consumed by downstream processing that
         # does the actual commenting out.
-        if field_name not in REQUIRED_KEYS:
+        if field_name not in DEFAULT_KEYS and (
+            source_config is None or field_name not in source_config
+        ):
             description = (
                 '\n'.join((description, COMMENTED_OUT_SENTINEL))
                 if description
@@ -217,21 +251,6 @@ def add_comments_to_configuration_object(config, schema, indent=0, skip_first=Fa
 RUAMEL_YAML_COMMENTS_INDEX = 1
 
 
-def remove_commented_out_sentinel(config, field_name):
-    '''
-    Given a configuration CommentedMap and a top-level field name in it, remove any "commented out"
-    sentinel found at the end of its YAML comments. This prevents the given field name from getting
-    commented out by downstream processing that consumes the sentinel.
-    '''
-    try:
-        last_comment_value = config.ca.items[field_name][RUAMEL_YAML_COMMENTS_INDEX][-1].value
-    except KeyError:
-        return
-
-    if last_comment_value == f'# {COMMENTED_OUT_SENTINEL}\n':
-        config.ca.items[field_name][RUAMEL_YAML_COMMENTS_INDEX].pop()
-
-
 def merge_source_configuration_into_destination(destination_config, source_config):
     '''
     Deep merge the given source configuration dict into the destination configuration CommentedMap,
@@ -246,12 +265,6 @@ def merge_source_configuration_into_destination(destination_config, source_confi
         return source_config
 
     for field_name, source_value in source_config.items():
-        # Since this key/value is from the source configuration, leave it uncommented and remove any
-        # sentinel that would cause it to get commented out.
-        remove_commented_out_sentinel(
-            ruamel.yaml.comments.CommentedMap(destination_config), field_name
-        )
-
         # This is a mapping. Recurse for this key/value.
         if isinstance(source_value, collections.abc.Mapping):
             destination_config[field_name] = merge_source_configuration_into_destination(
@@ -297,7 +310,7 @@ def generate_sample_configuration(
         normalize.normalize(source_filename, source_config)
 
     destination_config = merge_source_configuration_into_destination(
-        schema_to_sample_configuration(schema), source_config
+        schema_to_sample_configuration(schema, source_config), source_config
     )
 
     if dry_run:

borgmatic/config/normalize.py

@@ -58,6 +58,90 @@ def normalize_sections(config_filename, config):
     return []
 
 
+def make_command_hook_deprecation_log(config_filename, option_name):  # pragma: no cover
+    '''
+    Given a configuration filename and the name of a configuration option, return a deprecation
+    warning log for it.
+    '''
+    return logging.makeLogRecord(
+        dict(
+            levelno=logging.WARNING,
+            levelname='WARNING',
+            msg=f'{config_filename}: {option_name} is deprecated and support will be removed from a future release. Use commands: instead.',
+        )
+    )
+
+
+def normalize_commands(config_filename, config):
+    '''
+    Given a configuration filename and a configuration dict, transform any "before_*"- and
+    "after_*"-style command hooks into "commands:".
+    '''
+    logs = []
+
+    # Normalize "before_actions" and "after_actions".
+    for preposition in ('before', 'after'):
+        option_name = f'{preposition}_actions'
+        commands = config.pop(option_name, None)
+
+        if commands:
+            logs.append(make_command_hook_deprecation_log(config_filename, option_name))
+            config.setdefault('commands', []).append(
+                {
+                    preposition: 'repository',
+                    'run': commands,
+                }
+            )
+
+    # Normalize "before_backup", "before_prune", "after_backup", "after_prune", etc.
+    for action_name in ('create', 'prune', 'compact', 'check', 'extract'):
+        for preposition in ('before', 'after'):
+            option_name = f'{preposition}_{"backup" if action_name == "create" else action_name}'
+            commands = config.pop(option_name, None)
+
+            if not commands:
+                continue
+
+            logs.append(make_command_hook_deprecation_log(config_filename, option_name))
+            config.setdefault('commands', []).append(
+                {
+                    preposition: 'action',
+                    'when': [action_name],
+                    'run': commands,
+                }
+            )
+
+    # Normalize "on_error".
+    commands = config.pop('on_error', None)
+
+    if commands:
+        logs.append(make_command_hook_deprecation_log(config_filename, 'on_error'))
+        config.setdefault('commands', []).append(
+            {
+                'after': 'error',
+                'when': ['create', 'prune', 'compact', 'check'],
+                'run': commands,
+            }
+        )
+
+    # Normalize "before_everything" and "after_everything".
+    for preposition in ('before', 'after'):
+        option_name = f'{preposition}_everything'
+        commands = config.pop(option_name, None)
+
+        if commands:
+            logs.append(make_command_hook_deprecation_log(config_filename, option_name))
+            config.setdefault('commands', []).append(
+                {
+                    preposition: 'everything',
+                    'when': ['create'],
+                    'run': commands,
+                }
+            )
+
+    return logs
+
+
 def normalize(config_filename, config):
     '''
     Given a configuration filename and a configuration dict of its loaded contents, apply particular
@@ -67,6 +151,7 @@ def normalize(config_filename, config):
     Raise ValueError the configuration cannot be normalized.
     '''
     logs = normalize_sections(config_filename, config)
+    logs += normalize_commands(config_filename, config)
 
     if config.get('borgmatic_source_directory'):
         logs.append(

borgmatic/config/paths.py

@@ -134,7 +134,7 @@ class Runtime_directory:
         '''
         return self.runtime_path
 
-    def __exit__(self, exception, value, traceback):
+    def __exit__(self, exception_type, exception, traceback):
         '''
         Delete any temporary directory that was created as part of initialization.
         '''

borgmatic/config/schema.yaml

@@ -796,8 +796,9 @@ properties:
         items:
             type: string
         description: |
-            List of one or more shell commands or scripts to execute before all
-            the actions for each repository.
+            Deprecated. Use "commands:" instead. List of one or more shell
+            commands or scripts to execute before all the actions for each
+            repository.
         example:
             - "echo Starting actions."
     before_backup:
@@ -805,8 +806,9 @@ properties:
         items:
             type: string
         description: |
-            List of one or more shell commands or scripts to execute before
-            creating a backup, run once per repository.
+            Deprecated. Use "commands:" instead. List of one or more shell
+            commands or scripts to execute before creating a backup, run once
+            per repository.
         example:
             - "echo Starting a backup."
     before_prune:
@@ -814,8 +816,9 @@ properties:
         items:
             type: string
         description: |
-            List of one or more shell commands or scripts to execute before
-            pruning, run once per repository.
+            Deprecated. Use "commands:" instead. List of one or more shell
+            commands or scripts to execute before pruning, run once per
+            repository.
         example:
             - "echo Starting pruning."
     before_compact:
@@ -823,8 +826,9 @@ properties:
         items:
             type: string
         description: |
-            List of one or more shell commands or scripts to execute before
-            compaction, run once per repository.
+            Deprecated. Use "commands:" instead. List of one or more shell
+            commands or scripts to execute before compaction, run once per
+            repository.
         example:
             - "echo Starting compaction."
     before_check:
@@ -832,8 +836,9 @@ properties:
         items:
             type: string
         description: |
-            List of one or more shell commands or scripts to execute before
-            consistency checks, run once per repository.
+            Deprecated. Use "commands:" instead. List of one or more shell
+            commands or scripts to execute before consistency checks, run once
+            per repository.
         example:
             - "echo Starting checks."
     before_extract:
@@ -841,8 +846,9 @@ properties:
         items:
             type: string
         description: |
-            List of one or more shell commands or scripts to execute before
-            extracting a backup, run once per repository.
+            Deprecated. Use "commands:" instead. List of one or more shell
+            commands or scripts to execute before extracting a backup, run once
+            per repository.
         example:
             - "echo Starting extracting."
     after_backup:
@@ -850,8 +856,9 @@ properties:
         items:
             type: string
         description: |
-            List of one or more shell commands or scripts to execute after
-            creating a backup, run once per repository.
+            Deprecated. Use "commands:" instead. List of one or more shell
+            commands or scripts to execute after creating a backup, run once per
+            repository.
         example:
             - "echo Finished a backup."
     after_compact:
@@ -859,8 +866,9 @@ properties:
         items:
             type: string
         description: |
-            List of one or more shell commands or scripts to execute after
-            compaction, run once per repository.
+            Deprecated. Use "commands:" instead. List of one or more shell
+            commands or scripts to execute after compaction, run once per
+            repository.
         example:
             - "echo Finished compaction."
     after_prune:
@@ -868,8 +876,9 @@ properties:
         items:
             type: string
         description: |
-            List of one or more shell commands or scripts to execute after
-            pruning, run once per repository.
+            Deprecated. Use "commands:" instead. List of one or more shell
+            commands or scripts to execute after pruning, run once per
+            repository.
         example:
             - "echo Finished pruning."
     after_check:
@@ -877,8 +886,9 @@ properties:
         items:
             type: string
         description: |
-            List of one or more shell commands or scripts to execute after
-            consistency checks, run once per repository.
+            Deprecated. Use "commands:" instead. List of one or more shell
+            commands or scripts to execute after consistency checks, run once
+            per repository.
         example:
             - "echo Finished checks."
     after_extract:
@@ -886,8 +896,9 @@ properties:
         items:
             type: string
         description: |
-            List of one or more shell commands or scripts to execute after
-            extracting a backup, run once per repository.
+            Deprecated. Use "commands:" instead. List of one or more shell
+            commands or scripts to execute after extracting a backup, run once
+            per repository.
         example:
             - "echo Finished extracting."
     after_actions:
@@ -895,8 +906,9 @@ properties:
         items:
             type: string
         description: |
-            List of one or more shell commands or scripts to execute after all
-            actions for each repository.
+            Deprecated. Use "commands:" instead. List of one or more shell
+            commands or scripts to execute after all actions for each
+            repository.
         example:
             - "echo Finished actions."
     on_error:
@@ -904,9 +916,10 @@ properties:
         items:
             type: string
         description: |
-            List of one or more shell commands or scripts to execute when an
-            exception occurs during a "create", "prune", "compact", or "check"
-            action or an associated before/after hook.
+            Deprecated. Use "commands:" instead. List of one or more shell
+            commands or scripts to execute when an exception occurs during a
+            "create", "prune", "compact", or "check" action or an associated
+            before/after hook.
         example:
             - "echo Error during create/prune/compact/check."
     before_everything:
@@ -914,10 +927,10 @@ properties:
         items:
             type: string
         description: |
-            List of one or more shell commands or scripts to execute before
-            running all actions (if one of them is "create"). These are
-            collected from all configuration files and then run once before all
-            of them (prior to all actions).
+            Deprecated. Use "commands:" instead. List of one or more shell
+            commands or scripts to execute before running all actions (if one of
+            them is "create"). These are collected from all configuration files
+            and then run once before all of them (prior to all actions).
         example:
             - "echo Starting actions."
     after_everything:
@@ -925,12 +938,148 @@ properties:
         items:
             type: string
         description: |
-            List of one or more shell commands or scripts to execute after
-            running all actions (if one of them is "create"). These are
-            collected from all configuration files and then run once after all
-            of them (after any action).
+            Deprecated. Use "commands:" instead. List of one or more shell
+            commands or scripts to execute after running all actions (if one of
+            them is "create"). These are collected from all configuration files
+            and then run once after all of them (after any action).
         example:
             - "echo Completed actions."
+    commands:
+        type: array
+        items:
+            type: object
+            oneOf:
+                - required: [before, run]
+                  additionalProperties: false
+                  properties:
+                      before:
+                          type: string
+                          enum:
+                              - action
+                              - repository
+                              - configuration
+                              - everything
+                          description: |
+                              Name for the point in borgmatic's execution that
+                              the commands should be run before (required if
+                              "after" isn't set):
+                               * "action" runs before each action for each
+                              repository.
+                               * "repository" runs before all actions for each
+                              repository.
+                               * "configuration" runs before all actions and
+                              repositories in the current configuration file.
+                               * "everything" runs before all configuration
+                              files.
+                          example: action
+                      when:
+                          type: array
+                          items:
+                              type: string
+                              enum:
+                                  - repo-create
+                                  - transfer
+                                  - prune
+                                  - compact
+                                  - create
+                                  - check
+                                  - delete
+                                  - extract
+                                  - config
+                                  - export-tar
+                                  - mount
+                                  - umount
+                                  - repo-delete
+                                  - restore
+                                  - repo-list
+                                  - list
+                                  - repo-info
+                                  - info
+                                  - break-lock
+                                  - key
+                                  - borg
+                          description: |
+                              List of actions for which the commands will be
+                              run. Defaults to running for all actions.
+                          example: [create, prune, compact, check]
+                      run:
+                          type: array
+                          items:
+                              type: string
+                          description: |
+                              List of one or more shell commands or scripts to
+                              run when this command hook is triggered. Required.
+                          example:
+                              - "echo Doing stuff."
+                - required: [after, run]
+                  additionalProperties: false
+                  properties:
+                      after:
+                          type: string
+                          enum:
+                              - action
+                              - repository
+                              - configuration
+                              - everything
+                              - error
+                          description: |
+                              Name for the point in borgmatic's execution that
+                              the commands should be run after (required if
+                              "before" isn't set):
+                               * "action" runs after each action for each
+                              repository.
+                               * "repository" runs after all actions for each
+                              repository.
+                               * "configuration" runs after all actions and
+                              repositories in the current configuration file.
+                               * "everything" runs after all configuration
+                              files.
+                               * "error" runs after an error occurs.
+                          example: action
+                      when:
+                          type: array
+                          items:
+                              type: string
+                              enum:
+                                  - repo-create
+                                  - transfer
+                                  - prune
+                                  - compact
+                                  - create
+                                  - check
+                                  - delete
+                                  - extract
+                                  - config
+                                  - export-tar
+                                  - mount
+                                  - umount
+                                  - repo-delete
+                                  - restore
+                                  - repo-list
+                                  - list
+                                  - repo-info
+                                  - info
+                                  - break-lock
+                                  - key
+                                  - borg
+                          description: |
+                              Only trigger the hook when borgmatic is run with
+                              particular actions listed here. Defaults to
+                              running for all actions.
+                          example: [create, prune, compact, check]
+                      run:
+                          type: array
+                          items:
+                              type: string
+                          description: |
+                              List of one or more shell commands or scripts to
+                              run when this command hook is triggered. Required.
+                          example:
+                              - "echo Doing stuff."
+        description: |
+            List of one or more command hooks to execute, triggered at
+            particular points during borgmatic's execution. For each command
+            hook, specify one of "before" or "after", not both.
     bootstrap:
         type: object
         properties:
@@ -1040,6 +1189,18 @@ properties:
                         individual databases. See the pg_dump documentation for
                         more about formats.
                     example: directory
+                compression:
+                    type: ["string", "integer"]
+                    description: |
+                        Database dump compression level (integer) or method
+                        ("gzip", "lz4", "zstd", or "none") and optional
+                        colon-separated detail. Defaults to moderate "gzip" for
+                        "custom" and "directory" formats and no compression for
+                        the "plain" format. Compression is not supported for the
+                        "tar" format. Be aware that Borg does its own
+                        compression as well, so you may not need it in both
+                        places.
+                    example: none
                 ssl_mode:
                     type: string
                     enum: ['disable', 'allow', 'prefer',
@@ -1076,11 +1237,11 @@ properties:
                         Command to use instead of "pg_dump" or "pg_dumpall".
                         This can be used to run a specific pg_dump version
                         (e.g., one inside a running container). If you run it
-                        from within a container, make sure to mount your
-                        host's ".borgmatic" folder into the container using
-                        the same directory structure. Defaults to "pg_dump"
-                        for single database dump or "pg_dumpall" to dump all
-                        databases.
+                        from within a container, make sure to mount the path in
+                        the "user_runtime_directory" option from the host into
+                        the container at the same location. Defaults to
+                        "pg_dump" for single database dump or "pg_dumpall" to
+                        dump all databases.
                     example: docker exec my_pg_container pg_dump
                 pg_restore_command:
                     type: string
@@ -1198,15 +1359,30 @@ properties:
                         Defaults to the "password" option. Supports the
                         "{credential ...}" syntax.
                     example: trustsome1
+                tls:
+                    type: boolean
+                    description: |
+                        Whether to TLS-encrypt data transmitted between the
+                        client and server. The default varies based on the
+                        MariaDB version.
+                    example: false
+                restore_tls:
+                    type: boolean
+                    description: |
+                        Whether to TLS-encrypt data transmitted between the
+                        client and restore server. The default varies based on
+                        the MariaDB version.
+                    example: false
                 mariadb_dump_command:
                     type: string
                     description: |
                         Command to use instead of "mariadb-dump". This can be
                         used to run a specific mariadb_dump version (e.g., one
-                        inside a running container). If you run it from within
-                        a container, make sure to mount your host's
-                        ".borgmatic" folder into the container using the same
-                        directory structure. Defaults to "mariadb-dump".
+                        inside a running container). If you run it from within a
+                        container, make sure to mount the path in the
+                        "user_runtime_directory" option from the host into the
+                        container at the same location. Defaults to
+                        "mariadb-dump".
                     example: docker exec mariadb_container mariadb-dump
                 mariadb_command:
                     type: string
@@ -1328,15 +1504,29 @@ properties:
                         Defaults to the "password" option. Supports the
                         "{credential ...}" syntax.
                     example: trustsome1
+                tls:
+                    type: boolean
+                    description: |
+                        Whether to TLS-encrypt data transmitted between the
+                        client and server. The default varies based on the
+                        MySQL installation.
+                    example: false
+                restore_tls:
+                    type: boolean
+                    description: |
+                        Whether to TLS-encrypt data transmitted between the
+                        client and restore server. The default varies based on
+                        the MySQL installation.
+                    example: false
                 mysql_dump_command:
                     type: string
                     description: |
-                        Command to use instead of "mysqldump". This can be
-                        used to run a specific mysql_dump version (e.g., one
-                        inside a running container). If you run it from within
-                        a container, make sure to mount your host's
-                        ".borgmatic" folder into the container using the same
-                        directory structure. Defaults to "mysqldump".
+                        Command to use instead of "mysqldump". This can be used
+                        to run a specific mysql_dump version (e.g., one inside a
+                        running container). If you run it from within a
+                        container, make sure to mount the path in the
+                        "user_runtime_directory" option from the host into the
+                        container at the same location. Defaults to "mysqldump".
                     example: docker exec mysql_container mysqldump
                 mysql_command:
                     type: string
@@ -1423,6 +1613,24 @@ properties:
                         Path to the SQLite database file to restore to. Defaults
                         to the "path" option.
                     example: /var/lib/sqlite/users.db
+                sqlite_command:
+                    type: string
+                    description: |
+                        Command to use instead of "sqlite3". This can be used to
+                        run a specific sqlite3 version (e.g., one inside a
+                        running container). If you run it from within a
+                        container, make sure to mount the path in the
+                        "user_runtime_directory" option from the host into the
+                        container at the same location. Defaults to "sqlite3".
+                    example: docker exec sqlite_container sqlite3
+                sqlite_restore_command:
+                    type: string
+                    description: |
+                        Command to run when restoring a database instead
+                        of "sqlite3". This can be used to run a specific 
+                        sqlite3 version (e.g., one inside a running container). 
+                        Defaults to "sqlite3".
+                    example: docker exec sqlite_container sqlite3
     mongodb_databases:
         type: array
         items:
@@ -1518,6 +1726,25 @@ properties:
                         dump command, without performing any validation on them.
                         See mongorestore documentation for details.
                     example: --restoreDbUsersAndRoles
+                mongodump_command:
+                    type: string
+                    description: |
+                        Command to use instead of "mongodump". This can be used
+                        to run a specific mongodump version (e.g., one inside a
+                        running container). If you run it from within a
+                        container, make sure to mount the path in the
+                        "user_runtime_directory" option from the host into the
+                        container at the same location.  Defaults to
+                        "mongodump".
+                    example: docker exec mongodb_container mongodump
+                mongorestore_command:
+                    type: string
+                    description: |
+                        Command to run when restoring a database instead of
+                        "mongorestore". This can be used to run a specific
+                        mongorestore version (e.g., one inside a running
+                        container). Defaults to "mongorestore".
+                    example: docker exec mongodb_container mongorestore
         description: |
             List of one or more MongoDB databases to dump before creating a
             backup, run once per configuration file. The database dumps are
@@ -1907,6 +2134,8 @@ properties:
     zabbix:
         type: object
         additionalProperties: false
+        required:
+            - server
         properties:
             itemid:
                 type: integer
@@ -1929,7 +2158,8 @@ properties:
             server:
                 type: string
                 description: |
-                    The address of your Zabbix instance.
+                    The API endpoint URL of your Zabbix instance, usually ending
+                    with "/api_jsonrpc.php". Required.
                 example: https://zabbix.your-domain.com
             username:
                 type: string
@@ -2200,6 +2430,12 @@ properties:
                     - start
                     - finish
                     - fail
+            verify_tls:
+                type: boolean
+                description: |
+                    Verify the TLS certificate of the push URL host. Defaults to
+                    true.
+                example: false
         description: |
             Configuration for a monitoring integration with Uptime Kuma using
             the Push monitor type.
@@ -2230,6 +2466,12 @@ properties:
                     PagerDuty integration key used to notify PagerDuty when a
                     backup errors. Supports the "{credential ...}" syntax.
                 example: a177cad45bd374409f78906a810a3074
+            send_logs:
+                type: boolean
+                description: |
+                    Send borgmatic logs to PagerDuty when a backup errors.
+                    Defaults to true.
+                example: false
         description: |
             Configuration for a monitoring integration with PagerDuty. Create an
             account at https://www.pagerduty.com if you'd like to use this
@@ -2402,3 +2644,25 @@ properties:
         description: |
             Configuration for integration with Linux LVM (Logical Volume
             Manager).
+    container:
+        type: object
+        additionalProperties: false
+        properties:
+            secrets_directory:
+                type: string
+                description: |
+                    Secrets directory to use instead of "/run/secrets".
+                example: /path/to/secrets
+        description: |
+            Configuration for integration with Docker or Podman secrets.
+    keepassxc:
+        type: object
+        additionalProperties: false
+        properties:
+            keepassxc_cli_command:
+                type: string
+                description: |
+                    Command to use instead of "keepassxc-cli".
+                example: /usr/local/bin/keepassxc-cli
+        description: |
+            Configuration for integration with the KeePassXC password manager.

borgmatic/config/validate.py

@@ -138,16 +138,22 @@ def parse_configuration(config_filename, schema_filename, overrides=None, resolv
     return config, config_paths, logs
 
 
-def normalize_repository_path(repository):
+def normalize_repository_path(repository, base=None):
     '''
     Given a repository path, return the absolute path of it (for local repositories).
+    Optionally, use a base path for resolving relative paths, e.g. to the configured working directory.
     '''
     # A colon in the repository could mean that it's either a file:// URL or a remote repository.
     # If it's a remote repository, we don't want to normalize it. If it's a file:// URL, we do.
     if ':' not in repository:
-        return os.path.abspath(repository)
+        return (
+            os.path.abspath(os.path.join(base, repository)) if base else os.path.abspath(repository)
+        )
     elif repository.startswith('file://'):
-        return os.path.abspath(repository.partition('file://')[-1])
+        local_path = repository.partition('file://')[-1]
+        return (
+            os.path.abspath(os.path.join(base, local_path)) if base else os.path.abspath(local_path)
+        )
     else:
         return repository
 

borgmatic/execute.py

@@ -1,7 +1,6 @@
 import collections
 import enum
 import logging
-import os
 import select
 import subprocess
 import textwrap
@@ -243,6 +242,9 @@ def mask_command_secrets(full_command):
 MAX_LOGGED_COMMAND_LENGTH = 1000
 
 
+PREFIXES_OF_ENVIRONMENT_VARIABLES_TO_LOG = ('BORG_', 'PG', 'MARIADB_', 'MYSQL_')
+
+
 def log_command(full_command, input_file=None, output_file=None, environment=None):
     '''
     Log the given command (a sequence of command/argument strings), along with its input/output file
@@ -251,14 +253,21 @@ def log_command(full_command, input_file=None, output_file=None, environment=Non
     logger.debug(
         textwrap.shorten(
             ' '.join(
-                tuple(f'{key}=***' for key in (environment or {}).keys())
+                tuple(
+                    f'{key}=***'
+                    for key in (environment or {}).keys()
+                    if any(
+                        key.startswith(prefix)
+                        for prefix in PREFIXES_OF_ENVIRONMENT_VARIABLES_TO_LOG
+                    )
+                )
                 + mask_command_secrets(full_command)
             ),
             width=MAX_LOGGED_COMMAND_LENGTH,
             placeholder=' ...',
         )
-        + (f" < {getattr(input_file, 'name', '')}" if input_file else '')
-        + (f" > {getattr(output_file, 'name', '')}" if output_file else '')
+        + (f" < {getattr(input_file, 'name', input_file)}" if input_file else '')
+        + (f" > {getattr(output_file, 'name', output_file)}" if output_file else '')
     )
 
 
@@ -274,7 +283,7 @@ def execute_command(
     output_file=None,
     input_file=None,
     shell=False,
-    extra_environment=None,
+    environment=None,
     working_directory=None,
     borg_local_path=None,
     borg_exit_codes=None,
@@ -284,18 +293,17 @@ def execute_command(
     Execute the given command (a sequence of command/argument strings) and log its output at the
     given log level. If an open output file object is given, then write stdout to the file and only
     log stderr. If an open input file object is given, then read stdin from the file. If shell is
-    True, execute the command within a shell. If an extra environment dict is given, then use it to
-    augment the current environment, and pass the result into the command. If a working directory is
-    given, use that as the present working directory when running the command. If a Borg local path
-    is given, and the command matches it (regardless of arguments), treat exit code 1 as a warning
-    instead of an error. But if Borg exit codes are given as a sequence of exit code configuration
-    dicts, then use that configuration to decide what's an error and what's a warning. If run to
-    completion is False, then return the process for the command without executing it to completion.
+    True, execute the command within a shell. If an environment variables dict is given, then pass
+    it into the command. If a working directory is given, use that as the present working directory
+    when running the command. If a Borg local path is given, and the command matches it (regardless
+    of arguments), treat exit code 1 as a warning instead of an error. But if Borg exit codes are
+    given as a sequence of exit code configuration dicts, then use that configuration to decide
+    what's an error and what's a warning. If run to completion is False, then return the process for
+    the command without executing it to completion.
 
     Raise subprocesses.CalledProcessError if an error occurs while running the command.
     '''
-    log_command(full_command, input_file, output_file, extra_environment)
-    environment = {**os.environ, **extra_environment} if extra_environment else None
+    log_command(full_command, input_file, output_file, environment)
     do_not_capture = bool(output_file is DO_NOT_CAPTURE)
     command = ' '.join(full_command) if shell else full_command
 
@@ -307,8 +315,8 @@ def execute_command(
         shell=shell,
         env=environment,
         cwd=working_directory,
-        # Necessary for the passcommand credential hook to work.
-        close_fds=not bool((extra_environment or {}).get('BORG_PASSPHRASE_FD')),
+        # Necessary for passing credentials via anonymous pipe.
+        close_fds=False,
     )
     if not run_to_completion:
         return process
@@ -325,39 +333,40 @@ def execute_command(
 
 def execute_command_and_capture_output(
     full_command,
+    input_file=None,
     capture_stderr=False,
     shell=False,
-    extra_environment=None,
+    environment=None,
     working_directory=None,
     borg_local_path=None,
     borg_exit_codes=None,
 ):
     '''
     Execute the given command (a sequence of command/argument strings), capturing and returning its
-    output (stdout). If capture stderr is True, then capture and return stderr in addition to
-    stdout. If shell is True, execute the command within a shell. If an extra environment dict is
-    given, then use it to augment the current environment, and pass the result into the command. If
-    a working directory is given, use that as the present working directory when running the
-    command. If a Borg local path is given, and the command matches it (regardless of arguments),
-    treat exit code 1 as a warning instead of an error. But if Borg exit codes are given as a
-    sequence of exit code configuration dicts, then use that configuration to decide what's an error
-    and what's a warning.
+    output (stdout). If an input file descriptor is given, then pipe it to the command's stdin. If
+    capture stderr is True, then capture and return stderr in addition to stdout. If shell is True,
+    execute the command within a shell. If an environment variables dict is given, then pass it into
+    the command. If a working directory is given, use that as the present working directory when
+    running the command. If a Borg local path is given, and the command matches it (regardless of
+    arguments), treat exit code 1 as a warning instead of an error. But if Borg exit codes are given
+    as a sequence of exit code configuration dicts, then use that configuration to decide what's an
+    error and what's a warning.
 
     Raise subprocesses.CalledProcessError if an error occurs while running the command.
     '''
-    log_command(full_command, environment=extra_environment)
-    environment = {**os.environ, **extra_environment} if extra_environment else None
+    log_command(full_command, input_file, environment=environment)
     command = ' '.join(full_command) if shell else full_command
 
     try:
         output = subprocess.check_output(
             command,
+            stdin=input_file,
             stderr=subprocess.STDOUT if capture_stderr else None,
             shell=shell,
             env=environment,
             cwd=working_directory,
-            # Necessary for the passcommand credential hook to work.
-            close_fds=not bool((extra_environment or {}).get('BORG_PASSPHRASE_FD')),
+            # Necessary for passing credentials via anonymous pipe.
+            close_fds=False,
         )
     except subprocess.CalledProcessError as error:
         if (
@@ -377,7 +386,7 @@ def execute_command_with_processes(
     output_file=None,
     input_file=None,
     shell=False,
-    extra_environment=None,
+    environment=None,
     working_directory=None,
     borg_local_path=None,
     borg_exit_codes=None,
@@ -391,19 +400,17 @@ def execute_command_with_processes(
     If an open output file object is given, then write stdout to the file and only log stderr. But
     if output log level is None, instead suppress logging and return the captured output for (only)
     the given command. If an open input file object is given, then read stdin from the file. If
-    shell is True, execute the command within a shell. If an extra environment dict is given, then
-    use it to augment the current environment, and pass the result into the command. If a working
-    directory is given, use that as the present working directory when running the command. If a
-    Borg local path is given, then for any matching command or process (regardless of arguments),
-    treat exit code 1 as a warning instead of an error. But if Borg exit codes are given as a
-    sequence of exit code configuration dicts, then use that configuration to decide what's an error
-    and what's a warning.
+    shell is True, execute the command within a shell. If an environment variables dict is given,
+    then pass it into the command. If a working directory is given, use that as the present working
+    directory when running the command. If a Borg local path is given, then for any matching command
+    or process (regardless of arguments), treat exit code 1 as a warning instead of an error. But if
+    Borg exit codes are given as a sequence of exit code configuration dicts, then use that
+    configuration to decide what's an error and what's a warning.
 
     Raise subprocesses.CalledProcessError if an error occurs while running the command or in the
     upstream process.
     '''
-    log_command(full_command, input_file, output_file, extra_environment)
-    environment = {**os.environ, **extra_environment} if extra_environment else None
+    log_command(full_command, input_file, output_file, environment)
     do_not_capture = bool(output_file is DO_NOT_CAPTURE)
     command = ' '.join(full_command) if shell else full_command
 
@@ -418,8 +425,8 @@ def execute_command_with_processes(
             shell=shell,
             env=environment,
             cwd=working_directory,
-            # Necessary for the passcommand credential hook to work.
-            close_fds=not bool((extra_environment or {}).get('BORG_PASSPHRASE_FD')),
+            # Necessary for passing credentials via anonymous pipe.
+            close_fds=False,
         )
     except (subprocess.CalledProcessError, OSError):
         # Something has gone wrong. So vent each process' output buffer to prevent it from hanging.
@@ -430,13 +437,14 @@ def execute_command_with_processes(
                 process.kill()
         raise
 
-    captured_outputs = log_outputs(
-        tuple(processes) + (command_process,),
-        (input_file, output_file),
-        output_log_level,
-        borg_local_path,
-        borg_exit_codes,
-    )
+    with borgmatic.logger.Log_prefix(None):  # Log command output without any prefix.
+        captured_outputs = log_outputs(
+            tuple(processes) + (command_process,),
+            (input_file, output_file),
+            output_log_level,
+            borg_local_path,
+            borg_exit_codes,
+        )
 
     if output_log_level is None:
         return captured_outputs.get(command_process)

borgmatic/hooks/command.py

@@ -2,9 +2,11 @@ import logging
 import os
 import re
 import shlex
+import subprocess
 import sys
 
 import borgmatic.execute
+import borgmatic.logger
 
 logger = logging.getLogger(__name__)
 
@@ -30,66 +32,198 @@ def interpolate_context(hook_description, command, context):
 
 def make_environment(current_environment, sys_module=sys):
     '''
-    Given the existing system environment as a map from environment variable name to value, return
-    (in the same form) any extra environment variables that should be used when running command
-    hooks.
+    Given the existing system environment as a map from environment variable name to value, return a
+    copy of it, augmented with any extra environment variables that should be used when running
+    command hooks.
     '''
+    environment = dict(current_environment)
+
     # Detect whether we're running within a PyInstaller bundle. If so, set or clear LD_LIBRARY_PATH
     # based on the value of LD_LIBRARY_PATH_ORIG. This prevents library version information errors.
     if getattr(sys_module, 'frozen', False) and hasattr(sys_module, '_MEIPASS'):
-        return {'LD_LIBRARY_PATH': current_environment.get('LD_LIBRARY_PATH_ORIG', '')}
+        environment['LD_LIBRARY_PATH'] = environment.get('LD_LIBRARY_PATH_ORIG', '')
 
-    return {}
+    return environment
 
 
-def execute_hook(commands, umask, config_filename, description, dry_run, **context):
+def filter_hooks(command_hooks, before=None, after=None, hook_name=None, action_names=None):
     '''
-    Given a list of hook commands to execute, a umask to execute with (or None), a config filename,
-    a hook description, and whether this is a dry run, run the given commands. Or, don't run them
-    if this is a dry run.
+    Given a sequence of command hook dicts from configuration and one or more filters (before name,
+    after name, calling hook name, or a sequence of action names), filter down the command hooks to
+    just the ones that match the given filters.
+    '''
+    return tuple(
+        hook_config
+        for hook_config in command_hooks or ()
+        for config_action_names in (hook_config.get('when'),)
+        if before is None or hook_config.get('before') == before
+        if after is None or hook_config.get('after') == after
+        if action_names is None
+        or config_action_names is None
+        or set(config_action_names or ()).intersection(set(action_names))
+    )
+
+
+def execute_hooks(command_hooks, umask, working_directory, dry_run, **context):
+    '''
+    Given a sequence of command hook dicts from configuration, a umask to execute with (or None), a
+    working directory to execute with, and whether this is a dry run, run the commands for each
+    hook. Or don't run them if this is a dry run.
 
     The context contains optional values interpolated by name into the hook commands.
 
-    Raise ValueError if the umask cannot be parsed.
+    Raise ValueError if the umask cannot be parsed or a hook is invalid.
     Raise subprocesses.CalledProcessError if an error occurs in a hook.
     '''
-    if not commands:
-        logger.debug(f'No commands to run for {description} hook')
-        return
+    borgmatic.logger.add_custom_log_levels()
 
     dry_run_label = ' (dry run; not actually running hooks)' if dry_run else ''
 
-    context['configuration_filename'] = config_filename
-    commands = [interpolate_context(description, command, context) for command in commands]
+    for hook_config in command_hooks:
+        commands = hook_config.get('run')
 
-    if len(commands) == 1:
-        logger.info(f'Running command for {description} hook{dry_run_label}')
-    else:
-        logger.info(
-            f'Running {len(commands)} commands for {description} hook{dry_run_label}',
-        )
+        if 'before' in hook_config:
+            description = f'before {hook_config.get("before")}'
+        elif 'after' in hook_config:
+            description = f'after {hook_config.get("after")}'
+        else:
+            raise ValueError(f'Invalid hook configuration: {hook_config}')
 
-    if umask:
-        parsed_umask = int(str(umask), 8)
-        logger.debug(f'Set hook umask to {oct(parsed_umask)}')
-        original_umask = os.umask(parsed_umask)
-    else:
-        original_umask = None
+        if not commands:
+            logger.debug(f'No commands to run for {description} hook')
+            continue
 
-    try:
-        for command in commands:
-            if dry_run:
-                continue
+        commands = [interpolate_context(description, command, context) for command in commands]
 
-            borgmatic.execute.execute_command(
-                [command],
-                output_log_level=(logging.ERROR if description == 'on-error' else logging.WARNING),
-                shell=True,
-                extra_environment=make_environment(os.environ),
+        if len(commands) == 1:
+            logger.info(f'Running {description} command hook{dry_run_label}')
+        else:
+            logger.info(
+                f'Running {len(commands)} commands for {description} hook{dry_run_label}',
             )
-    finally:
-        if original_umask:
-            os.umask(original_umask)
+
+        if umask:
+            parsed_umask = int(str(umask), 8)
+            logger.debug(f'Setting hook umask to {oct(parsed_umask)}')
+            original_umask = os.umask(parsed_umask)
+        else:
+            original_umask = None
+
+        try:
+            for command in commands:
+                if dry_run:
+                    continue
+
+                borgmatic.execute.execute_command(
+                    [command],
+                    output_log_level=(
+                        logging.ERROR if hook_config.get('after') == 'error' else logging.ANSWER
+                    ),
+                    shell=True,
+                    environment=make_environment(os.environ),
+                    working_directory=working_directory,
+                )
+        finally:
+            if original_umask:
+                os.umask(original_umask)
+
+
+class Before_after_hooks:
+    '''
+    A Python context manager for executing command hooks both before and after the wrapped code.
+
+    Example use as a context manager:
+
+       with borgmatic.hooks.command.Before_after_hooks(
+           command_hooks=config.get('commands'),
+           before_after='do_stuff',
+           umask=config.get('umask'),
+           dry_run=dry_run,
+           hook_name='myhook',
+       ):
+            do()
+            some()
+            stuff()
+
+    With that context manager in place, "before" command hooks execute before the wrapped code runs,
+    and "after" command hooks execute after the wrapped code completes.
+    '''
+
+    def __init__(
+        self,
+        command_hooks,
+        before_after,
+        umask,
+        working_directory,
+        dry_run,
+        hook_name=None,
+        action_names=None,
+        **context,
+    ):
+        '''
+        Given a sequence of command hook configuration dicts, the before/after name, a umask to run
+        commands with, a working directory to run commands with, a dry run flag, the name of the
+        calling hook, a sequence of action names, and any context for the executed commands, save
+        those data points for use below.
+        '''
+        self.command_hooks = command_hooks
+        self.before_after = before_after
+        self.umask = umask
+        self.working_directory = working_directory
+        self.dry_run = dry_run
+        self.hook_name = hook_name
+        self.action_names = action_names
+        self.context = context
+
+    def __enter__(self):
+        '''
+        Run the configured "before" command hooks that match the initialized data points.
+        '''
+        try:
+            execute_hooks(
+                borgmatic.hooks.command.filter_hooks(
+                    self.command_hooks,
+                    before=self.before_after,
+                    hook_name=self.hook_name,
+                    action_names=self.action_names,
+                ),
+                self.umask,
+                self.working_directory,
+                self.dry_run,
+                **self.context,
+            )
+        except (OSError, subprocess.CalledProcessError) as error:
+            if considered_soft_failure(error):
+                return
+
+            # Trigger the after hook manually, since raising here will prevent it from being run
+            # otherwise.
+            self.__exit__(None, None, None)
+
+            raise ValueError(f'Error running before {self.before_after} hook: {error}')
+
+    def __exit__(self, exception_type, exception, traceback):
+        '''
+        Run the configured "after" command hooks that match the initialized data points.
+        '''
+        try:
+            execute_hooks(
+                borgmatic.hooks.command.filter_hooks(
+                    self.command_hooks,
+                    after=self.before_after,
+                    hook_name=self.hook_name,
+                    action_names=self.action_names,
+                ),
+                self.umask,
+                self.working_directory,
+                self.dry_run,
+                **self.context,
+            )
+        except (OSError, subprocess.CalledProcessError) as error:
+            if considered_soft_failure(error):
+                return
+
+            raise ValueError(f'Error running after {self.before_after} hook: {error}')
 
 
 def considered_soft_failure(error):

borgmatic/hooks/credential/container.py

@@ -0,0 +1,43 @@
+import logging
+import os
+import re
+
+logger = logging.getLogger(__name__)
+
+
+SECRET_NAME_PATTERN = re.compile(r'^\w+$')
+DEFAULT_SECRETS_DIRECTORY = '/run/secrets'
+
+
+def load_credential(hook_config, config, credential_parameters):
+    '''
+    Given the hook configuration dict, the configuration dict, and a credential parameters tuple
+    containing a secret name to load, read the secret from the corresponding container secrets file
+    and return it.
+
+    Raise ValueError if the credential parameters is not one element, the secret name is invalid, or
+    the secret file cannot be read.
+    '''
+    try:
+        (secret_name,) = credential_parameters
+    except ValueError:
+        name = ' '.join(credential_parameters)
+
+        raise ValueError(f'Cannot load invalid secret name: "{name}"')
+
+    if not SECRET_NAME_PATTERN.match(secret_name):
+        raise ValueError(f'Cannot load invalid secret name: "{secret_name}"')
+
+    try:
+        with open(
+            os.path.join(
+                config.get('working_directory', ''),
+                (hook_config or {}).get('secrets_directory', DEFAULT_SECRETS_DIRECTORY),
+                secret_name,
+            )
+        ) as secret_file:
+            return secret_file.read().rstrip(os.linesep)
+    except (FileNotFoundError, OSError) as error:
+        logger.warning(error)
+
+        raise ValueError(f'Cannot load secret "{secret_name}" from file: {error.filename}')

borgmatic/hooks/credential/file.py

@@ -0,0 +1,32 @@
+import logging
+import os
+
+logger = logging.getLogger(__name__)
+
+
+def load_credential(hook_config, config, credential_parameters):
+    '''
+    Given the hook configuration dict, the configuration dict, and a credential parameters tuple
+    containing a credential path to load, load the credential from file and return it.
+
+    Raise ValueError if the credential parameters is not one element or the secret file cannot be
+    read.
+    '''
+    try:
+        (credential_path,) = credential_parameters
+    except ValueError:
+        name = ' '.join(credential_parameters)
+
+        raise ValueError(f'Cannot load invalid credential: "{name}"')
+
+    expanded_credential_path = os.path.expanduser(credential_path)
+
+    try:
+        with open(
+            os.path.join(config.get('working_directory', ''), expanded_credential_path)
+        ) as credential_file:
+            return credential_file.read().rstrip(os.linesep)
+    except (FileNotFoundError, OSError) as error:
+        logger.warning(error)
+
+        raise ValueError(f'Cannot load credential file: {error.filename}')

borgmatic/hooks/credential/keepassxc.py

@@ -0,0 +1,44 @@
+import logging
+import os
+import shlex
+
+import borgmatic.execute
+
+logger = logging.getLogger(__name__)
+
+
+def load_credential(hook_config, config, credential_parameters):
+    '''
+    Given the hook configuration dict, the configuration dict, and a credential parameters tuple
+    containing a KeePassXC database path and an attribute name to load, run keepassxc-cli to fetch
+    the corresponidng KeePassXC credential and return it.
+
+    Raise ValueError if keepassxc-cli can't retrieve the credential.
+    '''
+    try:
+        (database_path, attribute_name) = credential_parameters
+    except ValueError:
+        path_and_name = ' '.join(credential_parameters)
+
+        raise ValueError(
+            f'Cannot load credential with invalid KeePassXC database path and attribute name: "{path_and_name}"'
+        )
+
+    expanded_database_path = os.path.expanduser(database_path)
+
+    if not os.path.exists(expanded_database_path):
+        raise ValueError(
+            f'Cannot load credential because KeePassXC database path does not exist: {database_path}'
+        )
+
+    return borgmatic.execute.execute_command_and_capture_output(
+        tuple(shlex.split((hook_config or {}).get('keepassxc_cli_command', 'keepassxc-cli')))
+        + (
+            'show',
+            '--show-protected',
+            '--attributes',
+            'Password',
+            expanded_database_path,
+            attribute_name,
+        )
+    ).rstrip(os.linesep)

borgmatic/hooks/credential/parse.py

@@ -1,42 +1,124 @@
 import functools
 import re
+import shlex
 
 import borgmatic.hooks.dispatch
 
 IS_A_HOOK = False
 
 
-CREDENTIAL_PATTERN = re.compile(
-    r'\{credential +(?P<hook_name>[A-Za-z0-9_]+) +(?P<credential_name>[A-Za-z0-9_]+)\}'
-)
-
-GENERAL_CREDENTIAL_PATTERN = re.compile(r'\{credential( +[^}]*)?\}')
-
-
-@functools.cache
-def resolve_credential(value):
+class Hash_adapter:
     '''
-    Given a configuration value containing a string like "{credential hookname credentialname}", resolve it by
-    calling the relevant hook to get the actual credential value. If the given value does not
-    actually contain a credential tag, then return it unchanged.
+    A Hash_adapter instance wraps an unhashable object and pretends it's hashable. This is intended
+    for passing to a @functools.cache-decorated function to prevent it from complaining that an
+    argument is unhashable. It should only be used for arguments that you don't want to actually
+    impact the cache hashing, because Hash_adapter doesn't actually hash the object's contents.
 
-    Cache the value so repeated calls to this function don't need to load the credential repeatedly.
+    Example usage:
+
+        @functools.cache
+        def func(a, b):
+            print(a, b.actual_value)
+            return a
+
+        func(5, Hash_adapter({1: 2, 3: 4}))  # Calls func(), prints, and returns.
+        func(5, Hash_adapter({1: 2, 3: 4}))  # Hits the cache and just returns the value.
+        func(5, Hash_adapter({5: 6, 7: 8}))  # Also uses cache, since the Hash_adapter is ignored.
+
+    In the above function, the "b" value is one that has been wrapped with Hash_adappter, and
+    therefore "b.actual_value" is necessary to access the original value.
+    '''
+
+    def __init__(self, actual_value):
+        self.actual_value = actual_value
+
+    def __eq__(self, other):
+        return True
+
+    def __hash__(self):
+        return 0
+
+
+UNHASHABLE_TYPES = (dict, list, set)
+
+
+def cache_ignoring_unhashable_arguments(function):
+    '''
+    A function decorator that caches calls to the decorated function but ignores any unhashable
+    arguments when performing cache lookups. This is intended to be a drop-in replacement for
+    functools.cache.
+
+    Example usage:
+
+        @cache_ignoring_unhashable_arguments
+        def func(a, b):
+            print(a, b)
+            return a
+
+        func(5, {1: 2, 3: 4})  # Calls func(), prints, and returns.
+        func(5, {1: 2, 3: 4})  # Hits the cache and just returns the value.
+        func(5, {5: 6, 7: 8})  # Also uses cache, since the unhashable value (the dict) is ignored.
+    '''
+
+    @functools.cache
+    def cached_function(*args, **kwargs):
+        return function(
+            *(arg.actual_value if isinstance(arg, Hash_adapter) else arg for arg in args),
+            **{
+                key: value.actual_value if isinstance(value, Hash_adapter) else value
+                for (key, value) in kwargs.items()
+            },
+        )
+
+    @functools.wraps(function)
+    def wrapper_function(*args, **kwargs):
+        return cached_function(
+            *(Hash_adapter(arg) if isinstance(arg, UNHASHABLE_TYPES) else arg for arg in args),
+            **{
+                key: Hash_adapter(value) if isinstance(value, UNHASHABLE_TYPES) else value
+                for (key, value) in kwargs.items()
+            },
+        )
+
+    wrapper_function.cache_clear = cached_function.cache_clear
+
+    return wrapper_function
+
+
+CREDENTIAL_PATTERN = re.compile(r'\{credential( +(?P<hook_and_parameters>.*))?\}')
+
+
+@cache_ignoring_unhashable_arguments
+def resolve_credential(value, config):
+    '''
+    Given a configuration value containing a string like "{credential hookname credentialname}" and
+    a configuration dict, resolve the credential by calling the relevant hook to get the actual
+    credential value. If the given value does not actually contain a credential tag, then return it
+    unchanged.
+
+    Cache the value (ignoring the config for purposes of caching), so repeated calls to this
+    function don't need to load the credential repeatedly.
 
     Raise ValueError if the config could not be parsed or the credential could not be loaded.
     '''
     if value is None:
         return value
 
-    result = CREDENTIAL_PATTERN.sub(
-        lambda matcher: borgmatic.hooks.dispatch.call_hook(
-            'load_credential', {}, matcher.group('hook_name'), matcher.group('credential_name')
-        ),
-        value,
-    )
+    matcher = CREDENTIAL_PATTERN.match(value)
 
-    # If we've tried to parse the credential, but the parsed result still looks kind of like a
-    # credential, it means it's invalid syntax.
-    if GENERAL_CREDENTIAL_PATTERN.match(result):
+    if not matcher:
+        return value
+
+    hook_and_parameters = matcher.group('hook_and_parameters')
+
+    if not hook_and_parameters:
         raise ValueError(f'Cannot load credential with invalid syntax "{value}"')
 
-    return result
+    (hook_name, *credential_parameters) = shlex.split(hook_and_parameters)
+
+    if not credential_parameters:
+        raise ValueError(f'Cannot load credential with invalid syntax "{value}"')
+
+    return borgmatic.hooks.dispatch.call_hook(
+        'load_credential', config, hook_name, tuple(credential_parameters)
+    )

borgmatic/hooks/credential/systemd.py

@@ -8,14 +8,22 @@ logger = logging.getLogger(__name__)
 CREDENTIAL_NAME_PATTERN = re.compile(r'^\w+$')
 
 
-def load_credential(hook_config, config, credential_name):
+def load_credential(hook_config, config, credential_parameters):
     '''
-    Given the hook configuration dict, the configuration dict, and a credential name to load, read
-    the credential from the corresponding systemd credential file and return it.
+    Given the hook configuration dict, the configuration dict, and a credential parameters tuple
+    containing a credential name to load, read the credential from the corresponding systemd
+    credential file and return it.
 
     Raise ValueError if the systemd CREDENTIALS_DIRECTORY environment variable is not set, the
     credential name is invalid, or the credential file cannot be read.
     '''
+    try:
+        (credential_name,) = credential_parameters
+    except ValueError:
+        name = ' '.join(credential_parameters)
+
+        raise ValueError(f'Cannot load invalid credential name: "{name}"')
+
     credentials_directory = os.environ.get('CREDENTIALS_DIRECTORY')
 
     if not credentials_directory:

borgmatic/hooks/data_source/bootstrap.py

@@ -55,9 +55,17 @@ def dump_data_sources(
             manifest_file,
         )
 
-    patterns.extend(borgmatic.borg.pattern.Pattern(config_path) for config_path in config_paths)
+    patterns.extend(
+        borgmatic.borg.pattern.Pattern(
+            config_path, source=borgmatic.borg.pattern.Pattern_source.HOOK
+        )
+        for config_path in config_paths
+    )
     patterns.append(
-        borgmatic.borg.pattern.Pattern(os.path.join(borgmatic_runtime_directory, 'bootstrap'))
+        borgmatic.borg.pattern.Pattern(
+            os.path.join(borgmatic_runtime_directory, 'bootstrap'),
+            source=borgmatic.borg.pattern.Pattern_source.HOOK,
+        )
     )
 
     return []

borgmatic/hooks/data_source/btrfs.py

@@ -48,13 +48,56 @@ def get_subvolume_mount_points(findmnt_command):
 Subvolume = collections.namedtuple('Subvolume', ('path', 'contained_patterns'), defaults=((),))
 
 
+def get_subvolume_property(btrfs_command, subvolume_path, property_name):
+    output = borgmatic.execute.execute_command_and_capture_output(
+        tuple(btrfs_command.split(' '))
+        + (
+            'property',
+            'get',
+            '-t',  # Type.
+            'subvol',
+            subvolume_path,
+            property_name,
+        ),
+    )
+
+    try:
+        value = output.strip().split('=')[1]
+    except IndexError:
+        raise ValueError(f'Invalid {btrfs_command} property output')
+
+    return {
+        'true': True,
+        'false': False,
+    }.get(value, value)
+
+
+def omit_read_only_subvolume_mount_points(btrfs_command, subvolume_paths):
+    '''
+    Given a Btrfs command to run and a sequence of Btrfs subvolume mount points, filter them down to
+    just those that are read-write. The idea is that Btrfs can't actually snapshot a read-only
+    subvolume, so we should just ignore them.
+    '''
+    retained_subvolume_paths = []
+
+    for subvolume_path in subvolume_paths:
+        if get_subvolume_property(btrfs_command, subvolume_path, 'ro'):
+            logger.debug(f'Ignoring Btrfs subvolume {subvolume_path} because it is read-only')
+        else:
+            retained_subvolume_paths.append(subvolume_path)
+
+    return tuple(retained_subvolume_paths)
+
+
 def get_subvolumes(btrfs_command, findmnt_command, patterns=None):
     '''
     Given a Btrfs command to run and a sequence of configured patterns, find the intersection
     between the current Btrfs filesystem and subvolume mount points and the paths of any patterns.
     The idea is that these pattern paths represent the requested subvolumes to snapshot.
 
-    If patterns is None, then return all subvolumes, sorted by path.
+    Only include subvolumes that contain at least one root pattern sourced from borgmatic
+    configuration (as opposed to generated elsewhere in borgmatic). But if patterns is None, then
+    return all subvolumes instead, sorted by path.
 
     Return the result as a sequence of matching subvolume mount points.
     '''
@@ -65,7 +108,11 @@ def get_subvolumes(btrfs_command, findmnt_command, patterns=None):
     # backup. Sort the subvolumes from longest to shortest mount points, so longer mount points get
     # a whack at the candidate pattern piñata before their parents do. (Patterns are consumed during
     # this process, so no two subvolumes end up with the same contained patterns.)
-    for mount_point in reversed(get_subvolume_mount_points(findmnt_command)):
+    for mount_point in reversed(
+        omit_read_only_subvolume_mount_points(
+            btrfs_command, get_subvolume_mount_points(findmnt_command)
+        )
+    ):
         subvolumes.extend(
             Subvolume(mount_point, contained_patterns)
             for contained_patterns in (
@@ -73,7 +120,12 @@ def get_subvolumes(btrfs_command, findmnt_command, patterns=None):
                     mount_point, candidate_patterns
                 ),
             )
-            if patterns is None or contained_patterns
+            if patterns is None
+            or any(
+                pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT
+                and pattern.source == borgmatic.borg.pattern.Pattern_source.CONFIG
+                for pattern in contained_patterns
+            )
         )
 
     return tuple(sorted(subvolumes, key=lambda subvolume: subvolume.path))
@@ -121,6 +173,7 @@ def make_snapshot_exclude_pattern(subvolume_path):  # pragma: no cover
         ),
         borgmatic.borg.pattern.Pattern_type.NO_RECURSE,
         borgmatic.borg.pattern.Pattern_style.FNMATCH,
+        source=borgmatic.borg.pattern.Pattern_source.HOOK,
     )
 
 
@@ -153,6 +206,7 @@ def make_borg_snapshot_pattern(subvolume_path, pattern):
         pattern.type,
         pattern.style,
         pattern.device,
+        source=borgmatic.borg.pattern.Pattern_source.HOOK,
     )
 
 
@@ -198,7 +252,8 @@ def dump_data_sources(
     dry_run_label = ' (dry run; not actually snapshotting anything)' if dry_run else ''
     logger.info(f'Snapshotting Btrfs subvolumes{dry_run_label}')
 
-    # Based on the configured patterns, determine Btrfs subvolumes to backup.
+    # Based on the configured patterns, determine Btrfs subvolumes to backup. Only consider those
+    # patterns that came from actual user configuration (as opposed to, say, other hooks).
     btrfs_command = hook_config.get('btrfs_command', 'btrfs')
     findmnt_command = hook_config.get('findmnt_command', 'findmnt')
     subvolumes = get_subvolumes(btrfs_command, findmnt_command, patterns)
@@ -299,9 +354,12 @@ def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, d
                 logger.debug(error)
                 return
 
-            # Strip off the subvolume path from the end of the snapshot path and then delete the
-            # resulting directory.
-            shutil.rmtree(snapshot_path.rsplit(subvolume.path, 1)[0])
+            # Remove the snapshot parent directory if it still exists. (It might not exist if the
+            # snapshot was for "/".)
+            snapshot_parent_dir = snapshot_path.rsplit(subvolume.path, 1)[0]
+
+            if os.path.isdir(snapshot_parent_dir):
+                shutil.rmtree(snapshot_parent_dir)
 
 
 def make_data_source_dump_patterns(

borgmatic/hooks/data_source/lvm.py

@@ -1,5 +1,6 @@
 import collections
 import glob
+import hashlib
 import json
 import logging
 import os
@@ -33,7 +34,9 @@ def get_logical_volumes(lsblk_command, patterns=None):
     between the current LVM logical volume mount points and the paths of any patterns. The idea is
     that these pattern paths represent the requested logical volumes to snapshot.
 
-    If patterns is None, include all logical volume mounts points, not just those in patterns.
+    Only include logical volumes that contain at least one root pattern sourced from borgmatic
+    configuration (as opposed to generated elsewhere in borgmatic). But if patterns is None, include
+    all logical volume mounts points instead, not just those in patterns.
 
     Return the result as a sequence of Logical_volume instances.
     '''
@@ -72,7 +75,12 @@ def get_logical_volumes(lsblk_command, patterns=None):
                     device['mountpoint'], candidate_patterns
                 ),
             )
-            if not patterns or contained_patterns
+            if not patterns
+            or any(
+                pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT
+                and pattern.source == borgmatic.borg.pattern.Pattern_source.CONFIG
+                for pattern in contained_patterns
+            )
         )
     except KeyError as error:
         raise ValueError(f'Invalid {lsblk_command} output: Missing key "{error}"')
@@ -124,10 +132,14 @@ def mount_snapshot(mount_command, snapshot_device, snapshot_mount_path):  # prag
     )
 
 
-def make_borg_snapshot_pattern(pattern, normalized_runtime_directory):
+MOUNT_POINT_HASH_LENGTH = 10
+
+
+def make_borg_snapshot_pattern(pattern, logical_volume, normalized_runtime_directory):
     '''
-    Given a Borg pattern as a borgmatic.borg.pattern.Pattern instance, return a new Pattern with its
-    path rewritten to be in a snapshot directory based on the given runtime directory.
+    Given a Borg pattern as a borgmatic.borg.pattern.Pattern instance and a Logical_volume
+    containing it, return a new Pattern with its path rewritten to be in a snapshot directory based
+    on both the given runtime directory and the given Logical_volume's mount point.
 
     Move any initial caret in a regular expression pattern path to the beginning, so as not to break
     the regular expression.
@@ -142,6 +154,13 @@ def make_borg_snapshot_pattern(pattern, normalized_runtime_directory):
     rewritten_path = initial_caret + os.path.join(
         normalized_runtime_directory,
         'lvm_snapshots',
+        # Including this hash prevents conflicts between snapshot patterns for different logical
+        # volumes. For instance, without this, snapshotting a logical volume at /var and another at
+        # /var/spool would result in overlapping snapshot patterns and therefore colliding mount
+        # attempts.
+        hashlib.shake_256(logical_volume.mount_point.encode('utf-8')).hexdigest(
+            MOUNT_POINT_HASH_LENGTH
+        ),
         '.',  # Borg 1.4+ "slashdot" hack.
         # Included so that the source directory ends up in the Borg archive at its "original" path.
         pattern.path.lstrip('^').lstrip(os.path.sep),
@@ -152,6 +171,7 @@ def make_borg_snapshot_pattern(pattern, normalized_runtime_directory):
         pattern.type,
         pattern.style,
         pattern.device,
+        source=borgmatic.borg.pattern.Pattern_source.HOOK,
     )
 
 
@@ -180,7 +200,8 @@ def dump_data_sources(
     dry_run_label = ' (dry run; not actually snapshotting anything)' if dry_run else ''
     logger.info(f'Snapshotting LVM logical volumes{dry_run_label}')
 
-    # List logical volumes to get their mount points.
+    # List logical volumes to get their mount points, but only consider those patterns that came
+    # from actual user configuration (as opposed to, say, other hooks).
     lsblk_command = hook_config.get('lsblk_command', 'lsblk')
     requested_logical_volumes = get_logical_volumes(lsblk_command, patterns)
 
@@ -218,6 +239,9 @@ def dump_data_sources(
         snapshot_mount_path = os.path.join(
             normalized_runtime_directory,
             'lvm_snapshots',
+            hashlib.shake_256(logical_volume.mount_point.encode('utf-8')).hexdigest(
+                MOUNT_POINT_HASH_LENGTH
+            ),
             logical_volume.mount_point.lstrip(os.path.sep),
         )
 
@@ -233,7 +257,9 @@ def dump_data_sources(
         )
 
         for pattern in logical_volume.contained_patterns:
-            snapshot_pattern = make_borg_snapshot_pattern(pattern, normalized_runtime_directory)
+            snapshot_pattern = make_borg_snapshot_pattern(
+                pattern, logical_volume, normalized_runtime_directory
+            )
 
             # Attempt to update the pattern in place, since pattern order matters to Borg.
             try:
@@ -337,6 +363,7 @@ def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, d
             os.path.normpath(borgmatic_runtime_directory),
         ),
         'lvm_snapshots',
+        '*',
     )
     logger.debug(f'Looking for snapshots to remove in {snapshots_glob}{dry_run_label}')
     umount_command = hook_config.get('umount_command', 'umount')
@@ -349,7 +376,10 @@ def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, d
             snapshot_mount_path = os.path.join(
                 snapshots_directory, logical_volume.mount_point.lstrip(os.path.sep)
             )
-            if not os.path.isdir(snapshot_mount_path):
+
+            # If the snapshot mount path is empty, this is probably just a "shadow" of a nested
+            # logical volume and therefore there's nothing to unmount.
+            if not os.path.isdir(snapshot_mount_path) or not os.listdir(snapshot_mount_path):
                 continue
 
             # This might fail if the directory is already mounted, but we swallow errors here since
@@ -374,7 +404,7 @@ def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, d
                 return
             except subprocess.CalledProcessError as error:
                 logger.debug(error)
-                return
+                continue
 
         if not dry_run:
             shutil.rmtree(snapshots_directory)

borgmatic/hooks/data_source/mariadb.py

@@ -1,6 +1,7 @@
 import copy
 import logging
 import os
+import re
 import shlex
 
 import borgmatic.borg.pattern
@@ -23,14 +24,92 @@ def make_dump_path(base_directory):  # pragma: no cover
     return dump.make_data_source_dump_path(base_directory, 'mariadb_databases')
 
 
-SYSTEM_DATABASE_NAMES = ('information_schema', 'mysql', 'performance_schema', 'sys')
+DEFAULTS_EXTRA_FILE_FLAG_PATTERN = re.compile('^--defaults-extra-file=(?P<filename>.*)$')
 
 
-def database_names_to_dump(database, extra_environment, dry_run):
+def parse_extra_options(extra_options):
     '''
-    Given a requested database config, return the corresponding sequence of database names to dump.
-    In the case of "all", query for the names of databases on the configured host and return them,
-    excluding any system databases that will cause problems during restore.
+    Given an extra options string, split the options into a tuple and return it. Additionally, if
+    the first option is "--defaults-extra-file=...", then remove it from the options and return the
+    filename.
+
+    So the return value is a tuple of: (parsed options, defaults extra filename).
+
+    The intent is to support downstream merging of multiple "--defaults-extra-file"s, as
+    MariaDB/MySQL only allows one at a time.
+    '''
+    split_extra_options = tuple(shlex.split(extra_options)) if extra_options else ()
+
+    if not split_extra_options:
+        return ((), None)
+
+    match = DEFAULTS_EXTRA_FILE_FLAG_PATTERN.match(split_extra_options[0])
+
+    if not match:
+        return (split_extra_options, None)
+
+    return (split_extra_options[1:], match.group('filename'))
+
+
+def make_defaults_file_options(username=None, password=None, defaults_extra_filename=None):
+    '''
+    Given a database username and/or password, write it to an anonymous pipe and return the flags
+    for passing that file descriptor to an executed command. The idea is that this is a more secure
+    way to transmit credentials to a database client than using an environment variable.
+
+    If no username or password are given, then return the options for the given defaults extra
+    filename (if any). But if there is a username and/or password and a defaults extra filename is
+    given, then "!include" it from the generated file, effectively allowing multiple defaults extra
+    files.
+
+    Do not use the returned value for multiple different command invocations. That will not work
+    because each pipe is "used up" once read.
+    '''
+    escaped_password = None if password is None else password.replace('\\', '\\\\')
+
+    values = '\n'.join(
+        (
+            (f'user={username}' if username is not None else ''),
+            (f'password="{escaped_password}"' if escaped_password is not None else ''),
+        )
+    ).strip()
+
+    if not values:
+        if defaults_extra_filename:
+            return (f'--defaults-extra-file={defaults_extra_filename}',)
+
+        return ()
+
+    fields_message = ' and '.join(
+        field_name
+        for field_name in (
+            (f'username ({username})' if username is not None else None),
+            ('password' if password is not None else None),
+        )
+        if field_name is not None
+    )
+    include_message = f' (including {defaults_extra_filename})' if defaults_extra_filename else ''
+    logger.debug(f'Writing database {fields_message} to defaults extra file pipe{include_message}')
+
+    include = f'!include {defaults_extra_filename}\n' if defaults_extra_filename else ''
+
+    read_file_descriptor, write_file_descriptor = os.pipe()
+    os.write(write_file_descriptor, f'{include}[client]\n{values}'.encode('utf-8'))
+    os.close(write_file_descriptor)
+
+    # This plus subprocess.Popen(..., close_fds=False) in execute.py is necessary for the database
+    # client child process to inherit the file descriptor.
+    os.set_inheritable(read_file_descriptor, True)
+
+    return (f'--defaults-extra-file=/dev/fd/{read_file_descriptor}',)
+
+
+def database_names_to_dump(database, config, username, password, environment, dry_run):
+    '''
+    Given a requested database config, a configuration dict, a database username and password, an
+    environment dict, and whether this is a dry run, return the corresponding sequence of database
+    names to dump. In the case of "all", query for the names of databases on the configured host and
+    return them, excluding any system databases that will cause problems during restore.
     '''
     if database['name'] != 'all':
         return (database['name'],)
@@ -40,24 +119,23 @@ def database_names_to_dump(database, extra_environment, dry_run):
     mariadb_show_command = tuple(
         shlex.quote(part) for part in shlex.split(database.get('mariadb_command') or 'mariadb')
     )
+    extra_options, defaults_extra_filename = parse_extra_options(database.get('list_options'))
     show_command = (
         mariadb_show_command
-        + (tuple(database['list_options'].split(' ')) if 'list_options' in database else ())
+        + make_defaults_file_options(username, password, defaults_extra_filename)
+        + extra_options
         + (('--host', database['hostname']) if 'hostname' in database else ())
         + (('--port', str(database['port'])) if 'port' in database else ())
         + (('--protocol', 'tcp') if 'hostname' in database or 'port' in database else ())
-        + (
-            ('--user', borgmatic.hooks.credential.parse.resolve_credential(database['username']))
-            if 'username' in database
-            else ()
-        )
+        + (('--ssl',) if database.get('tls') is True else ())
+        + (('--skip-ssl',) if database.get('tls') is False else ())
         + ('--skip-column-names', '--batch')
         + ('--execute', 'show schemas')
     )
+
     logger.debug('Querying for "all" MariaDB databases to dump')
-    show_output = execute_command_and_capture_output(
-        show_command, extra_environment=extra_environment
-    )
+
+    show_output = execute_command_and_capture_output(show_command, environment=environment)
 
     return tuple(
         show_name
@@ -66,8 +144,19 @@ def database_names_to_dump(database, extra_environment, dry_run):
     )
 
 
+SYSTEM_DATABASE_NAMES = ('information_schema', 'mysql', 'performance_schema', 'sys')
+
+
 def execute_dump_command(
-    database, dump_path, database_names, extra_environment, dry_run, dry_run_label
+    database,
+    config,
+    username,
+    password,
+    dump_path,
+    database_names,
+    environment,
+    dry_run,
+    dry_run_label,
 ):
     '''
     Kick off a dump for the given MariaDB database (provided as a configuration dict) to a named
@@ -94,18 +183,17 @@ def execute_dump_command(
         shlex.quote(part)
         for part in shlex.split(database.get('mariadb_dump_command') or 'mariadb-dump')
     )
+    extra_options, defaults_extra_filename = parse_extra_options(database.get('options'))
     dump_command = (
         mariadb_dump_command
-        + (tuple(database['options'].split(' ')) if 'options' in database else ())
+        + make_defaults_file_options(username, password, defaults_extra_filename)
+        + extra_options
         + (('--add-drop-database',) if database.get('add_drop_database', True) else ())
         + (('--host', database['hostname']) if 'hostname' in database else ())
         + (('--port', str(database['port'])) if 'port' in database else ())
         + (('--protocol', 'tcp') if 'hostname' in database or 'port' in database else ())
-        + (
-            ('--user', borgmatic.hooks.credential.parse.resolve_credential(database['username']))
-            if 'username' in database
-            else ()
-        )
+        + (('--ssl',) if database.get('tls') is True else ())
+        + (('--skip-ssl',) if database.get('tls') is False else ())
         + ('--databases',)
         + database_names
         + ('--result-file', dump_filename)
@@ -119,7 +207,7 @@ def execute_dump_command(
 
     return execute_command(
         dump_command,
-        extra_environment=extra_environment,
+        environment=environment,
         run_to_completion=False,
     )
 
@@ -161,12 +249,16 @@ def dump_data_sources(
 
     for database in databases:
         dump_path = make_dump_path(borgmatic_runtime_directory)
-        extra_environment = (
-            {'MYSQL_PWD': borgmatic.hooks.credential.parse.resolve_credential(database['password'])}
-            if 'password' in database
-            else None
+        username = borgmatic.hooks.credential.parse.resolve_credential(
+            database.get('username'), config
+        )
+        password = borgmatic.hooks.credential.parse.resolve_credential(
+            database.get('password'), config
+        )
+        environment = dict(os.environ)
+        dump_database_names = database_names_to_dump(
+            database, config, username, password, environment, dry_run
         )
-        dump_database_names = database_names_to_dump(database, extra_environment, dry_run)
 
         if not dump_database_names:
             if dry_run:
@@ -181,9 +273,12 @@ def dump_data_sources(
                 processes.append(
                     execute_dump_command(
                         renamed_database,
+                        config,
+                        username,
+                        password,
                         dump_path,
                         (dump_name,),
-                        extra_environment,
+                        environment,
                         dry_run,
                         dry_run_label,
                     )
@@ -192,9 +287,12 @@ def dump_data_sources(
             processes.append(
                 execute_dump_command(
                     database,
+                    config,
+                    username,
+                    password,
                     dump_path,
                     dump_database_names,
-                    extra_environment,
+                    environment,
                     dry_run,
                     dry_run_label,
                 )
@@ -203,7 +301,8 @@ def dump_data_sources(
     if not dry_run:
         patterns.append(
             borgmatic.borg.pattern.Pattern(
-                os.path.join(borgmatic_runtime_directory, 'mariadb_databases')
+                os.path.join(borgmatic_runtime_directory, 'mariadb_databases'),
+                source=borgmatic.borg.pattern.Pattern_source.HOOK,
             )
         )
 
@@ -264,32 +363,38 @@ def restore_data_source_dump(
     port = str(
         connection_params['port'] or data_source.get('restore_port', data_source.get('port', ''))
     )
+    tls = data_source.get('restore_tls', data_source.get('tls'))
     username = borgmatic.hooks.credential.parse.resolve_credential(
-        connection_params['username']
-        or data_source.get('restore_username', data_source.get('username'))
+        (
+            connection_params['username']
+            or data_source.get('restore_username', data_source.get('username'))
+        ),
+        config,
     )
     password = borgmatic.hooks.credential.parse.resolve_credential(
-        connection_params['password']
-        or data_source.get('restore_password', data_source.get('password'))
+        (
+            connection_params['password']
+            or data_source.get('restore_password', data_source.get('password'))
+        ),
+        config,
     )
 
     mariadb_restore_command = tuple(
         shlex.quote(part) for part in shlex.split(data_source.get('mariadb_command') or 'mariadb')
     )
+    extra_options, defaults_extra_filename = parse_extra_options(data_source.get('restore_options'))
     restore_command = (
         mariadb_restore_command
+        + make_defaults_file_options(username, password, defaults_extra_filename)
+        + extra_options
         + ('--batch',)
-        + (
-            tuple(data_source['restore_options'].split(' '))
-            if 'restore_options' in data_source
-            else ()
-        )
         + (('--host', hostname) if hostname else ())
         + (('--port', str(port)) if port else ())
         + (('--protocol', 'tcp') if hostname or port else ())
-        + (('--user', username) if username else ())
+        + (('--ssl',) if tls is True else ())
+        + (('--skip-ssl',) if tls is False else ())
     )
-    extra_environment = {'MYSQL_PWD': password} if password else None
+    environment = dict(os.environ)
 
     logger.debug(f"Restoring MariaDB database {data_source['name']}{dry_run_label}")
     if dry_run:
@@ -302,5 +407,5 @@ def restore_data_source_dump(
         [extract_process],
         output_log_level=logging.DEBUG,
         input_file=extract_process.stdout,
-        extra_environment=extra_environment,
+        environment=environment,
     )

borgmatic/hooks/data_source/mongodb.py

@@ -53,6 +53,7 @@ def dump_data_sources(
     logger.info(f'Dumping MongoDB databases{dry_run_label}')
 
     processes = []
+
     for database in databases:
         name = database['name']
         dump_filename = dump.make_data_source_dump_filename(
@@ -69,7 +70,7 @@ def dump_data_sources(
         if dry_run:
             continue
 
-        command = build_dump_command(database, dump_filename, dump_format)
+        command = build_dump_command(database, config, dump_filename, dump_format)
 
         if dump_format == 'directory':
             dump.create_parent_directory_for_dump(dump_filename)
@@ -81,21 +82,49 @@ def dump_data_sources(
     if not dry_run:
         patterns.append(
             borgmatic.borg.pattern.Pattern(
-                os.path.join(borgmatic_runtime_directory, 'mongodb_databases')
+                os.path.join(borgmatic_runtime_directory, 'mongodb_databases'),
+                source=borgmatic.borg.pattern.Pattern_source.HOOK,
             )
         )
 
     return processes
 
 
-def build_dump_command(database, dump_filename, dump_format):
+def make_password_config_file(password):
     '''
-    Return the mongodump command from a single database configuration.
+    Given a database password, write it as a MongoDB configuration file to an anonymous pipe and
+    return its filename. The idea is that this is a more secure way to transmit a password to
+    MongoDB than providing it directly on the command-line.
+
+    Do not use the returned value for multiple different command invocations. That will not work
+    because each pipe is "used up" once read.
+    '''
+    logger.debug('Writing MongoDB password to configuration file pipe')
+
+    read_file_descriptor, write_file_descriptor = os.pipe()
+    os.write(write_file_descriptor, f'password: {password}'.encode('utf-8'))
+    os.close(write_file_descriptor)
+
+    # This plus subprocess.Popen(..., close_fds=False) in execute.py is necessary for the database
+    # client child process to inherit the file descriptor.
+    os.set_inheritable(read_file_descriptor, True)
+
+    return f'/dev/fd/{read_file_descriptor}'
+
+
+def build_dump_command(database, config, dump_filename, dump_format):
+    '''
+    Return the custom mongodump_command from a single database configuration.
     '''
     all_databases = database['name'] == 'all'
 
+    password = borgmatic.hooks.credential.parse.resolve_credential(database.get('password'), config)
+
+    dump_command = tuple(
+        shlex.quote(part) for part in shlex.split(database.get('mongodump_command') or 'mongodump')
+    )
     return (
-        ('mongodump',)
+        dump_command
         + (('--out', shlex.quote(dump_filename)) if dump_format == 'directory' else ())
         + (('--host', shlex.quote(database['hostname'])) if 'hostname' in database else ())
         + (('--port', shlex.quote(str(database['port']))) if 'port' in database else ())
@@ -103,22 +132,15 @@ def build_dump_command(database, dump_filename, dump_format):
             (
                 '--username',
                 shlex.quote(
-                    borgmatic.hooks.credential.parse.resolve_credential(database['username'])
+                    borgmatic.hooks.credential.parse.resolve_credential(
+                        database['username'], config
+                    )
                 ),
             )
             if 'username' in database
             else ()
         )
-        + (
-            (
-                '--password',
-                shlex.quote(
-                    borgmatic.hooks.credential.parse.resolve_credential(database['password'])
-                ),
-            )
-            if 'password' in database
-            else ()
-        )
+        + (('--config', make_password_config_file(password)) if password else ())
         + (
             ('--authenticationDatabase', shlex.quote(database['authentication_database']))
             if 'authentication_database' in database
@@ -192,7 +214,7 @@ def restore_data_source_dump(
         data_source.get('hostname'),
     )
     restore_command = build_restore_command(
-        extract_process, data_source, dump_filename, connection_params
+        extract_process, data_source, config, dump_filename, connection_params
     )
 
     logger.debug(f"Restoring MongoDB database {data_source['name']}{dry_run_label}")
@@ -209,22 +231,33 @@ def restore_data_source_dump(
     )
 
 
-def build_restore_command(extract_process, database, dump_filename, connection_params):
+def build_restore_command(extract_process, database, config, dump_filename, connection_params):
     '''
-    Return the mongorestore command from a single database configuration.
+    Return the custom mongorestore_command from a single database configuration.
     '''
     hostname = connection_params['hostname'] or database.get(
         'restore_hostname', database.get('hostname')
     )
     port = str(connection_params['port'] or database.get('restore_port', database.get('port', '')))
     username = borgmatic.hooks.credential.parse.resolve_credential(
-        connection_params['username'] or database.get('restore_username', database.get('username'))
+        (
+            connection_params['username']
+            or database.get('restore_username', database.get('username'))
+        ),
+        config,
     )
     password = borgmatic.hooks.credential.parse.resolve_credential(
-        connection_params['password'] or database.get('restore_password', database.get('password'))
+        (
+            connection_params['password']
+            or database.get('restore_password', database.get('password'))
+        ),
+        config,
     )
 
-    command = ['mongorestore']
+    command = list(
+        shlex.quote(part)
+        for part in shlex.split(database.get('mongorestore_command') or 'mongorestore')
+    )
     if extract_process:
         command.append('--archive')
     else:
@@ -238,7 +271,7 @@ def build_restore_command(extract_process, database, dump_filename, connection_p
     if username:
         command.extend(('--username', username))
     if password:
-        command.extend(('--password', password))
+        command.extend(('--config', make_password_config_file(password)))
     if 'authentication_database' in database:
         command.extend(('--authenticationDatabase', database['authentication_database']))
     if 'restore_options' in database:

borgmatic/hooks/data_source/mysql.py

@@ -6,6 +6,7 @@ import shlex
 import borgmatic.borg.pattern
 import borgmatic.config.paths
 import borgmatic.hooks.credential.parse
+import borgmatic.hooks.data_source.mariadb
 from borgmatic.execute import (
     execute_command,
     execute_command_and_capture_output,
@@ -26,11 +27,12 @@ def make_dump_path(base_directory):  # pragma: no cover
 SYSTEM_DATABASE_NAMES = ('information_schema', 'mysql', 'performance_schema', 'sys')
 
 
-def database_names_to_dump(database, extra_environment, dry_run):
+def database_names_to_dump(database, config, username, password, environment, dry_run):
     '''
-    Given a requested database config, return the corresponding sequence of database names to dump.
-    In the case of "all", query for the names of databases on the configured host and return them,
-    excluding any system databases that will cause problems during restore.
+    Given a requested database config, a configuration dict, a database username and password, an
+    environment dict, and whether this is a dry run, return the corresponding sequence of database
+    names to dump. In the case of "all", query for the names of databases on the configured host and
+    return them, excluding any system databases that will cause problems during restore.
     '''
     if database['name'] != 'all':
         return (database['name'],)
@@ -40,24 +42,27 @@ def database_names_to_dump(database, extra_environment, dry_run):
     mysql_show_command = tuple(
         shlex.quote(part) for part in shlex.split(database.get('mysql_command') or 'mysql')
     )
+    extra_options, defaults_extra_filename = (
+        borgmatic.hooks.data_source.mariadb.parse_extra_options(database.get('list_options'))
+    )
     show_command = (
         mysql_show_command
-        + (tuple(database['list_options'].split(' ')) if 'list_options' in database else ())
+        + borgmatic.hooks.data_source.mariadb.make_defaults_file_options(
+            username, password, defaults_extra_filename
+        )
+        + extra_options
         + (('--host', database['hostname']) if 'hostname' in database else ())
         + (('--port', str(database['port'])) if 'port' in database else ())
         + (('--protocol', 'tcp') if 'hostname' in database or 'port' in database else ())
-        + (
-            ('--user', borgmatic.hooks.credential.parse.resolve_credential(database['username']))
-            if 'username' in database
-            else ()
-        )
+        + (('--ssl',) if database.get('tls') is True else ())
+        + (('--skip-ssl',) if database.get('tls') is False else ())
         + ('--skip-column-names', '--batch')
         + ('--execute', 'show schemas')
     )
+
     logger.debug('Querying for "all" MySQL databases to dump')
-    show_output = execute_command_and_capture_output(
-        show_command, extra_environment=extra_environment
-    )
+
+    show_output = execute_command_and_capture_output(show_command, environment=environment)
 
     return tuple(
         show_name
@@ -67,7 +72,15 @@ def database_names_to_dump(database, extra_environment, dry_run):
 
 
 def execute_dump_command(
-    database, dump_path, database_names, extra_environment, dry_run, dry_run_label
+    database,
+    config,
+    username,
+    password,
+    dump_path,
+    database_names,
+    environment,
+    dry_run,
+    dry_run_label,
 ):
     '''
     Kick off a dump for the given MySQL/MariaDB database (provided as a configuration dict) to a
@@ -93,18 +106,21 @@ def execute_dump_command(
     mysql_dump_command = tuple(
         shlex.quote(part) for part in shlex.split(database.get('mysql_dump_command') or 'mysqldump')
     )
+    extra_options, defaults_extra_filename = (
+        borgmatic.hooks.data_source.mariadb.parse_extra_options(database.get('options'))
+    )
     dump_command = (
         mysql_dump_command
-        + (tuple(database['options'].split(' ')) if 'options' in database else ())
+        + borgmatic.hooks.data_source.mariadb.make_defaults_file_options(
+            username, password, defaults_extra_filename
+        )
+        + extra_options
         + (('--add-drop-database',) if database.get('add_drop_database', True) else ())
         + (('--host', database['hostname']) if 'hostname' in database else ())
         + (('--port', str(database['port'])) if 'port' in database else ())
         + (('--protocol', 'tcp') if 'hostname' in database or 'port' in database else ())
-        + (
-            ('--user', borgmatic.hooks.credential.parse.resolve_credential(database['username']))
-            if 'username' in database
-            else ()
-        )
+        + (('--ssl',) if database.get('tls') is True else ())
+        + (('--skip-ssl',) if database.get('tls') is False else ())
         + ('--databases',)
         + database_names
         + ('--result-file', dump_filename)
@@ -118,7 +134,7 @@ def execute_dump_command(
 
     return execute_command(
         dump_command,
-        extra_environment=extra_environment,
+        environment=environment,
         run_to_completion=False,
     )
 
@@ -160,12 +176,16 @@ def dump_data_sources(
 
     for database in databases:
         dump_path = make_dump_path(borgmatic_runtime_directory)
-        extra_environment = (
-            {'MYSQL_PWD': borgmatic.hooks.credential.parse.resolve_credential(database['password'])}
-            if 'password' in database
-            else None
+        username = borgmatic.hooks.credential.parse.resolve_credential(
+            database.get('username'), config
+        )
+        password = borgmatic.hooks.credential.parse.resolve_credential(
+            database.get('password'), config
+        )
+        environment = dict(os.environ)
+        dump_database_names = database_names_to_dump(
+            database, config, username, password, environment, dry_run
         )
-        dump_database_names = database_names_to_dump(database, extra_environment, dry_run)
 
         if not dump_database_names:
             if dry_run:
@@ -180,9 +200,12 @@ def dump_data_sources(
                 processes.append(
                     execute_dump_command(
                         renamed_database,
+                        config,
+                        username,
+                        password,
                         dump_path,
                         (dump_name,),
-                        extra_environment,
+                        environment,
                         dry_run,
                         dry_run_label,
                     )
@@ -191,9 +214,12 @@ def dump_data_sources(
             processes.append(
                 execute_dump_command(
                     database,
+                    config,
+                    username,
+                    password,
                     dump_path,
                     dump_database_names,
-                    extra_environment,
+                    environment,
                     dry_run,
                     dry_run_label,
                 )
@@ -202,7 +228,8 @@ def dump_data_sources(
     if not dry_run:
         patterns.append(
             borgmatic.borg.pattern.Pattern(
-                os.path.join(borgmatic_runtime_directory, 'mysql_databases')
+                os.path.join(borgmatic_runtime_directory, 'mysql_databases'),
+                source=borgmatic.borg.pattern.Pattern_source.HOOK,
             )
         )
 
@@ -263,32 +290,42 @@ def restore_data_source_dump(
     port = str(
         connection_params['port'] or data_source.get('restore_port', data_source.get('port', ''))
     )
+    tls = data_source.get('restore_tls', data_source.get('tls'))
     username = borgmatic.hooks.credential.parse.resolve_credential(
-        connection_params['username']
-        or data_source.get('restore_username', data_source.get('username'))
+        (
+            connection_params['username']
+            or data_source.get('restore_username', data_source.get('username'))
+        ),
+        config,
     )
     password = borgmatic.hooks.credential.parse.resolve_credential(
-        connection_params['password']
-        or data_source.get('restore_password', data_source.get('password'))
+        (
+            connection_params['password']
+            or data_source.get('restore_password', data_source.get('password'))
+        ),
+        config,
     )
 
     mysql_restore_command = tuple(
         shlex.quote(part) for part in shlex.split(data_source.get('mysql_command') or 'mysql')
     )
+    extra_options, defaults_extra_filename = (
+        borgmatic.hooks.data_source.mariadb.parse_extra_options(data_source.get('restore_options'))
+    )
     restore_command = (
         mysql_restore_command
-        + ('--batch',)
-        + (
-            tuple(data_source['restore_options'].split(' '))
-            if 'restore_options' in data_source
-            else ()
+        + borgmatic.hooks.data_source.mariadb.make_defaults_file_options(
+            username, password, defaults_extra_filename
         )
+        + extra_options
+        + ('--batch',)
         + (('--host', hostname) if hostname else ())
         + (('--port', str(port)) if port else ())
         + (('--protocol', 'tcp') if hostname or port else ())
-        + (('--user', username) if username else ())
+        + (('--ssl',) if tls is True else ())
+        + (('--skip-ssl',) if tls is False else ())
     )
-    extra_environment = {'MYSQL_PWD': password} if password else None
+    environment = dict(os.environ)
 
     logger.debug(f"Restoring MySQL database {data_source['name']}{dry_run_label}")
     if dry_run:
@@ -301,5 +338,5 @@ def restore_data_source_dump(
         [extract_process],
         output_log_level=logging.DEBUG,
         input_file=extract_process.stdout,
-        extra_environment=extra_environment,
+        environment=environment,
     )

borgmatic/hooks/data_source/postgresql.py

@@ -25,49 +25,52 @@ def make_dump_path(base_directory):  # pragma: no cover
     return dump.make_data_source_dump_path(base_directory, 'postgresql_databases')
 
 
-def make_extra_environment(database, restore_connection_params=None):
+def make_environment(database, config, restore_connection_params=None):
     '''
-    Make the extra_environment dict from the given database configuration. If restore connection
-    params are given, this is for a restore operation.
+    Make an environment dict from the current environment variables and the given database
+    configuration. If restore connection params are given, this is for a restore operation.
     '''
-    extra = dict()
+    environment = dict(os.environ)
 
     try:
         if restore_connection_params:
-            extra['PGPASSWORD'] = borgmatic.hooks.credential.parse.resolve_credential(
-                restore_connection_params.get('password')
-                or database.get('restore_password', database['password'])
+            environment['PGPASSWORD'] = borgmatic.hooks.credential.parse.resolve_credential(
+                (
+                    restore_connection_params.get('password')
+                    or database.get('restore_password', database['password'])
+                ),
+                config,
             )
         else:
-            extra['PGPASSWORD'] = borgmatic.hooks.credential.parse.resolve_credential(
-                database['password']
+            environment['PGPASSWORD'] = borgmatic.hooks.credential.parse.resolve_credential(
+                database['password'], config
             )
     except (AttributeError, KeyError):
         pass
 
     if 'ssl_mode' in database:
-        extra['PGSSLMODE'] = database['ssl_mode']
+        environment['PGSSLMODE'] = database['ssl_mode']
     if 'ssl_cert' in database:
-        extra['PGSSLCERT'] = database['ssl_cert']
+        environment['PGSSLCERT'] = database['ssl_cert']
     if 'ssl_key' in database:
-        extra['PGSSLKEY'] = database['ssl_key']
+        environment['PGSSLKEY'] = database['ssl_key']
     if 'ssl_root_cert' in database:
-        extra['PGSSLROOTCERT'] = database['ssl_root_cert']
+        environment['PGSSLROOTCERT'] = database['ssl_root_cert']
     if 'ssl_crl' in database:
-        extra['PGSSLCRL'] = database['ssl_crl']
+        environment['PGSSLCRL'] = database['ssl_crl']
 
-    return extra
+    return environment
 
 
 EXCLUDED_DATABASE_NAMES = ('template0', 'template1')
 
 
-def database_names_to_dump(database, extra_environment, dry_run):
+def database_names_to_dump(database, config, environment, dry_run):
     '''
-    Given a requested database config, return the corresponding sequence of database names to dump.
-    In the case of "all" when a database format is given, query for the names of databases on the
-    configured host and return them. For "all" without a database format, just return a sequence
-    containing "all".
+    Given a requested database config and a configuration dict, return the corresponding sequence of
+    database names to dump. In the case of "all" when a database format is given, query for the
+    names of databases on the configured host and return them. For "all" without a database format,
+    just return a sequence containing "all".
     '''
     requested_name = database['name']
 
@@ -89,7 +92,7 @@ def database_names_to_dump(database, extra_environment, dry_run):
         + (
             (
                 '--username',
-                borgmatic.hooks.credential.parse.resolve_credential(database['username']),
+                borgmatic.hooks.credential.parse.resolve_credential(database['username'], config),
             )
             if 'username' in database
             else ()
@@ -97,9 +100,7 @@ def database_names_to_dump(database, extra_environment, dry_run):
         + (tuple(database['list_options'].split(' ')) if 'list_options' in database else ())
     )
     logger.debug('Querying for "all" PostgreSQL databases to dump')
-    list_output = execute_command_and_capture_output(
-        list_command, extra_environment=extra_environment
-    )
+    list_output = execute_command_and_capture_output(list_command, environment=environment)
 
     return tuple(
         row[0]
@@ -146,9 +147,9 @@ def dump_data_sources(
     logger.info(f'Dumping PostgreSQL databases{dry_run_label}')
 
     for database in databases:
-        extra_environment = make_extra_environment(database)
+        environment = make_environment(database, config)
         dump_path = make_dump_path(borgmatic_runtime_directory)
-        dump_database_names = database_names_to_dump(database, extra_environment, dry_run)
+        dump_database_names = database_names_to_dump(database, config, environment, dry_run)
 
         if not dump_database_names:
             if dry_run:
@@ -158,6 +159,7 @@ def dump_data_sources(
 
         for database_name in dump_database_names:
             dump_format = database.get('format', None if database_name == 'all' else 'custom')
+            compression = database.get('compression')
             default_dump_command = 'pg_dumpall' if database_name == 'all' else 'pg_dump'
             dump_command = tuple(
                 shlex.quote(part)
@@ -189,7 +191,7 @@ def dump_data_sources(
                         '--username',
                         shlex.quote(
                             borgmatic.hooks.credential.parse.resolve_credential(
-                                database['username']
+                                database['username'], config
                             )
                         ),
                     )
@@ -198,6 +200,7 @@ def dump_data_sources(
                 )
                 + (('--no-owner',) if database.get('no_owner', False) else ())
                 + (('--format', shlex.quote(dump_format)) if dump_format else ())
+                + (('--compress', shlex.quote(str(compression))) if compression is not None else ())
                 + (('--file', shlex.quote(dump_filename)) if dump_format == 'directory' else ())
                 + (
                     tuple(shlex.quote(option) for option in database['options'].split(' '))
@@ -222,7 +225,7 @@ def dump_data_sources(
                 execute_command(
                     command,
                     shell=True,
-                    extra_environment=extra_environment,
+                    environment=environment,
                 )
             else:
                 dump.create_named_pipe_for_dump(dump_filename)
@@ -230,7 +233,7 @@ def dump_data_sources(
                     execute_command(
                         command,
                         shell=True,
-                        extra_environment=extra_environment,
+                        environment=environment,
                         run_to_completion=False,
                     )
                 )
@@ -238,7 +241,8 @@ def dump_data_sources(
     if not dry_run:
         patterns.append(
             borgmatic.borg.pattern.Pattern(
-                os.path.join(borgmatic_runtime_directory, 'postgresql_databases')
+                os.path.join(borgmatic_runtime_directory, 'postgresql_databases'),
+                source=borgmatic.borg.pattern.Pattern_source.HOOK,
             )
         )
 
@@ -309,8 +313,11 @@ def restore_data_source_dump(
         connection_params['port'] or data_source.get('restore_port', data_source.get('port', ''))
     )
     username = borgmatic.hooks.credential.parse.resolve_credential(
-        connection_params['username']
-        or data_source.get('restore_username', data_source.get('username'))
+        (
+            connection_params['username']
+            or data_source.get('restore_username', data_source.get('username'))
+        ),
+        config,
     )
 
     all_databases = bool(data_source['name'] == 'all')
@@ -363,9 +370,7 @@ def restore_data_source_dump(
         )
     )
 
-    extra_environment = make_extra_environment(
-        data_source, restore_connection_params=connection_params
-    )
+    environment = make_environment(data_source, config, restore_connection_params=connection_params)
 
     logger.debug(f"Restoring PostgreSQL database {data_source['name']}{dry_run_label}")
     if dry_run:
@@ -378,6 +383,6 @@ def restore_data_source_dump(
         [extract_process] if extract_process else [],
         output_log_level=logging.DEBUG,
         input_file=extract_process.stdout if extract_process else None,
-        extra_environment=extra_environment,
+        environment=environment,
     )
-    execute_command(analyze_command, extra_environment=extra_environment)
+    execute_command(analyze_command, environment=environment)

borgmatic/hooks/data_source/snapshot.py

@@ -11,10 +11,10 @@ def get_contained_patterns(parent_directory, candidate_patterns):
     paths, but there's a parent directory (logical volume, dataset, subvolume, etc.) at /var, then
     /var is what we want to snapshot.
 
-    For this to work, a candidate pattern path can't have any globs or other non-literal characters
-    in the initial portion of the path that matches the parent directory. For instance, a parent
-    directory of /var would match a candidate pattern path of /var/log/*/data, but not a pattern
-    path like /v*/log/*/data.
+    For this function to work, a candidate pattern path can't have any globs or other non-literal
+    characters in the initial portion of the path that matches the parent directory. For instance, a
+    parent directory of /var would match a candidate pattern path of /var/log/*/data, but not a
+    pattern path like /v*/log/*/data.
 
     The one exception is that if a regular expression pattern path starts with "^", that will get
     stripped off for purposes of matching against a parent directory.
@@ -31,8 +31,10 @@ def get_contained_patterns(parent_directory, candidate_patterns):
         candidate
         for candidate in candidate_patterns
         for candidate_path in (pathlib.PurePath(candidate.path.lstrip('^')),)
-        if pathlib.PurePath(parent_directory) == candidate_path
-        or pathlib.PurePath(parent_directory) in candidate_path.parents
+        if (
+            pathlib.PurePath(parent_directory) == candidate_path
+            or pathlib.PurePath(parent_directory) in candidate_path.parents
+        )
     )
     candidate_patterns -= set(contained_patterns)
 

borgmatic/hooks/data_source/sqlite.py

@@ -71,13 +71,16 @@ def dump_data_sources(
             )
             continue
 
-        command = (
-            'sqlite3',
+        sqlite_command = tuple(
+            shlex.quote(part) for part in shlex.split(database.get('sqlite_command') or 'sqlite3')
+        )
+        command = sqlite_command + (
             shlex.quote(database_path),
             '.dump',
             '>',
             shlex.quote(dump_filename),
         )
+
         logger.debug(
             f'Dumping SQLite database at {database_path} to {dump_filename}{dry_run_label}'
         )
@@ -90,7 +93,8 @@ def dump_data_sources(
     if not dry_run:
         patterns.append(
             borgmatic.borg.pattern.Pattern(
-                os.path.join(borgmatic_runtime_directory, 'sqlite_databases')
+                os.path.join(borgmatic_runtime_directory, 'sqlite_databases'),
+                source=borgmatic.borg.pattern.Pattern_source.HOOK,
             )
         )
 
@@ -159,11 +163,11 @@ def restore_data_source_dump(
     except FileNotFoundError:  # pragma: no cover
         pass
 
-    restore_command = (
-        'sqlite3',
-        database_path,
+    sqlite_restore_command = tuple(
+        shlex.quote(part)
+        for part in shlex.split(data_source.get('sqlite_restore_command') or 'sqlite3')
     )
-
+    restore_command = sqlite_restore_command + (shlex.quote(database_path),)
     # Don't give Borg local path so as to error on warnings, as "borg extract" only gives a warning
     # if the restore paths don't exist in the archive.
     execute_command_with_processes(

borgmatic/hooks/data_source/zfs.py

@@ -1,5 +1,6 @@
 import collections
 import glob
+import hashlib
 import logging
 import os
 import shutil
@@ -38,6 +39,9 @@ def get_datasets_to_backup(zfs_command, patterns):
     pattern paths represent the requested datasets to snapshot. But also include any datasets tagged
     with a borgmatic-specific user property, whether or not they appear in the patterns.
 
+    Only include datasets that contain at least one root pattern sourced from borgmatic
+    configuration (as opposed to generated elsewhere in borgmatic).
+
     Return the result as a sequence of Dataset instances, sorted by mount point.
     '''
     list_output = borgmatic.execute.execute_command_and_capture_output(
@@ -48,7 +52,7 @@ def get_datasets_to_backup(zfs_command, patterns):
             '-t',
             'filesystem',
             '-o',
-            f'name,mountpoint,{BORGMATIC_USER_PROPERTY}',
+            f'name,mountpoint,canmount,{BORGMATIC_USER_PROPERTY}',
         )
     )
 
@@ -60,7 +64,12 @@ def get_datasets_to_backup(zfs_command, patterns):
             (
                 Dataset(dataset_name, mount_point, (user_property_value == 'auto'), ())
                 for line in list_output.splitlines()
-                for (dataset_name, mount_point, user_property_value) in (line.rstrip().split('\t'),)
+                for (dataset_name, mount_point, can_mount, user_property_value) in (
+                    line.rstrip().split('\t'),
+                )
+                # Skip datasets that are marked "canmount=off", because mounting their snapshots will
+                # result in completely empty mount points—thereby preventing us from backing them up.
+                if can_mount == 'on'
             ),
             key=lambda dataset: dataset.mount_point,
             reverse=True,
@@ -83,7 +92,12 @@ def get_datasets_to_backup(zfs_command, patterns):
                 for contained_patterns in (
                     (
                         (
-                            (borgmatic.borg.pattern.Pattern(dataset.mount_point),)
+                            (
+                                borgmatic.borg.pattern.Pattern(
+                                    dataset.mount_point,
+                                    source=borgmatic.borg.pattern.Pattern_source.HOOK,
+                                ),
+                            )
                             if dataset.auto_backup
                             else ()
                         )
@@ -92,7 +106,12 @@ def get_datasets_to_backup(zfs_command, patterns):
                         )
                     ),
                 )
-                if contained_patterns
+                if dataset.auto_backup
+                or any(
+                    pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT
+                    and pattern.source == borgmatic.borg.pattern.Pattern_source.CONFIG
+                    for pattern in contained_patterns
+                )
             ),
             key=lambda dataset: dataset.mount_point,
         )
@@ -115,7 +134,16 @@ def get_all_dataset_mount_points(zfs_command):
         )
     )
 
-    return tuple(sorted(line.rstrip() for line in list_output.splitlines()))
+    return tuple(
+        sorted(
+            {
+                mount_point
+                for line in list_output.splitlines()
+                for mount_point in (line.rstrip(),)
+                if mount_point != 'none'
+            }
+        )
+    )
 
 
 def snapshot_dataset(zfs_command, full_snapshot_name):  # pragma: no cover
@@ -155,10 +183,14 @@ def mount_snapshot(mount_command, full_snapshot_name, snapshot_mount_path):  # p
     )
 
 
-def make_borg_snapshot_pattern(pattern, normalized_runtime_directory):
+MOUNT_POINT_HASH_LENGTH = 10
+
+
+def make_borg_snapshot_pattern(pattern, dataset, normalized_runtime_directory):
     '''
-    Given a Borg pattern as a borgmatic.borg.pattern.Pattern instance, return a new Pattern with its
-    path rewritten to be in a snapshot directory based on the given runtime directory.
+    Given a Borg pattern as a borgmatic.borg.pattern.Pattern instance and the Dataset containing it,
+    return a new Pattern with its path rewritten to be in a snapshot directory based on both the
+    given runtime directory and the given Dataset's mount point.
 
     Move any initial caret in a regular expression pattern path to the beginning, so as not to break
     the regular expression.
@@ -173,6 +205,10 @@ def make_borg_snapshot_pattern(pattern, normalized_runtime_directory):
     rewritten_path = initial_caret + os.path.join(
         normalized_runtime_directory,
         'zfs_snapshots',
+        # Including this hash prevents conflicts between snapshot patterns for different datasets.
+        # For instance, without this, snapshotting a dataset at /var and another at /var/spool would
+        # result in overlapping snapshot patterns and therefore colliding mount attempts.
+        hashlib.shake_256(dataset.mount_point.encode('utf-8')).hexdigest(MOUNT_POINT_HASH_LENGTH),
         '.',  # Borg 1.4+ "slashdot" hack.
         # Included so that the source directory ends up in the Borg archive at its "original" path.
         pattern.path.lstrip('^').lstrip(os.path.sep),
@@ -183,6 +219,7 @@ def make_borg_snapshot_pattern(pattern, normalized_runtime_directory):
         pattern.type,
         pattern.style,
         pattern.device,
+        source=borgmatic.borg.pattern.Pattern_source.HOOK,
     )
 
 
@@ -209,7 +246,8 @@ def dump_data_sources(
     dry_run_label = ' (dry run; not actually snapshotting anything)' if dry_run else ''
     logger.info(f'Snapshotting ZFS datasets{dry_run_label}')
 
-    # List ZFS datasets to get their mount points.
+    # List ZFS datasets to get their mount points, but only consider those patterns that came from
+    # actual user configuration (as opposed to, say, other hooks).
     zfs_command = hook_config.get('zfs_command', 'zfs')
     requested_datasets = get_datasets_to_backup(zfs_command, patterns)
 
@@ -234,6 +272,9 @@ def dump_data_sources(
         snapshot_mount_path = os.path.join(
             normalized_runtime_directory,
             'zfs_snapshots',
+            hashlib.shake_256(dataset.mount_point.encode('utf-8')).hexdigest(
+                MOUNT_POINT_HASH_LENGTH
+            ),
             dataset.mount_point.lstrip(os.path.sep),
         )
 
@@ -249,7 +290,9 @@ def dump_data_sources(
         )
 
         for pattern in dataset.contained_patterns:
-            snapshot_pattern = make_borg_snapshot_pattern(pattern, normalized_runtime_directory)
+            snapshot_pattern = make_borg_snapshot_pattern(
+                pattern, dataset, normalized_runtime_directory
+            )
 
             # Attempt to update the pattern in place, since pattern order matters to Borg.
             try:
@@ -334,6 +377,7 @@ def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, d
             os.path.normpath(borgmatic_runtime_directory),
         ),
         'zfs_snapshots',
+        '*',
     )
     logger.debug(f'Looking for snapshots to remove in {snapshots_glob}{dry_run_label}')
     umount_command = hook_config.get('umount_command', 'umount')
@@ -346,7 +390,10 @@ def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, d
         # child datasets before the shorter mount point paths of parent datasets.
         for mount_point in reversed(dataset_mount_points):
             snapshot_mount_path = os.path.join(snapshots_directory, mount_point.lstrip(os.path.sep))
-            if not os.path.isdir(snapshot_mount_path):
+
+            # If the snapshot mount path is empty, this is probably just a "shadow" of a nested
+            # dataset and therefore there's nothing to unmount.
+            if not os.path.isdir(snapshot_mount_path) or not os.listdir(snapshot_mount_path):
                 continue
 
             # This might fail if the path is already mounted, but we swallow errors here since we'll
@@ -370,10 +417,10 @@ def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, d
                     return
                 except subprocess.CalledProcessError as error:
                     logger.debug(error)
-                    return
+                    continue
 
         if not dry_run:
-            shutil.rmtree(snapshots_directory)
+            shutil.rmtree(snapshot_mount_path, ignore_errors=True)
 
     # Destroy snapshots.
     full_snapshot_names = get_all_snapshots(zfs_command)

borgmatic/hooks/dispatch.py

@@ -3,6 +3,7 @@ import importlib
 import logging
 import pkgutil
 
+import borgmatic.hooks.command
 import borgmatic.hooks.credential
 import borgmatic.hooks.data_source
 import borgmatic.hooks.monitoring

borgmatic/hooks/monitoring/cronhub.py

@@ -28,7 +28,7 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     filename in any log entries. If this is a dry run, then don't actually ping anything.
     '''
     if state not in MONITOR_STATE_TO_CRONHUB:
-        logger.debug(f'Ignoring unsupported monitoring {state.name.lower()} in Cronhub hook')
+        logger.debug(f'Ignoring unsupported monitoring state {state.name.lower()} in Cronhub hook')
         return
 
     dry_run_label = ' (dry run; not actually pinging)' if dry_run else ''

borgmatic/hooks/monitoring/cronitor.py

@@ -28,7 +28,7 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     filename in any log entries. If this is a dry run, then don't actually ping anything.
     '''
     if state not in MONITOR_STATE_TO_CRONITOR:
-        logger.debug(f'Ignoring unsupported monitoring {state.name.lower()} in Cronitor hook')
+        logger.debug(f'Ignoring unsupported monitoring state {state.name.lower()} in Cronitor hook')
         return
 
     dry_run_label = ' (dry run; not actually pinging)' if dry_run else ''

borgmatic/hooks/monitoring/logs.py

@@ -64,7 +64,7 @@ def get_handler(identifier):
 def format_buffered_logs_for_payload(identifier):
     '''
     Get the handler previously added to the root logger, and slurp buffered logs out of it to
-    send to Healthchecks.
+    send to the monitoring service.
     '''
     try:
         buffering_handler = get_handler(identifier)

borgmatic/hooks/monitoring/ntfy.py

@@ -51,13 +51,13 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
 
         try:
             username = borgmatic.hooks.credential.parse.resolve_credential(
-                hook_config.get('username')
+                hook_config.get('username'), config
             )
             password = borgmatic.hooks.credential.parse.resolve_credential(
-                hook_config.get('password')
+                hook_config.get('password'), config
             )
             access_token = borgmatic.hooks.credential.parse.resolve_credential(
-                hook_config.get('access_token')
+                hook_config.get('access_token'), config
             )
         except ValueError as error:
             logger.warning(f'Ntfy credential error: {error}')

borgmatic/hooks/monitoring/pagerduty.py

@@ -6,20 +6,36 @@ import platform
 import requests
 
 import borgmatic.hooks.credential.parse
+import borgmatic.hooks.monitoring.logs
 from borgmatic.hooks.monitoring import monitor
 
 logger = logging.getLogger(__name__)
 
 EVENTS_API_URL = 'https://events.pagerduty.com/v2/enqueue'
+DEFAULT_LOGS_PAYLOAD_LIMIT_BYTES = 10000
+HANDLER_IDENTIFIER = 'pagerduty'
 
 
-def initialize_monitor(
-    integration_key, config, config_filename, monitoring_log_level, dry_run
-):  # pragma: no cover
+def initialize_monitor(hook_config, config, config_filename, monitoring_log_level, dry_run):
     '''
-    No initialization is necessary for this monitor.
+    Add a handler to the root logger that stores in memory the most recent logs emitted. That way,
+    we can send them all to PagerDuty upon a failure state. But skip this if the "send_logs" option
+    is false.
     '''
-    pass
+    if hook_config.get('send_logs') is False:
+        return
+
+    ping_body_limit = max(
+        DEFAULT_LOGS_PAYLOAD_LIMIT_BYTES
+        - len(borgmatic.hooks.monitoring.logs.PAYLOAD_TRUNCATION_INDICATOR),
+        0,
+    )
+
+    borgmatic.hooks.monitoring.logs.add_handler(
+        borgmatic.hooks.monitoring.logs.Forgetful_buffering_handler(
+            HANDLER_IDENTIFIER, ping_body_limit, monitoring_log_level
+        )
+    )
 
 
 def ping_monitor(hook_config, config, config_filename, state, monitoring_log_level, dry_run):
@@ -30,24 +46,25 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     '''
     if state != monitor.State.FAIL:
         logger.debug(
-            f'Ignoring unsupported monitoring {state.name.lower()} in PagerDuty hook',
+            f'Ignoring unsupported monitoring state {state.name.lower()} in PagerDuty hook',
         )
         return
 
     dry_run_label = ' (dry run; not actually sending)' if dry_run else ''
     logger.info(f'Sending failure event to PagerDuty {dry_run_label}')
 
-    if dry_run:
-        return
-
     try:
         integration_key = borgmatic.hooks.credential.parse.resolve_credential(
-            hook_config.get('integration_key')
+            hook_config.get('integration_key'), config
         )
     except ValueError as error:
         logger.warning(f'PagerDuty credential error: {error}')
         return
 
+    logs_payload = borgmatic.hooks.monitoring.logs.format_buffered_logs_for_payload(
+        HANDLER_IDENTIFIER
+    )
+
     hostname = platform.node()
     local_timestamp = datetime.datetime.now(datetime.timezone.utc).astimezone().isoformat()
     payload = json.dumps(
@@ -66,11 +83,14 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
                     'hostname': hostname,
                     'configuration filename': config_filename,
                     'server time': local_timestamp,
+                    'logs': logs_payload,
                 },
             },
         }
     )
-    logger.debug(f'Using PagerDuty payload: {payload}')
+
+    if dry_run:
+        return
 
     logging.getLogger('urllib3').setLevel(logging.ERROR)
     try:
@@ -83,6 +103,7 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
 
 def destroy_monitor(ping_url_or_uuid, config, monitoring_log_level, dry_run):  # pragma: no cover
     '''
-    No destruction is necessary for this monitor.
+    Remove the monitor handler that was added to the root logger. This prevents the handler from
+    getting reused by other instances of this monitor.
     '''
-    pass
+    borgmatic.hooks.monitoring.logs.remove_handler(HANDLER_IDENTIFIER)

borgmatic/hooks/monitoring/pushover.py

@@ -35,8 +35,10 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     state_config = hook_config.get(state.name.lower(), {})
 
     try:
-        token = borgmatic.hooks.credential.parse.resolve_credential(hook_config.get('token'))
-        user = borgmatic.hooks.credential.parse.resolve_credential(hook_config.get('user'))
+        token = borgmatic.hooks.credential.parse.resolve_credential(
+            hook_config.get('token'), config
+        )
+        user = borgmatic.hooks.credential.parse.resolve_credential(hook_config.get('user'), config)
     except ValueError as error:
         logger.warning(f'Pushover credential error: {error}')
         return

borgmatic/hooks/monitoring/uptime_kuma.py

@@ -37,7 +37,7 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     logging.getLogger('urllib3').setLevel(logging.ERROR)
 
     try:
-        response = requests.get(f'{push_url}?{query}')
+        response = requests.get(f'{push_url}?{query}', verify=hook_config.get('verify_tls', True))
         if not response.ok:
             response.raise_for_status()
     except requests.exceptions.RequestException as error:

borgmatic/hooks/monitoring/zabbix.py

@@ -16,6 +16,42 @@ def initialize_monitor(
     pass
 
 
+def send_zabbix_request(server, headers, data):
+    '''
+    Given a Zabbix server URL, HTTP headers as a dict, and valid Zabbix JSON payload data as a dict,
+    send a request to the Zabbix server via API.
+
+    Return the response "result" value or None.
+    '''
+    logging.getLogger('urllib3').setLevel(logging.ERROR)
+
+    logger.debug(f'Sending a "{data["method"]}" request to the Zabbix server')
+
+    try:
+        response = requests.post(server, headers=headers, json=data)
+
+        if not response.ok:
+            response.raise_for_status()
+    except requests.exceptions.RequestException as error:
+        logger.warning(f'Zabbix error: {error}')
+
+        return None
+
+    try:
+        result = response.json().get('result')
+        error_message = result['data'][0]['error']
+    except requests.exceptions.JSONDecodeError:
+        logger.warning('Zabbix error: Cannot parse API response')
+
+        return None
+    except (TypeError, KeyError, IndexError):
+        return result
+    else:
+        logger.warning(f'Zabbix error: {error_message}')
+
+        return None
+
+
 def ping_monitor(hook_config, config, config_filename, state, monitoring_log_level, dry_run):
     '''
     Update the configured Zabbix item using either the itemid, or a host and key.
@@ -37,11 +73,18 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     )
 
     try:
-        username = borgmatic.hooks.credential.parse.resolve_credential(hook_config.get('username'))
-        password = borgmatic.hooks.credential.parse.resolve_credential(hook_config.get('password'))
-        api_key = borgmatic.hooks.credential.parse.resolve_credential(hook_config.get('api_key'))
+        username = borgmatic.hooks.credential.parse.resolve_credential(
+            hook_config.get('username'), config
+        )
+        password = borgmatic.hooks.credential.parse.resolve_credential(
+            hook_config.get('password'), config
+        )
+        api_key = borgmatic.hooks.credential.parse.resolve_credential(
+            hook_config.get('api_key'), config
+        )
     except ValueError as error:
         logger.warning(f'Zabbix credential error: {error}')
+
         return
 
     server = hook_config.get('server')
@@ -51,13 +94,9 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     value = state_config.get('value')
     headers = {'Content-Type': 'application/json-rpc'}
 
-    logger.info(f'Updating Zabbix{dry_run_label}')
+    logger.info(f'Pinging Zabbix{dry_run_label}')
     logger.debug(f'Using Zabbix URL: {server}')
 
-    if server is None:
-        logger.warning('Server missing for Zabbix')
-        return
-
     # Determine the Zabbix method used to store the value: itemid or host/key
     if itemid is not None:
         logger.info(f'Updating {itemid} on Zabbix')
@@ -68,8 +107,8 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
             'id': 1,
         }
 
-    elif (host and key) is not None:
-        logger.info(f'Updating Host:{host} and Key:{key} on Zabbix')
+    elif host is not None and key is not None:
+        logger.info(f'Updating Host: "{host}" and Key: "{key}" on Zabbix')
         data = {
             'jsonrpc': '2.0',
             'method': 'history.push',
@@ -79,58 +118,63 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
 
     elif host is not None:
         logger.warning('Key missing for Zabbix')
-        return
 
+        return
     elif key is not None:
         logger.warning('Host missing for Zabbix')
+
         return
     else:
         logger.warning('No Zabbix itemid or host/key provided')
+
         return
 
     # Determine the authentication method: API key or username/password
     if api_key is not None:
         logger.info('Using API key auth for Zabbix')
-        headers['Authorization'] = 'Bearer ' + api_key
-
-    elif (username and password) is not None:
-        logger.info('Using user/pass auth with user {username} for Zabbix')
-        auth_data = {
+        headers['Authorization'] = f'Bearer {api_key}'
+    elif username is not None and password is not None:
+        logger.info(f'Using user/pass auth with user {username} for Zabbix')
+        login_data = {
             'jsonrpc': '2.0',
             'method': 'user.login',
             'params': {'username': username, 'password': password},
             'id': 1,
         }
+
         if not dry_run:
-            logging.getLogger('urllib3').setLevel(logging.ERROR)
-            try:
-                response = requests.post(server, headers=headers, json=auth_data)
-                data['auth'] = response.json().get('result')
-                if not response.ok:
-                    response.raise_for_status()
-            except requests.exceptions.RequestException as error:
-                logger.warning(f'Zabbix error: {error}')
+            result = send_zabbix_request(server, headers, login_data)
+
+            if not result:
                 return
 
+            headers['Authorization'] = f'Bearer {result}'
     elif username is not None:
         logger.warning('Password missing for Zabbix authentication')
-        return
 
+        return
     elif password is not None:
         logger.warning('Username missing for Zabbix authentication')
+
         return
     else:
         logger.warning('Authentication data missing for Zabbix')
+
         return
 
     if not dry_run:
-        logging.getLogger('urllib3').setLevel(logging.ERROR)
-        try:
-            response = requests.post(server, headers=headers, json=data)
-            if not response.ok:
-                response.raise_for_status()
-        except requests.exceptions.RequestException as error:
-            logger.warning(f'Zabbix error: {error}')
+        send_zabbix_request(server, headers, data)
+
+    if username is not None and password is not None:
+        logout_data = {
+            'jsonrpc': '2.0',
+            'method': 'user.logout',
+            'params': [],
+            'id': 1,
+        }
+
+        if not dry_run:
+            send_zabbix_request(server, headers, logout_data)
 
 
 def destroy_monitor(ping_url_or_uuid, config, monitoring_log_level, dry_run):  # pragma: no cover

borgmatic/logger.py

@@ -256,7 +256,7 @@ class Log_prefix:
         self.original_prefix = get_log_prefix()
         set_log_prefix(self.prefix)
 
-    def __exit__(self, exception, value, traceback):
+    def __exit__(self, exception_type, exception, traceback):
         '''
         Restore any original prefix.
         '''

borgmatic/signals.py

@@ -24,6 +24,9 @@ def handle_signal(signal_number, frame):
         logger.critical('Exiting due to TERM signal')
         sys.exit(EXIT_CODE_FROM_SIGNAL + signal.SIGTERM)
     elif signal_number == signal.SIGINT:
+        # Borg doesn't always exit on a SIGINT, so give it a little encouragement.
+        os.killpg(os.getpgrp(), signal.SIGTERM)
+
         raise KeyboardInterrupt()
 
 

docs/_includes/index.css

@@ -165,6 +165,7 @@ ul {
 }
 li {
 	padding: .25em 0;
+        line-height: 1.5;
 }
 li ul {
         list-style-type: disc;

docs/fetch-contributors

@@ -26,8 +26,7 @@ def list_merged_pulls(url):
 
 
 def list_contributing_issues(url):
-    # labels = bug, design finalized, etc.
-    response = requests.get(f'{url}?labels=19,20,22,23,32,52,53,54', headers={'Accept': 'application/json', 'Content-Type': 'application/json'})
+    response = requests.get(url, headers={'Accept': 'application/json', 'Content-Type': 'application/json'})
 
     if not response.ok:
         response.raise_for_status()
@@ -39,7 +38,7 @@ PULLS_API_ENDPOINT_URLS = (
     'https://projects.torsion.org/api/v1/repos/borgmatic-collective/borgmatic/pulls',
     'https://api.github.com/repos/borgmatic-collective/borgmatic/pulls',
 )
-ISSUES_API_ENDPOINT_URL = 'https://projects.torsion.org/api/v1/repos/borgmatic-collective/borgmatic/issues'
+ISSUES_API_ENDPOINT_URL = 'https://projects.torsion.org/api/v1/repos/borgmatic-collective/borgmatic/issues?state=all'
 RECENT_CONTRIBUTORS_CUTOFF_DAYS = 365
 
 

docs/how-to/add-preparation-and-cleanup-steps-to-backups.md

@@ -7,18 +7,112 @@ eleventyNavigation:
 ---
 ## Preparation and cleanup hooks
 
-If you find yourself performing preparation tasks before your backup runs, or
-cleanup work afterwards, borgmatic hooks may be of interest. Hooks are shell
-commands that borgmatic executes for you at various points as it runs, and
-they're configured in the `hooks` section of your configuration file. But if
-you're looking to backup a database, it's probably easier to use the [database
-backup
-feature](https://torsion.org/borgmatic/docs/how-to/backup-your-databases/)
-instead.
+If you find yourself performing preparation tasks before your backup runs or
+doing cleanup work afterwards, borgmatic command hooks may be of interest. These
+are custom shell commands you can configure borgmatic to execute at various
+points as it runs.
 
-You can specify `before_backup` hooks to perform preparation steps before
+(But if you're looking to backup a database, it's probably easier to use the
+[database backup
+feature](https://torsion.org/borgmatic/docs/how-to/backup-your-databases/)
+instead.)
+
+<span class="minilink minilink-addedin">New in version 2.0.0 (not yet
+released)</span> Command hooks are now configured via a list of `commands:` in
+your borgmatic configuration file. For example:
+
+```yaml
+commands:
+    - before: action
+      when: [create]
+      run:
+          - echo "Before create!"
+    - after: action
+      when:
+          - create
+          - prune
+      run:
+          - echo "After create or prune!"
+    - after: error
+      run:
+          - echo "Something went wrong!"
+```
+
+If you're coming from an older version of borgmatic, there is tooling to help
+you [upgrade your
+configuration](https://torsion.org/borgmatic/docs/how-to/upgrade/#upgrading-your-configuration)
+to this new command hook format.
+
+Note that if a `run:` command contains a special YAML character such as a colon,
+you may need to quote the entire string (or use a [multiline
+string](https://yaml-multiline.info/)) to avoid an error:
+
+```yaml
+commands:
+    - before: action
+      when: [create]
+      run:
+          - "echo Backup: start"
+```
+
+Each command in the `commands:` list has the following options:
+
+ * `before` or `after`: Name for the point in borgmatic's execution that the commands should be run before or after, one of:
+    * `action` runs before each action for each repository. This replaces the deprecated `before_create`, `after_prune`, etc.
+    * `repository` runs before or after all actions for each repository. This replaces the deprecated `before_actions` and `after_actions`.
+    * `configuration` runs before or after all actions and repositories in the current configuration file.
+    * `everything` runs before or after all configuration files. Errors here do not trigger `error` hooks or the `fail` state in monitoring hooks. This replaces the deprecated `before_everything` and `after_everything`.
+    * `error` runs after an error occurs—and it's only available for `after`. This replaces the deprecated `on_error` hook.
+ * `when`: Only trigger the hook when borgmatic is run with particular actions (`create`, `prune`, etc.) listed here. Defaults to running for all actions.
+ * `run`: List of one or more shell commands or scripts to run when this command hook is triggered.
+
+An `after` command hook runs even if an error occurs in the corresponding
+`before` hook or between those two hooks. This allows you to perform cleanup
+steps that correspond to `before` preparation commands—even when something goes
+wrong. This is a departure from the way that the deprecated `after_*` hooks
+worked in borgmatic prior to version 2.0.0.
+
+Additionally, when command hooks run, they respect the `working_directory`
+option if it is configured, meaning that the hook commands are run in that
+directory.
+
+
+### Order of execution
+
+Here's a way of visualizing how all of these command hooks slot into borgmatic's
+execution.
+
+Let's say you've got a borgmatic configuration file with a configured
+repository. And suppose you configure several command hooks and then run
+borgmatic for the `create` and `prune` actions. Here's the order of execution:
+