borgmatic-collective/borgmatic
borgmatic-collective/borgmatic
Watch
9
74
Fork
You've already forked borgmatic
 57
Code Issues 71 Pull requests 3 Releases 162 Packages 1 Activity Actions 1

Dry run on brand new setup fails #507

New issue
Closed
opened 2022-03-07 20:26:26 +00:00 by TheBekker · 4 comments
TheBekker commented 2022-03-07 20:26:26 +00:00
Copy link

What I'm trying to do and why

I'm trying to a dry run of my new borg repo.
After going trough the config.yaml and running borg --init i do a dry run. (init runs fine)

borgmatic -n -v 2

image

Don't see any good indication of what is actually going wrong.
I'm very new to using both borg and borgmatic, so i might have missed something obvious, sorry if thats the case.

Any help is greatly appreciated.

Environment

borgmatic version: 1.5.23

borgmatic installation method: Docker (b3vis/borgmatic)

Borg version: 1.2.0

Python version: 3.9.7

#### What I'm trying to do and why I'm trying to a dry run of my new borg repo. After going trough the config.yaml and running `borg --init` i do a dry run. (init runs fine) `borgmatic -n -v 2` ![image](/attachments/1403b34e-8888-473e-8ca3-fc29ceb6f49a) Don't see any good indication of what is actually going wrong. I'm very new to using both borg and borgmatic, so i might have missed something obvious, sorry if thats the case. Any help is greatly appreciated. #### Environment **borgmatic version:** 1.5.23 **borgmatic installation method:** Docker (b3vis/borgmatic) **Borg version:** 1.2.0 **Python version:** 3.9.7
image.png
96 KiB
TheBekker commented 2022-03-07 20:33:33 +00:00
Author
Copy link

Borgmatic config file: (sensitive info replaced with *)

Show full config 
# Where to look for files to backup, and where to store those backups.
# See https://borgbackup.readthedocs.io/en/stable/quickstart.html and
# https://borgbackup.readthedocs.io/en/stable/usage/create.html
# for details.
location:
    # List of source directories to backup (required). Globs and
    # tildes are expanded. Do not backslash spaces in path names.
    source_directories:
        - /mnt/user/Software/

    # Paths to local or remote repositories (required). Tildes are
    # expanded. Multiple repositories are backed up to in
    # sequence. Borg placeholders can be used. See the output of
    # "borg help placeholders" for details. See ssh_command for
    # SSH options like identity file or port. If systemd service
    # is used, then add local repository paths in the systemd
    # service file to the ReadWritePaths list.
    repositories:
        - 'ssh://*********.your-storagebox.de:23/./backups/thebeast'

    # Stay in same file system (do not cross mount points).
    # Defaults to false. But when a database hook is used, the
    # setting here is ignored and one_file_system is considered
    # true.
    # one_file_system: true

    # Only store/extract numeric user and group identifiers.
    # Defaults to false.
    # numeric_owner: true

    # Store atime into archive. Defaults to true in Borg < 1.2,
    # false in Borg 1.2+.
    # atime: false

    # Store ctime into archive. Defaults to true.
    # ctime: false

    # Store birthtime (creation date) into archive. Defaults to
    # true.
    # birthtime: false

    # Use Borg's --read-special flag to allow backup of block and
    # other special devices. Use with caution, as it will lead to
    # problems if used when backing up special devices such as
    # /dev/zero. Defaults to false. But when a database hook is
    # used, the setting here is ignored and read_special is
    # considered true.
    # read_special: false

    # Record bsdflags (e.g. NODUMP, IMMUTABLE) in archive.
    # Defaults to true.
    # bsd_flags: true

    # Mode in which to operate the files cache. See
    # http://borgbackup.readthedocs.io/en/stable/usage/create.html
    # for details. Defaults to "ctime,size,inode".
    # files_cache: ctime,size,inode

    # Alternate Borg local executable. Defaults to "borg".
    # local_path: borg1

    # Alternate Borg remote executable. Defaults to "borg".
    # remote_path: borg1

    # Any paths matching these patterns are included/excluded from
    # backups. Globs are expanded. (Tildes are not.) See the
    # output of "borg help patterns" for more details. Quote any
    # value if it contains leading punctuation, so it parses
    # correctly.
    # patterns:
        # - '+ /mnt/user/Software/*'
        # - R /
        # - '- /home/*/.cache'
        # - + /home/susan
        # - '- /home/*'

    # Read include/exclude patterns from one or more separate
    # named files, one pattern per line. Note that Borg considers
    # this option experimental. See the output of "borg help
    # patterns" for more details.
    # patterns_from:
        # - /etc/borgmatic/patterns

    # Any paths matching these patterns are excluded from backups.
    # Globs and tildes are expanded. (Note however that a glob
    # pattern must either start with a glob or be an absolute
    # path.) Do not backslash spaces in path names. See the output
    # of "borg help patterns" for more details.
    # exclude_patterns:
        # - '*.pyc'
        # - /home/*/.cache
        # - '*/.vim*.tmp'
        # - /etc/ssl
        # - /home/user/path with spaces

    # Read exclude patterns from one or more separate named files,
    # one pattern per line. See the output of "borg help patterns"
    # for more details.
    # exclude_from:
        # - /etc/borgmatic/excludes

    # Exclude directories that contain a CACHEDIR.TAG file. See
    # http://www.brynosaurus.com/cachedir/spec.html for details.
    # Defaults to false.
    # exclude_caches: true

    # Exclude directories that contain a file with the given
    # filenames. Defaults to not set.
    # exclude_if_present:
        # - .nobackup

    # If true, the exclude_if_present filename is included in
    # backups. Defaults to false, meaning that the
    # exclude_if_present filename is omitted from backups.
    # keep_exclude_tags: true

    # Exclude files with the NODUMP flag. Defaults to false.
    # exclude_nodump: true

    # Path for additional source files used for temporary internal
    # state like borgmatic database dumps. Note that changing this
    # path prevents "borgmatic restore" from finding any database
    # dumps created before the change. Defaults to ~/.borgmatic
    # borgmatic_source_directory: /tmp/borgmatic

# Repository storage options. See
# https://borgbackup.readthedocs.io/en/stable/usage/create.html and
# https://borgbackup.readthedocs.io/en/stable/usage/general.html for
# details.
storage:
    # The standard output of this command is used to unlock the
    # encryption key. Only use on repositories that were
    # initialized with passcommand/repokey/keyfile encryption.
    # Note that if both encryption_passcommand and
    # encryption_passphrase are set, then encryption_passphrase
    # takes precedence. Defaults to not set.
    # encryption_passcommand: secret-tool lookup borg-repository repo-name

    # Passphrase to unlock the encryption key with. Only use on
    # repositories that were initialized with
    # passphrase/repokey/keyfile encryption. Quote the value if it
    # contains punctuation, so it parses correctly. And backslash
    # any quote or backslash literals as well. Defaults to not
    # set.
    encryption_passphrase: "*********"

    # Number of seconds between each checkpoint during a
    # long-running backup. See
    # https://borgbackup.readthedocs.io/en/stable/faq.html
    # for details. Defaults to checkpoints every 1800 seconds (30
    # minutes).
    # checkpoint_interval: 1800

    # Specify the parameters passed to then chunker
    # (CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS,
    # HASH_WINDOW_SIZE). See
    # https://borgbackup.readthedocs.io/en/stable/internals.html
    # for details. Defaults to "19,23,21,4095".
    # chunker_params: 19,23,21,4095

    # Type of compression to use when creating archives. See
    # http://borgbackup.readthedocs.io/en/stable/usage/create.html
    # for details. Defaults to "lz4".
    # compression: lz4

    # Remote network upload rate limit in kiBytes/second. Defaults
    # to unlimited.
    # remote_rate_limit: 100

    # Number of times to retry a failing backup before giving up.
    # Defaults to 0 (i.e., does not attempt retry).
    retries: 3

    # Wait time between retries (in seconds) to allow transient
    # issues to pass. Increases after each retry as a form of
    # backoff. Defaults to 0 (no wait).
    retry_wait: 30

    # Directory where temporary files are stored. Defaults to
    # $TMPDIR
    # temporary_directory: /path/to/tmpdir

    # Command to use instead of "ssh". This can be used to specify
    # ssh options.  Defaults to not set.
    ssh_command: ssh -i /root/.ssh/storagebox-new

    # Base path used for various Borg directories. Defaults to
    # $HOME, ~$USER, or ~.
    # borg_base_directory: /path/to/base

    # Path for Borg configuration files. Defaults to
    # $borg_base_directory/.config/borg
    # borg_config_directory: /path/to/base/config

    # Path for Borg cache files. Defaults to
    # $borg_base_directory/.cache/borg
    # borg_cache_directory: /path/to/base/cache

    # Path for Borg security and encryption nonce files. Defaults
    # to $borg_base_directory/.config/borg/security
    # borg_security_directory: /path/to/base/config/security

    # Path for Borg encryption key files. Defaults to
    # $borg_base_directory/.config/borg/keys
    # borg_keys_directory: /path/to/base/config/keys

    # Umask to be used for borg create. Defaults to 0077.
    # umask: 0077

    # Maximum seconds to wait for acquiring a repository/cache
    # lock. Defaults to 1.
    lock_wait: 5

    # Name of the archive. Borg placeholders can be used. See the
    # output of "borg help placeholders" for details. Defaults to
    # "{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}". If you specify this
    # option, you must also specify a prefix in the retention
    # section to avoid accidental pruning of archives with a
    # different archive name format. And you should also specify a
    # prefix in the consistency section as well.
    # archive_name_format: 'thebeast-{now:%Y-%m-%dT%H:%M:%S.%f}'

    # Bypass Borg error about a repository that has been moved.
    # Defaults to false.
    # relocated_repo_access_is_ok: true

    # Bypass Borg error about a previously unknown unencrypted
    # repository. Defaults to false.
    # unknown_unencrypted_repo_access_is_ok: true

    # Additional options to pass directly to particular Borg
    # commands, handy for Borg options that borgmatic does not yet
    # support natively. Note that borgmatic does not perform any
    # validation on these options. Running borgmatic with
    # "--verbosity 2" shows the exact Borg command-line
    # invocation.
    # extra_borg_options:
        # Extra command-line options to pass to "borg init".
        # init: --extra-option

        # Extra command-line options to pass to "borg prune".
        # prune: --extra-option

        # Extra command-line options to pass to "borg compact".
        # compact: --extra-option

        # Extra command-line options to pass to "borg create".
        # create: --extra-option

        # Extra command-line options to pass to "borg check".
        # check: --extra-option

# Retention policy for how many backups to keep in each category. See
# https://borgbackup.readthedocs.io/en/stable/usage/prune.html for
# details. At least one of the "keep" options is required for pruning
# to work. To skip pruning entirely, run "borgmatic create" or "check"
# without the "prune" action. See borgmatic documentation for details.
retention:
    # Keep all archives within this time interval.
    # keep_within: 3H

    # Number of secondly archives to keep.
    # keep_secondly: 60

    # Number of minutely archives to keep.
    # keep_minutely: 60

    # Number of hourly archives to keep.
    # keep_hourly: 24

    # Number of daily archives to keep.
    keep_daily: 7

    # Number of weekly archives to keep.
    # keep_weekly: 4

    # Number of monthly archives to keep.
    # keep_monthly: 6

    # Number of yearly archives to keep.
    # keep_yearly: 1

    # When pruning, only consider archive names starting with this
    # prefix.  Borg placeholders can be used. See the output of
    # "borg help placeholders" for details. Defaults to
    # "{hostname}-". Use an empty value to disable the default.
    # prefix: sourcehostname

# Consistency checks to run after backups. See
# https://borgbackup.readthedocs.io/en/stable/usage/check.html and
# https://borgbackup.readthedocs.io/en/stable/usage/extract.html for
# details.
# consistency:
    # List of one or more consistency checks to run: "repository",
    # "archives", "data", and/or "extract". Defaults to
    # "repository" and "archives". Set to "disabled" to disable
    # all consistency checks. "repository" checks the consistency
    # of the repository, "archives" checks all of the archives,
    # "data" verifies the integrity of the data within the
    # archives, and "extract" does an extraction dry-run of the
    # most recent archive. Note that "data" implies "archives".
    # checks:
        # - repository
        # - archives

    # Paths to a subset of the repositories in the location
    # section on which to run consistency checks. Handy in case
    # some of your repositories are very large, and so running
    # consistency checks on them would take too long. Defaults to
    # running consistency checks on all repositories configured in
    # the location section.
    # check_repositories:
        # - user@backupserver:sourcehostname.borg

    # Restrict the number of checked archives to the last n.
    # Applies only to the "archives" check. Defaults to checking
    # all archives.
    # check_last: 3

    # When performing the "archives" check, only consider archive
    # names starting with this prefix. Borg placeholders can be
    # used. See the output of "borg help placeholders" for
    # details. Defaults to "{hostname}-". Use an empty value to
    # disable the default.
    # prefix: sourcehostname

# Options for customizing borgmatic's own output and logging.
# output:
    # Apply color to console output. Can be overridden with
    # --no-color command-line flag. Defaults to true.
    # color: false

# Shell commands, scripts, or integrations to execute at various
# points during a borgmatic run. IMPORTANT: All provided commands and
# scripts are executed with user permissions of borgmatic. Do not
# forget to set secure permissions on this configuration file (chmod
# 0600) as well as on any script called from a hook (chmod 0700) to
# prevent potential shell injection or privilege escalation.
hooks:
    # List of one or more shell commands or scripts to execute
    # before creating a backup, run once per configuration file.
    # before_backup:
        # - echo "Starting a backup."

    # List of one or more shell commands or scripts to execute
    # before pruning, run once per configuration file.
    # before_prune:
        # - echo "Starting pruning."

    # List of one or more shell commands or scripts to execute
    # before compaction, run once per configuration file.
    # before_compact:
        # - echo "Starting compaction."

    # List of one or more shell commands or scripts to execute
    # before consistency checks, run once per configuration file.
    # before_check:
        # - echo "Starting checks."

    # List of one or more shell commands or scripts to execute
    # before extracting a backup, run once per configuration file.
    # before_extract:
        # - echo "Starting extracting."

    # List of one or more shell commands or scripts to execute
    # after creating a backup, run once per configuration file.
    # after_backup:
        # - echo "Finished a backup."

    # List of one or more shell commands or scripts to execute
    # after compaction, run once per configuration file.
    # after_compact:
        # - echo "Finished compaction."

    # List of one or more shell commands or scripts to execute
    # after pruning, run once per configuration file.
    # after_prune:
        # - echo "Finished pruning."

    # List of one or more shell commands or scripts to execute
    # after consistency checks, run once per configuration file.
    # after_check:
        # - echo "Finished checks."

    # List of one or more shell commands or scripts to execute
    # after extracting a backup, run once per configuration file.
    # after_extract:
        # - echo "Finished extracting."

    # List of one or more shell commands or scripts to execute
    # when an exception occurs during a "prune", "compact",
    # "create", or "check" action or an associated before/after
    # hook.
    # on_error:
        # - echo "Error during prune/compact/create/check."

    # List of one or more shell commands or scripts to execute
    # before running all actions (if one of them is "create").
    # These are collected from all configuration files and then
    # run once before all of them (prior to all actions).
    # before_everything:
        # - echo "Starting actions."

    # List of one or more shell commands or scripts to execute
    # after running all actions (if one of them is "create").
    # These are collected from all configuration files and then
    # run once after all of them (after any action).
    # after_everything:
        # - echo "Completed actions."

    # List of one or more PostgreSQL databases to dump before
    # creating a backup, run once per configuration file. The
    # database dumps are added to your source directories at
    # runtime, backed up, and removed afterwards. Requires
    # pg_dump/pg_dumpall/pg_restore commands. See
    # https://www.postgresql.org/docs/current/app-pgdump.html and
    # https://www.postgresql.org/docs/current/libpq-ssl.html for
    # details.
    # postgresql_databases:
        # Database name (required if using this hook). Or
        # "all" to dump all databases on the host. Note
        # that using this database hook implicitly enables
        # both read_special and one_file_system (see
        # above) to support dump and restore streaming.
        # - name: users

          # Database hostname to connect to. Defaults to
          # connecting via local Unix socket.
          # hostname: database.example.org

          # Port to connect to. Defaults to 5432.
          # port: 5433

          # Username with which to connect to the database.
          # Defaults to the username of the current user.
          # You probably want to specify the "postgres"
          # superuser here when the database name is "all".
          # username: dbuser

          # Password with which to connect to the database.
          # Omitting a password will only work if PostgreSQL
          # is configured to trust the configured username
          # without a password or you create a ~/.pgpass
          # file.
          # password: trustsome1

          # Database dump output format. One of "plain",
          # "custom", "directory", or "tar". Defaults to
          # "custom" (unlike raw pg_dump). See pg_dump
          # documentation for details. Note that format is
          # ignored when the database name is "all".
          # format: directory

          # SSL mode to use to connect to the database
          # server. One of "disable", "allow", "prefer",
          # "require", "verify-ca" or "verify-full".
          # Defaults to "disable".
          # ssl_mode: require

          # Path to a client certificate.
          # ssl_cert: /root/.postgresql/postgresql.crt

          # Path to a private client key.
          # ssl_key: /root/.postgresql/postgresql.key

          # Path to a root certificate containing a list of
          # trusted certificate authorities.
          # ssl_root_cert: /root/.postgresql/root.crt

          # Path to a certificate revocation list.
          # ssl_crl: /root/.postgresql/root.crl

          # Additional pg_dump/pg_dumpall options to pass
          # directly to the dump command, without performing
          # any validation on them. See pg_dump
          # documentation for details.
          # options: --role=someone

    # List of one or more MySQL/MariaDB databases to dump before
    # creating a backup, run once per configuration file. The
    # database dumps are added to your source directories at
    # runtime, backed up, and removed afterwards. Requires
    # mysqldump/mysql commands (from either MySQL or MariaDB). See
    # https://dev.mysql.com/doc/refman/8.0/en/mysqldump.html or
    # https://mariadb.com/kb/en/library/mysqldump/ for details.
    # mysql_databases:
        # Database name (required if using this hook). Or
        # "all" to dump all databases on the host. Note
        # that using this database hook implicitly enables
        # both read_special and one_file_system (see
        # above) to support dump and restore streaming.
        # - name: users

          # Database hostname to connect to. Defaults to
          # connecting via local Unix socket.
          # hostname: database.example.org

          # Port to connect to. Defaults to 3306.
          # port: 3307

          # Username with which to connect to the database.
          # Defaults to the username of the current user.
          # username: dbuser

          # Password with which to connect to the database.
          # Omitting a password will only work if MySQL is
          # configured to trust the configured username
          # without a password.
          # password: trustsome1

          # Additional mysql options to pass directly to
          # the mysql command that lists available
          # databases, without performing any validation on
          # them. See mysql documentation for details.
          # list_options: --defaults-extra-file=my.cnf

          # Additional mysqldump options to pass directly to
          # the dump command, without performing any
          # validation on them. See mysqldump documentation
          # for details.
          # options: --skip-comments

    # List of one or more MongoDB databases to dump before
    # creating a backup, run once per configuration file. The
    # database dumps are added to your source directories at
    # runtime, backed up, and removed afterwards. Requires
    # mongodump/mongorestore commands. See
    # https://docs.mongodb.com/database-tools/mongodump/ and
    # https://docs.mongodb.com/database-tools/mongorestore/ for
    # details.
    # mongodb_databases:
        # Database name (required if using this hook). Or
        # "all" to dump all databases on the host. Note
        # that using this database hook implicitly enables
        # both read_special and one_file_system (see
        # above) to support dump and restore streaming.
        # - name: users

          # Database hostname to connect to. Defaults to
          # connecting to localhost.
          # hostname: database.example.org

          # Port to connect to. Defaults to 27017.
          # port: 27018

          # Username with which to connect to the database.
          # Skip it if no authentication is needed.
          # username: dbuser

          # Password with which to connect to the database.
          # Skip it if no authentication is needed.
          # password: trustsome1

          # Authentication database where the specified
          # username exists. If no authentication database
          # is specified, the database provided in "name"
          # is used. If "name" is "all", the "admin"
          # database is used.
          # authentication_database: admin

          # Database dump output format. One of "archive",
          # or "directory". Defaults to "archive". See 
          # mongodump documentation for details. Note that 
          # format is ignored when the database name is
          # "all".
          # format: directory

          # Additional mongodump options to pass
          # directly to the dump command, without performing
          # any validation on them. See mongodump
          # documentation for details.
          # options: --role=someone

    # Healthchecks ping URL or UUID to notify when a backup
    # begins, ends, or errors. Create an account at
    # https://healthchecks.io if you'd like to use this service.
    # See borgmatic monitoring documentation for details.
    # healthchecks: https://hc-ping.com/your-uuid-here

    # Cronitor ping URL to notify when a backup begins, ends, or
    # errors. Create an account at https://cronitor.io if you'd
    # like to use this service. See borgmatic monitoring
    # documentation for details.
    cronitor: https://cronitor.link/***********************

    # PagerDuty integration key used to notify PagerDuty when a
    # backup errors. Create an account at
    # https://www.pagerduty.com/ if you'd like to use this
    # service. See borgmatic monitoring documentation for details.
    # pagerduty: a177cad45bd374409f78906a810a3074

    # Cronhub ping URL to notify when a backup begins, ends, or
    # errors. Create an account at https://cronhub.io if you'd
    # like to use this service. See borgmatic monitoring
    # documentation for details.
    # cronhub: https://cronhub.io/start/1f5e3410-254c-11e8-b61d-55875966d01

    # Umask used when executing hooks. Defaults to the umask that
    # borgmatic is run with.
    # umask: 0077
Borgmatic config file: (sensitive info replaced with *) <details><summary>Show full config</summary> ``` # Where to look for files to backup, and where to store those backups. # See https://borgbackup.readthedocs.io/en/stable/quickstart.html and # https://borgbackup.readthedocs.io/en/stable/usage/create.html # for details. location: # List of source directories to backup (required). Globs and # tildes are expanded. Do not backslash spaces in path names. source_directories: - /mnt/user/Software/ # Paths to local or remote repositories (required). Tildes are # expanded. Multiple repositories are backed up to in # sequence. Borg placeholders can be used. See the output of # "borg help placeholders" for details. See ssh_command for # SSH options like identity file or port. If systemd service # is used, then add local repository paths in the systemd # service file to the ReadWritePaths list. repositories: - 'ssh://*********.your-storagebox.de:23/./backups/thebeast' # Stay in same file system (do not cross mount points). # Defaults to false. But when a database hook is used, the # setting here is ignored and one_file_system is considered # true. # one_file_system: true # Only store/extract numeric user and group identifiers. # Defaults to false. # numeric_owner: true # Store atime into archive. Defaults to true in Borg < 1.2, # false in Borg 1.2+. # atime: false # Store ctime into archive. Defaults to true. # ctime: false # Store birthtime (creation date) into archive. Defaults to # true. # birthtime: false # Use Borg's --read-special flag to allow backup of block and # other special devices. Use with caution, as it will lead to # problems if used when backing up special devices such as # /dev/zero. Defaults to false. But when a database hook is # used, the setting here is ignored and read_special is # considered true. # read_special: false # Record bsdflags (e.g. NODUMP, IMMUTABLE) in archive. # Defaults to true. # bsd_flags: true # Mode in which to operate the files cache. See # http://borgbackup.readthedocs.io/en/stable/usage/create.html # for details. Defaults to "ctime,size,inode". # files_cache: ctime,size,inode # Alternate Borg local executable. Defaults to "borg". # local_path: borg1 # Alternate Borg remote executable. Defaults to "borg". # remote_path: borg1 # Any paths matching these patterns are included/excluded from # backups. Globs are expanded. (Tildes are not.) See the # output of "borg help patterns" for more details. Quote any # value if it contains leading punctuation, so it parses # correctly. # patterns: # - '+ /mnt/user/Software/*' # - R / # - '- /home/*/.cache' # - + /home/susan # - '- /home/*' # Read include/exclude patterns from one or more separate # named files, one pattern per line. Note that Borg considers # this option experimental. See the output of "borg help # patterns" for more details. # patterns_from: # - /etc/borgmatic/patterns # Any paths matching these patterns are excluded from backups. # Globs and tildes are expanded. (Note however that a glob # pattern must either start with a glob or be an absolute # path.) Do not backslash spaces in path names. See the output # of "borg help patterns" for more details. # exclude_patterns: # - '*.pyc' # - /home/*/.cache # - '*/.vim*.tmp' # - /etc/ssl # - /home/user/path with spaces # Read exclude patterns from one or more separate named files, # one pattern per line. See the output of "borg help patterns" # for more details. # exclude_from: # - /etc/borgmatic/excludes # Exclude directories that contain a CACHEDIR.TAG file. See # http://www.brynosaurus.com/cachedir/spec.html for details. # Defaults to false. # exclude_caches: true # Exclude directories that contain a file with the given # filenames. Defaults to not set. # exclude_if_present: # - .nobackup # If true, the exclude_if_present filename is included in # backups. Defaults to false, meaning that the # exclude_if_present filename is omitted from backups. # keep_exclude_tags: true # Exclude files with the NODUMP flag. Defaults to false. # exclude_nodump: true # Path for additional source files used for temporary internal # state like borgmatic database dumps. Note that changing this # path prevents "borgmatic restore" from finding any database # dumps created before the change. Defaults to ~/.borgmatic # borgmatic_source_directory: /tmp/borgmatic # Repository storage options. See # https://borgbackup.readthedocs.io/en/stable/usage/create.html and # https://borgbackup.readthedocs.io/en/stable/usage/general.html for # details. storage: # The standard output of this command is used to unlock the # encryption key. Only use on repositories that were # initialized with passcommand/repokey/keyfile encryption. # Note that if both encryption_passcommand and # encryption_passphrase are set, then encryption_passphrase # takes precedence. Defaults to not set. # encryption_passcommand: secret-tool lookup borg-repository repo-name # Passphrase to unlock the encryption key with. Only use on # repositories that were initialized with # passphrase/repokey/keyfile encryption. Quote the value if it # contains punctuation, so it parses correctly. And backslash # any quote or backslash literals as well. Defaults to not # set. encryption_passphrase: "*********" # Number of seconds between each checkpoint during a # long-running backup. See # https://borgbackup.readthedocs.io/en/stable/faq.html # for details. Defaults to checkpoints every 1800 seconds (30 # minutes). # checkpoint_interval: 1800 # Specify the parameters passed to then chunker # (CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, # HASH_WINDOW_SIZE). See # https://borgbackup.readthedocs.io/en/stable/internals.html # for details. Defaults to "19,23,21,4095". # chunker_params: 19,23,21,4095 # Type of compression to use when creating archives. See # http://borgbackup.readthedocs.io/en/stable/usage/create.html # for details. Defaults to "lz4". # compression: lz4 # Remote network upload rate limit in kiBytes/second. Defaults # to unlimited. # remote_rate_limit: 100 # Number of times to retry a failing backup before giving up. # Defaults to 0 (i.e., does not attempt retry). retries: 3 # Wait time between retries (in seconds) to allow transient # issues to pass. Increases after each retry as a form of # backoff. Defaults to 0 (no wait). retry_wait: 30 # Directory where temporary files are stored. Defaults to # $TMPDIR # temporary_directory: /path/to/tmpdir # Command to use instead of "ssh". This can be used to specify # ssh options. Defaults to not set. ssh_command: ssh -i /root/.ssh/storagebox-new # Base path used for various Borg directories. Defaults to # $HOME, ~$USER, or ~. # borg_base_directory: /path/to/base # Path for Borg configuration files. Defaults to # $borg_base_directory/.config/borg # borg_config_directory: /path/to/base/config # Path for Borg cache files. Defaults to # $borg_base_directory/.cache/borg # borg_cache_directory: /path/to/base/cache # Path for Borg security and encryption nonce files. Defaults # to $borg_base_directory/.config/borg/security # borg_security_directory: /path/to/base/config/security # Path for Borg encryption key files. Defaults to # $borg_base_directory/.config/borg/keys # borg_keys_directory: /path/to/base/config/keys # Umask to be used for borg create. Defaults to 0077. # umask: 0077 # Maximum seconds to wait for acquiring a repository/cache # lock. Defaults to 1. lock_wait: 5 # Name of the archive. Borg placeholders can be used. See the # output of "borg help placeholders" for details. Defaults to # "{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}". If you specify this # option, you must also specify a prefix in the retention # section to avoid accidental pruning of archives with a # different archive name format. And you should also specify a # prefix in the consistency section as well. # archive_name_format: 'thebeast-{now:%Y-%m-%dT%H:%M:%S.%f}' # Bypass Borg error about a repository that has been moved. # Defaults to false. # relocated_repo_access_is_ok: true # Bypass Borg error about a previously unknown unencrypted # repository. Defaults to false. # unknown_unencrypted_repo_access_is_ok: true # Additional options to pass directly to particular Borg # commands, handy for Borg options that borgmatic does not yet # support natively. Note that borgmatic does not perform any # validation on these options. Running borgmatic with # "--verbosity 2" shows the exact Borg command-line # invocation. # extra_borg_options: # Extra command-line options to pass to "borg init". # init: --extra-option # Extra command-line options to pass to "borg prune". # prune: --extra-option # Extra command-line options to pass to "borg compact". # compact: --extra-option # Extra command-line options to pass to "borg create". # create: --extra-option # Extra command-line options to pass to "borg check". # check: --extra-option # Retention policy for how many backups to keep in each category. See # https://borgbackup.readthedocs.io/en/stable/usage/prune.html for # details. At least one of the "keep" options is required for pruning # to work. To skip pruning entirely, run "borgmatic create" or "check" # without the "prune" action. See borgmatic documentation for details. retention: # Keep all archives within this time interval. # keep_within: 3H # Number of secondly archives to keep. # keep_secondly: 60 # Number of minutely archives to keep. # keep_minutely: 60 # Number of hourly archives to keep. # keep_hourly: 24 # Number of daily archives to keep. keep_daily: 7 # Number of weekly archives to keep. # keep_weekly: 4 # Number of monthly archives to keep. # keep_monthly: 6 # Number of yearly archives to keep. # keep_yearly: 1 # When pruning, only consider archive names starting with this # prefix. Borg placeholders can be used. See the output of # "borg help placeholders" for details. Defaults to # "{hostname}-". Use an empty value to disable the default. # prefix: sourcehostname # Consistency checks to run after backups. See # https://borgbackup.readthedocs.io/en/stable/usage/check.html and # https://borgbackup.readthedocs.io/en/stable/usage/extract.html for # details. # consistency: # List of one or more consistency checks to run: "repository", # "archives", "data", and/or "extract". Defaults to # "repository" and "archives". Set to "disabled" to disable # all consistency checks. "repository" checks the consistency # of the repository, "archives" checks all of the archives, # "data" verifies the integrity of the data within the # archives, and "extract" does an extraction dry-run of the # most recent archive. Note that "data" implies "archives". # checks: # - repository # - archives # Paths to a subset of the repositories in the location # section on which to run consistency checks. Handy in case # some of your repositories are very large, and so running # consistency checks on them would take too long. Defaults to # running consistency checks on all repositories configured in # the location section. # check_repositories: # - user@backupserver:sourcehostname.borg # Restrict the number of checked archives to the last n. # Applies only to the "archives" check. Defaults to checking # all archives. # check_last: 3 # When performing the "archives" check, only consider archive # names starting with this prefix. Borg placeholders can be # used. See the output of "borg help placeholders" for # details. Defaults to "{hostname}-". Use an empty value to # disable the default. # prefix: sourcehostname # Options for customizing borgmatic's own output and logging. # output: # Apply color to console output. Can be overridden with # --no-color command-line flag. Defaults to true. # color: false # Shell commands, scripts, or integrations to execute at various # points during a borgmatic run. IMPORTANT: All provided commands and # scripts are executed with user permissions of borgmatic. Do not # forget to set secure permissions on this configuration file (chmod # 0600) as well as on any script called from a hook (chmod 0700) to # prevent potential shell injection or privilege escalation. hooks: # List of one or more shell commands or scripts to execute # before creating a backup, run once per configuration file. # before_backup: # - echo "Starting a backup." # List of one or more shell commands or scripts to execute # before pruning, run once per configuration file. # before_prune: # - echo "Starting pruning." # List of one or more shell commands or scripts to execute # before compaction, run once per configuration file. # before_compact: # - echo "Starting compaction." # List of one or more shell commands or scripts to execute # before consistency checks, run once per configuration file. # before_check: # - echo "Starting checks." # List of one or more shell commands or scripts to execute # before extracting a backup, run once per configuration file. # before_extract: # - echo "Starting extracting." # List of one or more shell commands or scripts to execute # after creating a backup, run once per configuration file. # after_backup: # - echo "Finished a backup." # List of one or more shell commands or scripts to execute # after compaction, run once per configuration file. # after_compact: # - echo "Finished compaction." # List of one or more shell commands or scripts to execute # after pruning, run once per configuration file. # after_prune: # - echo "Finished pruning." # List of one or more shell commands or scripts to execute # after consistency checks, run once per configuration file. # after_check: # - echo "Finished checks." # List of one or more shell commands or scripts to execute # after extracting a backup, run once per configuration file. # after_extract: # - echo "Finished extracting." # List of one or more shell commands or scripts to execute # when an exception occurs during a "prune", "compact", # "create", or "check" action or an associated before/after # hook. # on_error: # - echo "Error during prune/compact/create/check." # List of one or more shell commands or scripts to execute # before running all actions (if one of them is "create"). # These are collected from all configuration files and then # run once before all of them (prior to all actions). # before_everything: # - echo "Starting actions." # List of one or more shell commands or scripts to execute # after running all actions (if one of them is "create"). # These are collected from all configuration files and then # run once after all of them (after any action). # after_everything: # - echo "Completed actions." # List of one or more PostgreSQL databases to dump before # creating a backup, run once per configuration file. The # database dumps are added to your source directories at # runtime, backed up, and removed afterwards. Requires # pg_dump/pg_dumpall/pg_restore commands. See # https://www.postgresql.org/docs/current/app-pgdump.html and # https://www.postgresql.org/docs/current/libpq-ssl.html for # details. # postgresql_databases: # Database name (required if using this hook). Or # "all" to dump all databases on the host. Note # that using this database hook implicitly enables # both read_special and one_file_system (see # above) to support dump and restore streaming. # - name: users # Database hostname to connect to. Defaults to # connecting via local Unix socket. # hostname: database.example.org # Port to connect to. Defaults to 5432. # port: 5433 # Username with which to connect to the database. # Defaults to the username of the current user. # You probably want to specify the "postgres" # superuser here when the database name is "all". # username: dbuser # Password with which to connect to the database. # Omitting a password will only work if PostgreSQL # is configured to trust the configured username # without a password or you create a ~/.pgpass # file. # password: trustsome1 # Database dump output format. One of "plain", # "custom", "directory", or "tar". Defaults to # "custom" (unlike raw pg_dump). See pg_dump # documentation for details. Note that format is # ignored when the database name is "all". # format: directory # SSL mode to use to connect to the database # server. One of "disable", "allow", "prefer", # "require", "verify-ca" or "verify-full". # Defaults to "disable". # ssl_mode: require # Path to a client certificate. # ssl_cert: /root/.postgresql/postgresql.crt # Path to a private client key. # ssl_key: /root/.postgresql/postgresql.key # Path to a root certificate containing a list of # trusted certificate authorities. # ssl_root_cert: /root/.postgresql/root.crt # Path to a certificate revocation list. # ssl_crl: /root/.postgresql/root.crl # Additional pg_dump/pg_dumpall options to pass # directly to the dump command, without performing # any validation on them. See pg_dump # documentation for details. # options: --role=someone # List of one or more MySQL/MariaDB databases to dump before # creating a backup, run once per configuration file. The # database dumps are added to your source directories at # runtime, backed up, and removed afterwards. Requires # mysqldump/mysql commands (from either MySQL or MariaDB). See # https://dev.mysql.com/doc/refman/8.0/en/mysqldump.html or # https://mariadb.com/kb/en/library/mysqldump/ for details. # mysql_databases: # Database name (required if using this hook). Or # "all" to dump all databases on the host. Note # that using this database hook implicitly enables # both read_special and one_file_system (see # above) to support dump and restore streaming. # - name: users # Database hostname to connect to. Defaults to # connecting via local Unix socket. # hostname: database.example.org # Port to connect to. Defaults to 3306. # port: 3307 # Username with which to connect to the database. # Defaults to the username of the current user. # username: dbuser # Password with which to connect to the database. # Omitting a password will only work if MySQL is # configured to trust the configured username # without a password. # password: trustsome1 # Additional mysql options to pass directly to # the mysql command that lists available # databases, without performing any validation on # them. See mysql documentation for details. # list_options: --defaults-extra-file=my.cnf # Additional mysqldump options to pass directly to # the dump command, without performing any # validation on them. See mysqldump documentation # for details. # options: --skip-comments # List of one or more MongoDB databases to dump before # creating a backup, run once per configuration file. The # database dumps are added to your source directories at # runtime, backed up, and removed afterwards. Requires # mongodump/mongorestore commands. See # https://docs.mongodb.com/database-tools/mongodump/ and # https://docs.mongodb.com/database-tools/mongorestore/ for # details. # mongodb_databases: # Database name (required if using this hook). Or # "all" to dump all databases on the host. Note # that using this database hook implicitly enables # both read_special and one_file_system (see # above) to support dump and restore streaming. # - name: users # Database hostname to connect to. Defaults to # connecting to localhost. # hostname: database.example.org # Port to connect to. Defaults to 27017. # port: 27018 # Username with which to connect to the database. # Skip it if no authentication is needed. # username: dbuser # Password with which to connect to the database. # Skip it if no authentication is needed. # password: trustsome1 # Authentication database where the specified # username exists. If no authentication database # is specified, the database provided in "name" # is used. If "name" is "all", the "admin" # database is used. # authentication_database: admin # Database dump output format. One of "archive", # or "directory". Defaults to "archive". See # mongodump documentation for details. Note that # format is ignored when the database name is # "all". # format: directory # Additional mongodump options to pass # directly to the dump command, without performing # any validation on them. See mongodump # documentation for details. # options: --role=someone # Healthchecks ping URL or UUID to notify when a backup # begins, ends, or errors. Create an account at # https://healthchecks.io if you'd like to use this service. # See borgmatic monitoring documentation for details. # healthchecks: https://hc-ping.com/your-uuid-here # Cronitor ping URL to notify when a backup begins, ends, or # errors. Create an account at https://cronitor.io if you'd # like to use this service. See borgmatic monitoring # documentation for details. cronitor: https://cronitor.link/*********************** # PagerDuty integration key used to notify PagerDuty when a # backup errors. Create an account at # https://www.pagerduty.com/ if you'd like to use this # service. See borgmatic monitoring documentation for details. # pagerduty: a177cad45bd374409f78906a810a3074 # Cronhub ping URL to notify when a backup begins, ends, or # errors. Create an account at https://cronhub.io if you'd # like to use this service. See borgmatic monitoring # documentation for details. # cronhub: https://cronhub.io/start/1f5e3410-254c-11e8-b61d-55875966d01 # Umask used when executing hooks. Defaults to the umask that # borgmatic is run with. # umask: 0077 ``` </details>
witten commented 2022-03-07 23:07:56 +00:00
Owner
Copy link

Thank you for filing this and providing the helpful log and configuration info! So here's what I think is going on:

  • borgmatic is invoking borg compact --dry-run
  • Borg is responding by displaying its general command usage information and exiting with an error, which is Borg's way of saying "--dry-run is not supported for the compact sub-command." It's also possible that some of Borg's usage is getting truncated.

So I think the fix here is for borgmatic to be smart enough to skip calling borg compact altogether when a dry run is requested. In the interim as a work-around, you can try invoking borgmatic create --dry-run which will skip compacting.

Thank you for filing this and providing the helpful log and configuration info! So here's what I think is going on: * borgmatic is invoking `borg compact --dry-run` * Borg is responding by displaying its general command usage information and exiting with an error, which is Borg's way of saying "`--dry-run` is not supported for the `compact` sub-command." It's also possible that some of Borg's usage is getting truncated. So I think the fix here is for borgmatic to be smart enough to *skip* calling `borg compact` altogether when a dry run is requested. In the interim as a work-around, you can try invoking `borgmatic create --dry-run` which will skip compacting.
witten commented 2022-03-08 05:47:08 +00:00
Owner
Copy link

This has been fixed in borgmatic master. Thanks again for bringing it to my attention. This fix will be part of the next borgmatic release!

This has been fixed in borgmatic master. Thanks again for bringing it to my attention. This fix will be part of the next borgmatic release!
witten commented 2022-03-14 20:48:14 +00:00
Owner
Copy link

Released in borgmatic 1.5.24.

Released in borgmatic 1.5.24.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
parallel-borgmatic
command-hook-steps
2.1.7
2.1.6
2.1.5
2.1.4
2.1.3
2.1.2
2.1.1
2.1.0
2.0.13
2.0.12
2.0.11
2.0.10
2.0.9
2.0.8
2.0.7
2.0.6
2.0.5
2.0.4
2.0.3
2.0.2
2.0.1
2.0.0
1.9.14
1.9.13
1.9.12
1.9.11
1.9.10
1.9.9
1.9.8
1.9.7
1.9.6
1.9.5
1.9.4
1.9.3
1.9.2
1.9.1
1.9.0
1.8.14
1.8.13
1.8.12
1.8.11
1.8.10
1.8.9
1.8.8
1.8.7
1.8.6
1.8.5
1.8.4
1.8.3
1.8.2
1.8.1
1.8.0
1.7.15
1.7.14
1.7.13
1.7.12
1.7.11
1.7.10
1.7.9
1.7.8
1.7.7
1.7.6
1.7.5
1.7.4
1.7.3
1.7.2
1.7.1
1.7.0
1.6.6
1.6.5
1.6.4
1.6.3
1.6.2
1.6.1
1.6.0
1.5.24
1.5.23
1.5.22
1.5.21
1.5.20
1.5.19
1.5.18
1.5.17
1.5.16
1.5.15
1.5.14
1.5.13
1.5.12
1.5.11
1.5.10
1.5.9
1.5.8
1.5.7
1.5.7.dev0
1.5.6
1.5.5
1.5.4
1.5.3
1.5.2
1.5.1
1.5.0
1.4.22
1.4.21
1.4.20
1.4.19
1.4.18
1.4.17
1.4.16
1.4.15
1.4.14
1.4.13
1.4.12
1.4.11
1.4.10
1.4.9
1.4.8
1.4.7
1.4.6
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0
1.3.26
1.3.25
1.3.24
1.3.23
1.3.22
1.3.21
1.3.20
1.3.19
1.3.18
1.3.17
1.3.16
1.3.15
1.3.14
1.3.13
1.3.12
1.3.11
1.3.10
1.3.9
1.3.8
1.3.7
1.3.6
1.3.5
1.3.4
1.3.3
1.3.2
1.3.1
1.3.0
1.2.18
1.2.17
1.2.16
1.2.15
1.2.14
1.2.13
1.2.12
1.2.11
1.2.10
1.2.9
1.2.8
1.2.7
1.2.6
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.15
1.1.14
1.1.13
1.1.12
1.1.11
1.1.10
1.1.9
1.1.7
1.1.6
1.1.5
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.3
1.0.2
1.0.1
1.0.0
0.1.8
0.1.7
0.1.6
0.1.5
0.1.4
0.1.3
0.1.2
0.1.1
0.1.0
0.0.7
0.0.6
0.0.5
0.0.4
0.0.3
0.0.2
Labels
Clear labels   
blocked

   
breaking

   
bug

   
data loss

   
design finalized

   
good first issue

   
new feature area

   
question / support

   
security

   
waiting for response
No labels
blocked
breaking
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
borgmatic-collective/borgmatic#507
No description provided.