Compare commits

...

680 Commits
master ... main

Author SHA1 Message Date
Dan Helfman f21a2c06e3 Add documentation link to Loki dashboard for borgmatic (#843).
build / test (push) Successful in 6m37s Details
build / docs (push) Successful in 1m29s Details
2024-03-22 11:25:33 -07:00
Dan Helfman 36d10fecb1 Upgrade black in test requirements.
build / test (push) Successful in 5m56s Details
build / docs (push) Successful in 1m28s Details
2024-03-20 12:01:24 -07:00
Dan Helfman ecf5a7e294 When a command hook exits with a soft failure, ping the log and finish states for any configured monitoring hooks (#842).
build / test (push) Successful in 6m0s Details
build / docs (push) Successful in 1m28s Details
2024-03-18 23:15:28 -07:00
Dan Helfman 893fbcf9ff Add documentation about backing up containerized databases by configuring borgmatic to exec into a container to run a dump command.
build / test (push) Successful in 5m57s Details
build / docs (push) Successful in 1m29s Details
2024-03-14 18:00:52 -07:00
Dan Helfman f8f6560502 Fix handling of the NO_COLOR environment variable to ignore an empty value (#835).
build / test (push) Successful in 6m0s Details
build / docs (push) Successful in 1m29s Details
2024-03-13 09:35:19 -07:00
Dan Helfman 8c301ba688 Bump version for release.
build / test (push) Successful in 5m15s Details
build / docs (push) Successful in 1m5s Details
2024-03-11 13:27:08 -07:00
Dan Helfman 035e96156a Add an "access_token" option to the ntfy monitoring hook for authenticating without username/password (#811).
build / test (push) Successful in 5m2s Details
build / docs (push) Successful in 1m10s Details
2024-03-11 12:48:58 -07:00
Dan Helfman a08c7fc77a When running the "rcreate" action and the repository already exists but with a different encryption mode than requested, error (#840).
build / test (push) Successful in 4m55s Details
build / docs (push) Successful in 1m0s Details
2024-03-11 11:24:36 -07:00
Dan Helfman cf9e387811 Document a potentially breaking shell quoting edge case within error hooks (#839).
build / test (push) Successful in 6m43s Details
build / docs (push) Successful in 1m44s Details
2024-03-11 10:42:51 -07:00
Dan Helfman e37224606a Clarify dev-CI parity.
build / test (push) Successful in 5m12s Details
build / docs (push) Successful in 1m13s Details
2024-03-10 19:14:18 -07:00
Dan Helfman 9647301b99 Add log sending for the Apprise logging hook, enabled by default.
build / test (push) Successful in 7m4s Details
build / docs (push) Successful in 1m53s Details
2024-03-10 16:18:49 -07:00
Dan Helfman a0e5dbff96 Remove list of command in Bash script.
build / test (push) Successful in 5m5s Details
build / docs (push) Successful in 1m11s Details
2024-03-06 21:24:44 -08:00
Dan Helfman 86117edccf Remove build.torsion.org references from documentation.
build / test (push) Successful in 7m7s Details
build / docs (push) Successful in 1m56s Details
2024-03-06 20:01:32 -08:00
Dan Helfman 440f3eeb63 Remove Drone configuration/tests.
build / test (push) Successful in 5m1s Details
build / docs (push) Successful in 1m2s Details
2024-03-06 19:04:29 -08:00
Dan Helfman 181051eae1 Add new build server to NEWS.
build / test (push) Successful in 4m45s Details
build / docs (push) Successful in 2m5s Details
2024-03-06 18:52:27 -08:00
Dan Helfman ec0ee971ed Attempt to use secrets.
build / test (push) Successful in 4m46s Details
build / docs (push) Failing after 4s Details
2024-03-06 18:38:45 -08:00
Dan Helfman b83ffa0cf6 Attempt to fix trigger.
build / test (push) Successful in 4m45s Details
build / docs (push) Failing after 4s Details
2024-03-06 16:53:41 -08:00
Dan Helfman cf88665d37 Fix typo.
build / docs (push) Blocked by required conditions Details
build / test (push) Has been cancelled Details
2024-03-06 16:52:33 -08:00
Dan Helfman b233adba63 Fix build? 2024-03-06 16:51:49 -08:00
Dan Helfman 018f5e3315 Merge workflows, since Gitea doesn't yet support workflow_run. 2024-03-06 16:49:50 -08:00
Dan Helfman 284f26b49d Only run tests on pushes to main branch. 2024-03-06 16:40:39 -08:00
Dan Helfman 11b437794e Attempt to build documentation.
test / test (push) Successful in 4m53s Details
2024-03-06 16:38:34 -08:00
Dan Helfman 0665b50d57 Fixed debugging.
test / test (push) Successful in 4m40s Details
2024-03-06 16:17:12 -08:00
Dan Helfman 0586b80e5b More debugging.
test / test (push) Failing after 4m44s Details
2024-03-06 15:53:30 -08:00
Dan Helfman 272a7b4866 Actually kill other containers after tests finish.
test / test (push) Successful in 4m45s Details
2024-03-06 15:41:03 -08:00
Dan Helfman 98d4a59459 Another iteration.
test / test (push) Successful in 4m46s Details
2024-03-06 15:29:56 -08:00
Dan Helfman 744139cf97 Disable progress.
test / test (push) Has been cancelled Details
2024-03-06 15:21:45 -08:00
Dan Helfman 1339509e9b Flag order apparently matters to Docker Compose.
test / test (push) Waiting to run Details
2024-03-06 14:55:55 -08:00
Dan Helfman e14f61415b Fix spew in test script.
test / test (push) Failing after 2s Details
2024-03-06 14:54:53 -08:00
Dan Helfman 98cf8f7e20 Another try at exiting tests properly.
test / test (push) Has been cancelled Details
2024-03-06 14:42:06 -08:00
Dan Helfman 5f16b64639 Attempt to exit test containers on tests exit while also showing test output.
test / test (push) Failing after 3s Details
2024-03-06 14:39:23 -08:00
Dan Helfman fe62a81151 Add missing service name to test scrits.
test / test (push) Successful in 4m54s Details
2024-03-06 14:32:26 -08:00
Dan Helfman 585b1573ae Attempt to make containers stop after tests run.
test / test (push) Failing after 7s Details
2024-03-06 14:30:49 -08:00
Dan Helfman 141ba2771d Attempt to fix and debug read-only filesystem issue at build.
test / test (push) Has been cancelled Details
2024-03-06 11:10:20 -08:00
Dan Helfman a527f76d08 Add back checkout now that NodeJS is installed on the host.
continuous-integration/drone/push Build was killed Details
test / test (push) Has been cancelled Details
2024-03-06 08:49:53 -08:00
Dan Helfman a97c68b4c8 Debugging ls.
test / test (push) Failing after 0s Details
continuous-integration/drone/push Build was killed Details
2024-03-06 08:35:50 -08:00
Dan Helfman ef07005a75 Remove duplicative(?) checkout step.
test / test (push) Failing after 0s Details
continuous-integration/drone/push Build was killed Details
2024-03-06 08:35:05 -08:00
Dan Helfman 43c7c3b6be First attempt at using Gitea Actions to run tests.
test / test (push) Failing after 15s Details
continuous-integration/drone/push Build was killed Details
2024-03-06 08:32:55 -08:00
Dan Helfman 2f6ad9d173 Add NO_COLOR support to NEWS (#835).
continuous-integration/drone/push Build was killed Details
2024-03-04 13:49:54 -08:00
Dan Helfman 16bc0de3fb
Support for NO_COLOR environment variable (#835).
Merge pull request #82 from shivansh02/feature/support-no-color-env-var
2024-03-04 13:46:09 -08:00
shivansh02 458d157e62 NO_COLOR set to any value returns false 2024-03-05 00:15:52 +05:30
shivansh02 40c3a28620 support for NO_COLOR env var 2024-03-04 18:21:28 +05:30
Dan Helfman 60107f1ee8 Add custom dump/restore command options for MySQL and MariaDB (#311).
continuous-integration/drone/push Build was killed Details
2024-03-03 14:32:49 -08:00
Dan Helfman a1153a21fa
Custom dump command options for MySQL and MariaDB.
Merge pull request #81 from shivansh02/feature/custom-dump-restore-commands-mysql
2024-03-03 14:27:14 -08:00
shivansh02 b6cb7da98e custom dump commands for mariadb 2024-03-04 00:24:22 +05:30
shivansh02 9e3d19a406 custom commands escaped 2024-03-03 23:31:02 +05:30
shivansh02 2b755d8ade custom show command for mysql and schema description 2024-03-03 23:15:07 +05:30
shivansh02 925f99cfef custom dump command for mysql 2024-03-03 03:47:02 +05:30
Dan Helfman c9f20eb260 Fix "--override" values containing deprecated section headers not actually overriding configuration options under deprecated section headers (#829). 2024-02-15 21:12:42 -08:00
Dan Helfman f4744826fe When the "--json" flag is given, suppress console escape codes so as not to interfere with JSON output (#827).
continuous-integration/drone/push Build is passing Details
2024-02-11 17:44:43 -08:00
Dan Helfman 5586aab967 Clarify documentation about restoring a database: borgmatic does not create the database upon restore.
continuous-integration/drone/push Build is passing Details
2024-02-09 15:35:29 -08:00
Dan Helfman 6fa5dff79b Fix broken escaping logic for "pg_dump_command" (#822) + bonus shell injection fixes.
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/tag Build is passing Details
2024-01-31 10:53:32 -08:00
Dan Helfman 75d11aa9cd Pass the PostgreSQL "PGSSLMODE" environment variable through to Borg (#370).
continuous-integration/drone/push Build is passing Details
2024-01-25 14:18:01 -08:00
Dan Helfman ad1d104d65 Fix broken repository detection in the "rcreate" action with Borg 1.4 (#820).
continuous-integration/drone/push Build is passing Details
2024-01-24 15:45:51 -08:00
Dan Helfman 009062128d Remove Python 3.8+ restriction, as only Python 3.8+ is supported. 2024-01-22 09:41:43 -08:00
Dan Helfman e9813d2539 Allow the "--repository" flag to match across multiple configuration files (#818). 2024-01-21 18:25:44 -08:00
Dan Helfman f9998b50e8 Rephrase documentation and link to docs on exit codes feature (#798).
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/tag Build is passing Details
2024-01-21 14:47:21 -08:00
Dan Helfman 5f921a7f80 Add documentation heading (#798).
continuous-integration/drone/push Build is passing Details
2024-01-21 11:48:23 -08:00
Dan Helfman abf2b3a8c7 Elevate specific Borg warnings to errors or squash errors to warnings (#798).
continuous-integration/drone/push Build is passing Details
2024-01-21 11:34:40 -08:00
Dan Helfman 34f3c2bb16 Clarify "--override" command-line help (#814)
continuous-integration/drone/push Build is passing Details
2024-01-19 11:55:00 -08:00
Dan Helfman 4d79f582df Fix a traceback when providing an invalid "--override" value for a list option (#814).
continuous-integration/drone/push Build is passing Details
2024-01-18 10:39:40 -08:00
Dan Helfman 63198088c4 Store included configuration files within each backup archive in support of the "config bootstrap" action (#736).
continuous-integration/drone/push Build is passing Details
2024-01-09 13:47:20 -08:00
Dan Helfman 3c22a8ec16 Prevent various shell injection attacks (#810).
continuous-integration/drone/push Build is passing Details
2024-01-07 10:21:49 -08:00
Dan Helfman ca49109ce7 Bump version for release.
continuous-integration/drone/tag Build is passing Details
continuous-integration/drone/push Build is passing Details
2024-01-03 10:08:05 -08:00
Dan Helfman 6a7f71f92f Clarify prune action help concerning running compact afterwards (#808).
continuous-integration/drone/push Build is passing Details
2024-01-03 10:03:35 -08:00
Dan Helfman 5f3dc1cfb0 Stream SQLite databases directly to Borg instead of dumping to an intermediate file (#807).
continuous-integration/drone/push Build is passing Details
2023-12-31 11:07:59 -08:00
Dan Helfman f2023aed22 Fix typo.
continuous-integration/drone/push Build is passing Details
2023-12-30 15:48:55 -08:00
Dan Helfman a03c2744e5 Update docs/how-to/provide-your-passwords.md (#805).
continuous-integration/drone/push Build is passing Details
Reviewed-on: borgmatic-collective/borgmatic#805
Reviewed-by: Dan Helfman <witten@torsion.org>
2023-12-30 23:48:32 +00:00
axel simon 4176532317 Update docs/how-to/provide-your-passwords.md
Provide an explanation of encryption_passcommand.
Also, adjust headers for consistency.
2023-12-30 23:45:56 +00:00
Dan Helfman 9d6025e902 Validate the configured action names in the "skip_actions" option (#804).
continuous-integration/drone/push Build is passing Details
2023-12-28 20:07:57 -08:00
Dan Helfman cf739bc997 The "check --force" flag now runs checks even if "check" is in "skip_actions" (#802).
continuous-integration/drone/push Build is passing Details
2023-12-28 10:22:48 -08:00
Dan Helfman 84823dfb91 Clarify constants/placeholders interaction and improve examples (#763).
continuous-integration/drone/push Build is passing Details
2023-12-24 11:18:17 -08:00
Dan Helfman 20cf0f7089 Add an "--ssh-command" flag to the "config bootstrap" action (#767).
continuous-integration/drone/push Build is passing Details
2023-12-24 10:33:55 -08:00
Dan Helfman 67af0f5734 Document limitation with constant interpolation at the start of a value (#741).
continuous-integration/drone/push Build is passing Details
2023-12-22 21:39:44 -08:00
Dan Helfman e80e0a253c Add configured repository labels to the JSON output for all actions (#800). 2023-12-20 09:17:41 -08:00
Dan Helfman 72587a3b72 Merge branch 'main' of ssh://projects.torsion.org:3022/borgmatic-collective/borgmatic
continuous-integration/drone/push Build is passing Details
2023-12-04 11:17:59 -08:00
Dan Helfman 8b49a59aff Fix a traceback when the "repositories" option contains both strings and key/value pairs (#794). 2023-12-04 11:17:13 -08:00
Dan Helfman e120dff9ff Add debug message that logs borg version for every config (#714).
continuous-integration/drone/push Build is passing Details
Reviewed-on: borgmatic-collective/borgmatic#792
2023-11-25 03:59:40 +00:00
Tobias Hodapp 257678b66f Fixed borg -> Borg 2023-11-24 18:47:37 +01:00
Tobias Hodapp 422c5e32f4 Added debug message that logs borg version for every config 2023-11-23 11:46:10 +01:00
Dan Helfman c34ad7dde7 Update documentation about configuration includes and constants (#745).
continuous-integration/drone/push Build is passing Details
2023-11-19 21:22:10 -08:00
Dan Helfman fdb353d358 Bump version for release.
continuous-integration/drone/tag Build is passing Details
2023-11-19 21:14:56 -08:00
Dan Helfman 3b99f7c75a Constants support includes and command-line overrides (#745, #782) 2023-11-19 21:13:35 -08:00
Dan Helfman 8b9abc6cf8 Documentation clarifications (#791).
continuous-integration/drone/push Build is passing Details
2023-11-15 09:05:52 -08:00
Dan Helfman da034c316a Add another mention of "skip_actions" to the docs (#701).
continuous-integration/drone/push Build is passing Details
2023-11-08 18:22:17 -08:00
Dan Helfman 08d01d8bcd Documentation formatting.
continuous-integration/drone/push Build is passing Details
2023-11-08 17:57:31 -08:00
Dan Helfman eef69e23ee Document the possible units of times for a configured check frequency.
continuous-integration/drone/push Build is passing Details
2023-11-08 17:53:59 -08:00
Dan Helfman 26bb54a9dd Remove broken link in documentation (#786).
continuous-integration/drone/push Build is passing Details
2023-11-08 10:26:09 -08:00
Dan Helfman 715e2ac127 Add test support for Python 3.12.
continuous-integration/drone/push Build is passing Details
2023-11-07 10:17:55 -08:00
Dan Helfman f39cea4abf Remove additional Python 3.7-isms (#784).
continuous-integration/drone/push Build is passing Details
2023-11-07 10:17:16 -08:00
Dan Helfman 22101bdd49 Drop support for Python 3.7, which has been end-of-lifed (#784).
continuous-integration/drone/push Build is passing Details
2023-11-07 10:11:29 -08:00
Dan Helfman 13cf863d89 Fix tests (#783).
continuous-integration/drone/push Build is passing Details
2023-11-07 10:09:31 -08:00
Dan Helfman dcf25fa041 Upgrade ruamel.yaml dependency to support version 0.18.x (#783).
continuous-integration/drone/push Build is failing Details
2023-11-07 10:00:13 -08:00
Dan Helfman 12b75f9075 Update documentation about logging changes from version 1.8.3 (#665).
continuous-integration/drone/push Build is passing Details
2023-11-06 21:13:46 -08:00
Dan Helfman 9baf06a2f7
Fix typo.
continuous-integration/drone/push Build is passing Details
Typo
2023-11-04 08:56:39 -07:00
tdltdc 56302e22cd
Typo 2023-11-04 15:05:19 +01:00
Dan Helfman 6cc93c4eb9 Fix environment variable interpolation within configured repository paths (#782).
continuous-integration/drone/push Build is passing Details
2023-11-03 21:16:04 -07:00
Dan Helfman 2da43239f6 Fix docs: minor typos (#781).
continuous-integration/drone/push Build is passing Details
Reviewed-on: borgmatic-collective/borgmatic#781
2023-11-03 00:59:29 +00:00
debuglevel 4beef36d3c Update docs/how-to/inspect-your-backups.md 2023-11-02 23:14:31 +00:00
debuglevel eacfbd742b Typo 2023-11-02 23:13:45 +00:00
debuglevel 82a85986b6 Typo 2023-11-02 22:57:46 +00:00
Dan Helfman ef448e2dd1 Add a "skip_actions" option to skip running particular actions (#701).
continuous-integration/drone/push Build is passing Details
2023-10-31 21:54:41 -07:00
Dan Helfman c3efe1b90e Only parse "--override" values as complex data types when they're for options of those types (#779).
continuous-integration/drone/push Build is passing Details
2023-10-29 19:02:28 -07:00
Dan Helfman d85c1ee216 Correct changelog addition (#779).
continuous-integration/drone/push Build is passing Details
2023-10-29 16:25:40 -07:00
Dan Helfman b47088067c Add a "--match-archives" flag to the "check" action (#779).
continuous-integration/drone/push Build is passing Details
2023-10-29 16:22:39 -07:00
Dan Helfman c5732aa4fc Fix home page CSS layout to prevent overflow at certain window widths (#777).
continuous-integration/drone/push Build is passing Details
2023-10-27 14:12:35 -07:00
Dan Helfman a0323d9d6c Bump version for release.
continuous-integration/drone/tag Build is passing Details
continuous-integration/drone/push Build is passing Details
2023-10-26 22:20:26 -07:00
Dan Helfman 8ad7b473f1 When an archive filter causes no matching archives for the "rlist" or "info" actions, warn (#748).
continuous-integration/drone/push Build is passing Details
2023-10-26 22:12:13 -07:00
Dan Helfman 895a0ccb3c Upgrade to tox 4. (Now a minimum requirement.)
continuous-integration/drone/push Build is passing Details
2023-10-23 17:39:27 -07:00
Dan Helfman 257ab77bea Disallow the "--dry-run" flag with the "borg" action (#774).
continuous-integration/drone/push Build is passing Details
2023-10-23 17:23:04 -07:00
Dan Helfman dccaa4014b
Update systemd service example with better filesystem protection options.
continuous-integration/drone/push Build is passing Details
Merge pull request #78 from Alphix/update-systemd-service
2023-10-15 08:59:39 -07:00
David Härdeman 2f3c0bec5b Update systemd .service example
First, ProtectSystem=strict will make the entire file system hierarchy (except
/dev, /proc/ and /sys) read-only, so separate ReadOnlyPaths= is not necessary.

Second, ProtectHome=tmpfs will not just mount an empty tmpfs on /root, but also
on /home and /run/user. As it's likely quite common to want to backup /home,
this seems like a footgun.

Finally, it's quite likely that borgbackup will want access to root's SSH keys
in order to connect to remote backup servers.

Note that all these options are commented out by default, so this is more of
a documentation change than any real change in functionality.
2023-10-15 11:30:11 +02:00
Dan Helfman 487d8ffd32 Fix normalization of deprecated sections to support empty sections without erroring (#771).
continuous-integration/drone/push Build is passing Details
2023-10-14 13:04:18 -07:00
Dan Helfman 30523a7c89 Update home page example of Healthchecks configuration not to use deprecated config.
continuous-integration/drone/push Build is passing Details
2023-10-11 12:56:21 -07:00
Dan Helfman 77b1907d03 Update Healthchecks deprecation warning message for clarity.
continuous-integration/drone/push Build is passing Details
2023-10-11 12:17:57 -07:00
Dan Helfman 09594c85bf Be more explicit in documentation that you don't have to use an environment variable for passphrases.
continuous-integration/drone/push Build is passing Details
2023-10-10 09:34:55 -07:00
Dan Helfman e07efdf68f Add documentation note about using includes for specifying passphrases (#769).
continuous-integration/drone/push Build is passing Details
2023-10-10 09:16:58 -07:00
Dan Helfman 1fed44f905 Add documentation note about sudo and sudoers "secure_path" option (#757). 2023-10-09 14:15:54 -07:00
Dan Helfman c687dafdd2 Fix a traceback when an invalid command-line flag or action is used (#768).
continuous-integration/drone/push Build is passing Details
2023-10-06 21:00:23 -07:00
Dan Helfman 3eff2c4248 Add Grafana Loki badge to integrations documentation.
continuous-integration/drone/push Build is passing Details
2023-10-05 09:06:06 -07:00
Dan Helfman d94fdb6faf Add apprise logo to integrations in readme (#715).
continuous-integration/drone/push Build is passing Details
Reviewed-on: borgmatic-collective/borgmatic#765
2023-10-05 15:51:04 +00:00
Pim Kunis a83282faf0 add apprise logo to integrations in readme 2023-10-05 15:38:32 +02:00
Dan Helfman e7169f6fb2 Upgrade certifi test dependency to fix security alert.
continuous-integration/drone/push Build is passing Details
2023-10-04 22:59:15 -07:00
Dan Helfman 9587fc2366 Update Apprise documentation to use sudo for pipx install (#715).
continuous-integration/drone/push Build is passing Details
2023-10-04 22:54:11 -07:00
Dan Helfman 5f06884d5a Fix Apprise/PyYAML end-to-end test breakage (#715).
continuous-integration/drone/push Build encountered an error Details
2023-10-04 22:51:05 -07:00
Dan Helfman f011431463 Apprise hook documentation (#715).
continuous-integration/drone/push Build encountered an error Details
2023-10-04 19:23:53 -07:00
Dan Helfman 9e14f209f1 Merge branch 'main' of ssh://projects.torsion.org:3022/borgmatic-collective/borgmatic
continuous-integration/drone/push Build is failing Details
2023-10-04 14:58:48 -07:00
Dan Helfman 9d34d2eec5 Support for Apprise (#759).
continuous-integration/drone/push Build is failing Details
Reviewed-on: borgmatic-collective/borgmatic#759
2023-10-04 21:58:20 +00:00
Pim Kunis 7a9625cd44 fix PR comments 2023-10-04 13:19:40 +02:00
Pim Kunis 4763c323d0 add unit tests for apprise hook 2023-10-01 16:59:59 +00:00
Pim Kunis eaa22be3db fix PR comments 2023-10-01 16:59:59 +00:00
Pim Kunis a587e207f9 pin Apprise dependencies for test requirements 2023-10-01 16:59:59 +00:00
Pim Kunis db8079b699 fix typo in setup.py
handle if apprise cannot be imported
2023-10-01 16:59:59 +00:00
Pim Kunis 5a989826a1 convert map to list for apprise function call
fix apprise config schema
remove apprise from required dependencies
2023-10-01 16:59:59 +00:00
Pim Kunis 21f4266273 incorporate PR review comments 2023-10-01 16:59:59 +00:00
Pim Kunis e7252c7545 remove comments about tags 2023-10-01 16:59:59 +00:00
Pim Kunis 86011c8418 default apprise notify type per borgmatic state 2023-10-01 16:59:59 +00:00
Pim Kunis f3295ccb4a add support for apprise 2023-10-01 16:59:59 +00:00
Dan Helfman cacb81f086 Bump version for release. 2023-09-30 13:37:10 -07:00
Dan Helfman 06c2154e6a Build docs regardless of Drone "event" (push, etc.).
continuous-integration/drone/push Build was killed Details
continuous-integration/drone Build is passing Details
2023-09-29 19:49:09 -07:00
Dan Helfman ac1e1a9407 Simplify logging logic (#665).
continuous-integration/drone/push Build was killed Details
continuous-integration/drone Build is passing Details
2023-09-29 14:16:47 -07:00
Dan Helfman 10933fd55b Fix for borgmatic not stopping Borg immediately when the user presses ctrl-C (#761).
continuous-integration/drone/push Build was killed Details
2023-09-27 08:52:00 -07:00
Dan Helfman af422ad705 Add documentation note about upgrading multiple pipx installations of borgmatic.
continuous-integration/drone/push Build was killed Details
2023-09-18 13:46:41 -07:00
Dan Helfman d9d35491fb Fix tense typo.
continuous-integration/drone/push Build is passing Details
2023-09-17 23:25:57 -07:00
Dan Helfman b540e63c0e Updated documentation so "sudo borgmatic" works for pipx borgmatic installations (#757).
continuous-integration/drone Build was killed Details
2023-09-17 22:46:33 -07:00
Dan Helfman 5a56208922 Fix documentation typo.
continuous-integration/drone/push Build is passing Details
2023-09-15 10:18:35 -07:00
Dan Helfman 5912769273 Fix error handling to log command output as one record per line (#754).
continuous-integration/drone/push Build is passing Details
2023-09-14 21:10:52 -07:00
Dan Helfman bac2aabe66 Attempt to unbreak ticket filing.
continuous-integration/drone/push Build is passing Details
2023-09-12 09:50:38 -07:00
Dan Helfman 9f3328781b When "archive_name_format" is not set, filter archives using the default archive name format (#753).
continuous-integration/drone/push Build is passing Details
2023-09-06 23:13:40 -07:00
Dan Helfman 0205748db8 Update documentation to recommend installing/upgrading borgmatic with pipx instead of pip.
continuous-integration/drone/push Build is passing Details
2023-09-04 16:25:10 -07:00
Dan Helfman d0a8251ad2 Add borgmatic version introducing Loki hook to docs (#743).
continuous-integration/drone/push Build is passing Details
2023-08-27 20:30:13 -07:00
Dan Helfman 32019ea8f3 Add documentation for Grafana Loki hook (#743).
continuous-integration/drone/push Build is passing Details
2023-08-25 10:52:00 -07:00
Dan Helfman fa9a061033 Merge branch 'main' of ssh://projects.torsion.org:3022/borgmatic-collective/borgmatic 2023-08-25 09:29:02 -07:00
Dan Helfman b3d2560563 Added support for grafana loki (#743).
continuous-integration/drone/push Build is passing Details
Reviewed-on: borgmatic-collective/borgmatic#747
2023-08-25 16:28:19 +00:00
Dan Helfman 4b4f56da42 Fix another database -> data source instance (#685). 2023-08-24 14:00:29 -07:00
Dan Helfman b96d1898f7 Prep work for eventual container-dumping hooks: Generalize internal database hook "API" (#685).
continuous-integration/drone/push Build is passing Details
2023-08-24 13:50:10 -07:00
Tobias Hodapp 099a712e53 Added more documentation to the test
Split tests to integration tests
2023-08-24 13:17:42 +02:00
Tobias Hodapp 9e2674ea5a Added unit tests
Removed useless dry run check
2023-08-23 17:17:23 +02:00
Tobias Hodapp 7e419ec995 Fixed spelling errors
Added documentation
Added log messages for dry run
2023-08-22 23:03:14 +02:00
Tobias Hodapp a3edf757ee Added changes of formatting tools 2023-08-22 13:40:05 +02:00
Tobias Hodapp e576403b64 Added support for grafana loki 2023-08-22 03:13:39 +02:00
Dan Helfman 7313430178 Make warning about sections a little more explicit (#721).
continuous-integration/drone/push Build is passing Details
2023-08-19 22:51:20 -07:00
Dan Helfman 962daaa8b9 Bump version for release.
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/tag Build is passing Details
2023-08-14 12:54:38 -07:00
Dan Helfman cd51e9c1ea Fix for database "restore" action not actually restore anything (#738).
continuous-integration/drone/push Build is passing Details
2023-08-14 12:43:21 -07:00
Dan Helfman 6dca7c1c15 Add "key export" action to export a copy of the repository key (#345).
continuous-integration/drone/push Build is passing Details
2023-08-07 12:28:39 -07:00
Dan Helfman fd8c56c6be Add brief source code reference documentation.
continuous-integration/drone/push Build is passing Details
2023-08-06 23:44:31 -07:00
Dan Helfman 065057c966
Fix typos.
continuous-integration/drone/push Build is passing Details
Merge pull request #77 from hop/main
2023-08-05 17:19:57 -07:00
Christoph Schindler c04517f843 Fix typos. 2023-08-06 02:16:31 +02:00
Dan Helfman 5d80c366fb Fix "borg create" flags/argument interleaving.
continuous-integration/drone/push Build is passing Details
2023-08-04 20:02:09 -07:00
Dan Helfman 193dd93de2 Fork a MariaDB database hook from the MySQL database hook (#727).
continuous-integration/drone/push Build is passing Details
2023-08-04 13:22:44 -07:00
Dan Helfman 8a94b9e2f1 Mention "store_config_files" in docs (#725).
continuous-integration/drone/push Build is passing Details
2023-08-03 22:11:02 -07:00
Dan Helfman 3c8f6040e2 Bump version for release.
continuous-integration/drone/tag Build is passing Details
continuous-integration/drone/push Build is passing Details
2023-08-03 21:56:26 -07:00
Dan Helfman 14e2a6b89d Omit "--glob-archives" or "--match-archives" Borg flag when its value would be "*" (#734).
continuous-integration/drone/push Build is passing Details
2023-08-02 10:23:22 -07:00
Dan Helfman e607de7df1 Include multiple configuration files with a single "!include" (#732).
continuous-integration/drone/push Build is passing Details
Reviewed-on: borgmatic-collective/borgmatic#737
2023-08-02 04:46:33 +00:00
Dan Helfman e9bd5f4e1d Add documentation and NEWS link (#732).
continuous-integration/drone/push Build is passing Details
2023-08-01 21:12:49 -07:00
Dan Helfman 175003ff9b Additional test coverage (#732).
continuous-integration/drone/push Build is passing Details
2023-08-01 19:45:01 -07:00
Dan Helfman b8d349d048 Additional test coverage (#732).
continuous-integration/drone/push Build is passing Details
2023-08-01 16:27:53 -07:00
Dan Helfman f6f06551f0 Multiple configuration includes with a single "!include" (#732).
continuous-integration/drone/push Build is passing Details
2023-08-01 14:12:35 -07:00
Dan Helfman 69771fe7ce Add "store_config_files" option to NEWS (#725).
continuous-integration/drone/push Build is passing Details
2023-07-31 14:04:00 -07:00
Dan Helfman c5c3e2e0ce Code formatting (#725). 2023-07-31 14:01:20 -07:00
Dan Helfman 8491b2c416
Feat: optionally disable config bootstrap (#725).
Merge pull request #75 from diivi/feat/disable-config-bootstrap
2023-07-31 14:00:00 -07:00
Divyansh Singh 962267b3c5 add test for checking manifest creation 2023-08-01 02:26:36 +05:30
Divyansh Singh 562f4a281b add test 2023-08-01 02:01:15 +05:30
Divyansh Singh 6b09ca8022 exclude configs too and not just the manifest 2023-08-01 01:51:37 +05:30
Divyansh Singh f2ce2f387f feat: optionally disable config bootstrap 2023-08-01 01:32:06 +05:30
Dan Helfman 782a9bb70a Add new documentation to NEWS (#326, #697).
continuous-integration/drone/push Build is passing Details
2023-07-31 12:19:19 -07:00
Dan Helfman 88adb5b3de
Add docs for database restore params and config bootstrap (#326, #697).
continuous-integration/drone/push Build is passing Details
Merge pull request #74 from diivi/docs/database-restore-params-and-config-bootstrap
2023-07-31 11:56:32 -07:00
Divyansh Singh 59465b256d
Apply suggestions from code review 2023-07-31 23:50:46 +05:30
Divyansh Singh adfb89ee65 improve docs - wittens suggestions 2023-07-29 00:16:04 +05:30
Dan Helfman c11dcdef0a Fix some "borgmatic" capitalization issues.
continuous-integration/drone/push Build is passing Details
2023-07-26 23:38:08 -07:00
Dan Helfman 8a2514915c Fix for Borg's interactive prompt on the "check --repair" action automatically getting answered "NO" (#730).
continuous-integration/drone/push Build is passing Details
2023-07-22 15:25:55 -07:00
Dan Helfman 4d7a2876a5 Fix for "prune" action error when using the "keep_exclude_tags" option in configuration (#728).
continuous-integration/drone/push Build is passing Details
2023-07-22 10:26:52 -07:00
Dan Helfman 309f67e860 Fix documentation comma grammar issues.
continuous-integration/drone/push Build is passing Details
2023-07-18 23:27:45 -07:00
Dan Helfman 0ad7b4f408 Bump version for release.
continuous-integration/drone/tag Build is passing Details
continuous-integration/drone/push Build is passing Details
2023-07-18 22:50:09 -07:00
Divyansh Singh 1f415a24b8 add docs for database restore params and config bootstrap 2023-07-15 12:27:54 +05:30
Dan Helfman 5c941600a6 Clarify database version description.
continuous-integration/drone/push Build is passing Details
2023-07-14 22:25:45 -07:00
Dan Helfman 5e0283b391 Remove broken markdown item.
continuous-integration/drone/push Build is passing Details
2023-07-14 22:24:21 -07:00
Dan Helfman ac0b23901b Satisfy Gitea validation.
continuous-integration/drone/push Build is failing Details
2023-07-14 22:23:41 -07:00
Dan Helfman cde50a75e9 Attempt to add some vertical whitespace for clarity.
continuous-integration/drone/push Build is passing Details
2023-07-14 22:23:06 -07:00
Dan Helfman 521c0eb600 Fix text areas in issue templates.
continuous-integration/drone/push Build is passing Details
2023-07-14 22:17:39 -07:00
Dan Helfman bc5dc2253c Convert issue templates from Markdown to YAML to take advantage of forms.
continuous-integration/drone/push Build is passing Details
2023-07-14 22:16:04 -07:00
Dan Helfman 973ab01afb Disable submitting issues without a template.
continuous-integration/drone/push Build is passing Details
2023-07-14 20:10:20 -07:00
Dan Helfman 9413a19eca Try out multiple Gitea templates. 2023-07-14 20:06:22 -07:00
Dan Helfman 0055de08a4 Merge branch 'main' of ssh://projects.torsion.org:3022/borgmatic-collective/borgmatic
continuous-integration/drone/push Build is passing Details
2023-07-13 20:21:42 -07:00
Dan Helfman 04fb351a11 Order documentation navigation sections.
continuous-integration/drone/push Build is passing Details
2023-07-13 20:20:48 -07:00
Dan Helfman e913cddcd1 Remove configuration sections (#723).
continuous-integration/drone/push Build is passing Details
Reviewed-on: borgmatic-collective/borgmatic#723
2023-07-14 03:10:51 +00:00
Dan Helfman da78929415 To prevent argument parsing errors on ambiguous commands, drop support for multiple consecutive flag values.
continuous-integration/drone/push Build is passing Details
2023-07-13 19:25:14 -07:00
Dan Helfman 18b3b569d0 Fix broken bootstrap action (#721). 2023-07-12 09:39:45 -07:00
Dan Helfman 054bd52482 Fix incorrect merge from main that broke tests (#721).
continuous-integration/drone/push Build is passing Details
2023-07-11 22:16:42 -07:00
Dan Helfman 2453ecad42 More documentation updates based on section removal (#721).
continuous-integration/drone/push Build is failing Details
2023-07-11 22:10:57 -07:00
Dan Helfman cedf562a7e Merge branch 'main' into remove-sections 2023-07-11 20:05:16 -07:00
Dan Helfman 5b991b88dd Rewrite documentation navigation URLs when being run locally.
continuous-integration/drone/push Build is passing Details
2023-07-11 19:58:49 -07:00
Dan Helfman d2fa205476 Update documentation for section removal (#721).
continuous-integration/drone/push Build is passing Details
2023-07-11 19:42:14 -07:00
Dan Helfman ecd9e62147 Fix last end-to-end database test (#721).
continuous-integration/drone/push Build is passing Details
2023-07-11 00:37:36 -07:00
Dan Helfman 003d4eac93 Remove extra argument (#721).
continuous-integration/drone/push Build is failing Details
2023-07-10 11:41:43 -07:00
Dan Helfman e8dbca9d68 Truncate long command output without assuming an encoded byte string. 2023-07-10 11:16:18 -07:00
Dan Helfman bd9d592560 Truncate long command error output. 2023-07-10 10:20:51 -07:00
Dan Helfman 196a226a7e Add "check_i_know_what_i_am_doing" option to NEWS (#724).
continuous-integration/drone/push Build is passing Details
2023-07-10 09:44:00 -07:00
Dan Helfman a3f47a6418 Remove some sections from tests (#721).
continuous-integration/drone/push Build is failing Details
2023-07-10 09:38:28 -07:00
Dan Helfman d29667ca3c Add a config entry for BORG_CHECK_I_KNOW_WHAT_I_AM_DOING env var (#724).
continuous-integration/drone/push Build is passing Details
Reviewed-on: borgmatic-collective/borgmatic#724
2023-07-10 16:38:09 +00:00
Yoann Laissus 7d18f59079 Add a config entry for BORG_CHECK_I_KNOW_WHAT_I_AM_DOING env var 2023-07-10 10:36:56 +02:00
Dan Helfman ab351548d2 Fix (some) end-to-end tests (#721).
continuous-integration/drone/push Build is failing Details
2023-07-09 17:40:02 -07:00
Dan Helfman 48b6a1679a Remove sections (#721).
continuous-integration/drone/push Build is failing Details
2023-07-08 23:14:30 -07:00
Dan Helfman 9cafc16052 For "borgmatic borg", pass the repository to Borg via a Borg-supported environment variable (#575).
continuous-integration/drone/push Build is passing Details
2023-07-03 00:08:54 -07:00
Dan Helfman fbbfc684ce Add referral link for Hetzner.
continuous-integration/drone/push Build is passing Details
2023-07-02 22:14:36 -07:00
Dan Helfman 13a37a1d9b Reddit is dead.
continuous-integration/drone/push Build is passing Details
2023-06-30 22:55:47 -07:00
Dan Helfman 9cf27fa4ba Deprecated configuration options warning logging.
continuous-integration/drone/push Build is passing Details
2023-06-29 10:03:36 -07:00
Dan Helfman e2c95327fb Fix an error when dumping a MySQL database and the "exclude_nodump" option is set (#720).
continuous-integration/drone/push Build is passing Details
2023-06-28 09:15:11 -07:00
Dan Helfman f60e97d5bf When merging two configuration files, error gracefully if the two files do not adhere to the same format.
continuous-integration/drone/push Build is passing Details
2023-06-26 16:46:09 -07:00
Dan Helfman 44f9ab95f9 Fix typos (#575).
continuous-integration/drone/push Build is passing Details
2023-06-26 14:37:23 -07:00
Dan Helfman bb6004fc4f Revamp "borg" action to support REPOSITORY and ARCHIVE env vars instead of implicitly injecting repository/archive into the Borg command (#575). 2023-06-26 14:35:07 -07:00
Dan Helfman b242078f54 Fix an error when running "borg key export" through borgmatic (#719).
continuous-integration/drone/push Build is passing Details
2023-06-26 09:30:46 -07:00
Dan Helfman c3004c6090 Some brief documentation on running only checks (#364).
continuous-integration/drone/push Build is passing Details
2023-06-25 22:49:36 -07:00
Dan Helfman b9a11e860d Remove legacy configuration parsing code, no longer needed with upgrade-borgmatic-config gone (#529). 2023-06-25 15:36:25 -07:00
Dan Helfman 37a0a0c421 Bump version for release.
continuous-integration/drone/tag Build is passing Details
2023-06-24 22:23:01 -07:00
Dan Helfman 325b561296 Switch from "init" to "rcreate" for creating repos in end-to-end tests.
continuous-integration/drone/push Build is passing Details
2023-06-24 15:52:20 -07:00
Dan Helfman b62017be4b Fix edge case in which "--config somepath.yaml" followed by an action alias (e.g. init for rcreate) wasn't parsed correctly (#716).
continuous-integration/drone/push Build is passing Details
2023-06-24 15:35:10 -07:00
Dan Helfman 8debcbeaba Remove duplicated tests (#716).
continuous-integration/drone/push Build is failing Details
2023-06-24 14:28:50 -07:00
Dan Helfman 35a11559ac Fix error parsing arguments with multiple verbosity flags (#716).
continuous-integration/drone/push Build is failing Details
2023-06-24 14:10:47 -07:00
Dan Helfman e4e455ee45 Deprecate validate-borgmatic-config in favor of new "config validate" action (#529).
continuous-integration/drone/push Build is passing Details
2023-06-23 10:11:41 -07:00
Dan Helfman 23809e9060 More Docker build fun (#326).
continuous-integration/drone/push Build is passing Details
2023-06-22 15:11:49 -07:00
Dan Helfman bb0dd14f69 Attempt to fix CI test failures (#326).
continuous-integration/drone/push Build was killed Details
2023-06-22 14:55:32 -07:00
Dan Helfman 308c96aeb5 Add comment describing need for dev-CI parity test. 2023-06-22 14:37:08 -07:00
Dan Helfman 62a2f5a1d0 Code formatting.
continuous-integration/drone/push Build is failing Details
2023-06-22 14:25:26 -07:00
Dan Helfman e8c862659c Add missing services to build service configuration and add a test to catch this in the future (#326).
continuous-integration/drone/push Build is failing Details
2023-06-22 14:20:42 -07:00
Dan Helfman 69611681e2 Add database restore overrides to NEWS, add a test, and move some tests (#326).
continuous-integration/drone/push Build is failing Details
2023-06-22 12:40:57 -07:00
Dan Helfman 9e0df595c8 Merge branch 'main' of github.com:borgmatic-collective/borgmatic 2023-06-22 12:29:32 -07:00
Dan Helfman 68d90e1e40
feat: allow restoring to different port/host/username (#326).
Merge pull request #73 from diivi/feat/restore-with-different-hostname-port-username
2023-06-22 12:28:34 -07:00
Dan Helfman 248500c7be Accidentally a word.
continuous-integration/drone/push Build is passing Details
2023-06-22 09:21:06 -07:00
Dan Helfman 3addb60fb8 Actually link to the most recent version.
continuous-integration/drone/push Build is passing Details
2023-06-22 09:13:45 -07:00
Dan Helfman 01fffab898 Clarify that references docs are only for the most recent version of borgmatic.
continuous-integration/drone/push Build is passing Details
2023-06-22 09:12:06 -07:00
Dan Helfman bc93401a70 Codespell fixes.
continuous-integration/drone/push Build is passing Details
2023-06-21 13:14:54 -07:00
Dan Helfman 1b90da5bf1 Deprecate generate-borgmatic-config in favor if new "config generate" action (#529).
continuous-integration/drone/push Build is failing Details
2023-06-21 12:19:49 -07:00
Dan Helfman 803fc25848 Add a test for another edge case (#712).
continuous-integration/drone/push Build is passing Details
2023-06-21 10:47:53 -07:00
Dan Helfman 248f82d6f6 Fix for another subaction argument-parsing edge case (#712).
continuous-integration/drone/push Build is passing Details
2023-06-21 10:41:32 -07:00
Divyansh Singh 87c6e5b349 make sure restore params in config aren't used when cli args are supplied 2023-06-21 00:03:07 +05:30
Dan Helfman 147516ae3f Remove additional upgrade-borgmatic-config code (#529).
continuous-integration/drone/push Build is passing Details
2023-06-20 09:41:26 -07:00
Dan Helfman b10aee3070 Remove upgrade-borgmatic-config command for upgrading borgmatic 1.1.0 INI-style configuration (#529).
continuous-integration/drone/push Build is failing Details
2023-06-19 23:17:59 -07:00
Dan Helfman 6098005f5d Fix an error when "data" check time files are accessed without getting upgraded first (#711, #713).
continuous-integration/drone/push Build is passing Details
2023-06-19 23:07:57 -07:00
Dan Helfman 7b8be800a4 Refactor arguments parsing to fix bootstrap action CLI issues (#712).
continuous-integration/drone/push Build is passing Details
2023-06-19 16:18:47 -07:00
Divyansh Singh 1a21eb03cd add tests for all databases 2023-06-20 00:52:01 +05:30
Divyansh Singh e2d82e9bba actually test port restores 2023-06-19 01:10:01 +05:30
Divyansh Singh 384182172a add unit tests for cases when cli/config restore args are used 2023-06-18 06:29:11 +05:30
Divyansh Singh 9016dcc418 all e2e tests 2023-06-18 05:47:35 +05:30
Divyansh Singh e53dd3da87 fix witten reported mysql error 2023-06-17 22:58:59 +05:30
Divyansh Singh 6c87608548 add tests for password logic 2023-06-17 00:47:15 +05:30
Dan Helfman ee2ebb79b8 Find sub-actions for an action without an isinstance() check.
continuous-integration/drone/push Build is passing Details
2023-06-16 10:57:01 -07:00
Divyansh Singh 89602d1614 pass all existing tests (and formatting) 2023-06-16 15:14:00 +05:30
Dan Helfman c294e78715 Use absolute paths when storing configuration files in an archive for later bootstrapping (#697).
continuous-integration/drone/push Build is passing Details
2023-06-15 21:45:43 -07:00
Dan Helfman 9152fed249 Add a documentation troubleshooting note for MySQL/MariaDB authentication errors (#399).
continuous-integration/drone/push Build is passing Details
2023-06-15 14:55:57 -07:00
Divyansh Singh 8389851f2f fix bug where port becomes truthy when none is converted to str 2023-06-15 23:34:50 +05:30
Dan Helfman bbc7f0596c Fix Bash completion for sub-actions like "borgmatic config bootstrap" (#697 follow-on work).
continuous-integration/drone/push Build is passing Details
2023-06-15 10:55:31 -07:00
Divyansh Singh 82d851d891 add argument for restore path 2023-06-15 23:05:53 +05:30
Divyansh Singh 62b6f13299 add restore-path support for sqlite 2023-06-15 23:02:09 +05:30
Divyansh Singh b7423c488e refactor password assignment logic 2023-06-15 22:54:06 +05:30
Dan Helfman 1d7c7eaaa7 Add sample systemd user serivce for running borgmatic as a non-root user (#669).
continuous-integration/drone/push Build is failing Details
2023-06-14 14:57:57 -07:00
Divyansh Singh a9386b7a87 add mongodb support, and sqlite restore path (config option only) 2023-06-15 02:18:24 +05:30
Divyansh Singh 205e5b1524 mysql support 2023-06-15 01:47:46 +05:30
Divyansh Singh 67f4d43aec witten review 2023-06-15 01:37:18 +05:30
Dan Helfman e15bec30e6 Mention some hang edge cases in database limitations (#710).
continuous-integration/drone/push Build is passing Details
2023-06-13 23:34:58 -07:00
Divyansh Singh 230cf6adc4 support command line args for hostname port username password 2023-06-14 00:11:19 +05:30
Divyansh Singh 8e8e64d920 add no-owner and refactor 2023-06-13 23:42:50 +05:30
Divyansh Singh f558cb3156 feat: allow restoring to different port/host/username 2023-06-12 21:54:39 +05:30
Dan Helfman 41924f2400 A little activism.
continuous-integration/drone/push Build is passing Details
2023-06-11 09:50:57 -07:00
Dan Helfman 670bdffb3c Code formatting.
continuous-integration/drone/push Build is passing Details
2023-06-10 19:25:49 -07:00
Dan Helfman 691d4f887a Fix incorrect log message (#697).
continuous-integration/drone/push Build is failing Details
2023-06-10 16:02:03 -07:00
Dan Helfman beb899d6fb Make user-facing manifest loading error messages a little friendlier (#697).
continuous-integration/drone/push Build is failing Details
2023-06-10 15:50:11 -07:00
Dan Helfman 0f9756e739 Fix failing test and add "bootstrap" action to CLI reference docs (#697).
continuous-integration/drone/push Build is passing Details
2023-06-10 15:17:18 -07:00
Dan Helfman d84f1ec616 Add bootstrap action to NEWS and make post-PR tweaks (#697).
continuous-integration/drone/push Build is failing Details
2023-06-10 14:52:00 -07:00
Dan Helfman ef409ad23c
Store configs used to create an archive in the archive and add borgmatic bootstrap (#697).
Merge pull request #71 from diivi/feat/store-config-in-archive
2023-06-10 14:39:53 -07:00
Divyansh Singh d370ff958d mock expand directories thrice 2023-06-10 01:05:34 +05:30
Divyansh Singh 197920d9ef improve tests and some docstrings. 2023-06-09 17:31:57 +05:30
Divyansh Singh 425f260a22 test parser merging 2023-06-09 04:15:18 +05:30
Divyansh Singh 3315555d06 cleaner test 2023-06-09 00:21:41 +05:30
Divyansh Singh 6475345a8f attempt to test parse_subparser_arguments 2023-06-08 01:02:43 +05:30
Divyansh Singh f90d30e0e1 remove duplicate comments 2023-06-08 00:08:39 +05:30
Divyansh Singh 8384eaefb1 reformat 2023-06-08 00:07:36 +05:30
Divyansh Singh dcb90bba50 some tests remaining 2023-06-07 23:56:02 +05:30
Divyansh Singh dc56fd33a0 formatting 2023-06-07 01:47:16 +05:30
Divyansh Singh 2d761dd86b coverage at 100 2023-06-07 01:43:01 +05:30
Divyansh Singh f82631e3bb tests for arguments.py 2023-06-07 00:56:19 +05:30
Divyansh Singh 4b024daae0 pass all tests with wittens recommendation 2023-06-06 23:37:09 +05:30
Divyansh Singh 6a1d1a2e59 fix indentation error that caused too many test failures 2023-06-05 20:31:09 +05:30
Divyansh Singh 206a9c9607 edit schema comments and work on witten review 2023-06-05 20:05:10 +05:30
Dan Helfman a6425b8867 Fix moved Arch Linux borgmatic URL.
continuous-integration/drone/push Build is passing Details
2023-06-04 22:21:16 -07:00
Dan Helfman b5d9398910 Stop uploading GPG signatures to pypi since it no longer supports them.
continuous-integration/drone/push Build is passing Details
2023-06-03 22:37:46 -07:00
Dan Helfman a185eb73b0 Fix GitHub release script now that "master" has been renamed to "main".
continuous-integration/drone/push Build is passing Details
2023-06-03 22:26:49 -07:00
Dan Helfman e80f27f922 Bump version for release.
continuous-integration/drone/tag Build is passing Details
2023-06-03 22:14:21 -07:00
Dan Helfman 1a5b3c9e4e Add Fedora schema loading fix to NEWS (#703).
continuous-integration/drone/push Build is passing Details
2023-06-03 22:07:24 -07:00
Dan Helfman b3f70434df Fix error loading configuration schema on Fedora Linux (#703).
continuous-integration/drone/push Build is failing Details
Reviewed-on: borgmatic-collective/borgmatic#702
2023-06-04 05:04:41 +00:00
Felix Kaechele c61d63b235 Use open() to test for file existance and readability
Signed-off-by: Felix Kaechele <felix@kaechele.ca>
2023-06-04 00:54:29 -04:00
Felix Kaechele ba0899660d Verify that schema path exists before returning it
Signed-off-by: Felix Kaechele <felix@kaechele.ca>
2023-06-03 23:42:20 -04:00
Felix Kaechele 15cabb93ca Drop importlib_metadata entirely
The fallback option using the dirname of the config module location
seems to be more robust in a number of cases.

Signed-off-by: Felix Kaechele <felix@kaechele.ca>
2023-06-03 23:42:20 -04:00
Felix Kaechele ce6daff12f Fix importlib.metadata.files workaround
Some distributions, such as Fedora, do not install the RECORDS file as
part of a package's dist-info. As a result importlib.metadata.files will
return None.

Use the workaround for these cases as well.

Signed-off-by: Felix Kaechele <felix@kaechele.ca>
2023-06-03 23:42:20 -04:00
Dan Helfman caf654366c Document work-around for colons in YAML strings (#708).
continuous-integration/drone/push Build is passing Details
2023-06-03 10:19:34 -07:00
Divyansh Singh bb60b25399 merge subparsers and refactor 2023-06-02 02:04:35 +05:30
Divyansh Singh 74aa28e027 support more flags 2023-06-01 16:53:34 +05:30
Dan Helfman 4f49b345af NEWS wording fix for clarity (#706).
continuous-integration/drone/push Build is passing Details
2023-05-30 23:21:55 -07:00
Dan Helfman 1784ca5910 Fix "check" action error when repository and archive checks are configured but the archive check gets skipped due to the configured frequency (#704).
continuous-integration/drone/push Build is passing Details
2023-05-30 23:19:33 -07:00
Dan Helfman 8f4cce5fa5 Make dev docs message stand out a little more.
continuous-integration/drone/push Build is failing Details
2023-05-30 22:30:06 -07:00
Dan Helfman 518aeabb2a Document verbosity levels (#484).
continuous-integration/drone/push Build is failing Details
2023-05-30 22:25:27 -07:00
Dan Helfman 341bd4118d Fix "--archive latest" on "list" and "info" actions only working on the first of multiple configured repositories (#706).
continuous-integration/drone/push Build is failing Details
2023-05-30 16:53:55 -07:00
Dan Helfman b222f6a60b Mention new verbosity level to NEWS (#484).
continuous-integration/drone/push Build is failing Details
2023-05-30 15:52:49 -07:00
Dan Helfman c0aaba6891 Add option to disable syslog output (#484).
continuous-integration/drone/push Build is failing Details
Reviewed-on: borgmatic-collective/borgmatic#675
2023-05-30 20:03:56 +00:00
Soumik Dutta a7f81d538d nit changes
- help strings in borgmatic commands
- test fixes in test_logger and test_borgmatic

Signed-off-by: Soumik Dutta <shalearkane@gmail.com>
2023-05-29 01:09:00 +05:30
Divyansh Singh 4c60bf84d7 extract config files 2023-05-28 01:36:32 +05:30
Divyansh Singh dbb778a4d6 finish parsing and add error for empty config subcommand 2023-05-26 22:44:31 +05:30
Divyansh Singh f4a169fdf3
Merge pull request #2 from witten/feat/store-config-in-archive 2023-05-26 21:29:18 +05:30
Soumik Dutta 3d41ed3a34 add test to check that log_file is disabled
if logging is disabled

Signed-off-by: Soumik Dutta <shalearkane@gmail.com>
2023-05-26 09:58:53 +05:30
Soumik Dutta 0283f9ae2a fix help string
Signed-off-by: Soumik Dutta <shalearkane@gmail.com>
2023-05-26 08:57:12 +05:30
Soumik Dutta d556a23f97 update borgmatic tests
Signed-off-by: Soumik Dutta <shalearkane@gmail.com>
2023-05-26 08:57:12 +05:30
Soumik Dutta f98d07e8d8 fix logger test
Signed-off-by: Soumik Dutta <shalearkane@gmail.com>
2023-05-26 08:57:12 +05:30
Soumik Dutta 09f59ad97d disable monitoring hooks if monitoring_log_level is set to DISABLED
Signed-off-by: Soumik Dutta <shalearkane@gmail.com>
2023-05-26 08:57:12 +05:30
Soumik Dutta 24be6272ed add test for logger
Signed-off-by: Soumik Dutta <shalearkane@gmail.com>
2023-05-26 08:57:12 +05:30
Soumik Dutta 5a9bb4b97f update help strings
Signed-off-by: Soumik Dutta <shalearkane@gmail.com>
2023-05-26 08:57:12 +05:30
Soumik Dutta 6a2eb1f157 make value of disabled level higher
so that no other log has higher priority

Signed-off-by: Soumik Dutta <shalearkane@gmail.com>
2023-05-26 08:57:12 +05:30
Soumik Dutta 99473c30a8 disable sending logs in Healthchecks
Signed-off-by: Soumik Dutta <shalearkane@gmail.com>
2023-05-26 08:57:12 +05:30
Soumik Dutta f512d1e460 add verbosity level -2
Signed-off-by: Soumik Dutta <shalearkane@gmail.com>
2023-05-26 08:57:12 +05:30
Dan Helfman 96adee444b Potential fix for nested subparsers not parsing correctly. 2023-05-25 15:03:15 -07:00
Divyansh Singh 8b7996dfda removed parents and used reversed remaining_args 2023-05-26 01:07:11 +05:30
Divyansh Singh 2241de11c0 start work on borgmatic config bootstrap command 2023-05-26 00:26:13 +05:30
Dan Helfman 84c21b062f Fix incorrect argument ordering (#659).
continuous-integration/drone/push Build is passing Details
2023-05-23 16:55:40 -07:00
Dan Helfman 76138faaf3 Add integration test for mount action (#659). 2023-05-23 14:49:04 -07:00
Dan Helfman 9299841a5b Add date-based matching flags to NEWS (#659).
continuous-integration/drone/push Build is failing Details
2023-05-23 14:30:16 -07:00
Dan Helfman 35b5c62ca6 Add Borg 2 date-based matching flags for archive selection (#659).
continuous-integration/drone/push Build is failing Details
Reviewed-on: borgmatic-collective/borgmatic#661
2023-05-23 21:26:17 +00:00
Dan Helfman 05b989347c Upgrade requests test requirement (security).
continuous-integration/drone/push Build is passing Details
2023-05-23 08:43:45 -07:00
Chirag Aggarwal 00e9bb011a test should mock out make_flags_from_arguments
Signed-off-by: Chirag Aggarwal <thechiragaggarwal@gmail.com>
2023-05-20 09:23:09 -04:00
Dan Helfman 833796d1c4 Add archive check probing logic tweak to NEWS (#688).
continuous-integration/drone/push Build is passing Details
2023-05-17 08:48:54 -07:00
Divyansh Singh ee32b076eb update tests and formatting 2023-05-16 23:17:35 +05:30
Dan Helfman e3425f48be Instead of taking the first check time found, take the maximum value (#688)
continuous-integration/drone/push Build is passing Details
2023-05-16 10:20:52 -07:00
Dan Helfman 79b094d035 Bump version for release.
continuous-integration/drone/tag Build is passing Details
continuous-integration/drone/push Build is passing Details
2023-05-16 09:59:09 -07:00
Dan Helfman b45e45f161 Partial conversion of showing repository labels in logs instead of paths (part of #635).
continuous-integration/drone/push Build is running Details
2023-05-16 09:36:50 -07:00
Divyansh Singh b10148844b change config_paths var name to used_config_paths to avoid collisions 2023-05-16 14:00:23 +05:30
Dan Helfman ba845d4008 Codespell saves the day.
continuous-integration/drone/push Build is passing Details
2023-05-15 23:25:13 -07:00
Dan Helfman 645d29b040 Fix archive checks being skipped even when particular archives haven't been checked recently (#688).
continuous-integration/drone/push Build is failing Details
2023-05-15 23:17:45 -07:00
Divyansh Singh 49b4d371ce create and add content to borgmatic-manifest.json 2023-05-16 00:24:19 +05:30
Divyansh Singh 1bc7bb4971 feat: store configs used to create an archive in the archive 2023-05-15 23:04:42 +05:30
Dan Helfman e66e449c3b Merge branch 'main' of ssh://projects.torsion.org:3022/borgmatic-collective/borgmatic
continuous-integration/drone/push Build is passing Details
2023-05-14 12:51:23 -07:00
Dan Helfman 8eb05b840a Log a warning when "borgmatic borg" is run with an action that borgmatic natively supports (#694). 2023-05-14 09:59:28 -07:00
Dan Helfman f0fc638284 Docs: add Gentoo Linux to other ways to install (#696).
continuous-integration/drone/push Build is passing Details
Reviewed-on: borgmatic-collective/borgmatic#696
2023-05-13 16:33:11 +00:00
ennui c6126a9226 Docs: add Gentoo Linux to other ways to install 2023-05-13 11:22:47 +00:00
ennui 62b11ba16b Docs: add Gentoo Linux to other ways to install 2023-05-13 11:20:47 +00:00
Dan Helfman 403ae0f698 Clarify configuration comment about source_directories also accepting files (#693).
continuous-integration/drone/push Build is passing Details
2023-05-09 10:14:03 -07:00
Dan Helfman 92a2230a07 Add support for logging each log line as a JSON object via global "--log-json" flag (#680).
continuous-integration/drone/push Build is passing Details
2023-05-08 23:00:49 -07:00
Dan Helfman b3b08ee6d7 Fix error in "borgmatic restore" action when the configured repository path is relative (#691).
continuous-integration/drone/push Build is passing Details
2023-05-07 21:21:35 -07:00
Dan Helfman 15ef37d89f Add test coverage for exact_options_completion() raising (#686).
continuous-integration/drone/push Build is passing Details
2023-05-06 16:25:26 -07:00
Dan Helfman e84bac29e5 Remove value type for compatibility with Python 3.8 (#686). 2023-05-06 16:18:37 -07:00
Dan Helfman 1a956e8b05 Add fish shell completions to NEWS (#686).
continuous-integration/drone/push Build is failing Details
2023-05-06 16:04:15 -07:00
Dan Helfman 4aae7968b8
Add fish shell completions support (#686).
Merge pull request #70 from isaec/feat/fish-completions
2023-05-06 16:00:25 -07:00
Isaac 66964f613c
formatting! 2023-05-06 15:56:50 -07:00
Isaac 614c1bf2e4
rename test to make function under test clearer 2023-05-06 15:52:42 -07:00
Isaac aa770b98f9
follow unit test module convention 2023-05-06 15:50:37 -07:00
Isaac 453b78c852
drop messages 2023-05-06 15:49:07 -07:00
Isaac 0657106893
clarify dedent test name 2023-05-06 15:46:15 -07:00
Isaac 43c532bc57
add test for dedent strip 2023-05-06 11:51:35 -07:00
Isaac efb81fc2c1
rename last arg helper function to current arg for clarity 2023-05-06 11:42:32 -07:00
Isaac c8f4344f89
add more justification to checks 2023-05-06 11:39:02 -07:00
Isaac a047f856a1
tweak docstring, add comment 2023-05-06 11:37:38 -07:00
Isaac d732059979
fix rotted comments 2023-05-06 11:32:10 -07:00
Isaac ccfdd6806f
test the value of completions 2023-05-06 11:29:14 -07:00
Isaac aa564ac5fe
fix the error thrown, unit test for it, and add string explanations 2023-05-06 11:25:15 -07:00
Isaac 77dbb5c499
create way for test cases to be shared 2023-05-06 11:16:45 -07:00
Isaac e623f401b9
write more unit tests 2023-05-06 10:56:54 -07:00
Isaac 372622fbb1
add more doccomments, drop a check 2023-05-06 10:46:27 -07:00
Isaac 469e0ccace
create doccomments, start writing unit tests 2023-05-06 10:42:06 -07:00
Isaac 59a6ce1462
replace double quotes with single quotes 2023-05-05 00:03:43 -07:00
Isaac 5a7a1747f2
add safety check to avoid infinite cat hang 2023-05-05 00:01:45 -07:00
Isaac b557d635fd
async validity check 2023-05-04 23:57:37 -07:00
Isaac d59b9b817f
support required actions 2023-05-04 23:44:54 -07:00
Isaac 16ac4824a5
handle typed without default params 2023-05-04 23:42:04 -07:00
Isaac 3592ec3ddf
dont show deprecated options 2023-05-04 23:32:09 -07:00
Isaac 8f3039be23
handle the expanding filters better 2023-05-04 23:23:29 -07:00
Isaac b4a38d8be9
fix flag showing up for paths 2023-05-04 23:06:11 -07:00
Isaac d962376a9d
refactor to only show specific options if possible 2023-05-04 21:58:30 -07:00
Isaac 193731a017
rename function 2023-05-04 21:14:48 -07:00
Isaac bbc3e9d717
show possible choices 2023-05-04 21:12:24 -07:00
Isaac 639e88262e
create working file completion 2023-05-04 20:17:26 -07:00
Isaac f12a10d888
start work on conditional file completion 2023-05-04 19:50:49 -07:00
Isaac 28efc85660
rearrange to improve legability of the file 2023-05-04 18:11:13 -07:00
Isaac f1fd2e88dd
drop blank completion 2023-05-04 13:49:29 -07:00
Isaac 700f8e9d9c
replace .format with fstring 2023-05-04 13:39:48 -07:00
Isaac f04036e4a7
use fstring to produce completion lines 2023-05-04 13:33:21 -07:00
Isaac 062453af51
replace actionStr with action_name 2023-05-04 13:29:25 -07:00
Isaac b7fe2a5031
lowercase fish in docs 2023-05-04 13:27:57 -07:00
Isaac ca689505e5
add e2e fish test 2023-05-04 13:27:00 -07:00
Isaac 9ff5ea5240
add a unit test, fix isort and black 2023-05-04 13:22:09 -07:00
Dan Helfman 359afe5318 Error if --list is used with --json for create action (#680).
continuous-integration/drone/push Build is passing Details
2023-05-03 17:16:36 -07:00
Dan Helfman 0b397a5bf9 Fix borgmatic error when not finding the configuration schema for certain "pip install --editable" development installs (#687).
continuous-integration/drone/push Build is passing Details
2023-04-30 16:24:10 -07:00
Dan Helfman a60d7fd173 Run "borgmatic borg" action without capturing output so interactive prompts and flags like "--progress" still work.
continuous-integration/drone/push Build is passing Details
2023-04-30 15:43:41 -07:00
Isaac f7e4024fca
add to readme 2023-04-28 14:02:06 -07:00
Isaac 98e3a81fcf
allow file completions as applicable 2023-04-28 12:42:26 -07:00
Isaac 9c77ebb016
continue deduping 2023-04-28 12:15:01 -07:00
Isaac 23f478ce74
use less completion lines 2023-04-28 12:13:08 -07:00
Isaac d265b6ed6f
add comments in generated files 2023-04-28 11:57:16 -07:00
Dan Helfman 77c3161c77 Fix canonical home link in README.
continuous-integration/drone/push Build is passing Details
2023-04-28 08:36:03 -07:00
Isaac 2e658cfa56
only allow one parser 2023-04-27 21:57:50 -07:00
Isaac 412d18f218
show sub options 2023-04-27 21:31:53 -07:00
Isaac 8060586d8b
fix the script and drop unneeded options 2023-04-27 20:05:17 -07:00
Isaac 25b3db72a0
make more precise, fix the version check fn 2023-04-27 19:58:22 -07:00
Isaac 5678f3a96e
basic working version 2023-04-27 19:44:11 -07:00
Isaac 28b152aedd
make upgrade message a template 2023-04-27 19:31:42 -07:00
Isaac 0009471f67
start work on completion 2023-04-27 18:46:13 -07:00
jetchirag a62ac42cca Merge branch 'main' into borg2-archive-flags 2023-04-27 16:57:29 +00:00
Chirag Aggarwal 68ee9687f5 Added tests for all subcommands and used black formatter
Signed-off-by: Chirag Aggarwal <thechiragaggarwal@gmail.com>
2023-04-27 22:27:23 +05:30
Chirag Aggarwal 32395e47f9 Added duplicate flags test for prune
Signed-off-by: Chirag Aggarwal <thechiragaggarwal@gmail.com>
2023-04-24 20:49:41 +05:30
Chirag Aggarwal 8aaba9bb0a Added new flags to prune test for review
Signed-off-by: Chirag Aggarwal <thechiragaggarwal@gmail.com>
2023-04-24 20:43:34 +05:30
Chirag Aggarwal 96aca4f446 Updated existing tests to use new parameters
Signed-off-by: Chirag Aggarwal <thechiragaggarwal@gmail.com>
2023-04-24 20:24:41 +05:30
Dan Helfman 22b84a2fea Switch to Docker Compose for dev-docs script, so podman-docker is no longer needed for Podman users.
continuous-integration/drone/push Build is passing Details
2023-04-22 10:07:40 -07:00
Dan Helfman 5962fd473e Another try. Backing out psql error changes (#678).
continuous-integration/drone/push Build is passing Details
2023-04-21 10:34:50 -07:00
Dan Helfman 7e64f415ba Attempt to fix failing end-to-end database test that only fails in CI.
continuous-integration/drone/push Build is failing Details
2023-04-21 10:03:29 -07:00
Dan Helfman ae12ccd8e6 And fixing again...
continuous-integration/drone/push Build is failing Details
2023-04-21 09:31:37 -07:00
Dan Helfman 3cefeaa229 Fix end-to-end test command-line syntax.
continuous-integration/drone/push Build was killed Details
2023-04-21 09:30:08 -07:00
Dan Helfman 71b75800cd Get more verbose in the end-to-end test restore.
continuous-integration/drone/push Build is failing Details
2023-04-20 23:32:57 -07:00
Dan Helfman 9ca31530a0 Add missing test for check_all_source_directories_exist() raising.
continuous-integration/drone/push Build is failing Details
2023-04-20 23:15:22 -07:00
Dan Helfman b555fcb956 Add "source_directories_must_exist" expansion fix to NEWS (#682). 2023-04-20 23:08:21 -07:00
Dan Helfman 5829196b70 Expand source directories when checking for existence (#682).
continuous-integration/drone/push Build is failing Details
Reviewed-on: borgmatic-collective/borgmatic#683
2023-04-21 06:05:59 +00:00
Jesse Johnson a14870ce48 Expand source directories when checking for existence (#682). 2023-04-21 05:52:04 +00:00
Dan Helfman ee5c25f3bd Add additional tests for PostgreSQL hook fixes (#678).
continuous-integration/drone/push Build is failing Details
2023-04-20 21:44:42 -07:00
Dan Helfman da0f5a34f2 Fix multiple bugs in PostgreSQL hook (#678).
continuous-integration/drone/push Build is failing Details
Reviewed-on: borgmatic-collective/borgmatic#677
2023-04-21 04:05:22 +00:00
Dan Helfman 065be1d9d4 More inclusive language.
continuous-integration/drone/push Build is passing Details
2023-04-20 14:28:04 -07:00
Dan Helfman f2f6fb537a !!!
continuous-integration/drone/push Build is passing Details
2023-04-20 14:19:34 -07:00
Dan Helfman 7ff994a964 🤦
continuous-integration/drone/push Build was killed Details
2023-04-20 13:56:12 -07:00
Dan Helfman 08edecacae WTF?!
continuous-integration/drone/push Build was killed Details
2023-04-20 13:55:37 -07:00
Dan Helfman 1e03046d9a *Seriously?*
continuous-integration/drone/push Build is failing Details
2023-04-20 13:50:26 -07:00
Dan Helfman c9bf52ee45 Sigh again.
continuous-integration/drone/push Build was killed Details
2023-04-20 13:46:49 -07:00
Dan Helfman f947525fca ?
continuous-integration/drone/push Build was killed Details
2023-04-20 13:45:26 -07:00
Dan Helfman 7f7b89d79c Trying a different approach: Ditching Podman-in-Podman.
continuous-integration/drone/push Build is failing Details
2023-04-20 12:03:51 -07:00
Dan Helfman 499e42df35 😭
continuous-integration/drone/push Build was killed Details
2023-04-20 11:58:06 -07:00
Dan Helfman 4302a07c9b WTF.
continuous-integration/drone/push Build was killed Details
2023-04-20 11:53:52 -07:00
Dan Helfman 1721c05d2e Yet more.
continuous-integration/drone/push Build was killed Details
2023-04-20 11:52:23 -07:00
Dan Helfman 8a31c27078 To see what sticks.
continuous-integration/drone/push Build was killed Details
2023-04-20 11:50:25 -07:00
Dan Helfman d6e1cef356 Throwing stuff at the wall.
continuous-integration/drone/push Build was killed Details
2023-04-20 11:49:43 -07:00
Dan Helfman f82bf619ff More.
continuous-integration/drone/push Build is failing Details
2023-04-20 11:41:35 -07:00
Dan Helfman 02eeca1fc2 Hmm.
continuous-integration/drone/push Build is failing Details
2023-04-20 11:36:30 -07:00
Dan Helfman 4e78cf1b95 ಠ_ಠ
continuous-integration/drone/push Build was killed Details
2023-04-20 11:33:15 -07:00
Dan Helfman 9e9a7c50e5 😊🔫
continuous-integration/drone/push Build was killed Details
2023-04-20 11:30:30 -07:00
Dan Helfman 51bc53e5ca Whee.
continuous-integration/drone/push Build is failing Details
2023-04-20 11:24:59 -07:00
Dan Helfman b85538c54c Double sigh.
continuous-integration/drone/push Build is failing Details
2023-04-20 11:11:49 -07:00
Dan Helfman bb5028e484 Sigh.
continuous-integration/drone/push Build is failing Details
2023-04-20 11:11:08 -07:00
Dan Helfman 53ee0fcfad Another attempt at Podman-in-Podman incantations.
continuous-integration/drone/push Build is failing Details
2023-04-20 11:06:15 -07:00
Dan Helfman 5f8c79dd16 Attempt to get Podman-in-Podman builds working.
continuous-integration/drone/push Build is failing Details
2023-04-20 10:50:44 -07:00
Dan Helfman 0a6f5452f4 Fix broken Podman image name.
continuous-integration/drone Build was killed Details
continuous-integration/drone/push Build is failing Details
2023-04-19 23:16:15 -07:00
Dan Helfman 269fac074b Attempt to use Podman-in-Podman for building docs instead of Docker-in-Podman.
continuous-integration/drone/push Build encountered an error Details
2023-04-19 23:14:51 -07:00
Dan Helfman 3b21ce4ce8 Rename "master" development branch to "main" to use more inclusive language (#684).
continuous-integration/drone/push Build is failing Details
2023-04-19 21:43:08 -07:00
Dan Helfman 8bb7631f50 Fix missing mock in unit test.
continuous-integration/drone/push Build is failing Details
2023-04-19 21:22:51 -07:00
Dan Helfman 9f5769f87b Make docs/schema a little more container agnostic / less Docker specific.
continuous-integration/drone/push Build is passing Details
continuous-integration/drone Build is passing Details
2023-04-16 15:41:17 -07:00
Dan Helfman 991e08f16d Add Unraid borgmatic installation link to docs.
continuous-integration/drone/push Build is passing Details
2023-04-15 09:13:13 -07:00
Chirag Aggarwal 1ee56805f1 Merge remote-tracking branch 'upstream/master' into borg2-archive-flags 2023-04-15 17:29:20 +05:30
Dan Helfman 25506b8d2c Backing out upgrade of end-to-end test packages, because apparently we can't have nice things.
continuous-integration/drone/push Build is passing Details
2023-04-14 23:47:51 -07:00
Dan Helfman 28e62d824b Upgrade end-to-end test packages.
continuous-integration/drone/push Build is failing Details
2023-04-14 23:28:07 -07:00
Dan Helfman 7ee37a890e Fix broken end-to-end tests by no longer using an editable package there, a work-around for https://github.com/pypa/packaging-problems/issues/609
continuous-integration/drone/push Build is passing Details
2023-04-14 23:22:07 -07:00
Dan Helfman 8cb5a42a9e Drop deprecated pkg_resources in favor of importlib.metadata and packaging.
continuous-integration/drone/push Build is failing Details
2023-04-14 21:21:25 -07:00
Dan Helfman 5dbb71709c Upgrade test requirements and code style requirements. Auto-reformat code accordingly.
continuous-integration/drone/push Build is passing Details
2023-04-14 19:35:24 -07:00
Dan Helfman 1c67db5d62 Add documentation for "borgmatic restore --schema" (#375).
continuous-integration/drone/push Build is passing Details
2023-04-14 16:40:58 -07:00
Dan Helfman 96d4a8ee45 Add "borgmatic restore --schema" flag to NEWS (#375).
continuous-integration/drone/push Build is passing Details
2023-04-14 16:33:06 -07:00
Dan Helfman 81e167959b
feat: restore specific schemas (#375).
Merge pull request #67 from diivi/feat/restore-specific-schemas
2023-04-14 16:26:25 -07:00
Divyansh Singh f273e82d74 add tests 2023-04-15 02:57:51 +05:30
Jakub Jirutka 17f122bfe5 Use psql instead of pg_restore when format is "plain"
pg_restore: error: input file appears to be a text format dump. Please use psql.
2023-04-14 17:38:19 +02:00
Jakub Jirutka f0f43174c6 Swap if-else in restore_database_dump in postgresql hook for cleanliness 2023-04-14 17:38:19 +02:00
Jakub Jirutka dfccc1b94a Exit on error when restoring all PostgreSQL databases
"--set ON_ERROR_STOP=on" is equivalent to "--exit-on-error" in
pg_restore.
2023-04-14 17:38:18 +02:00
Jakub Jirutka 195024e505 Fix psql_command and pg_restore_command to accept command with arguments
These commands are executed without `shell=True`, so the subprocess
module treats e.g. "docker exec my_pg_container psql" as a single command
(resulting in Errno 2 "No such file or directory") instead of a command
with arguments.
2023-04-14 17:37:38 +02:00
Jakub Jirutka 19a00371f5 Run "psql" with "--no-psqlrc"
Some settings in user's .psqlrc, e.g. "linestyle unicode", may break the
CSV output. "--no-psqlrc" tells psql to not read startup file.

This is not necessary for the analyze_command and restore_command (with
all_databases), but it's generally recommended when running psql from a
script.
2023-04-14 17:37:37 +02:00
Jakub Jirutka 874fba7672 Fix PostgreSQL hook not using "psql_command" for list when dumping "all" 2023-04-14 15:13:49 +02:00
Dan Helfman 50b0a9ce38 Remove newline at end of file.
continuous-integration/drone/push Build is failing Details
2023-04-13 19:13:50 -07:00
Dan Helfman 8802f6888e Fix "TypeError: 'module' object is not callable" in test_commands.py' (#676).
continuous-integration/drone/push Build is failing Details
Reviewed-on: borgmatic-collective/borgmatic#676
2023-04-14 02:12:58 +00:00
polyzen ebe5c5e839 Fix "TypeError: 'module' object is not callable" in test_commands.py 2023-04-14 01:01:31 +00:00
Dan Helfman 613f6c602c Bump version for release.
continuous-integration/drone/tag Build is passing Details
continuous-integration/drone/push Build is passing Details
2023-04-13 15:12:19 -07:00
Dan Helfman 4a94c2c9bf Selectively omit list values when including configuration files (#672).
continuous-integration/drone/push Build is passing Details
2023-04-13 14:39:36 -07:00
Dan Helfman 08843d51d9 Replace "sequence" with "list" in docs for consistency.
continuous-integration/drone/push Build is passing Details
2023-04-12 10:30:23 -07:00
Dan Helfman ea9213cb03 Spelling.
continuous-integration/drone/push Build is passing Details
2023-04-11 22:12:57 -07:00
Dan Helfman 1ea4433aa9 Selectively shallow merge certain mappings or sequences when including configuration files (#672).
continuous-integration/drone/push Build is failing Details
2023-04-11 21:49:10 -07:00
Divyansh Singh 2fea429d78 collection restore for mongodb 2023-04-12 09:34:19 +05:30
Divyansh Singh 264cebd2b1 complete psql multi schema backup 2023-04-11 23:19:49 +05:30
Dan Helfman 4c0e2cab78 View the results of configuration file merging via "validate-borgmatic-config --show" flag (#673).
continuous-integration/drone/push Build is passing Details
2023-04-11 10:49:09 -07:00
Dan Helfman 31a2ac914a Add optional support for running end-to-end tests and building documentation with rootless Podman instead of Docker.
continuous-integration/drone/push Build is passing Details
2023-04-10 14:26:54 -07:00
Dan Helfman d6ef0df50d Mention #670 being fixed in NEWS.
continuous-integration/drone/push Build is passing Details
2023-04-09 10:01:08 -07:00
Dan Helfman cc60a71210 Clarify "log_file" NEWS (#413).
continuous-integration/drone/push Build is passing Details
2023-04-06 14:12:12 -07:00
Dan Helfman 4cd7556a34 Add "log_file" command hook context to NEWS and docs (#413).
continuous-integration/drone/push Build is passing Details
2023-04-06 13:58:37 -07:00
Dan Helfman b4b1fa939d
feat: add logfile name to hook context for interpolation
Merge pull request #68 from diivi/feat/add-log-filename-to-hook-context
2023-04-06 13:46:45 -07:00
Divyansh Singh 16d7131fb7 refactor tests 2023-04-07 01:00:38 +05:30
Divyansh Singh 091d60c226 refactor and improve tests 2023-04-06 12:36:10 +05:30
Divyansh Singh 0fbdf8d860 feat: add logfile name to hook context for interpolation 2023-04-06 09:31:24 +05:30
Dan Helfman 192bfe46a9 Fix error when running the "prune" action with both "archive_name_format" and "prefix" options set (#668).
continuous-integration/drone/push Build is passing Details
2023-04-05 14:58:05 -07:00
Dan Helfman 080c3afa0d Fix documentation referring to "archive_name_format" in wrong configuration section.
continuous-integration/drone/push Build is passing Details
2023-04-05 14:00:21 -07:00
Divyansh Singh 9bc2322f9a feat: restore specific schemas 2023-04-06 02:10:36 +05:30
Dan Helfman a9a65ebe54 Fix integration tests to actually assert (#666).
continuous-integration/drone/push Build is passing Details
2023-04-04 22:11:36 -07:00
Dan Helfman 616eb6b6da Fix error with "info --match-archives" and fix "--match-archives" overriding logic (#666).
continuous-integration/drone/push Build is passing Details
2023-04-04 21:25:10 -07:00
Dan Helfman 00d1dea94e Bump version for release.
continuous-integration/drone/tag Build is passing Details
continuous-integration/drone/push Build is passing Details
2023-04-03 16:11:25 -07:00
Dan Helfman 127ad1dd1f
Add favicon to documentation.
continuous-integration/drone/push Build is passing Details
Merge pull request #66 from diivi/add-favicon
2023-04-03 10:22:12 -07:00
Divyansh Singh fc58ba5763 add favicon to documentation 2023-04-03 17:36:24 +05:30
Dan Helfman 7e6bee84b0 Add "--log-file-format" flag for customizing the log message format (#658).
continuous-integration/drone/push Build is passing Details
2023-04-02 23:06:36 -07:00
Dan Helfman 01811e03ba Tagged the auto-matching archive behavior as breaking in NEWS.
continuous-integration/drone/push Build is passing Details
2023-04-02 14:38:35 -07:00
Dan Helfman 9712d00680 Add "match_archives" option (#588).
continuous-integration/drone/push Build is passing Details
2023-04-01 23:57:55 -07:00
Dan Helfman 275e99d0b9 Add codespell link to documentation.
continuous-integration/drone/push Build encountered an error Details
2023-04-01 14:38:52 -07:00
Dan Helfman b9328e6d42 Add spellchecking of source code to NEWS.
continuous-integration/drone/push Build is passing Details
2023-04-01 14:09:48 -07:00
Dan Helfman 2934d0902c Code spell checking on every test run!
continuous-integration/drone/push Build is passing Details
2023-04-01 11:03:59 -07:00
Dan Helfman 1ad43ad4b5
Fix: run typos to fix various typos in source code.
continuous-integration/drone/push Build is failing Details
Merge pull request #65 from diivi/fix/run-typos
2023-04-01 10:44:11 -07:00
Divyansh Singh 32ab17fa46 merge 2023-04-01 22:12:41 +05:30
Divyansh Singh 6054ced931 fix: run typos 2023-04-01 22:10:32 +05:30
Dan Helfman 1412038ed3
Fix randomly failing test: test_log_outputs_kills_other_processes_when_one_errors (#635).
continuous-integration/drone/push Build is passing Details
Merge pull request #64 from kxxt/master
2023-03-31 23:19:57 -07:00
kxxt fa8bc285c8 Fix randomly failing test. 2023-04-01 14:02:30 +08:00
Dan Helfman f256908b27 Document wording tweaks (#479).
continuous-integration/drone/push Build is passing Details
2023-03-31 15:36:59 -07:00
Dan Helfman 3f78ac4085 Automatically use the "archive_name_format" option to filter which archives get used for borgmatic actions that operate on multiple archives (#479).
continuous-integration/drone/push Build is passing Details
2023-03-31 15:21:08 -07:00
Dan Helfman 5f595f7ac3 Fix regression in which the "transfer" action produced a traceback (#663).
continuous-integration/drone/push Build is passing Details
2023-03-30 23:21:20 -07:00
Dan Helfman b27e625a77 Update schema comment for check_repositories to mention labels (#635).
continuous-integration/drone/push Build is passing Details
2023-03-28 15:44:38 -07:00
Dan Helfman fc2c181b74 Add missing Docker Compose depends.
continuous-integration/drone/push Build is passing Details
2023-03-28 15:31:37 -07:00
Dan Helfman 010b82d6d8 Remove unnecessary cd in dev documentation.
continuous-integration/drone/push Build is passing Details
2023-03-28 12:45:39 -07:00
Dan Helfman aaf3462d17 Fix Drone intentation.
continuous-integration/drone/push Build is passing Details
2023-03-28 12:03:12 -07:00
Dan Helfman f709125110 Error out if run-full-tests is run not inside a test container.
continuous-integration/drone/push Build encountered an error Details
2023-03-28 12:02:07 -07:00
Dan Helfman 3512191f3e Add check_repositories regression fix to NEWS (#662).
continuous-integration/drone/push Build is passing Details
2023-03-28 11:45:55 -07:00
Dan Helfman 06b5d81baa Merge branch 'master' of github.com:borgmatic-collective/borgmatic 2023-03-28 11:15:31 -07:00
Dan Helfman 9d71bf916e
fix: make check repositories work with dict and str repositories (#662).
Merge pull request #63 from diivi/fix/check-repositories-by-label
2023-03-28 11:15:01 -07:00
Dan Helfman 59fe01b56d Update script comment.
continuous-integration/drone/push Build is passing Details
2023-03-28 11:09:25 -07:00
Divyansh Singh 08e358e27f add and update tests 2023-03-28 22:51:35 +05:30
Divyansh Singh ce22d2d302 reformat 2023-03-28 22:29:21 +05:30
Divyansh Singh 2d08a63e60 fix: make check repositories work with dict and str repositories 2023-03-28 22:14:50 +05:30
Chirag Aggarwal 98c6aa6443 Use Square brackets to denote version specific flag
Signed-off-by: Chirag Aggarwal <thechiragaggarwal@gmail.com>
2023-03-28 18:15:49 +05:30
Chirag Aggarwal edd79ed86c removed individual action parameters, and used make_flags_from_arguments
Signed-off-by: Chirag Aggarwal <thechiragaggarwal@gmail.com>
2023-03-28 18:10:42 +05:30
Dan Helfman d96f2239c1 Update OpenBSD borgmatic link.
continuous-integration/drone/push Build is passing Details
2023-03-27 23:43:39 -07:00
Dan Helfman 67a349ae44 I had one job... (#461).
continuous-integration/drone/push Build is passing Details
2023-03-27 23:28:36 -07:00
Dan Helfman dcefded0fa Document that most command-line flags are not config-file-able (#461).
continuous-integration/drone/push Build is passing Details
2023-03-27 23:21:14 -07:00
Dan Helfman 1bcdebd1cc Fix multiple repositories example.
continuous-integration/drone/push Build is passing Details
2023-03-27 23:16:44 -07:00
Dan Helfman 7a8e0e89dd Mention prior versions of borgmatic in repositories schema.
continuous-integration/drone/push Build is passing Details
2023-03-27 21:54:01 -07:00
Dan Helfman 489ae080e5 Update docs with a few more "path:" repositories references (#635).
continuous-integration/drone/push Build is passing Details
2023-03-27 21:49:31 -07:00
Dan Helfman 0e3da7be63 Fix repository schema description.
continuous-integration/drone/push Build is passing Details
2023-03-27 16:15:24 -07:00
Dan Helfman c5ffb76dfa Bump version for release.
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/tag Build is passing Details
2023-03-27 15:56:49 -07:00
Dan Helfman 61c7b8f13c Add optional repository labels so you can select a repository via "--repository yourlabel" at the command-line (#635).
continuous-integration/drone/push Build is failing Details
2023-03-27 15:54:55 -07:00
Dan Helfman 3e8e38011b
Labels for repositories (#635).
Merge pull request #57 from diivi/feat/tag-repos
2023-03-27 15:46:22 -07:00
Chirag Aggarwal 4fa4fccab7 Use make_flags_from_arguments on mount; Pending test fixes
Signed-off-by: Chirag Aggarwal <thechiragaggarwal@gmail.com>
2023-03-27 23:24:17 +05:30
Dan Helfman d0d3a39833 When a database command errors, display and log the error message instead of swallowing it (#396).
continuous-integration/drone/push Build is passing Details
2023-03-27 10:36:39 -07:00
Divyansh Singh 8bef1c698b add feature to docs 2023-03-27 22:16:39 +05:30
Dan Helfman acbbd6670a Removing debugging command output.
continuous-integration/drone/push Build is passing Details
2023-03-26 21:26:35 -07:00
Divyansh Singh b336b9bedf add tests for repo labels 2023-03-27 00:19:23 +05:30
Divyansh Singh ec9def4e71 rename repository arg to repository_path in all borg actions 2023-03-26 23:52:25 +05:30
Divyansh Singh a136fda92d check all tests 2023-03-26 23:35:47 +05:30
Divyansh Singh b511e679ae remove optional label for repos from tests 2023-03-26 16:59:29 +05:30
Dan Helfman f56fdab7a9 Add troubleshooting documentation on PostgreSQL/MySQL authentication errors.
continuous-integration/drone/push Build is passing Details
2023-03-25 17:08:17 -07:00
jetchirag ff1f4dc09c minor fixes to prune argument help text 2023-03-26 02:06:46 +05:30
jetchirag 141474ff07 Added TIMESPAN flags to match archive in various commands (Borg2 feature)
Signed-off-by: jetchirag <thechiragaggarwal@gmail.com>
2023-03-26 01:58:03 +05:30
Dan Helfman 8c0eea7229 Add additional documentation link to environment variable feature. Rename constants section.
continuous-integration/drone/push Build is passing Details
2023-03-25 08:56:25 -07:00
Dan Helfman 19e95628c3 Add documentation and NEWS for custom constants feature (#612).
continuous-integration/drone/push Build is passing Details
2023-03-24 23:47:05 -07:00
Dan Helfman 4d01e53414
Fix: replace primitive values in config without quotes (#612).
Merge pull request #62 from diivi/fix/config-json-replacement
2023-03-24 23:45:36 -07:00
Divyansh Singh a082cb87cb fix: replace primitive values in config without quotes 2023-03-25 12:12:56 +05:30
Dan Helfman 1c51a8e229
Allow defining custom variables in config file (#612).
Merge pull request #60 from diivi/feat/constants-support
2023-03-24 22:50:57 -07:00
Dan Helfman d14a8df71a Hide obnoxious ruamel.yaml warnings during test runs.
continuous-integration/drone/push Build is passing Details
2023-03-24 22:43:10 -07:00
Dan Helfman 739a58fe47 Rename scripts/run-full-dev-tests to scripts/run-end-to-end-dev-tests and make it run end-to-end tests only.
continuous-integration/drone/push Build is passing Details
2023-03-24 16:24:00 -07:00
Dan Helfman af3431d6ae
fix: docs cli reference create spelling
continuous-integration/drone/push Build is passing Details
Merge pull request #61 from diivi/docs/cli-reference
2023-03-24 16:09:50 -07:00
Dan Helfman 9851abc2e1 Add documentation on backing up a database running in a container (#649).
continuous-integration/drone/push Build is passing Details
2023-03-24 15:18:49 -07:00
Divyansh Singh 61ce6f0473 fix: docs cli reference create spelling 2023-03-25 02:44:56 +05:30
Divyansh Singh 78e8bb6c8c reformat 2023-03-25 02:08:52 +05:30
Divyansh Singh af95134cd2 add test for complex constant 2023-03-25 02:03:36 +05:30
Divyansh Singh d6dfb8753a reformat 2023-03-25 01:50:47 +05:30
Divyansh Singh 1bc003560a Merge branch 'master' of https://github.com/diivi/borgmatic into feat/tag-repos 2023-03-25 01:39:26 +05:30
Divyansh Singh aeaf69f49e pass all tests 2023-03-25 01:34:03 +05:30
Divyansh Singh e83ad9e1e4 use repository["path"] instead of repository 2023-03-25 01:04:57 +05:30
Dan Helfman f42890430c Add code style plugins to enforce use of Python f-strings and prevent single-letter variables.
continuous-integration/drone/push Build is passing Details
2023-03-23 23:11:14 -07:00
Divyansh Singh 6f300b0079 feat: constants support 2023-03-24 02:39:37 +05:30
Dan Helfman 9bec029b4f
Fix: remove extra links from docs css.
continuous-integration/drone/push Build is failing Details
continuous-integration/drone Build is passing Details
Merge pull request #59 from diivi/fix/remove-extra-links-from-css
2023-03-23 12:57:55 -07:00
Divyansh Singh 08afad5d81 end with newline 2023-03-24 01:25:15 +05:30
Divyansh Singh a01dc62468 fix: remove extra links from docs css 2023-03-24 01:23:40 +05:30
Dan Helfman 8b61225b13
Copy to clipboard support in documentation.
continuous-integration/drone/push Build is passing Details
Merge pull request #58 from diivi/docs/copy-to-clipboard-support
2023-03-23 12:39:41 -07:00
Divyansh Singh 66d2f49f18 docs: copy to clipboard support 2023-03-23 14:45:23 +05:30
Dan Helfman 0a72c67c6c Add missing source directory error fix to NEWS (#655).
continuous-integration/drone/push Build is passing Details
2023-03-22 13:02:22 -07:00
Dan Helfman ab64b7ef67
Fix error when a source directory doesn't exist and databases are configured (#655).
Merge pull request #56 from diivi/fix/no-error-on-database-backup-without-source-dirs
2023-03-22 12:59:01 -07:00
Divyansh Singh 1e3a3bf1e7 review 2023-03-23 01:18:06 +05:30
Divyansh Singh 7a2f287918 reformat base 2023-03-23 01:08:30 +05:30
Divyansh Singh 8a63c49498 feat: tag repos 2023-03-23 01:01:26 +05:30
Divyansh Singh 3b5ede8044 remove extra parameter from function call 2023-03-22 23:11:44 +05:30
Divyansh Singh bd235f0426 use exit_code_indicates_error and modify it to accept a command 2023-03-22 16:23:53 +05:30
Divyansh Singh 09183464cd fix: no error on database backups without source dirs 2023-03-22 09:41:39 +05:30
Dan Helfman ca6fd6b061 Add confusing error message fix to NEWS (#623).
continuous-integration/drone/push Build is passing Details
2023-03-21 14:25:20 -07:00
Dan Helfman dd9a64f4b6
Fix confusing message when an error occurs running actions for a configuration file (#623).
Merge pull request #55 from diivi/fix/rephrase-error-message
2023-03-21 14:23:09 -07:00
Divyansh Singh 23e7f27ee4 fix: rephrase error when running from config
to avoid confusion, as the user might think the problem is with their config file
2023-03-22 02:22:43 +05:30
Dan Helfman f9ef52f9a5 Remove unused module and outdated test expectations (#576).
continuous-integration/drone/push Build is passing Details
2023-03-21 10:29:17 -07:00
Dan Helfman 3f17c355ca Add "file://" paths to NEWS (#576). 2023-03-21 10:24:51 -07:00
Dan Helfman c83fae5e5b
Support file:// paths for repositories (#576).
Merge pull request #54 from diivi/feat/file-urls-support
2023-03-21 10:22:39 -07:00
Divyansh Singh 39ad8f64c4 add tests and remove magic number 2023-03-21 17:06:03 +05:30
Divyansh Singh e86d223bbf Merge branch 'master' of https://github.com/diivi/borgmatic into feat/file-urls-support 2023-03-21 16:55:05 +05:30
Divyansh Singh 86587ab2dc send repo directly to extract and export_tar 2023-03-20 21:51:45 +05:30
Divyansh Singh 58c95d8015 feat: file:// URLs support 2023-03-20 02:43:23 +05:30
Dan Helfman 6351747da5 Add NixOS package link to installation docs.
continuous-integration/drone/push Build is passing Details
2023-03-19 09:02:47 -07:00
Dan Helfman 55c153409e Add "source_directories_must_exist" option to NEWS (#501).
continuous-integration/drone/push Build is passing Details
2023-03-18 14:07:38 -07:00
Dan Helfman b115fb2fbe Merge branch 'master' of github.com:borgmatic-collective/borgmatic 2023-03-18 14:01:52 -07:00
Dan Helfman 31d04d9ee3
Optionally error if a source directory does not exist.
feat: add optional check for existence of source directories
2023-03-18 13:59:20 -07:00
Divyansh Singh f803836416 reformat 2023-03-18 17:27:33 +05:30
Divyansh Singh 997f60b3e6 add tests 2023-03-18 17:24:21 +05:30
Dan Helfman c84b26499b Add "borg_files_cache_ttl" option to NEWS.
continuous-integration/drone/push Build is passing Details
2023-03-17 19:29:10 -07:00
Dan Helfman 214ae81cbb Add option to set borg_files_cache_ttl in config (#618).
continuous-integration/drone/push Build is passing Details
Reviewed-on: borgmatic-collective/borgmatic#654
2023-03-18 02:24:41 +00:00
Divyansh Singh d17b2c74db feat: add optional check for existence of source directories 2023-03-18 04:35:55 +05:30
Soumik Dutta fb9677230b add test to ensure integers are converted to string
before setting them up to be environment variable values

Signed-off-by: Soumik Dutta <shalearkane@gmail.com>
2023-03-18 02:57:56 +05:30
Soumik Dutta 0db137efdf add option to set borg_files_cache_ttl in config
Signed-off-by: Soumik Dutta <shalearkane@gmail.com>
2023-03-18 01:48:24 +05:30
Dan Helfman e6605c868d Clarify check frequency default behavior (#653).
continuous-integration/drone/push Build is passing Details
2023-03-17 10:09:36 -07:00
Dan Helfman bdfe4b61eb Bump version for release.
continuous-integration/drone/tag Build is passing Details
continuous-integration/drone/push Build is passing Details
2023-03-16 13:42:15 -07:00
Dan Helfman ca4461820d Add support for Python 3.11.
continuous-integration/drone/push Build is passing Details
2023-03-16 13:29:37 -07:00
Dan Helfman 7605838bfe Add "--repository" flag to all actions where it makes sense (#564).
continuous-integration/drone/push Build is passing Details
2023-03-16 13:27:08 -07:00
Dan Helfman 7a784b8eba Add "--repository" flag to common actions (where it makes sense) (#652).
continuous-integration/drone/push Build is passing Details
Reviewed-on: borgmatic-collective/borgmatic#652
2023-03-16 20:21:40 +00:00
Nain 3e22414613 Update tests
Make them more explicit. Also formatting.
2023-03-16 14:01:29 -04:00
Nain 5f87ea3ec5 Add "--repository" flag to the "create" action 2023-03-16 13:15:49 -04:00
Nain a8aeace5b5 Add "--repository" flag to the "compact" action 2023-03-16 11:13:45 -04:00
Nain 480addd7ce Add "--repository" flag to the "check" action 2023-03-16 10:41:13 -04:00
Nain ce0ce4cd1c Merge mostly repetetive tests 2023-03-16 08:23:21 -04:00
Nain 7de9260b0d Remove test now that --repository isn't expected to error
As discussed #652#issuecomment-5579
2023-03-15 14:59:12 -04:00
Nain cdbe6cdf3a Add "--repository" flag to the "prune" action
part of ticket #564
2023-03-15 14:43:17 -04:00
Dan Helfman 95dcc20d5f Better indicate position of additional docs on page (#651).
continuous-integration/drone/push Build is passing Details
Reviewed-on: borgmatic-collective/borgmatic#651
2023-03-15 18:13:27 +00:00
Dan Helfman 49e0494924 Fix --editable (mode) option given --user as arg (#648).
continuous-integration/drone/push Build is passing Details
Reviewed-on: borgmatic-collective/borgmatic#650
2023-03-15 18:06:46 +00:00
Nain 5fad2bd408 Better indicate position of additional docs on page
On wide screens, the position of the documentation (how-to and reference guide)
is at same level as #it's-your-data.-keep-it-that-way.

So the jump due to anchor link makes it seem like we're taken to top aka
main content. Indicate that links are to the left so reader doesn't recurse.
2023-03-15 07:54:49 -04:00
Nain c6829782a3 Fix --editable (mode) option given --user as arg
--user option should be before, or after `--editable .` not in between.
Before seems better.
2023-03-15 06:50:47 -04:00
Dan Helfman 8cec7c74d8 Add "--strip-components all" on the "extract" action to remove leading path components (#647).
continuous-integration/drone/push Build is passing Details
2023-03-09 10:09:16 -08:00
Dan Helfman d3086788eb Document how to list database dumps in an archive.
continuous-integration/drone/push Build is passing Details
2023-03-08 16:09:41 -08:00
Dan Helfman 8d860ea02c
Enhanced docs with info on fetching mysql database size
Merge pull request #46 from Jelle-SamsonIT/patch-3
2023-03-08 15:52:28 -08:00
Dan Helfman b343363bb8 Change the default action order to: "create", "prune", "compact", "check" (#304).
continuous-integration/drone/push Build is passing Details
2023-03-08 14:05:06 -08:00
Dan Helfman 9db31bd1e9 Run any command-line actions in the order specified instead of using a fixed ordering (#304).
continuous-integration/drone/push Build is passing Details
2023-03-08 13:19:41 -08:00
Dan Helfman d88bcc8be9 Add Healthchecks "log" state feature to NEWS.
continuous-integration/drone/push Build is passing Details
2023-03-07 15:45:23 -08:00
Dan Helfman 332f7c4bb6 Add support for healthchecks "log" feature (#628).
continuous-integration/drone/push Build is failing Details
Reviewed-on: borgmatic-collective/borgmatic#645
2023-03-07 22:21:30 +00:00
Dan Helfman 5d19d86e4a Add flake8-quotes to complain about incorrect quoting so I don't have to!
continuous-integration/drone/push Build is passing Details
2023-03-07 14:08:35 -08:00
Soumik Dutta 044ae7869a fix tests
Signed-off-by: Soumik Dutta <shalearkane@gmail.com>
2023-03-08 03:30:12 +05:30
Dan Helfman 62ae82f2c0 Mention searching for files in the extract a backup guide.
continuous-integration/drone/push Build is passing Details
2023-03-06 22:59:34 -08:00
Dan Helfman 66194b7304 Update dates in documentation examples.
continuous-integration/drone/push Build is passing Details
2023-03-06 22:41:43 -08:00
Soumik Dutta 98e429594e added tests to make sure unsupported log states are detected
Signed-off-by: Soumik Dutta <shalearkane@gmail.com>
2023-03-06 20:31:00 +05:30
Soumik Dutta 4fcfddbe08 return early if unsupported state is passed
Signed-off-by: Soumik Dutta <shalearkane@gmail.com>
2023-03-06 19:58:57 +05:30
Soumik Dutta f442aeae9c fix logs_monitor_start_error()
Signed-off-by: Soumik Dutta <shalearkane@gmail.com>
2023-03-06 05:21:56 +05:30
Soumik Dutta e211863cba update test_borgmatic.py
Signed-off-by: Soumik Dutta <shalearkane@gmail.com>
2023-03-06 05:12:24 +05:30
Soumik Dutta 45256ae33f add test for healthchecks
Signed-off-by: Soumik Dutta <shalearkane@gmail.com>
2023-03-06 03:38:08 +05:30
Soumik Dutta 1573d68fe2 update schema.yaml description
also add monitor.State.LOG to cronitor.

Signed-off-by: Soumik Dutta <shalearkane@gmail.com>
2023-03-05 21:57:13 +05:30
Soumik Dutta 69f6695253 Add support for healthchecks "log" feature #628
Signed-off-by: Soumik Dutta <shalearkane@gmail.com>
2023-03-05 19:27:32 +05:30
Dan Helfman a7c055264d
Fix incorrect documentation TOC background by removing extra dark mode styles.
continuous-integration/drone/push Build is passing Details
Merge pull request #52 from diivi/fix/remove-special-dark-mode-attributes
2023-03-04 16:18:04 -08:00
Divyansh Singh db18364a73 fix: remove extra dark mode styles 2023-03-05 03:16:46 +05:30
Dan Helfman 22498ebd4c In the documentation, mention what version of borgmatic introduced SQLite support.
continuous-integration/drone/push Build is passing Details
2023-03-04 10:50:28 -08:00
Dan Helfman e1f02d9fa5 Add SQLite feature to NEWS and also integrations.
continuous-integration/drone/push Build is passing Details
2023-03-04 09:59:16 -08:00
Dan Helfman 9ec220c600
Add SQLite database dump/restore hook (#295).
feat: add dump-restore support for sqlite databases
2023-03-04 09:47:21 -08:00
Divyansh Singh cf0275a3ed remove test path 2023-03-04 23:00:57 +05:30
Divyansh Singh c71eb60cd2 mock os.remove instead of actually removing a file 2023-03-04 13:08:30 +05:30
Divyansh Singh 675e54ba9f use os.remove and improve tests 2023-03-04 12:43:07 +05:30
Divyansh Singh 1793ad74bd add sqlite for e2e tests 2023-03-04 02:41:14 +05:30
Divyansh Singh 767a7d900b e2e tests schema update 2023-03-04 01:29:01 +05:30
Divyansh Singh 903507bd03 code review 2023-03-04 01:27:07 +05:30
Dan Helfman b6cf7d2adc Bump version for release.
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/tag Build is passing Details
2023-03-02 15:34:22 -08:00
Dan Helfman a071e02d20 With the "create" action and the "--list" ("--files") flag, only show excluded files at verbosity 2 (#620).
continuous-integration/drone/push Build is failing Details
2023-03-02 15:33:42 -08:00
Divyansh Singh 3aa88085ed formatting fix 2023-03-03 00:01:52 +05:30
Divyansh Singh af1cc27988 feat: add dump-restore support for sqlite databases 2023-03-02 23:55:16 +05:30
Dan Helfman dbf8301c19 Add "checkpoint_volume" configuration option to creates checkpoints every specified number of bytes.
continuous-integration/drone/push Build is passing Details
2023-02-27 10:47:17 -08:00
Dan Helfman 2a306bef12 Fix tests.
continuous-integration/drone/push Build is passing Details
2023-02-26 23:34:17 -08:00
Dan Helfman 2a36a2a312 Add "--repository" flag to the "rcreate" action. Add "--progress" flag to the "transfer" action.
continuous-integration/drone/push Build is failing Details
2023-02-26 23:22:23 -08:00
Dan Helfman d7a07f0428 Support status character changes in Borg 2.0.0b5 when filtering out special files that cause Borg to hang.
continuous-integration/drone/push Build is passing Details
2023-02-26 22:36:13 -08:00
Dan Helfman da321e180d Fix the "create" action with the "--dry-run" flag querying for databases when a PostgreSQL/MySQL "all" database is configured.
continuous-integration/drone/push Build is passing Details
2023-02-26 22:15:12 -08:00
Dan Helfman c6582e1171 Internally support new Borg 2.0.0b5 "--filter" status characters / item flags for the "create" action.
continuous-integration/drone/push Build is passing Details
2023-02-26 17:17:25 -08:00
Dan Helfman 9b83afe491 With the "create" action, only one of "--list" ("--files") and "--progress" flags can be used.
continuous-integration/drone/push Build is passing Details
2023-02-26 17:05:56 -08:00
Dan Helfman 2814ac3642 Update Borg 2.0 documentation links.
continuous-integration/drone/push Build is passing Details
2023-02-26 16:44:43 -08:00
Dan Helfman 8a9d5d93f5 Add ntfy authentication to NEWS.
continuous-integration/drone/push Build is passing Details
2023-02-25 14:23:42 -08:00
Dan Helfman 783a6d3b45 Add authentication to the ntfy hook (#621).
continuous-integration/drone/push Build is passing Details
Reviewed-on: borgmatic-collective/borgmatic#644
2023-02-25 22:04:37 +00:00
Tom Hubrecht 95575c3450 Add auth test for the ntfy hook 2023-02-25 20:04:39 +01:00
Tom Hubrecht 9b071ff92f Make the auth logic more explicit and warnings if necessary 2023-02-25 20:04:39 +01:00
Tom Hubrecht d80e716822 Add authentication to the ntfy hook 2023-02-24 17:35:53 +01:00
Dan Helfman 418ebc8843 Add MySQL database hook "add_drop_database" configuration option to control whether dumped MySQL databases get dropped right before restore (#642).
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/tag Build is passing Details
2023-02-20 15:32:47 -08:00
Dan Helfman f5a448c7c2 Fix for potential data loss (data not getting backed up) when dumping large "directory" format PostgreSQL/MongoDB databases (#643).
continuous-integration/drone/push Build is passing Details
2023-02-20 15:18:51 -08:00
Dan Helfman 37ac542b31 Merge pull request 'setup: Add link to MacPorts package' (#641) from neverpanic/borgmatic:cal-docs-macports-port into master
continuous-integration/drone/push Build is passing Details
Reviewed-on: borgmatic-collective/borgmatic#641
2023-02-15 17:31:03 +00:00
Clemens Lang 8c7d7e3e41 setup: Add link to MacPorts package 2023-02-15 10:47:59 +01:00
Dan Helfman b811f125b2 Clarify "checks" configuration documentation for older versions of borgmatic (#639).
continuous-integration/drone/push Build is passing Details
2023-02-12 21:42:43 -08:00
Jelle @ Samson-IT 3720f22234
reworded and added 'all' caveat 2022-07-13 22:03:51 +02:00
Jelle @ Samson-IT 1fdec480d6
Added some info about fetching mysql database size 2022-07-13 13:29:45 +02:00
252 changed files with 23015 additions and 8132 deletions

View File

@ -1,56 +0,0 @@
kind: pipeline
name: python-3-8-alpine-3-13
services:
- name: postgresql
image: postgres:13.1-alpine
environment:
POSTGRES_PASSWORD: test
POSTGRES_DB: test
- name: mysql
image: mariadb:10.5
environment:
MYSQL_ROOT_PASSWORD: test
MYSQL_DATABASE: test
- name: mongodb
image: mongo:5.0.5
environment:
MONGO_INITDB_ROOT_USERNAME: root
MONGO_INITDB_ROOT_PASSWORD: test
clone:
skip_verify: true
steps:
- name: build
image: alpine:3.13
pull: always
commands:
- scripts/run-full-tests
---
kind: pipeline
name: documentation
clone:
skip_verify: true
steps:
- name: build
image: plugins/docker
settings:
username:
from_secret: docker_username
password:
from_secret: docker_password
registry: projects.torsion.org
repo: projects.torsion.org/borgmatic-collective/borgmatic
tags: docs
dockerfile: docs/Dockerfile
trigger:
repo:
- borgmatic-collective/borgmatic
branch:
- master
event:
- push

View File

@ -1,4 +1,5 @@
const pluginSyntaxHighlight = require("@11ty/eleventy-plugin-syntaxhighlight");
const codeClipboard = require("eleventy-plugin-code-clipboard");
const inclusiveLangPlugin = require("@11ty/eleventy-plugin-inclusive-language");
const navigationPlugin = require("@11ty/eleventy-navigation");
@ -6,6 +7,7 @@ module.exports = function(eleventyConfig) {
eleventyConfig.addPlugin(pluginSyntaxHighlight);
eleventyConfig.addPlugin(inclusiveLangPlugin);
eleventyConfig.addPlugin(navigationPlugin);
eleventyConfig.addPlugin(codeClipboard);
let markdownIt = require("markdown-it");
let markdownItAnchor = require("markdown-it-anchor");
@ -31,6 +33,7 @@ module.exports = function(eleventyConfig) {
markdownIt(markdownItOptions)
.use(markdownItAnchor, markdownItAnchorOptions)
.use(markdownItReplaceLink)
.use(codeClipboard.markdownItCopyButton)
);
eleventyConfig.addPassthroughCopy({"docs/static": "static"});

1
.flake8 Normal file
View File

@ -0,0 +1 @@
select = Q0

View File

@ -1,35 +0,0 @@
#### What I'm trying to do and why
#### Steps to reproduce (if a bug)
Include (sanitized) borgmatic configuration files if applicable.
#### Actual behavior (if a bug)
Include (sanitized) `--verbosity 2` output if applicable.
#### Expected behavior (if a bug)
#### Other notes / implementation ideas
#### Environment
**borgmatic version:** [version here]
Use `sudo borgmatic --version` or `sudo pip show borgmatic | grep ^Version`
**borgmatic installation method:** [e.g., Debian package, Docker container, etc.]
**Borg version:** [version here]
Use `sudo borg --version`
**Python version:** [version here]
Use `python3 --version`
**Database version (if applicable):** [version here]
Use `psql --version` or `mysql --version` on client and server.
**operating system and version:** [OS here]

View File

@ -0,0 +1,77 @@
name: "Bug or question/support"
about: "For filing a bug or getting support"
body:
- type: textarea
id: problem
attributes:
label: What I'm trying to do and why
validations:
required: true
- type: textarea
id: repro_steps
attributes:
label: Steps to reproduce
description: Include (sanitized) borgmatic configuration files if applicable.
validations:
required: false
- type: textarea
id: actual_behavior
attributes:
label: Actual behavior
description: Include (sanitized) `--verbosity 2` output if applicable.
validations:
required: false
- type: textarea
id: expected_behavior
attributes:
label: Expected behavior
validations:
required: false
- type: textarea
id: notes
attributes:
label: Other notes / implementation ideas
validations:
required: false
- type: input
id: borgmatic_version
attributes:
label: borgmatic version
description: Use `sudo borgmatic --version` or `sudo pip show borgmatic | grep ^Version`
validations:
required: false
- type: input
id: borgmatic_install_method
attributes:
label: borgmatic installation method
description: e.g., pip install, Debian package, container, etc.
validations:
required: false
- type: input
id: borg_version
attributes:
label: Borg version
description: Use `sudo borg --version`
validations:
required: false
- type: input
id: python_version
attributes:
label: Python version
description: Use `python3 --version`
validations:
required: false
- type: input
id: database_version
attributes:
label: Database version (if applicable)
description: Use `psql --version` / `mysql --version` / `mongodump --version` / `sqlite3 --version`
validations:
required: false
- type: input
id: operating_system_version
attributes:
label: Operating system and version
description: On Linux, use `cat /etc/os-release`
validations:
required: false

View File

@ -0,0 +1 @@
blank_issues_enabled: true

View File

@ -0,0 +1,15 @@
name: "Feature"
about: "For filing a feature request or idea"
body:
- type: textarea
id: request
attributes:
label: What I'd like to do and why
validations:
required: true
- type: textarea
id: notes
attributes:
label: Other notes / implementation ideas
validations:
required: false

View File

@ -0,0 +1,28 @@
name: build
run-name: ${{ gitea.actor }} is building
on:
push:
branches: [main]
jobs:
test:
runs-on: host
steps:
- uses: actions/checkout@v4
- run: scripts/run-end-to-end-tests
docs:
needs: [test]
runs-on: host
env:
IMAGE_NAME: projects.torsion.org/borgmatic-collective/borgmatic:docs
steps:
- uses: actions/checkout@v4
- run: podman login --username "$USERNAME" --password "$PASSWORD" projects.torsion.org
env:
USERNAME: "${{ secrets.REGISTRY_USERNAME }}"
PASSWORD: "${{ secrets.REGISTRY_PASSWORD }}"
- run: podman build --tag "$IMAGE_NAME" --file docs/Dockerfile --storage-opt "overlay.mount_program=/usr/bin/fuse-overlayfs" .
- run: podman push "$IMAGE_NAME"

334
NEWS
View File

@ -1,3 +1,329 @@
1.8.10.dev0
* #842: When a command hook exits with a soft failure, ping the log and finish states for any
configured monitoring hooks.
* #843: Add documentation link to Loki dashboard for borgmatic:
https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#loki-hook
* Fix handling of the NO_COLOR environment variable to ignore an empty value.
* Add documentation about backing up containerized databases by configuring borgmatic to exec into
a container to run a dump command:
https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#containers
1.8.9
* #311: Add custom dump/restore command options for MySQL and MariaDB.
* #811: Add an "access_token" option to the ntfy monitoring hook for authenticating
without username/password.
* #827: When the "--json" flag is given, suppress console escape codes so as not to
interfere with JSON output.
* #829: Fix "--override" values containing deprecated section headers not actually overriding
configuration options under deprecated section headers.
* #835: Add support for the NO_COLOR environment variable. See the documentation for more
information:
https://torsion.org/borgmatic/docs/how-to/set-up-backups/#colored-output
* #839: Add log sending for the Apprise logging hook, enabled by default. See the documentation for
more information:
https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#apprise-hook
* #839: Document a potentially breaking shell quoting edge case within error hooks:
https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#error-hooks
* #840: When running the "rcreate" action and the repository already exists but with a different
encryption mode than requested, error.
* Switch from Drone to Gitea Actions for continuous integration.
* Rename scripts/run-end-to-end-dev-tests to scripts/run-end-to-end-tests and use it in both dev
and CI for better dev-CI parity.
* Clarify documentation about restoring a database: borgmatic does not create the database upon
restore.
1.8.8
* #370: For the PostgreSQL hook, pass the "PGSSLMODE" environment variable through to Borg when the
database's configuration omits the "ssl_mode" option.
* #818: Allow the "--repository" flag to match across multiple configuration files.
* #820: Fix broken repository detection in the "rcreate" action with Borg 1.4. The issue did not
occur with other versions of Borg.
* #822: Fix broken escaping logic in the PostgreSQL hook's "pg_dump_command" option.
* SECURITY: Prevent additional shell injection attacks within the PostgreSQL hook.
1.8.7
* #736: Store included configuration files within each backup archive in support of the "config
bootstrap" action. Previously, only top-level configuration files were stored.
* #798: Elevate specific Borg warnings to errors or squash errors to
* warnings. See the documentation for more information:
https://torsion.org/borgmatic/docs/how-to/customize-warnings-and-errors/
* #810: SECURITY: Prevent shell injection attacks within the PostgreSQL hook, the MongoDB hook, the
SQLite hook, the "borgmatic borg" action, and command hook variable/constant interpolation.
* #814: Fix a traceback when providing an invalid "--override" value for a list option.
1.8.6
* #767: Add an "--ssh-command" flag to the "config bootstrap" action for setting a custom SSH
command, as no configuration is available (including the "ssh_command" option) until
bootstrapping completes.
* #794: Fix a traceback when the "repositories" option contains both strings and key/value pairs.
* #800: Add configured repository labels to the JSON output for all actions.
* #802: The "check --force" flag now runs checks even if "check" is in "skip_actions".
* #804: Validate the configured action names in the "skip_actions" option.
* #807: Stream SQLite databases directly to Borg instead of dumping to an intermediate file.
* When logging commands that borgmatic executes, log the environment variables that
borgmatic sets for those commands. (But don't log their values, since they often contain
passwords.)
1.8.5
* #701: Add a "skip_actions" option to skip running particular actions, handy for append-only or
checkless configurations. See the documentation for more information:
https://torsion.org/borgmatic/docs/how-to/set-up-backups/#skipping-actions
* #701: Deprecate the "disabled" value for the "checks" option in favor of the new "skip_actions"
option.
* #745: Constants now apply to included configuration, not just the file doing the includes. As a
side effect of this change, constants no longer apply to option names and only substitute into
configuration values.
* #779: Add a "--match-archives" flag to the "check" action for selecting the archives to check,
overriding the existing "archive_name_format" and "match_archives" options in configuration.
* #779: Only parse "--override" values as complex data types when they're for options of those
types.
* #782: Fix environment variable interpolation within configured repository paths.
* #782: Add configuration constant overriding via the existing "--override" flag.
* #783: Upgrade ruamel.yaml dependency to support version 0.18.x.
* #784: Drop support for Python 3.7, which has been end-of-lifed.
1.8.4
* #715: Add a monitoring hook for sending backup status to a variety of monitoring services via the
Apprise library. See the documentation for more information:
https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#apprise-hook
* #748: When an archive filter causes no matching archives for the "rlist" or "info"
actions, warn the user and suggest how to remove the filter.
* #768: Fix a traceback when an invalid command-line flag or action is used.
* #771: Fix normalization of deprecated sections ("location:", "storage:", "hooks:", etc.) to
support empty sections without erroring.
* #774: Disallow the "--dry-run" flag with the "borg" action, as borgmatic can't guarantee the Borg
command won't have side effects.
1.8.3
* #665: BREAKING: Simplify logging logic as follows: Syslog verbosity is now disabled by
default, but setting the "--syslog-verbosity" flag enables it regardless of whether you're at an
interactive console. Additionally, "--log-file-verbosity" and "--monitoring-verbosity" now
default to 1 (info about steps borgmatic is taking) instead of 0. And both syslog logging and
file logging can be enabled simultaneously.
* #743: Add a monitoring hook for sending backup status and logs to Grafana Loki. See the
documentation for more information:
https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#loki-hook
* #753: When "archive_name_format" is not set, filter archives using the default archive name
format.
* #754: Fix error handling to log command output as one record per line instead of truncating
too-long output and swallowing the end of some Borg error messages.
* #757: Update documentation so "sudo borgmatic" works for pipx borgmatic installations.
* #761: Fix for borgmatic not stopping Borg immediately when the user presses ctrl-C.
* Update documentation to recommend installing/upgrading borgmatic with pipx instead of pip. See the
documentation for more information:
https://torsion.org/borgmatic/docs/how-to/set-up-backups/#installation
https://torsion.org/borgmatic/docs/how-to/upgrade/#upgrading-borgmatic
1.8.2
* #345: Add "key export" action to export a copy of the repository key for safekeeping in case
the original goes missing or gets damaged.
* #727: Add a MariaDB database hook that uses native MariaDB commands instead of the deprecated
MySQL ones. Be aware though that any existing backups made with the "mysql_databases:" hook are
only restorable with a "mysql_databases:" configuration.
* #738: Fix for potential data loss (data not getting restored) in which the database "restore"
action didn't actually restore anything and indicated success anyway.
* Remove the deprecated use of the MongoDB hook's "--db" flag for database restoration.
* Add source code reference documentation for getting oriented with the borgmatic code as a
developer: https://torsion.org/borgmatic/docs/reference/source-code/
1.8.1
* #326: Add documentation for restoring a database to an alternate host:
https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#restore-to-an-alternate-host
* #697: Add documentation for "bootstrap" action:
https://torsion.org/borgmatic/docs/how-to/extract-a-backup/#extract-the-configuration-files-used-to-create-an-archive
* #725: Add "store_config_files" option for disabling the automatic backup of configuration files
used by the "config bootstrap" action.
* #728: Fix for "prune" action error when using the "keep_exclude_tags" option.
* #730: Fix for Borg's interactive prompt on the "check --repair" action automatically getting
answered "NO" even when the "check_i_know_what_i_am_doing" option isn't set.
* #732: Include multiple configuration files with a single "!include". See the documentation for
more information:
https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#multiple-merge-includes
* #734: Omit "--glob-archives" or "--match-archives" Borg flag when its value would be "*" (meaning
all archives).
1.8.0
* #575: BREAKING: For the "borgmatic borg" action, instead of implicitly injecting
repository/archive into the resulting Borg command-line, pass repository to Borg via an
environment variable and make archive available for explicit use in your commands. See the
documentation for more information:
https://torsion.org/borgmatic/docs/how-to/run-arbitrary-borg-commands/
* #719: Fix an error when running "borg key export" through borgmatic.
* #720: Fix an error when dumping a database and the "exclude_nodump" option is set.
* #724: Add "check_i_know_what_i_am_doing" option to bypass Borg confirmation prompt when running
"check --repair".
* When merging two configuration files, error gracefully if the two files do not adhere to the same
format.
* #721: Remove configuration sections ("location:", "storage:", "hooks:", etc.), while still
keeping deprecated support for them. Now, all options are at the same level, and you don't need
to worry about commenting/uncommenting section headers when you change an option (if you remove
your sections first).
* #721: BREAKING: The retention prefix and the consistency prefix can no longer have different
values (unless one is not set).
* #721: BREAKING: The storage umask and the hooks umask can no longer have different values (unless
one is not set).
* BREAKING: Flags like "--config" that previously took multiple values now need to be given once
per value, e.g. "--config first.yaml --config second.yaml" instead of "--config first.yaml
second.yaml". This prevents argument parsing errors on ambiguous commands.
* BREAKING: Remove the deprecated (and silently ignored) "--successful" flag on the "list" action,
as newer versions of Borg list successful (non-checkpoint) archives by default.
* All deprecated configuration option values now generate warning logs.
* Remove the deprecated (and non-functional) "--excludes" flag in favor of excludes within
configuration.
* Fix an error when logging too-long command output during error handling. Now, long command output
is truncated before logging.
1.7.15
* #326: Add configuration options and command-line flags for backing up a database from one
location while restoring it somewhere else.
* #399: Add a documentation troubleshooting note for MySQL/MariaDB authentication errors.
* #529: Remove upgrade-borgmatic-config command for upgrading borgmatic 1.1.0 INI-style
configuration.
* #529: Deprecate generate-borgmatic-config in favor of new "config generate" action.
* #529: Deprecate validate-borgmatic-config in favor of new "config validate" action.
* #697, #712, #716: Extract borgmatic configuration from backup via new "config bootstrap"
action—even when borgmatic has no configuration yet!
* #669: Add sample systemd user service for running borgmatic as a non-root user.
* #711, #713: Fix an error when "data" check time files are accessed without getting upgraded
first.
1.7.14
* #484: Add a new verbosity level (-2) to disable output entirely (for console, syslog, log file,
or monitoring), so not even errors are shown.
* #688: Tweak archive check probing logic to use the newest timestamp found when multiple exist.
* #659: Add Borg 2 date-based matching flags to various actions for archive selection.
* #703: Fix an error when loading the configuration schema on Fedora Linux.
* #704: Fix "check" action error when repository and archive checks are configured but the archive
check gets skipped due to the configured frequency.
* #706: Fix "--archive latest" on "list" and "info" actions that only worked on the first of
multiple configured repositories.
1.7.13
* #375: Restore particular PostgreSQL schemas from a database dump via "borgmatic restore --schema"
flag. See the documentation for more information:
https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#restore-particular-schemas
* #678: Fix error from PostgreSQL when dumping a database with a "format" of "plain".
* #678: Fix PostgreSQL hook to support "psql_command" and "pg_restore_command" options containing
commands with arguments.
* #678: Fix calls to psql in PostgreSQL hook to ignore "~/.psqlrc", whose settings can break
database dumping.
* #680: Add support for logging each log line as a JSON object via global "--log-json" flag.
* #682: Fix "source_directories_must_exist" option to expand globs and tildes in source directories.
* #684: Rename "master" development branch to "main" to use more inclusive language. You'll need to
update your development checkouts accordingly.
* #686: Add fish shell completion script so you can tab-complete on the borgmatic command-line. See
the documentation for more information:
https://torsion.org/borgmatic/docs/how-to/set-up-backups/#shell-completion
* #687: Fix borgmatic error when not finding the configuration schema for certain "pip install
--editable" development installs.
* #688: Fix archive checks being skipped even when particular archives haven't been checked
recently. This occurred when using multiple borgmatic configuration files with different
"archive_name_format"s, for instance.
* #691: Fix error in "borgmatic restore" action when the configured repository path is relative
instead of absolute.
* #694: Run "borgmatic borg" action without capturing output so interactive prompts and flags like
"--progress" still work.
1.7.12
* #413: Add "log_file" context to command hooks so your scripts can consume the borgmatic log file.
See the documentation for more information:
https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/
* #666, #670: Fix error when running the "info" action with the "--match-archives" or "--archive"
flags. Also fix the "--match-archives"/"--archive" flags to correctly override the
"match_archives" configuration option for the "transfer", "list", "rlist", and "info" actions.
* #668: Fix error when running the "prune" action with both "archive_name_format" and "prefix"
options set.
* #672: Selectively shallow merge certain mappings or sequences when including configuration files.
See the documentation for more information:
https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#shallow-merge
* #672: Selectively omit list values when including configuration files. See the documentation for
more information:
https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#list-merge
* #673: View the results of configuration file merging via "validate-borgmatic-config --show" flag.
See the documentation for more information:
https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#debugging-includes
* Add optional support for running end-to-end tests and building documentation with rootless Podman
instead of Docker.
1.7.11
* #479, #588: BREAKING: Automatically use the "archive_name_format" option to filter which archives
get used for borgmatic actions that operate on multiple archives. Override this behavior with the
new "match_archives" option in the storage section. This change is "breaking" in that it silently
changes which archives get considered for "rlist", "prune", "check", etc. See the documentation
for more information:
https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#archive-naming
* #479, #588: The "prefix" options have been deprecated in favor of the new "archive_name_format"
auto-matching behavior and the "match_archives" option.
* #658: Add "--log-file-format" flag for customizing the log message format. See the documentation
for more information:
https://torsion.org/borgmatic/docs/how-to/inspect-your-backups/#logging-to-file
* #662: Fix regression in which the "check_repositories" option failed to match repositories.
* #663: Fix regression in which the "transfer" action produced a traceback.
* Add spellchecking of source code during test runs.
1.7.10
* #396: When a database command errors, display and log the error message instead of swallowing it.
* #501: Optionally error if a source directory does not exist via "source_directories_must_exist"
option in borgmatic's location configuration.
* #576: Add support for "file://" paths within "repositories" option.
* #612: Define and use custom constants in borgmatic configuration files. See the documentation for
more information:
https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#constant-interpolation
* #618: Add support for BORG_FILES_CACHE_TTL environment variable via "borg_files_cache_ttl" option
in borgmatic's storage configuration.
* #623: Fix confusing message when an error occurs running actions for a configuration file.
* #635: Add optional repository labels so you can select a repository via "--repository yourlabel"
at the command-line. See the configuration reference for more information:
https://torsion.org/borgmatic/docs/reference/configuration/
* #649: Add documentation on backing up a database running in a container:
https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#containers
* #655: Fix error when databases are configured and a source directory doesn't exist.
* Add code style plugins to enforce use of Python f-strings and prevent single-letter variables.
To join in the pedantry, refresh your test environment with "tox --recreate".
* Rename scripts/run-full-dev-tests to scripts/run-end-to-end-dev-tests and make it run end-to-end
tests only. Continue using tox to run unit and integration tests.
1.7.9
* #295: Add a SQLite database dump/restore hook.
* #304: Change the default action order when no actions are specified on the command-line to:
"create", "prune", "compact", "check". If you'd like to retain the old ordering ("prune" and
"compact" first), then specify actions explicitly on the command-line.
* #304: Run any command-line actions in the order specified instead of using a fixed ordering.
* #564: Add "--repository" flag to all actions where it makes sense, so you can run borgmatic on
a single configured repository instead of all of them.
* #628: Add a Healthchecks "log" state to send borgmatic logs to Healthchecks without signalling
success or failure.
* #647: Add "--strip-components all" feature on the "extract" action to remove leading path
components of files you extract. Must be used with the "--path" flag.
* Add support for Python 3.11.
1.7.8
* #620: With the "create" action and the "--list" ("--files") flag, only show excluded files at
verbosity 2.
* #621: Add optional authentication to the ntfy monitoring hook.
* With the "create" action, only one of "--list" ("--files") and "--progress" flags can be used.
This lines up with the new behavior in Borg 2.0.0b5.
* Internally support new Borg 2.0.0b5 "--filter" status characters / item flags for the "create"
action.
* Fix the "create" action with the "--dry-run" flag querying for databases when a PostgreSQL/MySQL
"all" database is configured. Now, these queries are skipped due to the dry run.
* Add "--repository" flag to the "rcreate" action to optionally select one configured repository to
create.
* Add "--progress" flag to the "transfer" action, new in Borg 2.0.0b5.
* Add "checkpoint_volume" configuration option to creates checkpoints every specified number of
bytes during a long-running backup, new in Borg 2.0.0b5.
1.7.7
* #642: Add MySQL database hook "add_drop_database" configuration option to control whether dumped
MySQL databases get dropped right before restore.
* #643: Fix for potential data loss (data not getting backed up) when dumping large "directory"
format PostgreSQL/MongoDB databases. Prior to the fix, these dumps would not finish writing to
disk before Borg consumed them. Now, the dumping process completes before Borg starts. This only
applies to "directory" format databases; other formats still stream to Borg without using
temporary disk space.
* Fix MongoDB "directory" format to work with mongodump/mongorestore without error. Prior to this
fix, only the "archive" format worked.
1.7.6
* #393, #438, #560: Optionally dump "all" PostgreSQL/MySQL databases to separate files instead of
one combined dump file, allowing more convenient restores of individual databases. You can enable
@ -262,7 +588,7 @@
* #398: Clarify canonical home of borgmatic in documentation.
* #406: Clarify that spaces in path names should not be backslashed in path names.
* #423: Fix error handling to error loudly when Borg gets killed due to running out of memory!
* Fix build so as not to attempt to build and push documentation for a non-master branch.
* Fix build so as not to attempt to build and push documentation for a non-main branch.
* "Fix" build failure with Alpine Edge by switching from Edge to Alpine 3.13.
* Move #borgmatic IRC channel from Freenode to Libera Chat due to Freenode takeover drama.
IRC connection info: https://torsion.org/borgmatic/#issues
@ -325,7 +651,7 @@
configuration schema descriptions.
1.5.6
* #292: Allow before_backup and similiar hooks to exit with a soft failure without altering the
* #292: Allow before_backup and similar hooks to exit with a soft failure without altering the
monitoring status on Healthchecks or other providers. Support this by waiting to ping monitoring
services with a "start" status until after before_* hooks finish. Failures in before_* hooks
still trigger a monitoring "fail" status.
@ -394,7 +720,7 @@
* For "list" and "info" actions, show repository names even at verbosity 0.
1.4.22
* #276, #285: Disable colored output when "--json" flag is used, so as to produce valid JSON ouput.
* #276, #285: Disable colored output when "--json" flag is used, so as to produce valid JSON output.
* After a backup of a database dump in directory format, properly remove the dump directory.
* In "borgmatic --help", don't expand $HOME in listing of default "--config" paths.
@ -766,7 +1092,7 @@
* #77: Skip non-"*.yaml" config filenames in /etc/borgmatic.d/ so as not to parse backup files,
editor swap files, etc.
* #81: Document user-defined hooks run before/after backup, or on error.
* Add code style guidelines to the documention.
* Add code style guidelines to the documentation.
1.2.0
* #61: Support for Borg --list option via borgmatic command-line to list all archives.

109
README.md
View File

@ -11,68 +11,64 @@ borgmatic is simple, configuration-driven backup software for servers and
workstations. Protect your files with client-side encryption. Backup your
databases too. Monitor it all with integrated third-party services.
The canonical home of borgmatic is at <a href="https://torsion.org/borgmatic">https://torsion.org/borgmatic</a>.
The canonical home of borgmatic is at <a href="https://torsion.org/borgmatic">https://torsion.org/borgmatic/</a>
Here's an example configuration file:
```yaml
location:
# List of source directories to backup.
source_directories:
- /home
- /etc
# List of source directories to backup.
source_directories:
- /home
- /etc
# Paths of local or remote repositories to backup to.
repositories:
- ssh://1234@usw-s001.rsync.net/./backups.borg
- ssh://k8pDxu32@k8pDxu32.repo.borgbase.com/./repo
- /var/lib/backups/local.borg
# Paths of local or remote repositories to backup to.
repositories:
- path: ssh://k8pDxu32@k8pDxu32.repo.borgbase.com/./repo
label: borgbase
- path: /var/lib/backups/local.borg
label: local
retention:
# Retention policy for how many backups to keep.
keep_daily: 7
keep_weekly: 4
keep_monthly: 6
# Retention policy for how many backups to keep.
keep_daily: 7
keep_weekly: 4
keep_monthly: 6
consistency:
# List of checks to run to validate your backups.
checks:
- name: repository
- name: archives
frequency: 2 weeks
# List of checks to run to validate your backups.
checks:
- name: repository
- name: archives
frequency: 2 weeks
hooks:
# Custom preparation scripts to run.
before_backup:
- prepare-for-backup.sh
# Custom preparation scripts to run.
before_backup:
- prepare-for-backup.sh
# Databases to dump and include in backups.
postgresql_databases:
- name: users
# Databases to dump and include in backups.
postgresql_databases:
- name: users
# Third-party services to notify you if backups aren't happening.
healthchecks: https://hc-ping.com/be067061-cf96-4412-8eae-62b0c50d6a8c
# Third-party services to notify you if backups aren't happening.
healthchecks:
ping_url: https://hc-ping.com/be067061-cf96-4412-8eae-62b0c50d6a8c
```
Want to see borgmatic in action? Check out the <a
href="https://asciinema.org/a/203761?autoplay=1" target="_blank">screencast</a>.
<a href="https://asciinema.org/a/203761?autoplay=1" target="_blank"><img src="https://asciinema.org/a/203761.png" width="480"></a>
borgmatic is powered by [Borg Backup](https://www.borgbackup.org/).
## Integrations
<a href="https://www.postgresql.org/"><img src="docs/static/postgresql.png" alt="PostgreSQL" height="60px" style="margin-bottom:20px;"></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<a href="https://www.mysql.com/"><img src="docs/static/mysql.png" alt="MySQL" height="60px" style="margin-bottom:20px;"></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<a href="https://mariadb.com/"><img src="docs/static/mariadb.png" alt="MariaDB" height="60px" style="margin-bottom:20px;"></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<a href="https://www.mongodb.com/"><img src="docs/static/mongodb.png" alt="MongoDB" height="60px" style="margin-bottom:20px;"></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<a href="https://healthchecks.io/"><img src="docs/static/healthchecks.png" alt="Healthchecks" height="60px" style="margin-bottom:20px;"></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<a href="https://cronitor.io/"><img src="docs/static/cronitor.png" alt="Cronitor" height="60px" style="margin-bottom:20px;"></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<a href="https://cronhub.io/"><img src="docs/static/cronhub.png" alt="Cronhub" height="60px" style="margin-bottom:20px;"></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<a href="https://www.pagerduty.com/"><img src="docs/static/pagerduty.png" alt="PagerDuty" height="60px" style="margin-bottom:20px;"></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<a href="https://ntfy.sh/"><img src="docs/static/ntfy.png" alt="ntfy" height="60px" style="margin-bottom:20px;"></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<a href="https://www.borgbase.com/?utm_source=borgmatic"><img src="docs/static/borgbase.png" alt="BorgBase" height="60px" style="margin-bottom:20px;"></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<a href="https://www.postgresql.org/"><img src="docs/static/postgresql.png" alt="PostgreSQL" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
<a href="https://www.mysql.com/"><img src="docs/static/mysql.png" alt="MySQL" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
<a href="https://mariadb.com/"><img src="docs/static/mariadb.png" alt="MariaDB" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
<a href="https://www.mongodb.com/"><img src="docs/static/mongodb.png" alt="MongoDB" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
<a href="https://sqlite.org/"><img src="docs/static/sqlite.png" alt="SQLite" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
<a href="https://healthchecks.io/"><img src="docs/static/healthchecks.png" alt="Healthchecks" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
<a href="https://cronitor.io/"><img src="docs/static/cronitor.png" alt="Cronitor" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
<a href="https://cronhub.io/"><img src="docs/static/cronhub.png" alt="Cronhub" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
<a href="https://www.pagerduty.com/"><img src="docs/static/pagerduty.png" alt="PagerDuty" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
<a href="https://ntfy.sh/"><img src="docs/static/ntfy.png" alt="ntfy" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
<a href="https://grafana.com/oss/loki/"><img src="docs/static/loki.png" alt="Loki" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
<a href="https://github.com/caronc/apprise/wiki"><img src="docs/static/apprise.png" alt="Apprise" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
<a href="https://www.borgbase.com/?utm_source=borgmatic"><img src="docs/static/borgbase.png" alt="BorgBase" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
## Getting started
@ -80,8 +76,8 @@ borgmatic is powered by [Borg Backup](https://www.borgbackup.org/).
Your first step is to [install and configure
borgmatic](https://torsion.org/borgmatic/docs/how-to/set-up-backups/).
For additional documentation, check out the links above for <a
href="https://torsion.org/borgmatic/#documentation">borgmatic how-to and
For additional documentation, check out the links above (left panel on wide screens)
for <a href="https://torsion.org/borgmatic/#documentation">borgmatic how-to and
reference guides</a>.
@ -89,16 +85,15 @@ reference guides</a>.
Need somewhere to store your encrypted off-site backups? The following hosting
providers include specific support for Borg/borgmatic—and fund borgmatic
development and hosting when you use these links to sign up. (These are
referral links, but without any tracking scripts or cookies.)
development and hosting when you use these referral links to sign up:
<ul>
<li class="referral"><a href="https://www.borgbase.com/?utm_source=borgmatic">BorgBase</a>: Borg hosting service with support for monitoring, 2FA, and append-only repos</li>
<li class="referral"><a href="https://hetzner.cloud/?ref=v9dOJ98Ic9I8">Hetzner</a>: A "storage box" that includes support for Borg</li>
</ul>
Additionally, [rsync.net](https://www.rsync.net/products/borg.html) and
[Hetzner](https://www.hetzner.com/storage/storage-box) have compatible storage
offerings, but do not currently fund borgmatic development or hosting.
Additionally, rsync.net has a compatible storage offering, but does not fund
borgmatic development or hosting.
## Support and contributing
@ -119,10 +114,7 @@ issues.
### Social
Check out the [Borg subreddit](https://www.reddit.com/r/BorgBackup/) for
general Borg and borgmatic discussion and support.
Also follow [borgmatic on Mastodon](https://fosstodon.org/@borgmatic).
Follow [borgmatic on Mastodon](https://fosstodon.org/@borgmatic).
### Chat
@ -162,6 +154,3 @@ general, contributions are very welcome. We don't bite!
Also, please check out the [borgmatic development
how-to](https://torsion.org/borgmatic/docs/how-to/develop-on-borgmatic/) for
info on cloning source code, running tests, etc.
<a href="https://build.torsion.org/borgmatic-collective/borgmatic" alt="build status">![Build Status](https://build.torsion.org/api/badges/borgmatic-collective/borgmatic/status.svg?ref=refs/heads/master)</a>

View File

@ -7,8 +7,8 @@ permalink: security-policy/index.html
While we want to hear about security vulnerabilities in all versions of
borgmatic, security fixes are only made to the most recently released version.
It's simply not practical for our small volunteer effort to maintain multiple
release branches and put out separate security patches for each.
It's not practical for our small volunteer effort to maintain multiple release
branches and put out separate security patches for each.
## Reporting a vulnerability

View File

@ -0,0 +1,9 @@
import argparse
def update_arguments(arguments, **updates):
'''
Given an argparse.Namespace instance of command-line arguments and one or more keyword argument
updates to perform, return a copy of the arguments with those updates applied.
'''
return argparse.Namespace(**dict(vars(arguments), **updates))

View File

@ -8,7 +8,13 @@ logger = logging.getLogger(__name__)
def run_borg(
repository, storage, local_borg_version, borg_arguments, local_path, remote_path,
repository,
config,
local_borg_version,
borg_arguments,
global_arguments,
local_path,
remote_path,
):
'''
Run the "borg" action for the given repository.
@ -16,18 +22,21 @@ def run_borg(
if borg_arguments.repository is None or borgmatic.config.validate.repositories_match(
repository, borg_arguments.repository
):
logger.info('{}: Running arbitrary Borg command'.format(repository))
logger.info(
f'{repository.get("label", repository["path"])}: Running arbitrary Borg command'
)
archive_name = borgmatic.borg.rlist.resolve_archive_name(
repository,
repository['path'],
borg_arguments.archive,
storage,
config,
local_borg_version,
global_arguments,
local_path,
remote_path,
)
borgmatic.borg.borg.run_arbitrary_borg(
repository,
storage,
repository['path'],
config,
local_borg_version,
options=borg_arguments.options,
archive=archive_name,

View File

@ -7,7 +7,13 @@ logger = logging.getLogger(__name__)
def run_break_lock(
repository, storage, local_borg_version, break_lock_arguments, local_path, remote_path,
repository,
config,
local_borg_version,
break_lock_arguments,
global_arguments,
local_path,
remote_path,
):
'''
Run the "break-lock" action for the given repository.
@ -15,7 +21,14 @@ def run_break_lock(
if break_lock_arguments.repository is None or borgmatic.config.validate.repositories_match(
repository, break_lock_arguments.repository
):
logger.info(f'{repository}: Breaking repository and cache locks')
borgmatic.borg.break_lock.break_lock(
repository, storage, local_borg_version, local_path=local_path, remote_path=remote_path,
logger.info(
f'{repository.get("label", repository["path"])}: Breaking repository and cache locks'
)
borgmatic.borg.break_lock.break_lock(
repository['path'],
config,
local_borg_version,
global_arguments,
local_path=local_path,
remote_path=remote_path,
)

View File

@ -1,6 +1,7 @@
import logging
import borgmatic.borg.check
import borgmatic.config.validate
import borgmatic.hooks.command
logger = logging.getLogger(__name__)
@ -9,10 +10,7 @@ logger = logging.getLogger(__name__)
def run_check(
config_filename,
repository,
location,
storage,
consistency,
hooks,
config,
hook_context,
local_borg_version,
check_arguments,
@ -23,31 +21,32 @@ def run_check(
'''
Run the "check" action for the given repository.
'''
if check_arguments.repository and not borgmatic.config.validate.repositories_match(
repository, check_arguments.repository
):
return
borgmatic.hooks.command.execute_hook(
hooks.get('before_check'),
hooks.get('umask'),
config.get('before_check'),
config.get('umask'),
config_filename,
'pre-check',
global_arguments.dry_run,
**hook_context,
)
logger.info('{}: Running consistency checks'.format(repository))
logger.info(f'{repository.get("label", repository["path"])}: Running consistency checks')
borgmatic.borg.check.check_archives(
repository,
location,
storage,
consistency,
repository['path'],
config,
local_borg_version,
check_arguments,
global_arguments,
local_path=local_path,
remote_path=remote_path,
progress=check_arguments.progress,
repair=check_arguments.repair,
only_checks=check_arguments.only,
force=check_arguments.force,
)
borgmatic.hooks.command.execute_hook(
hooks.get('after_check'),
hooks.get('umask'),
config.get('after_check'),
config.get('umask'),
config_filename,
'post-check',
global_arguments.dry_run,

View File

@ -2,6 +2,7 @@ import logging
import borgmatic.borg.compact
import borgmatic.borg.feature
import borgmatic.config.validate
import borgmatic.hooks.command
logger = logging.getLogger(__name__)
@ -10,9 +11,7 @@ logger = logging.getLogger(__name__)
def run_compact(
config_filename,
repository,
storage,
retention,
hooks,
config,
hook_context,
local_borg_version,
compact_arguments,
@ -24,21 +23,29 @@ def run_compact(
'''
Run the "compact" action for the given repository.
'''
if compact_arguments.repository and not borgmatic.config.validate.repositories_match(
repository, compact_arguments.repository
):
return
borgmatic.hooks.command.execute_hook(
hooks.get('before_compact'),
hooks.get('umask'),
config.get('before_compact'),
config.get('umask'),
config_filename,
'pre-compact',
global_arguments.dry_run,
**hook_context,
)
if borgmatic.borg.feature.available(borgmatic.borg.feature.Feature.COMPACT, local_borg_version):
logger.info('{}: Compacting segments{}'.format(repository, dry_run_label))
logger.info(
f'{repository.get("label", repository["path"])}: Compacting segments{dry_run_label}'
)
borgmatic.borg.compact.compact_segments(
global_arguments.dry_run,
repository,
storage,
repository['path'],
config,
local_borg_version,
global_arguments,
local_path=local_path,
remote_path=remote_path,
progress=compact_arguments.progress,
@ -46,10 +53,12 @@ def run_compact(
threshold=compact_arguments.threshold,
)
else: # pragma: nocover
logger.info('{}: Skipping compact (only available/needed in Borg 1.2+)'.format(repository))
logger.info(
f'{repository.get("label", repository["path"])}: Skipping compact (only available/needed in Borg 1.2+)'
)
borgmatic.hooks.command.execute_hook(
hooks.get('after_compact'),
hooks.get('umask'),
config.get('after_compact'),
config.get('umask'),
config_filename,
'post-compact',
global_arguments.dry_run,

View File

View File

@ -0,0 +1,103 @@
import json
import logging
import os
import borgmatic.borg.extract
import borgmatic.borg.rlist
import borgmatic.config.validate
import borgmatic.hooks.command
from borgmatic.borg.state import DEFAULT_BORGMATIC_SOURCE_DIRECTORY
logger = logging.getLogger(__name__)
def get_config_paths(bootstrap_arguments, global_arguments, local_borg_version):
'''
Given the bootstrap arguments as an argparse.Namespace (containing the repository and archive
name, borgmatic source directory, destination directory, and whether to strip components), the
global arguments as an argparse.Namespace (containing the dry run flag and the local borg
version), return the config paths from the manifest.json file in the borgmatic source directory
after extracting it from the repository.
Raise ValueError if the manifest JSON is missing, can't be decoded, or doesn't contain the
expected configuration path data.
'''
borgmatic_source_directory = (
bootstrap_arguments.borgmatic_source_directory or DEFAULT_BORGMATIC_SOURCE_DIRECTORY
)
borgmatic_manifest_path = os.path.expanduser(
os.path.join(borgmatic_source_directory, 'bootstrap', 'manifest.json')
)
config = {'ssh_command': bootstrap_arguments.ssh_command}
extract_process = borgmatic.borg.extract.extract_archive(
global_arguments.dry_run,
bootstrap_arguments.repository,
borgmatic.borg.rlist.resolve_archive_name(
bootstrap_arguments.repository,
bootstrap_arguments.archive,
config,
local_borg_version,
global_arguments,
),
[borgmatic_manifest_path],
config,
local_borg_version,
global_arguments,
extract_to_stdout=True,
)
manifest_json = extract_process.stdout.read()
if not manifest_json:
raise ValueError(
'Cannot read configuration paths from archive due to missing bootstrap manifest'
)
try:
manifest_data = json.loads(manifest_json)
except json.JSONDecodeError as error:
raise ValueError(
f'Cannot read configuration paths from archive due to invalid bootstrap manifest JSON: {error}'
)
try:
return manifest_data['config_paths']
except KeyError:
raise ValueError(
'Cannot read configuration paths from archive due to invalid bootstrap manifest'
)
def run_bootstrap(bootstrap_arguments, global_arguments, local_borg_version):
'''
Run the "bootstrap" action for the given repository.
Raise ValueError if the bootstrap configuration could not be loaded.
Raise CalledProcessError or OSError if Borg could not be run.
'''
manifest_config_paths = get_config_paths(
bootstrap_arguments, global_arguments, local_borg_version
)
config = {'ssh_command': bootstrap_arguments.ssh_command}
logger.info(f"Bootstrapping config paths: {', '.join(manifest_config_paths)}")
borgmatic.borg.extract.extract_archive(
global_arguments.dry_run,
bootstrap_arguments.repository,
borgmatic.borg.rlist.resolve_archive_name(
bootstrap_arguments.repository,
bootstrap_arguments.archive,
config,
local_borg_version,
global_arguments,
),
[config_path.lstrip(os.path.sep) for config_path in manifest_config_paths],
config,
local_borg_version,
global_arguments,
extract_to_stdout=False,
destination_path=bootstrap_arguments.destination,
strip_components=bootstrap_arguments.strip_components,
progress=bootstrap_arguments.progress,
)

View File

@ -0,0 +1,48 @@
import logging
import borgmatic.config.generate
import borgmatic.config.validate
import borgmatic.logger
logger = logging.getLogger(__name__)
def run_generate(generate_arguments, global_arguments):
'''
Given the generate arguments and the global arguments, each as an argparse.Namespace instance,
run the "generate" action.
Raise FileExistsError if a file already exists at the destination path and the generate
arguments do not have overwrite set.
'''
borgmatic.logger.add_custom_log_levels()
dry_run_label = ' (dry run; not actually writing anything)' if global_arguments.dry_run else ''
logger.answer(
f'Generating a configuration file at: {generate_arguments.destination_filename}{dry_run_label}'
)
borgmatic.config.generate.generate_sample_configuration(
global_arguments.dry_run,
generate_arguments.source_filename,
generate_arguments.destination_filename,
borgmatic.config.validate.schema_filename(),
overwrite=generate_arguments.overwrite,
)
if generate_arguments.source_filename:
logger.answer(
f'''
Merged in the contents of configuration file at: {generate_arguments.source_filename}
To review the changes made, run:
diff --unified {generate_arguments.source_filename} {generate_arguments.destination_filename}'''
)
logger.answer(
'''
This includes all available configuration options with example values, the few
required options as indicated. Please edit the file to suit your needs.
If you ever need help: https://torsion.org/borgmatic/#issues'''
)

View File

@ -0,0 +1,25 @@
import logging
import borgmatic.config.generate
import borgmatic.logger
logger = logging.getLogger(__name__)
def run_validate(validate_arguments, configs):
'''
Given the validate arguments as an argparse.Namespace instance and a dict of configuration
filename to corresponding parsed configuration, run the "validate" action.
Most of the validation is actually performed implicitly by the standard borgmatic configuration
loading machinery prior to here, so this function mainly exists to support additional validate
flags like "--show".
'''
borgmatic.logger.add_custom_log_levels()
if validate_arguments.show:
for config_path, config in configs.items():
if len(configs) > 1:
logger.answer('---')
logger.answer(borgmatic.config.generate.render_configuration(config))

View File

@ -1,7 +1,12 @@
import importlib.metadata
import json
import logging
import os
import borgmatic.actions.json
import borgmatic.borg.create
import borgmatic.borg.state
import borgmatic.config.validate
import borgmatic.hooks.command
import borgmatic.hooks.dispatch
import borgmatic.hooks.dump
@ -9,12 +14,40 @@ import borgmatic.hooks.dump
logger = logging.getLogger(__name__)
def create_borgmatic_manifest(config, config_paths, dry_run):
'''
Create a borgmatic manifest file to store the paths to the configuration files used to create
the archive.
'''
if dry_run:
return
borgmatic_source_directory = config.get(
'borgmatic_source_directory', borgmatic.borg.state.DEFAULT_BORGMATIC_SOURCE_DIRECTORY
)
borgmatic_manifest_path = os.path.expanduser(
os.path.join(borgmatic_source_directory, 'bootstrap', 'manifest.json')
)
if not os.path.exists(borgmatic_manifest_path):
os.makedirs(os.path.dirname(borgmatic_manifest_path), exist_ok=True)
with open(borgmatic_manifest_path, 'w') as config_list_file:
json.dump(
{
'borgmatic_version': importlib.metadata.version('borgmatic'),
'config_paths': config_paths,
},
config_list_file,
)
def run_create(
config_filename,
repository,
location,
storage,
hooks,
config,
config_paths,
hook_context,
local_borg_version,
create_arguments,
@ -28,39 +61,49 @@ def run_create(
If create_arguments.json is True, yield the JSON output from creating the archive.
'''
if create_arguments.repository and not borgmatic.config.validate.repositories_match(
repository, create_arguments.repository
):
return
borgmatic.hooks.command.execute_hook(
hooks.get('before_backup'),
hooks.get('umask'),
config.get('before_backup'),
config.get('umask'),
config_filename,
'pre-backup',
global_arguments.dry_run,
**hook_context,
)
logger.info('{}: Creating archive{}'.format(repository, dry_run_label))
logger.info(f'{repository.get("label", repository["path"])}: Creating archive{dry_run_label}')
borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
'remove_database_dumps',
hooks,
repository,
borgmatic.hooks.dump.DATABASE_HOOK_NAMES,
location,
'remove_data_source_dumps',
config,
repository['path'],
borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES,
global_arguments.dry_run,
)
active_dumps = borgmatic.hooks.dispatch.call_hooks(
'dump_databases',
hooks,
repository,
borgmatic.hooks.dump.DATABASE_HOOK_NAMES,
location,
'dump_data_sources',
config,
repository['path'],
borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES,
global_arguments.dry_run,
)
if config.get('store_config_files', True):
create_borgmatic_manifest(
config,
config_paths,
global_arguments.dry_run,
)
stream_processes = [process for processes in active_dumps.values() for process in processes]
json_output = borgmatic.borg.create.create_archive(
global_arguments.dry_run,
repository,
location,
storage,
repository['path'],
config,
config_paths,
local_borg_version,
global_arguments,
local_path=local_path,
remote_path=remote_path,
progress=create_arguments.progress,
@ -69,20 +112,19 @@ def run_create(
list_files=create_arguments.list_files,
stream_processes=stream_processes,
)
if json_output: # pragma: nocover
yield json.loads(json_output)
if json_output:
yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))
borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
'remove_database_dumps',
hooks,
'remove_data_source_dumps',
config,
config_filename,
borgmatic.hooks.dump.DATABASE_HOOK_NAMES,
location,
borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES,
global_arguments.dry_run,
)
borgmatic.hooks.command.execute_hook(
hooks.get('after_backup'),
hooks.get('umask'),
config.get('after_backup'),
config.get('umask'),
config_filename,
'post-backup',
global_arguments.dry_run,

View File

@ -0,0 +1,33 @@
import logging
import borgmatic.borg.export_key
import borgmatic.config.validate
logger = logging.getLogger(__name__)
def run_export_key(
repository,
config,
local_borg_version,
export_arguments,
global_arguments,
local_path,
remote_path,
):
'''
Run the "key export" action for the given repository.
'''
if export_arguments.repository is None or borgmatic.config.validate.repositories_match(
repository, export_arguments.repository
):
logger.info(f'{repository.get("label", repository["path"])}: Exporting repository key')
borgmatic.borg.export_key.export_key(
repository['path'],
config,
local_borg_version,
export_arguments,
global_arguments,
local_path=local_path,
remote_path=remote_path,
)

View File

@ -9,7 +9,7 @@ logger = logging.getLogger(__name__)
def run_export_tar(
repository,
storage,
config,
local_borg_version,
export_tar_arguments,
global_arguments,
@ -23,23 +23,25 @@ def run_export_tar(
repository, export_tar_arguments.repository
):
logger.info(
'{}: Exporting archive {} as tar file'.format(repository, export_tar_arguments.archive)
f'{repository["path"]}: Exporting archive {export_tar_arguments.archive} as tar file'
)
borgmatic.borg.export_tar.export_tar_archive(
global_arguments.dry_run,
repository,
repository['path'],
borgmatic.borg.rlist.resolve_archive_name(
repository,
repository['path'],
export_tar_arguments.archive,
storage,
config,
local_borg_version,
global_arguments,
local_path,
remote_path,
),
export_tar_arguments.paths,
export_tar_arguments.destination,
storage,
config,
local_borg_version,
global_arguments,
local_path=local_path,
remote_path=remote_path,
tar_filter=export_tar_arguments.tar_filter,

View File

@ -11,9 +11,7 @@ logger = logging.getLogger(__name__)
def run_extract(
config_filename,
repository,
location,
storage,
hooks,
config,
hook_context,
local_borg_version,
extract_arguments,
@ -25,8 +23,8 @@ def run_extract(
Run the "extract" action for the given repository.
'''
borgmatic.hooks.command.execute_hook(
hooks.get('before_extract'),
hooks.get('umask'),
config.get('before_extract'),
config.get('umask'),
config_filename,
'pre-extract',
global_arguments.dry_run,
@ -35,22 +33,25 @@ def run_extract(
if extract_arguments.repository is None or borgmatic.config.validate.repositories_match(
repository, extract_arguments.repository
):
logger.info('{}: Extracting archive {}'.format(repository, extract_arguments.archive))
logger.info(
f'{repository.get("label", repository["path"])}: Extracting archive {extract_arguments.archive}'
)
borgmatic.borg.extract.extract_archive(
global_arguments.dry_run,
repository,
repository['path'],
borgmatic.borg.rlist.resolve_archive_name(
repository,
repository['path'],
extract_arguments.archive,
storage,
config,
local_borg_version,
global_arguments,
local_path,
remote_path,
),
extract_arguments.paths,
location,
storage,
config,
local_borg_version,
global_arguments,
local_path=local_path,
remote_path=remote_path,
destination_path=extract_arguments.destination,
@ -58,8 +59,8 @@ def run_extract(
progress=extract_arguments.progress,
)
borgmatic.hooks.command.execute_hook(
hooks.get('after_extract'),
hooks.get('umask'),
config.get('after_extract'),
config.get('umask'),
config_filename,
'post-extract',
global_arguments.dry_run,

View File

@ -1,6 +1,7 @@
import json
import logging
import borgmatic.actions.arguments
import borgmatic.actions.json
import borgmatic.borg.info
import borgmatic.borg.rlist
import borgmatic.config.validate
@ -9,7 +10,13 @@ logger = logging.getLogger(__name__)
def run_info(
repository, storage, local_borg_version, info_arguments, local_path, remote_path,
repository,
config,
local_borg_version,
info_arguments,
global_arguments,
local_path,
remote_path,
):
'''
Run the "info" action for the given repository and archive.
@ -19,23 +26,27 @@ def run_info(
if info_arguments.repository is None or borgmatic.config.validate.repositories_match(
repository, info_arguments.repository
):
if not info_arguments.json: # pragma: nocover
logger.answer(f'{repository}: Displaying archive summary information')
info_arguments.archive = borgmatic.borg.rlist.resolve_archive_name(
repository,
if not info_arguments.json:
logger.answer(
f'{repository.get("label", repository["path"])}: Displaying archive summary information'
)
archive_name = borgmatic.borg.rlist.resolve_archive_name(
repository['path'],
info_arguments.archive,
storage,
config,
local_borg_version,
global_arguments,
local_path,
remote_path,
)
json_output = borgmatic.borg.info.display_archives_info(
repository,
storage,
repository['path'],
config,
local_borg_version,
info_arguments=info_arguments,
local_path=local_path,
remote_path=remote_path,
borgmatic.actions.arguments.update_arguments(info_arguments, archive=archive_name),
global_arguments,
local_path,
remote_path,
)
if json_output: # pragma: nocover
yield json.loads(json_output)
if json_output:
yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))

16
borgmatic/actions/json.py Normal file
View File

@ -0,0 +1,16 @@
import json
def parse_json(borg_json_output, label):
'''
Given a Borg JSON output string, parse it as JSON into a dict. Inject the given borgmatic
repository label into it and return the dict.
'''
json_data = json.loads(borg_json_output)
if 'repository' not in json_data:
return json_data
json_data['repository']['label'] = label or ''
return json_data

View File

@ -1,6 +1,7 @@
import json
import logging
import borgmatic.actions.arguments
import borgmatic.actions.json
import borgmatic.borg.list
import borgmatic.config.validate
@ -8,7 +9,13 @@ logger = logging.getLogger(__name__)
def run_list(
repository, storage, local_borg_version, list_arguments, local_path, remote_path,
repository,
config,
local_borg_version,
list_arguments,
global_arguments,
local_path,
remote_path,
):
'''
Run the "list" action for the given repository and archive.
@ -18,26 +25,29 @@ def run_list(
if list_arguments.repository is None or borgmatic.config.validate.repositories_match(
repository, list_arguments.repository
):
if not list_arguments.json: # pragma: nocover
if list_arguments.find_paths:
logger.answer(f'{repository}: Searching archives')
elif not list_arguments.archive:
logger.answer(f'{repository}: Listing archives')
list_arguments.archive = borgmatic.borg.rlist.resolve_archive_name(
repository,
if not list_arguments.json:
if list_arguments.find_paths: # pragma: no cover
logger.answer(f'{repository.get("label", repository["path"])}: Searching archives')
elif not list_arguments.archive: # pragma: no cover
logger.answer(f'{repository.get("label", repository["path"])}: Listing archives')
archive_name = borgmatic.borg.rlist.resolve_archive_name(
repository['path'],
list_arguments.archive,
storage,
config,
local_borg_version,
global_arguments,
local_path,
remote_path,
)
json_output = borgmatic.borg.list.list_archive(
repository,
storage,
repository['path'],
config,
local_borg_version,
list_arguments=list_arguments,
local_path=local_path,
remote_path=remote_path,
borgmatic.actions.arguments.update_arguments(list_arguments, archive=archive_name),
global_arguments,
local_path,
remote_path,
)
if json_output: # pragma: nocover
yield json.loads(json_output)
if json_output:
yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))

View File

@ -8,7 +8,13 @@ logger = logging.getLogger(__name__)
def run_mount(
repository, storage, local_borg_version, mount_arguments, local_path, remote_path,
repository,
config,
local_borg_version,
mount_arguments,
global_arguments,
local_path,
remote_path,
):
'''
Run the "mount" action for the given repository.
@ -17,26 +23,27 @@ def run_mount(
repository, mount_arguments.repository
):
if mount_arguments.archive:
logger.info('{}: Mounting archive {}'.format(repository, mount_arguments.archive))
logger.info(
f'{repository.get("label", repository["path"])}: Mounting archive {mount_arguments.archive}'
)
else: # pragma: nocover
logger.info('{}: Mounting repository'.format(repository))
logger.info(f'{repository.get("label", repository["path"])}: Mounting repository')
borgmatic.borg.mount.mount_archive(
repository,
repository['path'],
borgmatic.borg.rlist.resolve_archive_name(
repository,
repository['path'],
mount_arguments.archive,
storage,
config,
local_borg_version,
global_arguments,
local_path,
remote_path,
),
mount_arguments.mount_point,
mount_arguments.paths,
mount_arguments.foreground,
mount_arguments.options,
storage,
mount_arguments,
config,
local_borg_version,
global_arguments,
local_path=local_path,
remote_path=remote_path,
)

View File

@ -1,6 +1,7 @@
import logging
import borgmatic.borg.prune
import borgmatic.config.validate
import borgmatic.hooks.command
logger = logging.getLogger(__name__)
@ -9,9 +10,7 @@ logger = logging.getLogger(__name__)
def run_prune(
config_filename,
repository,
storage,
retention,
hooks,
config,
hook_context,
local_borg_version,
prune_arguments,
@ -23,29 +22,33 @@ def run_prune(
'''
Run the "prune" action for the given repository.
'''
if prune_arguments.repository and not borgmatic.config.validate.repositories_match(
repository, prune_arguments.repository
):
return
borgmatic.hooks.command.execute_hook(
hooks.get('before_prune'),
hooks.get('umask'),
config.get('before_prune'),
config.get('umask'),
config_filename,
'pre-prune',
global_arguments.dry_run,
**hook_context,
)
logger.info('{}: Pruning archives{}'.format(repository, dry_run_label))
logger.info(f'{repository.get("label", repository["path"])}: Pruning archives{dry_run_label}')
borgmatic.borg.prune.prune_archives(
global_arguments.dry_run,
repository,
storage,
retention,
repository['path'],
config,
local_borg_version,
prune_arguments,
global_arguments,
local_path=local_path,
remote_path=remote_path,
stats=prune_arguments.stats,
list_archives=prune_arguments.list_archives,
)
borgmatic.hooks.command.execute_hook(
hooks.get('after_prune'),
hooks.get('umask'),
config.get('after_prune'),
config.get('umask'),
config_filename,
'post-prune',
global_arguments.dry_run,

View File

@ -1,13 +1,14 @@
import logging
import borgmatic.borg.rcreate
import borgmatic.config.validate
logger = logging.getLogger(__name__)
def run_rcreate(
repository,
storage,
config,
local_borg_version,
rcreate_arguments,
global_arguments,
@ -17,12 +18,18 @@ def run_rcreate(
'''
Run the "rcreate" action for the given repository.
'''
logger.info('{}: Creating repository'.format(repository))
if rcreate_arguments.repository and not borgmatic.config.validate.repositories_match(
repository, rcreate_arguments.repository
):
return
logger.info(f'{repository.get("label", repository["path"])}: Creating repository')
borgmatic.borg.rcreate.create_repository(
global_arguments.dry_run,
repository,
storage,
repository['path'],
config,
local_borg_version,
global_arguments,
rcreate_arguments.encryption_mode,
rcreate_arguments.source_repository,
rcreate_arguments.copy_crypt_key,

View File

@ -17,161 +17,177 @@ logger = logging.getLogger(__name__)
UNSPECIFIED_HOOK = object()
def get_configured_database(
hooks, archive_database_names, hook_name, database_name, configuration_database_name=None
def get_configured_data_source(
config,
archive_data_source_names,
hook_name,
data_source_name,
configuration_data_source_name=None,
):
'''
Find the first database with the given hook name and database name in the configured hooks
dict and the given archive database names dict (from hook name to database names contained in
a particular backup archive). If UNSPECIFIED_HOOK is given as the hook name, search all database
hooks for the named database. If a configuration database name is given, use that instead of the
database name to lookup the database in the given hooks configuration.
Find the first data source with the given hook name and data source name in the configuration
dict and the given archive data source names dict (from hook name to data source names contained
in a particular backup archive). If UNSPECIFIED_HOOK is given as the hook name, search all data
source hooks for the named data source. If a configuration data source name is given, use that
instead of the data source name to lookup the data source in the given hooks configuration.
Return the found database as a tuple of (found hook name, database configuration dict).
Return the found data source as a tuple of (found hook name, data source configuration dict) or
(None, None) if not found.
'''
if not configuration_database_name:
configuration_database_name = database_name
if not configuration_data_source_name:
configuration_data_source_name = data_source_name
if hook_name == UNSPECIFIED_HOOK:
hooks_to_search = hooks
hooks_to_search = {
hook_name: value
for (hook_name, value) in config.items()
if hook_name in borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES
}
else:
hooks_to_search = {hook_name: hooks[hook_name]}
try:
hooks_to_search = {hook_name: config[hook_name]}
except KeyError:
return (None, None)
return next(
(
(name, hook_database)
(name, hook_data_source)
for (name, hook) in hooks_to_search.items()
for hook_database in hook
if hook_database['name'] == configuration_database_name
and database_name in archive_database_names.get(name, [])
for hook_data_source in hook
if hook_data_source['name'] == configuration_data_source_name
and data_source_name in archive_data_source_names.get(name, [])
),
(None, None),
)
def get_configured_hook_name_and_database(hooks, database_name):
'''
Find the hook name and first database dict with the given database name in the configured hooks
dict. This searches across all database hooks.
'''
def restore_single_database(
def restore_single_data_source(
repository,
location,
storage,
hooks,
config,
local_borg_version,
global_arguments,
local_path,
remote_path,
archive_name,
hook_name,
database,
data_source,
connection_params,
): # pragma: no cover
'''
Given (among other things) an archive name, a database hook name, and a configured database
configuration dict, restore that database from the archive.
Given (among other things) an archive name, a data source hook name, the hostname, port,
username/password as connection params, and a configured data source configuration dict, restore
that data source from the archive.
'''
logger.info(f'{repository}: Restoring database {database["name"]}')
logger.info(
f'{repository.get("label", repository["path"])}: Restoring data source {data_source["name"]}'
)
dump_pattern = borgmatic.hooks.dispatch.call_hooks(
'make_database_dump_pattern',
hooks,
repository,
borgmatic.hooks.dump.DATABASE_HOOK_NAMES,
location,
database['name'],
'make_data_source_dump_pattern',
config,
repository['path'],
borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES,
data_source['name'],
)[hook_name]
# Kick off a single database extract to stdout.
# Kick off a single data source extract to stdout.
extract_process = borgmatic.borg.extract.extract_archive(
dry_run=global_arguments.dry_run,
repository=repository,
repository=repository['path'],
archive=archive_name,
paths=borgmatic.hooks.dump.convert_glob_patterns_to_borg_patterns([dump_pattern]),
location_config=location,
storage_config=storage,
config=config,
local_borg_version=local_borg_version,
global_arguments=global_arguments,
local_path=local_path,
remote_path=remote_path,
destination_path='/',
# A directory format dump isn't a single file, and therefore can't extract
# to stdout. In this case, the extract_process return value is None.
extract_to_stdout=bool(database.get('format') != 'directory'),
extract_to_stdout=bool(data_source.get('format') != 'directory'),
)
# Run a single database restore, consuming the extract stdout (if any).
# Run a single data source restore, consuming the extract stdout (if any).
borgmatic.hooks.dispatch.call_hooks(
'restore_database_dump',
{hook_name: [database]},
repository,
borgmatic.hooks.dump.DATABASE_HOOK_NAMES,
location,
global_arguments.dry_run,
extract_process,
function_name='restore_data_source_dump',
config=config,
log_prefix=repository['path'],
hook_names=[hook_name],
data_source=data_source,
dry_run=global_arguments.dry_run,
extract_process=extract_process,
connection_params=connection_params,
)
def collect_archive_database_names(
repository, archive, location, storage, local_borg_version, local_path, remote_path,
def collect_archive_data_source_names(
repository,
archive,
config,
local_borg_version,
global_arguments,
local_path,
remote_path,
):
'''
Given a local or remote repository path, a resolved archive name, a location configuration dict,
a storage configuration dict, the local Borg version, and local and remote Borg paths, query the
archive for the names of databases it contains and return them as a dict from hook name to a
sequence of database names.
Given a local or remote repository path, a resolved archive name, a configuration dict, the
local Borg version, global_arguments an argparse.Namespace, and local and remote Borg paths,
query the archive for the names of data sources it contains as dumps and return them as a dict
from hook name to a sequence of data source names.
'''
borgmatic_source_directory = os.path.expanduser(
location.get(
config.get(
'borgmatic_source_directory', borgmatic.borg.state.DEFAULT_BORGMATIC_SOURCE_DIRECTORY
)
).lstrip('/')
parent_dump_path = os.path.expanduser(
borgmatic.hooks.dump.make_database_dump_path(borgmatic_source_directory, '*_databases/*/*')
)
dump_paths = borgmatic.borg.list.capture_archive_listing(
repository,
archive,
storage,
config,
local_borg_version,
list_path=parent_dump_path,
global_arguments,
list_paths=[
os.path.expanduser(
borgmatic.hooks.dump.make_data_source_dump_path(borgmatic_source_directory, pattern)
)
for pattern in ('*_databases/*/*',)
],
local_path=local_path,
remote_path=remote_path,
)
# Determine the database names corresponding to the dumps found in the archive and
# Determine the data source names corresponding to the dumps found in the archive and
# add them to restore_names.
archive_database_names = {}
archive_data_source_names = {}
for dump_path in dump_paths:
try:
(hook_name, _, database_name) = dump_path.split(
(hook_name, _, data_source_name) = dump_path.split(
borgmatic_source_directory + os.path.sep, 1
)[1].split(os.path.sep)[0:3]
except (ValueError, IndexError):
logger.warning(
f'{repository}: Ignoring invalid database dump path "{dump_path}" in archive {archive}'
f'{repository}: Ignoring invalid data source dump path "{dump_path}" in archive {archive}'
)
else:
if database_name not in archive_database_names.get(hook_name, []):
archive_database_names.setdefault(hook_name, []).extend([database_name])
if data_source_name not in archive_data_source_names.get(hook_name, []):
archive_data_source_names.setdefault(hook_name, []).extend([data_source_name])
return archive_database_names
return archive_data_source_names
def find_databases_to_restore(requested_database_names, archive_database_names):
def find_data_sources_to_restore(requested_data_source_names, archive_data_source_names):
'''
Given a sequence of requested database names to restore and a dict of hook name to the names of
databases found in an archive, return an expanded sequence of database names to restore,
replacing "all" with actual database names as appropriate.
Given a sequence of requested data source names to restore and a dict of hook name to the names
of data sources found in an archive, return an expanded sequence of data source names to
restore, replacing "all" with actual data source names as appropriate.
Raise ValueError if any of the requested database names cannot be found in the archive.
Raise ValueError if any of the requested data source names cannot be found in the archive.
'''
# A map from database hook name to the database names to restore for that hook.
# A map from data source hook name to the data source names to restore for that hook.
restore_names = (
{UNSPECIFIED_HOOK: requested_database_names}
if requested_database_names
{UNSPECIFIED_HOOK: requested_data_source_names}
if requested_data_source_names
else {UNSPECIFIED_HOOK: ['all']}
)
@ -180,64 +196,65 @@ def find_databases_to_restore(requested_database_names, archive_database_names):
if 'all' in restore_names[UNSPECIFIED_HOOK]:
restore_names[UNSPECIFIED_HOOK].remove('all')
for (hook_name, database_names) in archive_database_names.items():
restore_names.setdefault(hook_name, []).extend(database_names)
for hook_name, data_source_names in archive_data_source_names.items():
restore_names.setdefault(hook_name, []).extend(data_source_names)
# If a database is to be restored as part of "all", then remove it from restore names so
# it doesn't get restored twice.
for database_name in database_names:
if database_name in restore_names[UNSPECIFIED_HOOK]:
restore_names[UNSPECIFIED_HOOK].remove(database_name)
# If a data source is to be restored as part of "all", then remove it from restore names
# so it doesn't get restored twice.
for data_source_name in data_source_names:
if data_source_name in restore_names[UNSPECIFIED_HOOK]:
restore_names[UNSPECIFIED_HOOK].remove(data_source_name)
if not restore_names[UNSPECIFIED_HOOK]:
restore_names.pop(UNSPECIFIED_HOOK)
combined_restore_names = set(
name for database_names in restore_names.values() for name in database_names
name for data_source_names in restore_names.values() for name in data_source_names
)
combined_archive_database_names = set(
name for database_names in archive_database_names.values() for name in database_names
combined_archive_data_source_names = set(
name
for data_source_names in archive_data_source_names.values()
for name in data_source_names
)
missing_names = sorted(set(combined_restore_names) - combined_archive_database_names)
missing_names = sorted(set(combined_restore_names) - combined_archive_data_source_names)
if missing_names:
joined_names = ', '.join(f'"{name}"' for name in missing_names)
raise ValueError(
f"Cannot restore database{'s' if len(missing_names) > 1 else ''} {joined_names} missing from archive"
f"Cannot restore data source{'s' if len(missing_names) > 1 else ''} {joined_names} missing from archive"
)
return restore_names
def ensure_databases_found(restore_names, remaining_restore_names, found_names):
def ensure_data_sources_found(restore_names, remaining_restore_names, found_names):
'''
Given a dict from hook name to database names to restore, a dict from hook name to remaining
database names to restore, and a sequence of found (actually restored) database names, raise
ValueError if requested databases to restore were missing from the archive and/or configuration.
Given a dict from hook name to data source names to restore, a dict from hook name to remaining
data source names to restore, and a sequence of found (actually restored) data source names,
raise ValueError if requested data source to restore were missing from the archive and/or
configuration.
'''
combined_restore_names = set(
name
for database_names in tuple(restore_names.values())
for data_source_names in tuple(restore_names.values())
+ tuple(remaining_restore_names.values())
for name in database_names
for name in data_source_names
)
if not combined_restore_names and not found_names:
raise ValueError('No databases were found to restore')
raise ValueError('No data sources were found to restore')
missing_names = sorted(set(combined_restore_names) - set(found_names))
if missing_names:
joined_names = ', '.join(f'"{name}"' for name in missing_names)
raise ValueError(
f"Cannot restore database{'s' if len(missing_names) > 1 else ''} {joined_names} missing from borgmatic's configuration"
f"Cannot restore data source{'s' if len(missing_names) > 1 else ''} {joined_names} missing from borgmatic's configuration"
)
def run_restore(
repository,
location,
storage,
hooks,
config,
local_borg_version,
restore_arguments,
global_arguments,
@ -248,7 +265,7 @@ def run_restore(
Run the "restore" action for the given repository, but only if the repository matches the
requested repository in restore arguments.
Raise ValueError if a configured database could not be found to restore.
Raise ValueError if a configured data source could not be found to restore.
'''
if restore_arguments.repository and not borgmatic.config.validate.repositories_match(
repository, restore_arguments.repository
@ -256,90 +273,108 @@ def run_restore(
return
logger.info(
'{}: Restoring databases from archive {}'.format(repository, restore_arguments.archive)
f'{repository.get("label", repository["path"])}: Restoring data sources from archive {restore_arguments.archive}'
)
borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
'remove_database_dumps',
hooks,
repository,
borgmatic.hooks.dump.DATABASE_HOOK_NAMES,
location,
'remove_data_source_dumps',
config,
repository['path'],
borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES,
global_arguments.dry_run,
)
archive_name = borgmatic.borg.rlist.resolve_archive_name(
repository, restore_arguments.archive, storage, local_borg_version, local_path, remote_path,
repository['path'],
restore_arguments.archive,
config,
local_borg_version,
global_arguments,
local_path,
remote_path,
)
archive_database_names = collect_archive_database_names(
repository, archive_name, location, storage, local_borg_version, local_path, remote_path,
archive_data_source_names = collect_archive_data_source_names(
repository['path'],
archive_name,
config,
local_borg_version,
global_arguments,
local_path,
remote_path,
)
restore_names = find_data_sources_to_restore(
restore_arguments.data_sources, archive_data_source_names
)
restore_names = find_databases_to_restore(restore_arguments.databases, archive_database_names)
found_names = set()
remaining_restore_names = {}
connection_params = {
'hostname': restore_arguments.hostname,
'port': restore_arguments.port,
'username': restore_arguments.username,
'password': restore_arguments.password,
'restore_path': restore_arguments.restore_path,
}
for hook_name, database_names in restore_names.items():
for database_name in database_names:
found_hook_name, found_database = get_configured_database(
hooks, archive_database_names, hook_name, database_name
for hook_name, data_source_names in restore_names.items():
for data_source_name in data_source_names:
found_hook_name, found_data_source = get_configured_data_source(
config, archive_data_source_names, hook_name, data_source_name
)
if not found_database:
if not found_data_source:
remaining_restore_names.setdefault(found_hook_name or hook_name, []).append(
database_name
data_source_name
)
continue
found_names.add(database_name)
restore_single_database(
found_names.add(data_source_name)
restore_single_data_source(
repository,
location,
storage,
hooks,
config,
local_borg_version,
global_arguments,
local_path,
remote_path,
archive_name,
found_hook_name or hook_name,
found_database,
dict(found_data_source, **{'schemas': restore_arguments.schemas}),
connection_params,
)
# For any database that weren't found via exact matches in the hooks configuration, try to
# For any data sources that weren't found via exact matches in the configuration, try to
# fallback to "all" entries.
for hook_name, database_names in remaining_restore_names.items():
for database_name in database_names:
found_hook_name, found_database = get_configured_database(
hooks, archive_database_names, hook_name, database_name, 'all'
for hook_name, data_source_names in remaining_restore_names.items():
for data_source_name in data_source_names:
found_hook_name, found_data_source = get_configured_data_source(
config, archive_data_source_names, hook_name, data_source_name, 'all'
)
if not found_database:
if not found_data_source:
continue
found_names.add(database_name)
database = copy.copy(found_database)
database['name'] = database_name
found_names.add(data_source_name)
data_source = copy.copy(found_data_source)
data_source['name'] = data_source_name
restore_single_database(
restore_single_data_source(
repository,
location,
storage,
hooks,
config,
local_borg_version,
global_arguments,
local_path,
remote_path,
archive_name,
found_hook_name or hook_name,
database,
dict(data_source, **{'schemas': restore_arguments.schemas}),
connection_params,
)
borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
'remove_database_dumps',
hooks,
repository,
borgmatic.hooks.dump.DATABASE_HOOK_NAMES,
location,
'remove_data_source_dumps',
config,
repository['path'],
borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES,
global_arguments.dry_run,
)
ensure_databases_found(restore_names, remaining_restore_names, found_names)
ensure_data_sources_found(restore_names, remaining_restore_names, found_names)

View File

@ -1,6 +1,6 @@
import json
import logging
import borgmatic.actions.json
import borgmatic.borg.rinfo
import borgmatic.config.validate
@ -8,7 +8,13 @@ logger = logging.getLogger(__name__)
def run_rinfo(
repository, storage, local_borg_version, rinfo_arguments, local_path, remote_path,
repository,
config,
local_borg_version,
rinfo_arguments,
global_arguments,
local_path,
remote_path,
):
'''
Run the "rinfo" action for the given repository.
@ -18,15 +24,19 @@ def run_rinfo(
if rinfo_arguments.repository is None or borgmatic.config.validate.repositories_match(
repository, rinfo_arguments.repository
):
if not rinfo_arguments.json: # pragma: nocover
logger.answer('{}: Displaying repository summary information'.format(repository))
if not rinfo_arguments.json:
logger.answer(
f'{repository.get("label", repository["path"])}: Displaying repository summary information'
)
json_output = borgmatic.borg.rinfo.display_repository_info(
repository,
storage,
repository['path'],
config,
local_borg_version,
rinfo_arguments=rinfo_arguments,
global_arguments=global_arguments,
local_path=local_path,
remote_path=remote_path,
)
if json_output: # pragma: nocover
yield json.loads(json_output)
if json_output:
yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))

View File

@ -1,6 +1,6 @@
import json
import logging
import borgmatic.actions.json
import borgmatic.borg.rlist
import borgmatic.config.validate
@ -8,7 +8,13 @@ logger = logging.getLogger(__name__)
def run_rlist(
repository, storage, local_borg_version, rlist_arguments, local_path, remote_path,
repository,
config,
local_borg_version,
rlist_arguments,
global_arguments,
local_path,
remote_path,
):
'''
Run the "rlist" action for the given repository.
@ -18,15 +24,17 @@ def run_rlist(
if rlist_arguments.repository is None or borgmatic.config.validate.repositories_match(
repository, rlist_arguments.repository
):
if not rlist_arguments.json: # pragma: nocover
logger.answer('{}: Listing repository'.format(repository))
if not rlist_arguments.json:
logger.answer(f'{repository.get("label", repository["path"])}: Listing repository')
json_output = borgmatic.borg.rlist.list_repository(
repository,
storage,
repository['path'],
config,
local_borg_version,
rlist_arguments=rlist_arguments,
global_arguments=global_arguments,
local_path=local_path,
remote_path=remote_path,
)
if json_output: # pragma: nocover
yield json.loads(json_output)
if json_output:
yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))

View File

@ -7,7 +7,7 @@ logger = logging.getLogger(__name__)
def run_transfer(
repository,
storage,
config,
local_borg_version,
transfer_arguments,
global_arguments,
@ -17,13 +17,16 @@ def run_transfer(
'''
Run the "transfer" action for the given repository.
'''
logger.info(f'{repository}: Transferring archives to repository')
logger.info(
f'{repository.get("label", repository["path"])}: Transferring archives to repository'
)
borgmatic.borg.transfer.transfer_archives(
global_arguments.dry_run,
repository,
storage,
repository['path'],
config,
local_borg_version,
transfer_arguments,
global_arguments,
local_path=local_path,
remote_path=remote_path,
)

View File

@ -1,20 +1,20 @@
import logging
import shlex
import borgmatic.commands.arguments
import borgmatic.logger
from borgmatic.borg import environment, flags
from borgmatic.execute import execute_command
from borgmatic.execute import DO_NOT_CAPTURE, execute_command
logger = logging.getLogger(__name__)
REPOSITORYLESS_BORG_COMMANDS = {'serve', None}
BORG_SUBCOMMANDS_WITH_SUBCOMMANDS = {'key', 'debug'}
BORG_SUBCOMMANDS_WITHOUT_REPOSITORY = (('debug', 'info'), ('debug', 'convert-profile'), ())
def run_arbitrary_borg(
repository,
storage_config,
repository_path,
config,
local_borg_version,
options,
archive=None,
@ -22,12 +22,13 @@ def run_arbitrary_borg(
remote_path=None,
):
'''
Given a local or remote repository path, a storage config dict, the local Borg version, a
Given a local or remote repository path, a configuration dict, the local Borg version, a
sequence of arbitrary command-line Borg options, and an optional archive name, run an arbitrary
Borg command on the given repository/archive.
Borg command, passing in REPOSITORY and ARCHIVE environment variables for optional use in the
command.
'''
borgmatic.logger.add_custom_log_levels()
lock_wait = storage_config.get('lock_wait', None)
lock_wait = config.get('lock_wait', None)
try:
options = options[1:] if options[0] == '--' else options
@ -36,33 +37,36 @@ def run_arbitrary_borg(
command_options_start_index = 2 if options[0] in BORG_SUBCOMMANDS_WITH_SUBCOMMANDS else 1
borg_command = tuple(options[:command_options_start_index])
command_options = tuple(options[command_options_start_index:])
if borg_command and borg_command[0] in borgmatic.commands.arguments.ACTION_ALIASES.keys():
logger.warning(
f"Borg's {borg_command[0]} subcommand is supported natively by borgmatic. Try this instead: borgmatic {borg_command[0]}"
)
except IndexError:
borg_command = ()
command_options = ()
if borg_command in BORG_SUBCOMMANDS_WITHOUT_REPOSITORY:
repository_archive_flags = ()
elif archive:
repository_archive_flags = flags.make_repository_archive_flags(
repository, archive, local_borg_version
)
else:
repository_archive_flags = flags.make_repository_flags(repository, local_borg_version)
full_command = (
(local_path,)
+ borg_command
+ repository_archive_flags
+ command_options
+ (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+ (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+ flags.make_flags('remote-path', remote_path)
+ flags.make_flags('lock-wait', lock_wait)
+ command_options
)
return execute_command(
full_command,
output_log_level=logging.ANSWER,
tuple(shlex.quote(part) for part in full_command),
output_file=DO_NOT_CAPTURE,
shell=True,
extra_environment=dict(
(environment.make_environment(config) or {}),
**{
'BORG_REPO': repository_path,
'ARCHIVE': archive if archive else '',
},
),
borg_local_path=local_path,
extra_environment=environment.make_environment(storage_config),
borg_exit_codes=config.get('borg_exit_codes'),
)

View File

@ -7,25 +7,36 @@ logger = logging.getLogger(__name__)
def break_lock(
repository, storage_config, local_borg_version, local_path='borg', remote_path=None,
repository_path,
config,
local_borg_version,
global_arguments,
local_path='borg',
remote_path=None,
):
'''
Given a local or remote repository path, a storage configuration dict, the local Borg version,
and optional local and remote Borg paths, break any repository and cache locks leftover from Borg
aborting.
Given a local or remote repository path, a configuration dict, the local Borg version, an
argparse.Namespace of global arguments, and optional local and remote Borg paths, break any
repository and cache locks leftover from Borg aborting.
'''
umask = storage_config.get('umask', None)
lock_wait = storage_config.get('lock_wait', None)
umask = config.get('umask', None)
lock_wait = config.get('lock_wait', None)
full_command = (
(local_path, 'break-lock')
+ (('--remote-path', remote_path) if remote_path else ())
+ (('--umask', str(umask)) if umask else ())
+ (('--log-json',) if global_arguments.log_json else ())
+ (('--lock-wait', str(lock_wait)) if lock_wait else ())
+ (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+ (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+ flags.make_repository_flags(repository, local_borg_version)
+ flags.make_repository_flags(repository_path, local_borg_version)
)
borg_environment = environment.make_environment(storage_config)
execute_command(full_command, borg_local_path=local_path, extra_environment=borg_environment)
borg_environment = environment.make_environment(config)
execute_command(
full_command,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)

View File

@ -1,5 +1,7 @@
import argparse
import datetime
import hashlib
import itertools
import json
import logging
import os
@ -12,18 +14,17 @@ DEFAULT_CHECKS = (
{'name': 'repository', 'frequency': '1 month'},
{'name': 'archives', 'frequency': '1 month'},
)
DEFAULT_PREFIX = '{hostname}-'
logger = logging.getLogger(__name__)
def parse_checks(consistency_config, only_checks=None):
def parse_checks(config, only_checks=None):
'''
Given a consistency config with a "checks" sequence of dicts and an optional list of override
Given a configuration dict with a "checks" sequence of dicts and an optional list of override
checks, return a tuple of named checks to run.
For example, given a retention config of:
For example, given a config of:
{'checks': ({'name': 'repository'}, {'name': 'archives'})}
@ -35,11 +36,14 @@ def parse_checks(consistency_config, only_checks=None):
has a name of "disabled", return an empty tuple, meaning that no checks should be run.
'''
checks = only_checks or tuple(
check_config['name']
for check_config in (consistency_config.get('checks', None) or DEFAULT_CHECKS)
check_config['name'] for check_config in (config.get('checks', None) or DEFAULT_CHECKS)
)
checks = tuple(check.lower() for check in checks)
if 'disabled' in checks:
logger.warning(
'The "disabled" value for the "checks" option is deprecated and will be removed from a future release; use "skip_actions" instead'
)
if len(checks) > 1:
logger.warning(
'Multiple checks are configured, but one of them is "disabled"; not running any checks'
@ -89,17 +93,22 @@ def parse_frequency(frequency):
def filter_checks_on_frequency(
location_config, consistency_config, borg_repository_id, checks, force
config,
borg_repository_id,
checks,
force,
archives_check_id=None,
):
'''
Given a location config, a consistency config with a "checks" sequence of dicts, a Borg
repository ID, a sequence of checks, and whether to force checks to run, filter down those
checks based on the configured "frequency" for each check as compared to its check time file.
Given a configuration dict with a "checks" sequence of dicts, a Borg repository ID, a sequence
of checks, whether to force checks to run, and an ID for the archives check potentially being
run (if any), filter down those checks based on the configured "frequency" for each check as
compared to its check time file.
In other words, a check whose check time file's timestamp is too new (based on the configured
frequency) will get cut from the returned sequence of checks. Example:
consistency_config = {
config = {
'checks': [
{
'name': 'archives',
@ -108,18 +117,21 @@ def filter_checks_on_frequency(
]
}
When this function is called with that consistency_config and "archives" in checks, "archives"
will get filtered out of the returned result if its check time file is newer than 2 weeks old,
indicating that it's not yet time to run that check again.
When this function is called with that config and "archives" in checks, "archives" will get
filtered out of the returned result if its check time file is newer than 2 weeks old, indicating
that it's not yet time to run that check again.
Raise ValueError if a frequency cannot be parsed.
'''
if not checks:
return checks
filtered_checks = list(checks)
if force:
return tuple(filtered_checks)
for check_config in consistency_config.get('checks', DEFAULT_CHECKS):
for check_config in config.get('checks', DEFAULT_CHECKS):
check = check_config['name']
if checks and check not in checks:
continue
@ -128,9 +140,7 @@ def filter_checks_on_frequency(
if not frequency_delta:
continue
check_time = read_check_time(
make_check_time_path(location_config, borg_repository_id, check)
)
check_time = probe_for_check_time(config, borg_repository_id, check, archives_check_id)
if not check_time:
continue
@ -139,17 +149,68 @@ def filter_checks_on_frequency(
if datetime.datetime.now() < check_time + frequency_delta:
remaining = check_time + frequency_delta - datetime.datetime.now()
logger.info(
f"Skipping {check} check due to configured frequency; {remaining} until next check"
f'Skipping {check} check due to configured frequency; {remaining} until next check (use --force to check anyway)'
)
filtered_checks.remove(check)
return tuple(filtered_checks)
def make_check_flags(local_borg_version, checks, check_last=None, prefix=None):
def make_archive_filter_flags(
local_borg_version, config, checks, check_arguments, check_last=None, prefix=None
):
'''
Given the local Borg version and a parsed sequence of checks, transform the checks into tuple of
command-line flags.
Given the local Borg version, a configuration dict, a parsed sequence of checks, check arguments
as an argparse.Namespace instance, the check last value, and a consistency check prefix,
transform the checks into tuple of command-line flags for filtering archives in a check command.
If a check_last value is given and "archives" is in checks, then include a "--last" flag. And if
a prefix value is given and "archives" is in checks, then include a "--match-archives" flag.
'''
if 'archives' in checks or 'data' in checks:
return (('--last', str(check_last)) if check_last else ()) + (
(
('--match-archives', f'sh:{prefix}*')
if feature.available(feature.Feature.MATCH_ARCHIVES, local_borg_version)
else ('--glob-archives', f'{prefix}*')
)
if prefix
else (
flags.make_match_archives_flags(
check_arguments.match_archives or config.get('match_archives'),
config.get('archive_name_format'),
local_borg_version,
)
)
)
if check_last:
logger.warning(
'Ignoring check_last option, as "archives" or "data" are not in consistency checks'
)
if prefix:
logger.warning(
'Ignoring consistency prefix option, as "archives" or "data" are not in consistency checks'
)
return ()
def make_archives_check_id(archive_filter_flags):
'''
Given a sequence of flags to filter archives, return a unique hash corresponding to those
particular flags. If there are no flags, return None.
'''
if not archive_filter_flags:
return None
return hashlib.sha256(' '.join(archive_filter_flags).encode()).hexdigest()
def make_check_flags(checks, archive_filter_flags):
'''
Given a parsed sequence of checks and a sequence of flags to filter archives, transform the
checks into tuple of command-line check flags.
For example, given parsed checks of:
@ -161,10 +222,6 @@ def make_check_flags(local_borg_version, checks, check_last=None, prefix=None):
However, if both "repository" and "archives" are in checks, then omit them from the returned
flags because Borg does both checks by default. If "data" is in checks, that implies "archives".
Additionally, if a check_last value is given and "archives" is in checks, then include a
"--last" flag. And if a prefix value is given and "archives" is in checks, then include a
"--match-archives" flag.
'''
if 'data' in checks:
data_flags = ('--verify-data',)
@ -172,47 +229,38 @@ def make_check_flags(local_borg_version, checks, check_last=None, prefix=None):
else:
data_flags = ()
if 'archives' in checks:
last_flags = ('--last', str(check_last)) if check_last else ()
if feature.available(feature.Feature.MATCH_ARCHIVES, local_borg_version):
match_archives_flags = ('--match-archives', f'sh:{prefix}*') if prefix else ()
else:
match_archives_flags = ('--glob-archives', f'{prefix}*') if prefix else ()
else:
last_flags = ()
match_archives_flags = ()
if check_last:
logger.warning(
'Ignoring check_last option, as "archives" or "data" are not in consistency checks'
)
if prefix:
logger.warning(
'Ignoring consistency prefix option, as "archives" or "data" are not in consistency checks'
)
common_flags = last_flags + match_archives_flags + data_flags
common_flags = (archive_filter_flags if 'archives' in checks else ()) + data_flags
if {'repository', 'archives'}.issubset(set(checks)):
return common_flags
return (
tuple('--{}-only'.format(check) for check in checks if check in ('repository', 'archives'))
tuple(f'--{check}-only' for check in checks if check in ('repository', 'archives'))
+ common_flags
)
def make_check_time_path(location_config, borg_repository_id, check_type):
def make_check_time_path(config, borg_repository_id, check_type, archives_check_id=None):
'''
Given a location configuration dict, a Borg repository ID, and the name of a check type
("repository", "archives", etc.), return a path for recording that check's time (the time of
that check last occurring).
Given a configuration dict, a Borg repository ID, the name of a check type ("repository",
"archives", etc.), and a unique hash of the archives filter flags, return a path for recording
that check's time (the time of that check last occurring).
'''
borgmatic_source_directory = os.path.expanduser(
config.get('borgmatic_source_directory', state.DEFAULT_BORGMATIC_SOURCE_DIRECTORY)
)
if check_type in ('archives', 'data'):
return os.path.join(
borgmatic_source_directory,
'checks',
borg_repository_id,
check_type,
archives_check_id if archives_check_id else 'all',
)
return os.path.join(
os.path.expanduser(
location_config.get(
'borgmatic_source_directory', state.DEFAULT_BORGMATIC_SOURCE_DIRECTORY
)
),
borgmatic_source_directory,
'checks',
borg_repository_id,
check_type,
@ -242,24 +290,87 @@ def read_check_time(path):
return None
def probe_for_check_time(config, borg_repository_id, check, archives_check_id):
'''
Given a configuration dict, a Borg repository ID, the name of a check type ("repository",
"archives", etc.), and a unique hash of the archives filter flags, return a the corresponding
check time or None if such a check time does not exist.
When the check type is "archives" or "data", this function probes two different paths to find
the check time, e.g.:
~/.borgmatic/checks/1234567890/archives/9876543210
~/.borgmatic/checks/1234567890/archives/all
... and returns the maximum modification time of the files found (if any). The first path
represents a more specific archives check time (a check on a subset of archives), and the second
is a fallback to the last "all" archives check.
For other check types, this function reads from a single check time path, e.g.:
~/.borgmatic/checks/1234567890/repository
'''
check_times = (
read_check_time(group[0])
for group in itertools.groupby(
(
make_check_time_path(config, borg_repository_id, check, archives_check_id),
make_check_time_path(config, borg_repository_id, check),
)
)
)
try:
return max(check_time for check_time in check_times if check_time)
except ValueError:
return None
def upgrade_check_times(config, borg_repository_id):
'''
Given a configuration dict and a Borg repository ID, upgrade any corresponding check times on
disk from old-style paths to new-style paths.
Currently, the only upgrade performed is renaming an archive or data check path that looks like:
~/.borgmatic/checks/1234567890/archives
to:
~/.borgmatic/checks/1234567890/archives/all
'''
for check_type in ('archives', 'data'):
new_path = make_check_time_path(config, borg_repository_id, check_type, 'all')
old_path = os.path.dirname(new_path)
temporary_path = f'{old_path}.temp'
if not os.path.isfile(old_path) and not os.path.isfile(temporary_path):
continue
logger.debug(f'Upgrading archives check time from {old_path} to {new_path}')
try:
os.rename(old_path, temporary_path)
except FileNotFoundError:
pass
os.mkdir(old_path)
os.rename(temporary_path, new_path)
def check_archives(
repository,
location_config,
storage_config,
consistency_config,
repository_path,
config,
local_borg_version,
check_arguments,
global_arguments,
local_path='borg',
remote_path=None,
progress=None,
repair=None,
only_checks=None,
force=None,
):
'''
Given a local or remote repository path, a storage config dict, a consistency config dict,
local/remote commands to run, whether to include progress information, whether to attempt a
repair, and an optional list of checks to use instead of configured checks, check the contained
Borg archives for consistency.
Given a local or remote repository path, a configuration dict, the local Borg version, check
arguments as an argparse.Namespace instance, global arguments, and local/remote commands to run,
check the contained Borg archives for consistency.
If there are no consistency checks to run, skip running them.
@ -268,30 +379,40 @@ def check_archives(
try:
borg_repository_id = json.loads(
rinfo.display_repository_info(
repository,
storage_config,
repository_path,
config,
local_borg_version,
argparse.Namespace(json=True),
global_arguments,
local_path,
remote_path,
)
)['repository']['id']
except (json.JSONDecodeError, KeyError):
raise ValueError(f'Cannot determine Borg repository ID for {repository}')
raise ValueError(f'Cannot determine Borg repository ID for {repository_path}')
upgrade_check_times(config, borg_repository_id)
check_last = config.get('check_last', None)
prefix = config.get('prefix')
configured_checks = parse_checks(config, check_arguments.only_checks)
lock_wait = None
extra_borg_options = config.get('extra_borg_options', {}).get('check', '')
archive_filter_flags = make_archive_filter_flags(
local_borg_version, config, configured_checks, check_arguments, check_last, prefix
)
archives_check_id = make_archives_check_id(archive_filter_flags)
checks = filter_checks_on_frequency(
location_config,
consistency_config,
config,
borg_repository_id,
parse_checks(consistency_config, only_checks),
force,
configured_checks,
check_arguments.force,
archives_check_id,
)
check_last = consistency_config.get('check_last', None)
lock_wait = None
extra_borg_options = storage_config.get('extra_borg_options', {}).get('check', '')
if set(checks).intersection({'repository', 'archives', 'data'}):
lock_wait = storage_config.get('lock_wait', None)
lock_wait = config.get('lock_wait')
verbosity_flags = ()
if logger.isEnabledFor(logging.INFO):
@ -299,36 +420,53 @@ def check_archives(
if logger.isEnabledFor(logging.DEBUG):
verbosity_flags = ('--debug', '--show-rc')
prefix = consistency_config.get('prefix', DEFAULT_PREFIX)
full_command = (
(local_path, 'check')
+ (('--repair',) if repair else ())
+ make_check_flags(local_borg_version, checks, check_last, prefix)
+ (('--repair',) if check_arguments.repair else ())
+ make_check_flags(checks, archive_filter_flags)
+ (('--remote-path', remote_path) if remote_path else ())
+ (('--log-json',) if global_arguments.log_json else ())
+ (('--lock-wait', str(lock_wait)) if lock_wait else ())
+ verbosity_flags
+ (('--progress',) if progress else ())
+ (('--progress',) if check_arguments.progress else ())
+ (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+ flags.make_repository_flags(repository, local_borg_version)
+ flags.make_repository_flags(repository_path, local_borg_version)
)
borg_environment = environment.make_environment(storage_config)
borg_environment = environment.make_environment(config)
borg_exit_codes = config.get('borg_exit_codes')
# The Borg repair option triggers an interactive prompt, which won't work when output is
# captured. And progress messes with the terminal directly.
if repair or progress:
if check_arguments.repair or check_arguments.progress:
execute_command(
full_command, output_file=DO_NOT_CAPTURE, extra_environment=borg_environment
full_command,
output_file=DO_NOT_CAPTURE,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
else:
execute_command(full_command, extra_environment=borg_environment)
execute_command(
full_command,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
for check in checks:
write_check_time(make_check_time_path(location_config, borg_repository_id, check))
write_check_time(
make_check_time_path(config, borg_repository_id, check, archives_check_id)
)
if 'extract' in checks:
extract.extract_last_archive_dry_run(
storage_config, local_borg_version, repository, lock_wait, local_path, remote_path
config,
local_borg_version,
global_arguments,
repository_path,
lock_wait,
local_path,
remote_path,
)
write_check_time(make_check_time_path(location_config, borg_repository_id, 'extract'))
write_check_time(make_check_time_path(config, borg_repository_id, 'extract'))

View File

@ -8,9 +8,10 @@ logger = logging.getLogger(__name__)
def compact_segments(
dry_run,
repository,
storage_config,
repository_path,
config,
local_borg_version,
global_arguments,
local_path='borg',
remote_path=None,
progress=False,
@ -18,17 +19,18 @@ def compact_segments(
threshold=None,
):
'''
Given dry-run flag, a local or remote repository path, a storage config dict, and the local
Borg version, compact the segments in a repository.
Given dry-run flag, a local or remote repository path, a configuration dict, and the local Borg
version, compact the segments in a repository.
'''
umask = storage_config.get('umask', None)
lock_wait = storage_config.get('lock_wait', None)
extra_borg_options = storage_config.get('extra_borg_options', {}).get('compact', '')
umask = config.get('umask', None)
lock_wait = config.get('lock_wait', None)
extra_borg_options = config.get('extra_borg_options', {}).get('compact', '')
full_command = (
(local_path, 'compact')
+ (('--remote-path', remote_path) if remote_path else ())
+ (('--umask', str(umask)) if umask else ())
+ (('--log-json',) if global_arguments.log_json else ())
+ (('--lock-wait', str(lock_wait)) if lock_wait else ())
+ (('--progress',) if progress else ())
+ (('--cleanup-commits',) if cleanup_commits else ())
@ -36,16 +38,17 @@ def compact_segments(
+ (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+ (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+ (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+ flags.make_repository_flags(repository, local_borg_version)
+ flags.make_repository_flags(repository_path, local_borg_version)
)
if dry_run:
logging.info(f'{repository}: Skipping compact (dry run)')
logging.info(f'{repository_path}: Skipping compact (dry run)')
return
execute_command(
full_command,
output_log_level=logging.INFO,
extra_environment=environment.make_environment(config),
borg_local_path=local_path,
extra_environment=environment.make_environment(storage_config),
borg_exit_codes=config.get('borg_exit_codes'),
)

View File

@ -146,12 +146,12 @@ def ensure_files_readable(*filename_lists):
open(file_object).close()
def make_pattern_flags(location_config, pattern_filename=None):
def make_pattern_flags(config, pattern_filename=None):
'''
Given a location config dict with a potential patterns_from option, and a filename containing
any additional patterns, return the corresponding Borg flags for those files as a tuple.
Given a configuration dict with a potential patterns_from option, and a filename containing any
additional patterns, return the corresponding Borg flags for those files as a tuple.
'''
pattern_filenames = tuple(location_config.get('patterns_from') or ()) + (
pattern_filenames = tuple(config.get('patterns_from') or ()) + (
(pattern_filename,) if pattern_filename else ()
)
@ -162,12 +162,12 @@ def make_pattern_flags(location_config, pattern_filename=None):
)
def make_exclude_flags(location_config, exclude_filename=None):
def make_exclude_flags(config, exclude_filename=None):
'''
Given a location config dict with various exclude options, and a filename containing any exclude
Given a configuration dict with various exclude options, and a filename containing any exclude
patterns, return the corresponding Borg flags as a tuple.
'''
exclude_filenames = tuple(location_config.get('exclude_from') or ()) + (
exclude_filenames = tuple(config.get('exclude_from') or ()) + (
(exclude_filename,) if exclude_filename else ()
)
exclude_from_flags = tuple(
@ -175,17 +175,15 @@ def make_exclude_flags(location_config, exclude_filename=None):
('--exclude-from', exclude_filename) for exclude_filename in exclude_filenames
)
)
caches_flag = ('--exclude-caches',) if location_config.get('exclude_caches') else ()
caches_flag = ('--exclude-caches',) if config.get('exclude_caches') else ()
if_present_flags = tuple(
itertools.chain.from_iterable(
('--exclude-if-present', if_present)
for if_present in location_config.get('exclude_if_present', ())
for if_present in config.get('exclude_if_present', ())
)
)
keep_exclude_tags_flags = (
('--keep-exclude-tags',) if location_config.get('keep_exclude_tags') else ()
)
exclude_nodump_flags = ('--exclude-nodump',) if location_config.get('exclude_nodump') else ()
keep_exclude_tags_flags = ('--keep-exclude-tags',) if config.get('keep_exclude_tags') else ()
exclude_nodump_flags = ('--exclude-nodump',) if config.get('exclude_nodump') else ()
return (
exclude_from_flags
@ -196,7 +194,25 @@ def make_exclude_flags(location_config, exclude_filename=None):
)
DEFAULT_ARCHIVE_NAME_FORMAT = '{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}'
def make_list_filter_flags(local_borg_version, dry_run):
'''
Given the local Borg version and whether this is a dry run, return the corresponding flags for
passing to "--list --filter". The general idea is that excludes are shown for a dry run or when
the verbosity is debug.
'''
base_flags = 'AME'
show_excludes = logger.isEnabledFor(logging.DEBUG)
if feature.available(feature.Feature.EXCLUDED_FILES_MINUS, local_borg_version):
if show_excludes or dry_run:
return f'{base_flags}+-'
else:
return base_flags
if show_excludes:
return f'{base_flags}x-'
else:
return f'{base_flags}-'
def collect_borgmatic_source_directories(borgmatic_source_directory):
@ -256,26 +272,31 @@ def any_parent_directories(path, candidate_parents):
def collect_special_file_paths(
create_command, local_path, working_directory, borg_environment, skip_directories
create_command, config, local_path, working_directory, borg_environment, skip_directories
):
'''
Given a Borg create command as a tuple, a local Borg path, a working directory, and a dict of
Given a Borg create command as a tuple, a local Borg path, a working directory, a dict of
environment variables to pass to Borg, and a sequence of parent directories to skip, collect the
paths for any special files (character devices, block devices, and named pipes / FIFOs) that
Borg would encounter during a create. These are all paths that could cause Borg to hang if its
--read-special flag is used.
'''
# Omit "--exclude-nodump" from the Borg dry run command, because that flag causes Borg to open
# files including any named pipe we've created.
paths_output = execute_command_and_capture_output(
create_command + ('--dry-run', '--list'),
tuple(argument for argument in create_command if argument != '--exclude-nodump')
+ ('--dry-run', '--list'),
capture_stderr=True,
working_directory=working_directory,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
paths = tuple(
path_line.split(' ', 1)[1]
for path_line in paths_output.split('\n')
if path_line and path_line.startswith('- ')
if path_line and path_line.startswith('- ') or path_line.startswith('+ ')
)
return tuple(
@ -285,12 +306,27 @@ def collect_special_file_paths(
)
def check_all_source_directories_exist(source_directories):
'''
Given a sequence of source directories, check that they all exist. If any do not, raise an
exception.
'''
missing_directories = [
source_directory
for source_directory in source_directories
if not all([os.path.exists(directory) for directory in expand_directory(source_directory)])
]
if missing_directories:
raise ValueError(f"Source directories do not exist: {', '.join(missing_directories)}")
def create_archive(
dry_run,
repository,
location_config,
storage_config,
repository_path,
config,
config_paths,
local_borg_version,
global_arguments,
local_path='borg',
remote_path=None,
progress=False,
@ -300,66 +336,71 @@ def create_archive(
stream_processes=None,
):
'''
Given vebosity/dry-run flags, a local or remote repository path, a location config dict, and a
storage config dict, create a Borg archive and return Borg's JSON output (if any).
Given vebosity/dry-run flags, a local or remote repository path, a configuration dict, a
sequence of loaded configuration paths, the local Borg version, and global arguments as an
argparse.Namespace instance, create a Borg archive and return Borg's JSON output (if any).
If a sequence of stream processes is given (instances of subprocess.Popen), then execute the
create command while also triggering the given processes to produce output.
'''
borgmatic.logger.add_custom_log_levels()
borgmatic_source_directories = expand_directories(
collect_borgmatic_source_directories(location_config.get('borgmatic_source_directory'))
collect_borgmatic_source_directories(config.get('borgmatic_source_directory'))
)
if config.get('source_directories_must_exist', False):
check_all_source_directories_exist(config.get('source_directories'))
sources = deduplicate_directories(
map_directories_to_devices(
expand_directories(
tuple(location_config.get('source_directories', ())) + borgmatic_source_directories
tuple(config.get('source_directories', ()))
+ borgmatic_source_directories
+ tuple(config_paths if config.get('store_config_files', True) else ())
)
),
additional_directory_devices=map_directories_to_devices(
expand_directories(pattern_root_directories(location_config.get('patterns')))
expand_directories(pattern_root_directories(config.get('patterns')))
),
)
ensure_files_readable(location_config.get('patterns_from'), location_config.get('exclude_from'))
ensure_files_readable(config.get('patterns_from'), config.get('exclude_from'))
try:
working_directory = os.path.expanduser(location_config.get('working_directory'))
working_directory = os.path.expanduser(config.get('working_directory'))
except TypeError:
working_directory = None
pattern_file = (
write_pattern_file(location_config.get('patterns'), sources)
if location_config.get('patterns') or location_config.get('patterns_from')
write_pattern_file(config.get('patterns'), sources)
if config.get('patterns') or config.get('patterns_from')
else None
)
exclude_file = write_pattern_file(
expand_home_directories(location_config.get('exclude_patterns'))
)
checkpoint_interval = storage_config.get('checkpoint_interval', None)
chunker_params = storage_config.get('chunker_params', None)
compression = storage_config.get('compression', None)
upload_rate_limit = storage_config.get('upload_rate_limit', None)
umask = storage_config.get('umask', None)
lock_wait = storage_config.get('lock_wait', None)
files_cache = location_config.get('files_cache')
archive_name_format = storage_config.get('archive_name_format', DEFAULT_ARCHIVE_NAME_FORMAT)
extra_borg_options = storage_config.get('extra_borg_options', {}).get('create', '')
exclude_file = write_pattern_file(expand_home_directories(config.get('exclude_patterns')))
checkpoint_interval = config.get('checkpoint_interval', None)
checkpoint_volume = config.get('checkpoint_volume', None)
chunker_params = config.get('chunker_params', None)
compression = config.get('compression', None)
upload_rate_limit = config.get('upload_rate_limit', None)
umask = config.get('umask', None)
lock_wait = config.get('lock_wait', None)
list_filter_flags = make_list_filter_flags(local_borg_version, dry_run)
files_cache = config.get('files_cache')
archive_name_format = config.get('archive_name_format', flags.DEFAULT_ARCHIVE_NAME_FORMAT)
extra_borg_options = config.get('extra_borg_options', {}).get('create', '')
if feature.available(feature.Feature.ATIME, local_borg_version):
atime_flags = ('--atime',) if location_config.get('atime') is True else ()
atime_flags = ('--atime',) if config.get('atime') is True else ()
else:
atime_flags = ('--noatime',) if location_config.get('atime') is False else ()
atime_flags = ('--noatime',) if config.get('atime') is False else ()
if feature.available(feature.Feature.NOFLAGS, local_borg_version):
noflags_flags = ('--noflags',) if location_config.get('flags') is False else ()
noflags_flags = ('--noflags',) if config.get('flags') is False else ()
else:
noflags_flags = ('--nobsdflags',) if location_config.get('flags') is False else ()
noflags_flags = ('--nobsdflags',) if config.get('flags') is False else ()
if feature.available(feature.Feature.NUMERIC_IDS, local_borg_version):
numeric_ids_flags = ('--numeric-ids',) if location_config.get('numeric_ids') else ()
numeric_ids_flags = ('--numeric-ids',) if config.get('numeric_ids') else ()
else:
numeric_ids_flags = ('--numeric-owner',) if location_config.get('numeric_ids') else ()
numeric_ids_flags = ('--numeric-owner',) if config.get('numeric_ids') else ()
if feature.available(feature.Feature.UPLOAD_RATELIMIT, local_borg_version):
upload_ratelimit_flags = (
@ -370,42 +411,46 @@ def create_archive(
('--remote-ratelimit', str(upload_rate_limit)) if upload_rate_limit else ()
)
if stream_processes and location_config.get('read_special') is False:
if stream_processes and config.get('read_special') is False:
logger.warning(
f'{repository}: Ignoring configured "read_special" value of false, as true is needed for database hooks.'
f'{repository_path}: Ignoring configured "read_special" value of false, as true is needed for database hooks.'
)
create_command = (
create_flags = (
tuple(local_path.split(' '))
+ ('create',)
+ make_pattern_flags(location_config, pattern_file.name if pattern_file else None)
+ make_exclude_flags(location_config, exclude_file.name if exclude_file else None)
+ make_pattern_flags(config, pattern_file.name if pattern_file else None)
+ make_exclude_flags(config, exclude_file.name if exclude_file else None)
+ (('--checkpoint-interval', str(checkpoint_interval)) if checkpoint_interval else ())
+ (('--checkpoint-volume', str(checkpoint_volume)) if checkpoint_volume else ())
+ (('--chunker-params', chunker_params) if chunker_params else ())
+ (('--compression', compression) if compression else ())
+ upload_ratelimit_flags
+ (
('--one-file-system',)
if location_config.get('one_file_system') or stream_processes
else ()
)
+ (('--one-file-system',) if config.get('one_file_system') or stream_processes else ())
+ numeric_ids_flags
+ atime_flags
+ (('--noctime',) if location_config.get('ctime') is False else ())
+ (('--nobirthtime',) if location_config.get('birthtime') is False else ())
+ (('--read-special',) if location_config.get('read_special') or stream_processes else ())
+ (('--noctime',) if config.get('ctime') is False else ())
+ (('--nobirthtime',) if config.get('birthtime') is False else ())
+ (('--read-special',) if config.get('read_special') or stream_processes else ())
+ noflags_flags
+ (('--files-cache', files_cache) if files_cache else ())
+ (('--remote-path', remote_path) if remote_path else ())
+ (('--umask', str(umask)) if umask else ())
+ (('--log-json',) if global_arguments.log_json else ())
+ (('--lock-wait', str(lock_wait)) if lock_wait else ())
+ (('--list', '--filter', 'AMEx-') if list_files and not json and not progress else ())
+ (
('--list', '--filter', list_filter_flags)
if list_files and not json and not progress
else ()
)
+ (('--dry-run',) if dry_run else ())
+ (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+ flags.make_repository_archive_flags(repository, archive_name_format, local_borg_version)
+ (sources if not pattern_file else ())
)
create_positional_arguments = flags.make_repository_archive_flags(
repository_path, archive_name_format, local_borg_version
) + (sources if not pattern_file else ())
if json:
output_log_level = None
elif list_files or (stats and not dry_run):
@ -417,14 +462,15 @@ def create_archive(
# the terminal directly.
output_file = DO_NOT_CAPTURE if progress else None
borg_environment = environment.make_environment(storage_config)
borg_environment = environment.make_environment(config)
# If database hooks are enabled (as indicated by streaming processes), exclude files that might
# cause Borg to hang. But skip this if the user has explicitly set the "read_special" to True.
if stream_processes and not location_config.get('read_special'):
logger.debug(f'{repository}: Collecting special file paths')
if stream_processes and not config.get('read_special'):
logger.debug(f'{repository_path}: Collecting special file paths')
special_file_paths = collect_special_file_paths(
create_command,
create_flags + create_positional_arguments,
config,
local_path,
working_directory,
borg_environment,
@ -433,44 +479,51 @@ def create_archive(
if special_file_paths:
logger.warning(
f'{repository}: Excluding special files to prevent Borg from hanging: {", ".join(special_file_paths)}'
f'{repository_path}: Excluding special files to prevent Borg from hanging: {", ".join(special_file_paths)}'
)
exclude_file = write_pattern_file(
expand_home_directories(
tuple(location_config.get('exclude_patterns') or ()) + special_file_paths
tuple(config.get('exclude_patterns') or ()) + special_file_paths
),
pattern_file=exclude_file,
)
create_command += make_exclude_flags(location_config, exclude_file.name)
create_flags += make_exclude_flags(config, exclude_file.name)
create_command += (
create_flags += (
(('--info',) if logger.getEffectiveLevel() == logging.INFO and not json else ())
+ (('--stats',) if stats and not json and not dry_run else ())
+ (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) and not json else ())
+ (('--progress',) if progress else ())
+ (('--json',) if json else ())
)
borg_exit_codes = config.get('borg_exit_codes')
if stream_processes:
return execute_command_with_processes(
create_command,
create_flags + create_positional_arguments,
stream_processes,
output_log_level,
output_file,
borg_local_path=local_path,
working_directory=working_directory,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
elif output_log_level is None:
return execute_command_and_capture_output(
create_command, working_directory=working_directory, extra_environment=borg_environment,
create_flags + create_positional_arguments,
working_directory=working_directory,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
else:
execute_command(
create_command,
create_flags + create_positional_arguments,
output_log_level,
output_file,
borg_local_path=local_path,
working_directory=working_directory,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)

View File

@ -2,6 +2,7 @@ OPTION_TO_ENVIRONMENT_VARIABLE = {
'borg_base_directory': 'BORG_BASE_DIR',
'borg_config_directory': 'BORG_CONFIG_DIR',
'borg_cache_directory': 'BORG_CACHE_DIR',
'borg_files_cache_ttl': 'BORG_FILES_CACHE_TTL',
'borg_security_directory': 'BORG_SECURITY_DIR',
'borg_keys_directory': 'BORG_KEYS_DIR',
'encryption_passcommand': 'BORG_PASSCOMMAND',
@ -10,30 +11,47 @@ OPTION_TO_ENVIRONMENT_VARIABLE = {
'temporary_directory': 'TMPDIR',
}
DEFAULT_BOOL_OPTION_TO_ENVIRONMENT_VARIABLE = {
DEFAULT_BOOL_OPTION_TO_DOWNCASE_ENVIRONMENT_VARIABLE = {
'relocated_repo_access_is_ok': 'BORG_RELOCATED_REPO_ACCESS_IS_OK',
'unknown_unencrypted_repo_access_is_ok': 'BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK',
}
DEFAULT_BOOL_OPTION_TO_UPPERCASE_ENVIRONMENT_VARIABLE = {
'check_i_know_what_i_am_doing': 'BORG_CHECK_I_KNOW_WHAT_I_AM_DOING',
}
def make_environment(storage_config):
def make_environment(config):
'''
Given a borgmatic storage configuration dict, return its options converted to a Borg environment
Given a borgmatic configuration dict, return its options converted to a Borg environment
variable dict.
'''
environment = {}
for option_name, environment_variable_name in OPTION_TO_ENVIRONMENT_VARIABLE.items():
value = storage_config.get(option_name)
value = config.get(option_name)
if value:
environment[environment_variable_name] = value
environment[environment_variable_name] = str(value)
for (
option_name,
environment_variable_name,
) in DEFAULT_BOOL_OPTION_TO_ENVIRONMENT_VARIABLE.items():
value = storage_config.get(option_name, False)
environment[environment_variable_name] = 'yes' if value else 'no'
) in DEFAULT_BOOL_OPTION_TO_DOWNCASE_ENVIRONMENT_VARIABLE.items():
value = config.get(option_name)
if value is not None:
environment[environment_variable_name] = 'yes' if value else 'no'
for (
option_name,
environment_variable_name,
) in DEFAULT_BOOL_OPTION_TO_UPPERCASE_ENVIRONMENT_VARIABLE.items():
value = config.get(option_name)
if value is not None:
environment[environment_variable_name] = 'YES' if value else 'NO'
# On Borg 1.4.0a1+, take advantage of more specific exit codes. No effect on
# older versions of Borg.
environment['BORG_EXIT_CODES'] = 'modern'
return environment

View File

@ -0,0 +1,71 @@
import logging
import os
import borgmatic.logger
from borgmatic.borg import environment, flags
from borgmatic.execute import DO_NOT_CAPTURE, execute_command
logger = logging.getLogger(__name__)
def export_key(
repository_path,
config,
local_borg_version,
export_arguments,
global_arguments,
local_path='borg',
remote_path=None,
):
'''
Given a local or remote repository path, a configuration dict, the local Borg version, and
optional local and remote Borg paths, export the repository key to the destination path
indicated in the export arguments.
If the destination path is empty or "-", then print the key to stdout instead of to a file.
Raise FileExistsError if a path is given but it already exists on disk.
'''
borgmatic.logger.add_custom_log_levels()
umask = config.get('umask', None)
lock_wait = config.get('lock_wait', None)
if export_arguments.path and export_arguments.path != '-':
if os.path.exists(export_arguments.path):
raise FileExistsError(
f'Destination path {export_arguments.path} already exists. Aborting.'
)
output_file = None
else:
output_file = DO_NOT_CAPTURE
full_command = (
(local_path, 'key', 'export')
+ (('--remote-path', remote_path) if remote_path else ())
+ (('--umask', str(umask)) if umask else ())
+ (('--log-json',) if global_arguments.log_json else ())
+ (('--lock-wait', str(lock_wait)) if lock_wait else ())
+ (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+ (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+ flags.make_flags('paper', export_arguments.paper)
+ flags.make_flags('qr-html', export_arguments.qr_html)
+ flags.make_repository_flags(
repository_path,
local_borg_version,
)
+ ((export_arguments.path,) if output_file is None else ())
)
if global_arguments.dry_run:
logging.info(f'{repository_path}: Skipping key export (dry run)')
return
execute_command(
full_command,
output_file=output_file,
output_log_level=logging.ANSWER,
extra_environment=environment.make_environment(config),
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)

View File

@ -1,5 +1,4 @@
import logging
import os
import borgmatic.logger
from borgmatic.borg import environment, flags
@ -10,12 +9,13 @@ logger = logging.getLogger(__name__)
def export_tar_archive(
dry_run,
repository,
repository_path,
archive,
paths,
destination_path,
storage_config,
config,
local_borg_version,
global_arguments,
local_path='borg',
remote_path=None,
tar_filter=None,
@ -24,21 +24,22 @@ def export_tar_archive(
):
'''
Given a dry-run flag, a local or remote repository path, an archive name, zero or more paths to
export from the archive, a destination path to export to, a storage configuration dict, the
local Borg version, optional local and remote Borg paths, an optional filter program, whether to
include per-file details, and an optional number of path components to strip, export the archive
into the given destination path as a tar-formatted file.
export from the archive, a destination path to export to, a configuration dict, the local Borg
version, optional local and remote Borg paths, an optional filter program, whether to include
per-file details, and an optional number of path components to strip, export the archive into
the given destination path as a tar-formatted file.
If the destination path is "-", then stream the output to stdout instead of to a file.
'''
borgmatic.logger.add_custom_log_levels()
umask = storage_config.get('umask', None)
lock_wait = storage_config.get('lock_wait', None)
umask = config.get('umask', None)
lock_wait = config.get('lock_wait', None)
full_command = (
(local_path, 'export-tar')
+ (('--remote-path', remote_path) if remote_path else ())
+ (('--umask', str(umask)) if umask else ())
+ (('--log-json',) if global_arguments.log_json else ())
+ (('--lock-wait', str(lock_wait)) if lock_wait else ())
+ (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+ (('--list',) if list_files else ())
@ -47,7 +48,7 @@ def export_tar_archive(
+ (('--tar-filter', tar_filter) if tar_filter else ())
+ (('--strip-components', str(strip_components)) if strip_components else ())
+ flags.make_repository_archive_flags(
repository if ':' in repository else os.path.abspath(repository),
repository_path,
archive,
local_borg_version,
)
@ -61,13 +62,14 @@ def export_tar_archive(
output_log_level = logging.INFO
if dry_run:
logging.info('{}: Skipping export to tar file (dry run)'.format(repository))
logging.info(f'{repository_path}: Skipping export to tar file (dry run)')
return
execute_command(
full_command,
output_file=DO_NOT_CAPTURE if destination_path == '-' else None,
output_log_level=output_log_level,
extra_environment=environment.make_environment(config),
borg_local_path=local_path,
extra_environment=environment.make_environment(storage_config),
borg_exit_codes=config.get('borg_exit_codes'),
)

View File

@ -2,6 +2,7 @@ import logging
import os
import subprocess
import borgmatic.config.validate
from borgmatic.borg import environment, feature, flags, rlist
from borgmatic.execute import DO_NOT_CAPTURE, execute_command
@ -9,9 +10,10 @@ logger = logging.getLogger(__name__)
def extract_last_archive_dry_run(
storage_config,
config,
local_borg_version,
repository,
global_arguments,
repository_path,
lock_wait=None,
local_path='borg',
remote_path=None,
@ -20,8 +22,6 @@ def extract_last_archive_dry_run(
Perform an extraction dry-run of the most recent archive. If there are no archives, skip the
dry-run.
'''
remote_path_flags = ('--remote-path', remote_path) if remote_path else ()
lock_wait_flags = ('--lock-wait', str(lock_wait)) if lock_wait else ()
verbosity_flags = ()
if logger.isEnabledFor(logging.DEBUG):
verbosity_flags = ('--debug', '--show-rc')
@ -30,25 +30,38 @@ def extract_last_archive_dry_run(
try:
last_archive_name = rlist.resolve_archive_name(
repository, 'latest', storage_config, local_borg_version, local_path, remote_path
repository_path,
'latest',
config,
local_borg_version,
global_arguments,
local_path,
remote_path,
)
except ValueError:
logger.warning('No archives found. Skipping extract consistency check.')
return
list_flag = ('--list',) if logger.isEnabledFor(logging.DEBUG) else ()
borg_environment = environment.make_environment(storage_config)
borg_environment = environment.make_environment(config)
full_extract_command = (
(local_path, 'extract', '--dry-run')
+ remote_path_flags
+ lock_wait_flags
+ (('--remote-path', remote_path) if remote_path else ())
+ (('--log-json',) if global_arguments.log_json else ())
+ (('--lock-wait', str(lock_wait)) if lock_wait else ())
+ verbosity_flags
+ list_flag
+ flags.make_repository_archive_flags(repository, last_archive_name, local_borg_version)
+ flags.make_repository_archive_flags(
repository_path, last_archive_name, local_borg_version
)
)
execute_command(
full_extract_command, working_directory=None, extra_environment=borg_environment
full_extract_command,
working_directory=None,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
@ -57,9 +70,9 @@ def extract_archive(
repository,
archive,
paths,
location_config,
storage_config,
config,
local_borg_version,
global_arguments,
local_path='borg',
remote_path=None,
destination_path=None,
@ -69,29 +82,37 @@ def extract_archive(
):
'''
Given a dry-run flag, a local or remote repository path, an archive name, zero or more paths to
restore from the archive, the local Borg version string, location/storage configuration dicts,
optional local and remote Borg paths, and an optional destination path to extract to, extract
the archive into the current directory.
restore from the archive, the local Borg version string, an argparse.Namespace of global
arguments, a configuration dict, optional local and remote Borg paths, and an optional
destination path to extract to, extract the archive into the current directory.
If extract to stdout is True, then start the extraction streaming to stdout, and return that
extract process as an instance of subprocess.Popen.
'''
umask = storage_config.get('umask', None)
lock_wait = storage_config.get('lock_wait', None)
umask = config.get('umask', None)
lock_wait = config.get('lock_wait', None)
if progress and extract_to_stdout:
raise ValueError('progress and extract_to_stdout cannot both be set')
if feature.available(feature.Feature.NUMERIC_IDS, local_borg_version):
numeric_ids_flags = ('--numeric-ids',) if location_config.get('numeric_ids') else ()
numeric_ids_flags = ('--numeric-ids',) if config.get('numeric_ids') else ()
else:
numeric_ids_flags = ('--numeric-owner',) if location_config.get('numeric_ids') else ()
numeric_ids_flags = ('--numeric-owner',) if config.get('numeric_ids') else ()
if strip_components == 'all':
if not paths:
raise ValueError('The --strip-components flag with "all" requires at least one --path')
# Calculate the maximum number of leading path components of the given paths.
strip_components = max(0, *(len(path.split(os.path.sep)) - 1 for path in paths))
full_command = (
(local_path, 'extract')
+ (('--remote-path', remote_path) if remote_path else ())
+ numeric_ids_flags
+ (('--umask', str(umask)) if umask else ())
+ (('--log-json',) if global_arguments.log_json else ())
+ (('--lock-wait', str(lock_wait)) if lock_wait else ())
+ (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+ (('--debug', '--list', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
@ -100,14 +121,17 @@ def extract_archive(
+ (('--progress',) if progress else ())
+ (('--stdout',) if extract_to_stdout else ())
+ flags.make_repository_archive_flags(
repository if ':' in repository else os.path.abspath(repository),
# Make the repository path absolute so the working directory changes below don't
# prevent Borg from finding the repo.
borgmatic.config.validate.normalize_repository_path(repository),
archive,
local_borg_version,
)
+ (tuple(paths) if paths else ())
)
borg_environment = environment.make_environment(storage_config)
borg_environment = environment.make_environment(config)
borg_exit_codes = config.get('borg_exit_codes')
# The progress output isn't compatible with captured and logged output, as progress messes with
# the terminal directly.
@ -117,6 +141,8 @@ def extract_archive(
output_file=DO_NOT_CAPTURE,
working_directory=destination_path,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
return None
@ -127,10 +153,16 @@ def extract_archive(
working_directory=destination_path,
run_to_completion=False,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
# Don't give Borg local path so as to error on warnings, as "borg extract" only gives a warning
# if the restore paths don't exist in the archive.
execute_command(
full_command, working_directory=destination_path, extra_environment=borg_environment
full_command,
working_directory=destination_path,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)

View File

@ -1,6 +1,6 @@
from enum import Enum
from pkg_resources import parse_version
from packaging.version import parse
class Feature(Enum):
@ -14,19 +14,21 @@ class Feature(Enum):
RLIST = 8
RINFO = 9
MATCH_ARCHIVES = 10
EXCLUDED_FILES_MINUS = 11
FEATURE_TO_MINIMUM_BORG_VERSION = {
Feature.COMPACT: parse_version('1.2.0a2'), # borg compact
Feature.ATIME: parse_version('1.2.0a7'), # borg create --atime
Feature.NOFLAGS: parse_version('1.2.0a8'), # borg create --noflags
Feature.NUMERIC_IDS: parse_version('1.2.0b3'), # borg create/extract/mount --numeric-ids
Feature.UPLOAD_RATELIMIT: parse_version('1.2.0b3'), # borg create --upload-ratelimit
Feature.SEPARATE_REPOSITORY_ARCHIVE: parse_version('2.0.0a2'), # --repo with separate archive
Feature.RCREATE: parse_version('2.0.0a2'), # borg rcreate
Feature.RLIST: parse_version('2.0.0a2'), # borg rlist
Feature.RINFO: parse_version('2.0.0a2'), # borg rinfo
Feature.MATCH_ARCHIVES: parse_version('2.0.0b3'), # borg --match-archives
Feature.COMPACT: parse('1.2.0a2'), # borg compact
Feature.ATIME: parse('1.2.0a7'), # borg create --atime
Feature.NOFLAGS: parse('1.2.0a8'), # borg create --noflags
Feature.NUMERIC_IDS: parse('1.2.0b3'), # borg create/extract/mount --numeric-ids
Feature.UPLOAD_RATELIMIT: parse('1.2.0b3'), # borg create --upload-ratelimit
Feature.SEPARATE_REPOSITORY_ARCHIVE: parse('2.0.0a2'), # --repo with separate archive
Feature.RCREATE: parse('2.0.0a2'), # borg rcreate
Feature.RLIST: parse('2.0.0a2'), # borg rlist
Feature.RINFO: parse('2.0.0a2'), # borg rinfo
Feature.MATCH_ARCHIVES: parse('2.0.0b3'), # borg --match-archives
Feature.EXCLUDED_FILES_MINUS: parse('2.0.0b5'), # --list --filter uses "-" for excludes
}
@ -35,4 +37,4 @@ def available(feature, borg_version):
Given a Borg Feature constant and a Borg version string, return whether that feature is
available in that version of Borg.
'''
return FEATURE_TO_MINIMUM_BORG_VERSION[feature] <= parse_version(borg_version)
return FEATURE_TO_MINIMUM_BORG_VERSION[feature] <= parse(borg_version)

View File

@ -1,7 +1,12 @@
import itertools
import json
import logging
import re
from borgmatic.borg import feature
logger = logging.getLogger(__name__)
def make_flags(name, value):
'''
@ -10,7 +15,7 @@ def make_flags(name, value):
if not value:
return ()
flag = '--{}'.format(name.replace('_', '-'))
flag = f"--{name.replace('_', '-')}"
if value is True:
return (flag,)
@ -33,7 +38,7 @@ def make_flags_from_arguments(arguments, excludes=()):
)
def make_repository_flags(repository, local_borg_version):
def make_repository_flags(repository_path, local_borg_version):
'''
Given the path of a Borg repository and the local Borg version, return Borg-version-appropriate
command-line flags (as a tuple) for selecting that repository.
@ -42,17 +47,69 @@ def make_repository_flags(repository, local_borg_version):
('--repo',)
if feature.available(feature.Feature.SEPARATE_REPOSITORY_ARCHIVE, local_borg_version)
else ()
) + (repository,)
) + (repository_path,)
def make_repository_archive_flags(repository, archive, local_borg_version):
def make_repository_archive_flags(repository_path, archive, local_borg_version):
'''
Given the path of a Borg repository, an archive name or pattern, and the local Borg version,
return Borg-version-appropriate command-line flags (as a tuple) for selecting that repository
and archive.
'''
return (
('--repo', repository, archive)
('--repo', repository_path, archive)
if feature.available(feature.Feature.SEPARATE_REPOSITORY_ARCHIVE, local_borg_version)
else (f'{repository}::{archive}',)
else (f'{repository_path}::{archive}',)
)
DEFAULT_ARCHIVE_NAME_FORMAT = '{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}' # noqa: FS003
def make_match_archives_flags(match_archives, archive_name_format, local_borg_version):
'''
Return match archives flags based on the given match archives value, if any. If it isn't set,
return match archives flags to match archives created with the given (or default) archive name
format. This is done by replacing certain archive name format placeholders for ephemeral data
(like "{now}") with globs.
'''
if match_archives:
if feature.available(feature.Feature.MATCH_ARCHIVES, local_borg_version):
return ('--match-archives', match_archives)
else:
return ('--glob-archives', re.sub(r'^sh:', '', match_archives))
derived_match_archives = re.sub(
r'\{(now|utcnow|pid)([:%\w\.-]*)\}', '*', archive_name_format or DEFAULT_ARCHIVE_NAME_FORMAT
)
if derived_match_archives == '*':
return ()
if feature.available(feature.Feature.MATCH_ARCHIVES, local_borg_version):
return ('--match-archives', f'sh:{derived_match_archives}')
else:
return ('--glob-archives', f'{derived_match_archives}')
def warn_for_aggressive_archive_flags(json_command, json_output):
'''
Given a JSON archives command and the resulting JSON string output from running it, parse the
JSON and warn if the command used an archive flag but the output indicates zero archives were
found.
'''
archive_flags_used = {'--glob-archives', '--match-archives'}.intersection(set(json_command))
if not archive_flags_used:
return
try:
if len(json.loads(json_output)['archives']) == 0:
logger.warning('An archive filter was applied, but no matching archives were found.')
logger.warning(
'Try adding --match-archives "*" or adjusting archive_name_format/match_archives in configuration.'
)
except json.JSONDecodeError as error:
logger.debug(f'Cannot parse JSON output from archive command: {error}')
except (TypeError, KeyError):
logger.debug('Cannot parse JSON output from archive command: No "archives" key found')

View File

@ -1,3 +1,4 @@
import argparse
import logging
import borgmatic.logger
@ -7,23 +8,21 @@ from borgmatic.execute import execute_command, execute_command_and_capture_outpu
logger = logging.getLogger(__name__)
def display_archives_info(
repository,
storage_config,
def make_info_command(
repository_path,
config,
local_borg_version,
info_arguments,
local_path='borg',
remote_path=None,
global_arguments,
local_path,
remote_path,
):
'''
Given a local or remote repository path, a storage config dict, the local Borg version, and the
arguments to the info action, display summary information for Borg archives in the repository or
return JSON summary information.
Given a local or remote repository path, a configuration dict, the local Borg version, the
arguments to the info action as an argparse.Namespace, and global arguments, return a command
as a tuple to display summary information for archives in the repository.
'''
borgmatic.logger.add_custom_log_levels()
lock_wait = storage_config.get('lock_wait', None)
full_command = (
return (
(local_path, 'info')
+ (
('--info',)
@ -36,7 +35,8 @@ def display_archives_info(
else ()
)
+ flags.make_flags('remote-path', remote_path)
+ flags.make_flags('lock-wait', lock_wait)
+ flags.make_flags('log-json', global_arguments.log_json)
+ flags.make_flags('lock-wait', config.get('lock_wait'))
+ (
(
flags.make_flags('match-archives', f'sh:{info_arguments.prefix}*')
@ -44,27 +44,75 @@ def display_archives_info(
else flags.make_flags('glob-archives', f'{info_arguments.prefix}*')
)
if info_arguments.prefix
else ()
else (
flags.make_match_archives_flags(
info_arguments.match_archives
or info_arguments.archive
or config.get('match_archives'),
config.get('archive_name_format'),
local_borg_version,
)
)
)
+ flags.make_flags_from_arguments(
info_arguments, excludes=('repository', 'archive', 'prefix')
)
+ flags.make_repository_flags(repository, local_borg_version)
+ (
flags.make_flags('match-archives', info_arguments.archive)
if feature.available(feature.Feature.MATCH_ARCHIVES, local_borg_version)
else flags.make_flags('glob-archives', info_arguments.archive)
info_arguments, excludes=('repository', 'archive', 'prefix', 'match_archives')
)
+ flags.make_repository_flags(repository_path, local_borg_version)
)
def display_archives_info(
repository_path,
config,
local_borg_version,
info_arguments,
global_arguments,
local_path='borg',
remote_path=None,
):
'''
Given a local or remote repository path, a configuration dict, the local Borg version, the
arguments to the info action as an argparse.Namespace, and global arguments, display summary
information for Borg archives in the repository or return JSON summary information.
'''
borgmatic.logger.add_custom_log_levels()
main_command = make_info_command(
repository_path,
config,
local_borg_version,
info_arguments,
global_arguments,
local_path,
remote_path,
)
json_command = make_info_command(
repository_path,
config,
local_borg_version,
argparse.Namespace(**dict(info_arguments.__dict__, json=True)),
global_arguments,
local_path,
remote_path,
)
borg_exit_codes = config.get('borg_exit_codes')
json_info = execute_command_and_capture_output(
json_command,
extra_environment=environment.make_environment(config),
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
if info_arguments.json:
return execute_command_and_capture_output(
full_command, extra_environment=environment.make_environment(storage_config),
)
else:
execute_command(
full_command,
output_log_level=logging.ANSWER,
borg_local_path=local_path,
extra_environment=environment.make_environment(storage_config),
)
return json_info
flags.warn_for_aggressive_archive_flags(json_command, json_info)
execute_command(
main_command,
output_log_level=logging.ANSWER,
extra_environment=environment.make_environment(config),
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)

View File

@ -14,26 +14,26 @@ ARCHIVE_FILTER_FLAGS_MOVED_TO_RLIST = ('prefix', 'match_archives', 'sort_by', 'f
MAKE_FLAGS_EXCLUDES = (
'repository',
'archive',
'successful',
'paths',
'find_paths',
) + ARCHIVE_FILTER_FLAGS_MOVED_TO_RLIST
def make_list_command(
repository,
storage_config,
repository_path,
config,
local_borg_version,
list_arguments,
global_arguments,
local_path='borg',
remote_path=None,
):
'''
Given a local or remote repository path, a storage config dict, the arguments to the list
action, and local and remote Borg paths, return a command as a tuple to list archives or paths
within an archive.
Given a local or remote repository path, a configuration dict, the arguments to the list action,
and local and remote Borg paths, return a command as a tuple to list archives or paths within an
archive.
'''
lock_wait = storage_config.get('lock_wait', None)
lock_wait = config.get('lock_wait', None)
return (
(local_path, 'list')
@ -48,14 +48,15 @@ def make_list_command(
else ()
)
+ flags.make_flags('remote-path', remote_path)
+ flags.make_flags('log-json', global_arguments.log_json)
+ flags.make_flags('lock-wait', lock_wait)
+ flags.make_flags_from_arguments(list_arguments, excludes=MAKE_FLAGS_EXCLUDES)
+ (
flags.make_repository_archive_flags(
repository, list_arguments.archive, local_borg_version
repository_path, list_arguments.archive, local_borg_version
)
if list_arguments.archive
else flags.make_repository_flags(repository, local_borg_version)
else flags.make_repository_flags(repository_path, local_borg_version)
)
+ (tuple(list_arguments.paths) if list_arguments.paths else ())
)
@ -78,47 +79,54 @@ def make_find_paths(find_paths):
return ()
return tuple(
find_path
if re.compile(r'([-!+RrPp] )|(\w\w:)').match(find_path)
else f'sh:**/*{find_path}*/**'
(
find_path
if re.compile(r'([-!+RrPp] )|(\w\w:)').match(find_path)
else f'sh:**/*{find_path}*/**'
)
for find_path in find_paths
)
def capture_archive_listing(
repository,
repository_path,
archive,
storage_config,
config,
local_borg_version,
list_path=None,
global_arguments,
list_paths=None,
local_path='borg',
remote_path=None,
):
'''
Given a local or remote repository path, an archive name, a storage config dict, the local Borg
version, the archive path in which to list files, and local and remote Borg paths, capture the
output of listing that archive and return it as a list of file paths.
Given a local or remote repository path, an archive name, a configuration dict, the local Borg
version, global arguments as an argparse.Namespace, the archive paths in which to list files, and
local and remote Borg paths, capture the output of listing that archive and return it as a list
of file paths.
'''
borg_environment = environment.make_environment(storage_config)
borg_environment = environment.make_environment(config)
return tuple(
execute_command_and_capture_output(
make_list_command(
repository,
storage_config,
repository_path,
config,
local_borg_version,
argparse.Namespace(
repository=repository,
repository=repository_path,
archive=archive,
paths=[f'sh:{list_path}'],
paths=[f'sh:{path}' for path in list_paths] if list_paths else None,
find_paths=None,
json=None,
format='{path}{NL}',
format='{path}{NL}', # noqa: FS003
),
global_arguments,
local_path,
remote_path,
),
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
.strip('\n')
.split('\n')
@ -126,19 +134,21 @@ def capture_archive_listing(
def list_archive(
repository,
storage_config,
repository_path,
config,
local_borg_version,
list_arguments,
global_arguments,
local_path='borg',
remote_path=None,
):
'''
Given a local or remote repository path, a storage config dict, the local Borg version, the
arguments to the list action, and local and remote Borg paths, display the output of listing
the files of a Borg archive (or return JSON output). If list_arguments.find_paths are given,
list the files by searching across multiple archives. If neither find_paths nor archive name
are given, instead list the archives in the given repository.
Given a local or remote repository path, a configuration dict, the local Borg version, global
arguments as an argparse.Namespace, the arguments to the list action as an argparse.Namespace,
and local and remote Borg paths, display the output of listing the files of a Borg archive (or
return JSON output). If list_arguments.find_paths are given, list the files by searching across
multiple archives. If neither find_paths nor archive name are given, instead list the archives
in the given repository.
'''
borgmatic.logger.add_custom_log_levels()
@ -149,7 +159,7 @@ def list_archive(
)
rlist_arguments = argparse.Namespace(
repository=repository,
repository=repository_path,
short=list_arguments.short,
format=list_arguments.format,
json=list_arguments.json,
@ -160,7 +170,13 @@ def list_archive(
last=list_arguments.last,
)
return rlist.list_repository(
repository, storage_config, local_borg_version, rlist_arguments, local_path, remote_path
repository_path,
config,
local_borg_version,
rlist_arguments,
global_arguments,
local_path,
remote_path,
)
if list_arguments.archive:
@ -175,13 +191,14 @@ def list_archive(
'The --json flag on the list action is not supported when using the --archive/--find flags.'
)
borg_environment = environment.make_environment(storage_config)
borg_environment = environment.make_environment(config)
borg_exit_codes = config.get('borg_exit_codes')
# If there are any paths to find (and there's not a single archive already selected), start by
# getting a list of archives to search.
if list_arguments.find_paths and not list_arguments.archive:
rlist_arguments = argparse.Namespace(
repository=repository,
repository=repository_path,
short=True,
format=None,
json=None,
@ -196,14 +213,17 @@ def list_archive(
archive_lines = tuple(
execute_command_and_capture_output(
rlist.make_rlist_command(
repository,
storage_config,
repository_path,
config,
local_borg_version,
rlist_arguments,
global_arguments,
local_path,
remote_path,
),
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
.strip('\n')
.split('\n')
@ -213,7 +233,7 @@ def list_archive(
# For each archive listed by Borg, run list on the contents of that archive.
for archive in archive_lines:
logger.answer(f'{repository}: Listing archive {archive}')
logger.answer(f'{repository_path}: Listing archive {archive}')
archive_arguments = copy.copy(list_arguments)
archive_arguments.archive = archive
@ -224,10 +244,11 @@ def list_archive(
setattr(archive_arguments, name, None)
main_command = make_list_command(
repository,
storage_config,
repository_path,
config,
local_borg_version,
archive_arguments,
global_arguments,
local_path,
remote_path,
) + make_find_paths(list_arguments.find_paths)
@ -235,6 +256,7 @@ def list_archive(
execute_command(
main_command,
output_log_level=logging.ANSWER,
borg_local_path=local_path,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)

View File

@ -7,38 +7,40 @@ logger = logging.getLogger(__name__)
def mount_archive(
repository,
repository_path,
archive,
mount_point,
paths,
foreground,
options,
storage_config,
mount_arguments,
config,
local_borg_version,
global_arguments,
local_path='borg',
remote_path=None,
):
'''
Given a local or remote repository path, an optional archive name, a filesystem mount point,
zero or more paths to mount from the archive, extra Borg mount options, a storage configuration
dict, the local Borg version, and optional local and remote Borg paths, mount the archive onto
the mount point.
dict, the local Borg version, global arguments as an argparse.Namespace instance, and optional
local and remote Borg paths, mount the archive onto the mount point.
'''
umask = storage_config.get('umask', None)
lock_wait = storage_config.get('lock_wait', None)
umask = config.get('umask', None)
lock_wait = config.get('lock_wait', None)
full_command = (
(local_path, 'mount')
+ (('--remote-path', remote_path) if remote_path else ())
+ (('--umask', str(umask)) if umask else ())
+ (('--log-json',) if global_arguments.log_json else ())
+ (('--lock-wait', str(lock_wait)) if lock_wait else ())
+ (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+ (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+ (('--foreground',) if foreground else ())
+ (('-o', options) if options else ())
+ flags.make_flags_from_arguments(
mount_arguments,
excludes=('repository', 'archive', 'mount_point', 'paths', 'options'),
)
+ (('-o', mount_arguments.options) if mount_arguments.options else ())
+ (
(
flags.make_repository_flags(repository, local_borg_version)
flags.make_repository_flags(repository_path, local_borg_version)
+ (
('--match-archives', archive)
if feature.available(feature.Feature.MATCH_ARCHIVES, local_borg_version)
@ -47,25 +49,31 @@ def mount_archive(
)
if feature.available(feature.Feature.SEPARATE_REPOSITORY_ARCHIVE, local_borg_version)
else (
flags.make_repository_archive_flags(repository, archive, local_borg_version)
flags.make_repository_archive_flags(repository_path, archive, local_borg_version)
if archive
else flags.make_repository_flags(repository, local_borg_version)
else flags.make_repository_flags(repository_path, local_borg_version)
)
)
+ (mount_point,)
+ (tuple(paths) if paths else ())
+ (mount_arguments.mount_point,)
+ (tuple(mount_arguments.paths) if mount_arguments.paths else ())
)
borg_environment = environment.make_environment(storage_config)
borg_environment = environment.make_environment(config)
# Don't capture the output when foreground mode is used so that ctrl-C can work properly.
if foreground:
if mount_arguments.foreground:
execute_command(
full_command,
output_file=DO_NOT_CAPTURE,
borg_local_path=local_path,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
return
execute_command(full_command, borg_local_path=local_path, extra_environment=borg_environment)
execute_command(
full_command,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)

View File

@ -7,10 +7,10 @@ from borgmatic.execute import execute_command
logger = logging.getLogger(__name__)
def make_prune_flags(retention_config, local_borg_version):
def make_prune_flags(config, local_borg_version):
'''
Given a retention config dict mapping from option name to value, tranform it into an iterable of
command-line name-value flag pairs.
Given a configuration dict mapping from option name to value, transform it into an sequence of
command-line flags.
For example, given a retention config of:
@ -23,61 +23,70 @@ def make_prune_flags(retention_config, local_borg_version):
('--keep-monthly', '6'),
)
'''
config = retention_config.copy()
prefix = config.pop('prefix', '{hostname}-')
flag_pairs = (
('--' + option_name.replace('_', '-'), str(value))
for option_name, value in config.items()
if option_name.startswith('keep_') and option_name != 'keep_exclude_tags'
)
prefix = config.get('prefix')
if prefix:
if feature.available(feature.Feature.MATCH_ARCHIVES, local_borg_version):
config['match_archives'] = f'sh:{prefix}*'
else:
config['glob_archives'] = f'{prefix}*'
return (
('--' + option_name.replace('_', '-'), str(value)) for option_name, value in config.items()
return tuple(element for pair in flag_pairs for element in pair) + (
(
('--match-archives', f'sh:{prefix}*')
if feature.available(feature.Feature.MATCH_ARCHIVES, local_borg_version)
else ('--glob-archives', f'{prefix}*')
)
if prefix
else (
flags.make_match_archives_flags(
config.get('match_archives'),
config.get('archive_name_format'),
local_borg_version,
)
)
)
def prune_archives(
dry_run,
repository,
storage_config,
retention_config,
repository_path,
config,
local_borg_version,
prune_arguments,
global_arguments,
local_path='borg',
remote_path=None,
stats=False,
list_archives=False,
):
'''
Given dry-run flag, a local or remote repository path, a storage config dict, and a
retention config dict, prune Borg archives according to the retention policy specified in that
configuration.
Given dry-run flag, a local or remote repository path, and a configuration dict, prune Borg
archives according to the retention policy specified in that configuration.
'''
borgmatic.logger.add_custom_log_levels()
umask = storage_config.get('umask', None)
lock_wait = storage_config.get('lock_wait', None)
extra_borg_options = storage_config.get('extra_borg_options', {}).get('prune', '')
umask = config.get('umask', None)
lock_wait = config.get('lock_wait', None)
extra_borg_options = config.get('extra_borg_options', {}).get('prune', '')
full_command = (
(local_path, 'prune')
+ tuple(
element
for pair in make_prune_flags(retention_config, local_borg_version)
for element in pair
)
+ make_prune_flags(config, local_borg_version)
+ (('--remote-path', remote_path) if remote_path else ())
+ (('--umask', str(umask)) if umask else ())
+ (('--log-json',) if global_arguments.log_json else ())
+ (('--lock-wait', str(lock_wait)) if lock_wait else ())
+ (('--stats',) if stats and not dry_run else ())
+ (('--stats',) if prune_arguments.stats and not dry_run else ())
+ (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+ (('--list',) if list_archives else ())
+ flags.make_flags_from_arguments(
prune_arguments,
excludes=('repository', 'stats', 'list_archives'),
)
+ (('--list',) if prune_arguments.list_archives else ())
+ (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+ (('--dry-run',) if dry_run else ())
+ (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+ flags.make_repository_flags(repository, local_borg_version)
+ flags.make_repository_flags(repository_path, local_borg_version)
)
if stats or list_archives:
if prune_arguments.stats or prune_arguments.list_archives:
output_log_level = logging.ANSWER
else:
output_log_level = logging.INFO
@ -85,6 +94,7 @@ def prune_archives(
execute_command(
full_command,
output_log_level=output_log_level,
extra_environment=environment.make_environment(config),
borg_local_path=local_path,
extra_environment=environment.make_environment(storage_config),
borg_exit_codes=config.get('borg_exit_codes'),
)

View File

@ -1,4 +1,5 @@
import argparse
import json
import logging
import subprocess
@ -8,14 +9,15 @@ from borgmatic.execute import DO_NOT_CAPTURE, execute_command
logger = logging.getLogger(__name__)
RINFO_REPOSITORY_NOT_FOUND_EXIT_CODE = 2
RINFO_REPOSITORY_NOT_FOUND_EXIT_CODES = {2, 13}
def create_repository(
dry_run,
repository,
storage_config,
repository_path,
config,
local_borg_version,
global_arguments,
encryption_mode,
source_repository=None,
copy_crypt_key=False,
@ -26,27 +28,42 @@ def create_repository(
remote_path=None,
):
'''
Given a dry-run flag, a local or remote repository path, a storage configuration dict, the local
Borg version, a Borg encryption mode, the path to another repo whose key material should be
reused, whether the repository should be append-only, and the storage quota to use, create the
Given a dry-run flag, a local or remote repository path, a configuration dict, the local Borg
version, a Borg encryption mode, the path to another repo whose key material should be reused,
whether the repository should be append-only, and the storage quota to use, create the
repository. If the repository already exists, then log and skip creation.
Raise ValueError if the requested encryption mode does not match that of the repository.
Raise json.decoder.JSONDecodeError if the "borg info" JSON outputcannot be decoded.
Raise subprocess.CalledProcessError if "borg info" returns an error exit code.
'''
try:
rinfo.display_repository_info(
repository,
storage_config,
local_borg_version,
argparse.Namespace(json=True),
local_path,
remote_path,
info_data = json.loads(
rinfo.display_repository_info(
repository_path,
config,
local_borg_version,
argparse.Namespace(json=True),
global_arguments,
local_path,
remote_path,
)
)
logger.info(f'{repository}: Repository already exists. Skipping creation.')
repository_encryption_mode = info_data.get('encryption', {}).get('mode')
if repository_encryption_mode != encryption_mode:
raise ValueError(
f'Requested encryption mode "{encryption_mode}" does not match existing repository encryption mode "{repository_encryption_mode}"'
)
logger.info(f'{repository_path}: Repository already exists. Skipping creation.')
return
except subprocess.CalledProcessError as error:
if error.returncode != RINFO_REPOSITORY_NOT_FOUND_EXIT_CODE:
if error.returncode not in RINFO_REPOSITORY_NOT_FOUND_EXIT_CODES:
raise
extra_borg_options = storage_config.get('extra_borg_options', {}).get('rcreate', '')
lock_wait = config.get('lock_wait')
extra_borg_options = config.get('extra_borg_options', {}).get('rcreate', '')
rcreate_command = (
(local_path,)
@ -63,19 +80,22 @@ def create_repository(
+ (('--make-parent-dirs',) if make_parent_dirs else ())
+ (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+ (('--debug',) if logger.isEnabledFor(logging.DEBUG) else ())
+ (('--log-json',) if global_arguments.log_json else ())
+ (('--lock-wait', str(lock_wait)) if lock_wait else ())
+ (('--remote-path', remote_path) if remote_path else ())
+ (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+ flags.make_repository_flags(repository, local_borg_version)
+ flags.make_repository_flags(repository_path, local_borg_version)
)
if dry_run:
logging.info(f'{repository}: Skipping repository creation (dry run)')
logging.info(f'{repository_path}: Skipping repository creation (dry run)')
return
# Do not capture output here, so as to support interactive prompts.
execute_command(
rcreate_command,
output_file=DO_NOT_CAPTURE,
extra_environment=environment.make_environment(config),
borg_local_path=local_path,
extra_environment=environment.make_environment(storage_config),
borg_exit_codes=config.get('borg_exit_codes'),
)

View File

@ -8,20 +8,21 @@ logger = logging.getLogger(__name__)
def display_repository_info(
repository,
storage_config,
repository_path,
config,
local_borg_version,
rinfo_arguments,
global_arguments,
local_path='borg',
remote_path=None,
):
'''
Given a local or remote repository path, a storage config dict, the local Borg version, and the
arguments to the rinfo action, display summary information for the Borg repository or return
JSON summary information.
Given a local or remote repository path, a configuration dict, the local Borg version, the
arguments to the rinfo action, and global arguments as an argparse.Namespace, display summary
information for the Borg repository or return JSON summary information.
'''
borgmatic.logger.add_custom_log_levels()
lock_wait = storage_config.get('lock_wait', None)
lock_wait = config.get('lock_wait', None)
full_command = (
(local_path,)
@ -41,21 +42,27 @@ def display_repository_info(
else ()
)
+ flags.make_flags('remote-path', remote_path)
+ flags.make_flags('log-json', global_arguments.log_json)
+ flags.make_flags('lock-wait', lock_wait)
+ (('--json',) if rinfo_arguments.json else ())
+ flags.make_repository_flags(repository, local_borg_version)
+ flags.make_repository_flags(repository_path, local_borg_version)
)
extra_environment = environment.make_environment(storage_config)
extra_environment = environment.make_environment(config)
borg_exit_codes = config.get('borg_exit_codes')
if rinfo_arguments.json:
return execute_command_and_capture_output(
full_command, extra_environment=extra_environment,
full_command,
extra_environment=extra_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
else:
execute_command(
full_command,
output_log_level=logging.ANSWER,
borg_local_path=local_path,
extra_environment=extra_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)

View File

@ -1,3 +1,4 @@
import argparse
import logging
import borgmatic.logger
@ -8,63 +9,71 @@ logger = logging.getLogger(__name__)
def resolve_archive_name(
repository, archive, storage_config, local_borg_version, local_path='borg', remote_path=None
repository_path,
archive,
config,
local_borg_version,
global_arguments,
local_path='borg',
remote_path=None,
):
'''
Given a local or remote repository path, an archive name, a storage config dict, a local Borg
path, and a remote Borg path, simply return the archive name. But if the archive name is
"latest", then instead introspect the repository for the latest archive and return its name.
Given a local or remote repository path, an archive name, a configuration dict, the local Borg
version, global arguments as an argparse.Namespace, a local Borg path, and a remote Borg path,
return the archive name. But if the archive name is "latest", then instead introspect the
repository for the latest archive and return its name.
Raise ValueError if "latest" is given but there are no archives in the repository.
'''
if archive != "latest":
if archive != 'latest':
return archive
lock_wait = storage_config.get('lock_wait', None)
full_command = (
(
local_path,
'rlist' if feature.available(feature.Feature.RLIST, local_borg_version) else 'list',
)
+ flags.make_flags('remote-path', remote_path)
+ flags.make_flags('lock-wait', lock_wait)
+ flags.make_flags('log-json', global_arguments.log_json)
+ flags.make_flags('lock-wait', config.get('lock_wait'))
+ flags.make_flags('last', 1)
+ ('--short',)
+ flags.make_repository_flags(repository, local_borg_version)
+ flags.make_repository_flags(repository_path, local_borg_version)
)
output = execute_command_and_capture_output(
full_command, extra_environment=environment.make_environment(storage_config),
full_command,
extra_environment=environment.make_environment(config),
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
try:
latest_archive = output.strip().splitlines()[-1]
except IndexError:
raise ValueError('No archives found in the repository')
logger.debug('{}: Latest archive is {}'.format(repository, latest_archive))
logger.debug(f'{repository_path}: Latest archive is {latest_archive}')
return latest_archive
MAKE_FLAGS_EXCLUDES = ('repository', 'prefix')
MAKE_FLAGS_EXCLUDES = ('repository', 'prefix', 'match_archives')
def make_rlist_command(
repository,
storage_config,
repository_path,
config,
local_borg_version,
rlist_arguments,
global_arguments,
local_path='borg',
remote_path=None,
):
'''
Given a local or remote repository path, a storage config dict, the local Borg version, the
arguments to the rlist action, and local and remote Borg paths, return a command as a tuple to
list archives with a repository.
Given a local or remote repository path, a configuration dict, the local Borg version, the
arguments to the rlist action, global arguments as an argparse.Namespace instance, and local and
remote Borg paths, return a command as a tuple to list archives with a repository.
'''
lock_wait = storage_config.get('lock_wait', None)
return (
(
local_path,
@ -81,7 +90,8 @@ def make_rlist_command(
else ()
)
+ flags.make_flags('remote-path', remote_path)
+ flags.make_flags('lock-wait', lock_wait)
+ flags.make_flags('log-json', global_arguments.log_json)
+ flags.make_flags('lock-wait', config.get('lock_wait'))
+ (
(
flags.make_flags('match-archives', f'sh:{rlist_arguments.prefix}*')
@ -89,39 +99,73 @@ def make_rlist_command(
else flags.make_flags('glob-archives', f'{rlist_arguments.prefix}*')
)
if rlist_arguments.prefix
else ()
else (
flags.make_match_archives_flags(
rlist_arguments.match_archives or config.get('match_archives'),
config.get('archive_name_format'),
local_borg_version,
)
)
)
+ flags.make_flags_from_arguments(rlist_arguments, excludes=MAKE_FLAGS_EXCLUDES)
+ flags.make_repository_flags(repository, local_borg_version)
+ flags.make_repository_flags(repository_path, local_borg_version)
)
def list_repository(
repository,
storage_config,
repository_path,
config,
local_borg_version,
rlist_arguments,
global_arguments,
local_path='borg',
remote_path=None,
):
'''
Given a local or remote repository path, a storage config dict, the local Borg version, the
arguments to the list action, and local and remote Borg paths, display the output of listing
Borg archives in the given repository (or return JSON output).
Given a local or remote repository path, a configuration dict, the local Borg version, the
arguments to the list action, global arguments as an argparse.Namespace instance, and local and
remote Borg paths, display the output of listing Borg archives in the given repository (or
return JSON output).
'''
borgmatic.logger.add_custom_log_levels()
borg_environment = environment.make_environment(storage_config)
borg_environment = environment.make_environment(config)
main_command = make_rlist_command(
repository, storage_config, local_borg_version, rlist_arguments, local_path, remote_path
repository_path,
config,
local_borg_version,
rlist_arguments,
global_arguments,
local_path,
remote_path,
)
json_command = make_rlist_command(
repository_path,
config,
local_borg_version,
argparse.Namespace(**dict(rlist_arguments.__dict__, json=True)),
global_arguments,
local_path,
remote_path,
)
borg_exit_codes = config.get('borg_exit_codes')
json_listing = execute_command_and_capture_output(
json_command,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
if rlist_arguments.json:
return execute_command_and_capture_output(main_command, extra_environment=borg_environment,)
else:
execute_command(
main_command,
output_log_level=logging.ANSWER,
borg_local_path=local_path,
extra_environment=borg_environment,
)
return json_listing
flags.warn_for_aggressive_archive_flags(json_command, json_listing)
execute_command(
main_command,
output_log_level=logging.ANSWER,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)

View File

@ -2,23 +2,25 @@ import logging
import borgmatic.logger
from borgmatic.borg import environment, flags
from borgmatic.execute import execute_command
from borgmatic.execute import DO_NOT_CAPTURE, execute_command
logger = logging.getLogger(__name__)
def transfer_archives(
dry_run,
repository,
storage_config,
repository_path,
config,
local_borg_version,
transfer_arguments,
global_arguments,
local_path='borg',
remote_path=None,
):
'''
Given a dry-run flag, a local or remote repository path, a storage config dict, the local Borg
version, and the arguments to the transfer action, transfer archives to the given repository.
Given a dry-run flag, a local or remote repository path, a configuration dict, the local Borg
version, the arguments to the transfer action, and global arguments as an argparse.Namespace
instance, transfer archives to the given repository.
'''
borgmatic.logger.add_custom_log_levels()
@ -27,17 +29,24 @@ def transfer_archives(
+ (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+ (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+ flags.make_flags('remote-path', remote_path)
+ flags.make_flags('lock-wait', storage_config.get('lock_wait', None))
+ flags.make_flags('log-json', global_arguments.log_json)
+ flags.make_flags('lock-wait', config.get('lock_wait', None))
+ (
flags.make_flags(
'match-archives', transfer_arguments.match_archives or transfer_arguments.archive
flags.make_flags_from_arguments(
transfer_arguments,
excludes=('repository', 'source_repository', 'archive', 'match_archives'),
)
or (
flags.make_match_archives_flags(
transfer_arguments.match_archives
or transfer_arguments.archive
or config.get('match_archives'),
config.get('archive_name_format'),
local_borg_version,
)
)
)
+ flags.make_flags_from_arguments(
transfer_arguments,
excludes=('repository', 'source_repository', 'archive', 'match_archives'),
)
+ flags.make_repository_flags(repository, local_borg_version)
+ flags.make_repository_flags(repository_path, local_borg_version)
+ flags.make_flags('other-repo', transfer_arguments.source_repository)
+ flags.make_flags('dry-run', dry_run)
)
@ -45,6 +54,8 @@ def transfer_archives(
return execute_command(
full_command,
output_log_level=logging.ANSWER,
output_file=DO_NOT_CAPTURE if transfer_arguments.progress else None,
borg_local_path=local_path,
extra_environment=environment.make_environment(storage_config),
borg_exit_codes=config.get('borg_exit_codes'),
extra_environment=environment.make_environment(config),
)

View File

@ -5,7 +5,7 @@ from borgmatic.execute import execute_command
logger = logging.getLogger(__name__)
def unmount_archive(mount_point, local_path='borg'):
def unmount_archive(config, mount_point, local_path='borg'):
'''
Given a mounted filesystem mount point, and an optional local Borg paths, umount the filesystem
from the mount point.
@ -17,4 +17,6 @@ def unmount_archive(mount_point, local_path='borg'):
+ (mount_point,)
)
execute_command(full_command)
execute_command(
full_command, borg_local_path=local_path, borg_exit_codes=config.get('borg_exit_codes')
)

View File

@ -6,9 +6,9 @@ from borgmatic.execute import execute_command_and_capture_output
logger = logging.getLogger(__name__)
def local_borg_version(storage_config, local_path='borg'):
def local_borg_version(config, local_path='borg'):
'''
Given a storage configuration dict and a local Borg binary path, return a version string for it.
Given a configuration dict and a local Borg binary path, return a version string for it.
Raise OSError or CalledProcessError if there is a problem running Borg.
Raise ValueError if the version cannot be parsed.
@ -19,7 +19,10 @@ def local_borg_version(storage_config, local_path='borg'):
+ (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
)
output = execute_command_and_capture_output(
full_command, extra_environment=environment.make_environment(storage_config),
full_command,
extra_environment=environment.make_environment(config),
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
try:

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,36 @@
import borgmatic.commands.arguments
def upgrade_message(language: str, upgrade_command: str, completion_file: str):
return f'''
Your {language} completions script is from a different version of borgmatic than is
currently installed. Please upgrade your script so your completions match the
command-line flags in your installed borgmatic! Try this to upgrade:
{upgrade_command}
source {completion_file}
'''
def available_actions(subparsers, current_action=None):
'''
Given subparsers as an argparse._SubParsersAction instance and a current action name (if
any), return the actions names that can follow the current action on a command-line.
This takes into account which sub-actions that the current action supports. For instance, if
"bootstrap" is a sub-action for "config", then "bootstrap" should be able to follow a current
action of "config" but not "list".
'''
action_to_subactions = borgmatic.commands.arguments.get_subactions_for_actions(
subparsers.choices
)
current_subactions = action_to_subactions.get(current_action)
if current_subactions:
return current_subactions
all_subactions = set(
subaction for subactions in action_to_subactions.values() for subaction in subactions
)
return tuple(action for action in subparsers.choices.keys() if action not in all_subactions)

View File

@ -1,13 +1,5 @@
from borgmatic.commands import arguments
UPGRADE_MESSAGE = '''
Your bash completions script is from a different version of borgmatic than is
currently installed. Please upgrade your script so your completions match the
command-line flags in your installed borgmatic! Try this to upgrade:
sudo sh -c "borgmatic --bash-completion > $BASH_SOURCE"
source $BASH_SOURCE
'''
import borgmatic.commands.arguments
import borgmatic.commands.completion.actions
def parser_flags(parser):
@ -23,9 +15,12 @@ def bash_completion():
Return a bash completion script for the borgmatic command. Produce this by introspecting
borgmatic's command-line argument parsers.
'''
top_level_parser, subparsers = arguments.make_parsers()
global_flags = parser_flags(top_level_parser)
actions = ' '.join(subparsers.choices.keys())
(
unused_global_parser,
action_parsers,
global_plus_action_parser,
) = borgmatic.commands.arguments.make_parsers()
global_flags = parser_flags(global_plus_action_parser)
# Avert your eyes.
return '\n'.join(
@ -34,7 +29,11 @@ def bash_completion():
' local this_script="$(cat "$BASH_SOURCE" 2> /dev/null)"',
' local installed_script="$(borgmatic --bash-completion 2> /dev/null)"',
' if [ "$this_script" != "$installed_script" ] && [ "$installed_script" != "" ];'
' then cat << EOF\n%s\nEOF' % UPGRADE_MESSAGE,
f''' then cat << EOF\n{borgmatic.commands.completion.actions.upgrade_message(
'bash',
'sudo sh -c "borgmatic --bash-completion > $BASH_SOURCE"',
'$BASH_SOURCE',
)}\nEOF''',
' fi',
'}',
'complete_borgmatic() {',
@ -44,12 +43,22 @@ def bash_completion():
COMPREPLY=($(compgen -W "%s %s %s" -- "${COMP_WORDS[COMP_CWORD]}"))
return 0
fi'''
% (action, parser_flags(subparser), actions, global_flags)
for action, subparser in subparsers.choices.items()
% (
action,
parser_flags(action_parser),
' '.join(
borgmatic.commands.completion.actions.available_actions(action_parsers, action)
),
global_flags,
)
for action, action_parser in reversed(action_parsers.choices.items())
)
+ (
' COMPREPLY=($(compgen -W "%s %s" -- "${COMP_WORDS[COMP_CWORD]}"))'
% (actions, global_flags),
' COMPREPLY=($(compgen -W "%s %s" -- "${COMP_WORDS[COMP_CWORD]}"))' # noqa: FS003
% (
' '.join(borgmatic.commands.completion.actions.available_actions(action_parsers)),
global_flags,
),
' (check_version &)',
'}',
'\ncomplete -o bashdefault -o default -F complete_borgmatic borgmatic',

View File

@ -0,0 +1,176 @@
import shlex
from argparse import Action
from textwrap import dedent
import borgmatic.commands.arguments
import borgmatic.commands.completion.actions
def has_file_options(action: Action):
'''
Given an argparse.Action instance, return True if it takes a file argument.
'''
return action.metavar in (
'FILENAME',
'PATH',
) or action.dest in ('config_paths',)
def has_choice_options(action: Action):
'''
Given an argparse.Action instance, return True if it takes one of a predefined set of arguments.
'''
return action.choices is not None
def has_unknown_required_param_options(action: Action):
'''
A catch-all for options that take a required parameter, but we don't know what the parameter is.
This should be used last. These are actions that take something like a glob, a list of numbers, or a string.
Actions that match this pattern should not show the normal arguments, because those are unlikely to be valid.
'''
return (
action.required is True
or action.nargs
in (
'+',
'*',
)
or action.metavar in ('PATTERN', 'KEYS', 'N')
or (action.type is not None and action.default is None)
)
def has_exact_options(action: Action):
return (
has_file_options(action)
or has_choice_options(action)
or has_unknown_required_param_options(action)
)
def exact_options_completion(action: Action):
'''
Given an argparse.Action instance, return a completion invocation that forces file completions, options completion,
or just that some value follow the action, if the action takes such an argument and was the last action on the
command line prior to the cursor.
Otherwise, return an empty string.
'''
if not has_exact_options(action):
return ''
args = ' '.join(action.option_strings)
if has_file_options(action):
return f'''\ncomplete -c borgmatic -Fr -n "__borgmatic_current_arg {args}"'''
if has_choice_options(action):
return f'''\ncomplete -c borgmatic -f -a '{' '.join(map(str, action.choices))}' -n "__borgmatic_current_arg {args}"'''
if has_unknown_required_param_options(action):
return f'''\ncomplete -c borgmatic -x -n "__borgmatic_current_arg {args}"'''
raise ValueError(
f'Unexpected action: {action} passes has_exact_options but has no choices produced'
)
def dedent_strip_as_tuple(string: str):
'''
Dedent a string, then strip it to avoid requiring your first line to have content, then return a tuple of the string.
Makes it easier to write multiline strings for completions when you join them with a tuple.
'''
return (dedent(string).strip('\n'),)
def fish_completion():
'''
Return a fish completion script for the borgmatic command. Produce this by introspecting
borgmatic's command-line argument parsers.
'''
(
unused_global_parser,
action_parsers,
global_plus_action_parser,
) = borgmatic.commands.arguments.make_parsers()
all_action_parsers = ' '.join(action for action in action_parsers.choices.keys())
exact_option_args = tuple(
' '.join(action.option_strings)
for action_parser in action_parsers.choices.values()
for action in action_parser._actions
if has_exact_options(action)
) + tuple(
' '.join(action.option_strings)
for action in global_plus_action_parser._actions
if len(action.option_strings) > 0
if has_exact_options(action)
)
# Avert your eyes.
return '\n'.join(
dedent_strip_as_tuple(
f'''
function __borgmatic_check_version
set -fx this_filename (status current-filename)
fish -c '
if test -f "$this_filename"
set this_script (cat $this_filename 2> /dev/null)
set installed_script (borgmatic --fish-completion 2> /dev/null)
if [ "$this_script" != "$installed_script" ] && [ "$installed_script" != "" ]
echo "{borgmatic.commands.completion.actions.upgrade_message(
'fish',
'borgmatic --fish-completion | sudo tee $this_filename',
'$this_filename',
)}"
end
end
' &
end
__borgmatic_check_version
function __borgmatic_current_arg --description 'Check if any of the given arguments are the last on the command line before the cursor'
set -l all_args (commandline -poc)
# premature optimization to avoid iterating all args if there aren't enough
# to have a last arg beyond borgmatic
if [ (count $all_args) -lt 2 ]
return 1
end
for arg in $argv
if [ "$arg" = "$all_args[-1]" ]
return 0
end
end
return 1
end
set --local action_parser_condition "not __fish_seen_subcommand_from {all_action_parsers}"
set --local exact_option_condition "not __borgmatic_current_arg {' '.join(exact_option_args)}"
'''
)
+ ('\n# action_parser completions',)
+ tuple(
f'''complete -c borgmatic -f -n "$action_parser_condition" -n "$exact_option_condition" -a '{action_name}' -d {shlex.quote(action_parser.description)}'''
for action_name, action_parser in action_parsers.choices.items()
)
+ ('\n# global flags',)
+ tuple(
# -n is checked in order, so put faster / more likely to be true checks first
f'''complete -c borgmatic -f -n "$exact_option_condition" -a '{' '.join(action.option_strings)}' -d {shlex.quote(action.help)}{exact_options_completion(action)}'''
for action in global_plus_action_parser._actions
# ignore the noargs action, as this is an impossible completion for fish
if len(action.option_strings) > 0
if 'Deprecated' not in action.help
)
+ ('\n# action_parser flags',)
+ tuple(
f'''complete -c borgmatic -f -n "$exact_option_condition" -a '{' '.join(action.option_strings)}' -d {shlex.quote(action.help)} -n "__fish_seen_subcommand_from {action_name}"{exact_options_completion(action)}'''
for action_name, action_parser in action_parsers.choices.items()
for action in action_parser._actions
if 'Deprecated' not in (action.help or ())
)
)

View File

@ -1,110 +0,0 @@
import os
import sys
import textwrap
from argparse import ArgumentParser
from ruamel import yaml
from borgmatic.config import convert, generate, legacy, validate
DEFAULT_SOURCE_CONFIG_FILENAME = '/etc/borgmatic/config'
DEFAULT_SOURCE_EXCLUDES_FILENAME = '/etc/borgmatic/excludes'
DEFAULT_DESTINATION_CONFIG_FILENAME = '/etc/borgmatic/config.yaml'
def parse_arguments(*arguments):
'''
Given command-line arguments with which this script was invoked, parse the arguments and return
them as an ArgumentParser instance.
'''
parser = ArgumentParser(
description='''
Convert legacy INI-style borgmatic configuration and excludes files to a single YAML
configuration file. Note that this replaces any comments from the source files.
'''
)
parser.add_argument(
'-s',
'--source-config',
dest='source_config_filename',
default=DEFAULT_SOURCE_CONFIG_FILENAME,
help='Source INI-style configuration filename. Default: {}'.format(
DEFAULT_SOURCE_CONFIG_FILENAME
),
)
parser.add_argument(
'-e',
'--source-excludes',
dest='source_excludes_filename',
default=DEFAULT_SOURCE_EXCLUDES_FILENAME
if os.path.exists(DEFAULT_SOURCE_EXCLUDES_FILENAME)
else None,
help='Excludes filename',
)
parser.add_argument(
'-d',
'--destination-config',
dest='destination_config_filename',
default=DEFAULT_DESTINATION_CONFIG_FILENAME,
help='Destination YAML configuration filename. Default: {}'.format(
DEFAULT_DESTINATION_CONFIG_FILENAME
),
)
return parser.parse_args(arguments)
TEXT_WRAP_CHARACTERS = 80
def display_result(args): # pragma: no cover
result_lines = textwrap.wrap(
'Your borgmatic configuration has been upgraded. Please review the result in {}.'.format(
args.destination_config_filename
),
TEXT_WRAP_CHARACTERS,
)
delete_lines = textwrap.wrap(
'Once you are satisfied, you can safely delete {}{}.'.format(
args.source_config_filename,
' and {}'.format(args.source_excludes_filename)
if args.source_excludes_filename
else '',
),
TEXT_WRAP_CHARACTERS,
)
print('\n'.join(result_lines))
print()
print('\n'.join(delete_lines))
def main(): # pragma: no cover
try:
args = parse_arguments(*sys.argv[1:])
schema = yaml.round_trip_load(open(validate.schema_filename()).read())
source_config = legacy.parse_configuration(
args.source_config_filename, legacy.CONFIG_FORMAT
)
source_config_file_mode = os.stat(args.source_config_filename).st_mode
source_excludes = (
open(args.source_excludes_filename).read().splitlines()
if args.source_excludes_filename
else []
)
destination_config = convert.convert_legacy_parsed_config(
source_config, source_excludes, schema
)
generate.write_configuration(
args.destination_config_filename,
generate.render_configuration(destination_config),
mode=source_config_file_mode,
)
display_result(args)
except (ValueError, OSError) as error:
print(error, file=sys.stderr)
sys.exit(1)

View File

@ -1,69 +1,17 @@
import logging
import sys
from argparse import ArgumentParser
from borgmatic.config import generate, validate
DEFAULT_DESTINATION_CONFIG_FILENAME = '/etc/borgmatic/config.yaml'
import borgmatic.commands.borgmatic
def parse_arguments(*arguments):
'''
Given command-line arguments with which this script was invoked, parse the arguments and return
them as an ArgumentParser instance.
'''
parser = ArgumentParser(description='Generate a sample borgmatic YAML configuration file.')
parser.add_argument(
'-s',
'--source',
dest='source_filename',
help='Optional YAML configuration file to merge into the generated configuration, useful for upgrading your configuration',
)
parser.add_argument(
'-d',
'--destination',
dest='destination_filename',
default=DEFAULT_DESTINATION_CONFIG_FILENAME,
help='Destination YAML configuration file, default: {}'.format(
DEFAULT_DESTINATION_CONFIG_FILENAME
),
)
parser.add_argument(
'--overwrite',
default=False,
action='store_true',
help='Whether to overwrite any existing destination file, defaults to false',
)
return parser.parse_args(arguments)
def main(): # pragma: no cover
try:
args = parse_arguments(*sys.argv[1:])
generate.generate_sample_configuration(
args.source_filename,
args.destination_filename,
validate.schema_filename(),
overwrite=args.overwrite,
def main():
warning_log = logging.makeLogRecord(
dict(
levelno=logging.WARNING,
levelname='WARNING',
msg='generate-borgmatic-config is deprecated and will be removed from a future release. Please use "borgmatic config generate" instead.',
)
)
print('Generated a sample configuration file at {}.'.format(args.destination_filename))
print()
if args.source_filename:
print(
'Merged in the contents of configuration file at {}.'.format(args.source_filename)
)
print('To review the changes made, run:')
print()
print(
' diff --unified {} {}'.format(args.source_filename, args.destination_filename)
)
print()
print('This includes all available configuration options with example values. The few')
print('required options are indicated. Please edit the file to suit your needs.')
print()
print('If you ever need help: https://torsion.org/borgmatic/#issues')
except (ValueError, OSError) as error:
print(error, file=sys.stderr)
sys.exit(1)
sys.argv = ['borgmatic', 'config', 'generate'] + sys.argv[1:]
borgmatic.commands.borgmatic.main([warning_log])

View File

@ -1,56 +1,17 @@
import logging
import sys
from argparse import ArgumentParser
from borgmatic.config import collect, validate
logger = logging.getLogger(__name__)
import borgmatic.commands.borgmatic
def parse_arguments(*arguments):
'''
Given command-line arguments with which this script was invoked, parse the arguments and return
them as an ArgumentParser instance.
'''
config_paths = collect.get_default_config_paths()
parser = ArgumentParser(description='Validate borgmatic configuration file(s).')
parser.add_argument(
'-c',
'--config',
nargs='+',
dest='config_paths',
default=config_paths,
help='Configuration filenames or directories, defaults to: {}'.format(
' '.join(config_paths)
),
def main():
warning_log = logging.makeLogRecord(
dict(
levelno=logging.WARNING,
levelname='WARNING',
msg='validate-borgmatic-config is deprecated and will be removed from a future release. Please use "borgmatic config validate" instead.',
)
)
return parser.parse_args(arguments)
def main(): # pragma: no cover
args = parse_arguments(*sys.argv[1:])
logging.basicConfig(level=logging.INFO, format='%(message)s')
config_filenames = tuple(collect.collect_config_filenames(args.config_paths))
if len(config_filenames) == 0:
logger.critical('No files to validate found')
sys.exit(1)
found_issues = False
for config_filename in config_filenames:
try:
validate.parse_configuration(config_filename, validate.schema_filename())
except (ValueError, OSError, validate.Validation_error) as error:
logging.critical('{}: Error parsing configuration file'.format(config_filename))
logging.critical(error)
found_issues = True
if found_issues:
sys.exit(1)
else:
logger.info(
'All given configuration files are valid: {}'.format(', '.join(config_filenames))
)
sys.argv = ['borgmatic', 'config', 'validate'] + sys.argv[1:]
borgmatic.commands.borgmatic.main([warning_log])

View File

@ -1,9 +1,9 @@
def repository_enabled_for_checks(repository, consistency):
def repository_enabled_for_checks(repository, config):
'''
Given a repository name and a consistency configuration dict, return whether the repository
is enabled to have consistency checks run.
Given a repository name and a configuration dict, return whether the
repository is enabled to have consistency checks run.
'''
if not consistency.get('check_repositories'):
if not config.get('check_repositories'):
return True
return repository in consistency['check_repositories']
return repository in config['check_repositories']

View File

@ -16,17 +16,17 @@ def get_default_config_paths(expand_home=True):
return [
'/etc/borgmatic/config.yaml',
'/etc/borgmatic.d',
'%s/borgmatic/config.yaml' % user_config_directory,
'%s/borgmatic.d' % user_config_directory,
os.path.join(user_config_directory, 'borgmatic/config.yaml'),
os.path.join(user_config_directory, 'borgmatic.d'),
]
def collect_config_filenames(config_paths):
'''
Given a sequence of config paths, both filenames and directories, resolve that to an iterable
of files. Accomplish this by listing any given directories looking for contained config files
(ending with the ".yaml" or ".yml" extension). This is non-recursive, so any directories within the given
directories are ignored.
of absolute files. Accomplish this by listing any given directories looking for contained config
files (ending with the ".yaml" or ".yml" extension). This is non-recursive, so any directories
within the given directories are ignored.
Return paths even if they don't exist on disk, so the user can find out about missing
configuration paths. However, skip a default config path if it's missing, so the user doesn't
@ -41,7 +41,7 @@ def collect_config_filenames(config_paths):
continue
if not os.path.isdir(path) or not exists:
yield path
yield os.path.abspath(path)
continue
if not os.access(path, os.R_OK):
@ -51,4 +51,4 @@ def collect_config_filenames(config_paths):
full_filename = os.path.join(path, filename)
matching_filetype = full_filename.endswith('.yaml') or full_filename.endswith('.yml')
if matching_filetype and not os.path.isdir(full_filename):
yield full_filename
yield os.path.abspath(full_filename)

View File

@ -0,0 +1,61 @@
import shlex
def coerce_scalar(value):
'''
Given a configuration value, coerce it to an integer or a boolean as appropriate and return the
result.
'''
try:
return int(value)
except (TypeError, ValueError):
pass
if value == 'true' or value == 'True':
return True
if value == 'false' or value == 'False':
return False
return value
def apply_constants(value, constants, shell_escape=False):
'''
Given a configuration value (bool, dict, int, list, or string) and a dict of named constants,
replace any configuration string values of the form "{constant}" (or containing it) with the
value of the correspondingly named key from the constants. Recurse as necessary into nested
configuration to find values to replace.
For instance, if a configuration value contains "{foo}", replace it with the value of the "foo"
key found within the configuration's "constants".
If shell escape is True, then escape the constant's value before applying it.
Return the configuration value and modify the original.
'''
if not value or not constants:
return value
if isinstance(value, str):
for constant_name, constant_value in constants.items():
value = value.replace(
'{' + constant_name + '}',
shlex.quote(str(constant_value)) if shell_escape else str(constant_value),
)
# Support constants within non-string scalars by coercing the value to its appropriate type.
value = coerce_scalar(value)
elif isinstance(value, list):
for index, list_value in enumerate(value):
value[index] = apply_constants(list_value, constants, shell_escape)
elif isinstance(value, dict):
for option_name, option_value in value.items():
shell_escape = (
shell_escape
or option_name.startswith('before_')
or option_name.startswith('after_')
or option_name == 'on_error'
)
value[option_name] = apply_constants(option_value, constants, shell_escape)
return value

View File

@ -1,95 +0,0 @@
import os
from ruamel import yaml
from borgmatic.config import generate
def _convert_section(source_section_config, section_schema):
'''
Given a legacy Parsed_config instance for a single section, convert it to its corresponding
yaml.comments.CommentedMap representation in preparation for actual serialization to YAML.
Where integer types exist in the given section schema, convert their values to integers.
'''
destination_section_config = yaml.comments.CommentedMap(
[
(
option_name,
int(option_value)
if section_schema['properties'].get(option_name, {}).get('type') == 'integer'
else option_value,
)
for option_name, option_value in source_section_config.items()
]
)
return destination_section_config
def convert_legacy_parsed_config(source_config, source_excludes, schema):
'''
Given a legacy Parsed_config instance loaded from an INI-style config file and a list of exclude
patterns, convert them to a corresponding yaml.comments.CommentedMap representation in
preparation for serialization to a single YAML config file.
Additionally, use the given schema as a source of helpful comments to include within the
returned CommentedMap.
'''
destination_config = yaml.comments.CommentedMap(
[
(section_name, _convert_section(section_config, schema['properties'][section_name]))
for section_name, section_config in source_config._asdict().items()
]
)
# Split space-seperated values into actual lists, make "repository" into a list, and merge in
# excludes.
location = destination_config['location']
location['source_directories'] = source_config.location['source_directories'].split(' ')
location['repositories'] = [location.pop('repository')]
location['exclude_patterns'] = source_excludes
if source_config.consistency.get('checks'):
destination_config['consistency']['checks'] = source_config.consistency['checks'].split(' ')
# Add comments to each section, and then add comments to the fields in each section.
generate.add_comments_to_configuration_object(destination_config, schema)
for section_name, section_config in destination_config.items():
generate.add_comments_to_configuration_object(
section_config, schema['properties'][section_name], indent=generate.INDENT
)
return destination_config
class Legacy_configuration_not_upgraded(FileNotFoundError):
def __init__(self):
super(Legacy_configuration_not_upgraded, self).__init__(
'''borgmatic changed its configuration file format in version 1.1.0 from INI-style
to YAML. This better supports validation, and has a more natural way to express
lists of values. To upgrade your existing configuration, run:
sudo upgrade-borgmatic-config
That will generate a new YAML configuration file at /etc/borgmatic/config.yaml
(by default) using the values from both your existing configuration and excludes
files. The new version of borgmatic will consume the YAML configuration file
instead of the old one.'''
)
def guard_configuration_upgraded(source_config_filename, destination_config_filenames):
'''
If legacy source configuration exists but no destination upgraded configs do, raise
Legacy_configuration_not_upgraded.
The idea is that we want to alert the user about upgrading their config if they haven't already.
'''
destination_config_exists = any(
os.path.exists(filename) for filename in destination_config_filenames
)
if os.path.exists(source_config_filename) and not destination_config_exists:
raise Legacy_configuration_not_upgraded()

View File

@ -1,42 +1,51 @@
import os
import re
_VARIABLE_PATTERN = re.compile(
VARIABLE_PATTERN = re.compile(
r'(?P<escape>\\)?(?P<variable>\$\{(?P<name>[A-Za-z0-9_]+)((:?-)(?P<default>[^}]+))?\})'
)
def _resolve_string(matcher):
def resolve_string(matcher):
'''
Get the value from environment given a matcher containing a name and an optional default value.
If the variable is not defined in environment and no default value is provided, an Error is raised.
Given a matcher containing a name and an optional default value, get the value from environment.
Raise ValueError if the variable is not defined in environment and no default value is provided.
'''
if matcher.group('escape') is not None:
# in case of escaped envvar, unescape it
# In the case of an escaped environment variable, unescape it.
return matcher.group('variable')
# resolve the env var
# Resolve the environment variable.
name, default = matcher.group('name'), matcher.group('default')
out = os.getenv(name, default=default)
if out is None:
raise ValueError('Cannot find variable ${name} in environment'.format(name=name))
raise ValueError(f'Cannot find variable {name} in environment')
return out
def resolve_env_variables(item):
'''
Resolves variables like or ${FOO} from given configuration with values from process environment
Supported formats:
- ${FOO} will return FOO env variable
- ${FOO-bar} or ${FOO:-bar} will return FOO env variable if it exists, else "bar"
Resolves variables like or ${FOO} from given configuration with values from process environment.
If any variable is missing in environment and no default value is provided, an Error is raised.
Supported formats:
* ${FOO} will return FOO env variable
* ${FOO-bar} or ${FOO:-bar} will return FOO env variable if it exists, else "bar"
Raise if any variable is missing in environment and no default value is provided.
'''
if isinstance(item, str):
return _VARIABLE_PATTERN.sub(_resolve_string, item)
return VARIABLE_PATTERN.sub(resolve_string, item)
if isinstance(item, list):
for i, subitem in enumerate(item):
item[i] = resolve_env_variables(subitem)
for index, subitem in enumerate(item):
item[index] = resolve_env_variables(subitem)
if isinstance(item, dict):
for key, value in item.items():
item[key] = resolve_env_variables(value)
return item

View File

@ -3,7 +3,7 @@ import io
import os
import re
from ruamel import yaml
import ruamel.yaml
from borgmatic.config import load, normalize
@ -11,20 +11,20 @@ INDENT = 4
SEQUENCE_INDENT = 2
def _insert_newline_before_comment(config, field_name):
def insert_newline_before_comment(config, field_name):
'''
Using some ruamel.yaml black magic, insert a blank line in the config right before the given
field and its comments.
'''
config.ca.items[field_name][1].insert(
0, yaml.tokens.CommentToken('\n', yaml.error.CommentMark(0), None)
0, ruamel.yaml.tokens.CommentToken('\n', ruamel.yaml.error.CommentMark(0), None)
)
def _schema_to_sample_configuration(schema, level=0, parent_is_sequence=False):
def schema_to_sample_configuration(schema, level=0, parent_is_sequence=False):
'''
Given a loaded configuration schema, generate and return sample config for it. Include comments
for each section based on the schema "description".
for each option based on the schema "description".
'''
schema_type = schema.get('type')
example = schema.get('example')
@ -32,14 +32,14 @@ def _schema_to_sample_configuration(schema, level=0, parent_is_sequence=False):
return example
if schema_type == 'array':
config = yaml.comments.CommentedSeq(
[_schema_to_sample_configuration(schema['items'], level, parent_is_sequence=True)]
config = ruamel.yaml.comments.CommentedSeq(
[schema_to_sample_configuration(schema['items'], level, parent_is_sequence=True)]
)
add_comments_to_configuration_sequence(config, schema, indent=(level * INDENT))
elif schema_type == 'object':
config = yaml.comments.CommentedMap(
config = ruamel.yaml.comments.CommentedMap(
[
(field_name, _schema_to_sample_configuration(sub_schema, level + 1))
(field_name, schema_to_sample_configuration(sub_schema, level + 1))
for field_name, sub_schema in schema['properties'].items()
]
)
@ -48,18 +48,18 @@ def _schema_to_sample_configuration(schema, level=0, parent_is_sequence=False):
config, schema, indent=indent, skip_first=parent_is_sequence
)
else:
raise ValueError('Schema at level {} is unsupported: {}'.format(level, schema))
raise ValueError(f'Schema at level {level} is unsupported: {schema}')
return config
def _comment_out_line(line):
def comment_out_line(line):
# If it's already is commented out (or empty), there's nothing further to do!
stripped_line = line.lstrip()
if not stripped_line or stripped_line.startswith('#'):
return line
# Comment out the names of optional sections, inserting the '#' after any indent for aesthetics.
# Comment out the names of optional options, inserting the '#' after any indent for aesthetics.
matches = re.match(r'(\s*)', line)
indent_spaces = matches.group(0) if matches else ''
count_indent_spaces = len(indent_spaces)
@ -67,7 +67,7 @@ def _comment_out_line(line):
return '# '.join((indent_spaces, line[count_indent_spaces:]))
def _comment_out_optional_configuration(rendered_config):
def comment_out_optional_configuration(rendered_config):
'''
Post-process a rendered configuration string to comment out optional key/values, as determined
by a sentinel in the comment before each key.
@ -84,7 +84,7 @@ def _comment_out_optional_configuration(rendered_config):
for line in rendered_config.split('\n'):
# Upon encountering an optional configuration option, comment out lines until the next blank
# line.
if line.strip().startswith('# {}'.format(COMMENTED_OUT_SENTINEL)):
if line.strip().startswith(f'# {COMMENTED_OUT_SENTINEL}'):
optional = True
continue
@ -92,7 +92,7 @@ def _comment_out_optional_configuration(rendered_config):
if not line.strip():
optional = False
lines.append(_comment_out_line(line) if optional else line)
lines.append(comment_out_line(line) if optional else line)
return '\n'.join(lines)
@ -101,7 +101,7 @@ def render_configuration(config):
'''
Given a config data structure of nested OrderedDicts, render the config as YAML and return it.
'''
dumper = yaml.YAML()
dumper = ruamel.yaml.YAML(typ='rt')
dumper.indent(mapping=INDENT, sequence=INDENT + SEQUENCE_INDENT, offset=INDENT)
rendered = io.StringIO()
dumper.dump(config, rendered)
@ -117,9 +117,7 @@ def write_configuration(config_filename, rendered_config, mode=0o600, overwrite=
'''
if not overwrite and os.path.exists(config_filename):
raise FileExistsError(
'{} already exists. Aborting. Use --overwrite to replace the file.'.format(
config_filename
)
f'{config_filename} already exists. Aborting. Use --overwrite to replace the file.'
)
try:
@ -167,7 +165,6 @@ def add_comments_to_configuration_sequence(config, schema, indent=0):
return
REQUIRED_SECTION_NAMES = {'location', 'retention'}
REQUIRED_KEYS = {'source_directories', 'repositories', 'keep_daily'}
COMMENTED_OUT_SENTINEL = 'COMMENT_OUT'
@ -187,7 +184,7 @@ def add_comments_to_configuration_object(config, schema, indent=0, skip_first=Fa
# If this is an optional key, add an indicator to the comment flagging it to be commented
# out from the sample configuration. This sentinel is consumed by downstream processing that
# does the actual commenting out.
if field_name not in REQUIRED_SECTION_NAMES and field_name not in REQUIRED_KEYS:
if field_name not in REQUIRED_KEYS:
description = (
'\n'.join((description, COMMENTED_OUT_SENTINEL))
if description
@ -201,7 +198,7 @@ def add_comments_to_configuration_object(config, schema, indent=0, skip_first=Fa
config.yaml_set_comment_before_after_key(key=field_name, before=description, indent=indent)
if index > 0:
_insert_newline_before_comment(config, field_name)
insert_newline_before_comment(config, field_name)
RUAMEL_YAML_COMMENTS_INDEX = 1
@ -218,7 +215,7 @@ def remove_commented_out_sentinel(config, field_name):
except KeyError:
return
if last_comment_value == '# {}\n'.format(COMMENTED_OUT_SENTINEL):
if last_comment_value == f'# {COMMENTED_OUT_SENTINEL}\n':
config.ca.items[field_name][RUAMEL_YAML_COMMENTS_INDEX].pop()
@ -228,8 +225,7 @@ def merge_source_configuration_into_destination(destination_config, source_confi
favoring values from the source when there are collisions.
The purpose of this is to upgrade configuration files from old versions of borgmatic by adding
new
configuration keys and comments.
new configuration keys and comments.
'''
if not source_config:
return destination_config
@ -239,7 +235,9 @@ def merge_source_configuration_into_destination(destination_config, source_confi
for field_name, source_value in source_config.items():
# Since this key/value is from the source configuration, leave it uncommented and remove any
# sentinel that would cause it to get commented out.
remove_commented_out_sentinel(destination_config, field_name)
remove_commented_out_sentinel(
ruamel.yaml.comments.CommentedMap(destination_config), field_name
)
# This is a mapping. Recurse for this key/value.
if isinstance(source_value, collections.abc.Mapping):
@ -251,7 +249,7 @@ def merge_source_configuration_into_destination(destination_config, source_confi
# This is a sequence. Recurse for each item in it.
if isinstance(source_value, collections.abc.Sequence) and not isinstance(source_value, str):
destination_value = destination_config[field_name]
destination_config[field_name] = yaml.comments.CommentedSeq(
destination_config[field_name] = ruamel.yaml.comments.CommentedSeq(
[
merge_source_configuration_into_destination(
destination_value[index] if index < len(destination_value) else None,
@ -262,14 +260,14 @@ def merge_source_configuration_into_destination(destination_config, source_confi
)
continue
# This is some sort of scalar. Simply set it into the destination.
# This is some sort of scalar. Set it into the destination.
destination_config[field_name] = source_config[field_name]
return destination_config
def generate_sample_configuration(
source_filename, destination_filename, schema_filename, overwrite=False
dry_run, source_filename, destination_filename, schema_filename, overwrite=False
):
'''
Given an optional source configuration filename, and a required destination configuration
@ -278,7 +276,7 @@ def generate_sample_configuration(
schema. If a source filename is provided, merge the parsed contents of that configuration into
the generated configuration.
'''
schema = yaml.round_trip_load(open(schema_filename))
schema = ruamel.yaml.YAML(typ='safe').load(open(schema_filename))
source_config = None
if source_filename:
@ -286,11 +284,14 @@ def generate_sample_configuration(
normalize.normalize(source_filename, source_config)
destination_config = merge_source_configuration_into_destination(
_schema_to_sample_configuration(schema), source_config
schema_to_sample_configuration(schema), source_config
)
if dry_run:
return
write_configuration(
destination_filename,
_comment_out_optional_configuration(render_configuration(destination_config)),
comment_out_optional_configuration(render_configuration(destination_config)),
overwrite=overwrite,
)

View File

@ -1,152 +0,0 @@
from collections import OrderedDict, namedtuple
from configparser import RawConfigParser
Section_format = namedtuple('Section_format', ('name', 'options'))
Config_option = namedtuple('Config_option', ('name', 'value_type', 'required'))
def option(name, value_type=str, required=True):
'''
Given a config file option name, an expected type for its value, and whether it's required,
return a Config_option capturing that information.
'''
return Config_option(name, value_type, required)
CONFIG_FORMAT = (
Section_format(
'location',
(
option('source_directories'),
option('one_file_system', value_type=bool, required=False),
option('remote_path', required=False),
option('repository'),
),
),
Section_format(
'storage',
(
option('encryption_passphrase', required=False),
option('compression', required=False),
option('umask', required=False),
),
),
Section_format(
'retention',
(
option('keep_within', required=False),
option('keep_hourly', int, required=False),
option('keep_daily', int, required=False),
option('keep_weekly', int, required=False),
option('keep_monthly', int, required=False),
option('keep_yearly', int, required=False),
option('prefix', required=False),
),
),
Section_format(
'consistency', (option('checks', required=False), option('check_last', required=False))
),
)
def validate_configuration_format(parser, config_format):
'''
Given an open RawConfigParser and an expected config file format, validate that the parsed
configuration file has the expected sections, that any required options are present in those
sections, and that there aren't any unexpected options.
A section is required if any of its contained options are required.
Raise ValueError if anything is awry.
'''
section_names = set(parser.sections())
required_section_names = tuple(
section.name
for section in config_format
if any(option.required for option in section.options)
)
unknown_section_names = section_names - set(
section_format.name for section_format in config_format
)
if unknown_section_names:
raise ValueError(
'Unknown config sections found: {}'.format(', '.join(unknown_section_names))
)
missing_section_names = set(required_section_names) - section_names
if missing_section_names:
raise ValueError('Missing config sections: {}'.format(', '.join(missing_section_names)))
for section_format in config_format:
if section_format.name not in section_names:
continue
option_names = parser.options(section_format.name)
expected_options = section_format.options
unexpected_option_names = set(option_names) - set(
option.name for option in expected_options
)
if unexpected_option_names:
raise ValueError(
'Unexpected options found in config section {}: {}'.format(
section_format.name, ', '.join(sorted(unexpected_option_names))
)
)
missing_option_names = tuple(
option.name
for option in expected_options
if option.required
if option.name not in option_names
)
if missing_option_names:
raise ValueError(
'Required options missing from config section {}: {}'.format(
section_format.name, ', '.join(missing_option_names)
)
)
def parse_section_options(parser, section_format):
'''
Given an open RawConfigParser and an expected section format, return the option values from that
section as a dict mapping from option name to value. Omit those options that are not present in
the parsed options.
Raise ValueError if any option values cannot be coerced to the expected Python data type.
'''
type_getter = {str: parser.get, int: parser.getint, bool: parser.getboolean}
return OrderedDict(
(option.name, type_getter[option.value_type](section_format.name, option.name))
for option in section_format.options
if parser.has_option(section_format.name, option.name)
)
def parse_configuration(config_filename, config_format):
'''
Given a config filename and an expected config file format, return the parsed configuration
as a namedtuple with one attribute for each parsed section.
Raise IOError if the file cannot be read, or ValueError if the format is not as expected.
'''
parser = RawConfigParser()
if not parser.read(config_filename):
raise ValueError('Configuration file cannot be opened: {}'.format(config_filename))
validate_configuration_format(parser, config_format)
# Describes a parsed configuration, where each attribute is the name of a configuration file
# section and each value is a dict of that section's parsed options.
Parsed_config = namedtuple(
'Parsed_config', (section_format.name for section_format in config_format)
)
return Parsed_config(
*(parse_section_options(parser, section_format) for section_format in config_format)
)

View File

@ -1,5 +1,7 @@
import functools
import itertools
import logging
import operator
import os
import ruamel.yaml
@ -7,137 +9,277 @@ import ruamel.yaml
logger = logging.getLogger(__name__)
def include_configuration(loader, filename_node, include_directory):
def probe_and_include_file(filename, include_directories, config_paths):
'''
Given a ruamel.yaml.loader.Loader, a ruamel.yaml.serializer.ScalarNode containing the included
filename, and an include directory path to search for matching files, load the given YAML
filename (ignoring the given loader so we can use our own) and return its contents as a data
structure of nested dicts and lists. If the filename is relative, probe for it within 1. the
current working directory and 2. the given include directory.
Given a filename to include, a list of include directories to search for matching files, and a
set of configuration paths, probe for the file, load it, and return the loaded configuration as
a data structure of nested dicts, lists, etc. Add the filename to the given configuration paths.
Raise FileNotFoundError if the included file was not found.
'''
expanded_filename = os.path.expanduser(filename)
if os.path.isabs(expanded_filename):
return load_configuration(expanded_filename, config_paths)
candidate_filenames = {
os.path.join(directory, expanded_filename) for directory in include_directories
}
for candidate_filename in candidate_filenames:
if os.path.exists(candidate_filename):
return load_configuration(candidate_filename, config_paths)
raise FileNotFoundError(
f'Could not find include {filename} at {" or ".join(candidate_filenames)}'
)
def include_configuration(loader, filename_node, include_directory, config_paths):
'''
Given a ruamel.yaml.loader.Loader, a ruamel.yaml.nodes.ScalarNode containing the included
filename (or a list containing multiple such filenames), an include directory path to search for
matching files, and a set of configuration paths, load the given YAML filenames (ignoring the
given loader so we can use our own) and return their contents as data structure of nested dicts,
lists, etc. Add the names of included files to the given configuration paths. If the given
filename node's value is a scalar string, then the return value will be a single value. But if
the given node value is a list, then the return value will be a list of values, one per loaded
configuration file.
If a filename is relative, probe for it within: 1. the current working directory and 2. the
given include directory.
Raise FileNotFoundError if an included file was not found.
'''
include_directories = [os.getcwd(), os.path.abspath(include_directory)]
include_filename = os.path.expanduser(filename_node.value)
if not os.path.isabs(include_filename):
candidate_filenames = [
os.path.join(directory, include_filename) for directory in include_directories
if isinstance(filename_node.value, str):
return probe_and_include_file(filename_node.value, include_directories, config_paths)
if (
isinstance(filename_node.value, list)
and len(filename_node.value)
and isinstance(filename_node.value[0], ruamel.yaml.nodes.ScalarNode)
):
# Reversing the values ensures the correct ordering if these includes are subsequently
# merged together.
return [
probe_and_include_file(node.value, include_directories, config_paths)
for node in reversed(filename_node.value)
]
for candidate_filename in candidate_filenames:
if os.path.exists(candidate_filename):
include_filename = candidate_filename
break
else:
raise FileNotFoundError(
f'Could not find include {filename_node.value} at {" or ".join(candidate_filenames)}'
)
raise ValueError(
'!include value is not supported; use a single filename or a list of filenames'
)
return load_configuration(include_filename)
def raise_retain_node_error(loader, node):
'''
Given a ruamel.yaml.loader.Loader and a YAML node, raise an error about "!retain" usage.
Raise ValueError if a mapping or sequence node is given, as that indicates that "!retain" was
used in a configuration file without a merge. In configuration files with a merge, mapping and
sequence nodes with "!retain" tags are handled by deep_merge_nodes() below.
Also raise ValueError if a scalar node is given, as "!retain" is not supported on scalar nodes.
'''
if isinstance(node, (ruamel.yaml.nodes.MappingNode, ruamel.yaml.nodes.SequenceNode)):
raise ValueError(
'The !retain tag may only be used within a configuration file containing a merged !include tag.'
)
raise ValueError('The !retain tag may only be used on a mapping or list.')
def raise_omit_node_error(loader, node):
'''
Given a ruamel.yaml.loader.Loader and a YAML node, raise an error about "!omit" usage.
Raise ValueError unconditionally, as an "!omit" node here indicates it was used in a
configuration file without a merge. In configuration files with a merge, nodes with "!omit"
tags are handled by deep_merge_nodes() below.
'''
raise ValueError(
'The !omit tag may only be used on a scalar (e.g., string) or list element within a configuration file containing a merged !include tag.'
)
class Include_constructor(ruamel.yaml.SafeConstructor):
'''
A YAML "constructor" (a ruamel.yaml concept) that supports a custom "!include" tag for including
separate YAML configuration files. Example syntax: `retention: !include common.yaml`
separate YAML configuration files. Example syntax: `option: !include common.yaml`
'''
def __init__(self, preserve_quotes=None, loader=None, include_directory=None):
def __init__(
self, preserve_quotes=None, loader=None, include_directory=None, config_paths=None
):
super(Include_constructor, self).__init__(preserve_quotes, loader)
self.add_constructor(
'!include',
functools.partial(include_configuration, include_directory=include_directory),
functools.partial(
include_configuration,
include_directory=include_directory,
config_paths=config_paths,
),
)
# These are catch-all error handlers for tags that don't get applied and removed by
# deep_merge_nodes() below.
self.add_constructor('!retain', raise_retain_node_error)
self.add_constructor('!omit', raise_omit_node_error)
def flatten_mapping(self, node):
'''
Support the special case of deep merging included configuration into an existing mapping
using the YAML '<<' merge key. Example syntax:
```
retention:
keep_daily: 1
option:
sub_option: 1
<<: !include common.yaml
```
These includes are deep merged into the current configuration file. For instance, in this
example, any "retention" options in common.yaml will get merged into the "retention" section
in the example configuration file.
example, any "option" with sub-options in common.yaml will get merged into the corresponding
"option" with sub-options in the example configuration file.
'''
representer = ruamel.yaml.representer.SafeRepresenter()
for index, (key_node, value_node) in enumerate(node.value):
if key_node.tag == u'tag:yaml.org,2002:merge' and value_node.tag == '!include':
included_value = representer.represent_data(self.construct_object(value_node))
node.value[index] = (key_node, included_value)
# Replace the merge include with a sequence of included configuration nodes ready
# for merging. The construct_object() call here triggers include_configuration()
# among other constructors.
node.value[index] = (
key_node,
representer.represent_data(self.construct_object(value_node)),
)
# This super().flatten_mapping() call actually performs "<<" merges.
super(Include_constructor, self).flatten_mapping(node)
node.value = deep_merge_nodes(node.value)
def load_configuration(filename):
def load_configuration(filename, config_paths=None):
'''
Load the given configuration file and return its contents as a data structure of nested dicts
and lists.
and lists. Add the filename to the given configuration paths set, and also add any included
configuration filenames.
Raise ruamel.yaml.error.YAMLError if something goes wrong parsing the YAML, or RecursionError
if there are too many recursive includes.
'''
# Use an embedded derived class for the include constructor so as to capture the filename
# value. (functools.partial doesn't work for this use case because yaml.Constructor has to be
# an actual class.)
class Include_constructor_with_include_directory(Include_constructor):
if config_paths is None:
config_paths = set()
# Use an embedded derived class for the include constructor so as to capture the include
# directory and configuration paths values. (functools.partial doesn't work for this use case
# because yaml.Constructor has to be an actual class.)
class Include_constructor_with_extras(Include_constructor):
def __init__(self, preserve_quotes=None, loader=None):
super(Include_constructor_with_include_directory, self).__init__(
preserve_quotes, loader, include_directory=os.path.dirname(filename)
super(Include_constructor_with_extras, self).__init__(
preserve_quotes,
loader,
include_directory=os.path.dirname(filename),
config_paths=config_paths,
)
yaml = ruamel.yaml.YAML(typ='safe')
yaml.Constructor = Include_constructor_with_include_directory
yaml.Constructor = Include_constructor_with_extras
config_paths.add(filename)
return yaml.load(open(filename))
with open(filename) as file:
return yaml.load(file.read())
DELETED_NODE = object()
def filter_omitted_nodes(nodes, values):
'''
Given a nested borgmatic configuration data structure as a list of tuples in the form of:
[
(
ruamel.yaml.nodes.ScalarNode as a key,
ruamel.yaml.nodes.MappingNode or other Node as a value,
),
...
]
... and a combined list of all values for those nodes, return a filtered list of the values,
omitting any that have an "!omit" tag (or with a value matching such nodes).
But if only a single node is given, bail and return the given values unfiltered, as "!omit" only
applies when there are merge includes (and therefore multiple nodes).
'''
if len(nodes) <= 1:
return values
omitted_values = tuple(node.value for node in values if node.tag == '!omit')
return [node for node in values if node.value not in omitted_values]
def merge_values(nodes):
'''
Given a nested borgmatic configuration data structure as a list of tuples in the form of:
[
(
ruamel.yaml.nodes.ScalarNode as a key,
ruamel.yaml.nodes.MappingNode or other Node as a value,
),
...
]
... merge its sequence or mapping node values and return the result. For sequence nodes, this
means appending together its contained lists. For mapping nodes, it means merging its contained
dicts.
'''
return functools.reduce(operator.add, (value.value for key, value in nodes))
def deep_merge_nodes(nodes):
'''
Given a nested borgmatic configuration data structure as a list of tuples in the form of:
[
(
ruamel.yaml.nodes.ScalarNode as a key,
ruamel.yaml.nodes.MappingNode or other Node as a value,
),
...
]
... deep merge any node values corresponding to duplicate keys and return the result. If
there are colliding keys with non-MappingNode values (e.g., integers or strings), the last
of the values wins.
... deep merge any node values corresponding to duplicate keys and return the result. The
purpose of merging like this is to support, for instance, merging one borgmatic configuration
file into another for reuse, such that a configuration option with sub-options does not
completely replace the corresponding option in a merged file.
If there are colliding keys with scalar values (e.g., integers or strings), the last of the
values wins.
For instance, given node values of:
[
(
ScalarNode(tag='tag:yaml.org,2002:str', value='retention'),
ScalarNode(tag='tag:yaml.org,2002:str', value='option'),
MappingNode(tag='tag:yaml.org,2002:map', value=[
(
ScalarNode(tag='tag:yaml.org,2002:str', value='keep_hourly'),
ScalarNode(tag='tag:yaml.org,2002:int', value='24')
ScalarNode(tag='tag:yaml.org,2002:str', value='sub_option1'),
ScalarNode(tag='tag:yaml.org,2002:int', value='1')
),
(
ScalarNode(tag='tag:yaml.org,2002:str', value='keep_daily'),
ScalarNode(tag='tag:yaml.org,2002:int', value='7')
ScalarNode(tag='tag:yaml.org,2002:str', value='sub_option2'),
ScalarNode(tag='tag:yaml.org,2002:int', value='2')
),
]),
),
(
ScalarNode(tag='tag:yaml.org,2002:str', value='retention'),
ScalarNode(tag='tag:yaml.org,2002:str', value='option'),
MappingNode(tag='tag:yaml.org,2002:map', value=[
(
ScalarNode(tag='tag:yaml.org,2002:str', value='keep_daily'),
ScalarNode(tag='tag:yaml.org,2002:str', value='sub_option2'),
ScalarNode(tag='tag:yaml.org,2002:int', value='5')
),
]),
@ -148,69 +290,95 @@ def deep_merge_nodes(nodes):
[
(
ScalarNode(tag='tag:yaml.org,2002:str', value='retention'),
ScalarNode(tag='tag:yaml.org,2002:str', value='option'),
MappingNode(tag='tag:yaml.org,2002:map', value=[
(
ScalarNode(tag='tag:yaml.org,2002:str', value='keep_hourly'),
ScalarNode(tag='tag:yaml.org,2002:int', value='24')
ScalarNode(tag='tag:yaml.org,2002:str', value='sub_option1'),
ScalarNode(tag='tag:yaml.org,2002:int', value='1')
),
(
ScalarNode(tag='tag:yaml.org,2002:str', value='keep_daily'),
ScalarNode(tag='tag:yaml.org,2002:str', value='sub_option2'),
ScalarNode(tag='tag:yaml.org,2002:int', value='5')
),
]),
),
]
The purpose of deep merging like this is to support, for instance, merging one borgmatic
configuration file into another for reuse, such that a configuration section ("retention",
etc.) does not completely replace the corresponding section in a merged file.
This function supports multi-way merging, meaning that if the same option name exists three or
more times (at the same scope level), all of those instances get merged together.
If a mapping or sequence node has a YAML "!retain" tag, then that node is not merged.
Raise ValueError if a merge is implied using multiple incompatible types.
'''
# Map from original node key/value to the replacement merged node. DELETED_NODE as a replacement
# node indications deletion.
replaced_nodes = {}
merged_nodes = []
# To find nodes that require merging, compare each node with each other node.
for a_key, a_value in nodes:
for b_key, b_value in nodes:
# If we've already considered one of the nodes for merging, skip it.
if (a_key, a_value) in replaced_nodes or (b_key, b_value) in replaced_nodes:
continue
def get_node_key_name(node):
return node[0].value
# If the keys match and the values are different, we need to merge these two A and B nodes.
if a_key.tag == b_key.tag and a_key.value == b_key.value and a_value != b_value:
# Since we're merging into the B node, consider the A node a duplicate and remove it.
replaced_nodes[(a_key, a_value)] = DELETED_NODE
# Bucket the nodes by their keys. Then merge all of the values sharing the same key.
for key_name, grouped_nodes in itertools.groupby(
sorted(nodes, key=get_node_key_name), get_node_key_name
):
grouped_nodes = list(grouped_nodes)
# If we're dealing with MappingNodes, recurse and merge its values as well.
if isinstance(b_value, ruamel.yaml.nodes.MappingNode):
replaced_nodes[(b_key, b_value)] = (
b_key,
# The merged node inherits its attributes from the final node in the group.
(last_node_key, last_node_value) = grouped_nodes[-1]
value_types = set(type(value) for (_, value) in grouped_nodes)
if len(value_types) > 1:
raise ValueError(
f'Incompatible types found when trying to merge "{key_name}:" values across configuration files: {", ".join(value_type.id for value_type in value_types)}'
)
# If we're dealing with MappingNodes, recurse and merge its values as well.
if ruamel.yaml.nodes.MappingNode in value_types:
# A "!retain" tag says to skip deep merging for this node. Replace the tag so
# downstream schema validation doesn't break on our application-specific tag.
if last_node_value.tag == '!retain' and len(grouped_nodes) > 1:
last_node_value.tag = 'tag:yaml.org,2002:map'
merged_nodes.append((last_node_key, last_node_value))
else:
merged_nodes.append(
(
last_node_key,
ruamel.yaml.nodes.MappingNode(
tag=b_value.tag,
value=deep_merge_nodes(a_value.value + b_value.value),
start_mark=b_value.start_mark,
end_mark=b_value.end_mark,
flow_style=b_value.flow_style,
comment=b_value.comment,
anchor=b_value.anchor,
),
)
# If we're dealing with SequenceNodes, merge by appending one sequence to the other.
elif isinstance(b_value, ruamel.yaml.nodes.SequenceNode):
replaced_nodes[(b_key, b_value)] = (
b_key,
ruamel.yaml.nodes.SequenceNode(
tag=b_value.tag,
value=a_value.value + b_value.value,
start_mark=b_value.start_mark,
end_mark=b_value.end_mark,
flow_style=b_value.flow_style,
comment=b_value.comment,
anchor=b_value.anchor,
tag=last_node_value.tag,
value=deep_merge_nodes(merge_values(grouped_nodes)),
start_mark=last_node_value.start_mark,
end_mark=last_node_value.end_mark,
flow_style=last_node_value.flow_style,
comment=last_node_value.comment,
anchor=last_node_value.anchor,
),
)
)
return [
replaced_nodes.get(node, node) for node in nodes if replaced_nodes.get(node) != DELETED_NODE
]
continue
# If we're dealing with SequenceNodes, merge by appending sequences together.
if ruamel.yaml.nodes.SequenceNode in value_types:
if last_node_value.tag == '!retain' and len(grouped_nodes) > 1:
last_node_value.tag = 'tag:yaml.org,2002:seq'
merged_nodes.append((last_node_key, last_node_value))
else:
merged_nodes.append(
(
last_node_key,
ruamel.yaml.nodes.SequenceNode(
tag=last_node_value.tag,
value=filter_omitted_nodes(grouped_nodes, merge_values(grouped_nodes)),
start_mark=last_node_value.start_mark,
end_mark=last_node_value.end_mark,
flow_style=last_node_value.flow_style,
comment=last_node_value.comment,
anchor=last_node_value.anchor,
),
)
)
continue
merged_nodes.append((last_node_key, last_node_value))
return merged_nodes

View File

@ -1,4 +1,61 @@
import logging
import os
def normalize_sections(config_filename, config):
'''
Given a configuration filename and a configuration dict of its loaded contents, airlift any
options out of sections ("location:", etc.) to the global scope and delete those sections.
Return any log message warnings produced based on the normalization performed.
Raise ValueError if the "prefix" option is set in both "location" and "consistency" sections.
'''
try:
location = config.get('location') or {}
except AttributeError:
raise ValueError('Configuration does not contain any options')
storage = config.get('storage') or {}
consistency = config.get('consistency') or {}
hooks = config.get('hooks') or {}
if (
location.get('prefix')
and consistency.get('prefix')
and location.get('prefix') != consistency.get('prefix')
):
raise ValueError(
'The retention prefix and the consistency prefix cannot have different values (unless one is not set).'
)
if storage.get('umask') and hooks.get('umask') and storage.get('umask') != hooks.get('umask'):
raise ValueError(
'The storage umask and the hooks umask cannot have different values (unless one is not set).'
)
any_section_upgraded = False
# Move any options from deprecated sections into the global scope.
for section_name in ('location', 'storage', 'retention', 'consistency', 'output', 'hooks'):
section_config = config.get(section_name)
if section_config is not None:
any_section_upgraded = True
del config[section_name]
config.update(section_config)
if any_section_upgraded:
return [
logging.makeLogRecord(
dict(
levelno=logging.WARNING,
levelname='WARNING',
msg=f'{config_filename}: Configuration sections (like location:, storage:, retention:, consistency:, and hooks:) are deprecated and support will be removed from a future release. To prepare for this, move your options out of sections to the global scope.',
)
)
]
return []
def normalize(config_filename, config):
@ -6,83 +63,207 @@ def normalize(config_filename, config):
Given a configuration filename and a configuration dict of its loaded contents, apply particular
hard-coded rules to normalize the configuration to adhere to the current schema. Return any log
message warnings produced based on the normalization performed.
Raise ValueError the configuration cannot be normalized.
'''
logs = []
location = config.get('location') or {}
storage = config.get('storage') or {}
consistency = config.get('consistency') or {}
hooks = config.get('hooks') or {}
logs = normalize_sections(config_filename, config)
# Upgrade exclude_if_present from a string to a list.
exclude_if_present = location.get('exclude_if_present')
exclude_if_present = config.get('exclude_if_present')
if isinstance(exclude_if_present, str):
config['location']['exclude_if_present'] = [exclude_if_present]
logs.append(
logging.makeLogRecord(
dict(
levelno=logging.WARNING,
levelname='WARNING',
msg=f'{config_filename}: The exclude_if_present option now expects a list value. String values for this option are deprecated and support will be removed from a future release.',
)
)
)
config['exclude_if_present'] = [exclude_if_present]
# Upgrade various monitoring hooks from a string to a dict.
healthchecks = hooks.get('healthchecks')
healthchecks = config.get('healthchecks')
if isinstance(healthchecks, str):
config['hooks']['healthchecks'] = {'ping_url': healthchecks}
logs.append(
logging.makeLogRecord(
dict(
levelno=logging.WARNING,
levelname='WARNING',
msg=f'{config_filename}: The healthchecks hook now expects a key/value pair with "ping_url" as a key. String values for this option are deprecated and support will be removed from a future release.',
)
)
)
config['healthchecks'] = {'ping_url': healthchecks}
cronitor = hooks.get('cronitor')
cronitor = config.get('cronitor')
if isinstance(cronitor, str):
config['hooks']['cronitor'] = {'ping_url': cronitor}
logs.append(
logging.makeLogRecord(
dict(
levelno=logging.WARNING,
levelname='WARNING',
msg=f'{config_filename}: The healthchecks hook now expects key/value pairs. String values for this option are deprecated and support will be removed from a future release.',
)
)
)
config['cronitor'] = {'ping_url': cronitor}
pagerduty = hooks.get('pagerduty')
pagerduty = config.get('pagerduty')
if isinstance(pagerduty, str):
config['hooks']['pagerduty'] = {'integration_key': pagerduty}
logs.append(
logging.makeLogRecord(
dict(
levelno=logging.WARNING,
levelname='WARNING',
msg=f'{config_filename}: The healthchecks hook now expects key/value pairs. String values for this option are deprecated and support will be removed from a future release.',
)
)
)
config['pagerduty'] = {'integration_key': pagerduty}
cronhub = hooks.get('cronhub')
cronhub = config.get('cronhub')
if isinstance(cronhub, str):
config['hooks']['cronhub'] = {'ping_url': cronhub}
logs.append(
logging.makeLogRecord(
dict(
levelno=logging.WARNING,
levelname='WARNING',
msg=f'{config_filename}: The healthchecks hook now expects key/value pairs. String values for this option are deprecated and support will be removed from a future release.',
)
)
)
config['cronhub'] = {'ping_url': cronhub}
# Upgrade consistency checks from a list of strings to a list of dicts.
checks = consistency.get('checks')
checks = config.get('checks')
if isinstance(checks, list) and len(checks) and isinstance(checks[0], str):
config['consistency']['checks'] = [{'name': check_type} for check_type in checks]
logs.append(
logging.makeLogRecord(
dict(
levelno=logging.WARNING,
levelname='WARNING',
msg=f'{config_filename}: The checks option now expects a list of key/value pairs. Lists of strings for this option are deprecated and support will be removed from a future release.',
)
)
)
config['checks'] = [{'name': check_type} for check_type in checks]
# Rename various configuration options.
numeric_owner = location.pop('numeric_owner', None)
numeric_owner = config.pop('numeric_owner', None)
if numeric_owner is not None:
config['location']['numeric_ids'] = numeric_owner
logs.append(
logging.makeLogRecord(
dict(
levelno=logging.WARNING,
levelname='WARNING',
msg=f'{config_filename}: The numeric_owner option has been renamed to numeric_ids. numeric_owner is deprecated and support will be removed from a future release.',
)
)
)
config['numeric_ids'] = numeric_owner
bsd_flags = location.pop('bsd_flags', None)
bsd_flags = config.pop('bsd_flags', None)
if bsd_flags is not None:
config['location']['flags'] = bsd_flags
logs.append(
logging.makeLogRecord(
dict(
levelno=logging.WARNING,
levelname='WARNING',
msg=f'{config_filename}: The bsd_flags option has been renamed to flags. bsd_flags is deprecated and support will be removed from a future release.',
)
)
)
config['flags'] = bsd_flags
remote_rate_limit = storage.pop('remote_rate_limit', None)
remote_rate_limit = config.pop('remote_rate_limit', None)
if remote_rate_limit is not None:
config['storage']['upload_rate_limit'] = remote_rate_limit
logs.append(
logging.makeLogRecord(
dict(
levelno=logging.WARNING,
levelname='WARNING',
msg=f'{config_filename}: The remote_rate_limit option has been renamed to upload_rate_limit. remote_rate_limit is deprecated and support will be removed from a future release.',
)
)
)
config['upload_rate_limit'] = remote_rate_limit
# Upgrade remote repositories to ssh:// syntax, required in Borg 2.
repositories = location.get('repositories')
repositories = config.get('repositories')
if repositories:
config['location']['repositories'] = []
for repository in repositories:
if '~' in repository:
if any(isinstance(repository, str) for repository in repositories):
logs.append(
logging.makeLogRecord(
dict(
levelno=logging.WARNING,
levelname='WARNING',
msg=f'{config_filename}: The repositories option now expects a list of key/value pairs. Lists of strings for this option are deprecated and support will be removed from a future release.',
)
)
)
config['repositories'] = [
{'path': repository} if isinstance(repository, str) else repository
for repository in repositories
]
repositories = config['repositories']
config['repositories'] = []
for repository_dict in repositories:
repository_path = repository_dict['path']
if '~' in repository_path:
logs.append(
logging.makeLogRecord(
dict(
levelno=logging.WARNING,
levelname='WARNING',
msg=f'{config_filename}: Repository paths containing "~" are deprecated in borgmatic and no longer work in Borg 2.x+.',
msg=f'{config_filename}: Repository paths containing "~" are deprecated in borgmatic and support will be removed from a future release.',
)
)
)
if ':' in repository and not repository.startswith('ssh://'):
rewritten_repository = (
f"ssh://{repository.replace(':~', '/~').replace(':/', '/').replace(':', '/./')}"
)
logs.append(
logging.makeLogRecord(
if ':' in repository_path:
if repository_path.startswith('file://'):
updated_repository_path = os.path.abspath(
repository_path.partition('file://')[-1]
)
config['repositories'].append(
dict(
levelno=logging.WARNING,
levelname='WARNING',
msg=f'{config_filename}: Remote repository paths without ssh:// syntax are deprecated. Interpreting "{repository}" as "{rewritten_repository}"',
repository_dict,
path=updated_repository_path,
)
)
elif repository_path.startswith('ssh://'):
config['repositories'].append(repository_dict)
else:
rewritten_repository_path = f"ssh://{repository_path.replace(':~', '/~').replace(':/', '/').replace(':', '/./')}"
logs.append(
logging.makeLogRecord(
dict(
levelno=logging.WARNING,
levelname='WARNING',
msg=f'{config_filename}: Remote repository paths without ssh:// syntax are deprecated and support will be removed from a future release. Interpreting "{repository_path}" as "{rewritten_repository_path}"',
)
)
)
config['repositories'].append(
dict(
repository_dict,
path=rewritten_repository_path,
)
)
)
config['location']['repositories'].append(rewritten_repository)
else:
config['location']['repositories'].append(repository)
config['repositories'].append(repository_dict)
if config.get('prefix'):
logs.append(
logging.makeLogRecord(
dict(
levelno=logging.WARNING,
levelname='WARNING',
msg=f'{config_filename}: The prefix option is deprecated and support will be removed from a future release. Use archive_name_format or match_archives instead.',
)
)
)
return logs

View File

@ -13,6 +13,11 @@ def set_values(config, keys, value):
first_key = keys[0]
if len(keys) == 1:
if isinstance(config, list):
raise ValueError(
'When overriding a list option, the value must use list syntax (e.g., "[foo, bar]" or "[{key: value}]" as appropriate)'
)
config[first_key] = value
return
@ -22,29 +27,70 @@ def set_values(config, keys, value):
set_values(config[first_key], keys[1:], value)
def convert_value_type(value):
def convert_value_type(value, option_type):
'''
Given a string value, determine its logical type (string, boolean, integer, etc.), and return it
converted to that type.
Given a string value and its schema type as a string, determine its logical type (string,
boolean, integer, etc.), and return it converted to that type.
If the option type is a string, leave the value as a string so that special characters in it
don't get interpreted as YAML during conversion.
Raise ruamel.yaml.error.YAMLError if there's a parse issue with the YAML.
'''
if option_type == 'string':
return value
return ruamel.yaml.YAML(typ='safe').load(io.StringIO(value))
def parse_overrides(raw_overrides):
'''
Given a sequence of configuration file override strings in the form of "section.option=value",
parse and return a sequence of tuples (keys, values), where keys is a sequence of strings. For
instance, given the following raw overrides:
LEGACY_SECTION_NAMES = {'location', 'storage', 'retention', 'consistency', 'output', 'hooks'}
['section.my_option=value1', 'section.other_option=value2']
def strip_section_names(parsed_override_key):
'''
Given a parsed override key as a tuple of option and suboption names, strip out any initial
legacy section names, since configuration file normalization also strips them out.
'''
if parsed_override_key[0] in LEGACY_SECTION_NAMES:
return parsed_override_key[1:]
return parsed_override_key
def type_for_option(schema, option_keys):
'''
Given a configuration schema and a sequence of keys identifying an option, e.g.
('extra_borg_options', 'init'), return the schema type of that option as a string.
Return None if the option or its type cannot be found in the schema.
'''
option_schema = schema
for key in option_keys:
try:
option_schema = option_schema['properties'][key]
except KeyError:
return None
try:
return option_schema['type']
except KeyError:
return None
def parse_overrides(raw_overrides, schema):
'''
Given a sequence of configuration file override strings in the form of "option.suboption=value"
and a configuration schema dict, parse and return a sequence of tuples (keys, values), where
keys is a sequence of strings. For instance, given the following raw overrides:
['my_option.suboption=value1', 'other_option=value2']
... return this:
(
(('section', 'my_option'), 'value1'),
(('section', 'other_option'), 'value2'),
(('my_option', 'suboption'), 'value1'),
(('other_option'), 'value2'),
)
Raise ValueError if an override can't be parsed.
@ -57,10 +103,18 @@ def parse_overrides(raw_overrides):
for raw_override in raw_overrides:
try:
raw_keys, value = raw_override.split('=', 1)
parsed_overrides.append((tuple(raw_keys.split('.')), convert_value_type(value),))
keys = tuple(raw_keys.split('.'))
option_type = type_for_option(schema, keys)
parsed_overrides.append(
(
keys,
convert_value_type(value, option_type),
)
)
except ValueError:
raise ValueError(
f"Invalid override '{raw_override}'. Make sure you use the form: SECTION.OPTION=VALUE"
f"Invalid override '{raw_override}'. Make sure you use the form: OPTION=VALUE or OPTION.SUBOPTION=VALUE"
)
except ruamel.yaml.error.YAMLError as error:
raise ValueError(f"Invalid override '{raw_override}': {error.problem}")
@ -68,12 +122,18 @@ def parse_overrides(raw_overrides):
return tuple(parsed_overrides)
def apply_overrides(config, raw_overrides):
def apply_overrides(config, schema, raw_overrides):
'''
Given a configuration dict and a sequence of configuration file override strings in the form of
"section.option=value", parse each override and set it the configuration dict.
'''
overrides = parse_overrides(raw_overrides)
Given a configuration dict, a corresponding configuration schema dict, and a sequence of
configuration file override strings in the form of "option.suboption=value", parse each override
and set it into the configuration dict.
for (keys, value) in overrides:
Set the overrides into the configuration both with and without deprecated section names (if
used), so that the overrides work regardless of whether the configuration is also using
deprecated section names.
'''
overrides = parse_overrides(raw_overrides, schema)
for keys, value in overrides:
set_values(config, keys, value)
set_values(config, strip_section_names(keys), value)

File diff suppressed because it is too large Load Diff

View File

@ -1,18 +1,23 @@
import os
import jsonschema
import pkg_resources
import ruamel.yaml
from borgmatic.config import environment, load, normalize, override
import borgmatic.config
from borgmatic.config import constants, environment, load, normalize, override
def schema_filename():
'''
Path to the installed YAML configuration schema file, used to validate and parse the
configuration.
Raise FileNotFoundError when the schema path does not exist.
'''
return pkg_resources.resource_filename('borgmatic', 'config/schema.yaml')
schema_path = os.path.join(os.path.dirname(borgmatic.config.__file__), 'schema.yaml')
with open(schema_path):
return schema_path
def format_json_error_path_element(path_element):
@ -20,9 +25,9 @@ def format_json_error_path_element(path_element):
Given a path element into a JSON data structure, format it for display as a string.
'''
if isinstance(path_element, int):
return str('[{}]'.format(path_element))
return str(f'[{path_element}]')
return str('.{}'.format(path_element))
return str(f'.{path_element}')
def format_json_error(error):
@ -30,10 +35,10 @@ def format_json_error(error):
Given an instance of jsonschema.exceptions.ValidationError, format it for display as a string.
'''
if not error.path:
return 'At the top level: {}'.format(error.message)
return f'At the top level: {error.message}'
formatted_path = ''.join(format_json_error_path_element(element) for element in error.path)
return "At '{}': {}".format(formatted_path.lstrip('.'), error.message)
return f"At '{formatted_path.lstrip('.')}': {error.message}"
class Validation_error(ValueError):
@ -54,9 +59,10 @@ class Validation_error(ValueError):
'''
Render a validation error as a user-facing string.
'''
return 'An error occurred while parsing a configuration file at {}:\n'.format(
self.config_filename
) + '\n'.join(error for error in self.errors)
return (
f'An error occurred while parsing a configuration file at {self.config_filename}:\n'
+ '\n'.join(error for error in self.errors)
)
def apply_logical_validation(config_filename, parsed_configuration):
@ -65,17 +71,15 @@ def apply_logical_validation(config_filename, parsed_configuration):
below), run through any additional logical validation checks. If there are any such validation
problems, raise a Validation_error.
'''
location_repositories = parsed_configuration.get('location', {}).get('repositories')
check_repositories = parsed_configuration.get('consistency', {}).get('check_repositories', [])
repositories = parsed_configuration.get('repositories')
check_repositories = parsed_configuration.get('check_repositories', [])
for repository in check_repositories:
if repository not in location_repositories:
if not any(
repositories_match(repository, config_repository) for config_repository in repositories
):
raise Validation_error(
config_filename,
(
'Unknown repository in the "consistency" section\'s "check_repositories": {}'.format(
repository
),
),
(f'Unknown repository in "check_repositories": {repository}',),
)
@ -83,29 +87,38 @@ def parse_configuration(config_filename, schema_filename, overrides=None, resolv
'''
Given the path to a config filename in YAML format, the path to a schema filename in a YAML
rendition of JSON Schema format, a sequence of configuration file override strings in the form
of "section.option=value", return the parsed configuration as a data structure of nested dicts
of "option.suboption=value", return the parsed configuration as a data structure of nested dicts
and lists corresponding to the schema. Example return value:
{'location': {'source_directories': ['/home', '/etc'], 'repository': 'hostname.borg'},
'retention': {'keep_daily': 7}, 'consistency': {'checks': ['repository', 'archives']}}
{
'source_directories': ['/home', '/etc'],
'repository': 'hostname.borg',
'keep_daily': 7,
'checks': ['repository', 'archives'],
}
Also return a sequence of logging.LogRecord instances containing any warnings about the
configuration.
Also return a set of loaded configuration paths and a sequence of logging.LogRecord instances
containing any warnings about the configuration.
Raise FileNotFoundError if the file does not exist, PermissionError if the user does not
have permissions to read the file, or Validation_error if the config does not match the schema.
'''
config_paths = set()
try:
config = load.load_configuration(config_filename)
config = load.load_configuration(config_filename, config_paths)
schema = load.load_configuration(schema_filename)
except (ruamel.yaml.error.YAMLError, RecursionError) as error:
raise Validation_error(config_filename, (str(error),))
override.apply_overrides(config, overrides)
logs = normalize.normalize(config_filename, config)
override.apply_overrides(config, schema, overrides)
constants.apply_constants(config, config.get('constants') if config else {})
if resolve_env:
environment.resolve_env_variables(config)
logs = normalize.normalize(config_filename, config)
try:
validator = jsonschema.Draft7Validator(schema)
except AttributeError: # pragma: no cover
@ -119,35 +132,45 @@ def parse_configuration(config_filename, schema_filename, overrides=None, resolv
apply_logical_validation(config_filename, config)
return config, logs
return config, config_paths, logs
def normalize_repository_path(repository):
'''
Given a repository path, return the absolute path of it (for local repositories).
'''
# A colon in the repository indicates it's a remote repository. Bail.
if ':' in repository:
# A colon in the repository could mean that it's either a file:// URL or a remote repository.
# If it's a remote repository, we don't want to normalize it. If it's a file:// URL, we do.
if ':' not in repository:
return os.path.abspath(repository)
elif repository.startswith('file://'):
return os.path.abspath(repository.partition('file://')[-1])
else:
return repository
return os.path.abspath(repository)
def repositories_match(first, second):
'''
Given two repository paths (relative and/or absolute), return whether they match.
Given two repository dicts with keys 'path' (relative and/or absolute),
and 'label', or two repository paths, return whether they match.
'''
return normalize_repository_path(first) == normalize_repository_path(second)
if isinstance(first, str):
first = {'path': first, 'label': first}
if isinstance(second, str):
second = {'path': second, 'label': second}
return (first.get('label') == second.get('label')) or (
normalize_repository_path(first.get('path'))
== normalize_repository_path(second.get('path'))
)
def guard_configuration_contains_repository(repository, configurations):
'''
Given a repository path and a dict mapping from config filename to corresponding parsed config
dict, ensure that the repository is declared exactly once in all of the configurations. If no
dict, ensure that the repository is declared at least once in all of the configurations. If no
repository is given, skip this check.
Raise ValueError if the repository is not found in a configuration, or is declared multiple
times.
Raise ValueError if the repository is not found in any configurations.
'''
if not repository:
return
@ -156,15 +179,13 @@ def guard_configuration_contains_repository(repository, configurations):
tuple(
config_repository
for config in configurations.values()
for config_repository in config['location']['repositories']
if repositories_match(repository, config_repository)
for config_repository in config['repositories']
if repositories_match(config_repository, repository)
)
)
if count == 0:
raise ValueError('Repository {} not found in configuration files'.format(repository))
if count > 1:
raise ValueError('Repository {} found in multiple configuration files'.format(repository))
raise ValueError(f'Repository "{repository}" not found in configuration files')
def guard_single_repository_selected(repository, configurations):
@ -180,11 +201,11 @@ def guard_single_repository_selected(repository, configurations):
tuple(
config_repository
for config in configurations.values()
for config_repository in config['location']['repositories']
for config_repository in config['repositories']
)
)
if count != 1:
raise ValueError(
'Can\'t determine which repository to use. Use --repository to disambiguate'
"Can't determine which repository to use. Use --repository to disambiguate"
)

View File

@ -1,4 +1,5 @@
import collections
import enum
import logging
import os
import select
@ -8,24 +9,61 @@ logger = logging.getLogger(__name__)
ERROR_OUTPUT_MAX_LINE_COUNT = 25
BORG_ERROR_EXIT_CODE = 2
BORG_ERROR_EXIT_CODE_START = 2
BORG_ERROR_EXIT_CODE_END = 99
def exit_code_indicates_error(process, exit_code, borg_local_path=None):
class Exit_status(enum.Enum):
STILL_RUNNING = 1
SUCCESS = 2
WARNING = 3
ERROR = 4
def interpret_exit_code(command, exit_code, borg_local_path=None, borg_exit_codes=None):
'''
Return True if the given exit code from running a command corresponds to an error. If a Borg
local path is given and matches the process' command, then treat exit code 1 as a warning
instead of an error.
Return an Exit_status value (e.g. SUCCESS, ERROR, or WARNING) based on interpreting the given
exit code. If a Borg local path is given and matches the process' command, then interpret the
exit code based on Borg's documented exit code semantics. And if Borg exit codes are given as a
sequence of exit code configuration dicts, then take those configured preferences into account.
'''
if exit_code is None:
return False
command = process.args.split(' ') if isinstance(process.args, str) else process.args
return Exit_status.STILL_RUNNING
if exit_code == 0:
return Exit_status.SUCCESS
if borg_local_path and command[0] == borg_local_path:
return bool(exit_code < 0 or exit_code >= BORG_ERROR_EXIT_CODE)
# First try looking for the exit code in the borg_exit_codes configuration.
for entry in borg_exit_codes or ():
if entry.get('code') == exit_code:
treat_as = entry.get('treat_as')
return bool(exit_code != 0)
if treat_as == 'error':
logger.error(
f'Treating exit code {exit_code} as an error, as per configuration'
)
return Exit_status.ERROR
elif treat_as == 'warning':
logger.warning(
f'Treating exit code {exit_code} as a warning, as per configuration'
)
return Exit_status.WARNING
# If the exit code doesn't have explicit configuration, then fall back to the default Borg
# behavior.
return (
Exit_status.ERROR
if (
exit_code < 0
or (
exit_code >= BORG_ERROR_EXIT_CODE_START
and exit_code <= BORG_ERROR_EXIT_CODE_END
)
)
else Exit_status.WARNING
)
return Exit_status.ERROR
def command_for_process(process):
@ -45,7 +83,24 @@ def output_buffer_for_process(process, exclude_stdouts):
return process.stderr if process.stdout in exclude_stdouts else process.stdout
def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path):
def append_last_lines(last_lines, captured_output, line, output_log_level):
'''
Given a rolling list of last lines, a list of captured output, a line to append, and an output
log level, append the line to the last lines and (if necessary) the captured output. Then log
the line at the requested output log level.
'''
last_lines.append(line)
if len(last_lines) > ERROR_OUTPUT_MAX_LINE_COUNT:
last_lines.pop(0)
if output_log_level is None:
captured_output.append(line)
else:
logger.log(output_log_level, line)
def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path, borg_exit_codes):
'''
Given a sequence of subprocess.Popen() instances for multiple processes, log the output for each
process with the requested log level. Additionally, raise a CalledProcessError if a process
@ -53,7 +108,8 @@ def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path):
path).
If output log level is None, then instead of logging, capture output for each process and return
it as a dict from the process to its output.
it as a dict from the process to its output. Use the given Borg local path and exit code
configuration to decide what's an error and what's a warning.
For simplicity, it's assumed that the output buffer for each process is its stdout. But if any
stdouts are given to exclude, then for any matching processes, log from their stderr instead.
@ -100,15 +156,12 @@ def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path):
# Keep the last few lines of output in case the process errors, and we need the output for
# the exception below.
last_lines = buffer_last_lines[ready_buffer]
last_lines.append(line)
if len(last_lines) > ERROR_OUTPUT_MAX_LINE_COUNT:
last_lines.pop(0)
if output_log_level is None:
captured_outputs[ready_process].append(line)
else:
logger.log(output_log_level, line)
append_last_lines(
buffer_last_lines[ready_buffer],
captured_outputs[ready_process],
line,
output_log_level,
)
if not still_running:
break
@ -120,14 +173,28 @@ def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path):
if exit_code is None:
still_running = True
command = process.args.split(' ') if isinstance(process.args, str) else process.args
continue
# If any process errors, then raise accordingly.
if exit_code_indicates_error(process, exit_code, borg_local_path):
command = process.args.split(' ') if isinstance(process.args, str) else process.args
exit_status = interpret_exit_code(command, exit_code, borg_local_path, borg_exit_codes)
if exit_status in (Exit_status.ERROR, Exit_status.WARNING):
# If an error occurs, include its output in the raised exception so that we don't
# inadvertently hide error output.
output_buffer = output_buffer_for_process(process, exclude_stdouts)
last_lines = buffer_last_lines[output_buffer] if output_buffer else []
# Collect any straggling output lines that came in since we last gathered output.
while output_buffer: # pragma: no cover
line = output_buffer.readline().rstrip().decode()
if not line:
break
append_last_lines(
last_lines, captured_outputs[process], line, output_log_level=logging.ERROR
)
if len(last_lines) == ERROR_OUTPUT_MAX_LINE_COUNT:
last_lines.insert(0, '...')
@ -138,9 +205,13 @@ def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path):
other_process.stdout.read(0)
other_process.kill()
raise subprocess.CalledProcessError(
exit_code, command_for_process(process), '\n'.join(last_lines)
)
if exit_status == Exit_status.ERROR:
raise subprocess.CalledProcessError(
exit_code, command_for_process(process), '\n'.join(last_lines)
)
still_running = False
break
if captured_outputs:
return {
@ -148,19 +219,19 @@ def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path):
}
def log_command(full_command, input_file=None, output_file=None):
def log_command(full_command, input_file=None, output_file=None, environment=None):
'''
Log the given command (a sequence of command/argument strings), along with its input/output file
paths.
paths and extra environment variables (with omitted values in case they contain passwords).
'''
logger.debug(
' '.join(full_command)
+ (' < {}'.format(getattr(input_file, 'name', '')) if input_file else '')
+ (' > {}'.format(getattr(output_file, 'name', '')) if output_file else '')
' '.join(tuple(f'{key}=***' for key in (environment or {}).keys()) + tuple(full_command))
+ (f" < {getattr(input_file, 'name', '')}" if input_file else '')
+ (f" > {getattr(output_file, 'name', '')}" if output_file else '')
)
# An sentinel passed as an output file to execute_command() to indicate that the command's output
# A sentinel passed as an output file to execute_command() to indicate that the command's output
# should be allowed to flow through to stdout without being captured for logging. Useful for
# commands with interactive prompts or those that mess directly with the console.
DO_NOT_CAPTURE = object()
@ -175,6 +246,7 @@ def execute_command(
extra_environment=None,
working_directory=None,
borg_local_path=None,
borg_exit_codes=None,
run_to_completion=True,
):
'''
@ -185,12 +257,13 @@ def execute_command(
augment the current environment, and pass the result into the command. If a working directory is
given, use that as the present working directory when running the command. If a Borg local path
is given, and the command matches it (regardless of arguments), treat exit code 1 as a warning
instead of an error. If run to completion is False, then return the process for the command
without executing it to completion.
instead of an error. But if Borg exit codes are given as a sequence of exit code configuration
dicts, then use that configuration to decide what's an error and what's a warning. If run to
completion is False, then return the process for the command without executing it to completion.
Raise subprocesses.CalledProcessError if an error occurs while running the command.
'''
log_command(full_command, input_file, output_file)
log_command(full_command, input_file, output_file, extra_environment)
environment = {**os.environ, **extra_environment} if extra_environment else None
do_not_capture = bool(output_file is DO_NOT_CAPTURE)
command = ' '.join(full_command) if shell else full_command
@ -208,33 +281,55 @@ def execute_command(
return process
log_outputs(
(process,), (input_file, output_file), output_log_level, borg_local_path=borg_local_path
(process,),
(input_file, output_file),
output_log_level,
borg_local_path,
borg_exit_codes,
)
def execute_command_and_capture_output(
full_command, capture_stderr=False, shell=False, extra_environment=None, working_directory=None,
full_command,
capture_stderr=False,
shell=False,
extra_environment=None,
working_directory=None,
borg_local_path=None,
borg_exit_codes=None,
):
'''
Execute the given command (a sequence of command/argument strings), capturing and returning its
output (stdout). If capture stderr is True, then capture and return stderr in addition to
stdout. If shell is True, execute the command within a shell. If an extra environment dict is
given, then use it to augment the current environment, and pass the result into the command. If
a working directory is given, use that as the present working directory when running the command.
a working directory is given, use that as the present working directory when running the
command. If a Borg local path is given, and the command matches it (regardless of arguments),
treat exit code 1 as a warning instead of an error. But if Borg exit codes are given as a
sequence of exit code configuration dicts, then use that configuration to decide what's an error
and what's a warning.
Raise subprocesses.CalledProcessError if an error occurs while running the command.
'''
log_command(full_command)
log_command(full_command, environment=extra_environment)
environment = {**os.environ, **extra_environment} if extra_environment else None
command = ' '.join(full_command) if shell else full_command
output = subprocess.check_output(
command,
stderr=subprocess.STDOUT if capture_stderr else None,
shell=shell,
env=environment,
cwd=working_directory,
)
try:
output = subprocess.check_output(
command,
stderr=subprocess.STDOUT if capture_stderr else None,
shell=shell,
env=environment,
cwd=working_directory,
)
except subprocess.CalledProcessError as error:
if (
interpret_exit_code(command, error.returncode, borg_local_path, borg_exit_codes)
== Exit_status.ERROR
):
raise
output = error.output
return output.decode() if output is not None else None
@ -249,6 +344,7 @@ def execute_command_with_processes(
extra_environment=None,
working_directory=None,
borg_local_path=None,
borg_exit_codes=None,
):
'''
Execute the given command (a sequence of command/argument strings) and log its output at the
@ -263,12 +359,14 @@ def execute_command_with_processes(
use it to augment the current environment, and pass the result into the command. If a working
directory is given, use that as the present working directory when running the command. If a
Borg local path is given, then for any matching command or process (regardless of arguments),
treat exit code 1 as a warning instead of an error.
treat exit code 1 as a warning instead of an error. But if Borg exit codes are given as a
sequence of exit code configuration dicts, then use that configuration to decide what's an error
and what's a warning.
Raise subprocesses.CalledProcessError if an error occurs while running the command or in the
upstream process.
'''
log_command(full_command, input_file, output_file)
log_command(full_command, input_file, output_file, extra_environment)
environment = {**os.environ, **extra_environment} if extra_environment else None
do_not_capture = bool(output_file is DO_NOT_CAPTURE)
command = ' '.join(full_command) if shell else full_command
@ -278,9 +376,9 @@ def execute_command_with_processes(
command,
stdin=input_file,
stdout=None if do_not_capture else (output_file or subprocess.PIPE),
stderr=None
if do_not_capture
else (subprocess.PIPE if output_file else subprocess.STDOUT),
stderr=(
None if do_not_capture else (subprocess.PIPE if output_file else subprocess.STDOUT)
),
shell=shell,
env=environment,
cwd=working_directory,
@ -298,7 +396,8 @@ def execute_command_with_processes(
tuple(processes) + (command_process,),
(input_file, output_file),
output_log_level,
borg_local_path=borg_local_path,
borg_local_path,
borg_exit_codes,
)
if output_log_level is None:

109
borgmatic/hooks/apprise.py Normal file
View File

@ -0,0 +1,109 @@
import logging
import operator
import borgmatic.hooks.logs
import borgmatic.hooks.monitor
logger = logging.getLogger(__name__)
DEFAULT_LOGS_SIZE_LIMIT_BYTES = 100000
HANDLER_IDENTIFIER = 'apprise'
def initialize_monitor(hook_config, config, config_filename, monitoring_log_level, dry_run):
'''
Add a handler to the root logger that stores in memory the most recent logs emitted. That way,
we can send them all to an Apprise notification service upon a finish or failure state. But skip
this if the "send_logs" option is false.
'''
if hook_config.get('send_logs') is False:
return
logs_size_limit = max(
hook_config.get('logs_size_limit', DEFAULT_LOGS_SIZE_LIMIT_BYTES)
- len(borgmatic.hooks.logs.PAYLOAD_TRUNCATION_INDICATOR),
0,
)
borgmatic.hooks.logs.add_handler(
borgmatic.hooks.logs.Forgetful_buffering_handler(
HANDLER_IDENTIFIER, logs_size_limit, monitoring_log_level
)
)
def ping_monitor(hook_config, config, config_filename, state, monitoring_log_level, dry_run):
'''
Ping the configured Apprise service URLs. Use the given configuration filename in any log
entries. If this is a dry run, then don't actually ping anything.
'''
try:
import apprise
from apprise import NotifyFormat, NotifyType
except ImportError: # pragma: no cover
logger.warning('Unable to import Apprise in monitoring hook')
return
state_to_notify_type = {
'start': NotifyType.INFO,
'finish': NotifyType.SUCCESS,
'fail': NotifyType.FAILURE,
'log': NotifyType.INFO,
}
run_states = hook_config.get('states', ['fail'])
if state.name.lower() not in run_states:
return
state_config = hook_config.get(
state.name.lower(),
{
'title': f'A borgmatic {state.name} event happened',
'body': f'A borgmatic {state.name} event happened',
},
)
if not hook_config.get('services'):
logger.info(f'{config_filename}: No Apprise services to ping')
return
dry_run_string = ' (dry run; not actually pinging)' if dry_run else ''
labels_string = ', '.join(map(operator.itemgetter('label'), hook_config.get('services')))
logger.info(f'{config_filename}: Pinging Apprise services: {labels_string}{dry_run_string}')
apprise_object = apprise.Apprise()
apprise_object.add(list(map(operator.itemgetter('url'), hook_config.get('services'))))
if dry_run:
return
body = state_config.get('body')
if state in (
borgmatic.hooks.monitor.State.FINISH,
borgmatic.hooks.monitor.State.FAIL,
borgmatic.hooks.monitor.State.LOG,
):
formatted_logs = borgmatic.hooks.logs.format_buffered_logs_for_payload(HANDLER_IDENTIFIER)
if formatted_logs:
body += f'\n\n{formatted_logs}'
result = apprise_object.notify(
title=state_config.get('title', ''),
body=body,
body_format=NotifyFormat.TEXT,
notify_type=state_to_notify_type[state.name.lower()],
)
if result is False:
logger.warning(f'{config_filename}: Error sending some Apprise notifications')
def destroy_monitor(hook_config, config, config_filename, monitoring_log_level, dry_run):
'''
Remove the monitor handler that was added to the root logger. This prevents the handler from
getting reused by other instances of this monitor.
'''
borgmatic.hooks.logs.remove_handler(HANDLER_IDENTIFIER)

View File

@ -1,6 +1,7 @@
import logging
import os
import re
import shlex
from borgmatic import execute
@ -16,7 +17,7 @@ def interpolate_context(config_filename, hook_description, command, context):
names/values, interpolate the values by "{name}" into the command and return the result.
'''
for name, value in context.items():
command = command.replace('{%s}' % name, str(value))
command = command.replace(f'{{{name}}}', shlex.quote(str(value)))
for unsupported_variable in re.findall(r'{\w+}', command):
logger.warning(
@ -38,7 +39,7 @@ def execute_hook(commands, umask, config_filename, description, dry_run, **conte
Raise subprocesses.CalledProcessError if an error occurs in a hook.
'''
if not commands:
logger.debug('{}: No commands to run for {} hook'.format(config_filename, description))
logger.debug(f'{config_filename}: No commands to run for {description} hook')
return
dry_run_label = ' (dry run; not actually running hooks)' if dry_run else ''
@ -49,19 +50,15 @@ def execute_hook(commands, umask, config_filename, description, dry_run, **conte
]
if len(commands) == 1:
logger.info(
'{}: Running command for {} hook{}'.format(config_filename, description, dry_run_label)
)
logger.info(f'{config_filename}: Running command for {description} hook{dry_run_label}')
else:
logger.info(
'{}: Running {} commands for {} hook{}'.format(
config_filename, len(commands), description, dry_run_label
)
f'{config_filename}: Running {len(commands)} commands for {description} hook{dry_run_label}',
)
if umask:
parsed_umask = int(str(umask), 8)
logger.debug('{}: Set hook umask to {}'.format(config_filename, oct(parsed_umask)))
logger.debug(f'{config_filename}: Set hook umask to {oct(parsed_umask)}')
original_umask = os.umask(parsed_umask)
else:
original_umask = None
@ -71,9 +68,9 @@ def execute_hook(commands, umask, config_filename, description, dry_run, **conte
if not dry_run:
execute.execute_command(
[command],
output_log_level=logging.ERROR
if description == 'on-error'
else logging.WARNING,
output_log_level=(
logging.ERROR if description == 'on-error' else logging.WARNING
),
shell=True,
)
finally:
@ -93,9 +90,7 @@ def considered_soft_failure(config_filename, error):
if exit_code == SOFT_FAIL_EXIT_CODE:
logger.info(
'{}: Command hook exited with soft failure exit code ({}); skipping remaining actions'.format(
config_filename, SOFT_FAIL_EXIT_CODE
)
f'{config_filename}: Command hook exited with soft failure exit code ({SOFT_FAIL_EXIT_CODE}); skipping remaining actions',
)
return True

View File

@ -14,7 +14,7 @@ MONITOR_STATE_TO_CRONHUB = {
def initialize_monitor(
ping_url, config_filename, monitoring_log_level, dry_run
ping_url, config, config_filename, monitoring_log_level, dry_run
): # pragma: no cover
'''
No initialization is necessary for this monitor.
@ -22,23 +22,27 @@ def initialize_monitor(
pass
def ping_monitor(hook_config, config_filename, state, monitoring_log_level, dry_run):
def ping_monitor(hook_config, config, config_filename, state, monitoring_log_level, dry_run):
'''
Ping the configured Cronhub URL, modified with the monitor.State. Use the given configuration
filename in any log entries. If this is a dry run, then don't actually ping anything.
'''
if state not in MONITOR_STATE_TO_CRONHUB:
logger.debug(
f'{config_filename}: Ignoring unsupported monitoring {state.name.lower()} in Cronhub hook'
)
return
dry_run_label = ' (dry run; not actually pinging)' if dry_run else ''
formatted_state = '/{}/'.format(MONITOR_STATE_TO_CRONHUB[state])
formatted_state = f'/{MONITOR_STATE_TO_CRONHUB[state]}/'
ping_url = (
hook_config['ping_url']
.replace('/start/', formatted_state)
.replace('/ping/', formatted_state)
)
logger.info(
'{}: Pinging Cronhub {}{}'.format(config_filename, state.name.lower(), dry_run_label)
)
logger.debug('{}: Using Cronhub ping URL {}'.format(config_filename, ping_url))
logger.info(f'{config_filename}: Pinging Cronhub {state.name.lower()}{dry_run_label}')
logger.debug(f'{config_filename}: Using Cronhub ping URL {ping_url}')
if not dry_run:
logging.getLogger('urllib3').setLevel(logging.ERROR)
@ -51,7 +55,7 @@ def ping_monitor(hook_config, config_filename, state, monitoring_log_level, dry_
def destroy_monitor(
ping_url_or_uuid, config_filename, monitoring_log_level, dry_run
ping_url_or_uuid, config, config_filename, monitoring_log_level, dry_run
): # pragma: no cover
'''
No destruction is necessary for this monitor.

View File

@ -14,7 +14,7 @@ MONITOR_STATE_TO_CRONITOR = {
def initialize_monitor(
ping_url, config_filename, monitoring_log_level, dry_run
ping_url, config, config_filename, monitoring_log_level, dry_run
): # pragma: no cover
'''
No initialization is necessary for this monitor.
@ -22,18 +22,22 @@ def initialize_monitor(
pass
def ping_monitor(hook_config, config_filename, state, monitoring_log_level, dry_run):
def ping_monitor(hook_config, config, config_filename, state, monitoring_log_level, dry_run):
'''
Ping the configured Cronitor URL, modified with the monitor.State. Use the given configuration
filename in any log entries. If this is a dry run, then don't actually ping anything.
'''
dry_run_label = ' (dry run; not actually pinging)' if dry_run else ''
ping_url = '{}/{}'.format(hook_config['ping_url'], MONITOR_STATE_TO_CRONITOR[state])
if state not in MONITOR_STATE_TO_CRONITOR:
logger.debug(
f'{config_filename}: Ignoring unsupported monitoring {state.name.lower()} in Cronitor hook'
)
return
logger.info(
'{}: Pinging Cronitor {}{}'.format(config_filename, state.name.lower(), dry_run_label)
)
logger.debug('{}: Using Cronitor ping URL {}'.format(config_filename, ping_url))
dry_run_label = ' (dry run; not actually pinging)' if dry_run else ''
ping_url = f"{hook_config['ping_url']}/{MONITOR_STATE_TO_CRONITOR[state]}"
logger.info(f'{config_filename}: Pinging Cronitor {state.name.lower()}{dry_run_label}')
logger.debug(f'{config_filename}: Using Cronitor ping URL {ping_url}')
if not dry_run:
logging.getLogger('urllib3').setLevel(logging.ERROR)
@ -46,7 +50,7 @@ def ping_monitor(hook_config, config_filename, state, monitoring_log_level, dry_
def destroy_monitor(
ping_url_or_uuid, config_filename, monitoring_log_level, dry_run
ping_url_or_uuid, config, config_filename, monitoring_log_level, dry_run
): # pragma: no cover
'''
No destruction is necessary for this monitor.

View File

@ -1,58 +1,65 @@
import logging
from borgmatic.hooks import (
apprise,
cronhub,
cronitor,
healthchecks,
loki,
mariadb,
mongodb,
mysql,
ntfy,
pagerduty,
postgresql,
sqlite,
)
logger = logging.getLogger(__name__)
HOOK_NAME_TO_MODULE = {
'apprise': apprise,
'cronhub': cronhub,
'cronitor': cronitor,
'healthchecks': healthchecks,
'mariadb_databases': mariadb,
'mongodb_databases': mongodb,
'mysql_databases': mysql,
'ntfy': ntfy,
'pagerduty': pagerduty,
'postgresql_databases': postgresql,
'sqlite_databases': sqlite,
'loki': loki,
}
def call_hook(function_name, hooks, log_prefix, hook_name, *args, **kwargs):
def call_hook(function_name, config, log_prefix, hook_name, *args, **kwargs):
'''
Given the hooks configuration dict and a prefix to use in log entries, call the requested
function of the Python module corresponding to the given hook name. Supply that call with the
configuration for this hook (if any), the log prefix, and any given args and kwargs. Return any
return value.
Given a configuration dict and a prefix to use in log entries, call the requested function of
the Python module corresponding to the given hook name. Supply that call with the configuration
for this hook (if any), the log prefix, and any given args and kwargs. Return any return value.
Raise ValueError if the hook name is unknown.
Raise AttributeError if the function name is not found in the module.
Raise anything else that the called function raises.
'''
config = hooks.get(hook_name, {})
hook_config = config.get(hook_name, {})
try:
module = HOOK_NAME_TO_MODULE[hook_name]
except KeyError:
raise ValueError('Unknown hook name: {}'.format(hook_name))
raise ValueError(f'Unknown hook name: {hook_name}')
logger.debug('{}: Calling {} hook function {}'.format(log_prefix, hook_name, function_name))
return getattr(module, function_name)(config, log_prefix, *args, **kwargs)
logger.debug(f'{log_prefix}: Calling {hook_name} hook function {function_name}')
return getattr(module, function_name)(hook_config, config, log_prefix, *args, **kwargs)
def call_hooks(function_name, hooks, log_prefix, hook_names, *args, **kwargs):
def call_hooks(function_name, config, log_prefix, hook_names, *args, **kwargs):
'''
Given the hooks configuration dict and a prefix to use in log entries, call the requested
function of the Python module corresponding to each given hook name. Supply each call with the
configuration for that hook, the log prefix, and any given args and kwargs. Collect any return
values into a dict from hook name to return value.
Given a configuration dict and a prefix to use in log entries, call the requested function of
the Python module corresponding to each given hook name. Supply each call with the configuration
for that hook, the log prefix, and any given args and kwargs. Collect any return values into a
dict from hook name to return value.
If the hook name is not present in the hooks configuration, then don't call the function for it
and omit it from the return values.
@ -62,23 +69,23 @@ def call_hooks(function_name, hooks, log_prefix, hook_names, *args, **kwargs):
Raise anything else that a called function raises. An error stops calls to subsequent functions.
'''
return {
hook_name: call_hook(function_name, hooks, log_prefix, hook_name, *args, **kwargs)
hook_name: call_hook(function_name, config, log_prefix, hook_name, *args, **kwargs)
for hook_name in hook_names
if hooks.get(hook_name)
if config.get(hook_name)
}
def call_hooks_even_if_unconfigured(function_name, hooks, log_prefix, hook_names, *args, **kwargs):
def call_hooks_even_if_unconfigured(function_name, config, log_prefix, hook_names, *args, **kwargs):
'''
Given the hooks configuration dict and a prefix to use in log entries, call the requested
function of the Python module corresponding to each given hook name. Supply each call with the
configuration for that hook, the log prefix, and any given args and kwargs. Collect any return
values into a dict from hook name to return value.
Given a configuration dict and a prefix to use in log entries, call the requested function of
the Python module corresponding to each given hook name. Supply each call with the configuration
for that hook, the log prefix, and any given args and kwargs. Collect any return values into a
dict from hook name to return value.
Raise AttributeError if the function name is not found in the module.
Raise anything else that a called function raises. An error stops calls to subsequent functions.
'''
return {
hook_name: call_hook(function_name, hooks, log_prefix, hook_name, *args, **kwargs)
hook_name: call_hook(function_name, config, log_prefix, hook_name, *args, **kwargs)
for hook_name in hook_names
}

View File

@ -6,29 +6,35 @@ from borgmatic.borg.state import DEFAULT_BORGMATIC_SOURCE_DIRECTORY
logger = logging.getLogger(__name__)
DATABASE_HOOK_NAMES = ('postgresql_databases', 'mysql_databases', 'mongodb_databases')
DATA_SOURCE_HOOK_NAMES = (
'mariadb_databases',
'mysql_databases',
'mongodb_databases',
'postgresql_databases',
'sqlite_databases',
)
def make_database_dump_path(borgmatic_source_directory, database_hook_name):
def make_data_source_dump_path(borgmatic_source_directory, data_source_hook_name):
'''
Given a borgmatic source directory (or None) and a database hook name, construct a database dump
path.
Given a borgmatic source directory (or None) and a data source hook name, construct a data
source dump path.
'''
if not borgmatic_source_directory:
borgmatic_source_directory = DEFAULT_BORGMATIC_SOURCE_DIRECTORY
return os.path.join(borgmatic_source_directory, database_hook_name)
return os.path.join(borgmatic_source_directory, data_source_hook_name)
def make_database_dump_filename(dump_path, name, hostname=None):
def make_data_source_dump_filename(dump_path, name, hostname=None):
'''
Based on the given dump directory path, database name, and hostname, return a filename to use
for the database dump. The hostname defaults to localhost.
Based on the given dump directory path, data source name, and hostname, return a filename to use
for the data source dump. The hostname defaults to localhost.
Raise ValueError if the database name is invalid.
Raise ValueError if the data source name is invalid.
'''
if os.path.sep in name:
raise ValueError('Invalid database name {}'.format(name))
raise ValueError(f'Invalid data source name {name}')
return os.path.join(os.path.expanduser(dump_path), hostname or 'localhost', name)
@ -48,16 +54,14 @@ def create_named_pipe_for_dump(dump_path):
os.mkfifo(dump_path, mode=0o600)
def remove_database_dumps(dump_path, database_type_name, log_prefix, dry_run):
def remove_data_source_dumps(dump_path, data_source_type_name, log_prefix, dry_run):
'''
Remove all database dumps in the given dump directory path (including the directory itself). If
this is a dry run, then don't actually remove anything.
Remove all data source dumps in the given dump directory path (including the directory itself).
If this is a dry run, then don't actually remove anything.
'''
dry_run_label = ' (dry run; not actually removing anything)' if dry_run else ''
logger.debug(
'{}: Removing {} database dumps{}'.format(log_prefix, database_type_name, dry_run_label)
)
logger.debug(f'{log_prefix}: Removing {data_source_type_name} data source dumps{dry_run_label}')
expanded_path = os.path.expanduser(dump_path)
@ -73,4 +77,4 @@ def convert_glob_patterns_to_borg_patterns(patterns):
Convert a sequence of shell glob patterns like "/etc/*" to the corresponding Borg archive
patterns like "sh:etc/*".
'''
return ['sh:{}'.format(pattern.lstrip(os.path.sep)) for pattern in patterns]
return [f'sh:{pattern.lstrip(os.path.sep)}' for pattern in patterns]

View File

@ -2,6 +2,7 @@ import logging
import requests
import borgmatic.hooks.logs
from borgmatic.hooks import monitor
logger = logging.getLogger(__name__)
@ -10,66 +11,14 @@ MONITOR_STATE_TO_HEALTHCHECKS = {
monitor.State.START: 'start',
monitor.State.FINISH: None, # Healthchecks doesn't append to the URL for the finished state.
monitor.State.FAIL: 'fail',
monitor.State.LOG: 'log',
}
PAYLOAD_TRUNCATION_INDICATOR = '...\n'
DEFAULT_PING_BODY_LIMIT_BYTES = 100000
DEFAULT_PING_BODY_LIMIT_BYTES = 1500
HANDLER_IDENTIFIER = 'healthchecks'
class Forgetful_buffering_handler(logging.Handler):
'''
A buffering log handler that stores log messages in memory, and throws away messages (oldest
first) once a particular capacity in bytes is reached. But if the given byte capacity is zero,
don't throw away any messages.
'''
def __init__(self, byte_capacity, log_level):
super().__init__()
self.byte_capacity = byte_capacity
self.byte_count = 0
self.buffer = []
self.forgot = False
self.setLevel(log_level)
def emit(self, record):
message = record.getMessage() + '\n'
self.byte_count += len(message)
self.buffer.append(message)
if not self.byte_capacity:
return
while self.byte_count > self.byte_capacity and self.buffer:
self.byte_count -= len(self.buffer[0])
self.buffer.pop(0)
self.forgot = True
def format_buffered_logs_for_payload():
'''
Get the handler previously added to the root logger, and slurp buffered logs out of it to
send to Healthchecks.
'''
try:
buffering_handler = next(
handler
for handler in logging.getLogger().handlers
if isinstance(handler, Forgetful_buffering_handler)
)
except StopIteration:
# No handler means no payload.
return ''
payload = ''.join(message for message in buffering_handler.buffer)
if buffering_handler.forgot:
return PAYLOAD_TRUNCATION_INDICATOR + payload
return payload
def initialize_monitor(hook_config, config_filename, monitoring_log_level, dry_run):
def initialize_monitor(hook_config, config, config_filename, monitoring_log_level, dry_run):
'''
Add a handler to the root logger that stores in memory the most recent logs emitted. That way,
we can send them all to Healthchecks upon a finish or failure state. But skip this if the
@ -80,16 +29,18 @@ def initialize_monitor(hook_config, config_filename, monitoring_log_level, dry_r
ping_body_limit = max(
hook_config.get('ping_body_limit', DEFAULT_PING_BODY_LIMIT_BYTES)
- len(PAYLOAD_TRUNCATION_INDICATOR),
- len(borgmatic.hooks.logs.PAYLOAD_TRUNCATION_INDICATOR),
0,
)
logging.getLogger().addHandler(
Forgetful_buffering_handler(ping_body_limit, monitoring_log_level)
borgmatic.hooks.logs.add_handler(
borgmatic.hooks.logs.Forgetful_buffering_handler(
HANDLER_IDENTIFIER, ping_body_limit, monitoring_log_level
)
)
def ping_monitor(hook_config, config_filename, state, monitoring_log_level, dry_run):
def ping_monitor(hook_config, config, config_filename, state, monitoring_log_level, dry_run):
'''
Ping the configured Healthchecks URL or UUID, modified with the monitor.State. Use the given
configuration filename in any log entries, and log to Healthchecks with the giving log level.
@ -98,7 +49,7 @@ def ping_monitor(hook_config, config_filename, state, monitoring_log_level, dry_
ping_url = (
hook_config['ping_url']
if hook_config['ping_url'].startswith('http')
else 'https://hc-ping.com/{}'.format(hook_config['ping_url'])
else f"https://hc-ping.com/{hook_config['ping_url']}"
)
dry_run_label = ' (dry run; not actually pinging)' if dry_run else ''
@ -110,15 +61,13 @@ def ping_monitor(hook_config, config_filename, state, monitoring_log_level, dry_
healthchecks_state = MONITOR_STATE_TO_HEALTHCHECKS.get(state)
if healthchecks_state:
ping_url = '{}/{}'.format(ping_url, healthchecks_state)
ping_url = f'{ping_url}/{healthchecks_state}'
logger.info(
'{}: Pinging Healthchecks {}{}'.format(config_filename, state.name.lower(), dry_run_label)
)
logger.debug('{}: Using Healthchecks ping URL {}'.format(config_filename, ping_url))
logger.info(f'{config_filename}: Pinging Healthchecks {state.name.lower()}{dry_run_label}')
logger.debug(f'{config_filename}: Using Healthchecks ping URL {ping_url}')
if state in (monitor.State.FINISH, monitor.State.FAIL):
payload = format_buffered_logs_for_payload()
if state in (monitor.State.FINISH, monitor.State.FAIL, monitor.State.LOG):
payload = borgmatic.hooks.logs.format_buffered_logs_for_payload(HANDLER_IDENTIFIER)
else:
payload = ''
@ -134,13 +83,9 @@ def ping_monitor(hook_config, config_filename, state, monitoring_log_level, dry_
logger.warning(f'{config_filename}: Healthchecks error: {error}')
def destroy_monitor(hook_config, config_filename, monitoring_log_level, dry_run):
def destroy_monitor(hook_config, config, config_filename, monitoring_log_level, dry_run):
'''
Remove the monitor handler that was added to the root logger. This prevents the handler from
getting reused by other instances of this monitor.
'''
logger = logging.getLogger()
for handler in tuple(logger.handlers):
if isinstance(handler, Forgetful_buffering_handler):
logger.removeHandler(handler)
borgmatic.hooks.logs.remove_handler(HANDLER_IDENTIFIER)

91
borgmatic/hooks/logs.py Normal file
View File

@ -0,0 +1,91 @@
import logging
PAYLOAD_TRUNCATION_INDICATOR = '...\n'
class Forgetful_buffering_handler(logging.Handler):
'''
A buffering log handler that stores log messages in memory, and throws away messages (oldest
first) once a particular capacity in bytes is reached. But if the given byte capacity is zero,
don't throw away any messages.
The given identifier is used to distinguish the instance of this handler used for one monitoring
hook from those instances used for other monitoring hooks.
'''
def __init__(self, identifier, byte_capacity, log_level):
super().__init__()
self.identifier = identifier
self.byte_capacity = byte_capacity
self.byte_count = 0
self.buffer = []
self.forgot = False
self.setLevel(log_level)
def emit(self, record):
message = record.getMessage() + '\n'
self.byte_count += len(message)
self.buffer.append(message)
if not self.byte_capacity:
return
while self.byte_count > self.byte_capacity and self.buffer:
self.byte_count -= len(self.buffer[0])
self.buffer.pop(0)
self.forgot = True
def add_handler(handler): # pragma: no cover
'''
Add the given handler to the global logger.
'''
logging.getLogger().addHandler(handler)
def get_handler(identifier):
'''
Given the identifier for an existing Forgetful_buffering_handler instance, return the handler.
Raise ValueError if the handler isn't found.
'''
try:
return next(
handler
for handler in logging.getLogger().handlers
if isinstance(handler, Forgetful_buffering_handler) and handler.identifier == identifier
)
except StopIteration:
raise ValueError(f'A buffering handler for {identifier} was not found')
def format_buffered_logs_for_payload(identifier):
'''
Get the handler previously added to the root logger, and slurp buffered logs out of it to
send to Healthchecks.
'''
try:
buffering_handler = get_handler(identifier)
except ValueError:
# No handler means no payload.
return ''
payload = ''.join(message for message in buffering_handler.buffer)
if buffering_handler.forgot:
return PAYLOAD_TRUNCATION_INDICATOR + payload
return payload
def remove_handler(identifier):
'''
Given the identifier for an existing Forgetful_buffering_handler instance, remove it.
'''
logger = logging.getLogger()
try:
logger.removeHandler(get_handler(identifier))
except ValueError:
pass

154
borgmatic/hooks/loki.py Normal file
View File

@ -0,0 +1,154 @@
import json
import logging
import os
import platform
import time
import requests
from borgmatic.hooks import monitor
logger = logging.getLogger(__name__)
MONITOR_STATE_TO_LOKI = {
monitor.State.START: 'Started',
monitor.State.FINISH: 'Finished',
monitor.State.FAIL: 'Failed',
}
# Threshold at which logs get flushed to loki
MAX_BUFFER_LINES = 100
class Loki_log_buffer:
'''
A log buffer that allows to output the logs as loki requests in json. Allows
adding labels to the log stream and takes care of communication with loki.
'''
def __init__(self, url, dry_run):
self.url = url
self.dry_run = dry_run
self.root = {'streams': [{'stream': {}, 'values': []}]}
def add_value(self, value):
'''
Add a log entry to the stream.
'''
timestamp = str(time.time_ns())
self.root['streams'][0]['values'].append((timestamp, value))
def add_label(self, label, value):
'''
Add a label to the logging stream.
'''
self.root['streams'][0]['stream'][label] = value
def to_request(self):
return json.dumps(self.root)
def __len__(self):
'''
Gets the number of lines currently in the buffer.
'''
return len(self.root['streams'][0]['values'])
def flush(self):
if self.dry_run:
# Just empty the buffer and skip
self.root['streams'][0]['values'] = []
logger.info('Skipped uploading logs to loki due to dry run')
return
if len(self) == 0:
# Skip as there are not logs to send yet
return
request_body = self.to_request()
self.root['streams'][0]['values'] = []
request_header = {'Content-Type': 'application/json'}
try:
result = requests.post(self.url, headers=request_header, data=request_body, timeout=5)
result.raise_for_status()
except requests.RequestException:
logger.warning('Failed to upload logs to loki')
class Loki_log_handler(logging.Handler):
'''
A log handler that sends logs to loki.
'''
def __init__(self, url, dry_run):
super().__init__()
self.buffer = Loki_log_buffer(url, dry_run)
def emit(self, record):
'''
Add a log record from the logging module to the stream.
'''
self.raw(record.getMessage())
def add_label(self, key, value):
'''
Add a label to the logging stream.
'''
self.buffer.add_label(key, value)
def raw(self, msg):
'''
Add an arbitrary string as a log entry to the stream.
'''
self.buffer.add_value(msg)
if len(self.buffer) > MAX_BUFFER_LINES:
self.buffer.flush()
def flush(self):
'''
Send the logs to loki and empty the buffer.
'''
self.buffer.flush()
def initialize_monitor(hook_config, config, config_filename, monitoring_log_level, dry_run):
'''
Add a handler to the root logger to regularly send the logs to loki.
'''
url = hook_config.get('url')
loki = Loki_log_handler(url, dry_run)
for key, value in hook_config.get('labels').items():
if value == '__hostname':
loki.add_label(key, platform.node())
elif value == '__config':
loki.add_label(key, os.path.basename(config_filename))
elif value == '__config_path':
loki.add_label(key, config_filename)
else:
loki.add_label(key, value)
logging.getLogger().addHandler(loki)
def ping_monitor(hook_config, config, config_filename, state, monitoring_log_level, dry_run):
'''
Add an entry to the loki logger with the current state.
'''
for handler in tuple(logging.getLogger().handlers):
if isinstance(handler, Loki_log_handler):
if state in MONITOR_STATE_TO_LOKI.keys():
handler.raw(f'{config_filename}: {MONITOR_STATE_TO_LOKI[state]} backup')
def destroy_monitor(hook_config, config, config_filename, monitoring_log_level, dry_run):
'''
Remove the monitor handler that was added to the root logger.
'''
logger = logging.getLogger()
for handler in tuple(logger.handlers):
if isinstance(handler, Loki_log_handler):
handler.flush()
logger.removeHandler(handler)

249
borgmatic/hooks/mariadb.py Normal file
View File

@ -0,0 +1,249 @@
import copy
import logging
import os
import shlex
from borgmatic.execute import (
execute_command,
execute_command_and_capture_output,
execute_command_with_processes,
)
from borgmatic.hooks import dump
logger = logging.getLogger(__name__)
def make_dump_path(config): # pragma: no cover
'''
Make the dump path from the given configuration dict and the name of this hook.
'''
return dump.make_data_source_dump_path(
config.get('borgmatic_source_directory'), 'mariadb_databases'
)
SYSTEM_DATABASE_NAMES = ('information_schema', 'mysql', 'performance_schema', 'sys')
def database_names_to_dump(database, extra_environment, log_prefix, dry_run):
'''
Given a requested database config, return the corresponding sequence of database names to dump.
In the case of "all", query for the names of databases on the configured host and return them,
excluding any system databases that will cause problems during restore.
'''
if database['name'] != 'all':
return (database['name'],)
if dry_run:
return ()
mariadb_show_command = tuple(
shlex.quote(part) for part in shlex.split(database.get('mariadb_command') or 'mariadb')
)
show_command = (
mariadb_show_command
+ (tuple(database['list_options'].split(' ')) if 'list_options' in database else ())
+ (('--host', database['hostname']) if 'hostname' in database else ())
+ (('--port', str(database['port'])) if 'port' in database else ())
+ (('--protocol', 'tcp') if 'hostname' in database or 'port' in database else ())
+ (('--user', database['username']) if 'username' in database else ())
+ ('--skip-column-names', '--batch')
+ ('--execute', 'show schemas')
)
logger.debug(f'{log_prefix}: Querying for "all" MariaDB databases to dump')
show_output = execute_command_and_capture_output(
show_command, extra_environment=extra_environment
)
return tuple(
show_name
for show_name in show_output.strip().splitlines()
if show_name not in SYSTEM_DATABASE_NAMES
)
def execute_dump_command(
database, log_prefix, dump_path, database_names, extra_environment, dry_run, dry_run_label
):
'''
Kick off a dump for the given MariaDB database (provided as a configuration dict) to a named
pipe constructed from the given dump path and database name. Use the given log prefix in any
log entries.
Return a subprocess.Popen instance for the dump process ready to spew to a named pipe. But if
this is a dry run, then don't actually dump anything and return None.
'''
database_name = database['name']
dump_filename = dump.make_data_source_dump_filename(
dump_path, database['name'], database.get('hostname')
)
if os.path.exists(dump_filename):
logger.warning(
f'{log_prefix}: Skipping duplicate dump of MariaDB database "{database_name}" to {dump_filename}'
)
return None
mariadb_dump_command = tuple(
shlex.quote(part)
for part in shlex.split(database.get('mariadb_dump_command') or 'mariadb-dump')
)
dump_command = (
mariadb_dump_command
+ (tuple(database['options'].split(' ')) if 'options' in database else ())
+ (('--add-drop-database',) if database.get('add_drop_database', True) else ())
+ (('--host', database['hostname']) if 'hostname' in database else ())
+ (('--port', str(database['port'])) if 'port' in database else ())
+ (('--protocol', 'tcp') if 'hostname' in database or 'port' in database else ())
+ (('--user', database['username']) if 'username' in database else ())
+ ('--databases',)
+ database_names
+ ('--result-file', dump_filename)
)
logger.debug(
f'{log_prefix}: Dumping MariaDB database "{database_name}" to {dump_filename}{dry_run_label}'
)
if dry_run:
return None
dump.create_named_pipe_for_dump(dump_filename)
return execute_command(
dump_command,
extra_environment=extra_environment,
run_to_completion=False,
)
def dump_data_sources(databases, config, log_prefix, dry_run):
'''
Dump the given MariaDB databases to a named pipe. The databases are supplied as a sequence of
dicts, one dict describing each database as per the configuration schema. Use the given
configuration dict to construct the destination path and the given log prefix in any log
entries.
Return a sequence of subprocess.Popen instances for the dump processes ready to spew to a named
pipe. But if this is a dry run, then don't actually dump anything and return an empty sequence.
'''
dry_run_label = ' (dry run; not actually dumping anything)' if dry_run else ''
processes = []
logger.info(f'{log_prefix}: Dumping MariaDB databases{dry_run_label}')
for database in databases:
dump_path = make_dump_path(config)
extra_environment = {'MYSQL_PWD': database['password']} if 'password' in database else None
dump_database_names = database_names_to_dump(
database, extra_environment, log_prefix, dry_run
)
if not dump_database_names:
if dry_run:
continue
raise ValueError('Cannot find any MariaDB databases to dump.')
if database['name'] == 'all' and database.get('format'):
for dump_name in dump_database_names:
renamed_database = copy.copy(database)
renamed_database['name'] = dump_name
processes.append(
execute_dump_command(
renamed_database,
log_prefix,
dump_path,
(dump_name,),
extra_environment,
dry_run,
dry_run_label,
)
)
else:
processes.append(
execute_dump_command(
database,
log_prefix,
dump_path,
dump_database_names,
extra_environment,
dry_run,
dry_run_label,
)
)
return [process for process in processes if process]
def remove_data_source_dumps(databases, config, log_prefix, dry_run): # pragma: no cover
'''
Remove all database dump files for this hook regardless of the given databases. Use the given
configuration dict to construct the destination path and the log prefix in any log entries. If
this is a dry run, then don't actually remove anything.
'''
dump.remove_data_source_dumps(make_dump_path(config), 'MariaDB', log_prefix, dry_run)
def make_data_source_dump_pattern(databases, config, log_prefix, name=None): # pragma: no cover
'''
Given a sequence of configurations dicts, a configuration dict, a prefix to log with, and a
database name to match, return the corresponding glob patterns to match the database dump in an
archive.
'''
return dump.make_data_source_dump_filename(make_dump_path(config), name, hostname='*')
def restore_data_source_dump(
hook_config, config, log_prefix, data_source, dry_run, extract_process, connection_params
):
'''
Restore a database from the given extract stream. The database is supplied as a data source
configuration dict, but the given hook configuration is ignored. The given configuration dict is
used to construct the destination path, and the given log prefix is used for any log entries. If
this is a dry run, then don't actually restore anything. Trigger the given active extract
process (an instance of subprocess.Popen) to produce output to consume.
'''
dry_run_label = ' (dry run; not actually restoring anything)' if dry_run else ''
hostname = connection_params['hostname'] or data_source.get(
'restore_hostname', data_source.get('hostname')
)
port = str(
connection_params['port'] or data_source.get('restore_port', data_source.get('port', ''))
)
username = connection_params['username'] or data_source.get(
'restore_username', data_source.get('username')
)
password = connection_params['password'] or data_source.get(
'restore_password', data_source.get('password')
)
mariadb_restore_command = tuple(
shlex.quote(part) for part in shlex.split(data_source.get('mariadb_command') or 'mariadb')
)
restore_command = (
mariadb_restore_command
+ ('--batch',)
+ (
tuple(data_source['restore_options'].split(' '))
if 'restore_options' in data_source
else ()
)
+ (('--host', hostname) if hostname else ())
+ (('--port', str(port)) if port else ())
+ (('--protocol', 'tcp') if hostname or port else ())
+ (('--user', username) if username else ())
)
extra_environment = {'MYSQL_PWD': password} if password else None
logger.debug(f"{log_prefix}: Restoring MariaDB database {data_source['name']}{dry_run_label}")
if dry_run:
return
# Don't give Borg local path so as to error on warnings, as "borg extract" only gives a warning
# if the restore paths don't exist in the archive.
execute_command_with_processes(
restore_command,
[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
extra_environment=extra_environment,
)

View File

@ -1,4 +1,5 @@
import logging
import shlex
from borgmatic.execute import execute_command, execute_command_with_processes
from borgmatic.hooks import dump
@ -6,52 +7,50 @@ from borgmatic.hooks import dump
logger = logging.getLogger(__name__)
def make_dump_path(location_config): # pragma: no cover
def make_dump_path(config): # pragma: no cover
'''
Make the dump path from the given location configuration and the name of this hook.
Make the dump path from the given configuration dict and the name of this hook.
'''
return dump.make_database_dump_path(
location_config.get('borgmatic_source_directory'), 'mongodb_databases'
return dump.make_data_source_dump_path(
config.get('borgmatic_source_directory'), 'mongodb_databases'
)
def dump_databases(databases, log_prefix, location_config, dry_run):
def dump_data_sources(databases, config, log_prefix, dry_run):
'''
Dump the given MongoDB databases to a named pipe. The databases are supplied as a sequence of
dicts, one dict describing each database as per the configuration schema. Use the given log
prefix in any log entries. Use the given location configuration dict to construct the
destination path.
dicts, one dict describing each database as per the configuration schema. Use the configuration
dict to construct the destination path and the given log prefix in any log entries.
Return a sequence of subprocess.Popen instances for the dump processes ready to spew to a named
pipe. But if this is a dry run, then don't actually dump anything and return an empty sequence.
'''
dry_run_label = ' (dry run; not actually dumping anything)' if dry_run else ''
logger.info('{}: Dumping MongoDB databases{}'.format(log_prefix, dry_run_label))
logger.info(f'{log_prefix}: Dumping MongoDB databases{dry_run_label}')
processes = []
for database in databases:
name = database['name']
dump_filename = dump.make_database_dump_filename(
make_dump_path(location_config), name, database.get('hostname')
dump_filename = dump.make_data_source_dump_filename(
make_dump_path(config), name, database.get('hostname')
)
dump_format = database.get('format', 'archive')
logger.debug(
'{}: Dumping MongoDB database {} to {}{}'.format(
log_prefix, name, dump_filename, dry_run_label
)
f'{log_prefix}: Dumping MongoDB database {name} to {dump_filename}{dry_run_label}',
)
if dry_run:
continue
command = build_dump_command(database, dump_filename, dump_format)
if dump_format == 'directory':
dump.create_parent_directory_for_dump(dump_filename)
execute_command(command, shell=True)
else:
dump.create_named_pipe_for_dump(dump_filename)
command = build_dump_command(database, dump_filename, dump_format)
processes.append(execute_command(command, shell=True, run_to_completion=False))
processes.append(execute_command(command, shell=True, run_to_completion=False))
return processes
@ -61,73 +60,69 @@ def build_dump_command(database, dump_filename, dump_format):
Return the mongodump command from a single database configuration.
'''
all_databases = database['name'] == 'all'
command = ['mongodump', '--archive']
if dump_format == 'directory':
command.append(dump_filename)
if 'hostname' in database:
command.extend(('--host', database['hostname']))
if 'port' in database:
command.extend(('--port', str(database['port'])))
if 'username' in database:
command.extend(('--username', database['username']))
if 'password' in database:
command.extend(('--password', database['password']))
if 'authentication_database' in database:
command.extend(('--authenticationDatabase', database['authentication_database']))
if not all_databases:
command.extend(('--db', database['name']))
if 'options' in database:
command.extend(database['options'].split(' '))
if dump_format != 'directory':
command.extend(('>', dump_filename))
return command
return (
('mongodump',)
+ (('--out', shlex.quote(dump_filename)) if dump_format == 'directory' else ())
+ (('--host', shlex.quote(database['hostname'])) if 'hostname' in database else ())
+ (('--port', shlex.quote(str(database['port']))) if 'port' in database else ())
+ (('--username', shlex.quote(database['username'])) if 'username' in database else ())
+ (('--password', shlex.quote(database['password'])) if 'password' in database else ())
+ (
('--authenticationDatabase', shlex.quote(database['authentication_database']))
if 'authentication_database' in database
else ()
)
+ (('--db', shlex.quote(database['name'])) if not all_databases else ())
+ (
tuple(shlex.quote(option) for option in database['options'].split(' '))
if 'options' in database
else ()
)
+ (('--archive', '>', shlex.quote(dump_filename)) if dump_format != 'directory' else ())
)
def remove_database_dumps(databases, log_prefix, location_config, dry_run): # pragma: no cover
def remove_data_source_dumps(databases, config, log_prefix, dry_run): # pragma: no cover
'''
Remove all database dump files for this hook regardless of the given databases. Use the log
prefix in any log entries. Use the given location configuration dict to construct the
destination path. If this is a dry run, then don't actually remove anything.
prefix in any log entries. Use the given configuration dict to construct the destination path.
If this is a dry run, then don't actually remove anything.
'''
dump.remove_database_dumps(make_dump_path(location_config), 'MongoDB', log_prefix, dry_run)
dump.remove_data_source_dumps(make_dump_path(config), 'MongoDB', log_prefix, dry_run)
def make_database_dump_pattern(
databases, log_prefix, location_config, name=None
): # pragma: no cover
def make_data_source_dump_pattern(databases, config, log_prefix, name=None): # pragma: no cover
'''
Given a sequence of configurations dicts, a prefix to log with, a location configuration dict,
Given a sequence of database configurations dicts, a configuration dict, a prefix to log with,
and a database name to match, return the corresponding glob patterns to match the database dump
in an archive.
'''
return dump.make_database_dump_filename(make_dump_path(location_config), name, hostname='*')
return dump.make_data_source_dump_filename(make_dump_path(config), name, hostname='*')
def restore_database_dump(database_config, log_prefix, location_config, dry_run, extract_process):
def restore_data_source_dump(
hook_config, config, log_prefix, data_source, dry_run, extract_process, connection_params
):
'''
Restore the given MongoDB database from an extract stream. The database is supplied as a
one-element sequence containing a dict describing the database, as per the configuration schema.
Use the given log prefix in any log entries. If this is a dry run, then don't actually restore
anything. Trigger the given active extract process (an instance of subprocess.Popen) to produce
output to consume.
Restore a database from the given extract stream. The database is supplied as a data source
configuration dict, but the given hook configuration is ignored. The given configuration dict is
used to construct the destination path, and the given log prefix is used for any log entries. If
this is a dry run, then don't actually restore anything. Trigger the given active extract
process (an instance of subprocess.Popen) to produce output to consume.
If the extract process is None, then restore the dump from the filesystem rather than from an
extract stream.
'''
dry_run_label = ' (dry run; not actually restoring anything)' if dry_run else ''
if len(database_config) != 1:
raise ValueError('The database configuration value is invalid')
database = database_config[0]
dump_filename = dump.make_database_dump_filename(
make_dump_path(location_config), database['name'], database.get('hostname')
dump_filename = dump.make_data_source_dump_filename(
make_dump_path(config), data_source['name'], data_source.get('hostname')
)
restore_command = build_restore_command(extract_process, database, dump_filename)
logger.debug(
'{}: Restoring MongoDB database {}{}'.format(log_prefix, database['name'], dry_run_label)
restore_command = build_restore_command(
extract_process, data_source, dump_filename, connection_params
)
logger.debug(f"{log_prefix}: Restoring MongoDB database {data_source['name']}{dry_run_label}")
if dry_run:
return
@ -141,25 +136,42 @@ def restore_database_dump(database_config, log_prefix, location_config, dry_run,
)
def build_restore_command(extract_process, database, dump_filename):
def build_restore_command(extract_process, database, dump_filename, connection_params):
'''
Return the mongorestore command from a single database configuration.
'''
command = ['mongorestore', '--archive']
if not extract_process:
command.append(dump_filename)
hostname = connection_params['hostname'] or database.get(
'restore_hostname', database.get('hostname')
)
port = str(connection_params['port'] or database.get('restore_port', database.get('port', '')))
username = connection_params['username'] or database.get(
'restore_username', database.get('username')
)
password = connection_params['password'] or database.get(
'restore_password', database.get('password')
)
command = ['mongorestore']
if extract_process:
command.append('--archive')
else:
command.extend(('--dir', dump_filename))
if database['name'] != 'all':
command.extend(('--drop', '--db', database['name']))
if 'hostname' in database:
command.extend(('--host', database['hostname']))
if 'port' in database:
command.extend(('--port', str(database['port'])))
if 'username' in database:
command.extend(('--username', database['username']))
if 'password' in database:
command.extend(('--password', database['password']))
command.extend(('--drop',))
if hostname:
command.extend(('--host', hostname))
if port:
command.extend(('--port', str(port)))
if username:
command.extend(('--username', username))
if password:
command.extend(('--password', password))
if 'authentication_database' in database:
command.extend(('--authenticationDatabase', database['authentication_database']))
if 'restore_options' in database:
command.extend(database['restore_options'].split(' '))
if database.get('schemas'):
for schema in database['schemas']:
command.extend(('--nsInclude', schema))
return command

View File

@ -1,9 +1,10 @@
from enum import Enum
MONITOR_HOOK_NAMES = ('healthchecks', 'cronitor', 'cronhub', 'pagerduty', 'ntfy')
MONITOR_HOOK_NAMES = ('apprise', 'healthchecks', 'cronitor', 'cronhub', 'pagerduty', 'ntfy', 'loki')
class State(Enum):
START = 1
FINISH = 2
FAIL = 3
LOG = 4

View File

@ -1,6 +1,7 @@
import copy
import logging
import os
import shlex
from borgmatic.execute import (
execute_command,
@ -12,19 +13,19 @@ from borgmatic.hooks import dump
logger = logging.getLogger(__name__)
def make_dump_path(location_config): # pragma: no cover
def make_dump_path(config): # pragma: no cover
'''
Make the dump path from the given location configuration and the name of this hook.
Make the dump path from the given configuration dict and the name of this hook.
'''
return dump.make_database_dump_path(
location_config.get('borgmatic_source_directory'), 'mysql_databases'
return dump.make_data_source_dump_path(
config.get('borgmatic_source_directory'), 'mysql_databases'
)
SYSTEM_DATABASE_NAMES = ('information_schema', 'mysql', 'performance_schema', 'sys')
def database_names_to_dump(database, extra_environment, log_prefix, dry_run_label):
def database_names_to_dump(database, extra_environment, log_prefix, dry_run):
'''
Given a requested database config, return the corresponding sequence of database names to dump.
In the case of "all", query for the names of databases on the configured host and return them,
@ -32,9 +33,14 @@ def database_names_to_dump(database, extra_environment, log_prefix, dry_run_labe
'''
if database['name'] != 'all':
return (database['name'],)
if dry_run:
return ()
mysql_show_command = tuple(
shlex.quote(part) for part in shlex.split(database.get('mysql_command') or 'mysql')
)
show_command = (
('mysql',)
mysql_show_command
+ (tuple(database['list_options'].split(' ')) if 'list_options' in database else ())
+ (('--host', database['hostname']) if 'hostname' in database else ())
+ (('--port', str(database['port'])) if 'port' in database else ())
@ -43,9 +49,7 @@ def database_names_to_dump(database, extra_environment, log_prefix, dry_run_labe
+ ('--skip-column-names', '--batch')
+ ('--execute', 'show schemas')
)
logger.debug(
'{}: Querying for "all" MySQL databases to dump{}'.format(log_prefix, dry_run_label)
)
logger.debug(f'{log_prefix}: Querying for "all" MySQL databases to dump')
show_output = execute_command_and_capture_output(
show_command, extra_environment=extra_environment
)
@ -62,35 +66,37 @@ def execute_dump_command(
):
'''
Kick off a dump for the given MySQL/MariaDB database (provided as a configuration dict) to a
named pipe constructed from the given dump path and database names. Use the given log prefix in
named pipe constructed from the given dump path and database name. Use the given log prefix in
any log entries.
Return a subprocess.Popen instance for the dump process ready to spew to a named pipe. But if
this is a dry run, then don't actually dump anything and return None.
'''
database_name = database['name']
dump_filename = dump.make_database_dump_filename(
dump_filename = dump.make_data_source_dump_filename(
dump_path, database['name'], database.get('hostname')
)
if os.path.exists(dump_filename):
logger.warning(
f'{log_prefix}: Skipping duplicate dump of MySQL database "{database_name}" to {dump_filename}'
)
return None
mysql_dump_command = tuple(
shlex.quote(part) for part in shlex.split(database.get('mysql_dump_command') or 'mysqldump')
)
dump_command = (
('mysqldump',)
mysql_dump_command
+ (tuple(database['options'].split(' ')) if 'options' in database else ())
+ ('--add-drop-database',)
+ (('--add-drop-database',) if database.get('add_drop_database', True) else ())
+ (('--host', database['hostname']) if 'hostname' in database else ())
+ (('--port', str(database['port'])) if 'port' in database else ())
+ (('--protocol', 'tcp') if 'hostname' in database or 'port' in database else ())
+ (('--user', database['username']) if 'username' in database else ())
+ ('--databases',)
+ database_names
# Use shell redirection rather than execute_command(output_file=open(...)) to prevent
# the open() call on a named pipe from hanging the main borgmatic process.
+ ('>', dump_filename)
+ ('--result-file', dump_filename)
)
logger.debug(
@ -102,16 +108,17 @@ def execute_dump_command(
dump.create_named_pipe_for_dump(dump_filename)
return execute_command(
dump_command, shell=True, extra_environment=extra_environment, run_to_completion=False,
dump_command,
extra_environment=extra_environment,
run_to_completion=False,
)
def dump_databases(databases, log_prefix, location_config, dry_run):
def dump_data_sources(databases, config, log_prefix, dry_run):
'''
Dump the given MySQL/MariaDB databases to a named pipe. The databases are supplied as a sequence
of dicts, one dict describing each database as per the configuration schema. Use the given log
prefix in any log entries. Use the given location configuration dict to construct the
destination path.
of dicts, one dict describing each database as per the configuration schema. Use the given
configuration dict to construct the destination path and the given log prefix in any log entries.
Return a sequence of subprocess.Popen instances for the dump processes ready to spew to a named
pipe. But if this is a dry run, then don't actually dump anything and return an empty sequence.
@ -119,15 +126,19 @@ def dump_databases(databases, log_prefix, location_config, dry_run):
dry_run_label = ' (dry run; not actually dumping anything)' if dry_run else ''
processes = []
logger.info('{}: Dumping MySQL databases{}'.format(log_prefix, dry_run_label))
logger.info(f'{log_prefix}: Dumping MySQL databases{dry_run_label}')
for database in databases:
dump_path = make_dump_path(location_config)
dump_path = make_dump_path(config)
extra_environment = {'MYSQL_PWD': database['password']} if 'password' in database else None
dump_database_names = database_names_to_dump(
database, extra_environment, log_prefix, dry_run_label
database, extra_environment, log_prefix, dry_run
)
if not dump_database_names:
if dry_run:
continue
raise ValueError('Cannot find any MySQL databases to dump.')
if database['name'] == 'all' and database.get('format'):
@ -161,53 +172,67 @@ def dump_databases(databases, log_prefix, location_config, dry_run):
return [process for process in processes if process]
def remove_database_dumps(databases, log_prefix, location_config, dry_run): # pragma: no cover
def remove_data_source_dumps(databases, config, log_prefix, dry_run): # pragma: no cover
'''
Remove all database dump files for this hook regardless of the given databases. Use the log
prefix in any log entries. Use the given location configuration dict to construct the
destination path. If this is a dry run, then don't actually remove anything.
Remove all database dump files for this hook regardless of the given databases. Use the given
configuration dict to construct the destination path and the log prefix in any log entries. If
this is a dry run, then don't actually remove anything.
'''
dump.remove_database_dumps(make_dump_path(location_config), 'MySQL', log_prefix, dry_run)
dump.remove_data_source_dumps(make_dump_path(config), 'MySQL', log_prefix, dry_run)
def make_database_dump_pattern(
databases, log_prefix, location_config, name=None
): # pragma: no cover
def make_data_source_dump_pattern(databases, config, log_prefix, name=None): # pragma: no cover
'''
Given a sequence of configurations dicts, a prefix to log with, a location configuration dict,
and a database name to match, return the corresponding glob patterns to match the database dump
in an archive.
Given a sequence of configurations dicts, a configuration dict, a prefix to log with, and a
database name to match, return the corresponding glob patterns to match the database dump in an
archive.
'''
return dump.make_database_dump_filename(make_dump_path(location_config), name, hostname='*')
return dump.make_data_source_dump_filename(make_dump_path(config), name, hostname='*')
def restore_database_dump(database_config, log_prefix, location_config, dry_run, extract_process):
def restore_data_source_dump(
hook_config, config, log_prefix, data_source, dry_run, extract_process, connection_params
):
'''
Restore the given MySQL/MariaDB database from an extract stream. The database is supplied as a
one-element sequence containing a dict describing the database, as per the configuration schema.
Use the given log prefix in any log entries. If this is a dry run, then don't actually restore
anything. Trigger the given active extract process (an instance of subprocess.Popen) to produce
output to consume.
Restore a database from the given extract stream. The database is supplied as a data source
configuration dict, but the given hook configuration is ignored. The given configuration dict is
used to construct the destination path, and the given log prefix is used for any log entries. If
this is a dry run, then don't actually restore anything. Trigger the given active extract
process (an instance of subprocess.Popen) to produce output to consume.
'''
dry_run_label = ' (dry run; not actually restoring anything)' if dry_run else ''
hostname = connection_params['hostname'] or data_source.get(
'restore_hostname', data_source.get('hostname')
)
port = str(
connection_params['port'] or data_source.get('restore_port', data_source.get('port', ''))
)
username = connection_params['username'] or data_source.get(
'restore_username', data_source.get('username')
)
password = connection_params['password'] or data_source.get(
'restore_password', data_source.get('password')
)
if len(database_config) != 1:
raise ValueError('The database configuration value is invalid')
database = database_config[0]
mysql_restore_command = tuple(
shlex.quote(part) for part in shlex.split(data_source.get('mysql_command') or 'mysql')
)
restore_command = (
('mysql', '--batch')
+ (tuple(database['restore_options'].split(' ')) if 'restore_options' in database else ())
+ (('--host', database['hostname']) if 'hostname' in database else ())
+ (('--port', str(database['port'])) if 'port' in database else ())
+ (('--protocol', 'tcp') if 'hostname' in database or 'port' in database else ())
+ (('--user', database['username']) if 'username' in database else ())
mysql_restore_command
+ ('--batch',)
+ (
tuple(data_source['restore_options'].split(' '))
if 'restore_options' in data_source
else ()
)
+ (('--host', hostname) if hostname else ())
+ (('--port', str(port)) if port else ())
+ (('--protocol', 'tcp') if hostname or port else ())
+ (('--user', username) if username else ())
)
extra_environment = {'MYSQL_PWD': database['password']} if 'password' in database else None
extra_environment = {'MYSQL_PWD': password} if password else None
logger.debug(
'{}: Restoring MySQL database {}{}'.format(log_prefix, database['name'], dry_run_label)
)
logger.debug(f"{log_prefix}: Restoring MySQL database {data_source['name']}{dry_run_label}")
if dry_run:
return

View File

@ -2,19 +2,11 @@ import logging
import requests
from borgmatic.hooks import monitor
logger = logging.getLogger(__name__)
MONITOR_STATE_TO_NTFY = {
monitor.State.START: None,
monitor.State.FINISH: None,
monitor.State.FAIL: None,
}
def initialize_monitor(
ping_url, config_filename, monitoring_log_level, dry_run
ping_url, config, config_filename, monitoring_log_level, dry_run
): # pragma: no cover
'''
No initialization is necessary for this monitor.
@ -22,7 +14,7 @@ def initialize_monitor(
pass
def ping_monitor(hook_config, config_filename, state, monitoring_log_level, dry_run):
def ping_monitor(hook_config, config, config_filename, state, monitoring_log_level, dry_run):
'''
Ping the configured Ntfy topic. Use the given configuration filename in any log entries.
If this is a dry run, then don't actually ping anything.
@ -36,8 +28,8 @@ def ping_monitor(hook_config, config_filename, state, monitoring_log_level, dry_
state_config = hook_config.get(
state.name.lower(),
{
'title': f'A Borgmatic {state.name} event happened',
'message': f'A Borgmatic {state.name} event happened',
'title': f'A borgmatic {state.name} event happened',
'message': f'A borgmatic {state.name} event happened',
'priority': 'default',
'tags': 'borgmatic',
},
@ -56,18 +48,41 @@ def ping_monitor(hook_config, config_filename, state, monitoring_log_level, dry_
'X-Tags': state_config.get('tags'),
}
username = hook_config.get('username')
password = hook_config.get('password')
access_token = hook_config.get('access_token')
auth = None
if access_token is not None:
if username or password:
logger.warning(
f'{config_filename}: ntfy access_token is set but so is username/password, only using access_token'
)
auth = requests.auth.HTTPBasicAuth('', access_token)
elif (username and password) is not None:
auth = requests.auth.HTTPBasicAuth(username, password)
logger.info(f'{config_filename}: Using basic auth with user {username} for ntfy')
elif username is not None:
logger.warning(
f'{config_filename}: Password missing for ntfy authentication, defaulting to no auth'
)
elif password is not None:
logger.warning(
f'{config_filename}: Username missing for ntfy authentication, defaulting to no auth'
)
if not dry_run:
logging.getLogger('urllib3').setLevel(logging.ERROR)
try:
response = requests.post(f'{base_url}/{topic}', headers=headers)
response = requests.post(f'{base_url}/{topic}', headers=headers, auth=auth)
if not response.ok:
response.raise_for_status()
except requests.exceptions.RequestException as error:
logger.warning(f'{config_filename}: Ntfy error: {error}')
logger.warning(f'{config_filename}: ntfy error: {error}')
def destroy_monitor(
ping_url_or_uuid, config_filename, monitoring_log_level, dry_run
ping_url_or_uuid, config, config_filename, monitoring_log_level, dry_run
): # pragma: no cover
'''
No destruction is necessary for this monitor.

View File

@ -13,7 +13,7 @@ EVENTS_API_URL = 'https://events.pagerduty.com/v2/enqueue'
def initialize_monitor(
integration_key, config_filename, monitoring_log_level, dry_run
integration_key, config, config_filename, monitoring_log_level, dry_run
): # pragma: no cover
'''
No initialization is necessary for this monitor.
@ -21,7 +21,7 @@ def initialize_monitor(
pass
def ping_monitor(hook_config, config_filename, state, monitoring_log_level, dry_run):
def ping_monitor(hook_config, config, config_filename, state, monitoring_log_level, dry_run):
'''
If this is an error state, create a PagerDuty event with the configured integration key. Use
the given configuration filename in any log entries. If this is a dry run, then don't actually
@ -29,14 +29,12 @@ def ping_monitor(hook_config, config_filename, state, monitoring_log_level, dry_
'''
if state != monitor.State.FAIL:
logger.debug(
'{}: Ignoring unsupported monitoring {} in PagerDuty hook'.format(
config_filename, state.name.lower()
)
f'{config_filename}: Ignoring unsupported monitoring {state.name.lower()} in PagerDuty hook',
)
return
dry_run_label = ' (dry run; not actually sending)' if dry_run else ''
logger.info('{}: Sending failure event to PagerDuty {}'.format(config_filename, dry_run_label))
logger.info(f'{config_filename}: Sending failure event to PagerDuty {dry_run_label}')
if dry_run:
return
@ -50,7 +48,7 @@ def ping_monitor(hook_config, config_filename, state, monitoring_log_level, dry_
'routing_key': hook_config['integration_key'],
'event_action': 'trigger',
'payload': {
'summary': 'backup failed on {}'.format(hostname),
'summary': f'backup failed on {hostname}',
'severity': 'error',
'source': hostname,
'timestamp': local_timestamp,
@ -65,7 +63,7 @@ def ping_monitor(hook_config, config_filename, state, monitoring_log_level, dry_
},
}
)
logger.debug('{}: Using PagerDuty payload: {}'.format(config_filename, payload))
logger.debug(f'{config_filename}: Using PagerDuty payload: {payload}')
logging.getLogger('urllib3').setLevel(logging.ERROR)
try:
@ -77,7 +75,7 @@ def ping_monitor(hook_config, config_filename, state, monitoring_log_level, dry_
def destroy_monitor(
ping_url_or_uuid, config_filename, monitoring_log_level, dry_run
ping_url_or_uuid, config, config_filename, monitoring_log_level, dry_run
): # pragma: no cover
'''
No destruction is necessary for this monitor.

View File

@ -1,6 +1,8 @@
import csv
import itertools
import logging
import os
import shlex
from borgmatic.execute import (
execute_command,
@ -12,23 +14,34 @@ from borgmatic.hooks import dump
logger = logging.getLogger(__name__)
def make_dump_path(location_config): # pragma: no cover
def make_dump_path(config): # pragma: no cover
'''
Make the dump path from the given location configuration and the name of this hook.
Make the dump path from the given configuration dict and the name of this hook.
'''
return dump.make_database_dump_path(
location_config.get('borgmatic_source_directory'), 'postgresql_databases'
return dump.make_data_source_dump_path(
config.get('borgmatic_source_directory'), 'postgresql_databases'
)
def make_extra_environment(database):
def make_extra_environment(database, restore_connection_params=None):
'''
Make the extra_environment dict from the given database configuration.
Make the extra_environment dict from the given database configuration. If restore connection
params are given, this is for a restore operation.
'''
extra = dict()
if 'password' in database:
extra['PGPASSWORD'] = database['password']
extra['PGSSLMODE'] = database.get('ssl_mode', 'disable')
try:
if restore_connection_params:
extra['PGPASSWORD'] = restore_connection_params.get('password') or database.get(
'restore_password', database['password']
)
else:
extra['PGPASSWORD'] = database['password']
except (AttributeError, KeyError):
pass
if 'ssl_mode' in database:
extra['PGSSLMODE'] = database['ssl_mode']
if 'ssl_cert' in database:
extra['PGSSLCERT'] = database['ssl_cert']
if 'ssl_key' in database:
@ -37,13 +50,14 @@ def make_extra_environment(database):
extra['PGSSLROOTCERT'] = database['ssl_root_cert']
if 'ssl_crl' in database:
extra['PGSSLCRL'] = database['ssl_crl']
return extra
EXCLUDED_DATABASE_NAMES = ('template0', 'template1')
def database_names_to_dump(database, extra_environment, log_prefix, dry_run_label):
def database_names_to_dump(database, extra_environment, log_prefix, dry_run):
'''
Given a requested database config, return the corresponding sequence of database names to dump.
In the case of "all" when a database format is given, query for the names of databases on the
@ -56,17 +70,21 @@ def database_names_to_dump(database, extra_environment, log_prefix, dry_run_labe
return (requested_name,)
if not database.get('format'):
return ('all',)
if dry_run:
return ()
psql_command = tuple(
shlex.quote(part) for part in shlex.split(database.get('psql_command') or 'psql')
)
list_command = (
('psql', '--list', '--no-password', '--csv', '--tuples-only')
psql_command
+ ('--list', '--no-password', '--no-psqlrc', '--csv', '--tuples-only')
+ (('--host', database['hostname']) if 'hostname' in database else ())
+ (('--port', str(database['port'])) if 'port' in database else ())
+ (('--username', database['username']) if 'username' in database else ())
+ (tuple(database['list_options'].split(' ')) if 'list_options' in database else ())
)
logger.debug(
'{}: Querying for "all" PostgreSQL databases to dump{}'.format(log_prefix, dry_run_label)
)
logger.debug(f'{log_prefix}: Querying for "all" PostgreSQL databases to dump')
list_output = execute_command_and_capture_output(
list_command, extra_environment=extra_environment
)
@ -78,12 +96,12 @@ def database_names_to_dump(database, extra_environment, log_prefix, dry_run_labe
)
def dump_databases(databases, log_prefix, location_config, dry_run):
def dump_data_sources(databases, config, log_prefix, dry_run):
'''
Dump the given PostgreSQL databases to a named pipe. The databases are supplied as a sequence of
dicts, one dict describing each database as per the configuration schema. Use the given log
prefix in any log entries. Use the given location configuration dict to construct the
destination path.
dicts, one dict describing each database as per the configuration schema. Use the given
configuration dict to construct the destination path and the given log prefix in any log
entries.
Return a sequence of subprocess.Popen instances for the dump processes ready to spew to a named
pipe. But if this is a dry run, then don't actually dump anything and return an empty sequence.
@ -93,23 +111,29 @@ def dump_databases(databases, log_prefix, location_config, dry_run):
dry_run_label = ' (dry run; not actually dumping anything)' if dry_run else ''
processes = []
logger.info('{}: Dumping PostgreSQL databases{}'.format(log_prefix, dry_run_label))
logger.info(f'{log_prefix}: Dumping PostgreSQL databases{dry_run_label}')
for database in databases:
extra_environment = make_extra_environment(database)
dump_path = make_dump_path(location_config)
dump_path = make_dump_path(config)
dump_database_names = database_names_to_dump(
database, extra_environment, log_prefix, dry_run_label
database, extra_environment, log_prefix, dry_run
)
if not dump_database_names:
if dry_run:
continue
raise ValueError('Cannot find any PostgreSQL databases to dump.')
for database_name in dump_database_names:
dump_format = database.get('format', None if database_name == 'all' else 'custom')
default_dump_command = 'pg_dumpall' if database_name == 'all' else 'pg_dump'
dump_command = database.get('pg_dump_command') or default_dump_command
dump_filename = dump.make_database_dump_filename(
dump_command = tuple(
shlex.quote(part)
for part in shlex.split(database.get('pg_dump_command') or default_dump_command)
)
dump_filename = dump.make_data_source_dump_filename(
dump_path, database_name, database.get('hostname')
)
if os.path.exists(dump_filename):
@ -119,18 +143,32 @@ def dump_databases(databases, log_prefix, location_config, dry_run):
continue
command = (
(dump_command, '--no-password', '--clean', '--if-exists',)
+ (('--host', database['hostname']) if 'hostname' in database else ())
+ (('--port', str(database['port'])) if 'port' in database else ())
+ (('--username', database['username']) if 'username' in database else ())
+ (('--format', dump_format) if dump_format else ())
+ (('--file', dump_filename) if dump_format == 'directory' else ())
+ (tuple(database['options'].split(' ')) if 'options' in database else ())
+ (() if database_name == 'all' else (database_name,))
dump_command
+ (
'--no-password',
'--clean',
'--if-exists',
)
+ (('--host', shlex.quote(database['hostname'])) if 'hostname' in database else ())
+ (('--port', shlex.quote(str(database['port']))) if 'port' in database else ())
+ (
('--username', shlex.quote(database['username']))
if 'username' in database
else ()
)
+ (('--no-owner',) if database.get('no_owner', False) else ())
+ (('--format', shlex.quote(dump_format)) if dump_format else ())
+ (('--file', shlex.quote(dump_filename)) if dump_format == 'directory' else ())
+ (
tuple(shlex.quote(option) for option in database['options'].split(' '))
if 'options' in database
else ()
)
+ (() if database_name == 'all' else (shlex.quote(database_name),))
# Use shell redirection rather than the --file flag to sidestep synchronization issues
# when pg_dump/pg_dumpall tries to write to a named pipe. But for the directory dump
# format in a particular, a named destination is required, and redirection doesn't work.
+ (('>', dump_filename) if dump_format != 'directory' else ())
+ (('>', shlex.quote(dump_filename)) if dump_format != 'directory' else ())
)
logger.debug(
@ -141,90 +179,124 @@ def dump_databases(databases, log_prefix, location_config, dry_run):
if dump_format == 'directory':
dump.create_parent_directory_for_dump(dump_filename)
else:
dump.create_named_pipe_for_dump(dump_filename)
processes.append(
execute_command(
command,
shell=True,
extra_environment=extra_environment,
run_to_completion=False,
)
)
else:
dump.create_named_pipe_for_dump(dump_filename)
processes.append(
execute_command(
command,
shell=True,
extra_environment=extra_environment,
run_to_completion=False,
)
)
return processes
def remove_database_dumps(databases, log_prefix, location_config, dry_run): # pragma: no cover
def remove_data_source_dumps(databases, config, log_prefix, dry_run): # pragma: no cover
'''
Remove all database dump files for this hook regardless of the given databases. Use the log
prefix in any log entries. Use the given location configuration dict to construct the
destination path. If this is a dry run, then don't actually remove anything.
Remove all database dump files for this hook regardless of the given databases. Use the given
configuration dict to construct the destination path and the log prefix in any log entries. If
this is a dry run, then don't actually remove anything.
'''
dump.remove_database_dumps(make_dump_path(location_config), 'PostgreSQL', log_prefix, dry_run)
dump.remove_data_source_dumps(make_dump_path(config), 'PostgreSQL', log_prefix, dry_run)
def make_database_dump_pattern(
databases, log_prefix, location_config, name=None
): # pragma: no cover
def make_data_source_dump_pattern(databases, config, log_prefix, name=None): # pragma: no cover
'''
Given a sequence of configurations dicts, a prefix to log with, a location configuration dict,
and a database name to match, return the corresponding glob patterns to match the database dump
in an archive.
Given a sequence of configurations dicts, a configuration dict, a prefix to log with, and a
database name to match, return the corresponding glob patterns to match the database dump in an
archive.
'''
return dump.make_database_dump_filename(make_dump_path(location_config), name, hostname='*')
return dump.make_data_source_dump_filename(make_dump_path(config), name, hostname='*')
def restore_database_dump(database_config, log_prefix, location_config, dry_run, extract_process):
def restore_data_source_dump(
hook_config, config, log_prefix, data_source, dry_run, extract_process, connection_params
):
'''
Restore the given PostgreSQL database from an extract stream. The database is supplied as a
one-element sequence containing a dict describing the database, as per the configuration schema.
Use the given log prefix in any log entries. If this is a dry run, then don't actually restore
anything. Trigger the given active extract process (an instance of subprocess.Popen) to produce
output to consume.
Restore a database from the given extract stream. The database is supplied as a data source
configuration dict, but the given hook configuration is ignored. The given configuration dict is
used to construct the destination path, and the given log prefix is used for any log entries. If
this is a dry run, then don't actually restore anything. Trigger the given active extract
process (an instance of subprocess.Popen) to produce output to consume.
If the extract process is None, then restore the dump from the filesystem rather than from an
extract stream.
Use the given connection parameters to connect to the database. The connection parameters are
hostname, port, username, and password.
'''
dry_run_label = ' (dry run; not actually restoring anything)' if dry_run else ''
if len(database_config) != 1:
raise ValueError('The database configuration value is invalid')
database = database_config[0]
all_databases = bool(database['name'] == 'all')
dump_filename = dump.make_database_dump_filename(
make_dump_path(location_config), database['name'], database.get('hostname')
hostname = connection_params['hostname'] or data_source.get(
'restore_hostname', data_source.get('hostname')
)
port = str(
connection_params['port'] or data_source.get('restore_port', data_source.get('port', ''))
)
username = connection_params['username'] or data_source.get(
'restore_username', data_source.get('username')
)
all_databases = bool(data_source['name'] == 'all')
dump_filename = dump.make_data_source_dump_filename(
make_dump_path(config), data_source['name'], data_source.get('hostname')
)
psql_command = tuple(
shlex.quote(part) for part in shlex.split(data_source.get('psql_command') or 'psql')
)
psql_command = database.get('psql_command') or 'psql'
analyze_command = (
(psql_command, '--no-password', '--quiet')
+ (('--host', database['hostname']) if 'hostname' in database else ())
+ (('--port', str(database['port'])) if 'port' in database else ())
+ (('--username', database['username']) if 'username' in database else ())
+ (('--dbname', database['name']) if not all_databases else ())
+ (tuple(database['analyze_options'].split(' ')) if 'analyze_options' in database else ())
+ ('--command', 'ANALYZE')
)
pg_restore_command = database.get('pg_restore_command') or 'pg_restore'
restore_command = (
(psql_command if all_databases else pg_restore_command, '--no-password')
psql_command
+ ('--no-password', '--no-psqlrc', '--quiet')
+ (('--host', hostname) if hostname else ())
+ (('--port', port) if port else ())
+ (('--username', username) if username else ())
+ (('--dbname', data_source['name']) if not all_databases else ())
+ (
('--if-exists', '--exit-on-error', '--clean', '--dbname', database['name'])
if not all_databases
tuple(data_source['analyze_options'].split(' '))
if 'analyze_options' in data_source
else ()
)
+ (('--host', database['hostname']) if 'hostname' in database else ())
+ (('--port', str(database['port'])) if 'port' in database else ())
+ (('--username', database['username']) if 'username' in database else ())
+ (tuple(database['restore_options'].split(' ')) if 'restore_options' in database else ())
+ (() if extract_process else (dump_filename,))
+ ('--command', 'ANALYZE')
)
use_psql_command = all_databases or data_source.get('format') == 'plain'
pg_restore_command = tuple(
shlex.quote(part)
for part in shlex.split(data_source.get('pg_restore_command') or 'pg_restore')
)
restore_command = (
(psql_command if use_psql_command else pg_restore_command)
+ ('--no-password',)
+ (('--no-psqlrc',) if use_psql_command else ('--if-exists', '--exit-on-error', '--clean'))
+ (('--dbname', data_source['name']) if not all_databases else ())
+ (('--host', hostname) if hostname else ())
+ (('--port', port) if port else ())
+ (('--username', username) if username else ())
+ (('--no-owner',) if data_source.get('no_owner', False) else ())
+ (
tuple(data_source['restore_options'].split(' '))
if 'restore_options' in data_source
else ()
)
+ (() if extract_process else (dump_filename,))
+ tuple(
itertools.chain.from_iterable(('--schema', schema) for schema in data_source['schemas'])
if data_source.get('schemas')
else ()
)
)
extra_environment = make_extra_environment(
data_source, restore_connection_params=connection_params
)
extra_environment = make_extra_environment(database)
logger.debug(
'{}: Restoring PostgreSQL database {}{}'.format(log_prefix, database['name'], dry_run_label)
f"{log_prefix}: Restoring PostgreSQL database {data_source['name']}{dry_run_label}"
)
if dry_run:
return

127
borgmatic/hooks/sqlite.py Normal file
View File

@ -0,0 +1,127 @@
import logging
import os
import shlex
from borgmatic.execute import execute_command, execute_command_with_processes
from borgmatic.hooks import dump
logger = logging.getLogger(__name__)
def make_dump_path(config): # pragma: no cover
'''
Make the dump path from the given configuration dict and the name of this hook.
'''
return dump.make_data_source_dump_path(
config.get('borgmatic_source_directory'), 'sqlite_databases'
)
def dump_data_sources(databases, config, log_prefix, dry_run):
'''
Dump the given SQLite3 databases to a named pipe. The databases are supplied as a sequence of
configuration dicts, as per the configuration schema. Use the given configuration dict to
construct the destination path and the given log prefix in any log entries.
Return a sequence of subprocess.Popen instances for the dump processes ready to spew to a named
pipe. But if this is a dry run, then don't actually dump anything and return an empty sequence.
'''
dry_run_label = ' (dry run; not actually dumping anything)' if dry_run else ''
processes = []
logger.info(f'{log_prefix}: Dumping SQLite databases{dry_run_label}')
for database in databases:
database_path = database['path']
if database['name'] == 'all':
logger.warning('The "all" database name has no meaning for SQLite databases')
if not os.path.exists(database_path):
logger.warning(
f'{log_prefix}: No SQLite database at {database_path}; an empty database will be created and dumped'
)
dump_path = make_dump_path(config)
dump_filename = dump.make_data_source_dump_filename(dump_path, database['name'])
if os.path.exists(dump_filename):
logger.warning(
f'{log_prefix}: Skipping duplicate dump of SQLite database at {database_path} to {dump_filename}'
)
continue
command = (
'sqlite3',
shlex.quote(database_path),
'.dump',
'>',
shlex.quote(dump_filename),
)
logger.debug(
f'{log_prefix}: Dumping SQLite database at {database_path} to {dump_filename}{dry_run_label}'
)
if dry_run:
continue
dump.create_named_pipe_for_dump(dump_filename)
processes.append(execute_command(command, shell=True, run_to_completion=False))
return processes
def remove_data_source_dumps(databases, config, log_prefix, dry_run): # pragma: no cover
'''
Remove the given SQLite3 database dumps from the filesystem. The databases are supplied as a
sequence of configuration dicts, as per the configuration schema. Use the given configuration
dict to construct the destination path and the given log prefix in any log entries. If this is a
dry run, then don't actually remove anything.
'''
dump.remove_data_source_dumps(make_dump_path(config), 'SQLite', log_prefix, dry_run)
def make_data_source_dump_pattern(databases, config, log_prefix, name=None): # pragma: no cover
'''
Make a pattern that matches the given SQLite3 databases. The databases are supplied as a
sequence of configuration dicts, as per the configuration schema.
'''
return dump.make_data_source_dump_filename(make_dump_path(config), name)
def restore_data_source_dump(
hook_config, config, log_prefix, data_source, dry_run, extract_process, connection_params
):
'''
Restore a database from the given extract stream. The database is supplied as a data source
configuration dict, but the given hook configuration is ignored. The given configuration dict is
used to construct the destination path, and the given log prefix is used for any log entries. If
this is a dry run, then don't actually restore anything. Trigger the given active extract
process (an instance of subprocess.Popen) to produce output to consume.
'''
dry_run_label = ' (dry run; not actually restoring anything)' if dry_run else ''
database_path = connection_params['restore_path'] or data_source.get(
'restore_path', data_source.get('path')
)
logger.debug(f'{log_prefix}: Restoring SQLite database at {database_path}{dry_run_label}')
if dry_run:
return
try:
os.remove(database_path)
logger.warning(f'{log_prefix}: Removed existing SQLite database at {database_path}')
except FileNotFoundError: # pragma: no cover
pass
restore_command = (
'sqlite3',
database_path,
)
# Don't give Borg local path so as to error on warnings, as "borg extract" only gives a warning
# if the restore paths don't exist in the archive.
execute_command_with_processes(
restore_command,
[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
)

View File

@ -41,6 +41,9 @@ def should_do_markup(no_color, configs):
if any(config.get('output', {}).get('color') is False for config in configs.values()):
return False
if os.environ.get('NO_COLOR', None):
return False
py_colors = os.environ.get('PY_COLORS', None)
if py_colors is not None:
@ -68,7 +71,7 @@ class Multi_stream_handler(logging.Handler):
def emit(self, record):
'''
Dispatch the log record to the approriate stream handler for the record's log level.
Dispatch the log record to the appropriate stream handler for the record's log level.
'''
self.log_level_to_handler[record.levelno].emit(record)
@ -108,7 +111,7 @@ def color_text(color, message):
if not color:
return message
return '{}{}{}'.format(color, message, colorama.Style.RESET_ALL)
return f'{color}{message}{colorama.Style.RESET_ALL}'
def add_logging_level(level_name, level_number):
@ -141,6 +144,7 @@ def add_logging_level(level_name, level_number):
ANSWER = logging.WARN - 5
DISABLED = logging.CRITICAL + 10
def add_custom_log_levels(): # pragma: no cover
@ -148,6 +152,7 @@ def add_custom_log_levels(): # pragma: no cover
Add a custom log level between WARN and INFO for user-requested answers.
'''
add_logging_level('ANSWER', ANSWER)
add_logging_level('DISABLED', DISABLED)
def configure_logging(
@ -156,28 +161,32 @@ def configure_logging(
log_file_log_level=None,
monitoring_log_level=None,
log_file=None,
log_file_format=None,
color_enabled=True,
):
'''
Configure logging to go to both the console and (syslog or log file). Use the given log levels,
respectively.
respectively. If color is enabled, set up log formatting accordingly.
Raise FileNotFoundError or PermissionError if the log file could not be opened for writing.
'''
add_custom_log_levels()
if syslog_log_level is None:
syslog_log_level = console_log_level
syslog_log_level = logging.DISABLED
if log_file_log_level is None:
log_file_log_level = console_log_level
if monitoring_log_level is None:
monitoring_log_level = console_log_level
add_custom_log_levels()
# Log certain log levels to console stderr and others to stdout. This supports use cases like
# grepping (non-error) output.
console_disabled = logging.NullHandler()
console_error_handler = logging.StreamHandler(sys.stderr)
console_standard_handler = logging.StreamHandler(sys.stdout)
console_handler = Multi_stream_handler(
{
logging.DISABLED: console_disabled,
logging.CRITICAL: console_error_handler,
logging.ERROR: console_error_handler,
logging.WARN: console_error_handler,
@ -186,11 +195,17 @@ def configure_logging(
logging.DEBUG: console_standard_handler,
}
)
console_handler.setFormatter(Console_color_formatter())
if color_enabled:
console_handler.setFormatter(Console_color_formatter())
console_handler.setLevel(console_log_level)
syslog_path = None
if log_file is None:
handlers = [console_handler]
if syslog_log_level != logging.DISABLED:
syslog_path = None
if os.path.exists('/dev/log'):
syslog_path = '/dev/log'
elif os.path.exists('/var/run/syslog'):
@ -198,20 +213,25 @@ def configure_logging(
elif os.path.exists('/var/run/log'):
syslog_path = '/var/run/log'
if syslog_path and not interactive_console():
syslog_handler = logging.handlers.SysLogHandler(address=syslog_path)
syslog_handler.setFormatter(logging.Formatter('borgmatic: %(levelname)s %(message)s'))
syslog_handler.setLevel(syslog_log_level)
handlers = (console_handler, syslog_handler)
elif log_file:
if syslog_path:
syslog_handler = logging.handlers.SysLogHandler(address=syslog_path)
syslog_handler.setFormatter(
logging.Formatter('borgmatic: {levelname} {message}', style='{') # noqa: FS003
)
syslog_handler.setLevel(syslog_log_level)
handlers.append(syslog_handler)
if log_file and log_file_log_level != logging.DISABLED:
file_handler = logging.handlers.WatchedFileHandler(log_file)
file_handler.setFormatter(logging.Formatter('[%(asctime)s] %(levelname)s: %(message)s'))
file_handler.setFormatter(
logging.Formatter(
log_file_format or '[{asctime}] {levelname}: {message}', style='{' # noqa: FS003
)
)
file_handler.setLevel(log_file_log_level)
handlers = (console_handler, file_handler)
else:
handlers = (console_handler,)
handlers.append(file_handler)
logging.basicConfig(
level=min(console_log_level, syslog_log_level, log_file_log_level, monitoring_log_level),
level=min(handler.level for handler in handlers),
handlers=handlers,
)

View File

@ -23,12 +23,20 @@ def handle_signal(signal_number, frame):
if signal_number == signal.SIGTERM:
logger.critical('Exiting due to TERM signal')
sys.exit(EXIT_CODE_FROM_SIGNAL + signal.SIGTERM)
elif signal_number == signal.SIGINT:
raise KeyboardInterrupt()
def configure_signals():
'''
Configure borgmatic's signal handlers to pass relevant signals through to any child processes
like Borg. Note that SIGINT gets passed through even without these changes.
like Borg.
'''
for signal_number in (signal.SIGHUP, signal.SIGTERM, signal.SIGUSR1, signal.SIGUSR2):
for signal_number in (
signal.SIGHUP,
signal.SIGINT,
signal.SIGTERM,
signal.SIGUSR1,
signal.SIGUSR2,
):
signal.signal(signal_number, handle_signal)

View File

@ -2,6 +2,7 @@ import logging
import borgmatic.logger
VERBOSITY_DISABLED = -2
VERBOSITY_ERROR = -1
VERBOSITY_ANSWER = 0
VERBOSITY_SOME = 1
@ -15,6 +16,7 @@ def verbosity_to_log_level(verbosity):
borgmatic.logger.add_custom_log_levels()
return {
VERBOSITY_DISABLED: logging.DISABLED,
VERBOSITY_ERROR: logging.ERROR,
VERBOSITY_ANSWER: logging.ANSWER,
VERBOSITY_SOME: logging.INFO,

View File

@ -1,14 +1,14 @@
FROM alpine:3.17.1 as borgmatic
FROM docker.io/alpine:3.17.1 as borgmatic
COPY . /app
RUN apk add --no-cache py3-pip py3-ruamel.yaml py3-ruamel.yaml.clib
RUN pip install --no-cache /app && generate-borgmatic-config && chmod +r /etc/borgmatic/config.yaml
RUN borgmatic --help > /command-line.txt \
&& for action in rcreate transfer prune compact create check extract export-tar mount umount restore rlist list rinfo info break-lock borg; do \
&& for action in rcreate transfer create prune compact check extract config "config bootstrap" "config generate" "config validate" export-tar mount umount restore rlist list rinfo info break-lock borg; do \
echo -e "\n--------------------------------------------------------------------------------\n" >> /command-line.txt \
&& borgmatic "$action" --help >> /command-line.txt; done
&& borgmatic $action --help >> /command-line.txt; done
FROM node:19.5.0-alpine as html
FROM docker.io/node:19.5.0-alpine as html
ARG ENVIRONMENT=production
@ -18,6 +18,7 @@ RUN npm install @11ty/eleventy \
@11ty/eleventy-plugin-syntaxhighlight \
@11ty/eleventy-plugin-inclusive-language \
@11ty/eleventy-navigation \
eleventy-plugin-code-clipboard \
markdown-it \
markdown-it-anchor \
markdown-it-replace-link
@ -27,7 +28,7 @@ COPY . /source
RUN NODE_ENV=${ENVIRONMENT} npx eleventy --input=/source/docs --output=/output/docs \
&& mv /output/docs/index.html /output/index.html
FROM nginx:1.22.1-alpine
FROM docker.io/nginx:1.22.1-alpine
COPY --from=html /output /usr/share/nginx/html
COPY --from=borgmatic /etc/borgmatic/config.yaml /usr/share/nginx/html/docs/reference/config.yaml

View File

@ -16,4 +16,4 @@ each.
If you find a security vulnerability, please [file a
ticket](https://torsion.org/borgmatic/#issues) or [send email
directly](mailto:witten@torsion.org) as appropriate. You should expect to hear
back within a few days at most, and generally sooner.
back within a few days at most and generally sooner.

5
docs/_data/borgmatic.js Normal file
View File

@ -0,0 +1,5 @@
module.exports = function() {
return {
environment: process.env.NODE_ENV || "development"
};
};

Some files were not shown because too many files have changed in this diff Show More