Compare commits

...

1342 Commits

Author SHA1 Message Date
Dan Helfman 3561c93d74 Fix Healthchecks tests that leak global state, breaking downstream tests (discovered in #543). 2022-06-09 11:05:44 -07:00
Dan Helfman 331a503a25 Document the borgmatic version in which "borgmatic list --find" is available (#541). 2022-06-03 16:55:54 -07:00
Dan Helfman 9aefb5179f Fix None find paths (#541). 2022-06-03 15:20:05 -07:00
Dan Helfman d14f22e121 Add "borgmatic list --find" flag for searching for files across multiple archives (#541). 2022-06-03 15:12:14 -07:00
Dan Helfman b6893f6455 Exclude deprecated "borg list --successful" flag from getting passed to Borg. 2022-06-02 21:14:25 -07:00
Dan Helfman 80ec3e7d97 Deprecate "borgmatic list --successful" flag, as listing only non-checkpoint (successful) archives is now the default in newer versions of Borg. 2022-06-02 20:35:39 -07:00
Dan Helfman cd834311eb Clarify completion docs. 2022-06-01 10:57:23 -07:00
Dan Helfman d751cceeb0 Merge branch 'master' of ssh://projects.torsion.org:3022/borgmatic-collective/borgmatic 2022-06-01 10:38:05 -07:00
Dan Helfman ce78b07e4b Add macOs to install and Bash completion documentation.
Reviewed-on: borgmatic-collective/borgmatic#540
2022-06-01 17:37:51 +00:00
adidalal 87f3c50931 setup: add macOS 2022-06-01 15:56:40 +00:00
Dan Helfman 8e9e06afe6 Bump version for release. 2022-05-31 09:41:20 -07:00
Dan Helfman 2bc91ac3d2 Add "generate-borgmatic-config --overwrite" flag to replace an existing destination file (#539). 2022-05-29 16:03:55 -07:00
Dan Helfman 5b615d51a4 Add support for "borgmatic borg debug" command (#538). 2022-05-29 15:43:03 -07:00
Dan Helfman c7f5d5fd0b Fix broken Bash completion of filenames, as in "-c config.yaml". 2022-05-29 10:49:33 -07:00
Dan Helfman 6ef7538eb0 Fix typo in Bash completions script. 2022-05-28 19:34:13 -07:00
Dan Helfman 8fa90053cf Add "borgmatic check --force" flag to ignore configured check frequencies (#523). 2022-05-28 19:29:33 -07:00
Dan Helfman b3682b61d1 Add another note about the consistency checks schema in old versions (#523). 2022-05-28 19:03:45 -07:00
Dan Helfman ad0e2e0d7c Tweak default check frequency to 1 month (#523). 2022-05-28 15:49:50 -07:00
Dan Helfman 6629f40cab In bash completion script, warn when script is out of date using script contents instead of version. (Fewer spurious warnings that way.) 2022-05-28 15:27:11 -07:00
Dan Helfman e76bfa555f Reduce the default consistency check frequency and support configuring the frequency independently for each check (#523). 2022-05-28 14:42:19 -07:00
Dan Helfman 8ddb7268eb Reuse "borg info" function. 2022-05-27 13:51:11 -07:00
Dan Helfman cb5fe02ebd Fix broken Bash completion end-to-end test. 2022-05-26 11:18:46 -07:00
Dan Helfman 77b84f8a48 Add Bash completion script so you can tab-complete the borgmatic command-line. 2022-05-26 10:27:53 -07:00
Dan Helfman 691ec96909 Fix python_requires to support all versions of 3.7 (#537).
Reviewed-on: borgmatic-collective/borgmatic#537
2022-05-26 15:51:46 +00:00
Steve Atwell 29b4666205 Fix python_requires to support all versions of 3.7
This is the standard way to support "Python 3.7 and newer" and it also
fixes use of borgmatic with some tools that do custom dependency
resolution.  E.g., using pex with --platform.
2022-05-26 07:05:04 -07:00
Dan Helfman 316a22701f Add documentation note about multiple merge limitation (#380). 2022-05-25 23:12:42 -07:00
Dan Helfman be59a3e574 Fix generate-borgmatic-config with "--source" flag to support more complex schema changes like the new Healthchecks configuration options (#536). 2022-05-25 10:26:26 -07:00
Dan Helfman 37327379bc Merge branch 'master' of ssh://projects.torsion.org:3022/borgmatic-collective/borgmatic 2022-05-24 17:50:57 -07:00
Dan Helfman 22c2f13611 Remove trailing whitespace (#535).
Reviewed-on: borgmatic-collective/borgmatic#535
2022-05-25 00:50:12 +00:00
polyzen 8708ca07f4 Remove trailing whitespace 2022-05-25 00:43:40 +00:00
Dan Helfman 634d9e4946 Bump version for release. 2022-05-24 16:22:37 -07:00
Dan Helfman 54933ebef5 Change connection failures for monitoring hooks to be warnings instead of errors (#439). 2022-05-24 15:50:04 -07:00
Dan Helfman 157e59ac88 Add Healthchecks monitoring hook "send_logs" option to enable/disable sending borgmatic logs to the Healthchecks server (#460). 2022-05-24 14:44:33 -07:00
Dan Helfman 666f0dd751 Add missing Healthchecks "states" option example in configuration schema (#525). 2022-05-24 14:17:19 -07:00
Dan Helfman 8b179e4647 Reverse logic of Healtchecks "skip_states" option to just "states" (#525). 2022-05-24 14:09:42 -07:00
Dan Helfman 865eff7d98 Add Healthchecks monitoring hook "skip_states" option to disable pinging for particular monitoring states (#525). 2022-05-24 13:59:28 -07:00
Dan Helfman b9741f4d0b Add Healthchecks monitoring hook "ping_body_limit" option to configure how many bytes of logs to send to the Healthchecks server (#294). 2022-05-24 12:23:38 -07:00
Dan Helfman 02781662f8 Change monitoring hooks to specify the ping URL / integration key as a named option. 2022-05-23 20:02:10 -07:00
Dan Helfman 32a1043468 Remove the error when "archive_name_format" is specified but a retention prefix isn't (#402). 2022-05-23 16:11:24 -07:00
Dan Helfman 3e4aeec649 Warn when an unsupported variable is used in a hook command (#420). 2022-05-23 15:27:54 -07:00
Dan Helfman b98b827594 Remove stale comment. 2022-05-23 10:59:56 -07:00
Dan Helfman 255cc6ec23 When deep merging common configuration, merge colliding list values by appending them (#531). 2022-05-20 15:28:28 -07:00
Dan Helfman 51fc37d57a Improve the error message when a configuration override contains an invalid value (#528). 2022-05-20 13:38:53 -07:00
Dan Helfman 1921f55a9d Add emojis to documentation table of contents to make it easier to find particular how-to and reference guides at a glance. 2022-05-20 11:11:35 -07:00
Dan Helfman fbd381fcc1 Clarify manual database extraction documentation. 2022-05-20 10:06:19 -07:00
Dan Helfman cd88f9f2ea Better explain where to find the dump file when doing a manual restore (#510).
Reviewed-on: borgmatic-collective/borgmatic#510
2022-05-20 16:33:21 +00:00
Dan Helfman 788281cfb9 When a configuration include is a relative path, load it from either the current working directory or from the directory containing the file doing the including (#532). 2022-05-19 17:15:05 -07:00
Dan Helfman cd234b689d Link to additional borgmatic Docker image. 2022-05-12 12:00:12 -07:00
Dan Helfman 92354a77ee Mention that database dumps consumed disk space prior to borgmatic 1.5.3. 2022-05-09 16:08:47 -07:00
Dan Helfman 48ff3e70d1 Clarify documentation about include merging mappings vs. values. 2022-05-08 14:48:42 -07:00
Dan Helfman 7e9adfb899 Add NEWS entry for randomized systemd timer delay. 2022-05-07 23:11:26 -07:00
Dan Helfman e238e256f7
Add randomized delay to systemd timer.
Merge pull request from Daniel15/patch-1
2022-05-07 23:08:02 -07:00
Daniel Lo Nigro 3ecb92a8d2
Add randomized delay to systemd timer 2022-05-07 16:42:06 -07:00
Dan Helfman d58d450628 Remove stale borgmatic binary link. 2022-04-30 09:50:40 -07:00
Dan Helfman dee9c6e293 Remove link to stale borgmatic Docker image. 2022-04-30 09:46:08 -07:00
Dan Helfman 897c4487de Add mention in documentation about multiple backup scheduling needs (#511). 2022-04-28 11:16:31 -07:00
Dan Helfman 48b50b5209 Add documentation link to NEWS. 2022-04-26 10:24:25 -07:00
Dan Helfman 13bae8c23b Typo. 2022-04-26 10:12:02 -07:00
Dan Helfman 4a48e6aa04 Bump version for release. 2022-04-26 10:07:04 -07:00
Dan Helfman 525266ede6 Deep merging when including common configuration (#381). 2022-04-25 21:18:37 -07:00
Dan Helfman d045eb55ac Add mention of sudo's "secure_path" option in borgmatic installation documentation (#513). 2022-04-23 14:29:55 -07:00
Dan Helfman 0e6b425ac5 Fix "borgmatic borg key ..." to pass parameters to Borg in correct order (#515). 2022-04-23 14:03:15 -07:00
Dan Helfman bdc26f2117 Add note about old, pre-1.6.0 hooks behavior. 2022-04-22 19:58:28 -07:00
Dan Helfman ed7fe5c6d0 Instead of executing "before" command hooks before all borgmatic actions run (and "after" hooks after), execute these hooks right before/after the corresponding action (#473). 2022-04-21 22:08:25 -07:00
Dan Helfman cbce6707f4 Clarify one_file_system behavior in schema comment (#520). 2022-04-12 11:05:22 -07:00
Dan Helfman e40e726687 Change Healthchecks logs truncation size from 10k bytes to 100k bytes, corresponding to that same change on Healthchecks.io. 2022-04-06 22:00:18 -07:00
Dan Helfman 0c027a3050 Fix handling of TERM signal to exit borgmatic, not just forward the signal to Borg (#516). 2022-04-03 13:12:48 -07:00
Dan Helfman 9f44bbad65 Fix borgmatic exit code (so it's zero) when initial Borg calls fail but later retries succeed (#517). 2022-04-02 22:28:41 -07:00
Dan Helfman 413a079f51 Clarify Python version support. 2022-03-28 21:57:40 -07:00
gerdneuman 6f3accf691 Better explain where to find the dump file
I really had problem finding the dump file with the explanation as give before. I thought that the `~/.borgmatic/` would be my current user. So looked into `/home/gerd/.borgmatic` (wrong). Then I looked into `<EXTRACTED_DESTINATION_PATH/.borgmatic` (again wrong). Then finally (1h later and after having already prepared a bug ticketI figured out that the dump file is within `<EXTRACTED_DESTINATION_PATH/root/.borgmatic`. Hard to find because of course I d not only have `root` within `<EXTRACTED_DESTINATION_PATH/` but also all other backup'ed directories (including /etc/, /home/ on so on...)
2022-03-17 04:51:47 +00:00
Dan Helfman 5b3cfc542d Switch to PyPI API token. 2022-03-14 14:00:03 -07:00
Dan Helfman c838c1d11b Fix header placement in documentation guide. 2022-03-14 13:50:22 -07:00
Dan Helfman 4d1d8d7409 Bump version for release. 2022-03-14 13:43:24 -07:00
Dan Helfman db7499db82 Document "repositories" context to for "before_*" and "after_*" command action hooks (#469). 2022-03-14 13:34:14 -07:00
Dan Helfman 6b500c2a8b Add repositories context for command hooks.
Reviewed-on: borgmatic-collective/borgmatic#469
2022-03-14 20:13:15 +00:00
Dan Helfman 95c518e59b Documentation tip about dealing with hangs when database hook is enabled. 2022-03-12 13:17:32 -08:00
Dan Helfman 976516d0e1 When loading a configuration file that is unreadable due to file permissions, warn instead of erroring (#444). 2022-03-08 10:19:36 -08:00
Dan Helfman 574eb91921 Fix Borg usage error in the "compact" action when running "borgmatic --dry-run". Now, skip "compact" entirely during a dry run (#507). 2022-03-07 21:46:12 -08:00
Dan Helfman 28fef3264b Fix handling of "patterns_from" and "exclude_from" options to error instead of warning when referencing unreadable files and running "create" action (#486). 2022-03-07 15:32:07 -08:00
Dan Helfman 9161dbcb7d Removing unnecessary leading underscores from functions. 2022-03-07 11:58:29 -08:00
Dan Helfman 4b3027e4fc Add test for new working_directory option (#431). 2022-03-03 11:48:18 -08:00
Dan Helfman 0eb2634f9b Working directory option to support source directories with relative paths (#431).
Reviewed-on: borgmatic-collective/borgmatic#477
2022-03-03 19:28:17 +00:00
Dan Helfman 7c5b68c98f Bump version for release. 2022-02-10 10:29:18 -08:00
Dan Helfman 9317cbaaf0 Code formatting. 2022-02-10 10:23:34 -08:00
Dan Helfman 1b5f04b79f When using the "remote_rate_limit" option, tailor the flags passed to Borg depending on the Borg version (#394). 2022-02-10 10:16:09 -08:00
Dan Helfman 948c86f62c When using the "numeric_owner" option with the "extract" action, tailor the flags passed to Borg depending on the Borg version (#394). 2022-02-10 10:09:18 -08:00
Dan Helfman 7e7209322a When using the "numeric_owner" option, tailor the flags passed to Borg depending on the Borg version (#394). 2022-02-10 09:51:13 -08:00
Dan Helfman 00a57fd947 Code formatting. 2022-02-09 21:20:28 -08:00
Dan Helfman 6bf6ac310b When using the "bsd_flags" option, tailor the flags passed to Borg depending on the Borg version (#394). 2022-02-09 21:11:00 -08:00
Dan Helfman 4b5af2770d When the "atime" option is used, tailor the flags passed to Borg depending on version (#394). 2022-02-09 16:54:35 -08:00
Dan Helfman b525e70e1c Run "compact" action by default when no actions are specified (#394). 2022-02-09 14:33:12 -08:00
Dan Helfman 4498671233 Remove references to removed long-deprecated options (#394). 2022-02-09 11:08:02 -08:00
Dan Helfman 9997aa9a92 Fix capitalization on compact help. 2022-02-08 15:58:09 -08:00
Dan Helfman cbf7284f64 Add compact action to command-line reference documentation. 2022-02-08 15:37:24 -08:00
Dan Helfman ee466f870d Fixing ruamel.yaml.clib breakages harder. 2022-02-08 13:21:11 -08:00
Dan Helfman e3f4bf0293 Build fix for ruamel.yaml.clib error. 2022-02-08 12:52:45 -08:00
Dan Helfman 46688f10b1 Merge branch 'master' of ssh://projects.torsion.org:3022/borgmatic-collective/borgmatic 2022-02-08 12:10:57 -08:00
Dan Helfman 48f44d2f3d Add tests for compact action (#394). 2022-02-08 12:05:02 -08:00
Dan Helfman bff1347ba3 Fix some test failures (#394). 2022-02-08 09:35:03 -08:00
Dan Helfman 9582324c88 Compact repository segments with new "borgmatic compact" action (#394). 2022-02-07 23:29:44 -08:00
Dan Helfman bb0716421d Add comment about systemd service setting that may interfere with external commands in hooks (#492). 2022-01-25 09:26:11 -08:00
Dan Helfman bec73245e9 Fix traceback when a YAML validation error occurs (#480, #482). 2022-01-19 20:39:03 -08:00
Dan Helfman dcead12e86 Attempt to fix documentation build error introduced by Eleventy upgrade. 2022-01-09 14:21:27 -08:00
Dan Helfman 0119514c11 Add Python version requirements to setup.py. 2022-01-09 10:19:53 -08:00
fabianschilling b39f08694d Merge branch 'master' into pr-working-directory 2022-01-05 09:30:27 +00:00
Dan Helfman 80bdf1430b Bump version for release. 2022-01-04 20:20:13 -08:00
Dan Helfman 2ee75546f5 Add MongoDB database hook documentation. 2022-01-04 16:26:38 -08:00
Dan Helfman 07d7ae60d5 Add MongoDB database hook (#288).
Reviewed-on: borgmatic-collective/borgmatic#483
2022-01-04 23:50:25 +00:00
Andrea Ghensi 87001337b4 Merge master into mongodb_hook 2022-01-04 22:20:44 +01:00
Dan Helfman 2e9964c200 Remove references to Lima Labs (shut down their storage business).
Reviewed-on: borgmatic-collective/borgmatic#488
2022-01-03 17:34:38 +00:00
Ian Kerins 3ec3d8d045 Remove references to Lima Labs
From their homepage:
> Lima Labs is shutting down our storage business. We will try to keep data available as long as possible. No promises but we are targeting 3/1/2022 to bring down Archive and Canada.
2022-01-03 02:29:38 -05:00
Dan Helfman 96384d5ee1 Attempt to fix typed-ast build issue by relaxing version requirements in test. 2022-01-02 23:22:24 -08:00
Dan Helfman 8ed5467435 Drop support for Python 3.6. Add support for 3.10. 2022-01-02 23:17:57 -08:00
Andrea Ghensi 7c6ce9399c fix integration tests and mongodb auth 2021-12-29 22:18:50 +01:00
Andrea Ghensi 6b7653484b Add mongodb dump hook 2021-12-26 01:00:58 +01:00
Fabian Schilling 85e0334826 Add missing working_directory arg to pass tests 2021-12-10 18:24:41 +01:00
Fabian Schilling 2a80e48a92 Pass working directory to execute functions 2021-12-10 18:23:44 +01:00
Fabian Schilling 5821c6782e Add defaults to not set in schema 2021-12-10 18:23:08 +01:00
Fabian Schilling f15498f6d9 Add working_directory to borgmatic schema 2021-12-10 17:58:27 +01:00
Dan Helfman a1673d1fa1 Fix unicode error when restoring particular MySQL databases (#476). 2021-12-08 16:40:25 -08:00
Dan Helfman 2e99a1898c Fix f-string with missing expression. 2021-11-29 14:05:36 -08:00
Dan Helfman 7a086d8430 Fix import ordering. 2021-11-29 14:00:14 -08:00
Dan Helfman 0e8e9ced64 When command-line configuration override produces a parse error, error cleanly (#471). 2021-11-29 12:49:21 -08:00
Dan Helfman f34951c088 Add MySQL dump command adjustment to NEWS. 2021-11-29 12:10:04 -08:00
Dan Helfman c6f47d4d56 Move mysqldump options to the beginning of the command due to MySQL bug 30994 (#470).
Reviewed-on: borgmatic-collective/borgmatic#470
2021-11-29 20:08:59 +00:00
nebulon42 c3e76585fc
move mysqldump options to the beginning of the command due to MySQL bug 30994. 2021-11-26 17:16:03 +01:00
Chen Yufei 0014b149f8 remove configuration_filename as it's already set. 2021-11-26 11:38:58 +08:00
Chen Yufei 091c07bbe2 Add context for various hooks. 2021-11-26 11:35:10 +08:00
Dan Helfman 240547102f Enable auto-play on linked asciicast. 2021-11-25 13:09:55 -08:00
Dan Helfman 2bbd53e25a
Merge pull request #43 from acsfer/patch-1
Github doesn't allow script embedding
2021-11-25 13:06:43 -08:00
acsfer 58f2f63977
Switch to HTML 2021-11-25 22:03:26 +01:00
acsfer 7df6a78c30
Github doesn't allow script embedding 2021-11-25 21:36:31 +01:00
Dan Helfman c646edf2c7 Bump version for release. 2021-11-22 13:19:15 -08:00
Dan Helfman bcc820d646 Add list_options setting (#306).
Reviewed-on: borgmatic-collective/borgmatic#464
2021-11-22 21:14:02 +00:00
nebulon42 3729ba5ca3
add list_options setting, fixes #306 2021-11-20 15:43:58 +01:00
Dan Helfman 9c19591768 Revise hosting provider links. 2021-11-15 20:06:09 -08:00
Dan Helfman 38ebfd2969 Rename retry_timeout to retry_wait and standardize log formatting (#28). 2021-11-15 11:51:17 -08:00
Dan Helfman 180018fd81 Retry failing backups (#28, #432).
Reviewed-on: borgmatic-collective/borgmatic#432
2021-11-15 19:34:24 +00:00
Dan Helfman 794ae94ac4 Attempt to limit documentation pushing to commits (so, not pull requests). 2021-11-15 11:08:26 -08:00
Dan Helfman 4eb6359ed3 Remove now-unneeded build image workaround. 2021-11-15 10:56:12 -08:00
cadamswaite 976a877a25 Formatting 2021-11-14 22:37:42 +00:00
cadamswaite b4117916b8 Add timeout and tests 2021-11-14 22:15:22 +00:00
cadamswaite 19cad89978 Add some tests for retry logic 2021-11-14 21:35:23 +00:00
cadamswaite 6b182c9d2d Merge branch 'master' into master 2021-11-14 18:24:17 +00:00
Dan Helfman 4d6ed27f73 Add to changelog: Add support for old version (2.x) of jsonschema library. 2021-10-23 09:49:16 -07:00
Dan Helfman 745a8f9b8a Add support for both jsonschema v3 and old v2 (#459).
Reviewed-on: borgmatic-collective/borgmatic#459
2021-10-23 16:47:53 +00:00
Dan Helfman 6299d8115d Limit documentation build to master of main repo, as it pushes a Docker image. 2021-10-23 09:45:17 -07:00
Kim B. Heino 717cfd2d37 validate: add support for both jsonschema v3 and old v2
RHEL8 and RHEL7 have old jsonschema v2. Try v3 (Draft7) first but
fallback to v2 (Draft4) if needed.
2021-10-23 15:04:07 +03:00
Dan Helfman 7881327004 Upgrade CI test dependencies. 2021-10-22 14:07:14 -07:00
Dan Helfman 549aa9a25f Update editable link. 2021-10-22 14:06:27 -07:00
Dan Helfman 1c6890492b Bump version for release. 2021-10-11 17:02:32 -07:00
Dan Helfman a7c8e7c823 Bump version for release. 2021-10-11 11:13:32 -07:00
Dan Helfman c8fcf6b336 Mention changing borgmatic path in cron documentation (#455). 2021-10-11 11:02:08 -07:00
Dan Helfman 449896f661 Fix error when configured source directories are not present on the filesystem at the time of backup (#387). 2021-10-11 10:40:10 -07:00
Dan Helfman 1004500d65 Update sample systemd service file comments about more granular read-only filesystem settings. 2021-10-11 09:33:07 -07:00
Dan Helfman 0a8d4e5dfb
Add more strict ProtectHome to systemd sample configuration.
Merge pull request #42 from VTimofeenko/systemd_protecthome
2021-10-11 09:26:28 -07:00
Dan Helfman 38e35bdb12 Skip TLS verify in documentation build clone to work around old drone/git CA certs. 2021-10-04 14:31:15 -07:00
Dan Helfman 65503e38b6 Sigh. 2021-10-04 13:14:19 -07:00
Dan Helfman d0c5bf6f6f Another attempt to unbreak build. 2021-10-04 13:13:35 -07:00
Dan Helfman f129e4c301 Attempt to work-around outdated CA certificates in drone/git Docker image. 2021-10-04 13:09:44 -07:00
Dan Helfman fbbb096cec Note in documentation that borgmatic requires Python 3.6+. 2021-10-04 11:15:51 -07:00
Dan Helfman 77980511c6 Add another glob pattern example to exclude patterns. 2021-09-16 09:51:40 -07:00
Dan Helfman 4ba206f8f4 Update build server URL to new organization namespace. 2021-09-14 11:35:34 -07:00
Dan Helfman ecc849dd07 Move Gitea hosting from a personal namespace to an organization. 2021-09-14 11:32:01 -07:00
Dan Helfman 7ff6066d47 Move GitHub hosting from a personal namespace to an organization. 2021-09-14 10:18:10 -07:00
Dan Helfman 2bb1fc9826 Mention Docker Compose under installation options. 2021-09-12 13:15:34 -07:00
Vladimir Timofeenko 6df6176f3a
Added more strict ProtectHome to systemd unit
This commit changes the comment in sample systemd service.

Using a combination of 'ProtectHome' and 'BindPaths' it's possible to
hide the irrelevant paths inside /root from borgmatic service when it is
run.

ReadWritePaths are suggested to be used only for paths that contain borg
repositories and the backup sources can be specified as ReadOnlyPaths.
2021-08-30 11:20:34 -07:00
Dan Helfman acb2ca79d9 Fix traceback that can occur when dumping a database (#440). 2021-08-06 08:58:11 -07:00
Dan Helfman c9211320e1 Fix dev version in changelog. 2021-08-04 15:32:51 -07:00
Dan Helfman 760286abe1 Dev release bump. 2021-07-30 09:49:07 -07:00
Dan Helfman 5890a1cb48 Fix "message too long" error when logging to rsyslog (#389). 2021-07-30 09:48:13 -07:00
Dan Helfman b3f5a9d18f Fix error when configuration file contains "umask" option (#437). 2021-07-27 10:04:22 -07:00
Dan Helfman 80b33fbf8a Code style reformatting. 2021-07-27 09:39:48 -07:00
Dan Helfman 5389ff6160
Merge pull request #41 from mkszuba/tests_no_xxd
tests/integration/test_execute: use plain Python rather than xxd
2021-07-27 09:39:02 -07:00
Marek Szuba e8b8d86592 tests/integration/test_execute: use plain Python rather than xxd
Removes this test's dependencies on vim and /dev/urandom.

Signed-off-by: Marek Szuba <marek.szuba@cern.ch>
2021-07-27 13:50:16 +01:00
Dan Helfman 92d729a9dd Try temporary work around for Drone build bug: https://github.com/drone-plugins/drone-docker/pull/327 2021-07-26 16:33:41 -07:00
Dan Helfman c63219936e Wording tweaks to security policy. 2021-07-26 13:44:14 -07:00
Dan Helfman 0aff497430 Bump version for release. 2021-07-26 10:17:49 -07:00
Dan Helfman 1f3907a6a5 Fix for failing PostgreSQL directory format test (#430). 2021-07-26 09:42:14 -07:00
Dan Helfman 2a8692c64f Fix integration test to hopefully work on Alpine (#430). 2021-07-25 22:50:00 -07:00
Dan Helfman 1709f57ff0 Fix hang when restoring a PostgreSQL "tar" format database dump (#430). 2021-07-25 22:30:15 -07:00
cadamswaite 89baf757cf Sort imports 2021-07-14 23:17:35 +01:00
cadamswaite 4f36fe2b9f Run Black on changed file 2021-07-14 22:53:01 +01:00
cadamswaite 510449ce65 Change default retries to 0 2021-07-14 22:49:03 +01:00
cadamswaite 4cc4b8d484 Add queue based retry logic 2021-07-14 22:46:02 +01:00
Dan Helfman 9c972cb0e5 Add documentation note about systemd configuration with alternate install methods (#428). 2021-06-29 21:38:53 -07:00
Dan Helfman 9b1779065e Pin ruamel.yaml.clib to work around docs build issue. 2021-06-29 21:35:46 -07:00
Dan Helfman 057ec3e59b Add NEWS entry for #379: Suppress console output in sample crontab and systemd service files. 2021-06-23 10:35:41 -07:00
Dan Helfman bc2e611a74 Suppress console output in sample crontab/systemd service files (#379).
Reviewed-on: witten/borgmatic#379
2021-06-23 17:32:47 +00:00
Dan Helfman b6d3a1e02f Merge branch 'master' of ssh://projects.torsion.org:3022/witten/borgmatic 2021-06-23 10:22:07 -07:00
Dan Helfman 54d57e1349 Add test for #407: Fix syslog logging on FreeBSD. 2021-06-23 10:21:45 -07:00
Dan Helfman af0b3da8ed Fix syslog logging on FreeBSD (#407).
Reviewed-on: witten/borgmatic#407
2021-06-23 17:21:25 +00:00
Dan Helfman 27d37b606b Better error messages! Switch the library used for validating configuration files (from pykwalify to jsonschema). 2021-06-22 13:27:59 -07:00
Dan Helfman 77a860cc62 Link borgmatic Ansible role from installation documentation. 2021-06-19 19:04:22 -07:00
Dan Helfman 7bd6374751 Bump version for release. 2021-06-17 20:44:54 -07:00
Dan Helfman cf8882f2bc Run arbitrary Borg commands with new "borgmatic borg" action (#425). 2021-06-17 20:41:44 -07:00
Dan Helfman b37dd1a79e Document use case of running backups conditionally based on laptop power level (#419). 2021-06-09 10:03:35 -07:00
Dan Helfman fd59776f91 Bump version for release. 2021-06-08 11:44:53 -07:00
Dan Helfman 9fd28d2eed Fix error handling to error loudly when Borg gets killed due to running out of memory (#423)! 2021-06-08 11:43:55 -07:00
Dan Helfman f5c61c8013 Move #borgmatic IRC channel from Freenode to Libera Chat due to Freenode takeover drama. 2021-06-06 21:09:40 -07:00
Dan Helfman 88cb49dcc4 Fix release script based on GitHub authentication query parameter deprecation. 2021-04-24 20:27:53 -07:00
Dan Helfman 73235e59be Upgrade "py" test dependency (security). 2021-04-20 10:39:49 -07:00
Dan Helfman 7076a7ff86 Add link to Hetzner storage offering from the documentation (#390). 2021-04-18 18:03:43 -07:00
Dan Helfman d6e376d32d Fix end-to-end test broken by change in source directory examples. 2021-04-18 17:54:54 -07:00
Dan Helfman 9016f4be43 Clarify that spaces in path names should not be backslashed in path names (#406). 2021-04-18 17:28:11 -07:00
Jeffery To d1c403999f
Reduce console output in sample crontab/systemd service files.
As borgmatic will log to syslog in the sample crontab/systemd service
files, this makes console output redundant. (cron will mail any console
output to the root user; systemd will log any console output to syslog.)

This adds --verbosity -1 to both files to reduce console output to the
minimum.
2021-04-13 01:40:57 +08:00
Dan Helfman d543109ef4 "Fix" build failure with Alpine Edge by switching from Edge to Alpine 3.13. 2021-04-09 15:58:23 -07:00
Dan Helfman 7085a45649 Fix build so as not to attempt to build and push documentation for a non-master branch. 2021-04-09 15:04:09 -07:00
Dan Helfman cf4c603f1d Clarify canonical home of borgmatic in documentation (#398). 2021-04-09 14:54:21 -07:00
Victor Bouvier-Deleau d2533313bc
Fix syslog logging on FreeBSD
The UNIX domain socket to use on FreeBSD is /var/run/log.
See syslogd FreeBSD man page: https://www.freebsd.org/cgi/man.cgi?query=syslogd&sektion=8
2021-04-02 14:11:50 +02:00
Dan Helfman c43b50b6e6 Upgrade PyYAML. 2021-03-30 22:29:20 -07:00
Dan Helfman c072678936 Add support for ruamel.yaml 0.17.x YAML parsing library (#404). 2021-03-30 15:53:19 -07:00
Dan Helfman 631da1465e Add support for Python 3.9. 2021-03-30 15:36:26 -07:00
Dan Helfman f29519a5cd
Merge pull request #38 from lukehsiao/patch-1
Fix link to issue tracker in documentation
2021-03-20 15:45:15 -07:00
Luke Hsiao 5d82b42ab8
Fix link to issue tracker in documentation
Fixes: a1d986d952
2021-03-18 17:26:37 -07:00
Dan Helfman 4897a78fd3 Fix database tests broken by PostgreSQL upgrade in Alpine Edge. 2020-12-24 22:23:09 -08:00
Dan Helfman a1d986d952 Replace "improve this documentation" form with link to support and ticket tracker. 2020-12-24 14:57:51 -08:00
Dan Helfman 717c90a7d0 Clarify in systemd service file comment that security settings are optional. 2020-12-09 10:08:07 -08:00
Dan Helfman 8fde19a7dc Update systemd service example to return a permission error when a system call isn't permitted. 2020-11-30 22:14:28 -08:00
Dan Helfman ad7198ba66 Tweak to test failing on some machines. 2020-11-26 16:22:42 -08:00
Dan Helfman eb4b4cc92b Fix line length in schema. 2020-11-25 19:21:06 -08:00
Dan Helfman 41bf520585 Document that passphrase is used for Borg keyfile encryption, not just repokey encryption (#373). 2020-11-25 18:36:23 -08:00
Dan Helfman c0ae01f5d5 Code formatting. 2020-11-25 17:46:57 -08:00
Dan Helfman 8b8f92d717 Prevent newer (borgmatic-unsupported) version of Black code formatter installing in Alpine Edge. 2020-11-25 17:42:04 -08:00
Dan Helfman ccd1627175 Fix timing-related test error in Alpine Edge. 2020-11-25 15:48:33 -08:00
Dan Helfman b8a7e23f46 Add missing pip to test script. 2020-11-22 17:42:58 -08:00
Dan Helfman 1f4f28b4dc Drop support for Python 3.5. Only support black code formatter on Python 3.8+. 2020-11-22 17:27:21 -08:00
Dan Helfman ea6cd53067 Update versions of test dependencies (test_requirements.txt and test containers). 2020-11-22 14:48:07 -08:00
Dan Helfman 267138776d Add protection for accidentally releasing a dev version. 2020-11-21 14:03:39 -08:00
Dan Helfman 604b3d5e17 Bump version. 2020-11-21 13:56:19 -08:00
Dan Helfman 667e1e5b15 Update document about new --override behavior (#361). 2020-11-19 11:01:53 -08:00
Dan Helfman 9b819f32f8 Fix traceback when upgrading old INI-style configuration with upgrade-borgmatic-config (#367). 2020-11-18 15:01:10 -08:00
Dan Helfman b619bde037 Fix broken end-to-end test. 2020-11-18 14:20:07 -08:00
Dan Helfman 97af16bd86 Add Lima-Labs to examples. 2020-11-18 12:05:38 -08:00
Dan Helfman fa75f89acc Merge ssh://projects.torsion.org:3022/witten/borgmatic 2020-11-18 11:54:44 -08:00
Dan Helfman 222b61b577 Update changelog with note about #369. 2020-11-18 11:26:20 -08:00
Dan Helfman e77757f0fd Mention placeholders for repositories in documentation; fix #369
Reviewed-on: witten/borgmatic#371
2020-11-18 19:24:59 +00:00
François Poulain ebac02f118 Mention placeholders for repositories in documentation; fix #369 2020-11-18 20:22:41 +01:00
Dan Helfman 1c9ae81987 Fix signal forwarding from borgmatic to Borg resulting in recursion traceback (#368). 2020-11-18 10:29:03 -08:00
Dan Helfman 7b1fb68c18 Add referral link to Lima-Labs. 2020-11-17 14:04:24 -08:00
Dan Helfman 8aa7830f0d Fix broken "--override" action in Python 3.7 and below. 2020-11-15 13:39:15 -08:00
Dan Helfman 79bee755ee
Merge pull request #37 from ThorpeJosh/patch-1
Fix typo in large backup docs.
2020-11-03 22:22:12 -08:00
Josh Thorpe cde0ee96ff
Fix typo in large backup docs 2020-11-04 17:20:32 +11:00
Dan Helfman 1ea04aedf0 Merge override values when specifying the "--override" flag multiple times (#361). 2020-10-23 16:06:00 -07:00
Dan Helfman 446a2bc15a
Fixed minor typo in help for "info" command. 2020-10-11 15:08:43 -07:00
Diego Blanco 2d10e758e0 Fixed minor typo in help for "info" command 2020-10-12 00:04:15 +02:00
Dan Helfman 0e978299cf Fix traceback when a database hook value is null in a configuration file (#355). 2020-10-07 15:41:45 -07:00
Dan Helfman d06c1f2943 Minor edit. 2020-09-21 10:00:38 -07:00
Dan Helfman d768b50b97 Add note about empty source_directories.
Reviewed-on: witten/borgmatic#358
2020-09-21 16:59:18 +00:00
Luke Murphy 034ade48f2
Add note about empty source_directories 2020-09-21 17:35:26 +02:00
Dan Helfman d1e9f74087 Restore referal link. 2020-08-27 15:21:53 -07:00
Dan Helfman f262f77dbd Add temporary_directory option to changelog. 2020-08-26 09:51:55 -07:00
Dan Helfman a3387953a9 Add configuration option for settings Borg's temporary directory (#341).
Reviewed-on: witten/borgmatic#356
2020-08-26 16:54:49 +00:00
root 7cad5a8608 Add `temporary_directory` as a configuration option defaulting to $TMPDIR 2020-08-26 15:29:02 +00:00
Dan Helfman 9b83fcbf06 Add comment about MemoryDenyWriteExecute value and the tradeoffs thereof. 2020-08-23 14:11:19 -07:00
Dan Helfman 32a93ce8a2 Loosen systemd memory security setting to allow Healthchecks ping. 2020-08-22 13:37:34 -07:00
Dan Helfman e428329c03 Add systemd security improvement to NEWS. 2020-08-22 09:49:06 -07:00
Dan Helfman e844bbee15 Improve systemd security settings (#352). 2020-08-22 16:50:06 +00:00
Matthias 631c3068a9 systemd security settings 2020-08-22 15:41:25 +02:00
Dan Helfman 79d4888e22 Add documentation navigation links on left side of all documentation pages. 2020-08-21 14:27:47 -07:00
Dan Helfman de61fdef48 Mention "before_extract"/"after_extract" in README. 2020-08-12 14:12:58 -07:00
Dan Helfman 93caeba200 Add before_extract and after_extract hooks (#347).
Reviewed-on: witten/borgmatic#347
2020-08-12 21:14:15 +00:00
networkjanitor 3c723e8d99 Merge branch 'master' into master 2020-08-12 21:11:45 +00:00
networkjanitor c5776447b9 fixed description strings for before_extract and after_extract 2020-08-12 23:07:57 +02:00
Dan Helfman 5356f487a5 Move before/after_everything hooks so they're easier to find. 2020-08-12 13:46:31 -07:00
Dan Helfman 72bd96c656 Fix traceback when a configuration directory is non-readable due to directory permissions (#350). 2020-08-12 11:32:00 -07:00
Dan Helfman f611fe7be3 Clarify documentation overview of monitoring options. 2020-08-12 10:40:39 -07:00
Dan Helfman dd6ea40a36 Clarify encryption section: It's not just for cron users. 2020-08-10 14:43:03 -07:00
Dan Helfman ea1274d1c6 Remove inactive referral link. 2020-08-07 10:55:05 -07:00
Dan Helfman 8526468975
Fix configuration schema comment on after_everything option. 2020-08-07 09:49:25 -07:00
Jakub Duchateau 95c415f416
Change "before" to "after" in desc 2020-08-07 16:41:35 +02:00
Dan Helfman 06dc336481 Fix schema exclude patterns example path. 2020-08-06 09:59:48 -07:00
networkjanitor 893fca2816 Add before_extract and after_extract hooks 2020-08-01 16:08:32 +02:00
Dan Helfman 99590cb6b6 Clarify documentation on configuration overrides, specifically the portion about list syntax. 2020-07-23 21:33:42 -07:00
Dan Helfman b3fd1be5f6 Add "borgmatic export-tar" action to export an archive to a tar-formatted file or stream (#300). 2020-07-22 21:53:55 -07:00
Dan Helfman a23083f737 Clarify schema documentation about skipping pruning. 2020-07-22 10:35:16 -07:00
Dan Helfman 8306b758e8 Add documentation on how to make backups redundant with multiple repositories. 2020-07-17 16:00:50 -07:00
Dan Helfman 218cbd5289 In database documentation, include command for finding named pipes and special devices. 2020-07-13 10:56:59 -07:00
Dan Helfman 2ac58670d5 Clarify database documentation about excluding named pipes and character/block devices to prevent hangs. 2020-07-13 10:45:58 -07:00
Dan Helfman 6f82c9979b Add #339 to NEWS and add test. 2020-07-07 22:31:17 -07:00
Dan Helfman 0a659a397f Attempt to fix no-repro test failure of log_outputs() (#339). 2020-07-07 22:05:53 -07:00
Dan Helfman 2781873faf Bump version for release. 2020-06-30 21:47:34 -07:00
Dan Helfman 3aaa89fb08 Formatting. 2020-06-25 20:25:29 -07:00
Dan Helfman 35d542a676 Fix for traceback when running Cronitor, Cronhub, and PagerDuty monitor hooks (#336). 2020-06-25 20:23:25 -07:00
Dan Helfman d0b9c436b1 Bump version for release. 2020-06-23 11:21:43 -07:00
Dan Helfman 37cc229749 Fix duplicate logging to Healthchecks and send "after_*" hooks output to Healthchecks (#328). 2020-06-23 11:01:03 -07:00
Dan Helfman 17c2d109e5 Add tests for pass-through of BORG_* environment variables. 2020-06-21 14:41:22 -07:00
Dan Helfman c8d5de2179 Fix broken pass-through of BORG_* environment variables to Borg (#327).
Reviewed-on: witten/borgmatic#327
2020-06-21 21:29:59 +00:00
Dan Helfman 32e15dc905 Add a few more mocks to PostgreSQL SSL tests. 2020-06-20 14:39:16 -07:00
Dan Helfman f5ebca4907 Add SSL support to PostgreSQL database configuration (#331).
Reviewed-on: witten/borgmatic#331
2020-06-20 21:24:14 +00:00
Edward Shornock 01db676d68 Change the example for the ssl_mode parameter 2020-06-20 23:32:24 +03:00
Edward Shornock d2d92b1f1a Add tests for the PostgreSQL SSL options 2020-06-20 23:32:24 +03:00
Dan Helfman 27cbe9dfc0 Fix for potential data loss (data not getting backed up) when borgmatic omitted configured source directories in certain situations (#333). 2020-06-19 20:16:38 -07:00
Edward Shornock 8fb830099f Re-add the ilbpq-ssl documentation URL to the schema
It's been moved from describing `ssl_mode` to the general
postgresql_database description key.
2020-06-19 13:22:39 +03:00
Edward Shornock 463a133a63 Ensure schema lines are less than 80 characters in length 2020-06-19 13:22:39 +03:00
Edward Shornock a16fed8887 Rename PostgreSQL SSL config variables
e.g. s/sslmode/ssl_mode/g to conform with borgmatic naming conventions.
2020-06-19 13:20:14 +03:00
Edward Shornock 33113890f5 Reduce duplication with a common function 2020-06-19 12:32:36 +03:00
Edward Shornock abd47fc14e Add SSL support to PostgreSQL hooks 2020-06-19 02:19:17 +03:00
Dan Helfman 7fb4061759 Improve configuration reference documentation readability via more aggressive word-wrapping in configuration schema descriptions. 2020-06-17 23:15:12 -07:00
Dan Helfman b320e74ad5 Update documentation code fragments theme to better match the rest of the page. 2020-06-17 16:02:57 -07:00
Dan Helfman 0ed8f67b9d Documentation feedback: Clarify that a Borg manual install is required, separate from installing borgmatic. 2020-06-17 11:42:40 -07:00
Ralph Heinkel a12a1121b6 Use values from BORG_* env variables if they are not specified in config.yaml 2020-06-15 19:50:11 +02:00
Dan Helfman 795e18773b Bump version for release. 2020-06-06 15:01:56 -07:00
Dan Helfman aa14449857 Add "borgmatic extract --strip-components" flag to remove leading path components when extracting an archive (#324). 2020-06-06 14:57:14 -07:00
Dan Helfman ed7b1cd3d7 Add some no-cover pragmas on functions that don't need tests. 2020-06-06 14:33:06 -07:00
Dan Helfman a155eefa23 Fix for certain configuration options like ssh_command impacting Borg invocations for separate configuration files (#323). 2020-06-06 14:30:04 -07:00
Dan Helfman 398665be9e Allow before_backup and similiar hooks to exit with a soft failure without altering the monitoring status (#292). 2020-06-02 14:33:41 -07:00
Dan Helfman 6db232d4ac Link to Borgmacator GNOME AppIndicator from monitoring documentation. 2020-06-02 12:53:08 -07:00
Dan Helfman d7277893fb Fix hang when a stale database dump named pipe from an aborted borgmatic run remains on disk (#316). 2020-06-02 12:40:32 -07:00
Dan Helfman 00033bf0a8 Tweak comment indentation in generated configuration file for clarity. 2020-06-02 11:37:13 -07:00
Dan Helfman adda33dc4e Bump version for release. 2020-05-26 13:15:01 -07:00
Dan Helfman 097a09578a Fix enabled database hooks to implicitly set one_file_system configuration option to true to prevent Borg hang. (#315). 2020-05-26 09:20:14 -07:00
Dan Helfman 65472c8de2 Fix error message when there are no MySQL databases to dump for "all" databases (#319). 2020-05-26 08:59:04 -07:00
Dan Helfman 602ad9e7ee Add note about indirect dbus dependency. 2020-05-21 19:56:32 -07:00
Dan Helfman 96df52ec50 Fix hang when streaming a database dump to Borg with implicit duplicate source directories by deduplicating them first (#316). 2020-05-20 13:33:53 -07:00
Dan Helfman 244dc35bae Global install documentation. 2020-05-19 14:19:39 -07:00
Dan Helfman d9c9d7d2ee Improve documentation around the installation process. Specifically, making borgmatic commands runnable via the system PATH. 2020-05-18 20:38:43 -07:00
Dan Helfman 89cb5eb76d Fix regression in support for PostgreSQL's "directory" dump format (#314). 2020-05-18 11:31:29 -07:00
Dan Helfman 6d3802335e Adding docs note about upgrading to get --files flag. 2020-05-18 08:43:32 -07:00
Dan Helfman c1d6232b79 Fix documentation to mention new "--files" flag. 2020-05-15 10:45:58 -07:00
Dan Helfman 048a9ebb52 Add an additional end-to-end database test. 2020-05-15 10:12:49 -07:00
Dan Helfman de478f6ff7 Clarify wording in changelog. 2020-05-14 23:23:21 -07:00
Dan Helfman 3e5a19d95a Add missing test coverage. 2020-05-14 23:21:43 -07:00
Dan Helfman 2ddf38f99c Fix error handling when executing commands to handle more edge cases. 2020-05-14 23:04:01 -07:00
Dan Helfman d88f321cef Fix legitimate database dump command errors (exit code 1) not being treated as errors by borgmatic (#310). 2020-05-14 22:38:38 -07:00
Dan Helfman 74adac6c70 Bump version for release. 2020-05-12 21:03:09 -07:00
Dan Helfman 15ea70a71b Flesh out missing test coverage for execute.py. 2020-05-12 15:49:49 -07:00
Dan Helfman 8b91c01a4c Add some missing test coverage. 2020-05-11 11:34:14 -07:00
Dan Helfman 3bcef72050 Add some missing test mocks that were masking lack of unit coverage. 2020-05-11 11:17:24 -07:00
Dan Helfman 695c764a01 Merge log output functions into one. 2020-05-11 10:55:50 -07:00
Dan Helfman f7c93ea2e8 Wait for process to finish before trying to check exit status. 2020-05-09 23:09:48 -07:00
Dan Helfman 1ea047dd94 Remove "borgmatic restore --progress" flag, as it now conflicts with streaming database restores. 2020-05-09 21:53:16 -07:00
Dan Helfman 4b523f9e2c Make database restore output only show at verbosity 2. 2020-05-08 19:38:33 -07:00
Dan Helfman 6a61070d85 Use shell redirection rather than the --file flag to sidestep synchronization issues when pg_dump/pg_dumpall tries to write to a named pipe. 2020-05-08 19:11:26 -07:00
Dan Helfman f36082938e Additional test coverage. 2020-05-08 09:48:04 -07:00
Dan Helfman 1ba996ad93 Additional test coverage. 2020-05-07 12:14:27 -07:00
Dan Helfman a23fdf946d Stream database dumps and restores directly to/from Borg without using any additional filesystem space (#258). 2020-05-07 11:44:04 -07:00
Dan Helfman 12cf6913ef Remove unused function parameter. 2020-05-03 14:07:34 -07:00
Dan Helfman a4eef383c3 Spell out repository consistency check options in more detail. 2020-04-29 11:08:41 -07:00
Dan Helfman ac124612ad Documentation on macOS launchd permissions issues with work-around for Full Disk Access (#293). 2020-04-26 16:10:52 -07:00
Dan Helfman 95a479a86e Fix path in release script. 2020-04-24 16:05:50 -07:00
Dan Helfman e4eff0e3dc Bump version for release. 2020-04-24 15:56:56 -07:00
Dan Helfman dce1928dc4
Fix PostgreSQL restore error on "all" database dump. 2020-04-24 15:50:33 -07:00
Nathan Beals 3c8dc4929f Added `test_restore_all_database_dump` unit test.
Updated the other unit tests, as I had to re-arrange argument order
Added an 'all' test for the postgres end-to-end test.

Ran black formatter on it all.
2020-04-24 18:32:53 -04:00
Dan Helfman e511014a28 Fix MySQL restore error on "all" database dump by excluding system tables (#301). 2020-04-22 12:17:22 -07:00
Dan Helfman bae5f88824 Upgrade test database versions. 2020-04-21 10:01:26 -07:00
Dan Helfman 41ad98653a https://github.com/docker/compose/issues/2127 2020-04-21 09:39:20 -07:00
Dan Helfman 6a138aeb6e Move root vs. non-root instructions. 2020-04-18 13:14:35 -07:00
Dan Helfman f0ce37801b Add root vs. non-root to set up guide. 2020-04-17 20:30:10 -07:00
Dan Helfman 35f6aba365 Clarify that borgmatic should be run with sudo after a root --user install. 2020-04-17 09:46:50 -07:00
Nathan Beals f6407bafcb Remove the `--create` flag, was causing an error 2020-04-10 11:24:13 -04:00
Nathan Beals d5e9f67cec Finished. Now uses 'psql' to run the plain-text scripts that pg_dumpall creates 2020-04-10 10:55:53 -04:00
Nathan Beals b14f371c05 First attempt at fixing this pg_dumpall/restoring issue 2020-04-10 09:20:00 -04:00
Dan Helfman 31a5d1b9c4 Docs feedback: Clarify PagerDuty integration instructions. 2020-03-26 14:14:53 -07:00
Dan Helfman fb4305a953 Add link for Alpine packages of borgmatic to README. 2020-03-10 21:10:02 -07:00
Dan Helfman eab872823c Clarify license version. 2020-03-09 15:50:54 -07:00
Dan Helfman 3332750243 More documentation examples of a la carte actions. 2020-03-09 11:20:18 -07:00
Dan Helfman 4942b7ce4d Feedback on PagerDuty hook documentation. 2020-02-13 13:11:25 -08:00
Dan Helfman a2af77f363 Maybe fix release signing. 2020-02-03 09:57:34 -08:00
Dan Helfman a7490b56d1 Bump version for release. 2020-02-03 09:45:10 -08:00
Dan Helfman 66eb18d5ea Upgrade pip and tox in tests. 2020-02-02 14:30:26 -08:00
Dan Helfman 46486138b6 Fix PagerDuty link. 2020-02-01 13:43:37 -08:00
Dan Helfman d6562c4b1e Fix the "--stats" and "--files" flags so that they yield output at verbosity 0 (#290). 2020-01-31 10:23:36 -08:00
Dan Helfman 1ddde0910c Add security policy, such as it is. 2020-01-30 15:42:48 -08:00
Dan Helfman 79f3b84ca2 Documentation for "--archive latest" (#289). 2020-01-29 17:08:03 -08:00
Dan Helfman 55141bda67 Specify "--archive latest" to all actions that accept an archive (#289). 2020-01-29 16:59:02 -08:00
Dan Helfman bc02c123e6 Monitor backups with PagerDuty hook integration (#245). 2020-01-27 15:32:09 -08:00
Dan Helfman e76d5ad988 Fix tests. 2020-01-27 12:56:12 -08:00
Dan Helfman 8ad8a9c422 Add per-action hooks: "before_prune", "after_prune", "before_check", and "after_check" (#255). 2020-01-27 11:07:07 -08:00
Dan Helfman b15c9b7dab Add missing "how to" text. 2020-01-24 21:02:56 -08:00
Dan Helfman 2405e97c38 Backup to a removable drive or intermittent server via "soft failure" feature (#284). 2020-01-24 20:52:48 -08:00
Dan Helfman fdbb2ee905 View consistency check progress via "--progress" flag for "check" action (#287). 2020-01-24 11:27:16 -08:00
Dan Helfman 94b9ef56be Change "exclude_if_present" option to support multiple filenames, rather than just a single filename (#280). 2020-01-23 13:41:37 -08:00
Dan Helfman 952168ce25 Fix unwanted console log messages with "list --json" and "info --json". 2020-01-23 13:40:54 -08:00
Dan Helfman 5273037a94 For "list" and "info" actions, show repository names even at verbosity 0. 2020-01-23 11:17:39 -08:00
Dan Helfman 53e6ff9524 No longer list files or show stats by default at verbosity 2. 2020-01-22 15:23:49 -08:00
Dan Helfman f66fd1caaa Customize Healthchecks log level via borgmatic "--monitoring-verbosity" flag (#277). 2020-01-22 15:10:47 -08:00
Dan Helfman d93fdbc5ad Support "--files" and "--stats" flags at verbosity level 0. 2020-01-22 13:28:24 -08:00
Dan Helfman 58e0439daf Disable per-file logging by default at verbosity 1; opt-in via new --files option. 2020-01-22 20:36:59 +00:00
palto42 75b5e7254e changes as per comments in PR #283 2020-01-22 19:03:26 +01:00
Dan Helfman 39550a7fe9 Add ~/.config/borgmatic.d as another configuration directory default (#274). 2020-01-22 09:26:58 -08:00
palto42 5f0c084bee Merge 'upstream/master' into list-files 2020-01-22 18:12:26 +01:00
Dan Helfman 88f06f7921 Revert "Use absolute paths in systemd commands."
This reverts commit 24e1516ec5.
2020-01-21 16:03:24 -08:00
Dan Helfman 8d12079386 Bump version. 2020-01-21 10:47:29 -08:00
Dan Helfman 7824a034ca Add test for database dump directory removal. 2020-01-21 10:34:46 -08:00
Dan Helfman 8ef0ba2fae
After a backup of a database dump in directory format, properly remove the dump directory. 2020-01-21 10:29:40 -08:00
Dan Helfman cc384f4324 Second ticket for --json color bug. 2020-01-21 08:33:41 -08:00
Ronan Dunklau 8a91c79fb0 Support directory format dump cleanup.
Previously, only deleting a dump in a single-file format was supported.
This led to errors when performing a PostgreSQL directory format backup.
2020-01-19 15:15:47 +01:00
Dan Helfman ac1d63bb0d Use more realistic repository examples in README. 2020-01-18 20:00:18 -08:00
palto42 83632448be updated NEWS for mod. --stats & new --files opt. 2020-01-18 14:57:50 +01:00
palto42 e108526bab disable --stats by default 2020-01-18 14:38:59 +01:00
palto42 e27ba0d08a less detail at v1 + option "--files" for details 2020-01-11 16:38:07 +01:00
Dan Helfman 5afe0e3d63 Disable colored output when "--json" flag is used, so as to produce valid JSON ouput (#276). 2020-01-04 15:50:41 -08:00
Dan Helfman c52f82f9ce Documentation: Enable and start borgmatic with a single systemctl command. 2020-01-04 13:37:56 -08:00
Dan Helfman d0c533555e In "borgmatic --help", don't expand $HOME in listing of default "--config" paths. 2020-01-02 10:37:31 -08:00
Dan Helfman 1995c80e60 Add comment about old versions of systemd and option compatibility (#275). 2020-01-02 10:05:32 -08:00
Dan Helfman 24e1516ec5 Use absolute paths in systemd commands. 2020-01-01 17:14:55 -08:00
Dan Helfman 5b1beda82b Add logrotate documentation suggestion. 2019-12-31 15:06:53 -08:00
Dan Helfman e4f1094569 Bump version for release. 2019-12-20 14:04:49 -08:00
Dan Helfman 911668f0c8 Only trigger "on_error" hooks and monitoring failures for "prune", "create", and "check" actions, and not for other actions (#270). 2019-12-20 13:58:02 -08:00
Dan Helfman 6bfa0783b9 Clarify that the documentation suggestion form is only for documentation. 2019-12-17 20:16:13 -08:00
Dan Helfman d64bcd5e83 When pruning with verbosity level 1, list pruned and kept archives. 2019-12-17 20:12:41 -08:00
Dan Helfman ed2ca9f476 Sign release files. 2019-12-17 20:06:25 -08:00
Dan Helfman f787dfe809 Override particular configuration options from the command-line via "--override" flag (#268). 2019-12-17 11:46:27 -08:00
Dan Helfman afaabd14a8 Clarify documentation on how /etc/borgmatic.d/ configuration files are interpreted. 2019-12-13 11:42:17 -08:00
Dan Helfman e009bfeaa2 Update Healthchecks/Cronitor/Cronhub monitoring integrations to fire for "check" and "prune" actions, not just "create" (#249). 2019-12-12 22:54:45 -08:00
Dan Helfman f1358d52aa Add "borgmatic init" repository probing fix to NEWS. 2019-12-12 21:50:24 -08:00
Dan Helfman b04b333466
Use --remote-path, --debug and --info when checking for repo existence. 2019-12-13 05:47:47 +00:00
Matthew Daley dd16504329 Use --remote-path, --debug and --info when checking for repo existence
These are currently not being used in the call to `borg info` performed
as part of the borgmatic init command to check whether or not the repo
already exists.
2019-12-13 15:45:12 +13:00
Dan Helfman c6cb21a748 Switch to read-only container filesystem to avoid *.pyc getting created with busted permissions. 2019-12-11 21:24:37 -08:00
Dan Helfman 78aa4626fa Remove user switch in container due to CI permission issue. 2019-12-11 16:58:08 -08:00
Dan Helfman d2df224da8 Use busybox short option to su. 2019-12-11 16:46:24 -08:00
Dan Helfman 464ff2fe96 Run end-to-end tests on developer machines with Docker Compose for approximate parity with continuous integration tests. 2019-12-11 16:43:01 -08:00
Dan Helfman 0cc711173a Merge branch 'master' into end-to-end-database-tests 2019-12-11 12:27:14 -08:00
Dan Helfman 14e5cfc8f8 Support piping "borgmatic list" output to grep. Retain colored output when piping/redirecting (#271). 2019-12-11 12:12:25 -08:00
Dan Helfman b8b888090d Select Postgres service to work with particular client version. 2019-12-10 21:41:15 -08:00
Dan Helfman 68281339b7 Black. 2019-12-10 16:57:12 -08:00
Dan Helfman 2e5be3d3f1 Add missing psql. 2019-12-10 16:52:59 -08:00
Dan Helfman abd31a94fb Ports fix? 2019-12-10 16:47:09 -08:00
Dan Helfman 01e2cf08d1 Fix Drone CI services syntax. 2019-12-10 16:43:43 -08:00
Dan Helfman 9f821862b7 End-to-end tests for database dump and restore. 2019-12-10 16:41:01 -08:00
Dan Helfman 8660af745e Optionally change the internal database dump path via "borgmatic_source_directory" option in location configuration section (#259). 2019-12-10 16:04:34 -08:00
Dan Helfman 826e4352d1 Filter listed paths via "borgmatic list --path" flag (#269). 2019-12-08 14:07:02 -08:00
Dan Helfman b94999bba4 Fix "borgmatic umount" so it only runs Borg once instead of once per repository / configuration file. 2019-12-07 21:36:51 -08:00
Dan Helfman 65cc4c9429 Fix "--repository" flag to accept relative paths. 2019-12-06 16:29:41 -08:00
Dan Helfman df2be9620b Mount whole repositories via "borgmatic mount" without any "--archive" flag (#253). 2019-12-06 15:58:54 -08:00
Dan Helfman 2ab9daaa0f Attempt to repair any inconsistencies found during a consistency check via "borgmatic check --repair" flag (#266). 2019-12-04 16:07:00 -08:00
Dan Helfman 0c6c61a272 Pass extra options directly to particular Borg commands, handy for Borg options that borgmatic does not yet support natively (#235). 2019-12-04 15:48:10 -08:00
Dan Helfman 00f62ca023 Fix for "before_backup" hook not triggering an error when the command contains "borg" and has an exit code of 1 (#256). 2019-11-30 16:55:05 -08:00
Dan Helfman 9b2ca15de6 Fix for garbled Borg file listing when using "borgmatic create --progress" with verbosity level 1 or 2 (#257). 2019-11-30 15:31:36 -08:00
Dan Helfman c4aa34bf5c Fix for missing Healthchecks monitoring payload or HTTP 500 due to incorrect unicode encoding (#260). 2019-11-30 14:51:32 -08:00
Dan Helfman 4385f2a36a Merge branch 'master' of github.com:witten/borgmatic 2019-11-25 15:28:21 -08:00
Dan Helfman ed6a9dadf8
Fix for database dump removal incorrectly skipping some database dumps. 2019-11-25 23:28:15 +00:00
Dan Helfman d978a2d190 Fix for database dump removal incorrectly skipping some database dumps. 2019-11-25 15:27:59 -08:00
Dan Helfman 375036e409 Support for mounting an archive as a FUSE filesystem via "borgmatic mount" action, and unmounting via "borgmatic umount" (#123). 2019-11-25 14:56:20 -08:00
Raphael Heinrich 99168c1035 Fix error on cleanup multiple database dumps 2019-11-25 22:07:42 +01:00
Dan Helfman f4a231420f Show summary log errors regardless of verbosity. Log the "summary:" header with level based on the contained logs. 2019-11-25 10:31:09 -08:00
Dan Helfman 55ebfdda39 Show full error logs at "--verbosity 0" so you can see command output without upping the verbosity level. 2019-11-19 10:09:25 -08:00
Dan Helfman e63e2e0852 Rephrasing verbosity levels. 2019-11-17 22:52:26 -08:00
Dan Helfman edc4b9e60e Update the command-line help description. 2019-11-17 22:49:16 -08:00
Dan Helfman 78ff734e6c Add note about error logs only flowing to Healthchecks for the "create" action. 2019-11-17 19:10:11 -08:00
Dan Helfman 2cc743cf47 With "borgmatic check", consider Borg warnings as errors (#247). 2019-11-17 19:06:28 -08:00
Dan Helfman d99e6d1994 Remove obnoxious log message when Healthchecks hook sends failure for check action. 2019-11-17 18:55:40 -08:00
Dan Helfman 50f62d73b7 When using the Healthchecks monitoring hook, include borgmatic logs in the payloads for completion and failure pings (#241). 2019-11-17 16:54:27 -08:00
Dan Helfman 26a89de790 Higher resolution BorgBase logo. 2019-11-14 12:39:33 -08:00
Dan Helfman c2276b18c5 Add troubleshooting documentation for MySQL lock table errors (#244). 2019-11-14 10:33:47 -08:00
Dan Helfman 693434f8aa Icon spacing hack that'll show up on GitHub. 2019-11-14 09:45:38 -08:00
Dan Helfman 1e8edc05e9 Marketing. 2019-11-14 09:34:53 -08:00
Dan Helfman 1f166a47e9 Fix failing test due to dictionary order (or the lack thereof..) in Python 3.5. 2019-11-13 16:42:58 -08:00
Dan Helfman 9ee6151999 Add link to unofficial Fedora package of borgmatic that's more up-to-date. 2019-11-13 14:59:49 -08:00
Dan Helfman 6cdc92bd0c Fix for "borgmatic restore" showing success and incorrectly extracting archive files, even when no databases are configured to restore (#246). 2019-11-13 10:41:57 -08:00
Dan Helfman 612e1fea67 Add database version to issue template. 2019-11-13 08:38:58 -08:00
Dan Helfman 0a9f4e8708 Reopen the file given by "--log-file" flag if an external program rotates the log file while borgmatic is running. 2019-11-12 16:13:25 -08:00
Dan Helfman 781fac3266 Refactor ping monitors (Healthchecks, Cronitor, Cronhub) to share a common invocation and function signature. 2019-11-12 15:31:07 -08:00
Dan Helfman 4c38810a32 Add note about rsyslog rate limiting. 2019-11-12 12:32:32 -08:00
Dan Helfman bf0d38ff2a Update docs about MySQL/MariaDB database dumps and restores (#228). 2019-11-12 11:59:01 -08:00
Dan Helfman 04e5b42606 Fix repository does not exist error with "borgmatic extract" when repository is remote (#243). 2019-11-12 11:47:24 -08:00
Dan Helfman 30525c43bf Another edge case: Don't error on non-matching restore path globs (#228). 2019-11-12 11:24:31 -08:00
Dan Helfman ebeb5efe05 More accurately detecting restore of unconfigured database (#228). 2019-11-12 11:10:47 -08:00
Dan Helfman a3e939f34b Factor out filtering of database configuration to function with tests (#228). 2019-11-12 10:39:36 -08:00
Dan Helfman 2a771161e7 Finish test coverage for MySQL restore (#228). 2019-11-12 09:50:26 -08:00
Dan Helfman ded042d8cc First crack at MySQL/MariaDB database restore (#228). 2019-11-11 21:59:30 -08:00
Dan Helfman 4ed43ae4dc Indicate in AUTHORS that it's not an exhaustive list. 2019-11-10 14:50:28 -08:00
Dan Helfman 9d29ecf304 Remove MySQL/MariaDB database dumps after backing them up (#228). 2019-11-08 11:53:27 -08:00
Dan Helfman 427b57e2a9 Database dump hooks for MySQL/MariaDB (#228). 2019-11-08 11:17:52 -08:00
Dan Helfman e4f0a336c2 Fix installation path in docs. 2019-11-07 11:09:27 -08:00
Dan Helfman 68459c6795 Add note about setting PATH environment variable when installing borgmatic. 2019-11-07 11:05:41 -08:00
Dan Helfman 17fda7281a Monitor backups with Cronhub hook integration. Fix Healthchecks/Cronitor hooks to respect dry run. 2019-11-07 10:08:44 -08:00
Dan Helfman ac777965d0 Fix regression of generate-borgmatic-config working without --source flag (#239). 2019-11-06 09:52:21 -08:00
Dan Helfman 31d3bc9bd8 In documentation, clarify when Healthchecks/Cronitor hooks fire in relation to other hooks (#238). 2019-11-06 09:44:46 -08:00
Dan Helfman 2115eeb6a2 Upgrade your borgmatic configuration to get new options and comments via "generate-borgmatic-config --source" (#239). 2019-11-06 09:31:00 -08:00
Dan Helfman 08f017bc3e Remove broken Tox Python version filter when running Black (code formatter) environment directly. 2019-11-05 09:36:30 -08:00
Dan Helfman 7bc9482970 Add verbosity level -1 to NEWS. 2019-11-03 17:03:19 -08:00
Dan Helfman 57ffad4e04 Verbosity level "-1" for even quieter output: Errors only (#236). 2019-11-04 01:02:13 +00:00
Dan Helfman 5422d14f93 Add Python 3.8. 2019-11-03 14:12:46 -08:00
Dan Helfman e6d8c736d0 Documentation feedback: Don't hard-code Python 3.7 for Black. 2019-11-03 14:08:19 -08:00
palto42 18d3542fbc new verbosity level "-1" for errors 2019-11-03 09:55:19 +01:00
Dan Helfman 93f453cecf Add Python 3.8 to build matrix. 2019-11-02 16:38:50 -07:00
Dan Helfman 505bb778fa Fix logging docs typo. 2019-11-02 12:54:03 -07:00
Dan Helfman b09d464162 Add missing PR link. 2019-11-02 11:27:05 -07:00
Dan Helfman a9104ed090 Handle log file error more consistently with other error. Add --log-file-verbosity flag. Add docs. 2019-11-02 11:23:18 -07:00
Dan Helfman 06f134cc71 Log to file instead of syslog via command-line "--log-file" flag (#233). 2019-11-02 17:43:39 +00:00
palto42 584359b6c0 .gitignore 2019-11-02 14:39:16 +01:00
palto42 26a1a3d1e0 test cases for logfile option 2019-11-02 14:34:16 +01:00
palto42 6da05cbe2d Exception handling for logfile option 2019-11-02 14:33:57 +01:00
Dan Helfman f48f52079d Support for Borg --keep-exclude-tags and --exclude-nodump options (#234). 2019-11-01 15:50:59 -07:00
Dan Helfman 76c569cf84 Update test requirements. 2019-11-01 12:18:35 -07:00
palto42 b121290c0f Merge remote-tracking branch 'upstream/master' into logging 2019-11-01 19:44:04 +01:00
Dan Helfman 8fd46b8c70 Monitor backups with Cronitor hook integration. 2019-11-01 11:33:15 -07:00
Dan Helfman 603f525352 Clarify --archive help. 2019-11-01 10:50:20 -07:00
palto42 8c8640d0ab file-logger replaces syslog 2019-11-01 18:42:24 +01:00
Dan Helfman e3dd545345 Extract files to a particular directory via "borgmatic extract --destination" flag. Also rename "--restore-path" to "--path" to reduce confusion. 2019-11-01 10:00:26 -07:00
Dan Helfman 589fc30fc8 To orient the reader, add "borgmatic" to the top of each documentation page. 2019-10-31 22:04:38 -07:00
Dan Helfman bd3c51fc5a Documentation for database restores (#229). 2019-10-31 21:45:47 -07:00
Dan Helfman 2c46f53ef6 Attempt to upgrade version of pip used in CI. 2019-10-31 20:37:01 -07:00
Dan Helfman 939f4d4e3d Add additional tests for database restores. 2019-10-31 14:11:19 -07:00
Dan Helfman 3006db0cae Restore backed up PostgreSQL databases via "borgmatic restore" sub-command (#229). 2019-10-31 12:57:36 -07:00
palto42 22640a9ca0 new option for log-file 2019-10-31 10:44:22 +01:00
Dan Helfman ca23c3b8b3 Remove unnecessary word from documentation. 2019-10-30 10:55:40 -07:00
Dan Helfman 74607fdd43 Documentation on how to develop borgmatic's documentation. 2019-10-30 10:54:42 -07:00
Dan Helfman b53684a8f0 Fix incorrect link to hooks documentation. 2019-10-29 09:53:45 -07:00
Dan Helfman f055f5dea8 Clarify borgmatic restore example with no leading slash. 2019-10-28 12:00:54 -07:00
Dan Helfman 4dc4fe0b8d Reordering hooks a bit so that pre-backup ordering mirrors post-backup ordering. 2019-10-27 14:47:55 -07:00
Dan Helfman 5e3c2da79c Database dump hooks documentation (#225). 2019-10-23 15:35:37 -07:00
Dan Helfman 37dc94bc79 Add test for removal of database dumps. 2019-10-23 13:36:03 -07:00
Dan Helfman fc274b43f0 Rename "borgmatic list --pattern-from" flag to "--patterns-from" to match Borg (#230). 2019-10-22 22:42:36 -07:00
Dan Helfman 9ab12e4312 Tests for database dumping (#225). 2019-10-22 21:39:30 -07:00
Dan Helfman a5ff35c198 Update NEWS with PostgreSQL database dump hook. 2019-10-22 16:31:26 -07:00
Dan Helfman 458e7776c5 Database dump hooks for PostgreSQL, so you can easily dump your databases before backups run (#225). 2019-10-22 16:28:42 -07:00
Dan Helfman fa5fa1c11b Move hooks into directory, so there can be one source file per type of hook (#225). 2019-10-21 15:52:14 -07:00
Dan Helfman f8bc67be8d Config generation support for sequences of maps, needed for database dump hooks (#225). 2019-10-21 15:17:47 -07:00
Dan Helfman 17586d49ac Bump version of tox in CI. 2019-10-21 11:05:37 -07:00
Dan Helfman 2f75c9aa9e Bump Tox minimum version. 2019-10-20 21:47:57 +00:00
Luke Murphy 60650ccfc7
Follow latest Tox developments 2019-10-20 12:49:14 +02:00
Dan Helfman c12c47cace Fix "borgmatic list --successful" with a slightly better heuristic for listing successful (non-checkpoint) archives. 2019-10-16 10:24:58 -07:00
Dan Helfman d6aaab8a09 Remove parentheses from docs sentence. 2019-10-15 13:02:54 -07:00
Dan Helfman 128ebf04ce Dead man's switch via healthchecks.io integration (#223) + new monitoring documentation. 2019-10-15 10:49:14 -07:00
Dan Helfman b1941bcce9 Automatically rewrite links to localhost when developing on docs locally. 2019-10-14 13:13:41 -07:00
Dan Helfman 7b3b28616d Add "borgmatic list --successful" flag to only list successful (non-checkpoint) archives (#86). 2019-10-13 15:58:11 -07:00
Dan Helfman f3910f49ca Fix incorrect help on borg list --last flag. 2019-10-13 14:46:28 -07:00
Dan Helfman 59e1cac92c Correct Arch Linux borgmatic package link. 2019-10-11 14:35:07 -07:00
Dan Helfman b1f0287fdb Add documentation link to community AUR (Arch Linux) borgmatic package. 2019-10-11 13:35:57 -07:00
Dan Helfman 99c35d4077 "flags" -> "actions" a few places in the docs. 2019-10-11 10:46:30 -07:00
Dan Helfman 07b9ff61f2 Remove documentation link to the AUR (Arch Linux) borgmatic package, which apparently has been deleted. 2019-10-11 10:42:19 -07:00
Dan Helfman f573c1810a Add a suggestion form to all documentation pages, so users can submit ideas for improving the documentation. 2019-10-10 14:27:48 -07:00
Dan Helfman 1d37b14356 More detailed error alerting via runtime context available in "on_error" hook (#174). 2019-10-01 12:23:16 -07:00
Dan Helfman 6c617eddd5 When backups to one of several repositories fails, keep backing up to the other repositories (#144). 2019-09-30 22:19:31 -07:00
Dan Helfman e14ebee4e0 User-defined hooks for global setup or cleanup that run before/after all actions. (#192). 2019-09-28 16:18:10 -07:00
Dan Helfman a897ffd514 Fix "borgmatic create --progress" output so that it updates on the console in real-time (#221). 2019-09-25 12:03:10 -07:00
Dan Helfman a472735616 Merge sample cron files. 2019-09-24 10:49:46 -07:00
Dan Helfman b3fec03cf4 Up the syslog verbosity in sample cron files. 2019-09-24 10:47:39 -07:00
Dan Helfman 89dccc25c3 Add AC power condition for systemd service (#205). 2019-09-24 10:43:30 -07:00
Dan Helfman 3846155d62 More robust sample systemd service: boot delay, network dependency, lowered CPU/IO priority, etc (#205). 2019-09-24 10:16:30 -07:00
Dan Helfman 386979ebb4 Mention --stats option in documentation. 2019-09-23 13:13:34 -07:00
Dan Helfman 07222cd984 Fix visibility of "borgmatic prune --stats" output (#219). 2019-09-23 13:07:51 -07:00
Dan Helfman cf4c6c274d Upgrade build to Alpine 3.10. 2019-09-23 09:07:17 -07:00
Dan Helfman 340bd72176 Fix regression of argument parsing for default actions (#220). 2019-09-22 11:30:58 -07:00
Dan Helfman 1a1bb71af1 Fix error with "borgmatic check --only" command-line flag with "extract" consistency check (#217). 2019-09-20 11:43:27 -07:00
Dan Helfman ae45dfe63a Clarify command-like help for check --only. 2019-09-19 15:20:05 -07:00
Dan Helfman d6ac7a9192 Upgrade various dependencies. 2019-09-19 13:04:59 -07:00
Dan Helfman d959fdbf8d Document new "check --only" command-line flag. 2019-09-19 11:50:29 -07:00
Dan Helfman 81739791e0 Override configured consistency checks via "borgmatic check --only" command-line flag (#210). 2019-09-19 11:43:53 -07:00
Dan Helfman 4cdff74e9b Support for Borg check --verify-data flag via borgmatic "data" consistency check (#210). 2019-09-18 16:52:27 -07:00
Dan Helfman 11e830bb1d Fix flake8 warning. 2019-09-18 14:11:56 -07:00
Dan Helfman cba00a9c4e Add NEWS entry for generate-borgmatic-config comment change. 2019-09-18 14:06:03 -07:00
Dan Helfman f2198de151 Merge branch 'comments-white-space' of polyzen/borgmatic into master 2019-09-18 21:03:56 +00:00
Daniel M. Capella 0c439c0c02
Add space to separate comments from tokens
https://yaml.org/spec/1.2/spec.html#id2780069
2019-09-17 20:00:58 -04:00
Dan Helfman f11a9bb4aa Revert "Fix for spurious Borg traceback when initializing a repository in an empty directory (#201)."
This reverts commit 9585c8f908.
2019-09-14 16:14:20 -07:00
Dan Helfman ee6f390910 Merge branch 'point-to-stable-docs' of polyzen/borgmatic into master 2019-09-14 21:53:34 +00:00
Daniel M. Capella 9a5117db14
Consistently point to stable Borg docs 2019-09-14 17:30:28 -04:00
Dan Helfman 9585c8f908 Fix for spurious Borg traceback when initializing a repository in an empty directory (#201). 2019-09-13 13:08:23 -07:00
Dan Helfman 3495484ddd Bump version for release. 2019-09-12 21:35:00 -07:00
Dan Helfman 67ab2acb82 Fix for hook erroring with exit code 1 not being interpreted as an error (#214). 2019-09-12 16:37:43 -07:00
Dan Helfman c085bacccf Reorder arguments passed to Borg to fix duplicate directories when using Borg patterns (#213). 2019-09-12 15:27:04 -07:00
Dan Helfman 896401088e Fix for traceback when the "checks" option has an empty value (#208). 2019-08-26 09:52:32 -07:00
Dan Helfman ef3dda9213 Bypass Borg error about a moved repository (#209). 2019-08-26 09:39:41 -07:00
Dan Helfman c9f5d9b048 In issue template, use python3 instead of python. 2019-08-24 13:08:18 -07:00
Dan Helfman ccbd0b608b Do not treat Borg warnings (exit code 1) as failures (#204). 2019-08-03 15:13:54 -07:00
Dan Helfman a7cc2ea803 When validating configuration files, require strings instead of allowing any scalar type. 2019-08-03 14:52:12 -07:00
Dan Helfman 9ec75ccf3f Fit inadvertent conversion of ordered dict to dict. 2019-07-27 14:15:24 -07:00
Dan Helfman 7c890be76d Black formatting. 2019-07-27 14:08:47 -07:00
Dan Helfman 39e5aac479 If a "prefix" option in borgmatic's configuration has an empty value (blank or ""), then disable default prefix. 2019-07-27 14:04:13 -07:00
Dan Helfman e25f2c4e6c Clarify documentation/schema about on_error hook running if there's an error in another hook (#202). 2019-07-19 09:25:01 -07:00
Dan Helfman 7ad8f9ac6f Link to borgmatic-binary installation method. 2019-07-13 15:40:26 -07:00
Dan Helfman 2add3ff7ad Fix redirect. 2019-07-05 09:19:51 -07:00
Dan Helfman 0602ca1862 Add how-to redirect. Fix capitalization. 2019-07-05 09:03:08 -07:00
Dan Helfman e973802fc1 Iterate on how-to document name wording. 2019-07-05 08:57:25 -07:00
Dan Helfman 2bdf6dfd70 Merge branch 'master' of ssh://projects.torsion.org:3022/witten/borgmatic 2019-07-05 08:52:06 -07:00
Dan Helfman f894c49540 Merge branch 'rename_howto_guide' of duncanbetts/borgmatic into master 2019-07-05 15:52:21 +00:00
duncanbetts 7900e5ea53 Update 'README.md' 2019-07-05 14:40:41 +00:00
duncanbetts 5587f48bda Update 'docs/how-to/run-preparation-steps-before-backups.md' 2019-07-05 14:39:21 +00:00
duncanbetts de3ee07566 Update 'README.md'
Improved description of what the resource provides.
2019-07-05 14:37:42 +00:00
Dan Helfman fe39453598 Change example filename to be more descriptive. 2019-06-30 17:23:09 -07:00
Dan Helfman 9c75063c05 Unbreak console snippet in docs. 2019-06-30 17:09:34 -07:00
Dan Helfman 5cf2ef1732 Add note to documentation about using spaces instead of tabs for indentation, as YAML does not allow tabs (#199). 2019-06-30 16:58:01 -07:00
Dan Helfman f35e6ea7ad Upgrade base layers. 2019-06-27 15:38:00 -07:00
Dan Helfman 90595e9c18 Only log to syslog when run from a non-interactive console (e.g. a cron job). Related to #197. 2019-06-27 14:41:21 -07:00
Dan Helfman 032d4adee3 Remove unicode byte order mark from syslog output. (Related to #197.) 2019-06-27 10:03:49 -07:00
Dan Helfman 4444219e17 Support for Borg --noatime, --noctime, and --nobirthtime flags (mentioned in #193). 2019-06-25 11:30:55 -07:00
Dan Helfman 56fd78089d Sort generated flags before passing them to Borg. 2019-06-25 11:04:10 -07:00
Dan Helfman 86dbc00cbe Support for several more borgmatic/borg info command-line flags (#193). 2019-06-25 10:46:55 -07:00
Dan Helfman c644270599 Pass through several "borg list" flags (#193). 2019-06-25 10:18:30 -07:00
Dan Helfman 1676a98c51 Fix for Borg create error output not showing up at borgmatic verbosity level zero (#198). 2019-06-24 09:55:41 -07:00
Dan Helfman 358ed53da0 Only show build status badge for master branch. 2019-06-23 16:53:33 -07:00
Dan Helfman 90925c9428 Provide tips about old-style flags for those on older versions. 2019-06-23 16:42:23 -07:00
Dan Helfman cd192a6909 Bump version for release. 2019-06-23 16:30:16 -07:00
Dan Helfman 7185146481 A bunch of tests for parse_subparser_arguments(). 2019-06-23 16:06:39 -07:00
Dan Helfman c15e6c5fe5 More actions help. 2019-06-23 09:46:22 -07:00
Dan Helfman 79c2b9df06 Don't make major version bump after all. 2019-06-23 09:23:51 -07:00
Dan Helfman acd6772148 Update documentation to refer to dashless action sub-commands. 2019-06-22 22:09:50 -07:00
Dan Helfman cd91dbd4f7 Include sub-command help in documentation. 2019-06-22 22:04:56 -07:00
Dan Helfman 8fc4efff88 Add subcommand note to NEWS. 2019-06-22 21:35:40 -07:00
Dan Helfman 4bf3e906a1 Break out main borgmatic arguments-parsing code into a separate file. 2019-06-22 21:32:27 -07:00
Dan Helfman 0ca43ef67a Get tests passing. 2019-06-22 21:23:48 -07:00
Dan Helfman 603e055a39 Fix borgmatic command unit tests for new parsed arguments. 2019-06-22 16:29:25 -07:00
Dan Helfman 75c04611dc Refactor to support subparsed-based parsed arguments. 2019-06-22 16:10:07 -07:00
Dan Helfman 881dc9b01e Make each subparser get a crack all all arguments. 2019-06-21 23:12:37 -07:00
Dan Helfman 8c72e909a7 Initial stab at subparsers for argument parsing. Not yet fully working. 2019-06-21 22:27:16 -07:00
Dan Helfman 74ac148747 Disable console color via "color" option in borgmatic configuration output section (#191). 2019-06-19 20:48:54 -07:00
Dan Helfman be7887c071 Demote log level of unhelpful info log (#194). 2019-06-19 12:01:03 -07:00
Dan Helfman da459d95b8 Bump version for release. 2019-06-17 12:16:23 -07:00
Dan Helfman b3aa6af859 Don't color syslog output (#197). 2019-06-17 11:53:08 -07:00
Dan Helfman b816af1b13 Undo purge. 2019-06-16 22:10:25 -07:00
Dan Helfman 276aeb9875 Fix tests that assert on default syslog verbosity. 2019-06-16 21:58:41 -07:00
Dan Helfman de94001508 Change default syslog verbosity to show errors only. 2019-06-16 21:57:14 -07:00
Dan Helfman 7cfab3620b Don't prune docs image after push, so watchtower can pick it up. 2019-06-16 21:52:09 -07:00
Dan Helfman 6c136ebbf1 Fix for unclear error message for invalid YAML merge include (#196). 2019-06-16 21:33:40 -07:00
Dan Helfman eaa5eb4174 Note about including config file. 2019-06-15 14:28:32 -07:00
Dan Helfman acc2a39454 Include note about debug output. 2019-06-15 14:27:53 -07:00
Dan Helfman a10c7a8496 Trying out a Gitea issue template for somewhat more structured bug reports. 2019-06-15 14:23:45 -07:00
Dan Helfman de82919e39 Skip coverage on a particular annoying-to-trigger branch in execution code. 2019-06-13 21:38:06 -07:00
Dan Helfman 1ba56d5262 Fix tests in Python 3.6. 2019-06-13 21:34:04 -07:00
Dan Helfman 1c825b5d84 Bump version for release. 2019-06-13 20:50:06 -07:00
Dan Helfman d6d66de251 Set umask used when executing hooks via "umask" option in borgmatic hooks section (#189). 2019-06-13 17:05:26 -07:00
Dan Helfman 76d79f0331 Suppress part of an obnoxious warning about disabling coverage (for end-to-end tests). 2019-06-13 14:15:08 -07:00
Dan Helfman dc43c38e29 Complete test coverage for logging branch. 2019-06-13 11:11:42 -07:00
Dan Helfman 7d7308a80d Integration tests for execute.py. 2019-06-13 10:48:21 -07:00
Dan Helfman b43ef9d76d Move test file to correspond to its code under test. 2019-06-13 10:27:00 -07:00
Dan Helfman 28cdd67743 Error hook test. 2019-06-13 10:14:16 -07:00
Dan Helfman 7f126ce127 Move hook.py file up a level. 2019-06-13 10:09:16 -07:00
Dan Helfman a6c4debf78 Additional test coverage, and upgrade test requirements. 2019-06-13 10:01:55 -07:00
Dan Helfman a74ad5475e Run all Borg commands such that they log to syslog as well as console. 2019-06-12 20:56:20 -07:00
Dan Helfman fa293828df Run hooks such that their output goes to syslog as well as console. 2019-06-12 13:09:04 -07:00
Dan Helfman f5582b1754 Move borgmatic.borg.execute module up a level for broader use. 2019-06-12 12:13:59 -07:00
Dan Helfman 1af95714c2 Collapse two execute_command() parameters into one output log level parameter. 2019-06-12 12:11:36 -07:00
Dan Helfman 0406d18cfd Log Borg --stats output as warning so it shows up at any verbosity level. 2019-06-12 11:49:35 -07:00
Dan Helfman 66e9ec9c3c A few tests for JSON flag suppressing Borg output. 2019-06-12 11:31:46 -07:00
Dan Helfman 899a7c8318 Add some wheel metadata do .gitignore. 2019-06-11 21:40:28 -07:00
Dan Helfman 7c01b69498 Details on where to view logs. 2019-06-11 21:35:43 -07:00
Dan Helfman 4f0d3bf4ed Add docs/default about systemd journald rate limiting. 2019-06-11 17:03:40 -07:00
Dan Helfman 9a5e7a3abb Successfully convert Borg output to Python logging entries. 2019-06-11 16:42:04 -07:00
Dan Helfman 02eb6c7e09 Merge branch 'master' into logging 2019-06-10 10:27:22 -07:00
Dan Helfman 418c09398c Fix incorrect compression default in schema comment. 2019-06-09 21:21:46 -07:00
Dan Helfman cdbd4c55e8 Fix 404 links harder. 2019-06-01 13:23:48 -07:00
Dan Helfman 2374410891 Fix 404 documentation links by switching to absolute links. 2019-06-01 13:02:39 -07:00
Dan Helfman d2c46e91fe Add rsync.net to hosting providers; includ random link rotation. 2019-05-29 15:35:04 -07:00
Dan Helfman 12441331e6 Fix formatting / import ordering. 2019-05-27 15:46:38 -07:00
Dan Helfman 9ceeae2de0 Add separate syslog verbosity flag. 2019-05-27 15:44:48 -07:00
Dan Helfman e0e493c2f1 Factor out configuring of logging into a common function. 2019-05-27 15:05:26 -07:00
Dan Helfman 0f05f7ad93 Log to syslog in addition to existing standard out logging (#53). 2019-05-26 16:34:46 -07:00
Dan Helfman 9bc1b71017 Clarify description in setup.py. 2019-05-26 13:36:53 -07:00
Dan Helfman b3776871b5 Rewrite the borgmatic overview a bit to clarify its place in the world. 2019-05-26 13:35:51 -07:00
Dan Helfman 308cb31bf9 Remove some of the link rewriting to hopefully fix broken docs links. 2019-05-25 21:55:28 -07:00
Dan Helfman e1f4643215 In README, use absolute links to properly rendered documentation (to cut down on confusion). 2019-05-25 21:48:05 -07:00
Dan Helfman bc4fb322b5 Move documenation build step last in CI. Refactor docs build scripts. 2019-05-21 21:33:25 -07:00
Dan Helfman 2c4f192e43 Attempt to build documentation image in CI. 2019-05-21 21:13:35 -07:00
Dan Helfman fb7a6dccaa Link to docs on skipping pruning entirely. 2019-05-21 12:32:19 -07:00
Dan Helfman 2826b7bd7c Add files for building documentation into a Docker image for web serving. 2019-05-21 03:16:32 +00:00
Dan Helfman 932848f6c1 Merge branch 'master' into docs-image. 2019-05-20 19:23:12 -07:00
Dan Helfman 9255940c6b Upgrade Drone build file format from 0.8 to 1.x. 2019-05-21 02:21:46 +00:00
Dan Helfman 3eadd16856 Add build server upgrade to NEWS. 2019-05-20 19:20:05 -07:00
Dan Helfman 61f46c5ad5 Try without explicit "sh". 2019-05-20 19:06:13 -07:00
Dan Helfman aad47d1741 bash -> sh. 2019-05-20 18:19:42 -07:00
Dan Helfman 079dd3fe4c Another try. 2019-05-20 18:18:30 -07:00
Dan Helfman d47f1bff4d Try to run script. 2019-05-20 18:17:25 -07:00
Dan Helfman 53967f6324 Trigger build. 2019-05-20 17:48:39 -07:00
Dan Helfman f5a70dc2a5 Drone + jsonnet apparently requires an extension, so switching back to plain YAML. 2019-05-20 17:30:07 -07:00
Dan Helfman 31ae1013d7 Add missing close curly. 2019-05-20 17:18:30 -07:00
Dan Helfman 071945e558 Re-order. 2019-05-20 17:15:01 -07:00
Dan Helfman 5c4d6a6e83 Upgrade Drone build file format from 0.8 to 1.x. 2019-05-20 17:05:29 -07:00
Dan Helfman 9c9be65b2b Add files for building documentation into a Docker image for web serving. 2019-05-20 11:41:39 -07:00
Dan Helfman c164684703 Allow to only run unit tests with Tox. 2019-05-19 22:07:15 +00:00
Dan Helfman 842c9001ba Auto-join #borgmatic from IRC web chat. 2019-05-19 15:04:01 -07:00
Dan Helfman 481e47076e Add #borgmatic Freenode IRC channel to documentation. 2019-05-19 15:01:03 -07:00
Luke Murphy 917a0dd0a0
Pass posargs to pytest in main testenv 2019-05-19 23:53:43 +02:00
Luke Murphy 358aed7c31
Allow to only run unit tests with Tox 2019-05-19 23:53:42 +02:00
Dan Helfman 9893834e85 Pass positional arguments to Tox environments commands. 2019-05-19 21:28:38 +00:00
Dan Helfman 32cf3225c5 Update NEWS. 2019-05-19 14:17:00 -07:00
Dan Helfman 2bfd7518c5 Look for .yml configuration file extension in addition to .yaml (#178). 2019-05-19 21:16:25 +00:00
Dan Helfman 4ba56684d1 Update NEWS with remove Python cache files before each Tox run. 2019-05-19 10:07:28 -07:00
Dan Helfman 0b1e38e5f6 Remove Python cache files before each Tox run. 2019-05-19 17:07:11 +00:00
Luke Murphy 7974219389
Make sure to pass `posargs` for Tox testenvs 2019-05-19 13:11:22 +02:00
Luke Murphy 8424e443a9
Also read .yml ending configuration files
Closes witten/borgmatic#178.
2019-05-19 13:04:42 +02:00
Luke Murphy 85251cf5d4
Ensure to remove cache files for Tox runs 2019-05-19 12:46:32 +02:00
Dan Helfman 8f882ea3ea Switch to more standard "utm_source" for hosting provider link. 2019-05-18 21:33:39 -07:00
Dan Helfman 7a2bcc96bb Add Borg/borgmatic hosting providers to documentation. 2019-05-18 20:59:50 -07:00
Dan Helfman 8b41e58e1f Mention isort import ordering in documentation. 2019-05-16 12:06:55 -07:00
Dan Helfman 9417359da3 Fix for regression with missing verbose log entries (#177). 2019-05-16 10:50:19 -07:00
Dan Helfman 1cf0e1bd84 Support for various Borg directory environment variables (#153). 2019-05-16 10:34:52 -07:00
Dan Helfman 223f803e87 Fix formatting. 2019-05-14 13:09:36 -07:00
Dan Helfman 6cb901d083 Bump version for release. 2019-05-14 13:07:49 -07:00
Dan Helfman 096be14230 Run tests for all installed versions of Python (#166). 2019-05-14 20:06:08 +00:00
Dan Helfman bb8b1e58e8 PR feedback: Consistency. 2019-05-14 12:19:56 -07:00
Dan Helfman 06261d8c86 Merge branch 'master' into tox-skip-missing-interpreters 2019-05-14 12:18:30 -07:00
Dan Helfman 869cccf884 Upgrade pip to a particular version during local test and CI. 2019-05-14 19:17:30 +00:00
Dan Helfman 0defaf9cb5 Run tests for all installed versions of Python (#166). 2019-05-14 12:09:07 -07:00
Dan Helfman 60b1f9921d Don't use pip wrapper script in CI. 2019-05-14 12:01:40 -07:00
Dan Helfman f61bc91b0f Merge branch 'master' into upgrade-pip 2019-05-14 10:31:03 -07:00
Dan Helfman ed2c6053de Upgrade pip to a particular version during local test and CI. 2019-05-14 10:28:04 -07:00
Dan Helfman 2cffa8deaa Add missing ticket number to NEWS item. 2019-05-14 10:07:46 -07:00
Dan Helfman f0581271f6 Automatically sort Python imports in code. 2019-05-14 10:02:41 -07:00
Dan Helfman 99522234ea Automatically sort Python imports in code. 2019-05-14 17:02:37 +00:00
Dan Helfman 67f2862fb1 Change paths to reflect new pip install --user documentation. 2019-05-14 10:00:50 -07:00
Luke Murphy 1c0dc3f904
Run isort over the source 2019-05-14 18:59:19 +02:00
Luke Murphy b94dbff216
Add isort configuration
Closes witten/borgmatic#169.
2019-05-14 18:59:19 +02:00
Dan Helfman 7388c723cd Mention tox.ini refactoring in NEWS. 2019-05-14 09:45:39 -07:00
Dan Helfman 128be3c17d Factor out build/test configuration from tox.ini file. 2019-05-14 16:45:24 +00:00
Luke Murphy 4c30c94258
Add workaround for editable failure
See witten/borgmatic#165 (comment).
2019-05-14 13:17:15 +02:00
Luke Murphy 20b8b45aeb
Remove all configuration from Tox file
This puts tool configuration in their familiar and standard
locations and simplifies the Tox configuration to just laying
out the environments and factors.

This also allows users who do not want to deal with overhead of Tox (for
whatever reasons ...) to run pytest/black/etc. and have the same
behaviour.
2019-05-14 13:17:15 +02:00
Dan Helfman 2dd899f287 Linkify build status badge. 2019-05-13 22:33:28 -07:00
Dan Helfman a13cc0ab17 More tests for colored logging. 2019-05-13 21:10:26 -07:00
Dan Helfman 620f9e64d6 A few more tests for new colored logging. 2019-05-13 20:49:20 -07:00
Dan Helfman 25c320b281 Pin pip version: cherrypick of witten/borgmatic#172 2019-05-13 20:01:25 -07:00
Dan Helfman f19eec56ac Add tox pin to NEWS. 2019-05-13 17:07:20 -07:00
Dan Helfman 7cbcff2e9b Pin tox version. 2019-05-14 00:06:07 +00:00
Dan Helfman 9f6407ada6 Mention continuous integration badge in NEWS. 2019-05-13 14:54:24 -07:00
Dan Helfman e933ecf046 Add drone CI note and badge. 2019-05-13 21:51:00 +00:00
Luke Murphy 4010a2ed77
Add note about Drone CI to contributing docs 2019-05-13 23:43:11 +02:00
Luke Murphy 2f36096e1a
Add Drone build badge 2019-05-13 23:42:11 +02:00
Luke Murphy 82ec45e375
Pin tox version
Towards more reproducible results with Tox.
2019-05-13 23:12:18 +02:00
Dan Helfman 37362150fe Color records that are logged via logger.handle() as well. 2019-05-13 13:50:32 -07:00
Dan Helfman a7ba97803f Add colored output to NEWS. 2019-05-13 19:52:54 +00:00
Dan Helfman 31dc903877 Integrate colorama for colored output. 2019-05-13 19:50:36 +00:00
Dan Helfman 8943867433 Bump to dev version. 2019-05-13 19:07:27 +00:00
Dan Helfman d9cb110563 Document installing borgmatic with pip install --user instead of a system Python install. 2019-05-13 19:06:42 +00:00
Dan Helfman 32113cee67 Document installing borgmatic with pip install --user instead of a system Python install. 2019-05-13 19:04:24 +00:00
Luke Murphy a621ce199a
Add tests for borgmatic.logger.to_bool 2019-05-13 13:40:23 +02:00
Luke Murphy 1f524d6c87
Add borgmatic custom logger 2019-05-13 13:40:18 +02:00
Luke Murphy 0320d449ec
Add documentation about colorama 2019-05-13 13:40:17 +02:00
Luke Murphy 30f007687a
Add colorama to testing dependencies 2019-05-13 13:40:17 +02:00
Luke Murphy adf7856162
Add new colorama dependency 2019-05-13 13:40:17 +02:00
Luke Murphy f9dce8b2d3
Recommend user installs when upgrading 2019-05-13 13:18:59 +02:00
Luke Murphy 15cb6270ef
Recommend a tox user install for developing 2019-05-13 13:18:45 +02:00
Luke Murphy ed14fdbac9
Recommend root user package install
This can do bad things to a system Python install. So, we try to
mitigate this by recommending a root user user site installation.
2019-05-13 13:18:37 +02:00
Dan Helfman 8650a15db1 Document validate-borgmatic-config and add a few tests. 2019-05-11 14:05:16 -07:00
Dan Helfman 6a10022543 Add validate-borgmatic-config command. 2019-05-11 20:15:06 +00:00
Robin Schneider 52e4f48eb9
Add validate-borgmatic-config command
Useful when generating the borgmatic configuration file with
configuration management and before moving the generated file in place
checking if it is actually valid.
2019-05-10 00:10:28 +02:00
Dan Helfman f5e1e8bec9 In continuous integration build matrix, use newer Alpine 3.9 instead of 3.8. 2019-05-07 16:19:03 -07:00
Dan Helfman a291477c19 Fix for hooks executing when using --dry-run (#160). 2019-05-07 16:06:31 -07:00
Dan Helfman 1c88dda76a Fix for invalid JSON output when using multiple borgmatic configuration files (#155). 2019-04-02 22:30:14 -07:00
Dan Helfman 0b59c22c23 Fix for seemingly random filename ordering when running through a directory of configuration files (#157). 2019-03-30 16:41:21 -07:00
Dan Helfman 576377e2b2 Clarify differences between Docker images. 2019-03-16 15:04:48 -07:00
Dan Helfman 6ff1867312 Configuration files includes and merging (#148). 2019-03-06 12:06:27 -08:00
Dan Helfman 3cb52423d2 Support for Borg create/extract --numeric-owner flag (#147). 2019-03-05 09:11:35 -08:00
Dan Helfman 5a5b6491ac Add note about uncommenting section names. 2019-03-04 15:15:49 -08:00
Dan Helfman 4272c6b077 List the files within an archive via --list --archive option (#140). 2019-02-23 23:02:17 -08:00
Dan Helfman 26071de2e7 Update extraction docs. 2019-02-18 22:43:32 -08:00
Dan Helfman fe92d9e838 Fix restore paths list to tuple conversion. 2019-02-18 21:59:09 -08:00
Dan Helfman 5ea2d644a2 Fix error handling when --extract repository guard fails. 2019-02-18 21:52:56 -08:00
Dan Helfman c35f90154f Only guard repository when --extract is used. 2019-02-18 21:43:30 -08:00
Dan Helfman 36305c50b1 Update push script to support branches. 2019-02-18 13:51:33 -08:00
Dan Helfman 2b3b8eab71 Add archive extract to end-to-end test. 2019-02-18 13:47:18 -08:00
Dan Helfman aa7c7651e5 Fix config repositories consumption. 2019-02-18 13:27:35 -08:00
Dan Helfman c41ffb5ceb If no extract repository is given, then error if there are multiple configured repositories. 2019-02-18 13:22:14 -08:00
Dan Helfman 766a03375a Guard that the given repository occurs in config exactly once. 2019-02-18 12:58:39 -08:00
Dan Helfman 2a4d4247e3 Tests for extract_archive(). 2019-02-18 10:31:52 -08:00
Dan Helfman 9de5083a7e Additional test coverage for extract options in borgmatic command. 2019-02-18 09:52:56 -08:00
Dan Helfman d0557b2bcd Initial work on #123: Support for Borg extract. 2019-02-18 09:30:34 -08:00
Dan Helfman 1a980d6321 Organize options within command-line help into logical groups. 2019-02-12 22:27:04 -08:00
Dan Helfman fb21d4e645 Remove dead code. 2019-02-09 21:17:55 -08:00
Dan Helfman 5933a4d778 Note tests exclusion in changelog. 2019-02-08 20:59:50 -08:00
Dan Helfman 8cf16c7831 Exclude tests from distribution packages. 2019-02-09 05:02:19 +00:00
Julien Nicoulaud fcf4e03c2f
exclude tests from distribution packages 2019-02-08 19:22:56 +01:00
Dan Helfman d1b29e82da borgmatic command-line reference. 2019-02-04 22:27:40 -08:00
Dan Helfman 290e969a22 Include a sample borgmatic configuration file in the documentation (#119). 2019-02-04 22:12:33 -08:00
Dan Helfman 18ae91ea6e Strike some unnecessary words from docs. 2019-02-04 20:58:27 -08:00
Dan Helfman 0bce77a2ac Distribute troubleshooting among relevant how-to guides. 2019-02-04 20:53:47 -08:00
Dan Helfman 19155607af Include link to development how-to. 2019-02-03 22:42:33 -08:00
Dan Helfman f357c37e2c Fix/remove some documentation links. 2019-02-03 22:35:38 -08:00
Dan Helfman 2980c14728 Fix README links on GitHub. 2019-02-03 22:26:39 -08:00
Dan Helfman 7e0e00d45d Refactor documentation into multiple separate pages for clarity and findability. 2019-02-03 22:20:59 -08:00
Dan Helfman 8b4ac0017b Fix ticket number in changelog. 2019-01-27 14:00:24 -08:00
Dan Helfman 8ec1ec527e Bump version for release. 2019-01-27 13:54:26 -08:00
Dan Helfman 6096a7181c Leave exclude_patterns glob expansion to Borg, since doing it in borgmatic leads to confusing behavior (#132). 2019-01-27 13:47:26 -08:00
Dan Helfman fa9dfb8ff7 Remove date echo from schema example, as it's not a substitute for real logging (#127). 2019-01-27 12:22:22 -08:00
Dan Helfman 2dc006aab4 Allow use of --stats flag when --create or --prune flags are implied (#139). 2019-01-27 12:15:47 -08:00
Dan Helfman 031b9d6faf Handle and format validation errors raised during argument parsing (#136). 2019-01-27 11:58:04 -08:00
Dan Helfman d9018a47f6
Add link to openSUSE packages from README. 2019-01-21 09:57:05 -08:00
Antonio Larrosa e893a20dfd
Add link to openSUSE packages
Add a link to the software.opensuse.org page were both official and community packages of borgmatic are available to be downloaded or installed using 1-click-install.
2019-01-21 13:13:40 +01:00
Dan Helfman 09d521661f Remove weasel words like "easily" and "simply". 2018-12-25 22:03:34 -08:00
Dan Helfman fd46efb193 Add borgmatic --version command-line flag to get the current installed version number. 2018-12-25 21:01:08 -08:00
Dan Helfman 426f54c9cc When generating sample configuration, document the defaults for each option (#103). 2018-12-25 17:05:22 -08:00
Dan Helfman 45a537b6b1 When running multiple configuration files, attempt all of them even if one errors (#116). 2018-12-25 15:23:54 -08:00
Dan Helfman d6feca169c Fix duplicate issue number. 2018-12-24 22:35:16 -08:00
Dan Helfman 05e2900ab0 Rev version. 2018-12-24 22:29:41 -08:00
Dan Helfman 30b52e5523 With --init command-line flag, if a repository already exists, proceed without erroring (#117). 2018-12-24 22:28:02 -08:00
Dan Helfman 14aeddc11f Black re-formatting. 2018-12-24 14:38:57 -08:00
Dan Helfman 066399ecdb Mention --stats command-line flag in NEWS file. 2018-12-23 16:06:08 -08:00
Dan Helfman d4bbac4467 Support for --stats command-line flag independent of --verbosity (#100). 2018-12-24 00:04:23 +00:00
Felix Buehler 7516443a89 fix changes requested about stats 2018-12-22 23:46:03 +01:00
Felix Buehler 73d67e29b4 Support for Borg create & prune --stats via borgmatic command-line flag (#100) 2018-12-22 23:27:24 +01:00
Dan Helfman c3e7425f4c Some late-breaking README additions (new borgmatic packages). 2018-12-10 22:30:10 -08:00
Dan Helfman cc9dbb1def Support for Borg repository initialization via borgmatic --init command-line flag (#110). 2018-12-10 22:20:57 -08:00
Dan Helfman 2045edc11b Fix warning about classifiers as tuple. 2018-12-09 15:49:58 -08:00
Dan Helfman 1dcac44d6c Fix broken test of deprecated --excludes option. 2018-12-09 15:49:05 -08:00
Dan Helfman 300ead65d3 Error when deprecated --excludes command-line option is used. 2018-12-09 14:57:14 -08:00
Dan Helfman 6a0219a7a4 Update README with link to a new/forked Docker image (#113). 2018-12-02 15:16:52 -08:00
Dan Helfman 80c69aac05 Fix incomplete test coverage around --progress argument validation. 2018-12-02 15:08:42 -08:00
Dan Helfman 7417a3cd00 Update Borg create --filter values so a dry run lists files to back up. (#111). 2018-12-02 15:03:07 -08:00
Dan Helfman 9ca80a54d8 Support for Borg create --progress via borgmatic command-line flag (#108). 2018-11-21 22:03:39 -08:00
Dan Helfman 5c0b17ef39 Support for Borg --chunker-params create option via "chunker_params" in borgmatic's storage section (#105). 2018-10-27 15:57:28 -07:00
Dan Helfman 1697d8aaef Silence curl when posting release description to Gitea. 2018-10-15 22:32:13 -07:00
Dan Helfman fef441a8ff More concessions for Python 3.5 compatibility. 2018-10-15 09:37:26 -07:00
Dan Helfman c1ddc4268b We can't have nice things. 2018-10-15 09:30:04 -07:00
Dan Helfman e323290e61 Switch from bash to sh for black wrapper. 2018-10-15 09:25:57 -07:00
Dan Helfman 1ab44d4201 Wrap black with script that skips it if Python version < 3.6. 2018-10-15 09:20:35 -07:00
Dan Helfman 71b1c3dfb0 Make automated tests support running in Python 3.5. 2018-10-15 09:04:29 -07:00
Dan Helfman 695930a607 Fix for syntax error that occurred in Python 3.5 and below (#102). 2018-10-15 08:47:15 -07:00
Dan Helfman eb2a4ff1f0 Add Python 3.5 to continuous integration. 2018-10-15 08:17:34 -07:00
Dan Helfman 531d5c80c0 Fix quoting and escaping in release script. 2018-10-14 12:14:29 -07:00
Dan Helfman 067ed27689 Rev to 1.2.8. 2018-10-14 11:45:34 -07:00
Dan Helfman fa38de2de7 Enable consistency checks for only certain repositories via "check_repositories" (#73). 2018-10-13 20:34:51 -07:00
Dan Helfman e4d1b49c39 Switch some functions with many arguments to kwargs only. 2018-10-13 15:19:16 -07:00
Dan Helfman af7caec509 Mention minimum Borg version to install in README. 2018-10-13 13:35:42 -07:00
Dan Helfman 90c1f899fc Use newer Alpine (with newer version of Borg) in matrix builds. 2018-10-13 13:35:18 -07:00
Dan Helfman a0691ae4cd Run continuous integration tests on a matrix of Python and Borg versions. 2018-10-13 13:09:12 -07:00
Dan Helfman 2f20e6f808 Include link to issue tracker within various command output. 2018-10-07 22:29:56 -07:00
Dan Helfman 7a4636ae0f Remove curl --verbose in release script. 2018-10-06 22:35:00 -07:00
Dan Helfman 53435dcc3e Post release changelogs to projects.evoworx.org. 2018-10-06 22:24:46 -07:00
Dan Helfman 4d01278037 Update release file to post changelogs to GitHub release descriptions. 2018-10-06 15:18:21 -07:00
Dan Helfman 2299e5d41e Additional dependency version pins in test requirements. 2018-10-06 14:17:47 -07:00
Dan Helfman d16f5d5df3 Add backticks to path literal in README. 2018-10-06 13:23:54 -07:00
Dan Helfman da8e9638f4 Support for Borg --keep-secondly prune option (#98). 2018-10-04 21:54:23 -07:00
Dan Helfman 900ea80a42 Hack to uncomment all options in config file used for finding unsupported Borg options. 2018-10-04 21:45:31 -07:00
Dan Helfman 4b92d0f685 Remove unneeded Dockerfile for end-to-end tests. 2018-10-03 22:44:23 -07:00
Dan Helfman 3ce5533103 Make end-to-end test clean up after itself, and drop unnecessary use of Docker for it. 2018-10-03 22:36:25 -07:00
Dan Helfman 4a1ee8c911 Pull new base Docker images during CI. 2018-10-03 19:35:42 -07:00
Dan Helfman 3f22a99412 Rev pykwalify. 2018-10-03 08:59:08 -07:00
Dan Helfman caf95cc913 Rebuild. 2018-09-30 22:58:23 -07:00
Dan Helfman fd3130b4d9 Install tox before using it. 2018-09-30 22:47:07 -07:00
Dan Helfman 65bb5a49e2 CI? 2018-09-30 22:43:49 -07:00
Dan Helfman 4bcc517326 Attempted Drone CI configuration. 2018-09-30 22:09:53 -07:00
Dan Helfman 0b164973e0 Add an end-to-end automated test that actually integrates with Borg. 2018-09-30 17:30:04 -07:00
Dan Helfman a125df991b Move tests to the root of the repository, in keeping with more common convention. 2018-09-30 13:57:20 -07:00
Dan Helfman f9a9b42c58 A little introductory text for the screencast. 2018-09-30 11:11:07 -07:00
Dan Helfman 56ad1d164a Use Flake8 code checker as part of running automated tests. 2018-09-29 23:15:18 -07:00
Dan Helfman 3cce18919c Switch Black link to documentation. 2018-09-29 22:46:34 -07:00
Dan Helfman 76d6a69f5a Use Black code formatter as part of running automated tests. 2018-09-29 22:45:00 -07:00
Dan Helfman 3db17277b4 Replace broken screencast thumbnail with embedded player. 2018-09-29 21:38:38 -07:00
Dan Helfman ece49eb500 Update screencast. 2018-09-29 18:56:39 -07:00
Dan Helfman 746428ed44 Fix generated configuration to also include a "keep_daily" value so pruning works out of the box. 2018-09-29 15:44:37 -07:00
Dan Helfman 984702b3b2 Fix various warnings. 2018-09-29 15:06:57 -07:00
Dan Helfman 1bc71e1c5d Upgrade test requirements. 2018-09-29 15:04:42 -07:00
Dan Helfman 47efa88c9d In generate-borgmatic-config, comment out all optional config (#57). 2018-09-29 15:03:11 -07:00
Dan Helfman 3821636b77 Bump version. 2018-09-27 08:13:23 -07:00
Dan Helfman 596f6f9dac Update help/README about --create --json. 2018-09-27 08:12:54 -07:00
Dan Helfman 7ecdaea83a
Fix check_archives does not take json parameter. 2018-09-27 08:09:23 -07:00
Nils Hesse 98cb2644db
check_archives does not take json parameter 2018-09-27 12:21:14 +02:00
Dan Helfman 31db6faa19 Set to release version. 2018-09-26 21:32:28 -07:00
Dan Helfman 872d8b695a Flesh out NEWS line item a bit. 2018-09-24 21:37:45 -07:00
Dan Helfman 6db3e1dda5 Merge branch 'master' of floli/borgmatic into master 2018-09-25 04:36:09 +00:00
Florian Lindner 9aaf78b9dd Add --json option for --create command line.
Closes #94.
2018-09-24 21:53:09 +02:00
Dan Helfman 5d8ac158ce Merge ssh://projects.torsion.org:3022/witten/borgmatic 2018-09-17 22:34:08 -07:00
Dan Helfman d32a53d58f Mention log level fix in NEWS. 2018-09-17 22:33:34 -07:00
floli a836ec944f Limit argument range for --verbose, make default log level more explicit. (#93) 2018-09-18 05:31:27 +00:00
Dan Helfman e7b128e735 --read-special is now supported. 2018-09-09 11:21:06 -07:00
Dan Helfman ff3cb1d80f Attach #64 to logging rewrite in NEWS. 2018-09-09 11:18:26 -07:00
Dan Helfman c5ff08ee25 Remove now-gone verbosity parameter from test. 2018-09-09 11:14:33 -07:00
Dan Helfman 856db29180 Mention --read-special in NEWS. 2018-09-09 10:42:06 -07:00
Steve Kerrison 20e09b4ea8 Support for Borg create --read-special via "read_special" option (#25). 2018-09-09 10:39:56 -07:00
Dan Helfman 1dd0682661 Merge branch 'master' of ssh://projects.torsion.org:3022/witten/borgmatic 2018-09-08 13:54:18 -07:00
floli 7252b8d614 Rework logging/verbosity system (#90)
Looks great, merged! Thanks again for all your hard work here.
2018-09-08 20:53:37 +00:00
grerrg e5870a169b Add example for cron in Alpine Linux (#24) 2018-09-05 21:58:30 -07:00
Dan Helfman 94795a3560 Link to Borg home page instead of docs. 2018-09-02 22:06:57 -07:00
Dan Helfman 7705debab0 Switching back to table-like 11ty front matter. It looks less bad than JSON. 2018-09-01 22:45:13 -07:00
Dan Helfman f87df0527f Adding JSON front matter for 11ty. 2018-09-01 22:38:17 -07:00
Dan Helfman e4512a40e0 Removing 11ty front matter out of README since it renders as a table on GitHub. 2018-09-01 22:11:38 -07:00
Dan Helfman 1d4a9510b8 Upgrade pytest. 2018-09-01 20:29:05 -07:00
Dan Helfman 2648f07e7a Add missing syntax highlighting. 2018-08-29 23:01:11 -07:00
Dan Helfman 459bf1fcf6 Document --list and --info flags. 2018-08-29 22:57:32 -07:00
Dan Helfman 3930e63320 Merge branch 'master' of ssh://projects.torsion.org:3022/witten/borgmatic 2018-08-29 22:44:45 -07:00
Dan Helfman acecb1e397 README metadata changes to support 11ty static site generator. 2018-08-29 22:44:12 -07:00
Dan Helfman 9b48eb5a61 Clarify that --json can be used with --info command-line flag. 2018-08-19 12:57:52 -07:00
Dan Helfman 7d40a448cb Pass --show-rc option to Borg when at highest verbosity level (#89). 2018-08-19 12:44:40 -07:00
Dan Helfman da7aed3814 Support for Borg create --checkpoint-interval (#87). 2018-08-19 11:41:49 -07:00
Dan Helfman c7f4200417 Somewhat more robust mechanism to find unsupported Borg arguments. 2018-08-19 11:24:48 -07:00
Dan Helfman 5e2a5494af Fix declared pykwalify compatibility version range in setup.py (#88). 2018-08-18 14:07:18 -07:00
Dan Helfman 7b77fd2510 Fix compatibility issue between pykwalify and ruamel.yaml 0.15.52 (#85). 2018-08-11 13:59:27 -07:00
Dan Helfman ece5608677 Rev for release. 2018-07-28 22:27:39 -07:00
Dan Helfman 4644f613b2 Fix typo in README. 2018-07-28 22:24:24 -07:00
Dan Helfman 3afa5ac76d Document hooks (#81). 2018-07-28 22:22:25 -07:00
Dan Helfman 27f8a1df04 Switch to non-raw link to sample cron job. 2018-07-28 20:29:55 -07:00
Dan Helfman 8e5b0bbf17 Remove errant ctrl-F character from docs. 2018-07-28 20:27:18 -07:00
Dan Helfman 282e9565c9 Mentioning new --info --json option in NEWS. 2018-07-28 20:24:19 -07:00
Thomas LÉVEIL b714ffd48b add support for `--info --json` (#83) 2018-07-29 03:17:45 +00:00
Dan Helfman 9968a15ef8 Clarifying code style for multiline constructs. 2018-07-28 15:21:19 -07:00
Dan Helfman d93da55ce9 Add code style guidelines to the documention, and reformat some code accordingly. 2018-07-28 15:02:17 -07:00
Thomas LÉVEIL 789bcd402a add support for `--list --json` (#74) 2018-07-28 21:21:38 +00:00
Dan Helfman cf6ab60d2e Use XDG_CONFIG_HOME for user configuration directory, if set. (Thanks to floli.) (#71)
Thanks! This will go out in the next release.
2018-07-25 01:34:05 +00:00
Dan Helfman 64364b20ff Skip non-"*.yaml" config filenames in /etc/borgmatic.d/ so as not to parse backup files, editor swap files, etc. (#77) 2018-07-22 12:08:49 -07:00
Dan Helfman d29c7956bc Upgrade ruamel.yaml compatibility version range and fix support for Python 3.7 (#38, #76). 2018-07-22 11:25:06 -07:00
Dan Helfman e5ef485d6b Merge branch 'master' of ssh://projects.torsion.org:3022/witten/borgmatic 2018-07-01 14:54:15 -07:00
Dan Helfman fc8046edc4 Adding NEWS item about skipping before/after backup hooks. 2018-07-01 14:51:57 -07:00
Dan Helfman 4538017206 Merge branch 'fix-72-hooks-are-executed-when-list-or-info' of thomasleveil/borgmatic into master
Thanks for fixing this!

I agree about more specific hooks if and when `--check` or `--prune`-specific hook use cases are needed. I think what you've done here is fine until then.
2018-07-01 21:47:39 +00:00
Thomas LÉVEIL d664b6d253 only run hooks when creating an archive
fix #72
2018-07-01 21:09:45 +02:00
Dan Helfman f42aa0a6f2 Revving version for development. 2018-06-17 15:26:53 -07:00
Dan Helfman 9d4ba66f6e Revving version for release. 2018-06-17 15:14:45 -07:00
Dan Helfman cf846ab8ac Support for Borg prune --umask option (#69). 2018-06-17 15:12:43 -07:00
Dan Helfman 219e287c6c Document how to develop on and contribute to borgmatic. 2018-06-17 14:55:57 -07:00
Dan Helfman dede8f9d4b News for: ~/.config/borgmatic/config.yaml. 2018-06-17 14:30:47 -07:00
Dan Helfman 7a1e3f5639 Merge branch 'add_user_config' of floli/borgmatic into master 2018-06-17 21:26:36 +00:00
Florian Lindner 9bd77292ff Add default path for user configuration 2018-06-10 15:03:23 +02:00
Dan Helfman f1a143de5b Adding list and info Borg sub-commands to find-unsupported-borg-options script. 2018-05-26 20:53:03 -07:00
Dan Helfman 765e343c71 Support for Borg --info via borgmatic command-line (#61). 2018-05-26 16:19:05 -07:00
Dan Helfman af4b91a048 Support for Borg --list option via borgmatic command-line to list all archives (#61). 2018-05-26 16:09:08 -07:00
Dan Helfman cc9044487b Support for Borg --nobsdflags option to skip recording bsdflags (e.g. NODUMP, IMMUTABLE) in archive (#63). 2018-05-26 15:09:23 -07:00
Dan Helfman 11c30001c3 Add "Persistent" flag to systemd timer example. (#60) 2018-05-20 22:20:21 -07:00
Dan Helfman ac9161035a Merge branch 'master' of floli/borgmatic into master
Thanks for taking the time to add this! Makes sense.
2018-05-21 05:18:13 +00:00
Dan Helfman 007ec0644c Ignore "check_last" and consistency "prefix" when "archives" not in consistency checks. (#59) 2018-05-20 22:11:40 -07:00
Dan Helfman 1db808fb3d Link to OpenBSD port of borgmatic. 2018-05-19 16:16:54 -07:00
Dan Helfman 76656275c3 Update README to mention other ways of installing borgmatic. (#62) 2018-05-19 16:06:54 -07:00
Florian Lindner 64bdbc4bf0 Add Persistent, so that the timer is triggered if missed last time. 2018-05-17 21:47:58 +02:00
Dan Helfman 61033bb4e5 Update tox.ini to only assume Python 3.x instead of Python 3.4 specifically. 2018-04-09 20:34:59 -07:00
Dan Helfman e608b7924a Adding note about executable location. 2018-04-08 12:06:15 -07:00
Dan Helfman f7f852a28b Fix tests broken by addition of check --prefix default. 2018-03-03 22:36:51 -08:00
Dan Helfman 9b9c4c4abb Clarifying note in schema about adding prefix to consistency section. 2018-03-03 22:33:34 -08:00
Dan Helfman 1b59f5b190 Changing version in warning to correspond with next release version. 2018-03-03 22:30:30 -08:00
Dan Helfman 65ab230961 Noting new Borg check --prefix feature in release notes. 2018-03-03 22:21:48 -08:00
Nick Whyte c64d0100d5 Only check archives with matching prefix. 2018-03-03 22:17:39 -08:00
Dan Helfman 0112407250 Add introductory screencast link to documentation. 2018-02-19 17:44:20 -08:00
Dan Helfman 2d3f5fa05d Support for Borg --lock-wait option for the maximum wait for a repository/cache lock (#56). 2018-02-19 15:51:04 -08:00
Dan Helfman a87036ee46 Support for using tilde in exclude_patterns to reference home directory (#58). 2018-02-18 15:34:19 -08:00
Dan Helfman a72f5ff69a Tests for --dry-run + --verbosity fix. 2018-02-18 14:26:51 -08:00
newtonne bb99009191 Fix issue when using both --dry-run and -v options. 2018-02-18 14:18:25 -08:00
Dan Helfman 4c45d60529 Mentioning ssh_command for additional SSH configuration.. 2018-02-18 13:39:05 -08:00
Dan Helfman 2211f959ae AUTHORS addition for recent encryption pass command changes. 2018-01-18 21:13:20 -08:00
Dan Helfman cc1d6f53a0 55: Fix for missing tags/releases from Gitea and GitHub project hosting. 2018-01-17 20:27:09 -08:00
Dan Helfman 389778c716 Adding BORG_PASSCOMMAND update to NEWS. 2018-01-16 21:05:53 -08:00
newtonne e55e9e8139 Add `encryption_passcommand` configuration option 2018-01-16 21:03:25 -08:00
Dan Helfman ef76e87477 Bumping version for release. 2018-01-15 20:55:49 -08:00
Dan Helfman 62526038d6 47: Support for Borg --dry-run option via borgmatic command-line. 2018-01-15 20:55:27 -08:00
Dan Helfman bf2f39623e 49: Rename incorrect --pattern-from option to correct --patterns-from. 2018-01-15 20:22:53 -08:00
Dan Helfman 28c890a52d Bumping version for release. 2018-01-14 16:37:02 -08:00
Dan Helfman cd189c4fe4 48: Add "local_path" to configuration for specifying an alternative Borg executable path. 2018-01-14 16:35:24 -08:00
Dan Helfman b8f6bab12d 49: Support for Borg experimental --patterns-from and --patterns options for specifying mixed includes/excludes. 2018-01-14 15:52:19 -08:00
Dan Helfman 50b3240c4f 54: Fix for incorrect consistency check flags passed to Borg when all three checks in borgmatic config. 2018-01-14 14:09:20 -08:00
Dan Helfman 18fbc75e16 Revising history to account for off-by-one error when importing issue numbers into Gitea. 2018-01-04 21:49:43 -08:00
Dan Helfman 0881da4a82 New issue tracker. 2018-01-04 21:37:43 -08:00
Dan fa210766a2 Update for release. 2018-01-02 20:36:52 -08:00
Dan d4f52e3137 Update AUTHORS with recent changes + sort. 2018-01-02 20:33:27 -08:00
Dan 8b2ebdc5f7 Simplifying example. 2018-01-02 20:31:06 -08:00
Dan Helfman a00407256d Merge branch 'keep_minutely' of thomasleveil/borgmatic into master 2018-01-03 04:30:13 +00:00
Thomas LÉVEIL 24b5eccefc add support for Borg `--keep-minutely` prune option 2018-01-03 00:13:44 +01:00
Dan 815fb39a05 Declare dependency on pykwalify 1.6 or above, as older versions yield "Unknown key: version" rule errors. 2017-11-26 10:30:31 -08:00
Dan 24c196d2a4 Script to find unsupported Borg options in borgmatic, to assist with #13. 2017-11-11 16:07:54 -08:00
Dan 3e26e70d0c Fix for incorrect /etc/borgmatic.d/ configuration path probing on macOS. 2017-11-10 21:33:29 -08:00
Dan 5ce25e2790 Re-fixing logo image. 2017-11-04 11:55:48 -07:00
Dan 8243552c8c Fixing PNG path. 2017-11-04 11:54:29 -07:00
Dan 425e27dee5 Add "ssh_command" to configuration for specifying a custom SSH command or options. 2017-11-03 22:01:04 -07:00
Dan 9ec9269a18 Link to repository encryption section of Borg Quick Start. 2017-11-03 20:28:31 -07:00
Dan bf5cbd1deb Mentioning use of BORG_PASSPHRASE environment variable. 2017-11-03 20:27:21 -07:00
Dan 4c09cbf1a4 Releasing. 2017-11-02 22:38:16 -07:00
Dan fc077af4ce Mentioning borgmatic --config option in docs. 2017-11-02 22:28:53 -07:00
Dan ca4312bb85 Support for Borg --remote-ratelimit for limiting upload rate. And log Borg commands. 2017-11-02 22:22:40 -07:00
Dan fc3b1fccba Support for Borg --files-cache option for setting the files cache operation mode. 2017-11-02 22:03:11 -07:00
Dan f83346b9b3 Support for using tilde in repository paths to reference home directory. 2017-11-02 21:34:04 -07:00
Dan 63c7241aec Typo in comment. 2017-10-31 22:21:49 -07:00
Dan fd77dc579e Pass through several more Unix signals that Borg happens to consume. 2017-10-31 22:10:00 -07:00
Dan f017ed648f Disabling code coverage on this one-line functions. 2017-10-31 22:01:18 -07:00
Dan 27a6745743 Passing the Unix SIGTERM signal through to child processes like Borg. 2017-10-31 21:58:35 -07:00
Dan 95be0c8e46 Removing broken download URL. 2017-10-29 21:42:01 -07:00
Dan 17a774ba7e Removing .hg* files. 2017-10-29 20:32:21 -07:00
Dan a1d2bd173b Bumping version for release. 2017-10-29 20:29:53 -07:00
Dan f495550ad7 Default "prefix" to "{hostname}-" if not specified. 2017-10-29 20:14:18 -07:00
Dan 43d0e597a2 Require "prefix" in retention section when "archive_name_format" is set. 2017-10-29 19:36:26 -07:00
Dan f1c07b5cf5 Updated dead links to Borg documentation. 2017-10-29 17:05:49 -07:00
Dan f2782426d5 Comment typo. 2017-10-29 16:46:30 -07:00
Dan f13ed92b0e Breaking borgmatic command main() apart, since it was getting a little unwieldy. 2017-10-29 16:44:15 -07:00
Dan 6e9e7c45d7 Being explicit about markdown syntax highlighting in README. 2017-10-28 10:45:27 -07:00
Dan c1ca4b9421 Using absolute path for logo. 2017-10-28 10:33:36 -07:00
Dan 469feadbc0 Adding missing PNG logo. 2017-10-28 10:31:30 -07:00
Dan a5403a4373 Switched logo from SVG to PNG for compatibility reasons. 2017-10-28 10:29:18 -07:00
Dan 56c902258d Setting up download URL for new hosting location. 2017-10-27 22:40:34 -07:00
Dan 9c1660f467 Fix typo in README. 2017-10-27 22:27:28 -07:00
Dan dd926b5762 Updating links. 2017-10-27 22:26:33 -07:00
Dan 9d03351b5d Converted main source repository from Mercurial to Git. 2017-10-27 21:55:08 -07:00
Dan 719d9a9835 Merge branch 'master' of github.com:witten/borgmatic 2017-10-27 21:51:34 -07:00
Dan 731c8c9ad9 Adding push and release scripts. 2017-10-27 21:51:10 -07:00
Dan Helfman 2ae8ac2947 Add tests for verbosity mapping. 2017-10-25 22:36:23 -07:00
Dan Helfman cc94e5f52f Add tests for verbosity mapping. 2017-10-25 22:36:23 -07:00
Dan Helfman a09c9f248e Adding logging to hook execution! 2017-10-25 22:32:06 -07:00
Dan Helfman 16f0a3976c Adding logging to hook execution! 2017-10-25 22:32:06 -07:00
Dan Helfman cc78223164 Fixing inconsistent indentation. 2017-10-25 21:58:02 -07:00
Dan Helfman 30f56235c1 Fixing inconsistent indentation. 2017-10-25 21:58:02 -07:00
Dan Helfman 7458769cb3 Merge. 2017-10-25 21:54:50 -07:00
Dan Helfman a5aa9355f5 Merge. 2017-10-25 21:54:50 -07:00
Dan Helfman 5c229639f0 Improve clarity of logging spew at high verbosity levels. 2017-10-25 21:47:33 -07:00
Dan Helfman 059322b7f8 Improve clarity of logging spew at high verbosity levels. 2017-10-25 21:47:33 -07:00
Johannes Feichtner f1a98d82c6 #16, #38: Support for user-defined hooks before/after backup, or on error. 2017-10-25 21:38:27 -07:00
Johannes Feichtner 80e2c023dd #16, #38: Support for user-defined hooks before/after backup, or on error. 2017-10-26 06:38:27 +02:00
b3vis 86511deac4 Added section about docker (#18) 2017-10-25 21:24:24 -07:00
b3vis bb3475b3f8 Added section about docker (#18) 2017-10-26 05:24:24 +01:00
Dan Helfman bd196c1fb9 Removing "from __future__ import print_function". This isn't Python 2 anymore, Toto. 2017-09-09 17:38:14 -07:00
Dan Helfman 873fc22cfb Removing "from __future__ import print_function". This isn't Python 2 anymore, Toto. 2017-09-09 17:38:14 -07:00
Dan Helfman f3d6d7c0a3 #29: Support for using tilde in source directory path to reference home directory. 2017-09-09 17:23:31 -07:00
Dan Helfman 86cc6ca869 #29: Support for using tilde in source directory path to reference home directory. 2017-09-09 17:23:31 -07:00
Dan Helfman d30caa422e #39: Fix to make /etc/borgmatic/config.yaml optional rather than required when using the default config paths. 2017-09-08 21:25:42 -07:00
Dan Helfman 84c148fb3b #39: Fix to make /etc/borgmatic/config.yaml optional rather than required when using the default config paths. 2017-09-08 21:25:42 -07:00
Dan Helfman 6c4f641c1e Added tag 1.1.7 for changeset ec7949a14a20 2017-09-03 11:33:10 -07:00
Dan Helfman b44bc57548 Added tag 1.1.7 for changeset ec7949a14a20 2017-09-03 11:33:10 -07:00
Dan Helfman bb18a9a3f2 Update NEWS and AUTHORS for release. 2017-09-03 11:33:07 -07:00
Dan Helfman f7dcbe40d4 Update NEWS and AUTHORS for release. 2017-09-03 11:33:07 -07:00
Michele Lazzeri 95533d2b31 Added storage.archive_name_format to config (#16)
* Added storage.archive_name_format to config
2017-09-03 11:13:14 -07:00
Michele Lazzeri 867d3fceb0 Added storage.archive_name_format to config (#16)
* Added storage.archive_name_format to config
2017-09-03 20:13:14 +02:00
Dan Helfman 3af92f8b92 Fix for traceback when "exclude_from" value is empty in configuration file. 2017-08-27 10:01:49 -07:00
Dan Helfman 7c048d1989 Fix for traceback when "exclude_from" value is empty in configuration file. 2017-08-27 10:01:49 -07:00
Dan Helfman d127e73590 Clarification of Python 3 pip usage in documentation. 2017-08-26 16:18:53 -07:00
Dan Helfman 13ba5c84de Clarification of Python 3 pip usage in documentation. 2017-08-26 16:18:53 -07:00
Dan Helfman 50c4f6f2a1 Adding documentation note about pruning happening before archiving. 2017-08-26 16:13:41 -07:00
Dan Helfman 9588e111c4 Adding documentation note about pruning happening before archiving. 2017-08-26 16:13:41 -07:00
Dan Helfman 37ae34a432 When pruning, make highest verbosity level list archives kept and pruned. 2017-08-26 16:07:30 -07:00
Dan Helfman e3a559e13b When pruning, make highest verbosity level list archives kept and pruned. 2017-08-26 16:07:30 -07:00
Dan Helfman 3664ac7418 Added tag 1.1.6 for changeset 4daa944c122c 2017-08-05 23:33:08 -07:00
Dan Helfman 3f83788858 Added tag 1.1.6 for changeset 4daa944c122c 2017-08-05 23:33:08 -07:00
Dan Helfman 10cac46f4c #12, #35: Support for Borg --exclude-from, --exclude-caches, and --exclude-if-present options. 2017-08-05 23:32:39 -07:00
Dan Helfman b1f429f4b5 #12, #35: Support for Borg --exclude-from, --exclude-caches, and --exclude-if-present options. 2017-08-05 23:32:39 -07:00
Dan Helfman 51095cd419 Remove unused imports. 2017-08-05 22:26:38 -07:00
Dan Helfman ddd56bf2a7 Remove unused imports. 2017-08-05 22:26:38 -07:00
Dan Helfman 674a6153f3 Fix imports of borg/*.py modules now that they've been split out. 2017-08-05 22:26:28 -07:00
Dan Helfman 793ffbd048 Fix imports of borg/*.py modules now that they've been split out. 2017-08-05 22:26:28 -07:00
Dan Helfman aa04473521 Split out Borg integration code into multiple files, as it was getting kind of hairy all in one. 2017-08-05 16:21:39 -07:00
Dan Helfman 247d36a309 Split out Borg integration code into multiple files, as it was getting kind of hairy all in one. 2017-08-05 16:21:39 -07:00
Dan Helfman 77d3c66fb9 Added tag 1.1.5 for changeset 64ca13bfe050 2017-07-30 11:16:41 -07:00
Dan Helfman 9f5b808042 Added tag 1.1.5 for changeset 64ca13bfe050 2017-07-30 11:16:41 -07:00
Dan Helfman 9bea7ae5ed #34: New "extract" consistency check that performs a dry-run extraction of the most recent archive. 2017-07-30 11:16:26 -07:00
Dan Helfman e85d487c3a #34: New "extract" consistency check that performs a dry-run extraction of the most recent archive. 2017-07-30 11:16:26 -07:00
Dan Helfman 23679a6edd Removing Pelican-specific title metadata out of README markdown. 2017-07-29 16:05:11 -07:00
Dan Helfman 525ffa6a28 Removing Pelican-specific title metadata out of README markdown. 2017-07-29 16:05:11 -07:00
Dan Helfman 0f44fbedf4 Getting logo to show up on GitHub. 2017-07-28 22:36:16 -07:00
Dan Helfman ac47301a64 Getting logo to show up on GitHub. 2017-07-28 22:36:16 -07:00
Dan Helfman ae15e0f404 Added tag 1.1.4 for changeset 3d605962d891 2017-07-28 22:02:43 -07:00
Dan Helfman 9347c02268 Added tag 1.1.4 for changeset 3d605962d891 2017-07-28 22:02:43 -07:00
Dan Helfman a2e8abc537 #17: Added command-line flags for performing a borgmatic run with only pruning, creating, or checking enabled. 2017-07-28 22:02:18 -07:00
Dan Helfman ceeaf25443 #17: Added command-line flags for performing a borgmatic run with only pruning, creating, or checking enabled. 2017-07-28 22:02:18 -07:00
Dan Helfman 10404143c6 Added tag 1.1.3 for changeset 3f838f661546 2017-07-25 21:21:50 -07:00
Dan Helfman 62d2b267da Added tag 1.1.3 for changeset 3f838f661546 2017-07-25 21:21:50 -07:00
Dan Helfman 94aaf4554f Releasing. 2017-07-25 21:21:47 -07:00
Dan Helfman 03d50d74ca Releasing. 2017-07-25 21:21:47 -07:00
Dan Helfman 0c8816e6cc #14: Support for running multiple config files in /etc/borgmatic.d/ from a single borgmatic run. 2017-07-25 21:18:51 -07:00
Dan Helfman 7ed5b33db5 #14: Support for running multiple config files in /etc/borgmatic.d/ from a single borgmatic run. 2017-07-25 21:18:51 -07:00
Dan Helfman e3e4aeff94 Fix for generate-borgmatic-config writing config with invalid one_file_system value. 2017-07-25 20:32:32 -07:00
Dan Helfman 57b3066987 Fix for generate-borgmatic-config writing config with invalid one_file_system value. 2017-07-25 20:32:32 -07:00
Dan Helfman 89cd879529 Added tag 1.1.2 for changeset f052a77a8ad5 2017-07-24 19:29:28 -07:00
Dan Helfman 1527ff7898 Added tag 1.1.2 for changeset f052a77a8ad5 2017-07-24 19:29:28 -07:00
Dan Helfman 2c61c0bc08 #32: Fix for passing check_last as integer to subprocess when calling Borg. 2017-07-24 19:29:26 -07:00
Dan Helfman 3967e1b5f0 #32: Fix for passing check_last as integer to subprocess when calling Borg. 2017-07-24 19:29:26 -07:00
Dan Helfman bcd8b9982d Added tag 1.1.1 for changeset 7d3d11eff6c0 2017-07-24 08:41:05 -07:00
Dan Helfman 8cbd51512b Added tag 1.1.1 for changeset 7d3d11eff6c0 2017-07-24 08:41:05 -07:00
Dan Helfman b36b923c5d #32: Fix for upgrade-borgmatic-config converting check_last option as a string instead of an integer. 2017-07-24 08:41:02 -07:00
Dan Helfman c38f7a3693 #32: Fix for upgrade-borgmatic-config converting check_last option as a string instead of an integer. 2017-07-24 08:41:02 -07:00
Dan Helfman f44a7884e6 No longer producing univeral (Python 2 + 3) wheel. 2017-07-23 17:34:17 -07:00
Dan Helfman 7c77a5a8a5 No longer producing univeral (Python 2 + 3) wheel. 2017-07-23 17:34:17 -07:00
Dan Helfman b61b09f55c Added tag 1.1.0 for changeset 5a003056a8ff 2017-07-22 23:27:26 -07:00
Dan Helfman 9caaee18b5 Added tag 1.1.0 for changeset 5a003056a8ff 2017-07-22 23:27:26 -07:00
Dan Helfman 588955a467 Setting release version. 2017-07-22 23:27:21 -07:00
Dan Helfman 7c0407ed22 Setting release version. 2017-07-22 23:27:21 -07:00
Dan Helfman ee3edeaac2 Support for backing up to multiple repositories. 2017-07-22 22:56:46 -07:00
Dan Helfman 499f8aa0a4 Support for backing up to multiple repositories. 2017-07-22 22:56:46 -07:00
Dan Helfman 90a0d3b1e0 Renaming group to section for consistency. 2017-07-22 22:17:37 -07:00
Dan Helfman 548212274f Renaming group to section for consistency. 2017-07-22 22:17:37 -07:00
Dan Helfman cd8ceccfaf To free up space, now pruning backups prior to creating a new backup. 2017-07-22 21:50:29 -07:00
Dan Helfman 1292dd2162 To free up space, now pruning backups prior to creating a new backup. 2017-07-22 21:50:29 -07:00
Dan Helfman e5c12fc81c Mentioning test coverage addition in NEWS. 2017-07-22 21:23:01 -07:00
Dan Helfman b02ac44cfc Mentioning test coverage addition in NEWS. 2017-07-22 21:23:01 -07:00
Dan Helfman f5abe05ce9 Instructions to make cron file executable. 2017-07-22 21:20:48 -07:00
Dan Helfman 52963adfc9 Instructions to make cron file executable. 2017-07-22 21:20:48 -07:00
Dan Helfman 6af53d1163 Fixing gets on config group names. 2017-07-22 21:19:26 -07:00
Dan Helfman 2274cfe480 Fixing gets on config group names. 2017-07-22 21:19:26 -07:00
Dan Helfman 3cccac8cb1 Mentioning libyaml compile errors in troubleshooting. 2017-07-22 21:07:09 -07:00
Dan Helfman 8cf52651fe Mentioning libyaml compile errors in troubleshooting. 2017-07-22 21:07:09 -07:00
Dan Helfman 919d7573c3 Upgrading instructions to super clarify Python 3 upgrade. 2017-07-22 20:52:29 -07:00
Dan Helfman 166ef8faae Upgrading instructions to super clarify Python 3 upgrade. 2017-07-22 20:52:29 -07:00
Dan Helfman 8bfffd8cf7 Removing TODO that basically entails testing ruamel.yaml round-tripping, which in theory already has its own tests. 2017-07-22 20:31:26 -07:00
Dan Helfman ac2a63763f Removing TODO that basically entails testing ruamel.yaml round-tripping, which in theory already has its own tests. 2017-07-22 20:31:26 -07:00
Dan Helfman edb54b300b Fixing up borg module to deal with new parsed config file structures. 2017-07-22 20:11:49 -07:00
Dan Helfman 8b2b41eefc Fixing up borg module to deal with new parsed config file structures. 2017-07-22 20:11:49 -07:00
Dan Helfman 41d202c2e7 TODO about using the new exclude_patterns. 2017-07-10 16:26:32 -07:00
Dan Helfman fb172f018a TODO about using the new exclude_patterns. 2017-07-10 16:26:32 -07:00
Dan Helfman 8ef6c6fcbe Bail if "--excludes" argument is provided, as it's now deprecated in favor of configuration file. 2017-07-10 16:25:13 -07:00
Dan Helfman b1355e75c4 Bail if "--excludes" argument is provided, as it's now deprecated in favor of configuration file. 2017-07-10 16:25:13 -07:00
Dan Helfman 0691cda46f Mention generate-borgmatic-config in changelog. 2017-07-10 16:07:07 -07:00
Dan Helfman d2c143d39c Mention generate-borgmatic-config in changelog. 2017-07-10 16:07:07 -07:00
Dan Helfman 8bf07e4766 Provide helpful message when borgmatic is run with only legacy config present. 2017-07-10 16:06:02 -07:00
Dan Helfman ef32b292a8 Provide helpful message when borgmatic is run with only legacy config present. 2017-07-10 16:06:02 -07:00
Dan Helfman b3d0fb0cee When writing config, make containing directory if necessary. Also default to tighter permissions. 2017-07-10 15:20:50 -07:00
Dan Helfman 61f88228b0 When writing config, make containing directory if necessary. Also default to tighter permissions. 2017-07-10 15:20:50 -07:00
Dan Helfman ff28be7724 Documentation updates based on the new YAML configuration. 2017-07-10 11:06:28 -07:00
Dan Helfman f98558546c Documentation updates based on the new YAML configuration. 2017-07-10 11:06:28 -07:00
Dan Helfman 5ff016238e Don't overwrite config files. And retain file permissions when upgrading config. 2017-07-10 10:37:11 -07:00
Dan Helfman 9cc7c77ba9 Don't overwrite config files. And retain file permissions when upgrading config. 2017-07-10 10:37:11 -07:00
Dan Helfman 618e56b2a5 Display result of config upgrade. 2017-07-10 10:13:57 -07:00
Dan Helfman 3b1b058ffe Display result of config upgrade. 2017-07-10 10:13:57 -07:00
Dan Helfman 338b80903c Fixing tests broken by excludes merging. 2017-07-10 10:09:06 -07:00
Dan Helfman 9a3b52e1fd Fixing tests broken by excludes merging. 2017-07-10 10:09:06 -07:00
Dan Helfman fea97b5149 Merge excludes into config file format. 2017-07-10 09:43:25 -07:00
Dan Helfman 0dfc935af6 Merge excludes into config file format. 2017-07-10 09:43:25 -07:00
Dan Helfman 17c87f8758 Completed test coverage of commands (except for main()s). 2017-07-09 17:03:45 -07:00
Dan Helfman 2f7527a333 Completed test coverage of commands (except for main()s). 2017-07-09 17:03:45 -07:00
Dan Helfman d49be19544 Add a version to the schema, because inevitably I'll want to revise the schema. 2017-07-09 16:18:10 -07:00
Dan Helfman 263891f414 Add a version to the schema, because inevitably I'll want to revise the schema. 2017-07-09 16:18:10 -07:00
Dan Helfman d4ae7814a0 Adding TODO about a helpful notice about legacy config. 2017-07-09 11:49:51 -07:00
Dan Helfman 644c2e6612 Adding TODO about a helpful notice about legacy config. 2017-07-09 11:49:51 -07:00
Dan Helfman dc9b075d5a Rename convert-borgmatic-config to upgrade-borgmatic-config. 2017-07-09 11:48:24 -07:00
Dan Helfman 999feb81ca Rename convert-borgmatic-config to upgrade-borgmatic-config. 2017-07-09 11:48:24 -07:00
Dan Helfman 1bcb2a8be4 More test coverage, and simplification of config generation. 2017-07-09 11:41:55 -07:00
Dan Helfman f581f4b8d9 More test coverage, and simplification of config generation. 2017-07-09 11:41:55 -07:00
Dan Helfman a16d90ff46 Adding a "does not raise" test for displaying errors. 2017-07-09 10:27:34 -07:00
Dan Helfman c7803a2814 Adding a "does not raise" test for displaying errors. 2017-07-09 10:27:34 -07:00
Dan Helfman e50fd04750 Adding test coverage report. Making tests a little less brittle. 2017-07-08 23:01:41 -07:00
Dan Helfman f4e5dc8382 Adding test coverage report. Making tests a little less brittle. 2017-07-08 23:01:41 -07:00
Dan Helfman 745de200df Basic YAML generating / validating / converting to. 2017-07-08 22:33:51 -07:00
Dan Helfman f19a40ef9c Basic YAML generating / validating / converting to. 2017-07-08 22:33:51 -07:00
Dan Helfman bff6980eee Tests for YAML config code. 2017-07-04 18:32:37 -07:00
Dan Helfman 483bd50bdf Tests for YAML config code. 2017-07-04 18:32:37 -07:00
Dan Helfman 1dc60d2856 Integrating YAML config into borgmatic and updating README. 2017-07-04 18:23:59 -07:00
Dan Helfman 5110e64e63 Integrating YAML config into borgmatic and updating README. 2017-07-04 18:23:59 -07:00
Dan Helfman 6e85940d63 Basic YAML configuration file parsing. 2017-07-04 16:52:24 -07:00
Dan Helfman 4d7556f68b Basic YAML configuration file parsing. 2017-07-04 16:52:24 -07:00
Dan Helfman e00f74ddf7 Dropped Python 2 support. Now Python 3 only. 2017-07-02 17:18:33 -07:00
Dan Helfman 9212f87735 Dropped Python 2 support. Now Python 3 only. 2017-07-02 17:18:33 -07:00
Dan Helfman 1aaf27dfb2 Changed example umask config to be more realistic. 2017-06-25 10:36:36 -07:00
Dan Helfman ebd34f1695 Changed example umask config to be more realistic. 2017-06-25 10:36:36 -07:00
Dan Helfman 87c65fb723 Removing unnecessary curlies from bash command. 2016-07-04 09:35:51 -07:00
Dan Helfman a34dccbd27 Removing unnecessary curlies from bash command. 2016-07-04 09:35:51 -07:00
Dan Helfman abb6bed459 Sample files for triggering borgmatic from a systemd timer. 2016-07-04 09:19:34 -07:00
Dan Helfman 49c4f483fd Sample files for triggering borgmatic from a systemd timer. 2016-07-04 09:19:34 -07:00
Dan Helfman 5bd1cc5580 #18: Fix for README mention of sample files not included in package. Also, added logo. 2016-07-03 22:07:53 -07:00
Dan Helfman 4447956da7 #18: Fix for README mention of sample files not included in package. Also, added logo. 2016-07-03 22:07:53 -07:00
Dan Helfman f6d2e983d9 Added tag 1.0.3 for changeset 32c6341dda9f 2016-06-23 07:13:29 -07:00
Dan Helfman 9a96a277e6 Added tag 1.0.3 for changeset 32c6341dda9f 2016-06-23 07:13:29 -07:00
Dan Helfman 6bfe524bac #21: Fix for verbosity flag not actually causing verbose output. 2016-06-23 07:13:25 -07:00
Dan Helfman a45d7bec81 #21: Fix for verbosity flag not actually causing verbose output. 2016-06-23 07:13:25 -07:00
Dan Helfman ead991dcd1 Added tag 1.0.2 for changeset 9603d13910b3 2016-06-13 12:02:37 -07:00
Dan Helfman 63c4bf3bf9 Added tag 1.0.2 for changeset 9603d13910b3 2016-06-13 12:02:37 -07:00
Dan Helfman b22b552bf3 #20: Fix for traceback when remote_path option is missing. 2016-06-13 08:53:41 -07:00
Dan Helfman ed0127df91 #20: Fix for traceback when remote_path option is missing. 2016-06-13 08:53:41 -07:00
Dan Helfman 938392b25b Restricting issues list to open issues. 2016-06-12 22:40:04 -07:00
Dan Helfman 17e9f21fb9 Restricting issues list to open issues. 2016-06-12 22:40:04 -07:00
Dan Helfman 481dbc14c3 Rename issues URL. 2016-06-12 22:37:42 -07:00
Dan Helfman 16cc77fd9d Rename issues URL. 2016-06-12 22:37:42 -07:00
Dan Helfman 600c438951 Reverting to pre-rename issues link, because that link isn't yet renamed to borgmatic. 2016-06-10 17:11:28 -07:00
Dan Helfman 5d9bb13410 Reverting to pre-rename issues link, because that link isn't yet renamed to borgmatic. 2016-06-10 17:11:28 -07:00
Dan Helfman 2e3e68d2cb Added tag 1.0.1 for changeset de2d7721cdec 2016-06-10 13:34:23 -07:00
Dan Helfman da513c1089 Added tag 1.0.1 for changeset de2d7721cdec 2016-06-10 13:34:23 -07:00
Dan Helfman 331adca23e #19: Support for Borg's --remote-path option to use an alternate Borg executable. 2016-06-10 13:31:37 -07:00
Dan Helfman 3579dbe813 #19: Support for Borg's --remote-path option to use an alternate Borg executable. 2016-06-10 13:31:37 -07:00
Dan Helfman e1e5db22f8 Making a univeral wheel that supports both Python 2 and 3. 2016-06-10 12:34:49 -07:00
Dan Helfman 8e3a2c7a85 Making a univeral wheel that supports both Python 2 and 3. 2016-06-10 12:34:49 -07:00
Dan Helfman 377e3948ff Added tag 1.0.0 for changeset 0e1fbee9358d 2016-06-10 12:15:48 -07:00
Dan Helfman 706a31f189 Added tag 1.0.0 for changeset 0e1fbee9358d 2016-06-10 12:15:48 -07:00
Dan Helfman 4533fec167 Documenting how to upgrade from atticmatic to borgmatic. 2016-06-10 11:53:45 -07:00
Dan Helfman e5c772d8a5 Documenting how to upgrade from atticmatic to borgmatic. 2016-06-10 11:53:45 -07:00
Dan Helfman 633700c0af Dropping support for Attic. 2016-06-10 11:21:53 -07:00
Dan Helfman 40a215802f Dropping support for Attic. 2016-06-10 11:21:53 -07:00
Dan Helfman 007b3e6d4e Merge pull request #12 from dawez/master
fixed typos in readme
2016-04-17 21:21:44 -07:00
Dan Helfman 5d46acbe41 Merge pull request #12 from dawez/master
fixed typos in readme
2016-04-17 21:21:44 -07:00
dawez 175761c757 fixed typos in readme 2016-04-17 22:26:07 +02:00
dawez 9dc4960277 fixed typos in readme 2016-04-17 22:26:07 +02:00
Dan Helfman c7e23fe9ed Added tag 0.1.8 for changeset dbc96d3f83bd 2016-04-10 16:01:18 -07:00
Dan Helfman 7644a38574 Added tag 0.1.8 for changeset dbc96d3f83bd 2016-04-10 16:01:18 -07:00
Dan Helfman 9e45da75cb Cutting a release. 2016-04-10 16:01:05 -07:00
Dan Helfman 2359135327 Cutting a release. 2016-04-10 16:01:05 -07:00
Dan Helfman 0ea5824427 Switching from the no-longer-maintained nose test runner to pytest. 2016-04-10 15:59:36 -07:00
Dan Helfman df9cc6a7d2 Switching from the no-longer-maintained nose test runner to pytest. 2016-04-10 15:59:36 -07:00
Dan Helfman 5b66dc69a1 Refreshing flexmock version in test requirements. 2016-04-10 15:48:10 -07:00
Dan Helfman 448c2593ed Refreshing flexmock version in test requirements. 2016-04-10 15:48:10 -07:00
Dan Helfman 8210172d7f Fixing "check" backend tests to support new use of stderr=STDOUT. 2016-04-10 15:46:43 -07:00
Dan Helfman a364369f25 Fixing "check" backend tests to support new use of stderr=STDOUT. 2016-04-10 15:46:43 -07:00
Dan Helfman 82e8dae948 At verbosity zero, suppressing Borg check spew to stderr about "Checking segments". 2016-04-10 15:29:42 -07:00
Dan Helfman e430b8c281 At verbosity zero, suppressing Borg check spew to stderr about "Checking segments". 2016-04-10 15:29:42 -07:00
Dan Helfman fa87aed263 Normalizing recent changes. No new content. 2016-04-10 15:27:21 -07:00
Dan Helfman 219408647c Normalizing recent changes. No new content. 2016-04-10 15:27:21 -07:00
Dan Helfman c3b4cb21ed Fixed links to Borg documentation. 2016-04-10 10:23:32 -07:00
Dan Helfman 5bffa35741 Fixed links to Borg documentation. 2016-04-10 10:23:32 -07:00
Dan Helfman 030b321e39 Merge pull request #11 from jangondol/patch-1
Fix broken link to Borg quickstart
2016-04-10 10:21:23 -07:00
Dan Helfman a938a2ad61 Merge pull request #11 from jangondol/patch-1
Fix broken link to Borg quickstart
2016-04-10 10:21:23 -07:00
Jan Gondol 15bf273e6e Fix broken link to Borg quickstart 2016-04-06 14:54:06 +02:00
Jan Gondol 37b6f043dd Fix broken link to Borg quickstart 2016-04-06 14:54:06 +02:00
Dan Helfman cf545ae93a Mocking out glob() in test so it doesn't hit the filesystem, and simplifying comprehension. 2016-02-13 16:41:17 -08:00
Dan Helfman 0c97dba5f1 Mocking out glob() in test so it doesn't hit the filesystem, and simplifying comprehension. 2016-02-13 16:41:17 -08:00
Dan Helfman 45a2b9cded Merge pull request #5 from ypid/support-file-globs
Added support for file globs in source_directories.
2016-02-13 16:32:05 -08:00
Dan Helfman f149fa4b4b Merge pull request #5 from ypid/support-file-globs
Added support for file globs in source_directories.
2016-02-13 16:32:05 -08:00
Robin Schneider 953d08ba63
Made globing for source_directories the default.
Don’t remove non existing files/directories from the list and let
attic/borg handle this.
2016-02-13 21:10:05 +01:00
Robin Schneider 88da0c3039
Added support for file globs in source_directories.
source_directories_glob can be used to enable glob support (defaults to
disabled).
2016-02-13 21:07:07 +01:00
Robin Schneider f669e31305 Made globing for source_directories the default.
Don’t remove non existing files/directories from the list and let
attic/borg handle this.
2016-02-13 21:05:34 +01:00
Dan Helfman 0012e0cdea Support borg create --umask. (Merge PR from ypid.) 2016-02-13 10:59:43 -08:00
Dan Helfman dd2be365b1 Support borg create --umask. (Merge PR from ypid.) 2016-02-13 10:59:43 -08:00
Dan Helfman 049f9c8853 Added support for --one-file-system for Borg. 2016-02-13 10:43:31 -08:00
Dan Helfman d6585811d6 Added support for --one-file-system for Borg. 2016-02-13 10:43:31 -08:00
Dan Helfman 31482ee559 Merge pull request #8 from ypid/fixed-source-split-bug
Use /\s+/ to split source_directories to handle 1+ spaces.
2016-02-13 10:29:31 -08:00
Dan Helfman 14a277a64f Merge pull request #8 from ypid/fixed-source-split-bug
Use /\s+/ to split source_directories to handle 1+ spaces.
2016-02-13 10:29:31 -08:00
Dan Helfman 1ffff3255a Merge pull request #6 from ypid/added-gitignore
Added .gitignore file.
2016-02-13 10:27:55 -08:00
Dan Helfman 9723dbdd4c Merge pull request #6 from ypid/added-gitignore
Added .gitignore file.
2016-02-13 10:27:55 -08:00
Robin Schneider 9e52be6ffd
Use /\s+/ to split source_directories to handle 1+ spaces.
This bug is can be quite annoying because when you accidentally used
something like:

```ini
[location]
source_directories: backup_one  backup_two
;                              A (Additional space here)
```

It would call Attic/Borg with ('backup_one', '', 'backup_two') which in
turn backups your whole $PWD.
2016-02-07 23:30:54 +01:00
Robin Schneider 978096b402
Added .gitignore file. 2016-02-07 22:14:57 +01:00
Robin Schneider 9f5c5c8e13 Use /\s+/ to split source_directories to handle 1+ spaces.
This bug is can be quite annoying because when you accidentally used
something like:

```ini
[location]
source_directories: backup_one  backup_two
;                              A (Additional space here)
```

It would call Attic/Borg with ('backup_one', '', 'backup_two') which in
turn backups your whole $PWD.
2016-01-31 11:42:07 +01:00
Robin Schneider 4453f41967 Added .gitignore file. 2016-01-30 20:24:29 +01:00
Robin Schneider e4cf193cd7 Added support for file globs in source_directories.
source_directories_glob can be used to enable glob support (defaults to
disabled).
2016-01-25 23:52:16 +01:00
Dan Helfman cc6aa7af05 Merge pull request #3 from ypid/README-ini-hightlighting
Also allow the INI example to be highlighted on GitHub.
2016-01-24 20:52:01 -08:00
Dan Helfman e1605ae6ab Merge pull request #3 from ypid/README-ini-hightlighting
Also allow the INI example to be highlighted on GitHub.
2016-01-24 20:52:01 -08:00
Robin Schneider 32858fb0b4
Also allow the INI example to be highlighted on GitHub. 2016-01-20 13:11:15 +01:00
Robin Schneider d6ea9b6658 Also allow the INI example to be highlighted on GitHub. 2016-01-20 13:11:15 +01:00
Dan Helfman e59845d4e1 Added tag github/master for changeset 28434dd0440c 2015-11-08 17:06:48 -08:00
Dan Helfman 147996b234 Added tag github/master for changeset 28434dd0440c 2015-11-08 17:06:48 -08:00
Dan Helfman 9437e95849 Merge. 2015-11-08 17:04:49 -08:00
Dan Helfman ac374f89d2 Merge. 2015-11-08 17:04:49 -08:00
Dan Helfman 3a3851d2a5 Removed tag github/yaml_config_files 2015-11-08 17:04:14 -08:00
Dan Helfman ddde37ff59 Removed tag github/yaml_config_files 2015-11-08 17:04:14 -08:00
Dan Helfman 80318e6e30 Removed tag github/yaml_config_files 2015-11-08 17:03:40 -08:00
Dan Helfman 278aaabeb9 Removed tag github/yaml_config_files 2015-11-08 17:03:40 -08:00
Dan Helfman 6756ca8311 Merge pull request #2 from ThomasWaldmann/patch-1
fixed typo in README
2015-10-20 18:33:16 -07:00
Dan Helfman 06cdc2c46c Merge pull request #2 from ThomasWaldmann/patch-1
fixed typo in README
2015-10-20 18:33:16 -07:00
TW fa7955b8cf fixed typo in README 2015-10-20 23:08:43 +02:00
TW e5586f7d87 fixed typo in README 2015-10-20 23:08:43 +02:00
Dan Helfman 944c0212c3 Added tag 0.1.7 for changeset 5a458ebef804 2015-09-06 16:40:46 -07:00
Dan Helfman d044504693 Added tag 0.1.7 for changeset 5a458ebef804 2015-09-06 16:40:46 -07:00
Dan Helfman 2456fc67f1 Revving version. 2015-09-06 16:40:39 -07:00
Dan Helfman ec416cb152 Revving version. 2015-09-06 16:40:39 -07:00
Dan Helfman 8a58b72934 Better error message when configuration file is missing. 2015-09-06 15:55:14 -07:00
Dan Helfman 0da1c6ec7b Better error message when configuration file is missing. 2015-09-06 15:55:14 -07:00
Dan Helfman 6dc0173b74 #11: Fixed parsing of punctuation in configuration file. 2015-09-06 15:33:56 -07:00
Dan Helfman f54acc9bbf #11: Fixed parsing of punctuation in configuration file. 2015-09-06 15:33:56 -07:00
Dan Helfman 5c58f85be1 Added tag 0.1.6 for changeset 4c63f3d90ec2 2015-09-02 22:48:14 -07:00
Dan Helfman d86faa30d3 Added tag 0.1.6 for changeset 4c63f3d90ec2 2015-09-02 22:48:14 -07:00
Dan Helfman 3a9e32a411 #9: New configuration option for the encryption passphrase. #10: Support for Borg's new archive compression feature. 2015-09-02 22:48:07 -07:00
Dan Helfman a44212ff00 #9: New configuration option for the encryption passphrase. #10: Support for Borg's new archive compression feature. 2015-09-02 22:48:07 -07:00
Dan Helfman 30f6ec4f7d Adding documentation note about logging into the issue tracker in order to create issues. 2015-09-02 18:45:15 -07:00
Dan Helfman 178e56e77b Adding documentation note about logging into the issue tracker in order to create issues. 2015-09-02 18:45:15 -07:00
Dan Helfman c67ab09e4d Adding build to hgignore. 2015-08-09 11:04:57 -07:00
Dan Helfman 1cb44da6b3 Adding build to hgignore. 2015-08-09 11:04:57 -07:00
Dan Helfman 5299046b6b Added tag 0.1.5 for changeset 0afff209b902 2015-08-09 10:59:40 -07:00
Dan Helfman 7e8123379f Added tag 0.1.5 for changeset 0afff209b902 2015-08-09 10:59:40 -07:00
Dan Helfman 204e515bf7 Changes to support release on PyPI. Now pip installable by name! 2015-08-09 10:59:27 -07:00
Dan Helfman c554d1d36d Changes to support release on PyPI. Now pip installable by name! 2015-08-09 10:59:27 -07:00
Dan Helfman 1334da99e2 Added tag 0.1.4 for changeset e58246fc92bb 2015-07-30 08:13:32 -07:00
Dan Helfman a56529871c Added tag 0.1.4 for changeset e58246fc92bb 2015-07-30 08:13:32 -07:00
Dan Helfman 996ca19dac Adding version test. 2015-07-30 08:13:27 -07:00
Dan Helfman 820492f9a9 Adding version test. 2015-07-30 08:13:27 -07:00
Dan Helfman 61969d17a2 Added tag 0.1.4 for changeset 6dda59c12de8 2015-07-30 08:12:36 -07:00
Dan Helfman f803198f14 Added tag 0.1.4 for changeset 6dda59c12de8 2015-07-30 08:12:36 -07:00
Dan Helfman d041e23d35 Adding test that setup.py version matches release version. 2015-07-30 08:12:31 -07:00
Dan Helfman f59bc98d79 Adding test that setup.py version matches release version. 2015-07-30 08:12:31 -07:00
Dan Helfman e996e09657 Added tag 0.1.3 for changeset acc7fb61566f 2015-07-27 21:48:21 -07:00
Dan Helfman 158e889deb Added tag 0.1.3 for changeset acc7fb61566f 2015-07-27 21:48:21 -07:00
Dan Helfman 9c06874073 #1: Add support for "borg check --last N" to Borg backend. 2015-07-27 21:47:52 -07:00
Dan Helfman 2444c4b372 #1: Add support for "borg check --last N" to Borg backend. 2015-07-27 21:47:52 -07:00
Dan Helfman f5e0e10143 #6: Fixing example config file to use valid keep_within value. 2015-07-27 19:06:39 -07:00
Dan Helfman 9ecc207139 #6: Fixing example config file to use valid keep_within value. 2015-07-27 19:06:39 -07:00
Dan Helfman 952a691f60 Linking to both Attic and Borg check docs from sample config. 2015-07-26 22:02:43 -07:00
Dan Helfman 803fd3a851 Linking to both Attic and Borg check docs from sample config. 2015-07-26 22:02:43 -07:00
Dan Helfman f94181480c Removing some annoying Pelican metadata from docs. 2015-07-26 21:29:14 -07:00
Dan Helfman e18cfd6c80 Removing some annoying Pelican metadata from docs. 2015-07-26 21:29:14 -07:00
Dan Helfman c27b4a3497 Added tag 0.1.2 for changeset 83067f995dd3 2015-07-26 21:06:06 -07:00
Dan Helfman 7a287ba289 Added tag 0.1.2 for changeset 83067f995dd3 2015-07-26 21:06:06 -07:00
Dan Helfman 58d33503a1 As a convenience to new users, allow a missing default excludes file. 2015-07-26 21:06:03 -07:00
Dan Helfman 3f99dc6db2 As a convenience to new users, allow a missing default excludes file. 2015-07-26 21:06:03 -07:00
Dan Helfman 38322a3f6f Linking to both Attic and Borg prune docs from sample config. 2015-07-26 20:57:31 -07:00
Dan Helfman 837d25cfd8 Linking to both Attic and Borg prune docs from sample config. 2015-07-26 20:57:31 -07:00
Dan Helfman 52ab7cb881 New issue tracker, linked from documentation. 2015-07-21 21:29:40 -07:00
Dan Helfman fedc75cafc New issue tracker, linked from documentation. 2015-07-21 21:29:40 -07:00
Dan Helfman 17ac63aae6 Added tag 0.1.1 for changeset 7b6c87dca7ea 2015-07-18 23:49:06 -07:00
Dan Helfman 29106b9645 Added tag 0.1.1 for changeset 7b6c87dca7ea 2015-07-18 23:49:06 -07:00
Dan Helfman 1f1c8fdaba Bumping version. 2015-07-18 23:48:58 -07:00
Dan Helfman 4f6337c46e Bumping version. 2015-07-18 23:48:58 -07:00
Dan Helfman ce6196a5c6 Added tag 0.1.1 for changeset ac5dfa01e9d1 2015-07-18 18:44:14 -07:00
Dan Helfman 55183bf890 Added tag 0.1.1 for changeset ac5dfa01e9d1 2015-07-18 18:44:14 -07:00
Dan Helfman 6b0aa13856 Adding borgmatic cron example. 2015-07-18 18:44:11 -07:00
Dan Helfman cc09d7fc10 Adding borgmatic cron example. 2015-07-18 18:44:11 -07:00
Dan Helfman d25db4cd0d Added tag 0.1.0 for changeset 38d72677343f 2015-07-18 18:39:33 -07:00
Dan Helfman 96ce9309e0 Added tag 0.1.0 for changeset 38d72677343f 2015-07-18 18:39:33 -07:00
Dan Helfman 7097ed67a6 New "borgmatic" command to support Borg backup software, a fork of Attic. 2015-07-18 18:35:29 -07:00
Dan Helfman f2f8503e77 New "borgmatic" command to support Borg backup software, a fork of Attic. 2015-07-18 18:35:29 -07:00
Dan Helfman 52d5240fa0 Added tag 0.0.7 for changeset cf4c7065f071 2015-07-17 21:58:58 -07:00
Dan Helfman bdce17ae71 Added tag 0.0.7 for changeset cf4c7065f071 2015-07-17 21:58:58 -07:00
Dan Helfman 5bf3a4875c Flag for multiple levels of verbosity: some, and lots. 2015-07-17 21:58:50 -07:00
Dan Helfman b501a568aa Flag for multiple levels of verbosity: some, and lots. 2015-07-17 21:58:50 -07:00
Dan Helfman d9125451f5 Improved mocking of Python builtins in unit tests. 2015-06-14 11:00:46 -07:00
Dan Helfman 1578b44536 Improved mocking of Python builtins in unit tests. 2015-06-14 11:00:46 -07:00
Dan Helfman c3613e0637 Adding some explanitory text about consistency checks to README example. 2015-05-10 22:06:48 -07:00
Dan Helfman 932145f20b Adding some explanitory text about consistency checks to README example. 2015-05-10 22:06:48 -07:00
Dan Helfman c8f1af635f Added tag 0.0.6 for changeset 7ea93ca83f42 2015-05-10 22:00:34 -07:00
Dan Helfman 6c2e668262 Added tag 0.0.6 for changeset 7ea93ca83f42 2015-05-10 22:00:34 -07:00
Dan Helfman cfd61dc1d1 New configuration section for customizing which Attic consistency checks run, if any. 2015-05-10 22:00:31 -07:00
Dan Helfman df2d059af2 New configuration section for customizing which Attic consistency checks run, if any. 2015-05-10 22:00:31 -07:00
Dan Helfman 7750d2568c Passing through command-line options from tox to nosetests. 2015-03-15 11:15:40 -07:00
Dan Helfman 301eb4926e Passing through command-line options from tox to nosetests. 2015-03-15 11:15:40 -07:00
Dan Helfman 4e4f8c2670 Added tag 0.0.5 for changeset a03495a8e8b4 2015-03-15 10:47:58 -07:00
Dan Helfman 897046cf51 Added tag 0.0.5 for changeset a03495a8e8b4 2015-03-15 10:47:58 -07:00
Dan Helfman cb402d6846 Re-fixing version. 2015-03-15 10:47:49 -07:00
Dan Helfman 645c66c98a Re-fixing version. 2015-03-15 10:47:49 -07:00
Dan Helfman aa1178dc49 Added tag 0.0.5 for changeset 569aef47a9b2 2015-03-15 10:46:55 -07:00
Dan Helfman 811678396a Added tag 0.0.5 for changeset 569aef47a9b2 2015-03-15 10:46:55 -07:00
Dan Helfman 3506819511 Added tag 0.0.5 for changeset aa8a807f4ba2 2015-03-15 10:44:25 -07:00
Dan Helfman fd94debba6 Added tag 0.0.5 for changeset aa8a807f4ba2 2015-03-15 10:44:25 -07:00
Dan Helfman ac6c927a23 Backout out "helpful" error message that broke --verbose. 2015-03-15 10:44:18 -07:00
Dan Helfman 851dd69785 Backout out "helpful" error message that broke --verbose. 2015-03-15 10:44:18 -07:00
Dan Helfman bda6451c1d Added tag 0.0.5 for changeset b31d51b63370 2015-03-15 10:39:08 -07:00
Dan Helfman 6430e3799a Added tag 0.0.5 for changeset b31d51b63370 2015-03-15 10:39:08 -07:00
Dan Helfman d9e396e264 Added tag 0.0.4 for changeset 4bb2e81fc770 2015-03-15 10:19:12 -07:00
Dan Helfman c15a89b9f8 Added tag 0.0.4 for changeset 4bb2e81fc770 2015-03-15 10:19:12 -07:00
Dan Helfman 66286f92df Releasing 0.0.4. 2015-03-15 10:15:03 -07:00
Dan Helfman 0bf033de06 Releasing 0.0.4. 2015-03-15 10:15:03 -07:00
Dan Helfman 715b240589 Now using tox to run tests against multiple versions of Python in one go. 2015-03-15 10:14:30 -07:00
Dan Helfman 5bc7c04cdf Now using tox to run tests against multiple versions of Python in one go. 2015-03-15 10:14:30 -07:00
Dan Helfman ee5697ac37 Fixing Python 3 test incompatibility with builtins. 2015-03-15 10:14:16 -07:00
Dan Helfman a5eda0d305 Fixing Python 3 test incompatibility with builtins. 2015-03-15 10:14:16 -07:00
Dan Helfman aa48b95ee7 Bumping setup.py version. 2015-03-15 09:52:40 -07:00
Dan Helfman 04216922c4 Bumping setup.py version. 2015-03-15 09:52:40 -07:00
Dan Helfman 2639b7105a Added nosetests config file (setup.cfg) with defaults. 2015-03-15 09:41:58 -07:00
Dan Helfman aa482d088c Added nosetests config file (setup.cfg) with defaults. 2015-03-15 09:41:58 -07:00
Dan Helfman 02df59e964 Added a troubleshooting section with steps to deal with broken pipes. 2015-02-28 11:03:22 -08:00
Dan Helfman b755d09b0a Added a troubleshooting section with steps to deal with broken pipes. 2015-02-28 11:03:22 -08:00
Dan Helfman f23810f19a Updating install instructions so you can upgrade from one release of atticmatic to the next. 2015-02-14 09:31:42 -08:00
Dan Helfman 89703111d0 Updating install instructions so you can upgrade from one release of atticmatic to the next. 2015-02-14 09:31:42 -08:00
Dan Helfman 9f5dd6c10d Added tag 0.0.3 for changeset 7730ae34665c 2015-02-14 09:24:15 -08:00
Dan Helfman 0f21919626 Added tag 0.0.3 for changeset 7730ae34665c 2015-02-14 09:24:15 -08:00
Dan Helfman eaf2bd22c1 After pruning, run attic's consistency checks on all archives. 2015-02-14 09:23:40 -08:00
Dan Helfman e3fa18b892 After pruning, run attic's consistency checks on all archives. 2015-02-14 09:23:40 -08:00
Dan Helfman b1113d57ae Correcting doc string based on updated command-line arguments source. 2014-12-20 11:42:27 -08:00
Dan Helfman 6d4e40a819 Correcting doc string based on updated command-line arguments source. 2014-12-20 11:42:27 -08:00
Dan Helfman dbd312981e Integration tests for argument parsing. 2014-12-20 11:37:25 -08:00
Dan Helfman de40570db4 Integration tests for argument parsing. 2014-12-20 11:37:25 -08:00
Dan Helfman 511314a54a Adding a note about repository encryption. 2014-12-20 10:56:03 -08:00
Dan Helfman 2a45f43167 Adding a note about repository encryption. 2014-12-20 10:56:03 -08:00
Dan Helfman 18267b9677 Added tag 0.0.2 for changeset 467d3a3ce918 2014-12-06 18:35:28 -08:00
Dan Helfman f8c6da6ff4 Added tag 0.0.2 for changeset 467d3a3ce918 2014-12-06 18:35:28 -08:00
Dan Helfman 056ed7184b Configuration support for additional attic prune flags: keep_within, keep_hourly, keep_yearly, and prefix. 2014-12-06 18:35:20 -08:00
Dan Helfman b94c106a36 For convenience, adding some short-form arguments in addition to the long-form arguments. 2014-12-01 22:47:51 -08:00
Dan Helfman 965dd1aabe Adding sudo to installation of test dependencies, for consistency with installation of main dependencies. 2014-12-01 22:39:11 -08:00
Dan Helfman 626dd66254 Preventing ConfigParser from swallowing file read IOErrors, so that the user gets a more useful message. 2014-12-01 22:35:25 -08:00
Dan Helfman d46e370950 Fixing configparser import for Python 3. 2014-12-01 22:14:35 -08:00
Dan Helfman 126bb279cd Expanding description. 2014-12-01 20:36:43 -08:00
Dan Helfman d0eae19556 Adding authors/contributors file. 2014-12-01 20:30:07 -08:00
Dan Helfman 69971cd7e2 Python 3 ConfigParser compatibility. 2014-12-01 20:26:19 -08:00
Dan Helfman 45c6541266 Python 3 compatible exceptions. 2014-12-01 20:23:29 -08:00
Dan Helfman 65c837c828 Mentioning source code location explicitly. 2014-12-01 20:22:49 -08:00
Dan Helfman 8a4167b7a3 Saving README when rendered such that it can be served easily. 2014-12-01 20:15:21 -08:00
Dan Helfman 814770c2a9 Markdown metadata and link formatting updates. 2014-12-01 19:49:25 -08:00
Dan Helfman f862eda7d6 Renaming README to indicate markdown. 2014-11-27 09:34:13 -08:00
Dan Helfman 5472424d5a Playing nicely with markdown. 2014-11-27 09:29:31 -08:00
Dan Helfman 10a449fe1a Adding note about making etc configuration directory before copying a file to it. 2014-11-26 20:15:21 -08:00
Dan Helfman f557e2cbbd Merge pull request #1 from hajs/master
fixed README: copy cronjob to /etc/cron.d instead of /etc/init.d
2014-11-26 08:03:47 -08:00
Henning Schroder 704b97a636 fixed README: copy cronjob to /etc/cron.d instead of /etc/init.d (like comment in sample/atticmatic.cron correctly explains) 2014-11-26 13:04:14 +01:00
Dan Helfman 200a1bd63e Updating README with clarifications and examples. 2014-11-25 16:01:59 -08:00
Dan Helfman cf4c262226 Note about hosting arrangement. 2014-11-18 18:32:16 -08:00
Dan Helfman 7b5363ce14 Merge with Github head! 2014-11-18 18:28:20 -08:00
Dan Helfman 42d9e2bfd8 Adding GPL v3 license. 2014-11-18 18:22:51 -08:00
Dan Helfman d182509771 Unit tests for attic invocation code. 2014-11-17 22:19:34 -08:00
Dan Helfman e567158246 Adding unit tests for config module. 2014-11-17 21:57:44 -08:00
Dan Helfman db0f057b54 Adding contact info. 2014-11-17 18:35:47 -08:00
Dan Helfman 84922c7232 Adding PATH necessary to find the attic binary. 2014-11-01 17:46:04 -07:00
Dan Helfman 16bebe9832 Initial import. 2014-10-30 22:34:03 -07:00
200 changed files with 23444 additions and 487 deletions

2
.dockerignore Normal file
View File

@ -0,0 +1,2 @@
.git
.tox

54
.drone.yml Normal file
View File

@ -0,0 +1,54 @@
kind: pipeline
name: python-3-8-alpine-3-13
services:
- name: postgresql
image: postgres:13.1-alpine
environment:
POSTGRES_PASSWORD: test
POSTGRES_DB: test
- name: mysql
image: mariadb:10.5
environment:
MYSQL_ROOT_PASSWORD: test
MYSQL_DATABASE: test
- name: mongodb
image: mongo:5.0.5
environment:
MONGO_INITDB_ROOT_USERNAME: root
MONGO_INITDB_ROOT_PASSWORD: test
clone:
skip_verify: true
steps:
- name: build
image: alpine:3.13
pull: always
commands:
- scripts/run-full-tests
---
kind: pipeline
name: documentation
clone:
skip_verify: true
steps:
- name: build
image: plugins/docker
settings:
username:
from_secret: docker_username
password:
from_secret: docker_password
repo: witten/borgmatic-docs
dockerfile: docs/Dockerfile
trigger:
repo:
- borgmatic-collective/borgmatic
branch:
- master
event:
- push

48
.eleventy.js Normal file
View File

@ -0,0 +1,48 @@
const pluginSyntaxHighlight = require("@11ty/eleventy-plugin-syntaxhighlight");
const inclusiveLangPlugin = require("@11ty/eleventy-plugin-inclusive-language");
const navigationPlugin = require("@11ty/eleventy-navigation");
module.exports = function(eleventyConfig) {
eleventyConfig.addPlugin(pluginSyntaxHighlight);
eleventyConfig.addPlugin(inclusiveLangPlugin);
eleventyConfig.addPlugin(navigationPlugin);
let markdownIt = require("markdown-it");
let markdownItAnchor = require("markdown-it-anchor");
let markdownItReplaceLink = require("markdown-it-replace-link");
let markdownItOptions = {
html: true,
breaks: false,
linkify: true,
replaceLink: function (link, env) {
if (process.env.NODE_ENV == "production") {
return link;
}
return link.replace('https://torsion.org/borgmatic/', 'http://localhost:8080/');
}
};
let markdownItAnchorOptions = {
permalink: true,
permalinkClass: "direct-link"
};
eleventyConfig.setLibrary(
"md",
markdownIt(markdownItOptions)
.use(markdownItAnchor, markdownItAnchorOptions)
.use(markdownItReplaceLink)
);
eleventyConfig.addPassthroughCopy({"docs/static": "static"});
eleventyConfig.setLiquidOptions({dynamicPartials: false});
return {
templateFormats: [
"md",
"txt"
]
}
};

35
.gitea/issue_template.md Normal file
View File

@ -0,0 +1,35 @@
#### What I'm trying to do and why
#### Steps to reproduce (if a bug)
Include (sanitized) borgmatic configuration files if applicable.
#### Actual behavior (if a bug)
Include (sanitized) `--verbosity 2` output if applicable.
#### Expected behavior (if a bug)
#### Other notes / implementation ideas
#### Environment
**borgmatic version:** [version here]
Use `sudo borgmatic --version` or `sudo pip show borgmatic | grep ^Version`
**borgmatic installation method:** [e.g., Debian package, Docker container, etc.]
**Borg version:** [version here]
Use `sudo borg --version`
**Python version:** [version here]
Use `python3 --version`
**Database version (if applicable):** [version here]
Use `psql --version` or `mysql --version` on client and server.
**operating system and version:** [OS here]

11
.gitignore vendored Normal file
View File

@ -0,0 +1,11 @@
*.egg-info
*.pyc
*.swp
.cache
.coverage*
.pytest_cache
.tox
__pycache__
build/
dist/
pip-wheel-metadata/

View File

@ -1,4 +0,0 @@
syntax: glob
*.egg-info
*.pyc
*.swp

10
AUTHORS
View File

@ -1,4 +1,14 @@
Dan Helfman <witten@torsion.org>: Main developer
Alexander Görtz: Python 3 compatibility
Florian Lindner: Logging rewrite
Henning Schroeder: Copy editing
Johannes Feichtner: Support for user hooks
Michele Lazzeri: Custom archive names
Nick Whyte: Support prefix filtering for archive consistency checks
newtonne: Read encryption password from external file
Robin `ypid` Schneider: Support additional options of Borg and add validate-borgmatic-config command
Scott Squires: Custom archive names
Thomas LÉVEIL: Support for a keep_minutely prune option. Support for the --json option
And many others! See the output of "git log".

2
MANIFEST.in Normal file
View File

@ -0,0 +1,2 @@
include borgmatic/config/schema.yaml
graft sample/systemd

859
NEWS
View File

@ -1,3 +1,862 @@
1.6.3.dev0
* #541: Add "borgmatic list --find" flag for searching for files across multiple archives, useful
for hunting down that file you accidentally deleted so you can extract it. See the documentation
for more information:
https://torsion.org/borgmatic/docs/how-to/inspect-your-backups/#searching-for-a-file
* Deprecate "borgmatic list --successful" flag, as listing only non-checkpoint (successful)
archives is now the default in newer versions of Borg.
1.6.2
* #523: Reduce the default consistency check frequency and support configuring the frequency
independently for each check. Also add "borgmatic check --force" flag to ignore configured
frequencies. See the documentation for more information:
https://torsion.org/borgmatic/docs/how-to/deal-with-very-large-backups/#check-frequency
* #536: Fix generate-borgmatic-config to support more complex schema changes like the new
Healthchecks configuration options when the "--source" flag is used.
* #538: Add support for "borgmatic borg debug" command.
* #539: Add "generate-borgmatic-config --overwrite" flag to replace an existing destination file.
* Add Bash completion script so you can tab-complete the borgmatic command-line. See the
documentation for more information:
https://torsion.org/borgmatic/docs/how-to/set-up-backups/#shell-completion
1.6.1
* #294: Add Healthchecks monitoring hook "ping_body_limit" option to configure how many bytes of
logs to send to the Healthchecks server.
* #402: Remove the error when "archive_name_format" is specified but a retention prefix isn't.
* #420: Warn when an unsupported variable is used in a hook command.
* #439: Change connection failures for monitoring hooks (Healthchecks, Cronitor, PagerDuty, and
Cronhub) to be warnings instead of errors. This way, the monitoring system failing does not block
backups.
* #460: Add Healthchecks monitoring hook "send_logs" option to enable/disable sending borgmatic
logs to the Healthchecks server.
* #525: Add Healthchecks monitoring hook "states" option to only enable pinging for particular
monitoring states (start, finish, fail).
* #528: Improve the error message when a configuration override contains an invalid value.
* #531: BREAKING: When deep merging common configuration, merge colliding list values by appending
them. Previously, one list replaced the other.
* #532: When a configuration include is a relative path, load it from either the current working
directory or from the directory containing the file doing the including. Previously, only the
working directory was used.
* Add a randomized delay to the sample systemd timer to spread out the load on a server.
* Change the configuration format for borgmatic monitoring hooks (Healthchecks, Cronitor,
PagerDuty, and Cronhub) to specify the ping URL / integration key as a named option. The intent
is to support additional options (some in this release). This change is backwards-compatible.
* Add emojis to documentation table of contents to make it easier to find particular how-to and
reference guides at a glance.
1.6.0
* #381: BREAKING: Greatly simplify configuration file reuse by deep merging when including common
configuration. See the documentation for more information:
https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#include-merging
* #473: BREAKING: Instead of executing "before" command hooks before all borgmatic actions run (and
"after" hooks after), execute these hooks right before/after the corresponding action. E.g.,
"before_check" now runs immediately before the "check" action. This better supports running
timing-sensitive tasks like pausing containers. Side effect: before/after command hooks now run
once for each configured repository instead of once per configuration file. Additionally, the
"repositories" interpolated variable has been changed to "repository", containing the path to the
current repository for the hook. See the documentation for more information:
https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/
* #513: Add mention of sudo's "secure_path" option to borgmatic installation documentation.
* #515: Fix "borgmatic borg key ..." to pass parameters to Borg in the correct order.
* #516: Fix handling of TERM signal to exit borgmatic, not just forward the signal to Borg.
* #517: Fix borgmatic exit code (so it's zero) when initial Borg calls fail but later retries
succeed.
* Change Healthchecks logs truncation size from 10k bytes to 100k bytes, corresponding to that
same change on Healthchecks.io.
1.5.24
* #431: Add "working_directory" option to support source directories with relative paths.
* #444: When loading a configuration file that is unreadable due to file permissions, warn instead
of erroring. This supports running borgmatic as a non-root user with configuration in ~/.config
even if there is an unreadable global configuration file in /etc.
* #469: Add "repositories" context to "before_*" and "after_*" command action hooks. See the
documentation for more information:
https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/
* #486: Fix handling of "patterns_from" and "exclude_from" options to error instead of warning when
referencing unreadable files and "create" action is run.
* #507: Fix Borg usage error in the "compact" action when running "borgmatic --dry-run". Now, skip
"compact" entirely during a dry run.
1.5.23
* #394: Compact repository segments and free space with new "borgmatic compact" action. Borg 1.2+
only. Also run "compact" by default when no actions are specified, as "prune" in Borg 1.2 no
longer frees up space unless "compact" is run.
* #394: When using the "atime", "bsd_flags", "numeric_owner", or "remote_rate_limit" options,
tailor the flags passed to Borg depending on the Borg version.
* #480, #482: Fix traceback when a YAML validation error occurs.
1.5.22
* #288: Add database dump hook for MongoDB.
* #470: Move mysqldump options to the beginning of the command due to MySQL bug 30994.
* #471: When command-line configuration override produces a parse error, error cleanly instead of
tracebacking.
* #476: Fix unicode error when restoring particular MySQL databases.
* Drop support for Python 3.6, which has been end-of-lifed.
* Add support for Python 3.10.
1.5.21
* #28: Optionally retry failing backups via "retries" and "retry_wait" configuration options.
* #306: Add "list_options" MySQL configuration option for passing additional arguments to MySQL
list command.
* #459: Add support for old version (2.x) of jsonschema library.
1.5.20
* Re-release with correct version without dev0 tag.
1.5.19
* #387: Fix error when configured source directories are not present on the filesystem at the time
of backup. Now, Borg will complain, but the backup will still continue.
* #455: Mention changing borgmatic path in cron documentation.
* Update sample systemd service file with more granular read-only filesystem settings.
* Move Gitea and GitHub hosting from a personal namespace to an organization for better
collaboration with related projects.
* 1k ★s on GitHub!
1.5.18
* #389: Fix "message too long" error when logging to rsyslog.
* #440: Fix traceback that can occur when dumping a database.
1.5.17
* #437: Fix error when configuration file contains "umask" option.
* Remove test dependency on vim and /dev/urandom.
1.5.16
* #379: Suppress console output in sample crontab and systemd service files.
* #407: Fix syslog logging on FreeBSD.
* #430: Fix hang when restoring a PostgreSQL "tar" format database dump.
* Better error messages! Switch the library used for validating configuration files (from pykwalify
to jsonschema).
* Link borgmatic Ansible role from installation documentation:
https://torsion.org/borgmatic/docs/how-to/set-up-backups/#other-ways-to-install
1.5.15
* #419: Document use case of running backups conditionally based on laptop power level:
https://torsion.org/borgmatic/docs/how-to/backup-to-a-removable-drive-or-an-intermittent-server/
* #425: Run arbitrary Borg commands with new "borgmatic borg" action. See the documentation for
more information: https://torsion.org/borgmatic/docs/how-to/run-arbitrary-borg-commands/
1.5.14
* #390: Add link to Hetzner storage offering from the documentation.
* #398: Clarify canonical home of borgmatic in documentation.
* #406: Clarify that spaces in path names should not be backslashed in path names.
* #423: Fix error handling to error loudly when Borg gets killed due to running out of memory!
* Fix build so as not to attempt to build and push documentation for a non-master branch.
* "Fix" build failure with Alpine Edge by switching from Edge to Alpine 3.13.
* Move #borgmatic IRC channel from Freenode to Libera Chat due to Freenode takeover drama.
IRC connection info: https://torsion.org/borgmatic/#issues
1.5.13
* #373: Document that passphrase is used for Borg keyfile encryption, not just repokey encryption.
* #404: Add support for ruamel.yaml 0.17.x YAML parsing library.
* Update systemd service example to return a permission error when a system call isn't permitted
(instead of terminating borgmatic outright).
* Drop support for Python 3.5, which has been end-of-lifed.
* Add support for Python 3.9.
* Update versions of test dependencies (test_requirements.txt and test containers).
* Only support black code formatter on Python 3.8+. New black dependencies make installation
difficult on older versions of Python.
* Replace "improve this documentation" form with link to support and ticket tracker.
1.5.12
* Fix for previous release with incorrect version suffix in setup.py. No other changes.
1.5.11
* #341: Add "temporary_directory" option for changing Borg's temporary directory.
* #352: Lock down systemd security settings in sample systemd service file.
* #355: Fix traceback when a database hook value is null in a configuration file.
* #361: Merge override values when specifying the "--override" flag multiple times. The previous
behavior was to take the value of the last "--override" flag only.
* #367: Fix traceback when upgrading old INI-style configuration with upgrade-borgmatic-config.
* #368: Fix signal forwarding from borgmatic to Borg resulting in recursion traceback.
* #369: Document support for Borg placeholders in repository names.
1.5.10
* #347: Add hooks that run for the "extract" action: "before_extract" and "after_extract".
* #350: Fix traceback when a configuration directory is non-readable due to directory permissions.
* Add documentation navigation links on left side of all documentation pages.
* Clarify documentation on configuration overrides, specifically the portion about list syntax:
http://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#configuration-overrides
* Clarify documentation overview of monitoring options:
http://torsion.org/borgmatic/docs/how-to/monitor-your-backups/
1.5.9
* #300: Add "borgmatic export-tar" action to export an archive to a tar-formatted file or stream.
* #339: Fix for intermittent timing-related test failure of logging function.
* Clarify database documentation about excluding named pipes and character/block devices to prevent
hangs.
* Add documentation on how to make backups redundant with multiple repositories:
https://torsion.org/borgmatic/docs/how-to/make-backups-redundant/
1.5.8
* #336: Fix for traceback when running Cronitor, Cronhub, and PagerDuty monitor hooks.
1.5.7
* #327: Fix broken pass-through of BORG_* environment variables to Borg.
* #328: Fix duplicate logging to Healthchecks and send "after_*" hooks output to Healthchecks.
* #331: Add SSL support to PostgreSQL database configuration.
* #333: Fix for potential data loss (data not getting backed up) when borgmatic omitted configured
source directories in certain situations. Specifically, this occurred when two source directories
on different filesystems were related by parentage (e.g. "/foo" and "/foo/bar/baz") and the
one_file_system option was enabled.
* Update documentation code fragments theme to better match the rest of the page.
* Improve configuration reference documentation readability via more aggressive word-wrapping in
configuration schema descriptions.
1.5.6
* #292: Allow before_backup and similiar hooks to exit with a soft failure without altering the
monitoring status on Healthchecks or other providers. Support this by waiting to ping monitoring
services with a "start" status until after before_* hooks finish. Failures in before_* hooks
still trigger a monitoring "fail" status.
* #316: Fix hang when a stale database dump named pipe from an aborted borgmatic run remains on
disk.
* #323: Fix for certain configuration options like ssh_command impacting Borg invocations for
separate configuration files.
* #324: Add "borgmatic extract --strip-components" flag to remove leading path components when
extracting an archive.
* Tweak comment indentation in generated configuration file for clarity.
* Link to Borgmacator GNOME AppIndicator from monitoring documentation.
1.5.5
* #314: Fix regression in support for PostgreSQL's "directory" dump format. Unlike other dump
formats, the "directory" dump format does not stream directly to/from Borg.
* #315: Fix enabled database hooks to implicitly set one_file_system configuration option to true.
This prevents Borg from reading devices like /dev/zero and hanging.
* #316: Fix hang when streaming a database dump to Borg with implicit duplicate source directories
by deduplicating them first.
* #319: Fix error message when there are no MySQL databases to dump for "all" databases.
* Improve documentation around the installation process. Specifically, making borgmatic commands
runnable via the system PATH and offering a global install option.
1.5.4
* #310: Fix legitimate database dump command errors (exit code 1) not being treated as errors by
borgmatic.
* For database dumps, replace the named pipe on every borgmatic run. This prevent hangs on stale
pipes left over from previous runs.
* Fix error handling to handle more edge cases when executing commands.
1.5.3
* #258: Stream database dumps and restores directly to/from Borg without using any additional
filesystem space. This feature is automatic, and works even on restores from archives made with
previous versions of borgmatic.
* #293: Documentation on macOS launchd permissions issues with work-around for Full Disk Access.
* Remove "borgmatic restore --progress" flag, as it now conflicts with streaming database restores.
1.5.2
* #301: Fix MySQL restore error on "all" database dump by excluding system tables.
* Fix PostgreSQL restore error on "all" database dump by using "psql" for the restore instead of
"pg_restore".
1.5.1
* #289: Tired of looking up the latest successful archive name in order to pass it to borgmatic
actions? Me too. Now you can specify "--archive latest" to all actions that accept an archive
flag.
* #290: Fix the "--stats" and "--files" flags so that they yield output at verbosity 0.
* Reduce the default verbosity of borgmatic logs sent to Healthchecks monitoring hook. Now, it's
warnings and errors only. You can increase the verbosity via the "--monitoring-verbosity" flag.
* Add security policy documentation in SECURITY.md.
1.5.0
* #245: Monitor backups with PagerDuty hook integration. See the documentation for more
information: https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#pagerduty-hook
* #255: Add per-action hooks: "before_prune", "after_prune", "before_check", and "after_check".
* #274: Add ~/.config/borgmatic.d as another configuration directory default.
* #277: Customize Healthchecks log level via borgmatic "--monitoring-verbosity" flag.
* #280: Change "exclude_if_present" option to support multiple filenames that indicate a directory
should be excluded from backups, rather than just a single filename.
* #284: Backup to a removable drive or intermittent server via "soft failure" feature. See the
documentation for more information:
https://torsion.org/borgmatic/docs/how-to/backup-to-a-removable-drive-or-an-intermittent-server/
* #287: View consistency check progress via "--progress" flag for "check" action.
* For "create" and "prune" actions, no longer list files or show detailed stats at any verbosities
by default. You can opt back in with "--files" or "--stats" flags.
* For "list" and "info" actions, show repository names even at verbosity 0.
1.4.22
* #276, #285: Disable colored output when "--json" flag is used, so as to produce valid JSON ouput.
* After a backup of a database dump in directory format, properly remove the dump directory.
* In "borgmatic --help", don't expand $HOME in listing of default "--config" paths.
1.4.21
* #268: Override particular configuration options from the command-line via "--override" flag. See
the documentation for more information:
https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#configuration-overrides
* #270: Only trigger "on_error" hooks and monitoring failures for "prune", "create", and "check"
actions, and not for other actions.
* When pruning with verbosity level 1, list pruned and kept archives. Previously, this information
was only shown at verbosity level 2.
1.4.20
* Fix repository probing during "borgmatic init" to respect verbosity flag and remote_path option.
* #249: Update Healthchecks/Cronitor/Cronhub monitoring integrations to fire for "check" and
"prune" actions, not just "create".
1.4.19
* #259: Optionally change the internal database dump path via "borgmatic_source_directory" option
in location configuration section.
* #271: Support piping "borgmatic list" output to grep by logging certain log levels to console
stdout and others to stderr.
* Retain colored output when piping or redirecting in an interactive terminal.
* Add end-to-end tests for database dump and restore. These are run on developer machines with
Docker Compose for approximate parity with continuous integration tests.
1.4.18
* Fix "--repository" flag to accept relative paths.
* Fix "borgmatic umount" so it only runs Borg once instead of once per repository / configuration
file.
* #253: Mount whole repositories via "borgmatic mount" without any "--archive" flag.
* #269: Filter listed paths via "borgmatic list --path" flag.
1.4.17
* #235: Pass extra options directly to particular Borg commands, handy for Borg options that
borgmatic does not yet support natively. Use "extra_borg_options" in the storage configuration
section.
* #266: Attempt to repair any inconsistencies found during a consistency check via
"borgmatic check --repair" flag.
1.4.16
* #256: Fix for "before_backup" hook not triggering an error when the command contains "borg" and
has an exit code of 1.
* #257: Fix for garbled Borg file listing when using "borgmatic create --progress" with
verbosity level 1 or 2.
* #260: Fix for missing Healthchecks monitoring payload or HTTP 500 due to incorrect unicode
encoding.
1.4.15
* Fix for database dump removal incorrectly skipping some database dumps.
* #123: Support for mounting an archive as a FUSE filesystem via "borgmatic mount" action, and
unmounting via "borgmatic umount". See the documentation for more information:
https://torsion.org/borgmatic/docs/how-to/extract-a-backup/#mount-a-filesystem
1.4.14
* Show summary log errors regardless of verbosity level, and log the "summary:" header with a log
level based on the contained summary logs.
1.4.13
* Show full error logs at "--verbosity 0" so you can see command output without upping the
verbosity level.
1.4.12
* #247: With "borgmatic check", consider Borg warnings as errors.
* Dial back the display of inline error logs a bit, so failed command output doesn't appear
multiple times in the logs (well, except for the summary).
1.4.11
* #241: When using the Healthchecks monitoring hook, include borgmatic logs in the payloads for
completion and failure pings.
* With --verbosity level 1 or 2, show error logs both inline when they occur and in the summary
logs at the bottom. With lower verbosity levels, suppress the summary and show error logs when
they occur.
1.4.10
* #246: Fix for "borgmatic restore" showing success and incorrectly extracting archive files, even
when no databases are configured to restore. As this can overwrite files from the archive and
lead to data loss, please upgrade to get the fix before using "borgmatic restore".
* Reopen the file given by "--log-file" flag if an external program rotates the log file while
borgmatic is running.
1.4.9
* #228: Database dump hooks for MySQL/MariaDB, so you can easily dump your databases before backups
run.
* #243: Fix repository does not exist error with "borgmatic extract" when repository is remote.
1.4.8
* Monitor backups with Cronhub hook integration. See the documentation for more information:
https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#cronhub-hook
* Fix Healthchecks/Cronitor hooks to skip actions when the borgmatic "--dry-run" flag is used.
1.4.7
* #238: In documentation, clarify when Healthchecks/Cronitor hooks fire in relation to other hooks.
* #239: Upgrade your borgmatic configuration to get new options and comments via
"generate-borgmatic-config --source". See the documentation for more information:
https://torsion.org/borgmatic/docs/how-to/upgrade/#upgrading-your-configuration
1.4.6
* Verbosity level "-1" for even quieter output: Errors only (#236).
1.4.5
* Log to file instead of syslog via command-line "--log-file" flag (#233).
1.4.4
* #234: Support for Borg --keep-exclude-tags and --exclude-nodump options.
1.4.3
* Monitor backups with Cronitor hook integration. See the documentation for more information:
https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#cronitor-hook
1.4.2
* Extract files to a particular directory via "borgmatic extract --destination" flag.
* Rename "borgmatic extract --restore-path" flag to "--path" to reduce confusion with the separate
"borgmatic restore" action. Any uses of "--restore-path" will continue working.
1.4.1
* #229: Restore backed up PostgreSQL databases via "borgmatic restore" action. See the
documentation for more information:
https://torsion.org/borgmatic/docs/how-to/backup-your-databases/
* Documentation on how to develop borgmatic's documentation:
https://torsion.org/borgmatic/docs/how-to/develop-on-borgmatic/#documentation-development
1.4.0
* #225: Database dump hooks for PostgreSQL, so you can easily dump your databases before backups
run.
* #230: Rename "borgmatic list --pattern-from" flag to "--patterns-from" to match Borg.
1.3.26
* #224: Fix "borgmatic list --successful" with a slightly better heuristic for listing successful
(non-checkpoint) archives.
1.3.25
* #223: Dead man's switch to detect when backups start failing silently, implemented via
healthchecks.io hook integration. See the documentation for more information:
https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#healthchecks-hook
* Documentation on monitoring and alerting options for borgmatic backups:
https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/
* Automatically rewrite links when developing on documentation locally.
1.3.24
* #86: Add "borgmatic list --successful" flag to only list successful (non-checkpoint) archives.
* Add a suggestion form to all documentation pages, so users can submit ideas for improving the
documentation.
* Update documentation link to community Arch Linux borgmatic package.
1.3.23
* #174: More detailed error alerting via runtime context available in "on_error" hook.
1.3.22
* #144: When backups to one of several repositories fails, keep backing up to the other
repositories and report errors afterwards.
1.3.21
* #192: User-defined hooks for global setup or cleanup that run before/after all actions. See the
documentation for more information:
https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/
1.3.20
* #205: More robust sample systemd service: boot delay, network dependency, lowered CPU/IO
priority, etc.
* #221: Fix "borgmatic create --progress" output so that it updates on the console in real-time.
1.3.19
* #219: Fix visibility of "borgmatic prune --stats" output.
1.3.18
* #220: Fix regression of argument parsing for default actions.
1.3.17
* #217: Fix error with "borgmatic check --only" command-line flag with "extract" consistency check.
1.3.16
* #210: Support for Borg check --verify-data flag via borgmatic "data" consistency check.
* #210: Override configured consistency checks via "borgmatic check --only" command-line flag.
* When generating sample configuration with generate-borgmatic-config, add a space after each "#"
comment indicator.
1.3.15
* #208: Fix for traceback when the "checks" option has an empty value.
* #209: Bypass Borg error about a moved repository via "relocated_repo_access_is_ok" option in
borgmatic storage configuration section.
* #213: Reorder arguments passed to Borg to fix duplicate directories when using Borg patterns.
* #214: Fix for hook erroring with exit code 1 not being interpreted as an error.
1.3.14
* #204: Do not treat Borg warnings (exit code 1) as failures.
* When validating configuration files, require strings instead of allowing any scalar type.
1.3.13
* #199: Add note to documentation about using spaces instead of tabs for indentation, as YAML does
not allow tabs.
* #203: Fix compatibility with ruamel.yaml 0.16.x.
* If a "prefix" option in borgmatic's configuration has an empty value (blank or ""), then disable
default prefix.
1.3.12
* Only log to syslog when run from a non-interactive console (e.g. a cron job).
* Remove unicode byte order mark from syslog output so it doesn't show up as a literal in rsyslog
output. See discussion on #197.
1.3.11
* #193: Pass through several "borg list" and "borg info" flags like --short, --format, --sort-by,
--first, --last, etc. via borgmatic command-line flags.
* Add borgmatic info --repository and --archive command-line flags to display info for individual
repositories or archives.
* Support for Borg --noatime, --noctime, and --nobirthtime flags via corresponding options in
borgmatic configuration location section.
1.3.10
* #198: Fix for Borg create error output not showing up at borgmatic verbosity level zero.
1.3.9
* #195: Switch to command-line actions as more traditional sub-commands, e.g. "borgmatic create",
"borgmatic prune", etc. However, the classic dashed options like "--create" still work!
1.3.8
* #191: Disable console color via "color" option in borgmatic configuration output section.
1.3.7
* #196: Fix for unclear error message for invalid YAML merge include.
* #197: Don't color syslog output.
* Change default syslog verbosity to show errors only.
1.3.6
* #53: Log to syslog in addition to existing console logging. Add --syslog-verbosity flag to
customize the log level. See the documentation for more information:
https://torsion.org/borgmatic/docs/how-to/inspect-your-backups/
* #178: Look for .yml configuration file extension in addition to .yaml.
* #189: Set umask used when executing hooks via "umask" option in borgmatic hooks section.
* Remove Python cache files before each Tox run.
* Add #borgmatic Freenode IRC channel to documentation.
* Add Borg/borgmatic hosting providers section to documentation.
* Add files for building documentation into a Docker image for web serving.
* Upgrade project build server from Drone 0.8 to 1.1.
* Build borgmatic documentation during continuous integration.
* We're nearly at 500 ★s on GitHub. We can do this!
1.3.5
* #153: Support for various Borg directory environment variables (BORG_CONFIG_DIR, BORG_CACHE_DIR,
etc.) via options in borgmatic's storage configuration.
* #177: Fix for regression with missing verbose log entries.
1.3.4
* Part of #125: Color borgmatic (but not Borg) output when using an interactive terminal.
* #166: Run tests for all installed versions of Python.
* #168: Update README with continuous integration badge.
* #169: Automatically sort Python imports in code.
* Document installing borgmatic with pip install --user instead of a system Python install.
* Get more reproducible builds by pinning the versions of pip and tox used to run tests.
* Factor out build/test configuration from tox.ini file.
1.3.3
* Add validate-borgmatic-config command, useful for validating borgmatic config generated by
configuration management or even edited by hand.
1.3.2
* #160: Fix for hooks executing when using --dry-run. Now hooks are skipped during a dry run.
1.3.1
* #155: Fix for invalid JSON output when using multiple borgmatic configuration files.
* #157: Fix for seemingly random filename ordering when running through a directory of
configuration files.
* Fix for empty JSON output when using --create --json.
* Now capturing Borg output only when --json flag is used. Previously, borgmatic delayed Borg
output even without the --json flag.
1.3.0
* #148: Configuration file includes and merging via "!include" tag to support reuse of common
options across configuration files.
1.2.18
* #147: Support for Borg create/extract --numeric-owner flag via "numeric_owner" option in
borgmatic's location section.
1.2.17
* #140: List the files within an archive via --list --archive option.
1.2.16
* #119: Include a sample borgmatic configuration file in the documentation.
* #123: Support for Borg archive restoration via borgmatic --extract command-line flag.
* Refactor documentation into multiple separate pages for clarity and findability.
* Organize options within command-line help into logical groups.
* Exclude tests from distribution packages.
1.2.15
* #127: Remove date echo from schema example, as it's not a substitute for real logging.
* #132: Leave exclude_patterns glob expansion to Borg, since doing it in borgmatic leads to
confusing behavior.
* #136: Handle and format validation errors raised during argument parsing.
* #138: Allow use of --stats flag when --create or --prune flags are implied.
1.2.14
* #103: When generating sample configuration with generate-borgmatic-config, document the defaults
for each option.
* #116: When running multiple configuration files, attempt all configuration files even if one of
them errors. Log a summary of results at the end.
* Add borgmatic --version command-line flag to get the current installed version number.
1.2.13
* #100: Support for --stats command-line flag independent of --verbosity.
* #117: With borgmatic --init command-line flag, proceed without erroring if a repository already
exists.
1.2.12
* #110: Support for Borg repository initialization via borgmatic --init command-line flag.
* #111: Update Borg create --filter values so a dry run lists files to back up.
* #113: Update README with link to a new/forked Docker image.
* Prevent deprecated --excludes command-line option from being used.
* Refactor README a bit to flow better for first-time users.
* Update README with a few additional borgmatic packages (Debian and Ubuntu).
1.2.11
* #108: Support for Borg create --progress via borgmatic command-line flag.
1.2.10
* #105: Support for Borg --chunker-params create option via "chunker_params" option in borgmatic's
storage section.
1.2.9
* #102: Fix for syntax error that occurred in Python 3.5 and below.
* Make automated tests support running in Python 3.5.
1.2.8
* #73: Enable consistency checks for only certain repositories via "check_repositories" option in
borgmatic's consistency configuration. Handy for large repositories that take forever to check.
* Include link to issue tracker within various command output.
* Run continuous integration tests on a matrix of Python and Borg versions.
1.2.7
* #98: Support for Borg --keep-secondly prune option.
* Use Black code formatter and Flake8 code checker as part of running automated tests.
* Add an end-to-end automated test that actually integrates with Borg.
* Set up continuous integration for borgmatic automated tests on projects.evoworx.org.
1.2.6
* Fix generated configuration to also include a "keep_daily" value so pruning works out of the
box.
1.2.5
* #57: When generating sample configuration with generate-borgmatic-config, comment out all
optional configuration so as to streamline the initial configuration process.
1.2.4
* Fix for archive checking traceback due to parameter mismatch.
1.2.3
* #64, #90, #92: Rewrite of logging system. Now verbosity flags passed to Borg are derived from
borgmatic's log level. Note that the output of borgmatic might slightly change.
* Part of #80: Support for Borg create --read-special via "read_special" option in borgmatic's
location configuration.
* #87: Support for Borg create --checkpoint-interval via "checkpoint_interval" option in
borgmatic's storage configuration.
* #88: Fix declared pykwalify compatibility version range in setup.py to prevent use of ancient
versions of pykwalify with large version numbers.
* #89: Pass --show-rc option to Borg when at highest verbosity level.
* #94: Support for Borg --json option via borgmatic command-line to --create archives.
1.2.2
* #85: Fix compatibility issue between pykwalify and ruamel.yaml 0.15.52, which manifested in
borgmatic as a pykwalify RuleError.
1.2.1
* Skip before/after backup hooks when only doing --prune, --check, --list, and/or --info.
* #71: Support for XDG_CONFIG_HOME environment variable for specifying alternate user ~/.config/
path.
* #74, #83: Support for Borg --json option via borgmatic command-line to --list archives or show
archive --info in JSON format, ideal for programmatic consumption.
* #38, #76: Upgrade ruamel.yaml compatibility version range and fix support for Python 3.7.
* #77: Skip non-"*.yaml" config filenames in /etc/borgmatic.d/ so as not to parse backup files,
editor swap files, etc.
* #81: Document user-defined hooks run before/after backup, or on error.
* Add code style guidelines to the documention.
1.2.0
* #61: Support for Borg --list option via borgmatic command-line to list all archives.
* #61: Support for Borg --info option via borgmatic command-line to display summary information.
* #62: Update README to mention other ways of installing borgmatic.
* Support for Borg --prefix option for consistency checks via "prefix" option in borgmatic's
consistency configuration.
* Add introductory screencast link to documentation.
* #59: Ignore "check_last" and consistency "prefix" when "archives" not in consistency checks.
* #60: Add "Persistent" flag to systemd timer example.
* #63: Support for Borg --nobsdflags option to skip recording bsdflags (e.g. NODUMP, IMMUTABLE) in
archive.
* #69: Support for Borg prune --umask option using value of existing "umask" option in borgmatic's
storage configuration.
* Update tox.ini to only assume Python 3.x instead of Python 3.4 specifically.
* Add ~/.config/borgmatic/config.yaml to default configuration path probing.
* Document how to develop on and contribute to borgmatic.
1.1.15
* Support for Borg BORG_PASSCOMMAND environment variable to read a password from an external file.
* Fix for Borg create error when using borgmatic's --dry-run and --verbosity options together.
Work-around for behavior introduced in Borg 1.1.3: https://github.com/borgbackup/borg/issues/3298
* #55: Fix for missing tags/releases on Gitea and GitHub project hosting.
* #56: Support for Borg --lock-wait option for the maximum wait for a repository/cache lock.
* #58: Support for using tilde in exclude_patterns to reference home directory.
1.1.14
* #49: Fix for typo in --patterns-from option.
* #47: Support for Borg --dry-run option via borgmatic command-line.
1.1.13
* #54: Fix for incorrect consistency check flags passed to Borg when all three checks ("repository",
"archives", and "extract") are specified in borgmatic configuration.
* #48: Add "local_path" to configuration for specifying an alternative Borg executable path.
* #49: Support for Borg experimental --patterns-from and --patterns options for specifying mixed
includes/excludes.
* Moved issue tracker from Taiga to integrated Gitea tracker at
https://projects.torsion.org/borgmatic-collective/borgmatic/issues
1.1.12
* #46: Declare dependency on pykwalify 1.6 or above, as older versions yield "Unknown key: version"
rule errors.
* Support for Borg --keep-minutely prune option.
1.1.11
* #26: Add "ssh_command" to configuration for specifying a custom SSH command or options.
* Fix for incorrect /etc/borgmatic.d/ configuration path probing on macOS. This problem manifested
as an error on startup: "[Errno 2] No such file or directory: '/etc/borgmatic.d'".
1.1.10
* Pass several Unix signals through to child processes like Borg. This means that Borg now properly
shuts down if borgmatic is terminated (e.g. due to a system suspend).
* #30: Support for using tilde in repository paths to reference home directory.
* #43: Support for Borg --files-cache option for setting the files cache operation mode.
* #45: Support for Borg --remote-ratelimit option for limiting upload rate.
* Log invoked Borg commands when at highest verbosity level.
1.1.9
* #17, #39: Support for user-defined hooks before/after backup, or on error.
* #34: Improve clarity of logging spew at high verbosity levels.
* #30: Support for using tilde in source directory path to reference home directory.
* Require "prefix" in retention section when "archive_name_format" is set. This is to avoid
accidental pruning of archives with a different archive name format. For similar reasons, default
"prefix" to "{hostname}-" if not specified.
* Convert main source repository from Mercurial to Git.
* Update dead links to Borg documentation.
1.1.8
* #40: Fix to make /etc/borgmatic/config.yaml optional rather than required when using the default
config paths.
1.1.7
* #29: Add "archive_name_format" to configuration for customizing archive names.
* Fix for traceback when "exclude_from" value is empty in configuration file.
* When pruning, make highest verbosity level list archives kept and pruned.
* Clarification of Python 3 pip usage in documentation.
1.1.6
* #13, #36: Support for Borg --exclude-from, --exclude-caches, and --exclude-if-present options.
1.1.5
* #35: New "extract" consistency check that performs a dry-run extraction of the most recent
archive.
1.1.4
* #18: Added command-line flags for performing a borgmatic run with only pruning, creating, or
checking enabled. This supports use cases like running consistency checks from a different cron
job with a different frequency, or running pruning with a different verbosity level.
1.1.3
* #15: Support for running multiple config files in /etc/borgmatic.d/ from a single borgmatic run.
* Fix for generate-borgmatic-config writing config with invalid one_file_system value.
1.1.2
* #33: Fix for passing check_last as integer to subprocess when calling Borg.
1.1.1
* Part of #33: Fix for upgrade-borgmatic-config converting check_last option as a string instead of
an integer.
* Fix for upgrade-borgmatic-config erroring when consistency checks option is not present.
1.1.0
* Switched config file format to YAML. Run upgrade-borgmatic-config to upgrade.
* Added generate-borgmatic-config command for initial config creation.
* Dropped Python 2 support. Now Python 3 only.
* #19: Fix for README mention of sample files not included in package.
* #23: Sample files for triggering borgmatic from a systemd timer.
* Support for backing up to multiple repositories.
* To free up space, now pruning backups prior to creating a new backup.
* Enabled test coverage output during tox runs.
* Added logo.
1.0.3
* #22: Fix for verbosity flag not actually causing verbose output.
1.0.2
* #21: Fix for traceback when remote_path option is missing.
1.0.1
* #20: Support for Borg's --remote-path option to use an alternate Borg
executable. See sample/config.
1.0.0
* Attic is no longer supported, as there hasn't been any recent development on
it. Dropping Attic support will allow faster iteration on Borg-specific
features. If you're still using Attic, this is a good time to switch to Borg!
* Project renamed from atticmatic to borgmatic. See the borgmatic README for
information on upgrading.
0.1.8
* Fix for handling of spaces in source_directories which resulted in backup up everything.
* Fix for broken links to Borg documentation.
* At verbosity zero, suppressing Borg check stderr spew about "Checking segments".
* Support for Borg --one-file-system.
* Support for Borg create --umask.
* Support for file globs in source_directories.
0.1.7
* #12: Fixed parsing of punctuation in configuration file.
* Better error message when configuration file is missing.
0.1.6
* #10: New configuration option for the encryption passphrase.
* #11: Support for Borg's new archive compression feature.
0.1.5
* Changes to support release on PyPI. Now pip installable by name!
0.1.4
* Adding test that setup.py version matches release version.
0.1.3
* #2: Add support for "borg check --last N" to Borg backend.
0.1.2
* As a convenience to new users, allow a missing default excludes file.
* New issue tracker, linked from documentation.
0.1.1
* Adding borgmatic cron example, and updating documentation to refer to it.
0.1.0
* New "borgmatic" command to support Borg backup software, a fork of Attic.
0.0.7
* Flag for multiple levels of verbosity: some, and lots.
* Improved mocking of Python builtins in unit tests.
0.0.6
* New configuration section for customizing which Attic consistency checks run, if any.
0.0.5
* Fixed regression with --verbose output being buffered. This means dropping the helpful error
message introduced in 0.0.4.
0.0.4
* Now using tox to run tests against multiple versions of Python in one go.
* Helpful error message about how to create a repository if one is missing.
* Troubleshooting section with steps to deal with broken pipes.
* Nosetests config file (setup.cfg) with defaults.
0.0.3
* After pruning, run attic's consistency checks on all archives.
* Integration tests for argument parsing.
* Documentation updates about repository encryption.
0.0.2
* Configuration support for additional attic prune flags: keep_within, keep_hourly, keep_yearly,

176
README.md
View File

@ -1,91 +1,147 @@
title: Atticmatic
date:
save_as: atticmatic/index.html
---
title: borgmatic
permalink: index.html
---
## Overview
## It's your data. Keep it that way.
atticmatic is a simple Python wrapper script for the [Attic backup
software](https://attic-backup.org/) that initiates a backup and prunes any
old backups according to a retention policy. The script supports specifying
your settings in a declarative configuration file rather than having to put
them all on the command-line, and handles common errors.
<img src="docs/static/borgmatic.png" alt="borgmatic logo" width="150px" style="float: right; padding-left: 1em;">
Here's an example config file:
borgmatic is simple, configuration-driven backup software for servers and
workstations. Protect your files with client-side encryption. Backup your
databases too. Monitor it all with integrated third-party services.
[location]
# Space-separated list of source directories to backup.
source_directories: /home /etc
The canonical home of borgmatic is at <a href="https://torsion.org/borgmatic">https://torsion.org/borgmatic</a>.
# Path to local or remote Attic repository.
repository: user@backupserver:sourcehostname.attic
Here's an example configuration file:
[retention]
# Retention policy for how many backups to keep in each category.
```yaml
location:
# List of source directories to backup.
source_directories:
- /home
- /etc
# Paths of local or remote repositories to backup to.
repositories:
- 1234@usw-s001.rsync.net:backups.borg
- k8pDxu32@k8pDxu32.repo.borgbase.com:repo
- /var/lib/backups/local.borg
retention:
# Retention policy for how many backups to keep.
keep_daily: 7
keep_weekly: 4
keep_monthly: 6
Additionally, exclude patterns can be specified in a separate excludes config
file, one pattern per line.
consistency:
# List of checks to run to validate your backups.
checks:
- name: repository
- name: archives
frequency: 2 weeks
atticmatic is hosted at <https://torsion.org/atticmatic> with [source code
available](https://torsion.org/hg/atticmatic). It's also mirrored on
[GitHub](https://github.com/witten/atticmatic) and
[BitBucket](https://bitbucket.org/dhelfman/atticmatic) for convenience.
hooks:
# Custom preparation scripts to run.
before_backup:
- prepare-for-backup.sh
# Databases to dump and include in backups.
postgresql_databases:
- name: users
# Third-party services to notify you if backups aren't happening.
healthchecks: https://hc-ping.com/be067061-cf96-4412-8eae-62b0c50d6a8c
```
Want to see borgmatic in action? Check out the <a
href="https://asciinema.org/a/203761?autoplay=1" target="_blank">screencast</a>.
<a href="https://asciinema.org/a/203761?autoplay=1" target="_blank"><img src="https://asciinema.org/a/203761.png" width="480"></a>
borgmatic is powered by [Borg Backup](https://www.borgbackup.org/).
## Integrations
<a href="https://www.postgresql.org/"><img src="docs/static/postgresql.png" alt="PostgreSQL" height="60px" style="margin-bottom:20px;"></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<a href="https://www.mysql.com/"><img src="docs/static/mysql.png" alt="MySQL" height="60px" style="margin-bottom:20px;"></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<a href="https://mariadb.com/"><img src="docs/static/mariadb.png" alt="MariaDB" height="60px" style="margin-bottom:20px;"></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<a href="https://www.mongodb.com/"><img src="docs/static/mongodb.png" alt="MongoDB" height="60px" style="margin-bottom:20px;"></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<a href="https://healthchecks.io/"><img src="docs/static/healthchecks.png" alt="Healthchecks" height="60px" style="margin-bottom:20px;"></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<a href="https://cronitor.io/"><img src="docs/static/cronitor.png" alt="Cronitor" height="60px" style="margin-bottom:20px;"></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<a href="https://cronhub.io/"><img src="docs/static/cronhub.png" alt="Cronhub" height="60px" style="margin-bottom:20px;"></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<a href="https://www.pagerduty.com/"><img src="docs/static/pagerduty.png" alt="PagerDuty" height="60px" style="margin-bottom:20px;"></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<a href="https://www.borgbase.com/?utm_source=borgmatic"><img src="docs/static/borgbase.png" alt="BorgBase" height="60px" style="margin-bottom:20px;"></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
## Setup
## Getting started
To get up and running with Attic, follow the [Attic Quick
Start](https://attic-backup.org/quickstart.html) guide to create an Attic
repository on a local or remote host.
Your first step is to [install and configure
borgmatic](https://torsion.org/borgmatic/docs/how-to/set-up-backups/).
If the repository is on a remote host, make sure that your local root user has
key-based ssh access to the desired user account on the remote host.
To install atticmatic, run the following command to download and install it:
sudo pip install hg+https://torsion.org/hg/atticmatic
Then copy the following configuration files:
sudo cp sample/atticmatic.cron /etc/cron.d/atticmatic
sudo mkdir /etc/atticmatic/
sudo cp sample/config sample/excludes /etc/atticmatic/
Lastly, modify those files with your desired configuration.
For additional documentation, check out the links above for <a
href="https://torsion.org/borgmatic/#documentation">borgmatic how-to and
reference guides</a>.
## Usage
## Hosting providers
You can run atticmatic and start a backup simply by invoking it without
arguments:
Need somewhere to store your encrypted off-site backups? The following hosting
providers include specific support for Borg/borgmatic—and fund borgmatic
development and hosting when you use these links to sign up. (These are
referral links, but without any tracking scripts or cookies.)
atticmatic
<ul>
<li class="referral"><a href="https://www.borgbase.com/?utm_source=borgmatic">BorgBase</a>: Borg hosting service with support for monitoring, 2FA, and append-only repos</li>
</ul>
This will also prune any old backups as per the configured retention policy.
By default, the backup will proceed silently except in the case of errors. But
if you'd like to to get additional information about the progress of the
backup as it proceeds, use the verbose option instead:
Additionally, [rsync.net](https://www.rsync.net/products/borg.html) and
[Hetzner](https://www.hetzner.com/storage/storage-box) have compatible storage
offerings, but do not currently fund borgmatic development or hosting.
atticmattic --verbose
## Support and contributing
If you'd like to see the available command-line arguments, view the help:
### Issues
atticmattic --help
You've got issues? Or an idea for a feature enhancement? We've got an [issue
tracker](https://projects.torsion.org/borgmatic-collective/borgmatic/issues). In order to
create a new issue or comment on an issue, you'll need to [login
first](https://projects.torsion.org/user/login). Note that you can login with
an existing GitHub account if you prefer.
If you'd like to chat with borgmatic developers or users, head on over to the
`#borgmatic` IRC channel on Libera Chat, either via <a
href="https://web.libera.chat/#borgmatic">web chat</a> or a
native <a href="ircs://irc.libera.chat:6697">IRC client</a>. If you
don't get a response right away, please hang around a while—or file a ticket
instead.
Also see the [security
policy](https://torsion.org/borgmatic/docs/security-policy/) for any security
issues.
Other questions or comments? Contact
[witten@torsion.org](mailto:witten@torsion.org).
## Running tests
### Contributing
To install test-specific dependencies, first run:
borgmatic [source code is
available](https://projects.torsion.org/borgmatic-collective/borgmatic) and is also mirrored
on [GitHub](https://github.com/borgmatic-collective/borgmatic) for convenience.
sudo python setup.py test
borgmatic is licensed under the GNU General Public License version 3 or any
later version.
To actually run tests, run:
If you'd like to contribute to borgmatic development, please feel free to
submit a [Pull Request](https://projects.torsion.org/borgmatic-collective/borgmatic/pulls)
or open an [issue](https://projects.torsion.org/borgmatic-collective/borgmatic/issues) first
to discuss your idea. We also accept Pull Requests on GitHub, if that's more
your thing. In general, contributions are very welcome. We don't bite!
nosetests --detailed-errors
Also, please check out the [borgmatic development
how-to](https://torsion.org/borgmatic/docs/how-to/develop-on-borgmatic/) for
info on cloning source code, running tests, etc.
<a href="https://build.torsion.org/borgmatic-collective/borgmatic" alt="build status">![Build Status](https://build.torsion.org/api/badges/borgmatic-collective/borgmatic/status.svg?ref=refs/heads/master)</a>
## Feedback
Questions? Comments? Got a patch? Contact <mailto:witten@torsion.org>.

18
SECURITY.md Normal file
View File

@ -0,0 +1,18 @@
---
title: Security policy
permalink: security-policy/index.html
---
## Supported versions
While we want to hear about security vulnerabilities in all versions of
borgmatic, security fixes are only made to the most recently released version.
It's simply not practical for our small volunteer effort to maintain multiple
release branches and put out separate security patches for each.
## Reporting a vulnerability
If you find a security vulnerability, please [file a
ticket](https://torsion.org/borgmatic/#issues) or [send email
directly](mailto:witten@torsion.org) as appropriate. You should expect to hear
back within a few days at most and generally sooner.

View File

@ -1,65 +0,0 @@
from datetime import datetime
import platform
import subprocess
def create_archive(excludes_filename, verbose, source_directories, repository):
'''
Given an excludes filename, a vebosity flag, a space-separated list of source directories, and
a local or remote repository path, create an attic archive.
'''
sources = tuple(source_directories.split(' '))
command = (
'attic', 'create',
'--exclude-from', excludes_filename,
'{repo}::{hostname}-{timestamp}'.format(
repo=repository,
hostname=platform.node(),
timestamp=datetime.now().isoformat(),
),
) + sources + (
('--verbose', '--stats') if verbose else ()
)
subprocess.check_call(command)
def make_prune_flags(retention_config):
'''
Given a retention config dict mapping from option name to value, tranform it into an iterable of
command-line name-value flag pairs.
For example, given a retention config of:
{'keep_weekly': 4, 'keep_monthly': 6}
This will be returned as an iterable of:
(
('--keep-weekly', '4'),
('--keep-monthly', '6'),
)
'''
return (
('--' + option_name.replace('_', '-'), str(retention_config[option_name]))
for option_name, value in retention_config.items()
)
def prune_archives(verbose, repository, retention_config):
'''
Given a verbosity flag, a local or remote repository path, and a retention config dict, prune
attic archives according the the retention policy specified in that configuration.
'''
command = (
'attic', 'prune',
repository,
) + tuple(
element
for pair in make_prune_flags(retention_config)
for element in pair
) + (('--verbose',) if verbose else ())
subprocess.check_call(command)

View File

@ -1,45 +0,0 @@
from __future__ import print_function
from argparse import ArgumentParser
from subprocess import CalledProcessError
import sys
from atticmatic.attic import create_archive, prune_archives
from atticmatic.config import parse_configuration
def parse_arguments():
'''
Parse the command-line arguments from sys.argv and return them as an ArgumentParser instance.
'''
parser = ArgumentParser()
parser.add_argument(
'-c', '--config',
dest='config_filename',
default='/etc/atticmatic/config',
help='Configuration filename',
)
parser.add_argument(
'--excludes',
dest='excludes_filename',
default='/etc/atticmatic/excludes',
help='Excludes filename',
)
parser.add_argument(
'-v', '--verbose',
action='store_true',
help='Display verbose progress information',
)
return parser.parse_args()
def main():
try:
args = parse_arguments()
location_config, retention_config = parse_configuration(args.config_filename)
create_archive(args.excludes_filename, args.verbose, **location_config)
prune_archives(args.verbose, location_config['repository'], retention_config)
except (ValueError, IOError, CalledProcessError) as error:
print(error, file=sys.stderr)
sys.exit(1)

View File

@ -1,128 +0,0 @@
from collections import OrderedDict, namedtuple
try:
# Python 2
from ConfigParser import ConfigParser
except ImportError:
# Python 3
from configparser import ConfigParser
Section_format = namedtuple('Section_format', ('name', 'options'))
Config_option = namedtuple('Config_option', ('name', 'value_type', 'required'))
def option(name, value_type=str, required=True):
'''
Given a config file option name, an expected type for its value, and whether it's required,
return a Config_option capturing that information.
'''
return Config_option(name, value_type, required)
CONFIG_FORMAT = (
Section_format(
'location',
(
option('source_directories'),
option('repository'),
),
),
Section_format(
'retention',
(
option('keep_within', required=False),
option('keep_hourly', int, required=False),
option('keep_daily', int, required=False),
option('keep_weekly', int, required=False),
option('keep_monthly', int, required=False),
option('keep_yearly', int, required=False),
option('prefix', required=False),
),
)
)
def validate_configuration_format(parser, config_format):
'''
Given an open ConfigParser and an expected config file format, validate that the parsed
configuration file has the expected sections, that any required options are present in those
sections, and that there aren't any unexpected options.
Raise ValueError if anything is awry.
'''
section_names = parser.sections()
required_section_names = tuple(section.name for section in config_format)
if set(section_names) != set(required_section_names):
raise ValueError(
'Expected config sections {} but found sections: {}'.format(
', '.join(required_section_names),
', '.join(section_names)
)
)
for section_format in config_format:
option_names = parser.options(section_format.name)
expected_options = section_format.options
unexpected_option_names = set(option_names) - set(option.name for option in expected_options)
if unexpected_option_names:
raise ValueError(
'Unexpected options found in config section {}: {}'.format(
section_format.name,
', '.join(sorted(unexpected_option_names)),
)
)
missing_option_names = tuple(
option.name for option in expected_options if option.required
if option.name not in option_names
)
if missing_option_names:
raise ValueError(
'Required options missing from config section {}: {}'.format(
section_format.name,
', '.join(missing_option_names)
)
)
def parse_section_options(parser, section_format):
'''
Given an open ConfigParser and an expected section format, return the option values from that
section as a dict mapping from option name to value. Omit those options that are not present in
the parsed options.
Raise ValueError if any option values cannot be coerced to the expected Python data type.
'''
type_getter = {
str: parser.get,
int: parser.getint,
}
return OrderedDict(
(option.name, type_getter[option.value_type](section_format.name, option.name))
for option in section_format.options
if parser.has_option(section_format.name, option.name)
)
def parse_configuration(config_filename):
'''
Given a config filename of the expected format, return the parsed configuration as a tuple of
(location config, retention config) where each config is a dict of that section's options.
Raise IOError if the file cannot be read, or ValueError if the format is not as expected.
'''
parser = ConfigParser()
parser.readfp(open(config_filename))
validate_configuration_format(parser, CONFIG_FORMAT)
return tuple(
parse_section_options(parser, section_format)
for section_format in CONFIG_FORMAT
)

View File

@ -1,113 +0,0 @@
from collections import OrderedDict
from flexmock import flexmock
from atticmatic import attic as module
def insert_subprocess_mock(check_call_command):
subprocess = flexmock()
subprocess.should_receive('check_call').with_args(check_call_command).once()
flexmock(module).subprocess = subprocess
def insert_platform_mock():
flexmock(module).platform = flexmock().should_receive('node').and_return('host').mock
def insert_datetime_mock():
flexmock(module).datetime = flexmock().should_receive('now').and_return(
flexmock().should_receive('isoformat').and_return('now').mock
).mock
def test_create_archive_should_call_attic_with_parameters():
insert_subprocess_mock(
('attic', 'create', '--exclude-from', 'excludes', 'repo::host-now', 'foo', 'bar'),
)
insert_platform_mock()
insert_datetime_mock()
module.create_archive(
excludes_filename='excludes',
verbose=False,
source_directories='foo bar',
repository='repo',
)
def test_create_archive_with_verbose_should_call_attic_with_verbose_parameters():
insert_subprocess_mock(
(
'attic', 'create', '--exclude-from', 'excludes', 'repo::host-now', 'foo', 'bar',
'--verbose', '--stats',
),
)
insert_platform_mock()
insert_datetime_mock()
module.create_archive(
excludes_filename='excludes',
verbose=True,
source_directories='foo bar',
repository='repo',
)
BASE_PRUNE_FLAGS = (
('--keep-daily', '1'),
('--keep-weekly', '2'),
('--keep-monthly', '3'),
)
def test_make_prune_flags_should_return_flags_from_config():
retention_config = OrderedDict(
(
('keep_daily', 1),
('keep_weekly', 2),
('keep_monthly', 3),
)
)
result = module.make_prune_flags(retention_config)
assert tuple(result) == BASE_PRUNE_FLAGS
def test_prune_archives_should_call_attic_with_parameters():
retention_config = flexmock()
flexmock(module).should_receive('make_prune_flags').with_args(retention_config).and_return(
BASE_PRUNE_FLAGS,
)
insert_subprocess_mock(
(
'attic', 'prune', 'repo', '--keep-daily', '1', '--keep-weekly', '2', '--keep-monthly',
'3',
),
)
module.prune_archives(
verbose=False,
repository='repo',
retention_config=retention_config,
)
def test_prune_archives_with_verbose_should_call_attic_with_verbose_parameters():
retention_config = flexmock()
flexmock(module).should_receive('make_prune_flags').with_args(retention_config).and_return(
BASE_PRUNE_FLAGS,
)
insert_subprocess_mock(
(
'attic', 'prune', 'repo', '--keep-daily', '1', '--keep-weekly', '2', '--keep-monthly',
'3', '--verbose',
),
)
module.prune_archives(
repository='repo',
verbose=True,
retention_config=retention_config,
)

55
borgmatic/borg/borg.py Normal file
View File

@ -0,0 +1,55 @@
import logging
from borgmatic.borg.flags import make_flags
from borgmatic.execute import execute_command
logger = logging.getLogger(__name__)
REPOSITORYLESS_BORG_COMMANDS = {'serve', None}
BORG_COMMANDS_WITH_SUBCOMMANDS = {'key', 'debug'}
BORG_SUBCOMMANDS_WITHOUT_REPOSITORY = (('debug', 'info'), ('debug', 'convert-profile'))
def run_arbitrary_borg(
repository, storage_config, options, archive=None, local_path='borg', remote_path=None
):
'''
Given a local or remote repository path, a storage config dict, a sequence of arbitrary
command-line Borg options, and an optional archive name, run an arbitrary Borg command on the
given repository/archive.
'''
lock_wait = storage_config.get('lock_wait', None)
try:
options = options[1:] if options[0] == '--' else options
# Borg commands like "key" have a sub-command ("export", etc.) that must follow it.
command_options_start_index = 2 if options[0] in BORG_COMMANDS_WITH_SUBCOMMANDS else 1
borg_command = tuple(options[:command_options_start_index])
command_options = tuple(options[command_options_start_index:])
except IndexError:
borg_command = ()
command_options = ()
if borg_command in BORG_SUBCOMMANDS_WITHOUT_REPOSITORY:
repository_archive = None
else:
repository_archive = (
'::'.join((repository, archive)) if repository and archive else repository
)
full_command = (
(local_path,)
+ borg_command
+ ((repository_archive,) if borg_command and repository_archive else ())
+ command_options
+ (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+ (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+ make_flags('remote-path', remote_path)
+ make_flags('lock-wait', lock_wait)
)
return execute_command(
full_command, output_log_level=logging.WARNING, borg_local_path=local_path,
)

319
borgmatic/borg/check.py Normal file
View File

@ -0,0 +1,319 @@
import argparse
import datetime
import json
import logging
import os
import pathlib
from borgmatic.borg import extract, info, state
from borgmatic.execute import DO_NOT_CAPTURE, execute_command
DEFAULT_CHECKS = (
{'name': 'repository', 'frequency': '1 month'},
{'name': 'archives', 'frequency': '1 month'},
)
DEFAULT_PREFIX = '{hostname}-'
logger = logging.getLogger(__name__)
def parse_checks(consistency_config, only_checks=None):
'''
Given a consistency config with a "checks" sequence of dicts and an optional list of override
checks, return a tuple of named checks to run.
For example, given a retention config of:
{'checks': ({'name': 'repository'}, {'name': 'archives'})}
This will be returned as:
('repository', 'archives')
If no "checks" option is present in the config, return the DEFAULT_CHECKS. If a checks value
has a name of "disabled", return an empty tuple, meaning that no checks should be run.
If the "data" check is present, then make sure the "archives" check is included as well.
'''
checks = only_checks or tuple(
check_config['name']
for check_config in (consistency_config.get('checks', None) or DEFAULT_CHECKS)
)
checks = tuple(check.lower() for check in checks)
if 'disabled' in checks:
if len(checks) > 1:
logger.warning(
'Multiple checks are configured, but one of them is "disabled"; not running any checks'
)
return ()
if 'data' in checks and 'archives' not in checks:
return checks + ('archives',)
return checks
def parse_frequency(frequency):
'''
Given a frequency string with a number and a unit of time, return a corresponding
datetime.timedelta instance or None if the frequency is None or "always".
For instance, given "3 weeks", return datetime.timedelta(weeks=3)
Raise ValueError if the given frequency cannot be parsed.
'''
if not frequency:
return None
frequency = frequency.strip().lower()
if frequency == 'always':
return None
try:
number, time_unit = frequency.split(' ')
number = int(number)
except ValueError:
raise ValueError(f"Could not parse consistency check frequency '{frequency}'")
if not time_unit.endswith('s'):
time_unit += 's'
if time_unit == 'months':
number *= 30
time_unit = 'days'
elif time_unit == 'years':
number *= 365
time_unit = 'days'
try:
return datetime.timedelta(**{time_unit: number})
except TypeError:
raise ValueError(f"Could not parse consistency check frequency '{frequency}'")
def filter_checks_on_frequency(
location_config, consistency_config, borg_repository_id, checks, force
):
'''
Given a location config, a consistency config with a "checks" sequence of dicts, a Borg
repository ID, a sequence of checks, and whether to force checks to run, filter down those
checks based on the configured "frequency" for each check as compared to its check time file.
In other words, a check whose check time file's timestamp is too new (based on the configured
frequency) will get cut from the returned sequence of checks. Example:
consistency_config = {
'checks': [
{
'name': 'archives',
'frequency': '2 weeks',
},
]
}
When this function is called with that consistency_config and "archives" in checks, "archives"
will get filtered out of the returned result if its check time file is newer than 2 weeks old,
indicating that it's not yet time to run that check again.
Raise ValueError if a frequency cannot be parsed.
'''
filtered_checks = list(checks)
if force:
return tuple(filtered_checks)
for check_config in consistency_config.get('checks', DEFAULT_CHECKS):
check = check_config['name']
if checks and check not in checks:
continue
frequency_delta = parse_frequency(check_config.get('frequency'))
if not frequency_delta:
continue
check_time = read_check_time(
make_check_time_path(location_config, borg_repository_id, check)
)
if not check_time:
continue
# If we've not yet reached the time when the frequency dictates we're ready for another
# check, skip this check.
if datetime.datetime.now() < check_time + frequency_delta:
remaining = check_time + frequency_delta - datetime.datetime.now()
logger.info(
f"Skipping {check} check due to configured frequency; {remaining} until next check"
)
filtered_checks.remove(check)
return tuple(filtered_checks)
def make_check_flags(checks, check_last=None, prefix=None):
'''
Given a parsed sequence of checks, transform it into tuple of command-line flags.
For example, given parsed checks of:
('repository',)
This will be returned as:
('--repository-only',)
However, if both "repository" and "archives" are in checks, then omit them from the returned
flags because Borg does both checks by default.
Additionally, if a check_last value is given and "archives" is in checks, then include a
"--last" flag. And if a prefix value is given and "archives" is in checks, then include a
"--prefix" flag.
'''
if 'archives' in checks:
last_flags = ('--last', str(check_last)) if check_last else ()
prefix_flags = ('--prefix', prefix) if prefix else ()
else:
last_flags = ()
prefix_flags = ()
if check_last:
logger.info('Ignoring check_last option, as "archives" is not in consistency checks')
if prefix:
logger.info(
'Ignoring consistency prefix option, as "archives" is not in consistency checks'
)
common_flags = last_flags + prefix_flags + (('--verify-data',) if 'data' in checks else ())
if {'repository', 'archives'}.issubset(set(checks)):
return common_flags
return (
tuple('--{}-only'.format(check) for check in checks if check in ('repository', 'archives'))
+ common_flags
)
def make_check_time_path(location_config, borg_repository_id, check_type):
'''
Given a location configuration dict, a Borg repository ID, and the name of a check type
("repository", "archives", etc.), return a path for recording that check's time (the time of
that check last occurring).
'''
return os.path.join(
os.path.expanduser(
location_config.get(
'borgmatic_source_directory', state.DEFAULT_BORGMATIC_SOURCE_DIRECTORY
)
),
'checks',
borg_repository_id,
check_type,
)
def write_check_time(path): # pragma: no cover
'''
Record a check time of now as the modification time of the given path.
'''
logger.debug(f'Writing check time at {path}')
os.makedirs(os.path.dirname(path), mode=0o700, exist_ok=True)
pathlib.Path(path, mode=0o600).touch()
def read_check_time(path):
'''
Return the check time based on the modification time of the given path. Return None if the path
doesn't exist.
'''
logger.debug(f'Reading check time from {path}')
try:
return datetime.datetime.fromtimestamp(os.stat(path).st_mtime)
except FileNotFoundError:
return None
def check_archives(
repository,
location_config,
storage_config,
consistency_config,
local_path='borg',
remote_path=None,
progress=None,
repair=None,
only_checks=None,
force=None,
):
'''
Given a local or remote repository path, a storage config dict, a consistency config dict,
local/remote commands to run, whether to include progress information, whether to attempt a
repair, and an optional list of checks to use instead of configured checks, check the contained
Borg archives for consistency.
If there are no consistency checks to run, skip running them.
Raises ValueError if the Borg repository ID cannot be determined.
'''
try:
borg_repository_id = json.loads(
info.display_archives_info(
repository,
storage_config,
argparse.Namespace(json=True, archive=None),
local_path,
remote_path,
)
)['repository']['id']
except (json.JSONDecodeError, KeyError):
raise ValueError(f'Cannot determine Borg repository ID for {repository}')
checks = filter_checks_on_frequency(
location_config,
consistency_config,
borg_repository_id,
parse_checks(consistency_config, only_checks),
force,
)
check_last = consistency_config.get('check_last', None)
lock_wait = None
extra_borg_options = storage_config.get('extra_borg_options', {}).get('check', '')
if set(checks).intersection({'repository', 'archives', 'data'}):
lock_wait = storage_config.get('lock_wait', None)
verbosity_flags = ()
if logger.isEnabledFor(logging.INFO):
verbosity_flags = ('--info',)
if logger.isEnabledFor(logging.DEBUG):
verbosity_flags = ('--debug', '--show-rc')
prefix = consistency_config.get('prefix', DEFAULT_PREFIX)
full_command = (
(local_path, 'check')
+ (('--repair',) if repair else ())
+ make_check_flags(checks, check_last, prefix)
+ (('--remote-path', remote_path) if remote_path else ())
+ (('--lock-wait', str(lock_wait)) if lock_wait else ())
+ verbosity_flags
+ (('--progress',) if progress else ())
+ (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+ (repository,)
)
# The Borg repair option triggers an interactive prompt, which won't work when output is
# captured. And progress messes with the terminal directly.
if repair or progress:
execute_command(full_command, output_file=DO_NOT_CAPTURE)
else:
execute_command(full_command)
for check in checks:
write_check_time(make_check_time_path(location_config, borg_repository_id, check))
if 'extract' in checks:
extract.extract_last_archive_dry_run(repository, lock_wait, local_path, remote_path)
write_check_time(make_check_time_path(location_config, borg_repository_id, 'extract'))

41
borgmatic/borg/compact.py Normal file
View File

@ -0,0 +1,41 @@
import logging
from borgmatic.execute import execute_command
logger = logging.getLogger(__name__)
def compact_segments(
dry_run,
repository,
storage_config,
local_path='borg',
remote_path=None,
progress=False,
cleanup_commits=False,
threshold=None,
):
'''
Given dry-run flag, a local or remote repository path, and a storage config dict, compact Borg
segments in a repository.
'''
umask = storage_config.get('umask', None)
lock_wait = storage_config.get('lock_wait', None)
extra_borg_options = storage_config.get('extra_borg_options', {}).get('compact', '')
full_command = (
(local_path, 'compact')
+ (('--remote-path', remote_path) if remote_path else ())
+ (('--umask', str(umask)) if umask else ())
+ (('--lock-wait', str(lock_wait)) if lock_wait else ())
+ (('--progress',) if progress else ())
+ (('--cleanup-commits',) if cleanup_commits else ())
+ (('--threshold', str(threshold)) if threshold else ())
+ (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+ (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+ (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+ (repository,)
)
if not dry_run:
execute_command(full_command, output_log_level=logging.INFO, borg_local_path=local_path)

336
borgmatic/borg/create.py Normal file
View File

@ -0,0 +1,336 @@
import glob
import itertools
import logging
import os
import pathlib
import tempfile
from borgmatic.borg import feature, state
from borgmatic.execute import DO_NOT_CAPTURE, execute_command, execute_command_with_processes
logger = logging.getLogger(__name__)
def expand_directory(directory):
'''
Given a directory path, expand any tilde (representing a user's home directory) and any globs
therein. Return a list of one or more resulting paths.
'''
expanded_directory = os.path.expanduser(directory)
return glob.glob(expanded_directory) or [expanded_directory]
def expand_directories(directories):
'''
Given a sequence of directory paths, expand tildes and globs in each one. Return all the
resulting directories as a single flattened tuple.
'''
if directories is None:
return ()
return tuple(
itertools.chain.from_iterable(expand_directory(directory) for directory in directories)
)
def expand_home_directories(directories):
'''
Given a sequence of directory paths, expand tildes in each one. Do not perform any globbing.
Return the results as a tuple.
'''
if directories is None:
return ()
return tuple(os.path.expanduser(directory) for directory in directories)
def map_directories_to_devices(directories):
'''
Given a sequence of directories, return a map from directory to an identifier for the device on
which that directory resides or None if the path doesn't exist.
This is handy for determining whether two different directories are on the same filesystem (have
the same device identifier).
'''
return {
directory: os.stat(directory).st_dev if os.path.exists(directory) else None
for directory in directories
}
def deduplicate_directories(directory_devices):
'''
Given a map from directory to the identifier for the device on which that directory resides,
return the directories as a sorted tuple with all duplicate child directories removed. For
instance, if paths is ('/foo', '/foo/bar'), return just: ('/foo',)
The one exception to this rule is if two paths are on different filesystems (devices). In that
case, they won't get de-duplicated in case they both need to be passed to Borg (e.g. the
location.one_file_system option is true).
The idea is that if Borg is given a parent directory, then it doesn't also need to be given
child directories, because it will naturally spider the contents of the parent directory. And
there are cases where Borg coming across the same file twice will result in duplicate reads and
even hangs, e.g. when a database hook is using a named pipe for streaming database dumps to
Borg.
'''
deduplicated = set()
directories = sorted(directory_devices.keys())
for directory in directories:
deduplicated.add(directory)
parents = pathlib.PurePath(directory).parents
# If another directory in the given list is a parent of current directory (even n levels
# up) and both are on the same filesystem, then the current directory is a duplicate.
for other_directory in directories:
for parent in parents:
if (
pathlib.PurePath(other_directory) == parent
and directory_devices[directory] is not None
and directory_devices[other_directory] == directory_devices[directory]
):
if directory in deduplicated:
deduplicated.remove(directory)
break
return tuple(sorted(deduplicated))
def write_pattern_file(patterns=None):
'''
Given a sequence of patterns, write them to a named temporary file and return it. Return None
if no patterns are provided.
'''
if not patterns:
return None
pattern_file = tempfile.NamedTemporaryFile('w')
pattern_file.write('\n'.join(patterns))
pattern_file.flush()
return pattern_file
def ensure_files_readable(*filename_lists):
'''
Given a sequence of filename sequences, ensure that each filename is openable. This prevents
unreadable files from being passed to Borg, which in certain situations only warns instead of
erroring.
'''
for file_object in itertools.chain.from_iterable(
filename_list for filename_list in filename_lists if filename_list
):
open(file_object).close()
def make_pattern_flags(location_config, pattern_filename=None):
'''
Given a location config dict with a potential patterns_from option, and a filename containing
any additional patterns, return the corresponding Borg flags for those files as a tuple.
'''
pattern_filenames = tuple(location_config.get('patterns_from') or ()) + (
(pattern_filename,) if pattern_filename else ()
)
return tuple(
itertools.chain.from_iterable(
('--patterns-from', pattern_filename) for pattern_filename in pattern_filenames
)
)
def make_exclude_flags(location_config, exclude_filename=None):
'''
Given a location config dict with various exclude options, and a filename containing any exclude
patterns, return the corresponding Borg flags as a tuple.
'''
exclude_filenames = tuple(location_config.get('exclude_from') or ()) + (
(exclude_filename,) if exclude_filename else ()
)
exclude_from_flags = tuple(
itertools.chain.from_iterable(
('--exclude-from', exclude_filename) for exclude_filename in exclude_filenames
)
)
caches_flag = ('--exclude-caches',) if location_config.get('exclude_caches') else ()
if_present_flags = tuple(
itertools.chain.from_iterable(
('--exclude-if-present', if_present)
for if_present in location_config.get('exclude_if_present', ())
)
)
keep_exclude_tags_flags = (
('--keep-exclude-tags',) if location_config.get('keep_exclude_tags') else ()
)
exclude_nodump_flags = ('--exclude-nodump',) if location_config.get('exclude_nodump') else ()
return (
exclude_from_flags
+ caches_flag
+ if_present_flags
+ keep_exclude_tags_flags
+ exclude_nodump_flags
)
DEFAULT_ARCHIVE_NAME_FORMAT = '{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}'
def borgmatic_source_directories(borgmatic_source_directory):
'''
Return a list of borgmatic-specific source directories used for state like database backups.
'''
if not borgmatic_source_directory:
borgmatic_source_directory = state.DEFAULT_BORGMATIC_SOURCE_DIRECTORY
return (
[borgmatic_source_directory]
if os.path.exists(os.path.expanduser(borgmatic_source_directory))
else []
)
def create_archive(
dry_run,
repository,
location_config,
storage_config,
local_borg_version,
local_path='borg',
remote_path=None,
progress=False,
stats=False,
json=False,
files=False,
stream_processes=None,
):
'''
Given vebosity/dry-run flags, a local or remote repository path, a location config dict, and a
storage config dict, create a Borg archive and return Borg's JSON output (if any).
If a sequence of stream processes is given (instances of subprocess.Popen), then execute the
create command while also triggering the given processes to produce output.
'''
sources = deduplicate_directories(
map_directories_to_devices(
expand_directories(
location_config['source_directories']
+ borgmatic_source_directories(location_config.get('borgmatic_source_directory'))
)
)
)
try:
working_directory = os.path.expanduser(location_config.get('working_directory'))
except TypeError:
working_directory = None
pattern_file = write_pattern_file(location_config.get('patterns'))
exclude_file = write_pattern_file(
expand_home_directories(location_config.get('exclude_patterns'))
)
checkpoint_interval = storage_config.get('checkpoint_interval', None)
chunker_params = storage_config.get('chunker_params', None)
compression = storage_config.get('compression', None)
remote_rate_limit = storage_config.get('remote_rate_limit', None)
umask = storage_config.get('umask', None)
lock_wait = storage_config.get('lock_wait', None)
files_cache = location_config.get('files_cache')
archive_name_format = storage_config.get('archive_name_format', DEFAULT_ARCHIVE_NAME_FORMAT)
extra_borg_options = storage_config.get('extra_borg_options', {}).get('create', '')
if feature.available(feature.Feature.ATIME, local_borg_version):
atime_flags = ('--atime',) if location_config.get('atime') is True else ()
else:
atime_flags = ('--noatime',) if location_config.get('atime') is False else ()
if feature.available(feature.Feature.NOFLAGS, local_borg_version):
noflags_flags = ('--noflags',) if location_config.get('bsd_flags') is False else ()
else:
noflags_flags = ('--nobsdflags',) if location_config.get('bsd_flags') is False else ()
if feature.available(feature.Feature.NUMERIC_IDS, local_borg_version):
numeric_ids_flags = ('--numeric-ids',) if location_config.get('numeric_owner') else ()
else:
numeric_ids_flags = ('--numeric-owner',) if location_config.get('numeric_owner') else ()
if feature.available(feature.Feature.UPLOAD_RATELIMIT, local_borg_version):
upload_ratelimit_flags = (
('--upload-ratelimit', str(remote_rate_limit)) if remote_rate_limit else ()
)
else:
upload_ratelimit_flags = (
('--remote-ratelimit', str(remote_rate_limit)) if remote_rate_limit else ()
)
ensure_files_readable(location_config.get('patterns_from'), location_config.get('exclude_from'))
full_command = (
tuple(local_path.split(' '))
+ ('create',)
+ make_pattern_flags(location_config, pattern_file.name if pattern_file else None)
+ make_exclude_flags(location_config, exclude_file.name if exclude_file else None)
+ (('--checkpoint-interval', str(checkpoint_interval)) if checkpoint_interval else ())
+ (('--chunker-params', chunker_params) if chunker_params else ())
+ (('--compression', compression) if compression else ())
+ upload_ratelimit_flags
+ (
('--one-file-system',)
if location_config.get('one_file_system') or stream_processes
else ()
)
+ numeric_ids_flags
+ atime_flags
+ (('--noctime',) if location_config.get('ctime') is False else ())
+ (('--nobirthtime',) if location_config.get('birthtime') is False else ())
+ (('--read-special',) if (location_config.get('read_special') or stream_processes) else ())
+ noflags_flags
+ (('--files-cache', files_cache) if files_cache else ())
+ (('--remote-path', remote_path) if remote_path else ())
+ (('--umask', str(umask)) if umask else ())
+ (('--lock-wait', str(lock_wait)) if lock_wait else ())
+ (('--list', '--filter', 'AME-') if files and not json and not progress else ())
+ (('--info',) if logger.getEffectiveLevel() == logging.INFO and not json else ())
+ (('--stats',) if stats and not json and not dry_run else ())
+ (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) and not json else ())
+ (('--dry-run',) if dry_run else ())
+ (('--progress',) if progress else ())
+ (('--json',) if json else ())
+ (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+ (
'{repository}::{archive_name_format}'.format(
repository=repository, archive_name_format=archive_name_format
),
)
+ sources
)
if json:
output_log_level = None
elif (stats or files) and logger.getEffectiveLevel() == logging.WARNING:
output_log_level = logging.WARNING
else:
output_log_level = logging.INFO
# The progress output isn't compatible with captured and logged output, as progress messes with
# the terminal directly.
output_file = DO_NOT_CAPTURE if progress else None
if stream_processes:
return execute_command_with_processes(
full_command,
stream_processes,
output_log_level,
output_file,
borg_local_path=local_path,
working_directory=working_directory,
)
return execute_command(
full_command,
output_log_level,
output_file,
borg_local_path=local_path,
working_directory=working_directory,
)

View File

@ -0,0 +1,38 @@
import os
OPTION_TO_ENVIRONMENT_VARIABLE = {
'borg_base_directory': 'BORG_BASE_DIR',
'borg_config_directory': 'BORG_CONFIG_DIR',
'borg_cache_directory': 'BORG_CACHE_DIR',
'borg_security_directory': 'BORG_SECURITY_DIR',
'borg_keys_directory': 'BORG_KEYS_DIR',
'encryption_passcommand': 'BORG_PASSCOMMAND',
'encryption_passphrase': 'BORG_PASSPHRASE',
'ssh_command': 'BORG_RSH',
'temporary_directory': 'TMPDIR',
}
DEFAULT_BOOL_OPTION_TO_ENVIRONMENT_VARIABLE = {
'relocated_repo_access_is_ok': 'BORG_RELOCATED_REPO_ACCESS_IS_OK',
'unknown_unencrypted_repo_access_is_ok': 'BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK',
}
def initialize(storage_config):
for option_name, environment_variable_name in OPTION_TO_ENVIRONMENT_VARIABLE.items():
# Options from borgmatic configuration take precedence over already set BORG_* environment
# variables.
value = storage_config.get(option_name) or os.environ.get(environment_variable_name)
if value:
os.environ[environment_variable_name] = value
else:
os.environ.pop(environment_variable_name, None)
for (
option_name,
environment_variable_name,
) in DEFAULT_BOOL_OPTION_TO_ENVIRONMENT_VARIABLE.items():
value = storage_config.get(option_name, False)
os.environ[environment_variable_name] = 'yes' if value else 'no'

View File

@ -0,0 +1,64 @@
import logging
import os
from borgmatic.execute import DO_NOT_CAPTURE, execute_command
logger = logging.getLogger(__name__)
def export_tar_archive(
dry_run,
repository,
archive,
paths,
destination_path,
storage_config,
local_path='borg',
remote_path=None,
tar_filter=None,
files=False,
strip_components=None,
):
'''
Given a dry-run flag, a local or remote repository path, an archive name, zero or more paths to
export from the archive, a destination path to export to, a storage configuration dict, optional
local and remote Borg paths, an optional filter program, whether to include per-file details,
and an optional number of path components to strip, export the archive into the given
destination path as a tar-formatted file.
If the destination path is "-", then stream the output to stdout instead of to a file.
'''
umask = storage_config.get('umask', None)
lock_wait = storage_config.get('lock_wait', None)
full_command = (
(local_path, 'export-tar')
+ (('--remote-path', remote_path) if remote_path else ())
+ (('--umask', str(umask)) if umask else ())
+ (('--lock-wait', str(lock_wait)) if lock_wait else ())
+ (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+ (('--list',) if files else ())
+ (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+ (('--dry-run',) if dry_run else ())
+ (('--tar-filter', tar_filter) if tar_filter else ())
+ (('--strip-components', str(strip_components)) if strip_components else ())
+ ('::'.join((repository if ':' in repository else os.path.abspath(repository), archive)),)
+ (destination_path,)
+ (tuple(paths) if paths else ())
)
if files and logger.getEffectiveLevel() == logging.WARNING:
output_log_level = logging.WARNING
else:
output_log_level = logging.INFO
if dry_run:
logging.info('{}: Skipping export to tar file (dry run)'.format(repository))
return
execute_command(
full_command,
output_file=DO_NOT_CAPTURE if destination_path == '-' else None,
output_log_level=output_log_level,
borg_local_path=local_path,
)

127
borgmatic/borg/extract.py Normal file
View File

@ -0,0 +1,127 @@
import logging
import os
import subprocess
from borgmatic.borg import feature
from borgmatic.execute import DO_NOT_CAPTURE, execute_command
logger = logging.getLogger(__name__)
def extract_last_archive_dry_run(repository, lock_wait=None, local_path='borg', remote_path=None):
'''
Perform an extraction dry-run of the most recent archive. If there are no archives, skip the
dry-run.
'''
remote_path_flags = ('--remote-path', remote_path) if remote_path else ()
lock_wait_flags = ('--lock-wait', str(lock_wait)) if lock_wait else ()
verbosity_flags = ()
if logger.isEnabledFor(logging.DEBUG):
verbosity_flags = ('--debug', '--show-rc')
elif logger.isEnabledFor(logging.INFO):
verbosity_flags = ('--info',)
full_list_command = (
(local_path, 'list', '--short')
+ remote_path_flags
+ lock_wait_flags
+ verbosity_flags
+ (repository,)
)
list_output = execute_command(
full_list_command, output_log_level=None, borg_local_path=local_path
)
try:
last_archive_name = list_output.strip().splitlines()[-1]
except IndexError:
return
list_flag = ('--list',) if logger.isEnabledFor(logging.DEBUG) else ()
full_extract_command = (
(local_path, 'extract', '--dry-run')
+ remote_path_flags
+ lock_wait_flags
+ verbosity_flags
+ list_flag
+ (
'{repository}::{last_archive_name}'.format(
repository=repository, last_archive_name=last_archive_name
),
)
)
execute_command(full_extract_command, working_directory=None)
def extract_archive(
dry_run,
repository,
archive,
paths,
location_config,
storage_config,
local_borg_version,
local_path='borg',
remote_path=None,
destination_path=None,
strip_components=None,
progress=False,
extract_to_stdout=False,
):
'''
Given a dry-run flag, a local or remote repository path, an archive name, zero or more paths to
restore from the archive, the local Borg version string, location/storage configuration dicts,
optional local and remote Borg paths, and an optional destination path to extract to, extract
the archive into the current directory.
If extract to stdout is True, then start the extraction streaming to stdout, and return that
extract process as an instance of subprocess.Popen.
'''
umask = storage_config.get('umask', None)
lock_wait = storage_config.get('lock_wait', None)
if progress and extract_to_stdout:
raise ValueError('progress and extract_to_stdout cannot both be set')
if feature.available(feature.Feature.NUMERIC_IDS, local_borg_version):
numeric_ids_flags = ('--numeric-ids',) if location_config.get('numeric_owner') else ()
else:
numeric_ids_flags = ('--numeric-owner',) if location_config.get('numeric_owner') else ()
full_command = (
(local_path, 'extract')
+ (('--remote-path', remote_path) if remote_path else ())
+ numeric_ids_flags
+ (('--umask', str(umask)) if umask else ())
+ (('--lock-wait', str(lock_wait)) if lock_wait else ())
+ (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+ (('--debug', '--list', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+ (('--dry-run',) if dry_run else ())
+ (('--strip-components', str(strip_components)) if strip_components else ())
+ (('--progress',) if progress else ())
+ (('--stdout',) if extract_to_stdout else ())
+ ('::'.join((repository if ':' in repository else os.path.abspath(repository), archive)),)
+ (tuple(paths) if paths else ())
)
# The progress output isn't compatible with captured and logged output, as progress messes with
# the terminal directly.
if progress:
return execute_command(
full_command, output_file=DO_NOT_CAPTURE, working_directory=destination_path
)
return None
if extract_to_stdout:
return execute_command(
full_command,
output_file=subprocess.PIPE,
working_directory=destination_path,
run_to_completion=False,
)
# Don't give Borg local path, so as to error on warnings, as Borg only gives a warning if the
# restore paths don't exist in the archive!
execute_command(full_command, working_directory=destination_path)

28
borgmatic/borg/feature.py Normal file
View File

@ -0,0 +1,28 @@
from enum import Enum
from pkg_resources import parse_version
class Feature(Enum):
COMPACT = 1
ATIME = 2
NOFLAGS = 3
NUMERIC_IDS = 4
UPLOAD_RATELIMIT = 5
FEATURE_TO_MINIMUM_BORG_VERSION = {
Feature.COMPACT: parse_version('1.2.0a2'), # borg compact
Feature.ATIME: parse_version('1.2.0a7'), # borg create --atime
Feature.NOFLAGS: parse_version('1.2.0a8'), # borg create --noflags
Feature.NUMERIC_IDS: parse_version('1.2.0b3'), # borg create/extract/mount --numeric-ids
Feature.UPLOAD_RATELIMIT: parse_version('1.2.0b3'), # borg create --upload-ratelimit
}
def available(feature, borg_version):
'''
Given a Borg Feature constant and a Borg version string, return whether that feature is
available in that version of Borg.
'''
return FEATURE_TO_MINIMUM_BORG_VERSION[feature] <= parse_version(borg_version)

31
borgmatic/borg/flags.py Normal file
View File

@ -0,0 +1,31 @@
import itertools
def make_flags(name, value):
'''
Given a flag name and its value, return it formatted as Borg-compatible flags.
'''
if not value:
return ()
flag = '--{}'.format(name.replace('_', '-'))
if value is True:
return (flag,)
return (flag, str(value))
def make_flags_from_arguments(arguments, excludes=()):
'''
Given borgmatic command-line arguments as an instance of argparse.Namespace, and optionally a
list of named arguments to exclude, generate and return the corresponding Borg command-line
flags as a tuple.
'''
return tuple(
itertools.chain.from_iterable(
make_flags(name, value=getattr(arguments, name))
for name in sorted(vars(arguments))
if name not in excludes and not name.startswith('_')
)
)

45
borgmatic/borg/info.py Normal file
View File

@ -0,0 +1,45 @@
import logging
from borgmatic.borg.flags import make_flags, make_flags_from_arguments
from borgmatic.execute import execute_command
logger = logging.getLogger(__name__)
def display_archives_info(
repository, storage_config, info_arguments, local_path='borg', remote_path=None
):
'''
Given a local or remote repository path, a storage config dict, and the arguments to the info
action, display summary information for Borg archives in the repository or return JSON summary
information.
'''
lock_wait = storage_config.get('lock_wait', None)
full_command = (
(local_path, 'info')
+ (
('--info',)
if logger.getEffectiveLevel() == logging.INFO and not info_arguments.json
else ()
)
+ (
('--debug', '--show-rc')
if logger.isEnabledFor(logging.DEBUG) and not info_arguments.json
else ()
)
+ make_flags('remote-path', remote_path)
+ make_flags('lock-wait', lock_wait)
+ make_flags_from_arguments(info_arguments, excludes=('repository', 'archive'))
+ (
'::'.join((repository, info_arguments.archive))
if info_arguments.archive
else repository,
)
)
return execute_command(
full_command,
output_log_level=None if info_arguments.json else logging.WARNING,
borg_local_path=local_path,
)

57
borgmatic/borg/init.py Normal file
View File

@ -0,0 +1,57 @@
import argparse
import logging
import subprocess
from borgmatic.borg import info
from borgmatic.execute import DO_NOT_CAPTURE, execute_command
logger = logging.getLogger(__name__)
INFO_REPOSITORY_NOT_FOUND_EXIT_CODE = 2
def initialize_repository(
repository,
storage_config,
encryption_mode,
append_only=None,
storage_quota=None,
local_path='borg',
remote_path=None,
):
'''
Given a local or remote repository path, a storage configuration dict, a Borg encryption mode,
whether the repository should be append-only, and the storage quota to use, initialize the
repository. If the repository already exists, then log and skip initialization.
'''
try:
info.display_archives_info(
repository,
storage_config,
argparse.Namespace(json=True, archive=None),
local_path,
remote_path,
)
logger.info('Repository already exists. Skipping initialization.')
return
except subprocess.CalledProcessError as error:
if error.returncode != INFO_REPOSITORY_NOT_FOUND_EXIT_CODE:
raise
extra_borg_options = storage_config.get('extra_borg_options', {}).get('init', '')
init_command = (
(local_path, 'init')
+ (('--encryption', encryption_mode) if encryption_mode else ())
+ (('--append-only',) if append_only else ())
+ (('--storage-quota', storage_quota) if storage_quota else ())
+ (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+ (('--debug',) if logger.isEnabledFor(logging.DEBUG) else ())
+ (('--remote-path', remote_path) if remote_path else ())
+ (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+ (repository,)
)
# Do not capture output here, so as to support interactive prompts.
execute_command(init_command, output_file=DO_NOT_CAPTURE, borg_local_path=local_path)

160
borgmatic/borg/list.py Normal file
View File

@ -0,0 +1,160 @@
import copy
import logging
import re
from borgmatic.borg.flags import make_flags, make_flags_from_arguments
from borgmatic.execute import execute_command
logger = logging.getLogger(__name__)
def resolve_archive_name(repository, archive, storage_config, local_path='borg', remote_path=None):
'''
Given a local or remote repository path, an archive name, a storage config dict, a local Borg
path, and a remote Borg path, simply return the archive name. But if the archive name is
"latest", then instead introspect the repository for the latest archive and return its name.
Raise ValueError if "latest" is given but there are no archives in the repository.
'''
if archive != "latest":
return archive
lock_wait = storage_config.get('lock_wait', None)
full_command = (
(local_path, 'list')
+ (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+ (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+ make_flags('remote-path', remote_path)
+ make_flags('lock-wait', lock_wait)
+ make_flags('last', 1)
+ ('--short', repository)
)
output = execute_command(full_command, output_log_level=None, borg_local_path=local_path)
try:
latest_archive = output.strip().splitlines()[-1]
except IndexError:
raise ValueError('No archives found in the repository')
logger.debug('{}: Latest archive is {}'.format(repository, latest_archive))
return latest_archive
MAKE_FLAGS_EXCLUDES = ('repository', 'archive', 'successful', 'paths', 'find_paths')
def make_list_command(
repository, storage_config, list_arguments, local_path='borg', remote_path=None
):
'''
Given a local or remote repository path, a storage config dict, the arguments to the list
action, and local and remote Borg paths, return a command as a tuple to list archives or paths
within an archive.
'''
lock_wait = storage_config.get('lock_wait', None)
return (
(local_path, 'list')
+ (
('--info',)
if logger.getEffectiveLevel() == logging.INFO and not list_arguments.json
else ()
)
+ (
('--debug', '--show-rc')
if logger.isEnabledFor(logging.DEBUG) and not list_arguments.json
else ()
)
+ make_flags('remote-path', remote_path)
+ make_flags('lock-wait', lock_wait)
+ make_flags_from_arguments(list_arguments, excludes=MAKE_FLAGS_EXCLUDES,)
+ (
('::'.join((repository, list_arguments.archive)),)
if list_arguments.archive
else (repository,)
)
+ (tuple(list_arguments.paths) if list_arguments.paths else ())
)
def make_find_paths(find_paths):
'''
Given a sequence of path fragments or patterns as passed to `--find`, transform all path
fragments into glob patterns. Pass through existing patterns untouched.
For example, given find_paths of:
['foo.txt', 'pp:root/somedir']
... transform that into:
['sh:**/*foo.txt*/**', 'pp:root/somedir']
'''
if not find_paths:
return ()
return tuple(
find_path
if re.compile(r'([-!+RrPp] )|(\w\w:)').match(find_path)
else f'sh:**/*{find_path}*/**'
for find_path in find_paths
)
def list_archives(repository, storage_config, list_arguments, local_path='borg', remote_path=None):
'''
Given a local or remote repository path, a storage config dict, the arguments to the list
action, and local and remote Borg paths, display the output of listing Borg archives in the
repository or return JSON output. Or, if an archive name is given, list the files in that
archive. Or, if list_arguments.find_paths are given, list the files by searching across multiple
archives.
'''
# If there are any paths to find (and there's not a single archive already selected), start by
# getting a list of archives to search.
if list_arguments.find_paths and not list_arguments.archive:
repository_arguments = copy.copy(list_arguments)
repository_arguments.archive = None
repository_arguments.json = False
repository_arguments.format = None
# Ask Borg to list archives. Capture its output for use below.
archive_lines = tuple(
execute_command(
make_list_command(
repository, storage_config, repository_arguments, local_path, remote_path
),
output_log_level=None,
borg_local_path=local_path,
)
.strip('\n')
.split('\n')
)
else:
archive_lines = (list_arguments.archive,)
# For each archive listed by Borg, run list on the contents of that archive.
for archive_line in archive_lines:
try:
archive = archive_line.split()[0]
except (AttributeError, IndexError):
archive = None
if archive:
logger.warning(archive_line)
archive_arguments = copy.copy(list_arguments)
archive_arguments.archive = archive
main_command = make_list_command(
repository, storage_config, archive_arguments, local_path, remote_path
) + make_find_paths(list_arguments.find_paths)
output = execute_command(
main_command,
output_log_level=None if list_arguments.json else logging.WARNING,
borg_local_path=local_path,
)
if list_arguments.json:
return output

46
borgmatic/borg/mount.py Normal file
View File

@ -0,0 +1,46 @@
import logging
from borgmatic.execute import DO_NOT_CAPTURE, execute_command
logger = logging.getLogger(__name__)
def mount_archive(
repository,
archive,
mount_point,
paths,
foreground,
options,
storage_config,
local_path='borg',
remote_path=None,
):
'''
Given a local or remote repository path, an optional archive name, a filesystem mount point,
zero or more paths to mount from the archive, extra Borg mount options, a storage configuration
dict, and optional local and remote Borg paths, mount the archive onto the mount point.
'''
umask = storage_config.get('umask', None)
lock_wait = storage_config.get('lock_wait', None)
full_command = (
(local_path, 'mount')
+ (('--remote-path', remote_path) if remote_path else ())
+ (('--umask', str(umask)) if umask else ())
+ (('--lock-wait', str(lock_wait)) if lock_wait else ())
+ (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+ (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+ (('--foreground',) if foreground else ())
+ (('-o', options) if options else ())
+ (('::'.join((repository, archive)),) if archive else (repository,))
+ (mount_point,)
+ (tuple(paths) if paths else ())
)
# Don't capture the output when foreground mode is used so that ctrl-C can work properly.
if foreground:
execute_command(full_command, output_file=DO_NOT_CAPTURE, borg_local_path=local_path)
return
execute_command(full_command, borg_local_path=local_path)

75
borgmatic/borg/prune.py Normal file
View File

@ -0,0 +1,75 @@
import logging
from borgmatic.execute import execute_command
logger = logging.getLogger(__name__)
def _make_prune_flags(retention_config):
'''
Given a retention config dict mapping from option name to value, tranform it into an iterable of
command-line name-value flag pairs.
For example, given a retention config of:
{'keep_weekly': 4, 'keep_monthly': 6}
This will be returned as an iterable of:
(
('--keep-weekly', '4'),
('--keep-monthly', '6'),
)
'''
config = retention_config.copy()
if 'prefix' not in config:
config['prefix'] = '{hostname}-'
elif not config['prefix']:
config.pop('prefix')
return (
('--' + option_name.replace('_', '-'), str(value)) for option_name, value in config.items()
)
def prune_archives(
dry_run,
repository,
storage_config,
retention_config,
local_path='borg',
remote_path=None,
stats=False,
files=False,
):
'''
Given dry-run flag, a local or remote repository path, a storage config dict, and a
retention config dict, prune Borg archives according to the retention policy specified in that
configuration.
'''
umask = storage_config.get('umask', None)
lock_wait = storage_config.get('lock_wait', None)
extra_borg_options = storage_config.get('extra_borg_options', {}).get('prune', '')
full_command = (
(local_path, 'prune')
+ tuple(element for pair in _make_prune_flags(retention_config) for element in pair)
+ (('--remote-path', remote_path) if remote_path else ())
+ (('--umask', str(umask)) if umask else ())
+ (('--lock-wait', str(lock_wait)) if lock_wait else ())
+ (('--stats',) if stats and not dry_run else ())
+ (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+ (('--list',) if files else ())
+ (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+ (('--dry-run',) if dry_run else ())
+ (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+ (repository,)
)
if (stats or files) and logger.getEffectiveLevel() == logging.WARNING:
output_log_level = logging.WARNING
else:
output_log_level = logging.INFO
execute_command(full_command, output_log_level=output_log_level, borg_local_path=local_path)

1
borgmatic/borg/state.py Normal file
View File

@ -0,0 +1 @@
DEFAULT_BORGMATIC_SOURCE_DIRECTORY = '~/.borgmatic'

20
borgmatic/borg/umount.py Normal file
View File

@ -0,0 +1,20 @@
import logging
from borgmatic.execute import execute_command
logger = logging.getLogger(__name__)
def unmount_archive(mount_point, local_path='borg'):
'''
Given a mounted filesystem mount point, and an optional local Borg paths, umount the filesystem
from the mount point.
'''
full_command = (
(local_path, 'umount')
+ (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+ (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+ (mount_point,)
)
execute_command(full_command)

25
borgmatic/borg/version.py Normal file
View File

@ -0,0 +1,25 @@
import logging
from borgmatic.execute import execute_command
logger = logging.getLogger(__name__)
def local_borg_version(local_path='borg'):
'''
Given a local Borg binary path, return a version string for it.
Raise OSError or CalledProcessError if there is a problem running Borg.
Raise ValueError if the version cannot be parsed.
'''
full_command = (
(local_path, '--version')
+ (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+ (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
)
output = execute_command(full_command, output_log_level=None, borg_local_path=local_path)
try:
return output.split(' ')[1].strip()
except IndexError:
raise ValueError('Could not parse Borg version string')

View File

@ -0,0 +1,699 @@
import collections
from argparse import Action, ArgumentParser
from borgmatic.config import collect
SUBPARSER_ALIASES = {
'init': ['--init', '-I'],
'prune': ['--prune', '-p'],
'compact': [],
'create': ['--create', '-C'],
'check': ['--check', '-k'],
'extract': ['--extract', '-x'],
'export-tar': ['--export-tar'],
'mount': ['--mount', '-m'],
'umount': ['--umount', '-u'],
'restore': ['--restore', '-r'],
'list': ['--list', '-l'],
'info': ['--info', '-i'],
'borg': [],
}
def parse_subparser_arguments(unparsed_arguments, subparsers):
'''
Given a sequence of arguments and a dict from subparser name to argparse.ArgumentParser
instance, give each requested action's subparser a shot at parsing all arguments. This allows
common arguments like "--repository" to be shared across multiple subparsers.
Return the result as a tuple of (a dict mapping from subparser name to a parsed namespace of
arguments, a list of remaining arguments not claimed by any subparser).
'''
arguments = collections.OrderedDict()
remaining_arguments = list(unparsed_arguments)
alias_to_subparser_name = {
alias: subparser_name
for subparser_name, aliases in SUBPARSER_ALIASES.items()
for alias in aliases
}
# If the "borg" action is used, skip all other subparsers. This avoids confusion like
# "borg list" triggering borgmatic's own list action.
if 'borg' in unparsed_arguments:
subparsers = {'borg': subparsers['borg']}
for subparser_name, subparser in subparsers.items():
if subparser_name not in remaining_arguments:
continue
canonical_name = alias_to_subparser_name.get(subparser_name, subparser_name)
# If a parsed value happens to be the same as the name of a subparser, remove it from the
# remaining arguments. This prevents, for instance, "check --only extract" from triggering
# the "extract" subparser.
parsed, unused_remaining = subparser.parse_known_args(unparsed_arguments)
for value in vars(parsed).values():
if isinstance(value, str):
if value in subparsers:
remaining_arguments.remove(value)
elif isinstance(value, list):
for item in value:
if item in subparsers:
remaining_arguments.remove(item)
arguments[canonical_name] = parsed
# If no actions are explicitly requested, assume defaults: prune, compact, create, and check.
if not arguments and '--help' not in unparsed_arguments and '-h' not in unparsed_arguments:
for subparser_name in ('prune', 'compact', 'create', 'check'):
subparser = subparsers[subparser_name]
parsed, unused_remaining = subparser.parse_known_args(unparsed_arguments)
arguments[subparser_name] = parsed
remaining_arguments = list(unparsed_arguments)
# Now ask each subparser, one by one, to greedily consume arguments.
for subparser_name, subparser in subparsers.items():
if subparser_name not in arguments.keys():
continue
subparser = subparsers[subparser_name]
unused_parsed, remaining_arguments = subparser.parse_known_args(remaining_arguments)
# Special case: If "borg" is present in the arguments, consume all arguments after (+1) the
# "borg" action.
if 'borg' in arguments:
borg_options_index = remaining_arguments.index('borg') + 1
arguments['borg'].options = remaining_arguments[borg_options_index:]
remaining_arguments = remaining_arguments[:borg_options_index]
# Remove the subparser names themselves.
for subparser_name, subparser in subparsers.items():
if subparser_name in remaining_arguments:
remaining_arguments.remove(subparser_name)
return (arguments, remaining_arguments)
class Extend_action(Action):
'''
An argparse action to support Python 3.8's "extend" action in older versions of Python.
'''
def __call__(self, parser, namespace, values, option_string=None):
items = getattr(namespace, self.dest, None)
if items:
items.extend(values)
else:
setattr(namespace, self.dest, list(values))
def make_parsers():
'''
Build a top-level parser and its subparsers and return them as a tuple.
'''
config_paths = collect.get_default_config_paths(expand_home=True)
unexpanded_config_paths = collect.get_default_config_paths(expand_home=False)
global_parser = ArgumentParser(add_help=False)
global_parser.register('action', 'extend', Extend_action)
global_group = global_parser.add_argument_group('global arguments')
global_group.add_argument(
'-c',
'--config',
nargs='*',
dest='config_paths',
default=config_paths,
help='Configuration filenames or directories, defaults to: {}'.format(
' '.join(unexpanded_config_paths)
),
)
global_group.add_argument(
'--excludes',
dest='excludes_filename',
help='Deprecated in favor of exclude_patterns within configuration',
)
global_group.add_argument(
'-n',
'--dry-run',
dest='dry_run',
action='store_true',
help='Go through the motions, but do not actually write to any repositories',
)
global_group.add_argument(
'-nc', '--no-color', dest='no_color', action='store_true', help='Disable colored output'
)
global_group.add_argument(
'-v',
'--verbosity',
type=int,
choices=range(-1, 3),
default=0,
help='Display verbose progress to the console (from only errors to very verbose: -1, 0, 1, or 2)',
)
global_group.add_argument(
'--syslog-verbosity',
type=int,
choices=range(-1, 3),
default=0,
help='Log verbose progress to syslog (from only errors to very verbose: -1, 0, 1, or 2). Ignored when console is interactive or --log-file is given',
)
global_group.add_argument(
'--log-file-verbosity',
type=int,
choices=range(-1, 3),
default=0,
help='Log verbose progress to log file (from only errors to very verbose: -1, 0, 1, or 2). Only used when --log-file is given',
)
global_group.add_argument(
'--monitoring-verbosity',
type=int,
choices=range(-1, 3),
default=0,
help='Log verbose progress to monitoring integrations that support logging (from only errors to very verbose: -1, 0, 1, or 2)',
)
global_group.add_argument(
'--log-file',
type=str,
default=None,
help='Write log messages to this file instead of syslog',
)
global_group.add_argument(
'--override',
metavar='SECTION.OPTION=VALUE',
nargs='+',
dest='overrides',
action='extend',
help='One or more configuration file options to override with specified values',
)
global_group.add_argument(
'--bash-completion',
default=False,
action='store_true',
help='Show bash completion script and exit',
)
global_group.add_argument(
'--version',
dest='version',
default=False,
action='store_true',
help='Display installed version number of borgmatic and exit',
)
top_level_parser = ArgumentParser(
description='''
Simple, configuration-driven backup software for servers and workstations. If none of
the action options are given, then borgmatic defaults to: prune, compact, create, and
check.
''',
parents=[global_parser],
)
subparsers = top_level_parser.add_subparsers(
title='actions',
metavar='',
help='Specify zero or more actions. Defaults to prune, compact, create, and check. Use --help with action for details:',
)
init_parser = subparsers.add_parser(
'init',
aliases=SUBPARSER_ALIASES['init'],
help='Initialize an empty Borg repository',
description='Initialize an empty Borg repository',
add_help=False,
)
init_group = init_parser.add_argument_group('init arguments')
init_group.add_argument(
'-e',
'--encryption',
dest='encryption_mode',
help='Borg repository encryption mode',
required=True,
)
init_group.add_argument(
'--append-only',
dest='append_only',
action='store_true',
help='Create an append-only repository',
)
init_group.add_argument(
'--storage-quota',
dest='storage_quota',
help='Create a repository with a fixed storage quota',
)
init_group.add_argument('-h', '--help', action='help', help='Show this help message and exit')
prune_parser = subparsers.add_parser(
'prune',
aliases=SUBPARSER_ALIASES['prune'],
help='Prune archives according to the retention policy (with Borg 1.2+, run compact afterwards to actually free space)',
description='Prune archives according to the retention policy (with Borg 1.2+, run compact afterwards to actually free space)',
add_help=False,
)
prune_group = prune_parser.add_argument_group('prune arguments')
prune_group.add_argument(
'--stats',
dest='stats',
default=False,
action='store_true',
help='Display statistics of archive',
)
prune_group.add_argument(
'--files', dest='files', default=False, action='store_true', help='Show per-file details'
)
prune_group.add_argument('-h', '--help', action='help', help='Show this help message and exit')
compact_parser = subparsers.add_parser(
'compact',
aliases=SUBPARSER_ALIASES['compact'],
help='Compact segments to free space (Borg 1.2+ only)',
description='Compact segments to free space (Borg 1.2+ only)',
add_help=False,
)
compact_group = compact_parser.add_argument_group('compact arguments')
compact_group.add_argument(
'--progress',
dest='progress',
default=False,
action='store_true',
help='Display progress as each segment is compacted',
)
compact_group.add_argument(
'--cleanup-commits',
dest='cleanup_commits',
default=False,
action='store_true',
help='Cleanup commit-only 17-byte segment files left behind by Borg 1.1',
)
compact_group.add_argument(
'--threshold',
type=int,
dest='threshold',
help='Minimum saved space percentage threshold for compacting a segment, defaults to 10',
)
compact_group.add_argument(
'-h', '--help', action='help', help='Show this help message and exit'
)
create_parser = subparsers.add_parser(
'create',
aliases=SUBPARSER_ALIASES['create'],
help='Create archives (actually perform backups)',
description='Create archives (actually perform backups)',
add_help=False,
)
create_group = create_parser.add_argument_group('create arguments')
create_group.add_argument(
'--progress',
dest='progress',
default=False,
action='store_true',
help='Display progress for each file as it is backed up',
)
create_group.add_argument(
'--stats',
dest='stats',
default=False,
action='store_true',
help='Display statistics of archive',
)
create_group.add_argument(
'--files', dest='files', default=False, action='store_true', help='Show per-file details'
)
create_group.add_argument(
'--json', dest='json', default=False, action='store_true', help='Output results as JSON'
)
create_group.add_argument('-h', '--help', action='help', help='Show this help message and exit')
check_parser = subparsers.add_parser(
'check',
aliases=SUBPARSER_ALIASES['check'],
help='Check archives for consistency',
description='Check archives for consistency',
add_help=False,
)
check_group = check_parser.add_argument_group('check arguments')
check_group.add_argument(
'--progress',
dest='progress',
default=False,
action='store_true',
help='Display progress for each file as it is checked',
)
check_group.add_argument(
'--repair',
dest='repair',
default=False,
action='store_true',
help='Attempt to repair any inconsistencies found (for interactive use)',
)
check_group.add_argument(
'--only',
metavar='CHECK',
choices=('repository', 'archives', 'data', 'extract'),
dest='only',
action='append',
help='Run a particular consistency check (repository, archives, data, or extract) instead of configured checks (subject to configured frequency, can specify flag multiple times)',
)
check_group.add_argument(
'--force',
default=False,
action='store_true',
help='Ignore configured check frequencies and run checks unconditionally',
)
check_group.add_argument('-h', '--help', action='help', help='Show this help message and exit')
extract_parser = subparsers.add_parser(
'extract',
aliases=SUBPARSER_ALIASES['extract'],
help='Extract files from a named archive to the current directory',
description='Extract a named archive to the current directory',
add_help=False,
)
extract_group = extract_parser.add_argument_group('extract arguments')
extract_group.add_argument(
'--repository',
help='Path of repository to extract, defaults to the configured repository if there is only one',
)
extract_group.add_argument(
'--archive', help='Name of archive to extract (or "latest")', required=True
)
extract_group.add_argument(
'--path',
'--restore-path',
metavar='PATH',
nargs='+',
dest='paths',
help='Paths to extract from archive, defaults to the entire archive',
)
extract_group.add_argument(
'--destination',
metavar='PATH',
dest='destination',
help='Directory to extract files into, defaults to the current directory',
)
extract_group.add_argument(
'--strip-components',
type=int,
metavar='NUMBER',
dest='strip_components',
help='Number of leading path components to remove from each extracted path. Skip paths with fewer elements',
)
extract_group.add_argument(
'--progress',
dest='progress',
default=False,
action='store_true',
help='Display progress for each file as it is extracted',
)
extract_group.add_argument(
'-h', '--help', action='help', help='Show this help message and exit'
)
export_tar_parser = subparsers.add_parser(
'export-tar',
aliases=SUBPARSER_ALIASES['export-tar'],
help='Export an archive to a tar-formatted file or stream',
description='Export an archive to a tar-formatted file or stream',
add_help=False,
)
export_tar_group = export_tar_parser.add_argument_group('export-tar arguments')
export_tar_group.add_argument(
'--repository',
help='Path of repository to export from, defaults to the configured repository if there is only one',
)
export_tar_group.add_argument(
'--archive', help='Name of archive to export (or "latest")', required=True
)
export_tar_group.add_argument(
'--path',
metavar='PATH',
nargs='+',
dest='paths',
help='Paths to export from archive, defaults to the entire archive',
)
export_tar_group.add_argument(
'--destination',
metavar='PATH',
dest='destination',
help='Path to destination export tar file, or "-" for stdout (but be careful about dirtying output with --verbosity or --files)',
required=True,
)
export_tar_group.add_argument(
'--tar-filter', help='Name of filter program to pipe data through'
)
export_tar_group.add_argument(
'--files', default=False, action='store_true', help='Show per-file details'
)
export_tar_group.add_argument(
'--strip-components',
type=int,
metavar='NUMBER',
dest='strip_components',
help='Number of leading path components to remove from each exported path. Skip paths with fewer elements',
)
export_tar_group.add_argument(
'-h', '--help', action='help', help='Show this help message and exit'
)
mount_parser = subparsers.add_parser(
'mount',
aliases=SUBPARSER_ALIASES['mount'],
help='Mount files from a named archive as a FUSE filesystem',
description='Mount a named archive as a FUSE filesystem',
add_help=False,
)
mount_group = mount_parser.add_argument_group('mount arguments')
mount_group.add_argument(
'--repository',
help='Path of repository to use, defaults to the configured repository if there is only one',
)
mount_group.add_argument('--archive', help='Name of archive to mount (or "latest")')
mount_group.add_argument(
'--mount-point',
metavar='PATH',
dest='mount_point',
help='Path where filesystem is to be mounted',
required=True,
)
mount_group.add_argument(
'--path',
metavar='PATH',
nargs='+',
dest='paths',
help='Paths to mount from archive, defaults to the entire archive',
)
mount_group.add_argument(
'--foreground',
dest='foreground',
default=False,
action='store_true',
help='Stay in foreground until ctrl-C is pressed',
)
mount_group.add_argument('--options', dest='options', help='Extra Borg mount options')
mount_group.add_argument('-h', '--help', action='help', help='Show this help message and exit')
umount_parser = subparsers.add_parser(
'umount',
aliases=SUBPARSER_ALIASES['umount'],
help='Unmount a FUSE filesystem that was mounted with "borgmatic mount"',
description='Unmount a mounted FUSE filesystem',
add_help=False,
)
umount_group = umount_parser.add_argument_group('umount arguments')
umount_group.add_argument(
'--mount-point',
metavar='PATH',
dest='mount_point',
help='Path of filesystem to unmount',
required=True,
)
umount_group.add_argument('-h', '--help', action='help', help='Show this help message and exit')
restore_parser = subparsers.add_parser(
'restore',
aliases=SUBPARSER_ALIASES['restore'],
help='Restore database dumps from a named archive',
description='Restore database dumps from a named archive. (To extract files instead, use "borgmatic extract".)',
add_help=False,
)
restore_group = restore_parser.add_argument_group('restore arguments')
restore_group.add_argument(
'--repository',
help='Path of repository to restore from, defaults to the configured repository if there is only one',
)
restore_group.add_argument(
'--archive', help='Name of archive to restore from (or "latest")', required=True
)
restore_group.add_argument(
'--database',
metavar='NAME',
nargs='+',
dest='databases',
help='Names of databases to restore from archive, defaults to all databases. Note that any databases to restore must be defined in borgmatic\'s configuration',
)
restore_group.add_argument(
'-h', '--help', action='help', help='Show this help message and exit'
)
list_parser = subparsers.add_parser(
'list',
aliases=SUBPARSER_ALIASES['list'],
help='List archives',
description='List archives or the contents of an archive',
add_help=False,
)
list_group = list_parser.add_argument_group('list arguments')
list_group.add_argument(
'--repository', help='Path of repository to list, defaults to the configured repositories',
)
list_group.add_argument('--archive', help='Name of archive to list (or "latest")')
list_group.add_argument(
'--path',
metavar='PATH',
nargs='+',
dest='paths',
help='Paths or patterns to list from a single selected archive (via "--archive"), defaults to listing the entire archive',
)
list_group.add_argument(
'--find',
metavar='PATH',
nargs='+',
dest='find_paths',
help='Partial paths or patterns to search for and list across multiple archives',
)
list_group.add_argument(
'--short', default=False, action='store_true', help='Output only archive or path names'
)
list_group.add_argument('--format', help='Format for file listing')
list_group.add_argument(
'--json', default=False, action='store_true', help='Output results as JSON'
)
list_group.add_argument(
'-P', '--prefix', help='Only list archive names starting with this prefix'
)
list_group.add_argument(
'-a', '--glob-archives', metavar='GLOB', help='Only list archive names matching this glob'
)
list_group.add_argument(
'--successful',
default=True,
action='store_true',
help='Deprecated in favor of listing successful (non-checkpoint) backups by default in newer versions of Borg',
)
list_group.add_argument(
'--sort-by', metavar='KEYS', help='Comma-separated list of sorting keys'
)
list_group.add_argument(
'--first', metavar='N', help='List first N archives after other filters are applied'
)
list_group.add_argument(
'--last', metavar='N', help='List last N archives after other filters are applied'
)
list_group.add_argument(
'-e', '--exclude', metavar='PATTERN', help='Exclude paths matching the pattern'
)
list_group.add_argument(
'--exclude-from', metavar='FILENAME', help='Exclude paths from exclude file, one per line'
)
list_group.add_argument('--pattern', help='Include or exclude paths matching a pattern')
list_group.add_argument(
'--patterns-from',
metavar='FILENAME',
help='Include or exclude paths matching patterns from pattern file, one per line',
)
list_group.add_argument('-h', '--help', action='help', help='Show this help message and exit')
info_parser = subparsers.add_parser(
'info',
aliases=SUBPARSER_ALIASES['info'],
help='Display summary information on archives',
description='Display summary information on archives',
add_help=False,
)
info_group = info_parser.add_argument_group('info arguments')
info_group.add_argument(
'--repository',
help='Path of repository to show info for, defaults to the configured repository if there is only one',
)
info_group.add_argument('--archive', help='Name of archive to show info for (or "latest")')
info_group.add_argument(
'--json', dest='json', default=False, action='store_true', help='Output results as JSON'
)
info_group.add_argument(
'-P', '--prefix', help='Only show info for archive names starting with this prefix'
)
info_group.add_argument(
'-a',
'--glob-archives',
metavar='GLOB',
help='Only show info for archive names matching this glob',
)
info_group.add_argument(
'--sort-by', metavar='KEYS', help='Comma-separated list of sorting keys'
)
info_group.add_argument(
'--first',
metavar='N',
help='Show info for first N archives after other filters are applied',
)
info_group.add_argument(
'--last', metavar='N', help='Show info for last N archives after other filters are applied'
)
info_group.add_argument('-h', '--help', action='help', help='Show this help message and exit')
borg_parser = subparsers.add_parser(
'borg',
aliases=SUBPARSER_ALIASES['borg'],
help='Run an arbitrary Borg command',
description='Run an arbitrary Borg command based on borgmatic\'s configuration',
add_help=False,
)
borg_group = borg_parser.add_argument_group('borg arguments')
borg_group.add_argument(
'--repository',
help='Path of repository to pass to Borg, defaults to the configured repositories',
)
borg_group.add_argument('--archive', help='Name of archive to pass to Borg (or "latest")')
borg_group.add_argument(
'--',
metavar='OPTION',
dest='options',
nargs='+',
help='Options to pass to Borg, command first ("create", "list", etc). "--" is optional. To specify the repository or the archive, you must use --repository or --archive instead of providing them here.',
)
borg_group.add_argument('-h', '--help', action='help', help='Show this help message and exit')
return top_level_parser, subparsers
def parse_arguments(*unparsed_arguments):
'''
Given command-line arguments with which this script was invoked, parse the arguments and return
them as a dict mapping from subparser name (or "global") to an argparse.Namespace instance.
'''
top_level_parser, subparsers = make_parsers()
arguments, remaining_arguments = parse_subparser_arguments(
unparsed_arguments, subparsers.choices
)
arguments['global'] = top_level_parser.parse_args(remaining_arguments)
if arguments['global'].excludes_filename:
raise ValueError(
'The --excludes option has been replaced with exclude_patterns in configuration'
)
if 'init' in arguments and arguments['global'].dry_run:
raise ValueError('The init action cannot be used with the --dry-run option')
if (
'list' in arguments
and 'info' in arguments
and arguments['list'].json
and arguments['info'].json
):
raise ValueError('With the --json option, list and info actions cannot be used together')
return arguments

View File

@ -0,0 +1,934 @@
import collections
import copy
import json
import logging
import os
import sys
import time
from queue import Queue
from subprocess import CalledProcessError
import colorama
import pkg_resources
import borgmatic.commands.completion
from borgmatic.borg import borg as borg_borg
from borgmatic.borg import check as borg_check
from borgmatic.borg import compact as borg_compact
from borgmatic.borg import create as borg_create
from borgmatic.borg import environment as borg_environment
from borgmatic.borg import export_tar as borg_export_tar
from borgmatic.borg import extract as borg_extract
from borgmatic.borg import feature as borg_feature
from borgmatic.borg import info as borg_info
from borgmatic.borg import init as borg_init
from borgmatic.borg import list as borg_list
from borgmatic.borg import mount as borg_mount
from borgmatic.borg import prune as borg_prune
from borgmatic.borg import umount as borg_umount
from borgmatic.borg import version as borg_version
from borgmatic.commands.arguments import parse_arguments
from borgmatic.config import checks, collect, convert, validate
from borgmatic.hooks import command, dispatch, dump, monitor
from borgmatic.logger import configure_logging, should_do_markup
from borgmatic.signals import configure_signals
from borgmatic.verbosity import verbosity_to_log_level
logger = logging.getLogger(__name__)
LEGACY_CONFIG_PATH = '/etc/borgmatic/config'
def run_configuration(config_filename, config, arguments):
'''
Given a config filename, the corresponding parsed config dict, and command-line arguments as a
dict from subparser name to a namespace of parsed arguments, execute the defined prune, compact,
create, check, and/or other actions.
Yield a combination of:
* JSON output strings from successfully executing any actions that produce JSON
* logging.LogRecord instances containing errors from any actions or backup hooks that fail
'''
(location, storage, retention, consistency, hooks) = (
config.get(section_name, {})
for section_name in ('location', 'storage', 'retention', 'consistency', 'hooks')
)
global_arguments = arguments['global']
local_path = location.get('local_path', 'borg')
remote_path = location.get('remote_path')
retries = storage.get('retries', 0)
retry_wait = storage.get('retry_wait', 0)
borg_environment.initialize(storage)
encountered_error = None
error_repository = ''
using_primary_action = {'prune', 'compact', 'create', 'check'}.intersection(arguments)
monitoring_log_level = verbosity_to_log_level(global_arguments.monitoring_verbosity)
try:
local_borg_version = borg_version.local_borg_version(local_path)
except (OSError, CalledProcessError, ValueError) as error:
yield from log_error_records(
'{}: Error getting local Borg version'.format(config_filename), error
)
return
try:
if using_primary_action:
dispatch.call_hooks(
'initialize_monitor',
hooks,
config_filename,
monitor.MONITOR_HOOK_NAMES,
monitoring_log_level,
global_arguments.dry_run,
)
if using_primary_action:
dispatch.call_hooks(
'ping_monitor',
hooks,
config_filename,
monitor.MONITOR_HOOK_NAMES,
monitor.State.START,
monitoring_log_level,
global_arguments.dry_run,
)
except (OSError, CalledProcessError) as error:
if command.considered_soft_failure(config_filename, error):
return
encountered_error = error
yield from log_error_records('{}: Error pinging monitor'.format(config_filename), error)
if not encountered_error:
repo_queue = Queue()
for repo in location['repositories']:
repo_queue.put((repo, 0),)
while not repo_queue.empty():
repository_path, retry_num = repo_queue.get()
timeout = retry_num * retry_wait
if timeout:
logger.warning(f'{config_filename}: Sleeping {timeout}s before next retry')
time.sleep(timeout)
try:
yield from run_actions(
arguments=arguments,
config_filename=config_filename,
location=location,
storage=storage,
retention=retention,
consistency=consistency,
hooks=hooks,
local_path=local_path,
remote_path=remote_path,
local_borg_version=local_borg_version,
repository_path=repository_path,
)
except (OSError, CalledProcessError, ValueError) as error:
if retry_num < retries:
repo_queue.put((repository_path, retry_num + 1),)
tuple( # Consume the generator so as to trigger logging.
log_error_records(
'{}: Error running actions for repository'.format(repository_path),
error,
levelno=logging.WARNING,
log_command_error_output=True,
)
)
logger.warning(
f'{config_filename}: Retrying... attempt {retry_num + 1}/{retries}'
)
continue
if command.considered_soft_failure(config_filename, error):
return
yield from log_error_records(
'{}: Error running actions for repository'.format(repository_path), error
)
encountered_error = error
error_repository = repository_path
if not encountered_error:
try:
if using_primary_action:
dispatch.call_hooks(
'ping_monitor',
hooks,
config_filename,
monitor.MONITOR_HOOK_NAMES,
monitor.State.FINISH,
monitoring_log_level,
global_arguments.dry_run,
)
dispatch.call_hooks(
'destroy_monitor',
hooks,
config_filename,
monitor.MONITOR_HOOK_NAMES,
monitoring_log_level,
global_arguments.dry_run,
)
except (OSError, CalledProcessError) as error:
if command.considered_soft_failure(config_filename, error):
return
encountered_error = error
yield from log_error_records('{}: Error pinging monitor'.format(config_filename), error)
if encountered_error and using_primary_action:
try:
command.execute_hook(
hooks.get('on_error'),
hooks.get('umask'),
config_filename,
'on-error',
global_arguments.dry_run,
repository=error_repository,
error=encountered_error,
output=getattr(encountered_error, 'output', ''),
)
dispatch.call_hooks(
'ping_monitor',
hooks,
config_filename,
monitor.MONITOR_HOOK_NAMES,
monitor.State.FAIL,
monitoring_log_level,
global_arguments.dry_run,
)
dispatch.call_hooks(
'destroy_monitor',
hooks,
config_filename,
monitor.MONITOR_HOOK_NAMES,
monitoring_log_level,
global_arguments.dry_run,
)
except (OSError, CalledProcessError) as error:
if command.considered_soft_failure(config_filename, error):
return
yield from log_error_records(
'{}: Error running on-error hook'.format(config_filename), error
)
def run_actions(
*,
arguments,
config_filename,
location,
storage,
retention,
consistency,
hooks,
local_path,
remote_path,
local_borg_version,
repository_path,
):
'''
Given parsed command-line arguments as an argparse.ArgumentParser instance, the configuration
filename, several different configuration dicts, local and remote paths to Borg, a local Borg
version string, and a repository name, run all actions from the command-line arguments on the
given repository.
Yield JSON output strings from executing any actions that produce JSON.
Raise OSError or subprocess.CalledProcessError if an error occurs running a command for an
action or a hook. Raise ValueError if the arguments or configuration passed to action are
invalid.
'''
repository = os.path.expanduser(repository_path)
global_arguments = arguments['global']
dry_run_label = ' (dry run; not making any changes)' if global_arguments.dry_run else ''
hook_context = {
'repository': repository_path,
# Deprecated: For backwards compatibility with borgmatic < 1.6.0.
'repositories': ','.join(location['repositories']),
}
if 'init' in arguments:
logger.info('{}: Initializing repository'.format(repository))
borg_init.initialize_repository(
repository,
storage,
arguments['init'].encryption_mode,
arguments['init'].append_only,
arguments['init'].storage_quota,
local_path=local_path,
remote_path=remote_path,
)
if 'prune' in arguments:
command.execute_hook(
hooks.get('before_prune'),
hooks.get('umask'),
config_filename,
'pre-prune',
global_arguments.dry_run,
**hook_context,
)
logger.info('{}: Pruning archives{}'.format(repository, dry_run_label))
borg_prune.prune_archives(
global_arguments.dry_run,
repository,
storage,
retention,
local_path=local_path,
remote_path=remote_path,
stats=arguments['prune'].stats,
files=arguments['prune'].files,
)
command.execute_hook(
hooks.get('after_prune'),
hooks.get('umask'),
config_filename,
'post-prune',
global_arguments.dry_run,
**hook_context,
)
if 'compact' in arguments:
command.execute_hook(
hooks.get('before_compact'),
hooks.get('umask'),
config_filename,
'pre-compact',
global_arguments.dry_run,
)
if borg_feature.available(borg_feature.Feature.COMPACT, local_borg_version):
logger.info('{}: Compacting segments{}'.format(repository, dry_run_label))
borg_compact.compact_segments(
global_arguments.dry_run,
repository,
storage,
local_path=local_path,
remote_path=remote_path,
progress=arguments['compact'].progress,
cleanup_commits=arguments['compact'].cleanup_commits,
threshold=arguments['compact'].threshold,
)
else: # pragma: nocover
logger.info(
'{}: Skipping compact (only available/needed in Borg 1.2+)'.format(repository)
)
command.execute_hook(
hooks.get('after_compact'),
hooks.get('umask'),
config_filename,
'post-compact',
global_arguments.dry_run,
)
if 'create' in arguments:
command.execute_hook(
hooks.get('before_backup'),
hooks.get('umask'),
config_filename,
'pre-backup',
global_arguments.dry_run,
**hook_context,
)
logger.info('{}: Creating archive{}'.format(repository, dry_run_label))
dispatch.call_hooks(
'remove_database_dumps',
hooks,
repository,
dump.DATABASE_HOOK_NAMES,
location,
global_arguments.dry_run,
)
active_dumps = dispatch.call_hooks(
'dump_databases',
hooks,
repository,
dump.DATABASE_HOOK_NAMES,
location,
global_arguments.dry_run,
)
stream_processes = [process for processes in active_dumps.values() for process in processes]
json_output = borg_create.create_archive(
global_arguments.dry_run,
repository,
location,
storage,
local_borg_version,
local_path=local_path,
remote_path=remote_path,
progress=arguments['create'].progress,
stats=arguments['create'].stats,
json=arguments['create'].json,
files=arguments['create'].files,
stream_processes=stream_processes,
)
if json_output: # pragma: nocover
yield json.loads(json_output)
dispatch.call_hooks(
'remove_database_dumps',
hooks,
config_filename,
dump.DATABASE_HOOK_NAMES,
location,
global_arguments.dry_run,
)
command.execute_hook(
hooks.get('after_backup'),
hooks.get('umask'),
config_filename,
'post-backup',
global_arguments.dry_run,
**hook_context,
)
if 'check' in arguments and checks.repository_enabled_for_checks(repository, consistency):
command.execute_hook(
hooks.get('before_check'),
hooks.get('umask'),
config_filename,
'pre-check',
global_arguments.dry_run,
**hook_context,
)
logger.info('{}: Running consistency checks'.format(repository))
borg_check.check_archives(
repository,
location,
storage,
consistency,
local_path=local_path,
remote_path=remote_path,
progress=arguments['check'].progress,
repair=arguments['check'].repair,
only_checks=arguments['check'].only,
force=arguments['check'].force,
)
command.execute_hook(
hooks.get('after_check'),
hooks.get('umask'),
config_filename,
'post-check',
global_arguments.dry_run,
**hook_context,
)
if 'extract' in arguments:
command.execute_hook(
hooks.get('before_extract'),
hooks.get('umask'),
config_filename,
'pre-extract',
global_arguments.dry_run,
**hook_context,
)
if arguments['extract'].repository is None or validate.repositories_match(
repository, arguments['extract'].repository
):
logger.info(
'{}: Extracting archive {}'.format(repository, arguments['extract'].archive)
)
borg_extract.extract_archive(
global_arguments.dry_run,
repository,
borg_list.resolve_archive_name(
repository, arguments['extract'].archive, storage, local_path, remote_path
),
arguments['extract'].paths,
location,
storage,
local_borg_version,
local_path=local_path,
remote_path=remote_path,
destination_path=arguments['extract'].destination,
strip_components=arguments['extract'].strip_components,
progress=arguments['extract'].progress,
)
command.execute_hook(
hooks.get('after_extract'),
hooks.get('umask'),
config_filename,
'post-extract',
global_arguments.dry_run,
**hook_context,
)
if 'export-tar' in arguments:
if arguments['export-tar'].repository is None or validate.repositories_match(
repository, arguments['export-tar'].repository
):
logger.info(
'{}: Exporting archive {} as tar file'.format(
repository, arguments['export-tar'].archive
)
)
borg_export_tar.export_tar_archive(
global_arguments.dry_run,
repository,
borg_list.resolve_archive_name(
repository, arguments['export-tar'].archive, storage, local_path, remote_path
),
arguments['export-tar'].paths,
arguments['export-tar'].destination,
storage,
local_path=local_path,
remote_path=remote_path,
tar_filter=arguments['export-tar'].tar_filter,
files=arguments['export-tar'].files,
strip_components=arguments['export-tar'].strip_components,
)
if 'mount' in arguments:
if arguments['mount'].repository is None or validate.repositories_match(
repository, arguments['mount'].repository
):
if arguments['mount'].archive:
logger.info(
'{}: Mounting archive {}'.format(repository, arguments['mount'].archive)
)
else: # pragma: nocover
logger.info('{}: Mounting repository'.format(repository))
borg_mount.mount_archive(
repository,
borg_list.resolve_archive_name(
repository, arguments['mount'].archive, storage, local_path, remote_path
),
arguments['mount'].mount_point,
arguments['mount'].paths,
arguments['mount'].foreground,
arguments['mount'].options,
storage,
local_path=local_path,
remote_path=remote_path,
)
if 'restore' in arguments: # pragma: nocover
if arguments['restore'].repository is None or validate.repositories_match(
repository, arguments['restore'].repository
):
logger.info(
'{}: Restoring databases from archive {}'.format(
repository, arguments['restore'].archive
)
)
dispatch.call_hooks(
'remove_database_dumps',
hooks,
repository,
dump.DATABASE_HOOK_NAMES,
location,
global_arguments.dry_run,
)
restore_names = arguments['restore'].databases or []
if 'all' in restore_names:
restore_names = []
archive_name = borg_list.resolve_archive_name(
repository, arguments['restore'].archive, storage, local_path, remote_path
)
found_names = set()
for hook_name, per_hook_restore_databases in hooks.items():
if hook_name not in dump.DATABASE_HOOK_NAMES:
continue
for restore_database in per_hook_restore_databases:
database_name = restore_database['name']
if restore_names and database_name not in restore_names:
continue
found_names.add(database_name)
dump_pattern = dispatch.call_hooks(
'make_database_dump_pattern',
hooks,
repository,
dump.DATABASE_HOOK_NAMES,
location,
database_name,
)[hook_name]
# Kick off a single database extract to stdout.
extract_process = borg_extract.extract_archive(
dry_run=global_arguments.dry_run,
repository=repository,
archive=archive_name,
paths=dump.convert_glob_patterns_to_borg_patterns([dump_pattern]),
location_config=location,
storage_config=storage,
local_borg_version=local_borg_version,
local_path=local_path,
remote_path=remote_path,
destination_path='/',
# A directory format dump isn't a single file, and therefore can't extract
# to stdout. In this case, the extract_process return value is None.
extract_to_stdout=bool(restore_database.get('format') != 'directory'),
)
# Run a single database restore, consuming the extract stdout (if any).
dispatch.call_hooks(
'restore_database_dump',
{hook_name: [restore_database]},
repository,
dump.DATABASE_HOOK_NAMES,
location,
global_arguments.dry_run,
extract_process,
)
dispatch.call_hooks(
'remove_database_dumps',
hooks,
repository,
dump.DATABASE_HOOK_NAMES,
location,
global_arguments.dry_run,
)
if not restore_names and not found_names:
raise ValueError('No databases were found to restore')
missing_names = sorted(set(restore_names) - found_names)
if missing_names:
raise ValueError(
'Cannot restore database(s) {} missing from borgmatic\'s configuration'.format(
', '.join(missing_names)
)
)
if 'list' in arguments:
if arguments['list'].repository is None or validate.repositories_match(
repository, arguments['list'].repository
):
list_arguments = copy.copy(arguments['list'])
if not list_arguments.json: # pragma: nocover
logger.warning('{}: Listing archives'.format(repository))
list_arguments.archive = borg_list.resolve_archive_name(
repository, list_arguments.archive, storage, local_path, remote_path
)
json_output = borg_list.list_archives(
repository,
storage,
list_arguments=list_arguments,
local_path=local_path,
remote_path=remote_path,
)
if json_output: # pragma: nocover
yield json.loads(json_output)
if 'info' in arguments:
if arguments['info'].repository is None or validate.repositories_match(
repository, arguments['info'].repository
):
info_arguments = copy.copy(arguments['info'])
if not info_arguments.json: # pragma: nocover
logger.warning('{}: Displaying summary info for archives'.format(repository))
info_arguments.archive = borg_list.resolve_archive_name(
repository, info_arguments.archive, storage, local_path, remote_path
)
json_output = borg_info.display_archives_info(
repository,
storage,
info_arguments=info_arguments,
local_path=local_path,
remote_path=remote_path,
)
if json_output: # pragma: nocover
yield json.loads(json_output)
if 'borg' in arguments:
if arguments['borg'].repository is None or validate.repositories_match(
repository, arguments['borg'].repository
):
logger.warning('{}: Running arbitrary Borg command'.format(repository))
archive_name = borg_list.resolve_archive_name(
repository, arguments['borg'].archive, storage, local_path, remote_path
)
borg_borg.run_arbitrary_borg(
repository,
storage,
options=arguments['borg'].options,
archive=archive_name,
local_path=local_path,
remote_path=remote_path,
)
def load_configurations(config_filenames, overrides=None):
'''
Given a sequence of configuration filenames, load and validate each configuration file. Return
the results as a tuple of: dict of configuration filename to corresponding parsed configuration,
and sequence of logging.LogRecord instances containing any parse errors.
'''
# Dict mapping from config filename to corresponding parsed config dict.
configs = collections.OrderedDict()
logs = []
# Parse and load each configuration file.
for config_filename in config_filenames:
try:
configs[config_filename] = validate.parse_configuration(
config_filename, validate.schema_filename(), overrides
)
except PermissionError:
logs.extend(
[
logging.makeLogRecord(
dict(
levelno=logging.WARNING,
levelname='WARNING',
msg='{}: Insufficient permissions to read configuration file'.format(
config_filename
),
)
),
]
)
except (ValueError, OSError, validate.Validation_error) as error:
logs.extend(
[
logging.makeLogRecord(
dict(
levelno=logging.CRITICAL,
levelname='CRITICAL',
msg='{}: Error parsing configuration file'.format(config_filename),
)
),
logging.makeLogRecord(
dict(levelno=logging.CRITICAL, levelname='CRITICAL', msg=error)
),
]
)
return (configs, logs)
def log_record(suppress_log=False, **kwargs):
'''
Create a log record based on the given makeLogRecord() arguments, one of which must be
named "levelno". Log the record (unless suppress log is set) and return it.
'''
record = logging.makeLogRecord(kwargs)
if suppress_log:
return record
logger.handle(record)
return record
def log_error_records(
message, error=None, levelno=logging.CRITICAL, log_command_error_output=False
):
'''
Given error message text, an optional exception object, an optional log level, and whether to
log the error output of a CalledProcessError (if any), log error summary information and also
yield it as a series of logging.LogRecord instances.
Note that because the logs are yielded as a generator, logs won't get logged unless you consume
the generator output.
'''
level_name = logging._levelToName[levelno]
if not error:
yield log_record(levelno=levelno, levelname=level_name, msg=message)
return
try:
raise error
except CalledProcessError as error:
yield log_record(levelno=levelno, levelname=level_name, msg=message)
if error.output:
# Suppress these logs for now and save full error output for the log summary at the end.
yield log_record(
levelno=levelno,
levelname=level_name,
msg=error.output,
suppress_log=not log_command_error_output,
)
yield log_record(levelno=levelno, levelname=level_name, msg=error)
except (ValueError, OSError) as error:
yield log_record(levelno=levelno, levelname=level_name, msg=message)
yield log_record(levelno=levelno, levelname=level_name, msg=error)
except: # noqa: E722
# Raising above only as a means of determining the error type. Swallow the exception here
# because we don't want the exception to propagate out of this function.
pass
def get_local_path(configs):
'''
Arbitrarily return the local path from the first configuration dict. Default to "borg" if not
set.
'''
return next(iter(configs.values())).get('location', {}).get('local_path', 'borg')
def collect_configuration_run_summary_logs(configs, arguments):
'''
Given a dict of configuration filename to corresponding parsed configuration, and parsed
command-line arguments as a dict from subparser name to a parsed namespace of arguments, run
each configuration file and yield a series of logging.LogRecord instances containing summary
information about each run.
As a side effect of running through these configuration files, output their JSON results, if
any, to stdout.
'''
# Run cross-file validation checks.
if 'extract' in arguments:
repository = arguments['extract'].repository
elif 'list' in arguments and arguments['list'].archive:
repository = arguments['list'].repository
elif 'mount' in arguments:
repository = arguments['mount'].repository
else:
repository = None
if repository:
try:
validate.guard_configuration_contains_repository(repository, configs)
except ValueError as error:
yield from log_error_records(str(error))
return
if not configs:
yield from log_error_records(
'{}: No valid configuration files found'.format(
' '.join(arguments['global'].config_paths)
)
)
return
if 'create' in arguments:
try:
for config_filename, config in configs.items():
hooks = config.get('hooks', {})
command.execute_hook(
hooks.get('before_everything'),
hooks.get('umask'),
config_filename,
'pre-everything',
arguments['global'].dry_run,
)
except (CalledProcessError, ValueError, OSError) as error:
yield from log_error_records('Error running pre-everything hook', error)
return
# Execute the actions corresponding to each configuration file.
json_results = []
for config_filename, config in configs.items():
results = list(run_configuration(config_filename, config, arguments))
error_logs = tuple(result for result in results if isinstance(result, logging.LogRecord))
if error_logs:
yield from log_error_records(
'{}: Error running configuration file'.format(config_filename)
)
yield from error_logs
else:
yield logging.makeLogRecord(
dict(
levelno=logging.INFO,
levelname='INFO',
msg='{}: Successfully ran configuration file'.format(config_filename),
)
)
if results:
json_results.extend(results)
if 'umount' in arguments:
logger.info('Unmounting mount point {}'.format(arguments['umount'].mount_point))
try:
borg_umount.unmount_archive(
mount_point=arguments['umount'].mount_point, local_path=get_local_path(configs)
)
except (CalledProcessError, OSError) as error:
yield from log_error_records('Error unmounting mount point', error)
if json_results:
sys.stdout.write(json.dumps(json_results))
if 'create' in arguments:
try:
for config_filename, config in configs.items():
hooks = config.get('hooks', {})
command.execute_hook(
hooks.get('after_everything'),
hooks.get('umask'),
config_filename,
'post-everything',
arguments['global'].dry_run,
)
except (CalledProcessError, ValueError, OSError) as error:
yield from log_error_records('Error running post-everything hook', error)
def exit_with_help_link(): # pragma: no cover
'''
Display a link to get help and exit with an error code.
'''
logger.critical('')
logger.critical('Need some help? https://torsion.org/borgmatic/#issues')
sys.exit(1)
def main(): # pragma: no cover
configure_signals()
try:
arguments = parse_arguments(*sys.argv[1:])
except ValueError as error:
configure_logging(logging.CRITICAL)
logger.critical(error)
exit_with_help_link()
except SystemExit as error:
if error.code == 0:
raise error
configure_logging(logging.CRITICAL)
logger.critical('Error parsing arguments: {}'.format(' '.join(sys.argv)))
exit_with_help_link()
global_arguments = arguments['global']
if global_arguments.version:
print(pkg_resources.require('borgmatic')[0].version)
sys.exit(0)
if global_arguments.bash_completion:
print(borgmatic.commands.completion.bash_completion())
sys.exit(0)
config_filenames = tuple(collect.collect_config_filenames(global_arguments.config_paths))
configs, parse_logs = load_configurations(config_filenames, global_arguments.overrides)
any_json_flags = any(
getattr(sub_arguments, 'json', False) for sub_arguments in arguments.values()
)
colorama.init(
autoreset=True,
strip=not should_do_markup(global_arguments.no_color or any_json_flags, configs),
)
try:
configure_logging(
verbosity_to_log_level(global_arguments.verbosity),
verbosity_to_log_level(global_arguments.syslog_verbosity),
verbosity_to_log_level(global_arguments.log_file_verbosity),
verbosity_to_log_level(global_arguments.monitoring_verbosity),
global_arguments.log_file,
)
except (FileNotFoundError, PermissionError) as error:
configure_logging(logging.CRITICAL)
logger.critical('Error configuring logging: {}'.format(error))
exit_with_help_link()
logger.debug('Ensuring legacy configuration is upgraded')
convert.guard_configuration_upgraded(LEGACY_CONFIG_PATH, config_filenames)
summary_logs = parse_logs + list(collect_configuration_run_summary_logs(configs, arguments))
summary_logs_max_level = max(log.levelno for log in summary_logs)
for message in ('', 'summary:'):
log_record(
levelno=summary_logs_max_level,
levelname=logging.getLevelName(summary_logs_max_level),
msg=message,
)
for log in summary_logs:
logger.handle(log)
if summary_logs_max_level >= logging.CRITICAL:
exit_with_help_link()

View File

@ -0,0 +1,58 @@
from borgmatic.commands import arguments
UPGRADE_MESSAGE = '''
Your bash completions script is from a different version of borgmatic than is
currently installed. Please upgrade your script so your completions match the
command-line flags in your installed borgmatic! Try this to upgrade:
sudo sh -c "borgmatic --bash-completion > $BASH_SOURCE"
source $BASH_SOURCE
'''
def parser_flags(parser):
'''
Given an argparse.ArgumentParser instance, return its argument flags in a space-separated
string.
'''
return ' '.join(option for action in parser._actions for option in action.option_strings)
def bash_completion():
'''
Return a bash completion script for the borgmatic command. Produce this by introspecting
borgmatic's command-line argument parsers.
'''
top_level_parser, subparsers = arguments.make_parsers()
global_flags = parser_flags(top_level_parser)
actions = ' '.join(subparsers.choices.keys())
# Avert your eyes.
return '\n'.join(
(
'set -uo pipefail',
'check_version() {',
' local this_script="$(cat "$BASH_SOURCE" 2> /dev/null)"',
' local installed_script="$(borgmatic --bash-completion 2> /dev/null)"',
' if [ "$this_script" != "$installed_script" ] && [ "$installed_script" != "" ];'
' then cat << EOF\n%s\nEOF' % UPGRADE_MESSAGE,
' fi',
'}',
'complete_borgmatic() {',
)
+ tuple(
''' if [[ " ${COMP_WORDS[*]} " =~ " %s " ]]; then
COMPREPLY=($(compgen -W "%s %s %s" -- "${COMP_WORDS[COMP_CWORD]}"))
return 0
fi'''
% (action, parser_flags(subparser), actions, global_flags)
for action, subparser in subparsers.choices.items()
)
+ (
' COMPREPLY=($(compgen -W "%s %s" -- "${COMP_WORDS[COMP_CWORD]}"))'
% (actions, global_flags),
' (check_version &)',
'}',
'\ncomplete -o bashdefault -o default -F complete_borgmatic borgmatic',
)
)

View File

@ -0,0 +1,110 @@
import os
import sys
import textwrap
from argparse import ArgumentParser
from ruamel import yaml
from borgmatic.config import convert, generate, legacy, validate
DEFAULT_SOURCE_CONFIG_FILENAME = '/etc/borgmatic/config'
DEFAULT_SOURCE_EXCLUDES_FILENAME = '/etc/borgmatic/excludes'
DEFAULT_DESTINATION_CONFIG_FILENAME = '/etc/borgmatic/config.yaml'
def parse_arguments(*arguments):
'''
Given command-line arguments with which this script was invoked, parse the arguments and return
them as an ArgumentParser instance.
'''
parser = ArgumentParser(
description='''
Convert legacy INI-style borgmatic configuration and excludes files to a single YAML
configuration file. Note that this replaces any comments from the source files.
'''
)
parser.add_argument(
'-s',
'--source-config',
dest='source_config_filename',
default=DEFAULT_SOURCE_CONFIG_FILENAME,
help='Source INI-style configuration filename. Default: {}'.format(
DEFAULT_SOURCE_CONFIG_FILENAME
),
)
parser.add_argument(
'-e',
'--source-excludes',
dest='source_excludes_filename',
default=DEFAULT_SOURCE_EXCLUDES_FILENAME
if os.path.exists(DEFAULT_SOURCE_EXCLUDES_FILENAME)
else None,
help='Excludes filename',
)
parser.add_argument(
'-d',
'--destination-config',
dest='destination_config_filename',
default=DEFAULT_DESTINATION_CONFIG_FILENAME,
help='Destination YAML configuration filename. Default: {}'.format(
DEFAULT_DESTINATION_CONFIG_FILENAME
),
)
return parser.parse_args(arguments)
TEXT_WRAP_CHARACTERS = 80
def display_result(args): # pragma: no cover
result_lines = textwrap.wrap(
'Your borgmatic configuration has been upgraded. Please review the result in {}.'.format(
args.destination_config_filename
),
TEXT_WRAP_CHARACTERS,
)
delete_lines = textwrap.wrap(
'Once you are satisfied, you can safely delete {}{}.'.format(
args.source_config_filename,
' and {}'.format(args.source_excludes_filename)
if args.source_excludes_filename
else '',
),
TEXT_WRAP_CHARACTERS,
)
print('\n'.join(result_lines))
print()
print('\n'.join(delete_lines))
def main(): # pragma: no cover
try:
args = parse_arguments(*sys.argv[1:])
schema = yaml.round_trip_load(open(validate.schema_filename()).read())
source_config = legacy.parse_configuration(
args.source_config_filename, legacy.CONFIG_FORMAT
)
source_config_file_mode = os.stat(args.source_config_filename).st_mode
source_excludes = (
open(args.source_excludes_filename).read().splitlines()
if args.source_excludes_filename
else []
)
destination_config = convert.convert_legacy_parsed_config(
source_config, source_excludes, schema
)
generate.write_configuration(
args.destination_config_filename,
generate.render_configuration(destination_config),
mode=source_config_file_mode,
)
display_result(args)
except (ValueError, OSError) as error:
print(error, file=sys.stderr)
sys.exit(1)

View File

@ -0,0 +1,69 @@
import sys
from argparse import ArgumentParser
from borgmatic.config import generate, validate
DEFAULT_DESTINATION_CONFIG_FILENAME = '/etc/borgmatic/config.yaml'
def parse_arguments(*arguments):
'''
Given command-line arguments with which this script was invoked, parse the arguments and return
them as an ArgumentParser instance.
'''
parser = ArgumentParser(description='Generate a sample borgmatic YAML configuration file.')
parser.add_argument(
'-s',
'--source',
dest='source_filename',
help='Optional YAML configuration file to merge into the generated configuration, useful for upgrading your configuration',
)
parser.add_argument(
'-d',
'--destination',
dest='destination_filename',
default=DEFAULT_DESTINATION_CONFIG_FILENAME,
help='Destination YAML configuration file, default: {}'.format(
DEFAULT_DESTINATION_CONFIG_FILENAME
),
)
parser.add_argument(
'--overwrite',
default=False,
action='store_true',
help='Whether to overwrite any existing destination file, defaults to false',
)
return parser.parse_args(arguments)
def main(): # pragma: no cover
try:
args = parse_arguments(*sys.argv[1:])
generate.generate_sample_configuration(
args.source_filename,
args.destination_filename,
validate.schema_filename(),
overwrite=args.overwrite,
)
print('Generated a sample configuration file at {}.'.format(args.destination_filename))
print()
if args.source_filename:
print(
'Merged in the contents of configuration file at {}.'.format(args.source_filename)
)
print('To review the changes made, run:')
print()
print(
' diff --unified {} {}'.format(args.source_filename, args.destination_filename)
)
print()
print('Please edit the file to suit your needs. The values are representative.')
print('All fields are optional except where indicated.')
print()
print('If you ever need help: https://torsion.org/borgmatic/#issues')
except (ValueError, OSError) as error:
print(error, file=sys.stderr)
sys.exit(1)

View File

@ -0,0 +1,56 @@
import logging
import sys
from argparse import ArgumentParser
from borgmatic.config import collect, validate
logger = logging.getLogger(__name__)
def parse_arguments(*arguments):
'''
Given command-line arguments with which this script was invoked, parse the arguments and return
them as an ArgumentParser instance.
'''
config_paths = collect.get_default_config_paths()
parser = ArgumentParser(description='Validate borgmatic configuration file(s).')
parser.add_argument(
'-c',
'--config',
nargs='+',
dest='config_paths',
default=config_paths,
help='Configuration filenames or directories, defaults to: {}'.format(
' '.join(config_paths)
),
)
return parser.parse_args(arguments)
def main(): # pragma: no cover
args = parse_arguments(*sys.argv[1:])
logging.basicConfig(level=logging.INFO, format='%(message)s')
config_filenames = tuple(collect.collect_config_filenames(args.config_paths))
if len(config_filenames) == 0:
logger.critical('No files to validate found')
sys.exit(1)
found_issues = False
for config_filename in config_filenames:
try:
validate.parse_configuration(config_filename, validate.schema_filename())
except (ValueError, OSError, validate.Validation_error) as error:
logging.critical('{}: Error parsing configuration file'.format(config_filename))
logging.critical(error)
found_issues = True
if found_issues:
sys.exit(1)
else:
logger.info(
'All given configuration files are valid: {}'.format(', '.join(config_filenames))
)

View File

View File

@ -0,0 +1,9 @@
def repository_enabled_for_checks(repository, consistency):
'''
Given a repository name and a consistency configuration dict, return whether the repository
is enabled to have consistency checks run.
'''
if not consistency.get('check_repositories'):
return True
return repository in consistency['check_repositories']

View File

@ -0,0 +1,54 @@
import os
def get_default_config_paths(expand_home=True):
'''
Based on the value of the XDG_CONFIG_HOME and HOME environment variables, return a list of
default configuration paths. This includes both system-wide configuration and configuration in
the current user's home directory.
Don't expand the home directory ($HOME) if the expand home flag is False.
'''
user_config_directory = os.getenv('XDG_CONFIG_HOME') or os.path.join('$HOME', '.config')
if expand_home:
user_config_directory = os.path.expandvars(user_config_directory)
return [
'/etc/borgmatic/config.yaml',
'/etc/borgmatic.d',
'%s/borgmatic/config.yaml' % user_config_directory,
'%s/borgmatic.d' % user_config_directory,
]
def collect_config_filenames(config_paths):
'''
Given a sequence of config paths, both filenames and directories, resolve that to an iterable
of files. Accomplish this by listing any given directories looking for contained config files
(ending with the ".yaml" or ".yml" extension). This is non-recursive, so any directories within the given
directories are ignored.
Return paths even if they don't exist on disk, so the user can find out about missing
configuration paths. However, skip a default config path if it's missing, so the user doesn't
have to create a default config path unless they need it.
'''
real_default_config_paths = set(map(os.path.realpath, get_default_config_paths()))
for path in config_paths:
exists = os.path.exists(path)
if os.path.realpath(path) in real_default_config_paths and not exists:
continue
if not os.path.isdir(path) or not exists:
yield path
continue
if not os.access(path, os.R_OK):
continue
for filename in sorted(os.listdir(path)):
full_filename = os.path.join(path, filename)
matching_filetype = full_filename.endswith('.yaml') or full_filename.endswith('.yml')
if matching_filetype and not os.path.isdir(full_filename):
yield full_filename

View File

@ -0,0 +1,95 @@
import os
from ruamel import yaml
from borgmatic.config import generate
def _convert_section(source_section_config, section_schema):
'''
Given a legacy Parsed_config instance for a single section, convert it to its corresponding
yaml.comments.CommentedMap representation in preparation for actual serialization to YAML.
Where integer types exist in the given section schema, convert their values to integers.
'''
destination_section_config = yaml.comments.CommentedMap(
[
(
option_name,
int(option_value)
if section_schema['properties'].get(option_name, {}).get('type') == 'integer'
else option_value,
)
for option_name, option_value in source_section_config.items()
]
)
return destination_section_config
def convert_legacy_parsed_config(source_config, source_excludes, schema):
'''
Given a legacy Parsed_config instance loaded from an INI-style config file and a list of exclude
patterns, convert them to a corresponding yaml.comments.CommentedMap representation in
preparation for serialization to a single YAML config file.
Additionally, use the given schema as a source of helpful comments to include within the
returned CommentedMap.
'''
destination_config = yaml.comments.CommentedMap(
[
(section_name, _convert_section(section_config, schema['properties'][section_name]))
for section_name, section_config in source_config._asdict().items()
]
)
# Split space-seperated values into actual lists, make "repository" into a list, and merge in
# excludes.
location = destination_config['location']
location['source_directories'] = source_config.location['source_directories'].split(' ')
location['repositories'] = [location.pop('repository')]
location['exclude_patterns'] = source_excludes
if source_config.consistency.get('checks'):
destination_config['consistency']['checks'] = source_config.consistency['checks'].split(' ')
# Add comments to each section, and then add comments to the fields in each section.
generate.add_comments_to_configuration_object(destination_config, schema)
for section_name, section_config in destination_config.items():
generate.add_comments_to_configuration_object(
section_config, schema['properties'][section_name], indent=generate.INDENT
)
return destination_config
class Legacy_configuration_not_upgraded(FileNotFoundError):
def __init__(self):
super(Legacy_configuration_not_upgraded, self).__init__(
'''borgmatic changed its configuration file format in version 1.1.0 from INI-style
to YAML. This better supports validation, and has a more natural way to express
lists of values. To upgrade your existing configuration, run:
sudo upgrade-borgmatic-config
That will generate a new YAML configuration file at /etc/borgmatic/config.yaml
(by default) using the values from both your existing configuration and excludes
files. The new version of borgmatic will consume the YAML configuration file
instead of the old one.'''
)
def guard_configuration_upgraded(source_config_filename, destination_config_filenames):
'''
If legacy source configuration exists but no destination upgraded configs do, raise
Legacy_configuration_not_upgraded.
The idea is that we want to alert the user about upgrading their config if they haven't already.
'''
destination_config_exists = any(
os.path.exists(filename) for filename in destination_config_filenames
)
if os.path.exists(source_config_filename) and not destination_config_exists:
raise Legacy_configuration_not_upgraded()

View File

@ -0,0 +1,296 @@
import collections
import io
import os
import re
from ruamel import yaml
from borgmatic.config import load, normalize
INDENT = 4
SEQUENCE_INDENT = 2
def _insert_newline_before_comment(config, field_name):
'''
Using some ruamel.yaml black magic, insert a blank line in the config right before the given
field and its comments.
'''
config.ca.items[field_name][1].insert(
0, yaml.tokens.CommentToken('\n', yaml.error.CommentMark(0), None)
)
def _schema_to_sample_configuration(schema, level=0, parent_is_sequence=False):
'''
Given a loaded configuration schema, generate and return sample config for it. Include comments
for each section based on the schema "description".
'''
schema_type = schema.get('type')
example = schema.get('example')
if example is not None:
return example
if schema_type == 'array':
config = yaml.comments.CommentedSeq(
[_schema_to_sample_configuration(schema['items'], level, parent_is_sequence=True)]
)
add_comments_to_configuration_sequence(config, schema, indent=(level * INDENT))
elif schema_type == 'object':
config = yaml.comments.CommentedMap(
[
(field_name, _schema_to_sample_configuration(sub_schema, level + 1))
for field_name, sub_schema in schema['properties'].items()
]
)
indent = (level * INDENT) + (SEQUENCE_INDENT if parent_is_sequence else 0)
add_comments_to_configuration_object(
config, schema, indent=indent, skip_first=parent_is_sequence
)
else:
raise ValueError('Schema at level {} is unsupported: {}'.format(level, schema))
return config
def _comment_out_line(line):
# If it's already is commented out (or empty), there's nothing further to do!
stripped_line = line.lstrip()
if not stripped_line or stripped_line.startswith('#'):
return line
# Comment out the names of optional sections, inserting the '#' after any indent for aesthetics.
matches = re.match(r'(\s*)', line)
indent_spaces = matches.group(0) if matches else ''
count_indent_spaces = len(indent_spaces)
return '# '.join((indent_spaces, line[count_indent_spaces:]))
def _comment_out_optional_configuration(rendered_config):
'''
Post-process a rendered configuration string to comment out optional key/values, as determined
by a sentinel in the comment before each key.
The idea is that the pre-commented configuration prevents the user from having to comment out a
bunch of configuration they don't care about to get to a minimal viable configuration file.
Ideally ruamel.yaml would support commenting out keys during configuration generation, but it's
not terribly easy to accomplish that way.
'''
lines = []
optional = False
for line in rendered_config.split('\n'):
# Upon encountering an optional configuration option, comment out lines until the next blank
# line.
if line.strip().startswith('# {}'.format(COMMENTED_OUT_SENTINEL)):
optional = True
continue
# Hit a blank line, so reset commenting.
if not line.strip():
optional = False
lines.append(_comment_out_line(line) if optional else line)
return '\n'.join(lines)
def render_configuration(config):
'''
Given a config data structure of nested OrderedDicts, render the config as YAML and return it.
'''
dumper = yaml.YAML()
dumper.indent(mapping=INDENT, sequence=INDENT + SEQUENCE_INDENT, offset=INDENT)
rendered = io.StringIO()
dumper.dump(config, rendered)
return rendered.getvalue()
def write_configuration(config_filename, rendered_config, mode=0o600, overwrite=False):
'''
Given a target config filename and rendered config YAML, write it out to file. Create any
containing directories as needed. But if the file already exists and overwrite is False,
abort before writing anything.
'''
if not overwrite and os.path.exists(config_filename):
raise FileExistsError(
'{} already exists. Aborting. Use --overwrite to replace the file.'.format(
config_filename
)
)
try:
os.makedirs(os.path.dirname(config_filename), mode=0o700)
except (FileExistsError, FileNotFoundError):
pass
with open(config_filename, 'w') as config_file:
config_file.write(rendered_config)
os.chmod(config_filename, mode)
def add_comments_to_configuration_sequence(config, schema, indent=0):
'''
If the given config sequence's items are object, then mine the schema for the description of the
object's first item, and slap that atop the sequence. Indent the comment the given number of
characters.
Doing this for sequences of maps results in nice comments that look like:
```
things:
# First key description. Added by this function.
- key: foo
# Second key description. Added by add_comments_to_configuration_object().
other: bar
```
'''
if schema['items'].get('type') != 'object':
return
for field_name in config[0].keys():
field_schema = schema['items']['properties'].get(field_name, {})
description = field_schema.get('description')
# No description to use? Skip it.
if not field_schema or not description:
return
config[0].yaml_set_start_comment(description, indent=indent)
# We only want the first key's description here, as the rest of the keys get commented by
# add_comments_to_configuration_object().
return
REQUIRED_SECTION_NAMES = {'location', 'retention'}
REQUIRED_KEYS = {'source_directories', 'repositories', 'keep_daily'}
COMMENTED_OUT_SENTINEL = 'COMMENT_OUT'
def add_comments_to_configuration_object(config, schema, indent=0, skip_first=False):
'''
Using descriptions from a schema as a source, add those descriptions as comments to the given
config mapping, before each field. Indent the comment the given number of characters.
'''
for index, field_name in enumerate(config.keys()):
if skip_first and index == 0:
continue
field_schema = schema['properties'].get(field_name, {})
description = field_schema.get('description', '').strip()
# If this is an optional key, add an indicator to the comment flagging it to be commented
# out from the sample configuration. This sentinel is consumed by downstream processing that
# does the actual commenting out.
if field_name not in REQUIRED_SECTION_NAMES and field_name not in REQUIRED_KEYS:
description = (
'\n'.join((description, COMMENTED_OUT_SENTINEL))
if description
else COMMENTED_OUT_SENTINEL
)
# No description to use? Skip it.
if not field_schema or not description: # pragma: no cover
continue
config.yaml_set_comment_before_after_key(key=field_name, before=description, indent=indent)
if index > 0:
_insert_newline_before_comment(config, field_name)
RUAMEL_YAML_COMMENTS_INDEX = 1
def remove_commented_out_sentinel(config, field_name):
'''
Given a configuration CommentedMap and a top-level field name in it, remove any "commented out"
sentinel found at the end of its YAML comments. This prevents the given field name from getting
commented out by downstream processing that consumes the sentinel.
'''
try:
last_comment_value = config.ca.items[field_name][RUAMEL_YAML_COMMENTS_INDEX][-1].value
except KeyError:
return
if last_comment_value == '# {}\n'.format(COMMENTED_OUT_SENTINEL):
config.ca.items[field_name][RUAMEL_YAML_COMMENTS_INDEX].pop()
def merge_source_configuration_into_destination(destination_config, source_config):
'''
Deep merge the given source configuration dict into the destination configuration CommentedMap,
favoring values from the source when there are collisions.
The purpose of this is to upgrade configuration files from old versions of borgmatic by adding
new
configuration keys and comments.
'''
if not source_config:
return destination_config
if not destination_config or not isinstance(source_config, collections.abc.Mapping):
return source_config
for field_name, source_value in source_config.items():
# Since this key/value is from the source configuration, leave it uncommented and remove any
# sentinel that would cause it to get commented out.
remove_commented_out_sentinel(destination_config, field_name)
# This is a mapping. Recurse for this key/value.
if isinstance(source_value, collections.abc.Mapping):
destination_config[field_name] = merge_source_configuration_into_destination(
destination_config[field_name], source_value
)
continue
# This is a sequence. Recurse for each item in it.
if isinstance(source_value, collections.abc.Sequence) and not isinstance(source_value, str):
destination_value = destination_config[field_name]
destination_config[field_name] = yaml.comments.CommentedSeq(
[
merge_source_configuration_into_destination(
destination_value[index] if index < len(destination_value) else None,
source_item,
)
for index, source_item in enumerate(source_value)
]
)
continue
# This is some sort of scalar. Simply set it into the destination.
destination_config[field_name] = source_config[field_name]
return destination_config
def generate_sample_configuration(
source_filename, destination_filename, schema_filename, overwrite=False
):
'''
Given an optional source configuration filename, and a required destination configuration
filename, the path to a schema filename in a YAML rendition of the JSON Schema format, and
whether to overwrite a destination file, write out a sample configuration file based on that
schema. If a source filename is provided, merge the parsed contents of that configuration into
the generated configuration.
'''
schema = yaml.round_trip_load(open(schema_filename))
source_config = None
if source_filename:
source_config = load.load_configuration(source_filename)
normalize.normalize(source_config)
destination_config = merge_source_configuration_into_destination(
_schema_to_sample_configuration(schema), source_config
)
write_configuration(
destination_filename,
_comment_out_optional_configuration(render_configuration(destination_config)),
overwrite=overwrite,
)

152
borgmatic/config/legacy.py Normal file
View File

@ -0,0 +1,152 @@
from collections import OrderedDict, namedtuple
from configparser import RawConfigParser
Section_format = namedtuple('Section_format', ('name', 'options'))
Config_option = namedtuple('Config_option', ('name', 'value_type', 'required'))
def option(name, value_type=str, required=True):
'''
Given a config file option name, an expected type for its value, and whether it's required,
return a Config_option capturing that information.
'''
return Config_option(name, value_type, required)
CONFIG_FORMAT = (
Section_format(
'location',
(
option('source_directories'),
option('one_file_system', value_type=bool, required=False),
option('remote_path', required=False),
option('repository'),
),
),
Section_format(
'storage',
(
option('encryption_passphrase', required=False),
option('compression', required=False),
option('umask', required=False),
),
),
Section_format(
'retention',
(
option('keep_within', required=False),
option('keep_hourly', int, required=False),
option('keep_daily', int, required=False),
option('keep_weekly', int, required=False),
option('keep_monthly', int, required=False),
option('keep_yearly', int, required=False),
option('prefix', required=False),
),
),
Section_format(
'consistency', (option('checks', required=False), option('check_last', required=False))
),
)
def validate_configuration_format(parser, config_format):
'''
Given an open RawConfigParser and an expected config file format, validate that the parsed
configuration file has the expected sections, that any required options are present in those
sections, and that there aren't any unexpected options.
A section is required if any of its contained options are required.
Raise ValueError if anything is awry.
'''
section_names = set(parser.sections())
required_section_names = tuple(
section.name
for section in config_format
if any(option.required for option in section.options)
)
unknown_section_names = section_names - set(
section_format.name for section_format in config_format
)
if unknown_section_names:
raise ValueError(
'Unknown config sections found: {}'.format(', '.join(unknown_section_names))
)
missing_section_names = set(required_section_names) - section_names
if missing_section_names:
raise ValueError('Missing config sections: {}'.format(', '.join(missing_section_names)))
for section_format in config_format:
if section_format.name not in section_names:
continue
option_names = parser.options(section_format.name)
expected_options = section_format.options
unexpected_option_names = set(option_names) - set(
option.name for option in expected_options
)
if unexpected_option_names:
raise ValueError(
'Unexpected options found in config section {}: {}'.format(
section_format.name, ', '.join(sorted(unexpected_option_names))
)
)
missing_option_names = tuple(
option.name
for option in expected_options
if option.required
if option.name not in option_names
)
if missing_option_names:
raise ValueError(
'Required options missing from config section {}: {}'.format(
section_format.name, ', '.join(missing_option_names)
)
)
def parse_section_options(parser, section_format):
'''
Given an open RawConfigParser and an expected section format, return the option values from that
section as a dict mapping from option name to value. Omit those options that are not present in
the parsed options.
Raise ValueError if any option values cannot be coerced to the expected Python data type.
'''
type_getter = {str: parser.get, int: parser.getint, bool: parser.getboolean}
return OrderedDict(
(option.name, type_getter[option.value_type](section_format.name, option.name))
for option in section_format.options
if parser.has_option(section_format.name, option.name)
)
def parse_configuration(config_filename, config_format):
'''
Given a config filename and an expected config file format, return the parsed configuration
as a namedtuple with one attribute for each parsed section.
Raise IOError if the file cannot be read, or ValueError if the format is not as expected.
'''
parser = RawConfigParser()
if not parser.read(config_filename):
raise ValueError('Configuration file cannot be opened: {}'.format(config_filename))
validate_configuration_format(parser, config_format)
# Describes a parsed configuration, where each attribute is the name of a configuration file
# section and each value is a dict of that section's parsed options.
Parsed_config = namedtuple(
'Parsed_config', (section_format.name for section_format in config_format)
)
return Parsed_config(
*(parse_section_options(parser, section_format) for section_format in config_format)
)

215
borgmatic/config/load.py Normal file
View File

@ -0,0 +1,215 @@
import logging
import os
import ruamel.yaml
logger = logging.getLogger(__name__)
class Yaml_with_loader_stream(ruamel.yaml.YAML):
'''
A derived class of ruamel.yaml.YAML that simply tacks the loaded stream (file object) onto the
loader class so that it's available anywhere that's passed a loader (in this case,
include_configuration() below).
'''
def get_constructor_parser(self, stream):
constructor, parser = super(Yaml_with_loader_stream, self).get_constructor_parser(stream)
constructor.loader.stream = stream
return constructor, parser
def load_configuration(filename):
'''
Load the given configuration file and return its contents as a data structure of nested dicts
and lists.
Raise ruamel.yaml.error.YAMLError if something goes wrong parsing the YAML, or RecursionError
if there are too many recursive includes.
'''
yaml = Yaml_with_loader_stream(typ='safe')
yaml.Constructor = Include_constructor
return yaml.load(open(filename))
def include_configuration(loader, filename_node):
'''
Load the given YAML filename (ignoring the given loader so we can use our own) and return its
contents as a data structure of nested dicts and lists. If the filename is relative, probe for
it within 1. the current working directory and 2. the directory containing the YAML file doing
the including.
Raise FileNotFoundError if an included file was not found.
'''
include_directories = [os.getcwd(), os.path.abspath(os.path.dirname(loader.stream.name))]
include_filename = os.path.expanduser(filename_node.value)
if not os.path.isabs(include_filename):
candidate_filenames = [
os.path.join(directory, include_filename) for directory in include_directories
]
for candidate_filename in candidate_filenames:
if os.path.exists(candidate_filename):
include_filename = candidate_filename
break
else:
raise FileNotFoundError(
f'Could not find include {filename_node.value} at {" or ".join(candidate_filenames)}'
)
return load_configuration(include_filename)
DELETED_NODE = object()
def deep_merge_nodes(nodes):
'''
Given a nested borgmatic configuration data structure as a list of tuples in the form of:
(
ruamel.yaml.nodes.ScalarNode as a key,
ruamel.yaml.nodes.MappingNode or other Node as a value,
),
... deep merge any node values corresponding to duplicate keys and return the result. If
there are colliding keys with non-MappingNode values (e.g., integers or strings), the last
of the values wins.
For instance, given node values of:
[
(
ScalarNode(tag='tag:yaml.org,2002:str', value='retention'),
MappingNode(tag='tag:yaml.org,2002:map', value=[
(
ScalarNode(tag='tag:yaml.org,2002:str', value='keep_hourly'),
ScalarNode(tag='tag:yaml.org,2002:int', value='24')
),
(
ScalarNode(tag='tag:yaml.org,2002:str', value='keep_daily'),
ScalarNode(tag='tag:yaml.org,2002:int', value='7')
),
]),
),
(
ScalarNode(tag='tag:yaml.org,2002:str', value='retention'),
MappingNode(tag='tag:yaml.org,2002:map', value=[
(
ScalarNode(tag='tag:yaml.org,2002:str', value='keep_daily'),
ScalarNode(tag='tag:yaml.org,2002:int', value='5')
),
]),
),
]
... the returned result would be:
[
(
ScalarNode(tag='tag:yaml.org,2002:str', value='retention'),
MappingNode(tag='tag:yaml.org,2002:map', value=[
(
ScalarNode(tag='tag:yaml.org,2002:str', value='keep_hourly'),
ScalarNode(tag='tag:yaml.org,2002:int', value='24')
),
(
ScalarNode(tag='tag:yaml.org,2002:str', value='keep_daily'),
ScalarNode(tag='tag:yaml.org,2002:int', value='5')
),
]),
),
]
The purpose of deep merging like this is to support, for instance, merging one borgmatic
configuration file into another for reuse, such that a configuration section ("retention",
etc.) does not completely replace the corresponding section in a merged file.
'''
# Map from original node key/value to the replacement merged node. DELETED_NODE as a replacement
# node indications deletion.
replaced_nodes = {}
# To find nodes that require merging, compare each node with each other node.
for a_key, a_value in nodes:
for b_key, b_value in nodes:
# If we've already considered one of the nodes for merging, skip it.
if (a_key, a_value) in replaced_nodes or (b_key, b_value) in replaced_nodes:
continue
# If the keys match and the values are different, we need to merge these two A and B nodes.
if a_key.tag == b_key.tag and a_key.value == b_key.value and a_value != b_value:
# Since we're merging into the B node, consider the A node a duplicate and remove it.
replaced_nodes[(a_key, a_value)] = DELETED_NODE
# If we're dealing with MappingNodes, recurse and merge its values as well.
if isinstance(b_value, ruamel.yaml.nodes.MappingNode):
replaced_nodes[(b_key, b_value)] = (
b_key,
ruamel.yaml.nodes.MappingNode(
tag=b_value.tag,
value=deep_merge_nodes(a_value.value + b_value.value),
start_mark=b_value.start_mark,
end_mark=b_value.end_mark,
flow_style=b_value.flow_style,
comment=b_value.comment,
anchor=b_value.anchor,
),
)
# If we're dealing with SequenceNodes, merge by appending one sequence to the other.
elif isinstance(b_value, ruamel.yaml.nodes.SequenceNode):
replaced_nodes[(b_key, b_value)] = (
b_key,
ruamel.yaml.nodes.SequenceNode(
tag=b_value.tag,
value=a_value.value + b_value.value,
start_mark=b_value.start_mark,
end_mark=b_value.end_mark,
flow_style=b_value.flow_style,
comment=b_value.comment,
anchor=b_value.anchor,
),
)
return [
replaced_nodes.get(node, node) for node in nodes if replaced_nodes.get(node) != DELETED_NODE
]
class Include_constructor(ruamel.yaml.SafeConstructor):
'''
A YAML "constructor" (a ruamel.yaml concept) that supports a custom "!include" tag for including
separate YAML configuration files. Example syntax: `retention: !include common.yaml`
'''
def __init__(self, preserve_quotes=None, loader=None):
super(Include_constructor, self).__init__(preserve_quotes, loader)
self.add_constructor('!include', include_configuration)
def flatten_mapping(self, node):
'''
Support the special case of deep merging included configuration into an existing mapping
using the YAML '<<' merge key. Example syntax:
```
retention:
keep_daily: 1
<<: !include common.yaml
```
These includes are deep merged into the current configuration file. For instance, in this
example, any "retention" options in common.yaml will get merged into the "retention" section
in the example configuration file.
'''
representer = ruamel.yaml.representer.SafeRepresenter()
for index, (key_node, value_node) in enumerate(node.value):
if key_node.tag == u'tag:yaml.org,2002:merge' and value_node.tag == '!include':
included_value = representer.represent_data(self.construct_object(value_node))
node.value[index] = (key_node, included_value)
super(Include_constructor, self).flatten_mapping(node)
node.value = deep_merge_nodes(node.value)

View File

@ -0,0 +1,31 @@
def normalize(config):
'''
Given a configuration dict, apply particular hard-coded rules to normalize its contents to
adhere to the configuration schema.
'''
# Upgrade exclude_if_present from a string to a list.
exclude_if_present = config.get('location', {}).get('exclude_if_present')
if isinstance(exclude_if_present, str):
config['location']['exclude_if_present'] = [exclude_if_present]
# Upgrade various monitoring hooks from a string to a dict.
healthchecks = config.get('hooks', {}).get('healthchecks')
if isinstance(healthchecks, str):
config['hooks']['healthchecks'] = {'ping_url': healthchecks}
cronitor = config.get('hooks', {}).get('cronitor')
if isinstance(cronitor, str):
config['hooks']['cronitor'] = {'ping_url': cronitor}
pagerduty = config.get('hooks', {}).get('pagerduty')
if isinstance(pagerduty, str):
config['hooks']['pagerduty'] = {'integration_key': pagerduty}
cronhub = config.get('hooks', {}).get('cronhub')
if isinstance(cronhub, str):
config['hooks']['cronhub'] = {'ping_url': cronhub}
# Upgrade consistency checks from a list of strings to a list of dicts.
checks = config.get('consistency', {}).get('checks')
if isinstance(checks, list) and len(checks) and isinstance(checks[0], str):
config['consistency']['checks'] = [{'name': check_type} for check_type in checks]

View File

@ -0,0 +1,79 @@
import io
import ruamel.yaml
def set_values(config, keys, value):
'''
Given a hierarchy of configuration dicts, a sequence of parsed key strings, and a string value,
descend into the hierarchy based on the keys to set the value into the right place.
'''
if not keys:
return
first_key = keys[0]
if len(keys) == 1:
config[first_key] = value
return
if first_key not in config:
config[first_key] = {}
set_values(config[first_key], keys[1:], value)
def convert_value_type(value):
'''
Given a string value, determine its logical type (string, boolean, integer, etc.), and return it
converted to that type.
Raise ruamel.yaml.error.YAMLError if there's a parse issue with the YAML.
'''
return ruamel.yaml.YAML(typ='safe').load(io.StringIO(value))
def parse_overrides(raw_overrides):
'''
Given a sequence of configuration file override strings in the form of "section.option=value",
parse and return a sequence of tuples (keys, values), where keys is a sequence of strings. For
instance, given the following raw overrides:
['section.my_option=value1', 'section.other_option=value2']
... return this:
(
(('section', 'my_option'), 'value1'),
(('section', 'other_option'), 'value2'),
)
Raise ValueError if an override can't be parsed.
'''
if not raw_overrides:
return ()
parsed_overrides = []
for raw_override in raw_overrides:
try:
raw_keys, value = raw_override.split('=', 1)
parsed_overrides.append((tuple(raw_keys.split('.')), convert_value_type(value),))
except ValueError:
raise ValueError(
f"Invalid override '{raw_override}'. Make sure you use the form: SECTION.OPTION=VALUE"
)
except ruamel.yaml.error.YAMLError as error:
raise ValueError(f"Invalid override '{raw_override}': {error.problem}")
return tuple(parsed_overrides)
def apply_overrides(config, raw_overrides):
'''
Given a sequence of configuration file override strings in the form of "section.option=value"
and a configuration dict, parse each override and set it the configuration dict.
'''
overrides = parse_overrides(raw_overrides)
for (keys, value) in overrides:
set_values(config, keys, value)

1002
borgmatic/config/schema.yaml Normal file

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,176 @@
import os
import jsonschema
import pkg_resources
import ruamel.yaml
from borgmatic.config import load, normalize, override
def schema_filename():
'''
Path to the installed YAML configuration schema file, used to validate and parse the
configuration.
'''
return pkg_resources.resource_filename('borgmatic', 'config/schema.yaml')
def format_json_error_path_element(path_element):
'''
Given a path element into a JSON data structure, format it for display as a string.
'''
if isinstance(path_element, int):
return str('[{}]'.format(path_element))
return str('.{}'.format(path_element))
def format_json_error(error):
'''
Given an instance of jsonschema.exceptions.ValidationError, format it for display as a string.
'''
if not error.path:
return 'At the top level: {}'.format(error.message)
formatted_path = ''.join(format_json_error_path_element(element) for element in error.path)
return "At '{}': {}".format(formatted_path.lstrip('.'), error.message)
class Validation_error(ValueError):
'''
A collection of error messages generated when attempting to validate a particular
configuration file.
'''
def __init__(self, config_filename, errors):
'''
Given a configuration filename path and a sequence of string error messages, create a
Validation_error.
'''
self.config_filename = config_filename
self.errors = errors
def __str__(self):
'''
Render a validation error as a user-facing string.
'''
return 'An error occurred while parsing a configuration file at {}:\n'.format(
self.config_filename
) + '\n'.join(error for error in self.errors)
def apply_logical_validation(config_filename, parsed_configuration):
'''
Given a parsed and schematically valid configuration as a data structure of nested dicts (see
below), run through any additional logical validation checks. If there are any such validation
problems, raise a Validation_error.
'''
location_repositories = parsed_configuration.get('location', {}).get('repositories')
check_repositories = parsed_configuration.get('consistency', {}).get('check_repositories', [])
for repository in check_repositories:
if repository not in location_repositories:
raise Validation_error(
config_filename,
(
'Unknown repository in the consistency section\'s check_repositories: {}'.format(
repository
),
),
)
def parse_configuration(config_filename, schema_filename, overrides=None):
'''
Given the path to a config filename in YAML format, the path to a schema filename in a YAML
rendition of JSON Schema format, a sequence of configuration file override strings in the form
of "section.option=value", return the parsed configuration as a data structure of nested dicts
and lists corresponding to the schema. Example return value:
{'location': {'source_directories': ['/home', '/etc'], 'repository': 'hostname.borg'},
'retention': {'keep_daily': 7}, 'consistency': {'checks': ['repository', 'archives']}}
Raise FileNotFoundError if the file does not exist, PermissionError if the user does not
have permissions to read the file, or Validation_error if the config does not match the schema.
'''
try:
config = load.load_configuration(config_filename)
schema = load.load_configuration(schema_filename)
except (ruamel.yaml.error.YAMLError, RecursionError) as error:
raise Validation_error(config_filename, (str(error),))
override.apply_overrides(config, overrides)
normalize.normalize(config)
try:
validator = jsonschema.Draft7Validator(schema)
except AttributeError: # pragma: no cover
validator = jsonschema.Draft4Validator(schema)
validation_errors = tuple(validator.iter_errors(config))
if validation_errors:
raise Validation_error(
config_filename, tuple(format_json_error(error) for error in validation_errors)
)
apply_logical_validation(config_filename, config)
return config
def normalize_repository_path(repository):
'''
Given a repository path, return the absolute path of it (for local repositories).
'''
# A colon in the repository indicates it's a remote repository. Bail.
if ':' in repository:
return repository
return os.path.abspath(repository)
def repositories_match(first, second):
'''
Given two repository paths (relative and/or absolute), return whether they match.
'''
return normalize_repository_path(first) == normalize_repository_path(second)
def guard_configuration_contains_repository(repository, configurations):
'''
Given a repository path and a dict mapping from config filename to corresponding parsed config
dict, ensure that the repository is declared exactly once in all of the configurations.
If no repository is given, then error if there are multiple configured repositories.
Raise ValueError if the repository is not found in a configuration, or is declared multiple
times.
'''
if not repository:
count = len(
tuple(
config_repository
for config in configurations.values()
for config_repository in config['location']['repositories']
)
)
if count > 1:
raise ValueError(
'Can\'t determine which repository to use. Use --repository option to disambiguate'
)
return
count = len(
tuple(
config_repository
for config in configurations.values()
for config_repository in config['location']['repositories']
if repositories_match(repository, config_repository)
)
)
if count == 0:
raise ValueError('Repository {} not found in configuration files'.format(repository))
if count > 1:
raise ValueError('Repository {} found in multiple configuration files'.format(repository))

281
borgmatic/execute.py Normal file
View File

@ -0,0 +1,281 @@
import collections
import logging
import os
import select
import subprocess
logger = logging.getLogger(__name__)
ERROR_OUTPUT_MAX_LINE_COUNT = 25
BORG_ERROR_EXIT_CODE = 2
def exit_code_indicates_error(process, exit_code, borg_local_path=None):
'''
Return True if the given exit code from running a command corresponds to an error. If a Borg
local path is given and matches the process' command, then treat exit code 1 as a warning
instead of an error.
'''
if exit_code is None:
return False
command = process.args.split(' ') if isinstance(process.args, str) else process.args
if borg_local_path and command[0] == borg_local_path:
return bool(exit_code < 0 or exit_code >= BORG_ERROR_EXIT_CODE)
return bool(exit_code != 0)
def command_for_process(process):
'''
Given a process as an instance of subprocess.Popen, return the command string that was used to
invoke it.
'''
return process.args if isinstance(process.args, str) else ' '.join(process.args)
def output_buffer_for_process(process, exclude_stdouts):
'''
Given a process as an instance of subprocess.Popen and a sequence of stdouts to exclude, return
either the process's stdout or stderr. The idea is that if stdout is excluded for a process, we
still have stderr to log.
'''
return process.stderr if process.stdout in exclude_stdouts else process.stdout
def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path):
'''
Given a sequence of subprocess.Popen() instances for multiple processes, log the output for each
process with the requested log level. Additionally, raise a CalledProcessError if a process
exits with an error (or a warning for exit code 1, if that process matches the Borg local path).
For simplicity, it's assumed that the output buffer for each process is its stdout. But if any
stdouts are given to exclude, then for any matching processes, log from their stderr instead.
Note that stdout for a process can be None if output is intentionally not captured. In which
case it won't be logged.
'''
# Map from output buffer to sequence of last lines.
buffer_last_lines = collections.defaultdict(list)
process_for_output_buffer = {
output_buffer_for_process(process, exclude_stdouts): process
for process in processes
if process.stdout or process.stderr
}
output_buffers = list(process_for_output_buffer.keys())
# Log output for each process until they all exit.
while True:
if output_buffers:
(ready_buffers, _, _) = select.select(output_buffers, [], [])
for ready_buffer in ready_buffers:
ready_process = process_for_output_buffer.get(ready_buffer)
# The "ready" process has exited, but it might be a pipe destination with other
# processes (pipe sources) waiting to be read from. So as a measure to prevent
# hangs, vent all processes when one exits.
if ready_process and ready_process.poll() is not None:
for other_process in processes:
if (
other_process.poll() is None
and other_process.stdout
and other_process.stdout not in output_buffers
):
# Add the process's output to output_buffers to ensure it'll get read.
output_buffers.append(other_process.stdout)
line = ready_buffer.readline().rstrip().decode()
if not line or not ready_process:
continue
# Keep the last few lines of output in case the process errors, and we need the output for
# the exception below.
last_lines = buffer_last_lines[ready_buffer]
last_lines.append(line)
if len(last_lines) > ERROR_OUTPUT_MAX_LINE_COUNT:
last_lines.pop(0)
logger.log(output_log_level, line)
still_running = False
for process in processes:
exit_code = process.poll() if output_buffers else process.wait()
if exit_code is None:
still_running = True
# If any process errors, then raise accordingly.
if exit_code_indicates_error(process, exit_code, borg_local_path):
# If an error occurs, include its output in the raised exception so that we don't
# inadvertently hide error output.
output_buffer = output_buffer_for_process(process, exclude_stdouts)
last_lines = buffer_last_lines[output_buffer] if output_buffer else []
if len(last_lines) == ERROR_OUTPUT_MAX_LINE_COUNT:
last_lines.insert(0, '...')
# Something has gone wrong. So vent each process' output buffer to prevent it from
# hanging. And then kill the process.
for other_process in processes:
if other_process.poll() is None:
other_process.stdout.read(0)
other_process.kill()
raise subprocess.CalledProcessError(
exit_code, command_for_process(process), '\n'.join(last_lines)
)
if not still_running:
break
# Consume any remaining output that we missed (if any).
for process in processes:
output_buffer = output_buffer_for_process(process, exclude_stdouts)
if not output_buffer:
continue
while True: # pragma: no cover
remaining_output = output_buffer.readline().rstrip().decode()
if not remaining_output:
break
logger.log(output_log_level, remaining_output)
def log_command(full_command, input_file, output_file):
'''
Log the given command (a sequence of command/argument strings), along with its input/output file
paths.
'''
logger.debug(
' '.join(full_command)
+ (' < {}'.format(getattr(input_file, 'name', '')) if input_file else '')
+ (' > {}'.format(getattr(output_file, 'name', '')) if output_file else '')
)
# An sentinel passed as an output file to execute_command() to indicate that the command's output
# should be allowed to flow through to stdout without being captured for logging. Useful for
# commands with interactive prompts or those that mess directly with the console.
DO_NOT_CAPTURE = object()
def execute_command(
full_command,
output_log_level=logging.INFO,
output_file=None,
input_file=None,
shell=False,
extra_environment=None,
working_directory=None,
borg_local_path=None,
run_to_completion=True,
):
'''
Execute the given command (a sequence of command/argument strings) and log its output at the
given log level. If output log level is None, instead capture and return the output. (Implies
run_to_completion.) If an open output file object is given, then write stdout to the file and
only log stderr (but only if an output log level is set). If an open input file object is given,
then read stdin from the file. If shell is True, execute the command within a shell. If an extra
environment dict is given, then use it to augment the current environment, and pass the result
into the command. If a working directory is given, use that as the present working directory
when running the command. If a Borg local path is given, and the command matches it (regardless
of arguments), treat exit code 1 as a warning instead of an error. If run to completion is
False, then return the process for the command without executing it to completion.
Raise subprocesses.CalledProcessError if an error occurs while running the command.
'''
log_command(full_command, input_file, output_file)
environment = {**os.environ, **extra_environment} if extra_environment else None
do_not_capture = bool(output_file is DO_NOT_CAPTURE)
command = ' '.join(full_command) if shell else full_command
if output_log_level is None:
output = subprocess.check_output(
command, shell=shell, env=environment, cwd=working_directory
)
return output.decode() if output is not None else None
process = subprocess.Popen(
command,
stdin=input_file,
stdout=None if do_not_capture else (output_file or subprocess.PIPE),
stderr=None if do_not_capture else (subprocess.PIPE if output_file else subprocess.STDOUT),
shell=shell,
env=environment,
cwd=working_directory,
)
if not run_to_completion:
return process
log_outputs(
(process,), (input_file, output_file), output_log_level, borg_local_path=borg_local_path
)
def execute_command_with_processes(
full_command,
processes,
output_log_level=logging.INFO,
output_file=None,
input_file=None,
shell=False,
extra_environment=None,
working_directory=None,
borg_local_path=None,
):
'''
Execute the given command (a sequence of command/argument strings) and log its output at the
given log level. Simultaneously, continue to poll one or more active processes so that they
run as well. This is useful, for instance, for processes that are streaming output to a named
pipe that the given command is consuming from.
If an open output file object is given, then write stdout to the file and only log stderr (but
only if an output log level is set). If an open input file object is given, then read stdin from
the file. If shell is True, execute the command within a shell. If an extra environment dict is
given, then use it to augment the current environment, and pass the result into the command. If
a working directory is given, use that as the present working directory when running the
command. If a Borg local path is given, then for any matching command or process (regardless of
arguments), treat exit code 1 as a warning instead of an error.
Raise subprocesses.CalledProcessError if an error occurs while running the command or in the
upstream process.
'''
log_command(full_command, input_file, output_file)
environment = {**os.environ, **extra_environment} if extra_environment else None
do_not_capture = bool(output_file is DO_NOT_CAPTURE)
command = ' '.join(full_command) if shell else full_command
try:
command_process = subprocess.Popen(
command,
stdin=input_file,
stdout=None if do_not_capture else (output_file or subprocess.PIPE),
stderr=None
if do_not_capture
else (subprocess.PIPE if output_file else subprocess.STDOUT),
shell=shell,
env=environment,
cwd=working_directory,
)
except (subprocess.CalledProcessError, OSError):
# Something has gone wrong. So vent each process' output buffer to prevent it from hanging.
# And then kill the process.
for process in processes:
if process.poll() is None:
process.stdout.read(0)
process.kill()
raise
log_outputs(
tuple(processes) + (command_process,),
(input_file, output_file),
output_log_level,
borg_local_path=borg_local_path,
)

View File

102
borgmatic/hooks/command.py Normal file
View File

@ -0,0 +1,102 @@
import logging
import os
import re
from borgmatic import execute
logger = logging.getLogger(__name__)
SOFT_FAIL_EXIT_CODE = 75
def interpolate_context(config_filename, hook_description, command, context):
'''
Given a config filename, a hook description, a single hook command, and a dict of context
names/values, interpolate the values by "{name}" into the command and return the result.
'''
for name, value in context.items():
command = command.replace('{%s}' % name, str(value))
for unsupported_variable in re.findall(r'{\w+}', command):
logger.warning(
f"{config_filename}: Variable '{unsupported_variable}' is not supported in {hook_description} hook"
)
return command
def execute_hook(commands, umask, config_filename, description, dry_run, **context):
'''
Given a list of hook commands to execute, a umask to execute with (or None), a config filename,
a hook description, and whether this is a dry run, run the given commands. Or, don't run them
if this is a dry run.
The context contains optional values interpolated by name into the hook commands.
Raise ValueError if the umask cannot be parsed.
Raise subprocesses.CalledProcessError if an error occurs in a hook.
'''
if not commands:
logger.debug('{}: No commands to run for {} hook'.format(config_filename, description))
return
dry_run_label = ' (dry run; not actually running hooks)' if dry_run else ''
context['configuration_filename'] = config_filename
commands = [
interpolate_context(config_filename, description, command, context) for command in commands
]
if len(commands) == 1:
logger.info(
'{}: Running command for {} hook{}'.format(config_filename, description, dry_run_label)
)
else:
logger.info(
'{}: Running {} commands for {} hook{}'.format(
config_filename, len(commands), description, dry_run_label
)
)
if umask:
parsed_umask = int(str(umask), 8)
logger.debug('{}: Set hook umask to {}'.format(config_filename, oct(parsed_umask)))
original_umask = os.umask(parsed_umask)
else:
original_umask = None
try:
for command in commands:
if not dry_run:
execute.execute_command(
[command],
output_log_level=logging.ERROR
if description == 'on-error'
else logging.WARNING,
shell=True,
)
finally:
if original_umask:
os.umask(original_umask)
def considered_soft_failure(config_filename, error):
'''
Given a configuration filename and an exception object, return whether the exception object
represents a subprocess.CalledProcessError with a return code of SOFT_FAIL_EXIT_CODE. If so,
that indicates that the error is a "soft failure", and should not result in an error.
'''
exit_code = getattr(error, 'returncode', None)
if exit_code is None:
return False
if exit_code == SOFT_FAIL_EXIT_CODE:
logger.info(
'{}: Command hook exited with soft failure exit code ({}); skipping remaining actions'.format(
config_filename, SOFT_FAIL_EXIT_CODE
)
)
return True
return False

View File

@ -0,0 +1,57 @@
import logging
import requests
from borgmatic.hooks import monitor
logger = logging.getLogger(__name__)
MONITOR_STATE_TO_CRONHUB = {
monitor.State.START: 'start',
monitor.State.FINISH: 'finish',
monitor.State.FAIL: 'fail',
}
def initialize_monitor(
ping_url, config_filename, monitoring_log_level, dry_run
): # pragma: no cover
'''
No initialization is necessary for this monitor.
'''
pass
def ping_monitor(hook_config, config_filename, state, monitoring_log_level, dry_run):
'''
Ping the configured Cronhub URL, modified with the monitor.State. Use the given configuration
filename in any log entries. If this is a dry run, then don't actually ping anything.
'''
dry_run_label = ' (dry run; not actually pinging)' if dry_run else ''
formatted_state = '/{}/'.format(MONITOR_STATE_TO_CRONHUB[state])
ping_url = (
hook_config['ping_url']
.replace('/start/', formatted_state)
.replace('/ping/', formatted_state)
)
logger.info(
'{}: Pinging Cronhub {}{}'.format(config_filename, state.name.lower(), dry_run_label)
)
logger.debug('{}: Using Cronhub ping URL {}'.format(config_filename, ping_url))
if not dry_run:
logging.getLogger('urllib3').setLevel(logging.ERROR)
try:
requests.get(ping_url)
except requests.exceptions.RequestException as error:
logger.warning(f'{config_filename}: Cronhub error: {error}')
def destroy_monitor(
ping_url_or_uuid, config_filename, monitoring_log_level, dry_run
): # pragma: no cover
'''
No destruction is necessary for this monitor.
'''
pass

View File

@ -0,0 +1,52 @@
import logging
import requests
from borgmatic.hooks import monitor
logger = logging.getLogger(__name__)
MONITOR_STATE_TO_CRONITOR = {
monitor.State.START: 'run',
monitor.State.FINISH: 'complete',
monitor.State.FAIL: 'fail',
}
def initialize_monitor(
ping_url, config_filename, monitoring_log_level, dry_run
): # pragma: no cover
'''
No initialization is necessary for this monitor.
'''
pass
def ping_monitor(hook_config, config_filename, state, monitoring_log_level, dry_run):
'''
Ping the configured Cronitor URL, modified with the monitor.State. Use the given configuration
filename in any log entries. If this is a dry run, then don't actually ping anything.
'''
dry_run_label = ' (dry run; not actually pinging)' if dry_run else ''
ping_url = '{}/{}'.format(hook_config['ping_url'], MONITOR_STATE_TO_CRONITOR[state])
logger.info(
'{}: Pinging Cronitor {}{}'.format(config_filename, state.name.lower(), dry_run_label)
)
logger.debug('{}: Using Cronitor ping URL {}'.format(config_filename, ping_url))
if not dry_run:
logging.getLogger('urllib3').setLevel(logging.ERROR)
try:
requests.get(ping_url)
except requests.exceptions.RequestException as error:
logger.warning(f'{config_filename}: Cronitor error: {error}')
def destroy_monitor(
ping_url_or_uuid, config_filename, monitoring_log_level, dry_run
): # pragma: no cover
'''
No destruction is necessary for this monitor.
'''
pass

View File

@ -0,0 +1,63 @@
import logging
from borgmatic.hooks import cronhub, cronitor, healthchecks, mongodb, mysql, pagerduty, postgresql
logger = logging.getLogger(__name__)
HOOK_NAME_TO_MODULE = {
'healthchecks': healthchecks,
'cronitor': cronitor,
'cronhub': cronhub,
'pagerduty': pagerduty,
'postgresql_databases': postgresql,
'mysql_databases': mysql,
'mongodb_databases': mongodb,
}
def call_hook(function_name, hooks, log_prefix, hook_name, *args, **kwargs):
'''
Given the hooks configuration dict and a prefix to use in log entries, call the requested
function of the Python module corresponding to the given hook name. Supply that call with the
configuration for this hook, the log prefix, and any given args and kwargs. Return any return
value.
If the hook name is not present in the hooks configuration, then bail without calling anything.
Raise ValueError if the hook name is unknown.
Raise AttributeError if the function name is not found in the module.
Raise anything else that the called function raises.
'''
config = hooks.get(hook_name)
if not config:
logger.debug('{}: No {} hook configured.'.format(log_prefix, hook_name))
return
try:
module = HOOK_NAME_TO_MODULE[hook_name]
except KeyError:
raise ValueError('Unknown hook name: {}'.format(hook_name))
logger.debug('{}: Calling {} hook function {}'.format(log_prefix, hook_name, function_name))
return getattr(module, function_name)(config, log_prefix, *args, **kwargs)
def call_hooks(function_name, hooks, log_prefix, hook_names, *args, **kwargs):
'''
Given the hooks configuration dict and a prefix to use in log entries, call the requested
function of the Python module corresponding to each given hook name. Supply each call with the
configuration for that hook, the log prefix, and any given args and kwargs. Collect any return
values into a dict from hook name to return value.
If the hook name is not present in the hooks configuration, then don't call the function for it,
and omit it from the return values.
Raise ValueError if the hook name is unknown.
Raise AttributeError if the function name is not found in the module.
Raise anything else that a called function raises. An error stops calls to subsequent functions.
'''
return {
hook_name: call_hook(function_name, hooks, log_prefix, hook_name, *args, **kwargs)
for hook_name in hook_names
if hooks.get(hook_name)
}

76
borgmatic/hooks/dump.py Normal file
View File

@ -0,0 +1,76 @@
import logging
import os
import shutil
from borgmatic.borg.state import DEFAULT_BORGMATIC_SOURCE_DIRECTORY
logger = logging.getLogger(__name__)
DATABASE_HOOK_NAMES = ('postgresql_databases', 'mysql_databases', 'mongodb_databases')
def make_database_dump_path(borgmatic_source_directory, database_hook_name):
'''
Given a borgmatic source directory (or None) and a database hook name, construct a database dump
path.
'''
if not borgmatic_source_directory:
borgmatic_source_directory = DEFAULT_BORGMATIC_SOURCE_DIRECTORY
return os.path.join(borgmatic_source_directory, database_hook_name)
def make_database_dump_filename(dump_path, name, hostname=None):
'''
Based on the given dump directory path, database name, and hostname, return a filename to use
for the database dump. The hostname defaults to localhost.
Raise ValueError if the database name is invalid.
'''
if os.path.sep in name:
raise ValueError('Invalid database name {}'.format(name))
return os.path.join(os.path.expanduser(dump_path), hostname or 'localhost', name)
def create_parent_directory_for_dump(dump_path):
'''
Create a directory to contain the given dump path.
'''
os.makedirs(os.path.dirname(dump_path), mode=0o700, exist_ok=True)
def create_named_pipe_for_dump(dump_path):
'''
Create a named pipe at the given dump path.
'''
create_parent_directory_for_dump(dump_path)
os.mkfifo(dump_path, mode=0o600)
def remove_database_dumps(dump_path, database_type_name, log_prefix, dry_run):
'''
Remove all database dumps in the given dump directory path (including the directory itself). If
this is a dry run, then don't actually remove anything.
'''
dry_run_label = ' (dry run; not actually removing anything)' if dry_run else ''
logger.info(
'{}: Removing {} database dumps{}'.format(log_prefix, database_type_name, dry_run_label)
)
expanded_path = os.path.expanduser(dump_path)
if dry_run:
return
if os.path.exists(expanded_path):
shutil.rmtree(expanded_path)
def convert_glob_patterns_to_borg_patterns(patterns):
'''
Convert a sequence of shell glob patterns like "/etc/*" to the corresponding Borg archive
patterns like "sh:etc/*".
'''
return ['sh:{}'.format(pattern.lstrip(os.path.sep)) for pattern in patterns]

View File

@ -0,0 +1,142 @@
import logging
import requests
from borgmatic.hooks import monitor
logger = logging.getLogger(__name__)
MONITOR_STATE_TO_HEALTHCHECKS = {
monitor.State.START: 'start',
monitor.State.FINISH: None, # Healthchecks doesn't append to the URL for the finished state.
monitor.State.FAIL: 'fail',
}
PAYLOAD_TRUNCATION_INDICATOR = '...\n'
DEFAULT_PING_BODY_LIMIT_BYTES = 100000
class Forgetful_buffering_handler(logging.Handler):
'''
A buffering log handler that stores log messages in memory, and throws away messages (oldest
first) once a particular capacity in bytes is reached. But if the given byte capacity is zero,
don't throw away any messages.
'''
def __init__(self, byte_capacity, log_level):
super().__init__()
self.byte_capacity = byte_capacity
self.byte_count = 0
self.buffer = []
self.forgot = False
self.setLevel(log_level)
def emit(self, record):
message = record.getMessage() + '\n'
self.byte_count += len(message)
self.buffer.append(message)
if not self.byte_capacity:
return
while self.byte_count > self.byte_capacity and self.buffer:
self.byte_count -= len(self.buffer[0])
self.buffer.pop(0)
self.forgot = True
def format_buffered_logs_for_payload():
'''
Get the handler previously added to the root logger, and slurp buffered logs out of it to
send to Healthchecks.
'''
try:
buffering_handler = next(
handler
for handler in logging.getLogger().handlers
if isinstance(handler, Forgetful_buffering_handler)
)
except StopIteration:
# No handler means no payload.
return ''
payload = ''.join(message for message in buffering_handler.buffer)
if buffering_handler.forgot:
return PAYLOAD_TRUNCATION_INDICATOR + payload
return payload
def initialize_monitor(hook_config, config_filename, monitoring_log_level, dry_run):
'''
Add a handler to the root logger that stores in memory the most recent logs emitted. That way,
we can send them all to Healthchecks upon a finish or failure state. But skip this if the
"send_logs" option is false.
'''
if hook_config.get('send_logs') is False:
return
ping_body_limit = max(
hook_config.get('ping_body_limit', DEFAULT_PING_BODY_LIMIT_BYTES)
- len(PAYLOAD_TRUNCATION_INDICATOR),
0,
)
logging.getLogger().addHandler(
Forgetful_buffering_handler(ping_body_limit, monitoring_log_level)
)
def ping_monitor(hook_config, config_filename, state, monitoring_log_level, dry_run):
'''
Ping the configured Healthchecks URL or UUID, modified with the monitor.State. Use the given
configuration filename in any log entries, and log to Healthchecks with the giving log level.
If this is a dry run, then don't actually ping anything.
'''
ping_url = (
hook_config['ping_url']
if hook_config['ping_url'].startswith('http')
else 'https://hc-ping.com/{}'.format(hook_config['ping_url'])
)
dry_run_label = ' (dry run; not actually pinging)' if dry_run else ''
if 'states' in hook_config and state.name.lower() not in hook_config['states']:
logger.info(
f'{config_filename}: Skipping Healthchecks {state.name.lower()} ping due to configured states'
)
return
healthchecks_state = MONITOR_STATE_TO_HEALTHCHECKS.get(state)
if healthchecks_state:
ping_url = '{}/{}'.format(ping_url, healthchecks_state)
logger.info(
'{}: Pinging Healthchecks {}{}'.format(config_filename, state.name.lower(), dry_run_label)
)
logger.debug('{}: Using Healthchecks ping URL {}'.format(config_filename, ping_url))
if state in (monitor.State.FINISH, monitor.State.FAIL):
payload = format_buffered_logs_for_payload()
else:
payload = ''
if not dry_run:
logging.getLogger('urllib3').setLevel(logging.ERROR)
try:
requests.post(ping_url, data=payload.encode('utf-8'))
except requests.exceptions.RequestException as error:
logger.warning(f'{config_filename}: Healthchecks error: {error}')
def destroy_monitor(hook_config, config_filename, monitoring_log_level, dry_run):
'''
Remove the monitor handler that was added to the root logger. This prevents the handler from
getting reused by other instances of this monitor.
'''
logger = logging.getLogger()
for handler in tuple(logger.handlers):
if isinstance(handler, Forgetful_buffering_handler):
logger.removeHandler(handler)

162
borgmatic/hooks/mongodb.py Normal file
View File

@ -0,0 +1,162 @@
import logging
from borgmatic.execute import execute_command, execute_command_with_processes
from borgmatic.hooks import dump
logger = logging.getLogger(__name__)
def make_dump_path(location_config): # pragma: no cover
'''
Make the dump path from the given location configuration and the name of this hook.
'''
return dump.make_database_dump_path(
location_config.get('borgmatic_source_directory'), 'mongodb_databases'
)
def dump_databases(databases, log_prefix, location_config, dry_run):
'''
Dump the given MongoDB databases to a named pipe. The databases are supplied as a sequence of
dicts, one dict describing each database as per the configuration schema. Use the given log
prefix in any log entries. Use the given location configuration dict to construct the
destination path.
Return a sequence of subprocess.Popen instances for the dump processes ready to spew to a named
pipe. But if this is a dry run, then don't actually dump anything and return an empty sequence.
'''
dry_run_label = ' (dry run; not actually dumping anything)' if dry_run else ''
logger.info('{}: Dumping MongoDB databases{}'.format(log_prefix, dry_run_label))
processes = []
for database in databases:
name = database['name']
dump_filename = dump.make_database_dump_filename(
make_dump_path(location_config), name, database.get('hostname')
)
dump_format = database.get('format', 'archive')
logger.debug(
'{}: Dumping MongoDB database {} to {}{}'.format(
log_prefix, name, dump_filename, dry_run_label
)
)
if dry_run:
continue
if dump_format == 'directory':
dump.create_parent_directory_for_dump(dump_filename)
else:
dump.create_named_pipe_for_dump(dump_filename)
command = build_dump_command(database, dump_filename, dump_format)
processes.append(execute_command(command, shell=True, run_to_completion=False))
return processes
def build_dump_command(database, dump_filename, dump_format):
'''
Return the mongodump command from a single database configuration.
'''
all_databases = database['name'] == 'all'
command = ['mongodump', '--archive']
if dump_format == 'directory':
command.append(dump_filename)
if 'hostname' in database:
command.extend(('--host', database['hostname']))
if 'port' in database:
command.extend(('--port', str(database['port'])))
if 'username' in database:
command.extend(('--username', database['username']))
if 'password' in database:
command.extend(('--password', database['password']))
if 'authentication_database' in database:
command.extend(('--authenticationDatabase', database['authentication_database']))
if not all_databases:
command.extend(('--db', database['name']))
if 'options' in database:
command.extend(database['options'].split(' '))
if dump_format != 'directory':
command.extend(('>', dump_filename))
return command
def remove_database_dumps(databases, log_prefix, location_config, dry_run): # pragma: no cover
'''
Remove all database dump files for this hook regardless of the given databases. Use the log
prefix in any log entries. Use the given location configuration dict to construct the
destination path. If this is a dry run, then don't actually remove anything.
'''
dump.remove_database_dumps(make_dump_path(location_config), 'MongoDB', log_prefix, dry_run)
def make_database_dump_pattern(
databases, log_prefix, location_config, name=None
): # pragma: no cover
'''
Given a sequence of configurations dicts, a prefix to log with, a location configuration dict,
and a database name to match, return the corresponding glob patterns to match the database dump
in an archive.
'''
return dump.make_database_dump_filename(make_dump_path(location_config), name, hostname='*')
def restore_database_dump(database_config, log_prefix, location_config, dry_run, extract_process):
'''
Restore the given MongoDB database from an extract stream. The database is supplied as a
one-element sequence containing a dict describing the database, as per the configuration schema.
Use the given log prefix in any log entries. If this is a dry run, then don't actually restore
anything. Trigger the given active extract process (an instance of subprocess.Popen) to produce
output to consume.
If the extract process is None, then restore the dump from the filesystem rather than from an
extract stream.
'''
dry_run_label = ' (dry run; not actually restoring anything)' if dry_run else ''
if len(database_config) != 1:
raise ValueError('The database configuration value is invalid')
database = database_config[0]
dump_filename = dump.make_database_dump_filename(
make_dump_path(location_config), database['name'], database.get('hostname')
)
restore_command = build_restore_command(extract_process, database, dump_filename)
logger.debug(
'{}: Restoring MongoDB database {}{}'.format(log_prefix, database['name'], dry_run_label)
)
if dry_run:
return
execute_command_with_processes(
restore_command,
[extract_process] if extract_process else [],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout if extract_process else None,
borg_local_path=location_config.get('local_path', 'borg'),
)
def build_restore_command(extract_process, database, dump_filename):
'''
Return the mongorestore command from a single database configuration.
'''
command = ['mongorestore', '--archive']
if not extract_process:
command.append(dump_filename)
if database['name'] != 'all':
command.extend(('--drop', '--db', database['name']))
if 'hostname' in database:
command.extend(('--host', database['hostname']))
if 'port' in database:
command.extend(('--port', str(database['port'])))
if 'username' in database:
command.extend(('--username', database['username']))
if 'password' in database:
command.extend(('--password', database['password']))
if 'authentication_database' in database:
command.extend(('--authenticationDatabase', database['authentication_database']))
return command

View File

@ -0,0 +1,9 @@
from enum import Enum
MONITOR_HOOK_NAMES = ('healthchecks', 'cronitor', 'cronhub', 'pagerduty')
class State(Enum):
START = 1
FINISH = 2
FAIL = 3

176
borgmatic/hooks/mysql.py Normal file
View File

@ -0,0 +1,176 @@
import logging
from borgmatic.execute import execute_command, execute_command_with_processes
from borgmatic.hooks import dump
logger = logging.getLogger(__name__)
def make_dump_path(location_config): # pragma: no cover
'''
Make the dump path from the given location configuration and the name of this hook.
'''
return dump.make_database_dump_path(
location_config.get('borgmatic_source_directory'), 'mysql_databases'
)
SYSTEM_DATABASE_NAMES = ('information_schema', 'mysql', 'performance_schema', 'sys')
def database_names_to_dump(database, extra_environment, log_prefix, dry_run_label):
'''
Given a requested database name, return the corresponding sequence of database names to dump.
In the case of "all", query for the names of databases on the configured host and return them,
excluding any system databases that will cause problems during restore.
'''
requested_name = database['name']
if requested_name != 'all':
return (requested_name,)
show_command = (
('mysql',)
+ (tuple(database['list_options'].split(' ')) if 'list_options' in database else ())
+ (('--host', database['hostname']) if 'hostname' in database else ())
+ (('--port', str(database['port'])) if 'port' in database else ())
+ (('--protocol', 'tcp') if 'hostname' in database or 'port' in database else ())
+ (('--user', database['username']) if 'username' in database else ())
+ ('--skip-column-names', '--batch')
+ ('--execute', 'show schemas')
)
logger.debug(
'{}: Querying for "all" MySQL databases to dump{}'.format(log_prefix, dry_run_label)
)
show_output = execute_command(
show_command, output_log_level=None, extra_environment=extra_environment
)
return tuple(
show_name
for show_name in show_output.strip().splitlines()
if show_name not in SYSTEM_DATABASE_NAMES
)
def dump_databases(databases, log_prefix, location_config, dry_run):
'''
Dump the given MySQL/MariaDB databases to a named pipe. The databases are supplied as a sequence
of dicts, one dict describing each database as per the configuration schema. Use the given log
prefix in any log entries. Use the given location configuration dict to construct the
destination path.
Return a sequence of subprocess.Popen instances for the dump processes ready to spew to a named
pipe. But if this is a dry run, then don't actually dump anything and return an empty sequence.
'''
dry_run_label = ' (dry run; not actually dumping anything)' if dry_run else ''
processes = []
logger.info('{}: Dumping MySQL databases{}'.format(log_prefix, dry_run_label))
for database in databases:
requested_name = database['name']
dump_filename = dump.make_database_dump_filename(
make_dump_path(location_config), requested_name, database.get('hostname')
)
extra_environment = {'MYSQL_PWD': database['password']} if 'password' in database else None
dump_database_names = database_names_to_dump(
database, extra_environment, log_prefix, dry_run_label
)
if not dump_database_names:
raise ValueError('Cannot find any MySQL databases to dump.')
dump_command = (
('mysqldump',)
+ (tuple(database['options'].split(' ')) if 'options' in database else ())
+ ('--add-drop-database',)
+ (('--host', database['hostname']) if 'hostname' in database else ())
+ (('--port', str(database['port'])) if 'port' in database else ())
+ (('--protocol', 'tcp') if 'hostname' in database or 'port' in database else ())
+ (('--user', database['username']) if 'username' in database else ())
+ ('--databases',)
+ dump_database_names
# Use shell redirection rather than execute_command(output_file=open(...)) to prevent
# the open() call on a named pipe from hanging the main borgmatic process.
+ ('>', dump_filename)
)
logger.debug(
'{}: Dumping MySQL database {} to {}{}'.format(
log_prefix, requested_name, dump_filename, dry_run_label
)
)
if dry_run:
continue
dump.create_named_pipe_for_dump(dump_filename)
processes.append(
execute_command(
dump_command,
shell=True,
extra_environment=extra_environment,
run_to_completion=False,
)
)
return processes
def remove_database_dumps(databases, log_prefix, location_config, dry_run): # pragma: no cover
'''
Remove all database dump files for this hook regardless of the given databases. Use the log
prefix in any log entries. Use the given location configuration dict to construct the
destination path. If this is a dry run, then don't actually remove anything.
'''
dump.remove_database_dumps(make_dump_path(location_config), 'MySQL', log_prefix, dry_run)
def make_database_dump_pattern(
databases, log_prefix, location_config, name=None
): # pragma: no cover
'''
Given a sequence of configurations dicts, a prefix to log with, a location configuration dict,
and a database name to match, return the corresponding glob patterns to match the database dump
in an archive.
'''
return dump.make_database_dump_filename(make_dump_path(location_config), name, hostname='*')
def restore_database_dump(database_config, log_prefix, location_config, dry_run, extract_process):
'''
Restore the given MySQL/MariaDB database from an extract stream. The database is supplied as a
one-element sequence containing a dict describing the database, as per the configuration schema.
Use the given log prefix in any log entries. If this is a dry run, then don't actually restore
anything. Trigger the given active extract process (an instance of subprocess.Popen) to produce
output to consume.
'''
dry_run_label = ' (dry run; not actually restoring anything)' if dry_run else ''
if len(database_config) != 1:
raise ValueError('The database configuration value is invalid')
database = database_config[0]
restore_command = (
('mysql', '--batch')
+ (('--host', database['hostname']) if 'hostname' in database else ())
+ (('--port', str(database['port'])) if 'port' in database else ())
+ (('--protocol', 'tcp') if 'hostname' in database or 'port' in database else ())
+ (('--user', database['username']) if 'username' in database else ())
)
extra_environment = {'MYSQL_PWD': database['password']} if 'password' in database else None
logger.debug(
'{}: Restoring MySQL database {}{}'.format(log_prefix, database['name'], dry_run_label)
)
if dry_run:
return
execute_command_with_processes(
restore_command,
[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
extra_environment=extra_environment,
borg_local_path=location_config.get('local_path', 'borg'),
)

View File

@ -0,0 +1,83 @@
import datetime
import json
import logging
import platform
import requests
from borgmatic.hooks import monitor
logger = logging.getLogger(__name__)
EVENTS_API_URL = 'https://events.pagerduty.com/v2/enqueue'
def initialize_monitor(
integration_key, config_filename, monitoring_log_level, dry_run
): # pragma: no cover
'''
No initialization is necessary for this monitor.
'''
pass
def ping_monitor(hook_config, config_filename, state, monitoring_log_level, dry_run):
'''
If this is an error state, create a PagerDuty event with the configured integration key. Use
the given configuration filename in any log entries. If this is a dry run, then don't actually
create an event.
'''
if state != monitor.State.FAIL:
logger.debug(
'{}: Ignoring unsupported monitoring {} in PagerDuty hook'.format(
config_filename, state.name.lower()
)
)
return
dry_run_label = ' (dry run; not actually sending)' if dry_run else ''
logger.info('{}: Sending failure event to PagerDuty {}'.format(config_filename, dry_run_label))
if dry_run:
return
hostname = platform.node()
local_timestamp = (
datetime.datetime.utcnow().replace(tzinfo=datetime.timezone.utc).astimezone().isoformat()
)
payload = json.dumps(
{
'routing_key': hook_config['integration_key'],
'event_action': 'trigger',
'payload': {
'summary': 'backup failed on {}'.format(hostname),
'severity': 'error',
'source': hostname,
'timestamp': local_timestamp,
'component': 'borgmatic',
'group': 'backups',
'class': 'backup failure',
'custom_details': {
'hostname': hostname,
'configuration filename': config_filename,
'server time': local_timestamp,
},
},
}
)
logger.debug('{}: Using PagerDuty payload: {}'.format(config_filename, payload))
logging.getLogger('urllib3').setLevel(logging.ERROR)
try:
requests.post(EVENTS_API_URL, data=payload.encode('utf-8'))
except requests.exceptions.RequestException as error:
logger.warning(f'{config_filename}: PagerDuty error: {error}')
def destroy_monitor(
ping_url_or_uuid, config_filename, monitoring_log_level, dry_run
): # pragma: no cover
'''
No destruction is necessary for this monitor.
'''
pass

View File

@ -0,0 +1,179 @@
import logging
from borgmatic.execute import execute_command, execute_command_with_processes
from borgmatic.hooks import dump
logger = logging.getLogger(__name__)
def make_dump_path(location_config): # pragma: no cover
'''
Make the dump path from the given location configuration and the name of this hook.
'''
return dump.make_database_dump_path(
location_config.get('borgmatic_source_directory'), 'postgresql_databases'
)
def make_extra_environment(database):
'''
Make the extra_environment dict from the given database configuration.
'''
extra = dict()
if 'password' in database:
extra['PGPASSWORD'] = database['password']
extra['PGSSLMODE'] = database.get('ssl_mode', 'disable')
if 'ssl_cert' in database:
extra['PGSSLCERT'] = database['ssl_cert']
if 'ssl_key' in database:
extra['PGSSLKEY'] = database['ssl_key']
if 'ssl_root_cert' in database:
extra['PGSSLROOTCERT'] = database['ssl_root_cert']
if 'ssl_crl' in database:
extra['PGSSLCRL'] = database['ssl_crl']
return extra
def dump_databases(databases, log_prefix, location_config, dry_run):
'''
Dump the given PostgreSQL databases to a named pipe. The databases are supplied as a sequence of
dicts, one dict describing each database as per the configuration schema. Use the given log
prefix in any log entries. Use the given location configuration dict to construct the
destination path.
Return a sequence of subprocess.Popen instances for the dump processes ready to spew to a named
pipe. But if this is a dry run, then don't actually dump anything and return an empty sequence.
'''
dry_run_label = ' (dry run; not actually dumping anything)' if dry_run else ''
processes = []
logger.info('{}: Dumping PostgreSQL databases{}'.format(log_prefix, dry_run_label))
for database in databases:
name = database['name']
dump_filename = dump.make_database_dump_filename(
make_dump_path(location_config), name, database.get('hostname')
)
all_databases = bool(name == 'all')
dump_format = database.get('format', 'custom')
command = (
(
'pg_dumpall' if all_databases else 'pg_dump',
'--no-password',
'--clean',
'--if-exists',
)
+ (('--host', database['hostname']) if 'hostname' in database else ())
+ (('--port', str(database['port'])) if 'port' in database else ())
+ (('--username', database['username']) if 'username' in database else ())
+ (() if all_databases else ('--format', dump_format))
+ (('--file', dump_filename) if dump_format == 'directory' else ())
+ (tuple(database['options'].split(' ')) if 'options' in database else ())
+ (() if all_databases else (name,))
# Use shell redirection rather than the --file flag to sidestep synchronization issues
# when pg_dump/pg_dumpall tries to write to a named pipe. But for the directory dump
# format in a particular, a named destination is required, and redirection doesn't work.
+ (('>', dump_filename) if dump_format != 'directory' else ())
)
extra_environment = make_extra_environment(database)
logger.debug(
'{}: Dumping PostgreSQL database {} to {}{}'.format(
log_prefix, name, dump_filename, dry_run_label
)
)
if dry_run:
continue
if dump_format == 'directory':
dump.create_parent_directory_for_dump(dump_filename)
else:
dump.create_named_pipe_for_dump(dump_filename)
processes.append(
execute_command(
command, shell=True, extra_environment=extra_environment, run_to_completion=False
)
)
return processes
def remove_database_dumps(databases, log_prefix, location_config, dry_run): # pragma: no cover
'''
Remove all database dump files for this hook regardless of the given databases. Use the log
prefix in any log entries. Use the given location configuration dict to construct the
destination path. If this is a dry run, then don't actually remove anything.
'''
dump.remove_database_dumps(make_dump_path(location_config), 'PostgreSQL', log_prefix, dry_run)
def make_database_dump_pattern(
databases, log_prefix, location_config, name=None
): # pragma: no cover
'''
Given a sequence of configurations dicts, a prefix to log with, a location configuration dict,
and a database name to match, return the corresponding glob patterns to match the database dump
in an archive.
'''
return dump.make_database_dump_filename(make_dump_path(location_config), name, hostname='*')
def restore_database_dump(database_config, log_prefix, location_config, dry_run, extract_process):
'''
Restore the given PostgreSQL database from an extract stream. The database is supplied as a
one-element sequence containing a dict describing the database, as per the configuration schema.
Use the given log prefix in any log entries. If this is a dry run, then don't actually restore
anything. Trigger the given active extract process (an instance of subprocess.Popen) to produce
output to consume.
If the extract process is None, then restore the dump from the filesystem rather than from an
extract stream.
'''
dry_run_label = ' (dry run; not actually restoring anything)' if dry_run else ''
if len(database_config) != 1:
raise ValueError('The database configuration value is invalid')
database = database_config[0]
all_databases = bool(database['name'] == 'all')
dump_filename = dump.make_database_dump_filename(
make_dump_path(location_config), database['name'], database.get('hostname')
)
analyze_command = (
('psql', '--no-password', '--quiet')
+ (('--host', database['hostname']) if 'hostname' in database else ())
+ (('--port', str(database['port'])) if 'port' in database else ())
+ (('--username', database['username']) if 'username' in database else ())
+ (('--dbname', database['name']) if not all_databases else ())
+ ('--command', 'ANALYZE')
)
restore_command = (
('psql' if all_databases else 'pg_restore', '--no-password')
+ (
('--if-exists', '--exit-on-error', '--clean', '--dbname', database['name'])
if not all_databases
else ()
)
+ (('--host', database['hostname']) if 'hostname' in database else ())
+ (('--port', str(database['port'])) if 'port' in database else ())
+ (('--username', database['username']) if 'username' in database else ())
+ (() if extract_process else (dump_filename,))
)
extra_environment = make_extra_environment(database)
logger.debug(
'{}: Restoring PostgreSQL database {}{}'.format(log_prefix, database['name'], dry_run_label)
)
if dry_run:
return
execute_command_with_processes(
restore_command,
[extract_process] if extract_process else [],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout if extract_process else None,
extra_environment=extra_environment,
borg_local_path=location_config.get('local_path', 'borg'),
)
execute_command(analyze_command, extra_environment=extra_environment)

174
borgmatic/logger.py Normal file
View File

@ -0,0 +1,174 @@
import logging
import logging.handlers
import os
import sys
import colorama
def to_bool(arg):
'''
Return a boolean value based on `arg`.
'''
if arg is None or isinstance(arg, bool):
return arg
if isinstance(arg, str):
arg = arg.lower()
if arg in ('yes', 'on', '1', 'true', 1):
return True
return False
def interactive_console():
'''
Return whether the current console is "interactive". Meaning: Capable of
user input and not just something like a cron job.
'''
return sys.stderr.isatty() and os.environ.get('TERM') != 'dumb'
def should_do_markup(no_color, configs):
'''
Given the value of the command-line no-color argument, and a dict of configuration filename to
corresponding parsed configuration, determine if we should enable colorama marking up.
'''
if no_color:
return False
if any(config.get('output', {}).get('color') is False for config in configs.values()):
return False
py_colors = os.environ.get('PY_COLORS', None)
if py_colors is not None:
return to_bool(py_colors)
return interactive_console()
class Multi_stream_handler(logging.Handler):
'''
A logging handler that dispatches each log record to one of multiple stream handlers depending
on the record's log level.
'''
def __init__(self, log_level_to_stream_handler):
super(Multi_stream_handler, self).__init__()
self.log_level_to_handler = log_level_to_stream_handler
self.handlers = set(self.log_level_to_handler.values())
def flush(self): # pragma: no cover
super(Multi_stream_handler, self).flush()
for handler in self.handlers:
handler.flush()
def emit(self, record):
'''
Dispatch the log record to the approriate stream handler for the record's log level.
'''
self.log_level_to_handler[record.levelno].emit(record)
def setFormatter(self, formatter): # pragma: no cover
super(Multi_stream_handler, self).setFormatter(formatter)
for handler in self.handlers:
handler.setFormatter(formatter)
def setLevel(self, level): # pragma: no cover
super(Multi_stream_handler, self).setLevel(level)
for handler in self.handlers:
handler.setLevel(level)
LOG_LEVEL_TO_COLOR = {
logging.CRITICAL: colorama.Fore.RED,
logging.ERROR: colorama.Fore.RED,
logging.WARN: colorama.Fore.YELLOW,
logging.INFO: colorama.Fore.GREEN,
logging.DEBUG: colorama.Fore.CYAN,
}
class Console_color_formatter(logging.Formatter):
def format(self, record):
color = LOG_LEVEL_TO_COLOR.get(record.levelno)
return color_text(color, record.msg)
def color_text(color, message):
'''
Give colored text.
'''
if not color:
return message
return '{}{}{}'.format(color, message, colorama.Style.RESET_ALL)
def configure_logging(
console_log_level,
syslog_log_level=None,
log_file_log_level=None,
monitoring_log_level=None,
log_file=None,
):
'''
Configure logging to go to both the console and (syslog or log file). Use the given log levels,
respectively.
Raise FileNotFoundError or PermissionError if the log file could not be opened for writing.
'''
if syslog_log_level is None:
syslog_log_level = console_log_level
if log_file_log_level is None:
log_file_log_level = console_log_level
if monitoring_log_level is None:
monitoring_log_level = console_log_level
# Log certain log levels to console stderr and others to stdout. This supports use cases like
# grepping (non-error) output.
console_error_handler = logging.StreamHandler(sys.stderr)
console_standard_handler = logging.StreamHandler(sys.stdout)
console_handler = Multi_stream_handler(
{
logging.CRITICAL: console_error_handler,
logging.ERROR: console_error_handler,
logging.WARN: console_standard_handler,
logging.INFO: console_standard_handler,
logging.DEBUG: console_standard_handler,
}
)
console_handler.setFormatter(Console_color_formatter())
console_handler.setLevel(console_log_level)
syslog_path = None
if log_file is None:
if os.path.exists('/dev/log'):
syslog_path = '/dev/log'
elif os.path.exists('/var/run/syslog'):
syslog_path = '/var/run/syslog'
elif os.path.exists('/var/run/log'):
syslog_path = '/var/run/log'
if syslog_path and not interactive_console():
syslog_handler = logging.handlers.SysLogHandler(address=syslog_path)
syslog_handler.setFormatter(logging.Formatter('borgmatic: %(levelname)s %(message)s'))
syslog_handler.setLevel(syslog_log_level)
handlers = (console_handler, syslog_handler)
elif log_file:
file_handler = logging.handlers.WatchedFileHandler(log_file)
file_handler.setFormatter(logging.Formatter('[%(asctime)s] %(levelname)s: %(message)s'))
file_handler.setLevel(log_file_log_level)
handlers = (console_handler, file_handler)
else:
handlers = (console_handler,)
logging.basicConfig(
level=min(console_log_level, syslog_log_level, log_file_log_level, monitoring_log_level),
handlers=handlers,
)

34
borgmatic/signals.py Normal file
View File

@ -0,0 +1,34 @@
import logging
import os
import signal
import sys
logger = logging.getLogger(__name__)
EXIT_CODE_FROM_SIGNAL = 128
def handle_signal(signal_number, frame):
'''
Send the signal to all processes in borgmatic's process group, which includes child processes.
'''
# Prevent infinite signal handler recursion. If the parent frame is this very same handler
# function, we know we're recursing.
if frame.f_back.f_code.co_name == handle_signal.__name__:
return
os.killpg(os.getpgrp(), signal_number)
if signal_number == signal.SIGTERM:
logger.critical('Exiting due to TERM signal')
sys.exit(EXIT_CODE_FROM_SIGNAL + signal.SIGTERM)
def configure_signals():
'''
Configure borgmatic's signal handlers to pass relevant signals through to any child processes
like Borg. Note that SIGINT gets passed through even without these changes.
'''
for signal_number in (signal.SIGHUP, signal.SIGTERM, signal.SIGUSR1, signal.SIGUSR2):
signal.signal(signal_number, handle_signal)

18
borgmatic/verbosity.py Normal file
View File

@ -0,0 +1,18 @@
import logging
VERBOSITY_ERROR = -1
VERBOSITY_WARNING = 0
VERBOSITY_SOME = 1
VERBOSITY_LOTS = 2
def verbosity_to_log_level(verbosity):
'''
Given a borgmatic verbosity value, return the corresponding Python log level.
'''
return {
VERBOSITY_ERROR: logging.ERROR,
VERBOSITY_WARNING: logging.WARNING,
VERBOSITY_SOME: logging.INFO,
VERBOSITY_LOTS: logging.DEBUG,
}.get(verbosity, logging.WARNING)

33
docs/Dockerfile Normal file
View File

@ -0,0 +1,33 @@
FROM python:3.8-alpine3.13 as borgmatic
COPY . /app
RUN apk add --no-cache py3-ruamel.yaml py3-ruamel.yaml.clib
RUN pip install --no-cache /app && generate-borgmatic-config && chmod +r /etc/borgmatic/config.yaml
RUN borgmatic --help > /command-line.txt \
&& for action in init prune compact create check extract export-tar mount umount restore list info borg; do \
echo -e "\n--------------------------------------------------------------------------------\n" >> /command-line.txt \
&& borgmatic "$action" --help >> /command-line.txt; done
FROM node:15.2.1-alpine as html
ARG ENVIRONMENT=production
WORKDIR /source
RUN npm install @11ty/eleventy \
@11ty/eleventy-plugin-syntaxhighlight \
@11ty/eleventy-plugin-inclusive-language \
@11ty/eleventy-navigation \
markdown-it \
markdown-it-anchor \
markdown-it-replace-link
COPY --from=borgmatic /etc/borgmatic/config.yaml /source/docs/_includes/borgmatic/config.yaml
COPY --from=borgmatic /command-line.txt /source/docs/_includes/borgmatic/command-line.txt
COPY . /source
RUN NODE_ENV=${ENVIRONMENT} npx eleventy --input=/source/docs --output=/output/docs \
&& mv /output/docs/index.html /output/index.html
FROM nginx:1.19.4-alpine
COPY --from=html /output /usr/share/nginx/html
COPY --from=borgmatic /etc/borgmatic/config.yaml /usr/share/nginx/html/docs/reference/config.yaml

1
docs/README.md Symbolic link
View File

@ -0,0 +1 @@
../README.md

19
docs/SECURITY.md Normal file
View File

@ -0,0 +1,19 @@
---
title: Security policy
permalink: security-policy/index.html
---
## Supported versions
While we want to hear about security vulnerabilities in all versions of
borgmatic, security fixes will only be made to the most recently released
version. It's not practical for our small volunteer effort to maintain
multiple different release branches and put out separate security patches for
each.
## Reporting a vulnerability
If you find a security vulnerability, please [file a
ticket](https://torsion.org/borgmatic/#issues) or [send email
directly](mailto:witten@torsion.org) as appropriate. You should expect to hear
back within a few days at most, and generally sooner.

1
docs/_data/layout.json Normal file
View File

@ -0,0 +1 @@
"layouts/main.njk"

View File

@ -0,0 +1,3 @@
.asciicast > iframe {
width: 100% !important;
}

View File

@ -0,0 +1,12 @@
/* External links */
a[href^="http://"]:not(.minilink):not(.elv-externalexempt),
a[href^="https://"]:not(.minilink):not(.elv-externalexempt),
a[href^="//"]:not(.minilink):not(.elv-externalexempt) {
text-decoration-color: inherit;
}
/* External link hovers */
a[href^="http://"]:not(.minilink):not(.elv-externalexempt):hover,
a[href^="https://"]:not(.minilink):not(.elv-externalexempt):hover,
a[href^="//"]:not(.minilink):not(.elv-externalexempt):hover {
text-decoration-color: #00bcd4;
}

View File

@ -0,0 +1,34 @@
/* Warning */
.elv-info {
line-height: 1.5;
padding: 0.8125em 1em 0.75em; /* 13px 16px 12px /16 */
margin-left: -1rem;
margin-right: -1rem;
margin-bottom: 2em;
background-color: #dff7ff;
}
.elv-info:before {
content: " ";
}
.elv-info-warn {
background-color: #ffa;
}
.elv-info-warn:before {
content: "⚠️ ";
}
.elv-info:first-child {
margin-top: 0;
}
body > .elv-info {
margin-left: 0;
margin-right: 0;
padding: .5rem 1rem;
}
@media (min-width: 37.5em) and (min-height: 25em) { /* 600px / 400px */
body > .elv-info-sticky {
position: sticky;
top: 0;
z-index: 2;
box-shadow: 0 3px 0 0 rgba(0,0,0,.08);
}
}

View File

@ -0,0 +1,278 @@
/* Buzzwords */
@keyframes rainbow {
0% { background-position: 0% 50%; }
100% { background-position: 100% 50%; }
}
.buzzword-list,
.inlinelist {
padding: 0;
}
.inlinelist:first-child:last-child {
margin: 0;
}
.buzzword,
.buzzword-list li,
.inlinelist .inlinelist-item {
display: inline;
-webkit-box-decoration-break: clone;
box-decoration-break: clone;
font-family: Georgia, serif;
font-size: 116%;
white-space: normal;
line-height: 1.85;
padding: .2em .5em;
margin: 4px 4px 4px 0;
transition: .15s linear outline;
}
.inlinelist .inlinelist-item.active {
background-color: #222;
color: #fff;
font-weight: inherit;
}
.inlinelist .inlinelist-item.active :link,
.inlinelist .inlinelist-item.active :visited {
color: #fff;
}
.inlinelist .inlinelist-item code {
background-color: transparent;
font-size: 80%;
margin-left: 6px;
padding-left: 6px;
display: inline-block;
position: relative;
}
@media (max-width: 26.8125em) { /* 429px */
.inlinelist .inlinelist-item {
overflow: hidden;
}
.inlinelist .inlinelist-item code {
float: right;
line-height: 1.75;
}
}
@media (min-width: 26.875em) { /* 430px */
.inlinelist .inlinelist-item code {
float: none;
}
.inlinelist .inlinelist-item code:before {
content: " ";
border-left: 1px solid rgba(255,255,255,.8);
position: absolute;
left: -2px;
top: -2px;
bottom: 2px;
}
@media (prefers-color-scheme: dark) {
.inlinelist .inlinelist-item code:before {
border-left-color: rgba(0,0,0,.8);
}
}
}
a.buzzword {
text-decoration: underline;
}
.buzzword-list a,
.inlinelist a {
text-decoration: none;
}
.inlinelist .inlinelist-item {
font-size: 100%;
line-height: 2;
}
@supports not(-webkit-box-decoration-break: clone) {
.buzzword,
.buzzword-list li,
.inlinelist .inlinelist-item {
display: inline-block;
}
}
.buzzword-list li,
.buzzword {
background-color: #f7f7f7;
}
@media (prefers-color-scheme: dark) {
.buzzword-list li,
.buzzword {
background-color: #080808;
}
}
.inlinelist .inlinelist-item {
background-color: #e9e9e9;
}
@media (prefers-color-scheme: dark) {
.inlinelist .inlinelist-item {
background-color: #000;
}
.inlinelist .inlinelist-item a {
color: #fff;
}
.inlinelist .inlinelist-item code {
color: inherit;
}
}
.inlinelist .inlinelist-item:hover,
.inlinelist .inlinelist-item:focus,
.buzzword-list li:hover,
.buzzword-list li:focus,
.buzzword:hover,
.buzzword:focus,
.rainbow-active:hover,
.rainbow-active:focus {
position: relative;
background-image: linear-gradient(238deg, #ff0000, #ff8000, #ffff00, #80ff00, #00ff00, #00ff80, #00ffff, #0080ff, #0000ff, #8000ff, #ff0080);
background-size: 1200% 1200%;
background-position: 2% 80%;
color: #fff;
text-shadow: 0 0 2px rgba(0,0,0,.9);
animation: rainbow 4s ease-out alternate infinite;
}
.rainbow-active-noanim {
animation: none !important;
}
.inlinelist .inlinelist-item:hover a,
.inlinelist .inlinelist-item:focus a,
.buzzword-list li:hover a,
.buzzword-list li:focus a,
a.buzzword:hover,
a.buzzword:focus,
a.rainbow-active:hover,
a.rainbow-active:focus {
color: #fff;
text-decoration: none;
}
@media (prefers-reduced-motion: reduce) {
.inlinelist .inlinelist-item:hover,
.inlinelist .inlinelist-item:focus,
.buzzword-list li:hover,
.buzzword-list li:focus,
.buzzword:hover,
.buzzword:focus,
.rainbow-active:hover,
.rainbow-active:focus {
animation: none;
}
}
.buzzword-list li:hover:after,
.buzzword-list li:focus:after,
.buzzword:hover:after,
.buzzword:focus:after {
font-family: system-ui, -apple-system, sans-serif;
content: "Buzzword alert!!!";
position: absolute;
left: 0;
top: 0;
max-width: 8em;
color: #f00;
font-weight: 700;
text-transform: uppercase;
transform: rotate(-10deg) translate(-25%, -125%);
text-shadow: 1px 1px 5px rgba(0,0,0,.6);
line-height: 1.2;
pointer-events: none;
}
main h2 .buzzword,
main h3 .buzzword,
main p .buzzword {
padding: 0px 7px;
font-size: 1em; /* 18px /18 */
margin: 0;
line-height: 1.444444444444; /* 26px /18 */
font-family: inherit;
}
main h2 a.buzzword,
main h3 a.buzzword,
main p a.buzzword {
text-decoration: underline;
}
/* Small viewport */
@media (max-width: 26.8125em) { /* 429px */
.inlinelist .inlinelist-item {
display: block;
width: auto;
padding: 0;
line-height: 1.4;
}
.inlinelist .inlinelist-item > a {
display: block;
padding: .2em .5em;
}
}
@media (min-width: 26.875em) { /* 430px */
.inlinelist .inlinelist-item > a {
display: inline-block;
white-space: nowrap;
}
}
.numberflag {
display: inline-flex;
align-items: center;
justify-content: center;
background-color: #dff7ff;
border-radius: 50%;
width: 1.75em;
height: 1.75em;
font-weight: 600;
}
@media (prefers-color-scheme: dark) {
.numberflag {
background-color: #00bcd4;
color: #222;
}
}
h1 .numberflag,
h2 .numberflag,
h3 .numberflag,
h4 .numberflag,
h5 .numberflag {
width: 1.25em;
height: 1.25em;
}
h2 .numberflag {
position: relative;
margin-right: 0.25em; /* 10px /40 */
}
h2 .numberflag:after {
content: " ";
position: absolute;
bottom: -1px;
left: 0;
height: 1px;
background-color: #fff;
width: calc(100% + 0.4em); /* 16px /40 */
}
@media (prefers-color-scheme: dark) {
h2 .numberflag:after {
background-color: #222;
}
}
/* Super featured list on home page */
.list-superfeatured .avatar {
width: calc(30px + 5vw);
height: calc(30px + 5vw);
max-width: 60px;
max-height: 60px;
margin-left: 0;
}
@media (max-width: 26.8125em) { /* 429px */
.list-superfeatured .inlinelist-item > a {
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
}
}
@media (min-width: 26.875em) { /* 430px */
.list-superfeatured .inlinelist-item {
font-size: 110%;
}
}
/* Only top level */
.inlinelist-no-nest ul,
.inlinelist-no-nest ol {
display: none;
}

View File

@ -0,0 +1,107 @@
/* Mini link */
.minilink {
display: inline-block;
padding: .125em .375em;
text-transform: uppercase;
font-size: 0.875rem; /* 14px /16 */
text-decoration: none;
background-color: #ddd;
border-radius: 0.1875em; /* 3px /16 */
font-weight: 500;
margin: 0 0.4285714285714em 0.07142857142857em 0; /* 0 6px 1px 0 /14 */
line-height: 1.285714285714; /* 18px /14 */
font-family: system-ui, -apple-system, sans-serif;
}
@media (prefers-color-scheme: dark) {
.minilink {
background-color: #222;
/*
!important to override .elv-callout a
see _includes/components/callout.css
*/
color: #fff !important;
}
}
table .minilink {
margin-top: 6px;
}
.minilink[href] {
box-shadow: 0 1px 1px 0 rgba(0,0,0,.5);
}
.minilink[href]:hover,
.minilink[href]:focus {
background-color: #bbb;
}
@media (prefers-color-scheme: dark) {
.minilink[href]:hover,
.minilink[href]:focus {
background-color: #444;
}
}
pre + .minilink {
color: #fff;
border-radius: 0 0 0.2857142857143em 0.2857142857143em; /* 4px /14 */
float: right;
background-color: #444;
color: #fff;
}
pre[class*=language-] + .minilink {
position: relative;
top: -0.7142857142857em; /* -10px /14 */
}
p.minilink {
float: right;
margin-left: 2em;
margin-bottom: 2em;
}
h1 .minilink,
h2 .minilink,
h3 .minilink,
h4 .minilink {
font-size: 0.9375rem; /* 15px /16 */
vertical-align: middle;
margin-left: 1em;
}
h3 .minilink,
h4 .minilink {
font-size: 0.8125rem; /* 13px /16 */
}
.minilink + pre[class*=language-] {
clear: both;
}
.minilink-addedin {
text-transform: none;
box-shadow: 0 0 0 1px rgba(0,0,0,0.3);
}
@media (prefers-color-scheme: dark) {
.minilink-addedin {
box-shadow: 0 0 0 1px rgba(255,255,255,0.3);
}
}
.minilink-addedin:not(:first-child) {
margin-left: .5em;
}
.minilink-addedin.minilink-inline {
margin: 0 4px;
background-color: #fff;
}
.minilink-lower {
text-transform: none;
background-color: transparent;
}
.minilink-lower[href] {
box-shadow: 0 0 0 1px rgba(0,0,0,0.5);
}
.minilink-lower[href]:hover,
.minilink-lower[href]:focus {
background-color: #eee;
}
.minilink > .minilink {
margin: -.125em .375em -.125em -.375em;
box-shadow: none;
border-top-right-radius: 0;
border-bottom-right-radius: 0;
}

View File

@ -0,0 +1,5 @@
<h2>Improve this documentation</h2>
<p>Have an idea on how to make this documentation even better? Use our <a
href="https://projects.torsion.org/borgmatic-collective/borgmatic/issues">issue tracker</a> to send your
feedback!</p>

View File

@ -0,0 +1,111 @@
.elv-toc {
font-size: 1rem; /* Reset */
}
.elv-toc details {
--details-force-closed: (max-width: 63.9375em); /* 1023px */
}
.elv-toc details > summary {
font-size: 1.375rem; /* 22px /16 */
margin-bottom: .5em;
}
@media (min-width: 64em) { /* 1024px */
.elv-toc {
position: absolute;
left: 3rem;
width: 16rem;
z-index: 1;
}
.elv-toc details > summary {
margin-top: 0;
}
.js .elv-toc details > summary {
display: none;
}
}
.elv-toc-list {
display: flex;
flex-wrap: wrap;
justify-content: space-between;
padding-left: 0;
padding-right: 0;
margin: 0 0 2.5em;
list-style: none;
}
.elv-toc-list li {
font-size: 0.9375em; /* 15px /16 */
line-height: 1.466666666667; /* 22px /15 */
}
/* Nested lists */
.elv-toc-list ul {
padding: 0 0 .75em 0;
margin: 0;
list-style: none;
}
/* Menus nested 2 or more deep */
.elv-toc-list ul ul {
padding-bottom: 0;
padding-left: 0.625rem; /* 10px /16 */
}
/* Hide inactive menus 3 or more deep */
.elv-toc-list ul ul > li:not(.elv-toc-active) > ul > li:not(.elv-toc-active) {
display: none;
}
/* List items */
.elv-toc summary,
.elv-toc-list a {
padding: .15em .25em;
}
.elv-toc-list a {
display: block;
}
.elv-toc-list a:not(:hover) {
text-decoration: none;
}
.elv-toc-list li {
margin: 0;
padding: 0;
}
.elv-toc-list > li {
flex-grow: 1;
flex-basis: 14.375rem; /* 230px /16 */
}
/* Top level links */
.elv-toc-list > li > a {
color: #222;
font-weight: 600;
border-bottom: 1px solid #ddd;
margin-bottom: 0.25em; /* 4px /16 */
}
@media (prefers-color-scheme: dark) {
.elv-toc-list > li > a {
color: #fff;
border-color: #444;
}
}
/* Active links */
.elv-toc-list li.elv-toc-active > a {
background-color: #dff7ff;
}
@media (prefers-color-scheme: dark) {
.elv-toc-list li.elv-toc-active > a {
background-color: #353535;
}
}
.elv-toc-list ul .elv-toc-active > a:after {
content: "";
}
/* Show only active nested lists */
.elv-toc-list ul.elv-toc-active,
.elv-toc-list li.elv-toc-active > ul {
display: block;
}
/* Footer catgory navigation */
.elv-cat-list-active {
font-weight: 600;
}

View File

@ -0,0 +1,4 @@
<header class="elv-layout elv-layout-full elv-header{% if headerClass %} {{ headerClass }}{% endif %}">
{% if page.url != '/' %}<h3><a href="https://torsion.org/borgmatic/">borgmatic</a></h3>{% endif %}
<h1 class="elv-hed">{{ title | safe }}</h1>
</header>

532
docs/_includes/index.css Normal file

File diff suppressed because one or more lines are too long

View File

@ -0,0 +1,27 @@
<!doctype html>
<html lang="en"{% if templateClass %} class="{{ templateClass }}"{% endif %}>
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>{{ subtitle + ' - ' if subtitle}}{{ title }}</title>
{%- set css %}
{% include 'index.css' %}
{% include 'components/lists.css' %}
{% include 'components/external-links.css' %}
{% include 'components/minilink.css' %}
{% include 'components/toc.css' %}
{% include 'components/info-blocks.css' %}
{% include 'prism-theme.css' %}
{% include 'asciinema.css' %}
{% endset %}
<style>{{ css | safe }}</style>
{% if feedTitle and feedUrl %}
<link rel="alternate" href="{{ feedUrl }}" title="{{ feedTitle }}" type="application/atom+xml">
{% endif %}
</head>
<body>
{{ content | safe }}
</body>
</html>

View File

@ -0,0 +1,32 @@
---
layout: layouts/base.njk
templateClass: elv-default
headerClass: elv-header-default
---
{% include "header.njk" %}
<main class="elv-layout{% if layoutClass %} {{ layoutClass }}{% endif %}">
<div id="documentation" class="elv-toc">
<div>
{% set navPages = collections.all | eleventyNavigation %}
{% macro renderNavListItem(entry) -%}
<li{% if entry.url == page.url %} class="elv-toc-active"{% endif %}>
<a {% if entry.url %}href="https://torsion.org/borgmatic/docs{{ entry.url | url }}"{% endif %}>{{ entry.title }}</a>
{%- if entry.children.length -%}
<ul>
{%- for child in entry.children %}{{ renderNavListItem(child) }}{% endfor -%}
</ul>
{%- endif -%}
</li>
{%- endmacro %}
<ul class="elv-toc-list">
{%- for entry in navPages %}{{ renderNavListItem(entry) }}{%- endfor -%}
</ul>
</div>
</div>
{{ content | safe }}
{% include 'components/suggestion-link.html' %}
</main>

View File

@ -0,0 +1,177 @@
/**
* prism.js default theme for JavaScript, CSS and HTML
* Based on dabblet (http://dabblet.com)
* @author Lea Verou
*/
/*
* Modified with an approximation of the One Light syntax highlighting theme.
*/
code[class*="language-"],
pre[class*="language-"] {
color: #494b53;
background: none;
font-family: Consolas, Monaco, 'Andale Mono', 'Ubuntu Mono', monospace;
text-align: left;
white-space: pre;
word-spacing: normal;
word-break: normal;
word-wrap: normal;
line-height: 1.5;
-moz-tab-size: 4;
-o-tab-size: 4;
tab-size: 4;
-webkit-hyphens: none;
-moz-hyphens: none;
-ms-hyphens: none;
hyphens: none;
}
pre[class*="language-"]::-moz-selection, pre[class*="language-"] ::-moz-selection,
code[class*="language-"]::-moz-selection, code[class*="language-"] ::-moz-selection {
text-shadow: none;
color: #232324;
background: #dbdbdc;
}
pre[class*="language-"]::selection, pre[class*="language-"] ::selection,
code[class*="language-"]::selection, code[class*="language-"] ::selection {
text-shadow: none;
color: #232324;
background: #dbdbdc;
}
@media print {
code[class*="language-"],
pre[class*="language-"] {
text-shadow: none;
}
}
/* Code blocks */
pre[class*="language-"] {
padding: 1em;
margin: .5em 0;
overflow: auto;
}
:not(pre) > code[class*="language-"],
pre[class*="language-"] {
background: #fafafa;
}
/* Inline code */
:not(pre) > code[class*="language-"] {
padding: .1em;
border-radius: .3em;
white-space: normal;
}
.token.comment,
.token.prolog,
.token.doctype,
.token.cdata {
color: #505157;
}
.token.punctuation {
color: #526fff;
}
.token.selector,
.token.tag {
color: none;
}
.token.property,
.token.boolean,
.token.number,
.token.constant,
.token.symbol,
.token.attr-name,
.token.deleted {
color: #986801;
}
.token.string,
.token.char,
.token.attr-value,
.token.builtin,
.token.inserted {
color: #50a14f;
}
.token.operator,
.token.entity,
.token.url,
.language-css .token.string,
.style .token.string {
color: #526fff;
}
.token.atrule,
.token.keyword {
color: #e45649;
}
.token.function {
color: #4078f2;
}
.token.regex,
.token.important,
.token.variable {
color: #e45649;
}
.token.important,
.token.bold {
font-weight: bold;
}
.token.italic {
font-style: italic;
}
.token.entity {
cursor: help;
}
pre.line-numbers {
position: relative;
padding-left: 3.8em;
counter-reset: linenumber;
}
pre.line-numbers > code {
position: relative;
}
.line-numbers .line-numbers-rows {
position: absolute;
pointer-events: none;
top: 0;
font-size: 100%;
left: -3.8em;
width: 3em; /* works for line-numbers below 1000 lines */
letter-spacing: -1px;
border-right: 0;
-webkit-user-select: none;
-moz-user-select: none;
-ms-user-select: none;
user-select: none;
}
.line-numbers-rows > span {
pointer-events: none;
display: block;
counter-increment: linenumber;
}
.line-numbers-rows > span:before {
content: counter(linenumber);
color: #5C6370;
display: block;
padding-right: 0.8em;
text-align: right;
}

View File

@ -0,0 +1,107 @@
---
title: How to add preparation and cleanup steps to backups
eleventyNavigation:
key: 🧹 Add preparation and cleanup steps
parent: How-to guides
order: 8
---
## Preparation and cleanup hooks
If you find yourself performing preparation tasks before your backup runs, or
cleanup work afterwards, borgmatic hooks may be of interest. Hooks are shell
commands that borgmatic executes for you at various points as it runs, and
they're configured in the `hooks` section of your configuration file. But if
you're looking to backup a database, it's probably easier to use the [database
backup
feature](https://torsion.org/borgmatic/docs/how-to/backup-your-databases/)
instead.
You can specify `before_backup` hooks to perform preparation steps before
running backups, and specify `after_backup` hooks to perform cleanup steps
afterwards. Here's an example:
```yaml
hooks:
before_backup:
- mount /some/filesystem
after_backup:
- umount /some/filesystem
```
The `before_backup` and `after_backup` hooks each run once per repository in a
configuration file. `before_backup` hooks runs right before the `create`
action for a particular repository, and `after_backup` hooks run afterwards,
but not if an error occurs in a previous hook or in the backups themselves.
(Prior to borgmatic 1.6.0, these hooks instead ran once per configuration file
rather than once per repository.)
There are additional hooks that run before/after other actions as well. For
instance, `before_prune` runs before a `prune` action for a repository, while
`after_prune` runs after it.
## Variable interpolation
The before and after action hooks support interpolating particular runtime
variables into the hook command. Here's an example that assumes you provide a
separate shell script:
```yaml
hooks:
after_prune:
- record-prune.sh "{configuration_filename}" "{repository}"
```
In this example, when the hook is triggered, borgmatic interpolates runtime
values into the hook command: the borgmatic configuration filename and the
paths of the current Borg repository. Here's the full set of supported
variables you can use here:
* `configuration_filename`: borgmatic configuration filename in which the
hook was defined
* `repository`: path of the current repository as configured in the current
borgmatic configuration file
## Global hooks
You can also use `before_everything` and `after_everything` hooks to perform
global setup or cleanup:
```yaml
hooks:
before_everything:
- set-up-stuff-globally
after_everything:
- clean-up-stuff-globally
```
`before_everything` hooks collected from all borgmatic configuration files run
once before all configuration files (prior to all actions), but only if there
is a `create` action. An error encountered during a `before_everything` hook
causes borgmatic to exit without creating backups.
`after_everything` hooks run once after all configuration files and actions,
but only if there is a `create` action. It runs even if an error occurs during
a backup or a backup hook, but not if an error occurs during a
`before_everything` hook.
## Error hooks
borgmatic also runs `on_error` hooks if an error occurs, either when creating
a backup or running a backup hook. See the [monitoring and alerting
documentation](https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/)
for more information.
## Hook output
Any output produced by your hooks shows up both at the console and in syslog
(when run in a non-interactive console). For more information, read about <a
href="https://torsion.org/borgmatic/docs/how-to/inspect-your-backups/">inspecting
your backups</a>.
## Security
An important security note about hooks: borgmatic executes all hook commands
with the user permissions of borgmatic itself. So to prevent potential shell
injection or privilege escalation, do not forget to set secure permissions
on borgmatic configuration files (`chmod 0600`) and scripts (`chmod 0700`)
invoked by hooks.

View File

@ -0,0 +1,120 @@
---
title: How to backup to a removable drive or an intermittent server
eleventyNavigation:
key: 💾 Backup to a removable drive/server
parent: How-to guides
order: 9
---
## Occasional backups
A common situation is backing up to a repository that's only sometimes online.
For instance, you might send most of your backups to the cloud, but
occasionally you want to plug in an external hard drive or backup to your
buddy's sometimes-online server for that extra level of redundancy.
But if you run borgmatic and your hard drive isn't plugged in, or your buddy's
server is offline, then you'll get an annoying error message and the overall
borgmatic run will fail (even if individual repositories still complete).
Another variant is when the source machine is only sometimes available for
backups, e.g. a laptop where you want to skip backups when the battery falls
below a certain level.
So what if you want borgmatic to swallow the error of a missing drive
or an offline server or a low battery—and exit gracefully? That's where the
concept of "soft failure" come in.
## Soft failure command hooks
This feature leverages [borgmatic command
hooks](https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/),
so first familiarize yourself with them. The idea is that you write a simple
test in the form of a borgmatic hook to see if backups should proceed or not.
The way the test works is that if any of your hook commands return a special
exit status of 75, that indicates to borgmatic that it's a temporary failure,
and borgmatic should skip all subsequent actions for that configuration file.
If you return any other status, then it's a standard success or error. (Zero is
success; anything else other than 75 is an error).
So for instance, if you have an external drive that's only sometimes mounted,
declare its repository in its own [separate configuration
file](https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/),
say at `/etc/borgmatic.d/removable.yaml`:
```yaml
location:
source_directories:
- /home
repositories:
- /mnt/removable/backup.borg
```
Then, write a `before_backup` hook in that same configuration file that uses
the external `findmnt` utility to see whether the drive is mounted before
proceeding.
```yaml
hooks:
before_backup:
- findmnt /mnt/removable > /dev/null || exit 75
```
What this does is check if the `findmnt` command errors when probing for a
particular mount point. If it does error, then it returns exit code 75 to
borgmatic. borgmatic logs the soft failure, skips all further actions in that
configurable file, and proceeds onward to any other borgmatic configuration
files you may have.
You can imagine a similar check for the sometimes-online server case:
```yaml
location:
source_directories:
- /home
repositories:
- me@buddys-server.org:backup.borg
hooks:
before_backup:
- ping -q -c 1 buddys-server.org > /dev/null || exit 75
```
Or to only run backups if the battery level is high enough:
```yaml
hooks:
before_backup:
- is_battery_percent_at_least.sh 25
```
(Writing the battery script is left as an exercise to the reader.)
## Caveats and details
There are some caveats you should be aware of with this feature.
* You'll generally want to put a soft failure command in the `before_backup`
hook, so as to gate whether the backup action occurs. While a soft failure is
also supported in the `after_backup` hook, returning a soft failure there
won't prevent any actions from occuring, because they've already occurred!
Similiarly, you can return a soft failure from an `on_error` hook, but at
that point it's too late to prevent the error.
* Returning a soft failure does prevent further commands in the same hook from
executing. So, like a standard error, it is an "early out". Unlike a standard
error, borgmatic does not display it in angry red text or consider it a
failure.
* The soft failure only applies to the scope of a single borgmatic
configuration file. So put anything that you don't want soft-failed, like
always-online cloud backups, in separate configuration files from your
soft-failing repositories.
* The soft failure doesn't have to apply to a repository. You can even perform
a test to make sure that individual source directories are mounted and
available. Use your imagination!
* The soft failure feature also works for before/after hooks for other
actions as well. But it is not implemented for `before_everything` or
`after_everything`.

View File

@ -0,0 +1,261 @@
---
title: How to backup your databases
eleventyNavigation:
key: 🗄️ Backup your databases
parent: How-to guides
order: 7
---
## Database dump hooks
If you want to backup a database, it's best practice with most database
systems to backup an exported database dump, rather than backing up your
database's internal file storage. That's because the internal storage can
change while you're reading from it. In contrast, a database dump creates a
consistent snapshot that is more suited for backups.
Fortunately, borgmatic includes built-in support for creating database dumps
prior to running backups. For example, here is everything you need to dump and
backup a couple of local PostgreSQL databases, a MySQL/MariaDB database, and a
MongoDB database:
```yaml
hooks:
postgresql_databases:
- name: users
- name: orders
mysql_databases:
- name: posts
mongodb_databases:
- name: messages
```
As part of each backup, borgmatic streams a database dump for each configured
database directly to Borg, so it's included in the backup without consuming
additional disk space. (The exceptions are the PostgreSQL/MongoDB "directory"
dump formats, which can't stream and therefore do consume temporary disk
space. Additionally, prior to borgmatic 1.5.3, all database dumps consumed
temporary disk space.)
To support this, borgmatic creates temporary named pipes in `~/.borgmatic` by
default. To customize this path, set the `borgmatic_source_directory` option
in the `location` section of borgmatic's configuration.
Also note that using a database hook implicitly enables both the
`read_special` and `one_file_system` configuration settings (even if they're
disabled in your configuration) to support this dump and restore streaming.
See Limitations below for more on this.
Here's a more involved example that connects to remote databases:
```yaml
hooks:
postgresql_databases:
- name: users
hostname: database1.example.org
port: 5433
username: postgres
password: trustsome1
format: tar
options: "--role=someone"
mysql_databases:
- name: posts
hostname: database2.example.org
port: 3307
username: root
password: trustsome1
options: "--skip-comments"
mongodb_databases:
- name: messages
hostname: database3.example.org
port: 27018
username: dbuser
password: trustsome1
authentication_database: mongousers
options: "--ssl"
```
If you want to dump all databases on a host, use `all` for the database name:
```yaml
hooks:
postgresql_databases:
- name: all
mysql_databases:
- name: all
mongodb_databases:
- name: all
```
Note that you may need to use a `username` of the `postgres` superuser for
this to work with PostgreSQL.
If you would like to backup databases only and not source directories, you can
specify an empty `source_directories` value (as it is a mandatory field):
```yaml
location:
source_directories: []
hooks:
mysql_databases:
- name: all
```
### Configuration backups
An important note about this database configuration: You'll need the
configuration to be present in order for borgmatic to restore a database. So
to prepare for this situation, it's a good idea to include borgmatic's own
configuration files as part of your regular backups. That way, you can always
bring back any missing configuration files in order to restore a database.
## Supported databases
As of now, borgmatic supports PostgreSQL, MySQL/MariaDB, and MongoDB databases
directly. But see below about general-purpose preparation and cleanup hooks as
a work-around with other database systems. Also, please [file a
ticket](https://torsion.org/borgmatic/#issues) for additional database systems
that you'd like supported.
## Database restoration
To restore a database dump from an archive, use the `borgmatic restore`
action. But the first step is to figure out which archive to restore from. A
good way to do that is to use the `list` action:
```bash
borgmatic list
```
(No borgmatic `list` action? Try the old-style `--list`, or upgrade
borgmatic!)
That should yield output looking something like:
```text
host-2019-01-01T04:05:06.070809 Tue, 2019-01-01 04:05:06 [...]
host-2019-01-02T04:06:07.080910 Wed, 2019-01-02 04:06:07 [...]
```
Assuming that you want to restore all database dumps from the archive with the
most up-to-date files and therefore the latest timestamp, run a command like:
```bash
borgmatic restore --archive host-2019-01-02T04:06:07.080910
```
(No borgmatic `restore` action? Upgrade borgmatic!)
With newer versions of borgmatic, you can simplify this to:
```bash
borgmatic restore --archive latest
```
The `--archive` value is the name of the archive to restore from. This
restores all databases dumps that borgmatic originally backed up to that
archive.
This is a destructive action! `borgmatic restore` replaces live databases by
restoring dumps from the selected archive. So be very careful when and where
you run it.
### Repository selection
If you have a single repository in your borgmatic configuration file(s), no
problem: the `restore` action figures out which repository to use.
But if you have multiple repositories configured, then you'll need to specify
the repository path containing the archive to restore. Here's an example:
```bash
borgmatic restore --repository repo.borg --archive host-2019-...
```
### Restore particular databases
If you've backed up multiple databases into an archive, and you'd only like to
restore one of them, use the `--database` flag to select one or more
databases. For instance:
```bash
borgmatic restore --archive host-2019-... --database users
```
### Limitations
There are a few important limitations with borgmatic's current database
restoration feature that you should know about:
1. You must restore as the same Unix user that created the archive containing
the database dump. That's because the user's home directory path is encoded
into the path of the database dump within the archive.
2. As mentioned above, borgmatic can only restore a database that's defined in
borgmatic's own configuration file. So include your configuration file in
backups to avoid getting caught without a way to restore a database.
3. borgmatic does not currently support backing up or restoring multiple
databases that share the exact same name on different hosts.
4. Because database hooks implicitly enable the `read_special` configuration
setting to support dump and restore streaming, you'll need to ensure that any
special files are excluded from backups (named pipes, block devices,
character devices, and sockets) to prevent hanging. Try a command like
`find /your/source/path -type c,b,p,s` to find such files. Common directories
to exclude are `/dev` and `/run`, but that may not be exhaustive.
### Manual restoration
If you prefer to restore a database without the help of borgmatic, first
[extract](https://torsion.org/borgmatic/docs/how-to/extract-a-backup/) an
archive containing a database dump.
borgmatic extracts the dump file into the *`username`*`/.borgmatic/` directory
within the extraction destination path, where *`username`* is the user that
created the backup. For example, if you created the backup with the `root`
user and you're extracting to `/tmp`, then the dump will be in
`/tmp/root/.borgmatic`.
After extraction, you can manually restore the dump file using native database
commands like `pg_restore`, `mysql`, `mongorestore` or similar.
## Preparation and cleanup hooks
If this database integration is too limited for needs, borgmatic also supports
general-purpose [preparation and cleanup
hooks](https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/).
These hooks allows you to trigger arbitrary commands or scripts before and
after backups. So if necessary, you can use these hooks to create database
dumps with any database system.
## Troubleshooting
### MySQL table lock errors
If you encounter table lock errors during a database dump with MySQL/MariaDB,
you may need to [use a
transaction](https://dev.mysql.com/doc/refman/8.0/en/mysqldump.html#option_mysqldump_single-transaction).
You can add any additional flags to the `options:` in your database
configuration. Here's an example:
```yaml
hooks:
mysql_databases:
- name: posts
options: "--single-transaction --quick"
```
### borgmatic hangs during backup
See Limitations above about `read_special`. You may need to exclude certain
paths with named pipes, block devices, character devices, or sockets on which
borgmatic is hanging.
Alternatively, if excluding special files is too onerous, you can create two
separate borgmatic configuration files—one for your source files and a
separate one for backing up databases. That way, the database `read_special`
option will not be active when backing up special files.

View File

@ -0,0 +1,160 @@
---
title: How to deal with very large backups
eleventyNavigation:
key: 📏 Deal with very large backups
parent: How-to guides
order: 3
---
## Biggish data
Borg itself is great for efficiently de-duplicating data across successive
backup archives, even when dealing with very large repositories. But you may
find that while borgmatic's default mode of `prune`, `compact`, `create`, and
`check` works well on small repositories, it's not so great on larger ones.
That's because running the default pruning, compact, and consistency checks
take a long time on large repositories.
### A la carte actions
If you find yourself in this situation, you have some options. First, you can
run borgmatic's `prune`, `compact`, `create`, or `check` actions separately.
For instance, the following optional actions are available:
```bash
borgmatic prune
borgmatic compact
borgmatic create
borgmatic check
```
(No borgmatic `prune`, `create`, or `check` actions? Try the old-style
`--prune`, `--create`, or `--check`. Or upgrade borgmatic!)
You can run with only one of these actions provided, or you can mix and match
any number of them in a single borgmatic run. This supports approaches like
skipping certain actions while running others. For instance, this skips
`prune` and `compact` and only runs `create` and `check`:
```bash
borgmatic create check
```
Or, you can make backups with `create` on a frequent schedule (e.g. with
`borgmatic create` called from one cron job), while only running expensive
consistency checks with `check` on a much less frequent basis (e.g. with
`borgmatic check` called from a separate cron job).
### Consistency check configuration
Another option is to customize your consistency checks. The default
consistency checks run both full-repository checks and per-archive checks
within each repository no more than once a month.
But if you find that archive checks are too slow, for example, you can
configure borgmatic to run repository checks only. Configure this in the
`consistency` section of borgmatic configuration:
```yaml
consistency:
checks:
- name: repository
```
(Prior to borgmatic 1.6.2, `checks` was a plain list of strings without the `name:` part.)
Here are the available checks from fastest to slowest:
* `repository`: Checks the consistency of the repository itself.
* `archives`: Checks all of the archives in the repository.
* `extract`: Performs an extraction dry-run of the most recent archive.
* `data`: Verifies the data integrity of all archives contents, decrypting and decompressing all data (implies `archives` as well).
See [Borg's check documentation](https://borgbackup.readthedocs.io/en/stable/usage/check.html) for more information.
### Check frequency
As of borgmatic 1.6.2, you can optionally configure checks to run on a
periodic basis rather than every time borgmatic runs checks. For instance:
```yaml
consistency:
checks:
- name: repository
frequency: 2 weeks
```
This tells borgmatic to run this consistency check at most once every two
weeks for a given repository. The `frequency` value is a number followed by a
unit of time, e.g. "3 days", "1 week", "2 months", etc. The `frequency`
defaults to "always", which means run this check every time checks run.
Unlike a real scheduler like cron, borgmatic only makes a best effort to run
checks on the configured frequency. It compares that frequency with how long
it's been since the last check for a given repository (as recorded in a file
within `~/.borgmatic/checks`). If it hasn't been long enough, the check is
skipped. And you still have to run `borgmatic check` (or just `borgmatic`) in
order for checks to run, even when a `frequency` is configured!
If you want to temporarily ignore your configured frequencies, you can invoke
`borgmatic check --force` to run checks unconditionally.
### Disabling checks
If that's still too slow, you can disable consistency checks entirely,
either for a single repository or for all repositories.
Disabling all consistency checks looks like this:
```yaml
consistency:
checks:
- name: disabled
```
Or, if you have multiple repositories in your borgmatic configuration file,
you can keep running consistency checks, but only against a subset of the
repositories:
```yaml
consistency:
check_repositories:
- path/of/repository_to_check.borg
```
Finally, you can override your configuration file's consistency checks, and
run particular checks via the command-line. For instance:
```bash
borgmatic check --only data --only extract
```
This is useful for running slow consistency checks on an infrequent basis,
separate from your regular checks. It is still subject to any configured
check frequencies unless the `--force` flag is used.
## Troubleshooting
### Broken pipe with remote repository
When running borgmatic on a large remote repository, you may receive errors
like the following, particularly while "borg check" is validating backups for
consistency:
```text
Write failed: Broken pipe
borg: Error: Connection closed by remote host
```
This error can be caused by an ssh timeout, which you can rectify by adding
the following to the `~/.ssh/config` file on the client:
```text
Host *
ServerAliveInterval 120
```
This should make the client keep the connection alive while validating
backups.

View File

@ -0,0 +1,143 @@
---
title: How to develop on borgmatic
eleventyNavigation:
key: 🏗️ Develop on borgmatic
parent: How-to guides
order: 12
---
## Source code
To get set up to hack on borgmatic, first clone master via HTTPS or SSH:
```bash
git clone https://projects.torsion.org/borgmatic-collective/borgmatic.git
```
Or:
```bash
git clone ssh://git@projects.torsion.org:3022/borgmatic-collective/borgmatic.git
```
Then, install borgmatic
"[editable](https://pip.pypa.io/en/stable/cli/pip_install/#editable-installs)"
so that you can run borgmatic commands while you're hacking on them to
make sure your changes work.
```bash
cd borgmatic/
pip3 install --editable --user .
```
Note that this will typically install the borgmatic commands into
`~/.local/bin`, which may or may not be on your PATH. There are other ways to
install borgmatic editable as well, for instance into the system Python
install (so without `--user`, as root), or even into a
[virtualenv](https://virtualenv.pypa.io/en/stable/). How or where you install
borgmatic is up to you, but generally an editable install makes development
and testing easier.
## Automated tests
Assuming you've cloned the borgmatic source code as described above, and
you're in the `borgmatic/` working copy, install tox, which is used for
setting up testing environments:
```bash
pip3 install --user tox
```
Finally, to actually run tests, run:
```bash
cd borgmatic
tox
```
### Code formatting
If when running tests, you get an error from the
[Black](https://black.readthedocs.io/en/stable/) code formatter about files
that would be reformatted, you can ask Black to format them for you via the
following:
```bash
tox -e black
```
And if you get a complaint from the
[isort](https://github.com/timothycrosley/isort) Python import orderer, you
can ask isort to order your imports for you:
```bash
tox -e isort
```
### End-to-end tests
borgmatic additionally includes some end-to-end tests that integration test
with Borg and supported databases for a few representative scenarios. These
tests don't run by default when running `tox`, because they're relatively slow
and depend on Docker containers for runtime dependencies. These tests tests do
run on the continuous integration (CI) server, and running them on your
developer machine is the closest thing to CI test parity.
If you would like to run the full test suite, first install Docker and [Docker
Compose](https://docs.docker.com/compose/install/). Then run:
```bash
scripts/run-full-dev-tests
```
Note that this scripts assumes you have permission to run Docker. If you
don't, then you may need to run with `sudo`.
## Code style
Start with [PEP 8](https://www.python.org/dev/peps/pep-0008/). But then, apply
the following deviations from it:
* For strings, prefer single quotes over double quotes.
* Limit all lines to a maximum of 100 characters.
* Use trailing commas within multiline values or argument lists.
* For multiline constructs, put opening and closing delimeters on lines
separate from their contents.
* Within multiline constructs, use standard four-space indentation. Don't align
indentation with an opening delimeter.
borgmatic code uses the [Black](https://black.readthedocs.io/en/stable/) code
formatter, the [Flake8](http://flake8.pycqa.org/en/latest/) code checker, and
the [isort](https://github.com/timothycrosley/isort) import orderer, so
certain code style requirements will be enforced when running automated tests.
See the Black, Flake8, and isort documentation for more information.
## Continuous integration
Each pull request triggers a continuous integration build which runs the test
suite. You can view these builds on
[build.torsion.org](https://build.torsion.org/borgmatic-collective/borgmatic), and they're
also linked from the commits list on each pull request.
## Documentation development
Updates to borgmatic's documentation are welcome. It's formatted in Markdown
and located in the `docs/` directory in borgmatic's source, plus the
`README.md` file at the root.
To build and view a copy of the documentation with your local changes, run the
following from the root of borgmatic's source code:
```bash
sudo scripts/dev-docs
```
This requires Docker to be installed on your system. You may not need to use
sudo if your non-root user has permissions to run Docker.
After you run the script, you can point your web browser at
http://localhost:8080 to view the documentation with your changes.
To close the documentation server, ctrl-C the script. Note that it does not
currently auto-reload, so you'll need to stop it and re-run it for any
additional documentation changes to take effect.

View File

@ -0,0 +1,138 @@
---
title: How to extract a backup
eleventyNavigation:
key: 📤 Extract a backup
parent: How-to guides
order: 6
---
## Extract
When the worst happens—or you want to test your backups—the first step is
to figure out which archive to extract. A good way to do that is to use the
`list` action:
```bash
borgmatic list
```
(No borgmatic `list` action? Try the old-style `--list`, or upgrade
borgmatic!)
That should yield output looking something like:
```text
host-2019-01-01T04:05:06.070809 Tue, 2019-01-01 04:05:06 [...]
host-2019-01-02T04:06:07.080910 Wed, 2019-01-02 04:06:07 [...]
```
Assuming that you want to extract the archive with the most up-to-date files
and therefore the latest timestamp, run a command like:
```bash
borgmatic extract --archive host-2019-01-02T04:06:07.080910
```
(No borgmatic `extract` action? Try the old-style `--extract`, or upgrade
borgmatic!)
With newer versions of borgmatic, you can simplify this to:
```bash
borgmatic extract --archive latest
```
The `--archive` value is the name of the archive to extract. This extracts the
entire contents of the archive to the current directory, so make sure you're
in the right place before running the command.
## Repository selection
If you have a single repository in your borgmatic configuration file(s), no
problem: the `extract` action figures out which repository to use.
But if you have multiple repositories configured, then you'll need to specify
the repository path containing the archive to extract. Here's an example:
```bash
borgmatic extract --repository repo.borg --archive host-2019-...
```
## Extract particular files
Sometimes, you want to extract a single deleted file, rather than extracting
everything from an archive. To do that, tack on one or more `--path` values.
For instance:
```bash
borgmatic extract --archive host-2019-... --path path/1 path/2
```
Note that the specified restore paths should not have a leading slash. Like a
whole-archive extract, this also extracts into the current directory. So for
example, if you happen to be in the directory `/var` and you run the `extract`
command above, borgmatic will extract `/var/path/1` and `/var/path/2`.
## Extract to a particular destination
By default, borgmatic extracts files into the current directory. To instead
extract files to a particular destination directory, use the `--destination`
flag:
```bash
borgmatic extract --archive host-2019-... --destination /tmp
```
When using the `--destination` flag, be careful not to overwrite your system's
files with extracted files unless that is your intent.
## Database restoration
The `borgmatic extract` command only extracts files. To restore a database,
please see the [documentation on database backups and
restores](https://torsion.org/borgmatic/docs/how-to/backup-your-databases/).
borgmatic does not perform database restoration as part of `borgmatic extract`
so that you can extract files from your archive without impacting your live
databases.
## Mount a filesystem
If instead of extracting files, you'd like to explore the files from an
archive as a [FUSE](https://en.wikipedia.org/wiki/Filesystem_in_Userspace)
filesystem, you can use the `borgmatic mount` action. Here's an example:
```bash
borgmatic mount --archive host-2019-... --mount-point /mnt
```
This mounts the entire archive on the given mount point `/mnt`, so that you
can look in there for your files.
Omit the `--archive` flag to mount all archives (lazy-loaded):
```bash
borgmatic mount --mount-point /mnt
```
Or use the "latest" value for the archive to mount the latest archive:
```bash
borgmatic mount --archive latest --mount-point /mnt
```
If you'd like to restrict the mounted filesystem to only particular paths from
your archive, use the `--path` flag, similar to the `extract` action above.
For instance:
```bash
borgmatic mount --archive host-2019-... --mount-point /mnt --path var/lib
```
When you're all done exploring your files, unmount your mount point. No
`--archive` flag is needed:
```bash
borgmatic umount --mount-point /mnt
```

5
docs/how-to/index.md Normal file
View File

@ -0,0 +1,5 @@
---
eleventyNavigation:
key: How-to guides
permalink: false
---

View File

@ -0,0 +1,129 @@
---
title: How to inspect your backups
eleventyNavigation:
key: 🔎 Inspect your backups
parent: How-to guides
order: 4
---
## Backup progress
By default, borgmatic runs proceed silently except in the case of errors. But
if you'd like to to get additional information about the progress of the
backup as it proceeds, use the verbosity option:
```bash
borgmatic --verbosity 1
```
This lists the files that borgmatic is archiving, which are those that are new
or changed since the last backup.
Or, for even more progress and debug spew:
```bash
borgmatic --verbosity 2
```
## Backup summary
If you're less concerned with progress during a backup, and you only want to
see the summary of archive statistics at the end, you can use the stats
option when performing a backup:
```bash
borgmatic --stats
```
## Existing backups
borgmatic provides convenient actions for Borg's
[list](https://borgbackup.readthedocs.io/en/stable/usage/list.html) and
[info](https://borgbackup.readthedocs.io/en/stable/usage/info.html)
functionality:
```bash
borgmatic list
borgmatic info
```
(No borgmatic `list` or `info` actions? Try the old-style `--list` or
`--info`. Or upgrade borgmatic!)
### Searching for a file
Let's say you've accidentally deleted a file and want to find the backup
archive(s) containing it. `borgmatic list` provides a `--find` flag for
exactly this purpose (as of borgmatic 1.6.3). For instance, if you're looking
for a `foo.txt`:
```bash
borgmatic list --find foo.txt
```
This will list your archives and indicate those with files matching
`*foo.txt*` anywhere in the archive. The `--find` parameter can alternatively
be a [Borg
pattern](https://borgbackup.readthedocs.io/en/stable/usage/help.html#borg-patterns).
To limit the archives searched, use the standard `list` parameters for
filtering archives such as `--last`, `--archive`, `--glob-archives`, etc. For
example, to search only the last five archives:
```bash
borgmatic list --find foo.txt --last 5
```
## Logging
By default, borgmatic logs to a local syslog-compatible daemon if one is
present and borgmatic is running in a non-interactive console. Where those
logs show up depends on your particular system. If you're using systemd, try
running `journalctl -xe`. Otherwise, try viewing `/var/log/syslog` or
similiar.
You can customize the log level used for syslog logging with the
`--syslog-verbosity` flag, and this is independent from the console logging
`--verbosity` flag described above. For instance, to get additional
information about the progress of the backup as it proceeds:
```bash
borgmatic --syslog-verbosity 1
```
Or to increase syslog logging to include debug spew:
```bash
borgmatic --syslog-verbosity 2
```
### Rate limiting
If you are using rsyslog or systemd's journal, be aware that by default they
both throttle the rate at which logging occurs. So you may need to change
either [the global rate
limit](https://www.rootusers.com/how-to-change-log-rate-limiting-in-linux/) or
[the per-service rate
limit](https://www.freedesktop.org/software/systemd/man/journald.conf.html#RateLimitIntervalSec=)
if you're finding that borgmatic logs are missing.
Note that the [sample borgmatic systemd service
file](https://torsion.org/borgmatic/docs/how-to/set-up-backups/#systemd)
already has this rate limit disabled for systemd's journal.
### Logging to file
If you don't want to use syslog, and you'd rather borgmatic log to a plain
file, use the `--log-file` flag:
```bash
borgmatic --log-file /path/to/file.log
```
Note that if you use the `--log-file` flag, you are responsible for rotating
the log file so it doesn't grow too large, for example with
[logrotate](https://wiki.archlinux.org/index.php/Logrotate). Also, there is a
`--log-file-verbosity` flag to customize the log file's log level.

View File

@ -0,0 +1,44 @@
---
title: How to make backups redundant
eleventyNavigation:
key: ☁️ Make backups redundant
parent: How-to guides
order: 2
---
## Multiple repositories
If you really care about your data, you probably want more than one backup of
it. borgmatic supports this in its configuration by specifying multiple backup
repositories. Here's an example:
```yaml
location:
# List of source directories to backup.
source_directories:
- /home
- /etc
# Paths of local or remote repositories to backup to.
repositories:
- 1234@usw-s001.rsync.net:backups.borg
- k8pDxu32@k8pDxu32.repo.borgbase.com:repo
- /var/lib/backups/local.borg
```
When you run borgmatic with this configuration, it invokes Borg once for each
configured repository in sequence. (So, not in parallel.) That means—in each
repository—borgmatic creates a single new backup archive containing all of
your source directories.
Here's a way of visualizing what borgmatic does with the above configuration:
1. Backup `/home` and `/etc` to `1234@usw-s001.rsync.net:backups.borg`
2. Backup `/home` and `/etc` to `k8pDxu32@k8pDxu32.repo.borgbase.com:repo`
3. Backup `/home` and `/etc` to `/var/lib/backups/local.borg`
This gives you redundancy of your data across repositories and even
potentially across providers.
See [Borg repository URLs
documentation](https://borgbackup.readthedocs.io/en/stable/usage/general.html#repository-urls)
for more information on how to specify local and remote repository paths.

View File

@ -0,0 +1,205 @@
---
title: How to make per-application backups
eleventyNavigation:
key: 🔀 Make per-application backups
parent: How-to guides
order: 1
---
## Multiple backup configurations
You may find yourself wanting to create different backup policies for
different applications on your system. For instance, you may want one backup
configuration for your database data directory, and a different configuration
for your user home directories.
The way to accomplish that is pretty simple: Create multiple separate
configuration files and place each one in a `/etc/borgmatic.d/` directory. For
instance:
```bash
sudo mkdir /etc/borgmatic.d
sudo generate-borgmatic-config --destination /etc/borgmatic.d/app1.yaml
sudo generate-borgmatic-config --destination /etc/borgmatic.d/app2.yaml
```
When you set up multiple configuration files like this, borgmatic will run
each one in turn from a single borgmatic invocation. This includes, by
default, the traditional `/etc/borgmatic/config.yaml` as well.
Each configuration file is interpreted independently, as if you ran borgmatic
for each configuration file one at a time. In other words, borgmatic does not
perform any merging of configuration files by default. If you'd like borgmatic
to merge your configuration files, see below about configuration includes.
Additionally, the `~/.config/borgmatic.d/` directory works the same way as
`/etc/borgmatic.d`.
If you need even more customizability, you can specify alternate configuration
paths on the command-line with borgmatic's `--config` flag. (See `borgmatic
--help` for more information.) For instance, if you want to schedule your
various borgmatic backups to run at different times, you'll need multiple
entries in your [scheduling software of
choice](https://torsion.org/borgmatic/docs/how-to/set-up-backups/#autopilot),
each entry using borgmatic's `--config` flag instead of relying on
`/etc/borgmatic.d`.
## Configuration includes
Once you have multiple different configuration files, you might want to share
common configuration options across these files with having to copy and paste
them. To achieve this, you can put fragments of common configuration options
into a file, and then include or inline that file into one or more borgmatic
configuration files.
Let's say that you want to include common retention configuration across all
of your configuration files. You could do that in each configuration file with
the following:
```yaml
location:
...
retention:
!include /etc/borgmatic/common_retention.yaml
```
And then the contents of `common_retention.yaml` could be:
```yaml
keep_hourly: 24
keep_daily: 7
```
To prevent borgmatic from trying to load these configuration fragments by
themselves and complaining that they are not valid configuration files, you
should put them in a directory other than `/etc/borgmatic.d/`. (A subdirectory
is fine.)
When a configuration include is a relative path, borgmatic loads it from either
the current working directory or from the directory containing the file doing
the including.
Note that this form of include must be a YAML value rather than a key. For
example, this will not work:
```yaml
location:
...
# Don't do this. It won't work!
!include /etc/borgmatic/common_retention.yaml
```
But if you do want to merge in a YAML key *and* its values, keep reading!
## Include merging
If you need to get even fancier and pull in common configuration options while
potentially overriding individual options, you can perform a YAML merge of
included configuration using the YAML `<<` key. For instance, here's an
example of a main configuration file that pulls in two retention options via
an include and then overrides one of them locally:
```yaml
<<: !include /etc/borgmatic/common.yaml
location:
...
retention:
keep_daily: 5
```
This is what `common.yaml` might look like:
```yaml
retention:
keep_hourly: 24
keep_daily: 7
```
Once this include gets merged in, the resulting configuration would have a
`keep_hourly` value of `24` and an overridden `keep_daily` value of `5`.
When there's an option collision between the local file and the merged
include, the local file's option takes precedence. And as of borgmatic 1.6.0,
this feature performs a deep merge, meaning that values are merged at all
levels in the two configuration files. Colliding list values are appended
together. This allows you to include common configuration—up to full borgmatic
configuration files—while overriding only the parts you want to customize.
Note that this `<<` include merging syntax is only for merging in mappings
(configuration options and their values). But if you'd like to include a
single value directly, please see the section above about standard includes.
Additionally, there is a limitation preventing multiple `<<` include merges
per section. So for instance, that means you can do one `<<` merge at the
global level, another `<<` within each configuration section, etc. (This is a
YAML limitation.)
## Configuration overrides
In more complex multi-application setups, you may want to override particular
borgmatic configuration file options at the time you run borgmatic. For
instance, you could reuse a common configuration file for multiple
applications, but then set the repository for each application at runtime. Or
you might want to try a variant of an option for testing purposes without
actually touching your configuration file.
Whatever the reason, you can override borgmatic configuration options at the
command-line via the `--override` flag. Here's an example:
```bash
borgmatic create --override location.remote_path=/usr/local/bin/borg1
```
What this does is load your configuration files, and for each one, disregard
the configured value for the `remote_path` option in the `location` section,
and use the value of `/usr/local/bin/borg1` instead.
You can even override multiple values at once. For instance:
```bash
borgmatic create --override section.option1=value1 section.option2=value2
```
This will accomplish the same thing:
```bash
borgmatic create --override section.option1=value1 --override section.option2=value2
```
Note that each value is parsed as an actual YAML string, so you can even set
list values by using brackets. For instance:
```bash
borgmatic create --override location.repositories=[test1.borg,test2.borg]
```
Or even a single list element:
```bash
borgmatic create --override location.repositories=[/root/test.borg]
```
If your override value contains special YAML characters like colons, then
you'll need quotes for it to parse correctly:
```bash
borgmatic create --override location.repositories="['user@server:test.borg']"
```
There is not currently a way to override a single element of a list without
replacing the whole list.
Note that if you override an option of the list type (like
`location.repositories`), you do need to use the `[ ]` list syntax. See the
[configuration
reference](https://torsion.org/borgmatic/docs/reference/configuration/) for
which options are list types. (YAML list values look like `- this` with an
indentation and a leading dash.)
Be sure to quote your overrides if they contain spaces or other characters
that your shell may interpret.

View File

@ -0,0 +1,298 @@
---
title: How to monitor your backups
eleventyNavigation:
key: 🚨 Monitor your backups
parent: How-to guides
order: 5
---
## Monitoring and alerting
Having backups is great, but they won't do you a lot of good unless you have
confidence that they're running on a regular basis. That's where monitoring
and alerting comes in.
There are several different ways you can monitor your backups and find out
whether they're succeeding. Which of these you choose to do is up to you and
your particular infrastructure.
### Job runner alerts
The easiest place to start is with failure alerts from the [scheduled job
runner](https://torsion.org/borgmatic/docs/how-to/set-up-backups/#autopilot)
(cron, systemd, etc.) that's running borgmatic. But note that if the job
doesn't even get scheduled (e.g. due to the job runner not running), you
probably won't get an alert at all! Still, this is a decent first line of
defense, especially when combined with some of the other approaches below.
### Commands run on error
The `on_error` hook allows you to run an arbitrary command or script when
borgmatic itself encounters an error running your backups. So for instance,
you can run a script to send yourself a text message alert. But note that if
borgmatic doesn't actually run, this alert won't fire. See [error
hooks](https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#error-hooks)
below for how to configure this.
### Third-party monitoring services
borgmatic integrates with monitoring services like
[Healthchecks](https://healthchecks.io/), [Cronitor](https://cronitor.io),
[Cronhub](https://cronhub.io), and [PagerDuty](https://www.pagerduty.com/) and
pings these services whenever borgmatic runs. That way, you'll receive an
alert when something goes wrong or (for certain hooks) the service doesn't
hear from borgmatic for a configured interval. See [Healthchecks
hook](https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#healthchecks-hook),
[Cronitor
hook](https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#cronitor-hook),
[Cronhub
hook](https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#cronhub-hook),
and [PagerDuty
hook](https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#pagerduty-hook)
below for how to configure this.
While these services offer different features, you probably only need to use
one of them at most.
### Third-party monitoring software
You can use traditional monitoring software to consume borgmatic JSON output
and track when the last successful backup occurred. See [scripting
borgmatic](https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#scripting-borgmatic)
and [related
software](https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#related-software)
below for how to configure this.
### Borg hosting providers
Most [Borg hosting
providers](https://torsion.org/borgmatic/#hosting-providers) include
monitoring and alerting as part of their offering. This gives you a dashboard
to check on all of your backups, and can alert you if the service doesn't hear
from borgmatic for a configured interval.
### Consistency checks
While not strictly part of monitoring, if you really want confidence that your
backups are not only running but are restorable as well, you can configure
particular [consistency
checks](https://torsion.org/borgmatic/docs/how-to/deal-with-very-large-backups/#consistency-check-configuration)
or even script full [extract
tests](https://torsion.org/borgmatic/docs/how-to/extract-a-backup/).
## Error hooks
When an error occurs during a `prune`, `compact`, `create`, or `check` action,
borgmatic can run configurable shell commands to fire off custom error
notifications or take other actions, so you can get alerted as soon as
something goes wrong. Here's a not-so-useful example:
```yaml
hooks:
on_error:
- echo "Error while creating a backup or running a backup hook."
```
The `on_error` hook supports interpolating particular runtime variables into
the hook command. Here's an example that assumes you provide a separate shell
script to handle the alerting:
```yaml
hooks:
on_error:
- send-text-message.sh "{configuration_filename}" "{repository}"
```
In this example, when the error occurs, borgmatic interpolates runtime values
into the hook command: the borgmatic configuration filename, and the path of
the repository. Here's the full set of supported variables you can use here:
* `configuration_filename`: borgmatic configuration filename in which the
error occurred
* `repository`: path of the repository in which the error occurred (may be
blank if the error occurs in a hook)
* `error`: the error message itself
* `output`: output of the command that failed (may be blank if an error
occurred without running a command)
Note that borgmatic runs the `on_error` hooks only for `prune`, `compact`,
`create`, or `check` actions or hooks in which an error occurs, and not other
actions. borgmatic does not run `on_error` hooks if an error occurs within a
`before_everything` or `after_everything` hook. For more about hooks, see the
[borgmatic hooks
documentation](https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/),
especially the security information.
## Healthchecks hook
[Healthchecks](https://healthchecks.io/) is a service that provides "instant
alerts when your cron jobs fail silently", and borgmatic has built-in
integration with it. Once you create a Healthchecks account and project on
their site, all you need to do is configure borgmatic with the unique "Ping
URL" for your project. Here's an example:
```yaml
hooks:
healthchecks:
ping_url: https://hc-ping.com/addffa72-da17-40ae-be9c-ff591afb942a
```
With this hook in place, borgmatic pings your Healthchecks project when a
backup begins, ends, or errors. Specifically, after the <a
href="https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/">`before_backup`
hooks</a> run, borgmatic lets Healthchecks know that it has started if any of
the `prune`, `compact`, `create`, or `check` actions are run.
Then, if the actions complete successfully, borgmatic notifies Healthchecks of
the success after the `after_backup` hooks run, and includes borgmatic logs in
the payload data sent to Healthchecks. This means that borgmatic logs show up
in the Healthchecks UI, although be aware that Healthchecks currently has a
10-kilobyte limit for the logs in each ping.
If an error occurs during any action or hook, borgmatic notifies Healthchecks
after the `on_error` hooks run, also tacking on logs including the error
itself. But the logs are only included for errors that occur when a `prune`,
`compact`, `create`, or `check` action is run.
You can customize the verbosity of the logs that are sent to Healthchecks with
borgmatic's `--monitoring-verbosity` flag. The `--files` and `--stats` flags
may also be of use. See `borgmatic --help` for more information. Additionally,
see the [borgmatic configuration
file](https://torsion.org/borgmatic/docs/reference/configuration/) for
additional Healthchecks options.
You can configure Healthchecks to notify you by a [variety of
mechanisms](https://healthchecks.io/#welcome-integrations) when backups fail
or it doesn't hear from borgmatic for a certain period of time.
## Cronitor hook
[Cronitor](https://cronitor.io/) provides "Cron monitoring and uptime healthchecks
for websites, services and APIs", and borgmatic has built-in
integration with it. Once you create a Cronitor account and cron job monitor on
their site, all you need to do is configure borgmatic with the unique "Ping
API URL" for your monitor. Here's an example:
```yaml
hooks:
cronitor:
ping_url: https://cronitor.link/d3x0c1
```
With this hook in place, borgmatic pings your Cronitor monitor when a backup
begins, ends, or errors. Specifically, after the <a
href="https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/">`before_backup`
hooks</a> run, borgmatic lets Cronitor know that it has started if any of the
`prune`, `compact`, `create`, or `check` actions are run. Then, if the actions
complete successfully, borgmatic notifies Cronitor of the success after the
`after_backup` hooks run. And if an error occurs during any action or hook,
borgmatic notifies Cronitor after the `on_error` hooks run.
You can configure Cronitor to notify you by a [variety of
mechanisms](https://cronitor.io/docs/cron-job-notifications) when backups fail
or it doesn't hear from borgmatic for a certain period of time.
## Cronhub hook
[Cronhub](https://cronhub.io/) provides "instant alerts when any of your
background jobs fail silently or run longer than expected", and borgmatic has
built-in integration with it. Once you create a Cronhub account and monitor on
their site, all you need to do is configure borgmatic with the unique "Ping
URL" for your monitor. Here's an example:
```yaml
hooks:
cronhub:
ping_url: https://cronhub.io/start/1f5e3410-254c-11e8-b61d-55875966d031
```
With this hook in place, borgmatic pings your Cronhub monitor when a backup
begins, ends, or errors. Specifically, after the <a
href="https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/">`before_backup`
hooks</a> run, borgmatic lets Cronhub know that it has started if any of the
`prune`, `compact`, `create`, or `check` actions are run. Then, if the actions
complete successfully, borgmatic notifies Cronhub of the success after the
`after_backup` hooks run. And if an error occurs during any action or hook,
borgmatic notifies Cronhub after the `on_error` hooks run.
Note that even though you configure borgmatic with the "start" variant of the
ping URL, borgmatic substitutes the correct state into the URL when pinging
Cronhub ("start", "finish", or "fail").
You can configure Cronhub to notify you by a [variety of
mechanisms](https://docs.cronhub.io/integrations.html) when backups fail
or it doesn't hear from borgmatic for a certain period of time.
## PagerDuty hook
In case you're new here: [borgmatic](https://torsion.org/borgmatic/) is
simple, configuration-driven backup software for servers and workstations,
powered by [Borg Backup](https://www.borgbackup.org/).
[PagerDuty](https://www.pagerduty.com/) provides incident monitoring and
alerting. borgmatic has built-in integration that can notify you via PagerDuty
as soon as a backup fails, so you can make sure your backups keep working.
First, create a PagerDuty account and <a
href="https://support.pagerduty.com/docs/services-and-integrations">service</a>
on their site. On the service, add an integration and set the Integration Type
to "borgmatic".
Then, configure borgmatic with the unique "Integration Key" for your service.
Here's an example:
```yaml
hooks:
pagerduty:
integration_key: a177cad45bd374409f78906a810a3074
```
With this hook in place, borgmatic creates a PagerDuty event for your service
whenever backups fail. Specifically, if an error occurs during a `create`,
`prune`, `compact`, or `check` action, borgmatic sends an event to PagerDuty
before the `on_error` hooks run. Note that borgmatic does not contact
PagerDuty when a backup starts or ends without error.
You can configure PagerDuty to notify you by a [variety of
mechanisms](https://support.pagerduty.com/docs/notifications) when backups
fail.
If you have any issues with the integration, [please contact
us](https://torsion.org/borgmatic/#support-and-contributing).
## Scripting borgmatic
To consume the output of borgmatic in other software, you can include an
optional `--json` flag with `create`, `list`, or `info` to get the output
formatted as JSON.
Note that when you specify the `--json` flag, Borg's other non-JSON output is
suppressed so as not to interfere with the captured JSON. Also note that JSON
output only shows up at the console, and not in syslog.
## Related software
* [Borgmacator GNOME AppIndicator](https://github.com/N-Coder/borgmacator/)
### Latest backups
All borgmatic actions that accept an "--archive" flag allow you to specify an
archive name of "latest". This lets you get the latest archive without having
to first run "borgmatic list" manually, which can be handy in automated
scripts. Here's an example:
```bash
borgmatic info --archive latest
```

View File

@ -0,0 +1,3 @@
<head>
<meta http-equiv='refresh' content='0; URL=https://torsion.org/borgmatic/docs/how-to/extract-a-backup/'>
</head>

View File

@ -0,0 +1,94 @@
---
title: How to run arbitrary Borg commands
eleventyNavigation:
key: 🔧 Run arbitrary Borg commands
parent: How-to guides
order: 10
---
## Running Borg with borgmatic
Borg has several commands (and options) that borgmatic does not currently
support. Sometimes though, as a borgmatic user, you may find yourself wanting
to take advantage of these off-the-beaten-path Borg features. You could of
course drop down to running Borg directly. But then you'd give up all the
niceties of your borgmatic configuration. You could file a [borgmatic
ticket](https://torsion.org/borgmatic/#issues) or even a [pull
request](https://torsion.org/borgmatic/#contributing) to add the feature. But
what if you need it *now*?
That's where borgmatic's support for running "arbitrary" Borg commands comes
in. Running Borg commands with borgmatic takes advantage of the following, all
based on your borgmatic configuration files or command-line arguments:
* configured repositories (automatically runs your Borg command once for each
one)
* local and remote Borg binary paths
* SSH settings and Borg environment variables
* lock wait settings
* verbosity
### borg action
The way you run Borg with borgmatic is via the `borg` action. Here's a simple
example:
```bash
borgmatic borg break-lock
```
(No `borg` action in borgmatic? Time to upgrade!)
This runs Borg's `break-lock` command once on each configured borgmatic
repository. Notice how the repository isn't present in the specified Borg
options, as that part is provided by borgmatic.
You can also specify Borg options for relevant commands:
```bash
borgmatic borg list --progress
```
This runs Borg's `list` command once on each configured borgmatic
repository. However, the native `borgmatic list` action should be preferred
for most use.
What if you only want to run Borg on a single configured borgmatic repository
when you've got several configured? Not a problem.
```bash
borgmatic borg --repository repo.borg break-lock
```
And what about a single archive?
```bash
borgmatic borg --archive your-archive-name list
```
### Limitations
borgmatic's `borg` action is not without limitations:
* The Borg command you want to run (`create`, `list`, etc.) *must* come first
after the `borg` action. If you have any other Borg options to specify,
provide them after. For instance, `borgmatic borg list --progress` will work,
but `borgmatic borg --progress list` will not.
* borgmatic supplies the repository/archive name to Borg for you (based on
your borgmatic configuration or the `borgmatic borg --repository`/`--archive`
arguments), so do not specify the repository/archive otherwise.
* The `borg` action will not currently work for any Borg commands like `borg
serve` that do not accept a repository/archive name.
* Do not specify any global borgmatic arguments to the right of the `borg`
action. (They will be passed to Borg instead of borgmatic.) If you have
global borgmatic arguments, specify them *before* the `borg` action.
* Unlike other borgmatic actions, you cannot combine the `borg` action with
other borgmatic actions. This is to prevent ambiguity in commands like
`borgmatic borg list`, in which `list` is both a valid Borg command and a
borgmatic action. In this case, only the Borg command is run.
* Unlike normal borgmatic actions that support JSON, the `borg` action will
not disable certain borgmatic logs to avoid interfering with JSON output.
In general, this `borgmatic borg` feature should be considered an escape
valve—a feature of second resort. In the long run, it's preferable to wrap
Borg commands with borgmatic actions that can support them fully.

View File

@ -0,0 +1,3 @@
<head>
<meta http-equiv='refresh' content='0; URL=https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/'>
</head>

View File

@ -0,0 +1,376 @@
---
title: How to set up backups
eleventyNavigation:
key: 📥 Set up backups
parent: How-to guides
order: 0
---
## Installation
Many users need to backup system files that require privileged access, so
these instructions install and run borgmatic as root. If you don't need to
backup such files, then you are welcome to install and run borgmatic as a
non-root user.
First, manually [install
Borg](https://borgbackup.readthedocs.io/en/stable/installation.html), at least
version 1.1. borgmatic does not install Borg automatically so as to avoid
conflicts with existing Borg installations.
Then, download and install borgmatic as a [user site
installation](https://packaging.python.org/tutorials/installing-packages/#installing-to-the-user-site)
by running the following command:
```bash
sudo pip3 install --user --upgrade borgmatic
```
This installs borgmatic and its commands at the `/root/.local/bin` path.
Your pip binary may have a different name than "pip3". Make sure you're using
Python 3.7+, as borgmatic does not support older versions of Python.
The next step is to ensure that borgmatic's commands available are on your
system `PATH`, so that you can run borgmatic:
```bash
echo export 'PATH="$PATH:/root/.local/bin"' >> ~/.bashrc
source ~/.bashrc
```
This adds `/root/.local/bin` to your non-root user's system `PATH`.
If you're using a command shell other than Bash, you may need to use different
commands here.
You can check whether all of this worked with:
```bash
sudo borgmatic --version
```
If borgmatic is properly installed, that should output your borgmatic version.
As an alternative to adding the path to `~/.bashrc` file, if you're using sudo
to run borgmatic, you can configure [sudo's
`secure_path` option](https://man.archlinux.org/man/sudoers.5) to include
borgmatic's path.
### Global install option
If you try the user site installation above, and have problems making
borgmatic commands runnable on your system `PATH`, an alternate approach is to
install borgmatic globally.
The following uninstalls borgmatic, and then reinstalls it such that borgmatic
commands are on the default system `PATH`:
```bash
sudo pip3 uninstall borgmatic
sudo pip3 install --upgrade borgmatic
```
The main downside of a global install is that borgmatic is less cleanly
separated from the rest of your Python software, and there's the theoretical
possibility of library conflicts. But if you're okay with that, for instance
on a relatively dedicated system, then a global install can work out fine.
### Other ways to install
Besides the approaches described above, there are several other options for
installing borgmatic:
* [Docker image with scheduled backups](https://hub.docker.com/r/b3vis/borgmatic/) (+ Docker Compose files)
* [Docker image with multi-arch and Docker CLI support](https://hub.docker.com/r/modem7/borgmatic-docker/)
* [Debian](https://tracker.debian.org/pkg/borgmatic)
* [Ubuntu](https://launchpad.net/ubuntu/+source/borgmatic)
* [Fedora official](https://bodhi.fedoraproject.org/updates/?search=borgmatic)
* [Fedora unofficial](https://copr.fedorainfracloud.org/coprs/heffer/borgmatic/)
* [Arch Linux](https://www.archlinux.org/packages/community/any/borgmatic/)
* [Alpine Linux](https://pkgs.alpinelinux.org/packages?name=borgmatic)
* [OpenBSD](http://ports.su/sysutils/borgmatic)
* [openSUSE](https://software.opensuse.org/package/borgmatic)
* [macOS (via Homebrew)](https://formulae.brew.sh/formula/borgmatic)
* [Ansible role](https://github.com/borgbase/ansible-role-borgbackup)
* [virtualenv](https://virtualenv.pypa.io/en/stable/)
## Hosting providers
Need somewhere to store your encrypted off-site backups? The following hosting
providers include specific support for Borg/borgmatic—and fund borgmatic
development and hosting when you use these links to sign up. (These are
referral links, but without any tracking scripts or cookies.)
<ul>
<li class="referral"><a href="https://www.borgbase.com/?utm_source=borgmatic">BorgBase</a>: Borg hosting service with support for monitoring, 2FA, and append-only repos</li>
</ul>
Additionally, [rsync.net](https://www.rsync.net/products/borg.html) and
[Hetzner](https://www.hetzner.com/storage/storage-box) have compatible storage
offerings, but do not currently fund borgmatic development or hosting.
## Configuration
After you install borgmatic, generate a sample configuration file:
```bash
sudo generate-borgmatic-config
```
If that command is not found, then it may be installed in a location that's
not in your system `PATH` (see above). Try looking in `~/.local/bin/`.
This generates a sample configuration file at `/etc/borgmatic/config.yaml` by
default. If you'd like to use another path, use the `--destination` flag, for
instance: `--destination ~/.config/borgmatic/config.yaml`.
You should edit the configuration file to suit your needs, as the generated
values are only representative. All options are optional except where
indicated, so feel free to ignore anything you don't need.
Note that the configuration file is organized into distinct sections, each
with a section name like `location:` or `storage:`. So take care that if you
uncomment a particular option, also uncomment its containing section name, or
else borgmatic won't recognize the option. Also be sure to use spaces rather
than tabs for indentation; YAML does not allow tabs.
You can get the same sample configuration file from the [configuration
reference](https://torsion.org/borgmatic/docs/reference/configuration/), the
authoritative set of all configuration options. This is handy if borgmatic has
added new options since you originally created your configuration file. Also
check out how to [upgrade your
configuration](https://torsion.org/borgmatic/docs/how-to/upgrade/#upgrading-your-configuration).
### Encryption
If you encrypt your Borg repository with a passphrase or a key file, you'll
either need to set the borgmatic `encryption_passphrase` configuration
variable or set the `BORG_PASSPHRASE` environment variable. See the
[repository encryption
section](https://borgbackup.readthedocs.io/en/stable/quickstart.html#repository-encryption)
of the Borg Quick Start for more info.
Alternatively, you can specify the passphrase programatically by setting
either the borgmatic `encryption_passcommand` configuration variable or the
`BORG_PASSCOMMAND` environment variable. See the [Borg Security
FAQ](http://borgbackup.readthedocs.io/en/stable/faq.html#how-can-i-specify-the-encryption-passphrase-programmatically)
for more info.
### Redundancy
If you'd like to configure your backups to go to multiple different
repositories, see the documentation on how to [make backups
redundant](https://torsion.org/borgmatic/docs/how-to/make-backups-redundant/).
### Validation
If you'd like to validate that your borgmatic configuration is valid, the
following command is available for that:
```bash
sudo validate-borgmatic-config
```
This command's exit status (`$?` in Bash) is zero when configuration is valid
and non-zero otherwise.
Validating configuration can be useful if you generate your configuration
files via configuration management, or you want to double check that your hand
edits are valid.
## Initialization
Before you can create backups with borgmatic, you first need to initialize a
Borg repository so you have a destination for your backup archives. (But skip
this step if you already have a Borg repository.) To create a repository, run
a command like the following:
```bash
sudo borgmatic init --encryption repokey
```
(No borgmatic `init` action? Try the old-style `--init` flag, or upgrade
borgmatic!)
This uses the borgmatic configuration file you created above to determine
which local or remote repository to create, and encrypts it with the
encryption passphrase specified there if one is provided. Read about [Borg
encryption
modes](https://borgbackup.readthedocs.io/en/stable/usage/init.html#encryption-modes)
for the menu of available encryption modes.
Also, optionally check out the [Borg Quick
Start](https://borgbackup.readthedocs.org/en/stable/quickstart.html) for more
background about repository initialization.
Note that borgmatic skips repository initialization if the repository already
exists. This supports use cases like ensuring a repository exists prior to
performing a backup.
If the repository is on a remote host, make sure that your local user has
key-based SSH access to the desired user account on the remote host.
## Backups
Now that you've configured borgmatic and initialized a repository, it's a
good idea to test that borgmatic is working. So to run borgmatic and start a
backup, you can invoke it like this:
```bash
sudo borgmatic --verbosity 1 --files
```
(No borgmatic `--files` flag? It's only present in newer versions of
borgmatic. So try leaving it out, or upgrade borgmatic!)
By default, this will also prune any old backups as per the configured
retention policy, compact segments to free up space (with Borg 1.2+), and
check backups for consistency problems due to things like file damage.
The verbosity flag makes borgmatic show the steps it's performing. And the
files flag lists each file that's new or changed since the last backup.
Eyeball the list and see if it matches your expectations based on the
configuration.
If you'd like to specify an alternate configuration file path, use the
`--config` flag. See `borgmatic --help` for more information.
## Autopilot
Running backups manually is good for validating your configuration, but I'm
guessing that you want to run borgmatic automatically, say once a day. To do
that, you can configure a separate job runner to invoke it periodically.
### cron
If you're using cron, download the [sample cron
file](https://projects.torsion.org/borgmatic-collective/borgmatic/src/master/sample/cron/borgmatic).
Then, from the directory where you downloaded it:
```bash
sudo mv borgmatic /etc/cron.d/borgmatic
sudo chmod +x /etc/cron.d/borgmatic
```
If borgmatic is installed at a different location than
`/root/.local/bin/borgmatic`, edit the cron file with the correct path. You
can also modify the cron file if you'd like to run borgmatic more or less
frequently.
### systemd
If you're using systemd instead of cron to run jobs, you can still configure
borgmatic to run automatically.
(If you installed borgmatic from [Other ways to
install](https://torsion.org/borgmatic/docs/how-to/set-up-backups/#other-ways-to-install),
you may already have borgmatic systemd service and timer files. If so, you may
be able to skip some of the steps below.)
First, download the [sample systemd service
file](https://projects.torsion.org/borgmatic-collective/borgmatic/raw/branch/master/sample/systemd/borgmatic.service)
and the [sample systemd timer
file](https://projects.torsion.org/borgmatic-collective/borgmatic/raw/branch/master/sample/systemd/borgmatic.timer).
Then, from the directory where you downloaded them:
```bash
sudo mv borgmatic.service borgmatic.timer /etc/systemd/system/
sudo systemctl enable --now borgmatic.timer
```
Review the security settings in the service file and update them as needed.
If `ProtectSystem=strict` is enabled and local repositories are used, then
the repository path must be added to the `ReadWritePaths` list.
Feel free to modify the timer file based on how frequently you'd like
borgmatic to run.
### launchd in macOS
If you run borgmatic in macOS with launchd, you may encounter permissions
issues when reading files to backup. If that happens to you, you may be
interested in an [unofficial work-around for Full Disk
Access](https://projects.torsion.org/borgmatic-collective/borgmatic/issues/293).
## Niceties
### Shell completion
borgmatic includes a shell completion script (currently only for Bash) to
support tab-completing borgmatic command-line actions and flags. Depending on
how you installed borgmatic, this may be enabled by default. But if it's not,
start by installing the `bash-completion` Linux package or the
[`bash-completion@2`](https://formulae.brew.sh/formula/bash-completion@2)
macOS Homebrew formula. Then, install the shell completion script globally:
```bash
sudo su -c "borgmatic --bash-completion > $(pkg-config --variable=completionsdir bash-completion)/borgmatic"
```
If you don't have `pkg-config` installed, you can try the following path
instead:
```bash
sudo su -c "borgmatic --bash-completion > /usr/share/bash-completion/completions/borgmatic"
```
Or, if you'd like to install the script for just the current user:
```bash
mkdir --parents ~/.local/share/bash-completion/completions
borgmatic --bash-completion > ~/.local/share/bash-completion/completions/borgmatic
```
Finally, restart your shell (`exit` and open a new shell) so the completions
take effect.
### Colored output
borgmatic produces colored terminal output by default. It is disabled when a
non-interactive terminal is detected (like a cron job), or when you use the
`--json` flag. Otherwise, you can disable it by passing the `--no-color` flag,
setting the environment variable `PY_COLORS=False`, or setting the `color`
option to `false` in the `output` section of configuration.
## Troubleshooting
### "found character that cannot start any token" error
If you run borgmatic and see an error looking something like this, it probably
means you've used tabs instead of spaces:
```
test.yaml: Error parsing configuration file
An error occurred while parsing a configuration file at config.yaml:
while scanning for the next token
found character that cannot start any token
in "config.yaml", line 230, column 1
```
YAML does not allow tabs. So to fix this, replace any tabs in your
configuration file with the requisite number of spaces.
### libyaml compilation errors
borgmatic depends on a Python YAML library (ruamel.yaml) that will optionally
use a C YAML library (libyaml) if present. But if it's not installed, then
when installing or upgrading borgmatic, you may see errors about compiling the
YAML library. If so, not to worry. borgmatic should install and function
correctly even without the C YAML library. And borgmatic won't be any faster
with the C library present, so you don't need to go out of your way to install
it.

117
docs/how-to/upgrade.md Normal file
View File

@ -0,0 +1,117 @@
---
title: How to upgrade borgmatic
eleventyNavigation:
key: 📦 Upgrade borgmatic
parent: How-to guides
order: 11
---
## Upgrading
In general, all you should need to do to upgrade borgmatic is run the
following:
```bash
sudo pip3 install --user --upgrade borgmatic
```
See below about special cases with old versions of borgmatic. Additionally, if
you installed borgmatic [without using `pip3 install
--user`](https://torsion.org/borgmatic/docs/how-to/set-up-backups/#other-ways-to-install),
then your upgrade process may be different.
### Upgrading your configuration
The borgmatic configuration file format is almost always backwards-compatible
from release to release without any changes, but you may still want to update
your configuration file when you upgrade to take advantage of new
configuration options. This is completely optional. If you prefer, you can add
new configuration options manually.
If you do want to upgrade your configuration file to include new options, use
the `generate-borgmatic-config` script with its optional `--source` flag that
takes the path to your original configuration file. If provided with this
path, `generate-borgmatic-config` merges your original configuration into the
generated configuration file, so you get all the newest options and comments.
Here's an example:
```bash
generate-borgmatic-config --source config.yaml --destination config-new.yaml
```
New options start as commented out, so you can edit the file and decide
whether you want to use each one.
There are a few caveats to this process. First, when generating the new
configuration file, `generate-borgmatic-config` replaces any comments you've
written in your original configuration file with the newest generated
comments. Second, the script adds back any options you had originally deleted,
although it does so with the options commented out. And finally, any YAML
includes you've used in the source configuration get flattened out into a
single generated file.
As a safety measure, `generate-borgmatic-config` refuses to modify
configuration files in-place. So it's up to you to review the generated file
and, if desired, replace your original configuration file with it.
### Upgrading from borgmatic 1.0.x
borgmatic changed its configuration file format in version 1.1.0 from
INI-style to YAML. This better supports validation, and has a more natural way
to express lists of values. To upgrade your existing configuration, first
upgrade to the new version of borgmatic.
As of version 1.1.0, borgmatic no longer supports Python 2. If you were
already running borgmatic with Python 3, then you can upgrade borgmatic
in-place:
```bash
sudo pip3 install --user --upgrade borgmatic
```
But if you were running borgmatic with Python 2, uninstall and reinstall instead:
```bash
sudo pip uninstall borgmatic
sudo pip3 install --user borgmatic
```
The pip binary names for different versions of Python can differ, so the above
commands may need some tweaking to work on your machine.
Once borgmatic is upgraded, run:
```bash
sudo upgrade-borgmatic-config
```
That will generate a new YAML configuration file at /etc/borgmatic/config.yaml
(by default) using the values from both your existing configuration and
excludes files. The new version of borgmatic will consume the YAML
configuration file instead of the old one.
### Upgrading from atticmatic
You can ignore this section if you're not an atticmatic user (the former name
of borgmatic).
borgmatic only supports Borg now and no longer supports Attic. So if you're
an Attic user, consider switching to Borg. See the [Borg upgrade
command](https://borgbackup.readthedocs.io/en/stable/usage.html#borg-upgrade)
for more information. Then, follow the instructions above about setting up
your borgmatic configuration files.
If you were already using Borg with atticmatic, then you can upgrade
from atticmatic to borgmatic by running the following commands:
```bash
sudo pip3 uninstall atticmatic
sudo pip3 install --user borgmatic
```
That's it! borgmatic will continue using your /etc/borgmatic configuration
files.

Some files were not shown because too many files have changed in this diff Show More