A wrapper script for Borg backup software that creates and prunes backups
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

README.md 17KB

5 years ago
5 years ago
5 years ago
5 years ago
5 years ago
5 years ago
2 years ago
5 years ago
5 years ago
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532
  1. ---
  2. title: borgmatic
  3. permalink: borgmatic/index.html
  4. ---
  5. ## Overview
  6. <img src="https://projects.torsion.org/witten/borgmatic/raw/branch/master/static/borgmatic.png" width="150px" style="float: right; padding-left: 1em;">
  7. borgmatic is a simple Python wrapper script for the
  8. [Borg](https://www.borgbackup.org/) backup software that initiates a backup,
  9. prunes any old backups according to a retention policy, and validates backups
  10. for consistency. The script supports specifying your settings in a declarative
  11. configuration file rather than having to put them all on the command-line, and
  12. handles common errors.
  13. Here's an example config file:
  14. ```yaml
  15. location:
  16. # List of source directories to backup. Globs are expanded.
  17. source_directories:
  18. - /home
  19. - /etc
  20. - /var/log/syslog*
  21. # Paths to local or remote repositories.
  22. repositories:
  23. - user@backupserver:sourcehostname.borg
  24. # Any paths matching these patterns are excluded from backups.
  25. exclude_patterns:
  26. - /home/*/.cache
  27. retention:
  28. # Retention policy for how many backups to keep in each category.
  29. keep_daily: 7
  30. keep_weekly: 4
  31. keep_monthly: 6
  32. consistency:
  33. # List of consistency checks to run: "repository", "archives", or both.
  34. checks:
  35. - repository
  36. - archives
  37. ```
  38. borgmatic is hosted at <https://torsion.org/borgmatic> with [source code
  39. available](https://projects.torsion.org/witten/borgmatic). It's also mirrored
  40. on [GitHub](https://github.com/witten/borgmatic) for convenience.
  41. Want to see borgmatic in action? Check out the <a
  42. href="https://asciinema.org/a/203761" target="_blank">screencast</a>.
  43. <script src="https://asciinema.org/a/203761.js" id="asciicast-203761" async></script>
  44. ## Installation
  45. To get up and running, first [install
  46. Borg](https://borgbackup.readthedocs.io/en/latest/installation.html), at
  47. least version 1.1.
  48. Then, run the following command to download and install borgmatic:
  49. ```bash
  50. sudo pip3 install --upgrade borgmatic
  51. ```
  52. Note that your pip binary may have a different name than "pip3". Make sure
  53. you're using Python 3, as borgmatic does not support Python 2.
  54. ### Other ways to install
  55. * [A borgmatic Docker image](https://hub.docker.com/r/monachus/borgmatic/) based
  56. on Alpine Linux.
  57. * [Another borgmatic Docker image](https://hub.docker.com/r/b3vis/borgmatic/) based
  58. on Alpine Linux.
  59. * [A borgmatic package for
  60. Fedora](https://bodhi.fedoraproject.org/updates/?search=borgmatic).
  61. * [A borgmatic package for Arch
  62. Linux](https://aur.archlinux.org/packages/borgmatic/).
  63. * [A borgmatic package for OpenBSD](http://ports.su/sysutils/borgmatic).
  64. <br><br>
  65. ## Configuration
  66. After you install borgmatic, generate a sample configuration file:
  67. ```bash
  68. sudo generate-borgmatic-config
  69. ```
  70. If that command is not found, then it may be installed in a location that's
  71. not in your system `PATH`. Try looking in `/usr/local/bin/`.
  72. This generates a sample configuration file at /etc/borgmatic/config.yaml (by
  73. default). You should edit the file to suit your needs, as the values are just
  74. representative. All fields are optional except where indicated, so feel free
  75. to ignore anything you don't need.
  76. You can also have a look at the [full configuration
  77. schema](https://projects.torsion.org/witten/borgmatic/src/master/borgmatic/config/schema.yaml)
  78. for the authoritative set of all configuration options. This is handy if
  79. borgmatic has added new options since you originally created your
  80. configuration file.
  81. ### Encryption
  82. Note that if you plan to run borgmatic on a schedule with cron, and you
  83. encrypt your Borg repository with a passphrase instead of a key file, you'll
  84. either need to set the borgmatic `encryption_passphrase` configuration
  85. variable or set the `BORG_PASSPHRASE` environment variable. See the
  86. [repository encryption
  87. section](https://borgbackup.readthedocs.io/en/latest/quickstart.html#repository-encryption)
  88. of the Quick Start for more info.
  89. Alternatively, the passphrase can be specified programatically by setting
  90. either the borgmatic `encryption_passcommand` configuration variable or the
  91. `BORG_PASSCOMMAND` environment variable. See the [Borg Security
  92. FAQ](http://borgbackup.readthedocs.io/en/stable/faq.html#how-can-i-specify-the-encryption-passphrase-programmatically)
  93. for more info.
  94. ## Usage
  95. ### Initialization
  96. Before you can create backups with borgmatic, you first need to initialize a
  97. Borg repository so you have a destination for your backup archives. (But skip
  98. this step if you already have a Borg repository.) To create a repository, run
  99. a command like the following:
  100. ```bash
  101. borgmatic --init --encryption repokey
  102. ```
  103. This uses the borgmatic configuration file you created above to determine
  104. which local or remote repository to create, and encrypts it with the
  105. encryption passphrase specified there if one is provided. Read about [Borg
  106. encryption
  107. modes](https://borgbackup.readthedocs.io/en/latest/usage/init.html#encryption-modes)
  108. for the menu of available encryption modes.
  109. Also, optionally check out the [Borg Quick
  110. Start](https://borgbackup.readthedocs.org/en/latest/quickstart.html) for more
  111. background about repository initialization.
  112. If the repository is on a remote host, make sure that your local user has
  113. key-based SSH access to the desired user account on the remote host.
  114. ### Backups
  115. You can run borgmatic and start a backup simply by invoking it without
  116. arguments:
  117. ```bash
  118. borgmatic
  119. ```
  120. This will also prune any old backups as per the configured retention policy,
  121. and check backups for consistency problems due to things like file damage.
  122. If you'd like to see the available command-line arguments, view the help:
  123. ```bash
  124. borgmatic --help
  125. ```
  126. Note that borgmatic prunes archives *before* creating an archive, so as to
  127. free up space for archiving. This means that when a borgmatic run finishes,
  128. there may still be prune-able archives. Not to worry, as they will get cleaned
  129. up at the start of the next run.
  130. ### Verbosity
  131. By default, the backup will proceed silently except in the case of errors. But
  132. if you'd like to to get additional information about the progress of the
  133. backup as it proceeds, use the verbosity option:
  134. ```bash
  135. borgmatic --verbosity 1
  136. ```
  137. Or, for even more progress spew:
  138. ```bash
  139. borgmatic --verbosity 2
  140. ```
  141. ### À la carte
  142. If you want to run borgmatic with only pruning, creating, or checking enabled,
  143. the following optional flags are available:
  144. ```bash
  145. borgmatic --prune
  146. borgmatic --create
  147. borgmatic --check
  148. ```
  149. You can run with only one of these flags provided, or you can mix and match
  150. any number of them. This supports use cases like running consistency checks
  151. from a different cron job with a different frequency, or running pruning with
  152. a different verbosity level.
  153. Additionally, borgmatic provides convenient flags for Borg's
  154. [list](https://borgbackup.readthedocs.io/en/stable/usage/list.html) and
  155. [info](https://borgbackup.readthedocs.io/en/stable/usage/info.html)
  156. functionality:
  157. ```bash
  158. borgmatic --list
  159. borgmatic --info
  160. ```
  161. You can include an optional `--json` flag with `--create`, `--list`, or
  162. `--info` to get the output formatted as JSON.
  163. ## Autopilot
  164. If you want to run borgmatic automatically, say once a day, the you can
  165. configure a job runner to invoke it periodically.
  166. ### cron
  167. If you're using cron, download the [sample cron
  168. file](https://projects.torsion.org/witten/borgmatic/src/master/sample/cron/borgmatic).
  169. Then, from the directory where you downloaded it:
  170. ```bash
  171. sudo mv borgmatic /etc/cron.d/borgmatic
  172. sudo chmod +x /etc/cron.d/borgmatic
  173. ```
  174. You can modify the cron file if you'd like to run borgmatic more or less frequently.
  175. ### systemd
  176. If you're using systemd instead of cron to run jobs, download the [sample
  177. systemd service
  178. file](https://projects.torsion.org/witten/borgmatic/src/master/sample/systemd/borgmatic.service)
  179. and the [sample systemd timer
  180. file](https://projects.torsion.org/witten/borgmatic/src/master/sample/systemd/borgmatic.timer).
  181. Then, from the directory where you downloaded them:
  182. ```bash
  183. sudo mv borgmatic.service borgmatic.timer /etc/systemd/system/
  184. sudo systemctl enable borgmatic.timer
  185. sudo systemctl start borgmatic.timer
  186. ```
  187. Feel free to modify the timer file based on how frequently you'd like
  188. borgmatic to run.
  189. ## Advanced configuration
  190. ### Multiple configuration files
  191. A more advanced usage is to create multiple separate configuration files and
  192. place each one in an /etc/borgmatic.d directory. For instance:
  193. ```bash
  194. sudo mkdir /etc/borgmatic.d
  195. sudo generate-borgmatic-config --destination /etc/borgmatic.d/app1.yaml
  196. sudo generate-borgmatic-config --destination /etc/borgmatic.d/app2.yaml
  197. ```
  198. With this approach, you can have entirely different backup policies for
  199. different applications on your system. For instance, you may want one backup
  200. configuration for your database data directory, and a different configuration
  201. for your user home directories.
  202. When you set up multiple configuration files like this, borgmatic will run
  203. each one in turn from a single borgmatic invocation. This includes, by
  204. default, the traditional /etc/borgmatic/config.yaml as well.
  205. And if you need even more customizability, you can specify alternate
  206. configuration paths on the command-line with borgmatic's `--config` option.
  207. See `borgmatic --help` for more information.
  208. ### Hooks
  209. If you find yourself performing prepraration tasks before your backup runs, or
  210. cleanup work afterwards, borgmatic hooks may be of interest. They're simply
  211. shell commands that borgmatic executes for you at various points, and they're
  212. configured in the `hooks` section of your configuration file.
  213. For instance, you can specify `before_backup` hooks to dump a database to file
  214. before backing it up, and specify `after_backup` hooks to delete the temporary
  215. file afterwards.
  216. borgmatic hooks run once per configuration file. `before_backup` hooks run
  217. prior to backups of all repositories. `after_backup` hooks run afterwards, but
  218. not if an error occurs in a previous hook or in the backups themselves. And
  219. borgmatic runs `on_error` hooks if an error occurs.
  220. An important security note about hooks: borgmatic executes all hook commands
  221. with the user permissions of borgmatic itself. So to prevent potential shell
  222. injection or privilege escalation, do not forget to set secure permissions
  223. (`chmod 0700`) on borgmatic configuration files and scripts invoked by hooks.
  224. See the sample generated configuration file mentioned above for specifics
  225. about hook configuration syntax.
  226. ## Upgrading
  227. In general, all you should need to do to upgrade borgmatic is run the
  228. following:
  229. ```bash
  230. sudo pip3 install --upgrade borgmatic
  231. ```
  232. However, see below about special cases.
  233. ### Upgrading from borgmatic 1.0.x
  234. borgmatic changed its configuration file format in version 1.1.0 from
  235. INI-style to YAML. This better supports validation, and has a more natural way
  236. to express lists of values. To upgrade your existing configuration, first
  237. upgrade to the new version of borgmatic.
  238. As of version 1.1.0, borgmatic no longer supports Python 2. If you were
  239. already running borgmatic with Python 3, then you can simply upgrade borgmatic
  240. in-place:
  241. ```bash
  242. sudo pip3 install --upgrade borgmatic
  243. ```
  244. But if you were running borgmatic with Python 2, uninstall and reinstall instead:
  245. ```bash
  246. sudo pip uninstall borgmatic
  247. sudo pip3 install borgmatic
  248. ```
  249. The pip binary names for different versions of Python can differ, so the above
  250. commands may need some tweaking to work on your machine.
  251. Once borgmatic is upgraded, run:
  252. ```bash
  253. sudo upgrade-borgmatic-config
  254. ```
  255. That will generate a new YAML configuration file at /etc/borgmatic/config.yaml
  256. (by default) using the values from both your existing configuration and
  257. excludes files. The new version of borgmatic will consume the YAML
  258. configuration file instead of the old one.
  259. ### Upgrading from atticmatic
  260. You can ignore this section if you're not an atticmatic user (the former name
  261. of borgmatic).
  262. borgmatic only supports Borg now and no longer supports Attic. So if you're
  263. an Attic user, consider switching to Borg. See the [Borg upgrade
  264. command](https://borgbackup.readthedocs.io/en/stable/usage.html#borg-upgrade)
  265. for more information. Then, follow the instructions above about setting up
  266. your borgmatic configuration files.
  267. If you were already using Borg with atticmatic, then you can easily upgrade
  268. from atticmatic to borgmatic. Simply run the following commands:
  269. ```bash
  270. sudo pip3 uninstall atticmatic
  271. sudo pip3 install borgmatic
  272. ```
  273. That's it! borgmatic will continue using your /etc/borgmatic configuration
  274. files.
  275. ## Support and contributing
  276. ### Issues
  277. You've got issues? Or an idea for a feature enhancement? We've got an [issue
  278. tracker](https://projects.torsion.org/witten/borgmatic/issues). In order to
  279. create a new issue or comment on an issue, you'll need to [login
  280. first](https://projects.torsion.org/user/login). Note that you can login with
  281. an existing GitHub account if you prefer.
  282. Other questions or comments? Contact <mailto:witten@torsion.org>.
  283. ### Contributing
  284. If you'd like to contribute to borgmatic development, please feel free to
  285. submit a [Pull Request](https://projects.torsion.org/witten/borgmatic/pulls)
  286. or open an [issue](https://projects.torsion.org/witten/borgmatic/issues) first
  287. to discuss your idea. We also accept Pull Requests on GitHub, if that's more
  288. your thing. In general, contributions are very welcome. We don't bite!
  289. ### Code style
  290. Start with [PEP 8](https://www.python.org/dev/peps/pep-0008/). But then, apply
  291. the following deviations from it:
  292. * For strings, prefer single quotes over double quotes.
  293. * Limit all lines to a maximum of 100 characters.
  294. * Use trailing commas within multiline values or argument lists.
  295. * For multiline constructs, put opening and closing delimeters on lines
  296. separate from their contents.
  297. * Within multiline constructs, use standard four-space indentation. Don't align
  298. indentation with an opening delimeter.
  299. borgmatic code uses the [Black](https://black.readthedocs.io/en/stable/) code
  300. formatter and [Flake8](http://flake8.pycqa.org/en/latest/) code checker, so
  301. certain code style requirements will be enforced when running automated tests.
  302. See the Black and Flake8 documentation for more information.
  303. ### Development
  304. To get set up to hack on borgmatic, first clone master via HTTPS or SSH:
  305. ```bash
  306. git clone https://projects.torsion.org/witten/borgmatic.git
  307. ```
  308. Or:
  309. ```bash
  310. git clone ssh://git@projects.torsion.org:3022/witten/borgmatic.git
  311. ```
  312. Then, install borgmatic
  313. "[editable](https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs)"
  314. so that you can easily run borgmatic commands while you're hacking on them to
  315. make sure your changes work.
  316. ```bash
  317. cd borgmatic/
  318. pip3 install --editable --user .
  319. ```
  320. Note that this will typically install the borgmatic commands into
  321. `~/.local/bin`, which may or may not be on your PATH. There are other ways to
  322. install borgmatic editable as well, for instance into the system Python
  323. install (so without `--user`, as root), or even into a
  324. [virtualenv](https://virtualenv.pypa.io/en/stable/). How or where you install
  325. borgmatic is up to you, but generally an editable install makes development
  326. and testing easier.
  327. ### Running tests
  328. Assuming you've cloned the borgmatic source code as described above, and
  329. you're in the `borgmatic/` working copy, install tox, which is used for
  330. setting up testing environments:
  331. ```bash
  332. sudo pip3 install tox
  333. ```
  334. Finally, to actually run tests, run:
  335. ```bash
  336. cd borgmatic
  337. tox
  338. ```
  339. If when running tests, you get an error from the
  340. [Black](https://black.readthedocs.io/en/stable/) code formatter about files
  341. that would be reformatted, you can ask Black to format them for you via the
  342. following:
  343. ```bash
  344. tox -e black
  345. ```
  346. ### End-to-end tests
  347. borgmatic additionally includes some end-to-end tests that integration test
  348. with Borg for a few representative scenarios. These tests don't run by default
  349. because they're relatively slow and depend on Borg. If you would like to run
  350. them:
  351. ```bash
  352. tox -e end-to-end
  353. ```
  354. ## Troubleshooting
  355. ### Broken pipe with remote repository
  356. When running borgmatic on a large remote repository, you may receive errors
  357. like the following, particularly while "borg check" is validating backups for
  358. consistency:
  359. ```text
  360. Write failed: Broken pipe
  361. borg: Error: Connection closed by remote host
  362. ```
  363. This error can be caused by an ssh timeout, which you can rectify by adding
  364. the following to the `~/.ssh/config` file on the client:
  365. ```text
  366. Host *
  367. ServerAliveInterval 120
  368. ```
  369. This should make the client keep the connection alive while validating
  370. backups.
  371. ### libyaml compilation errors
  372. borgmatic depends on a Python YAML library (ruamel.yaml) that will optionally
  373. use a C YAML library (libyaml) if present. But if it's not installed, then
  374. when installing or upgrading borgmatic, you may see errors about compiling the
  375. YAML library. If so, not to worry. borgmatic should install and function
  376. correctly even without the C YAML library. And borgmatic won't be any faster
  377. with the C library present, so you don't need to go out of your way to install
  378. it.