Browse Source

Dropping support for Attic.

master
Dan Helfman 3 years ago
parent
commit
40a215802f

+ 6
- 0
NEWS View File

@@ -1,3 +1,9 @@
1.0.0

* Attic is no longer supported, as there hasn't been any recent development on it.
This will allow faster iteration on Borg-specific configuration.
* Project renamed from atticmatic to borgmatic.

0.1.8

* Fix for handling of spaces in source_directories which resulted in backup up everything.

+ 24
- 37
README.md View File

@@ -1,9 +1,8 @@
title: Atticmatic
title: Borgmatic

## Overview

atticmatic is a simple Python wrapper script for the
[Attic](https://attic-backup.org/) and
borgmatic (formerly atticmatic) is a simple Python wrapper script for the
[Borg](https://borgbackup.readthedocs.org/en/stable/) backup software that
initiates a backup, prunes any old backups according to a retention policy,
and validates backups for consistency. The script supports specifying your
@@ -19,7 +18,7 @@ Here's an example config file:
source_directories: /home /etc /var/log/syslog*

# Path to local or remote backup repository.
repository: user@backupserver:sourcehostname.attic
repository: user@backupserver:sourcehostname.borg

[retention]
# Retention policy for how many backups to keep in each category.
@@ -35,37 +34,30 @@ checks: repository archives
Additionally, exclude patterns can be specified in a separate excludes config
file, one pattern per line.

atticmatic is hosted at <https://torsion.org/atticmatic> with [source code
available](https://torsion.org/hg/atticmatic). It's also mirrored on
[GitHub](https://github.com/witten/atticmatic) and
[BitBucket](https://bitbucket.org/dhelfman/atticmatic) for convenience.
borgmatic is hosted at <https://torsion.org/borgmatic> with [source code
available](https://torsion.org/hg/borgmatic). It's also mirrored on
[GitHub](https://github.com/witten/borgmatic) and
[BitBucket](https://bitbucket.org/dhelfman/borgmatic) for convenience.


## Setup

To get up and running, follow the [Attic Quick
Start](https://attic-backup.org/quickstart.html) or the [Borg Quick
Start](https://borgbackup.readthedocs.org/en/latest/quickstart.html) to create a
repository on a local or remote host. Note that if you plan to run atticmatic
on a schedule with cron, and you encrypt your attic repository with a
passphrase instead of a key file, you'll need to set the atticmatic
To get up and running, follow the [Borg Quick
Start](https://borgbackup.readthedocs.org/en/latest/quickstart.html) to create
a repository on a local or remote host. Note that if you plan to run
borgmatic on a schedule with cron, and you encrypt your Borg repository with
a passphrase instead of a key file, you'll need to set the borgmatic
`encryption_passphrase` configuration variable. See the repository encryption
section of the Quick Start for more info.

If the repository is on a remote host, make sure that your local root user has
key-based ssh access to the desired user account on the remote host.

To install atticmatic, run the following command to download and install it:
To install borgmatic, run the following command to download and install it:

sudo pip install --upgrade atticmatic
sudo pip install --upgrade borgmatic

If you are using Attic, copy the following configuration files:

sudo cp sample/atticmatic.cron /etc/cron.d/atticmatic
sudo mkdir /etc/atticmatic/
sudo cp sample/config sample/excludes /etc/atticmatic/

If you are using Borg, copy the files like this instead:
Then, copy the following configuration files:

sudo cp sample/borgmatic.cron /etc/cron.d/borgmatic
sudo mkdir /etc/borgmatic/
@@ -76,14 +68,9 @@ Lastly, modify the /etc files with your desired configuration.

## Usage

You can run atticmatic and start a backup simply by invoking it without
You can run borgmatic and start a backup simply by invoking it without
arguments:

atticmatic

Or, if you're using Borg, use this command instead to make use of the Borg
backend:

borgmatic

This will also prune any old backups as per the configured retention policy,
@@ -93,15 +80,15 @@ By default, the backup will proceed silently except in the case of errors. But
if you'd like to to get additional information about the progress of the
backup as it proceeds, use the verbosity option:

atticmatic --verbosity 1
borgmatic --verbosity 1

Or, for even more progress spew:

atticmatic --verbosity 2
borgmatic --verbosity 2

If you'd like to see the available command-line arguments, view the help:

atticmatic --help
borgmatic --help


## Running tests
@@ -119,12 +106,12 @@ Then, to actually run tests, run:

### Broken pipe with remote repository

When running atticmatic on a large remote repository, you may receive errors
like the following, particularly while "attic check" is validating backups for
When running borgmatic on a large remote repository, you may receive errors
like the following, particularly while "borg check" is validating backups for
consistency:

Write failed: Broken pipe
attic: Error: Connection closed by remote host
borg: Error: Connection closed by remote host

This error can be caused by an ssh timeout, which you can rectify by adding
the following to the ~/.ssh/config file on the client:
@@ -138,8 +125,8 @@ backups.

## Issues and feedback

Got an issue or an idea for a feature enhancement? Check out the [atticmatic
issue tracker](https://tree.taiga.io/project/witten-atticmatic/issues). In
Got an issue or an idea for a feature enhancement? Check out the [borgmatic
issue tracker](https://tree.taiga.io/project/witten-borgmatic/issues). In
order to create a new issue or comment on an issue, you'll need to [login
first](https://tree.taiga.io/login).


+ 0
- 14
atticmatic/backends/attic.py View File

@@ -1,14 +0,0 @@
from functools import partial

from atticmatic.backends import shared

# An atticmatic backend that supports Attic for actually handling backups.

COMMAND = 'attic'
CONFIG_FORMAT = shared.CONFIG_FORMAT


initialize = partial(shared.initialize, command=COMMAND)
create_archive = partial(shared.create_archive, command=COMMAND)
prune_archives = partial(shared.prune_archives, command=COMMAND)
check_archives = partial(shared.check_archives, command=COMMAND)

+ 0
- 41
atticmatic/backends/borg.py View File

@@ -1,41 +0,0 @@
from functools import partial

from atticmatic.config import Section_format, option
from atticmatic.backends import shared

# An atticmatic backend that supports Borg for actually handling backups.

COMMAND = 'borg'
CONFIG_FORMAT = (
Section_format(
'location',
(
option('source_directories'),
option('one_file_system', value_type=bool, required=False),
option('repository'),
),
),
Section_format(
'storage',
(
option('encryption_passphrase', required=False),
option('compression', required=False),
option('umask', required=False),
),
),
shared.CONFIG_FORMAT[2], # retention
Section_format(
'consistency',
(
option('checks', required=False),
option('check_last', required=False),
),
)
)


initialize = partial(shared.initialize, command=COMMAND)

create_archive = partial(shared.create_archive, command=COMMAND)
prune_archives = partial(shared.prune_archives, command=COMMAND)
check_archives = partial(shared.check_archives, command=COMMAND)

+ 0
- 75
atticmatic/command.py View File

@@ -1,75 +0,0 @@
from __future__ import print_function
from argparse import ArgumentParser
from importlib import import_module
import os
from subprocess import CalledProcessError
import sys

from atticmatic.config import parse_configuration


DEFAULT_CONFIG_FILENAME_PATTERN = '/etc/{}/config'
DEFAULT_EXCLUDES_FILENAME_PATTERN = '/etc/{}/excludes'


def parse_arguments(command_name, *arguments):
'''
Given the name of the command with which this script was invoked and command-line arguments,
parse the arguments and return them as an ArgumentParser instance. Use the command name to
determine the default configuration and excludes paths.
'''
config_filename_default = DEFAULT_CONFIG_FILENAME_PATTERN.format(command_name)
excludes_filename_default = DEFAULT_EXCLUDES_FILENAME_PATTERN.format(command_name)

parser = ArgumentParser()
parser.add_argument(
'-c', '--config',
dest='config_filename',
default=config_filename_default,
help='Configuration filename',
)
parser.add_argument(
'--excludes',
dest='excludes_filename',
default=excludes_filename_default if os.path.exists(excludes_filename_default) else None,
help='Excludes filename',
)
parser.add_argument(
'-v', '--verbosity',
type=int,
help='Display verbose progress (1 for some, 2 for lots)',
)

return parser.parse_args(arguments)


def load_backend(command_name):
'''
Given the name of the command with which this script was invoked, return the corresponding
backend module responsible for actually dealing with backups.
'''
backend_name = {
'atticmatic': 'attic',
'borgmatic': 'borg',
}.get(command_name, 'attic')

return import_module('atticmatic.backends.{}'.format(backend_name))


def main():
try:
command_name = os.path.basename(sys.argv[0])
args = parse_arguments(command_name, *sys.argv[1:])
backend = load_backend(command_name)
config = parse_configuration(args.config_filename, backend.CONFIG_FORMAT)
repository = config.location['repository']

backend.initialize(config.storage)
backend.create_archive(
args.excludes_filename, args.verbosity, config.storage, **config.location
)
backend.prune_archives(args.verbosity, repository, config.retention)
backend.check_archives(args.verbosity, repository, config.consistency)
except (ValueError, IOError, CalledProcessError) as error:
print(error, file=sys.stderr)
sys.exit(1)

+ 0
- 0
atticmatic/tests/unit/__init__.py View File


+ 0
- 0
atticmatic/tests/unit/backends/__init__.py View File


+ 0
- 33
atticmatic/tests/unit/test_command.py View File

@@ -1,33 +0,0 @@
from flexmock import flexmock

from atticmatic import command as module


def test_load_backend_with_atticmatic_command_should_return_attic_backend():
backend = flexmock()
(
flexmock(module).should_receive('import_module').with_args('atticmatic.backends.attic')
.and_return(backend).once()
)

assert module.load_backend('atticmatic') == backend


def test_load_backend_with_unknown_command_should_return_attic_backend():
backend = flexmock()
(
flexmock(module).should_receive('import_module').with_args('atticmatic.backends.attic')
.and_return(backend).once()
)

assert module.load_backend('unknownmatic') == backend


def test_load_backend_with_borgmatic_command_should_return_borg_backend():
backend = flexmock()
(
flexmock(module).should_receive('import_module').with_args('atticmatic.backends.borg')
.and_return(backend).once()
)

assert module.load_backend('borgmatic') == backend

atticmatic/__init__.py → borgmatic/__init__.py View File


atticmatic/backends/shared.py → borgmatic/borg.py View File

@@ -6,52 +6,16 @@ import subprocess
from glob import glob
from itertools import chain

from atticmatic.config import Section_format, option
from atticmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
from borgmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS


# Common backend functionality shared by Attic and Borg. As the two backup
# commands diverge, these shared functions will likely need to be replaced
# with non-shared functions within atticmatic.backends.attic and
# atticmatic.backends.borg.
# Integration with Borg for actually handling backups.


CONFIG_FORMAT = (
Section_format(
'location',
(
option('source_directories'),
option('repository'),
),
),
Section_format(
'storage',
(
option('encryption_passphrase', required=False),
),
),
Section_format(
'retention',
(
option('keep_within', required=False),
option('keep_hourly', int, required=False),
option('keep_daily', int, required=False),
option('keep_weekly', int, required=False),
option('keep_monthly', int, required=False),
option('keep_yearly', int, required=False),
option('prefix', required=False),
),
),
Section_format(
'consistency',
(
option('checks', required=False),
),
)
)
COMMAND = 'borg'


def initialize(storage_config, command):
def initialize(storage_config, command=COMMAND):
passphrase = storage_config.get('encryption_passphrase')

if passphrase:
@@ -59,7 +23,7 @@ def initialize(storage_config, command):


def create_archive(
excludes_filename, verbosity, storage_config, source_directories, repository, command,
excludes_filename, verbosity, storage_config, source_directories, repository, command=COMMAND,
one_file_system=None
):
'''
@@ -115,7 +79,7 @@ def _make_prune_flags(retention_config):
)


def prune_archives(verbosity, repository, retention_config, command):
def prune_archives(verbosity, repository, retention_config, command=COMMAND):
'''
Given a verbosity flag, a local or remote repository path, a retention config dict, and a
command to run, prune attic archives according the the retention policy specified in that
@@ -191,7 +155,7 @@ def _make_check_flags(checks, check_last=None):
) + last_flag


def check_archives(verbosity, repository, consistency_config, command):
def check_archives(verbosity, repository, consistency_config, command=COMMAND):
'''
Given a verbosity flag, a local or remote repository path, a consistency config dict, and a
command to run, check the contained attic archives for consistency.

+ 57
- 0
borgmatic/command.py View File

@@ -0,0 +1,57 @@
from __future__ import print_function
from argparse import ArgumentParser
import os
from subprocess import CalledProcessError
import sys

from borgmatic import borg
from borgmatic.config import parse_configuration, CONFIG_FORMAT


DEFAULT_CONFIG_FILENAME = '/etc/borgmatic/config'
DEFAULT_EXCLUDES_FILENAME = '/etc/borgmatic/excludes'


def parse_arguments(*arguments):
'''
Given the name of the command with which this script was invoked and command-line arguments,
parse the arguments and return them as an ArgumentParser instance. Use the command name to
determine the default configuration and excludes paths.
'''
parser = ArgumentParser()
parser.add_argument(
'-c', '--config',
dest='config_filename',
default=DEFAULT_CONFIG_FILENAME,
help='Configuration filename',
)
parser.add_argument(
'--excludes',
dest='excludes_filename',
default=DEFAULT_EXCLUDES_FILENAME if os.path.exists(DEFAULT_EXCLUDES_FILENAME) else None,
help='Excludes filename',
)
parser.add_argument(
'-v', '--verbosity',
type=int,
help='Display verbose progress (1 for some, 2 for lots)',
)

return parser.parse_args(arguments)


def main():
try:
args = parse_arguments(*sys.argv[1:])
config = parse_configuration(args.config_filename, CONFIG_FORMAT)
repository = config.location['repository']

borg.initialize(config.storage)
borg.create_archive(
args.excludes_filename, args.verbosity, config.storage, **config.location
)
borg.prune_archives(args.verbosity, repository, config.retention)
borg.check_archives(args.verbosity, repository, config.consistency)
except (ValueError, IOError, CalledProcessError) as error:
print(error, file=sys.stderr)
sys.exit(1)

atticmatic/config.py → borgmatic/config.py View File

@@ -20,6 +20,45 @@ def option(name, value_type=str, required=True):
return Config_option(name, value_type, required)


CONFIG_FORMAT = (
Section_format(
'location',
(
option('source_directories'),
option('one_file_system', value_type=bool, required=False),
option('repository'),
),
),
Section_format(
'storage',
(
option('encryption_passphrase', required=False),
option('compression', required=False),
option('umask', required=False),
),
),
Section_format(
'retention',
(
option('keep_within', required=False),
option('keep_hourly', int, required=False),
option('keep_daily', int, required=False),
option('keep_weekly', int, required=False),
option('keep_monthly', int, required=False),
option('keep_yearly', int, required=False),
option('prefix', required=False),
),
),
Section_format(
'consistency',
(
option('checks', required=False),
option('check_last', required=False),
),
)
)


def validate_configuration_format(parser, config_format):
'''
Given an open RawConfigParser and an expected config file format, validate that the parsed

atticmatic/backends/__init__.py → borgmatic/tests/__init__.py View File


atticmatic/tests/builtins.py → borgmatic/tests/builtins.py View File


atticmatic/tests/__init__.py → borgmatic/tests/integration/__init__.py View File


atticmatic/tests/integration/test_command.py → borgmatic/tests/integration/test_command.py View File

@@ -4,26 +4,23 @@ import sys
from flexmock import flexmock
import pytest

from atticmatic import command as module


COMMAND_NAME = 'foomatic'
from borgmatic import command as module


def test_parse_arguments_with_no_arguments_uses_defaults():
flexmock(os.path).should_receive('exists').and_return(True)

parser = module.parse_arguments(COMMAND_NAME)
parser = module.parse_arguments()

assert parser.config_filename == module.DEFAULT_CONFIG_FILENAME_PATTERN.format(COMMAND_NAME)
assert parser.excludes_filename == module.DEFAULT_EXCLUDES_FILENAME_PATTERN.format(COMMAND_NAME)
assert parser.config_filename == module.DEFAULT_CONFIG_FILENAME
assert parser.excludes_filename == module.DEFAULT_EXCLUDES_FILENAME
assert parser.verbosity == None


def test_parse_arguments_with_filename_arguments_overrides_defaults():
flexmock(os.path).should_receive('exists').and_return(True)

parser = module.parse_arguments(COMMAND_NAME, '--config', 'myconfig', '--excludes', 'myexcludes')
parser = module.parse_arguments('--config', 'myconfig', '--excludes', 'myexcludes')

assert parser.config_filename == 'myconfig'
assert parser.excludes_filename == 'myexcludes'
@@ -33,9 +30,9 @@ def test_parse_arguments_with_filename_arguments_overrides_defaults():
def test_parse_arguments_with_missing_default_excludes_file_sets_filename_to_none():
flexmock(os.path).should_receive('exists').and_return(False)

parser = module.parse_arguments(COMMAND_NAME)
parser = module.parse_arguments()

assert parser.config_filename == module.DEFAULT_CONFIG_FILENAME_PATTERN.format(COMMAND_NAME)
assert parser.config_filename == module.DEFAULT_CONFIG_FILENAME
assert parser.excludes_filename == None
assert parser.verbosity == None

@@ -43,9 +40,9 @@ def test_parse_arguments_with_missing_default_excludes_file_sets_filename_to_non
def test_parse_arguments_with_missing_overridden_excludes_file_retains_filename():
flexmock(os.path).should_receive('exists').and_return(False)

parser = module.parse_arguments(COMMAND_NAME, '--excludes', 'myexcludes')
parser = module.parse_arguments('--excludes', 'myexcludes')

assert parser.config_filename == module.DEFAULT_CONFIG_FILENAME_PATTERN.format(COMMAND_NAME)
assert parser.config_filename == module.DEFAULT_CONFIG_FILENAME
assert parser.excludes_filename == 'myexcludes'
assert parser.verbosity == None

@@ -53,10 +50,10 @@ def test_parse_arguments_with_missing_overridden_excludes_file_retains_filename(
def test_parse_arguments_with_verbosity_flag_overrides_default():
flexmock(os.path).should_receive('exists').and_return(True)

parser = module.parse_arguments(COMMAND_NAME, '--verbosity', '1')
parser = module.parse_arguments('--verbosity', '1')

assert parser.config_filename == module.DEFAULT_CONFIG_FILENAME_PATTERN.format(COMMAND_NAME)
assert parser.excludes_filename == module.DEFAULT_EXCLUDES_FILENAME_PATTERN.format(COMMAND_NAME)
assert parser.config_filename == module.DEFAULT_CONFIG_FILENAME
assert parser.excludes_filename == module.DEFAULT_EXCLUDES_FILENAME
assert parser.verbosity == 1


@@ -67,6 +64,6 @@ def test_parse_arguments_with_invalid_arguments_exits():

try:
with pytest.raises(SystemExit):
module.parse_arguments(COMMAND_NAME, '--posix-me-harder')
module.parse_arguments('--posix-me-harder')
finally:
sys.stderr = original_stderr

atticmatic/tests/integration/test_config.py → borgmatic/tests/integration/test_config.py View File

@@ -8,7 +8,7 @@ except ImportError:
from collections import OrderedDict
import string

from atticmatic import config as module
from borgmatic import config as module


def test_parse_section_options_with_punctuation_should_return_section_options():

atticmatic/tests/integration/test_version.py → borgmatic/tests/integration/test_version.py View File


atticmatic/tests/integration/__init__.py → borgmatic/tests/unit/__init__.py View File


atticmatic/tests/unit/backends/test_shared.py → borgmatic/tests/unit/test_borg.py View File

@@ -4,9 +4,9 @@ import os

from flexmock import flexmock

from atticmatic.backends import shared as module
from atticmatic.tests.builtins import builtins_mock
from atticmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
from borgmatic import borg as module
from borgmatic.tests.builtins import builtins_mock
from borgmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS


def test_initialize_with_passphrase_should_set_environment():

atticmatic/tests/unit/test_config.py → borgmatic/tests/unit/test_config.py View File

@@ -3,7 +3,7 @@ from collections import OrderedDict
from flexmock import flexmock
import pytest

from atticmatic import config as module
from borgmatic import config as module


def test_option_should_create_config_option():

atticmatic/verbosity.py → borgmatic/verbosity.py View File


+ 0
- 3
sample/atticmatic.cron View File

@@ -1,3 +0,0 @@
# You can drop this file into /etc/cron.d/ to run atticmatic nightly.

0 3 * * * root PATH=$PATH:/usr/local/bin /usr/local/bin/atticmatic

+ 7
- 8
sample/config View File

@@ -8,22 +8,21 @@ source_directories: /home /etc /var/log/syslog*
#one_file_system: True

# Path to local or remote repository.
repository: user@backupserver:sourcehostname.attic
repository: user@backupserver:sourcehostname.borg

[storage]
# Passphrase to unlock the encryption key with. Only use on repositories that
# were initialized with passphrase/repokey encryption.
#encryption_passphrase: foo
# For Borg only, you can specify the type of compression to use when creating
# archives. See https://borgbackup.readthedocs.org/en/stable/usage.html#borg-create
# Type of compression to use when creating archives. See
# https://borgbackup.readthedocs.org/en/stable/usage.html#borg-create
# for details. Defaults to no compression.
#compression: lz4
# For Borg only, you can specify the umask to be used for borg create.
# Umask to be used for borg create.
#umask: 0740

[retention]
# Retention policy for how many backups to keep in each category. See
# https://attic-backup.org/usage.html#attic-prune or
# https://borgbackup.readthedocs.org/en/stable/usage.html#borg-prune for details.
#keep_within: 3H
#keep_hourly: 24
@@ -36,8 +35,8 @@ keep_yearly: 1
[consistency]
# Space-separated list of consistency checks to run: "repository", "archives",
# or both. Defaults to both. Set to "disabled" to disable all consistency
# checks. See https://attic-backup.org/usage.html#attic-check or
# https://borgbackup.readthedocs.org/en/stable/usage.html#borg-check for details.
# checks. See https://borgbackup.readthedocs.org/en/stable/usage.html#borg-check
# for details.
checks: repository archives
# For Borg only, you can restrict the number of checked archives to the last n.
# Restrict the number of checked archives to the last n.
#check_last: 3

+ 9
- 7
setup.py View File

@@ -1,17 +1,17 @@
from setuptools import setup, find_packages


VERSION = '0.1.8'
VERSION = '1.0.0'


setup(
name='atticmatic',
name='borgmatic',
version=VERSION,
description='A wrapper script for Attic/Borg backup software that creates and prunes backups',
description='A wrapper script for Borg backup software that creates and prunes backups',
author='Dan Helfman',
author_email='witten@torsion.org',
url='https://torsion.org/atticmatic',
download_url='https://torsion.org/hg/atticmatic/archive/%s.tar.gz' % VERSION,
url='https://torsion.org/borgmatic',
download_url='https://torsion.org/hg/borgmatic/archive/%s.tar.gz' % VERSION,
classifiers=(
'Development Status :: 5 - Production/Stable',
'Environment :: Console',
@@ -24,10 +24,12 @@ setup(
packages=find_packages(),
entry_points={
'console_scripts': [
'atticmatic = atticmatic.command:main',
'borgmatic = atticmatic.command:main',
'borgmatic = borgmatic.command:main',
]
},
obsoletes=(
'atticmatic',
),
tests_require=(
'flexmock',
'pytest',

Loading…
Cancel
Save