#17: Added command-line flags for performing a borgmatic run with only pruning, creating, or checking enabled.

.hgignore

@@ -2,6 +2,8 @@ syntax: glob
 *.egg-info
 *.pyc
 *.swp
+.cache
+.coverage
 .tox
 build
 dist

.hgtags

@@ -29,3 +29,8 @@ dbc96d3f83bd5570b6826537616d4160b3374836 0.1.8
 0e1fbee9358de4f062fa9539e1355db83db70caa 1.0.0
 de2d7721cdec93a52d20222a9ddd579ed93c1017 1.0.1
 9603d13910b32d57a887765cab694ac5d0acc1f4 1.0.2
+32c6341dda9fad77a3982641bce8a3a45821842e 1.0.3
+5a003056a8ff4709c5bd4d6d33354199423f8a1d 1.1.0
+7d3d11eff6c0773883c48f221431f157bc7995eb 1.1.1
+f052a77a8ad5a0fea7fa86a902e0e401252f7d80 1.1.2
+3f838f661546e04529b453aa443529b432afc243 1.1.3

MANIFEST.in

@@ -0,0 +1 @@
+include borgmatic/config/schema.yaml

NEWS

@@ -1,3 +1,36 @@
+1.1.4
+
+ * #17: Added command-line flags for performing a borgmatic run with only pruning, creating, or
+   checking enabled. This supports use cases like running consistency checks from a different cron
+   job with a different frequency, or running pruning with a different verbosity level.
+
+1.1.3
+
+ * #14: Support for running multiple config files in /etc/borgmatic.d/ from a single borgmatic run.
+ * Fix for generate-borgmatic-config writing config with invalid one_file_system value.
+
+1.1.2
+
+ * #32: Fix for passing check_last as integer to subprocess when calling Borg.
+
+1.1.1
+
+ * Part of #32: Fix for upgrade-borgmatic-config converting check_last option as a string instead of
+   an integer.
+ * Fix for upgrade-borgmatic-config erroring when consistency checks option is not present.
+
+1.1.0
+
+ * Switched config file format to YAML. Run upgrade-borgmatic-config to upgrade.
+ * Added generate-borgmatic-config command for initial config creation.
+ * Dropped Python 2 support. Now Python 3 only.
+ * #18: Fix for README mention of sample files not included in package.
+ * #22: Sample files for triggering borgmatic from a systemd timer.
+ * Support for backing up to multiple repositories.
+ * To free up space, now pruning backups prior to creating a new backup.
+ * Enabled test coverage output during tox runs.
+ * Added logo.
+
 1.0.3
 
  * #21: Fix for verbosity flag not actually causing verbose output.

README.md

@@ -1,4 +1,6 @@
-title: Borgmatic
+title: borgmatic
+
+<img src="static/borgmatic.svg" alt="borgmatic logo" style="width: 8em; float: right; padding-left: 1em;" />
 
 ## Overview
 
@@ -11,36 +13,42 @@ all on the command-line, and handles common errors.
 
 Here's an example config file:
 
-```INI
-[location]
-# Space-separated list of source directories to backup.
-# Globs are expanded.
-source_directories: /home /etc /var/log/syslog*
+```yaml
+location:
+    # List of source directories to backup. Globs are expanded.
+    source_directories:
+        - /home
+        - /etc
+        - /var/log/syslog*
 
-# Path to local or remote backup repository.
-repository: user@backupserver:sourcehostname.borg
+    # Paths to local or remote repositories.
+    repositories:
+        - user@backupserver:sourcehostname.borg
 
-[retention]
-# Retention policy for how many backups to keep in each category.
-keep_daily: 7
-keep_weekly: 4
-keep_monthly: 6
+    # Any paths matching these patterns are excluded from backups.
+    exclude_patterns:
+        - /home/*/.cache
 
-[consistency]
-# Consistency checks to run, or "disabled" to prevent checks.
-checks: repository archives
+retention:
+    # Retention policy for how many backups to keep in each category.
+    keep_daily: 7
+    keep_weekly: 4
+    keep_monthly: 6
+
+consistency:
+    # List of consistency checks to run: "repository", "archives", or both.
+    checks:
+        - repository
+        - archives
 ```
 
-Additionally, exclude patterns can be specified in a separate excludes config
-file, one pattern per line.
-
 borgmatic is hosted at <https://torsion.org/borgmatic> with [source code
 available](https://torsion.org/hg/borgmatic). It's also mirrored on
 [GitHub](https://github.com/witten/borgmatic) and
 [BitBucket](https://bitbucket.org/dhelfman/borgmatic) for convenience.
 
 
-## Setup
+## Installation
 
 To get up and running, follow the [Borg Quick
 Start](https://borgbackup.readthedocs.org/en/latest/quickstart.html) to create
@@ -57,16 +65,86 @@ To install borgmatic, run the following command to download and install it:
 
     sudo pip install --upgrade borgmatic
 
-Then, copy the following configuration files:
+Make sure you're using Python 3, as borgmatic does not support Python 2. (You
+may have to use "pip3" or similar instead of "pip".)
 
-    sudo cp sample/borgmatic.cron /etc/cron.d/borgmatic
-    sudo mkdir /etc/borgmatic/
-    sudo cp sample/config sample/excludes /etc/borgmatic/
+## Configuration
 
-Lastly, modify the /etc files with your desired configuration.
+After you install borgmatic, generate a sample configuration file:
+
+    sudo generate-borgmatic-config
+
+This generates a sample configuration file at /etc/borgmatic/config.yaml (by
+default). You should edit the file to suit your needs, as the values are just
+representative. All fields are optional except where indicated, so feel free
+to remove anything you don't need.
 
 
-## Upgrading from atticmatic
+### Multiple configuration files
+
+A more advanced usage is to create multiple separate configuration files and
+place each one in an /etc/borgmatic.d directory. For instance:
+
+    sudo mkdir /etc/borgmatic.d
+    sudo generate-borgmatic-config --destination /etc/borgmatic.d/app1.yaml
+    sudo generate-borgmatic-config --destination /etc/borgmatic.d/app2.yaml
+
+With this approach, you can have entirely different backup policies for
+different applications on your system. For instance, you may want one backup
+configuration for your database data directory, and a different configuration
+for your user home directories.
+
+When you set up multiple configuration files like this, borgmatic will run
+each one in turn from a single borgmatic invocation. This includes, by
+default, the traditional /etc/borgmatic/config.yaml as well.
+
+
+## Upgrading
+
+In general, all you should need to do to upgrade borgmatic is run the
+following:
+
+    sudo pip install --upgrade borgmatic
+
+(You may have to use "pip3" or similar instead of "pip", so Python 3 gets
+used.)
+
+However, see below about special cases.
+
+
+### Upgrading from borgmatic 1.0.x
+
+borgmatic changed its configuration file format in version 1.1.0 from
+INI-style to YAML. This better supports validation, and has a more natural way
+to express lists of values. To upgrade your existing configuration, first
+upgrade to the new version of borgmatic:
+
+As of version 1.1.0, borgmatic no longer supports Python 2. If you were
+already running borgmatic with Python 3, then you can simply upgrade borgmatic
+in-place:
+
+    sudo pip install --upgrade borgmatic
+
+But if you were running borgmatic with Python 2, uninstall and reinstall instead:
+
+    sudo pip uninstall borgmatic
+    sudo pip3 install borgmatic
+
+The pip binary names for different versions of Python can differ, so the above
+commands may need some tweaking to work on your machine.
+
+
+Once borgmatic is upgraded, run:
+
+    sudo upgrade-borgmatic-config
+
+That will generate a new YAML configuration file at /etc/borgmatic/config.yaml
+(by default) using the values from both your existing configuration and
+excludes files. The new version of borgmatic will consume the YAML
+configuration file instead of the old one.
+
+
+### Upgrading from atticmatic
 
 You can ignore this section if you're not an atticmatic user (the former name
 of borgmatic).
@@ -86,6 +164,7 @@ from atticmatic to borgmatic. Simply run the following commands:
 That's it! borgmatic will continue using your /etc/borgmatic configuration
 files.
 
+
 ## Usage
 
 You can run borgmatic and start a backup simply by invoking it without
@@ -96,6 +175,12 @@ arguments:
 This will also prune any old backups as per the configured retention policy,
 and check backups for consistency problems due to things like file damage.
 
+If you'd like to see the available command-line arguments, view the help:
+
+    borgmatic --help
+
+### Verbosity
+
 By default, the backup will proceed silently except in the case of errors. But
 if you'd like to to get additional information about the progress of the
 backup as it proceeds, use the verbosity option:
@@ -106,9 +191,52 @@ Or, for even more progress spew:
 
     borgmatic --verbosity 2
 
-If you'd like to see the available command-line arguments, view the help:
+### À la carte
 
-    borgmatic --help
+If you want to run borgmatic with only pruning, creating, or checking enabled,
+the following optional flags are available:
+
+    borgmatic --prune
+    borgmatic --create
+    borgmatic --check
+
+You can run with only one of these flags provided, or you can mix and match
+any number of them. This supports use cases like running consistency checks
+from a different cron job with a different frequency, or running pruning with
+a different verbosity level.
+
+
+## Autopilot
+
+If you want to run borgmatic automatically, say once a day, the you can
+configure a job runner to invoke it periodically.
+
+### cron
+
+If you're using cron, download the [sample cron
+file](https://torsion.org/hg/borgmatic/raw-file/tip/sample/cron/borgmatic).
+Then, from the directory where you downloaded it:
+
+    sudo mv borgmatic /etc/cron.d/borgmatic
+    sudo chmod +x /etc/cron.d/borgmatic
+
+You can modify the cron file if you'd like to run borgmatic more or less frequently.
+
+### systemd
+
+If you're using systemd instead of cron to run jobs, download the [sample
+systemd service
+file](https://torsion.org/hg/borgmatic/raw-file/tip/sample/systemd/borgmatic.service)
+and the [sample systemd timer
+file](https://torsion.org/hg/borgmatic/raw-file/tip/sample/systemd/borgmatic.timer).
+Then, from the directory where you downloaded them:
+
+    sudo mv borgmatic.service borgmatic.timer /etc/systemd/system/
+    sudo systemctl enable borgmatic.timer
+    sudo systemctl start borgmatic.timer
+
+Feel free to modify the timer file based on how frequently you'd like
+borgmatic to run.
 
 
 ## Running tests
@@ -143,6 +271,17 @@ This should make the client keep the connection alive while validating
 backups.
 
 
+### libyaml compilation errors
+
+borgmatic depends on a Python YAML library (ruamel.yaml) that will optionally
+use a C YAML library (libyaml) if present. But if it's not installed, then
+when installing or upgrading borgmatic, you may see errors about compiling the
+YAML library. If so, not to worry. borgmatic should install and function
+correctly even without the C YAML library. And borgmatic won't be any faster
+with the C library present, so you don't need to go out of your way to install
+it.
+
+
 ## Issues and feedback
 
 Got an issue or an idea for a feature enhancement? Check out the [borgmatic

borgmatic/borg.py

@@ -1,10 +1,11 @@
 from datetime import datetime
+import glob
+import itertools
 import os
-import re
 import platform
+import re
 import subprocess
-from glob import glob
-from itertools import chain
+import tempfile
 
 from borgmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
 
@@ -22,23 +23,43 @@ def initialize(storage_config, command=COMMAND):
         os.environ['{}_PASSPHRASE'.format(command.upper())] = passphrase
 
 
+def _write_exclude_file(exclude_patterns=None):
+    '''
+    Given a sequence of exclude patterns, write them to a named temporary file and return it. Return
+    None if no patterns are provided.
+    '''
+    if not exclude_patterns:
+        return None
+
+    exclude_file = tempfile.NamedTemporaryFile('w')
+    exclude_file.write('\n'.join(exclude_patterns))
+    exclude_file.flush()
+
+    return exclude_file
+
+
 def create_archive(
-    excludes_filename, verbosity, storage_config, source_directories, repository, command=COMMAND,
-    one_file_system=None, remote_path=None,
+    verbosity, repository, location_config, storage_config, command=COMMAND,
 ):
     '''
-    Given an excludes filename (or None), a vebosity flag, a storage config dict, a space-separated
-    list of source directories, a local or remote repository path, and a command to run, create an
-    attic archive.
+    Given a vebosity flag, a storage config dict, a list of source directories, a local or remote
+    repository path, a list of exclude patterns, and a command to run, create an attic archive.
     '''
-    sources = re.split('\s+', source_directories)
-    sources = tuple(chain.from_iterable(glob(x) or [x] for x in sources))
-    exclude_flags = ('--exclude-from', excludes_filename) if excludes_filename else ()
+    sources = tuple(
+        itertools.chain.from_iterable(
+            glob.glob(directory) or [directory]
+            for directory in location_config['source_directories']
+        )
+    )
+
+    exclude_file = _write_exclude_file(location_config.get('exclude_patterns'))
+    exclude_flags = ('--exclude-from', exclude_file.name) if exclude_file else ()
     compression = storage_config.get('compression', None)
     compression_flags = ('--compression', compression) if compression else ()
     umask = storage_config.get('umask', None)
     umask_flags = ('--umask', str(umask)) if umask else ()
-    one_file_system_flags = ('--one-file-system',) if one_file_system else ()
+    one_file_system_flags = ('--one-file-system',) if location_config.get('one_file_system') else ()
+    remote_path = location_config.get('remote_path')
     remote_path_flags = ('--remote-path', remote_path) if remote_path else ()
     verbosity_flags = {
         VERBOSITY_SOME: ('--info', '--stats',),
@@ -109,12 +130,11 @@ DEFAULT_CHECKS = ('repository', 'archives')
 
 def _parse_checks(consistency_config):
     '''
-    Given a consistency config with a space-separated "checks" option, transform it to a tuple of
-    named checks to run.
+    Given a consistency config with a "checks" list, transform it to a tuple of named checks to run.
 
     For example, given a retention config of:
 
-        {'checks': 'repository archives'}
+        {'checks': ['repository', 'archives']}
 
     This will be returned as:
 
@@ -123,14 +143,11 @@ def _parse_checks(consistency_config):
     If no "checks" option is present, return the DEFAULT_CHECKS. If the checks value is the string
     "disabled", return an empty tuple, meaning that no checks should be run.
     '''
-    checks = consistency_config.get('checks', '').strip()
-    if not checks:
-        return DEFAULT_CHECKS
+    checks = consistency_config.get('checks', [])
+    if checks == ['disabled']:
+        return ()
 
-    return tuple(
-        check for check in consistency_config['checks'].split(' ')
-        if check.lower() not in ('disabled', '')
-    )
+    return tuple(check for check in checks if check.lower() not in ('disabled', '')) or DEFAULT_CHECKS
 
 
 def _make_check_flags(checks, check_last=None):
@@ -145,10 +162,9 @@ def _make_check_flags(checks, check_last=None):
 
         ('--repository-only',)
 
-    Additionally, if a check_last value is given, a "--last" flag will be added. Note that only
-    Borg supports this flag.
+    Additionally, if a check_last value is given, a "--last" flag will be added.
     '''
-    last_flag = ('--last', check_last) if check_last else ()
+    last_flag = ('--last', str(check_last)) if check_last else ()
     if checks == DEFAULT_CHECKS:
         return last_flag
 

borgmatic/command.py

@@ -1,58 +0,0 @@
-from __future__ import print_function
-from argparse import ArgumentParser
-import os
-from subprocess import CalledProcessError
-import sys
-
-from borgmatic import borg
-from borgmatic.config import parse_configuration, CONFIG_FORMAT
-
-
-DEFAULT_CONFIG_FILENAME = '/etc/borgmatic/config'
-DEFAULT_EXCLUDES_FILENAME = '/etc/borgmatic/excludes'
-
-
-def parse_arguments(*arguments):
-    '''
-    Given the name of the command with which this script was invoked and command-line arguments,
-    parse the arguments and return them as an ArgumentParser instance. Use the command name to
-    determine the default configuration and excludes paths.
-    '''
-    parser = ArgumentParser()
-    parser.add_argument(
-        '-c', '--config',
-        dest='config_filename',
-        default=DEFAULT_CONFIG_FILENAME,
-        help='Configuration filename',
-    )
-    parser.add_argument(
-        '--excludes',
-        dest='excludes_filename',
-        default=DEFAULT_EXCLUDES_FILENAME if os.path.exists(DEFAULT_EXCLUDES_FILENAME) else None,
-        help='Excludes filename',
-    )
-    parser.add_argument(
-        '-v', '--verbosity',
-        type=int,
-        help='Display verbose progress (1 for some, 2 for lots)',
-    )
-
-    return parser.parse_args(arguments)
-
-
-def main():
-    try:
-        args = parse_arguments(*sys.argv[1:])
-        config = parse_configuration(args.config_filename, CONFIG_FORMAT)
-        repository = config.location['repository']
-        remote_path = config.location.get('remote_path')
-
-        borg.initialize(config.storage)
-        borg.create_archive(
-            args.excludes_filename, args.verbosity, config.storage, **config.location
-        )
-        borg.prune_archives(args.verbosity, repository, config.retention, remote_path=remote_path)
-        borg.check_archives(args.verbosity, repository, config.consistency, remote_path=remote_path)
-    except (ValueError, IOError, CalledProcessError) as error:
-        print(error, file=sys.stderr)
-        sys.exit(1)

borgmatic/commands/borgmatic.py

@@ -0,0 +1,111 @@
+from __future__ import print_function
+from argparse import ArgumentParser
+import os
+from subprocess import CalledProcessError
+import sys
+
+from borgmatic import borg
+from borgmatic.config import collect, convert, validate
+
+
+LEGACY_CONFIG_PATH = '/etc/borgmatic/config'
+DEFAULT_CONFIG_PATHS = ['/etc/borgmatic/config.yaml', '/etc/borgmatic.d']
+DEFAULT_EXCLUDES_PATH = '/etc/borgmatic/excludes'
+
+
+def parse_arguments(*arguments):
+    '''
+    Given command-line arguments with which this script was invoked, parse the arguments and return
+    them as an ArgumentParser instance.
+    '''
+    parser = ArgumentParser(
+        description=
+            '''
+            A simple wrapper script for the Borg backup software that creates and prunes backups.
+            If none of the --prune, --create, or --check options are given, then borgmatic defaults
+            to all three: prune, create, and check archives.
+            '''
+    )
+    parser.add_argument(
+        '-c', '--config',
+        nargs='+',
+        dest='config_paths',
+        default=DEFAULT_CONFIG_PATHS,
+        help='Configuration filenames or directories, defaults to: {}'.format(' '.join(DEFAULT_CONFIG_PATHS)),
+    )
+    parser.add_argument(
+        '--excludes',
+        dest='excludes_filename',
+        help='Deprecated in favor of exclude_patterns within configuration',
+    )
+    parser.add_argument(
+        '-p', '--prune',
+        dest='prune',
+        action='store_true',
+        help='Prune archives according to the retention policy',
+    )
+    parser.add_argument(
+        '-C', '--create',
+        dest='create',
+        action='store_true',
+        help='Create archives (actually perform backups)',
+    )
+    parser.add_argument(
+        '-k', '--check',
+        dest='check',
+        action='store_true',
+        help='Check archives for consistency',
+    )
+    parser.add_argument(
+        '-v', '--verbosity',
+        type=int,
+        help='Display verbose progress (1 for some, 2 for lots)',
+    )
+
+    args = parser.parse_args(arguments)
+
+    # If any of the three action flags in the given parse arguments have been explicitly requested,
+    # leave them as-is. Otherwise, assume defaults: Mutate the given arguments to enable all the
+    # actions.
+    if not args.prune and not args.create and not args.check:
+        args.prune = True
+        args.create = True
+        args.check = True
+
+    return args
+
+
+def main():  # pragma: no cover
+    try:
+        args = parse_arguments(*sys.argv[1:])
+        config_filenames = tuple(collect.collect_config_filenames(args.config_paths))
+        convert.guard_configuration_upgraded(LEGACY_CONFIG_PATH, config_filenames)
+
+        if len(config_filenames) == 0:
+            raise ValueError('Error: No configuration files found in: {}'.format(' '.join(args.config_paths)))
+
+        for config_filename in config_filenames:
+            config = validate.parse_configuration(config_filename, validate.schema_filename())
+            (location, storage, retention, consistency) = (
+                config.get(section_name, {})
+                for section_name in ('location', 'storage', 'retention', 'consistency')
+            )
+            remote_path = location.get('remote_path')
+
+            borg.initialize(storage)
+
+            for repository in location['repositories']:
+                if args.prune:
+                    borg.prune_archives(args.verbosity, repository, retention, remote_path=remote_path)
+                if args.create:
+                    borg.create_archive(
+                        args.verbosity,
+                        repository,
+                        location,
+                        storage,
+                    )
+                if args.check:
+                    borg.check_archives(args.verbosity, repository, consistency, remote_path=remote_path)
+    except (ValueError, OSError, CalledProcessError) as error:
+        print(error, file=sys.stderr)
+        sys.exit(1)

borgmatic/commands/convert_config.py

@@ -0,0 +1,99 @@
+from __future__ import print_function
+from argparse import ArgumentParser
+import os
+from subprocess import CalledProcessError
+import sys
+import textwrap
+
+from ruamel import yaml
+
+from borgmatic import borg
+from borgmatic.config import convert, generate, legacy, validate
+
+
+DEFAULT_SOURCE_CONFIG_FILENAME = '/etc/borgmatic/config'
+DEFAULT_SOURCE_EXCLUDES_FILENAME = '/etc/borgmatic/excludes'
+DEFAULT_DESTINATION_CONFIG_FILENAME = '/etc/borgmatic/config.yaml'
+
+
+def parse_arguments(*arguments):
+    '''
+    Given command-line arguments with which this script was invoked, parse the arguments and return
+    them as an ArgumentParser instance.
+    '''
+    parser = ArgumentParser(
+        description='''
+            Convert legacy INI-style borgmatic configuration and excludes files to a single YAML
+            configuration file. Note that this replaces any comments from the source files.
+        '''
+    )
+    parser.add_argument(
+        '-s', '--source-config',
+        dest='source_config_filename',
+        default=DEFAULT_SOURCE_CONFIG_FILENAME,
+        help='Source INI-style configuration filename. Default: {}'.format(DEFAULT_SOURCE_CONFIG_FILENAME),
+    )
+    parser.add_argument(
+        '-e', '--source-excludes',
+        dest='source_excludes_filename',
+        default=DEFAULT_SOURCE_EXCLUDES_FILENAME if os.path.exists(DEFAULT_SOURCE_EXCLUDES_FILENAME) else None,
+        help='Excludes filename',
+    )
+    parser.add_argument(
+        '-d', '--destination-config',
+        dest='destination_config_filename',
+        default=DEFAULT_DESTINATION_CONFIG_FILENAME,
+        help='Destination YAML configuration filename. Default: {}'.format(DEFAULT_DESTINATION_CONFIG_FILENAME),
+    )
+
+    return parser.parse_args(arguments)
+
+
+TEXT_WRAP_CHARACTERS = 80
+
+
+def display_result(args):  # pragma: no cover
+    result_lines = textwrap.wrap(
+        'Your borgmatic configuration has been upgraded. Please review the result in {}.'.format(
+            args.destination_config_filename
+        ),
+        TEXT_WRAP_CHARACTERS,
+    )
+    
+    delete_lines = textwrap.wrap(
+        'Once you are satisfied, you can safely delete {}{}.'.format(
+            args.source_config_filename,
+            ' and {}'.format(args.source_excludes_filename) if args.source_excludes_filename else '',
+        ),
+        TEXT_WRAP_CHARACTERS,
+    )
+
+    print('\n'.join(result_lines))
+    print()
+    print('\n'.join(delete_lines))
+
+
+def main():  # pragma: no cover
+    try:
+        args = parse_arguments(*sys.argv[1:])
+        schema = yaml.round_trip_load(open(validate.schema_filename()).read())
+        source_config = legacy.parse_configuration(args.source_config_filename, legacy.CONFIG_FORMAT)
+        source_config_file_mode = os.stat(args.source_config_filename).st_mode
+        source_excludes = (
+            open(args.source_excludes_filename).read().splitlines()
+            if args.source_excludes_filename
+            else []
+        )
+
+        destination_config = convert.convert_legacy_parsed_config(source_config, source_excludes, schema)
+
+        generate.write_configuration(
+            args.destination_config_filename,
+            destination_config,
+            mode=source_config_file_mode,
+        )
+
+        display_result(args)
+    except (ValueError, OSError) as error:
+        print(error, file=sys.stderr)
+        sys.exit(1)

borgmatic/commands/generate_config.py

@@ -0,0 +1,44 @@
+from __future__ import print_function
+from argparse import ArgumentParser
+import os
+from subprocess import CalledProcessError
+import sys
+
+from ruamel import yaml
+
+from borgmatic import borg
+from borgmatic.config import convert, generate, validate
+
+
+DEFAULT_DESTINATION_CONFIG_FILENAME = '/etc/borgmatic/config.yaml'
+
+
+def parse_arguments(*arguments):
+    '''
+    Given command-line arguments with which this script was invoked, parse the arguments and return
+    them as an ArgumentParser instance.
+    '''
+    parser = ArgumentParser(description='Generate a sample borgmatic YAML configuration file.')
+    parser.add_argument(
+        '-d', '--destination',
+        dest='destination_filename',
+        default=DEFAULT_DESTINATION_CONFIG_FILENAME,
+        help='Destination YAML configuration filename. Default: {}'.format(DEFAULT_DESTINATION_CONFIG_FILENAME),
+    )
+
+    return parser.parse_args(arguments)
+
+
+def main():  # pragma: no cover
+    try:
+        args = parse_arguments(*sys.argv[1:])
+
+        generate.generate_sample_configuration(args.destination_filename, validate.schema_filename())
+
+        print('Generated a sample configuration file at {}.'.format(args.destination_filename))
+        print()
+        print('Please edit the file to suit your needs. The values are just representative.')
+        print('All fields are optional except where indicated.')
+    except (ValueError, OSError) as error:
+        print(error, file=sys.stderr)
+        sys.exit(1)

borgmatic/config/collect.py

@@ -0,0 +1,27 @@
+import os
+
+
+def collect_config_filenames(config_paths):
+    '''
+    Given a sequence of config paths, both filenames and directories, resolve that to just an
+    iterable of files. Accomplish this by listing any given directories looking for contained config
+    files. This is non-recursive, so any directories within the given directories are ignored.
+
+    Return paths even if they don't exist on disk, so the user can find out about missing
+    configuration paths. However, skip /etc/borgmatic.d if it's missing, so the user doesn't have to
+    create it unless they need it.
+    '''
+    for path in config_paths:
+        exists = os.path.exists(path)
+
+        if os.path.realpath(path) == '/etc/borgmatic.d' and not exists:
+            continue
+
+        if not os.path.isdir(path) or not exists:
+            yield path
+            continue
+
+        for filename in os.listdir(path):
+            full_filename = os.path.join(path, filename)
+            if not os.path.isdir(full_filename):
+                yield full_filename

borgmatic/config/convert.py

@@ -0,0 +1,111 @@
+import os
+
+from ruamel import yaml
+
+from borgmatic.config import generate
+
+
+def _convert_section(source_section_config, section_schema):
+    '''
+    Given a legacy Parsed_config instance for a single section, convert it to its corresponding
+    yaml.comments.CommentedMap representation in preparation for actual serialization to YAML.
+
+    Where integer types exist in the given section schema, convert their values to integers.
+    '''
+    destination_section_config = yaml.comments.CommentedMap([
+        (
+            option_name,
+            int(option_value)
+                if section_schema['map'].get(option_name, {}).get('type') == 'int' else option_value
+        )
+        for option_name, option_value in source_section_config.items()
+    ])
+
+    return destination_section_config
+
+
+def convert_legacy_parsed_config(source_config, source_excludes, schema):
+    '''
+    Given a legacy Parsed_config instance loaded from an INI-style config file and a list of exclude
+    patterns, convert them to a corresponding yaml.comments.CommentedMap representation in
+    preparation for serialization to a single YAML config file.
+
+    Additionally, use the given schema as a source of helpful comments to include within the
+    returned CommentedMap.
+    '''
+    destination_config = yaml.comments.CommentedMap([
+        (section_name, _convert_section(section_config, schema['map'][section_name]))
+        for section_name, section_config in source_config._asdict().items()
+    ])
+
+    # Split space-seperated values into actual lists, make "repository" into a list, and merge in
+    # excludes.
+    location = destination_config['location']
+    location['source_directories'] = source_config.location['source_directories'].split(' ')
+    location['repositories'] = [location.pop('repository')]
+    location['exclude_patterns'] = source_excludes
+
+    if source_config.consistency.get('checks'):
+        destination_config['consistency']['checks'] = source_config.consistency['checks'].split(' ')
+
+    # Add comments to each section, and then add comments to the fields in each section.
+    generate.add_comments_to_configuration(destination_config, schema)
+
+    for section_name, section_config in destination_config.items():
+        generate.add_comments_to_configuration(
+            section_config,
+            schema['map'][section_name],
+            indent=generate.INDENT,
+        )
+
+    return destination_config
+
+
+class LegacyConfigurationNotUpgraded(FileNotFoundError):
+    def __init__(self):
+        super(LegacyConfigurationNotUpgraded, self).__init__(
+            '''borgmatic changed its configuration file format in version 1.1.0 from INI-style
+to YAML. This better supports validation, and has a more natural way to express
+lists of values. To upgrade your existing configuration, run:
+
+    sudo upgrade-borgmatic-config
+
+That will generate a new YAML configuration file at /etc/borgmatic/config.yaml
+(by default) using the values from both your existing configuration and excludes
+files. The new version of borgmatic will consume the YAML configuration file
+instead of the old one.'''
+        )
+
+
+def guard_configuration_upgraded(source_config_filename, destination_config_filenames):
+    '''
+    If legacy source configuration exists but no destination upgraded configs do, raise
+    LegacyConfigurationNotUpgraded.
+
+    The idea is that we want to alert the user about upgrading their config if they haven't already.
+    '''
+    destination_config_exists = any(
+        os.path.exists(filename)
+        for filename in destination_config_filenames
+    )
+
+    if os.path.exists(source_config_filename) and not destination_config_exists:
+        raise LegacyConfigurationNotUpgraded()
+
+
+class LegacyExcludesFilenamePresent(FileNotFoundError):
+    def __init__(self):
+        super(LegacyExcludesFilenamePresent, self).__init__(
+            '''borgmatic changed its configuration file format in version 1.1.0 from INI-style
+to YAML. This better supports validation, and has a more natural way to express
+lists of values. The new configuration file incorporates excludes, so you no
+longer need to provide an excludes filename on the command-line with an
+"--excludes" argument.
+
+Please remove the "--excludes" argument and run borgmatic again.'''
+        )
+
+
+def guard_excludes_filename_omitted(excludes_filename):
+    if excludes_filename != None:
+        raise LegacyExcludesFilenamePresent()

borgmatic/config/generate.py

@@ -0,0 +1,93 @@
+from collections import OrderedDict
+import os
+
+from ruamel import yaml
+
+
+INDENT = 4
+
+
+def _insert_newline_before_comment(config, field_name):
+    '''
+    Using some ruamel.yaml black magic, insert a blank line in the config right befor the given
+    field and its comments.
+    '''
+    config.ca.items[field_name][1].insert(
+        0,
+        yaml.tokens.CommentToken('\n', yaml.error.CommentMark(0), None),
+    )
+
+
+def _schema_to_sample_configuration(schema, level=0):
+    '''
+    Given a loaded configuration schema, generate and return sample config for it. Include comments
+    for each section based on the schema "desc" description.
+    '''
+    example = schema.get('example')
+    if example:
+        return example
+
+    config = yaml.comments.CommentedMap([
+        (
+            section_name,
+            _schema_to_sample_configuration(section_schema, level + 1),
+        )
+        for section_name, section_schema in schema['map'].items()
+    ])
+
+    add_comments_to_configuration(config, schema, indent=(level * INDENT))
+
+    return config
+
+
+def write_configuration(config_filename, config, mode=0o600):
+    '''
+    Given a target config filename and a config data structure of nested OrderedDicts, write out the
+    config to file as YAML. Create any containing directories as needed.
+    '''
+    if os.path.exists(config_filename):
+        raise FileExistsError('{} already exists. Aborting.'.format(config_filename))
+
+    try:
+        os.makedirs(os.path.dirname(config_filename), mode=0o700)
+    except (FileExistsError, FileNotFoundError):
+        pass
+
+    with open(config_filename, 'w') as config_file:
+        config_file.write(yaml.round_trip_dump(config, indent=INDENT, block_seq_indent=INDENT))
+
+    os.chmod(config_filename, mode)
+
+
+def add_comments_to_configuration(config, schema, indent=0):
+    '''
+    Using descriptions from a schema as a source, add those descriptions as comments to the given
+    config before each field. This function only adds comments for the top-most config map level.
+    Indent the comment the given number of characters.
+    '''
+    for index, field_name in enumerate(config.keys()):
+        field_schema = schema['map'].get(field_name, {})
+        description = field_schema.get('desc')
+
+        # No description to use? Skip it.
+        if not field_schema or not description:
+            continue
+
+        config.yaml_set_comment_before_after_key(
+            key=field_name,
+            before=description,
+            indent=indent,
+        )
+        if index > 0:
+            _insert_newline_before_comment(config, field_name)
+
+
+def generate_sample_configuration(config_filename, schema_filename):
+    '''
+    Given a target config filename and the path to a schema filename in pykwalify YAML schema
+    format, write out a sample configuration file based on that schema.
+    '''
+    schema = yaml.round_trip_load(open(schema_filename))
+    config = _schema_to_sample_configuration(schema)
+
+    write_configuration(config_filename, config)

borgmatic/config/legacy.py

@@ -1,11 +1,5 @@
 from collections import OrderedDict, namedtuple
-
-try:
-    # Python 2
-    from ConfigParser import RawConfigParser
-except ImportError:
-    # Python 3
-    from configparser import RawConfigParser
+from configparser import RawConfigParser
 
 
 Section_format = namedtuple('Section_format', ('name', 'options'))

borgmatic/config/schema.yaml

@@ -0,0 +1,126 @@
+name: Borgmatic configuration file schema
+version: 1
+map:
+    location:
+        desc: |
+            Where to look for files to backup, and where to store those backups. See
+            https://borgbackup.readthedocs.io/en/stable/quickstart.html and
+            https://borgbackup.readthedocs.io/en/stable/usage.html#borg-create for details.
+        required: true
+        map:
+            source_directories:
+                required: true
+                seq:
+                    - type: scalar
+                desc: List of source directories to backup (required). Globs are expanded.
+                example:
+                    - /home
+                    - /etc
+                    - /var/log/syslog*
+            one_file_system:
+                type: bool
+                desc: Stay in same file system (do not cross mount points).
+                example: true
+            remote_path:
+                type: scalar
+                desc: Alternate Borg remote executable. Defaults to "borg".
+                example: borg1
+            repositories:
+                required: true
+                seq:
+                    - type: scalar
+                desc: |
+                    Paths to local or remote repositories (required). Multiple repositories are
+                    backed up to in sequence.
+                example:
+                    - user@backupserver:sourcehostname.borg
+            exclude_patterns:
+                seq:
+                    - type: scalar
+                desc: |
+                    Any paths matching these patterns are excluded from backups. Globs are expanded.
+                    See https://borgbackup.readthedocs.io/en/stable/usage.html#borg-help-patterns
+                    for details.
+                example:
+                    - '*.pyc'
+                    - /home/*/.cache
+                    - /etc/ssl
+    storage:
+        desc: |
+            Repository storage options. See
+            https://borgbackup.readthedocs.io/en/stable/usage.html#borg-create and
+            https://borgbackup.readthedocs.io/en/stable/usage.html#environment-variables for details.
+        map:
+            encryption_passphrase:
+                type: scalar
+                desc: |
+                    Passphrase to unlock the encryption key with. Only use on repositories that were
+                    initialized with passphrase/repokey encryption. Quote the value if it contains
+                    punctuation, so it parses correctly. And backslash any quote or backslash
+                    literals as well.
+                example: "!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~"
+            compression:
+                type: scalar
+                desc: |
+                    Type of compression to use when creating archives. See
+                    https://borgbackup.readthedocs.org/en/stable/usage.html#borg-create for details.
+                    Defaults to no compression.
+                example: lz4
+            umask:
+                type: scalar
+                desc: Umask to be used for borg create.
+                example: 0077
+    retention:
+        desc: |
+            Retention policy for how many backups to keep in each category. See
+            https://borgbackup.readthedocs.org/en/stable/usage.html#borg-prune for details.
+        map:
+            keep_within:
+                type: scalar
+                desc: Keep all archives within this time interval.
+                example: 3H
+            keep_hourly:
+                type: int
+                desc: Number of hourly archives to keep.
+                example: 24
+            keep_daily:
+                type: int
+                desc: Number of daily archives to keep.
+                example: 7
+            keep_weekly:
+                type: int
+                desc: Number of weekly archives to keep.
+                example: 4
+            keep_monthly:
+                type: int
+                desc: Number of monthly archives to keep.
+                example: 6
+            keep_yearly:
+                type: int
+                desc: Number of yearly archives to keep.
+                example: 1
+            prefix:
+                type: scalar
+                desc: When pruning, only consider archive names starting with this prefix.
+                example: sourcehostname
+    consistency:
+        desc: |
+            Consistency checks to run after backups. See
+            https://borgbackup.readthedocs.org/en/stable/usage.html#borg-check for details.
+        map:
+            checks:
+                seq:
+                    - type: str
+                      enum: ['repository', 'archives', 'disabled']
+                      unique: true
+                desc: |
+                    List of consistency checks to run: "repository", "archives", or both. Defaults
+                    to both. Set to "disabled" to disable all consistency checks. See
+                    https://borgbackup.readthedocs.org/en/stable/usage.html#borg-check for details.
+                example:
+                    - repository
+                    - archives
+            check_last:
+                type: int
+                desc: Restrict the number of checked archives to the last n.
+                example: 3

borgmatic/config/validate.py

@@ -0,0 +1,74 @@
+import logging
+import sys
+import warnings
+
+import pkg_resources
+import pykwalify.core
+import pykwalify.errors
+from ruamel import yaml
+
+
+def schema_filename():
+    '''
+    Path to the installed YAML configuration schema file, used to validate and parse the
+    configuration.
+    '''
+    return pkg_resources.resource_filename('borgmatic', 'config/schema.yaml')
+
+
+class Validation_error(ValueError):
+    '''
+    A collection of error message strings generated when attempting to validate a particular
+    configurartion file.
+    '''
+    def __init__(self, config_filename, error_messages):
+        self.config_filename = config_filename
+        self.error_messages = error_messages
+
+
+def parse_configuration(config_filename, schema_filename):
+    '''
+    Given the path to a config filename in YAML format and the path to a schema filename in
+    pykwalify YAML schema format, return the parsed configuration as a data structure of nested
+    dicts and lists corresponding to the schema. Example return value:
+
+       {'location': {'source_directories': ['/home', '/etc'], 'repository': 'hostname.borg'},
+       'retention': {'keep_daily': 7}, 'consistency': {'checks': ['repository', 'archives']}}
+
+    Raise FileNotFoundError if the file does not exist, PermissionError if the user does not
+    have permissions to read the file, or Validation_error if the config does not match the schema.
+    '''
+    try:
+        config = yaml.round_trip_load(open(config_filename))
+        schema = yaml.round_trip_load(open(schema_filename))
+    except yaml.error.YAMLError as error:
+        raise Validation_error(config_filename, (str(error),))
+
+    # pykwalify gets angry if the example field is not a string. So rather than bend to its will,
+    # simply remove all examples before passing the schema to pykwalify.
+    for section_name, section_schema in schema['map'].items():
+        for field_name, field_schema in section_schema['map'].items():
+            field_schema.pop('example')
+
+    validator = pykwalify.core.Core(source_data=config, schema_data=schema)
+    parsed_result = validator.validate(raise_exception=False)
+
+    if validator.validation_errors:
+        raise Validation_error(config_filename, validator.validation_errors)
+
+    return parsed_result
+
+
+def display_validation_error(validation_error):
+    '''
+    Given a Validation_error, display its error messages to stderr.
+    '''
+    print(
+        'An error occurred while parsing a configuration file at {}:'.format(
+            validation_error.config_filename
+        ),
+        file=sys.stderr,
+    )
+
+    for error in validation_error.error_messages:
+        print(error, file=sys.stderr)

borgmatic/tests/builtins.py

@@ -1,11 +0,0 @@
-from flexmock import flexmock
-import sys
-
-
-def builtins_mock():
-    try:
-        # Python 2
-        return flexmock(sys.modules['__builtin__'])
-    except KeyError:
-        # Python 3
-        return flexmock(sys.modules['builtins'])

borgmatic/tests/integration/commands/test_borgmatic.py

@@ -0,0 +1,66 @@
+import os
+
+from flexmock import flexmock
+import pytest
+
+from borgmatic.commands import borgmatic as module
+
+
+def test_parse_arguments_with_no_arguments_uses_defaults():
+    parser = module.parse_arguments()
+
+    assert parser.config_paths == module.DEFAULT_CONFIG_PATHS
+    assert parser.excludes_filename == None
+    assert parser.verbosity is None
+
+
+def test_parse_arguments_with_path_arguments_overrides_defaults():
+    parser = module.parse_arguments('--config', 'myconfig', '--excludes', 'myexcludes')
+
+    assert parser.config_paths == ['myconfig']
+    assert parser.excludes_filename == 'myexcludes'
+    assert parser.verbosity is None
+
+
+def test_parse_arguments_with_multiple_config_paths_parses_as_list():
+    parser = module.parse_arguments('--config', 'myconfig', 'otherconfig')
+
+    assert parser.config_paths == ['myconfig', 'otherconfig']
+    assert parser.verbosity is None
+
+
+def test_parse_arguments_with_verbosity_flag_overrides_default():
+    parser = module.parse_arguments('--verbosity', '1')
+
+    assert parser.config_paths == module.DEFAULT_CONFIG_PATHS
+    assert parser.excludes_filename == None
+    assert parser.verbosity == 1
+
+
+def test_parse_arguments_with_no_actions_defaults_to_all_actions_enabled():
+    parser = module.parse_arguments()
+
+    assert parser.prune is True
+    assert parser.create is True
+    assert parser.check is True
+
+
+def test_parse_arguments_with_prune_action_leaves_other_actions_disabled():
+    parser = module.parse_arguments('--prune')
+
+    assert parser.prune is True
+    assert parser.create is False
+    assert parser.check is False
+
+
+def test_parse_arguments_with_multiple_actions_leaves_other_action_disabled():
+    parser = module.parse_arguments('--create', '--check')
+
+    assert parser.prune is False
+    assert parser.create is True
+    assert parser.check is True
+
+
+def test_parse_arguments_with_invalid_arguments_exits():
+    with pytest.raises(SystemExit):
+        module.parse_arguments('--posix-me-harder')

borgmatic/tests/integration/commands/test_convert_config.py

@@ -0,0 +1,47 @@
+import os
+
+from flexmock import flexmock
+import pytest
+
+from borgmatic.commands import convert_config as module
+
+
+def test_parse_arguments_with_no_arguments_uses_defaults():
+    flexmock(os.path).should_receive('exists').and_return(True)
+
+    parser = module.parse_arguments()
+
+    assert parser.source_config_filename == module.DEFAULT_SOURCE_CONFIG_FILENAME
+    assert parser.source_excludes_filename == module.DEFAULT_SOURCE_EXCLUDES_FILENAME
+    assert parser.destination_config_filename == module.DEFAULT_DESTINATION_CONFIG_FILENAME
+
+
+def test_parse_arguments_with_filename_arguments_overrides_defaults():
+    flexmock(os.path).should_receive('exists').and_return(True)
+
+    parser = module.parse_arguments(
+        '--source-config', 'config',
+        '--source-excludes', 'excludes',
+        '--destination-config', 'config.yaml',
+    )
+
+    assert parser.source_config_filename == 'config'
+    assert parser.source_excludes_filename == 'excludes'
+    assert parser.destination_config_filename == 'config.yaml'
+
+
+def test_parse_arguments_with_missing_default_excludes_file_sets_filename_to_none():
+    flexmock(os.path).should_receive('exists').and_return(False)
+
+    parser = module.parse_arguments()
+
+    assert parser.source_config_filename == module.DEFAULT_SOURCE_CONFIG_FILENAME
+    assert parser.source_excludes_filename is None
+    assert parser.destination_config_filename == module.DEFAULT_DESTINATION_CONFIG_FILENAME
+
+
+def test_parse_arguments_with_invalid_arguments_exits():
+    flexmock(os.path).should_receive('exists').and_return(True)
+
+    with pytest.raises(SystemExit):
+        module.parse_arguments('--posix-me-harder')

borgmatic/tests/integration/commands/test_generate_config.py

@@ -0,0 +1,12 @@
+from borgmatic.commands import generate_config as module
+
+
+def test_parse_arguments_with_no_arguments_uses_defaults():
+    parser = module.parse_arguments()
+
+    assert parser.destination_filename == module.DEFAULT_DESTINATION_CONFIG_FILENAME
+
+def test_parse_arguments_with_filename_argument_overrides_defaults():
+    parser = module.parse_arguments('--destination', 'config.yaml')
+
+    assert parser.destination_filename == 'config.yaml'

borgmatic/tests/integration/config/test_generate.py

@@ -0,0 +1,65 @@
+from io import StringIO
+import os
+import sys
+
+from flexmock import flexmock
+import pytest
+
+from borgmatic.config import generate as module
+
+
+def test_insert_newline_before_comment_does_not_raise():
+    field_name = 'foo'
+    config = module.yaml.comments.CommentedMap([(field_name, 33)])
+    config.yaml_set_comment_before_after_key(key=field_name, before='Comment',)
+
+    module._insert_newline_before_comment(config, field_name)
+
+
+def test_write_configuration_does_not_raise():
+    flexmock(os.path).should_receive('exists').and_return(False)
+    flexmock(os).should_receive('makedirs')
+    builtins = flexmock(sys.modules['builtins'])
+    builtins.should_receive('open').and_return(StringIO())
+    flexmock(os).should_receive('chmod')
+
+    module.write_configuration('config.yaml', {})
+
+
+def test_write_configuration_with_already_existing_file_raises():
+    flexmock(os.path).should_receive('exists').and_return(True)
+
+    with pytest.raises(FileExistsError):
+        module.write_configuration('config.yaml', {})
+
+
+def test_write_configuration_with_already_existing_directory_does_not_raise():
+    flexmock(os.path).should_receive('exists').and_return(False)
+    flexmock(os).should_receive('makedirs').and_raise(FileExistsError)
+    builtins = flexmock(sys.modules['builtins'])
+    builtins.should_receive('open').and_return(StringIO())
+    flexmock(os).should_receive('chmod')
+
+    module.write_configuration('config.yaml', {})
+
+
+def test_add_comments_to_configuration_does_not_raise():
+    # Ensure that it can deal with fields both in the schema and missing from the schema.
+    config = module.yaml.comments.CommentedMap([('foo', 33), ('bar', 44), ('baz', 55)])
+    schema = {
+        'map': {
+            'foo': {'desc': 'Foo'},
+            'bar': {'desc': 'Bar'},
+        }
+    }
+
+    module.add_comments_to_configuration(config, schema)
+
+
+def test_generate_sample_configuration_does_not_raise():
+    builtins = flexmock(sys.modules['builtins'])
+    builtins.should_receive('open').with_args('schema.yaml').and_return('')
+    flexmock(module).should_receive('write_configuration')
+    flexmock(module).should_receive('_schema_to_sample_configuration')
+
+    module.generate_sample_configuration('config.yaml', 'schema.yaml')

borgmatic/tests/integration/config/test_legacy.py

@@ -1,14 +1,9 @@
-try:
-    # Python 2
-    from cStringIO import StringIO
-except ImportError:
-    # Python 3
-    from io import StringIO
+from io import StringIO
 
 from collections import OrderedDict
 import string
 
-from borgmatic import config as module
+from borgmatic.config import legacy as module
 
 
 def test_parse_section_options_with_punctuation_should_return_section_options():

borgmatic/tests/integration/config/test_validate.py

@@ -0,0 +1,123 @@
+import io
+import string
+import sys
+import os
+
+from flexmock import flexmock
+import pytest
+
+from borgmatic.config import validate as module
+
+
+def test_schema_filename_returns_plausable_path():
+    schema_path = module.schema_filename()    
+
+    assert schema_path.endswith('/schema.yaml')
+
+
+def mock_config_and_schema(config_yaml):
+    '''
+    Set up mocks for the config config YAML string and the default schema so that the code under
+    test consumes them when parsing the configuration.
+    '''
+    config_stream = io.StringIO(config_yaml)
+    schema_stream = open(module.schema_filename())
+    builtins = flexmock(sys.modules['builtins'])
+    builtins.should_receive('open').with_args('config.yaml').and_return(config_stream)
+    builtins.should_receive('open').with_args('schema.yaml').and_return(schema_stream)
+
+
+def test_parse_configuration_transforms_file_into_mapping():
+    mock_config_and_schema(
+        '''
+        location:
+            source_directories:
+                - /home
+                - /etc
+
+            repositories:
+                - hostname.borg
+
+        retention:
+            keep_daily: 7
+
+        consistency:
+            checks:
+                - repository
+                - archives
+        '''
+    )
+
+    result = module.parse_configuration('config.yaml', 'schema.yaml')
+
+    assert result == {
+        'location': {'source_directories': ['/home', '/etc'], 'repositories': ['hostname.borg']},
+        'retention': {'keep_daily': 7},
+        'consistency': {'checks': ['repository', 'archives']},
+    }
+
+
+def test_parse_configuration_passes_through_quoted_punctuation():
+    escaped_punctuation = string.punctuation.replace('\\', r'\\').replace('"', r'\"')
+
+    mock_config_and_schema(
+        '''
+        location:
+            source_directories:
+                - /home
+
+            repositories:
+                - "{}.borg"
+        '''.format(escaped_punctuation)
+    )
+
+    result = module.parse_configuration('config.yaml', 'schema.yaml')
+
+    assert result == {
+        'location': {
+            'source_directories': ['/home'],
+            'repositories': ['{}.borg'.format(string.punctuation)],
+        },
+    }
+
+
+def test_parse_configuration_raises_for_missing_config_file():
+    with pytest.raises(FileNotFoundError):
+        module.parse_configuration('config.yaml', 'schema.yaml')
+
+
+def test_parse_configuration_raises_for_missing_schema_file():
+    mock_config_and_schema('')
+    builtins = flexmock(sys.modules['builtins'])
+    builtins.should_receive('open').with_args('schema.yaml').and_raise(FileNotFoundError)
+
+    with pytest.raises(FileNotFoundError):
+        module.parse_configuration('config.yaml', 'schema.yaml')
+
+
+def test_parse_configuration_raises_for_syntax_error():
+    mock_config_and_schema('foo:\nbar')
+
+    with pytest.raises(ValueError):
+        module.parse_configuration('config.yaml', 'schema.yaml')
+
+
+def test_parse_configuration_raises_for_validation_error():
+    mock_config_and_schema(
+        '''
+        location:
+            source_directories: yes
+            repositories:
+                - hostname.borg
+        '''
+    )
+
+    with pytest.raises(module.Validation_error):
+        module.parse_configuration('config.yaml', 'schema.yaml')
+
+
+def test_display_validation_error_does_not_raise():
+    flexmock(sys.modules['builtins']).should_receive('print')
+    error = module.Validation_error('config.yaml', ('oops', 'uh oh'))
+
+    module.display_validation_error(error)

borgmatic/tests/integration/test_command.py

@@ -1,69 +0,0 @@
-import os
-import sys
-
-from flexmock import flexmock
-import pytest
-
-from borgmatic import command as module
-
-
-def test_parse_arguments_with_no_arguments_uses_defaults():
-    flexmock(os.path).should_receive('exists').and_return(True)
-
-    parser = module.parse_arguments()
-
-    assert parser.config_filename == module.DEFAULT_CONFIG_FILENAME
-    assert parser.excludes_filename == module.DEFAULT_EXCLUDES_FILENAME
-    assert parser.verbosity == None
-
-
-def test_parse_arguments_with_filename_arguments_overrides_defaults():
-    flexmock(os.path).should_receive('exists').and_return(True)
-
-    parser = module.parse_arguments('--config', 'myconfig', '--excludes', 'myexcludes')
-
-    assert parser.config_filename == 'myconfig'
-    assert parser.excludes_filename == 'myexcludes'
-    assert parser.verbosity == None
-
-
-def test_parse_arguments_with_missing_default_excludes_file_sets_filename_to_none():
-    flexmock(os.path).should_receive('exists').and_return(False)
-
-    parser = module.parse_arguments()
-
-    assert parser.config_filename == module.DEFAULT_CONFIG_FILENAME
-    assert parser.excludes_filename == None
-    assert parser.verbosity == None
-
-
-def test_parse_arguments_with_missing_overridden_excludes_file_retains_filename():
-    flexmock(os.path).should_receive('exists').and_return(False)
-
-    parser = module.parse_arguments('--excludes', 'myexcludes')
-
-    assert parser.config_filename == module.DEFAULT_CONFIG_FILENAME
-    assert parser.excludes_filename == 'myexcludes'
-    assert parser.verbosity == None
-
-
-def test_parse_arguments_with_verbosity_flag_overrides_default():
-    flexmock(os.path).should_receive('exists').and_return(True)
-
-    parser = module.parse_arguments('--verbosity', '1')
-
-    assert parser.config_filename == module.DEFAULT_CONFIG_FILENAME
-    assert parser.excludes_filename == module.DEFAULT_EXCLUDES_FILENAME
-    assert parser.verbosity == 1
-
-
-def test_parse_arguments_with_invalid_arguments_exits():
-    flexmock(os.path).should_receive('exists').and_return(True)
-    original_stderr = sys.stderr
-    sys.stderr = sys.stdout
-
-    try:
-        with pytest.raises(SystemExit):
-            module.parse_arguments('--posix-me-harder')
-    finally:
-        sys.stderr = original_stderr

borgmatic/tests/unit/config/test_collect.py

@@ -0,0 +1,58 @@
+from flexmock import flexmock
+
+from borgmatic.config import collect as module
+
+
+def test_collect_config_filenames_collects_given_files():
+    config_paths = ('config.yaml', 'other.yaml')
+    flexmock(module.os.path).should_receive('isdir').and_return(False)
+
+    config_filenames = tuple(module.collect_config_filenames(config_paths))
+
+    assert config_filenames == config_paths
+
+
+def test_collect_config_filenames_collects_files_from_given_directories_and_ignores_sub_directories():
+    config_paths = ('config.yaml', '/etc/borgmatic.d')
+    mock_path = flexmock(module.os.path)
+    mock_path.should_receive('exists').and_return(True)
+    mock_path.should_receive('isdir').with_args('config.yaml').and_return(False)
+    mock_path.should_receive('isdir').with_args('/etc/borgmatic.d').and_return(True)
+    mock_path.should_receive('isdir').with_args('/etc/borgmatic.d/foo.yaml').and_return(False)
+    mock_path.should_receive('isdir').with_args('/etc/borgmatic.d/bar').and_return(True)
+    mock_path.should_receive('isdir').with_args('/etc/borgmatic.d/baz.yaml').and_return(False)
+    flexmock(module.os).should_receive('listdir').and_return(['foo.yaml', 'bar', 'baz.yaml'])
+
+    config_filenames = tuple(module.collect_config_filenames(config_paths))
+
+    assert config_filenames == (
+        'config.yaml',
+        '/etc/borgmatic.d/foo.yaml',
+        '/etc/borgmatic.d/baz.yaml',
+    )
+
+
+def test_collect_config_filenames_skips_etc_borgmatic_dot_d_if_it_does_not_exist():
+    config_paths = ('config.yaml', '/etc/borgmatic.d')
+    mock_path = flexmock(module.os.path)
+    mock_path.should_receive('exists').with_args('config.yaml').and_return(True)
+    mock_path.should_receive('exists').with_args('/etc/borgmatic.d').and_return(False)
+    mock_path.should_receive('isdir').with_args('config.yaml').and_return(False)
+    mock_path.should_receive('isdir').with_args('/etc/borgmatic.d').and_return(True)
+
+    config_filenames = tuple(module.collect_config_filenames(config_paths))
+
+    assert config_filenames == ('config.yaml',)
+
+
+def test_collect_config_filenames_includes_directory_if_it_does_not_exist():
+    config_paths = ('config.yaml', '/my/directory')
+    mock_path = flexmock(module.os.path)
+    mock_path.should_receive('exists').with_args('config.yaml').and_return(True)
+    mock_path.should_receive('exists').with_args('/my/directory').and_return(False)
+    mock_path.should_receive('isdir').with_args('config.yaml').and_return(False)
+    mock_path.should_receive('isdir').with_args('/my/directory').and_return(True)
+
+    config_filenames = tuple(module.collect_config_filenames(config_paths))
+
+    assert config_filenames == config_paths

borgmatic/tests/unit/config/test_convert.py

@@ -0,0 +1,118 @@
+from collections import defaultdict, OrderedDict, namedtuple
+import os
+
+from flexmock import flexmock
+import pytest
+
+from borgmatic.config import convert as module
+
+
+Parsed_config = namedtuple('Parsed_config', ('location', 'storage', 'retention', 'consistency'))
+
+
+def test_convert_section_generates_integer_value_for_integer_type_in_schema():
+    flexmock(module.yaml.comments).should_receive('CommentedMap').replace_with(OrderedDict)
+    source_section_config = OrderedDict([('check_last', '3')])
+    section_schema = {'map': {'check_last': {'type': 'int'}}}
+
+    destination_config = module._convert_section(source_section_config, section_schema)
+
+    assert destination_config == OrderedDict([('check_last', 3)])
+
+
+def test_convert_legacy_parsed_config_transforms_source_config_to_mapping():
+    flexmock(module.yaml.comments).should_receive('CommentedMap').replace_with(OrderedDict)
+    source_config = Parsed_config(
+        location=OrderedDict([('source_directories', '/home'), ('repository', 'hostname.borg')]),
+        storage=OrderedDict([('encryption_passphrase', 'supersecret')]),
+        retention=OrderedDict([('keep_daily', 7)]),
+        consistency=OrderedDict([('checks', 'repository')]),
+    )
+    source_excludes = ['/var']
+    schema = {'map': defaultdict(lambda: {'map': {}})}
+
+    destination_config = module.convert_legacy_parsed_config(source_config, source_excludes, schema)
+
+    assert destination_config == OrderedDict([
+        (
+            'location',
+            OrderedDict([
+                ('source_directories', ['/home']),
+                ('repositories', ['hostname.borg']),
+                ('exclude_patterns', ['/var']),
+            ]),
+        ),
+        ('storage', OrderedDict([('encryption_passphrase', 'supersecret')])),
+        ('retention', OrderedDict([('keep_daily', 7)])),
+        ('consistency', OrderedDict([('checks', ['repository'])])),
+    ])
+
+
+def test_convert_legacy_parsed_config_splits_space_separated_values():
+    flexmock(module.yaml.comments).should_receive('CommentedMap').replace_with(OrderedDict)
+    source_config = Parsed_config(
+        location=OrderedDict([('source_directories', '/home /etc'), ('repository', 'hostname.borg')]),
+        storage=OrderedDict(),
+        retention=OrderedDict(),
+        consistency=OrderedDict([('checks', 'repository archives')]),
+    )
+    source_excludes = ['/var']
+    schema = {'map': defaultdict(lambda: {'map': {}})}
+
+    destination_config = module.convert_legacy_parsed_config(source_config, source_excludes, schema) 
+
+    assert destination_config == OrderedDict([
+        (
+            'location',
+            OrderedDict([
+                ('source_directories', ['/home', '/etc']),
+                ('repositories', ['hostname.borg']),
+                ('exclude_patterns', ['/var']),
+            ]),
+        ),
+        ('storage', OrderedDict()),
+        ('retention', OrderedDict()),
+        ('consistency', OrderedDict([('checks', ['repository', 'archives'])])),
+    ])
+
+
+def test_guard_configuration_upgraded_raises_when_only_source_config_present():
+    flexmock(os.path).should_receive('exists').with_args('config').and_return(True)
+    flexmock(os.path).should_receive('exists').with_args('config.yaml').and_return(False)
+    flexmock(os.path).should_receive('exists').with_args('other.yaml').and_return(False)
+
+    with pytest.raises(module.LegacyConfigurationNotUpgraded):
+        module.guard_configuration_upgraded('config', ('config.yaml', 'other.yaml'))
+
+
+def test_guard_configuration_upgraded_does_not_raise_when_only_destination_config_present():
+    flexmock(os.path).should_receive('exists').with_args('config').and_return(False)
+    flexmock(os.path).should_receive('exists').with_args('config.yaml').and_return(False)
+    flexmock(os.path).should_receive('exists').with_args('other.yaml').and_return(True)
+
+    module.guard_configuration_upgraded('config', ('config.yaml', 'other.yaml'))
+
+
+def test_guard_configuration_upgraded_does_not_raise_when_both_configs_present():
+    flexmock(os.path).should_receive('exists').with_args('config').and_return(True)
+    flexmock(os.path).should_receive('exists').with_args('config.yaml').and_return(False)
+    flexmock(os.path).should_receive('exists').with_args('other.yaml').and_return(True)
+
+    module.guard_configuration_upgraded('config', ('config.yaml', 'other.yaml'))
+
+
+def test_guard_configuration_upgraded_does_not_raise_when_neither_config_present():
+    flexmock(os.path).should_receive('exists').with_args('config').and_return(False)
+    flexmock(os.path).should_receive('exists').with_args('config.yaml').and_return(False)
+    flexmock(os.path).should_receive('exists').with_args('other.yaml').and_return(False)
+
+    module.guard_configuration_upgraded('config', ('config.yaml', 'other.yaml'))
+
+
+def test_guard_excludes_filename_omitted_raises_when_filename_provided():
+    with pytest.raises(module.LegacyExcludesFilenamePresent):
+        module.guard_excludes_filename_omitted(excludes_filename='/etc/borgmatic/excludes')
+
+
+def test_guard_excludes_filename_omitted_does_not_raise_when_filename_not_provided():
+    module.guard_excludes_filename_omitted(excludes_filename=None)

borgmatic/tests/unit/config/test_generate.py

@@ -0,0 +1,49 @@
+from collections import OrderedDict
+
+from flexmock import flexmock
+
+from borgmatic.config import generate as module
+
+
+def test_schema_to_sample_configuration_generates_config_with_examples():
+    flexmock(module.yaml.comments).should_receive('CommentedMap').replace_with(OrderedDict)
+    flexmock(module).should_receive('add_comments_to_configuration')
+    schema = {
+        'map': OrderedDict([
+            (
+                'section1', {
+                    'map': {
+                        'field1': OrderedDict([
+                            ('example', 'Example 1')
+                        ]),
+                    },
+                },
+            ),
+            (
+                'section2', {
+                    'map': OrderedDict([
+                        ('field2', {'example': 'Example 2'}),
+                        ('field3', {'example': 'Example 3'}),
+                    ]),
+                }
+            ),
+        ])
+    }
+
+    config = module._schema_to_sample_configuration(schema)
+
+    assert config == OrderedDict([
+        (
+            'section1',
+            OrderedDict([
+                ('field1', 'Example 1'),
+            ]),
+        ),
+        (
+            'section2',
+            OrderedDict([
+                ('field2', 'Example 2'),
+                ('field3', 'Example 3'),
+            ]),
+        )
+    ])

borgmatic/tests/unit/config/test_legacy.py

@@ -3,7 +3,7 @@ from collections import OrderedDict
 from flexmock import flexmock
 import pytest
 
-from borgmatic import config as module
+from borgmatic.config import legacy as module
 
 
 def test_option_should_create_config_option():

borgmatic/tests/unit/test_borg.py

@@ -1,11 +1,11 @@
 from collections import OrderedDict
 from subprocess import STDOUT
+import sys
 import os
 
 from flexmock import flexmock
 
 from borgmatic import borg as module
-from borgmatic.tests.builtins import builtins_mock
 from borgmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
 
 
@@ -30,6 +30,20 @@ def test_initialize_without_passphrase_should_not_set_environment():
     finally:
         os.environ = orig_environ
 
+def test_write_exclude_file_does_not_raise():
+    temporary_file = flexmock(
+        name='filename',
+        write=lambda mode: None,
+        flush=lambda: None,
+    )
+    flexmock(module.tempfile).should_receive('NamedTemporaryFile').and_return(temporary_file)
+
+    module._write_exclude_file(['exclude'])
+
+
+def test_write_exclude_file_with_empty_exclude_patterns_does_not_raise():
+    module._write_exclude_file([])
+
 
 def insert_subprocess_mock(check_call_command, **kwargs):
     subprocess = flexmock(STDOUT=STDOUT)
@@ -53,191 +67,219 @@ def insert_datetime_mock():
     ).mock
 
 
-CREATE_COMMAND_WITHOUT_EXCLUDES = ('borg', 'create', 'repo::host-now', 'foo', 'bar')
-CREATE_COMMAND = CREATE_COMMAND_WITHOUT_EXCLUDES + ('--exclude-from', 'excludes')
+CREATE_COMMAND = ('borg', 'create', 'repo::host-now', 'foo', 'bar')
 
 
 def test_create_archive_should_call_borg_with_parameters():
+    flexmock(module).should_receive('_write_exclude_file')
     insert_subprocess_mock(CREATE_COMMAND)
     insert_platform_mock()
     insert_datetime_mock()
 
     module.create_archive(
-        excludes_filename='excludes',
         verbosity=None,
-        storage_config={},
-        source_directories='foo bar',
         repository='repo',
+        location_config={
+            'source_directories': ['foo', 'bar'],
+            'repositories': ['repo'],
+            'exclude_patterns': None,
+        },
+        storage_config={},
         command='borg',
     )
 
 
-def test_create_archive_with_two_spaces_in_source_directories():
-    insert_subprocess_mock(CREATE_COMMAND)
+def test_create_archive_with_exclude_patterns_should_call_borg_with_excludes():
+    flexmock(module).should_receive('_write_exclude_file').and_return(flexmock(name='excludes'))
+    insert_subprocess_mock(CREATE_COMMAND + ('--exclude-from', 'excludes'))
     insert_platform_mock()
     insert_datetime_mock()
 
     module.create_archive(
-        excludes_filename='excludes',
         verbosity=None,
-        storage_config={},
-        source_directories='foo  bar',
         repository='repo',
-        command='borg',
-    )
-
-
-def test_create_archive_with_none_excludes_filename_should_call_borg_without_excludes():
-    insert_subprocess_mock(CREATE_COMMAND_WITHOUT_EXCLUDES)
-    insert_platform_mock()
-    insert_datetime_mock()
-
-    module.create_archive(
-        excludes_filename=None,
-        verbosity=None,
+        location_config={
+            'source_directories': ['foo', 'bar'],
+            'repositories': ['repo'],
+            'exclude_patterns': ['exclude'],
+        },
         storage_config={},
-        source_directories='foo bar',
-        repository='repo',
         command='borg',
     )
 
 
 def test_create_archive_with_verbosity_some_should_call_borg_with_info_parameter():
+    flexmock(module).should_receive('_write_exclude_file')
     insert_subprocess_mock(CREATE_COMMAND + ('--info', '--stats',))
     insert_platform_mock()
     insert_datetime_mock()
 
     module.create_archive(
-        excludes_filename='excludes',
         verbosity=VERBOSITY_SOME,
-        storage_config={},
-        source_directories='foo bar',
         repository='repo',
+        location_config={
+            'source_directories': ['foo', 'bar'],
+            'repositories': ['repo'],
+            'exclude_patterns': None,
+        },
+        storage_config={},
         command='borg',
     )
 
 
 def test_create_archive_with_verbosity_lots_should_call_borg_with_debug_parameter():
+    flexmock(module).should_receive('_write_exclude_file')
     insert_subprocess_mock(CREATE_COMMAND + ('--debug', '--list', '--stats'))
     insert_platform_mock()
     insert_datetime_mock()
 
     module.create_archive(
-        excludes_filename='excludes',
         verbosity=VERBOSITY_LOTS,
-        storage_config={},
-        source_directories='foo bar',
         repository='repo',
+        location_config={
+            'source_directories': ['foo', 'bar'],
+            'repositories': ['repo'],
+            'exclude_patterns': None,
+        },
+        storage_config={},
         command='borg',
     )
 
 
 def test_create_archive_with_compression_should_call_borg_with_compression_parameters():
+    flexmock(module).should_receive('_write_exclude_file')
     insert_subprocess_mock(CREATE_COMMAND + ('--compression', 'rle'))
     insert_platform_mock()
     insert_datetime_mock()
 
     module.create_archive(
-        excludes_filename='excludes',
         verbosity=None,
-        storage_config={'compression': 'rle'},
-        source_directories='foo bar',
         repository='repo',
+        location_config={
+            'source_directories': ['foo', 'bar'],
+            'repositories': ['repo'],
+            'exclude_patterns': None,
+        },
+        storage_config={'compression': 'rle'},
         command='borg',
     )
 
 
 def test_create_archive_with_one_file_system_should_call_borg_with_one_file_system_parameters():
+    flexmock(module).should_receive('_write_exclude_file')
     insert_subprocess_mock(CREATE_COMMAND + ('--one-file-system',))
     insert_platform_mock()
     insert_datetime_mock()
 
     module.create_archive(
-        excludes_filename='excludes',
         verbosity=None,
-        storage_config={},
-        source_directories='foo bar',
         repository='repo',
+        location_config={
+            'source_directories': ['foo', 'bar'],
+            'repositories': ['repo'],
+            'one_file_system': True,
+            'exclude_patterns': None,
+        },
+        storage_config={},
         command='borg',
-        one_file_system=True,
     )
 
 
 def test_create_archive_with_remote_path_should_call_borg_with_remote_path_parameters():
+    flexmock(module).should_receive('_write_exclude_file')
     insert_subprocess_mock(CREATE_COMMAND + ('--remote-path', 'borg1'))
     insert_platform_mock()
     insert_datetime_mock()
 
     module.create_archive(
-        excludes_filename='excludes',
         verbosity=None,
-        storage_config={},
-        source_directories='foo bar',
         repository='repo',
+        location_config={
+            'source_directories': ['foo', 'bar'],
+            'repositories': ['repo'],
+            'remote_path': 'borg1',
+            'exclude_patterns': None,
+        },
+        storage_config={},
         command='borg',
-        remote_path='borg1',
     )
 
 
 def test_create_archive_with_umask_should_call_borg_with_umask_parameters():
+    flexmock(module).should_receive('_write_exclude_file')
     insert_subprocess_mock(CREATE_COMMAND + ('--umask', '740'))
     insert_platform_mock()
     insert_datetime_mock()
 
     module.create_archive(
-        excludes_filename='excludes',
         verbosity=None,
-        storage_config={'umask': 740},
-        source_directories='foo bar',
         repository='repo',
+        location_config={
+            'source_directories': ['foo', 'bar'],
+            'repositories': ['repo'],
+            'exclude_patterns': None,
+        },
+        storage_config={'umask': 740},
         command='borg',
     )
 
 
 def test_create_archive_with_source_directories_glob_expands():
+    flexmock(module).should_receive('_write_exclude_file')
     insert_subprocess_mock(('borg', 'create', 'repo::host-now', 'foo', 'food'))
     insert_platform_mock()
     insert_datetime_mock()
-    flexmock(module).should_receive('glob').with_args('foo*').and_return(['foo', 'food'])
+    flexmock(module.glob).should_receive('glob').with_args('foo*').and_return(['foo', 'food'])
 
     module.create_archive(
-        excludes_filename=None,
         verbosity=None,
-        storage_config={},
-        source_directories='foo*',
         repository='repo',
+        location_config={
+            'source_directories': ['foo*'],
+            'repositories': ['repo'],
+            'exclude_patterns': None,
+        },
+        storage_config={},
         command='borg',
     )
 
 
 def test_create_archive_with_non_matching_source_directories_glob_passes_through():
+    flexmock(module).should_receive('_write_exclude_file')
     insert_subprocess_mock(('borg', 'create', 'repo::host-now', 'foo*'))
     insert_platform_mock()
     insert_datetime_mock()
-    flexmock(module).should_receive('glob').with_args('foo*').and_return([])
+    flexmock(module.glob).should_receive('glob').with_args('foo*').and_return([])
 
     module.create_archive(
-        excludes_filename=None,
         verbosity=None,
-        storage_config={},
-        source_directories='foo*',
         repository='repo',
+        location_config={
+            'source_directories': ['foo*'],
+            'repositories': ['repo'],
+            'exclude_patterns': None,
+        },
+        storage_config={},
         command='borg',
     )
 
 
 def test_create_archive_with_glob_should_call_borg_with_expanded_directories():
+    flexmock(module).should_receive('_write_exclude_file')
     insert_subprocess_mock(('borg', 'create', 'repo::host-now', 'foo', 'food'))
     insert_platform_mock()
     insert_datetime_mock()
-    flexmock(module).should_receive('glob').with_args('foo*').and_return(['foo', 'food'])
+    flexmock(module.glob).should_receive('glob').with_args('foo*').and_return(['foo', 'food'])
 
     module.create_archive(
-        excludes_filename=None,
         verbosity=None,
-        storage_config={},
-        source_directories='foo*',
         repository='repo',
+        location_config={
+            'source_directories': ['foo*'],
+            'repositories': ['repo'],
+            'exclude_patterns': None,
+        },
+        storage_config={},
         command='borg',
     )
 
@@ -329,7 +371,7 @@ def test_prune_archive_with_remote_path_should_call_borg_with_remote_path_parame
 
 
 def test_parse_checks_returns_them_as_tuple():
-    checks = module._parse_checks({'checks': 'foo disabled bar'})
+    checks = module._parse_checks({'checks': ['foo', 'disabled', 'bar']})
 
     assert checks == ('foo', 'bar')
 
@@ -341,13 +383,13 @@ def test_parse_checks_with_missing_value_returns_defaults():
 
 
 def test_parse_checks_with_blank_value_returns_defaults():
-    checks = module._parse_checks({'checks': ''})
+    checks = module._parse_checks({'checks': []})
 
     assert checks == module.DEFAULT_CHECKS
 
 
 def test_parse_checks_with_disabled_returns_no_checks():
-    checks = module._parse_checks({'checks': 'disabled'})
+    checks = module._parse_checks({'checks': ['disabled']})
 
     assert checks == ()
 
@@ -367,13 +409,13 @@ def test_make_check_flags_with_default_checks_returns_no_flags():
 def test_make_check_flags_with_checks_and_last_returns_flags_including_last():
     flags = module._make_check_flags(('foo', 'bar'), check_last=3)
 
-    assert flags == ('--foo-only', '--bar-only', '--last', 3)
+    assert flags == ('--foo-only', '--bar-only', '--last', '3')
 
 
 def test_make_check_flags_with_last_returns_last_flag():
     flags = module._make_check_flags(module.DEFAULT_CHECKS, check_last=3)
 
-    assert flags == ('--last', 3)
+    assert flags == ('--last', '3')
 
 
 def test_check_archives_should_call_borg_with_parameters():
@@ -389,7 +431,7 @@ def test_check_archives_should_call_borg_with_parameters():
     )
     insert_platform_mock()
     insert_datetime_mock()
-    builtins_mock().should_receive('open').and_return(stdout)
+    flexmock(sys.modules['builtins']).should_receive('open').and_return(stdout)
     flexmock(module.os).should_receive('devnull')
 
     module.check_archives(
@@ -464,7 +506,7 @@ def test_check_archives_with_remote_path_should_call_borg_with_remote_path_param
     )
     insert_platform_mock()
     insert_datetime_mock()
-    builtins_mock().should_receive('open').and_return(stdout)
+    flexmock(sys.modules['builtins']).should_receive('open').and_return(stdout)
     flexmock(module.os).should_receive('devnull')
 
     module.check_archives(

sample/config

@@ -1,48 +0,0 @@
-[location]
-# Space-separated list of source directories to backup.
-# Globs are expanded.
-source_directories: /home /etc /var/log/syslog*
-
-# Stay in same file system (do not cross mount points).
-#one_file_system: True
-
-# Alternate Borg remote executable (defaults to "borg"):
-#remote_path: borg1
-
-# Path to local or remote repository.
-repository: user@backupserver:sourcehostname.borg
-
-[storage]
-# Passphrase to unlock the encryption key with. Only use on repositories that
-# were initialized with passphrase/repokey encryption.
-#encryption_passphrase: foo
-
-# Type of compression to use when creating archives. See
-# https://borgbackup.readthedocs.org/en/stable/usage.html#borg-create
-# for details. Defaults to no compression.
-#compression: lz4
-
-# Umask to be used for borg create.
-#umask: 0740
-
-[retention]
-# Retention policy for how many backups to keep in each category. See
-# https://borgbackup.readthedocs.org/en/stable/usage.html#borg-prune for details.
-#keep_within: 3H
-#keep_hourly: 24
-keep_daily: 7
-keep_weekly: 4
-keep_monthly: 6
-keep_yearly: 1
-
-#prefix: sourcehostname
-
-[consistency]
-# Space-separated list of consistency checks to run: "repository", "archives",
-# or both. Defaults to both. Set to "disabled" to disable all consistency
-# checks. See https://borgbackup.readthedocs.org/en/stable/usage.html#borg-check
-# for details.
-checks: repository archives
-
-# Restrict the number of checked archives to the last n.
-#check_last: 3

sample/excludes

@@ -1,3 +0,0 @@
-*.pyc
-/home/*/.cache
-/etc/ssl

sample/systemd/borgmatic.service

@@ -0,0 +1,6 @@
+[Unit]
+Description=borgmatic backup
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/bin/borgmatic

sample/systemd/borgmatic.timer

@@ -0,0 +1,8 @@
+[Unit]
+Description=Run borgmatic backup
+
+[Timer]
+OnCalendar=daily
+
+[Install]
+WantedBy=timers.target

setup.cfg

@@ -1,5 +1,2 @@
 [metadata]
 description-file=README.md
-
-[bdist_wheel]
-universal=1

setup.py

@@ -1,7 +1,7 @@
 from setuptools import setup, find_packages
 
 
-VERSION = '1.0.3'
+VERSION = '1.1.4'
 
 
 setup(
@@ -24,14 +24,22 @@ setup(
     packages=find_packages(),
     entry_points={
         'console_scripts': [
-            'borgmatic = borgmatic.command:main',
+            'borgmatic = borgmatic.commands.borgmatic:main',
+            'upgrade-borgmatic-config = borgmatic.commands.convert_config:main',
+            'generate-borgmatic-config = borgmatic.commands.generate_config:main',
         ]
     },
     obsoletes=[
         'atticmatic',
     ],
+    install_requires=(
+        'pykwalify',
+        'ruamel.yaml<=0.15',
+        'setuptools',
+    ),
     tests_require=(
         'flexmock',
         'pytest',
-    )
+    ),
+    include_package_data=True,
 )

static/borgmatic.svg

@@ -0,0 +1 @@
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" x="0px" y="0px" viewBox="0 0 100 100" enable-background="new 0 0 100 100" xml:space="preserve"><g><circle cx="63.461" cy="59.615" r="5.626"></circle><path d="M81.432,48.015c0.779-10.849-1.758-19.894-7.407-26.293C68.447,15.403,59.915,11.923,50,11.923   c-6.729,0-12.834,1.645-17.81,4.724c-0.405-0.06-0.823-0.073-1.248,0.005C8.128,20.817,0,34.623,0,46.154   c0,6.155,1.856,11.005,5.519,14.417c2.082,1.941,4.461,3.223,6.659,3.921C13.041,74.256,18.345,81,23.077,81h1.438   C30.168,94,44.847,99.614,50,99.614c6.708,0,24.493-7.875,27.38-25.392c4.758-1.888,10.697-7.373,10.697-13.711   C88.077,54.639,85.395,49.855,81.432,48.015z M7.691,46.154c0-3.397,1.238-13.399,14.747-19.083   c-2.043,3.821-3.31,8.234-3.771,13.136c-3.06,2.521-5.577,7.792-6.366,15.873c-0.534-0.316-1.056-0.686-1.538-1.135   C8.725,53.045,7.691,50.088,7.691,46.154z M74.23,67.947c-1.806,0-3.309,1.389-3.451,3.188C69.608,85.928,53.914,92.575,50,92.575   c-3.275,0-14.607-4.474-19.03-14.325c1.001-0.709,2.168-1.25,3.646-1.25c3.809,0,3.847,0,3.847,0c2.126,0,7.692-4.023,7.692-19.494   C46.154,46.241,40.588,42,38.462,42c0,0-0.195,0-3.847,0c-3.65,0-5.566-4-7.692-4h-1.046c0.813-4,2.646-8.756,5.473-11.94   c4.321-4.87,10.771-7.329,18.65-7.329c8.016,0,14.528,2.636,18.835,7.515c4.821,5.462,6.697,13.762,5.424,23.95   c-0.123,0.984,0.183,1.99,0.84,2.734s1.602,1.179,2.595,1.179c1.671,0,3.46,2.562,3.46,6.371   C81.153,64.016,75.754,67.812,74.23,67.947z M30.432,59.615c0-3.377,2.734-6.107,6.106-6.107c3.373,0,6.107,2.73,6.107,6.107   s-2.734,6.107-6.107,6.107C33.166,65.723,30.432,62.992,30.432,59.615z"></path></g></svg>

test_requirements.txt

@@ -1,2 +1,5 @@
 flexmock==0.10.2
+pykwalify==1.6.0
 pytest==2.9.1
+pytest-cov==2.5.1
+ruamel.yaml==0.15.18

tox.ini

@@ -1,8 +1,8 @@
 [tox]
-envlist=py27,py34
+envlist=py34
 skipsdist=True
 
 [testenv]
 usedevelop=True
 deps=-rtest_requirements.txt
-commands = py.test borgmatic []
+commands = py.test --cov-report term-missing:skip-covered --cov=borgmatic borgmatic []