borgmatic.d per app configs don't run in the order I want them to... why? #157

New Issue

abclution · 2019-03-30T22:18:44Z

abclution commented

2019-03-30 22:18:44 +00:00

This is the order I want borgmatic to run them in, and the corresponding file names of the configs.

ls -A1 
config-1-BAREMETAL-CONFIGS-LOGS-HOME.yaml
config-2-proxmox-passthrough-mounts-vps1.website.com.yaml
config-3-proxmox-passthrough-mounts-cloud.website.com.yaml
config-4-VIRTUALMIN-DOMAIN-BACKUPS-DB-AND-CONFIG-ARCHIVES.yaml
config-5-VIRTUALMIN-DOMAIN-BACKUPS-ARCHIVES.yaml
config-6-PROXMOX-DIFFERENTIAL-BACKUPS.yaml
hetznerbackup-1-MAIN-DATA-BACKUP.yaml
hetznerbackup-2-VIRTUALMIN-DOMAIN-BACKUPS-DB-AND-CONFIG-ARCHIVES.yaml
hetznerbackup-3-proxmox-passthrough-mounts-vps1.website.com.yaml
hetznerbackup-4-PROXMOX-DIFFERENTIAL-BACKUPS.yaml

But when running borgmatic it seems to run them in an out of order way, I think the order it ran it was: (a bit hard to tell because i only get output when I up the verbosity but v 1 is too much info and v 0 outputs nothing much)

 borgmatic -n -v 1
/etc/borgmatic.d/hetznerbackup-2-VIRTUALMIN-DOMAIN-BACKUPS-DB-AND-CONFIG-ARCHIVES.yaml: Parsing configuration file
/etc/borgmatic.d/config-3-proxmox-passthrough-mounts-cloud.website.com.yaml: Parsing configuration file
/etc/borgmatic.d/hetznerbackup-4-PROXMOX-DIFFERENTIAL-BACKUPS.yaml: Parsing configuration file
/etc/borgmatic.d/config-2-proxmox-passthrough-mounts-vps1.website.com.yaml: Parsing configuration file
/etc/borgmatic.d/config-4-VIRTUALMIN-DOMAIN-BACKUPS-DB-AND-CONFIG-ARCHIVES.yaml: Parsing configuration file
/etc/borgmatic.d/hetznerbackup-1-MAIN-DATA-BACKUP.yaml: Parsing configuration file
/etc/borgmatic.d/hetznerbackup-3-proxmox-passthrough-mounts-vps1.website.com.yaml: Parsing configuration file
/etc/borgmatic.d/config-1-BAREMETAL-CONFIGS-LOGS-HOME.yaml: Parsing configuration file
/etc/borgmatic.d/config-6-PROXMOX-DIFFERENTIAL-BACKUPS.yaml: Parsing configuration file
/etc/borgmatic.d/config-5-VIRTUALMIN-DOMAIN-BACKUPS-ARCHIVES.yaml: Parsing configuration file

and so it, it seems kind of random although surely it has to do with the file names.

I would like it to run it in the same order as they are listed in the directory listing but I would like to keep the descriptive naming I have for ease of editing.

No idea how to paste in code view so that formatting isn't messed up.. sorry
I attached a txt file with a better listing

This is the order I want borgmatic to run them in, and the corresponding file names of the configs. ``` ls -A1 config-1-BAREMETAL-CONFIGS-LOGS-HOME.yaml config-2-proxmox-passthrough-mounts-vps1.website.com.yaml config-3-proxmox-passthrough-mounts-cloud.website.com.yaml config-4-VIRTUALMIN-DOMAIN-BACKUPS-DB-AND-CONFIG-ARCHIVES.yaml config-5-VIRTUALMIN-DOMAIN-BACKUPS-ARCHIVES.yaml config-6-PROXMOX-DIFFERENTIAL-BACKUPS.yaml hetznerbackup-1-MAIN-DATA-BACKUP.yaml hetznerbackup-2-VIRTUALMIN-DOMAIN-BACKUPS-DB-AND-CONFIG-ARCHIVES.yaml hetznerbackup-3-proxmox-passthrough-mounts-vps1.website.com.yaml hetznerbackup-4-PROXMOX-DIFFERENTIAL-BACKUPS.yaml ``` But when running borgmatic it seems to run them in an out of order way, I think the order it ran it was: (a bit hard to tell because i only get output when I up the verbosity but v 1 is too much info and v 0 outputs nothing much) ``` borgmatic -n -v 1 /etc/borgmatic.d/hetznerbackup-2-VIRTUALMIN-DOMAIN-BACKUPS-DB-AND-CONFIG-ARCHIVES.yaml: Parsing configuration file /etc/borgmatic.d/config-3-proxmox-passthrough-mounts-cloud.website.com.yaml: Parsing configuration file /etc/borgmatic.d/hetznerbackup-4-PROXMOX-DIFFERENTIAL-BACKUPS.yaml: Parsing configuration file /etc/borgmatic.d/config-2-proxmox-passthrough-mounts-vps1.website.com.yaml: Parsing configuration file /etc/borgmatic.d/config-4-VIRTUALMIN-DOMAIN-BACKUPS-DB-AND-CONFIG-ARCHIVES.yaml: Parsing configuration file /etc/borgmatic.d/hetznerbackup-1-MAIN-DATA-BACKUP.yaml: Parsing configuration file /etc/borgmatic.d/hetznerbackup-3-proxmox-passthrough-mounts-vps1.website.com.yaml: Parsing configuration file /etc/borgmatic.d/config-1-BAREMETAL-CONFIGS-LOGS-HOME.yaml: Parsing configuration file /etc/borgmatic.d/config-6-PROXMOX-DIFFERENTIAL-BACKUPS.yaml: Parsing configuration file /etc/borgmatic.d/config-5-VIRTUALMIN-DOMAIN-BACKUPS-ARCHIVES.yaml: Parsing configuration file ``` and so it, it seems kind of random although surely it has to do with the file names. I would like it to run it in the same order as they are listed in the directory listing but I would like to keep the descriptive naming I have for ease of editing. No idea how to paste in code view so that formatting isn't messed up.. sorry I attached a txt file with a better listing

abclution commented

2019-03-30 22:22:11 +00:00

No description provided.

notepad_inMXjm1SPh.png

39 KiB

abclution commented

2019-03-30 22:44:46 +00:00

No description provided.

chrome_mAGh6401Yv.png

20 KiB

witten commented

2019-03-30 22:45:49 +00:00

Good catch. borgmatic is using Python's os.listdir() under the hood to list the directory, which apparently has questionable ordering semantics.

Example consistent with the behavior you're seeing:

$ touch config-1-BAREMETAL-CONFIGS-LOGS-HOME.yaml config-2-proxmox-passthrough-mounts-vps1.website.com.yaml config-3-proxmox-passthrough-mounts-cloud.website.com.yaml config-4-VIRTUALMIN-DOMAIN-BACKUPS-DB-AND-CONFIG-ARCHIVES.yaml config-5-VIRTUALMIN-DOMAIN-BACKUPS-ARCHIVES.yaml config-6-PROXMOX-DIFFERENTIAL-BACKUPS.yaml hetznerbackup-1-MAIN-DATA-BACKUP.yaml hetznerbackup-2-VIRTUALMIN-DOMAIN-BACKUPS-DB-AND-CONFIG-ARCHIVES.yaml hetznerbackup-3-proxmox-passthrough-mounts-vps1.website.com.yaml hetznerbackup-4-PROXMOX-DIFFERENTIAL-BACKUPS.yaml
$ ls -1
config-1-BAREMETAL-CONFIGS-LOGS-HOME.yaml
config-2-proxmox-passthrough-mounts-vps1.website.com.yaml
config-3-proxmox-passthrough-mounts-cloud.website.com.yaml
config-4-VIRTUALMIN-DOMAIN-BACKUPS-DB-AND-CONFIG-ARCHIVES.yaml
config-5-VIRTUALMIN-DOMAIN-BACKUPS-ARCHIVES.yaml
config-6-PROXMOX-DIFFERENTIAL-BACKUPS.yaml
hetznerbackup-1-MAIN-DATA-BACKUP.yaml
hetznerbackup-2-VIRTUALMIN-DOMAIN-BACKUPS-DB-AND-CONFIG-ARCHIVES.yaml
hetznerbackup-3-proxmox-passthrough-mounts-vps1.website.com.yaml
hetznerbackup-4-PROXMOX-DIFFERENTIAL-BACKUPS.yaml
$ ipython
Python 3.7.2 (default, Jan 10 2019, 23:51:51) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.3.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import os                                                                                                                                 

In [2]: for x in os.listdir('.'): 
   ...:     print(x) 
   ...:                                                                                                                                           
config-6-PROXMOX-DIFFERENTIAL-BACKUPS.yaml
config-3-proxmox-passthrough-mounts-cloud.website.com.yaml
hetznerbackup-2-VIRTUALMIN-DOMAIN-BACKUPS-DB-AND-CONFIG-ARCHIVES.yaml
hetznerbackup-4-PROXMOX-DIFFERENTIAL-BACKUPS.yaml
hetznerbackup-3-proxmox-passthrough-mounts-vps1.website.com.yaml
config-5-VIRTUALMIN-DOMAIN-BACKUPS-ARCHIVES.yaml
config-2-proxmox-passthrough-mounts-vps1.website.com.yaml
config-1-BAREMETAL-CONFIGS-LOGS-HOME.yaml
config-4-VIRTUALMIN-DOMAIN-BACKUPS-DB-AND-CONFIG-ARCHIVES.yaml
hetznerbackup-1-MAIN-DATA-BACKUP.yaml

(FYI, you can format code here by surrounding it with ``` on an empty line.)

So I think the fix is to explicitly perform an alphanumeric sort when getting the nonsensical order back from os.listdir().

Good catch. borgmatic is using Python's `os.listdir()` under the hood to list the directory, which apparently has [questionable ordering semantics](https://stackoverflow.com/questions/4813061/non-alphanumeric-list-order-from-os-listdir). Example consistent with the behavior you're seeing: ```bash $ touch config-1-BAREMETAL-CONFIGS-LOGS-HOME.yaml config-2-proxmox-passthrough-mounts-vps1.website.com.yaml config-3-proxmox-passthrough-mounts-cloud.website.com.yaml config-4-VIRTUALMIN-DOMAIN-BACKUPS-DB-AND-CONFIG-ARCHIVES.yaml config-5-VIRTUALMIN-DOMAIN-BACKUPS-ARCHIVES.yaml config-6-PROXMOX-DIFFERENTIAL-BACKUPS.yaml hetznerbackup-1-MAIN-DATA-BACKUP.yaml hetznerbackup-2-VIRTUALMIN-DOMAIN-BACKUPS-DB-AND-CONFIG-ARCHIVES.yaml hetznerbackup-3-proxmox-passthrough-mounts-vps1.website.com.yaml hetznerbackup-4-PROXMOX-DIFFERENTIAL-BACKUPS.yaml $ ls -1 config-1-BAREMETAL-CONFIGS-LOGS-HOME.yaml config-2-proxmox-passthrough-mounts-vps1.website.com.yaml config-3-proxmox-passthrough-mounts-cloud.website.com.yaml config-4-VIRTUALMIN-DOMAIN-BACKUPS-DB-AND-CONFIG-ARCHIVES.yaml config-5-VIRTUALMIN-DOMAIN-BACKUPS-ARCHIVES.yaml config-6-PROXMOX-DIFFERENTIAL-BACKUPS.yaml hetznerbackup-1-MAIN-DATA-BACKUP.yaml hetznerbackup-2-VIRTUALMIN-DOMAIN-BACKUPS-DB-AND-CONFIG-ARCHIVES.yaml hetznerbackup-3-proxmox-passthrough-mounts-vps1.website.com.yaml hetznerbackup-4-PROXMOX-DIFFERENTIAL-BACKUPS.yaml $ ipython Python 3.7.2 (default, Jan 10 2019, 23:51:51) Type 'copyright', 'credits' or 'license' for more information IPython 7.3.0 -- An enhanced Interactive Python. Type '?' for help. In [1]: import os In [2]: for x in os.listdir('.'): ...: print(x) ...: config-6-PROXMOX-DIFFERENTIAL-BACKUPS.yaml config-3-proxmox-passthrough-mounts-cloud.website.com.yaml hetznerbackup-2-VIRTUALMIN-DOMAIN-BACKUPS-DB-AND-CONFIG-ARCHIVES.yaml hetznerbackup-4-PROXMOX-DIFFERENTIAL-BACKUPS.yaml hetznerbackup-3-proxmox-passthrough-mounts-vps1.website.com.yaml config-5-VIRTUALMIN-DOMAIN-BACKUPS-ARCHIVES.yaml config-2-proxmox-passthrough-mounts-vps1.website.com.yaml config-1-BAREMETAL-CONFIGS-LOGS-HOME.yaml config-4-VIRTUALMIN-DOMAIN-BACKUPS-DB-AND-CONFIG-ARCHIVES.yaml hetznerbackup-1-MAIN-DATA-BACKUP.yaml ``` (FYI, you can format code here by surrounding it with \`\`\` on an empty line.) So I think the fix is to explicitly perform an alphanumeric sort when getting the nonsensical order back from `os.listdir()`.

witten added the

design finalized

bug

labels 2019-03-30 22:47:02 +00:00

witten referenced this issue from a commit

2019-03-30 23:41:56 +00:00

Fix for seemingly random filename ordering when running through a directory of configuration files (#157).

witten commented

2019-03-30 23:42:13 +00:00

Fixed in master. Will go out as part of the next release.

abclution commented

2019-03-30 23:48:24 +00:00

Hey Dan, thanks for the reply.

Did not expect it to be a bug but rather a user error, but appreciate the humorous explanation re "questionable ordering"

Also just to clarify further in case someone else encounters this, having the configs run in the order I expect them to run is important as some of these backups are sent to much slower storage and I would prefer in this case for those configs to be run last and not block the earlier running configs.

Thanks very much.

Borgmatic is an excellent abstraction layer on top of an excellent backup system. I have been using it since atticmatic days and it has just gotten better and better.

I just recently moved my backup configs to the yaml format, which I had never encountered before. Took me quite a few tries to understand it (yaml) is a config format that very much cares about space/spacing characters. :P

I'm not a programmer but I understand python as well very much cares about spaces/tabs as, if recall correctly.

Hey Dan, thanks for the reply. Did not expect it to be a bug but rather a user error, but appreciate the humorous explanation re "questionable ordering" Also just to clarify further in case someone else encounters this, having the configs run in the order I expect them to run is important as some of these backups are sent to much slower storage and I would prefer in this case for those configs to be run last and not block the earlier running configs. Thanks very much. Borgmatic is an excellent abstraction layer on top of an excellent backup system. I have been using it since atticmatic days and it has just gotten better and better. I just recently moved my backup configs to the yaml format, which I had never encountered before. Took me quite a few tries to understand it (yaml) is a config format that very much cares about space/spacing characters. :P I'm not a programmer but I understand python as well very much cares about spaces/tabs as, if recall correctly.

abclution closed this issue

2019-03-30 23:48:24 +00:00

witten commented

2019-03-31 04:06:05 +00:00

Yeah, YAML is certainly a trade-off. The upside is that it's supports more complex data structures (lists, etc.), and the downside is that it's more complex. Glad to hear borgmatic has been working out for you though!