borgmatic-collective/borgmatic
borgmatic-collective
/
borgmatic
15
57
Fork 40
Code Issues 52 Pull Requests 2 Actions Packages Releases 121 Activity

Listing by script the content of archives gives an error when the same command executed interactively not #448

New Issue
Closed
opened 2021-09-07 10:33:02 +00:00 by Cartache · 8 comments
Cartache commented 2021-09-07 10:33:02 +00:00
Copy Link

Hi, first thing first: Thank again you for your dedication and this great piece of software that works smoothly for months.

What I'm trying to do and why

I have split my backups in small chunks executed at different moment (day, week month).
Each separate job generates an archive.
I am trying to automate the process of listing the content of each archive file.

To do this, I first extract the list of all archives by executing the command and putting the list in a source file:

borgmatic list --repository repo_path > source_file.txt

Thereafter, I try to list the content of archive by looping through the source_file.txt and executing the following command:

borgmatic list --repository /mnt/borg-repository --archive $archive_var > $archive_var.txt

When I run the command from the command line it runs without any issue.

Doing it thanks to the script I am getting an error:

TAM-verified manifest
security: read previous location '/mnt/borg-repository'
security: read manifest timestamp '2021-09-04T03:02:45.766094'
security: determined newest manifest timestamp as 2021-09-04T03:02:45.766094
security: repository checks ok, allowing access
 does not exist2021-05-23T18:19:36
Traceback (most recent call last):
  File "/usr/lib/python3.8/borg/archiver.py", line 4621, in main
/mnt/borg-repository: Error running actions for repository
' returned non-zero exit status 2.rc /mnt/borg-repository::backup-2021-05-23T18:19:36
/etc/borgmatic.d/config.yaml: Error running configuration file

summary:
/etc/borgmatic.d/config.yaml: Error running configuration file
/mnt/borg-repository: Error running actions for repository
using builtin fallback logging configuration
35 self tests completed in 0.14 seconds
Verified integrity of /mnt/borg-repository/index.35330
TAM-verified manifest
security: read previous location '/mnt/borg-repository'
security: read manifest timestamp '2021-09-04T03:02:45.766094'
security: determined newest manifest timestamp as 2021-09-04T03:02:45.766094
security: repository checks ok, allowing access
 does not exist2021-05-23T18:19:36
Traceback (most recent call last):
  File "/usr/lib/python3.8/borg/archiver.py", line 4621, in main
' returned non-zero exit status 2.rc /mnt/borg-repository::backup-2021-05-23T18:19:36

Need some help? https://torsion.org/borgmatic/#issues

Here is the script I am using:

#!/bin/ash

export BORG_PASSPHRASE='************************************************************'

job_file=/mnt/borg-repository/Jobs_list.txt
while read -r job_unit; do
borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive "$job_unit"
echo "/mnt/borg-repository::$job_unit"
done < "$job_file"

Environment

borgmatic version: 1.5.12

borgmatic installation method: docker

Borg version: borg 1.1.15

Python version: Python 3.8.10

Database version (if applicable): psql (PostgreSQL) 13.2

operating system and version: Borgmatic in docker container from b3vis/borgmatic on Unraid server

Hi, first thing first: Thank again you for your dedication and this great piece of software that works smoothly for months. #### What I'm trying to do and why I have split my backups in small chunks executed at different moment (day, week month). Each separate job generates an archive. I am trying to automate the process of listing the content of each archive file. To do this, I first extract the list of all archives by executing the command and putting the list in a source file: borgmatic list --repository repo_path > source_file.txt Thereafter, I try to list the content of archive by looping through the source_file.txt and executing the following command: borgmatic list --repository /mnt/borg-repository --archive $archive_var > $archive_var.txt When I run the command from the command line it runs without any issue. Doing it thanks to the script I am getting an error: ``` TAM-verified manifest security: read previous location '/mnt/borg-repository' security: read manifest timestamp '2021-09-04T03:02:45.766094' security: determined newest manifest timestamp as 2021-09-04T03:02:45.766094 security: repository checks ok, allowing access does not exist2021-05-23T18:19:36 Traceback (most recent call last): File "/usr/lib/python3.8/borg/archiver.py", line 4621, in main /mnt/borg-repository: Error running actions for repository ' returned non-zero exit status 2.rc /mnt/borg-repository::backup-2021-05-23T18:19:36 /etc/borgmatic.d/config.yaml: Error running configuration file summary: /etc/borgmatic.d/config.yaml: Error running configuration file /mnt/borg-repository: Error running actions for repository using builtin fallback logging configuration 35 self tests completed in 0.14 seconds Verified integrity of /mnt/borg-repository/index.35330 TAM-verified manifest security: read previous location '/mnt/borg-repository' security: read manifest timestamp '2021-09-04T03:02:45.766094' security: determined newest manifest timestamp as 2021-09-04T03:02:45.766094 security: repository checks ok, allowing access does not exist2021-05-23T18:19:36 Traceback (most recent call last): File "/usr/lib/python3.8/borg/archiver.py", line 4621, in main ' returned non-zero exit status 2.rc /mnt/borg-repository::backup-2021-05-23T18:19:36 Need some help? https://torsion.org/borgmatic/#issues ``` Here is the script I am using: ``` #!/bin/ash export BORG_PASSPHRASE='************************************************************' job_file=/mnt/borg-repository/Jobs_list.txt while read -r job_unit; do borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive "$job_unit" echo "/mnt/borg-repository::$job_unit" done < "$job_file" ``` #### Environment **borgmatic version:** 1.5.12 **borgmatic installation method:** docker **Borg version:** borg 1.1.15 **Python version:** Python 3.8.10 **Database version (if applicable):** psql (PostgreSQL) 13.2 **operating system and version:** Borgmatic in docker container from b3vis/borgmatic on Unraid server
ams_tschoening commented 2021-09-09 06:39:04 +00:00
Copy Link
security: repository checks ok, allowing access
 does not exist2021-05-23T18:19:36
Traceback (most recent call last):

That output looks strange, either your archive names are missing prefixes like backup- in your text file or that seems to get lost somewhere for some reason. Thought, the does not exist looks strange already, I would expect the missing part would be placed before or after that message, in the latter case with some , : or alike to distinguish things. I can't find does not exist in this combination in the codebase of Borg as well. Things are especially strange because when you echo on your own, there seems to be some prefix like backup- available.

So I suggest printing some additional output here, some lines of the text file your are reading, some lines directly of the borgmatic command output creating the content of the text file etc.

Additionally, try to enable debugging of the shell itself, something like the following in bash. If ash doesn't provide that, simply try bash instead.

#set -o errexit
#set -o functrace
#set -o xtrace

I suggest instead of running borgmatic, put an echo in front to see the actually rendered command line as well:

echo borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive "$job_unit"

Besides all of that, using a more current version of BorgMatic might be an option as well.

```bash security: repository checks ok, allowing access does not exist2021-05-23T18:19:36 Traceback (most recent call last): ``` That output looks strange, either your archive names are missing prefixes like `backup-` in your text file or that seems to get lost somewhere for some reason. Thought, the `does not exist` looks strange already, I would expect the missing part would be placed before or after that message, in the latter case with some ` `, `:` or alike to distinguish things. I can't find `does not exist` in this combination in the codebase of Borg as well. Things are especially strange because when you `echo` on your own, there seems to be some prefix like `backup-` available. So I suggest printing some additional output here, some lines of the text file your are reading, some lines directly of the borgmatic command output creating the content of the text file etc. Additionally, try to enable debugging of the shell itself, something like the following in `bash`. If `ash` doesn't provide that, simply try `bash` instead. ```bash #set -o errexit #set -o functrace #set -o xtrace ``` I suggest instead of running `borgmatic`, put an `echo` in front to see the actually rendered command line as well: ```bash echo borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive "$job_unit" ``` Besides all of that, using a more current version of BorgMatic might be an option as well.
Cartache commented 2021-09-13 15:48:32 +00:00
Author
Copy Link

Hello,

Thank you for taking time to provide me some support.

Please here is the echo of the command:

borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-18T01:30:01
borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-19T01:30:01
borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-19T03:00:10
borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-20T01:30:02
borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-20T04:04:53
borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-21T01:30:01
borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-21T03:00:10
borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-22T01:30:01
borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-22T03:00:10
borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-23T01:30:01
borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-24T01:30:01
borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-24T03:00:10
borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-25T01:30:01
borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-26T01:30:01
borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-26T03:00:10
borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-27T01:30:01
borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-27T04:04:55
borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-28T01:30:01
borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-28T03:00:10
borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-29T01:30:01
borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-29T03:00:10
borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-30T01:30:01
borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-31T01:30:02

As a test, I took the last line of the echo and executed it as is. And it went successfully.

Hereunder you find an extract of my source file:

backup-2021-05-23T18:19:36           Sun, 2021-05-23 18:19:49 [4edc658e2013c35ac6323f2a0a7f361c374f6ee1adf2c8fbba5bd8b79b5f6bd9]
backup-2021-05-28T19:23:19           Fri, 2021-05-28 19:23:28 [9292a7cd669a02f86dc9e0cd894c5d1bc2bcb72167644f6d284c61d8cc3582e8]
backup-2021-05-30T15:51:18           Sun, 2021-05-30 15:51:25 [b88b475308edd93b6bbfc0828692d93a33153325bff32bc23838e70576289dee]
backup-2021-06-11T12:13:41           Fri, 2021-06-11 12:13:46 [71fe157a8bbbe343faaca4e7b08641cf6ed919ae392ecac633c2a3d321afc090]
backup-2021-06-12T13:00:07           Sat, 2021-06-12 13:00:08 [c568adb9a8648ce7491609ba4d6ba9c58bf8a03ac8f462f0971e14f333572cf4]
backup-2021-06-13T03:44:20           Sun, 2021-06-13 03:44:25 [08320d5361b49172038742e357dff0944fd80920cac21b58ddb48683dd3fd042]
backup-2021-06-14T23:47:07           Mon, 2021-06-14 23:47:10 [d9eb0c0a502d619200aa0aeee5535a9171471351dfb19e742583e6edc5e88d3b]
backup-2021-06-15T00:04:06           Tue, 2021-06-15 00:04:08 [e7647d9540c5d8eed0574c077b15aa0aa7bacda61c1f71497c9d956c3f1d2dc5]
backup-2021-07-19T09:24:27           Mon, 2021-07-19 09:24:39 [4f79ba131ffbbd18f4302af292581fcad7d0dfda7084b3920541f7d355a0461b]
backup-2021-07-19T09:27:05           Mon, 2021-07-19 09:27:07 [faefdac48aed9918f8c0394a8715a56288aed42ea7aff6ec5e05d65e1c5f44bd]
backup-2021-07-19T12:10:12           Mon, 2021-07-19 12:10:18 [972547387c3409246e9231677002317e0ce8ea1cd8b7024994a0b3d3d40fc8e4]
backup-2021-07-19T12:47:46           Mon, 2021-07-19 12:47:54 [e7aff27c952ac1a081f87a4fe5e6a6020dd388948c5dda247e6cbd4d81e3c992]
backup-2021-07-19T12:49:11           Mon, 2021-07-19 12:49:14 [50ba44fc4777c70be7ecb4ebc8b8896534e0cae7da3bba869b5e810dfd256ea1]
backup-2021-07-19T12:49:35           Mon, 2021-07-19 12:49:50 [59a08547f93e0d0fc4ff89990dbfa46035ef67ac647fc9e3b41e07cc183fdd85]
backup-2021-07-19T17:03:02           Mon, 2021-07-19 17:03:09 [a6ba0b03e4d75428bee14070543a84fb3bc90c3c63d7904a2acf93b1a4be4ac0]
backup-2021-07-19T17:18:48           Mon, 2021-07-19 17:18:55 [0b3d1ff49056147503455b29ca570b2363a6aa5b06ad6e6a1932616cec0e2803]
backup-2021-07-19T17:19:35           Mon, 2021-07-19 17:19:37 [2eba0b0411ba64080abd345793fe69f44d62a45927e9dcd1675e252e8b875b7c]
backup-2021-07-19T17:22:13           Mon, 2021-07-19 17:22:19 [1629383fef26ffefa732784ba703dee0dcd2070762a0024777ce97f0363d7827]
backup-2021-07-19T17:22:58           Mon, 2021-07-19 17:23:00 [eff500fada9bc2779f7f8f5a9666bcceca865fd6cd439248984bbb915d370dab]
backup-2021-07-19T17:52:28           Mon, 2021-07-19 17:52:34 [7d05b5938f11b04c7f33f8be1c7ddf7efff9bb9af570db313e92c7f9dfbd4085]
backup-2021-07-19T17:53:45           Mon, 2021-07-19 17:53:47 [3dc2c9fa08232ddb1ebc5b4f4485ae00c5c2f58d601af5198776c7b87592a8b3]
backup-2021-07-19T18:26:18           Mon, 2021-07-19 18:26:25 [0fc4b1f2e4d2ed8f6fed6a36fdda69814397035594aa3af57fe63dde28b1a72f]
backup-2021-07-19T18:27:13           Mon, 2021-07-19 18:27:15 [6aa6db57afb4bf43f8e556678753ee6e633d0bdff6805e8cbab081c15863407a]
backup-2021-07-19T19:26:18           Mon, 2021-07-19 19:26:25 [cc7730cd520258df4336681d520649d7e2cbe80e71e58afe29437d65c9cc4048]
backup-2021-07-19T19:27:10           Mon, 2021-07-19 19:27:13 [30ab3877303ab9d2015a3705541fc266c7ed94ea2bc2c3c76cf681511cc58d9b]
backup-2021-07-19T20:26:18           Mon, 2021-07-19 20:26:26 [d46916d7ab15c6cb1dd87a1e045a5904ea1616553574c02cde1fce5332c96ecc]
backup-2021-07-19T20:27:13           Mon, 2021-07-19 20:27:15 [2099b3262df3e061d6eee91ecde2d55a783c2d006b015dd5e6c4d0ba3c960c7f]

Here is an extract of the error with the additional parameters you provided. I currently am not able to use those parameters with bash as the container is compiled with ash:


/ # ash /borg_scripts/borgmatic/borgmatic_list_jobcontent.sh
Ensuring legacy configuration is upgraded
/mnt/borg-repository: Listing archives
borg list --debug --show-rc /mnt/borg-repository::backup-2021-05-23T18:19:36
using builtin fallback logging configuration
35 self tests completed in 0.14 seconds
Verified integrity of /mnt/borg-repository/index.35603
TAM-verified manifest
security: read previous location '/mnt/borg-repository'
security: read manifest timestamp '2021-09-13T03:03:26.358714'
security: determined newest manifest timestamp as 2021-09-13T03:03:26.358714
security: repository checks ok, allowing access
 does not exist2021-05-23T18:19:36
Traceback (most recent call last):
/mnt/borg-repository: Error running actions for repository
' returned non-zero exit status 2.rc /mnt/borg-repository::backup-2021-05-23T18:19:36
/etc/borgmatic.d/config.yaml: Error running configuration file

summary:
/etc/borgmatic.d/config.yaml: Error running configuration file
/mnt/borg-repository: Error running actions for repository
using builtin fallback logging configuration
35 self tests completed in 0.14 seconds
Verified integrity of /mnt/borg-repository/index.35603
TAM-verified manifest
security: read previous location '/mnt/borg-repository'
security: read manifest timestamp '2021-09-13T03:03:26.358714'
security: determined newest manifest timestamp as 2021-09-13T03:03:26.358714
security: repository checks ok, allowing access
 does not exist2021-05-23T18:19:36
Traceback (most recent call last):
' returned non-zero exit status 2.rc /mnt/borg-repository::backup-2021-05-23T18:19:36

Need some help? https://torsion.org/borgmatic/#issues
Ensuring legacy configuration is upgraded
/mnt/borg-repository: Listing archives
borg list --debug --show-rc /mnt/borg-repository::backup-2021-05-28T19:23:19
using builtin fallback logging configuration
35 self tests completed in 0.14 seconds
Verified integrity of /mnt/borg-repository/index.35603
TAM-verified manifest
security: read previous location '/mnt/borg-repository'
security: read manifest timestamp '2021-09-13T03:03:26.358714'
security: determined newest manifest timestamp as 2021-09-13T03:03:26.358714
security: repository checks ok, allowing access
 does not exist2021-05-28T19:23:19
Traceback (most recent call last):
  File "/usr/lib/python3.8/borg/archiver.py", line 4703, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3.8/borg/archiver.py", line 4635, in run
    return set_ec(func(args))
  File "/usr/lib/python3.8/borg/archiver.py", line 177, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib/python3.8/borg/archiver.py", line 1460, in do_list
    return self._list_archive(args, repository, manifest, key)
  File "/usr/lib/python3.8/borg/archiver.py", line 1489, in _list_archive
    _list_inner(cache=None)
  File "/usr/lib/python3.8/borg/archiver.py", line 1477, in _list_inner
    archive = Archive(repository, key, manifest, args.location.archive, cache=cache,
  File "/usr/lib/python3.8/borg/archive.py", line 369, in __init__
    raise self.DoesNotExist(name)
 does not existchive.DoesNotExist: Archive backup-2021-05-28T19:23:19
Platform: Linux ket-borgmatic 5.10.28-Unraid #1 SMP Wed Apr 7 08:23:18 PDT 2021 x86_64
Linux: Unknown Linux
Borg: 1.1.17  Python: CPython 3.8.10 msgpack: 0.5.6.+borg1
PID: 141  CWD: /
sys.argv: ['/usr/bin/borg', 'list', '--debug', '--show-rc', '/mnt/borg-repository::backup-2021-05-28T19:23:19\r']
SSH_ORIGINAL_COMMAND: None
terminating with error status, rc 2
/mnt/borg-repository: Error running actions for repository
' returned non-zero exit status 2.rc /mnt/borg-repository::backup-2021-05-28T19:23:19
/etc/borgmatic.d/config.yaml: Error running configuration file

summary:
/etc/borgmatic.d/config.yaml: Error running configuration file
/mnt/borg-repository: Error running actions for repository
...
security: repository checks ok, allowing access
 does not exist2021-05-28T19:23:19
Traceback (most recent call last):
  File "/usr/lib/python3.8/borg/archiver.py", line 4703, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3.8/borg/archiver.py", line 4635, in run
    return set_ec(func(args))
  File "/usr/lib/python3.8/borg/archiver.py", line 177, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib/python3.8/borg/archiver.py", line 1460, in do_list
    return self._list_archive(args, repository, manifest, key)
  File "/usr/lib/python3.8/borg/archiver.py", line 1489, in _list_archive
    _list_inner(cache=None)
  File "/usr/lib/python3.8/borg/archiver.py", line 1477, in _list_inner
    archive = Archive(repository, key, manifest, args.location.archive, cache=cache,
  File "/usr/lib/python3.8/borg/archive.py", line 369, in __init__
    raise self.DoesNotExist(name)
 does not existchive.DoesNotExist: Archive backup-2021-05-28T19:23:19
Platform: Linux ket-borgmatic 5.10.28-Unraid #1 SMP Wed Apr 7 08:23:18 PDT 2021 x86_64
Linux: Unknown Linux
Borg: 1.1.17  Python: CPython 3.8.10 msgpack: 0.5.6.+borg1
PID: 141  CWD: /
sys.argv: ['/usr/bin/borg', 'list', '--debug', '--show-rc', '/mnt/borg-repository::backup-2021-05-28T19:23:19\r']
SSH_ORIGINAL_COMMAND: None
terminating with error status, rc 2
' returned non-zero exit status 2.rc /mnt/borg-repository::backup-2021-05-28T19:23:19

Need some help? https://torsion.org/borgmatic/#issues
Ensuring legacy configuration is upgraded
/mnt/borg-repository: Listing archives
borg list --debug --show-rc /mnt/borg-repository::backup-2021-05-30T15:51:18
using builtin fallback logging configuration
35 self tests completed in 0.14 seconds
Verified integrity of /mnt/borg-repository/index.35603
TAM-verified manifest
security: read previous location '/mnt/borg-repository'
security: read manifest timestamp '2021-09-13T03:03:26.358714'
security: determined newest manifest timestamp as 2021-09-13T03:03:26.358714
security: repository checks ok, allowing access
 does not exist2021-05-30T15:51:18
Traceback (most recent call last):
  File "/usr/lib/python3.8/borg/archiver.py", line 4703, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3.8/borg/archiver.py", line 4635, in run
    return set_ec(func(args))
  File "/usr/lib/python3.8/borg/archiver.py", line 177, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib/python3.8/borg/archiver.py", line 1460, in do_list
    return self._list_archive(args, repository, manifest, key)
  File "/usr/lib/python3.8/borg/archiver.py", line 1489, in _list_archive
    _list_inner(cache=None)
  File "/usr/lib/python3.8/borg/archiver.py", line 1477, in _list_inner
    archive = Archive(repository, key, manifest, args.location.archive, cache=cache,
  File "/usr/lib/python3.8/borg/archive.py", line 369, in __init__
    raise self.DoesNotExist(name)
 does not existchive.DoesNotExist: Archive backup-2021-05-30T15:51:18
Platform: Linux ket-borgmatic 5.10.28-Unraid #1 SMP Wed Apr 7 08:23:18 PDT 2021 x86_64
Linux: Unknown Linux
Borg: 1.1.17  Python: CPython 3.8.10 msgpack: 0.5.6.+borg1
PID: 147  CWD: /
sys.argv: ['/usr/bin/borg', 'list', '--debug', '--show-rc', '/mnt/borg-repository::backup-2021-05-30T15:51:18\r']
SSH_ORIGINAL_COMMAND: None
terminating with error status, rc 2
/mnt/borg-repository: Error running actions for repository
' returned non-zero exit status 2.rc /mnt/borg-repository::backup-2021-05-30T15:51:18
/etc/borgmatic.d/config.yaml: Error running configuration file

summary:
/etc/borgmatic.d/config.yaml: Error running configuration file
/mnt/borg-repository: Error running actions for repository
...
security: repository checks ok, allowing access
 does not exist2021-05-30T15:51:18
Traceback (most recent call last):
  File "/usr/lib/python3.8/borg/archiver.py", line 4703, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3.8/borg/archiver.py", line 4635, in run
    return set_ec(func(args))
  File "/usr/lib/python3.8/borg/archiver.py", line 177, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib/python3.8/borg/archiver.py", line 1460, in do_list
    return self._list_archive(args, repository, manifest, key)
  File "/usr/lib/python3.8/borg/archiver.py", line 1489, in _list_archive
    _list_inner(cache=None)
  File "/usr/lib/python3.8/borg/archiver.py", line 1477, in _list_inner
    archive = Archive(repository, key, manifest, args.location.archive, cache=cache,
  File "/usr/lib/python3.8/borg/archive.py", line 369, in __init__
    raise self.DoesNotExist(name)
 does not existchive.DoesNotExist: Archive backup-2021-05-30T15:51:18
Platform: Linux ket-borgmatic 5.10.28-Unraid #1 SMP Wed Apr 7 08:23:18 PDT 2021 x86_64
Linux: Unknown Linux
Borg: 1.1.17  Python: CPython 3.8.10 msgpack: 0.5.6.+borg1
PID: 147  CWD: /
sys.argv: ['/usr/bin/borg', 'list', '--debug', '--show-rc', '/mnt/borg-repository::backup-2021-05-30T15:51:18\r']
SSH_ORIGINAL_COMMAND: None
terminating with error status, rc 2
' returned non-zero exit status 2.rc /mnt/borg-repository::backup-2021-05-30T15:51:18

Need some help? https://torsion.org/borgmatic/#issues
Hello, Thank you for taking time to provide me some support. Please here is the echo of the command: ``` borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-18T01:30:01 borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-19T01:30:01 borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-19T03:00:10 borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-20T01:30:02 borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-20T04:04:53 borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-21T01:30:01 borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-21T03:00:10 borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-22T01:30:01 borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-22T03:00:10 borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-23T01:30:01 borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-24T01:30:01 borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-24T03:00:10 borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-25T01:30:01 borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-26T01:30:01 borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-26T03:00:10 borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-27T01:30:01 borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-27T04:04:55 borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-28T01:30:01 borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-28T03:00:10 borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-29T01:30:01 borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-29T03:00:10 borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-30T01:30:01 borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-08-31T01:30:02 ``` As a test, I took the last line of the echo and executed it as is. And it went successfully. Hereunder you find an extract of my source file: ``` backup-2021-05-23T18:19:36 Sun, 2021-05-23 18:19:49 [4edc658e2013c35ac6323f2a0a7f361c374f6ee1adf2c8fbba5bd8b79b5f6bd9] backup-2021-05-28T19:23:19 Fri, 2021-05-28 19:23:28 [9292a7cd669a02f86dc9e0cd894c5d1bc2bcb72167644f6d284c61d8cc3582e8] backup-2021-05-30T15:51:18 Sun, 2021-05-30 15:51:25 [b88b475308edd93b6bbfc0828692d93a33153325bff32bc23838e70576289dee] backup-2021-06-11T12:13:41 Fri, 2021-06-11 12:13:46 [71fe157a8bbbe343faaca4e7b08641cf6ed919ae392ecac633c2a3d321afc090] backup-2021-06-12T13:00:07 Sat, 2021-06-12 13:00:08 [c568adb9a8648ce7491609ba4d6ba9c58bf8a03ac8f462f0971e14f333572cf4] backup-2021-06-13T03:44:20 Sun, 2021-06-13 03:44:25 [08320d5361b49172038742e357dff0944fd80920cac21b58ddb48683dd3fd042] backup-2021-06-14T23:47:07 Mon, 2021-06-14 23:47:10 [d9eb0c0a502d619200aa0aeee5535a9171471351dfb19e742583e6edc5e88d3b] backup-2021-06-15T00:04:06 Tue, 2021-06-15 00:04:08 [e7647d9540c5d8eed0574c077b15aa0aa7bacda61c1f71497c9d956c3f1d2dc5] backup-2021-07-19T09:24:27 Mon, 2021-07-19 09:24:39 [4f79ba131ffbbd18f4302af292581fcad7d0dfda7084b3920541f7d355a0461b] backup-2021-07-19T09:27:05 Mon, 2021-07-19 09:27:07 [faefdac48aed9918f8c0394a8715a56288aed42ea7aff6ec5e05d65e1c5f44bd] backup-2021-07-19T12:10:12 Mon, 2021-07-19 12:10:18 [972547387c3409246e9231677002317e0ce8ea1cd8b7024994a0b3d3d40fc8e4] backup-2021-07-19T12:47:46 Mon, 2021-07-19 12:47:54 [e7aff27c952ac1a081f87a4fe5e6a6020dd388948c5dda247e6cbd4d81e3c992] backup-2021-07-19T12:49:11 Mon, 2021-07-19 12:49:14 [50ba44fc4777c70be7ecb4ebc8b8896534e0cae7da3bba869b5e810dfd256ea1] backup-2021-07-19T12:49:35 Mon, 2021-07-19 12:49:50 [59a08547f93e0d0fc4ff89990dbfa46035ef67ac647fc9e3b41e07cc183fdd85] backup-2021-07-19T17:03:02 Mon, 2021-07-19 17:03:09 [a6ba0b03e4d75428bee14070543a84fb3bc90c3c63d7904a2acf93b1a4be4ac0] backup-2021-07-19T17:18:48 Mon, 2021-07-19 17:18:55 [0b3d1ff49056147503455b29ca570b2363a6aa5b06ad6e6a1932616cec0e2803] backup-2021-07-19T17:19:35 Mon, 2021-07-19 17:19:37 [2eba0b0411ba64080abd345793fe69f44d62a45927e9dcd1675e252e8b875b7c] backup-2021-07-19T17:22:13 Mon, 2021-07-19 17:22:19 [1629383fef26ffefa732784ba703dee0dcd2070762a0024777ce97f0363d7827] backup-2021-07-19T17:22:58 Mon, 2021-07-19 17:23:00 [eff500fada9bc2779f7f8f5a9666bcceca865fd6cd439248984bbb915d370dab] backup-2021-07-19T17:52:28 Mon, 2021-07-19 17:52:34 [7d05b5938f11b04c7f33f8be1c7ddf7efff9bb9af570db313e92c7f9dfbd4085] backup-2021-07-19T17:53:45 Mon, 2021-07-19 17:53:47 [3dc2c9fa08232ddb1ebc5b4f4485ae00c5c2f58d601af5198776c7b87592a8b3] backup-2021-07-19T18:26:18 Mon, 2021-07-19 18:26:25 [0fc4b1f2e4d2ed8f6fed6a36fdda69814397035594aa3af57fe63dde28b1a72f] backup-2021-07-19T18:27:13 Mon, 2021-07-19 18:27:15 [6aa6db57afb4bf43f8e556678753ee6e633d0bdff6805e8cbab081c15863407a] backup-2021-07-19T19:26:18 Mon, 2021-07-19 19:26:25 [cc7730cd520258df4336681d520649d7e2cbe80e71e58afe29437d65c9cc4048] backup-2021-07-19T19:27:10 Mon, 2021-07-19 19:27:13 [30ab3877303ab9d2015a3705541fc266c7ed94ea2bc2c3c76cf681511cc58d9b] backup-2021-07-19T20:26:18 Mon, 2021-07-19 20:26:26 [d46916d7ab15c6cb1dd87a1e045a5904ea1616553574c02cde1fce5332c96ecc] backup-2021-07-19T20:27:13 Mon, 2021-07-19 20:27:15 [2099b3262df3e061d6eee91ecde2d55a783c2d006b015dd5e6c4d0ba3c960c7f] ``` Here is an extract of the error with the additional parameters you provided. I currently am not able to use those parameters with bash as the container is compiled with ash: ``` / # ash /borg_scripts/borgmatic/borgmatic_list_jobcontent.sh Ensuring legacy configuration is upgraded /mnt/borg-repository: Listing archives borg list --debug --show-rc /mnt/borg-repository::backup-2021-05-23T18:19:36 using builtin fallback logging configuration 35 self tests completed in 0.14 seconds Verified integrity of /mnt/borg-repository/index.35603 TAM-verified manifest security: read previous location '/mnt/borg-repository' security: read manifest timestamp '2021-09-13T03:03:26.358714' security: determined newest manifest timestamp as 2021-09-13T03:03:26.358714 security: repository checks ok, allowing access does not exist2021-05-23T18:19:36 Traceback (most recent call last): /mnt/borg-repository: Error running actions for repository ' returned non-zero exit status 2.rc /mnt/borg-repository::backup-2021-05-23T18:19:36 /etc/borgmatic.d/config.yaml: Error running configuration file summary: /etc/borgmatic.d/config.yaml: Error running configuration file /mnt/borg-repository: Error running actions for repository using builtin fallback logging configuration 35 self tests completed in 0.14 seconds Verified integrity of /mnt/borg-repository/index.35603 TAM-verified manifest security: read previous location '/mnt/borg-repository' security: read manifest timestamp '2021-09-13T03:03:26.358714' security: determined newest manifest timestamp as 2021-09-13T03:03:26.358714 security: repository checks ok, allowing access does not exist2021-05-23T18:19:36 Traceback (most recent call last): ' returned non-zero exit status 2.rc /mnt/borg-repository::backup-2021-05-23T18:19:36 Need some help? https://torsion.org/borgmatic/#issues Ensuring legacy configuration is upgraded /mnt/borg-repository: Listing archives borg list --debug --show-rc /mnt/borg-repository::backup-2021-05-28T19:23:19 using builtin fallback logging configuration 35 self tests completed in 0.14 seconds Verified integrity of /mnt/borg-repository/index.35603 TAM-verified manifest security: read previous location '/mnt/borg-repository' security: read manifest timestamp '2021-09-13T03:03:26.358714' security: determined newest manifest timestamp as 2021-09-13T03:03:26.358714 security: repository checks ok, allowing access does not exist2021-05-28T19:23:19 Traceback (most recent call last): File "/usr/lib/python3.8/borg/archiver.py", line 4703, in main exit_code = archiver.run(args) File "/usr/lib/python3.8/borg/archiver.py", line 4635, in run return set_ec(func(args)) File "/usr/lib/python3.8/borg/archiver.py", line 177, in wrapper return method(self, args, repository=repository, **kwargs) File "/usr/lib/python3.8/borg/archiver.py", line 1460, in do_list return self._list_archive(args, repository, manifest, key) File "/usr/lib/python3.8/borg/archiver.py", line 1489, in _list_archive _list_inner(cache=None) File "/usr/lib/python3.8/borg/archiver.py", line 1477, in _list_inner archive = Archive(repository, key, manifest, args.location.archive, cache=cache, File "/usr/lib/python3.8/borg/archive.py", line 369, in __init__ raise self.DoesNotExist(name) does not existchive.DoesNotExist: Archive backup-2021-05-28T19:23:19 Platform: Linux ket-borgmatic 5.10.28-Unraid #1 SMP Wed Apr 7 08:23:18 PDT 2021 x86_64 Linux: Unknown Linux Borg: 1.1.17 Python: CPython 3.8.10 msgpack: 0.5.6.+borg1 PID: 141 CWD: / sys.argv: ['/usr/bin/borg', 'list', '--debug', '--show-rc', '/mnt/borg-repository::backup-2021-05-28T19:23:19\r'] SSH_ORIGINAL_COMMAND: None terminating with error status, rc 2 /mnt/borg-repository: Error running actions for repository ' returned non-zero exit status 2.rc /mnt/borg-repository::backup-2021-05-28T19:23:19 /etc/borgmatic.d/config.yaml: Error running configuration file summary: /etc/borgmatic.d/config.yaml: Error running configuration file /mnt/borg-repository: Error running actions for repository ... security: repository checks ok, allowing access does not exist2021-05-28T19:23:19 Traceback (most recent call last): File "/usr/lib/python3.8/borg/archiver.py", line 4703, in main exit_code = archiver.run(args) File "/usr/lib/python3.8/borg/archiver.py", line 4635, in run return set_ec(func(args)) File "/usr/lib/python3.8/borg/archiver.py", line 177, in wrapper return method(self, args, repository=repository, **kwargs) File "/usr/lib/python3.8/borg/archiver.py", line 1460, in do_list return self._list_archive(args, repository, manifest, key) File "/usr/lib/python3.8/borg/archiver.py", line 1489, in _list_archive _list_inner(cache=None) File "/usr/lib/python3.8/borg/archiver.py", line 1477, in _list_inner archive = Archive(repository, key, manifest, args.location.archive, cache=cache, File "/usr/lib/python3.8/borg/archive.py", line 369, in __init__ raise self.DoesNotExist(name) does not existchive.DoesNotExist: Archive backup-2021-05-28T19:23:19 Platform: Linux ket-borgmatic 5.10.28-Unraid #1 SMP Wed Apr 7 08:23:18 PDT 2021 x86_64 Linux: Unknown Linux Borg: 1.1.17 Python: CPython 3.8.10 msgpack: 0.5.6.+borg1 PID: 141 CWD: / sys.argv: ['/usr/bin/borg', 'list', '--debug', '--show-rc', '/mnt/borg-repository::backup-2021-05-28T19:23:19\r'] SSH_ORIGINAL_COMMAND: None terminating with error status, rc 2 ' returned non-zero exit status 2.rc /mnt/borg-repository::backup-2021-05-28T19:23:19 Need some help? https://torsion.org/borgmatic/#issues Ensuring legacy configuration is upgraded /mnt/borg-repository: Listing archives borg list --debug --show-rc /mnt/borg-repository::backup-2021-05-30T15:51:18 using builtin fallback logging configuration 35 self tests completed in 0.14 seconds Verified integrity of /mnt/borg-repository/index.35603 TAM-verified manifest security: read previous location '/mnt/borg-repository' security: read manifest timestamp '2021-09-13T03:03:26.358714' security: determined newest manifest timestamp as 2021-09-13T03:03:26.358714 security: repository checks ok, allowing access does not exist2021-05-30T15:51:18 Traceback (most recent call last): File "/usr/lib/python3.8/borg/archiver.py", line 4703, in main exit_code = archiver.run(args) File "/usr/lib/python3.8/borg/archiver.py", line 4635, in run return set_ec(func(args)) File "/usr/lib/python3.8/borg/archiver.py", line 177, in wrapper return method(self, args, repository=repository, **kwargs) File "/usr/lib/python3.8/borg/archiver.py", line 1460, in do_list return self._list_archive(args, repository, manifest, key) File "/usr/lib/python3.8/borg/archiver.py", line 1489, in _list_archive _list_inner(cache=None) File "/usr/lib/python3.8/borg/archiver.py", line 1477, in _list_inner archive = Archive(repository, key, manifest, args.location.archive, cache=cache, File "/usr/lib/python3.8/borg/archive.py", line 369, in __init__ raise self.DoesNotExist(name) does not existchive.DoesNotExist: Archive backup-2021-05-30T15:51:18 Platform: Linux ket-borgmatic 5.10.28-Unraid #1 SMP Wed Apr 7 08:23:18 PDT 2021 x86_64 Linux: Unknown Linux Borg: 1.1.17 Python: CPython 3.8.10 msgpack: 0.5.6.+borg1 PID: 147 CWD: / sys.argv: ['/usr/bin/borg', 'list', '--debug', '--show-rc', '/mnt/borg-repository::backup-2021-05-30T15:51:18\r'] SSH_ORIGINAL_COMMAND: None terminating with error status, rc 2 /mnt/borg-repository: Error running actions for repository ' returned non-zero exit status 2.rc /mnt/borg-repository::backup-2021-05-30T15:51:18 /etc/borgmatic.d/config.yaml: Error running configuration file summary: /etc/borgmatic.d/config.yaml: Error running configuration file /mnt/borg-repository: Error running actions for repository ... security: repository checks ok, allowing access does not exist2021-05-30T15:51:18 Traceback (most recent call last): File "/usr/lib/python3.8/borg/archiver.py", line 4703, in main exit_code = archiver.run(args) File "/usr/lib/python3.8/borg/archiver.py", line 4635, in run return set_ec(func(args)) File "/usr/lib/python3.8/borg/archiver.py", line 177, in wrapper return method(self, args, repository=repository, **kwargs) File "/usr/lib/python3.8/borg/archiver.py", line 1460, in do_list return self._list_archive(args, repository, manifest, key) File "/usr/lib/python3.8/borg/archiver.py", line 1489, in _list_archive _list_inner(cache=None) File "/usr/lib/python3.8/borg/archiver.py", line 1477, in _list_inner archive = Archive(repository, key, manifest, args.location.archive, cache=cache, File "/usr/lib/python3.8/borg/archive.py", line 369, in __init__ raise self.DoesNotExist(name) does not existchive.DoesNotExist: Archive backup-2021-05-30T15:51:18 Platform: Linux ket-borgmatic 5.10.28-Unraid #1 SMP Wed Apr 7 08:23:18 PDT 2021 x86_64 Linux: Unknown Linux Borg: 1.1.17 Python: CPython 3.8.10 msgpack: 0.5.6.+borg1 PID: 147 CWD: / sys.argv: ['/usr/bin/borg', 'list', '--debug', '--show-rc', '/mnt/borg-repository::backup-2021-05-30T15:51:18\r'] SSH_ORIGINAL_COMMAND: None terminating with error status, rc 2 ' returned non-zero exit status 2.rc /mnt/borg-repository::backup-2021-05-30T15:51:18 Need some help? https://torsion.org/borgmatic/#issues ```
ams_tschoening commented 2021-09-13 16:09:16 +00:00
Copy Link

Your proivided output doesn't make too much sense: The archive names used in your first quoted commands are NOT present in what you say is the source file you are basing your script on. OTOH, the source files contains multiple columns of output, while your script does NOT seem prepared to process multiple columns, but only one line with one column instead.

So either you mixed something totally up somewhere or are only showing some excerpts and the spaces-divided columns in your source file are the problem already. The latter might make even sense with the error message:

security: repository checks ok, allowing access
 does not exist2021-05-23T18:19:36
Traceback (most recent call last):

There might be one of the spaces of your source file which is not present.

Your proivided output doesn't make too much sense: The archive names used in your first quoted commands are NOT present in what you say is the source file you are basing your script on. OTOH, the source files contains multiple columns of output, while your script does NOT seem prepared to process multiple columns, but only one line with one column instead. So either you mixed something totally up somewhere or are only showing some excerpts and the spaces-divided columns in your source file are the problem already. The latter might make even sense with the error message: ```bash security: repository checks ok, allowing access does not exist2021-05-23T18:19:36 Traceback (most recent call last): ``` There might be one of the spaces of your source file which is not present.
witten commented 2021-09-14 01:40:58 +00:00
Owner
Copy Link

A couple of additional questions:

  • Does borgmatic_list_jobcontent.sh correspond with the script containing job_unit pasted above? If so, how does job_unit end up getting set? Are values piped into borgmatic_list_jobcontent.sh? If so, can we see that full invocation including the values passed in?
  • You said above that you're "trying to automate the process of listing the content of each archive file." But can you say a little more about your intent behind that so that I can understand the problem you're trying to solve? Why do you want to see all files in all archives?

Thanks! I appreciate your patience here.

A couple of additional questions: * Does `borgmatic_list_jobcontent.sh` correspond with the script containing `job_unit` pasted above? If so, how does `job_unit` end up getting set? Are values piped into `borgmatic_list_jobcontent.sh`? If so, can we see that full invocation including the values passed in? * You said above that you're "trying to automate the process of listing the content of each archive file." But can you say a little more about your intent behind that so that I can understand the problem you're trying to solve? Why do you want to see all files in all archives? Thanks! I appreciate your patience here.
witten added the
question / support
 label 2021-09-14 01:41:10 +00:00
Cartache commented 2021-09-14 08:25:01 +00:00
Author
Copy Link

Hi,

I think I replied a little bit in a rush and messed up my answer.

So let's start with the purpose of the script.
I have a borgmatic container executing different jobs based on cron jobs. Roughly, there are 3 types of jobs:

  • docker
  • vm
  • files and foders

For each type of job there are preparation and ending operations (stop container/vm, check, backup, check restart container/vm). Based on the criticality of the data the jobs are executed more or less frequently. For the "files and folders" type, the job is executed everyday.
Each job is named with a basic naming convention (backup-2021-07-20T05:13:56). Maybe I should already start naming the jobs by adding the job type in the name.

The problem: When I want to restore a specific file or folder I need to know which archive needs to be provided.
Currently, I am not able to do this because I have a very long list of archives which will constantly increase everyday.

My idea was to generate automatically a library containing the archives names and its content to be able to index and search through it.

To proceed with this idea, I created a script which outputs the list of all archives in a source file (job_list.txt).

For what concerns the source file, I agree there was one step I omitted to mention: I removed the useless columns on the right. Sorry for that.

In a second stage, I take the job_list.txt and parse the file using each line (job_unit) to build a new borgmatic command to output the content of each archive represented by the job_unit. borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive "$job_unit" > "$job_unit.txt"

When I execute the command manually it works perfectly. In borg and borgmatic

borg list --debug --show-rc /mnt/borg-repository::backup-2021-09-14T05:00:02
using builtin fallback logging configuration
35 self tests completed in 0.13 seconds
Verified integrity of /mnt/borg-repository/index.35640
TAM-verified manifest
security: read previous location '/mnt/borg-repository'
security: read manifest timestamp '2021-09-14T03:02:20.760196'
security: determined newest manifest timestamp as 2021-09-14T03:02:20.760196
security: repository checks ok, allowing access

....

terminating with success status, rc 0

/ # borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-09-14T05:00:02
Ensuring legacy configuration is upgraded
/mnt/borg-repository: Listing archives
borg list --debug --show-rc /mnt/borg-repository::backup-2021-09-14T05:00:02
using builtin fallback logging configuration
35 self tests completed in 0.13 seconds
Verified integrity of /mnt/borg-repository/index.35640
TAM-verified manifest
security: read previous location '/mnt/borg-repository'
security: read manifest timestamp '2021-09-14T03:02:20.760196'
security: determined newest manifest timestamp as 2021-09-14T03:02:20.760196
security: repository checks ok, allowing access
terminating with success status, rc 0

summary:
/etc/borgmatic.d/config.yaml: Successfully ran configuration file

When I echo the exact command I get the error shown above.

When I remove the file redirection of the command and echo it, I get it working

borgmatic --verbosity 2 list --repository /mnt/borg-repository/ --archive backup-2021-08-31T01:30:02
borgmatic --verbosity 2 list --repository /mnt/borg-repository/ --archive backup-2021-08-31T03:00:11
borgmatic --verbosity 2 list --repository /mnt/borg-repository/ --archive backup-2021-09-01T01:30:01
borgmatic --verbosity 2 list --repository /mnt/borg-repository/ --archive backup-2021-09-01T05:00:12
borgmatic --verbosity 2 list --repository /mnt/borg-repository/ --archive backup-2021-09-02T01:30:01

I tried the same with bash and with borg. Get the same error and same success.

borg list --short /mnt/user/Backup/repo::backup-2021-08-19T03:00:10
borg list --short /mnt/user/Backup/repo::backup-2021-08-20T01:30:02
borg list --short /mnt/user/Backup/repo::backup-2021-08-20T04:04:53
borg list --short /mnt/user/Backup/repo::backup-2021-08-21T01:30:01
borg list --short /mnt/user/Backup/repo::backup-2021-08-21T03:00:10
borg list --short /mnt/user/Backup/repo::backup-2021-08-22T01:30:01
borg list --short /mnt/user/Backup/repo::backup-2021-08-22T03:00:10
Hi, I think I replied a little bit in a rush and messed up my answer. **So let's start with the purpose of the script.** I have a borgmatic container executing different jobs based on cron jobs. Roughly, there are 3 types of jobs: * docker * vm * files and foders For each type of job there are preparation and ending operations (stop container/vm, check, backup, check restart container/vm). Based on the criticality of the data the jobs are executed more or less frequently. For the "files and folders" type, the job is executed everyday. Each job is named with a basic naming convention (backup-2021-07-20T05:13:56). *Maybe I should already start naming the jobs by adding the job type in the name.* The problem: When I want to restore a specific file or folder I need to know which archive needs to be provided. Currently, I am not able to do this because I have a very long list of archives which will constantly increase everyday. My idea was to generate automatically a library containing the archives names and its content to be able to index and search through it. To proceed with this idea, I created a script which outputs the list of all archives in a source file (job_list.txt). ### **For what concerns the source file, I agree there was one step I omitted to mention: I removed the useless columns on the right. Sorry for that.** In a second stage, I take the job_list.txt and parse the file using each line (job_unit) to build a new borgmatic command to output the content of each archive represented by the job_unit. `borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive "$job_unit" > "$job_unit.txt"` When I execute the command manually it works perfectly. In borg and borgmatic ``` borg list --debug --show-rc /mnt/borg-repository::backup-2021-09-14T05:00:02 using builtin fallback logging configuration 35 self tests completed in 0.13 seconds Verified integrity of /mnt/borg-repository/index.35640 TAM-verified manifest security: read previous location '/mnt/borg-repository' security: read manifest timestamp '2021-09-14T03:02:20.760196' security: determined newest manifest timestamp as 2021-09-14T03:02:20.760196 security: repository checks ok, allowing access .... terminating with success status, rc 0 ``` ``` / # borgmatic --verbosity 2 list --repository /mnt/borg-repository --archive backup-2021-09-14T05:00:02 Ensuring legacy configuration is upgraded /mnt/borg-repository: Listing archives borg list --debug --show-rc /mnt/borg-repository::backup-2021-09-14T05:00:02 using builtin fallback logging configuration 35 self tests completed in 0.13 seconds Verified integrity of /mnt/borg-repository/index.35640 TAM-verified manifest security: read previous location '/mnt/borg-repository' security: read manifest timestamp '2021-09-14T03:02:20.760196' security: determined newest manifest timestamp as 2021-09-14T03:02:20.760196 security: repository checks ok, allowing access terminating with success status, rc 0 summary: /etc/borgmatic.d/config.yaml: Successfully ran configuration file ``` When I echo the exact command I get the error shown above. When I remove the file redirection of the command and echo it, I get it working ``` borgmatic --verbosity 2 list --repository /mnt/borg-repository/ --archive backup-2021-08-31T01:30:02 borgmatic --verbosity 2 list --repository /mnt/borg-repository/ --archive backup-2021-08-31T03:00:11 borgmatic --verbosity 2 list --repository /mnt/borg-repository/ --archive backup-2021-09-01T01:30:01 borgmatic --verbosity 2 list --repository /mnt/borg-repository/ --archive backup-2021-09-01T05:00:12 borgmatic --verbosity 2 list --repository /mnt/borg-repository/ --archive backup-2021-09-02T01:30:01 ``` I tried the same with bash and with borg. Get the same error and same success. ``` borg list --short /mnt/user/Backup/repo::backup-2021-08-19T03:00:10 borg list --short /mnt/user/Backup/repo::backup-2021-08-20T01:30:02 borg list --short /mnt/user/Backup/repo::backup-2021-08-20T04:04:53 borg list --short /mnt/user/Backup/repo::backup-2021-08-21T01:30:01 borg list --short /mnt/user/Backup/repo::backup-2021-08-21T03:00:10 borg list --short /mnt/user/Backup/repo::backup-2021-08-22T01:30:01 borg list --short /mnt/user/Backup/repo::backup-2021-08-22T03:00:10 ```
ams_tschoening commented 2021-09-14 09:08:37 +00:00
Copy Link

It's still not clear to me what your are executing when, it additionally seems you seem to change what gets executed with each comment. E.g. the redirection > "$job_unit.txt" wasn't mentioned before? This can't be debugged under those circumstances, so I suggest taking a step back and look at what you said before:

[...]Maybe I should already start naming the jobs by adding the job type in the name.

Not only that: Am I understanding correctly that you use one repo only to store all data? Then I suggest changing that to one repo per job type. The data of the different jobs will most likely be that different, that you won't get too much benefit of de-duplication within one and the same repo anyway. Use one repo for docker, one for VMs and one for individual files and folders of interest. Depending on the actual backed up data, you can additionally name archives by customers or alike. With that approach you always know which repo contains your files and folders of interest and only care about the concrete archive to restore.

In the easiest case, you wouldn't need to maintain some index at all only to work around your "one repo to rule them all"-approach.

It's still not clear to me what your are executing when, it additionally seems you seem to change what gets executed with each comment. E.g. the redirection `> "$job_unit.txt"` wasn't mentioned before? This can't be debugged under those circumstances, so I suggest taking a step back and look at what you said before: > [...]Maybe I should already start naming the jobs by adding the job type in the name. Not only that: Am I understanding correctly that you use one repo only to store all data? Then I suggest changing that to one repo per job type. The data of the different jobs will most likely be that different, that you won't get too much benefit of de-duplication within one and the same repo anyway. Use one repo for docker, one for VMs and one for individual files and folders of interest. Depending on the actual backed up data, you can additionally name archives by customers or alike. With that approach you always know which repo contains your files and folders of interest and only care about the concrete archive to restore. In the easiest case, you wouldn't need to maintain some index at all only to work around your "one repo to rule them all"-approach.
Cartache commented 2021-09-14 09:48:52 +00:00
Author
Copy Link

Dear,

Thank you for your time and your answer. I certainly will take your proposal into account and review the architecture of my backups.

Concerning your remark about the redirection it was mentionned in the begining of the post "borgmatic list --repository /mnt/borg-repository --archive $archive_var > $archive_var.txt"

As an additional remark I think I found the source of the problem, it is linked to a special character located at the end of the source file which caused the issues. This is very strange and the errors raised did not easily helped to pinpoint it.

Thanks again for your help and support.

Denis

Dear, Thank you for your time and your answer. I certainly will take your proposal into account and review the architecture of my backups. Concerning your remark about the redirection it was mentionned in the begining of the post "borgmatic list --repository /mnt/borg-repository --archive $archive_var > $archive_var.txt" As an additional remark I think I found the source of the problem, it is linked to a special character located at the end of the source file which caused the issues. This is very strange and the errors raised did not easily helped to pinpoint it. Thanks again for your help and support. Denis
witten commented 2021-09-15 06:15:09 +00:00
Owner
Copy Link

I'm glad to hear you've found the source of the problem! You may already be aware of the feature, but borgmatic accepts "latest" as an archive name, which will select the most recent archive in a repository. I don't know if that's of use for your particular use case, but it could be useful if you switch to a separate repository per backup source. In any case, if you have any further issues, please feel free to file another ticket! And thanks for using borgmatic.

I'm glad to hear you've found the source of the problem! You may already be aware of the feature, but borgmatic accepts "latest" as an archive name, which will select the most recent archive in a repository. I don't know if that's of use for your particular use case, but it could be useful if you switch to a separate repository per backup source. In any case, if you have any further issues, please feel free to file another ticket! And thanks for using borgmatic.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
main
spot-check
1.8.10
1.8.9
1.8.8
1.8.7
1.8.6
1.8.5
1.8.4
1.8.3
1.8.2
1.8.1
1.8.0
1.7.15
1.7.14
1.7.13
1.7.12
1.7.11
1.7.10
1.7.9
1.7.8
1.7.7
1.7.6
1.7.5
1.7.4
1.7.3
1.7.2
1.7.1
1.7.0
1.6.6
1.6.5
1.6.4
1.6.3
1.6.2
1.6.1
1.6.0
1.5.24
1.5.23
1.5.22
1.5.21
1.5.20
1.5.19
1.5.18
1.5.17
1.5.16
1.5.15
1.5.14
1.5.13
1.5.12
1.5.11
1.5.10
1.5.9
1.5.8
1.5.7
1.5.7.dev0
1.5.6
1.5.5
1.5.4
1.5.3
1.5.2
1.5.1
1.5.0
1.4.22
1.4.21
1.4.20
1.4.19
1.4.18
1.4.17
1.4.16
1.4.15
1.4.14
1.4.13
1.4.12
1.4.11
1.4.10
1.4.9
1.4.8
1.4.7
1.4.6
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0
1.3.26
1.3.25
1.3.24
1.3.23
1.3.22
1.3.21
1.3.20
1.3.19
1.3.18
1.3.17
1.3.16
1.3.15
1.3.14
1.3.13
1.3.12
1.3.11
1.3.10
1.3.9
1.3.8
1.3.7
1.3.6
1.3.5
1.3.4
1.3.3
1.3.2
1.3.1
1.3.0
1.2.18
1.2.17
1.2.16
1.2.15
1.2.14
1.2.13
1.2.12
1.2.11
1.2.10
1.2.9
1.2.8
1.2.7
1.2.6
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.15
1.1.14
1.1.13
1.1.12
1.1.11
1.1.10
1.1.9
1.1.7
1.1.6
1.1.5
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.3
1.0.2
1.0.1
1.0.0
0.1.8
0.1.7
0.1.6
0.1.5
0.1.4
0.1.3
0.1.2
0.1.1
0.1.0
0.0.7
0.0.6
0.0.5
0.0.4
0.0.3
0.0.2
Labels
Clear labels   
bug

   
data loss

   
design finalized

   
good first issue

   
new feature area

   
question / support

   
security

   
waiting for response
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#448
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?