borgmatic-collective/borgmatic
borgmatic-collective
/
borgmatic
15
57
Fork 40
Code Issues 54 Pull Requests 1 Actions Packages Releases 121 Activity

borgmatic restore/extract fails in Ubuntu 18.04 with remote repository over SSH setup #243

New Issue
Closed
opened 2019-11-12 13:14:12 +00:00 by earthquakesan · 10 comments
earthquakesan commented 2019-11-12 13:14:12 +00:00
Copy Link

What I'm trying to do and why

Running the following commands in the sequence:

borgmatic
borgmatic list
borgmatic restore --archive client-timedate-stamp

I am trying out the borg/borgmatic before deploying it in our org.

Steps to reproduce (if a bug)

The setup requires vagrant with virtualbox installed as well as vagrant-vbguest (see README.md in the repository):

git clone https://github.com/earthquakesan/borg-lab && cd borg-lab
git checkout af15592da8d75a4767dce2d3dcfe5876c17c4264
vagrant up
make install
make ssh-client
borgmatic
borgmatic list
borgmatic restore --archive client-datetime-stamp

Actual behavior (if a bug)

The restore does not work, throws the following error:

root@client:/home/vagrant# borgmatic list
client-2019-11-12T05:08:04.381881    Tue, 2019-11-12 05:08:05 [0b3621a643e45a72af72e34c82114932cb380360015853d591fcf4516087a4cb]
client-2019-11-12T05:09:04.334387    Tue, 2019-11-12 05:09:05 [1a071e35ac7c41110031dc7e3055704b4d0098e1f03024eeef1461504cfa6e2e]
root@client:/home/vagrant# borgmatic restore --archive client-2019-11-12T05:08:04.381881
/etc/borgmatic/config.yaml: Error running configuration file
root@borg.eccenca.local:/data/borg: Error running actions for repository
Repository /home/vagrant/root@borg.eccenca.local:/data/borg does not exist.
Command 'borg extract /home/vagrant/root@borg.eccenca.local:/data/borg::client-2019-11-12T05:08:04.381881 sh:root/.borgmatic/postgresql_databases/*/*' returned non-zero exit status 2.

Need some help? https://torsion.org/borgmatic/#issues

Expected behavior (if a bug)

Restore works as expected.

Other notes / implementation ideas

I can execute extract using borg without any problems:

root@client:/home/vagrant/archive# ls
root@client:/home/vagrant/archive# borg list
client-2019-11-12T05:10:04.062028    Tue, 2019-11-12 05:10:05 [1d3d95b54856e0e79de5e84c7b66a956a1c5983e7f37e50639ca0db173984ed7]
client-2019-11-12T05:11:04.713546    Tue, 2019-11-12 05:11:06 [7ffa77ead2d304993d11ab96844326fbf5bb68779f2df0e0b612877378788412]
oroot@client:/home/vagrant/archive# borg extract ::client-2019-11-12T05:10:04.062028
root@client:/home/vagrant/archive# ls
etc  home  var
root@client:/home/vagrant/archive#

Environment

borgmatic version: 1.4.8

Use sudo borgmatic --version or sudo pip show borgmatic | grep ^Version

borgmatic installation method: installed with pip3

Borg version: 1.1.10

Use sudo borg --version

Python version: 3.6.8

Use python3 --version

operating system and version: Ubuntu 18.04

#### What I'm trying to do and why Running the following commands in the sequence: ``` borgmatic borgmatic list borgmatic restore --archive client-timedate-stamp ``` I am trying out the borg/borgmatic before deploying it in our org. #### Steps to reproduce (if a bug) The setup requires vagrant with virtualbox installed as well as vagrant-vbguest (see README.md in the repository): ``` git clone https://github.com/earthquakesan/borg-lab && cd borg-lab git checkout af15592da8d75a4767dce2d3dcfe5876c17c4264 vagrant up make install make ssh-client borgmatic borgmatic list borgmatic restore --archive client-datetime-stamp ``` #### Actual behavior (if a bug) The restore does not work, throws the following error: ``` root@client:/home/vagrant# borgmatic list client-2019-11-12T05:08:04.381881 Tue, 2019-11-12 05:08:05 [0b3621a643e45a72af72e34c82114932cb380360015853d591fcf4516087a4cb] client-2019-11-12T05:09:04.334387 Tue, 2019-11-12 05:09:05 [1a071e35ac7c41110031dc7e3055704b4d0098e1f03024eeef1461504cfa6e2e] root@client:/home/vagrant# borgmatic restore --archive client-2019-11-12T05:08:04.381881 /etc/borgmatic/config.yaml: Error running configuration file root@borg.eccenca.local:/data/borg: Error running actions for repository Repository /home/vagrant/root@borg.eccenca.local:/data/borg does not exist. Command 'borg extract /home/vagrant/root@borg.eccenca.local:/data/borg::client-2019-11-12T05:08:04.381881 sh:root/.borgmatic/postgresql_databases/*/*' returned non-zero exit status 2. Need some help? https://torsion.org/borgmatic/#issues ``` #### Expected behavior (if a bug) Restore works as expected. #### Other notes / implementation ideas I can execute extract using borg without any problems: ``` root@client:/home/vagrant/archive# ls root@client:/home/vagrant/archive# borg list client-2019-11-12T05:10:04.062028 Tue, 2019-11-12 05:10:05 [1d3d95b54856e0e79de5e84c7b66a956a1c5983e7f37e50639ca0db173984ed7] client-2019-11-12T05:11:04.713546 Tue, 2019-11-12 05:11:06 [7ffa77ead2d304993d11ab96844326fbf5bb68779f2df0e0b612877378788412] oroot@client:/home/vagrant/archive# borg extract ::client-2019-11-12T05:10:04.062028 root@client:/home/vagrant/archive# ls etc home var root@client:/home/vagrant/archive# ``` #### Environment **borgmatic version:** 1.4.8 Use `sudo borgmatic --version` or `sudo pip show borgmatic | grep ^Version` **borgmatic installation method:** installed with pip3 **Borg version:** 1.1.10 Use `sudo borg --version` **Python version:** 3.6.8 Use `python3 --version` **operating system and version:** Ubuntu 18.04
earthquakesan commented 2019-11-12 13:23:44 +00:00
Author
Copy Link

Ok, I found the problem. in borg/extract.py on line 85:

85         + ('::'.join((os.path.abspath(repository), archive)),)

In debugger:

ipdb> repository
'root@borg.eccenca.local:/data/borg'
ipdb> os.path.abspath(repository)
'/usr/local/lib/python3.6/dist-packages/borgmatic/root@borg.eccenca.local:/data/borg'
Ok, I found the problem. in `borg/extract.py` on line 85: ``` 85 + ('::'.join((os.path.abspath(repository), archive)),) ``` In debugger: ``` ipdb> repository 'root@borg.eccenca.local:/data/borg' ipdb> os.path.abspath(repository) '/usr/local/lib/python3.6/dist-packages/borgmatic/root@borg.eccenca.local:/data/borg' ```
earthquakesan commented 2019-11-12 13:26:31 +00:00
Author
Copy Link

The defaul behaviour of abspath is exactly how it goes here:

Return a normalized absolutized version of the pathname path. On most platforms, this is equivalent to calling the function normpath() as follows: normpath(join(os.getcwd(), path)).

It basically appends my cwd to the repository, which makes it inoperable.

The defaul behaviour of abspath is exactly how it goes here: ``` Return a normalized absolutized version of the pathname path. On most platforms, this is equivalent to calling the function normpath() as follows: normpath(join(os.getcwd(), path)). ``` It basically appends my cwd to the repository, which makes it inoperable.
earthquakesan commented 2019-11-12 13:34:32 +00:00
Author
Copy Link

After removing os.path.abspath, I am still running into the issue with PostgreSQL:

/etc/borgmatic/config.yaml: Error running configuration file
root@borg.eccenca.local:/data/borg: Error running actions for repository
Include pattern 'root/.borgmatic/postgresql_databases/*/*' never matched.
Command 'borg extract root@borg.eccenca.local:/data/borg::client-2019-11-12T05:17:04.453003 sh:root/.borgmatic/postgresql_databases/*/*' returned non-zero exit status 1.

I am not trying to backup any PostgreSQL DBs (yet), what is it trying to do here? oO

After removing os.path.abspath, I am still running into the issue with PostgreSQL: ``` /etc/borgmatic/config.yaml: Error running configuration file root@borg.eccenca.local:/data/borg: Error running actions for repository Include pattern 'root/.borgmatic/postgresql_databases/*/*' never matched. Command 'borg extract root@borg.eccenca.local:/data/borg::client-2019-11-12T05:17:04.453003 sh:root/.borgmatic/postgresql_databases/*/*' returned non-zero exit status 1. ``` I am not trying to backup any PostgreSQL DBs (yet), what is it trying to do here? oO
earthquakesan commented 2019-11-12 13:43:54 +00:00
Author
Copy Link

I commented out the line responsible for inclusion patterns:

 87         #+ (tuple(paths) if paths else ())

And now have this error:

root@client:/usr/local/lib/python3.6/dist-packages/borgmatic# borgmatic restore --archive client-2019-11-12T05:17:04.453003
Traceback (most recent call last):
  File "/usr/local/bin/borgmatic", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/dist-packages/borgmatic/commands/borgmatic.py", line 511, in main
    summary_logs = list(collect_configuration_run_summary_logs(configs, arguments))
  File "/usr/local/lib/python3.6/dist-packages/borgmatic/commands/borgmatic.py", line 425, in collect_configuration_run_summary_logs
    results = list(run_configuration(config_filename, config, arguments))
  File "/usr/local/lib/python3.6/dist-packages/borgmatic/commands/borgmatic.py", line 93, in run_configuration
    repository_path=repository_path,
  File "/usr/local/lib/python3.6/dist-packages/borgmatic/commands/borgmatic.py", line 275, in run_actions
    restore_names or postgresql.get_database_names_from_dumps(dump_patterns),
  File "/usr/local/lib/python3.6/dist-packages/borgmatic/hooks/postgresql.py", line 126, in get_database_configurations
    named_databases = {database['name']: database for database in databases}
TypeError: 'NoneType' object is not iterable

I will try to use borgmatic version prior to PostgreSQL update --> 1.3.26

I commented out the line responsible for inclusion patterns: ``` 87 #+ (tuple(paths) if paths else ()) ``` And now have this error: ``` root@client:/usr/local/lib/python3.6/dist-packages/borgmatic# borgmatic restore --archive client-2019-11-12T05:17:04.453003 Traceback (most recent call last): File "/usr/local/bin/borgmatic", line 11, in <module> sys.exit(main()) File "/usr/local/lib/python3.6/dist-packages/borgmatic/commands/borgmatic.py", line 511, in main summary_logs = list(collect_configuration_run_summary_logs(configs, arguments)) File "/usr/local/lib/python3.6/dist-packages/borgmatic/commands/borgmatic.py", line 425, in collect_configuration_run_summary_logs results = list(run_configuration(config_filename, config, arguments)) File "/usr/local/lib/python3.6/dist-packages/borgmatic/commands/borgmatic.py", line 93, in run_configuration repository_path=repository_path, File "/usr/local/lib/python3.6/dist-packages/borgmatic/commands/borgmatic.py", line 275, in run_actions restore_names or postgresql.get_database_names_from_dumps(dump_patterns), File "/usr/local/lib/python3.6/dist-packages/borgmatic/hooks/postgresql.py", line 126, in get_database_configurations named_databases = {database['name']: database for database in databases} TypeError: 'NoneType' object is not iterable ``` I will try to use borgmatic version prior to PostgreSQL update --> 1.3.26
earthquakesan commented 2019-11-12 13:53:59 +00:00
Author
Copy Link

There is no restore in 1.3.26, but extract command works as expected -- i.e. os.path.abspath is not there yet.

There is no `restore` in 1.3.26, but `extract` command works as expected -- i.e. `os.path.abspath` is not there yet.
witten commented 2019-11-12 16:57:27 +00:00
Owner
Copy Link

Thank you so much for the detailed bug report. I'll have to dig into this. However, some initial reactions off the bat: abspath() is indeed a problem here, and never should have been used on a repository because, as you discovered, it breaks if the repository is remote.

I am not trying to backup any PostgreSQL DBs (yet), what is it trying to do here? oO

borgmatic extract is for extracting files from an archive, and borgmatic restore is for restoring databases! So what it's trying to do here is look for database dump files in your archive and restore them. If there's any way I can make that clearer in the docs or help, please let me know.

Thank you so much for the detailed bug report. I'll have to dig into this. However, some initial reactions off the bat: `abspath()` is indeed a problem here, and never should have been used on a repository because, as you discovered, it breaks if the repository is remote. > I am not trying to backup any PostgreSQL DBs (yet), what is it trying to do here? oO `borgmatic extract` is for extracting files from an archive, and `borgmatic restore` is for restoring databases! So what it's trying to do here is look for database dump files in your archive and restore them. If there's any way I can make that clearer in the docs or help, please let me know.
witten added the
bug
 label 2019-11-12 16:57:33 +00:00
witten commented 2019-11-12 20:08:07 +00:00
Owner
Copy Link

Fixed in borgmatic 1.4.9, just released. Thanks again for reporting, and please let me know whether or not this fix works for you!

Fixed in borgmatic 1.4.9, just released. Thanks again for reporting, and please let me know whether or not this fix works for you!
earthquakesan commented 2019-11-13 09:30:36 +00:00
Author
Copy Link

@witten, thanks for the quick fix!

I don't think improving the docs are necessary, I was just surprised by the command behaviour. It works as expected now:

root@client:/home/vagrant# borgmatic
root@client:/home/vagrant# borgmatic list
client-2019-11-13T01:27:05.230074    Wed, 2019-11-13 01:27:06 [bf2a5108ff6eeaf2940d659258e30821ac61684a16f84e20866624aecd1f6210]
client-2019-11-13T01:27:25.505990    Wed, 2019-11-13 01:27:26 [963cdff124ac7cf29aea80c787bbc925cac922723d32a395c5ee37f6e5e3f358]
root@client:/home/vagrant# borgmatic extract --archive client-2019-11-13T01:27:05.230074
root@client:/home/vagrant# borgmatic restore --archive client-2019-11-13T01:27:05.230074
root@client:/home/vagrant# ls
etc  extract  home  var

restore simply does nothing as I don't have any databases configured there.

@witten, thanks for the quick fix! I don't think improving the docs are necessary, I was just surprised by the command behaviour. It works as expected now: ``` root@client:/home/vagrant# borgmatic root@client:/home/vagrant# borgmatic list client-2019-11-13T01:27:05.230074 Wed, 2019-11-13 01:27:06 [bf2a5108ff6eeaf2940d659258e30821ac61684a16f84e20866624aecd1f6210] client-2019-11-13T01:27:25.505990 Wed, 2019-11-13 01:27:26 [963cdff124ac7cf29aea80c787bbc925cac922723d32a395c5ee37f6e5e3f358] root@client:/home/vagrant# borgmatic extract --archive client-2019-11-13T01:27:05.230074 root@client:/home/vagrant# borgmatic restore --archive client-2019-11-13T01:27:05.230074 root@client:/home/vagrant# ls etc extract home var ``` `restore` simply does nothing as I don't have any databases configured there.
witten commented 2019-11-13 16:51:51 +00:00
Owner
Copy Link

Cool, glad to hear it's working!

Cool, glad to hear it's working!
witten commented 2019-11-13 18:31:16 +00:00
Owner
Copy Link

You should be aware of this related bug, as you may have experienced it: #246.

You should be aware of this related bug, as you may have experienced it: #246.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
main
spot-check
1.8.10
1.8.9
1.8.8
1.8.7
1.8.6
1.8.5
1.8.4
1.8.3
1.8.2
1.8.1
1.8.0
1.7.15
1.7.14
1.7.13
1.7.12
1.7.11
1.7.10
1.7.9
1.7.8
1.7.7
1.7.6
1.7.5
1.7.4
1.7.3
1.7.2
1.7.1
1.7.0
1.6.6
1.6.5
1.6.4
1.6.3
1.6.2
1.6.1
1.6.0
1.5.24
1.5.23
1.5.22
1.5.21
1.5.20
1.5.19
1.5.18
1.5.17
1.5.16
1.5.15
1.5.14
1.5.13
1.5.12
1.5.11
1.5.10
1.5.9
1.5.8
1.5.7
1.5.7.dev0
1.5.6
1.5.5
1.5.4
1.5.3
1.5.2
1.5.1
1.5.0
1.4.22
1.4.21
1.4.20
1.4.19
1.4.18
1.4.17
1.4.16
1.4.15
1.4.14
1.4.13
1.4.12
1.4.11
1.4.10
1.4.9
1.4.8
1.4.7
1.4.6
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0
1.3.26
1.3.25
1.3.24
1.3.23
1.3.22
1.3.21
1.3.20
1.3.19
1.3.18
1.3.17
1.3.16
1.3.15
1.3.14
1.3.13
1.3.12
1.3.11
1.3.10
1.3.9
1.3.8
1.3.7
1.3.6
1.3.5
1.3.4
1.3.3
1.3.2
1.3.1
1.3.0
1.2.18
1.2.17
1.2.16
1.2.15
1.2.14
1.2.13
1.2.12
1.2.11
1.2.10
1.2.9
1.2.8
1.2.7
1.2.6
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.15
1.1.14
1.1.13
1.1.12
1.1.11
1.1.10
1.1.9
1.1.7
1.1.6
1.1.5
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.3
1.0.2
1.0.1
1.0.0
0.1.8
0.1.7
0.1.6
0.1.5
0.1.4
0.1.3
0.1.2
0.1.1
0.1.0
0.0.7
0.0.6
0.0.5
0.0.4
0.0.3
0.0.2
Labels
Clear labels   
bug

   
data loss

   
design finalized

   
good first issue

   
new feature area

   
question / support

   
security

   
waiting for response
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#243
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?