borgmatic-collective/borgmatic
borgmatic-collective
/
borgmatic
15
56
Fork 39
Code Issues 52 Pull Requests 1 Actions Packages Releases 121 Activity

Deep merging issue on Ubuntu ARM64 22.04 #622

New Issue
Closed
opened 2022-12-21 13:43:28 +00:00 by maxhamon · 12 comments
maxhamon commented 2022-12-21 13:43:28 +00:00
Copy Link

Hi

What I'm trying to do and why

I try to use deep merging configuration. Deploying same installation and configurations on different OS but not working on Ubuntu.
No issue without deep merge.

Steps to reproduce (if a bug)

2 yaml files test.yml and common.yml with very basic configuration

<<: !include /tmp/common.yml

location:
  source_directories:
    - /tmp/test
  repositories:
    - 'ssh://na7c728u@na7c728u.repo.borgbase.com/./repo'

storage:
  archive_name_format: 'test-{now}'

storage:
  encryption_passphrase: this-is-a-test

Actual behavior (if a bug)

With previous configuration, trying to list,create,... crash

$ borgmatic -c test.yml --verbosity 2 list

Traceback (most recent call last):
  File "/opt/borgmatic/bin/borgmatic", line 8, in <module>
    sys.exit(main())
  File "/opt/borgmatic/lib/python3.10/site-packages/borgmatic/commands/borgmatic.py", line 1010, in main
    configs, parse_logs = load_configurations(
  File "/opt/borgmatic/lib/python3.10/site-packages/borgmatic/commands/borgmatic.py", line 780, in load_configurations
    configs[config_filename], parse_logs = validate.parse_configuration(
  File "/opt/borgmatic/lib/python3.10/site-packages/borgmatic/config/validate.py", line 99, in parse_configuration
    config = load.load_configuration(config_filename)
  File "/opt/borgmatic/lib/python3.10/site-packages/borgmatic/config/load.py", line 33, in load_configuration
    return yaml.load(open(filename))
  File "/opt/borgmatic/lib/python3.10/site-packages/ruamel/yaml/main.py", line 434, in load
    return constructor.get_single_data()
  File "/opt/borgmatic/lib/python3.10/site-packages/ruamel/yaml/constructor.py", line 121, in get_single_data
    return self.construct_document(node)
  File "/opt/borgmatic/lib/python3.10/site-packages/ruamel/yaml/constructor.py", line 131, in construct_document
    for _dummy in generator:
  File "/opt/borgmatic/lib/python3.10/site-packages/ruamel/yaml/constructor.py", line 674, in construct_yaml_map
    value = self.construct_mapping(node)
  File "/opt/borgmatic/lib/python3.10/site-packages/ruamel/yaml/constructor.py", line 444, in construct_mapping
    self.flatten_mapping(node)
  File "/opt/borgmatic/lib/python3.10/site-packages/borgmatic/config/load.py", line 210, in flatten_mapping
    included_value = representer.represent_data(self.construct_object(value_node))
  File "/opt/borgmatic/lib/python3.10/site-packages/ruamel/yaml/constructor.py", line 154, in construct_object
    data = self.construct_non_recursive_object(node)
  File "/opt/borgmatic/lib/python3.10/site-packages/ruamel/yaml/constructor.py", line 189, in construct_non_recursive_object
    data = constructor(self, node)
  File "/opt/borgmatic/lib/python3.10/site-packages/borgmatic/config/load.py", line 45, in include_configuration
    include_directories = [os.getcwd(), os.path.abspath(os.path.dirname(loader.stream.name))]
AttributeError: 'Include_constructor' object has no attribute 'stream'

Expected behavior (if a bug)

Listing repository or create new backup
Without using deep merging it's working well

Other notes / implementation ideas

Environment

borgmatic version: [version here]

pip show borgmatic | grep ^Version
Version: 1.7.5

borgmatic installation method: [e.g., Debian package, Docker container, etc.]

Pip

Borg version: [version here]

borg --version
borg 1.2.2

Python version: [version here]

python3 --version
Python 3.10.6

operating system and version: [OS here]

lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.1 LTS
Release:	22.04
Codename:	jammy

Same configuration files, using same version of borgmatic/borg on Debian and Archlinux are ok.

Hi #### What I'm trying to do and why I try to use deep merging configuration. Deploying same installation and configurations on different OS but not working on Ubuntu. No issue without deep merge. #### Steps to reproduce (if a bug) 2 yaml files `test.yml` and `common.yml` with very basic configuration ```yaml <<: !include /tmp/common.yml location: source_directories: - /tmp/test repositories: - 'ssh://na7c728u@na7c728u.repo.borgbase.com/./repo' storage: archive_name_format: 'test-{now}' ``` ```yaml storage: encryption_passphrase: this-is-a-test ``` #### Actual behavior (if a bug) With previous configuration, trying to list,create,... crash ```bash $ borgmatic -c test.yml --verbosity 2 list Traceback (most recent call last): File "/opt/borgmatic/bin/borgmatic", line 8, in <module> sys.exit(main()) File "/opt/borgmatic/lib/python3.10/site-packages/borgmatic/commands/borgmatic.py", line 1010, in main configs, parse_logs = load_configurations( File "/opt/borgmatic/lib/python3.10/site-packages/borgmatic/commands/borgmatic.py", line 780, in load_configurations configs[config_filename], parse_logs = validate.parse_configuration( File "/opt/borgmatic/lib/python3.10/site-packages/borgmatic/config/validate.py", line 99, in parse_configuration config = load.load_configuration(config_filename) File "/opt/borgmatic/lib/python3.10/site-packages/borgmatic/config/load.py", line 33, in load_configuration return yaml.load(open(filename)) File "/opt/borgmatic/lib/python3.10/site-packages/ruamel/yaml/main.py", line 434, in load return constructor.get_single_data() File "/opt/borgmatic/lib/python3.10/site-packages/ruamel/yaml/constructor.py", line 121, in get_single_data return self.construct_document(node) File "/opt/borgmatic/lib/python3.10/site-packages/ruamel/yaml/constructor.py", line 131, in construct_document for _dummy in generator: File "/opt/borgmatic/lib/python3.10/site-packages/ruamel/yaml/constructor.py", line 674, in construct_yaml_map value = self.construct_mapping(node) File "/opt/borgmatic/lib/python3.10/site-packages/ruamel/yaml/constructor.py", line 444, in construct_mapping self.flatten_mapping(node) File "/opt/borgmatic/lib/python3.10/site-packages/borgmatic/config/load.py", line 210, in flatten_mapping included_value = representer.represent_data(self.construct_object(value_node)) File "/opt/borgmatic/lib/python3.10/site-packages/ruamel/yaml/constructor.py", line 154, in construct_object data = self.construct_non_recursive_object(node) File "/opt/borgmatic/lib/python3.10/site-packages/ruamel/yaml/constructor.py", line 189, in construct_non_recursive_object data = constructor(self, node) File "/opt/borgmatic/lib/python3.10/site-packages/borgmatic/config/load.py", line 45, in include_configuration include_directories = [os.getcwd(), os.path.abspath(os.path.dirname(loader.stream.name))] AttributeError: 'Include_constructor' object has no attribute 'stream' ``` #### Expected behavior (if a bug) Listing repository or create new backup Without using deep merging it's working well #### Other notes / implementation ideas #### Environment **borgmatic version:** [version here] ```bash pip show borgmatic | grep ^Version Version: 1.7.5 ``` **borgmatic installation method:** [e.g., Debian package, Docker container, etc.] Pip **Borg version:** [version here] ```bash borg --version borg 1.2.2 ``` **Python version:** [version here] ```bash python3 --version Python 3.10.6 ``` **operating system and version:** [OS here] ```bash lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 22.04.1 LTS Release: 22.04 Codename: jammy ``` Same configuration files, using same version of borgmatic/borg on Debian and Archlinux are ok.
witten commented 2022-12-21 16:50:37 +00:00
Owner
Copy Link

Thanks for the detailed issue report! Looking at the error message, my guess is that the version of ruamel.yaml you're using doesn't work with borgmatic. You can find the version installed via: pip show ruamel.yaml | grep ^Version

Thanks for the detailed issue report! Looking at the error message, my guess is that the version of `ruamel.yaml` you're using doesn't work with borgmatic. You can find the version installed via: `pip show ruamel.yaml | grep ^Version`
maxhamon commented 2022-12-21 17:45:48 +00:00
Author
Copy Link

I think ruamel.yaml is installed as a dependancies via pip, sadly it's latest

$ pip show ruamel.yaml | grep ^Version
Version: 0.17.21

Below a list of all pip packages in my venv

Package            Version
------------------ ---------
attrs              22.2.0
borgbackup         1.2.2
borgmatic          1.7.5
build              0.9.0
certifi            2022.12.7
charset-normalizer 2.1.1
click              8.1.3
colorama           0.4.6
Cython             0.29.32
idna               3.4
jsonschema         4.17.3
msgpack            1.0.4
packaging          22.0
pep517             0.13.0
pip                22.3.1
pip-tools          6.12.1
pkgconfig          1.5.5
pyrsistent         0.19.2
requests           2.28.1
ruamel.yaml        0.17.21
ruamel.yaml.clib   0.2.7
setuptools         65.6.3
tomli              2.0.1
urllib3            1.26.13
wheel              0.38.4

Can't understand / reproduce what's wrong....Attach a requirements.txt

I think `ruamel.yaml` is installed as a dependancies via pip, sadly it's latest ```bash $ pip show ruamel.yaml | grep ^Version Version: 0.17.21 ``` Below a list of all pip packages in my venv ```bash Package Version ------------------ --------- attrs 22.2.0 borgbackup 1.2.2 borgmatic 1.7.5 build 0.9.0 certifi 2022.12.7 charset-normalizer 2.1.1 click 8.1.3 colorama 0.4.6 Cython 0.29.32 idna 3.4 jsonschema 4.17.3 msgpack 1.0.4 packaging 22.0 pep517 0.13.0 pip 22.3.1 pip-tools 6.12.1 pkgconfig 1.5.5 pyrsistent 0.19.2 requests 2.28.1 ruamel.yaml 0.17.21 ruamel.yaml.clib 0.2.7 setuptools 65.6.3 tomli 2.0.1 urllib3 1.26.13 wheel 0.38.4 ``` Can't understand / reproduce what's wrong....Attach a requirements.txt
requirements.txt
373 B
witten commented 2022-12-21 20:49:57 +00:00
Owner
Copy Link

That's odd.. I'm using the same version of ruamel.yaml locally but I'm not getting any errors with includes. Someone else is getting the same error as you using Python 3.11, but you're on 3.10 (as am I). Let me look into this and see if I can manage to reproduce the issue.

That's odd.. I'm using the same version of `ruamel.yaml` locally but I'm not getting any errors with includes. [Someone else is getting the same error as you](https://github.com/borgmatic-collective/docker-borgmatic/issues/188) using Python 3.11, but you're on 3.10 (as am I). Let me look into this and see if I can manage to reproduce the issue.
witten added the
bug
 label 2022-12-21 20:50:05 +00:00
witten commented 2022-12-22 00:35:27 +00:00
Owner
Copy Link

Unfortunately, I still haven't been able to reproduce this. What I tried was running an Ubuntu 22.04 Docker image, creating a venv inside of it, and then installing the packages from your requirements.txt (and the system packages necessary for them to build). Upon running borgmatic with your sample configuration (changed to use a local repository), it runs just fine without error!

Is there anything else particular about your Ubuntu system that might be relevant to this issue? And are you sure the reported version of ruamel.yaml, for instance, is the version inside of your venv instead of a package installed at the system level?

Thanks for your patience here.

Unfortunately, I still haven't been able to reproduce this. What I tried was running an Ubuntu 22.04 Docker image, creating a venv inside of it, and then installing the packages from your `requirements.txt` (and the system packages necessary for them to build). Upon running borgmatic with your sample configuration (changed to use a local repository), it runs just fine without error! Is there anything else particular about your Ubuntu system that might be relevant to this issue? And are you sure the reported version of `ruamel.yaml`, for instance, is the version inside of your venv instead of a package installed at the system level? Thanks for your patience here.
maxhamon commented 2022-12-22 13:32:46 +00:00
Author
Copy Link

Thanks, I have try to pop a new vm with same configuration and can't reproduce too. Yes runamel.yaml is in my venv.

I'll try to remove every irrelevant python package at system level to see if it helps. One thing I didn't mention (not sure it's relevant) : it's an arm64 system

Thanks, I have try to pop a new vm with same configuration and can't reproduce too. Yes `runamel.yaml` is in my venv. I'll try to remove every irrelevant python package at system level to see if it helps. One thing I didn't mention ~~(not sure it's relevant)~~ : it's an arm64 system
maxhamon commented 2022-12-22 14:38:57 +00:00
Author
Copy Link

I've found a way to reproduce it ! Using a docker ubuntu:22.04 arm64 image (I'm using an oracle cloud instance)

$ apt update && apt install vim libssl-dev libacl1-dev libacl1 build-essential python3-setuptools python3-dev python3-pip python3-pkgconfig python3-msgpack python3-venv
$ python3 -m venv /opt/borgmatic
$ . /opt/borgmatic/bin/activate
$ pip3 install -r requirements.txt
I've found a way to reproduce it ! Using a docker ubuntu:22.04 arm64 image (I'm using an oracle cloud instance) ```bash $ apt update && apt install vim libssl-dev libacl1-dev libacl1 build-essential python3-setuptools python3-dev python3-pip python3-pkgconfig python3-msgpack python3-venv $ python3 -m venv /opt/borgmatic $ . /opt/borgmatic/bin/activate $ pip3 install -r requirements.txt ```
witten commented 2022-12-22 17:10:50 +00:00
Owner
Copy Link

Awesome, thank you! I'll see if I can get access to an arm64 machine and give that a shot. I may also have an alternate approach to integrating the includes with ruamel.yaml that may work better.

Awesome, thank you! I'll see if I can get access to an arm64 machine and give that a shot. I may also have an alternate approach to integrating the includes with `ruamel.yaml` that may work better.
witten commented 2022-12-23 06:54:28 +00:00
Owner
Copy Link

Finally got a repro, and I have a fix in hand, too! I'll do some additional testing and let you know when it's integrated.

Finally got a repro, and I have a fix in hand, too! I'll do some additional testing and let you know when it's integrated.
maxhamon changed title from Deep merging issue on Ubuntu 22.04 to Deep merging issue on Ubuntu ARM64 22.04 2022-12-23 09:01:54 +00:00
maxhamon commented 2022-12-23 09:04:02 +00:00
Author
Copy Link

Awesome, excellent work.Will be happy to test.

Awesome, excellent work.Will be happy to test.
witten commented 2022-12-23 18:13:46 +00:00
Owner
Copy Link

Okay, this should be fixed in master and will be part of the next release. If you'd like to test before then (totally optional), feel free to download an updated copy of load.py and use it to overwrite your local /opt/borgmatic/lib/python3.10/site-packages/borgmatic/config/load.py file. Thanks again for all your help in diagnosing this!

Okay, this should be fixed in master and will be part of the next release. If you'd like to test before then (totally optional), feel free to download an updated copy of [`load.py`](https://projects.torsion.org/borgmatic-collective/borgmatic/raw/branch/master/borgmatic/config/load.py) and use it to overwrite your local `/opt/borgmatic/lib/python3.10/site-packages/borgmatic/config/load.py` file. Thanks again for all your help in diagnosing this!
❤️ 1
s1shed commented 2023-01-09 12:33:46 +00:00
Contributor
Copy Link

Okay, this should be fixed in master and will be part of the next release. If you'd like to test before then (totally optional), feel free to download an updated copy of load.py and use it to overwrite your local /opt/borgmatic/lib/python3.10/site-packages/borgmatic/config/load.py file. Thanks again for all your help in diagnosing this!

This updated copy of load.py fixed my problem with Python 3.11.

> Okay, this should be fixed in master and will be part of the next release. If you'd like to test before then (totally optional), feel free to download an updated copy of [`load.py`](https://projects.torsion.org/borgmatic-collective/borgmatic/raw/branch/master/borgmatic/config/load.py) and use it to overwrite your local `/opt/borgmatic/lib/python3.10/site-packages/borgmatic/config/load.py` file. Thanks again for all your help in diagnosing this! This updated copy of `load.py` fixed my problem with Python 3.11.
witten commented 2023-01-26 23:25:22 +00:00
Owner
Copy Link

Just released in borgmatic 1.7.6!

Just released in borgmatic 1.7.6!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
main
spot-check
1.8.10
1.8.9
1.8.8
1.8.7
1.8.6
1.8.5
1.8.4
1.8.3
1.8.2
1.8.1
1.8.0
1.7.15
1.7.14
1.7.13
1.7.12
1.7.11
1.7.10
1.7.9
1.7.8
1.7.7
1.7.6
1.7.5
1.7.4
1.7.3
1.7.2
1.7.1
1.7.0
1.6.6
1.6.5
1.6.4
1.6.3
1.6.2
1.6.1
1.6.0
1.5.24
1.5.23
1.5.22
1.5.21
1.5.20
1.5.19
1.5.18
1.5.17
1.5.16
1.5.15
1.5.14
1.5.13
1.5.12
1.5.11
1.5.10
1.5.9
1.5.8
1.5.7
1.5.7.dev0
1.5.6
1.5.5
1.5.4
1.5.3
1.5.2
1.5.1
1.5.0
1.4.22
1.4.21
1.4.20
1.4.19
1.4.18
1.4.17
1.4.16
1.4.15
1.4.14
1.4.13
1.4.12
1.4.11
1.4.10
1.4.9
1.4.8
1.4.7
1.4.6
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0
1.3.26
1.3.25
1.3.24
1.3.23
1.3.22
1.3.21
1.3.20
1.3.19
1.3.18
1.3.17
1.3.16
1.3.15
1.3.14
1.3.13
1.3.12
1.3.11
1.3.10
1.3.9
1.3.8
1.3.7
1.3.6
1.3.5
1.3.4
1.3.3
1.3.2
1.3.1
1.3.0
1.2.18
1.2.17
1.2.16
1.2.15
1.2.14
1.2.13
1.2.12
1.2.11
1.2.10
1.2.9
1.2.8
1.2.7
1.2.6
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.15
1.1.14
1.1.13
1.1.12
1.1.11
1.1.10
1.1.9
1.1.7
1.1.6
1.1.5
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.3
1.0.2
1.0.1
1.0.0
0.1.8
0.1.7
0.1.6
0.1.5
0.1.4
0.1.3
0.1.2
0.1.1
0.1.0
0.0.7
0.0.6
0.0.5
0.0.4
0.0.3
0.0.2
Labels
Clear labels   
bug

   
data loss

   
design finalized

   
good first issue

   
new feature area

   
question / support

   
security

   
waiting for response
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#622
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?