borgmatic-collective/borgmatic
borgmatic-collective/borgmatic
12
62
Fork 49
Code Issues 49 Pull Requests 1 Actions Packages Releases 132 Activity

PyInstaller - Unsetting LD_LIBRARY_PATH when executing programs #926

New Issue
Closed
opened 2024-10-30 23:13:25 +00:00 by ccarney16 · 18 comments
ccarney16 commented 2024-10-30 23:13:25 +00:00
Copy Link

What I'm trying to do and why

Hello,

I was unsure of posting this as a bug or a feature request, but I am currently building a portable version of borgbackup/borgmatic using PyInstaller. The purpose of which is to have a stable version of both programs that I can deploy to various distributions I work with. This works fine until I need to run external scripts or programs to do other work.

When executing a program, there is the potential of seeing this type of warning: bash: /opt/borgbackup/lib/libtinfo.so.6: no version information available (required by bash), which is due to PyInstaller setting the environment variable LD_LIBRARY_PATH. This problem has been resolved by borgbackup in prior releases here: https://borgbackup.readthedocs.io/en/1.4.0/changes.html#version-0-30-0-2016-01-23.

Steps to reproduce

I have a private git repository setup for this, but I can provide both the source/ci scripts and already built binaries if asked via an upload.

Warning is being thrown here and with hooks that require executing external programs:

after_backup:
- bash -c 'find /<directory>/ -maxdepth 1 -name "*.tar" -type f -printf "%T@\t%p\n" | sort --numeric-sort | head --lines=-8 | cut --fields=2- | xargs --replace={} rm --verbose --force {}'

Actual behavior

No response

Expected behavior

No response

Other notes / implementation ideas

This is not a bug of borgmatic itself, but rather an issue of using borgmatic with PyInstaller and maybe other similar tools. The best course of action would be to detect if LD_LIBRARY_PATH is set and to unset it for executing programs. PyInstaller has provided documentation on this here: https://pyinstaller.org/en/stable/runtime-information.html#ld-library-path-libpath-considerations

borgmatic version

1.8.13

borgmatic installation method

custom (pyinstaller)

Borg version

borg 1.4.0

Python version

3.11

Database version (if applicable)

No response

Operating system and version

No response

### What I'm trying to do and why Hello, I was unsure of posting this as a bug or a feature request, but I am currently building a portable version of borgbackup/borgmatic using PyInstaller. The purpose of which is to have a stable version of both programs that I can deploy to various distributions I work with. This works fine until I need to run external scripts or programs to do other work. When executing a program, there is the potential of seeing this type of warning: `bash: /opt/borgbackup/lib/libtinfo.so.6: no version information available (required by bash)`, which is due to PyInstaller setting the environment variable `LD_LIBRARY_PATH`. This problem has been resolved by borgbackup in prior releases here: https://borgbackup.readthedocs.io/en/1.4.0/changes.html#version-0-30-0-2016-01-23. ### Steps to reproduce I have a private git repository setup for this, but I can provide both the source/ci scripts and already built binaries if asked via an upload. Warning is being thrown here and with hooks that require executing external programs: ``` after_backup: - bash -c 'find /<directory>/ -maxdepth 1 -name "*.tar" -type f -printf "%T@\t%p\n" | sort --numeric-sort | head --lines=-8 | cut --fields=2- | xargs --replace={} rm --verbose --force {}' ``` ### Actual behavior _No response_ ### Expected behavior _No response_ ### Other notes / implementation ideas This is not a bug of borgmatic itself, but rather an issue of using borgmatic with PyInstaller and maybe other similar tools. The best course of action would be to detect if LD_LIBRARY_PATH is set and to unset it for executing programs. PyInstaller has provided documentation on this here: https://pyinstaller.org/en/stable/runtime-information.html#ld-library-path-libpath-considerations ### borgmatic version 1.8.13 ### borgmatic installation method custom (pyinstaller) ### Borg version borg 1.4.0 ### Python version 3.11 ### Database version (if applicable) _No response_ ### Operating system and version _No response_
witten commented 2024-11-04 00:47:26 +00:00
Owner
Copy Link

Thanks for taking the time to file this and a explain the issue. A couple of questions though:

  • The example you linked to talks about reading from and using LD_LIBRARY_PATH_ORIG. Should borgmatic implement similar logic? Or is just unsetting LD_LIBRARY_PATH for executed subprocesses sufficient?
  • What about a user not using PyInstaller who legitimately wants to set LD_LIBRARY_PATH before running borgmatic so that a subprocess can find a particular library?
Thanks for taking the time to file this and a explain the issue. A couple of questions though: * The example you linked to talks about reading from and using `LD_LIBRARY_PATH_ORIG`. Should borgmatic implement similar logic? Or is just unsetting `LD_LIBRARY_PATH` for executed subprocesses sufficient? * What about a user not using PyInstaller who legitimately wants to set `LD_LIBRARY_PATH` before running borgmatic so that a subprocess can find a particular library?
ccarney16 commented 2024-11-05 02:25:51 +00:00
Author
Copy Link

At the moment for me, I do not see a need for setting LD_LIBRARY_PATH for my projects, however it would be useful for someone who may need to set it. I would be inclined to support making LD_LIBRARY_PATH be unset or set by LD_LIBRARY_PATH_ORIG to ensure a pyinstaller or similar installation works similar to pip/pipx.

At the moment for me, I do not see a need for setting `LD_LIBRARY_PATH` for my projects, however it would be useful for someone who may need to set it. I would be inclined to support making `LD_LIBRARY_PATH` be unset or set by `LD_LIBRARY_PATH_ORIG` to ensure a pyinstaller or similar installation works similar to pip/pipx.
witten commented 2024-11-10 03:54:10 +00:00
Owner
Copy Link

Okay, how about something like this:

  • Detect whether borgmatic is running under PyInstaller. If so:
    • When running an external program, if LD_LIBRARY_PATH_ORIG is set, replace LD_LIBRARY_PATH with the value of LD_LIBRARY_PATH_ORIG.
    • Otherwise, clear LD_LIBRARY_PATH.

So sort of a combination of code examples from https://pyinstaller.org/en/stable/runtime-information.html and https://pyinstaller.org/en/stable/runtime-information.html#ld-library-path-libpath-considerations

This way, non-PyInstaller users can still use LD_LIBRARY_PATH, while PyInstaller users get (roughly) the behavior you're requesting in this ticket.

Okay, how about something like this: * Detect whether borgmatic is running under PyInstaller. If so: * When running an external program, if `LD_LIBRARY_PATH_ORIG` is set, replace `LD_LIBRARY_PATH` with the value of `LD_LIBRARY_PATH_ORIG`. * Otherwise, clear `LD_LIBRARY_PATH`. So sort of a combination of code examples from https://pyinstaller.org/en/stable/runtime-information.html and https://pyinstaller.org/en/stable/runtime-information.html#ld-library-path-libpath-considerations This way, non-PyInstaller users can still use LD_LIBRARY_PATH, while PyInstaller users get (roughly) the behavior you're requesting in this ticket.
ccarney16 commented 2024-11-10 05:43:26 +00:00
Author
Copy Link

That sounds good, I do not see any issue with that logic.

That sounds good, I do not see any issue with that logic.
đź‘Ť 1
witten commented 2024-11-20 00:00:54 +00:00
Owner
Copy Link

Do you happen to have a mostly working PyInstaller spec file for borgmatic—or just the magical PyInstaller command-line incantation necessary to build a binary of borgmatic that can actually find its dependencies? I'm new to PyInstaller and everything I try makes a binary unable to import borgmatic's third-party modules. Thanks!

Do you happen to have a mostly working PyInstaller spec file for borgmatic—or just the magical PyInstaller command-line incantation necessary to build a binary of borgmatic that can actually find its dependencies? I'm new to PyInstaller and everything I try makes a binary unable to import borgmatic's third-party modules. Thanks!
ccarney16 commented 2024-11-20 00:22:04 +00:00
Author
Copy Link

I do have a complete working spec file that contains both borgbackup and borgmatic in one file. I can provide the entire main branch including the CI scripts and instructions on building the environment. This does require the use of podman and or docker to work. I do have a repository but it is currently on a private gitea instance. Would you prefer me to upload a archive here or can it be sent as an email attachment?

I do have a complete working spec file that contains both borgbackup and borgmatic in one file. I can provide the entire main branch including the CI scripts and instructions on building the environment. This does require the use of podman and or docker to work. I do have a repository but it is currently on a private gitea instance. Would you prefer me to upload a archive here or can it be sent as an email attachment?
witten commented 2024-11-20 03:14:59 +00:00
Owner
Copy Link

Getting that spec file and the CI scripts would be helpful. I do have Podman here, so assuming the setup is fairly replicable, that should work. And uploading an archive on this ticket is fine assuming it's not huge. Another option would be for you to create a new repository here (public or private) with the source.

Thank you!

Getting that spec file and the CI scripts would be helpful. I do have Podman here, so assuming the setup is fairly replicable, that should work. And uploading an archive on this ticket is fine assuming it's not _huge_. Another option would be for you to create a new repository here (public or private) with the source. Thank you!
ccarney16 commented 2024-11-20 20:09:28 +00:00
Author
Copy Link

Here is the copy of the project I am working on. I was unable to create a repository to upload my project, so I will provide a tar.gz of the main branch. Here is the basic set of instructions needed to set this up:

Once downloaded, you just need to go into the project directory borgbackup-master/borgbackup/ You then need to run the following script to setup the development environment, manifest/build/dev-container.sh, though with podman you may need to modify commands to get it working. After that its just run manifest/build/setup-env.sh and manifest/build/build.sh in the development container. Build artifacts will be in the dist/ folder.

Here is the copy of the project I am working on. I was unable to create a repository to upload my project, so I will provide a tar.gz of the main branch. Here is the basic set of instructions needed to set this up: Once downloaded, you just need to go into the project directory `borgbackup-master/borgbackup/` You then need to run the following script to setup the development environment, `manifest/build/dev-container.sh`, though with podman you may need to modify commands to get it working. After that its just run `manifest/build/setup-env.sh` and `manifest/build/build.sh` in the development container. Build artifacts will be in the `dist/` folder.
borgbackup-pyinstaller.tar.gz
4.8 KiB
witten commented 2024-11-25 06:21:51 +00:00
Owner
Copy Link

Unfortunately I can't seem to repro this even with your dev container. I went through all the steps and ended up with a build of borgmatic that can apparently run your after_backup command (with a real directory substituted) without a problem. I even tried creating twenty tar files, and indeed some of them got removed by the after_backup command. Here's my borgmatic config files (comments removed):

source_directories:
    - /workdir/.woodpecker
repositories:
    - path: test.borg
      label: test
local_path: dist/borgbackup.linux.x86_64/borg
keep_daily: 7
after_backup:
    - bash -c 'find /workdir/ -maxdepth 1 -name "*.tar" -type f -printf "%T@\t%p\n" | sort --numeric-sort | head --lines=-8 | cut --fields=2- | xargs --replace={} rm --verbose --force {}'

And here's my borgmatic invocation:

# ./dist/borgbackup.linux.x86_64/borgmatic -c test.yaml create -v 2
...
/workdir/test.yaml: Running command for post-backup hook
bash -c 'find /workdir/ -maxdepth 1 -name "*.tar" -type f -printf "%T@\t%p\n" | sort --numeric-sort | head --lines=-8 | cut --fields=2- | xargs --replace={} rm --verbose --force {}'
removed '/workdir/w.tar'
removed '/workdir/w2.tar'
removed '/workdir/w3.tar'
removed '/workdir/w4.tar'
removed '/workdir/w5.tar'
removed '/workdir/w6.tar'
removed '/workdir/w7.tar'
removed '/workdir/w8.tar'
removed '/workdir/w9.tar'
removed '/workdir/w10.tar'
removed '/workdir/w11.tar'
removed '/workdir/w12.tar'
...

So any ideas what I can do to trigger the library error?

Unfortunately I can't seem to repro this even with your dev container. I went through all the steps and ended up with a build of borgmatic that can apparently run your `after_backup` command (with a real directory substituted) without a problem. I even tried creating twenty tar files, and indeed some of them got removed by the `after_backup` command. Here's my borgmatic config files (comments removed): ```yaml source_directories: - /workdir/.woodpecker repositories: - path: test.borg label: test local_path: dist/borgbackup.linux.x86_64/borg keep_daily: 7 after_backup: - bash -c 'find /workdir/ -maxdepth 1 -name "*.tar" -type f -printf "%T@\t%p\n" | sort --numeric-sort | head --lines=-8 | cut --fields=2- | xargs --replace={} rm --verbose --force {}' ``` And here's my borgmatic invocation: ``` # ./dist/borgbackup.linux.x86_64/borgmatic -c test.yaml create -v 2 ... /workdir/test.yaml: Running command for post-backup hook bash -c 'find /workdir/ -maxdepth 1 -name "*.tar" -type f -printf "%T@\t%p\n" | sort --numeric-sort | head --lines=-8 | cut --fields=2- | xargs --replace={} rm --verbose --force {}' removed '/workdir/w.tar' removed '/workdir/w2.tar' removed '/workdir/w3.tar' removed '/workdir/w4.tar' removed '/workdir/w5.tar' removed '/workdir/w6.tar' removed '/workdir/w7.tar' removed '/workdir/w8.tar' removed '/workdir/w9.tar' removed '/workdir/w10.tar' removed '/workdir/w11.tar' removed '/workdir/w12.tar' ... ``` So any ideas what I can do to trigger the library error?
ccarney16 commented 2024-11-25 16:33:37 +00:00
Author
Copy Link

The problem does not exist if you use the same distribution that pyinstaller is building on. You should try running the same build on debian or ubuntu as that is where I get those warnings from.

The problem does not exist if you use the same distribution that pyinstaller is building on. You should try running the same build on debian or ubuntu as that is where I get those warnings from.
witten commented 2024-11-25 19:00:26 +00:00
Owner
Copy Link

Got it, thanks. I have a repro on Debian now.

Got it, thanks. I have a repro on Debian now.
witten commented 2024-11-25 19:14:14 +00:00
Owner
Copy Link

Actually, I'm not sure I actually have a repro now. I am getting a library error, but it's immediately upon running borgmatic rather than once the before_backup hook is triggered:

$ ./borgmatic -c test.yaml -v 2 repo-create -e repokey
[PYI-2973817:ERROR] Failed to load Python shared library '/home/witten/lib/libpython3.11.so.1.0': dlopen: /home/witten/lib/libpython3.11.so.1.0: cannot open shared object file: No such file or directory

So that looks like a problem with the PyInstaller binary itself. Any ideas here?

Actually, I'm not sure I actually have a repro now. I am getting a library error, but it's immediately upon running borgmatic rather than once the `before_backup` hook is triggered: ``` $ ./borgmatic -c test.yaml -v 2 repo-create -e repokey [PYI-2973817:ERROR] Failed to load Python shared library '/home/witten/lib/libpython3.11.so.1.0': dlopen: /home/witten/lib/libpython3.11.so.1.0: cannot open shared object file: No such file or directory ``` So that looks like a problem with the PyInstaller binary itself. Any ideas here?
ccarney16 commented 2024-11-25 19:49:30 +00:00
Author
Copy Link

Is this running in the stock docker hub Debian 11/12 container? I am unable to reproduce that shared library error for borgmatic itself since it should be included with the build artifacts in the lib folder. Is /home/witten/lib/libpython3.11.so.1.0 a file that exists? I forgot to mention the build is not exactly a single binary in my configuration, and I opted to use a onedir build using lib as the folder name since it does not require to be decompressed into /tmp. This must exist next to the borg/borgmatic binaries in the same folder.

Is this running in the stock docker hub Debian 11/12 container? I am unable to reproduce that shared library error for borgmatic itself since it should be included with the build artifacts in the `lib` folder. Is `/home/witten/lib/libpython3.11.so.1.0` a file that exists? I forgot to mention the build is not exactly a single binary in my configuration, and I opted to use a onedir build using `lib` as the folder name since it does not require to be decompressed into `/tmp`. This must exist next to the borg/borgmatic binaries in the same folder.
witten commented 2024-11-25 20:55:21 +00:00
Owner
Copy Link

Thanks! This is on a non-container Debian 12 system (in a VPS VM). I didn't realize the produced lib/ directory needed to be present as well. Once I copied that over, I do seem to have an actual repro now... borgmatic is giving me bash: /home/witten/lib/libtinfo.so.6: no version information available (required by bash) from running the before_backup hook, although it does appear to also remove the tar files without error.

Thanks! This is on a non-container Debian 12 system (in a VPS VM). I didn't realize the produced `lib/` directory needed to be present as well. Once I copied that over, I do seem to have an actual repro now... borgmatic is giving me `bash: /home/witten/lib/libtinfo.so.6: no version information available (required by bash)` from running the `before_backup` hook, although it does appear to also remove the tar files without error.
ccarney16 commented 2024-11-26 01:24:35 +00:00
Author
Copy Link

The program does work, and for the machine I was testing this on, there were no unintended side effects so far. This is just a warning due to the library path being set by pyinstaller. This may cause unintended side effects for programs that need to run. I don't have a use case right now that has been causing crashes, as I am currently still testing this installation method out. It does look like that the borgbackup developers did mention this kind of problem in one of their commits: e006a6f368.

The program does work, and for the machine I was testing this on, there were no unintended side effects so far. This is just a warning due to the library path being set by pyinstaller. This may cause unintended side effects for programs that need to run. I don't have a use case right now that has been causing crashes, as I am currently still testing this installation method out. It does look like that the borgbackup developers did mention this kind of problem in one of their commits: https://github.com/ThomasWaldmann/borg/commit/e006a6f368aba610d6086af8aa1b486708c3f319.
witten commented 2024-11-26 04:18:24 +00:00
Owner
Copy Link

Okay, this is now addressed in main and will be part of the next release! The fix only applies to command hooks (before_backup, etc.) currently—not the calls to Borg itself or other built-in command invocations (database dumping, etc.).

Thanks again for filing this and proving an entire dev environment!

Okay, this is now addressed in main and will be part of the next release! The fix only applies to command hooks (`before_backup`, etc.) currently—not the calls to Borg itself or other built-in command invocations (database dumping, etc.). Thanks again for filing this and proving an entire dev environment!
ccarney16 commented 2024-11-26 16:17:23 +00:00
Author
Copy Link

Thank you so much for looking into and fixing this. By the way, feel free to use and adapt the spec file and other scripts provided if you want to offer a single binary/distribution option. What I sent was an early build and contains some hacks to get it building.

Thank you so much for looking into and fixing this. By the way, feel free to use and adapt the spec file and other scripts provided if you want to offer a single binary/distribution option. What I sent was an early build and contains some hacks to get it building.
đź‘Ť 1
witten commented 2024-12-07 04:33:25 +00:00
Owner
Copy Link

Released in borgmatic 1.9.4!

Released in borgmatic 1.9.4!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
blocked
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No Label
Milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#926
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?