Create automated binaries for borgmatic #151

Closed
opened 2019-03-12 14:14:46 +00:00 by dverhelst · 18 comments

Hi,

Enterprise linux operating systems don't easily support Python3/Pip3.
Even from a deployment perspective, providing a binary makes sense for a python3 package.

Working on https://github.com/FiaasCo/borgbackup taught us deploying a binary of borg made much more sense.

Are there plans to implement a packaging system (such as https://github.com/cmarquardt/borgmatic-binary) for borgmatic?

Can we get this into a feature and start working on it?

Please share your thoughts.

Hi, Enterprise linux operating systems don't easily support Python3/Pip3. Even from a deployment perspective, providing a binary makes sense for a python3 package. Working on https://github.com/FiaasCo/borgbackup taught us deploying a binary of borg made much more sense. Are there plans to implement a packaging system (such as https://github.com/cmarquardt/borgmatic-binary) for borgmatic? Can we get this into a feature and start working on it? Please share your thoughts.
Owner

You've probably seen it already, but there are a bunch of third-party distribution-specific options for installing borgmatic.

The borgmatic-binary project you linked to though looks like exactly what you're asking about here. Is the issue that it's not updated on every borgmatic release, and you'd like something more official or automated?

Also, could you say more about your use case, your Linux distribution, etc? Is the problem that pip just isn't available at all, or is it an outdated version? Do you normally install things with the enterprise Linux's package manager, or do you usually install binaries?

Thanks.

You've probably seen it already, but there are a bunch of third-party distribution-specific options for [installing borgmatic](https://torsion.org/borgmatic/docs/how-to/set-up-backups/#other-ways-to-install). The borgmatic-binary project you linked to though looks like exactly what you're asking about here. Is the issue that it's not updated on every borgmatic release, and you'd like something more official or automated? Also, could you say more about your use case, your Linux distribution, etc? Is the problem that pip just isn't available at all, or is it an outdated version? Do you normally install things with the enterprise Linux's package manager, or do you usually install binaries? Thanks.
Author

The borgmatic-binary would be perfect if it was implemented into borgmatic just as borg does for releasing cross-platform binaries. That way I am able to quickly run borg on a variety of platform/cpus and deploy at scale.

Having widespread packages available for different distributions really is a good thing, but from an automation point of view, it requires adding 1 or more different repositories, packages, .. just to be able to get it to work. Every distribution is almost a custom-job to get it done (ever tried 'managing' python3/pip3 packages on CentOS?).
(note: i'm all in favor of using regular packages (rpm/deb/...) but in some cases it causes more trouble than it fixes things if they are not in the main repositories of the distributions)

Having to go through that hassle vs. being able to just take a binary is just a much cleaner option from an automation point of view. It also allows for easy upgrades/downgrades. Borgmatic will be instantly available (platforms,cpus) and much easier to start playing with.

I would be willing to help out and try to implement this with pleasure provided we decide on what needs to be done. (I'm a automation/cicd fanatic, but I lack python dev skills).

The alternative will be to spawn a separate disconnected system to get these binaries built and deployed (much like the person who created the borgmatic-binary project). This is rarely a good thing.

The borgmatic-binary would be perfect if it was implemented into borgmatic just as borg does for releasing cross-platform binaries. That way I am able to quickly run borg on a variety of platform/cpus and deploy at scale. Having widespread packages available for different distributions really is a good thing, but from an automation point of view, it requires adding 1 or more different repositories, packages, .. just to be able to get it to work. Every distribution is almost a custom-job to get it done (ever tried 'managing' python3/pip3 packages on CentOS?). (note: i'm all in favor of using regular packages (rpm/deb/...) but in some cases it causes more trouble than it fixes things if they are not in the main repositories of the distributions) Having to go through that hassle vs. being able to just take a binary is just a much cleaner option from an automation point of view. It also allows for easy upgrades/downgrades. Borgmatic will be instantly available (platforms,cpus) and much easier to start playing with. I would be willing to help out and try to implement this with pleasure provided we decide on what needs to be done. (I'm a automation/cicd fanatic, but I lack python dev skills). The alternative will be to spawn a separate disconnected system to get these binaries built and deployed (much like the person who created the borgmatic-binary project). This is rarely a good thing.

Hello,

I would also like to have a pre-compiled binary for borgmatic!

It would help with running it on my Unraid server. Currently, I need to re-install it every reboot because packages installed do not persist after a reboot.

With a pre-compiled binary like I am using with Borg, I could just create a symbolic link at boot time rather than have to re-install via pip3 every boot...

Thank you!

Hello, I would also like to have a pre-compiled binary for borgmatic! It would help with running it on my Unraid server. Currently, I need to re-install it every reboot because packages installed do not persist after a reboot. With a pre-compiled binary like I am using with Borg, I could just create a symbolic link at boot time rather than have to re-install via pip3 every boot... Thank you!
Owner

Binaries seems like a reasonable thing to have for use cases as described here, but also a fair amount of work to set up and maintain compared to the platform-agnostic packages that are built and distributed for borgmatic today. For instance, the very meager CI infrastructure that I'm running today may not be sufficient for building binaries.

Given that this need sounds like it's for an enterprise Linux distribution in at least one case (maybe less so for Unraid), does a binary distribution of borgmatic + Borg sound like something your employer would purchase a license for? If so, that would certainly help me find the time to work on this sooner rather than later. (I'm assuming you'd be running enterprise Linux in a company of some sort with a budget for such things. If not, let me know!)

Binaries seems like a reasonable thing to have for use cases as described here, but also a fair amount of work to set up and maintain compared to the platform-agnostic packages that are built and distributed for borgmatic today. For instance, the very meager CI infrastructure that I'm running today may not be sufficient for building binaries. Given that this need sounds like it's for an enterprise Linux distribution in at least one case (maybe less so for Unraid), does a binary distribution of borgmatic + Borg sound like something your employer would purchase a license for? If so, that would certainly help me find the time to work on this sooner rather than later. (I'm assuming you'd be running enterprise Linux in a company of some sort with a budget for such things. If not, let me know!)
Author

This request is not for an Enterprise-need, but it's based on experiences on both open-source and enterprise.
It's the sheer variety of distro's and older packages, obsolete python's, ... that makes using borgmatic difficult.
Imagine a mix of:

  • el6
  • el7
  • squeeze
  • bionic
  • freebsd (freebsd)
  • ....

Currently I use a borg-wrapper-script which uses a built borg-binary and this approach works on (almost) all platforms. I can prebuild the binary and put it on the system without any problem and I can back up all my machines using borg.

This approach is what I would like for borgmatic.

If you point me in the correct direction on how to optimally build that binary I might be able to work something out for CI.

(again, this is not an enterprise-need, but an identified shortcoming of Borgmatic for me to be able to use it on all platforms borg support)

This request is not for an Enterprise-need, but it's based on experiences on both open-source and enterprise. It's the sheer variety of distro's and older packages, obsolete python's, ... that makes using borgmatic difficult. Imagine a mix of: - el6 - el7 - squeeze - bionic - freebsd (freebsd) - .... Currently I use a borg-wrapper-script which uses a built borg-binary and this approach works on (almost) all platforms. I can prebuild the binary and put it on the system without any problem and I can back up all my machines using borg. This approach is what I would like for borgmatic. If you point me in the correct direction on how to optimally build that binary I might be able to work something out for CI. (again, this is not an enterprise-need, but an identified shortcoming of Borgmatic for me to be able to use it on all platforms borg support)
Owner

Thanks for clarifying! One thing I'm still a little bit unclear on though is the gap you see for something like borgmatic-binary to meet this need.

Earlier, you said:

The borgmatic-binary would be perfect if it was implemented into borgmatic just as borg does for releasing cross-platform binaries. That way I am able to quickly run borg on a variety of platform/cpus and deploy at scale.

So is the problem just the number of platforms that borgmatic-binary has binary releases for? It looks like right now, there are only releases for x86_64 Linux. Sounds like you ideally would like FreeBSD releases as well. Are there other platforms or architectures? And are there other concerns with using binary releases from a third-party project?

It appears that borgmatic-binary includes build instructions for any platform supporting Borg even if there aren't pre-built binaries. So if you have your own CI, that may get your started building borgmatic for your target architectures/platforms.

I'm asking about all this because I want to understand your platform requirements here. borgmatic's existing CI server is x86_64 Linux, so it would require some investment (both in time and hosting cost) to support other platforms like: FreeBSD, non-x86_64, etc.

Thanks for clarifying! One thing I'm still a little bit unclear on though is the gap you see for something like [borgmatic-binary](https://github.com/cmarquardt/borgmatic-binary) to meet this need. Earlier, you said: > The borgmatic-binary would be perfect if it was implemented into borgmatic just as borg does for releasing cross-platform binaries. That way I am able to quickly run borg on a variety of platform/cpus and deploy at scale. So is the problem just the number of platforms that borgmatic-binary has binary releases for? It looks like right now, there are only [releases for x86_64 Linux](https://github.com/cmarquardt/borgmatic-binary/releases). Sounds like you ideally would like FreeBSD releases as well. Are there other platforms or architectures? And are there other concerns with using binary releases from a third-party project? It appears that borgmatic-binary includes build [instructions](https://github.com/cmarquardt/borgmatic-binary#building-manually) for any platform supporting Borg even if there aren't pre-built binaries. So if you have your own CI, that may get your started building borgmatic for your target architectures/platforms. I'm asking about all this because I want to understand your platform requirements here. borgmatic's existing CI server is x86_64 Linux, so it would require some investment (both in time and hosting cost) to support other platforms like: FreeBSD, non-x86_64, etc.

Hi,

sorry for joining the discussion so late - and thanks to @witten for making me aware of it.

I started borgmatic-binary for the reasons @dverheslt mentions. Similar to the borg binaries, they are also built with PyInstaller. They thus should have the same advantages (and occasional issues).

Technically, I'm creating release files via Travis (GitHub's CI) whenever there's a new tag; that pushes a Linux release to GitHub. I usually do a manual build for OS-X as well and upload it, but I don't have other OSs like FreeBSD available. And as you've found out, there's no automatism in generating new releases based on borgmatic - sorry...

Building such an automatism isn't overly difficult. It would be even easier if @witten's CI would trigger them directly. I'm more than happy to contribute the scripts and PyInstaller configuration if Dan is willing to take them. That said - the binaries do require additional effort to maintain (and test) them. For example, I'm currently running into trouble when putting them on Linux machines with different Linux variants just because they have different versions of openSSL. I'm sure that can be solved by configuring PyInstaller correctly, but solving the problem generically may cost a lot of effort (I have little experience with that particular tool). Supporting more platforms, as Dan mentions, means even more work. I believe borg does it with virtual machines and vagrant, but that kind of infrastructure needs to be built and maintained in the first place.

In practice, I would agree with Dan that building the binaries on a development machine and then deploying them to the rest of your machines is by far the easiest solution.

Hi, sorry for joining the discussion so late - and thanks to @witten for making me aware of it. I started borgmatic-binary for the reasons @dverheslt mentions. Similar to the [borg binaries](https://borgbackup.readthedocs.io/en/stable/installation.html#pyinstaller-binary), they are also built with PyInstaller. They thus should have the same advantages (and occasional issues). Technically, I'm creating release files via Travis (GitHub's CI) whenever there's a new tag; that pushes a Linux release to GitHub. I usually do a manual build for OS-X as well and upload it, but I don't have other OSs like FreeBSD available. And as you've found out, there's no automatism in generating new releases based on borgmatic - sorry... Building such an automatism isn't overly difficult. It would be even easier if @witten's CI would trigger them directly. I'm more than happy to contribute the scripts and PyInstaller configuration if Dan is willing to take them. That said - the binaries do require additional effort to maintain (and test) them. For example, I'm currently running into trouble when putting them on Linux machines with different Linux variants just because they have different versions of openSSL. I'm sure that can be solved by configuring PyInstaller correctly, but solving the problem generically may cost a lot of effort (I have little experience with that particular tool). Supporting more platforms, as Dan mentions, means even more work. I believe `borg` does it with virtual machines and `vagrant`, but that kind of infrastructure needs to be built and maintained in the first place. In practice, I would agree with Dan that building the binaries on a development machine and then deploying them to the rest of your machines is by far the easiest solution.
Owner

FWIW, I'd be happy to look into triggering an external build (like borgmatic-binary) from borgmatic's CI. An alternative though might be for something external to poll for new borgmatic releases and use that information to trigger binary builds. Example:

$ curl --silent -X GET "https://projects.torsion.org/api/v1/repos/witten/borgmatic/releases" -H  "accept: application/json" | jq .[0]

{
  "id": 78,
  "tag_name": "1.3.4",
  "target_commitish": "",
  "name": "borgmatic 1.3.4",
  ...
}

But before jumping to solutions, it would be good to understand what problems (if any) we're solving for!

FWIW, I'd be happy to look into triggering an external build (like borgmatic-binary) from borgmatic's CI. An alternative though might be for something external to poll for new borgmatic releases and use that information to trigger binary builds. Example: ``` $ curl --silent -X GET "https://projects.torsion.org/api/v1/repos/witten/borgmatic/releases" -H "accept: application/json" | jq .[0] { "id": 78, "tag_name": "1.3.4", "target_commitish": "", "name": "borgmatic 1.3.4", ... } ``` But before jumping to solutions, it would be good to understand what problems (if any) we're solving for!
Owner

Closing due to inactivity, but please feel free to file a new ticket with specific additions! Or re-open this one with more info.

Closing due to inactivity, but please feel free to file a new ticket with specific additions! Or re-open this one with more info.

Just chiming in as a new borg and borgmatic user. Having pre-built binaries would be great, even though I currently only want to run borg in 64-bit Linux.

For now I'm using the asdf version manager to install Python 3 and then using pip to install borgmatic. This doesn't seem like a big deal on machines on which I'm already using asdf-managed Python(s), but does seem like somewhat of a barrier to entry on those machines where asdf/Python 3 isn't otherwise installed.

The release at borgmatic-binary is significantly out of date. I thought I'd try going through its build process but didn't manage to build even the out-of-date version let alone start looking into how one might use the latest version of borgmatic. (I suspect doing so would be beyond me anyway; I don't have much Python experience.) I also tried building without borgmatic-binary in a venv in Ubuntu's own Python 3 with the latest release from the borgmatic repository and pyinstaller borgmatic/commands/borgmatic.py, which did complete but gives a binary that says the colorama module is not found. So in short, a load of fail. :)

For now I hope to be able to put borgmatic on each server of mine via asdf and pip, but thought I'd leave my experience here for what it's worth.

Many thanks for borgmatic! It's obvious a lot of care and attention has been put into it and its documentation.

Just chiming in as a new borg and borgmatic user. Having pre-built binaries would be great, even though I currently only want to run borg in 64-bit Linux. For now I'm using the asdf version manager to install Python 3 and then using pip to install borgmatic. This doesn't seem like a big deal on machines on which I'm already using asdf-managed Python(s), but does seem like somewhat of a barrier to entry on those machines where asdf/Python 3 isn't otherwise installed. The release at borgmatic-binary is significantly out of date. I thought I'd try going through its build process but didn't manage to build even the out-of-date version let alone start looking into how one might use the latest version of borgmatic. (I suspect doing so would be beyond me anyway; I don't have much Python experience.) I also tried building without borgmatic-binary in a venv in Ubuntu's own Python 3 with the latest release from the borgmatic repository and `pyinstaller borgmatic/commands/borgmatic.py`, which did complete but gives a binary that says the colorama module is not found. So in short, a load of fail. :) For now I hope to be able to put borgmatic on each server of mine via asdf and pip, but thought I'd leave my experience here for what it's worth. Many thanks for borgmatic! It's obvious a lot of care and attention has been put into it and its documentation.
Owner

Thanks for posting your findings here. Since you mentioned you're using Ubuntu, I assume that you've looked at the Ubuntu borgmatic package and it doesn't meet your needs? Too far out of date?

Thanks for posting your findings here. Since you mentioned you're using Ubuntu, I assume that you've looked at the [Ubuntu borgmatic package](https://packages.ubuntu.com/eoan/borgmatic) and it doesn't meet your needs? Too far out of date?

Yes, I wanted to start off with latest stable, and use PostgreSQL dump hooks. :)

Yes, I wanted to start off with latest stable, and use PostgreSQL dump hooks. :)
Owner

Understood. I'll reopen this ticket in case someone would like to pick it up!

Understood. I'll reopen this ticket in case someone would like to pick it up!
witten reopened this issue 2020-01-14 17:37:48 +00:00

Created at Github Action to build the stuff:

https://github.com/psi-4ward/borgmatic-binary

Let's see if @cmarquardt adopts it.

Created at Github Action to build the stuff: https://github.com/psi-4ward/borgmatic-binary Let's see if @cmarquardt adopts it.

Our scenario is that we want to use the same borgmatic and borg backup version on all machines, but currently it is not an easy task (when pip is not an option).

Would be great if the above could be part of the official borgmatic community, not having to download binaries from a rather unknown 3rd party repo.

I appreciate the efforts done by @cmarquardt and @psi-4ward

Our scenario is that we want to use the same borgmatic and borg backup version on all machines, but currently it is not an easy task (when pip is not an option). Would be great if the above could be part of the official borgmatic community, not having to download binaries from a rather unknown 3rd party repo. I appreciate the efforts done by @cmarquardt and @psi-4ward
Owner

Some thoughts on this for anyone looking to pick up the ticket: A good place to start may be the (now outdated) borgmatic-binary, maybe even forking it outright. However, it would be nice to do #529 first so there's only a single borgmatic binary to wrap instead of four separate binaries!

Some thoughts on this for anyone looking to pick up the ticket: A good place to start may be the (now outdated) [borgmatic-binary](https://github.com/cmarquardt/borgmatic-binary), maybe even forking it outright. However, it would be nice to do #529 first so there's only a single borgmatic binary to wrap instead of four separate binaries!
Owner

#529 is done now, so now there's only a single borgmatic executable to wrap if you ignore the two compatibility executables (generate-borgmatic-config and validate-borgmatic-config).

#529 is done now, so now there's only a single borgmatic executable to wrap if you ignore the two compatibility executables (`generate-borgmatic-config` and `validate-borgmatic-config`).
witten added the
new feature area
label 2023-06-28 18:53:57 +00:00
Owner

Given that a solution for this hasn't materialized in the past few years (e.g. via a PR), I'm closing this for now. However I'd be happy to revisit and reopen if someone wants to work on this.

Given that a solution for this hasn't materialized in the past few years (e.g. via a PR), I'm closing this for now. However I'd be happy to revisit and reopen if someone wants to work on this.
Sign in to join this conversation.
No Milestone
No Assignees
7 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#151
No description provided.