Compiled Binary Size Questions #1314
Labels
No labels
blocked
breaking
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No milestone
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
borgmatic-collective/borgmatic#1314
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What I'm trying to do and why
Borgmatic now ships a precompiled binary (for x86-64).
It looks like the binary size is larger than it needs to be.
The release versions are: v2.1.4 is ~28M and v2.1.5 is ~22M.
My compiles seem to be in the ~14M range.
Not being a python expert, I have no idea why there's such a large size difference but I wanted to just raise a request to look into it (just to avoid including unwanted libraries as well).
Steps to reproduce
No response
Actual behavior
No response
Expected behavior
No response
Other notes / implementation ideas
No response
borgmatic version
No response
borgmatic installation method
No response
Borg version
No response
Python version
No response
Database version (if applicable)
No response
Operating system and version
Debian Trixie (13.5)
I think I found the culprits - likely to be libraries (and their own dependencies) that are required for e.g. for apprise integration and the like.
Closing this issue as it requires a more thorough investigation to confirm which libraries are missing resulting in library size changes.
Thanks for filing this and following up. So to get the smaller binary, were you building without Apprise? I had originally thought of producing multiple binaries (e.g. with and without Apprise), but that seemed like a bit much and I ended up just going with a single binary. But this might require rethinking because I'm about to add yet more optional dependencies...
I'll admit that was a surprise to me; I'm not sure why there was so much variation in release to release.
@witten wrote in #1314 (comment):
Yes. It seems like I'm building a minimal version without any of the extra bells and whistles.
@witten wrote in #1314 (comment):
It would be great if you could reconsider this and actually build multiple binaries. Consdering you have most of the setup already properly configured, it would take very little effort for you to ship a minimal (self sufficient) borgmatic binary as well as a more feature rich (or multiple) with everything compiled in.
As you test, could you please also check the sizes of the final binaries and share it here? If the sizes end up being significantly different (like >8-10MB variations I saw initially), it would definitely be worth it for users to pick and chose from the binaries based on their requirements.
Also, could you please consider different build options/scripts as part of the code base (with instructions) so that even if you don't ship them, there is a way for users to rebuild artifacts with their specific requirements compiled in?
I'm going to check out borgbackup as well to see if self-compiling it reduces the size significantly since I am using precompiled binaries for borg (and they have been shipping binaries for quite some time now).
Documenting the build process makes sense to me. Could you say a little more about your motivation with the streamlined binaries? Is it the disk space? Network bandwidth? Memory usage? The surface area from a security perspective? Something else? It's just that building multiple binaries for every release is not without costs, so I want to make sure I understand the benefits. Thank you!
The main reason is to have a minimal binary (esp. with no extraneous dependencies/features, avoiding all unwanted dependencies and libraries - definitely a key security consideration in many environments).
Space in absolute terms is not a critical factor (+10MB isn't earth shattering in this day and age) but it goes back to keeping things as simple and minimal as possible. I haven't evaluated the memory footprint as such, but I assume not using Apprise will not result in unnecessary memory bloat?
I totally understand your concerns on multiple-binary builds. At a minimum, please consider a barebones build that has nothing other than vanilla borgmatic and a full featured build that has everything. This should cover all use cases without a problem.
If the build scripts and process is made available, one could build whatever special build one wanted (including a bare-bones minimal binary).
Thanks for the explanation. And I get it.. The reason I made the Apprise feature optional to begin with is that it has some extra dependencies I just didn't want to pull in for everyone by default.
I wouldn't imagine the memory consumption difference would be that big, given that Linux mmap()s in sections of the binary from disk on demand.
@quietquark wrote in #1314 (comment):
I can look into it. It probably should involve moving the build process into CI. (Right now it's just run as part of a manual release script.)
Yup, that should be made available regardless!
Well, I produced a minimal binary as part of the standard, manually invoked release process. It omits Apprise and shaves off a few MB. It'll be available for the next release. The process is still not documented (because that probably means moving it out of the release script), so I'm leaving this ticket open.
Released in 2.1.6!