#143 Relative repository path support

Closed
opened 2 months ago by gemarcano · 9 comments

I’m not able to assign a label, although this is more of a question (and perhaps a suggestion?).

Some background: I dual-boot Linux and Windows, and depending on what I’m doing that week (gaming vs. programming for fun), I’ll spend more time on one OS over the other. I keep all of my data (Documents, Music, Projects, etc.) on a shared NTFS disk that both OSs can read. I installed borg on my Cygwin installation on Windows, and I’ve verified that borgmatic is working well when I’m backing up data that isn’t shared between the two systems.

The part I’m a bit stuck on is that I would like to have both Linux and Windows borgmatic installations/configurations be able to back up the shared data onto the same repository/archive. Thing is, on Linux this is in /mnt/Data/ or something like that, and on Windows, because of cygwin, it’s under /cygdrive/h/. In other words, the absolute paths differ.

I know Borg allows using relative paths from the current working directory, so I tried to see if I could do something similar by leveraging the before_backup hook to cd to the root of the shared disk, but it didn’t work. After looking how it is that borgmatic handles hooks, it makes sense, since from what I could tell it calls each hook in a subprocess, so my cd does nothing to affect the working directory of borgmatic.

So, the question I really care about is: is there a way to synchronize 2 different absolute paths that have identical data into the same repository? I think the easiest way would be to somehow run borgmatic using relative paths for the repository path. Any thoughts or alternatives? I might just hack something on my local copy of borgmatic, probably add a new configuration option or something to set the working directory of borgmatic, since I think that would require the least amount of work.

I'm not able to assign a label, although this is more of a question (and perhaps a suggestion?). Some background: I dual-boot Linux and Windows, and depending on what I'm doing that week (gaming vs. programming for fun), I'll spend more time on one OS over the other. I keep all of my data (Documents, Music, Projects, etc.) on a shared NTFS disk that both OSs can read. I installed borg on my Cygwin installation on Windows, and I've verified that borgmatic is working well when I'm backing up data that isn't shared between the two systems. The part I'm a bit stuck on is that I would like to have both Linux and Windows borgmatic installations/configurations be able to back up the shared data onto the same repository/archive. Thing is, on Linux this is in `/mnt/Data/` or something like that, and on Windows, because of cygwin, it's under `/cygdrive/h/`. In other words, the absolute paths differ. I know Borg allows using relative paths from the current working directory, so I tried to see if I could do something similar by leveraging the `before_backup` hook to `cd` to the root of the shared disk, but it didn't work. After looking how it is that borgmatic handles hooks, it makes sense, since from what I could tell it calls each hook in a subprocess, so my `cd` does nothing to affect the working directory of borgmatic. So, the question I really care about is: is there a way to synchronize 2 different absolute paths that have identical data into the same repository? I think the easiest way would be to somehow run borgmatic using relative paths for the repository path. Any thoughts or alternatives? I might just hack something on my local copy of borgmatic, probably add a new configuration option or something to set the working directory of borgmatic, since I think that would require the least amount of work.
witten added the
question / support
label 2 months ago
gemarcano commented 2 months ago
Poster

Alright, after reading a bit more about how everything is implemented, and how Python handles subprocess current working directory, I ran a test. If I run borgmatic from the root of my shared drive, the current working directory is passed to all subprocesses, and the relative path in the configuration file (location.repositories) works as expected. I suppose this is a workaround, and I might be able to get the effect I want using cron by chaining a cd and the borgmatic invocation. It would still be nice to be able to specify the current working directory (unless there’s a better way to do what I tried to describe earlier).

Alright, after reading a bit more about how everything is implemented, and how Python handles subprocess current working directory, I ran a test. If I run borgmatic from the root of my shared drive, the current working directory is passed to all subprocesses, and the relative path in the configuration file (location.repositories) works as expected. I suppose this is a workaround, and I might be able to get the effect I want using cron by chaining a cd and the borgmatic invocation. It would still be nice to be able to specify the current working directory (unless there's a better way to do what I tried to describe earlier).
witten commented 2 months ago
Owner

Interesting situation! I haven’t come across a set up like this before, so I don’t know of any direct solutions. However, I was actually in the process of writing up the exact same suggestion you propose in your second comment when you posted it: Try to change the working directory right before you invoke borgmatic.

Please give that a shot, and let me know how it works for you in your dual-boot setup. However, regardless of whether that works, what would be a more ideal solution for you? An option in borgmatic’s configuration file to set the current working directory?

Interesting situation! I haven't come across a set up like this before, so I don't know of any direct solutions. However, I was actually in the process of writing up the exact same suggestion you propose in your second comment when you posted it: Try to change the working directory right before you invoke borgmatic. Please give that a shot, and let me know how it works for you in your dual-boot setup. However, regardless of whether that works, what would be a more ideal solution for you? An option in borgmatic's configuration file to set the current working directory?
gemarcano commented 2 months ago
Poster

I was able to hack in the ability to change the current working directory per configuration file, by adding a new field to location, and in run_configuration add a check that if that field is specified, just set the application’s working directory to that value.

I would submit a PR, but I need to get approval from work first (asinine contract IP assignment requirements for work). Honestly, if this is something you wanted to add, you might be able to get it added long before I received approval from work.

I was able to hack in the ability to change the current working directory per configuration file, by adding a new field to location, and in `run_configuration` add a check that if that field is specified, just set the application's working directory to that value. I would submit a PR, but I need to get approval from work first (asinine contract IP assignment requirements for work). Honestly, if this is something you wanted to add, you might be able to get it added long before I received approval from work.
witten commented 2 months ago
Owner

Glad to hear that at least the hack worked. I know how that is with IP requirements! Hopefully someone will be able to pick this up (or you’ll get approval for a PR).

Glad to hear that at least the hack worked. I know how that is with IP requirements! Hopefully someone will be able to pick this up (or you'll get approval for a PR).
witten removed the
question / support
label 2 months ago
witten added the
design finalized
label 2 months ago
gemarcano commented 2 months ago
Poster

Somehow I missed your comment (my browser probably updated the page once I commented).

I’m still not 100% convinced that adding a new configuration field is the right approach, but my hack appears to be working as intended anyway. The root of the problem is that borg (I think, right?) uses actual paths to organize the data being backed up. Since cygwin and Linux have different layouts, this (in theory) confuses borg (although the deduplication would limit the damage) in the extreme case my use case presents. If I were using borg directly, I’d have to do something with cd and relative paths anyway.

Maybe providing a new optional field in the configuration file (per location?) is the solution that borgmatic can take (looks like someone has to cd or chdir at some point, due to borg). I don’t know if my use case is common enough to justify adding a config field, though. I realize I have a really strange setup, few people I know even know about Windows NTFS junctions, I just wanted to make sure Documents, Music, etc. were available to both systems, and with symlinks and junctions I was able to move them from my main OS SSD to a much larger conventional drive (in Linux I could just re-bind all of those folders too, but symlinks work just fine).

Another work-around I just thought of for me would be to create a symlink mimicking the /mnt/Media path found in Linux in Cygwin, that way paths would look the same on both systems.

EDIT: All of that is a long-winded way of saying I should probably sleep on it to see if I can think of a better approach.

Somehow I missed your comment (my browser probably updated the page once I commented). I'm still not 100% convinced that adding a new configuration field is the right approach, but my hack appears to be working as intended anyway. The root of the problem is that borg (I think, right?) uses actual paths to organize the data being backed up. Since cygwin and Linux have different layouts, this (in theory) confuses borg (although the deduplication would limit the damage) in the extreme case my use case presents. If I were using borg directly, I'd have to do something with cd and relative paths anyway. Maybe providing a new optional field in the configuration file (per location?) is the solution that borgmatic can take (looks like someone has to `cd` or `chdir` at some point, due to borg). I don't know if my use case is common enough to justify adding a config field, though. I realize I have a really strange setup, few people I know even know about Windows NTFS junctions, I just wanted to make sure Documents, Music, etc. were available to both systems, and with symlinks and junctions I was able to move them from my main OS SSD to a much larger conventional drive (in Linux I could just re-bind all of those folders too, but symlinks work just fine). Another work-around I just thought of for me would be to create a symlink mimicking the `/mnt/Media` path found in Linux in Cygwin, that way paths would look the same on both systems. EDIT: All of that is a long-winded way of saying I should probably sleep on it to see if I can think of a better approach.
witten commented 2 months ago
Owner

Okay, let me know what you come up with! I’m open to ideas.. This is certainly a new use case to me.

Okay, let me know what you come up with! I'm open to ideas.. This is certainly a new use case to me.
witten removed the
design finalized
label 2 months ago
witten commented 5 days ago
Owner

Any more thoughts on this one? Is changing the working directory still working out for you? Thanks.

Any more thoughts on this one? Is changing the working directory still working out for you? Thanks.
gemarcano commented 4 days ago
Poster

Sorry for not reporting back. Yeah, the patch I developed is working properly, and it ended up being a one or two liner (plus adding an entry to the configuration), I forget exactly right now. Effectively, I just change the directory to the one specified in the configuration file if that configuration is present, before beginning to execute borg commands.

Now that it has been a while since I did it I need to sit down and take a look again at my use case to see if I can do without the patch, since it’s such a weird use case. Again, due to my IP waiver with my employer I can’t submit the patch, at least not yet.

Sorry for not reporting back. Yeah, the patch I developed is working properly, and it ended up being a one or two liner (plus adding an entry to the configuration), I forget exactly right now. Effectively, I just change the directory to the one specified in the configuration file if that configuration is present, before beginning to execute borg commands. Now that it has been a while since I did it I need to sit down and take a look again at my use case to see if I can do without the patch, since it's such a weird use case. Again, due to my IP waiver with my employer I can't submit the patch, at least not yet.
witten commented 3 days ago
Owner

Got it. In that case, I’ll close this for now. But please feel free to re-open at any point if maintaining the patch gets to be a huge pain and/or you get the opportunity to submit the patch here. Thanks!

Got it. In that case, I'll close this for now. But please feel free to re-open at any point if maintaining the patch gets to be a huge pain and/or you get the opportunity to submit the patch here. Thanks!
Sign in to join this conversation.
No Milestone
No Assignees
2 Participants
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Loading…
Cancel
Save
There is no content yet.