Relative repository path support #143

Closed
opened 2019-02-24 05:55:36 +00:00 by gemarcano · 13 comments

I'm not able to assign a label, although this is more of a question (and perhaps a suggestion?).

Some background: I dual-boot Linux and Windows, and depending on what I'm doing that week (gaming vs. programming for fun), I'll spend more time on one OS over the other. I keep all of my data (Documents, Music, Projects, etc.) on a shared NTFS disk that both OSs can read. I installed borg on my Cygwin installation on Windows, and I've verified that borgmatic is working well when I'm backing up data that isn't shared between the two systems.

The part I'm a bit stuck on is that I would like to have both Linux and Windows borgmatic installations/configurations be able to back up the shared data onto the same repository/archive. Thing is, on Linux this is in /mnt/Data/ or something like that, and on Windows, because of cygwin, it's under /cygdrive/h/. In other words, the absolute paths differ.

I know Borg allows using relative paths from the current working directory, so I tried to see if I could do something similar by leveraging the before_backup hook to cd to the root of the shared disk, but it didn't work. After looking how it is that borgmatic handles hooks, it makes sense, since from what I could tell it calls each hook in a subprocess, so my cd does nothing to affect the working directory of borgmatic.

So, the question I really care about is: is there a way to synchronize 2 different absolute paths that have identical data into the same repository? I think the easiest way would be to somehow run borgmatic using relative paths for the repository path. Any thoughts or alternatives? I might just hack something on my local copy of borgmatic, probably add a new configuration option or something to set the working directory of borgmatic, since I think that would require the least amount of work.

I'm not able to assign a label, although this is more of a question (and perhaps a suggestion?). Some background: I dual-boot Linux and Windows, and depending on what I'm doing that week (gaming vs. programming for fun), I'll spend more time on one OS over the other. I keep all of my data (Documents, Music, Projects, etc.) on a shared NTFS disk that both OSs can read. I installed borg on my Cygwin installation on Windows, and I've verified that borgmatic is working well when I'm backing up data that isn't shared between the two systems. The part I'm a bit stuck on is that I would like to have both Linux and Windows borgmatic installations/configurations be able to back up the shared data onto the same repository/archive. Thing is, on Linux this is in `/mnt/Data/` or something like that, and on Windows, because of cygwin, it's under `/cygdrive/h/`. In other words, the absolute paths differ. I know Borg allows using relative paths from the current working directory, so I tried to see if I could do something similar by leveraging the `before_backup` hook to `cd` to the root of the shared disk, but it didn't work. After looking how it is that borgmatic handles hooks, it makes sense, since from what I could tell it calls each hook in a subprocess, so my `cd` does nothing to affect the working directory of borgmatic. So, the question I really care about is: is there a way to synchronize 2 different absolute paths that have identical data into the same repository? I think the easiest way would be to somehow run borgmatic using relative paths for the repository path. Any thoughts or alternatives? I might just hack something on my local copy of borgmatic, probably add a new configuration option or something to set the working directory of borgmatic, since I think that would require the least amount of work.
witten added the
question / support
label 2019-02-24 06:21:47 +00:00
Author

Alright, after reading a bit more about how everything is implemented, and how Python handles subprocess current working directory, I ran a test. If I run borgmatic from the root of my shared drive, the current working directory is passed to all subprocesses, and the relative path in the configuration file (location.repositories) works as expected. I suppose this is a workaround, and I might be able to get the effect I want using cron by chaining a cd and the borgmatic invocation. It would still be nice to be able to specify the current working directory (unless there's a better way to do what I tried to describe earlier).

Alright, after reading a bit more about how everything is implemented, and how Python handles subprocess current working directory, I ran a test. If I run borgmatic from the root of my shared drive, the current working directory is passed to all subprocesses, and the relative path in the configuration file (location.repositories) works as expected. I suppose this is a workaround, and I might be able to get the effect I want using cron by chaining a cd and the borgmatic invocation. It would still be nice to be able to specify the current working directory (unless there's a better way to do what I tried to describe earlier).
Owner

Interesting situation! I haven't come across a set up like this before, so I don't know of any direct solutions. However, I was actually in the process of writing up the exact same suggestion you propose in your second comment when you posted it: Try to change the working directory right before you invoke borgmatic.

Please give that a shot, and let me know how it works for you in your dual-boot setup. However, regardless of whether that works, what would be a more ideal solution for you? An option in borgmatic's configuration file to set the current working directory?

Interesting situation! I haven't come across a set up like this before, so I don't know of any direct solutions. However, I was actually in the process of writing up the exact same suggestion you propose in your second comment when you posted it: Try to change the working directory right before you invoke borgmatic. Please give that a shot, and let me know how it works for you in your dual-boot setup. However, regardless of whether that works, what would be a more ideal solution for you? An option in borgmatic's configuration file to set the current working directory?
Author

I was able to hack in the ability to change the current working directory per configuration file, by adding a new field to location, and in run_configuration add a check that if that field is specified, just set the application's working directory to that value.

I would submit a PR, but I need to get approval from work first (asinine contract IP assignment requirements for work). Honestly, if this is something you wanted to add, you might be able to get it added long before I received approval from work.

I was able to hack in the ability to change the current working directory per configuration file, by adding a new field to location, and in `run_configuration` add a check that if that field is specified, just set the application's working directory to that value. I would submit a PR, but I need to get approval from work first (asinine contract IP assignment requirements for work). Honestly, if this is something you wanted to add, you might be able to get it added long before I received approval from work.
Owner

Glad to hear that at least the hack worked. I know how that is with IP requirements! Hopefully someone will be able to pick this up (or you'll get approval for a PR).

Glad to hear that at least the hack worked. I know how that is with IP requirements! Hopefully someone will be able to pick this up (or you'll get approval for a PR).
witten added
design finalized
and removed
question / support
labels 2019-02-24 06:59:51 +00:00
Author

Somehow I missed your comment (my browser probably updated the page once I commented).

I'm still not 100% convinced that adding a new configuration field is the right approach, but my hack appears to be working as intended anyway. The root of the problem is that borg (I think, right?) uses actual paths to organize the data being backed up. Since cygwin and Linux have different layouts, this (in theory) confuses borg (although the deduplication would limit the damage) in the extreme case my use case presents. If I were using borg directly, I'd have to do something with cd and relative paths anyway.

Maybe providing a new optional field in the configuration file (per location?) is the solution that borgmatic can take (looks like someone has to cd or chdir at some point, due to borg). I don't know if my use case is common enough to justify adding a config field, though. I realize I have a really strange setup, few people I know even know about Windows NTFS junctions, I just wanted to make sure Documents, Music, etc. were available to both systems, and with symlinks and junctions I was able to move them from my main OS SSD to a much larger conventional drive (in Linux I could just re-bind all of those folders too, but symlinks work just fine).

Another work-around I just thought of for me would be to create a symlink mimicking the /mnt/Media path found in Linux in Cygwin, that way paths would look the same on both systems.

EDIT: All of that is a long-winded way of saying I should probably sleep on it to see if I can think of a better approach.

Somehow I missed your comment (my browser probably updated the page once I commented). I'm still not 100% convinced that adding a new configuration field is the right approach, but my hack appears to be working as intended anyway. The root of the problem is that borg (I think, right?) uses actual paths to organize the data being backed up. Since cygwin and Linux have different layouts, this (in theory) confuses borg (although the deduplication would limit the damage) in the extreme case my use case presents. If I were using borg directly, I'd have to do something with cd and relative paths anyway. Maybe providing a new optional field in the configuration file (per location?) is the solution that borgmatic can take (looks like someone has to `cd` or `chdir` at some point, due to borg). I don't know if my use case is common enough to justify adding a config field, though. I realize I have a really strange setup, few people I know even know about Windows NTFS junctions, I just wanted to make sure Documents, Music, etc. were available to both systems, and with symlinks and junctions I was able to move them from my main OS SSD to a much larger conventional drive (in Linux I could just re-bind all of those folders too, but symlinks work just fine). Another work-around I just thought of for me would be to create a symlink mimicking the `/mnt/Media` path found in Linux in Cygwin, that way paths would look the same on both systems. EDIT: All of that is a long-winded way of saying I should probably sleep on it to see if I can think of a better approach.
Owner

Okay, let me know what you come up with! I'm open to ideas.. This is certainly a new use case to me.

Okay, let me know what you come up with! I'm open to ideas.. This is certainly a new use case to me.
witten removed the
design finalized
label 2019-02-24 22:16:17 +00:00
Owner

Any more thoughts on this one? Is changing the working directory still working out for you? Thanks.

Any more thoughts on this one? Is changing the working directory still working out for you? Thanks.
Author

Sorry for not reporting back. Yeah, the patch I developed is working properly, and it ended up being a one or two liner (plus adding an entry to the configuration), I forget exactly right now. Effectively, I just change the directory to the one specified in the configuration file if that configuration is present, before beginning to execute borg commands.

Now that it has been a while since I did it I need to sit down and take a look again at my use case to see if I can do without the patch, since it's such a weird use case. Again, due to my IP waiver with my employer I can't submit the patch, at least not yet.

Sorry for not reporting back. Yeah, the patch I developed is working properly, and it ended up being a one or two liner (plus adding an entry to the configuration), I forget exactly right now. Effectively, I just change the directory to the one specified in the configuration file if that configuration is present, before beginning to execute borg commands. Now that it has been a while since I did it I need to sit down and take a look again at my use case to see if I can do without the patch, since it's such a weird use case. Again, due to my IP waiver with my employer I can't submit the patch, at least not yet.
Owner

Got it. In that case, I'll close this for now. But please feel free to re-open at any point if maintaining the patch gets to be a huge pain and/or you get the opportunity to submit the patch here. Thanks!

Got it. In that case, I'll close this for now. But please feel free to re-open at any point if maintaining the patch gets to be a huge pain and/or you get the opportunity to submit the patch here. Thanks!

Looking for the same support now. Specifying e.g.

  source_directories:
    - ~/Videos

will result in absolute paths, which may or may not be desired. I think according to the borg documentation, changing to the desired directory and then specifying . as a directory to archive should produce relative paths in resulting archive. Meaning this is surely something for borgmatic to implement as per configuration, where it would change to each directory listed under source_directories and create archive for the current directory (.).

Looking for the same support now. Specifying e.g. ```yaml source_directories: - ~/Videos ``` will result in absolute paths, which may or may not be desired. I think according to the `borg` documentation, changing to the desired directory and then specifying `.` as a directory to archive should produce relative paths in resulting archive. Meaning this is surely something for `borgmatic` to implement as per configuration, where it would change to each directory listed under `source_directories` and `create` archive for the current directory (`.`).
Owner

where it would change to each directory listed under source_directories and create archive for the current directory (.).

borgmatic currently creates a single archive containing all configured source directories. So are you proposing for borgmatic to instead (optionally) create a separate archive per source directory? Can you talk a little more about your use case for that?

> where it would change to each directory listed under source_directories and create archive for the current directory (.). borgmatic currently creates a single archive containing all configured source directories. So are you proposing for borgmatic to instead (optionally) create a separate archive per source directory? Can you talk a little more about your use case for that?

If I'm getting it right this is the same requirement as #431.

If I'm getting it right this is the same requirement as #431.
Owner

Yes, and #431 is now implemented via a new working_directory option.

Yes, and #431 is now implemented via a new `working_directory` option.
Sign in to join this conversation.
No Milestone
No Assignees
4 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#143
No description provided.