Borg with-lock? How to use.? #79

New Issue

patricko · 2018-07-21T22:38:10Z

patricko commented

2018-07-21 22:38:10 +00:00

Hello, I use borgmatic because it simplifies stuff and allows me to discover config option easily

but how to activate borg with-lock to prevent errors?

Hello, I use borgmatic because it simplifies stuff and allows me to discover config option easily but how to activate borg with-lock to prevent errors?

witten added the

question / support

label 2018-07-22 17:29:06 +00:00

witten commented

2018-07-22 17:33:19 +00:00

You should still be able to run the Borg with-lock sub-command manually outside of borgmatic. Could you tell me a little bit more about your use case / how you're wanting to use with-lock with borgmatic? Thanks!

Also note that borgmatic has support for "hooks" that will execute your custom commands before or after a backup. However, right now they don't run within Borg's with-lock. Maybe that would be a good enhancement to make? Finding out about your use case will help me determine whether that'll actually solve your problem.

You should still be able to run the Borg `with-lock` sub-command manually outside of borgmatic. Could you tell me a little bit more about your use case / how you're wanting to use `with-lock` with borgmatic? Thanks! Also note that borgmatic has support for "hooks" that will execute your custom commands before or after a backup. However, right now they don't run within Borg's `with-lock`. Maybe that would be a good enhancement to make? Finding out about your use case will help me determine whether that'll actually solve your problem.

patricko commented

2018-07-23 00:08:39 +00:00

I have to make a tar archive of my borg repo and send it through ftp so that would come in handy to avoid corruption I guess?

witten commented

2018-07-23 04:56:58 +00:00

Got it. Is that the sort of thing you'd want to trigger after a borgmatic backup completes? For reference, here's the current documentation for borgmatic hooks:

# Shell commands or scripts to execute before and after a backup or if an error has occurred.
# IMPORTANT: All provided commands and scripts are executed with user permissions of borgmatic.
# Do not forget to set secure permissions on this file as well as on any script listed (chmod 0700) to
# prevent potential shell injection or privilege escalation.
hooks:
    # List of one or more shell commands or scripts to execute before creating a backup.
    before_backup:
        - echo "`date` - Starting a backup job."

    # List of one or more shell commands or scripts to execute after creating a backup.
    after_backup:
        - echo "`date` - Backup created."

    # List of one or more shell commands or scripts to execute in case an exception has occurred.
    on_error:
        - echo "`date` - Error while creating a backup."

So does after_backup sounds like it would work for your needs if the hook command uses borg with-lock?

Got it. Is that the sort of thing you'd want to trigger after a borgmatic backup completes? For reference, here's the current documentation for borgmatic hooks: ```shell # Shell commands or scripts to execute before and after a backup or if an error has occurred. # IMPORTANT: All provided commands and scripts are executed with user permissions of borgmatic. # Do not forget to set secure permissions on this file as well as on any script listed (chmod 0700) to # prevent potential shell injection or privilege escalation. hooks: # List of one or more shell commands or scripts to execute before creating a backup. before_backup: - echo "`date` - Starting a backup job." # List of one or more shell commands or scripts to execute after creating a backup. after_backup: - echo "`date` - Backup created." # List of one or more shell commands or scripts to execute in case an exception has occurred. on_error: - echo "`date` - Error while creating a backup." ``` So does `after_backup` sounds like it would work for your needs if the hook command uses `borg with-lock`?

patricko commented

2018-07-23 14:26:45 +00:00

Hey, sorry I'm a real beginner, I'm not even sure about the use of borg with-lock but Thomas Waldmann said to use it in my case to prevent corruption

Actually I made a script like this:

launch borgmatic
tar the repo
send through ftp

Do you suggest me to remove "launch borgmatic" from the script and add the script in the hooks part? and just run borgmatic regularly

but the with-lock part must be added into borgmatic's source right?

Hey, sorry I'm a real beginner, I'm not even sure about the use of borg with-lock but Thomas Waldmann said to use it in my case to prevent corruption Actually I made a script like this: ``` launch borgmatic tar the repo send through ftp ``` Do you suggest me to remove "launch borgmatic" from the script and add the script in the hooks part? and just run borgmatic regularly but the with-lock part must be added into borgmatic's source right?

witten commented

2018-07-24 03:22:13 +00:00

Okay, one last question (hopefully): Can you clarify what part you'd like to run within the protection of the with-lock? Is it actually making the backup and tarring the repo and sending it through FTP? Or is it just tarring the repo and sending it through FTP? Thanks.

Okay, one last question (hopefully): Can you clarify what part you'd like to run within the protection of the `with-lock`? Is it actually making the backup *and* tarring the repo *and* sending it through FTP? Or is it just tarring the repo and sending it through FTP? Thanks.

patricko commented

2018-07-24 14:23:10 +00:00

I really don't know, it's just for added security. That's what borg author told me to do.

I have nothing that conflicts with the repo, no operation is ran during the backup IN THEORY. At first sight I don't need it but why does it exist and why did the author advise to use it then?

I really don't know, it's just for added security. That's what borg author told me to do. I have nothing that conflicts with the repo, no operation is ran during the backup IN THEORY. At first sight I don't need it but why does it exist and why did the author advise to use it then?

witten commented

2018-07-25 01:47:25 +00:00

Not sure what the Borg author had in mind, but my guess is that the operation of tarring up the Borg repo could result in an inconsistent/broken tarball if another process happens to be writing to the Borg repository at the same time.

Given what you've said so far, and looking at the contents of your script, there shouldn't be another process writing to the Borg repository at the same time as the tar operation. But if you wanted to protect against that just to be safe, here's what I'd recommend. You can implement one of the two following options:

Change the tar command in your script to be wrapped with a borg with-lock invocation, referencing the desired repository. That should work with no changes to borgmatic. I'm saying that based on my understanding that, anytime borgmatic invokes Borg, Borg takes out its own repository lock.
Or, move both the tar invocation and the "send through ftp" invocation into borgmatic's hooks configuration as an after_backup hook. There, you'd still have to wrap the tar invocation with borg with-lock. Given that you'd be calling with-lock yourself, that wouldn't require any borgmatic changes either.

Which one you do depends on whether you want to keep your own script around to do the tarring and FTPing, or whether you'd prefer borgmatic to handle that for you from a common configuration file.

Let me know if that sounds like either of those will work for you.

Not sure what the Borg author had in mind, but my *guess* is that the operation of tarring up the Borg repo could result in an inconsistent/broken tarball if another process happens to be writing to the Borg repository at the same time. Given what you've said so far, and looking at the contents of your script, there *shouldn't* be another process writing to the Borg repository at the same time as the `tar` operation. But if you wanted to protect against that just to be safe, here's what I'd recommend. You can implement one of the two following options: 1. Change the `tar` command in your script to be wrapped with a `borg with-lock` invocation, referencing the desired repository. That should work with no changes to borgmatic. I'm saying that based on my understanding that, anytime borgmatic invokes Borg, Borg takes out its own repository lock. 2. Or, move both the `tar` invocation and the "send through ftp" invocation into borgmatic's hooks configuration as an `after_backup` hook. There, you'd still have to wrap the `tar` invocation with `borg with-lock`. Given that you'd be calling `with-lock` yourself, that wouldn't require any borgmatic changes either. Which one you do depends on whether you want to keep your own script around to do the tarring and FTPing, or whether you'd prefer borgmatic to handle that for you from a common configuration file. Let me know if that sounds like either of those will work for you.

patricko commented

2018-07-30 18:37:42 +00:00

Ok so, I just discovered how the command is used thanks to you... I think borgbackup docs are not so obvious lol

so I can use it like that:

borg with-lock repo ls -l repo

both options are the same, right? it's just that if I choose option 2 I need to remake my script

Ok so, I just discovered how the command is used thanks to you... I think borgbackup docs are not so obvious lol so I can use it like that: borg with-lock repo ls -l repo both options are the same, right? it's just that if I choose option 2 I need to remake my script

witten commented

2018-07-31 04:56:32 +00:00

Yup, basically the same thing in either case. Really a matter of preference.

patricko commented

2018-07-31 17:47:40 +00:00

ok then I understand better, borgmatic will automatically lock the repo as it's calling borg (and I suppose it's a default setting?)

If I lock the repo when doing tar, it will prevent me from doing anything to the repo manually while the script is running.

So I guess the solution is found and sorry, in the end it was not really related to borgmatic (well I learned about the "after_backup" hook though)

Thank you :P

ok then I understand better, borgmatic will automatically lock the repo as it's calling borg (and I suppose it's a default setting?) If I lock the repo when doing tar, it will prevent me from doing anything to the repo manually while the script is running. So I guess the solution is found and sorry, in the end it was not really related to borgmatic (well I learned about the "after_backup" hook though) Thank you :P

witten commented

2018-08-01 05:09:44 +00:00

That's all correct. No need to apologize.. Always happy to help!

witten closed this issue

2018-08-01 05:09:45 +00:00

Sign in to join this conversation.