borgmatic-collective/borgmatic
borgmatic-collective
/
borgmatic
15
57
Fork 41
Code Issues 55 Pull Requests 1 Actions Packages Releases 121 Activity

[Errno 17] File exists while running borgmatic check #711

New Issue
Closed
opened 2023-06-11 17:05:19 +00:00 by flyinggreenfrog · 13 comments
flyinggreenfrog commented 2023-06-11 17:05:19 +00:00
Copy Link

What I'm trying to do and why

After upgrading from borgmatic 1.7.10 to 1.7.13 borgmatic check fails with following error:

ssh://XXXX@XXXX/./YYYY: Error running actions for repository
[Errno 17] File exists: '/root/.borgmatic/checks/971dd166ae0ba8f621740de76f43e0984a0ee1cdbf378f7776adec3c7d41d132/data'

Steps to reproduce (if a bug)

Having run borgmatic check in version 1.7.10 for repository created the file /root/.borgmatic/checks/.../data.

With newer version 1.7.13 borgmatic check wants to create a file /root/.borgmatic/checks/.../data/all instead, but can't handle the situation, when data as file already exists, but needs to be now a directory.

Actual behavior (if a bug)

borgmatic check fails with

ssh://XXXX@XXXX/./YYYY: Error running actions for repository
[Errno 17] File exists: '/root/.borgmatic/checks/971dd166ae0ba8f621740de76f43e0984a0ee1cdbf378f7776adec3c7d41d132/data'

Expected behavior (if a bug)

borgmatic check should run successfully

Other notes / implementation ideas

Workaround/Solution: after upgrading to borgmatic 1.7.13 manually delete the /root/.borgmatic/checks/.../data file, and a new run will create subdirectory data with now used file data/all.

Alternatively create empty file data/all before a new run.

Environment

borgmatic version: 1.7.13

borgmatic installation method: openSUSE package

Borg version: borg 1.2.4

Python version: 3.10.11

Database version (if applicable): -

operating system and version: openSUSE Tumbleweed

#### What I'm trying to do and why After upgrading from borgmatic 1.7.10 to 1.7.13 `borgmatic check` fails with following error: ``` ssh://XXXX@XXXX/./YYYY: Error running actions for repository [Errno 17] File exists: '/root/.borgmatic/checks/971dd166ae0ba8f621740de76f43e0984a0ee1cdbf378f7776adec3c7d41d132/data' ``` #### Steps to reproduce (if a bug) Having run `borgmatic check` in version 1.7.10 for repository created the file `/root/.borgmatic/checks/.../data`. With newer version 1.7.13 `borgmatic check` wants to create a file `/root/.borgmatic/checks/.../data/all` instead, but can't handle the situation, when `data` as file already exists, but needs to be now a directory. #### Actual behavior (if a bug) `borgmatic check` fails with ``` ssh://XXXX@XXXX/./YYYY: Error running actions for repository [Errno 17] File exists: '/root/.borgmatic/checks/971dd166ae0ba8f621740de76f43e0984a0ee1cdbf378f7776adec3c7d41d132/data' ``` #### Expected behavior (if a bug) `borgmatic check` should run successfully #### Other notes / implementation ideas **Workaround/Solution:** after upgrading to borgmatic 1.7.13 manually delete the `/root/.borgmatic/checks/.../data` file, and a new run will create subdirectory `data` with now used file `data/all`. Alternatively create empty file `data/all` before a new run. #### Environment **borgmatic version:** 1.7.13 **borgmatic installation method:** openSUSE package **Borg version:** borg 1.2.4 **Python version:** 3.10.11 **Database version (if applicable):** - **operating system and version:** openSUSE Tumbleweed
witten commented 2023-06-11 17:10:50 +00:00
Owner
Copy Link

Ugh, thanks for filing this. The new check code is supposed to gracefully upgrade the old check file layout in place, but it sounds like that's not working in your case. I'll see if I can repro this locally.

Ugh, thanks for filing this. The new check code is supposed to gracefully upgrade the old check file layout in place, but it sounds like that's not working in your case. I'll see if I can repro this locally.
witten added the
bug
 label 2023-06-11 17:10:57 +00:00
witten commented 2023-06-11 17:49:56 +00:00
Owner
Copy Link

Can I get a look at your borgmatic --verbosity 2 logs when this error occurs? In particular, I'm looking for a log like:

Upgrading archives check time from /root/.borgmatic/checks/971dd166ae0ba8f621740de76f43e0984a0ee1cdbf378f7776adec3c7d41d132/data to /root/.borgmatic/checks/971dd166ae0ba8f621740de76f43e0984a0ee1cdbf378f7776adec3c7d41d132/data/all

Thanks!

(FWIW, I've not been able to reproduce the error locally.)

Can I get a look at your `borgmatic --verbosity 2` logs when this error occurs? In particular, I'm looking for a log like: ``` Upgrading archives check time from /root/.borgmatic/checks/971dd166ae0ba8f621740de76f43e0984a0ee1cdbf378f7776adec3c7d41d132/data to /root/.borgmatic/checks/971dd166ae0ba8f621740de76f43e0984a0ee1cdbf378f7776adec3c7d41d132/data/all ``` Thanks! (FWIW, I've not been able to reproduce the error locally.)
flyinggreenfrog commented 2023-06-11 20:16:00 +00:00
Author
Copy Link

Thanks for your reply.

I will try to provide logs, but in the error logs (borgmatic check -v 2) from my attempts earlier no such line Upgrading archives check time ... appeared.

At the moment it works again for me, as I was manually deleting that data file.

Hopefully on next weekend I would try to reproduce it again with old version, before upgrading again.

Thanks for your reply. I will try to provide logs, but in the error logs (`borgmatic check -v 2`) from my attempts earlier no such line `Upgrading archives check time ...` appeared. At the moment it works again for me, as I was manually deleting that `data` file. Hopefully on next weekend I would try to reproduce it again with old version, before upgrading again.
witten commented 2023-06-11 23:23:59 +00:00
Owner
Copy Link

Okay. If it comes to it, I can give you a modified source file instrumented with even more debug logging to determine what might be going on with these check upgrades on your system. Keep me updated!

Okay. If it comes to it, I can give you a modified source file instrumented with even _more_ debug logging to determine what might be going on with these check upgrades on your system. Keep me updated!
witten added the
waiting for response
 label 2023-06-11 23:24:06 +00:00
witten commented 2023-06-18 05:51:14 +00:00
Owner
Copy Link

Possibly related: #713.

Possibly related: #713.
ahjohannessen commented 2023-06-22 15:04:14 +00:00
Copy Link

@witten I am getting something similar in my borg extract tests from rsync.net:

...
archives
root/.borgmatic/checks/45c964b6df60b853411b175cff99e289f90957e1486142ca4e3f62678bbf1de8/archives: open: [Errno 21] Is a directory: '/alloc/data/archives'
repository
archives
root/.borgmatic/checks/5104269ce93ba0783c8b115fd072b71f947e1e6b9970d650b1a375eb3d018cee/archives: open: [Errno 21] Is a directory: '/alloc/data/archives'
repository
archives
root/.borgmatic/checks/f2e38ccc990b318897dc030d786be9b4e97818f87ba2abcece7ea4a3178bd4fd/archives: open: [Errno 21] Is a directory: '/alloc/data/archives'
repository
archives
root/.borgmatic/checks/404985c24bc47e7856b17ea72cf6a9f9aa85dc9e96c3a02f14d95d153d5ff9c3/archives: open: [Errno 21] Is a directory: '/alloc/data/archives'
archives
archives/all
repository
archives
archives/all
repository
RemoteRepository: 149.98 kB bytes sent, 2.74 GB bytes received, 2837 messages sent

I am running something like this on in a Docker container on Nomad (Hashicorp):

cd $NOMAD_ALLOC_DIR/data && borg --bypass-lock list --last 1 --short $BORG_REPO_TARGET | head -n1 | xargs -n1 -I% borg --bypass-lock extract --strip-components 4 $BORG_REPO_TARGET::%

This was working fine until I moved from borgmatic 1.7.12 to 1.7.14.

@witten I am getting something similar in my borg extract tests from rsync.net: ``` ... archives root/.borgmatic/checks/45c964b6df60b853411b175cff99e289f90957e1486142ca4e3f62678bbf1de8/archives: open: [Errno 21] Is a directory: '/alloc/data/archives' repository archives root/.borgmatic/checks/5104269ce93ba0783c8b115fd072b71f947e1e6b9970d650b1a375eb3d018cee/archives: open: [Errno 21] Is a directory: '/alloc/data/archives' repository archives root/.borgmatic/checks/f2e38ccc990b318897dc030d786be9b4e97818f87ba2abcece7ea4a3178bd4fd/archives: open: [Errno 21] Is a directory: '/alloc/data/archives' repository archives root/.borgmatic/checks/404985c24bc47e7856b17ea72cf6a9f9aa85dc9e96c3a02f14d95d153d5ff9c3/archives: open: [Errno 21] Is a directory: '/alloc/data/archives' archives archives/all repository archives archives/all repository RemoteRepository: 149.98 kB bytes sent, 2.74 GB bytes received, 2837 messages sent ``` I am running something like this on in a Docker container on Nomad (Hashicorp): ``` cd $NOMAD_ALLOC_DIR/data && borg --bypass-lock list --last 1 --short $BORG_REPO_TARGET | head -n1 | xargs -n1 -I% borg --bypass-lock extract --strip-components 4 $BORG_REPO_TARGET::% ``` This was working fine until I moved from borgmatic 1.7.12 to 1.7.14.
witten commented 2023-06-22 16:27:59 +00:00
Owner
Copy Link

Thanks for chiming in! If you have the ability to install a development version of borgmatic, I have a potential fix you can try. You would upgrade borgmatic with the version in that branch. Or, if it's easier, I can give you a single file to override one of your local borgmatic source files with.

But if you're not comfortable with that for whatever reason, you can just wait for the next release and this should hopefully be fixed.

Thanks for chiming in! If you have the ability to install a development version of borgmatic, I have [a potential fix you can try](https://projects.torsion.org/borgmatic-collective/borgmatic/src/branch/713-check-time-upgrade-debugging). You would upgrade borgmatic with the version in that branch. Or, if it's easier, I can give you a single file to override one of your local borgmatic source files with. But if you're not comfortable with that for whatever reason, you can just wait for the next release and this should hopefully be fixed.
ahjohannessen commented 2023-06-22 16:47:03 +00:00
Copy Link

@witten I figured out why root/.borgmatic/checks/404985c24bc47e7856b17ea72cf6a9f9aa85dc9e96c3a02f14d95d153d5ff9c3/archives: open: [Errno 21] Is a directory: '/alloc/data/archives' happened.

I take borg backups of zfs snapshots that are mounted to something like /mnt/borg/r02/data/

Then the database folder is inside that.

However, when stripping 4 components root/.borgmatic/checks/404985c24bc47e7856b17ea72cf6a9f9aa85dc9e96c3a02f14d95d153d5ff9c3/archives

ended up in same place as borg's archives folder.

@witten I figured out why `root/.borgmatic/checks/404985c24bc47e7856b17ea72cf6a9f9aa85dc9e96c3a02f14d95d153d5ff9c3/archives: open: [Errno 21] Is a directory: '/alloc/data/archives'` happened. I take borg backups of zfs snapshots that are mounted to something like `/mnt/borg/r02/data/` Then the database folder is inside that. However, when stripping 4 components `root/.borgmatic/checks/404985c24bc47e7856b17ea72cf6a9f9aa85dc9e96c3a02f14d95d153d5ff9c3/archives` ended up in same place as borg's archives folder.
witten commented 2023-06-22 17:48:54 +00:00
Owner
Copy Link

Interesting. I assume those ZFS snapshots read-only? That might explain why borgmatic couldn't upgrade their file layout (which changed from 1.7.12. to 1.7.14). The ~/.borgmatic/checks/ directory needs to be read-write.

Interesting. I assume those ZFS snapshots read-only? That might explain why borgmatic couldn't upgrade their file layout (which changed from 1.7.12. to 1.7.14). The `~/.borgmatic/checks/` directory needs to be read-write.
ahjohannessen commented 2023-06-22 21:04:46 +00:00
Copy Link

@witten Yep, snapshots mounted as read-only locally, then shipped to rsync.net with borgmatic.
Verification is done in a docker/nomad combo using borgmatic docker container. I got it running again without that error after removal of --strip-components

@witten Yep, snapshots mounted as read-only locally, then shipped to rsync.net with borgmatic. Verification is done in a docker/nomad combo using borgmatic docker container. I got it running again without that error after removal of --strip-components
witten commented 2023-06-23 03:32:48 +00:00
Owner
Copy Link

Got it, thanks.

Got it, thanks.
witten removed the
waiting for response
 label 2023-06-23 17:35:42 +00:00
witten commented 2023-06-23 17:36:05 +00:00
Owner
Copy Link

I believe this ticket to be fixed now as part of the work on #713. The fix should be part of the next release.

I believe this ticket to be fixed now as part of the work on #713. The fix should be part of the next release.
witten commented 2023-06-25 05:32:15 +00:00
Owner
Copy Link

Just released in borgmatic 1.7.15.

Just released in borgmatic 1.7.15.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
main
spot-check
1.8.10
1.8.9
1.8.8
1.8.7
1.8.6
1.8.5
1.8.4
1.8.3
1.8.2
1.8.1
1.8.0
1.7.15
1.7.14
1.7.13
1.7.12
1.7.11
1.7.10
1.7.9
1.7.8
1.7.7
1.7.6
1.7.5
1.7.4
1.7.3
1.7.2
1.7.1
1.7.0
1.6.6
1.6.5
1.6.4
1.6.3
1.6.2
1.6.1
1.6.0
1.5.24
1.5.23
1.5.22
1.5.21
1.5.20
1.5.19
1.5.18
1.5.17
1.5.16
1.5.15
1.5.14
1.5.13
1.5.12
1.5.11
1.5.10
1.5.9
1.5.8
1.5.7
1.5.7.dev0
1.5.6
1.5.5
1.5.4
1.5.3
1.5.2
1.5.1
1.5.0
1.4.22
1.4.21
1.4.20
1.4.19
1.4.18
1.4.17
1.4.16
1.4.15
1.4.14
1.4.13
1.4.12
1.4.11
1.4.10
1.4.9
1.4.8
1.4.7
1.4.6
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0
1.3.26
1.3.25
1.3.24
1.3.23
1.3.22
1.3.21
1.3.20
1.3.19
1.3.18
1.3.17
1.3.16
1.3.15
1.3.14
1.3.13
1.3.12
1.3.11
1.3.10
1.3.9
1.3.8
1.3.7
1.3.6
1.3.5
1.3.4
1.3.3
1.3.2
1.3.1
1.3.0
1.2.18
1.2.17
1.2.16
1.2.15
1.2.14
1.2.13
1.2.12
1.2.11
1.2.10
1.2.9
1.2.8
1.2.7
1.2.6
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.15
1.1.14
1.1.13
1.1.12
1.1.11
1.1.10
1.1.9
1.1.7
1.1.6
1.1.5
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.3
1.0.2
1.0.1
1.0.0
0.1.8
0.1.7
0.1.6
0.1.5
0.1.4
0.1.3
0.1.2
0.1.1
0.1.0
0.0.7
0.0.6
0.0.5
0.0.4
0.0.3
0.0.2
Labels
Clear labels   
bug

   
data loss

   
design finalized

   
good first issue

   
new feature area

   
question / support

   
security

   
waiting for response
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#711
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?