[Errno 7] Argument list too long: 'xxh64sum' #866

New Issue

Closed

opened 2024-05-15 14:23:33 +00:00 by michaeltoohig · 4 comments

michaeltoohig commented 2024-05-15 14:23:33 +00:00

Copy Link

What I'm trying to do and why

Create a new archive. Every day I run backup via cron and I am receiving a fail alert but this did not occur the first time I made a full backup only the 2nd and 3rd night so far. If I had to guess knowing xxhash is for comparing files it may be my spot check config failing to run. Is this a bug or am I comparing too many files, my config has a 10% compare configured.

Steps to reproduce

Create a new archive with the following config.

source_directories:
  - /mnt/source

repositories:
  - path: ssh://xxx@xxx.repo.borgbase.com/./repo
    label: borgbase-family-photos

one_file_system: false
archive_name_format: 'backup-{utcnow}'

exclude_if_present:
  - .nobackup
  - .NOBACKUP

keep_daily: 7
keep_weekly: 8
keep_monthly: 6
keep_yearly: 2

checks:
  - name: repository
    frequency: 1 month
  - name: archives
    frequency: 1 month
  - name: spot
    frequency: 1 week
    count_tolerance_percentage: 10
    data_sample_percentage: 10
    data_tolerance_percentage: 0.5

check_last: 3

before_everything:
  - echo "Starting a backup job."
after_everything:
  - echo "Backup created."
on_error:
  - echo "Error while creating a backup."

apprise:
  services:
    - url: tgram://xxx:xxx_xxx/xxx/
      label: telegram

  states:
    - start
    - finish
    - fail

Actual behavior

After the backup is complete the following error occurs which raises an fail message to my notifications.

CRITICAL:None:borgbase-family-photos: Error running actions for repository
CRITICAL:None:[Errno 7] Argument list too long: 'xxh64sum'
ERROR:borgmatic.execute:Error while creating a backup.
CRITICAL:None:/etc/borgmatic.d/config.yaml: An error occurred
WARNING:borgmatic.execute:Backup created.
CRITICAL:None:
CRITICAL:None:summary:
CRITICAL:None:/etc/borgmatic.d/config.yaml: An error occurred
CRITICAL:None:borgbase-family-photos: Error running actions for repository
CRITICAL:None:[Errno 7] Argument list too long: 'xxh64sum'
CRITICAL:borgmatic.commands.borgmatic:
CRITICAL:borgmatic.commands.borgmatic:Need some help? https://torsion.org/borgmatic/#issues

Expected behavior

No issue with xxh64sum.

Other notes / implementation ideas

No response

borgmatic version

1.8.11

borgmatic installation method

container

Borg version

1.2.8

Python version

3.12.3

Database version (if applicable)

mysql Ver 15.1 Distrib 10.11.6-MariaDB, for Linux (x86_64) using readline 5.1

Operating system and version

Alpine 3.19

### What I'm trying to do and why Create a new archive. Every day I run backup via cron and I am receiving a `fail` alert but this did not occur the first time I made a full backup only the 2nd and 3rd night so far. If I had to guess knowing xxhash is for comparing files it may be my spot check config failing to run. Is this a bug or am I comparing too many files, my config has a 10% compare configured. ### Steps to reproduce Create a new archive with the following config. ``` source_directories: - /mnt/source repositories: - path: ssh://xxx@xxx.repo.borgbase.com/./repo label: borgbase-family-photos one_file_system: false archive_name_format: 'backup-{utcnow}' exclude_if_present: - .nobackup - .NOBACKUP keep_daily: 7 keep_weekly: 8 keep_monthly: 6 keep_yearly: 2 checks: - name: repository frequency: 1 month - name: archives frequency: 1 month - name: spot frequency: 1 week count_tolerance_percentage: 10 data_sample_percentage: 10 data_tolerance_percentage: 0.5 check_last: 3 before_everything: - echo "Starting a backup job." after_everything: - echo "Backup created." on_error: - echo "Error while creating a backup." apprise: services: - url: tgram://xxx:xxx_xxx/xxx/ label: telegram states: - start - finish - fail ``` ### Actual behavior After the backup is complete the following error occurs which raises an `fail` message to my notifications. ``` CRITICAL:None:borgbase-family-photos: Error running actions for repository CRITICAL:None:[Errno 7] Argument list too long: 'xxh64sum' ERROR:borgmatic.execute:Error while creating a backup. CRITICAL:None:/etc/borgmatic.d/config.yaml: An error occurred WARNING:borgmatic.execute:Backup created. CRITICAL:None: CRITICAL:None:summary: CRITICAL:None:/etc/borgmatic.d/config.yaml: An error occurred CRITICAL:None:borgbase-family-photos: Error running actions for repository CRITICAL:None:[Errno 7] Argument list too long: 'xxh64sum' CRITICAL:borgmatic.commands.borgmatic: CRITICAL:borgmatic.commands.borgmatic:Need some help? https://torsion.org/borgmatic/#issues ``` ### Expected behavior No issue with `xxh64sum`. ### Other notes / implementation ideas _No response_ ### borgmatic version 1.8.11 ### borgmatic installation method container ### Borg version 1.2.8 ### Python version 3.12.3 ### Database version (if applicable) mysql Ver 15.1 Distrib 10.11.6-MariaDB, for Linux (x86_64) using readline 5.1 ### Operating system and version Alpine 3.19

👀 1

witten commented 2024-05-15 16:12:02 +00:00

Owner

Copy Link

Thanks so much for filing this. Yes, this is ostensibly a bug with the spot check. It looks like xxh64sum can't handle more than a certain number of files at once. Until this is fixed, you can try lowering your data_sample_percentage as a work-around.

Thanks so much for filing this. Yes, this is ostensibly a bug with the `spot` check. It looks like `xxh64sum` can't handle more than a certain number of files at once. Until this is fixed, you can try lowering your `data_sample_percentage` as a work-around.

witten added the

bug

label 2024-05-15 16:12:09 +00:00

michaeltoohig commented 2024-05-15 17:16:01 +00:00

Author

Copy Link

Thanks for the confirmation :)

Thanks for the confirmation :)

michaeltoohig referenced this issue 2024-05-15 17:29:47 +00:00

Q: Spot check schedule with config.yml & requesting clarification of spot check parameters #868

witten referenced this issue 2024-06-09 22:30:45 +00:00

Q: Spot check schedule with config.yml & requesting clarification of spot check parameters #868

witten referenced this issue from a commit 2024-06-10 05:54:33 +00:00

Fix "Argument list too long" error in the "spot" check when checking 100k+ files (#866).

witten commented 2024-06-10 05:57:17 +00:00

Owner

Copy Link

This has been fixed in main and will be part of the next release. This fix is performed by batching calls to xxh64sum (and also to Borg) so that it works on only 10,000 files at a time, looping as necessary to get through all the files that need hashing as part of this check. It's possible this could still error on some systems or when using super long filenames, so please let me know if you still see errors when this is released.

Thanks again for bringing this to my attention!

This has been fixed in main and will be part of the next release. This fix is performed by batching calls to xxh64sum (and also to Borg) so that it works on only 10,000 files at a time, looping as necessary to get through all the files that need hashing as part of this check. It's possible this could still error on some systems or when using super long filenames, so please let me know if you still see errors when this is released. Thanks again for bringing this to my attention!

witten closed this issue 2024-06-10 05:57:17 +00:00

witten commented 2024-06-24 00:05:51 +00:00

Owner

Copy Link

Released in borgmatic 1.8.12!

Released in borgmatic 1.8.12!

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

config-command-line

1.9.14

1.9.13

1.9.12

1.9.11

1.9.10

1.9.9

1.9.8

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.14

1.8.13

1.8.12

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels

blocked

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

No Label

bug

Milestone

No items

No Milestone

Assignees

Clear assignees

diivi import_bot witten

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#866

No description provided.