borgmatic-collective/borgmatic
11
70
Fork 40
Code Issues 49 Pull Requests 7 Actions 1 Packages Releases 152 Activity

Data integrity error: Invalid segment entry header [segment 4439, offset 179348663]: unpack requires a buffer of 41 bytes #585

New Issue
Closed
opened 2022-09-12 08:46:59 +00:00 by modem7 · 3 comments
modem7 commented 2022-09-12 08:46:59 +00:00
Copy Link

What I'm trying to do and why

Attempting backup

Steps to reproduce (if a bug)

docker exec Borgmatic sh -c "cd && borgmatic --stats -v 1 --files 2>&1"

location:
    source_directories:
         - ${BORG_SOURCE_1}
         - ${BORG_SOURCE_2}
    repositories:
        - ${BORG_REPO_LOCAL}
        - ${BORG_REPO_REMOTE}
    one_file_system: true
    exclude_caches: true

storage:
#   Passphase is set in variable $BORG_PASSPHRASE
    compression: lz4
    archive_name_format: 'backup-{now}'

retention:
    keep_hourly: 0
    keep_daily: 7
    keep_weekly: 4
    keep_monthly: 12
    keep_yearly: 1
    prefix: 'backup-'

consistency:
    check_repositories:
      - ${BORG_REPO_LOCAL}
    checks:
      - name: repository
        frequency: 1 week
      - name: archives
        frequency: always
      - name: extract
        frequency: 2 weeks
      - name: data
        frequency: 1 month
    prefix: 'backup-'

hooks:
    before_everything:
        - echo "Starting a backup job."
        - echo "Stopping containers."
        - exec /borgscripts/docker-stop.sh
    after_everything:
        - echo "Starting containers."
        - exec /borgscripts/docker-start.sh
        - echo "Backup created."
    on_error:
        - echo "Error while creating a backup."
        - exec /borgscripts/docker-start.sh
    # https://torsion.org/borgmatic/docs/how-to/backup-your-databases/

    # Third-party services to notify you if backups aren't happening.
    healthchecks: ${BORG_HEALTHCHECK_URL}

    # https://torsion.org/borgmatic/docs/reference/configuration/

Actual behavior (if a bug)

Remote: check_free_space: required bytes 671482478, free bytes 6130344710144
Remote: Storage quota: 27.20 GB out of 394.90 GB used.
security: saving state for <id> to /root/.config/borg/security/<id>
security: current location   <id>
security: key type           5
security: manifest timestamp 2022-09-12T08:43:22.478181
Remote: Verified integrity of <id>
------------------------------------------------------------------------------
Repository: ssh://<borgbaseID>.repo.borgbase.com/./repo
Archive name: backup-2022-09-12T09:42:52
Archive fingerprint: <id>
Time (start): Mon, 2022-09-12 09:42:54
Time (end):   Mon, 2022-09-12 09:43:22
Duration: 27.96 seconds
Number of files: 58802
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:               14.79 GB              9.41 GB             27.85 MB
All archives:              291.47 GB            176.33 GB             26.52 GB
                       Unique chunks         Total chunks
Chunk index:                  360837              1966319
------------------------------------------------------------------------------
RemoteRepository: 28.08 MB bytes sent, 140.35 kB bytes received, 122 messages sent
terminating with success status, rc 0
/etc/borgmatic.d/config.yml: No commands to run for post-backup hook
/etc/borgmatic.d/config.yml: No commands to run for post-actions hook
/etc/borgmatic.d/config.yml: Running 2 commands for on-error hook
echo "Error while creating a backup."
Error while creating a backup.
exec /borgscripts/docker-start.sh
/etc/borgmatic.d/config.yml: Calling healthchecks hook function ping_monitor
/etc/borgmatic.d/config.yml: Pinging Healthchecks fail
/etc/borgmatic.d/config.yml: Using Healthchecks ping URL https://hc-ping.com/<id>/fail
/etc/borgmatic.d/config.yml: Calling healthchecks hook function destroy_monitor
/etc/borgmatic.d/config.yml: Error running configuration file
/etc/borgmatic.d/config.yml: Running 3 commands for post-everything hook
echo "Starting containers."
Starting containers.
exec /borgscripts/docker-start.sh
echo "Backup created."
Backup created.

summary:
/etc/borgmatic.d/config.yml: Error running configuration file
/mnt/borg-repository: Error running actions for repository
...
  File "/usr/local/lib/python3.10/site-packages/borg/archiver.py", line 5159, in main
    exit_code = archiver.run(args)
  File "/usr/local/lib/python3.10/site-packages/borg/archiver.py", line 5090, in run
    return set_ec(func(args))
  File "/usr/local/lib/python3.10/site-packages/borg/archiver.py", line 183, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/borg/archiver.py", line 1591, in do_prune
    archive.delete(stats, forced=args.forced)
  File "/usr/local/lib/python3.10/site-packages/borg/archive.py", line 1001, in delete
    for (i, (items_id, data)) in enumerate(zip(items_ids, self.repository.get_many(items_ids))):
  File "/usr/local/lib/python3.10/site-packages/borg/repository.py", line 1209, in get_many
    yield self.get(id_)
  File "/usr/local/lib/python3.10/site-packages/borg/repository.py", line 1203, in get
    return self.io.read(segment, offset, id)
  File "/usr/local/lib/python3.10/site-packages/borg/repository.py", line 1572, in read
    size, tag, key, data = self._read(fd, self.put_header_fmt, header, segment, offset, (TAG_PUT, ), read_data)
  File "/usr/local/lib/python3.10/site-packages/borg/repository.py", line 1587, in _read
    raise IntegrityError('Invalid segment entry header [segment {}, offset {}]: {}'.format(
borg.helpers.errors.IntegrityError: Data integrity error: Invalid segment entry header [segment 4439, offset 179348663]: unpack requires a buffer of 41 bytes
Platform: Linux a9626fc2faf9 5.15.0-47-generic #51-Ubuntu SMP Thu Aug 11 07:51:15 UTC 2022 x86_64
Linux: Unknown Linux
Borg: 1.2.2  Python: CPython 3.10.7 msgpack: 1.0.4 fuse: llfuse 1.4.2 [pyfuse3,llfuse]
PID: 294  CWD: /root
sys.argv: ['/usr/local/bin/borg', 'prune', '--keep-hourly', '0', '--keep-daily', '7', '--keep-weekly', '4', '--keep-monthly', '12', '--keep-yearly', '1', '--glob-archives', 'backup-*', '--stats', '--debug', '--show-rc', '/mnt/borg-repository']
SSH_ORIGINAL_COMMAND: None
Command 'borg prune --keep-hourly 0 --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --keep-yearly 1 --glob-archives backup-* --stats --debug --show-rc /mnt/borg-repository' returned non-zero exit status 2.

Have attempted:
borgmatic check --progress --repair

Environment

borgmatic version: borgmatic 1.7.2

borgmatic installation method: Docker

Borg version: borg 1.2.2

Python version: Python 3.10.7

operating system and version: Ubuntu 22.04

#### What I'm trying to do and why Attempting backup #### Steps to reproduce (if a bug) ```bash docker exec Borgmatic sh -c "cd && borgmatic --stats -v 1 --files 2>&1" ``` ```yaml location: source_directories: - ${BORG_SOURCE_1} - ${BORG_SOURCE_2} repositories: - ${BORG_REPO_LOCAL} - ${BORG_REPO_REMOTE} one_file_system: true exclude_caches: true storage: # Passphase is set in variable $BORG_PASSPHRASE compression: lz4 archive_name_format: 'backup-{now}' retention: keep_hourly: 0 keep_daily: 7 keep_weekly: 4 keep_monthly: 12 keep_yearly: 1 prefix: 'backup-' consistency: check_repositories: - ${BORG_REPO_LOCAL} checks: - name: repository frequency: 1 week - name: archives frequency: always - name: extract frequency: 2 weeks - name: data frequency: 1 month prefix: 'backup-' hooks: before_everything: - echo "Starting a backup job." - echo "Stopping containers." - exec /borgscripts/docker-stop.sh after_everything: - echo "Starting containers." - exec /borgscripts/docker-start.sh - echo "Backup created." on_error: - echo "Error while creating a backup." - exec /borgscripts/docker-start.sh # https://torsion.org/borgmatic/docs/how-to/backup-your-databases/ # Third-party services to notify you if backups aren't happening. healthchecks: ${BORG_HEALTHCHECK_URL} # https://torsion.org/borgmatic/docs/reference/configuration/ ``` #### Actual behavior (if a bug) ```shell Remote: check_free_space: required bytes 671482478, free bytes 6130344710144 Remote: Storage quota: 27.20 GB out of 394.90 GB used. security: saving state for <id> to /root/.config/borg/security/<id> security: current location <id> security: key type 5 security: manifest timestamp 2022-09-12T08:43:22.478181 Remote: Verified integrity of <id> ------------------------------------------------------------------------------ Repository: ssh://<borgbaseID>.repo.borgbase.com/./repo Archive name: backup-2022-09-12T09:42:52 Archive fingerprint: <id> Time (start): Mon, 2022-09-12 09:42:54 Time (end): Mon, 2022-09-12 09:43:22 Duration: 27.96 seconds Number of files: 58802 Utilization of max. archive size: 0% ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size This archive: 14.79 GB 9.41 GB 27.85 MB All archives: 291.47 GB 176.33 GB 26.52 GB Unique chunks Total chunks Chunk index: 360837 1966319 ------------------------------------------------------------------------------ RemoteRepository: 28.08 MB bytes sent, 140.35 kB bytes received, 122 messages sent terminating with success status, rc 0 /etc/borgmatic.d/config.yml: No commands to run for post-backup hook /etc/borgmatic.d/config.yml: No commands to run for post-actions hook /etc/borgmatic.d/config.yml: Running 2 commands for on-error hook echo "Error while creating a backup." Error while creating a backup. exec /borgscripts/docker-start.sh /etc/borgmatic.d/config.yml: Calling healthchecks hook function ping_monitor /etc/borgmatic.d/config.yml: Pinging Healthchecks fail /etc/borgmatic.d/config.yml: Using Healthchecks ping URL https://hc-ping.com/<id>/fail /etc/borgmatic.d/config.yml: Calling healthchecks hook function destroy_monitor /etc/borgmatic.d/config.yml: Error running configuration file /etc/borgmatic.d/config.yml: Running 3 commands for post-everything hook echo "Starting containers." Starting containers. exec /borgscripts/docker-start.sh echo "Backup created." Backup created. summary: /etc/borgmatic.d/config.yml: Error running configuration file /mnt/borg-repository: Error running actions for repository ... File "/usr/local/lib/python3.10/site-packages/borg/archiver.py", line 5159, in main exit_code = archiver.run(args) File "/usr/local/lib/python3.10/site-packages/borg/archiver.py", line 5090, in run return set_ec(func(args)) File "/usr/local/lib/python3.10/site-packages/borg/archiver.py", line 183, in wrapper return method(self, args, repository=repository, **kwargs) File "/usr/local/lib/python3.10/site-packages/borg/archiver.py", line 1591, in do_prune archive.delete(stats, forced=args.forced) File "/usr/local/lib/python3.10/site-packages/borg/archive.py", line 1001, in delete for (i, (items_id, data)) in enumerate(zip(items_ids, self.repository.get_many(items_ids))): File "/usr/local/lib/python3.10/site-packages/borg/repository.py", line 1209, in get_many yield self.get(id_) File "/usr/local/lib/python3.10/site-packages/borg/repository.py", line 1203, in get return self.io.read(segment, offset, id) File "/usr/local/lib/python3.10/site-packages/borg/repository.py", line 1572, in read size, tag, key, data = self._read(fd, self.put_header_fmt, header, segment, offset, (TAG_PUT, ), read_data) File "/usr/local/lib/python3.10/site-packages/borg/repository.py", line 1587, in _read raise IntegrityError('Invalid segment entry header [segment {}, offset {}]: {}'.format( borg.helpers.errors.IntegrityError: Data integrity error: Invalid segment entry header [segment 4439, offset 179348663]: unpack requires a buffer of 41 bytes Platform: Linux a9626fc2faf9 5.15.0-47-generic #51-Ubuntu SMP Thu Aug 11 07:51:15 UTC 2022 x86_64 Linux: Unknown Linux Borg: 1.2.2 Python: CPython 3.10.7 msgpack: 1.0.4 fuse: llfuse 1.4.2 [pyfuse3,llfuse] PID: 294 CWD: /root sys.argv: ['/usr/local/bin/borg', 'prune', '--keep-hourly', '0', '--keep-daily', '7', '--keep-weekly', '4', '--keep-monthly', '12', '--keep-yearly', '1', '--glob-archives', 'backup-*', '--stats', '--debug', '--show-rc', '/mnt/borg-repository'] SSH_ORIGINAL_COMMAND: None Command 'borg prune --keep-hourly 0 --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --keep-yearly 1 --glob-archives backup-* --stats --debug --show-rc /mnt/borg-repository' returned non-zero exit status 2. ``` Have attempted: `borgmatic check --progress --repair` #### Environment **borgmatic version:** borgmatic 1.7.2 **borgmatic installation method:** Docker **Borg version:** borg 1.2.2 **Python version:** Python 3.10.7 **operating system and version:** Ubuntu 22.04
witten commented 2022-09-12 18:36:48 +00:00
Owner
Copy Link

Uh oh.. This looks like a Borg data error with the repository! My recommendation:

  1. Read this Borg information about integrity errors and perhaps follow some of the suggestions there for checking your hardware.
  2. borgmatic check. That may provide more details about the repository problem.
  3. borgmatic check --repair. That may fix the problem, depending on how severe it is. It's probably a good idea to backup (copy) your repository first, though, as there's no guarantee --repair will make things better.
  4. If that still doesn't fix it, perhaps seek help from Borg project: File a ticket, ask on IRC, post on r/borgbackup, etc. It is possible your repository is toast, which means you may want to make a new repository and backup into it as soon as possible. Or, ideally, multiple repositories on different machines....

Hope that helps!

Uh oh.. This looks like a Borg data error with the repository! My recommendation: 1. Read this [Borg information about integrity errors](https://borgbackup.readthedocs.io/en/stable/faq.html#i-get-an-integrityerror-or-similar-what-now) and perhaps follow some of the suggestions there for checking your hardware. 2. `borgmatic check`. That may provide more details about the repository problem. 3. `borgmatic check --repair`. That *may* fix the problem, depending on how severe it is. It's probably a good idea to backup (copy) your repository first, though, as there's no guarantee `--repair` will make things better. 4. If that still doesn't fix it, perhaps seek help from Borg project: File a ticket, ask on IRC, post on r/borgbackup, etc. It *is* possible your repository is toast, which means you may want to make a new repository and backup into it as soon as possible. Or, ideally, multiple repositories on different machines.... Hope that helps!
witten added the question / support label 2022-09-12 18:36:58 +00:00
witten commented 2022-10-05 06:04:54 +00:00
Owner
Copy Link

Closing this for now, but feel free to follow up. We can always reopen it. Thanks!

Closing this for now, but feel free to follow up. We can always reopen it. Thanks!
👍 1
modem7 commented 2022-10-16 15:18:48 +00:00
Author
Copy Link

Another repair did the trick.

Sorry for lack of comms!

Another repair did the trick. Sorry for lack of comms!
👍 1
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
blocked
breaking
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No Label question / support
Milestone
No items
No Milestone
Assignees
Clear assignees
witten
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#585
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?