Data integrity error: Invalid segment entry header [segment 4439, offset 179348663]: unpack requires a buffer of 41 bytes #585

New Issue

modem7 · 2022-09-12T08:46:59Z

modem7 commented

2022-09-12 08:46:59 +00:00

What I'm trying to do and why

Attempting backup

Steps to reproduce (if a bug)

docker exec Borgmatic sh -c "cd && borgmatic --stats -v 1 --files 2>&1"

location:
    source_directories:
         - ${BORG_SOURCE_1}
         - ${BORG_SOURCE_2}
    repositories:
        - ${BORG_REPO_LOCAL}
        - ${BORG_REPO_REMOTE}
    one_file_system: true
    exclude_caches: true

storage:
#   Passphase is set in variable $BORG_PASSPHRASE
    compression: lz4
    archive_name_format: 'backup-{now}'

retention:
    keep_hourly: 0
    keep_daily: 7
    keep_weekly: 4
    keep_monthly: 12
    keep_yearly: 1
    prefix: 'backup-'

consistency:
    check_repositories:
      - ${BORG_REPO_LOCAL}
    checks:
      - name: repository
        frequency: 1 week
      - name: archives
        frequency: always
      - name: extract
        frequency: 2 weeks
      - name: data
        frequency: 1 month
    prefix: 'backup-'

hooks:
    before_everything:
        - echo "Starting a backup job."
        - echo "Stopping containers."
        - exec /borgscripts/docker-stop.sh
    after_everything:
        - echo "Starting containers."
        - exec /borgscripts/docker-start.sh
        - echo "Backup created."
    on_error:
        - echo "Error while creating a backup."
        - exec /borgscripts/docker-start.sh
    # https://torsion.org/borgmatic/docs/how-to/backup-your-databases/

    # Third-party services to notify you if backups aren't happening.
    healthchecks: ${BORG_HEALTHCHECK_URL}

    # https://torsion.org/borgmatic/docs/reference/configuration/

Actual behavior (if a bug)

Remote: check_free_space: required bytes 671482478, free bytes 6130344710144
Remote: Storage quota: 27.20 GB out of 394.90 GB used.
security: saving state for <id> to /root/.config/borg/security/<id>
security: current location   <id>
security: key type           5
security: manifest timestamp 2022-09-12T08:43:22.478181
Remote: Verified integrity of <id>
------------------------------------------------------------------------------
Repository: ssh://<borgbaseID>.repo.borgbase.com/./repo
Archive name: backup-2022-09-12T09:42:52
Archive fingerprint: <id>
Time (start): Mon, 2022-09-12 09:42:54
Time (end):   Mon, 2022-09-12 09:43:22
Duration: 27.96 seconds
Number of files: 58802
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:               14.79 GB              9.41 GB             27.85 MB
All archives:              291.47 GB            176.33 GB             26.52 GB
                       Unique chunks         Total chunks
Chunk index:                  360837              1966319
------------------------------------------------------------------------------
RemoteRepository: 28.08 MB bytes sent, 140.35 kB bytes received, 122 messages sent
terminating with success status, rc 0
/etc/borgmatic.d/config.yml: No commands to run for post-backup hook
/etc/borgmatic.d/config.yml: No commands to run for post-actions hook
/etc/borgmatic.d/config.yml: Running 2 commands for on-error hook
echo "Error while creating a backup."
Error while creating a backup.
exec /borgscripts/docker-start.sh
/etc/borgmatic.d/config.yml: Calling healthchecks hook function ping_monitor
/etc/borgmatic.d/config.yml: Pinging Healthchecks fail
/etc/borgmatic.d/config.yml: Using Healthchecks ping URL https://hc-ping.com/<id>/fail
/etc/borgmatic.d/config.yml: Calling healthchecks hook function destroy_monitor
/etc/borgmatic.d/config.yml: Error running configuration file
/etc/borgmatic.d/config.yml: Running 3 commands for post-everything hook
echo "Starting containers."
Starting containers.
exec /borgscripts/docker-start.sh
echo "Backup created."
Backup created.

summary:
/etc/borgmatic.d/config.yml: Error running configuration file
/mnt/borg-repository: Error running actions for repository
...
  File "/usr/local/lib/python3.10/site-packages/borg/archiver.py", line 5159, in main
    exit_code = archiver.run(args)
  File "/usr/local/lib/python3.10/site-packages/borg/archiver.py", line 5090, in run
    return set_ec(func(args))
  File "/usr/local/lib/python3.10/site-packages/borg/archiver.py", line 183, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/borg/archiver.py", line 1591, in do_prune
    archive.delete(stats, forced=args.forced)
  File "/usr/local/lib/python3.10/site-packages/borg/archive.py", line 1001, in delete
    for (i, (items_id, data)) in enumerate(zip(items_ids, self.repository.get_many(items_ids))):
  File "/usr/local/lib/python3.10/site-packages/borg/repository.py", line 1209, in get_many
    yield self.get(id_)
  File "/usr/local/lib/python3.10/site-packages/borg/repository.py", line 1203, in get
    return self.io.read(segment, offset, id)
  File "/usr/local/lib/python3.10/site-packages/borg/repository.py", line 1572, in read
    size, tag, key, data = self._read(fd, self.put_header_fmt, header, segment, offset, (TAG_PUT, ), read_data)
  File "/usr/local/lib/python3.10/site-packages/borg/repository.py", line 1587, in _read
    raise IntegrityError('Invalid segment entry header [segment {}, offset {}]: {}'.format(
borg.helpers.errors.IntegrityError: Data integrity error: Invalid segment entry header [segment 4439, offset 179348663]: unpack requires a buffer of 41 bytes
Platform: Linux a9626fc2faf9 5.15.0-47-generic #51-Ubuntu SMP Thu Aug 11 07:51:15 UTC 2022 x86_64
Linux: Unknown Linux
Borg: 1.2.2  Python: CPython 3.10.7 msgpack: 1.0.4 fuse: llfuse 1.4.2 [pyfuse3,llfuse]
PID: 294  CWD: /root
sys.argv: ['/usr/local/bin/borg', 'prune', '--keep-hourly', '0', '--keep-daily', '7', '--keep-weekly', '4', '--keep-monthly', '12', '--keep-yearly', '1', '--glob-archives', 'backup-*', '--stats', '--debug', '--show-rc', '/mnt/borg-repository']
SSH_ORIGINAL_COMMAND: None
Command 'borg prune --keep-hourly 0 --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --keep-yearly 1 --glob-archives backup-* --stats --debug --show-rc /mnt/borg-repository' returned non-zero exit status 2.

Have attempted:
borgmatic check --progress --repair

Environment

borgmatic version: borgmatic 1.7.2

borgmatic installation method: Docker

Borg version: borg 1.2.2

Python version: Python 3.10.7

operating system and version: Ubuntu 22.04

#### What I'm trying to do and why Attempting backup #### Steps to reproduce (if a bug) ```bash docker exec Borgmatic sh -c "cd && borgmatic --stats -v 1 --files 2>&1" ``` ```yaml location: source_directories: - ${BORG_SOURCE_1} - ${BORG_SOURCE_2} repositories: - ${BORG_REPO_LOCAL} - ${BORG_REPO_REMOTE} one_file_system: true exclude_caches: true storage: # Passphase is set in variable $BORG_PASSPHRASE compression: lz4 archive_name_format: 'backup-{now}' retention: keep_hourly: 0 keep_daily: 7 keep_weekly: 4 keep_monthly: 12 keep_yearly: 1 prefix: 'backup-' consistency: check_repositories: - ${BORG_REPO_LOCAL} checks: - name: repository frequency: 1 week - name: archives frequency: always - name: extract frequency: 2 weeks - name: data frequency: 1 month prefix: 'backup-' hooks: before_everything: - echo "Starting a backup job." - echo "Stopping containers." - exec /borgscripts/docker-stop.sh after_everything: - echo "Starting containers." - exec /borgscripts/docker-start.sh - echo "Backup created." on_error: - echo "Error while creating a backup." - exec /borgscripts/docker-start.sh # https://torsion.org/borgmatic/docs/how-to/backup-your-databases/ # Third-party services to notify you if backups aren't happening. healthchecks: ${BORG_HEALTHCHECK_URL} # https://torsion.org/borgmatic/docs/reference/configuration/ ``` #### Actual behavior (if a bug) ```shell Remote: check_free_space: required bytes 671482478, free bytes 6130344710144 Remote: Storage quota: 27.20 GB out of 394.90 GB used. security: saving state for <id> to /root/.config/borg/security/<id> security: current location <id> security: key type 5 security: manifest timestamp 2022-09-12T08:43:22.478181 Remote: Verified integrity of <id> ------------------------------------------------------------------------------ Repository: ssh://<borgbaseID>.repo.borgbase.com/./repo Archive name: backup-2022-09-12T09:42:52 Archive fingerprint: <id> Time (start): Mon, 2022-09-12 09:42:54 Time (end): Mon, 2022-09-12 09:43:22 Duration: 27.96 seconds Number of files: 58802 Utilization of max. archive size: 0% ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size This archive: 14.79 GB 9.41 GB 27.85 MB All archives: 291.47 GB 176.33 GB 26.52 GB Unique chunks Total chunks Chunk index: 360837 1966319 ------------------------------------------------------------------------------ RemoteRepository: 28.08 MB bytes sent, 140.35 kB bytes received, 122 messages sent terminating with success status, rc 0 /etc/borgmatic.d/config.yml: No commands to run for post-backup hook /etc/borgmatic.d/config.yml: No commands to run for post-actions hook /etc/borgmatic.d/config.yml: Running 2 commands for on-error hook echo "Error while creating a backup." Error while creating a backup. exec /borgscripts/docker-start.sh /etc/borgmatic.d/config.yml: Calling healthchecks hook function ping_monitor /etc/borgmatic.d/config.yml: Pinging Healthchecks fail /etc/borgmatic.d/config.yml: Using Healthchecks ping URL https://hc-ping.com/<id>/fail /etc/borgmatic.d/config.yml: Calling healthchecks hook function destroy_monitor /etc/borgmatic.d/config.yml: Error running configuration file /etc/borgmatic.d/config.yml: Running 3 commands for post-everything hook echo "Starting containers." Starting containers. exec /borgscripts/docker-start.sh echo "Backup created." Backup created. summary: /etc/borgmatic.d/config.yml: Error running configuration file /mnt/borg-repository: Error running actions for repository ... File "/usr/local/lib/python3.10/site-packages/borg/archiver.py", line 5159, in main exit_code = archiver.run(args) File "/usr/local/lib/python3.10/site-packages/borg/archiver.py", line 5090, in run return set_ec(func(args)) File "/usr/local/lib/python3.10/site-packages/borg/archiver.py", line 183, in wrapper return method(self, args, repository=repository, **kwargs) File "/usr/local/lib/python3.10/site-packages/borg/archiver.py", line 1591, in do_prune archive.delete(stats, forced=args.forced) File "/usr/local/lib/python3.10/site-packages/borg/archive.py", line 1001, in delete for (i, (items_id, data)) in enumerate(zip(items_ids, self.repository.get_many(items_ids))): File "/usr/local/lib/python3.10/site-packages/borg/repository.py", line 1209, in get_many yield self.get(id_) File "/usr/local/lib/python3.10/site-packages/borg/repository.py", line 1203, in get return self.io.read(segment, offset, id) File "/usr/local/lib/python3.10/site-packages/borg/repository.py", line 1572, in read size, tag, key, data = self._read(fd, self.put_header_fmt, header, segment, offset, (TAG_PUT, ), read_data) File "/usr/local/lib/python3.10/site-packages/borg/repository.py", line 1587, in _read raise IntegrityError('Invalid segment entry header [segment {}, offset {}]: {}'.format( borg.helpers.errors.IntegrityError: Data integrity error: Invalid segment entry header [segment 4439, offset 179348663]: unpack requires a buffer of 41 bytes Platform: Linux a9626fc2faf9 5.15.0-47-generic #51-Ubuntu SMP Thu Aug 11 07:51:15 UTC 2022 x86_64 Linux: Unknown Linux Borg: 1.2.2 Python: CPython 3.10.7 msgpack: 1.0.4 fuse: llfuse 1.4.2 [pyfuse3,llfuse] PID: 294 CWD: /root sys.argv: ['/usr/local/bin/borg', 'prune', '--keep-hourly', '0', '--keep-daily', '7', '--keep-weekly', '4', '--keep-monthly', '12', '--keep-yearly', '1', '--glob-archives', 'backup-*', '--stats', '--debug', '--show-rc', '/mnt/borg-repository'] SSH_ORIGINAL_COMMAND: None Command 'borg prune --keep-hourly 0 --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --keep-yearly 1 --glob-archives backup-* --stats --debug --show-rc /mnt/borg-repository' returned non-zero exit status 2. ``` Have attempted: `borgmatic check --progress --repair` #### Environment **borgmatic version:** borgmatic 1.7.2 **borgmatic installation method:** Docker **Borg version:** borg 1.2.2 **Python version:** Python 3.10.7 **operating system and version:** Ubuntu 22.04

witten commented

2022-09-12 18:36:48 +00:00

Uh oh.. This looks like a Borg data error with the repository! My recommendation:

Read this Borg information about integrity errors and perhaps follow some of the suggestions there for checking your hardware.
borgmatic check. That may provide more details about the repository problem.
borgmatic check --repair. That may fix the problem, depending on how severe it is. It's probably a good idea to backup (copy) your repository first, though, as there's no guarantee --repair will make things better.
If that still doesn't fix it, perhaps seek help from Borg project: File a ticket, ask on IRC, post on r/borgbackup, etc. It is possible your repository is toast, which means you may want to make a new repository and backup into it as soon as possible. Or, ideally, multiple repositories on different machines....

Hope that helps!

Uh oh.. This looks like a Borg data error with the repository! My recommendation: 1. Read this [Borg information about integrity errors](https://borgbackup.readthedocs.io/en/stable/faq.html#i-get-an-integrityerror-or-similar-what-now) and perhaps follow some of the suggestions there for checking your hardware. 2. `borgmatic check`. That may provide more details about the repository problem. 3. `borgmatic check --repair`. That *may* fix the problem, depending on how severe it is. It's probably a good idea to backup (copy) your repository first, though, as there's no guarantee `--repair` will make things better. 4. If that still doesn't fix it, perhaps seek help from Borg project: File a ticket, ask on IRC, post on r/borgbackup, etc. It *is* possible your repository is toast, which means you may want to make a new repository and backup into it as soon as possible. Or, ideally, multiple repositories on different machines.... Hope that helps!

witten added the

question / support

label 2022-09-12 18:36:58 +00:00

witten commented

2022-10-05 06:04:54 +00:00

Closing this for now, but feel free to follow up. We can always reopen it. Thanks!

👍 1

witten closed this issue

2022-10-05 06:04:54 +00:00

modem7 commented

2022-10-16 15:18:48 +00:00

Another repair did the trick.