borgmatic-collective/borgmatic
borgmatic-collective
/
borgmatic
15
57
Fork 40
Code Issues 53 Pull Requests 2 Actions Packages Releases 121 Activity

Restoring a specific database conflicts with creating a backup that includes all databases #560

New Issue
Closed
opened 2022-07-05 18:07:38 +00:00 by Alex789 · 4 comments
Alex789 commented 2022-07-05 18:07:38 +00:00
Copy Link

What I'm trying to do and why

I made borgmatic create a backup including all databases by writing "- name: all" in the /etc/borgmatic/config.yaml file.
This is according to the guideline "If you want to dump all databases on a host, use all for the database name".

I have a script that automatically creates new databases. The "- name: all" option comes in handy. Otherwise, I would have to update the config file everytime I create a new database.

Steps to reproduce

Make a backup including all databases with this config:

location:
    source_directories:
         - /var/www/html
         - /etc
         - /root
         - /var/log/syslog*
    repositories:
         - blablabla@blabla.repo.borgbase.com:repo
    exclude_patterns:
        - '*.pyc'
        - /home/*/.cache
    exclude_caches: true
    exclude_if_present:
        - .nobackup

storage:
    encryption_passphrase: "blablablablabla"
    ssh_command: ssh -i /root/.ssh/id_ed25519_blabla
    archive_name_format: '{hostname}-{now}'

retention:
    keep_within: 24H
    keep_daily: 7
    keep_weekly: 4
    keep_monthly: 6
    keep_yearly: 1

consistency:
    checks:
        - name: repository
        - name: archives
          frequency: 2 weeks

hooks:
  mysql_databases:
        - name: all
  healthchecks:
      ping_url: https://hc-ping.com/blablablabla

Run borgmatic create
Delete the database using mysql> drop database at409171;
Restore the database using borgmatic restore --archive blabla-2022-07-05T15:03:38 --database at409171

Actual behavior

After restoring the database, the database is still not there. (mysql> show databases; doesn't show the database.)

I get this output:

borgmatic restore --archive blabla-2022-07-05T15:03:38 --database at409171 -v 2
Ensuring legacy configuration is upgraded
borg --version --debug --show-rc
blablabla@blabla.repo.borgbase.com:repo: Restoring databases from archive blabla-2022-07-05T15:03:38
blablabla@blabla.repo.borgbase.com:repo: Calling mysql_databases hook function remove_database_dumps
blablabla@blabla.repo.borgbase.com:repo: Removing MySQL database dumps
blablabla@blabla.repo.borgbase.com:repo: Calling mysql_databases hook function make_database_dump_pattern
borg extract --debug --list --show-rc --stdout blablabla@blabla.repo.borgbase.com:repo::blabla-2022-07-05T15:03:38 sh:root/.borgmatic/mysql_databases/*/at409171 > 
blablabla@blabla.repo.borgbase.com:repo: Calling mysql_databases hook function restore_database_dump
blablabla@blabla.repo.borgbase.com:repo: Restoring MySQL database at409171
mysql --batch < 3
using builtin fallback logging configuration
35 self tests completed in 0.12 seconds
SSH command line: ['ssh', '-i', '/root/.ssh/id_ed25519_blabla', 'blablabla@blabla.repo.borgbase.com', 'borg', 'serve', '--umask=077', '--debug']
Remote: using builtin fallback logging configuration
Remote: borg selftest disabled via BORG_SELFTEST env variable
Remote: using builtin fallback logging configuration
Remote: Initialized logging system for JSON-based protocol
Remote: Resolving repository path b'repo'
Remote: Resolved repository path to '/srv/repos/blabla/repo'
Remote: Verified integrity of /srv/repos/blabla/repo/index.815
TAM-verified manifest
security: read previous location 'ssh://blablabla@blabla.repo.borgbase.com/./repo'
security: read manifest timestamp '2022-07-05T16:48:42.003492'
security: determined newest manifest timestamp as 2022-07-05T16:48:42.003492
security: repository checks ok, allowing access
Include pattern 'root/.borgmatic/mysql_databases/*/at409171' never matched.
RemoteRepository: 5.72 kB bytes sent, 4.51 MB bytes received, 113 messages sent
terminating with warning status, rc 1
blablabla@blabla.repo.borgbase.com:repo: Calling mysql_databases hook function remove_database_dumps
blablabla@blabla.repo.borgbase.com:repo: Removing MySQL database dumps

summary:
/etc/borgmatic/config.yaml: Successfully ran configuration file

There is no error notification or similar.

Note the line: Include pattern 'root/.borgmatic/mysql_databases/*/at409171' never matched.
It seems that borgmatic is searching for a file called at409171 (name of the database).
However, when I normally extract the repo, I'll find the dumps of all the databases in one common file called root/.borgmatic/mysql_databases/localhost/all.

Expected behavior

The database should be restored.
If the database can't be restored (file not found), an error notification should be displayed.

Other notes / implementation ideas

When creating the backup, dump the databases in separate files for each database.

Environment

borgmatic version: 1.6.5

borgmatic installation method: sudo pip install --user --upgrade borgmatic

Borg version: borg 1.1.15

Python version: Python 3.8.10

Database version: mysql Ver 8.0.29-0ubuntu0.20.04.3 for Linux on x86_64 ((Ubuntu))

operating system and version: Ubuntu 20.04.4 LTS

#### What I'm trying to do and why I made borgmatic create a backup including all databases by writing "- name: all" in the /etc/borgmatic/config.yaml file. This is according to the [guideline](https://https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#database-dump-hooks) "If you want to dump all databases on a host, use all for the database name". I have a script that automatically creates new databases. The "- name: all" option comes in handy. Otherwise, I would have to update the config file everytime I create a new database. #### Steps to reproduce Make a backup including all databases with this config: ``` location: source_directories: - /var/www/html - /etc - /root - /var/log/syslog* repositories: - blablabla@blabla.repo.borgbase.com:repo exclude_patterns: - '*.pyc' - /home/*/.cache exclude_caches: true exclude_if_present: - .nobackup storage: encryption_passphrase: "blablablablabla" ssh_command: ssh -i /root/.ssh/id_ed25519_blabla archive_name_format: '{hostname}-{now}' retention: keep_within: 24H keep_daily: 7 keep_weekly: 4 keep_monthly: 6 keep_yearly: 1 consistency: checks: - name: repository - name: archives frequency: 2 weeks hooks: mysql_databases: - name: all healthchecks: ping_url: https://hc-ping.com/blablablabla ``` Run `borgmatic create` Delete the database using `mysql> drop database at409171; ` Restore the database using `borgmatic restore --archive blabla-2022-07-05T15:03:38 --database at409171` #### Actual behavior After restoring the database, the database is still not there. (`mysql> show databases;` doesn't show the database.) I get this output: ``` borgmatic restore --archive blabla-2022-07-05T15:03:38 --database at409171 -v 2 Ensuring legacy configuration is upgraded borg --version --debug --show-rc blablabla@blabla.repo.borgbase.com:repo: Restoring databases from archive blabla-2022-07-05T15:03:38 blablabla@blabla.repo.borgbase.com:repo: Calling mysql_databases hook function remove_database_dumps blablabla@blabla.repo.borgbase.com:repo: Removing MySQL database dumps blablabla@blabla.repo.borgbase.com:repo: Calling mysql_databases hook function make_database_dump_pattern borg extract --debug --list --show-rc --stdout blablabla@blabla.repo.borgbase.com:repo::blabla-2022-07-05T15:03:38 sh:root/.borgmatic/mysql_databases/*/at409171 > blablabla@blabla.repo.borgbase.com:repo: Calling mysql_databases hook function restore_database_dump blablabla@blabla.repo.borgbase.com:repo: Restoring MySQL database at409171 mysql --batch < 3 using builtin fallback logging configuration 35 self tests completed in 0.12 seconds SSH command line: ['ssh', '-i', '/root/.ssh/id_ed25519_blabla', 'blablabla@blabla.repo.borgbase.com', 'borg', 'serve', '--umask=077', '--debug'] Remote: using builtin fallback logging configuration Remote: borg selftest disabled via BORG_SELFTEST env variable Remote: using builtin fallback logging configuration Remote: Initialized logging system for JSON-based protocol Remote: Resolving repository path b'repo' Remote: Resolved repository path to '/srv/repos/blabla/repo' Remote: Verified integrity of /srv/repos/blabla/repo/index.815 TAM-verified manifest security: read previous location 'ssh://blablabla@blabla.repo.borgbase.com/./repo' security: read manifest timestamp '2022-07-05T16:48:42.003492' security: determined newest manifest timestamp as 2022-07-05T16:48:42.003492 security: repository checks ok, allowing access Include pattern 'root/.borgmatic/mysql_databases/*/at409171' never matched. RemoteRepository: 5.72 kB bytes sent, 4.51 MB bytes received, 113 messages sent terminating with warning status, rc 1 blablabla@blabla.repo.borgbase.com:repo: Calling mysql_databases hook function remove_database_dumps blablabla@blabla.repo.borgbase.com:repo: Removing MySQL database dumps summary: /etc/borgmatic/config.yaml: Successfully ran configuration file ``` There is no error notification or similar. **Note the line**: `Include pattern 'root/.borgmatic/mysql_databases/*/at409171' never matched.` It seems that borgmatic is searching for a file called at409171 (name of the database). However, when I normally extract the repo, I'll find the dumps of all the databases in one common file called `root/.borgmatic/mysql_databases/localhost/all`. #### Expected behavior The database should be restored. If the database can't be restored (file not found), an error notification should be displayed. #### Other notes / implementation ideas When creating the backup, dump the databases in separate files for each database. #### Environment **borgmatic version:** 1.6.5 **borgmatic installation method:** sudo pip install --user --upgrade borgmatic **Borg version:** borg 1.1.15 **Python version:** Python 3.8.10 **Database version:** mysql Ver 8.0.29-0ubuntu0.20.04.3 for Linux on x86_64 ((Ubuntu)) **operating system and version:** Ubuntu 20.04.4 LTS
witten commented 2022-07-05 18:12:41 +00:00
Owner
Copy Link

Thanks for filing this! Yeah, borgmatic restores are currently all-or-nothing when the database is all. Your idea for supporting a single database restore with separate files is probably the way to go. See #438 for a similar idea.

Thanks for filing this! Yeah, borgmatic restores are currently all-or-nothing when the database is `all`. Your idea for supporting a single database restore with separate files is probably the way to go. See #438 for a similar idea.
witten added the
bug
 label 2022-07-05 18:13:59 +00:00
witten commented 2022-07-07 06:23:23 +00:00
Owner
Copy Link

If the database can't be restored (file not found), an error notification should be displayed.

This part of the ticket is done in master and will be part of the next release. I'm still trying to figure out what to do for the rest of the ticket.

> If the database can't be restored (file not found), an error notification should be displayed. This part of the ticket is done in master and will be part of the next release. I'm still trying to figure out what to do for the rest of the ticket.
witten commented 2022-07-20 03:35:30 +00:00
Owner
Copy Link

The error notifcation portion of the ticket has been released in borgmatic 1.6.6.

The error notifcation portion of the ticket has been released in borgmatic 1.6.6.
witten commented 2023-01-26 23:22:41 +00:00
Owner
Copy Link

This has been implemented in borgmatic 1.7.6! To enable it, add a format option to your "all" database configuration. Then borgmatic will create individual dump files per database. See the documentation for more information: https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#all-databases

This has been implemented in borgmatic 1.7.6! To enable it, add a `format` option to your "all" database configuration. Then borgmatic will create individual dump files per database. See the documentation for more information: https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#all-databases
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
main
spot-check
1.8.10
1.8.9
1.8.8
1.8.7
1.8.6
1.8.5
1.8.4
1.8.3
1.8.2
1.8.1
1.8.0
1.7.15
1.7.14
1.7.13
1.7.12
1.7.11
1.7.10
1.7.9
1.7.8
1.7.7
1.7.6
1.7.5
1.7.4
1.7.3
1.7.2
1.7.1
1.7.0
1.6.6
1.6.5
1.6.4
1.6.3
1.6.2
1.6.1
1.6.0
1.5.24
1.5.23
1.5.22
1.5.21
1.5.20
1.5.19
1.5.18
1.5.17
1.5.16
1.5.15
1.5.14
1.5.13
1.5.12
1.5.11
1.5.10
1.5.9
1.5.8
1.5.7
1.5.7.dev0
1.5.6
1.5.5
1.5.4
1.5.3
1.5.2
1.5.1
1.5.0
1.4.22
1.4.21
1.4.20
1.4.19
1.4.18
1.4.17
1.4.16
1.4.15
1.4.14
1.4.13
1.4.12
1.4.11
1.4.10
1.4.9
1.4.8
1.4.7
1.4.6
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0
1.3.26
1.3.25
1.3.24
1.3.23
1.3.22
1.3.21
1.3.20
1.3.19
1.3.18
1.3.17
1.3.16
1.3.15
1.3.14
1.3.13
1.3.12
1.3.11
1.3.10
1.3.9
1.3.8
1.3.7
1.3.6
1.3.5
1.3.4
1.3.3
1.3.2
1.3.1
1.3.0
1.2.18
1.2.17
1.2.16
1.2.15
1.2.14
1.2.13
1.2.12
1.2.11
1.2.10
1.2.9
1.2.8
1.2.7
1.2.6
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.15
1.1.14
1.1.13
1.1.12
1.1.11
1.1.10
1.1.9
1.1.7
1.1.6
1.1.5
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.3
1.0.2
1.0.1
1.0.0
0.1.8
0.1.7
0.1.6
0.1.5
0.1.4
0.1.3
0.1.2
0.1.1
0.1.0
0.0.7
0.0.6
0.0.5
0.0.4
0.0.3
0.0.2
Labels
Clear labels   
bug

   
data loss

   
design finalized

   
good first issue

   
new feature area

   
question / support

   
security

   
waiting for response
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#560
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?