Save all databases in seperate files #393

New Issue

Closed

opened 2021-02-23 17:58:58 +00:00 by eipi-1 · 3 comments

eipi-1 commented 2021-02-23 17:58:58 +00:00

Copy Link

What I'm trying to do and why

I would like to save databases using the name: all feature in different files. This would allow deletion (and extraction) of seperate databases.

I need to backup a mariadb server with around 200 databases and growing. I currently use borgmatic to backup all databases using the name: all feature. Borgmatic creates a single file .borgmatic/mysql_databases/db_server/all. I would like to be able to delete a single database from all archives (using borg recreate). Therefor I need all databases in seperate paths.

Other notes / implementation ideas

In order to keep backwards compatibility this could by implemented using a config flag like seperate_files: true or by using a different keyword name: all_seperated.

Environment

borgmatic version: 1.5.10

borgmatic installation method: Docker

Borg version: 1.1.3

Python version: 3.6.8

Database version (if applicable): mariadb 10.5.9

#### What I'm trying to do and why I would like to save databases using the `name: all` feature in different files. This would allow deletion (and extraction) of seperate databases. I need to backup a mariadb server with around 200 databases and growing. I currently use borgmatic to backup all databases using the `name: all` feature. Borgmatic creates a single file `.borgmatic/mysql_databases/db_server/all`. I would like to be able to delete a single database from all archives (using [borg recreate](https://borgbackup.readthedocs.io/en/stable/usage/recreate.html)). Therefor I need all databases in seperate paths. #### Other notes / implementation ideas In order to keep backwards compatibility this could by implemented using a config flag like `seperate_files: true` or by using a different keyword `name: all_seperated`. #### Environment **borgmatic version:** 1.5.10 **borgmatic installation method:** [Docker](https://hub.docker.com/r/monachus/borgmatic) **Borg version:** 1.1.3 **Python version:** 3.6.8 **Database version (if applicable):** mariadb 10.5.9

👍 1

sbi commented 2021-05-16 09:35:45 +00:00

Copy Link

Seconded. I'd be willing to contribute a patch if we find an agreement on what the configuration should look like. I like introducing name: all_separated.

Seconded. I'd be willing to contribute a patch if we find an agreement on what the configuration should look like. I like introducing `name: all_separated`.

👍 1

ams_tschoening commented 2021-07-15 08:32:43 +00:00

Copy Link

I have exactly the same use case with PostgreSQL. For various historical reasons, my database sizes vary a lot as well, some store files with multiple GiB each, while others are somewhat small. All manual processes would benefit a lot if I was able to access only those files for individual databases of interest.

Additionally, it might even improve compression and de-duplication to "group" only that data which belongs together, regardless of any other. Would be great if pg_dumpall simply had such a feature already, so that BorgMatic would only need to forward some arguments, but that doesn't seem to be the case. Other people with similar use cases seem to simply split created dump files afterwards. While that might be an idea for a hook in BorgMatic, I gues it's difficult to implement with the streaming approach of database dumps, temporarily consumes more space etc.

I have exactly the same use case with PostgreSQL. For various historical reasons, my database sizes vary a lot as well, some store files with multiple GiB each, while others are somewhat small. All manual processes would benefit a lot if I was able to access only those files for individual databases of interest. Additionally, it might even improve compression and de-duplication to "group" only that data which belongs together, regardless of any other. Would be great if `pg_dumpall` simply had such a feature already, so that BorgMatic would only need to forward some arguments, [but that doesn't seem to be the case](https://www.postgresql.org/message-id/1225386069.20210715182946%40am-soft.de). Other people with similar use cases seem to simply [split created dump files afterwards](https://github.com/kruckenb/split_postgres_dump/blob/master/split-pg-dump.pl). While that might be an idea for a hook in BorgMatic, I gues it's difficult to implement with the streaming approach of database dumps, temporarily consumes more space etc.

witten commented 2023-01-26 23:21:47 +00:00

Owner

Copy Link

This has been implemented in borgmatic 1.7.6! To enable it, add a format option to your "all" database configuration. Then borgmatic will create individual dump files per database. See the documentation for more information: https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#all-databases

This has been implemented in borgmatic 1.7.6! To enable it, add a `format` option to your "all" database configuration. Then borgmatic will create individual dump files per database. See the documentation for more information: https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#all-databases

witten closed this issue 2023-01-26 23:21:47 +00:00

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

spot-check

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels   

bug

  

data loss

  

design finalized

  

good first issue

  

new feature area

  

question / support

  

security

  

waiting for response

No Label

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

Milestone

Clear milestone

No items

No Milestone

Assignees

Clear assignees

No Assignees

4 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#393

No description provided.