PostgreSQL directory format dump not supported anymore #314

New Issue

Closed

opened 2020-05-18 13:11:26 +00:00 by rdunklau · 4 comments

rdunklau commented 2020-05-18 13:11:26 +00:00

Copy Link

Trying to perform a directory format dump with PostgreSQL

This used to be supported, and is very useful to avoid bloating a repository since in a directory backup, many files will not change from one backup to the next. This allows for a better reuse of chunks.

Steps to reproduce (if a bug)

Use the following configuration to dump the database:

postgresql_databases:
- name: mydb
format: directory
options: --compress 0

Actual behavior (if a bug)

The french pg_dump error says "no target directory given"

mai 18 14:33:02 tilleuls borgmatic[72086]: Command 'pg_dump --no-password --clean --if-exists --format directory --compress 0 mydb > /home/borg/.borgmatic/postgresql_databases/localhost/mydb' returned non-zero exit status 1.
mai 18 14:33:02 tilleuls borgmatic[72086]: CRITICAL pg_dump: erreur : aucun répertoire cible indiqué
mai 18 14:33:02 tilleuls borgmatic[72086]: CRITICAL Command 'pg_dump --no-password --clean --if-exists --format directory --compress 0 mydb > /home/borg/.borgmatic/postgresql_databases/localhost/mydb' returned non-zero exit s>
mai 18 14:33:02 tilleuls borgmatic[72086]: Need some help? https://torsion.org/borgmatic/#issues

Expected behavior (if a bug)

The dump should be performed in a (temp-ish) directory, then backed up

Other notes / implementation ideas

I think we should special case the directory format, since it can't redirect it's output to stdout.

Environment

borgmatic version: 1.5.4

borgmatic installation method: Archlinux package

Borg version: borg 1.1.11

Python version: Python 3.8.2

Database version (if applicable): PostgreSQL 12.3

operating system and version: Arch Linux

#### Trying to perform a directory format dump with PostgreSQL This used to be supported, and is very useful to avoid bloating a repository since in a directory backup, many files will not change from one backup to the next. This allows for a better reuse of chunks. #### Steps to reproduce (if a bug) Use the following configuration to dump the database: postgresql_databases: - name: mydb format: directory options: --compress 0 #### Actual behavior (if a bug) The french pg_dump error says "no target directory given" ``` mai 18 14:33:02 tilleuls borgmatic[72086]: Command 'pg_dump --no-password --clean --if-exists --format directory --compress 0 mydb > /home/borg/.borgmatic/postgresql_databases/localhost/mydb' returned non-zero exit status 1. mai 18 14:33:02 tilleuls borgmatic[72086]: CRITICAL pg_dump: erreur : aucun répertoire cible indiqué mai 18 14:33:02 tilleuls borgmatic[72086]: CRITICAL Command 'pg_dump --no-password --clean --if-exists --format directory --compress 0 mydb > /home/borg/.borgmatic/postgresql_databases/localhost/mydb' returned non-zero exit s> mai 18 14:33:02 tilleuls borgmatic[72086]: Need some help? https://torsion.org/borgmatic/#issues ``` #### Expected behavior (if a bug) The dump should be performed in a (temp-ish) directory, then backed up #### Other notes / implementation ideas I think we should special case the directory format, since it can't redirect it's output to stdout. #### Environment **borgmatic version:** 1.5.4 **borgmatic installation method:** Archlinux package **Borg version:** borg 1.1.11 **Python version:** Python 3.8.2 **Database version (if applicable):** PostgreSQL 12.3 **operating system and version:** Arch Linux

witten added the

bug

label 2020-05-18 15:38:39 +00:00

witten commented 2020-05-18 15:40:36 +00:00

Owner

Copy Link

Thank you for bringing this to my attention! I obviously overlooked support for this directory format with the most recent release. I'll have a look at the code and see what, if anything, I can do given the new streaming database dump approach.

Thank you for bringing this to my attention! I obviously overlooked support for this directory format with the most recent release. I'll have a look at the code and see what, if anything, I can do given the new streaming database dump approach.

witten referenced this issue from a commit 2020-05-18 18:34:16 +00:00

Fix regression in support for PostgreSQL's "directory" dump format (#314).

witten commented 2020-05-18 18:37:17 +00:00

Owner

Copy Link

Okay, I have a fix for this implemented in master, and it'll go out as part of the next release. I'll mention it on this ticket when that happens.

Some details: When the "directory" dump format is selected now, borgmatic does not stream that database to/from Borg for either the dump or the restore. Instead, it writes out the directory dump to a temporary location on disk, and restores from there as well.

I also added a regression test that should catch this sort of issue in the future.

Thanks again for reporting this!

Okay, I have a fix for this implemented in master, and it'll go out as part of the next release. I'll mention it on this ticket when that happens. Some details: When the "directory" dump format is selected now, borgmatic does not stream that database to/from Borg for either the dump or the restore. Instead, it writes out the directory dump to a temporary location on disk, and restores from there as well. I also added a regression test that should catch this sort of issue in the future. Thanks again for reporting this!

witten closed this issue 2020-05-18 18:37:17 +00:00

witten commented 2020-05-26 20:20:01 +00:00

Owner

Copy Link

Released in borgmatic 1.5.5!

Released in borgmatic 1.5.5!

rdunklau commented 2020-05-27 10:20:52 +00:00

Author

Copy Link

Thanks !

Thanks !

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

spot-check

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels   

bug

  

data loss

  

design finalized

  

good first issue

  

new feature area

  

question / support

  

security

  

waiting for response

No Label

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

Milestone

Clear milestone

No items

No Milestone

Assignees

Clear assignees

No Assignees

2 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#314

No description provided.