pg_dump in directory format can't be done if extracted using docker exec #978

New issue

Closed

opened 2025-01-24 16:58:53 +00:00 by vitoyucepi · 5 comments

vitoyucepi commented 2025-01-24 16:58:53 +00:00

Copy link

What I'm trying to do and why

I was just curious and tried to test pg_dump for #975.

postgresql_databases:
  - name: postgres
    username: postgres
    password: password
    format: directory
    pg_dump_command: docker exec postgres pg_dump

Steps to reproduce

1. docker run --name database -e POSTGRES_PASSWORD=password -d postgres
2. /etc/borgmatic.d/postgres.yaml

   repositories:
     - path: /tmp
   
   read_special: true
   
   postgresql_databases:
     - name: postgres
       username: postgres
       password: password
       format: directory
       pg_dump_command: docker exec postgres pg_dump
   

3. borgmatic -c /etc/borgmatic.d/postgres.yaml create -v2

Actual behavior

The database tries to create dump in the database container, but borgmatic expects it in the local directory.

pg_dump: error: could not create directory "/tmp/borgmatic-aaaabbbb/./borgmatic/postgresql_databases/localhost/postgres": No such file or directory

Expected behavior

The problem is not so simple, I don't expect anything.

Other notes / implementation ideas

Add a note about this behavior in the documentation.

borgmatic version

1.9.6

borgmatic installation method

self built package in a container

Borg version

1.4.0

Python version

Python 3.12.8

Database version (if applicable)

15.2

Operating system and version

alpine:3.21

### What I'm trying to do and why I was just curious and tried to test `pg_dump` for #975. ```yaml postgresql_databases: - name: postgres username: postgres password: password format: directory pg_dump_command: docker exec postgres pg_dump ``` ### Steps to reproduce 1. `docker run --name database -e POSTGRES_PASSWORD=password -d postgres` 2. `/etc/borgmatic.d/postgres.yaml` ```yaml repositories: - path: /tmp read_special: true postgresql_databases: - name: postgres username: postgres password: password format: directory pg_dump_command: docker exec postgres pg_dump ``` 1. `borgmatic -c /etc/borgmatic.d/postgres.yaml create -v2` ### Actual behavior The database tries to create dump in the database container, but borgmatic expects it in the local directory. ``` pg_dump: error: could not create directory "/tmp/borgmatic-aaaabbbb/./borgmatic/postgresql_databases/localhost/postgres": No such file or directory ``` ### Expected behavior The problem is not so simple, I don't expect anything. ### Other notes / implementation ideas Add a note about this behavior in the documentation. ### borgmatic version 1.9.6 ### borgmatic installation method self built package in a container ### Borg version 1.4.0 ### Python version Python 3.12.8 ### Database version (if applicable) 15.2 ### Operating system and version alpine:3.21

witten referenced this issue from a commit 2025-01-24 22:32:47 +00:00

Clarify/correct documentation about dumping databases when using containers (#978).

witten commented 2025-01-24 22:35:27 +00:00

Owner

Copy link

Thanks for taking the time to file and describe this. Have you tried mounting the borgmatic runtime directory into the database container? See the docs here, which should be updated shortly with a correction about the runtime directory: https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#containers

Thanks for taking the time to file and describe this. Have you tried mounting the borgmatic runtime directory into the database container? See the docs here, which should be updated shortly with [a correction](https://projects.torsion.org/borgmatic-collective/borgmatic/commit/7551810ea60e43972d58fa725d765b08d84e590f) about the runtime directory: https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#containers

witten added the

question / support

label 2025-01-24 22:35:34 +00:00

vitoyucepi commented 2025-01-25 12:41:08 +00:00

Author

Copy link

tried mounting the borgmatic runtime directory

In my case this could be achieved by adding -v /tmp/borgmatic:/tmp/borgmatic to the docker run command and setting user_runtime_directory: /tmp/borgmatic. I think this would work fine, but without it I can't extract the directory from the container.

The configuration reference mentions borgmatic_runtime_directory only in one place.

borgmatic/config/schema.yaml

Lines 203 to 218 in 7551810

	borgmatic_source_directory:
	type: string
	description: |
	Deprecated. Only used for locating database dumps and bootstrap
	metadata within backup archives created prior to deprecation.
	Replaced by borgmatic_runtime_directory and
	borgmatic_state_directory. Defaults to ~/.borgmatic
	example: /tmp/borgmatic
	user_runtime_directory:
	type: string
	description: |
	Path for storing temporary runtime data like streaming database
	dumps and bootstrap metadata. borgmatic automatically creates and
	uses a "borgmatic" subdirectory here. Defaults to $XDG_RUNTIME_DIR
	or or $TMPDIR or $TEMP or /run/user/$UID.
	example: /run/user/1001

> tried mounting the borgmatic runtime directory In my case this could be achieved by adding `-v /tmp/borgmatic:/tmp/borgmatic` to the `docker run` command and setting `user_runtime_directory: /tmp/borgmatic`. I think this would work fine, but without it I can't extract the directory from the container. The configuration reference mentions `borgmatic_runtime_directory` only in one place. https://projects.torsion.org/borgmatic-collective/borgmatic/src/commit/7551810ea60e43972d58fa725d765b08d84e590f/borgmatic/config/schema.yaml#L203-L218

witten commented 2025-01-25 16:11:51 +00:00

Owner

Copy link

I'm glad to hear it's working for you with a volume mount in place. The configuration file reference is unfortunately not exhaustive. It documents the individual options but not every use case of borgmatic. For that, the rest of the docs are needed. (Tangentially related: #942.) Please let me know though if there are any clarifications I can make to the docs so that this is easier for future users. Thanks!

I'm glad to hear it's working for you with a volume mount in place. The configuration file reference is unfortunately not exhaustive. It documents the individual options but not every use case of borgmatic. For that, the rest of the docs are needed. (Tangentially related: #942.) Please let me know though if there are any clarifications I can make to the docs so that this is easier for future users. Thanks!

vitoyucepi commented 2025-01-25 16:35:03 +00:00

Author

Copy link

if there are any clarifications I can make to the docs

I'd say that 7551810e should explicitly mention that mounting the directory should be done when dumping the database in directory format. I have not experienced any problems when the format is custom.

> if there are any clarifications I can make to the docs I'd say that 7551810e should explicitly mention that mounting the directory should be done when dumping the database in directory format. I have not experienced any problems when the format is custom.

witten referenced this issue from a commit 2025-01-25 17:17:48 +00:00

Further database container dump documentation clarifications (#978).

witten commented 2025-01-25 17:18:53 +00:00

Owner

Copy link

Good point! I forgot that non-directory-format dumps get conveniently exfiltrated from the database container via shell redirection. This is hopefully addressed now.

Good point! I forgot that non-directory-format dumps get conveniently exfiltrated from the database container via shell redirection. This is [hopefully addressed](https://projects.torsion.org/borgmatic-collective/borgmatic/commit/02451a8b303508c8cfbde8f60163ecf22c8188da) now.

witten closed this issue 2025-01-25 17:18:53 +00:00

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

parallel-borgmatic

command-hook-steps

2.1.7

2.1.6

2.1.5

2.1.4

2.1.3

2.1.2

2.1.1

2.1.0

2.0.13

2.0.12

2.0.11

2.0.10

2.0.9

2.0.8

2.0.7

2.0.6

2.0.5

2.0.4

2.0.3

2.0.2

2.0.1

2.0.0

1.9.14

1.9.13

1.9.12

1.9.11

1.9.10

1.9.9

1.9.8

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.14

1.8.13

1.8.12

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels   

blocked

  

breaking

  

bug

  

data loss

  

design finalized

  

good first issue

  

new feature area

  

question / support

  

security

  

waiting for response

No labels

blocked

breaking

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

Milestone

Clear milestone

No items

No milestone

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

borgmatic-collective/borgmatic#978

No description provided.