Backup of Database in Docker Container fails when using backup tool within that container and following docs #906

New Issue

Closed

opened 2024-08-15 14:05:56 +00:00 by JonnyBnator · 5 comments

JonnyBnator commented 2024-08-15 14:05:56 +00:00

Contributor

Copy Link

What I'm trying to do and why

I am trying to backup a Database (MariaDB, although I assume other DBs are affected too) within a Docker Container.
Borgmatic is not installed in a Container, but on the host instead.
I also do not have the database backup tool (mariadb-dump) installed on the host, so I am using the one provided inside the Container.

So I followed the docs (specifically this part), to setup the backup.

It is failing however, because the command executed by borgmatic for the db backup includes the --result-file option. Since with this configuration the mariadb-dump tool gets executed inside the container, it tries to output to the result file inside the container, which does not exist.
(/home/USERNAME/.borgmatic/mariadb_databases/localhost/DATABASE_NAME in my case)

After playing around a bit and figuring out that it was because the file by borgmatic to pipe the db backup into was created on the host, but the actual mariadb backup tool tried to put the content into that file inside the container, I was able to solve that issue by mounting the .borgmatic directory inside the container with read and write permissions.

I am not sure however, if this has any security implications and if there is a better way to achieve the result without mounting.
For example using the mariadb-dump command with a shell pipe (> OUTPUT file) on the host system worked fine instead of the --result-file config.
The --result-file option is hardcoded though.

Steps to reproduce

1. Install Docker Container with MariaDB
2. Install Borg and Borgmatic on Host
3. Try to backup MariaDB inside Docker Container with mariadb-dump tool inside Container (docker exec NAME mariadb-dump [...])

relevant part of docker compose file:

services:
  db:
    image: mariadb:10.6
    container_name: CONTAINER_NAME
    restart: always
    volumes:
      - ./data:/var/lib/mysql
      - ./ssl:/etc/mysql/ssl:ro
    network_mode: "host"

relevant part of borgmatic config file:

mariadb_databases:
    - name: all

      # Command to use instead of "mariadb-dump". This can be
      # used to run a specific mariadb_dump version (e.g., one
      # inside a running container). Defaults to "mariadb-dump".
      mariadb_dump_command: docker exec CONTAINER_NAME mariadb-dump

      # Command to run instead of "mariadb". This can be used to
      # run a specific mariadb version (e.g., one inside a
      # running container). Defaults to "mariadb".
      mariadb_command: docker exec CONTAINER_NAME mariadb

      # Database dump output format. Currently only "sql" is
      # supported. Defaults to "sql" for a single database. Or,
      # when database name is "all" and format is blank, dumps
      # all databases to a single file. But if a format is
      # specified with an "all" database name, dumps each
      # database to a separate file of that format, allowing
      # more convenient restores of individual databases.
      format: sql

Actual behavior

Borgmatic fails to create database backup

mariadb-dump: Can't create/write to file '/home/USERNAME/.borgmatic/mariadb_databases/localhost/DATABASE_NAME' (Errcode: 2 "No such file or directory")
backup_nas: Error running actions for repository
Command 'docker exec CONTAINER_NAME mariadb-dump --databases DATABASE_NAME --result-file /home/USERNAME/.borgmatic/mariadb_databases/localhost/DATABASE_NAME' returned non-zero exit status 1.

Expected behavior

Since borgmatic creates the file on the host system, the db backup content should be put inside the same file.

Other notes / implementation ideas

As described earlier, if I include the following volume mount in the Docker compose file for the DB, it works.

    volumes:
      [...]
      - /home/USERNAME/.borgmatic:/home/USERNAME/.borgmatic

If there are no security implications to mounting the .borgmatic folder inside the Docker Container, I would add this to the docs.
(Happy to create a PR for it myself, if this is the chosen solution)

borgmatic version

1.8.13

borgmatic installation method

pip install

Borg version

borg 1.2.4

Python version

Python 3.11.2

Database version (if applicable)

mariadb Ver 15.1 Distrib 10.6.19-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2

Operating system and version

PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" NAME="Debian GNU/Linux" VERSION_ID="12" VERSION="12 (bookworm)" VERSION_CODENAME=bookworm ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"

### What I'm trying to do and why I am trying to backup a Database (MariaDB, although I assume other DBs are affected too) within a Docker Container. Borgmatic is not installed in a Container, but on the host instead. I also do not have the database backup tool (`mariadb-dump`) installed on the host, so I am using the one provided inside the Container. So I followed the docs *(specifically this [part](https://projects.torsion.org/borgmatic-collective/borgmatic/src/commit/548aceb3d5d94622806c4bcdd3433412a998cb6c/docs/how-to/backup-your-databases.md?display=source#L209-L237))*, to setup the backup. It is failing however, because the command executed by borgmatic for the db backup includes the `--result-file` option. Since with this configuration the `mariadb-dump` tool gets executed **inside** the container, it tries to output to the result file **inside** the container, which does not exist. *(`/home/USERNAME/.borgmatic/mariadb_databases/localhost/DATABASE_NAME` in my case)* After playing around a bit and figuring out that it was because the file by borgmatic to pipe the db backup into was created on the host, but the actual mariadb backup tool tried to put the content into that file inside the container, I was able to solve that issue by mounting the .borgmatic directory inside the container with read and write permissions. I am not sure however, if this has any security implications and if there is a better way to achieve the result without mounting. For example using the mariadb-dump command with a shell pipe (`> OUTPUT file`) on the host system worked fine instead of the --result-file config. The --result-file option is hardcoded though. ### Steps to reproduce 1. Install Docker Container with MariaDB 2. Install Borg and Borgmatic on Host 3. Try to backup MariaDB inside Docker Container with `mariadb-dump` tool inside Container (`docker exec NAME mariadb-dump [...]`) relevant part of docker compose file: ```yaml services: db: image: mariadb:10.6 container_name: CONTAINER_NAME restart: always volumes: - ./data:/var/lib/mysql - ./ssl:/etc/mysql/ssl:ro network_mode: "host" ``` relevant part of borgmatic config file: ```yaml mariadb_databases: - name: all # Command to use instead of "mariadb-dump". This can be # used to run a specific mariadb_dump version (e.g., one # inside a running container). Defaults to "mariadb-dump". mariadb_dump_command: docker exec CONTAINER_NAME mariadb-dump # Command to run instead of "mariadb". This can be used to # run a specific mariadb version (e.g., one inside a # running container). Defaults to "mariadb". mariadb_command: docker exec CONTAINER_NAME mariadb # Database dump output format. Currently only "sql" is # supported. Defaults to "sql" for a single database. Or, # when database name is "all" and format is blank, dumps # all databases to a single file. But if a format is # specified with an "all" database name, dumps each # database to a separate file of that format, allowing # more convenient restores of individual databases. format: sql ``` ### Actual behavior Borgmatic fails to create database backup ``` mariadb-dump: Can't create/write to file '/home/USERNAME/.borgmatic/mariadb_databases/localhost/DATABASE_NAME' (Errcode: 2 "No such file or directory") backup_nas: Error running actions for repository Command 'docker exec CONTAINER_NAME mariadb-dump --databases DATABASE_NAME --result-file /home/USERNAME/.borgmatic/mariadb_databases/localhost/DATABASE_NAME' returned non-zero exit status 1. ``` ### Expected behavior Since borgmatic creates the file on the host system, the db backup content should be put inside the same file. ### Other notes / implementation ideas As described earlier, if I include the following volume mount in the Docker compose file for the DB, it works. ```yaml volumes: [...] - /home/USERNAME/.borgmatic:/home/USERNAME/.borgmatic ``` If there are no security implications to mounting the .borgmatic folder inside the Docker Container, I would add this to the docs. (Happy to create a PR for it myself, if this is the chosen solution) ### borgmatic version 1.8.13 ### borgmatic installation method pip install ### Borg version borg 1.2.4 ### Python version Python 3.11.2 ### Database version (if applicable) mariadb Ver 15.1 Distrib 10.6.19-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2 ### Operating system and version PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" NAME="Debian GNU/Linux" VERSION_ID="12" VERSION="12 (bookworm)" VERSION_CODENAME=bookworm ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"

witten commented 2024-08-15 15:59:32 +00:00

Owner

Copy Link

Good catch, and thank you for the detailed ticket! I think your analysis is completely correct here.

For example using the mariadb-dump command with a shell pipe (> OUTPUT file) on the host system worked fine instead of the --result-file config.

That makes sense, but it wouldn't work when running mariadb-dump inside of the container, right? Because then the shell pipe would still go to a file inside the container.

I think the ~/.borgmatic volume mount is probably the right solution here. There are some security implications, namely that a database container would have read-write access to: 1. database dump temporary files (which is what we want) and 2. records of when borgmatic check last ran (not a big deal). So I think the security risks are low and the worst an attacker could do is take control of the database container and prevent database backups from working and maybe prevent borgmatic checks from running as frequently. If you're okay with that, I'm okay with that.

A PR would be welcome! Besides adding the "fix" in the docs, it might be good to at least mention it in the configuration schema.

Good catch, and thank you for the detailed ticket! I think your analysis is completely correct here. > For example using the mariadb-dump command with a shell pipe (> OUTPUT file) on the host system worked fine instead of the --result-file config. That makes sense, but it wouldn't work when running `mariadb-dump` inside of the container, right? Because then the shell pipe would still go to a file inside the container. I think the `~/.borgmatic` volume mount is probably the right solution here. There _are_ some security implications, namely that a database container would have read-write access to: 1. database dump temporary files (which is what we want) and 2. records of when borgmatic check last ran (not a big deal). So I think the security risks are low and the worst an attacker could do is take control of the database container and prevent database backups from working and _maybe_ prevent borgmatic checks from running as frequently. If you're okay with that, I'm okay with that. A PR would be welcome! Besides adding the "fix" in the docs, it might be good to at least mention it in the configuration schema.

witten added the

bug

label 2024-08-15 16:01:06 +00:00

JonnyBnator commented 2024-08-16 11:36:56 +00:00

Author

Contributor

Copy Link

That makes sense, but it wouldn't work when running mariadb-dump inside of the container, right? Because then the shell pipe would still go to a file inside the container.

True, didn't think about that.

I think the ~/.borgmatic volume mount is probably the right solution here. There are some security implications, namely that a database container would have read-write access to: 1. database dump temporary files (which is what we want) and 2. records of when borgmatic check last ran (not a big deal). So I think the security risks are low and the worst an attacker could do is take control of the database container and prevent database backups from working and maybe prevent borgmatic checks from running as frequently. If you're okay with that, I'm okay with that.

I am okay with that.

> That makes sense, but it wouldn't work when running `mariadb-dump` inside of the container, right? Because then the shell pipe would still go to a file inside the container. True, didn't think about that. > I think the `~/.borgmatic` volume mount is probably the right solution here. There _are_ some security implications, namely that a database container would have read-write access to: 1. database dump temporary files (which is what we want) and 2. records of when borgmatic check last ran (not a big deal). So I think the security risks are low and the worst an attacker could do is take control of the database container and prevent database backups from working and _maybe_ prevent borgmatic checks from running as frequently. If you're okay with that, I'm okay with that. I am okay with that.

JonnyBnator referenced this issue 2024-08-16 12:31:21 +00:00

Documentation and config schema update for #906 #907

dn-achim commented 2024-08-20 07:48:19 +00:00

Copy Link

That makes sense, but it wouldn't work when running mariadb-dump inside of the container, right? Because then the shell pipe would still go to a file inside the container.

Sorry to ask, but isn't that exactly what you do (redirection) in the Postgresql binding?
It works pretty well for me, instead of the volume solution described above.

+ (('>', shlex.quote(dump_filename)) if dump_format != 'directory' else ())

> That makes sense, but it wouldn't work when running mariadb-dump inside of the container, right? Because then the shell pipe would still go to a file inside the container. Sorry to ask, but isn't that exactly what you do (redirection) in the Postgresql binding? It works pretty well for me, instead of the volume solution described above. ` + (('>', shlex.quote(dump_filename)) if dump_format != 'directory' else ())`

witten commented 2024-08-20 21:08:55 +00:00

Owner

Copy Link

That's correct; the PostgreSQL hook does perform the same approach, and it would presumably have the same problem if run in a container without a directory format database dump.

That's correct; the PostgreSQL hook does perform the same approach, and it would presumably have the same problem if run in a container without a directory format database dump.

witten referenced this issue from a commit 2024-08-20 21:51:52 +00:00

Documentation and config schema update about ~/.borgmatic volume mount (#906).

witten referenced this issue from a commit 2024-08-20 22:04:14 +00:00

: Add documentation details for how to run custom database dump commands using binaries from running containers (#906).

witten commented 2024-08-20 22:05:52 +00:00

Owner

Copy Link

Implemented in main—thank you for the PR!—and will be part of the next release. (I tweaked some of the wording hopefully for clarity.)

Implemented in main—thank you for the PR!—and will be part of the next release. (I tweaked some of the wording hopefully for clarity.)

witten closed this issue 2024-08-20 22:05:52 +00:00

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

config-command-line

1.9.14

1.9.13

1.9.12

1.9.11

1.9.10

1.9.9

1.9.8

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.14

1.8.13

1.8.12

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels

blocked

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

No Label

bug

Milestone

No items

No Milestone

Assignees

Clear assignees

diivi import_bot witten

No Assignees

3 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#906

No description provided.