Environment variables interpolation does not work when using before/after hooks #860

New Issue

Closed

opened 2024-04-28 16:24:48 +00:00 by valfur03 · 8 comments

valfur03 commented 2024-04-28 16:24:48 +00:00

Copy Link

What I'm trying to do and why

I want to backup a database while performing some actions using the before_backup hook.

Steps to reproduce

1. Setup the database connection in mariadb_databases. Here is my configuration:

   # /etc/borgmatic.d/database.yaml
   <<: !include
       - hooks/before.yaml
   
   constants:
       mariadb_name: name
       mariadb_host: host
       mariadb_user: user
       mariadb_pass: ${MARIADB_PASSWORD}
   
   mariadb_databases:
       !include sources/mariadb-database.yaml
   
   repositories:
       !include repositories/all.yaml
   
   # /etc/borgmatic.d/hooks/before.yaml
   before_backup: []
   
   # sources/mariadb-database.yaml
   - name: "{mariadb_name}"
     hostname: "{mariadb_host}"
     username: "{mariadb_user}"
     password: "{mariadb_pass}"
   

2. Run a create action (borgmatic --stats --list --verbosity 1)

Actual behavior

Borgmatic exits with error code 1 and prints this:

/etc/borgmatic.d/database.yaml: An error occurred                            
local: Error running actions for repository                                            
mariadb-dump: Got error: 1045: "Access denied for user 'user'@'192.168.16.6' (using password: YES)" when trying to connect
Command 'mariadb-dump --add-drop-database --host host --protocol tcp --user user --databases name --result-file /root/.borgmatic/mariadb_databases/database/name returned non-zero exit status 2.                                            

Expected behavior

It should perform the backup successfully.

Other notes / implementation ideas

I tried hardcoding the password constant:

mariadb_pass: mydbpass

This way it works fine. That is why my guess is that environment variable iterpolation stops working when before/after hooks are set, even empty.

I had no idea what to try to be sure about that.

Note that environment variables exist and are set if I try to echo them in before_backup hook.

borgmatic version

1.8.10

borgmatic installation method

Docker container

Borg version

1.2.8

Python version

3.12.3

Database version (if applicable)

mariadb v11.2

Operating system and version

Alpine Linux 3.19.1

### What I'm trying to do and why I want to backup a database while performing some actions using the `before_backup` hook. ### Steps to reproduce 1. Setup the database connection in `mariadb_databases`. Here is my configuration: ```yaml # /etc/borgmatic.d/database.yaml <<: !include - hooks/before.yaml constants: mariadb_name: name mariadb_host: host mariadb_user: user mariadb_pass: ${MARIADB_PASSWORD} mariadb_databases: !include sources/mariadb-database.yaml repositories: !include repositories/all.yaml # /etc/borgmatic.d/hooks/before.yaml before_backup: [] # sources/mariadb-database.yaml - name: "{mariadb_name}" hostname: "{mariadb_host}" username: "{mariadb_user}" password: "{mariadb_pass}" ``` 2. Run a create action (`borgmatic --stats --list --verbosity 1`) ### Actual behavior Borgmatic exits with error code 1 and prints this: ``` /etc/borgmatic.d/database.yaml: An error occurred local: Error running actions for repository mariadb-dump: Got error: 1045: "Access denied for user 'user'@'192.168.16.6' (using password: YES)" when trying to connect Command 'mariadb-dump --add-drop-database --host host --protocol tcp --user user --databases name --result-file /root/.borgmatic/mariadb_databases/database/name returned non-zero exit status 2. ``` ### Expected behavior It should perform the backup successfully. ### Other notes / implementation ideas I tried hardcoding the password constant: ```yaml mariadb_pass: mydbpass ``` This way it works fine. That is why my guess is that environment variable iterpolation stops working when before/after hooks are set, even empty. I had no idea what to try to be sure about that. Note that environment variables exist and are set if I try to `echo` them in `before_backup` hook. ### borgmatic version 1.8.10 ### borgmatic installation method Docker container ### Borg version 1.2.8 ### Python version 3.12.3 ### Database version (if applicable) mariadb v11.2 ### Operating system and version Alpine Linux 3.19.1

witten commented 2024-04-28 22:34:30 +00:00

Owner

Copy Link

Thanks for filing this! Unfortunately I haven't been able to reproduce it thus far on my system. I've tried a very similar configuration: A constant consuming an environment variable, and then using that constant in included database configuration. It all works as expected here.

So a couple of ideas on how to proceed:

• Have you double checked that the password as set from the environment variable is correct?
• Is it possible that the password contains a special character that's tripping up your shell (or borgmatic), a special character doesn't cause problems when hard-coded in the constant? If you think that might be a problem, can you try a simpler password temporarily to see if that solves it?
• Maybe try simpler variants of the configuration, like using the environment variable directly without an intermediate constant.. or using the environment variable and constant but cutting out includes.

Let me know what you find out!

Thanks for filing this! Unfortunately I haven't been able to reproduce it thus far on my system. I've tried a very similar configuration: A constant consuming an environment variable, and then using that constant in included database configuration. It all works as expected here. So a couple of ideas on how to proceed: * Have you double checked that the password as set from the environment variable is correct? * Is it possible that the password contains a special character that's tripping up your shell (or borgmatic), a special character doesn't cause problems when hard-coded in the constant? If you think that might be a problem, can you try a simpler password temporarily to see if that solves it? * Maybe try simpler variants of the configuration, like using the environment variable directly without an intermediate constant.. or using the environment variable and constant but cutting out includes. Let me know what you find out!

valfur03 commented 2024-04-29 18:19:40 +00:00

Author

Copy Link

Thanks for your answer. Unfortunately, none of the solutions worked on my part.

I have provided you a sample repository for you to try to reproduce this error: https://github.com/valfur03/borgmatic-env-issue-860

Some additional notes, I have tested on both amd64 and armv8 architectures.

Thanks again for your help!

Thanks for your answer. Unfortunately, none of the solutions worked on my part. I have provided you a sample repository for you to try to reproduce this error: https://github.com/valfur03/borgmatic-env-issue-860 Some additional notes, I have tested on both amd64 and armv8 architectures. Thanks again for your help!

witten commented 2024-04-30 16:18:35 +00:00

Owner

Copy Link

Thank you so much for setting up a sample repro! I have indeed managed to reproduce the issue with it, and it looks like what's going on is that borgmatic's constants code is very intentionally shell escaping all constant values, in theory to prevent shell injection attacks. Unfortunately that also prevents environment variable substitution within constant values, which is exactly what you're trying to do here. I'll have to think about what can be done about this.

Thank you so much for setting up a sample repro! I have indeed managed to reproduce the issue with it, and it looks like what's going on is that borgmatic's constants code is very intentionally shell escaping all constant values, in theory to prevent shell injection attacks. Unfortunately that also prevents environment variable substitution within constant values, which is exactly what you're trying to do here. I'll have to think about what can be done about this.

witten commented 2024-04-30 16:25:27 +00:00

Owner

Copy Link

Update! I have a fix in hand.. I just need to write tests for it. It turned out that there was already code to selectively apply the shell escaping logic, and it wasn't supposed to apply to constants used in database passwords, but it was being applied anyway. I'll let you know once this fix is pushed.

Update! I have a fix in hand.. I just need to write tests for it. It turned out that there was already code to selectively apply the shell escaping logic, and it wasn't _supposed_ to apply to constants used in database passwords, but it was being applied anyway. I'll let you know once this fix is pushed.

witten referenced this issue from a commit 2024-04-30 16:37:17 +00:00

Fix interaction between environment variable interpolation in constants and shell escaping (#860).

witten commented 2024-04-30 16:38:41 +00:00

Owner

Copy Link

Okay, the fix is pushed and will be part of the next release. Thanks again!

Okay, the fix is pushed and will be part of the next release. Thanks again!

witten closed this issue 2024-04-30 16:38:41 +00:00

witten added the

bug

label 2024-04-30 16:38:46 +00:00

valfur03 commented 2024-04-30 17:55:34 +00:00

Author

Copy Link

I have tested locally and everything works fine!

Thank you so much for your help!

I have tested locally and everything works fine! Thank you so much for your help!

witten commented 2024-04-30 18:04:43 +00:00

Owner

Copy Link

Awesome, glad to hear it!

Awesome, glad to hear it!

witten commented 2024-06-24 00:05:26 +00:00

Owner

Copy Link

Released in borgmatic 1.8.12!

Released in borgmatic 1.8.12!

🎉 1 🚀 1

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

1.8.14

1.8.13

1.8.12

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels   

bug

  

data loss

  

design finalized

  

good first issue

  

new feature area

  

question / support

  

security

  

waiting for response

No Label

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

Milestone

Clear milestone

No items

No Milestone

Assignees

Clear assignees

No Assignees

2 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#860

No description provided.