Commands working on terminal are not working in borgmatic hooks #1157

New Issue

elhananjair · 2025-10-03T18:39:46Z

elhananjair commented

2025-10-03 18:39:46 +00:00

What I'm trying to do and why

Hello there, I am trying to run some commands before the backup starts, and I have used the same command which is perfectly working when running on the terminal, but when I start the Borgmatic.service file, I am getting an error with that specific command. This is the command

podman exec -ti liyuerp_backend_1 /bin/bash -c "bench --site erp.liyumfi.net backup --with-files --compress --backup-path /home/frappe/frappe-bench/backups" && podman cp liyuerp_backend_1:/home/frappe/frappe-bench/backups/ /home/admin/erp_backup/

The command above makes sure to create a backup of the ERP I am running and then copies the backup files from the container to the host path. I have added the erp_backup path in one of the paths to be backed up. I tried creating a shell file containing the same command and put sudo /etc/borgmatic/pre_back.sh in place of the whole command. But still receiving the same issue

Steps to reproduce

Add the following in config.yaml to execute before back up starts,

commands:
    # Name for the point in borgmatic's execution that
    # the commands should be run before (required if
    # "after" isn't set):
    #  * "action" runs before each action for each
    # repository.
    #  * "repository" runs before all actions for each
    # repository.
    #  * "configuration" runs before all actions and
    # repositories in the current configuration file.
    #  * "everything" runs before all configuration
    # files.
    - before: action

      # Name for the point in borgmatic's execution that
      # the commands should be run after (required if
      # "before" isn't set):
      #  * "action" runs after each action for each
      # repository.
      #  * "repository" runs after all actions for each
      # repository.
      #  * "configuration" runs after all actions and
      # repositories in the current configuration file.
      #  * "everything" runs after all configuration
      # files.
      #  * "error" runs after an error occurs.
      # after: action

      # Only trigger the hook when borgmatic is run with
      # particular actions listed here. Defaults to
      # running for all actions.
      when:
          - create
          # - prune
          # - compact
          # - check

      # List of one or more shell commands or scripts to
      # run when this command hook is triggered. Required.
      run:
          - echo "Stoping DMS for backup..."
          - echo "Staring ERP backup process and copying into host back up folder..."
          - sudo /etc/borgmatic/pre_backup.sh
          - systemctl stop cells.service

Actual behavior

I am getting an error

 CRITICAL An error occurred
CRITICAL Error running actions for repository
CRITICAL Error running before action hook: Command 'sudo /etc/borgmatic/pre_backup.sh' returned non-zero exit status 1.
Critical Error running configuration
CRITICAL
CRITICAL Need some help? https://torsion.org/borgmatic/#issues
 /usr/bin/borgmatic failed with exit status 1.

Expected behavior

Just work as it worked on terminal

Other notes / implementation ideas

No response

borgmatic version

2.0.6

borgmatic installation method

Official Fedora Repository

Borg version

1.4.1

Python version

3.13.7

Database version (if applicable)

No response

Operating system and version

Fedora Linux 42 (WSL) x86_64

### What I'm trying to do and why Hello there, I am trying to run some commands before the backup starts, and I have used the same command which is perfectly working when running on the terminal, but when I start the Borgmatic.service file, I am getting an error with that specific command. This is the command ``` podman exec -ti liyuerp_backend_1 /bin/bash -c "bench --site erp.liyumfi.net backup --with-files --compress --backup-path /home/frappe/frappe-bench/backups" && podman cp liyuerp_backend_1:/home/frappe/frappe-bench/backups/ /home/admin/erp_backup/ ``` The command above makes sure to create a backup of the ERP I am running and then copies the backup files from the container to the host path. I have added the erp_backup path in one of the paths to be backed up. I tried creating a shell file containing the same command and put `sudo /etc/borgmatic/pre_back.sh` in place of the whole command. But still receiving the same issue ### Steps to reproduce Add the following in config.yaml to execute before back up starts, ``` commands: # Name for the point in borgmatic's execution that # the commands should be run before (required if # "after" isn't set): # * "action" runs before each action for each # repository. # * "repository" runs before all actions for each # repository. # * "configuration" runs before all actions and # repositories in the current configuration file. # * "everything" runs before all configuration # files. - before: action # Name for the point in borgmatic's execution that # the commands should be run after (required if # "before" isn't set): # * "action" runs after each action for each # repository. # * "repository" runs after all actions for each # repository. # * "configuration" runs after all actions and # repositories in the current configuration file. # * "everything" runs after all configuration # files. # * "error" runs after an error occurs. # after: action # Only trigger the hook when borgmatic is run with # particular actions listed here. Defaults to # running for all actions. when: - create # - prune # - compact # - check # List of one or more shell commands or scripts to # run when this command hook is triggered. Required. run: - echo "Stoping DMS for backup..." - echo "Staring ERP backup process and copying into host back up folder..." - sudo /etc/borgmatic/pre_backup.sh - systemctl stop cells.service ``` ### Actual behavior I am getting an error ``` CRITICAL An error occurred CRITICAL Error running actions for repository CRITICAL Error running before action hook: Command 'sudo /etc/borgmatic/pre_backup.sh' returned non-zero exit status 1. Critical Error running configuration CRITICAL CRITICAL Need some help? https://torsion.org/borgmatic/#issues /usr/bin/borgmatic failed with exit status 1. ``` ### Expected behavior Just work as it worked on terminal ### Other notes / implementation ideas _No response_ ### borgmatic version 2.0.6 ### borgmatic installation method Official Fedora Repository ### Borg version 1.4.1 ### Python version 3.13.7 ### Database version (if applicable) _No response_ ### Operating system and version Fedora Linux 42 (WSL) x86_64

witten commented

2025-10-03 20:00:51 +00:00

A few thoughts on this:

Is there anything above that error message in the borgmatic output that might be relevant here?
Are you running borgmatic with --verbosity 2?
Is your borgmatic systemd service file running as the root user?
Do you have any of the security-related options enabled in the systemd service file that might be preventing podman from running? If so, you can start by commenting them out and reloading the service to see if that fixes it. If it does, you know at least one of them is causing the problem.
Does borgmatic itself work when run manually through the terminal? Or is it also failing there?

A few thoughts on this: * Is there anything above that error message in the borgmatic output that might be relevant here? * Are you running borgmatic with `--verbosity 2`? * Is your borgmatic systemd service file running as the root user? * Do you have any of the security-related options enabled in the systemd service file that might be preventing `podman` from running? If so, you can start by commenting them out and reloading the service to see if that fixes it. If it does, you know at least one of them is causing the problem. * Does borgmatic itself work when run manually through the terminal? Or is it also failing there?

elhananjair commented

2025-10-03 20:17:04 +00:00

Hello @witten thanks again,

Is there anything above that error message in the borgmatic output that might be relevant here?

Nothing else, journalctl -xeu borgmatic.service is not outputting anything.

Are you running borgmatic with --verbosity 2?

I have used the default borgmatic.service and I didn't change anything, but it seems using that # ExecStart=systemd-inhibit --who="borgmatic" --what="sleep:shutdown" --why="Prevent interrupting scheduled backup" /usr/bin/borgmatic --verbosity -2 --syslog-verbosity 1

Is your borgmatic systemd service file running as the root user?

I haven't specified username and group on borgmatic.service, I am using the default file

Do you have any of the security-related options enabled in the systemd service file that might be preventing podman from running? If so, you can start by commenting them out and reloading the service to see if that fixes it. If it does, you know at least one of them is causing the problem.

I was checking with Gemeni, and it suggested that using -ti on the podman exec command might cause a problem, as it requires an interactive shell.

Does borgmatic itself work when run manually through the terminal? Or is it also failing there?

It shows the error when I tried "borgmatic create --list --stat" on terminal

Hello @witten thanks again, > Is there anything above that error message in the borgmatic output that might be relevant here? Nothing else, journalctl -xeu borgmatic.service is not outputting anything. > Are you running borgmatic with --verbosity 2? I have used the default borgmatic.service and I didn't change anything, but it seems using that `# ExecStart=systemd-inhibit --who="borgmatic" --what="sleep:shutdown" --why="Prevent interrupting scheduled backup" /usr/bin/borgmatic --verbosity -2 --syslog-verbosity 1` > Is your borgmatic systemd service file running as the root user? I haven't specified username and group on borgmatic.service, I am using the default file > Do you have any of the security-related options enabled in the systemd service file that might be preventing podman from running? If so, you can start by commenting them out and reloading the service to see if that fixes it. If it does, you know at least one of them is causing the problem. I was checking with Gemeni, and it suggested that using -ti on the podman exec command might cause a problem, as it requires an interactive shell. > Does borgmatic itself work when run manually through the terminal? Or is it also failing there? It shows the error when I tried "borgmatic create --list --stat" on terminal

witten commented

2025-10-03 20:50:47 +00:00

I have used the default borgmatic.service and I didn't change anything, but it seems using that # ExecStart=systemd-inhibit --who="borgmatic" --what="sleep:shutdown" --why="Prevent interrupting scheduled backup" /usr/bin/borgmatic --verbosity -2 --syslog-verbosity 1

Try changing --syslog-verbosity to 2. That should give you some more verbose output, viewable with journalctl.

Do you have any of the security-related options enabled in the systemd service file that might be preventing podman from running? If so, you can start by commenting them out and reloading the service to see if that fixes it. If it does, you know at least one of them is causing the problem.

I was checking with Gemeni, and it suggested that using -ti on the podman exec command might cause a problem, as it requires an interactive shell.

That's a separate issue, but still possibly valid. You could try dropping the -ti in the podman exec command.

What I was talking about is all of the security options in the default borgmatic system service file. Everything from LockPersonality=true through CapabilityBoundingSet=.... You could try commenting those out and running systemctl daemon-reload to pick up the changes.

However, reading on, it sounds like the problem still occurs when you run borgmatic manually, so that probably indicates that the problem isn't something introduced by the systemd service file.

Does borgmatic itself work when run manually through the terminal? Or is it also failing there?

It shows the error when I tried "borgmatic create --list --stat" on terminal

Okay, in that case I'd recommend:

Get borgmatic working at the terminal before bothering with systemd.
Try removing -ti from the podman exec command.
Add --verbosity 2 to your borgmatic create command to see if that gives a more useful error.

> I have used the default borgmatic.service and I didn't change anything, but it seems using that `# ExecStart=systemd-inhibit --who="borgmatic" --what="sleep:shutdown" --why="Prevent interrupting scheduled backup" /usr/bin/borgmatic --verbosity -2 --syslog-verbosity 1` Try changing `--syslog-verbosity` to `2`. That should give you some more verbose output, viewable with `journalctl`. > > Do you have any of the security-related options enabled in the systemd service file that might be preventing podman from running? If so, you can start by commenting them out and reloading the service to see if that fixes it. If it does, you know at least one of them is causing the problem. > > I was checking with Gemeni, and it suggested that using -ti on the podman exec command might cause a problem, as it requires an interactive shell. That's a separate issue, but still possibly valid. You could try dropping the `-ti` in the `podman exec` command. ~~What I was talking about is all of the security options in [the default borgmatic system service file](https://projects.torsion.org/borgmatic-collective/borgmatic/raw/branch/main/sample/systemd/borgmatic.service). Everything from `LockPersonality=true` through `CapabilityBoundingSet=...`. You could try commenting those out and running `systemctl daemon-reload` to pick up the changes.~~ However, reading on, it sounds like the problem still occurs when you run borgmatic manually, so that probably indicates that the problem isn't something introduced by the systemd service file. > > Does borgmatic itself work when run manually through the terminal? Or is it also failing there? > > It shows the error when I tried "borgmatic create --list --stat" on terminal Okay, in that case I'd recommend: * Get borgmatic working at the terminal before bothering with systemd. * Try removing `-ti` from the `podman exec` command. * Add `--verbosity 2` to your `borgmatic create` command to see if that gives a more useful error.

witten added the question / support label 2025-10-03 20:50:54 +00:00

elhananjair commented

2025-10-04 09:02:04 +00:00

As you suggested, I am running sudo borgmatic create --list --stat --verbosity -2 from the terminal before starting the systemd service, and there is no output when I run the above command.

I removed -ti from podman exec, edited the .sh file, but as I said, there is no output trying from the terminal.

I added --verbosity 2 while running the borgmaic create, and I have created logfile.txt, now after removing -ti I get this in logs "Error: no container with name or ID "liyuerp_backend_1" found: no such container" which indicates the process is running as root, I guess, since I am running rootless podman. Since I am running Borgmatic as root, is that maybe an issue?

As you suggested, I am running `sudo borgmatic create --list --stat --verbosity -2 ` from the terminal before starting the systemd service, and there is no output when I run the above command. I removed -ti from podman exec, edited the .sh file, but as I said, there is no output trying from the terminal. I added --verbosity 2 while running the borgmaic create, and I have created logfile.txt, now after removing -ti I get this in logs "Error: no container with name or ID "liyuerp_backend_1" found: no such container" which indicates the process is running as root, I guess, since I am running rootless podman. Since I am running Borgmatic as root, is that maybe an issue?

elhananjair commented

2025-10-04 09:52:39 +00:00

Ok After checking on AI, it seems working, now I am dealing with ssh but that's another problem, as I said the problem with the user who executing the podman exec command, since borgmatic is running with root and podman container is owned by non root user that was the reason...

#!/bin/bash

# --- Configuration Variables ---
CONTAINER_USER="liyuadmin"
CONTAINER_NAME="liyuerp_backend_1"
BACKUP_COMMAND='bench --site erp.liyumfi.net backup --with-files --compress --backup-path /home/frappe/frappe-bench/backups'
CONTAINER_PATH="/home/frappe/frappe-bench/backups/"
HOST_PATH="/home/liyuadmin/erp_backup/"

# --- Step 1: Execute the backup command inside the container ---
# Note: The command is wrapped in single quotes '...' to prevent host shell expansion.
echo "Starting container backup as user $CONTAINER_USER..."
su -c "/usr/bin/podman exec -i $CONTAINER_NAME /bin/bash -c \"$BACKUP_COMMAND\"" "$CONTAINER_USER"

# Check the exit status of the backup command
BACKUP_EXIT_CODE=$?

if [ $BACKUP_EXIT_CODE -ne 0 ]; then
    echo "ERROR: Internal container backup failed with exit code $BACKUP_EXIT_CODE"
    exit 1  # Exit the script to stop borgmatic
fi

echo "Backup successful. Copying files to host..."

# --- Step 2: Copy the backups from the container to the host ---
# Note: podman cp command must also run as the container owner!
# The -i flag is unnecessary for exec and is not used here.

su -c "/usr/bin/podman cp $CONTAINER_NAME:$CONTAINER_PATH $HOST_PATH" "$CONTAINER_USER"

# Check the exit status of the copy command
CP_EXIT_CODE=$?

if [ $CP_EXIT_CODE -eq 0 ]; then
    echo "Copy operation successful!"
else
    echo "ERROR: podman cp failed with exit code $CP_EXIT_CODE"
    exit 1 # Exit the script to stop borgmatic
fi

Ok After checking on AI, it seems working, now I am dealing with ssh but that's another problem, as I said the problem with the user who executing the podman exec command, since borgmatic is running with root and podman container is owned by non root user that was the reason... ``` #!/bin/bash # --- Configuration Variables --- CONTAINER_USER="liyuadmin" CONTAINER_NAME="liyuerp_backend_1" BACKUP_COMMAND='bench --site erp.liyumfi.net backup --with-files --compress --backup-path /home/frappe/frappe-bench/backups' CONTAINER_PATH="/home/frappe/frappe-bench/backups/" HOST_PATH="/home/liyuadmin/erp_backup/" # --- Step 1: Execute the backup command inside the container --- # Note: The command is wrapped in single quotes '...' to prevent host shell expansion. echo "Starting container backup as user $CONTAINER_USER..." su -c "/usr/bin/podman exec -i $CONTAINER_NAME /bin/bash -c \"$BACKUP_COMMAND\"" "$CONTAINER_USER" # Check the exit status of the backup command BACKUP_EXIT_CODE=$? if [ $BACKUP_EXIT_CODE -ne 0 ]; then echo "ERROR: Internal container backup failed with exit code $BACKUP_EXIT_CODE" exit 1 # Exit the script to stop borgmatic fi echo "Backup successful. Copying files to host..." # --- Step 2: Copy the backups from the container to the host --- # Note: podman cp command must also run as the container owner! # The -i flag is unnecessary for exec and is not used here. su -c "/usr/bin/podman cp $CONTAINER_NAME:$CONTAINER_PATH $HOST_PATH" "$CONTAINER_USER" # Check the exit status of the copy command CP_EXIT_CODE=$? if [ $CP_EXIT_CODE -eq 0 ]; then echo "Copy operation successful!" else echo "ERROR: podman cp failed with exit code $CP_EXIT_CODE" exit 1 # Exit the script to stop borgmatic fi ```

witten commented

2025-10-04 15:10:41 +00:00

Ah, glad to hear you've got it figured out now!

witten closed this issue

2025-10-04 15:10:41 +00:00

Sign in to join this conversation.

Branches Tags

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#1157