[Docs] Clarify execution order of shell commands in same borgmatic step #1099

New issue

Closed

opened 2025-05-24 21:17:32 +00:00 by holocronweaver · 2 comments

holocronweaver commented

2025-05-24 21:17:32 +00:00

Contributor

Problem

What order do multiple command: entries run in when they are for the same borgmatic step?

Example:

commands:
  - after: configuration
    run:
      - ./start_services.sh
  - after: configuration
    run:
      - ./services_remote_backup.py
    states:
      - finish

Note both commands run after: configuration, but the last one only runs when state is finish, whereas the first runs in all states.

Is it guaranteed in this case that start_services.sh will run before services_remote_backup? That is, for a given borgmatic step, do multiple commands entries for that step run in the order specified in the config file?

Background

In my particular case the concern is that services_remote_backup.py is lengthy and slow (uses rclone on my borg repo), and I do not want to delay start_services.sh while the remote backup could take hours. However I also do not want to perform a remote backup if any borg steps fail (i.e., states: [finish]), and I want borgmatic to fail and send a message to healthchecks.io if the remote backup fails (hence running after: configuration and not after: everything). Services should be started again regardless of if borgmatic succeeds (i.e., all states).

Analysis

I suspect they are guaranteed to run in order given in the config because load_configuration relies on ruamel.yaml which claims it:

supports roundtrip preservation of comments, seq/map flow style, and map key order

But I wanted to double check that the maintainers intend to preserve this config-based order behavior in the future before I rely on it.

I suggest clarifying this in the preparation and cleanup steps docs page, https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/

Kudos

Thank you so much for making borgmatic, which has saved my rear on at least a dozen occasions.

## Problem What order do multiple `command:` entries run in when they are for the same borgmatic step? Example: ```yaml commands: - after: configuration run: - ./start_services.sh - after: configuration run: - ./services_remote_backup.py states: - finish ``` Note both commands run `after: configuration`, but the last one only runs when state is `finish`, whereas the first runs in all states. Is it guaranteed in this case that `start_services.sh` will run before `services_remote_backup`? That is, for a given borgmatic step, do multiple `commands` entries for that step run in the order specified in the config file? ## Background In my particular case the concern is that `services_remote_backup.py` is lengthy and slow (uses `rclone` on my borg repo), and I do not want to delay `start_services.sh` while the remote backup could take hours. However I also do not want to perform a remote backup if any borg steps fail (i.e., `states: [finish]`), and I want borgmatic to fail and send a message to healthchecks.io if the remote backup fails (hence running `after: configuration` and not `after: everything`). Services should be started again regardless of if borgmatic succeeds (i.e., all states). ## Analysis I suspect they are guaranteed to run in order given in the config because [`load_configuration`](https://projects.torsion.org/borgmatic-collective/borgmatic/src/commit/b208c5352ad39cf6d4eb87df15175185df29dee7/borgmatic/config/load.py#L165) relies on [ruamel.yaml](https://pypi.org/project/ruamel.yaml/) which claims it: > supports roundtrip preservation of comments, seq/map flow style, and map key order But I wanted to double check that the maintainers intend to preserve this config-based order behavior in the future before I rely on it. I suggest clarifying this in the preparation and cleanup steps docs page, https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/ ## Kudos Thank you so much for making borgmatic, which has saved my rear on at least a dozen occasions.

witten commented

2025-05-25 02:30:53 +00:00

Owner

Great question! Hooks of the same "step" run in the order as defined in the configuration file. The way it works is:

The defined commands: values, once parsed, are filtered down based on the current state. So if the current state being evaluated is finish, then only command hooks implicitly or explicitly matching that state are used. The original ordering is maintained during this filtering step.
The resulting command hooks are run in order. More specifically, each command hook is looked at, and each of its run: commands is executed in turn.

I'll leave this ticket open to document this (intentional) behavior. And I'm glad to hear borgmatic is working out for you!

Great question! Hooks of the same "step" run in the order as defined in the configuration file. The way it works is: 1. The defined `commands:` values, once parsed, are filtered down based on the current state. So if the current state being evaluated is `finish`, then only command hooks implicitly or explicitly matching that state are used. The original ordering is maintained during this filtering step. 2. The resulting command hooks are run in order. More specifically, each command hook is looked at, and each of its `run:` commands is executed in turn. I'll leave this ticket open to document this (intentional) behavior. And I'm glad to hear borgmatic is working out for you!

witten referenced this issue

2025-05-25 02:40:28 +00:00

A command hook with the "finish" state gets run even when "borg --version" fails #1100

witten referenced this issue from a commit

2025-06-02 19:01:53 +00:00

Clarify documentation on command hooks order of execution (#1099).

witten commented

2025-06-02 19:02:01 +00:00

Owner

Added to docs. Should be deployed shortly!

witten closed this issue

2025-06-02 19:02:01 +00:00