Make it possible to overwrite config in command line call #268

New Issue

errnesto · 2019-12-05T01:26:26Z

errnesto commented

2019-12-05 01:26:26 +00:00

I think it would be nice to overwrite some of the options
of the config file with paramters on the command line.

I think this would be a more general solution to issues like #73 and (as i understand it) is what @thomasleveil means by 'using it like docker-compose' in the discussion of that issue.

My use case is that I would like to store the config in a git repository with my code, but the fact that I have to supply a repository makes this not a good solution for me.

One reason is that I just don't like to store that information in git.
But the other is that i have multiple instances of that app on the same server and it would be nice to just create backups for all of them with one simple script.
That script would find all (identical) config files and run borgmatic for each of them and add a repository for each app instance based on some environment variables.

I can see that this could get kind of messy when using multiple config files with a single command though…

I think it would be nice to overwrite some of the options of the config file with paramters on the command line. I think this would be a more general solution to issues like #73 and (as i understand it) is what @thomasleveil means by 'using it like docker-compose' in the discussion of that issue. My use case is that I would like to store the config in a git repository with my code, but the fact that I have to supply a repository makes this not a good solution for me. One reason is that I just don't like to store that information in git. But the other is that i have multiple instances of that app on the same server and it would be nice to just create backups for all of them with one simple script. That script would find all (identical) config files and run borgmatic for each of them and add a repository for each app instance based on some environment variables. I can see that this could get kind of messy when using multiple config files with a single command though…

witten commented

2019-12-05 07:08:55 +00:00

This is one of those features that I've thought of adding before, but said to myself: "Naaah, if someone really wants it, they'll file a ticket". So here we are. 😄

I will admit though that I am a little bit of two minds on this one. On the one hand, overriding the occasional configuration option via the command-line makes theoretical sense. There's plenty of prior art in other command-line applications. On the other hand though, 1. borgmatic's main premise/philosophy is that you can replace most Borg command-line flags with a declarative config file, and 2. If there's a major gap in borgmatic's configuration functionality (as per witten/borgmatic#73, etc.), maybe we should fix that instead of working around it on the command-line?

My use case is that I would like to store the config in a git repository with my code, but the fact that I have to supply a repository makes this not a good solution for me.

Can you say a little more about your use case? Does your repository name contain a secret? Is it only known at runtime? What's the mismatch with git-backed config files?

But the other is that i have multiple instances of that app on the same server and it would be nice to just create backups for all of them with one simple script.

So in this case, the only thing that varies is the repository? I assume the source directories vary as well?

That script would find all (identical) config files and run borgmatic for each of them and add a repository for each app instance based on some environment variables.

Where would the environment variables come from? To be clear, you're not expecting borgmatic to consume the environment variables directly (necessarily)? But instead command-line arguments that you could populate with environment variables would be fine?

For the sake of completeness, have you looked into configuration includes, and determined that they're not a good fit for your needs?

I know this is a lot of questions.. Just trying to make sure I understand this need!

This is one of those features that I've thought of adding before, but said to myself: "Naaah, if someone really wants it, they'll file a ticket". So here we are. :smile: I will admit though that I am a little bit of two minds on this one. On the one hand, overriding the occasional configuration option via the command-line makes theoretical sense. There's plenty of [prior art](https://helm.sh/docs/intro/using_helm/#customizing-the-chart-before-installing) in other command-line applications. On the other hand though, 1. borgmatic's main premise/philosophy is that you can replace most Borg command-line flags with a declarative config file, and 2. If there's a major gap in borgmatic's configuration functionality (as per https://projects.torsion.org/witten/borgmatic/issues/73, etc.), maybe we should fix that instead of working around it on the command-line? > My use case is that I would like to store the config in a git repository with my code, but the fact that I have to supply a repository makes this not a good solution for me. Can you say a little more about your use case? Does your repository name contain a secret? Is it only known at runtime? What's the mismatch with git-backed config files? > But the other is that i have multiple instances of that app on the same server and it would be nice to just create backups for all of them with one simple script. So in this case, the only thing that varies is the repository? I assume the source directories vary as well? > That script would find all (identical) config files and run borgmatic for each of them and add a repository for each app instance based on some environment variables. Where would the environment variables come from? To be clear, you're not expecting borgmatic to consume the environment variables directly (necessarily)? But instead command-line arguments that you could populate with environment variables would be fine? For the sake of completeness, have you looked into [configuration includes](https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#configuration-includes), and determined that they're not a good fit for your needs? I know this is a lot of questions.. Just trying to make sure I understand this need!

errnesto commented

2019-12-05 23:05:20 +00:00

I know this is a lot of questions.. Just trying to make sure I understand this need!

No thank you for your quick reply, and your interest :-)

Can you say a little more about your use case? Does your repository name contain a secret? Is it only known at runtime? What’s the mismatch with git-backed config files?

Well the fact that it feels wrong to me to store that config with the code might be just a feeling that makes no sense at all…

But that is how I understand the 12 factor app https://12factor.net/backing-services and https://12factor.net/config
And those rules kind of do make sense to me. Say I want to test locally if my backup setup is working: I could just set the repository environment variable to some local location and check if everything is fine without making changes to the code.
Or I could set up a staging server, that – for some reason – has backups but stores them somewhere else…

The real use case is I have a server with multiple services on it (running in lxc containers).
So there are two instances of service A (with different config) and one of B and C.
There might be a service D later.
Each service knows how to backup itself but I want to have a single script that can collect all the backups and then send them off to my backup server.

So in this case, the only thing that varies is the repository? I assume the source directories vary as well?

Yes. You can imagine it as cloning the same git repo into two different folders on the same server.

Where would the environment variables come from?

The borg repository would basically be named after the container the app is running in.

command-line arguments that you could populate with environment variables would be fine?

Yes

For the sake of completeness, have you looked into configuration includes, and determined that they’re not a good fit for your needs?

Yes - I think this is the other way around of what I'm trying to do.
I try to have "multiple values in the same config" not share values between configs.

But also maybe I'm thinking about this all wrong. Or I'm trying to use the wrong tool for the task…

Anyway thank you for your time :-)

> I know this is a lot of questions.. Just trying to make sure I understand this need! No thank you for your quick reply, and your interest :-) > Can you say a little more about your use case? Does your repository name contain a secret? Is it only known at runtime? What’s the mismatch with git-backed config files? Well the fact that it feels wrong to me to store that config with the code might be just a feeling that makes no sense at all… But that is how I understand the 12 factor app https://12factor.net/backing-services and https://12factor.net/config And those rules kind of do make sense to me. Say I want to test locally if my backup setup is working: I could just set the repository environment variable to some local location and check if everything is fine without making changes to the code. Or I could set up a staging server, that – for some reason – has backups but stores them somewhere else… The real use case is I have a server with multiple services on it (running in lxc containers). So there are two instances of service A (with different config) and one of B and C. There might be a service D later. Each service knows how to backup itself but I want to have a single script that can collect all the backups and then send them off to my backup server. > So in this case, the only thing that varies is the repository? I assume the source directories vary as well? Yes. You can imagine it as cloning the same git repo into two different folders on the same server. > Where would the environment variables come from? The borg repository would basically be named after the container the app is running in. > command-line arguments that you could populate with environment variables would be fine? Yes > For the sake of completeness, have you looked into configuration includes, and determined that they’re not a good fit for your needs? Yes - I think this is the other way around of what I'm trying to do. I try to have "multiple values in the same config" not share values between configs. But also maybe I'm thinking about this all wrong. Or I'm trying to use the wrong tool for the task… Anyway thank you for your time :-)

witten commented

2019-12-09 18:54:44 +00:00

Thanks for the context. That's super helpful. And understood about 12-factor apps. So in your case it sounds like your borgmatic configuration file is stored in Git along with the source code of the application, and you're wanting to vary the repository / source directories per installed instance of the application. And you have a couple of different such applications.

I'll focus on that use case here. I think that the testing use cases you mention are legit too, but I don't want to assume the solution for all use cases is necessarily the same.

Each service knows how to backup itself but I want to have a single script that can collect all the backups and then send them off to my backup server.

borgmatic does have a feature for configuring multiple different applications, in case you haven't seen it. Basically you drop one config file for each application into /etc/borgmatic.d/, and borgmatic runs each one in succession.

I mention this because I think I'd lean toward any solution that allows borgmatic itself to act as the backup orchestrator rather than a separate script. Not because I have anything against external scripts, but because I view orchestrating backups as pretty central to the borgmatic value proposition! And so I want borgmatic to be good at that.

Yes - I think this is the other way around of what I’m trying to do. I try to have “multiple values in the same config” not share values between configs.

Have you tried something like this?

<<: !include common.yaml

location:
    source_directories:
        - /root/tmp

    repositories:
        - test.borg

... where common.yaml contains your entire application config, just without your instance-specific repositories and source_directories.

To be clear, I'm not against adding features to satisfy your use cases. I just want to make sure I'm performing due diligence here.

Thanks for the context. That's super helpful. And understood about 12-factor apps. So in your case it sounds like your borgmatic configuration file is stored in Git along with the source code of the application, and you're wanting to vary the repository / source directories per installed instance of the application. And you have a couple of different such applications. I'll focus on that use case here. I think that the testing use cases you mention are legit too, but I don't want to assume the solution for all use cases is necessarily the same. > Each service knows how to backup itself but I want to have a single script that can collect all the backups and then send them off to my backup server. borgmatic does have a feature for configuring multiple different applications, in case you [haven't seen it](https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#multiple-backup-configurations). Basically you drop one config file for each application into `/etc/borgmatic.d/`, and borgmatic runs each one in succession. I mention this because I think I'd lean toward any solution that allows borgmatic itself to act as the backup orchestrator rather than a separate script. Not because I have anything against external scripts, but because I view orchestrating backups as pretty central to the borgmatic value proposition! And so I want borgmatic to be good at that. > Yes - I think this is the other way around of what I’m trying to do. I try to have “multiple values in the same config” not share values between configs. Have you tried something like this? ``` <<: !include common.yaml location: source_directories: - /root/tmp repositories: - test.borg ``` ... where `common.yaml` contains your entire application config, just without your instance-specific `repositories` and `source_directories`. To be clear, I'm not against adding features to satisfy your use cases. I just want to make sure I'm performing due diligence here.

errnesto commented

2019-12-09 23:47:46 +00:00

Have you tried something like this?
…

Oh I like that! Now it seams obvoius to do it that way but I did noth think of it.

Still I think my usecase is not covered.
I'd love borgmatic to do the orchestraion.
But sadly I think, in my case, I do need an extra script because the apps live in containers.
And I have to run borgmatic in each container so it has access to the files and can run the hooks (the host just runs the cronjob and passes the repositories so I don't have to set it up for every container)

But yeah that is my very specific use case. I don't know if that is a valid argument that borgmatic needs this functionality :-)

> Have you tried something like this? > … Oh I like that! Now it seams obvoius to do it that way but I did noth think of it. Still I think my usecase is not covered. I'd love borgmatic to do the orchestraion. But sadly I think, in my case, I do need an extra script because the apps live in containers. And I have to run borgmatic in each container so it has access to the files and can run the hooks (the host just runs the cronjob and passes the repositories so I don't have to set it up for every container) But yeah that is my very specific use case. I don't know if that is a valid argument that borgmatic needs this functionality :-)

witten commented

2019-12-10 18:21:18 +00:00

Gotcha! Separate borgmatic instances in containers was the piece I was missing. Thanks for getting me there. Now I get the motivation for the separate script. I'll look into the command-line override approach and see what it would take.

errnesto commented

2019-12-13 14:49:09 +00:00

:D cool

witten referenced this issue from a commit

2019-12-17 19:50:05 +00:00

Override particular configuration options from the command-line via "--override" flag (#268).

witten commented

2019-12-17 19:53:50 +00:00

Okay, this is implemented now in master! The syntax for overriding both source_directories and repositories would look something like this:

borgmatic create --override location.source_directories=[/foo,/bar] location.repositories=[test.borg]

Docs will be here momentarily: https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#configuration-overrides

It's not released yet, but I'll follow-up here when the next release goes out.

Okay, this is implemented now in master! The syntax for overriding both `source_directories` and `repositories` would look something like this: ```bash borgmatic create --override location.source_directories=[/foo,/bar] location.repositories=[test.borg] ``` Docs will be here momentarily: https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#configuration-overrides It's not released yet, but I'll follow-up here when the next release goes out.

witten closed this issue

2019-12-17 19:53:50 +00:00

witten commented

2019-12-20 22:11:13 +00:00

This is released now in borgmatic 1.4.21! Let me know how it works out for you.