borgmatic-collective/borgmatic
borgmatic-collective/borgmatic
Watch
9
74
Fork
You've already forked borgmatic
 57
Code Issues 71 Pull requests 3 Releases 162 Packages 1 Activity Actions 1

Loki hook not sending logs #1152

New issue
Closed
opened 2025-09-25 16:43:06 +00:00 by maxhamon · 7 comments
maxhamon commented 2025-09-25 16:43:06 +00:00
Contributor
Copy link

What I'm trying to do and why

Using Borgmatic 2.0.8, I’m trying to send logs to my Loki instance, as I was able to do in version 2.0.7

Steps to reproduce

I have made a simple configuration file for testing

➜ cat test.yaml 
---

compression: auto,zstd

source_directories:
  - '/root/test'

repositories:
  - path: 'ssh://redacted@redacted.repo.borgbase.com/./repo'
encryption_passphrase: 'redacted'

archive_name_format: 'test-{now}'

loki:
  url: 'https://redacted:redacted@loki.fqdn/loki/api/v1/push'
  labels:
    job: logs/borgmatic
    instance: __hostname
    service: test

Actual behavior

➜ borgmatic -c test.yaml create --verbosity 2
/root/test.yaml: Calling loki hook function initialize_monitor
/root/test.yaml: Calling loki hook function ping_monitor
/root/test.yaml: BORG_PASSPHRASE_FD=*** BORG_DEBUG_PASSPHRASE=*** BORG_DISPLAY_PASSPHRASE=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_USE_CHUNKS_ARCHIVE=*** BORG_EXIT_CODES=*** borg --version --debug --show-rc
/root/test.yaml: Borg 1.4.1
ssh://redacted@redacted.repo.borgbase.com/./repo: Running actions for repository
ssh://redacted@redacted.repo.borgbase.com/./repo: Creating archive
ssh://redacted@redacted.repo.borgbase.com/./repo: Using runtime directory /tmp/borgmatic-tk9m_g91/borgmatic
ssh://redacted@redacted.repo.borgbase.com/./repo: Calling bootstrap hook function remove_data_source_dumps
ssh://redacted@redacted.repo.borgbase.com/./repo: Looking for bootstrap manifest files to remove in /tmp/borgmatic-*/borgmatic/bootstrap
ssh://redacted@redacted.repo.borgbase.com/./repo: Calling btrfs hook function remove_data_source_dumps
ssh://redacted@redacted.repo.borgbase.com/./repo: Calling lvm hook function remove_data_source_dumps
ssh://redacted@redacted.repo.borgbase.com/./repo: Calling mariadb hook function remove_data_source_dumps
ssh://redacted@redacted.repo.borgbase.com/./repo: Removing MariaDB data source dumps
ssh://redacted@redacted.repo.borgbase.com/./repo: Calling mongodb hook function remove_data_source_dumps
ssh://redacted@redacted.repo.borgbase.com/./repo: Removing MongoDB data source dumps
ssh://redacted@redacted.repo.borgbase.com/./repo: Calling mysql hook function remove_data_source_dumps
ssh://redacted@redacted.repo.borgbase.com/./repo: Removing MySQL data source dumps
ssh://redacted@redacted.repo.borgbase.com/./repo: Calling postgresql hook function remove_data_source_dumps
ssh://redacted@redacted.repo.borgbase.com/./repo: Removing PostgreSQL data source dumps
ssh://redacted@redacted.repo.borgbase.com/./repo: Calling sqlite hook function remove_data_source_dumps
ssh://redacted@redacted.repo.borgbase.com/./repo: Removing SQLite data source dumps
ssh://redacted@redacted.repo.borgbase.com/./repo: Calling zfs hook function remove_data_source_dumps
ssh://redacted@redacted.repo.borgbase.com/./repo: Calling bootstrap hook function dump_data_sources
ssh://redacted@redacted.repo.borgbase.com/./repo: Writing patterns to /tmp/borgmatic-tk9m_g91/borgmatic/tmpwv1zda0u:
R /root/test
R /root/test.yaml
R /tmp/borgmatic-tk9m_g91/./borgmatic/bootstrap
ssh://redacted@redacted.repo.borgbase.com/./repo: Checking file paths Borg plans to include
ssh://redacted@redacted.repo.borgbase.com/./repo: BORG_PASSPHRASE_FD=*** BORG_DEBUG_PASSPHRASE=*** BORG_DISPLAY_PASSPHRASE=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_USE_CHUNKS_ARCHIVE=*** BORG_EXIT_CODES=*** borg create --patterns-from /tmp/borgmatic-tk9m_g91/borgmatic/tmpwv1zda0u --compression auto,zstd ssh://redacted@redacted.repo.borgbase.com/./repo::test-{now} --dry-run --list
ssh://redacted@redacted.repo.borgbase.com/./repo: BORG_PASSPHRASE_FD=*** BORG_DEBUG_PASSPHRASE=*** BORG_DISPLAY_PASSPHRASE=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_USE_CHUNKS_ARCHIVE=*** BORG_EXIT_CODES=*** borg create --patterns-from /tmp/borgmatic-tk9m_g91/borgmatic/tmpwv1zda0u --compression auto,zstd --debug --show-rc ssh://redacted@redacted.repo.borgbase.com/./repo::test-{now}
using builtin fallback logging configuration
33 self tests completed in 0.03 seconds
SSH command line: ['ssh', 'redacted@redacted.repo.borgbase.com', 'borg', 'serve', '--debug']
Remote: using builtin fallback logging configuration
Remote: borg selftest disabled via BORG_SELFTEST env variable
Remote: using builtin fallback logging configuration
Remote: Initialized logging system for JSON-based protocol
Remote: Resolving repository path b'/./repo'
Remote: Resolved repository path to '/srv/repos/redacted/repo'
Remote: Verified integrity of /srv/repos/redacted/repo/index.21
TAM-verified manifest
security: read previous location 'ssh://redacted@redacted.repo.borgbase.com/./repo'
security: read manifest timestamp '2025-09-25T16:29:31.402252'
security: determined newest manifest timestamp as 2025-09-25T16:29:31.402252
security: repository checks ok, allowing access
Creating archive at "ssh://redacted@redacted.repo.borgbase.com/./repo::test-2025-09-25T18:30:45"
Verified integrity of /root/.cache/borg/182c4f4402b084caea41006063aff730d6cbcab7a6b39d37f7c95df4960860b7/chunks
Reading files cache ...
Verified integrity of /root/.cache/borg/182c4f4402b084caea41006063aff730d6cbcab7a6b39d37f7c95df4960860b7/files
security: read previous location 'ssh://redacted@redacted.repo.borgbase.com/./repo'
security: read manifest timestamp '2025-09-25T16:29:31.402252'
security: determined newest manifest timestamp as 2025-09-25T16:29:31.402252
security: repository checks ok, allowing access
Processing files ...
Remote: Cleaned up 0 uncommitted segment files (== everything after segment 21).
Remote: Verified integrity of /srv/repos/redacted/repo/hints.21
Remote: Storage quota: 10.47 kB out of 3.21 TB used.
Remote: check_free_space: required bytes 545429078, free bytes 17810533048320
Remote: Storage quota: 12.42 kB out of 3.21 TB used.
security: saving state for 182c4f4402b084caea41006063aff730d6cbcab7a6b39d37f7c95df4960860b7 to /root/.config/borg/security/182c4f4402b084caea41006063aff730d6cbcab7a6b39d37f7c95df4960860b7
security: current location   ssh://redacted@redacted.repo.borgbase.com/./repo
security: key type           5
security: manifest timestamp 2025-09-25T16:30:46.992380
RemoteRepository: 2.39 kB bytes sent, 4.21 kB bytes received, 11 messages sent
terminating with success status, rc 0
ssh://redacted@redacted.repo.borgbase.com/./repo: Calling bootstrap hook function remove_data_source_dumps
ssh://redacted@redacted.repo.borgbase.com/./repo: Looking for bootstrap manifest files to remove in /tmp/borgmatic-*/borgmatic/bootstrap
ssh://redacted@redacted.repo.borgbase.com/./repo: Removing bootstrap manifest at /tmp/borgmatic-tk9m_g91/borgmatic/bootstrap/manifest.json
ssh://redacted@redacted.repo.borgbase.com/./repo: Calling btrfs hook function remove_data_source_dumps
ssh://redacted@redacted.repo.borgbase.com/./repo: Calling lvm hook function remove_data_source_dumps
ssh://redacted@redacted.repo.borgbase.com/./repo: Calling mariadb hook function remove_data_source_dumps
ssh://redacted@redacted.repo.borgbase.com/./repo: Removing MariaDB data source dumps
ssh://redacted@redacted.repo.borgbase.com/./repo: Calling mongodb hook function remove_data_source_dumps
ssh://redacted@redacted.repo.borgbase.com/./repo: Removing MongoDB data source dumps
ssh://redacted@redacted.repo.borgbase.com/./repo: Calling mysql hook function remove_data_source_dumps
ssh://redacted@redacted.repo.borgbase.com/./repo: Removing MySQL data source dumps
ssh://redacted@redacted.repo.borgbase.com/./repo: Calling postgresql hook function remove_data_source_dumps
ssh://redacted@redacted.repo.borgbase.com/./repo: Removing PostgreSQL data source dumps
ssh://redacted@redacted.repo.borgbase.com/./repo: Calling sqlite hook function remove_data_source_dumps
ssh://redacted@redacted.repo.borgbase.com/./repo: Removing SQLite data source dumps
ssh://redacted@redacted.repo.borgbase.com/./repo: Calling zfs hook function remove_data_source_dumps
/root/test.yaml: Calling loki hook function ping_monitor
/root/test.yaml: Calling loki hook function ping_monitor
/root/test.yaml: Calling loki hook function destroy_monitor
/root/test.yaml: Starting new HTTPS connection (1): loki.fqdn:443
/root/test.yaml: https://loki.fqdn:443 "POST /loki/api/v1/push HTTP/1.1" 204 0

Expected behavior

The hooks appear to be called, but no logs are received in Loki, like it was in 2.0.7 with the exact same configuration.

Other notes / implementation ideas

➜ borgmatic --version
2.0.8
➜ borgmatic -c test.yaml create
➜ ./test_loki.sh 
Fetching logs from 2025-09-25 18:26:33 to 2025-09-25 18:27:33.
Empty

➜ borgmatic --version                             
2.0.7
➜ borgmatic -c test.yaml create                   
➜ ./test_loki.sh                                  
Fetching logs from 2025-09-25 18:27:51 to 2025-09-25 18:28:51.
Finished backup
Started backup

borgmatic version

2.0.8

borgmatic installation method

pip

Borg version

1.4.1

Python version

3.13.5

Database version (if applicable)

No response

Operating system and version

Debian 13

### What I'm trying to do and why Using Borgmatic 2.0.8, I’m trying to send logs to my Loki instance, as I was able to do in version 2.0.7 ### Steps to reproduce I have made a simple configuration file for testing ``` ➜ cat test.yaml --- compression: auto,zstd source_directories: - '/root/test' repositories: - path: 'ssh://redacted@redacted.repo.borgbase.com/./repo' encryption_passphrase: 'redacted' archive_name_format: 'test-{now}' loki: url: 'https://redacted:redacted@loki.fqdn/loki/api/v1/push' labels: job: logs/borgmatic instance: __hostname service: test ``` ### Actual behavior ``` ➜ borgmatic -c test.yaml create --verbosity 2 /root/test.yaml: Calling loki hook function initialize_monitor /root/test.yaml: Calling loki hook function ping_monitor /root/test.yaml: BORG_PASSPHRASE_FD=*** BORG_DEBUG_PASSPHRASE=*** BORG_DISPLAY_PASSPHRASE=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_USE_CHUNKS_ARCHIVE=*** BORG_EXIT_CODES=*** borg --version --debug --show-rc /root/test.yaml: Borg 1.4.1 ssh://redacted@redacted.repo.borgbase.com/./repo: Running actions for repository ssh://redacted@redacted.repo.borgbase.com/./repo: Creating archive ssh://redacted@redacted.repo.borgbase.com/./repo: Using runtime directory /tmp/borgmatic-tk9m_g91/borgmatic ssh://redacted@redacted.repo.borgbase.com/./repo: Calling bootstrap hook function remove_data_source_dumps ssh://redacted@redacted.repo.borgbase.com/./repo: Looking for bootstrap manifest files to remove in /tmp/borgmatic-*/borgmatic/bootstrap ssh://redacted@redacted.repo.borgbase.com/./repo: Calling btrfs hook function remove_data_source_dumps ssh://redacted@redacted.repo.borgbase.com/./repo: Calling lvm hook function remove_data_source_dumps ssh://redacted@redacted.repo.borgbase.com/./repo: Calling mariadb hook function remove_data_source_dumps ssh://redacted@redacted.repo.borgbase.com/./repo: Removing MariaDB data source dumps ssh://redacted@redacted.repo.borgbase.com/./repo: Calling mongodb hook function remove_data_source_dumps ssh://redacted@redacted.repo.borgbase.com/./repo: Removing MongoDB data source dumps ssh://redacted@redacted.repo.borgbase.com/./repo: Calling mysql hook function remove_data_source_dumps ssh://redacted@redacted.repo.borgbase.com/./repo: Removing MySQL data source dumps ssh://redacted@redacted.repo.borgbase.com/./repo: Calling postgresql hook function remove_data_source_dumps ssh://redacted@redacted.repo.borgbase.com/./repo: Removing PostgreSQL data source dumps ssh://redacted@redacted.repo.borgbase.com/./repo: Calling sqlite hook function remove_data_source_dumps ssh://redacted@redacted.repo.borgbase.com/./repo: Removing SQLite data source dumps ssh://redacted@redacted.repo.borgbase.com/./repo: Calling zfs hook function remove_data_source_dumps ssh://redacted@redacted.repo.borgbase.com/./repo: Calling bootstrap hook function dump_data_sources ssh://redacted@redacted.repo.borgbase.com/./repo: Writing patterns to /tmp/borgmatic-tk9m_g91/borgmatic/tmpwv1zda0u: R /root/test R /root/test.yaml R /tmp/borgmatic-tk9m_g91/./borgmatic/bootstrap ssh://redacted@redacted.repo.borgbase.com/./repo: Checking file paths Borg plans to include ssh://redacted@redacted.repo.borgbase.com/./repo: BORG_PASSPHRASE_FD=*** BORG_DEBUG_PASSPHRASE=*** BORG_DISPLAY_PASSPHRASE=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_USE_CHUNKS_ARCHIVE=*** BORG_EXIT_CODES=*** borg create --patterns-from /tmp/borgmatic-tk9m_g91/borgmatic/tmpwv1zda0u --compression auto,zstd ssh://redacted@redacted.repo.borgbase.com/./repo::test-{now} --dry-run --list ssh://redacted@redacted.repo.borgbase.com/./repo: BORG_PASSPHRASE_FD=*** BORG_DEBUG_PASSPHRASE=*** BORG_DISPLAY_PASSPHRASE=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_USE_CHUNKS_ARCHIVE=*** BORG_EXIT_CODES=*** borg create --patterns-from /tmp/borgmatic-tk9m_g91/borgmatic/tmpwv1zda0u --compression auto,zstd --debug --show-rc ssh://redacted@redacted.repo.borgbase.com/./repo::test-{now} using builtin fallback logging configuration 33 self tests completed in 0.03 seconds SSH command line: ['ssh', 'redacted@redacted.repo.borgbase.com', 'borg', 'serve', '--debug'] Remote: using builtin fallback logging configuration Remote: borg selftest disabled via BORG_SELFTEST env variable Remote: using builtin fallback logging configuration Remote: Initialized logging system for JSON-based protocol Remote: Resolving repository path b'/./repo' Remote: Resolved repository path to '/srv/repos/redacted/repo' Remote: Verified integrity of /srv/repos/redacted/repo/index.21 TAM-verified manifest security: read previous location 'ssh://redacted@redacted.repo.borgbase.com/./repo' security: read manifest timestamp '2025-09-25T16:29:31.402252' security: determined newest manifest timestamp as 2025-09-25T16:29:31.402252 security: repository checks ok, allowing access Creating archive at "ssh://redacted@redacted.repo.borgbase.com/./repo::test-2025-09-25T18:30:45" Verified integrity of /root/.cache/borg/182c4f4402b084caea41006063aff730d6cbcab7a6b39d37f7c95df4960860b7/chunks Reading files cache ... Verified integrity of /root/.cache/borg/182c4f4402b084caea41006063aff730d6cbcab7a6b39d37f7c95df4960860b7/files security: read previous location 'ssh://redacted@redacted.repo.borgbase.com/./repo' security: read manifest timestamp '2025-09-25T16:29:31.402252' security: determined newest manifest timestamp as 2025-09-25T16:29:31.402252 security: repository checks ok, allowing access Processing files ... Remote: Cleaned up 0 uncommitted segment files (== everything after segment 21). Remote: Verified integrity of /srv/repos/redacted/repo/hints.21 Remote: Storage quota: 10.47 kB out of 3.21 TB used. Remote: check_free_space: required bytes 545429078, free bytes 17810533048320 Remote: Storage quota: 12.42 kB out of 3.21 TB used. security: saving state for 182c4f4402b084caea41006063aff730d6cbcab7a6b39d37f7c95df4960860b7 to /root/.config/borg/security/182c4f4402b084caea41006063aff730d6cbcab7a6b39d37f7c95df4960860b7 security: current location ssh://redacted@redacted.repo.borgbase.com/./repo security: key type 5 security: manifest timestamp 2025-09-25T16:30:46.992380 RemoteRepository: 2.39 kB bytes sent, 4.21 kB bytes received, 11 messages sent terminating with success status, rc 0 ssh://redacted@redacted.repo.borgbase.com/./repo: Calling bootstrap hook function remove_data_source_dumps ssh://redacted@redacted.repo.borgbase.com/./repo: Looking for bootstrap manifest files to remove in /tmp/borgmatic-*/borgmatic/bootstrap ssh://redacted@redacted.repo.borgbase.com/./repo: Removing bootstrap manifest at /tmp/borgmatic-tk9m_g91/borgmatic/bootstrap/manifest.json ssh://redacted@redacted.repo.borgbase.com/./repo: Calling btrfs hook function remove_data_source_dumps ssh://redacted@redacted.repo.borgbase.com/./repo: Calling lvm hook function remove_data_source_dumps ssh://redacted@redacted.repo.borgbase.com/./repo: Calling mariadb hook function remove_data_source_dumps ssh://redacted@redacted.repo.borgbase.com/./repo: Removing MariaDB data source dumps ssh://redacted@redacted.repo.borgbase.com/./repo: Calling mongodb hook function remove_data_source_dumps ssh://redacted@redacted.repo.borgbase.com/./repo: Removing MongoDB data source dumps ssh://redacted@redacted.repo.borgbase.com/./repo: Calling mysql hook function remove_data_source_dumps ssh://redacted@redacted.repo.borgbase.com/./repo: Removing MySQL data source dumps ssh://redacted@redacted.repo.borgbase.com/./repo: Calling postgresql hook function remove_data_source_dumps ssh://redacted@redacted.repo.borgbase.com/./repo: Removing PostgreSQL data source dumps ssh://redacted@redacted.repo.borgbase.com/./repo: Calling sqlite hook function remove_data_source_dumps ssh://redacted@redacted.repo.borgbase.com/./repo: Removing SQLite data source dumps ssh://redacted@redacted.repo.borgbase.com/./repo: Calling zfs hook function remove_data_source_dumps /root/test.yaml: Calling loki hook function ping_monitor /root/test.yaml: Calling loki hook function ping_monitor /root/test.yaml: Calling loki hook function destroy_monitor /root/test.yaml: Starting new HTTPS connection (1): loki.fqdn:443 /root/test.yaml: https://loki.fqdn:443 "POST /loki/api/v1/push HTTP/1.1" 204 0 ``` ### Expected behavior The hooks appear to be called, but no logs are received in Loki, like it was in 2.0.7 with the exact same configuration. ### Other notes / implementation ideas ``` ➜ borgmatic --version 2.0.8 ➜ borgmatic -c test.yaml create ➜ ./test_loki.sh Fetching logs from 2025-09-25 18:26:33 to 2025-09-25 18:27:33. Empty ``` ``` ➜ borgmatic --version 2.0.7 ➜ borgmatic -c test.yaml create ➜ ./test_loki.sh Fetching logs from 2025-09-25 18:27:51 to 2025-09-25 18:28:51. Finished backup Started backup ``` ### borgmatic version 2.0.8 ### borgmatic installation method pip ### Borg version 1.4.1 ### Python version 3.13.5 ### Database version (if applicable) _No response_ ### Operating system and version Debian 13
witten commented 2025-09-25 17:43:15 +00:00
Owner
Copy link

Thanks for filing this. The only change to the Loki hook itself in borgmatic 2.0.8 is that it now adds a user agent header to all outgoing Loki requests. I suppose it's possible there was some other logging change in 2.0.8 outside of the Loki hook that could impact this, although I'm not seeing anything obvious. In any case, have you tried invoking borgmatic with --monitoring-verbosity 2 to see if that changes anything?

Also, if you have easy access to borgmatic's installed source on your machine, you could try modifying the Loki hook to remove the user agent header temporarily to see if that changes the behavior. You'll find that at borgmatic/hooks/monitoring/loki.py, and you can try commenting out this line:

                    'User-Agent': 'borgmatic',
Thanks for filing this. The only change to the Loki hook itself in borgmatic 2.0.8 is that it now adds a user agent header to all outgoing Loki requests. I suppose it's possible there was some other logging change in 2.0.8 outside of the Loki hook that could impact this, although I'm not seeing anything obvious. In any case, have you tried invoking borgmatic with `--monitoring-verbosity 2` to see if that changes anything? Also, if you have easy access to borgmatic's installed source on your machine, you could try modifying the Loki hook to remove the user agent header temporarily to see if that changes the behavior. You'll find that at `borgmatic/hooks/monitoring/loki.py`, and you can try commenting out this line: ```python 'User-Agent': 'borgmatic', ```
maxhamon commented 2025-09-25 19:10:07 +00:00
Author
Contributor
Copy link

I don’t see any difference when using --monitoring-verbosity 2. I also tried removing the User-Agent, but it didn’t make much difference.

The strange thing is that the last POST hitting my reverse proxy in front of Loki was when I tested with Borgmatic 2.0.7:

X.X.X.X - - [25/Sep/2025:18:40:39 +0000] "POST /loki/api/v1/push HTTP/1.1" 204 0 "-" "-" 784497 "loki@docker" "http://172.16.0.133:3100" 7ms

It seems like Borgmatic 2.0.8 isn’t sending any POST requests at all, I'll try to add some more verbose on my reverse proxy tomorrow.

I don’t see any difference when using `--monitoring-verbosity 2`. I also tried removing the User-Agent, but it didn’t make much difference. The strange thing is that the last POST hitting my reverse proxy in front of Loki was when I tested with Borgmatic 2.0.7: ``` X.X.X.X - - [25/Sep/2025:18:40:39 +0000] "POST /loki/api/v1/push HTTP/1.1" 204 0 "-" "-" 784497 "loki@docker" "http://172.16.0.133:3100" 7ms ``` It seems like Borgmatic 2.0.8 isn’t sending any POST requests at all, I'll try to add some more verbose on my reverse proxy tomorrow.
witten commented 2025-09-25 21:38:15 +00:00
Owner
Copy link

Really interesting that it appears not to be sending requests at all. Yeah, if you could confirm that on your reverse proxy, that would be a good idea. Also, is that log you posted complete? Are there any log entries after /root/test.yaml: https://loki.fqdn:443 "POST /loki/api/v1/push HTTP/1.1" 204 0 or does it just end there? And how does this compare to the logs when running borgmatic 2.0.7? Do they get further?

Really interesting that it appears not to be sending requests at all. Yeah, if you could confirm that on your reverse proxy, that would be a good idea. Also, is that log you posted complete? Are there any log entries after `/root/test.yaml: https://loki.fqdn:443 "POST /loki/api/v1/push HTTP/1.1" 204 0` or does it just end there? And how does this compare to the logs when running borgmatic 2.0.7? Do they get further?
maxhamon commented 2025-09-26 07:48:43 +00:00
Author
Contributor
Copy link

I've dug a bit deeper and I was wrong about the reverse proxy (the request is actually going through).
I added a tcpdump to inspect the traffic and noticed something strange in 2.0.8: the values field is empty (with or without the Borgmatic User-Agent).

2.0.8

{"streams": [{"stream": {"instance": "myhost.fqdn", "job": "logs/borgmatic", "service": "test"}, "values": []}]}

2.0.7

[{"stream": {"instance": "", "job": "logs/borgmatic", "service": "test"}, "values": [["1758871691603873776", "Starting new HTTPS connection (1): loki.fqdn:443"], ["17588716916.....truncated

I managed to send logs by playing around with self.root['streams'][0]['values'] = [] but I don’t fully understand how it works.

If I add request_body = self.to_request() before flushing values it works !

  request_body = self.to_request()
  self.root['streams'][0]['values'] = []
  
  try:
      result = requests.post(
          self.url,
          data=request_body,
          timeout=TIMEOUT_SECONDS,
          headers={
              'Content-Type': 'application/json',
              'User-Agent': 'borgmatic',
          },
      )
I've dug a bit deeper and I was wrong about the reverse proxy (the request is actually going through). I added a tcpdump to inspect the traffic and noticed something strange in 2.0.8: the values field is empty (with or without the Borgmatic User-Agent). 2.0.8 ``` {"streams": [{"stream": {"instance": "myhost.fqdn", "job": "logs/borgmatic", "service": "test"}, "values": []}]} ``` 2.0.7 ``` [{"stream": {"instance": "", "job": "logs/borgmatic", "service": "test"}, "values": [["1758871691603873776", "Starting new HTTPS connection (1): loki.fqdn:443"], ["17588716916.....truncated ``` I managed to send logs by playing around with [self.root['streams'][0]['values'] = []](https://projects.torsion.org/borgmatic-collective/borgmatic/src/commit/119245171816e641853e26b8a20d9552907c5e6d/borgmatic/hooks/monitoring/loki.py#L68) but I don’t fully understand how it works. If I add `request_body = self.to_request()` before flushing values it works ! ``` request_body = self.to_request() self.root['streams'][0]['values'] = [] try: result = requests.post( self.url, data=request_body, timeout=TIMEOUT_SECONDS, headers={ 'Content-Type': 'application/json', 'User-Agent': 'borgmatic', }, ) ```
witten commented 2025-09-28 22:54:08 +00:00
Owner
Copy link

Excellent detective work! I think what's going on is that as part of the user agent change, I naively moved self.to_request() to after the values are emptied out. Which means no values get sent now.

Excellent detective work! I think what's going on is that as part of the user agent change, I naively moved `self.to_request()` to *after* the values are emptied out. Which means no values get sent now.
😆 1
witten commented 2025-09-28 23:45:29 +00:00
Owner
Copy link

Okay, a fix for this is implemented in main and will be part of the release! Thanks again.

Okay, a fix for this is implemented in main and will be part of the release! Thanks again.
❤️ 1
witten commented 2025-10-07 17:00:44 +00:00
Owner
Copy link

Released in borgmatic 2.0.9!

Released in borgmatic 2.0.9!
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
parallel-borgmatic
command-hook-steps
2.1.7
2.1.6
2.1.5
2.1.4
2.1.3
2.1.2
2.1.1
2.1.0
2.0.13
2.0.12
2.0.11
2.0.10
2.0.9
2.0.8
2.0.7
2.0.6
2.0.5
2.0.4
2.0.3
2.0.2
2.0.1
2.0.0
1.9.14
1.9.13
1.9.12
1.9.11
1.9.10
1.9.9
1.9.8
1.9.7
1.9.6
1.9.5
1.9.4
1.9.3
1.9.2
1.9.1
1.9.0
1.8.14
1.8.13
1.8.12
1.8.11
1.8.10
1.8.9
1.8.8
1.8.7
1.8.6
1.8.5
1.8.4
1.8.3
1.8.2
1.8.1
1.8.0
1.7.15
1.7.14
1.7.13
1.7.12
1.7.11
1.7.10
1.7.9
1.7.8
1.7.7
1.7.6
1.7.5
1.7.4
1.7.3
1.7.2
1.7.1
1.7.0
1.6.6
1.6.5
1.6.4
1.6.3
1.6.2
1.6.1
1.6.0
1.5.24
1.5.23
1.5.22
1.5.21
1.5.20
1.5.19
1.5.18
1.5.17
1.5.16
1.5.15
1.5.14
1.5.13
1.5.12
1.5.11
1.5.10
1.5.9
1.5.8
1.5.7
1.5.7.dev0
1.5.6
1.5.5
1.5.4
1.5.3
1.5.2
1.5.1
1.5.0
1.4.22
1.4.21
1.4.20
1.4.19
1.4.18
1.4.17
1.4.16
1.4.15
1.4.14
1.4.13
1.4.12
1.4.11
1.4.10
1.4.9
1.4.8
1.4.7
1.4.6
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0
1.3.26
1.3.25
1.3.24
1.3.23
1.3.22
1.3.21
1.3.20
1.3.19
1.3.18
1.3.17
1.3.16
1.3.15
1.3.14
1.3.13
1.3.12
1.3.11
1.3.10
1.3.9
1.3.8
1.3.7
1.3.6
1.3.5
1.3.4
1.3.3
1.3.2
1.3.1
1.3.0
1.2.18
1.2.17
1.2.16
1.2.15
1.2.14
1.2.13
1.2.12
1.2.11
1.2.10
1.2.9
1.2.8
1.2.7
1.2.6
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.15
1.1.14
1.1.13
1.1.12
1.1.11
1.1.10
1.1.9
1.1.7
1.1.6
1.1.5
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.3
1.0.2
1.0.1
1.0.0
0.1.8
0.1.7
0.1.6
0.1.5
0.1.4
0.1.3
0.1.2
0.1.1
0.1.0
0.0.7
0.0.6
0.0.5
0.0.4
0.0.3
0.0.2
Labels
Clear labels   
blocked

   
breaking

   
bug

   
data loss

   
design finalized

   
good first issue

   
new feature area

   
question / support

   
security

   
waiting for response
No labels
blocked
breaking
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
borgmatic-collective/borgmatic#1152
No description provided.