Remote Version: Add async polling to speed up the scraping /metrics endpoint #102
Conversation
hansmi
left a comment
hansmi
left a comment
There was a problem hiding this comment.
Partial review only. Code needs some changes.
|
|
||
| // run executes the periodic polling loop. | ||
| func (c *remoteVersionCollector) run() { | ||
| ticker := time.NewTicker(c.interval) |
There was a problem hiding this comment.
Unfortunately this code will keep polling even when there are no metric requests. With the (current) default that's once per minute.
Please restructure such that a new request is issued if the last result is older than a configured interval. That way no "stopping" is necessary either.
There was a problem hiding this comment.
I may not understand fully your point. Can you elaborate a bit?
The basic idea is to keep polling the information in the background to speed up the /metrics request and not doing a call each /metric call. The new default is 1h, but can also be set to 1d.
There was a problem hiding this comment.
@hansmi Any deeper elaboration for me? Happy to adjust the code here.
There was a problem hiding this comment.
I see. Is the long /metrics request a big problem?
Would it be better to introduce a separate endpoint to get the remote version metrics (the result of which could still be reflected in /metrics in a passive manner)? For that endpoint Prometheus can be configured to use a longer timeout plus a longer poll interval.
Don't worry about the code yet.
There was a problem hiding this comment.
Another endpoint would also be good, a different polling interval can be configured.
Is the long /metrics request a big problem?
Not in this particular situation. I would like
- to poll the paperless metrics very frequently
- to poll / refresh the remote version only once a day
As not much new versions are getting released.
(the result of which could still be reflected in /metrics in a passive manner)
How would this be reflected in the code?
I see two options:
- This one here with a different timing polling mechanism
- Another
/metrics-remoteendpoint
The additional /metrics-remote is rather uncommon for tooling that exposes metrics.
Let me know how you would like to proceed.
andygrunwald
force-pushed
the
remote-version-async
branch
5 times, most recently
from
ba77841 to
e4a2606
Compare
andygrunwald
force-pushed
the
remote-version-async
branch
from
e4a2606 to
fb95478
Compare
andygrunwald
force-pushed
the
remote-version-async
branch
from
fb95478 to
fff54a6
Compare
|
I found out, that during startup, i do two initial fetches. With 6ded949 this is now only one |
2 participants
This PR adds the functionality to not call the remote version endpoint everytime
/metricsis called, but in an async way.The interval can be configured via the
--remote-version-intervalflag.