Our metrics infrastructure is pretty weird: two Tailscale networks, prom+grafana in one of them, a lots of hosts in another.
The problem is: you have to share all machines from the second network, and/or deal with ACLs to make them accessible by a metrics scrapper from the first network.
The idea is: share only one machine, use it as a proxy or metrics gateway for the whole network.
I can see two possible approaches for this:
expose many ports "/metrics_n" on a proxy node, each path handler forwards the request to the "/metrics" handler of its own service or machine.
metrics_path is a thing, and I can implement a proxy with just lots of "/metrics_n" handlers.write a metrics scrapper that will annotate metrics with machine or service names, and expose them all at once, in a single /metrics handler for a scrapper in the another network.
The second approach sounds weird enough to give it a try, and so learn something new about prometheus protocol, and its internals. But that's the only pros on that, so let's quickly hack the first solution.
two hours alter: the solution, now manages metrics collection in my personal Tailscale network.