Prometheus - Match all metrics but one

Question

How can I match all Prometheus metrics except some?

E.g: {__name__!~"metric_to_discard"}

Does not work, it returns Error executing query: parse error at char 35: vector selector must contain at least one non-empty matcher.

Thanks

Answer 1

This is some kind of safety mechanism to avoid heavy load on Prometheus by accidentally using too wide selectors. Usually you shouldn't have a need for such a query that matches (almost) every metric with all label dimensions.

There is some hacky solution to workaround this safety check by adding another dimension with a ..* or .+ match-all selector. (The safety check also forbids using .*)

{job=~"..*", __name__!~"metric_to_discard"}

Caution: Be aware that this query will probably hit your Prometheus instance pretty hard and this should not be used on a production system.

As already said, there is probably a better way to get the result you like without querying for almost each and every metric. Please consider posting a new question describing what result you really want to achieve.

Answer 2

Prometheus supports {__name__!=""} query, which returns all the time series. E.g. this query is equivalent to the following SQL query: SELECT * FROM metrics (note that this query may crash Prometheus if Prometheus contains many time series). Prometheus allows using multiple filters against the same label (including __name__ label, which maps to metric name). So the following query would return all the time series except the metric_to_discard metric:

{__name__!="",__name__!~"metric_to_discard"}

P.S. The __name__!="" hack isn't needed in VictoriaMetrics. It just returns the expected results from {__name__!~"metric_to_discard"} query.