Skip to content

Monitoring prometheus-k8s-n pods RBAC bug

In namespace monitoring the prometheus-k8s-0 andprometheus-k8s-1 pods are showing the following error in the logs:

=== Namespace: monitoring ===
--- Pod: prometheus-k8s-0 (current) ---
2025-11-11T13:35:17.097789845+01:00 time=2025-11-11T12:35:17.097Z level=ERROR source=reflector.go:205 msg="Failed to watch" component=k8s_client_runtime logger=UnhandledError err="failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\" in the namespace \"kepler\"" reflector=pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290 type=*v1.EndpointSlice
2025-11-11T13:36:11.428626045+01:00 time=2025-11-11T12:36:11.428Z level=ERROR source=reflector.go:205 msg="Failed to watch" component=k8s_client_runtime logger=UnhandledError err="failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\" in the namespace \"kepler\"" reflector=pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290 type=*v1.EndpointSlice
2025-11-11T13:36:51.149358752+01:00 time=2025-11-11T12:36:51.149Z level=ERROR source=reflector.go:205 msg="Failed to watch" component=k8s_client_runtime logger=UnhandledError err="failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\" in the namespace \"kepler\"" reflector=pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290 type=*v1.EndpointSlice
2025-11-11T13:37:27.504477282+01:00 time=2025-11-11T12:37:27.504Z level=ERROR source=reflector.go:205 msg="Failed to watch" component=k8s_client_runtime logger=UnhandledError err="failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\" in the namespace \"kepler\"" reflector=pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290 type=*v1.EndpointSlice
2025-11-11T13:38:14.099689659+01:00 time=2025-11-11T12:38:14.099Z level=ERROR source=reflector.go:205 msg="Failed to watch" component=k8s_client_runtime logger=UnhandledError err="failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\" in the namespace \"kepler\"" reflector=pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290 type=*v1.EndpointSlice
2025-11-11T13:38:48.396849627+01:00 time=2025-11-11T12:38:48.396Z level=ERROR source=reflector.go:205 msg="Failed to watch" component=k8s_client_runtime logger=UnhandledError err="failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\" in the namespace \"kepler\"" reflector=pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290 type=*v1.EndpointSlice
2025-11-11T13:39:25.887465226+01:00 time=2025-11-11T12:39:25.887Z level=ERROR source=reflector.go:205 msg="Failed to watch" component=k8s_client_runtime logger=UnhandledError err="failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\" in the namespace \"kepler\"" reflector=pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290 type=*v1.EndpointSlice
2025-11-11T13:40:07.885681154+01:00 time=2025-11-11T12:40:07.885Z level=ERROR source=reflector.go:205 msg="Failed to watch" component=k8s_client_runtime logger=UnhandledError err="failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\" in the namespace \"kepler\"" reflector=pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290 type=*v1.EndpointSlice
2025-11-11T13:40:47.507800436+01:00 time=2025-11-11T12:40:47.507Z level=ERROR source=reflector.go:205 msg="Failed to watch" component=k8s_client_runtime logger=UnhandledError err="failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\" in the namespace \"kepler\"" reflector=pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290 type=*v1.EndpointSlice
2025-11-11T13:41:31.834897850+01:00 time=2025-11-11T12:41:31.834Z level=ERROR source=reflector.go:205 msg="Failed to watch" component=k8s_client_runtime logger=UnhandledError err="failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\" in the namespace \"kepler\"" reflector=pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290 type=*v1.EndpointSlice
2025-11-11T13:42:26.597865789+01:00 time=2025-11-11T12:42:26.597Z level=ERROR source=reflector.go:205 msg="Failed to watch" component=k8s_client_runtime logger=UnhandledError err="failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\" in the namespace \"kepler\"" reflector=pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290 type=*v1.EndpointSlice
2025-11-11T13:43:02.578684112+01:00 time=2025-11-11T12:43:02.578Z level=ERROR source=reflector.go:205 msg="Failed to watch" component=k8s_client_runtime logger=UnhandledError err="failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\" in the namespace \"kepler\"" reflector=pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290 type=*v1.EndpointSlice
2025-11-11T13:43:37.214526337+01:00 time=2025-11-11T12:43:37.214Z level=ERROR source=reflector.go:205 msg="Failed to watch" component=k8s_client_runtime logger=UnhandledError err="failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\" in the namespace \"kepler\"" reflector=pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290 type=*v1.EndpointSlice
2025-11-11T13:44:13.756085001+01:00 time=2025-11-11T12:44:13.755Z level=ERROR source=reflector.go:205 msg="Failed to watch" component=k8s_client_runtime logger=UnhandledError err="failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\" in the namespace \"kepler\"" reflector=pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290 type=*v1.EndpointSlice
2025-11-11T13:44:52.784653308+01:00 time=2025-11-11T12:44:52.784Z level=ERROR source=reflector.go:205 msg="Failed to watch" component=k8s_client_runtime logger=UnhandledError err="failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\" in the namespace \"kepler\"" reflector=pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290 type=*v1.EndpointSlice

=== Namespace: monitoring ===
--- Pod: prometheus-k8s-1 (current) ---
2025-11-11T12:35:18.698374053Z time=2025-11-11T12:35:18.698Z level=ERROR source=reflector.go:205 msg="Failed to watch" component=k8s_client_runtime logger=UnhandledError err="failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\" in the namespace \"kepler\"" reflector=pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290 type=*v1.EndpointSlice
2025-11-11T12:36:06.962645986Z time=2025-11-11T12:36:06.962Z level=ERROR source=reflector.go:205 msg="Failed to watch" component=k8s_client_runtime logger=UnhandledError err="failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\" in the namespace \"kepler\"" reflector=pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290 type=*v1.EndpointSlice
2025-11-11T12:36:48.590548378Z time=2025-11-11T12:36:48.590Z level=ERROR source=reflector.go:205 msg="Failed to watch" component=k8s_client_runtime logger=UnhandledError err="failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\" in the namespace \"kepler\"" reflector=pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290 type=*v1.EndpointSlice
2025-11-11T12:37:44.819668742Z time=2025-11-11T12:37:44.819Z level=ERROR source=reflector.go:205 msg="Failed to watch" component=k8s_client_runtime logger=UnhandledError err="failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\" in the namespace \"kepler\"" reflector=pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290 type=*v1.EndpointSlice
2025-11-11T12:38:32.796575366Z time=2025-11-11T12:38:32.796Z level=ERROR source=reflector.go:205 msg="Failed to watch" component=k8s_client_runtime logger=UnhandledError err="failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\" in the namespace \"kepler\"" reflector=pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290 type=*v1.EndpointSlice
2025-11-11T12:39:06.275975113Z time=2025-11-11T12:39:06.275Z level=ERROR source=reflector.go:205 msg="Failed to watch" component=k8s_client_runtime logger=UnhandledError err="failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\" in the namespace \"kepler\"" reflector=pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290 type=*v1.EndpointSlice
2025-11-11T12:39:56.743797415Z time=2025-11-11T12:39:56.743Z level=ERROR source=reflector.go:205 msg="Failed to watch" component=k8s_client_runtime logger=UnhandledError err="failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\" in the namespace \"kepler\"" reflector=pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290 type=*v1.EndpointSlice
2025-11-11T12:40:45.705894933Z time=2025-11-11T12:40:45.705Z level=ERROR source=reflector.go:205 msg="Failed to watch" component=k8s_client_runtime logger=UnhandledError err="failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\" in the namespace \"kepler\"" reflector=pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290 type=*v1.EndpointSlice
2025-11-11T12:41:23.527138955Z time=2025-11-11T12:41:23.526Z level=ERROR source=reflector.go:205 msg="Failed to watch" component=k8s_client_runtime logger=UnhandledError err="failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\" in the namespace \"kepler\"" reflector=pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290 type=*v1.EndpointSlice
2025-11-11T12:42:14.692145887Z time=2025-11-11T12:42:14.692Z level=ERROR source=reflector.go:205 msg="Failed to watch" component=k8s_client_runtime logger=UnhandledError err="failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\" in the namespace \"kepler\"" reflector=pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290 type=*v1.EndpointSlice
2025-11-11T12:42:52.999982639Z time=2025-11-11T12:42:52.999Z level=ERROR source=reflector.go:205 msg="Failed to watch" component=k8s_client_runtime logger=UnhandledError err="failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\" in the namespace \"kepler\"" reflector=pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290 type=*v1.EndpointSlice
2025-11-11T12:43:27.830548332Z time=2025-11-11T12:43:27.830Z level=ERROR source=reflector.go:205 msg="Failed to watch" component=k8s_client_runtime logger=UnhandledError err="failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\" in the namespace \"kepler\"" reflector=pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290 type=*v1.EndpointSlice
2025-11-11T12:44:13.165422227Z time=2025-11-11T12:44:13.165Z level=ERROR source=reflector.go:205 msg="Failed to watch" component=k8s_client_runtime logger=UnhandledError err="failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\" in the namespace \"kepler\"" reflector=pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290 type=*v1.EndpointSlice
2025-11-11T12:45:01.472393087Z time=2025-11-11T12:45:01.472Z level=ERROR source=reflector.go:205 msg="Failed to watch" component=k8s_client_runtime logger=UnhandledError err="failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\" in the namespace \"kepler\"" reflector=pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290 type=*v1.EndpointSlice

This is a RBAC problem, something with incorrect rights of the service account and the EndpointSlice-resource not readable in the Keplernamespace

Edited by Chiel van Diepen