Skip to content

acm-operator-controller-manager crashloopbackoff after reboot

After rebooting the control plane node, the followin error apperars.

│ manager Already exists: Updating SWM app                                                                                                                                                                                                                                                                               │
│ manager Wed Nov  5 12:49:09 UTC 2025 ------ Waiting 8 seconds -------                                                                                                                                                                                                                                                  │
│ manager Wed Nov  5 12:49:17 UTC 2025 SWM App : acm-swm-app                                                                                                                                                                                                                                                             │
│ manager Wed Nov  5 12:49:17 UTC 2025 SWM App Phase3 : Pending                                                                                                                                                                                                                                                          │
│ manager Wed Nov  5 12:49:17 UTC 2025 SWM App W1 Basename: backend                                                                                                                                                                                                                                                      │
│ manager Wed Nov  5 12:49:17 UTC 2025 ---------------------- Update ACM App status metrics -----------------------                                                                                                                                                                                                      │
│ manager Wed Nov  5 12:49:17 UTC 2025 ---------------------- Get node level metrics -----------------------                                                                                                                                                                                                             │
│ manager Node: steroidpi                                                                                                                                                                                                                                                                                                │
│ manager Node: taart5                                                                                                                                                                                                                                                                                                   │
│ manager Node: codeco-almende                                                                                                                                                                                                                                                                                           │
│ manager Wed Nov  5 12:49:17 UTC 2025 ---------------------- Get app level metrics -----------------------                                                                                                                                                                                                              │
│ manager Wed Nov  5 12:49:17 UTC 2025 ---------------------- Get service level metrics -----------------------                                                                                                                                                                                                          │
│ manager Pod: acm-swm-app-front-end, Node: codeco-almende                                                                                                                                                                                                                                                               │
│ manager Pod: acm-swm-app-backend, Node: codeco-almende                                                                                                                                                                                                                                                                 │
│ manager Pod: acm-swm-app-backend-1, Node:                                                                                                                                                                                                                                                                              │
│ manager 2025-11-05T12:49:17Z    INFO    Observed a panic in reconciler: runtime error: index out of range [2] with length 2    {"controller": "codecoapp", "controllerGroup": "codeco.he-codeco.eu", "controllerKind": "CodecoApp", "CodecoApp": {"name":"codecoappinstance3","namespace":"he-codeco-acm"}, "namespace │
│ manager panic: runtime error: index out of range [2] with length 2 [recovered]                                                                                                                                                                                                                                         │
│ manager     panic: runtime error: index out of range [2] with length 2                                                                                                                                                                                                                                                 │
│ manager                                                                                                                                                                                                                                                                                                                │
│ manager goroutine 129 [running]:                                                                                                                                                                                                                                                                                       │
│ manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()                                                                                                                                                                                                                         │
│ manager     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:116 +0x1a4                                                                                                                                                                                                        │
│ manager panic({0x1424e20?, 0x40006a81c8?})                                                                                                                                                                                                                                                                             │
│ manager     /usr/local/go/src/runtime/panic.go:770 +0x124                                                                                                                                                                                                                                                              │
│ manager gitlab.eclipse.org/eclipse-research-labs/codeco-project/acm/controllers.getPodInfo(0x40000e2008, {0x17f46e0?, 0x400026be70?})                                                                                                                                                                                  │
│ manager     /workspace/controllers/codecoapp_controller.go:165 +0x2a4                                                                                                                                                                                                                                                  │
│ manager gitlab.eclipse.org/eclipse-research-labs/codeco-project/acm/controllers.getServiceMetricsFromPrometheus(0x40000e2008, {0x17f46e0, 0x400026be70})                                                                                                                                                               │
│ manager     /workspace/controllers/codecoapp_controller.go:183 +0x104                                                                                                                                                                                                                                                  │
│ manager gitlab.eclipse.org/eclipse-research-labs/codeco-project/acm/controllers.getAppMetricsFromPrometheus(0x40000e2008, {0x17f46e0, 0x400026be70})                                                                                                                                                                   │
│ manager     /workspace/controllers/codecoapp_controller.go:250 +0x2a4                                                                                                                                                                                                                                                  │
│ manager gitlab.eclipse.org/eclipse-research-labs/codeco-project/acm/controllers.getMetricsFromPrometheus(0x40000e2008)                                                                                                                                                                                                 │
│ manager     /workspace/controllers/codecoapp_controller.go:331 +0x14c                                                                                                                                                                                                                                                  │
│ manager gitlab.eclipse.org/eclipse-research-labs/codeco-project/acm/controllers.(*CodecoAppReconciler).Reconcile(0x400041d818, {0x17e55f8, 0x40001c2510}, {{{0x4000734d50?, 0x5?}, {0x4000585188?, 0x400046dd08?}}})                                                                                                   │
│ manager     /workspace/controllers/codecoapp_controller.go:527 +0x16d4                                                                                                                                                                                                                                                 │
│ manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x17e97f0?, {0x17e55f8?, 0x40001c2510?}, {{{0x4000734d50?, 0xb?}, {0x4000585188?, 0x0?}}})                                                                                                                                      │
│ manager     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119 +0x8c                                                                                                                                                                                                         │
│ manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0x4000273360, {0x17e5630, 0x400026c6e0}, {0x13907c0, 0x4000342020})                                                                                                                                                      │
│ manager     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316 +0x2dc                                                                                                                                                                                                        │
│ manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0x4000273360, {0x17e5630, 0x400026c6e0})                                                                                                                                                                              │
│ manager     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266 +0x16c                                                                                                                                                                                                        │
│ manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()                                                                                                                                                                                                                           │
│ manager     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227 +0x74                                                                                                                                                                                                         │
│ manager created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 74                                                                                                                                                                                                    │
│ manager     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:223 +0x404  

Also, the already deployed example app got duplicated (backend only) and keeps on being pending:

image

@gpapathan