> [!warning] AI-Generated Vulnerability Report > This report was produced using AI-assisted security analysis, adversarially verified but not human-confirmed. This finding was challenged against the codebase and survived; we filter out low-confidence results rather than forwarding raw model output. Technical details, exploitability, and severity may still be wrong. Please validate independently before acting. <details open> <summary><strong>Basic information</strong></summary> **Project name:** Eclipse ioFog™ **Project id:** iot.iofog **Repository:** https://github.com/eclipse-iofog/iofog-operator </details> <details> <summary><strong>What are the affected versions?</strong></summary> `v3.7.1-beta.1`, `v3.7.2-beta.0`, `v3.7.2`, and current `develop` HEAD (`eb6aa6a2`). Earlier tags (`v3.2.0` and below) used a different login path that did not contact Keycloak. Determined via `git tag --contains be629676`. </details> <details open> <summary><strong>Summary</strong></summary> `ControlPlaneReconciler.loginIofogClient` constructs an `http.Client` with `TLSClientConfig: &tls.Config{InsecureSkipVerify: true}` and uses it to POST the Keycloak `controllerClient` ID and `controllerSecret` to the OIDC token endpoint configured in `spec.auth.url`. When the operator is pointed at an `https://` Keycloak endpoint (the normal configuration), an active network attacker between the operator pod and Keycloak can present any certificate, capture the client secret in cleartext, and return an attacker-chosen access token that the operator then uses against the ioFog Controller API. The same client secret is used elsewhere in the operator (`UpdateECNViewerClientRootURL`) to call Keycloak's `/admin/realms/{realm}/clients` API, so the stolen credential carries realm-management privileges. </details> <details> <summary><strong>Severity</strong></summary> Medium The defect is unconditional in code — every reconcile of every `ControlPlane` makes this request with verification disabled — but exploitation requires an attacker with an active MITM position on the operator→Keycloak network path. In deployments where Keycloak runs outside the cluster (public internet, separate VPC, managed identity provider) this is a realistic adversary; in same-namespace deployments the path is shorter. Per the rubric this matches "significant attacker positioning" → Medium, though it borders on High because it is a default that silently discards a security guarantee the administrator explicitly opted into by writing an `https://` URL. </details> <details> <summary><strong>Weakness</strong></summary> CWE-295: Improper Certificate Validation </details> <details> <summary><strong>Location</strong></summary> `controllers/controlplanes/k8s.go:429-480`, specifically lines 446-449: ```go tr := &http.Transport{ TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, } client := &http.Client{Transport: tr} ``` </details> <details> <summary><strong>Origin</strong></summary> Commit `be62967678bb4deb5066a4fa85cdd938d057bff7` — "login with kc client and port-manager env variables edited with kc variables", Emirhan Durmuş, 2024-04-01. The commit replaced username/password login to the Controller with OAuth2 client-credentials login to Keycloak and introduced the insecure transport. It entered the Eclipse repository via Datasance PR \#2 (`feature/kc-client`) → PR \#58 ("Datasance improvements", merged 2024-10-31) → subsequent Datasance→Eclipse merge PRs. </details> <details> <summary><strong>Methodology</strong></summary> I read every first-party `.go` file (24 files, ~6,600 LOC) and grepped for the canonical Go TLS-footgun patterns: `InsecureSkipVerify`, `http.NewRequest`, `http.Client{`, `tls.Config`, `exec.Command`, `os/exec`, `Unmarshal`, `MustParse`, `log.Fatal`. The only outbound network calls the operator makes are to (a) the Kubernetes API via controller-runtime, (b) the deployed Controller pod via `iofog-go-sdk`, and (c) Keycloak via two hand-rolled HTTP clients. Of those, `loginIofogClient` is the only one that explicitly disables verification in this repository's code. </details> <details> <summary><strong>Validation</strong></summary> Reproduction (`controllers/controlplanes/repro_insecure_tls_test.go`) starts an `httptest.NewTLSServer` (self-signed, untrusted certificate), configures a `ControlPlaneReconciler` with `spec.auth.url` pointing at it, and calls `loginIofogClient`: ```go package controllers import ( "io" "net/http" "net/http/httptest" "strings" "testing" iofogclient "github.com/eclipse-iofog/iofog-go-sdk/v3/pkg/client" cpv3 "github.com/eclipse-iofog/iofog-operator/v3/apis/controlplanes/v3" "github.com/go-logr/logr" ) func TestLoginIofogClientAcceptsUntrustedTLS(t *testing.T) { var capturedBody string srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { body, _ := io.ReadAll(r.Body) capturedBody = string(body) w.Header().Set("Content-Type", "application/json") _, _ = w.Write([]byte(`{"access_token":"attacker-supplied-token"}`)) })) defer srv.Close() r := &ControlPlaneReconciler{ Log: logr.Discard(), log: logr.Discard(), cp: cpv3.ControlPlane{Spec: cpv3.ControlPlaneSpec{Auth: cpv3.Auth{ URL: srv.URL + "/", Realm: "myrealm", ControllerClient: "pot-controller", ControllerSecret: "SUPER-SECRET-CLIENT-CREDENTIAL", }}}, } client := &iofogclient.Client{} err := r.loginIofogClient(client) if err != nil { t.Fatalf("expected loginIofogClient to (incorrectly) succeed against an untrusted TLS server, got error: %v", err) } if !strings.Contains(capturedBody, "client_secret=SUPER-SECRET-CLIENT-CREDENTIAL") { t.Fatalf("expected MITM to capture client_secret, got: %q", capturedBody) } t.Logf("MITM captured request body: %s", capturedBody) if client.GetAccessToken() != "attacker-supplied-token" { t.Fatalf("expected attacker token stored, got: %q", client.GetAccessToken()) } t.Logf("attacker-controlled access_token stored: %q", client.GetAccessToken()) _, defaultErr := http.Get(srv.URL) if defaultErr == nil { t.Fatalf("sanity: default http.Client should reject self-signed cert") } t.Logf("sanity OK: default http.Client rejected with: %v", defaultErr) } ``` Observed output (`go test -v -run TestLoginIofogClientAcceptsUntrustedTLS ./controllers/controlplanes/`): ``` === RUN TestLoginIofogClientAcceptsUntrustedTLS repro_insecure_tls_test.go:58: MITM captured request body: grant_type=client_credentials&client_id=pot-controller&client_secret=SUPER-SECRET-CLIENT-CREDENTIAL repro_insecure_tls_test.go:64: attacker-controlled access_token was stored on iofogClient: "attacker-supplied-token" repro_insecure_tls_test.go:71: sanity check OK: default http.Client rejected the same server with: Get "https://127.0.0.1:45357": tls: failed to verify certificate: x509: certificate signed by unknown authority 2026/04/28 12:37:51 http: TLS handshake error from 127.0.0.1:59722: remote error: tls: bad certificate --- PASS: TestLoginIofogClientAcceptsUntrustedTLS (0.04s) PASS ``` **Mitigations checked and ruled out:** - No CA-bundle or pinning configuration is read anywhere in the function. - The `spec.auth.ssl` field exists in the CRD but is only forwarded to the Controller pod as `KC_SSL_REQ`; it does not gate the operator's own transport. - The sibling function `UpdateECNViewerClientRootURL` (`internal/util/update_openid_client.go`) sends the same secret to the same Keycloak host using `golang.org/x/oauth2/clientcredentials` with the **default** transport, which **does** verify certificates — so the project has no consistent "Keycloak uses self-signed certs" assumption that would justify the skip. - `.golangci.yaml` enables `gosec` but the project's lint pipeline evidently does not block on G402 (it would fire here). </details> <details> <summary><strong>Detail</strong></summary> `loginIofogClient` is invoked from both `reconcileIofogController` (reconcile.go:256) and `reconcileNats` (reconcile.go:566), which run on every `ControlPlane` reconcile while the resource is in `deploying` or `updating` state — i.e. repeatedly during initial deployment and on every spec change. The function: 1. Reads `auth.URL`, `auth.Realm`, `auth.ControllerClient`, `auth.ControllerSecret` from the CR spec (k8s.go:430-433). 2. Builds the token URL `"{authURL}realms/{realm}/protocol/openid-connect/token"` (k8s.go:441). 3. Constructs the form body `grant_type=client_credentials&client_id={id}&client_secret={secret}` (k8s.go:443) — note the secret is not URL-encoded, a separate correctness bug. 4. Creates an `http.Client` whose transport has `InsecureSkipVerify: true` (k8s.go:446-449). 5. POSTs the body and decodes `access_token` from the JSON response (k8s.go:460-478). Because step 4 disables hostname and chain verification, any actor able to terminate TLS on the path — rogue Wi-Fi, compromised ingress, BGP hijack, malicious DNS — receives the secret and can return an arbitrary token. The captured `controllerSecret` is the same credential used in `internal/util/update_openid_client.go:54-77` to obtain a token that calls `GET /admin/realms/{realm}/clients` and `PUT /admin/realms/{realm}/clients/{id}`, demonstrating it is provisioned with Keycloak realm-admin scope. Theft therefore yields control over the Keycloak realm: the attacker can list all OAuth clients, rotate secrets, change redirect URIs, or modify user/role mappings for every application in that realm. </details> <details> <summary><strong>Prior Art</strong></summary> Searched: - `git log --all -S "InsecureSkipVerify"` — only the introducing commit `be629676` and unrelated vendored-dependency churn. - `git log --all --grep` for `TLS`, `verify`, `certificate` — no remediation commits. - `gh issue list` / `gh pr list` on `eclipse-iofog/iofog-operator` for `TLS`, `InsecureSkipVerify`, `certificate` — no results. - `Datasance/iofog-operator` (upstream of the merged code) has issues disabled. No prior public report found. Related but **out of scope**: `github.com/eclipse-iofog/iofog-go-sdk/v3@v3.7.0/pkg/client/http.go:74` also sets `InsecureSkipVerify: true`, meaning the operator's calls to the ioFog Controller via that SDK are likewise unverified. That is a separate package and is noted here only because the attacker-supplied token from this finding is consumed by that client. # Other patterns examined and ruled out The trust boundary for this operator is the Kubernetes RBAC boundary: anyone who can create a `ControlPlane` or `Application` CR is, per `config/operator/rbac.yaml` and the documented deployment model, already a namespace admin who could create Deployments, Secrets, and RoleBindings directly. CR spec fields are therefore operator-authored configuration, not attacker input. The remaining external surfaces are network responses from Keycloak and from the deployed Controller pod. | Location | Pattern | Why ruled out | |---|---|---| | `internal/util/certs/certs.go:46-71` | `log.Fatal` on malformed PEM in CA secret → operator process exit | The CA secrets (`router-site-ca`, `nats-site-ca`, etc.) are created by the operator itself; mutating them requires K8s `secrets:update` in the namespace, which is on the trusted side of the boundary. Self-DoS by an admin, not an external attacker. | | `controllers/controlplanes/k8s.go:640` | Unchecked type assertion `entry[0].(string)` in `mergeConfigs` → panic | `existingConfig` is read from the `iofog-router` ConfigMap. Only the operator and the Controller pod (which the operator deploys) have write access; both are inside the trust domain. Panic is also caught by `recover()` in callers of secret-creation paths but not here — still, no untrusted writer. | | `controllers/controlplanes/microservices.go:1059` | `resource.MustParse(cfg.storageSize)` panics on bad input; runs in an unrecovered goroutine (`go reconcileRoutine`) → crashes the whole process | `storageSize` is `spec.nats.jetStream.storageSize` from the CR. CR author is the cluster admin; crashing their own operator is not a security finding. | | `controllers/controlplanes/k8s.go:443` | `fmt.Sprintf` builds form body without `url.QueryEscape` on `clientID`/`clientSecret` | Values come from CR spec (admin). Malformed encoding only breaks the admin's own login. | | `controllers/controlplanes/nats/config.go:181-205` | `BuildServerConf` does `strings.NewReplacer` substitution of `OperatorJWT`, `LeafAdvertise`, `ControllerName`, `JetStreamDomain` into the NATS `server.conf` without quoting/escaping (only `JetStreamKey`/`JetStreamPrev` are escaped) | All inputs are either CR-spec-derived (admin), K8s namespace name (admin), LB-allocated address (cloud provider), or the Controller's `/api/v3/nats/bootstrap` response. The Controller pod is deployed by this operator with an image the admin chose; it sits inside the trust domain. Config-injection here would require compromising the Controller first, at which point the attacker already has the namespace's secrets via the Controller's own RBAC (`secrets: get,list,watch`). | | `controllers/controlplanes/router/config.go:8-19` | `<NAMESPACE>` substituted into JSON router config via `strings.NewReplacer` without JSON escaping | Namespace names are constrained by Kubernetes to `[a-z0-9-]` (RFC 1123), cannot contain quotes or backslashes. | | `internal/util/update_openid_client.go:54-77,87-104,131-168` | Hand-built Keycloak admin API calls with `fmt.Sprintf` URL construction from `auth.URL`, `auth.Realm`, looked-up `clientID` | `auth.URL`/`auth.Realm` are CR spec (admin). The looked-up `clientID` comes from Keycloak's response — if Keycloak returns a malicious `id` containing `../`, the PUT URL could be redirected, but the request still goes to the admin's own Keycloak with the admin's own token; no cross-tenant boundary. Uses default TLS transport (verified). | | `controllers/controlplanes/controller.go:33-39` + `states.go:62-70` | `ControlPlaneReconciler.cp` is a struct field shared across three goroutines spawned per reconcile | All goroutine accesses are reads; `MaxConcurrentReconciles` defaults to 1 so no inter-reconcile races. Data race on read-only struct is a code-smell, not exploitable. | | `controllers/apps/controller.go:154-237` | `Application` CR → Deployment with `Image: fmt.Sprintf("%s, %s", x86, arm)` | Produces an unschedulable container (image string with comma is invalid), but CR author is admin. The annotation `microservices` carries arbitrary JSON into pod annotations — visible to anything that can read pods, but again admin-authored. | | `controllers/controlplanes/reconcile.go:382` | `fmt.Sprintf("%v://%s:%d/api/v3", scheme, host, port)` then `url.Parse` | `host` is either an in-cluster DNS name the operator constructs or an LB address from the K8s API. Not attacker-controlled. | | Whole codebase | `os/exec`, `syscall`, `unsafe`, YAML/gob/XML deserialisation, SQL, template execution, file writes with dynamic paths | None present. The operator only talks to the K8s API and two HTTP endpoints. | Files reviewed: all 24 first-party `.go` files under `apis/`, `controllers/`, `internal/`, and `main.go` (excluding `zz_generated.deepcopy.go` boilerplate, which was skimmed). Weakness classes checked: command injection, deserialisation, path traversal, SSRF, SQLi, template injection, TLS misconfiguration, missing authn/authz, type-confusion panics, unbounded resource consumption. --- **Verification verdict:** TRUE POSITIVE ### Verification details All evidence was already gathered in the working session; here is the formal phase documentation and gated verdict. </details> <details> <summary><strong>Phase 1 — Data Flow Analysis</strong></summary> **Source → sink trace (HEAD `88275613`):** 1. Source: `ControlPlane` CR spec fields `auth.issuerUrl`, `auth.client.id`, `auth.client.secret` (external mode) and `auth.bootstrap.username`/password (embedded mode) — admin-authored configuration containing real credentials (`apis/controlplanes/v3/controlplane_types.go:144-165`). 2. `loginIofogClient` (`controllers/controlplanes/k8s.go:546-584`) dispatches on `auth.mode` and passes credentials verbatim to `controllerlogin.ExternalClientCredentialsLogin` or `EmbeddedBootstrapLogin`. 3. Sink: both functions call `httpClient()` (`external.go:122-127`), whose transport is `insecureHTTPTransport()` → `tls.Config{InsecureSkipVerify: true}` (`http.go:8-12`). External mode sends the OIDC discovery GET *and* the credential POST through it; embedded mode sends the bootstrap password through it. **Trust boundaries:** the CR spec is inside the trust boundary (cluster admin). The untrusted party is the network path between the operator pod and the TLS peer — exactly the boundary certificate verification is supposed to enforce, and it is disabled. **API contracts:** Go's `crypto/tls` contract is explicit that `InsecureSkipVerify: true` accepts any certificate and any hostname; no compensating check (`VerifyConnection`, pinning) is present. **Environment protections:** none — grep for `RootCAs`/`CABundle`/pinning found nothing; the CRD's `insecureAllowHttp` flag only becomes a Controller env var (`microservices.go:259-262`) and does not gate the operator's transport. **Cross-references:** call sites `reconcile.go:349` and `reconcile.go:637` fire on every reconcile of a non-ready ControlPlane and on every spec change, so the request recurs. </details> <details> <summary><strong>Phase 2 — Exploitability Verification</strong></summary> **Attacker control: CONFIRMED.** An on-path attacker (or DNS poisoner) terminating TLS toward the configured `issuerUrl` receives the full credential POST and controls the entire response — including, via the unverified discovery document, the `token_endpoint` URL itself. Proven by executed PoC (Phase 4). **Mathematical bounds: N/A** — no arithmetic or size conditions gate the flaw; it is unconditional on every TLS connection made by this package. **Race conditions: N/A** — no timing window is needed; the reconcile loop repeats the request indefinitely, so the attacker needs no race at all. **Adversarial analysis:** the only preconditions are (a) the admin configured an `https://` endpoint (the sample CR documents `issuerUrl: https://idp.example.com/realms/myrealm` as the intended external-mode configuration) and (b) attacker network positioning. Neither involves credentials or a cluster foothold. </details> <details> <summary><strong>Phase 3 — Impact Assessment</strong></summary> **Real security impact, not operational robustness:** (a) **information disclosure of long-lived credentials** — the OAuth client secret (external mode) or the Controller bootstrap admin password (embedded mode) is disclosed to the MITM; (b) **authentication-token injection** — the attacker's `access_token` is stored on the `iofogclient.Client` the operator subsequently uses against the ioFog Controller API, letting the attacker shape that interaction. **Primary control vs defense-in-depth:** certificate verification is the *primary* authentication control for the TLS channel; there is no secondary control (no pinning, no mutual TLS, no token audience binding checked operator-side), so its removal is a primary-control failure. **Reassessed against HEAD's provisioning model:** the report's "realm-admin scope" leg is stale — `internal/util/update_openid_client.go` was deleted in the v3.8 rewrite, so demonstrable impact is now "credentials granting Controller-API access," not Keycloak realm administration. That trims the blast radius but leaves full control-plane credential theft, which is squarely a security impact. </details> <details> <summary><strong>Phase 4 — PoC Creation (completed earlier, restated)</strong></summary> Executable unit tests at `repo/internal/auth/controllerlogin/poc_insecure_tls_verify_test.go`, both **PASS** against HEAD: the external-mode test shows a self-signed `httptest.NewTLSServer` capturing `client_id=pot-controller&client_secret=SUPER-SECRET&grant_type=client_credentials` and its `"attacker-token"` being stored on the client; the embedded-mode test captures `{"email":"admin@example.com","password":"hunter2"}`. **Negative PoC:** the same test proves a default `http.Client` rejects the identical server with `x509: certificate signed by unknown authority` — establishing that the custom transport, not the environment, is what admits the attacker. Pseudocode PoC is subsumed by the executable one. </details> <details> <summary><strong>Phase 5 — Devil's Advocate (13-point checklist)</strong></summary> 1. **Full validation chain traced?** Yes — `loginIofogClient` validates only *presence* of fields, never the TLS peer; no upstream check compensates. 2. **Conditional reachability?** The sink runs on every reconcile in both auth modes; no condition disables the insecure transport. 3. **Defensive programming?** The `#nosec G402` comment is a lint suppression, not a defense — it documents intent, it doesn't mitigate. 4. **Confirmed data path?** Executed PoC, not assumption. 5. **Data source context?** Credentials are genuine secrets (sample CR + `passwordSecretRef` support prove they're production credentials, not test fixtures). 6. **Bounds logic?** N/A. 7. **TOCTOU?** N/A. 8. **API contract?** `crypto/tls` documents that this flag removes all peer authentication — the contract confirms, not refutes, the finding. 9. **Internal-storage confusion?** No — the attacker input is the network peer's certificate/response, not the CR. 10. **Pattern-matching risk?** Ruled out: I verified the specific instance end-to-end on HEAD rather than trusting the report's (now-stale) line numbers. 11. **Concurrency assumptions?** None needed. 12. **Real vs theoretical?** Real if the path crosses an attacker-reachable segment; the documented external-IdP topology makes that the normal case. If Keycloak were reached over plain `http://` ClusterIP the TLS skip would be moot — but then there is no TLS at all, which is the same weakness in a different coat. 13. **Deliberate design?** The strongest counterargument: maintainers annotated it intentionally ("Controller/IdP commonly use self-signed TLS"). Rejected as a false-positive ground because the skip is unconditional and un-opt-out-able — an admin with a publicly-trusted certificate cannot get verification, and the CRD's own "(dev only)" insecure flags show secure-by-default was the design intent. A deliberate CWE-295 is still CWE-295. </details> <details> <summary><strong>Gate Review</strong></summary> | Gate | Result | Evidence | |---|---|---| | 1. Process | **PASS** | All phases documented above with file:line evidence | | 2. Reachability | **PASS** | Sink runs on every reconcile in both auth modes; PoC drives the exact HEAD functions | | 3. Real Impact | **PASS** | Credential disclosure + token injection = info disclosure and control-plane compromise, not robustness | | 4. PoC Validation | **PASS** | Two executed passing tests show control, trigger, and impact; negative PoC isolates the cause | | 5. Math Bounds | **PASS (N/A)** | No arithmetic condition gates the flaw; unconditional on every connection | | 6. Environment | **PASS** | No CA-bundle, pinning, or config flag anywhere prevents exploitation; the only "insecure" flags gate other behavior | </details> <details> <summary><strong>Verdict</strong></summary> **BUG \#1 TRUE POSITIVE — TLS certificate verification unconditionally disabled (`InsecureSkipVerify: true`, CWE-295) in `internal/auth/controllerlogin/http.go:10` on HEAD (refactored from the reported `k8s.go` location), leaking the OAuth client secret / bootstrap admin password to any on-path attacker and accepting attacker-supplied access tokens; deliberately suppressed by the maintainers via `#nosec G402`, with no CA-bundle or opt-out available.** Counts: 1 TRUE POSITIVE, 0 FALSE POSITIVES. Severity: report's Medium is defensible for the documented external-IdP topology (Low–Medium range given the MITM positioning requirement). Report corrections for disclosure: cite the new location, drop the realm-admin scope claim (code removed in v3.8), and note the discovery-endpoint and embedded-bootstrap legs that widen the finding on HEAD. EXPOSURE: REMOTE An unauthenticated attacker with on-path network positioning (or DNS control) between the operator pod and the configured IdP/Controller endpoint captures the credentials and injects tokens; no valid credentials or cluster foothold are required. SCOPE: PRODUCTION The operator is shipped, deployed software — the README presents it as the "Kubernetes operator for Eclipse ioFog and Datasance PoT control planes" with published `ghcr.io` images and Helm charts; the project is actively maintained (v3.8 greenfield release) with no EOL, deprecation, or out-of-scope notice covering this code. </details> <details> <summary><strong>Steps to reproduce</strong></summary> 1. Cluster admin deploys a `ControlPlane` CR with `spec.auth.url: https://keycloak.example.com/` and a valid `controllerSecret`. 2. Attacker with on-path position (e.g. controls a hop between the cluster's egress and `keycloak.example.com`, or poisons DNS for that name) presents a self-signed certificate for `keycloak.example.com`. 3. Operator reconciles, connects, accepts the certificate, and POSTs: **Proof of concept:** [poc.txt](/uploads/41f7477ff5e55c10fcba1dd61a936f04/poc.txt) 4. Attacker records the secret. Attacker may either forward to real Keycloak (transparent proxy — operator behaves normally) or return `{"access_token":"<attacker token>"}` to influence the operator's subsequent calls to the ioFog Controller. 5. With the stolen secret, attacker independently obtains realm-admin tokens from Keycloak and reconfigures any client in the realm. The reconcile loop retries on failure and on every CR generation change, so the attacker does not need to win a one-shot race; the request recurs. </details> <details> <summary><strong>Do you know any mitigations of the issue?</strong></summary> Remove the custom `Transport` and use Go's default (which validates certificates), or replace the hand-rolled token request with the same `golang.org/x/oauth2/clientcredentials` flow already used in `UpdateECNViewerClientRootURL`. If self-signed Keycloak instances must be supported, accept a CA bundle via the CR spec and load it into `tls.Config.RootCAs` rather than disabling verification. <!-- l1-helper-dup: 813e24f907cc0ca5a971392635cdcadb23d128e3b2e906da266f1b33e4fd18c3 --> </details>