Skip to content
Snippets Groups Projects
Commit 3ec7cc3f authored by Athanasios Liatifis's avatar Athanasios Liatifis
Browse files

update to v1.1

parent 7628abf5
No related branches found
No related tags found
No related merge requests found
Showing
with 139 additions and 17762 deletions
......@@ -58,13 +58,128 @@ helm install kepler kepler/kepler --namespace enact --create-namespace \
--set serviceMonitor.enabled=true \
--set serviceMonitor.labels.release=prometheus
```
### Secret creation
Create a directory in the core cluster and execute the following.
#### Generate the CA
```bash
openssl req -x509 -new -nodes -key ca.key -sha256 -days 3650 -out ca.crt
```
#### Generate core Cluster Certificate
```bash
# Create key and CSR
openssl req -new -nodes -out core.csr -keyout core.key -subj "/CN=elasticsearch-master-core"
# Sign core CSR with CA
openssl x509 -req -in core.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out core.crt -days 365 -sha256
```
#### Generate remote Cluster Certificate
```bash
# Create key and CSR
openssl req -new -nodes -out remote.csr -keyout remote.key -subj "/CN=elasticsearch-master-remote"
# Sign Remote CSR with CA
openssl x509 -req -in remote.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out remote.crt -days 365 -sha256
```
Navigate to the directory containing your certificates (`ca.crt`, `core.key`, `core.crt`, etc.).
#### Core Cluster Secrets
```bash
kubectl create secret generic core-master-certs \
--from-file=tls.crt=core.crt \
--from-file=tls.key=core.key \
--from-file=ca.crt=ca.crt \
-n enact
```
#### Remote Cluster Secrets
```bash
kubectl create secret generic remote-master-certs \
--from-file=tls.crt=remote.crt \
--from-file=tls.key=remote.key \
--from-file=ca.crt=ca.crt \
-n enact
```
#### Shared CA Secret (for all clusters)
```bash
kubectl create secret generic elasticsearch-shared-ca \
--from-file=elastic-ca.crt=ca.crt \
-n enact
```
**Note:** The shared CA (`elasticsearch-shared-ca`) must be present in every cluster.
Copy `remote.crt`, `remote.key`, and `ca.crt` to the remote cluster.
Create:
```bash
kubectl create secret generic remote-master-certs \
--from-file=tls.crt=remote.crt \
--from-file=tls.key=remote.key \
--from-file=ca.crt=ca.crt \
-n enact
```
and
```bash
kubectl create secret generic elasticsearch-shared-ca \
--from-file=elastic-ca.crt=ca.crt \
-n enact
```
---
### Configure Elasticsearch and Metricbeat values
### Elasticsearch Configuration
In the `values.yaml` of each Elasticsearch cluster, add the appropriate `secretMounts` entry.
> **Note:**
> - In the core cluster, also add the secrets of all remote clusters.
> - Prefer using this path for local cluster secret: `/usr/share/elasticsearch/config/certs`
> - For remote cluster certificates, use: `/usr/share/elasticsearch/config/remote-cas`
```yaml
secretMounts:
- name: core-master-certs # Replace with the appropriate name for each cluster
secretName: core-master-certs
path: /usr/share/elasticsearch/config/certs
```
---
### Metricbeat Configuration
For each cluster, update the `secretMounts` in both the **DaemonSet** section (not the Deployment) and the **global** section.
Use the corresponding secret name per cluster:
```yaml
secretMounts:
- name: core-master-certs # Replace with the appropriate name for each cluster
secretName: core-master-certs
path: /usr/share/metricbeat/certs/
```
### Install ELK stack using helmfile
Helmfile is a tool that will help in deploying ELK stack. To install helmfile:
```
2. helm plugin install https://github.com/databus23/helm-diff
helm plugin install https://github.com/databus23/helm-diff
```
To install elastic-stack on Linux machines:
......@@ -106,6 +221,27 @@ helm upgrade --install tdcme-api-topology ./topology_exporter/chart/ --namespace
helm uninstall -n enact elasticsearch infra kepler kibana metricbeat tdcme-api tdcme-api-topology
```
## Cluster Registration/Removal
To register a remote cluster with the API, use the following `curl` command:
```bash
curl -X PUT "http://<api-node-ip>:32554/clusters" \
-H "Content-Type: application/json" \
-d '{
"cluster_name": "<cluter-name>",
"proxy_address": "<remote-node-ip>:<transport-nodeport>",
"host": "https://<remote-node-ip>:32555",
"username": "elastic",
"password": "4w2v4DFXtzL4M1oMJ2Y5"
}'
```
To remove a registered cluster:
```bash
curl -X DELETE "http://<api-node-ip>:32554/clusters/<cluster-name>"
```
---
## API Endpoints
| Endpoint | Description |
|--------------------------------------------------------|----------------------------------------------------------------------------------------|
......@@ -118,6 +254,7 @@ helm uninstall -n enact elasticsearch infra kepler kibana metricbeat tdcme-api t
| http://<endpoint\>:32554/namespaces/<namespace\>/pods | returns all pods under the <namespace\>. |
| http://<endpoint\>:32554/nodes | return all registered nodes |
| http://<endpoint\>:32553/topology | return the Pod topology graph |
| http://<endpoint\>:32554/pod/<pod_name\>/<container_name\>/metrics | returns CPU and memory metrics of a container inside the specified pod. |
| http://<endpoint\>:32554/pod/<pod_name\>/<container_name\>/metrics | returns CPU and memory metrics of a container inside the specified pod. |
| http://<endpoint\>:32554/pod/<pod_name\>/energy | returns the energy consumption of the specified pod. |
| http://<endpoint\>:32554/latency/<source_node\>/<destination-node\> | returns the latency between two Kubernetes nodes across the Edge-to-Cloud continuum |
\ No newline at end of file
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.github/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*.orig
*~
# Various IDEs
.project
.idea/
*.tmproj
.vscode/
annotations:
artifacthub.io/crds: "- kind: CiliumNetworkPolicy\n version: v2\n name: ciliumnetworkpolicies.cilium.io\n
\ displayName: Cilium Network Policy\n description: |\n Cilium Network Policies
provide additional functionality beyond what\n is provided by standard Kubernetes
NetworkPolicy such as the ability\n to allow traffic based on FQDNs, or to
filter at Layer 7.\n- kind: CiliumClusterwideNetworkPolicy\n version: v2\n name:
ciliumclusterwidenetworkpolicies.cilium.io\n displayName: Cilium Clusterwide
Network Policy\n description: |\n Cilium Clusterwide Network Policies support
configuring network traffic\n policiies across the entire cluster, including
applying node firewalls.\n- kind: CiliumExternalWorkload\n version: v2\n name:
ciliumexternalworkloads.cilium.io\n displayName: Cilium External Workload\n description:
|\n Cilium External Workload supports configuring the ability for external\n
\ non-Kubernetes workloads to join the cluster.\n- kind: CiliumLocalRedirectPolicy\n
\ version: v2\n name: ciliumlocalredirectpolicies.cilium.io\n displayName: Cilium
Local Redirect Policy\n description: |\n Cilium Local Redirect Policy allows
local redirects to be configured\n within a node to support use cases like
Node-Local DNS or KIAM.\n- kind: CiliumNode\n version: v2\n name: ciliumnodes.cilium.io\n
\ displayName: Cilium Node\n description: |\n Cilium Node represents a node
managed by Cilium. It contains a\n specification to control various node specific
configuration aspects\n and a status section to represent the status of the
node.\n- kind: CiliumIdentity\n version: v2\n name: ciliumidentities.cilium.io\n
\ displayName: Cilium Identity\n description: |\n Cilium Identity allows introspection
into security identities that\n Cilium allocates which identify sets of labels
that are assigned to\n individual endpoints in the cluster.\n- kind: CiliumEndpoint\n
\ version: v2\n name: ciliumendpoints.cilium.io\n displayName: Cilium Endpoint\n
\ description: |\n Cilium Endpoint represents the status of individual pods
or nodes in\n the cluster which are managed by Cilium, including enforcement
status,\n IP addressing and whether the networking is successfully operational.\n-
kind: CiliumEndpointSlice\n version: v2alpha1\n name: ciliumendpointslices.cilium.io\n
\ displayName: Cilium Endpoint Slice\n description: |\n Cilium Endpoint Slice
represents the status of groups of pods or nodes\n in the cluster which are
managed by Cilium, including enforcement status,\n IP addressing and whether
the networking is successfully operational.\n- kind: CiliumEgressGatewayPolicy\n
\ version: v2\n name: ciliumegressgatewaypolicies.cilium.io\n displayName: Cilium
Egress Gateway Policy\n description: |\n Cilium Egress Gateway Policy provides
control over the way that traffic\n leaves the cluster and which source addresses
to use for that traffic.\n- kind: CiliumClusterwideEnvoyConfig\n version: v2\n
\ name: ciliumclusterwideenvoyconfigs.cilium.io\n displayName: Cilium Clusterwide
Envoy Config\n description: |\n Cilium Clusterwide Envoy Config specifies
Envoy resources and K8s service mappings\n to be provisioned into Cilium host
proxy instances in cluster context.\n- kind: CiliumEnvoyConfig\n version: v2\n
\ name: ciliumenvoyconfigs.cilium.io\n displayName: Cilium Envoy Config\n description:
|\n Cilium Envoy Config specifies Envoy resources and K8s service mappings\n
\ to be provisioned into Cilium host proxy instances in namespace context.\n-
kind: CiliumBGPPeeringPolicy\n version: v2alpha1\n name: ciliumbgppeeringpolicies.cilium.io\n
\ displayName: Cilium BGP Peering Policy\n description: |\n Cilium BGP Peering
Policy instructs Cilium to create specific BGP peering\n configurations.\n-
kind: CiliumBGPClusterConfig\n version: v2alpha1\n name: ciliumbgpclusterconfigs.cilium.io\n
\ displayName: Cilium BGP Cluster Config\n description: |\n Cilium BGP Cluster
Config instructs Cilium operator to create specific BGP cluster\n configurations.\n-
kind: CiliumBGPPeerConfig\n version: v2alpha1\n name: ciliumbgppeerconfigs.cilium.io\n
\ displayName: Cilium BGP Peer Config\n description: |\n CiliumBGPPeerConfig
is a common set of BGP peer configurations. It can be referenced \n by multiple
peers from CiliumBGPClusterConfig.\n- kind: CiliumBGPAdvertisement\n version:
v2alpha1\n name: ciliumbgpadvertisements.cilium.io\n displayName: Cilium BGP
Advertisement\n description: |\n CiliumBGPAdvertisement is used to define
source of BGP advertisement as well as BGP attributes \n to be advertised with
those prefixes.\n- kind: CiliumBGPNodeConfig\n version: v2alpha1\n name: ciliumbgpnodeconfigs.cilium.io\n
\ displayName: Cilium BGP Node Config\n description: |\n CiliumBGPNodeConfig
is read only node specific BGP configuration. It is constructed by Cilium operator.\n
\ It will also contain node local BGP state information.\n- kind: CiliumBGPNodeConfigOverride\n
\ version: v2alpha1\n name: ciliumbgpnodeconfigoverrides.cilium.io\n displayName:
Cilium BGP Node Config Override\n description: |\n CiliumBGPNodeConfigOverride
can be used to override node specific BGP configuration.\n- kind: CiliumLoadBalancerIPPool\n
\ version: v2alpha1\n name: ciliumloadbalancerippools.cilium.io\n displayName:
Cilium Load Balancer IP Pool\n description: |\n Defining a Cilium Load Balancer
IP Pool instructs Cilium to assign IPs to LoadBalancer Services.\n- kind: CiliumNodeConfig\n
\ version: v2alpha1\n name: ciliumnodeconfigs.cilium.io\n displayName: Cilium
Node Configuration\n description: |\n CiliumNodeConfig is a list of configuration
key-value pairs. It is applied to\n nodes indicated by a label selector.\n-
kind: CiliumCIDRGroup\n version: v2alpha1\n name: ciliumcidrgroups.cilium.io\n
\ displayName: Cilium CIDR Group\n description: |\n CiliumCIDRGroup is a list
of CIDRs that can be referenced as a single entity from CiliumNetworkPolicies.\n-
kind: CiliumL2AnnouncementPolicy\n version: v2alpha1\n name: ciliuml2announcementpolicies.cilium.io\n
\ displayName: Cilium L2 Announcement Policy\n description: |\n CiliumL2AnnouncementPolicy
is a policy which determines which service IPs will be announced to\n the local
area network, by which nodes, and via which interfaces.\n- kind: CiliumPodIPPool\n
\ version: v2alpha1\n name: ciliumpodippools.cilium.io\n displayName: Cilium
Pod IP Pool\n description: |\n CiliumPodIPPool defines an IP pool that can
be used for pooled IPAM (i.e. the multi-pool IPAM mode).\n"
apiVersion: v2
appVersion: 1.15.6
description: eBPF-based Networking, Security, and Observability
home: https://cilium.io/
icon: https://cdn.jsdelivr.net/gh/cilium/cilium@main/Documentation/images/logo-solo.svg
keywords:
- BPF
- eBPF
- Kubernetes
- Networking
- Security
- Observability
- Troubleshooting
kubeVersion: '>= 1.16.0-0'
name: cilium
sources:
- https://github.com/cilium/cilium
version: 1.15.6
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "{}"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright {yyyy} Authors of Cilium
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
This diff is collapsed.
{{ template "chart.header" . }}
{{ template "chart.deprecationWarning" . }}
{{ template "chart.versionBadge" . }}{{ template "chart.typeBadge" . }}{{ template "chart.appVersionBadge" . }}
Cilium is open source software for providing and transparently securing
network connectivity and loadbalancing between application workloads such as
application containers or processes. Cilium operates at Layer 3/4 to provide
traditional networking and security services as well as Layer 7 to protect and
secure use of modern application protocols such as HTTP, gRPC and Kafka.
A new Linux kernel technology called eBPF is at the foundation of Cilium.
It supports dynamic insertion of eBPF bytecode into the Linux kernel at various
integration points such as: network IO, application sockets, and tracepoints
to implement security, networking and visibility logic. eBPF is highly
efficient and flexible.
![Cilium feature overview](https://raw.githubusercontent.com/cilium/cilium/main/Documentation/images/cilium_overview.png)
## Prerequisites
* Kubernetes: `{{ template "chart.kubeVersion" . }}`
* Helm: `>= 3.0`
## Getting Started
Try Cilium on any Kubernetes distribution in under 15 minutes:
| Minikube | Self-Managed K8s | Amazon EKS | Google GKE | Microsoft AKS |
|:-:|:-:|:-:|:-:|:-:|
| [![Minikube](https://raw.githubusercontent.com/cilium/charts/master/images/minikube.svg)](https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/) | [![Self-Managed Kubernetes](https://raw.githubusercontent.com/cilium/charts/master/images/k8s.png)](https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/) | [![Amazon EKS](https://raw.githubusercontent.com/cilium/charts/master/images/aws.svg)](https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/) | [![Google GKE](https://raw.githubusercontent.com/cilium/charts/master/images/google-cloud.svg)](https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/) | [![Microsoft AKS](https://raw.githubusercontent.com/cilium/charts/master/images/azure.svg)](https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/) |
Or, for a quick install with the default configuration:
```
$ helm repo add cilium https://helm.cilium.io/
$ helm install cilium cilium/cilium --namespace=kube-system
```
After Cilium is installed, you can explore the features that Cilium has to
offer from the [Getting Started Guides page](https://docs.cilium.io/en/stable/gettingstarted/).
{{ template "chart.maintainersSection" . }}
{{ template "chart.sourcesSection" . }}
## Getting Help
The best way to get help if you get stuck is to ask a question on the
[Cilium Slack channel](https://cilium.herokuapp.com/). With Cilium
contributors across the globe, there is almost always someone available to help.
{{ template "chart.valuesSection" . }}
set -o errexit
set -o pipefail
set -o nounset
# When running in AWS ENI mode, it's likely that 'aws-node' has
# had a chance to install SNAT iptables rules. These can result
# in dropped traffic, so we should attempt to remove them.
# We do it using a 'postStart' hook since this may need to run
# for nodes which might have already been init'ed but may still
# have dangling rules. This is safe because there are no
# dependencies on anything that is part of the startup script
# itself, and can be safely run multiple times per node (e.g. in
# case of a restart).
if [[ "$(iptables-save | grep -E -c 'AWS-SNAT-CHAIN|AWS-CONNMARK-CHAIN')" != "0" ]];
then
echo 'Deleting iptables rules created by the AWS CNI VPC plugin'
iptables-save | grep -E -v 'AWS-SNAT-CHAIN|AWS-CONNMARK-CHAIN' | iptables-restore
fi
echo 'Done!'
This diff is collapsed.
{
"node": {
"id": "host~127.0.0.1~no-id~localdomain",
"cluster": "ingress-cluster"
},
"staticResources": {
"listeners": [
{{- if and .Values.proxy.prometheus.enabled .Values.envoy.prometheus.enabled }}
{
"name": "envoy-prometheus-metrics-listener",
"address": {
"socket_address": {
"address": "0.0.0.0",
"port_value": {{ .Values.proxy.prometheus.port | default .Values.envoy.prometheus.port }}
}
},
"filter_chains": [
{
"filters": [
{
"name": "envoy.filters.network.http_connection_manager",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"stat_prefix": "envoy-prometheus-metrics-listener",
"route_config": {
"virtual_hosts": [
{
"name": "prometheus_metrics_route",
"domains": [
"*"
],
"routes": [
{
"name": "prometheus_metrics_route",
"match": {
"prefix": "/metrics"
},
"route": {
"cluster": "/envoy-admin",
"prefix_rewrite": "/stats/prometheus"
}
}
]
}
]
},
"http_filters": [
{
"name": "envoy.filters.http.router",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
}
}
],
"stream_idle_timeout": "0s"
}
}
]
}
]
},
{{- end }}
{
"name": "envoy-health-listener",
"address": {
"socket_address": {
"address": {{ .Values.ipv4.enabled | ternary "127.0.0.1" "::1" | quote }},
"port_value": {{ .Values.envoy.healthPort }}
}
},
{{- if and .Values.ipv4.enabled .Values.ipv6.enabled }}
"additional_addresses": [
{
"address": {
"socket_address": {
"address": "::1",
"port_value": {{ .Values.envoy.healthPort }}
}
}
}
],
{{- end }}
"filter_chains": [
{
"filters": [
{
"name": "envoy.filters.network.http_connection_manager",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"stat_prefix": "envoy-health-listener",
"route_config": {
"virtual_hosts": [
{
"name": "health",
"domains": [
"*"
],
"routes": [
{
"name": "health",
"match": {
"prefix": "/healthz"
},
"route": {
"cluster": "/envoy-admin",
"prefix_rewrite": "/ready"
}
}
]
}
]
},
"http_filters": [
{
"name": "envoy.filters.http.router",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
}
}
],
"stream_idle_timeout": "0s"
}
}
]
}
]
}
],
"clusters": [
{
"name": "ingress-cluster",
"type": "ORIGINAL_DST",
"connectTimeout": "{{ .Values.envoy.connectTimeoutSeconds }}s",
"lbPolicy": "CLUSTER_PROVIDED",
"typedExtensionProtocolOptions": {
"envoy.extensions.upstreams.http.v3.HttpProtocolOptions": {
"@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions",
"commonHttpProtocolOptions": {
"idleTimeout": "{{ .Values.envoy.idleTimeoutDurationSeconds }}s",
"maxConnectionDuration": "{{ .Values.envoy.maxConnectionDurationSeconds }}s",
"maxRequestsPerConnection": {{ .Values.envoy.maxRequestsPerConnection }}
},
"useDownstreamProtocolConfig": {}
}
},
"cleanupInterval": "{{ .Values.envoy.connectTimeoutSeconds }}.500s"
},
{
"name": "egress-cluster-tls",
"type": "ORIGINAL_DST",
"connectTimeout": "{{ .Values.envoy.connectTimeoutSeconds }}s",
"lbPolicy": "CLUSTER_PROVIDED",
"typedExtensionProtocolOptions": {
"envoy.extensions.upstreams.http.v3.HttpProtocolOptions": {
"@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions",
"commonHttpProtocolOptions": {
"idleTimeout": "{{ .Values.envoy.idleTimeoutDurationSeconds }}s",
"maxConnectionDuration": "{{ .Values.envoy.maxConnectionDurationSeconds }}s",
"maxRequestsPerConnection": {{ .Values.envoy.maxRequestsPerConnection }}
},
"upstreamHttpProtocolOptions": {},
"useDownstreamProtocolConfig": {}
}
},
"cleanupInterval": "{{ .Values.envoy.connectTimeoutSeconds }}.500s",
"transportSocket": {
"name": "cilium.tls_wrapper",
"typedConfig": {
"@type": "type.googleapis.com/cilium.UpstreamTlsWrapperContext"
}
}
},
{
"name": "egress-cluster",
"type": "ORIGINAL_DST",
"connectTimeout": "{{ .Values.envoy.connectTimeoutSeconds }}s",
"lbPolicy": "CLUSTER_PROVIDED",
"typedExtensionProtocolOptions": {
"envoy.extensions.upstreams.http.v3.HttpProtocolOptions": {
"@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions",
"commonHttpProtocolOptions": {
"idleTimeout": "{{ .Values.envoy.idleTimeoutDurationSeconds }}s",
"maxConnectionDuration": "{{ .Values.envoy.maxConnectionDurationSeconds }}s",
"maxRequestsPerConnection": {{ .Values.envoy.maxRequestsPerConnection }}
},
"useDownstreamProtocolConfig": {}
}
},
"cleanupInterval": "{{ .Values.envoy.connectTimeoutSeconds }}.500s"
},
{
"name": "ingress-cluster-tls",
"type": "ORIGINAL_DST",
"connectTimeout": "{{ .Values.envoy.connectTimeoutSeconds }}s",
"lbPolicy": "CLUSTER_PROVIDED",
"typedExtensionProtocolOptions": {
"envoy.extensions.upstreams.http.v3.HttpProtocolOptions": {
"@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions",
"commonHttpProtocolOptions": {
"idleTimeout": "{{ .Values.envoy.idleTimeoutDurationSeconds }}s",
"maxConnectionDuration": "{{ .Values.envoy.maxConnectionDurationSeconds }}s",
"maxRequestsPerConnection": {{ .Values.envoy.maxRequestsPerConnection }}
},
"upstreamHttpProtocolOptions": {},
"useDownstreamProtocolConfig": {}
}
},
"cleanupInterval": "{{ .Values.envoy.connectTimeoutSeconds }}.500s",
"transportSocket": {
"name": "cilium.tls_wrapper",
"typedConfig": {
"@type": "type.googleapis.com/cilium.UpstreamTlsWrapperContext"
}
}
},
{
"name": "xds-grpc-cilium",
"type": "STATIC",
"connectTimeout": "{{ .Values.envoy.connectTimeoutSeconds }}s",
"loadAssignment": {
"clusterName": "xds-grpc-cilium",
"endpoints": [
{
"lbEndpoints": [
{
"endpoint": {
"address": {
"pipe": {
"path": "/var/run/cilium/envoy/sockets/xds.sock"
}
}
}
}
]
}
]
},
"typedExtensionProtocolOptions": {
"envoy.extensions.upstreams.http.v3.HttpProtocolOptions": {
"@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions",
"explicitHttpConfig": {
"http2ProtocolOptions": {}
}
}
}
},
{
"name": "/envoy-admin",
"type": "STATIC",
"connectTimeout": "{{ .Values.envoy.connectTimeoutSeconds }}s",
"loadAssignment": {
"clusterName": "/envoy-admin",
"endpoints": [
{
"lbEndpoints": [
{
"endpoint": {
"address": {
"pipe": {
"path": "/var/run/cilium/envoy/sockets/admin.sock"
}
}
}
}
]
}
]
}
}
]
},
"dynamicResources": {
"ldsConfig": {
"apiConfigSource": {
"apiType": "GRPC",
"transportApiVersion": "V3",
"grpcServices": [
{
"envoyGrpc": {
"clusterName": "xds-grpc-cilium"
}
}
],
"setNodeOnFirstMessageOnly": true
},
"resourceApiVersion": "V3"
},
"cdsConfig": {
"apiConfigSource": {
"apiType": "GRPC",
"transportApiVersion": "V3",
"grpcServices": [
{
"envoyGrpc": {
"clusterName": "xds-grpc-cilium"
}
}
],
"setNodeOnFirstMessageOnly": true
},
"resourceApiVersion": "V3"
}
},
"bootstrapExtensions": [
{
"name": "envoy.bootstrap.internal_listener",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.bootstrap.internal_listener.v3.InternalListener"
}
}
],
"layeredRuntime": {
"layers": [
{
"name": "static_layer_0",
"staticLayer": {
"overload": {
"global_downstream_max_connections": 50000
}
}
}
]
},
"admin": {
"address": {
"pipe": {
"path": "/var/run/cilium/envoy/sockets/admin.sock"
}
}
}
}
This diff is collapsed.
{
"__inputs": [
{
"name": "DS_PROMETHEUS",
"label": "Prometheus",
"description": "",
"type": "datasource",
"pluginId": "prometheus",
"pluginName": "Prometheus"
}
],
"__elements": {},
"__requires": [
{
"type": "panel",
"id": "bargauge",
"name": "Bar gauge",
"version": ""
},
{
"type": "grafana",
"id": "grafana",
"name": "Grafana",
"version": "9.4.7"
},
{
"type": "datasource",
"id": "prometheus",
"name": "Prometheus",
"version": "1.0.0"
},
{
"type": "panel",
"id": "timeseries",
"name": "Time series",
"version": ""
}
],
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "datasource",
"uid": "grafana"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"target": {
"limit": 100,
"matchAny": false,
"tags": [],
"type": "dashboard"
},
"type": "dashboard"
}
]
},
"description": "",
"editable": true,
"fiscalYearStartMonth": 0,
"gnetId": 16612,
"graphTooltip": 0,
"id": null,
"links": [
{
"asDropdown": true,
"icon": "external link",
"includeVars": true,
"keepTime": true,
"tags": [
"cilium-overview"
],
"targetBlank": false,
"title": "Cilium Overviews",
"tooltip": "",
"type": "dashboards",
"url": ""
},
{
"asDropdown": true,
"icon": "external link",
"includeVars": false,
"keepTime": true,
"tags": [
"hubble"
],
"targetBlank": false,
"title": "Hubble",
"tooltip": "",
"type": "dashboards",
"url": ""
}
],
"liveNow": false,
"panels": [
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 0
},
"id": 2,
"panels": [],
"title": "DNS",
"type": "row"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "normal"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "reqps"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 12,
"x": 0,
"y": 1
},
"id": 37,
"options": {
"legend": {
"calcs": [
"mean",
"lastNotNull"
],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "sum(rate(hubble_dns_queries_total{cluster=~\"$cluster\", source_namespace=~\"$source_namespace\", destination_namespace=~\"$destination_namespace\"}[$__rate_interval])) by (source) > 0",
"legendFormat": "{{source}}",
"range": true,
"refId": "A"
}
],
"title": "DNS queries",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "reqps"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 12,
"x": 12,
"y": 1
},
"id": 41,
"options": {
"displayMode": "gradient",
"minVizHeight": 10,
"minVizWidth": 0,
"orientation": "horizontal",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"showUnfilled": true
},
"pluginVersion": "9.4.7",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "topk(10, sum(rate(hubble_dns_queries_total{cluster=~\"$cluster\", source_namespace=~\"$source_namespace\", destination_namespace=~\"$destination_namespace\"}[$__rate_interval])*60) by (query))",
"legendFormat": "{{query}}",
"range": true,
"refId": "A"
}
],
"title": "Top 10 DNS queries",
"type": "bargauge"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "normal"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "reqps"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 12,
"x": 0,
"y": 10
},
"id": 39,
"options": {
"legend": {
"calcs": [
"mean",
"lastNotNull"
],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "round(sum(rate(hubble_dns_queries_total{cluster=~\"$cluster\", source_namespace=~\"$source_namespace\", destination_namespace=~\"$destination_namespace\"}[$__rate_interval])) by (source) - sum(label_replace(sum(rate(hubble_dns_responses_total{cluster=~\"$cluster\", source_namespace=~\"$destination_namespace\", destination_namespace=~\"$source_namespace\"}[$__rate_interval])) by (destination), \"source\", \"$1\", \"destination\", \"(.*)\")) without (destination), 0.001) > 0",
"legendFormat": "{{source}}",
"range": true,
"refId": "A"
}
],
"title": "Missing DNS responses",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "normal"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "reqps"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 12,
"x": 12,
"y": 10
},
"id": 43,
"options": {
"legend": {
"calcs": [
"mean",
"lastNotNull"
],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "sum(rate(hubble_dns_responses_total{cluster=~\"$cluster\", source_namespace=~\"$destination_namespace\", destination_namespace=~\"$source_namespace\", rcode!=\"No Error\"}[$__rate_interval])) by (destination, rcode) > 0",
"legendFormat": "{{destination}}: {{rcode}}",
"range": true,
"refId": "A"
}
],
"title": "DNS errors",
"type": "timeseries"
}
],
"refresh": "",
"revision": 1,
"schemaVersion": 38,
"style": "dark",
"tags": [
"kubecon-demo"
],
"templating": {
"list": [
{
"current": {
"selected": false,
"text": "default",
"value": "default"
},
"hide": 0,
"includeAll": false,
"label": "Data Source",
"multi": false,
"name": "prometheus_datasource",
"options": [],
"query": "prometheus",
"queryValue": "",
"refresh": 1,
"regex": "(?!grafanacloud-usage|grafanacloud-ml-metrics).+",
"skipUrlSync": false,
"type": "datasource"
},
{
"current": {},
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"definition": "label_values(cilium_version, cluster)",
"hide": 0,
"includeAll": true,
"multi": true,
"name": "cluster",
"options": [],
"query": {
"query": "label_values(cilium_version, cluster)",
"refId": "StandardVariableQuery"
},
"refresh": 1,
"regex": "",
"skipUrlSync": false,
"sort": 0,
"type": "query"
},
{
"allValue": ".*",
"current": {},
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"definition": "label_values(source_namespace)",
"hide": 0,
"includeAll": true,
"label": "Source Namespace",
"multi": true,
"name": "source_namespace",
"options": [],
"query": {
"query": "label_values(source_namespace)",
"refId": "StandardVariableQuery"
},
"refresh": 1,
"regex": "",
"skipUrlSync": false,
"sort": 0,
"type": "query"
},
{
"allValue": ".*",
"current": {},
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"definition": "label_values(destination_namespace)",
"hide": 0,
"includeAll": true,
"label": "Destination Namespace",
"multi": true,
"name": "destination_namespace",
"options": [],
"query": {
"query": "label_values(destination_namespace)",
"refId": "StandardVariableQuery"
},
"refresh": 1,
"regex": "",
"skipUrlSync": false,
"sort": 0,
"type": "query"
}
]
},
"time": {
"from": "now-1h",
"to": "now"
},
"timepicker": {
"refresh_intervals": [
"10s",
"30s",
"1m",
"5m",
"15m",
"30m",
"1h",
"2h",
"1d"
],
"time_options": [
"5m",
"15m",
"1h",
"6h",
"12h",
"24h",
"2d",
"7d",
"30d"
]
},
"timezone": "",
"title": "Hubble / DNS Overview (Namespace)",
"uid": "_f0DUpY4k",
"version": 26,
"weekStart": ""
}
\ No newline at end of file
This diff is collapsed.
#!/bin/bash
set -o errexit
set -o pipefail
set -o nounset
{{ .Values.nodeinit.prestop.preScript }}
if stat /tmp/node-deinit.cilium.io > /dev/null 2>&1; then
exit 0
fi
echo "Waiting on pods to stop..."
if [ ! -f /etc/crictl.yaml ] || grep -q 'docker' /etc/crictl.yaml; then
# Works for COS, ubuntu
while docker ps | grep -v "node-init" | grep -q "POD_cilium"; do sleep 1; done
else
# COS-beta (with containerd). Some versions of COS have crictl in /home/kubernetes/bin.
while PATH="${PATH}:/home/kubernetes/bin" crictl ps | grep -v "node-init" | grep -q "POD_cilium"; do sleep 1; done
fi
if ip link show cilium_host; then
echo "Deleting cilium_host interface..."
ip link del cilium_host
fi
{{- if not (eq .Values.nodeinit.bootstrapFile "") }}
rm -f {{ .Values.nodeinit.bootstrapFile | quote }}
{{- end }}
rm -f /tmp/node-init.cilium.io
touch /tmp/node-deinit.cilium.io
{{- if .Values.nodeinit.reconfigureKubelet }}
# Check if we're running on a GKE containerd flavor.
GKE_KUBERNETES_BIN_DIR="/home/kubernetes/bin"
if [[ -f "${GKE_KUBERNETES_BIN_DIR}/gke" ]] && command -v containerd &>/dev/null; then
CONTAINERD_CONFIG="/etc/containerd/config.toml"
echo "Reverting changes to the containerd configuration"
sed -Ei "s/^\#(\s+conf_template)/\1/g" "${CONTAINERD_CONFIG}"
echo "Removing the kubelet wrapper"
[[ -f "${GKE_KUBERNETES_BIN_DIR}/the-kubelet" ]] && mv "${GKE_KUBERNETES_BIN_DIR}/the-kubelet" "${GKE_KUBERNETES_BIN_DIR}/kubelet"
else
echo "Changing kubelet configuration to --network-plugin=kubenet"
sed -i "s:--network-plugin=cni\ --cni-bin-dir={{ .Values.cni.binPath }}:--network-plugin=kubenet:g" /etc/default/kubelet
fi
echo "Restarting the kubelet"
systemctl restart kubelet
{{- end }}
{{- if (and .Values.gke.enabled (or .Values.enableIPv4Masquerade .Values.gke.disableDefaultSnat))}}
# If the IP-MASQ chain exists, add back default jump rule from the GKE instance configure script
if iptables -w -t nat -L IP-MASQ > /dev/null; then
iptables -w -t nat -A POSTROUTING -m comment --comment "ip-masq: ensure nat POSTROUTING directs all non-LOCAL destination traffic to our custom IP-MASQ chain" -m addrtype ! --dst-type LOCAL -j IP-MASQ
fi
{{- end }}
{{ .Values.nodeinit.prestop.postScript }}
echo "Node de-initialization complete"
#!/bin/bash
set -o errexit
set -o pipefail
set -o nounset
echo "Link information:"
ip link
echo "Routing table:"
ip route
echo "Addressing:"
ip -4 a
ip -6 a
{{ .Values.nodeinit.startup.preScript }}
{{- if .Values.nodeinit.removeCbrBridge }}
if ip link show cbr0; then
echo "Detected cbr0 bridge. Deleting interface..."
ip link del cbr0
fi
{{- end }}
{{- if .Values.nodeinit.reconfigureKubelet }}
# Check if we're running on a GKE containerd flavor as indicated by the presence
# of the '--container-runtime-endpoint' flag in '/etc/default/kubelet'.
GKE_KUBERNETES_BIN_DIR="/home/kubernetes/bin"
KUBELET_DEFAULTS_FILE="/etc/default/kubelet"
if [[ -f "${GKE_KUBERNETES_BIN_DIR}/gke" ]] && [[ $(grep -cF -- '--container-runtime-endpoint' "${KUBELET_DEFAULTS_FILE}") == "1" ]]; then
echo "GKE *_containerd flavor detected..."
# (GKE *_containerd) Upon node restarts, GKE's containerd images seem to reset
# the /etc directory and our changes to the kubelet and Cilium's CNI
# configuration are removed. This leaves room for containerd and its CNI to
# take over pods previously managed by Cilium, causing Cilium to lose
# ownership over these pods. We rely on the empirical observation that
# /home/kubernetes/bin/kubelet is not changed across node reboots, and replace
# it with a wrapper script that performs some initialization steps when
# required and then hands over control to the real kubelet.
# Only create the kubelet wrapper if we haven't previously done so.
if [[ ! -f "${GKE_KUBERNETES_BIN_DIR}/the-kubelet" ]];
then
echo "Installing the kubelet wrapper..."
# Rename the real kubelet.
mv "${GKE_KUBERNETES_BIN_DIR}/kubelet" "${GKE_KUBERNETES_BIN_DIR}/the-kubelet"
# Initialize the kubelet wrapper which lives in the place of the real kubelet.
touch "${GKE_KUBERNETES_BIN_DIR}/kubelet"
chmod a+x "${GKE_KUBERNETES_BIN_DIR}/kubelet"
# Populate the kubelet wrapper. It will perform the initialization steps we
# need and then become the kubelet.
cat <<'EOF' | tee "${GKE_KUBERNETES_BIN_DIR}/kubelet"
#!/bin/bash
set -euo pipefail
CNI_CONF_DIR="/etc/cni/net.d"
CONTAINERD_CONFIG="/etc/containerd/config.toml"
# kubelet version string format is "Kubernetes v1.24-gke.900"
K8S_VERSION=$(/home/kubernetes/bin/the-kubelet --version)
# Helper to check if a version string, passed as first parameter, is greater than or
# equal the one passed as second parameter.
function version_gte() {
[[ "$(printf '%s\n' "${2}" "${1}" | sort -V | head -n1)" = "${2}" ]] && return
}
# Only stop and start containerd if the Cilium CNI configuration does not exist,
# or if the 'conf_template' property is present in the containerd config file,
# in order to avoid unnecessarily restarting containerd.
if [[ -z "$(find "${CNI_CONF_DIR}" -type f -name '*cilium*')" || \
"$(grep -cE '^\s+conf_template' "${CONTAINERD_CONFIG}")" != "0" ]];
then
# Stop containerd as it starts by creating a CNI configuration from a template
# causing pods to start with IPs assigned by GKE's CNI.
# 'disable --now' is used instead of stop as this script runs concurrently
# with containerd on node startup, and hence containerd might not have been
# started yet, in which case 'disable' prevents it from starting.
echo "Disabling and stopping containerd"
systemctl disable --now containerd
# Remove any pre-existing files in the CNI configuration directory. We skip
# any possibly existing Cilium configuration file for the obvious reasons.
echo "Removing undesired CNI configuration files"
find "${CNI_CONF_DIR}" -type f -not -name '*cilium*' -exec rm {} \;
# As mentioned above, the containerd configuration needs a little tweak in
# order not to create the default CNI configuration, so we update its config.
echo "Fixing containerd configuration"
sed -Ei 's/^(\s+conf_template)/\#\1/g' "${CONTAINERD_CONFIG}"
if version_gte "${K8S_VERSION#"Kubernetes "}" "v1.24"; then
# Starting from GKE node version 1.24, containerd version used is 1.6.
# Since that version containerd no longer allows missing configuration for the CNI,
# not even for pods with hostNetwork set to true. Thus, we add a temporary one.
# This will be replaced with the real config by the agent pod.
echo -e '{\n\t"cniVersion": "0.3.1",\n\t"name": "cilium",\n\t"type": "cilium-cni"\n}' > /etc/cni/net.d/05-cilium.conf
fi
# Start containerd. It won't create it's CNI configuration file anymore.
echo "Enabling and starting containerd"
systemctl enable --now containerd
fi
# Become the real kubelet and, for k8s < 1.24, pass it additional dockershim
# flags (and place these last so they have precedence).
if version_gte "${K8S_VERSION#"Kubernetes "}" "v1.24"; then
exec /home/kubernetes/bin/the-kubelet "${@}"
else
exec /home/kubernetes/bin/the-kubelet "${@}" --network-plugin=cni --cni-bin-dir={{ .Values.cni.binPath }}
fi
EOF
else
echo "Kubelet wrapper already exists, skipping..."
fi
else
# kubelet version string format is "Kubernetes v1.24-gke.900"
K8S_VERSION=$(kubelet --version)
# Helper to check if a version string, passed as first parameter, is greater than or
# equal the one passed as second parameter.
function version_gte() {
[[ "$(printf '%s\n' "${2}" "${1}" | sort -V | head -n1)" = "${2}" ]] && return
}
# Dockershim flags have been removed since k8s 1.24.
if ! version_gte "${K8S_VERSION#"Kubernetes "}" "v1.24"; then
# (Generic) Alter the kubelet configuration to run in CNI mode
echo "Changing kubelet configuration to --network-plugin=cni --cni-bin-dir={{ .Values.cni.binPath }}"
mkdir -p {{ .Values.cni.binPath }}
sed -i "s:--network-plugin=kubenet:--network-plugin=cni\ --cni-bin-dir={{ .Values.cni.binPath }}:g" "${KUBELET_DEFAULTS_FILE}"
fi
fi
echo "Restarting the kubelet..."
systemctl restart kubelet
{{- end }}
{{- if (and .Values.gke.enabled (or .Values.enableIPv4Masquerade .Values.gke.disableDefaultSnat))}}
# If Cilium is configured to manage masquerading of traffic leaving the node,
# we need to disable the IP-MASQ chain because even if ip-masq-agent
# is not installed, the node init script installs some default rules into
# the IP-MASQ chain.
# If we remove the jump to that ip-masq chain, then we ensure the ip masquerade
# configuration is solely managed by Cilium.
# Also, if Cilium is installed, it may be expected that it would be solely responsible
# for the networking configuration on that node. So provide the same functionality
# as the --disable-snat-flag for existing GKE clusters.
iptables -w -t nat -D POSTROUTING -m comment --comment "ip-masq: ensure nat POSTROUTING directs all non-LOCAL destination traffic to our custom IP-MASQ chain" -m addrtype ! --dst-type LOCAL -j IP-MASQ || true
{{- end }}
{{- if not (eq .Values.nodeinit.bootstrapFile "") }}
mkdir -p {{ .Values.nodeinit.bootstrapFile | dir | quote }}
date > {{ .Values.nodeinit.bootstrapFile | quote }}
{{- end }}
{{- if .Values.azure.enabled }}
# AKS: If azure-vnet is installed on the node, and (still) configured in bridge mode,
# configure it as 'transparent' to be consistent with Cilium's CNI chaining config.
# If the azure-vnet CNI config is not removed, kubelet will execute CNI CHECK commands
# against it every 5 seconds and write 'bridge' to its state file, causing inconsistent
# behaviour when Pods are removed.
if [ -f /etc/cni/net.d/10-azure.conflist ]; then
echo "Ensuring azure-vnet is configured in 'transparent' mode..."
sed -i 's/"mode":\s*"bridge"/"mode":"transparent"/g' /etc/cni/net.d/10-azure.conflist
fi
# The azure0 interface being present means the node was booted with azure-vnet configured
# in bridge mode. This means there might be ebtables rules and neight entries interfering
# with pod connectivity if we deploy with Azure IPAM.
if ip l show dev azure0 >/dev/null 2>&1; then
# In Azure IPAM mode, also remove the azure-vnet state file, otherwise ebtables rules get
# restored by the azure-vnet CNI plugin on every CNI CHECK, which can cause connectivity
# issues in Cilium-managed Pods. Since azure-vnet is no longer called on scheduling events,
# this file can be removed.
rm -f /var/run/azure-vnet.json
# This breaks connectivity for existing workload Pods when Cilium is scheduled, but we need
# to flush these to prevent Cilium-managed Pod IPs conflicting with Pod IPs previously allocated
# by azure-vnet. These ebtables DNAT rules contain fixed MACs that are no longer bound on the node,
# causing packets for these Pods to be redirected back out to the gateway, where they are dropped.
echo 'Flushing ebtables pre/postrouting rules in nat table.. (disconnecting non-Cilium Pods!)'
ebtables -t nat -F PREROUTING || true
ebtables -t nat -F POSTROUTING || true
# ip-masq-agent periodically injects PERM neigh entries towards the gateway
# for all other k8s nodes in the cluster. These are safe to flush, as ARP can
# resolve these nodes as usual. PERM entries will be automatically restored later.
echo 'Deleting all permanent neighbour entries on azure0...'
ip neigh show dev azure0 nud permanent | cut -d' ' -f1 | xargs -r -n1 ip neigh del dev azure0 to || true
fi
{{- end }}
{{- if .Values.nodeinit.revertReconfigureKubelet }}
rm -f /tmp/node-deinit.cilium.io
{{- end }}
{{ .Values.nodeinit.startup.postScript }}
echo "Node initialization complete"
This diff is collapsed.
set -e
echo "Waiting for spire server to be reachable to start"
{{- if .Values.authentication.mutual.spire.serverAddress }}
ADDR="{{ .Values.authentication.mutual.spire.serverAddress }}"
{{- else }}
ADDR="spire-server.{{ .Values.authentication.mutual.spire.install.namespace}}.svc:8081"
{{- end }}
CONN_TIMEOUT="3"
TIMEOUT="60"
call_tcp_endpoint_with_timeout() {
local addr="$1"
local timeout="$2"
nc -z "$addr" -w "$timeout" &> /dev/null
}
# wait for SPIRE server to be reachable till $TIMEOUT is reached
start_time=$(date +%s)
while true; do
if call_tcp_endpoint_with_timeout "$ADDR" "$CONN_TIMEOUT"; then
echo "SPIRE server is reachable"
break
fi
if [ $(( $(date +%s) - start_time )) -gt "$TIMEOUT" ]; then
echo "Timed out waiting for spire server to be reachable"
exit 1
fi
echo "Waiting for spire server to be reachable"
sleep 1
done
This diff is collapsed.
This diff is collapsed.
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment