Skip to content
Snippets Groups Projects
Commit 50ac7cc9 authored by Alex ubuntu vm's avatar Alex ubuntu vm
Browse files

L2S-M installation deploy: Fixed dependencies between components. 

- Added health checks in the containers to enable readiness and liveness probes

- Implemented Probes and initContainers so the installation follows a specific order without having to manage it.
parent 85457a2f
No related branches found
No related tags found
1 merge request!2repo: added new directory where utils scripts will be
Hide whitespace changes
Inline Side-by-side
deployments/README.md
+ 14
− 108
View file @ 50ac7cc9
...@@ -18,127 +18,33 @@ kubectl taint nodes --all node-role.kubernetes.io/control-plane- node-role.kuber ...@@ -18,127 +18,33 @@ kubectl taint nodes --all node-role.kubernetes.io/control-plane- node-role.kuber
## Install L2S-M ## Install L2S-M
1. Create the virtual interface definitions using the following command: Installing L2S-M can be done by using a single command:
```bash
kubectl create -f ./deployments/custom-installation/interfaces_definitions
```
2. Create the Kubernetes account Service Account and apply their configuration by applying the following command:
```bash
kubectl create -f ./deployments/config/
```
3. Create the Kubernetes Persistent Volume by using the following kubectl command:
```bash
kubectl create -f ./deployments/custom-installation/mysql/
```
4. Before deploying the L2S-M operator, it is neccessary to label your master node as the "master" of the cluster. To do so, get the names of your Kubernetes nodes, select the master and apply the "master" label with the following command:
```bash
kubectl get nodes
kubectl label nodes [your-master-node] dedicated=master
```
5. Deploy the L2S-M Controller by using the following command:
```bash ```bash
kubectl create -f ./deployments/custom-installation/deployController.yaml kubectl create -f ./deployments/l2sm-deployment.yaml
```
You can check that the deployment was successful if the pod enters the "running" state using the *kubectl get pods* command.
6. After the previous preparation, (make sure the controller is running) you can deploy the operator in your cluster using the YAML deployment file:
```bash
kubectl create -f ./deployments/custom-installation/deployOperator.yaml
``` ```
Once these two pods are in running state, you can finally deploy the virtual switches The installation will take around a minute to finish, and to check that everyting is running properly, you may run the following command:
7. This is done by:
**First deploying the virtual OVS Daemonset:**
```bash ```bash
kubectl create -f ./deployments/custom-installation/deploySwitch.yaml kubectl get pods -o wide
```
And check there is a pod running in each node, with ```kubectl get pods -o wide```
**Lastly, we configure the Vxlans:**
In order to connect the switches between themselves, an additional configuarion must be done. A configuration file specifying which nodes we want to connect and which IP addresses their switches have will be made, and then a script will be run in each l2sm switch, using this configuration file.
a. Create a file anywhere or use the reference in ./configs/sampleFile.json. In this installation, this file will be used as a reference.
b. In this file, you will specify, using the template shown in the reference file, the name of the nodes in the cluster and the IP addresses of **the switches** running on them. For example:
```bash
$ kubectl get pods -o wide
>NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
>l2sm-controller-d647b7fb5-lpp2h 1/1 Running 0 30m 10.1.14.55 l2sm1 <none> <none>
>l2sm-operator-7d487d8468-lhgkx 2/2 Running 0 2m11s 10.1.14.56 l2sm1 <none> <none>
>l2sm-switch-8p5td 1/1 Running 0 71s 10.1.14.58 l2sm1 <none> <none>
>l2sm-switch-xdkvz 1/1 Running 0 71s 10.1.72.111 l2sm2 <none> <none>
```
In this example we have two nodes: l2sm1 and l2sm2, with two switches, with IP addresses 10.1.14.58 and 10.1.72.111.
We want to connect them directly, so we modify the reference file, ./configs/sampleFile.json:
```json
[
{
"name": "<NODE_SWITCH_1>",
"nodeIP": "<IP_SWITCH_1>",
"neighborNodes": ["<NODE_SWITCH_2>"]
},
{
"name": "<NODE_SWITCH_2>",
"nodeIP": "<IP_SWITCH_2>",
"neighborNodes": ["<NODE_SWITCH_1>"]
}
]
```
Note: The parameters to be changed are shown in the NODE and IP columns of the table above.
Example of how it looks:
```json
[
{
"name": "l2sm1",
"nodeIP": "10.1.14.58",
"neighborNodes": ["l2sm2"]
},
{
"name": "l2sm2",
"nodeIP": "10.1.72.111",
"neighborNodes": ["l2sm1"]
}
]
``` ```
Note: Any number of nodes can be configured, as long as the entry is in this file. The desired connections are under the neighborNodes field, in an array, such as this other example, where we add a neighbor to l2sm2: ["l2sm1","l2sm3"]
Once this file is created, we inject it to each node using the kubectl cp command: Which should give you an output like this:
```bash ```bash
kubectl cp ./configs/sampleFile.json <pod-name>:/etc/l2sm/switchConfig.json NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
``` l2sm-controller-56b45487b7-nglns 1/1 Running 0 129m 10.1.72.72 l2sm2 <none> <none>
And then executing the script in the switch-pod: l2sm-operator-7794c5f66d-b9nsf 2/2 Running 0 119m 10.1.14.45 l2sm1 <none> <none>
```bash l2sm-switch-49qpq 1/1 Running 0 129m 10.1.14.63 l2sm1 <none> <none>
kubectl exec -it <switch-pod-name> -- setup_switch.sh l2sm-switch-2g696 1/1 Running 0 129m 10.1.72.73 l2sm2 <none> <none>
``` ```
With the components: _l2sm-controller_, _l2sm-operator_ and one _l2sm-switch_ for **each** node in the cluster.
This must be done in each switch-pod. In the provided example, using two nodes, l2sm1 and l2sm2, we have to do it twice, in l2-ps-8p5td and l2-ps-xdkvz. ## Configuring VxLANs
When the exec command is done, we should see an output like this:
```bash Each Node enables the creation of custom L2S-M networks, as can be seen in the [examples section](../../examples/) section. But for communicating pods that are in different Nodes of the cluster, additional configuration must be done, the VxLAN tunnels between them.
$ kubectl exec -it l2-ps-xdkvz -- setup_switch.sh
2023-10-30T10:22:18Z|00001|ovs_numa|INFO|Discovered 1 CPU cores on NUMA node 0
2023-10-30T10:22:18Z|00002|ovs_numa|INFO|Discovered 1 NUMA nodes and 1 CPU cores
2023-10-30T10:22:18Z|00003|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2023-10-30T10:22:18Z|00004|netlink_socket|INFO|netlink: could not enable listening to all nsid (Operation not permitted)
2023-10-30T10:22:18Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
initializing switch, connected to controller: 10.1.14.8
Switch initialized and connected to the controller.
Created vxlan between node l2sm2 and node l2sm1.
```
You can proceed to configure Vxlans by following the steps outlined in [the vxlan configuration guide.](../deployment/vxlans.md)
You are all set! If you want to learn how to create virtual networks and use them in your applications, [check the following section of the repository](https://github.com/Networks-it-uc3m/L2S-M/tree/release-2.0/examples/) You are all set! If you want to learn how to create virtual networks and use them in your applications, [check the following section of the repository](https://github.com/Networks-it-uc3m/L2S-M/tree/release-2.0/examples/)
deployments/custom-installation/deployController.yaml
+ 2
− 0
View file @ 50ac7cc9
...@@ -19,6 +19,8 @@ spec: ...@@ -19,6 +19,8 @@ spec:
httpGet: httpGet:
path: /onos/ui path: /onos/ui
port: 8181 port: 8181
initialDelaySeconds: 30
periodSeconds: 10
ports: ports:
- containerPort: 6633 - containerPort: 6633
- containerPort: 8181 - containerPort: 8181
......
deployments/custom-installation/deployOperator.yaml
+ 26
− 5
View file @ 50ac7cc9
...@@ -17,17 +17,23 @@ spec: ...@@ -17,17 +17,23 @@ spec:
serviceAccountName: l2sm-operator serviceAccountName: l2sm-operator
initContainers: initContainers:
- name: wait-for-l2sm-controller - name: wait-for-l2sm-controller
image: busybox:1.31 image: curlimages/curl
command: ["/bin/sh", "-c", "set -x; while true; do status_code=$(wget --spider -S http://l2sm-controller-service:8181/onos/ui 2>&1 | grep 'HTTP/' | awk '{print $2}'); [ -z $status_code ] && sleep 15 || [ $status_code -eq 200 ] && break || sleep 15; done"] args:
- /bin/sh
- -c
- >
set -x;
while [ $(curl -sw '%{http_code}' "http://l2sm-controller-service:8181/onos/ui" -o /dev/null) -ne 302 ]; do
sleep 15;
done;
sleep 5;
containers: containers:
- name: wait-test
image: busybox:1.31
command: ["sleep","infinity"]
- image: alexdecb/l2sm-operator:2.2 - image: alexdecb/l2sm-operator:2.2
name: l2sm-opt-pod name: l2sm-opt-pod
env: env:
- name: CONTROLLER_IP - name: CONTROLLER_IP
value: l2sm-controller-service value: l2sm-controller-service
#imagePullPolicy: Always
- image: mysql/mysql-server:5.7 - image: mysql/mysql-server:5.7
name: mysql name: mysql
env: env:
...@@ -56,3 +62,18 @@ spec: ...@@ -56,3 +62,18 @@ spec:
operator: Equal operator: Equal
value: master value: master
effect: NoSchedule effect: NoSchedule
---
apiVersion: v1
kind: Service
metadata:
name: l2sm-operator-service
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8080
selector:
l2sm-component: l2sm-opt
deployments/custom-installation/deploySwitch.yaml
+ 12
− 1
View file @ 50ac7cc9
...@@ -22,10 +22,21 @@ spec: ...@@ -22,10 +22,21 @@ spec:
- key: node-role.kubernetes.io/master - key: node-role.kubernetes.io/master
operator: Exists operator: Exists
effect: NoSchedule effect: NoSchedule
initContainers:
- name: wait-for-l2sm-operator
image: curlimages/curl
args:
- /bin/sh
- -c
- >
set -x;
while [ $(curl -sw '%{http_code}' "http://l2sm-operator-service:8080/healthz" -o /dev/null) -ne 200 ]; do
sleep 15;
done;
sleep 5;
containers: containers:
- name: l2sm-switch - name: l2sm-switch
image: alexdecb/l2sm-switch:2.2 image: alexdecb/l2sm-switch:2.2
command: ["sleep","infinity"]
#args: ["setup_switch.sh && sleep infinity"] #args: ["setup_switch.sh && sleep infinity"]
env: env:
- name: NODENAME - name: NODENAME
......
deployments/l2sm-deployment.yaml
+ 47
− 10
View file @ 50ac7cc9
...@@ -191,11 +191,13 @@ spec: ...@@ -191,11 +191,13 @@ spec:
spec: spec:
containers: containers:
- name: l2sm-controller - name: l2sm-controller
image: alexdecb/l2sm-controller:latest image: alexdecb/l2sm-controller:2.2
# readinessProbe: readinessProbe:
# httpGet: httpGet:
# path: /onos/v1/l2sm/networks/status path: /onos/ui
# port: 8181 port: 8181
initialDelaySeconds: 30
periodSeconds: 10
ports: ports:
- containerPort: 6633 - containerPort: 6633
- containerPort: 8181 - containerPort: 8181
...@@ -235,13 +237,25 @@ spec: ...@@ -235,13 +237,25 @@ spec:
l2sm-component: l2sm-opt l2sm-component: l2sm-opt
spec: spec:
serviceAccountName: l2sm-operator serviceAccountName: l2sm-operator
initContainers:
- name: wait-for-l2sm-controller
image: curlimages/curl
args:
- /bin/sh
- -c
- >
set -x;
while [ $(curl -sw '%{http_code}' "http://l2sm-controller-service:8181/onos/ui" -o /dev/null) -ne 302 ]; do
sleep 15;
done;
sleep 5;
containers: containers:
- image: alexdecb/l2sm-operator:2.1 - image: alexdecb/l2sm-operator:2.2
name: l2sm-opt-pod name: l2sm-opt-pod
env: env:
- name: CONTROLLER_IP - name: CONTROLLER_IP
value: l2sm-controller-service value: l2sm-controller-service
#command: ["sleep","infinity"] #imagePullPolicy: Always
- image: mysql/mysql-server:5.7 - image: mysql/mysql-server:5.7
name: mysql name: mysql
env: env:
...@@ -271,6 +285,18 @@ spec: ...@@ -271,6 +285,18 @@ spec:
value: master value: master
effect: NoSchedule effect: NoSchedule
--- ---
apiVersion: v1
kind: Service
metadata:
name: l2sm-operator-service
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8080
selector:
l2sm-component: l2sm-opt
---
apiVersion: apps/v1 apiVersion: apps/v1
kind: DaemonSet kind: DaemonSet
metadata: metadata:
...@@ -295,11 +321,22 @@ spec: ...@@ -295,11 +321,22 @@ spec:
- key: node-role.kubernetes.io/master - key: node-role.kubernetes.io/master
operator: Exists operator: Exists
effect: NoSchedule effect: NoSchedule
initContainers:
- name: wait-for-l2sm-operator
image: curlimages/curl
args:
- /bin/sh
- -c
- >
set -x;
while [ $(curl -sw '%{http_code}' "http://l2sm-operator-service:8080/healthz" -o /dev/null) -ne 200 ]; do
sleep 15;
done;
sleep 5;
containers: containers:
- name: l2sm-switch - name: l2sm-switch
image: alexdecb/l2sm-switch:2.1 image: alexdecb/l2sm-switch:2.2
command: ["/bin/sh","-c"] #args: ["setup_switch.sh && sleep infinity"]
args: ["setup_switch.sh && sleep infinity"]
env: env:
- name: NODENAME - name: NODENAME
valueFrom: valueFrom:
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment

Copyright © Eclipse Foundation, Inc. All Rights Reserved.     Privacy Policy | Terms of Use | Copyright Agent