VNG Lab
Using nodeSelectors
Let’s deploy our workload, in our examples we will a deploy simple nginx Pod just to simulate the behaviour of Ocean alongside the Kubernetes Scheduler.
Create a file vng-example.yaml with the following content
apiVersion: apps/v1
kind: Deployment
metadata:
name: example-1
spec:
selector:
matchLabels:
app: nginx
replicas: 1
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx-dev
image: nginx
resources:
requests:
memory: "100Mi"
cpu: "256m"
nodeSelector:
env: ocean-workshop
example: "1"
Apply the file using kubectl, run kubectl apply -f vng-example.yaml
➜ kubectl apply -f vng-example.yaml
deployment.apps/example-1 created
List the pods,nodes from the cluster, you’ll be able to see the created Pod in a Pending state, run kubectl get pods,nodes
➜ kubectl get pods,nodes
NAME READY STATUS RESTARTS AGE
pod/example-1-7f8b5549bb-4k8rp 0/1 Pending 0 14s
NAME STATUS ROLES AGE VERSION
aks-agentpool-46800815-vmss000000 Ready <none> 64d v1.24.15
Following a couple of minutes, run kubectl get pods,nodes -o wide, we will see that a new node joined our cluster and that the Pod is being created/running on top of it
➜ kubectl get pods,nodes -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/example-1-7f8b5549bb-4k8rp 0/1 ContainerCreating 0 83s <none> aks-agentpool-46800815-vmss000000 <none> <none>
NAME STATUS ROLES AGE VERSION INTERNAL-IP
aks-agentpool-46800815-vmss000000 Ready agent 64d v1.24.15 10.224.0.4
EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
<none> Ubuntu 18.04.6 LTS 5.4.0-1112-azure containerd://1.7.1+azure-1
Describe the node to verify the nodeLabels are assigned to it, run kubectl describe <node-name>
➜ kubectl describe node aks-agentpool-46800815-vmss000000
Name: aks-agentpool-46800815-vmss000000
Roles: agent
Labels: agentpool=agentpool
beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=Standard_DS2_v2
beta.kubernetes.io/os=linux
failure-domain.beta.kubernetes.io/region=eastus
failure-domain.beta.kubernetes.io/zone=eastus-1
kubernetes.azure.com/agentpool=agentpool
kubernetes.azure.com/cluster=MC_yaros-rbac-test_yar-private-preview_eastus
kubernetes.azure.com/consolidated-additional-properties=3ad9cafe-3324-11ee-a619-16bb608ba0be
kubernetes.azure.com/kubelet-identity-client-id=c82dc65e-5484-4309-8da6-efd14c60d472
kubernetes.azure.com/mode=system
kubernetes.azure.com/node-image-version=AKSUbuntu-1804gen2containerd-202307.27.0
kubernetes.azure.com/nodepool-type=VirtualMachineScaleSets
kubernetes.azure.com/os-sku=Ubuntu
kubernetes.azure.com/role=agent
kubernetes.azure.com/storageprofile=managed
kubernetes.azure.com/storagetier=Premium_LRS
kubernetes.io/arch=amd64
kubernetes.io/hostname=aks-agentpool-46800815-vmss000000
kubernetes.io/os=linux
kubernetes.io/role=agent
node-role.kubernetes.io/agent=
node.kubernetes.io/instance-type=Standard_DS2_v2
storageprofile=managed
storagetier=Premium_LRS
topology.disk.csi.azure.com/zone=eastus-1
topology.kubernetes.io/region=eastus
topology.kubernetes.io/zone=eastus-1
env=ocean-workshop
example=1
In addition to node labels you define, nodes come with pre-populated standard set of labels. You can review kubernetes Well-Known Labels, Annotations and Taints and Azure Reserved system labels.
Ocean expands the built in Kubernetes node-labels functionality with some proprietary labels supported by Ocean. See Labels & Taints for further details.
We are now able to specify when we want to force a specific workload to run on machines of the example-1 VNG, in the current state, Pods with no specific constraints (e.g not directed toward specific type of nodes) might be placed on nodes from the example-1 VNG, in order to prevent that we will also need to assign a Taint to the VNG, the Pods that we want to run on the tainted VNG should Tolerate that Taint.
Using Taints/Tolerations
In order to demonstrate that, let’s add a new VNG, follow the same process but specify the following configurations - in the Name specify example-2, in the Node Selection section specify:
Node Labels (formatted as key: value)
- env: ocean-workshop
- example: 2
Node Taints
- Key: example
Value: 2
Effect: NoSchedule
Click the save button on the bottom of the page
Create a file vng-example-2.yaml file with the following content:
apiVersion: apps/v1
kind: Deployment
metadata:
name: example-2-1
spec:
selector:
matchLabels:
app: nginx
replicas: 1
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx-dev
image: nginx
resources:
requests:
memory: "100Mi"
cpu: "256m"
nodeSelector:
env: ocean-workshop
example: "2"
tolerations:
- key: "example"
operator: "Equal"
value: "2"
effect: "NoSchedule"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: example-2-2
spec:
selector:
matchLabels:
app: nginx
replicas: 1
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx-dev
image: nginx
resources:
requests:
memory: "100Mi"
cpu: "256m"
nodeSelector:
env: ocean-workshop
example: "2"
While both Pods require our nodeLabels only one of them tolerates the node taints, let’s view it’s effect:
Run kubectl apply -f vng-example-2.yaml
➜ kubectl apply -f vng-example-2.yaml
deployment.apps/example-2-1 created
deployment.apps/example-2-2 created
We will now see that both our Pods are pending, run kubectl get pods,nodes
➜ kubectl get pods,nodes
NAME READY STATUS RESTARTS AGE
example-1-7f8b5549bb-xnsfq 1/1 Running 0 48m
example-2-1-64d9d4877d-rv2pg 0/1 Pending 0 39s
example-2-2-7d55b5b8cf-dfkpw 0/1 Pending 0 37s
NAME STATUS ROLES AGE VERSION
aks-agentpool-46800815-vmss000000 Ready <none> 64d v1.24.15
A couple of minutes later Ocean will scale up a new machine to satisfy the supported Pod, we will see that only one of the Pods (example-2-1) will be scheduled while the other wont since it does not tolerate the nodes taint.
Run kubectl get pods,nodes -o wide
➜ kubectl get pods,nodes -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
example-1-7f8b5549bb-xnsfq 1/1 Running 0 50m 192.168.48.21 aks-agentpool-46800815-vmss000000 <none> <none>
example-2-1-64d9d4877d-rv2pg 1/1 Running 0 2m26s 192.168.24.234 aks-agentpool-46800815-vmss000000 <none> <none>
example-2-2-7d55b5b8cf-dfkpw 0/1 Pending 0 2m24s <none> <none> <none> <none>
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
aks-agentpool-46805500-vmss000005 Ready <none> 87s v1.24.15 192.168.24.193 54.149.39.228 Ubuntu 18.04.6 LTS 5.4.0-1112-azure containerd://1.7.1+azure-1
aks-agentpool-46800815-vmss000000 Ready <none> 64d v1.24.15 192.168.46.105 35.160.241.120 Ubuntu 18.04.6 LTS 5.4.0-1112-azure containerd://1.7.1+azure-1
Run kubectl describe <pod-name>
Under the Events section you’ll see that the reason for the Pod not being scheduled is because there is a node with a taint that the pod did not tolerate
➜ kubectl describe pod/example-2-2-7d55b5b8cf-dfkpw
Name: example-2-2-7d55b5b8cf-dfkpw
Namespace: default
Priority: 0
Service Account: default
Node: <none>
Labels: app=nginx
pod-template-hash=df4d44b8c
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/example-2-2-7d55b5b8cf
Containers:
nginx-dev:
Image: nginx
Port: <none>
Host Port: <none>
Requests:
cpu: 256m
memory: 100Mi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5dqmd (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
kube-api-access-5dqmd:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: env=ocean-workshop
example=2
Tolerations: example=2:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 39s default-scheduler 0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.