Guided Exercise: Kubernetes Pod and Service Networks

Deploy a database server and access it through a Kubernetes service.

Outcomes

You should be able to deploy a database server, and access it indirectly through a Kubernetes service, and also directly pod-to-pod for troubleshooting.

As the student user on the workstation machine, use the lab command to prepare your system for this exercise.

This command ensures that all resources are available for this exercise. It also creates the deploy-services project and the /home/student/DO180/labs/deploy-services/resources.txt file. The resources.txt file contains some commands that you use during the exercise. You can use the file to copy and paste these commands.

[student@workstation ~]$ lab start deploy-services

Note

It is safe to ignore pod security warnings for exercises in this course. OpenShift uses the Security Context Constraints controller to provide safe defaults for pod security.

Instructions

  1. Log in to the OpenShift cluster as the developer user with the developer password. Use the deploy-services project.

    1. Log in to the OpenShift cluster.

      [student@workstation ~]$ oc login -u developer -p developer \
  https://api.ocp4.example.com:6443
Login successful.
...output omitted...

    2. Set the deploy-services project as the active project.

      [student@workstation ~]$ oc project deploy-services
...output omitted...

  2. Use the registry.ocp4.example.com:8443/rhel8/mysql-80 container image to create a MySQL deployment named db-pod. Add the missing environment variables for the pod to run.

    1. Create the db-pod deployment.

      [student@workstation ~]$ oc create deployment db-pod --port 3306 \
  --image registry.ocp4.example.com:8443/rhel8/mysql-80
deployment.apps/db-pod created

    2. Add the environment variables.

      [student@workstation ~]$ oc set env deployment/db-pod \
  MYSQL_USER=user1 \
  MYSQL_PASSWORD=mypa55w0rd \
  MYSQL_DATABASE=items
deployment.apps/db-pod updated

    3. Confirm that the pod is running.

      [student@workstation ~]$ oc get pods
NAME                      READY   STATUS    RESTARTS   AGE
db-pod-6ccc485cfc-vrc4r   1/1     Running   0          2m30s

      Your pod name might differ from the previous output.

  3. Expose the db-pod deployment to create a ClusterIP service.

    1. View the deployment for the pod.

      [student@workstation ~]$ oc get deployment
NAME     READY   UP-TO-DATE   AVAILABLE   AGE
db-pod   1/1     1            1           3m36s

    2. Expose the db-pod deployment to create a service.

      [student@workstation ~]$ oc expose deployment/db-pod
service/db-pod exposed

  4. Validate the service. Confirm that the service selector matches the label on the pod. Then, confirm that the db-pod service endpoint matches the IP of the pod.

    1. Identify the selector for the db-pod service. Use the oc get service command with the -o wide option to retrieve the selector that the service uses.

      [student@workstation ~]$ oc get service db-pod -o wide
NAME     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE    SELECTOR
db-pod   ClusterIP   172.30.108.92   <none>        3306/TCP   108s   app=db-pod

      The selector shows an app=db-pod key:value pair.

    2. Capture the name of the pod in a variable.

      [student@workstation ~]$ PODNAME=$(oc get pods \
  -o jsonpath='{.items[0].metadata.name}')

    3. Query the label on the pod.

      [student@workstation ~]$ oc get pod $PODNAME --show-labels
NAME                     READY   STATUS    RESTARTS   AGE     LABELS
db-pod-6ccc485cfc-vrc4r  1/1     Running   0          6m50s   app=db-pod ...

      Notice that the label list includes the app=db-pod key-value pair, which is the selector for the db-pod service.

    4. Retrieve the endpoints for the db-pod service.

      [student@workstation ~]$ oc get endpoints
NAME     ENDPOINTS        AGE
db-pod   10.8.0.85:3306   4m38s

      Your endpoints values might differ from the previous output.

    5. Verify that the service endpoint matches the db-pod IP address. Use the oc get pods command with the -o wide option to view the pod IP address.

      [student@workstation ~]$ oc get pods -o wide
NAME                      READY   STATUS    RESTARTS   AGE   IP        ...
db-pod-6ccc485cfc-vrc4r   1/1     Running   0          54m   10.8.0.85 ...

      The service endpoint resolves to the IP address that is assigned to the pod.

  5. Delete and then re-create the db-pod deployment. Confirm that the db-pod service endpoint automatically resolves to the IP address of the new pod.

    1. Delete the db-pod deployment.

      [student@workstation ~]$ oc delete deployment.apps/db-pod
deployment.apps "db-pod" deleted

    2. Verify that the service still exists without the deployment.

      [student@workstation ~]$ oc get service
NAME     TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
db-pod   ClusterIP   172.30.108.92  <none>        3306/TCP   9m53s

    3. The list of endpoints for the service is now empty.

      [student@workstation ~]$ oc get endpoints
NAME     ENDPOINTS   AGE
db-pod   <none>      12m

    4. Re-create the db-pod deployment.

      [student@workstation ~]$ oc create deployment db-pod --port 3306 \
  --image registry.ocp4.example.com:8443/rhel8/mysql-80
deployment.apps/db-pod created

    5. Add the environment variables.

      [student@workstation ~]$ oc set env deployment/db-pod \
  MYSQL_USER=user1 \
  MYSQL_PASSWORD=mypa55w0rd \
  MYSQL_DATABASE=items
deployment.apps/db-pod updated

    6. Confirm that the newly created pod has the app=db-pod selector.

      [student@workstation ~]$ oc get pods --selector app=db-pod -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP        ...
db-pod-6ccc485cfc-l2x   1/1     Running   0          32s   10.8.0.85 ...

      Notice the change in the pod name. The pod IP address might also change. Your pod name and IP address might differ from the previous output.

    7. Confirm that the endpoints for the db-pod service include the newly created pod.

      [student@workstation ~]$ oc get endpoints
NAME     ENDPOINTS        AGE
db-pod   10.8.0.85:3306   16m

  6. Create a pod to identify the available DNS name assignments for the service.

    1. Create a pod named shell to use for troubleshooting. Use the oc run command and the registry.ocp4.example.com:8443/openshift4/network-tools-rhel8 container image.

      [student@workstation ~]$ oc run shell -it \
  --image registry.ocp4.example.com:8443/openshift4/network-tools-rhel8
If you don't see a command prompt, try pressing enter.
bash-4.4$

    2. From the prompt inside the shell pod, view the /etc/resolv.conf file to identify the cluster-domain name.

      bash-4.4$ cat /etc/resolv.conf
search deploy-services.svc.cluster.local svc.cluster.local ...
nameserver 172.30.0.10
options ndots:5

      The container uses the values from the search directive as suffix values on DNS searches. The container appends these values to a DNS query, in the written order, to resolve the search. The cluster-domain name is the last few components of these values that start after svc.

    3. Use the nc and echo commands to test the available DNS names for the service.

      nc -z <service>_<server>_ <port>

      The long version of the DNS name is required when accessing the service from a different project. When the pod is in the same project, you can use a shorter version of the DNS name.

      bash-4.4$ nc -z db-pod.deploy-services 3306 && \
 echo "Connection success to db-pod.deploy-services:3306" || \
 echo "Connection failed"
Connection success to db-pod.deploy-services:3306

    4. Exit the interactive session.

      bash-4.4$ exit
Session ended, resume using 'oc attach shell -c shell -i -t' command when the pod is running

    5. Delete the pod for the shell.

      [student@workstation ~]$ oc delete pod shell
pod "shell" deleted

  7. Use a new project to test pod communications across namespaces.

    1. Create a second namespace with the oc new-project command.

      [student@workstation ~]$ oc new-project deploy-services-2
Now using project "deploy-services-2" on server "https://api.ocp4.example.com:6443".
...output omitted...

    2. Execute the nc and echo commands from a pod to test the DNS name access to another namespace.

      [student@workstation ~]$ oc run shell -it --rm \
  --image registry.ocp4.example.com:8443/openshift4/network-tools-rhel8 \
  --restart Never -- nc -z db-pod.deploy-services.svc.cluster.local 3306 && \
  echo "Connection success to db-pod.deploy-services.svc.cluster.local:3306" \
  || echo "Connection failed"
pod "shell" deleted
Connection success to db-pod.deploy-services.svc.cluster.local:3306

    3. Return to the deploy-services project.

      [student@workstation ~]$ oc project deploy-services
Now using project "deploy-services" on server "https://api.ocp4.example.com:6443".

  8. Use a Kubernetes job to add initialization data to the database.

    1. Create a job named mysql-init that uses the registry.ocp4.example.com:8443/redhattraining/do180-dbinit:v1 container image. This image uses the mysql-80 container image as a base image, and it includes a script that adds a few initial records to the database.

      [student@workstation ~]$ oc create job mysql-init \
  --image registry.ocp4.example.com:8443/redhattraining/do180-dbinit:v1 \
  -- /bin/bash -c "mysql -uuser1 -pmypa55w0rd --protocol tcp \
  -h db-pod -P3306 items </tmp/db-init.sql"
job.batch/mysql-init created

      The -h option of the mysql command directs the command to communicate with the DNS short name of the db-pod service. The db-pod short name can be used here, because the pod for the job is created in the same namespace as the service.

      The double dash -- before /bin/bash separates the oc command arguments from the command in the pod. The -c option of /bin/bash directs the command interpreter in the container to execute the command string. The /tmp/db-init.sql file is redirected as input for the command. The db-init.sql file is included in the image, and contains the following script.

      DROP TABLE IF EXISTS `Item`;
CREATE TABLE `Item` (`id` BIGINT not null auto_increment primary key, `description` VARCHAR(100), `done` BIT);
INSERT INTO `Item` (`id`,`description`,`done`) VALUES (1,'Pick up newspaper', 0);
INSERT INTO `Item` (`id`,`description`,`done`) VALUES (2,'Buy groceries', 1);

    2. Confirm the status of the mysql-init job. Wait for the job to complete.

      [student@workstation ~]$ oc get job
NAME         COMPLETIONS   DURATION   AGE
mysql-init   1/1                      22m

    3. Retrieve the status of the mysql-init job pod, to confirm that the pod has a Completed status.

      [student@workstation ~]$ oc get pods
NAME                      READY   STATUS      RESTARTS      AGE
db-pod-6ccc485cfc-2lklx   1/1     Running     0             4h24m
mysql-init-ln9cg          0/1     Completed   0             23m

    4. Delete the mysql-init job, because it is no longer needed.

      [student@workstation ~]$ oc delete job mysql-init
job.batch "mysql-init" deleted

    5. Verify that the corresponding mysql-init pod is also deleted.

      [student@workstation ~]$ oc get pods
NAME                      READY   STATUS    RESTARTS      AGE
db-pod-6ccc485cfc-2lklx   1/1     Running   0             4h2

  9. Create a query-db pod by using the oc run command and the registry.ocp4.example.com:8443/redhattraining/do180-dbinit:v1 container image. Use the pod to execute a query against the database service.

    1. Create the query-db pod. Configure the pod to use the MySQL client to execute a query against the db-pod service. You can use the db-pod service short name, which provides a stable reference.

      [student@workstation ~]$ oc run query-db -it --rm \
  --image registry.ocp4.example.com:8443/redhattraining/do180-dbinit:v1 \
  --restart Never \
  -- mysql -uuser1 -pmypa55w0rd --protocol tcp \
  -h db-pod -P3306 items -e 'select * from Item;'
mysql: [Warning] Using a password on the command line interface can be insecure.
+----+-------------------+------------+
| id | description       | done       |
+----+-------------------+------------+
|  1 | Pick up newspaper | 0x00       |
|  2 | Buy groceries     | 0x01       |
+----+-------------------+------------+
pod "query-db" deleted

  10. It might be necessary to use pod-to-pod communications for troubleshooting. Use the oc run command to create a pod that executes a network test against the IP address of the database pod.

    1. Confirm the IP address of the MySQL database pod. Your pod IP address might differ from the output.

      [student@workstation ~]$ oc get pods -o wide
NAME                     READY  STATUS     RESTARTS     AGE  IP         ...
db-pod-6ccc485cfc-2lklx  1/1    Running    0            4h5  10.8.0.69  ...

    2. Capture the IP address in an environment variable.

      [student@workstation ~]$ POD_IP=$(oc get pod -l app=db-pod \
  -o jsonpath='{.items[0].status.podIP}')

    3. Create a test pod named shell with the oc run command. Execute the nc command to test against the $POD_IP environment variable and the 3306 port for the database.

      [student@workstation ~]$ oc run shell --env POD_IP=$POD_IP -it --rm \
  --image registry.ocp4.example.com:8443/openshift4/network-tools-rhel8 \
  --restart Never \
  -- nc -z $POD_IP 3306 && echo "Connection success to $POD_IP:3306" \
 || echo "Connection failed"
pod "shell" deleted
Connection success to 10.8.0.69:3306

Finish

On the workstation machine, use the lab command to complete this exercise. This step is important to ensure that resources from previous exercises do not impact upcoming exercises.

[student@workstation ~]$ lab finish deploy-services