Scale and Expose Applications to External Access

Objectives

  • Expose applications to clients outside the cluster by using Kubernetes ingress and OpenShift routes.

IP Addresses for Pods and Services

Most real-world applications do not run as a single pod. Because applications need to scale horizontally, many pods run the same containers from the same pod resource definition, to meet growing user demand. A service defines a single IP/port combination, and provides a single IP address to a pool of pods, and a load-balancing client request among member pods.

By default, services connect clients to pods in a round-robin fashion, and each service is assigned a unique IP address for clients to connect to. This IP address comes from an internal OpenShift virtual network, which although distinct from the pods' internal network, is visible only to pods. Each pod that matches the selector is added to the service resource as an endpoint.

Containers inside Kubernetes pods must not connect to each other's dynamic IP address directly. Services resolve this problem by linking more stable IP addresses from the SDN to the pods. If pods are restarted, replicated, or rescheduled to different nodes, then services are updated, to provide scalability and fault tolerance.

Service Types

You can choose between several service types depending on your application needs, cluster infrastructure, and security requirements.

ClusterIP

This type is the default, unless you explicitly specify a type for a service. The ClusterIP type exposes the service on a cluster-internal IP address. If you choose this value, then the service is reachable only from within the cluster.

The ClusterIP service type is used for pod-to-pod routing within the RHOCP cluster, and enables pods to communicate with and to access each other. IP addresses for the ClusterIP services are assigned from a dedicated service network that is accessible only from inside the cluster. Most applications should use this service type, for which Kubernetes automates the management.

Load balancer

This resource instructs RHOCP to activate a load balancer in a cloud environment. A load balancer instructs Kubernetes to interact with the cloud provider that the cluster is running in, to provision a load balancer. The load balancer then provides an externally accessible IP address to the application.

Take all necessary precautions before deploying this service type. Load balancers are typically too expensive to assign one for each application in a cluster. Furthermore, applications that use this service type become accessible from networks outside the cluster. Additional security configuration is required to prevent unintended access.

NodePort

With this method, Kubernetes exposes a service on a port on the node IP address. The port is exposed on all cluster nodes, and each node redirects traffic to the endpoints (pods) of the service.

A NodePort service requires allowing direct network connections to a cluster node, which is a security risk.

ExternalName

This service tells Kubernetes that the DNS name in the externalName field is the location of the resource that backs the service. When a DNS request is made against the Kubernetes DNS server, it returns the externalName in a Canonical Name (CNAME) record, and directs the client to look up the returned name to get the IP address.

Using Routes for External Connectivity

RHOCP provides resources to expose your applications to external networks outside the cluster. You can expose HTTP and HTTPS traffic, TCP applications, and also non-TCP traffic. However, you should expose only HTTP and TLS-based applications to external access. Applications that use other protocols, such as databases, are usually not exposed to external access (from outside a cluster). Routes and ingress are the main resources for handling ingress traffic.

RHOCP provides the route resource to expose your applications to external networks. With routes, you can access your application with a unique hostname that is publicly accessible. Routes rely on a Kubernetes ingress controller to redirect the traffic from the public IP address to pods. By default, Kubernetes provides an ingress controller, starting from the 1.24 release. For RHOCP clusters, the OpenShift ingress operator provides the ingress controller. RHOCP clusters can also use various third-party ingress controllers that can be deployed in parallel with the OpenShift ingress controller.

Routes provide ingress traffic to services in the cluster. Routes were created before Kubernetes ingress objects, and provide more features. Routes provide advanced features that Kubernetes ingress controllers might not support through a standard interface, such as TLS re-encryption, TLS passthrough, and split traffic for blue-green deployments.

To create a route with the oc CLI, use the oc expose service service-name command. Include the --hostname option to provide a custom hostname for the route.

[user@host ~]$ oc expose service api-frontend \
--hostname api.apps.acme.com

If you omit the hostname, then RHOCP automatically generates a hostname with the following structure: <route-name>-<project-name>.<default-domain>. For example, if you create a frontend route in an api project, in a cluster that uses apps.example.com as the wildcard domain, then the route hostname is as follows:

frontend-api.apps.example.com

Important

The DNS server that hosts the wildcard domain is unaware of any route hostnames; it resolves any name only to the configured IPs. Only the RHOCP router knows about route hostnames, and treats each one as an HTTP virtual host.

Invalid wildcard domain hostnames, or hostnames that do not correspond to any route, are blocked by the RHOCP router and result in an HTTP 503 error.

Consider the following settings when creating a route:

  • The name of a service. The route uses the service to determine the pods to direct the traffic to.

  • A hostname for the route. A route is always a subdomain of your cluster wildcard domain. For example, if you are using a wildcard domain of apps.dev-cluster.acme.com, and need to expose a frontend service through a route, then the route name is as follows:

    frontend.apps.dev-cluster.acme.com.

    RHOCP can also automatically generate a hostname for the route.

  • An optional path, for path-based routes.

  • A target port that the application listens to. The target port corresponds to the port that you define in the targetPort key of the service.

  • An encryption strategy, depending on whether you need a secure or an insecure route.

The following listing shows a minimal definition for a route:

kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: a-simple-route 1
  labels: 2
    app: API
    name: api-frontend
spec:
  host: api.apps.acme.com 3
  to:
    kind: Service
    name: api-frontend 4
  port: 8080 5
    targetPort: 8443

1

The name of the route. This name must be unique.

2

A set of labels that you can use as selectors.

3

The hostname of the route. This hostname must be a subdomain of your wildcard domain, because RHOCP routes the wildcard domain to the routers.

4

The service to redirect the traffic to. Although you use a service name, the route uses this information only to determine the list of pods that receive the traffic.

5

Port mapping from a router to an endpoint in the service endpoints. The target port on pods that are selected by the service that this route points to.

Note

Some ecosystem components have an integration with ingress resources, but not with route resources. For this case, RHOCP automatically creates managed route objects when an ingress object is created. These route objects are deleted when the corresponding ingress objects are deleted.

You can delete a route by using the oc delete route route-name command.

[user@host ~]$ oc delete route myapp-route

You can also expose a service from the web console by clicking the Networking → Routes menu. Click Create Route and customize the name, the hostname, the path, and the service to route to by using the form view or the YAML manifest.

Using Ingress Objects for External Connectivity

An ingress is a Kubernetes resource that provides some of the same features as routes (which are an RHOCP resource). Ingress objects accept external requests and transfer the requests based on the route. You can enable only certain types of traffic: HTTP, HTTPS and server name identification (SNI), and TLS with SNI. Standard Kubernetes ingress resources are typically minimal. Many common features that applications rely on, such as TLS termination, path redirecting, and sticky sessions, depend on the ingress controller. Kubernetes does not define the configuration syntax. In RHOCP, routes are generated to meet the conditions that the ingress object specifies.

Note

The ingress resource is commonly used for Kubernetes. However, the route resource is the preferred method for external connectivity in RHOCP.

To create an ingress object, use the oc create ingress ingress-name --rule=URL_route=service-name:port-number command. Use the --rule option to provide a custom rule in the host/path=service:port[,tls=secretname] format. If the TLS option is omitted, then an insecure route is created.

[user@host ~]$ oc create ingress ingr-sakila \
--rule="ingr-sakila.apps.ocp4.example.com/*=sakila-service:8080"

The following listing shows a minimal definition for an ingress object:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: frontend 1
spec:
  rules: 2
  - host: "www.example.com 3
    http:
      paths:
      - backend: 4
          service:
            name: frontend
            port:
              number: 80
        pathType: Prefix
        path: /
  tls: 5
  - hosts:
    - www.example.com
    secretName: example-com-tls-certificate

1

The name of the ingress object. This name must be unique.

2

The HTTP or HTTPS rule for the ingress object.

3

The host for the ingress object. Applies the HTTP rule to the inbound HTTP traffic of the specified host.

4

The backend to redirect traffic to. Defines the service name, port number, and port names for the ingress object. To connect to the back end, incoming requests must match the host and path of the rule.

5

The configuration of TLS for the ingress object; it is required for secured paths. The host in the TLS object must match the host in the rules object.

You can delete an ingress object by using the oc delete ingress ingress-name command.

[user@host ~]$ oc delete ingress example-ingress

Sticky Sessions

Sticky sessions enable stateful application traffic by ensuring that all requests reach the same endpoint. RHOCP uses cookies to configure session persistence for ingress and route resources. The ingress controller selects an endpoint to handle any user requests, and creates a cookie for the session. The cookie is passed back in response to the request, and the user sends back the cookie with the next request in the session. The cookie tells the ingress controller which endpoint is handling the session, to ensure that client requests use the cookie so that they are routed to the same pod.

RHOCP auto-generates the cookie name for ingress and route resources. You can overwrite the default cookie name by using the annotate command with either the kubectl or the oc commands. With this annotation, the application that receives route traffic knows the cookie name.

The following example configures a cookie for an ingress object:

[user@host ~]$ oc annotate ingress ingr-example \
ingress.kubernetes.io/affinity=cookie

The following example configures a cookie named myapp for a route object:

[user@host ~]$ oc annotate route route-example \
router.openshift.io/cookie_name=myapp

After you annotate the route, capture the route hostname in a variable:

[user@host ~]$ ROUTE_NAME=$(oc get route <route_name> \
-o jsonpath='{.spec.host}')

Then, use the curl command to save the cookie and access the route:

[user@host ~]$ curl $ROUTE_NAME -k -c /tmp/cookie_jar

The cookie is passed back in response to the request, and is saved to the /tmp/cookie_jar directory. Use the curl command and the cookie that was saved from the previous command to connect to the route:

[user@host ~]$ curl $ROUTE_NAME -k -b /tmp/cookie_jar

By using the saved cookie, the request is sent to the same pod as the previous request.

Load Balance and Scale Applications

Developers and administrators can choose to manually scale the number of replica pods in a deployment. More pods might be needed for an anticipated surge in traffic, or the pod count might be reduced to reclaim resources that the cluster can use elsewhere.

You can change the number of replicas in a deployment resource manually by using the oc scale command.

[user@host ~]$ oc scale --replicas 5 deployment/scale

The deployment resource propagates the change to the replica set. The replica set reacts to the change by creating pods (replicas) or by deleting existing ones, depending on whether the new intended replica count is less than or greater than the existing count.

Although you can manipulate a replica set resource directly, the recommended practice is to manipulate the deployment resource instead. A new deployment creates either a replica set or a replication controller, and direct changes to a previous replica set or replication controller are ignored.

Load Balance Pods

A Kubernetes service serves as an internal load balancer. Standard services act as a load balancer or a proxy, and give access to the workload object by using the service name. A service identifies a set of replicated pods to transfer the connections that it receives.

A router uses the service selector to find the service and the endpoints, or pods, that back the service. When both a router and a service provide load balancing, RHOCP uses the router to load-balance traffic to pods. A router detects relevant changes in the IP addresses of its services, and adapts its configuration accordingly. Custom routers can thereby communicate modifications of API objects to an external routing solution.

RHOCP routers map external hostnames, and load-balance service endpoints over protocols that pass distinguishing information directly to the router. The hostname must exist in the protocol for the router to determine where to send it.

References

For more information, refer to the About Networking section in the Red Hat OpenShift Container Platform 4.14 Networking documentation at https://docs.redhat.com/en/documentation/openshift_container_platform/4.14/html-single/networking/index#about-networking