When installing Openshift on-premise or in the cloud, by default you’ll get a single ingress controller. It’s the well known *.apps ingress controllers and it forwards all traffic from *.apps.yourcluster.yourdomain to the pods. However, for on of our installations we needed multiple ingress controllers. In other words, we had multiple entry points to our cluster, ranging from *.management for the console, to *.dev and *.prod for our various workloads. You can set this up, but it’s not part of the vanilla installation. Also, setting it up required some fiddeling with the default wildcard certificates.
You might wonder why someone would need this?
For example, out of the box you can perfectly set up all routes on the default ingress controller. The routes would look like: frontend-dev.apps.yourcluster.yourdomain, frontend-prod.apps.yourcluster.yourdomain etc. This would all fit within the default ingress controller and it’s all covered with your wildcard certificate.
However in our case we were dealing with multiple (seperated) networks. The controlplane was installed on network A, the DEV infranodes and workers needed to be installed on network B and the PROD workload needed to be installed in network C. Even though the cluster is stretched on top of those networks and all pods can talk directly though the SDN, ingress traffic was expected to come in at the corresponding network level. And as infranodes are not part of any subscriptions, you can build as many as you need!
First we needed to rename the default ingress controller. You need to do this before you install your cluster, as this cannot be renamed afterwards! When first running the installation for openshift 4.x, you will create some manifests using
./openshift-install create manifests --dir=<installation_directory>
After you get these manifest files, go to the folder ‘manifests’ inside the installation_directory and edit the file ‘cluster-ingress-02-config.yml’. In here, set your ‘domain’ value to match your new default ingress controller value. You can find the link to the Red Hat support article here. After adjusting the file, you can run the next step of the installer. Make sure you set up dedicated infranodes using this guide.
Next, when you have your cluster you need to label the infranodes. Let’s assume you label the infranodes ‘my/zone=management’, ‘my/zone=dev’ and ‘my/zone=prod’ for the management, dev and infranodes respectively. Out of the box, the ingress controller will push router pods on all infranodes, but you don’t want this, as *.management traffic is only allowed on the orange ‘my/zone=management’ infranodes shown above. To fix this, patch your default ingress controller:
oc edit ingresscontroller default -n openshift-ingress-operator -o yaml
and set
spec:
nodePlacement:
nodeSelector:
matchLabels:
node-role.kubernetes.io/infra: ""
my/zone: management
you can check if the default router pods are only running on the ‘management’ infranodes by running
oc get pods -n openshift-ingress -owide
Now for the fun part.. setting up additional ingress controllers. Take a look at this ingresscontrollers.yaml file:
apiVersion: v1
items:
- apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
name: dev
namespace: openshift-ingress-operator
spec:
domain: dev.yourcluster.yourdomain
nodePlacement:
nodeSelector:
matchLabels:
my/zone: dev
node-role.kubernetes.io/infra: ''
routeSelector:
matchLabels:
my/env: dev
routeAdmission:
namespaceOwnership: InterNamespaceAllowed
- apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
name: prod
namespace: openshift-ingress-operator
spec:
domain: prod.yourcluster.yourdomain
nodePlacement:
nodeSelector:
matchLabels:
my/zone: prod
node-role.kubernetes.io/infra: ''
routeSelector:
matchLabels:
my/env: prod
routeAdmission:
namespaceOwnership: InterNamespaceAllowed
This will create two additional ingress controllers: ‘dev’ and ‘prod’ in the ‘openshift-ingress-operator’ namespace. The ‘domain’ section for each ingress controller matches the FQDN on the loadbalancer (with the wildcard CNAME) and the router pods are only pushed to the infra nodes with the proper label.
To apply these ingress controllers, use
oc apply -f ingresscontrollers.yaml
To view your ingress controllers, type
oc get ingresscontroller -n openshift-ingress-operator
(this might seem confusing, as your actual router pods are pushed to ‘openshift-ingress’ instead of ‘openshift-ingress-operator’.)
By default, some secrets are created containing wildcard certificates for these new ingress controllers. As these are signed by the ingress operator, they’re probably not valid inside your organisation. To replace them, run
cat <dev wildcard certificate> <intermediate certificate> <root certificate> > dev.crt oc delete secret/router-certs-dev -n openshift-ingress oc create secret tls router-certs-dev --cert=dev.crt --key=dev.key -n openshift-ingress
repeat this step for your prod wildcard certificate.
Now that you have set up each ingress controller on their own set of infranodes, you are almost done. There is one thing you need to do. If you create a route for e.g. your frontend webserver, the route will attach itself to each ingress controller. You might have seen the section in the yaml that is designed to prevent this:
routeSelector:
matchLabels:
my/env: prod
By creating routes with ‘my/env=prod’, you are certain that the route is accepted by the prod ingress controller. However, the default ingress controller doesn’t have a ‘routeSelector’ and will accept any route! To make sure the route is only exposed on the ingress controller you want, patch the default ingress controller to ignore routes with ‘my/env=dev’ or ‘my/env=prod’ labels:
oc patch --type=merge -p '{"spec":{"routeSelector":{"matchExpressions":[{"key":"my/env","operator":"NotIn","values":["dev","prod"]}]}}}' ingresscontroller default -n openshift-ingress-operator
And that’s it! Any out-of-the-box route is now exposed in the orange ‘ops’ network and any route with a dev or prod label is exposed on the proper corresponding network.
In my next blogpost I’ll describe how to set up additional ingresscontrollers on the same infranodes using nodeportservices. The goal there is to get traffic into multiple Openshift clusters.
