In this blogpost I’ll explain how we have set up Openshift 4.5 on a VMWare environment using the ‘User Provided Infrastructure’ installation method. Whereas Openshift 4,1 and 4.2 had mandatory requirements to set the network up using DHCP, version 4.3 was the first Openshift version to mention that static IP’s were possible. Unfortunately, the standard documentation doesn’t describe how this can be achieved. We used a hands-on approach adjusting the official ignition files for each node. As it turns out, a similar method was later described in a new support article from Red Hat. To indicate this was a real issue for a lot for several companies, the release notes for Openshift 4.6 even say that the new CoreOS OVA file contains an out-of-the-box approach for setting static IP’s, even though this isn’t described in the documentation yet.
Before we can explain how the static files are added to each node, you must understand how the installation of Openshift on VMware is achieved. First you download the Openshift-Installer binary which is used to generate ignition files. The Openshift-Installer expects an install-config.yaml, a simple configuration file where you enter the cluster name, pull secret (for subscriptions) etc. When the Openshift-Installer is finished, you’ll get 3 ignition files: one for bootstrap, one for masters and one for workers. Note that there is no dedicated ignition file for ‘infra nodes’, as these are simply workers with an additional label. To build your cluster, simply download the latest CoreOS OVA (a virtual disk), import it to VMWare and add corresponding ignition files as base64 encoded strings.
The VM should be connected to a vSwitch which connects to your company’s network. If you can manipulate the DHCP server of this network, the official documentation states you should set an indefinite IP Address to your VM’s MAC address. If you can do this, you’re all set. In our case, we had no direct access to the DHCP so we could not match the VM MAC address to an IP address. Instead, we used DHCP to get an initial IP and added a static IP file which was active after reboot. Unfortunately this means you always need DHCP to give you an initial IP Address (unless you use Openshift 4.6 or above).
To set a static IP for a VM, first we used the following template:
TYPE=Ethernet PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=none IPADDR=__IP__ PREFIX=__PREFIX__ GATEWAY=__GW__ DNS1=__NAMESERVER1__ DNS2=__NAMESERVER2__ DEFROUTE=yes IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no IPV6_ADDR_GEN_MODE=stable-privacy NAME="ens192" DEVICE=ens192 ONBOOT=yes AUTOCONNECT_PRIORITY=-999
For each host we would fill in the variables (indicated with the double underscore) . As you can imagine, this turned out to be incredibly cumbersome for a lot of nodes, which is why we generated these files using a simple bash script. After the file for the specific node was created, we would simply base64 encode the file, using:
cat <filename> | base64 -w0
The output will look like this
VFlQRT1FdGhlcm5ldApQUk9YWV9NRVRIT0Q9bm9uZQpCUk9XU0VSX09OTFk9bm8KQk9PVFBST1RPPW5vbmUKSVBBRERSPTE5Mi4xNjguMC4xMApQUkVGSVg9MjQKR0FURVdBWT0xOTIuMTY4LjAuMQpETlMxPTguOC44LjgKRE5TMj04LjguNC40CkRFRlJPVVRFPXllcwpJUFY0X0ZBSUxVUkVfRkFUQUw9bm8KSVBWNklOSVQ9eWVzCklQVjZfQVVUT0NPTkY9eWVzCklQVjZfREVGUk9VVEU9eWVzCklQVjZfRkFJTFVSRV9GQVRBTD1ubwpJUFY2X0FERFJfR0VOX01PREU9c3RhYmxlLXByaXZhY3kKTkFNRT0iZW5zMTkyIgpERVZJQ0U9ZW5zMTkyCk9OQk9PVD15ZXMKQVVUT0NPTk5FQ1RfUFJJT1JJVFk9LTk5OQo=
If you’re not sure if the base64 encoding worked, or if you simply want to check the contents of a base64 string, just type
base64 -d <your_base64_string>
Now that we have the static IP configuration file set for the VM, it’s time to manipulate the default ignition files you’d get from the Openshift-Installer binary. If you look at these ignition files, you’ll see that they contain an empty “storage” section.
{
"ignition": {
"config": {
"append": [
{
"source": "https://<your_internal_api_loadbalancer>:22623/config/worker",
"verification": {}
}
]
},
"security": {
"tls": {
"certificateAuthorities": [
{
"source": "data:text/plain;charset=utf-8;base64,<some_generated_base64_string>",
"verification": {}
}
]
}
},
"timeouts": {},
"version": "2.2.0"
},
"networkd": {},
"passwd": {},
"storage": {}, #<------ alter this section
"systemd": {}
}
In our script, we’d replace that section with the contents of the base64 encoded static file. The end result should look like this:
"storage": {
"files": [
{
"filesystem": "root",
"path": "/etc/sysconfig/network-scripts/ifcfg-ens192",
"mode": 420,
"contents": {
"source": "data:text/plain;charset=utf-8;base64,VFlQRT1FdGhlcm5ldApQUk9YWV9NRVRIT0Q9bm9uZQpCUk9XU0VSX09OTFk9bm8KQk9PVFBST1RPPW5vbmUKSVBBRERSPTE5Mi4xNjguMC4xMApQUkVGSVg9MjQKR0FURVdBWT0xOTIuMTY4LjAuMQpETlMxPTguOC44LjgKRE5TMj04LjguNC40CkRFRlJPVVRFPXllcwpJUFY0X0ZBSUxVUkVfRkFUQUw9bm8KSVBWNklOSVQ9eWVzCklQVjZfQVVUT0NPTkY9eWVzCklQVjZfREVGUk9VVEU9eWVzCklQVjZfRkFJTFVSRV9GQVRBTD1ubwpJUFY2X0FERFJfR0VOX01PREU9c3RhYmxlLXByaXZhY3kKTkFNRT0iZW5zMTkyIgpERVZJQ0U9ZW5zMTkyCk9OQk9PVD15ZXMKQVVUT0NPTk5FQ1RfUFJJT1JJVFk9LTk5OQo="
}
}
]
},
Note that we’ll create an ignition file for each node, instead of using an ignition file for a node-type as described in the official installation! Also, note that you should modify the append-bootstrap.ign file rather then the bootstrap.ign file itself.
Next, we would encode the entire modified ignitionfile in base64, (just as you normally would) and add it to the guestinfo.ignition.config.data parameter. When the VM boots, it will get a DHCP IP address and it will try to reach the URL defined in the ignition file. If you look in the sample ignition file above, you’ll see it resolves to https://<your_internal_api_loadbalancer>:22623/config/worker. The URL should resolve to the etcd (or bootstrap) node, where ignition downloads some additional configuration and reboots. After booting, it will read the new file you wrote to /etc/sysconfig/network-scripts/ifcfg-ens192 and the static IP Address will be used. If this IP Address is in your company’s DNS, it will also grab the hostname which matches that IP.
Mission accomplished!
We did run into a small issue using this method. The control plane in our scenario was not in the same network as the worker nodes, so a firewall was preventing us from reaching the control plane when booting up worker nodes. We couldn’t figure out why, as we made firewall exceptions for the static IP’s we added. As it turns out, the initial DHCP IP Address obviously didn’t match the static IP’s in the firewall whitelist and access to https://<your_internal_api_loadbalancer>:22623/config/worker was blocked. As a result, the CoreOS Ignition never rebooted and never received the static IP Address (which was whitelisted in the firewall). A real catch-22 as a manual reboot didn’t help either. If you hit this issue, you still have 3 options:
- You can use Openshift 4.6 which shouldn’t need the initial DHCP IP Adress
- You can whitelist the DHCP IP range in your firewall (which was not accepted in our case)
- You can add the IP manually in GRUB as a kernel parameter.
Using option 3 is obviously really cumbersome but this was the only option in our case. While booting the VM, simply hit ‘e’ when the grub menu shows up in the VMware Console. There, change the kernel options and replace
'ip=dhcp,dhcp6'
to
'ip=<your_ip>::<your_gateway>:<your_subnet>:::none nameserver=<your_nameserver>'
Next, hit CTRL-X and the VM should now boot with the proper IP Address. As this IP Address is whitelisted in your firewall, it will download the proper configuration and reboot. With the reboot your manual kernel parameters will disappear and the file in /etc/sysconfig/network-scripts/ifcfg-ens192 will finally be used.
Hopefully you’ll never need this trick as Static IP support for future Openshift releases should improve.
__ATA.cmd.push(function() {
__ATA.initDynamicSlot({
id: ‘atatags-26942-5fab069c6702c’,
location: 120,
formFactor: ‘001’,
label: {
text: ‘Advertisements’,
},
creative: {
reportAd: {
text: ‘Report this ad’,
},
privacySettings: {
text: ‘Privacy settings’,
}
}
});
});
