KubeOVN AirGap Issue On Harvester: Image Pull Backoff

Hey guys! Today, we're diving deep into a tricky issue encountered while trying to set up KubeOVN in an air-gapped Harvester environment. Specifically, we're talking about those pesky image pull backoffs that can halt your deployment in its tracks. If you've been wrestling with this, you're in the right place. Let's break down the problem, explore the steps to reproduce it, discuss the expected behavior, and even touch on potential workarounds. So, buckle up, and let's get started!

Describe the Bug

The main problem we're tackling is image pull backoffs that occur when trying to deploy KubeOVN in an air-gapped Harvester setup. The specific image causing the trouble is docker.io/kubeovn/kube-ovn:v1.14.10. This means that the system is unable to retrieve the necessary image from the Docker Hub registry, which is a critical component for KubeOVN to function correctly. In an air-gapped environment, this is especially problematic because the system doesn't have direct access to external registries.

When you encounter an image pull backoff, it signifies that Kubernetes is repeatedly trying and failing to pull the specified image. This can be due to various reasons, such as incorrect image name, network issues, or, in our case, the unavailability of external registries in an air-gapped setup. The error messages associated with image pull backoffs usually provide clues about the root cause, but in this scenario, the primary culprit is the lack of connectivity to docker.io.

Diagnosing an image pull backoff typically involves inspecting the Kubernetes events and pod status. You can use commands like kubectl describe pod <pod-name> to view the events related to a specific pod. These events often contain detailed information about why the image pull failed, including the specific error message from the container runtime (like Docker or containerd). Understanding these error messages is crucial for pinpointing the exact reason for the failure and devising an appropriate solution.

Moreover, the version of KubeOVN being used, v1.14.10, is a key piece of information. It's essential to ensure that this version is compatible with your Harvester setup and that any required dependencies are also available in your air-gapped environment. Sometimes, specific versions of images have dependencies on other images or external resources, which can further complicate the deployment process in air-gapped scenarios.

To Reproduce

To reliably reproduce this bug, follow these steps meticulously:

1. AirGap Harvester: The first and foremost step is to ensure that your Harvester environment is completely air-gapped. This means it should have absolutely no direct connection to the internet. This is crucial because the issue specifically arises when the system cannot reach external image registries.
2. Enable OVN addon: Once your Harvester environment is air-gapped, the next step is to enable the OVN (Open Virtual Networking) addon. This addon is responsible for providing the network fabric for your Kubernetes cluster. You can usually enable this addon through the Harvester UI or via command-line tools, depending on your setup.
3. Deployment Hangs: After enabling the OVN addon, initiate the deployment process. What you'll observe is that the deployment will not complete successfully. Instead, it will get stuck in an image pull backoff loop. This is because the system is trying to pull the docker.io/kubeovn/kube-ovn:v1.14.10 image, but it cannot reach the Docker Hub registry to download it.

During the reproduction process, it's helpful to monitor the status of the pods and deployments using kubectl. This allows you to observe the image pull errors in real-time and confirm that the issue is indeed an image pull backoff. Pay close attention to the events associated with the failing pods, as they will provide detailed error messages that can help you further diagnose the problem.

It's also worth noting that the exact steps to enable the OVN addon may vary depending on the version of Harvester you are using. Consult the official Harvester documentation for specific instructions on enabling addons in your environment. Additionally, ensure that your air-gapped environment is properly configured to use a local image registry, if you have one, as this can help mitigate the issue once you have the necessary images available locally.

Expected Behavior

In an ideal scenario, enabling embedded addons, even if they are marked as experimental, should succeed without encountering image pull backoff errors. This is especially crucial in air-gapped environments where direct access to external registries is not available. The expected behavior is that the system should either:

1. Use Pre-existing Images: If the required images are already available in a local image registry or pre-loaded on the nodes, the system should seamlessly use those images without attempting to pull from external registries.
2. Provide Clear Error Messages: If the required images are not available, the system should provide clear and actionable error messages indicating that the images are missing and need to be manually loaded or made available in a local registry.
3. Offer Configuration Options: The system could offer configuration options to specify the location of the image registry to use. This would allow users to point the system to a local registry where the required images are stored.

Essentially, the goal is to ensure that the deployment process is as smooth and straightforward as possible, even in the absence of direct internet connectivity. This requires careful planning and consideration of how images are managed and distributed in air-gapped environments. The system should be designed to handle such scenarios gracefully, providing users with the necessary tools and information to successfully deploy and run their applications.

Support Bundle for Troubleshooting

Unfortunately, in this particular bug report, a support bundle for troubleshooting is not available (n/a). A support bundle typically contains a collection of logs, configuration files, and other diagnostic information that can be invaluable for debugging issues. However, even without a support bundle, the information provided in the bug report can still be used to investigate the problem and identify potential solutions.

If you encounter this issue and are able to generate a support bundle, it would be highly beneficial to include it in your bug report. This will provide developers with more context and make it easier for them to diagnose the root cause of the problem. Support bundles can often be generated using command-line tools or through the Harvester UI, depending on the version you are using. Consult the official Harvester documentation for instructions on how to generate a support bundle.

In the absence of a support bundle, you can still gather relevant information manually. This includes:

• Kubernetes Events: Examine the events associated with the failing pods and deployments using kubectl describe. These events often contain detailed error messages and can provide clues about the cause of the failure.
• Node Logs: Check the logs on the nodes where the failing pods are running. These logs may contain information about image pull failures or other errors that are preventing the pods from starting.
• Harvester Configuration: Review the Harvester configuration to ensure that it is properly configured for an air-gapped environment. This includes settings related to image registries, network connectivity, and addon management.

Environment

• Harvester version: master-head. This indicates that the bug was observed on the master-head version of Harvester. This is important because it helps developers understand the specific codebase where the bug is occurring. The master-head version typically refers to the latest development version of the software, which may contain features and bug fixes that are not yet available in stable releases.

When reporting bugs, it's always a good idea to include the specific version of the software you are using. This helps developers reproduce the issue and ensure that their fixes are targeted at the correct version. Additionally, it's helpful to provide information about your environment, such as the operating system, hardware configuration, and any other relevant details that may be contributing to the problem.

In this case, knowing that the bug occurs on the master-head version of Harvester allows developers to focus their efforts on the most recent changes and identify any regressions that may have been introduced. It also helps them prioritize the bug fix based on the impact it has on users who are running the latest development version of the software.

Additional Context

The included image visually confirms the image pull backoff issue, showing the system's inability to retrieve the necessary KubeOVN image. This visual evidence reinforces the problem description and provides additional context for developers to understand the issue. Screenshots and other visual aids can be incredibly helpful when reporting bugs, as they can often convey information more effectively than text alone.

In this case, the screenshot clearly shows the error messages associated with the image pull backoff, including the specific image that is failing to download and the reason for the failure. This allows developers to quickly identify the problem and begin investigating potential solutions. Additionally, the screenshot may contain other relevant information, such as the timestamps of the errors and the names of the affected pods and deployments.

When including screenshots in bug reports, it's important to ensure that they are clear and easy to read. Crop the image to focus on the relevant area and use annotations to highlight specific details. Additionally, provide a brief description of what the screenshot is showing and why it is relevant to the bug report.

Workaround and Mitigation

The suggested workaround involves manually adding or sideloading the image. This is a common approach in air-gapped environments where direct access to external registries is not available. Sideloading an image typically involves downloading the image from a machine with internet access and then transferring it to the air-gapped environment using a USB drive, network share, or other means.

Once the image is transferred, it can be loaded into the local image registry using commands like docker load or ctr image import, depending on the container runtime being used. After the image is loaded, Kubernetes should be able to pull it from the local registry without encountering the image pull backoff error.

While sideloading images can be an effective workaround, it can also be a time-consuming and error-prone process. It's important to ensure that the correct image is downloaded and that it is loaded into the local registry correctly. Additionally, it's important to keep track of the images that have been sideloaded and to update them periodically to ensure that they are up-to-date with the latest security patches and bug fixes.

In addition to sideloading images, another mitigation strategy is to use a local image registry that mirrors the images from external registries. This allows you to download the images once and then serve them from the local registry, eliminating the need to sideload them individually. Local image registries can be set up using tools like Harbor, Nexus, or Artifactory.