Problem Alignment

The Problem

Customers typically run more than one cluster and/or applications deployed across different regions. In such a hybrid cloud environment, aggregating metrics is a key requirement to avoid admins and or applications owners to drop in into individual clusters to troubleshoot specific problems. And since Red Hat does not offer a standalone metrics aggregation service, customers have started to use existing, home-grown technologies based on, for example, InfluxDB or Kafka to achieve that.

In summary:

• OpenShift Monitoring is optimized for short-term retention only.
• Red Hat does not offer a central metrics aggregation service yet.
• Customers use existing, home-grown technologies to distribute information across other stakeholders in their company.

High-Level Approach

Expose Prometheus remote-write configuration via our OpenShift Monitoring (Cluster and User Workload) ConfigMap to allow customers to push time-series data to a remote location.

Please note that we do not plan to support certain third party “receivers” with this solution. Customers will be responsible to ensure an appropriate receiving component is up and running that implements the “remote-write” API. Here is a list of possible “receiver” plugins.

Goal & Success

• Introduce some “ease of use” features to configure certain parts for remote-write to decrease possible misconfigurations.
• Allow customers to push metrics off the cluster to allow aggregation use cases and more options for our partners to integrate into OpenShift - e.g. to allow long-term retention or security/analytics scenarios.

Solution Alignment

Key Capabilities

• As an OpenShift administrator, I want to configure remote-write for both the OOTB infrastructure bundle and the user workload stack, so that time-series data will be available on the system of my choice.
• As an OpenShift administrator, I want to easily build an allow list of metrics that should be pushed externally.

Key Flows

User configures one of the available ConfigMaps to allow node_cpu_seconds_total to be written into a remote Thanos system.

• Administrator opens the cluster-monitoring-config ConfigMap.
• They add a new field to configure remote write.
• They add the node_cpu_seconds_total metric to the allow list.
• They add the remote URL for the Thanos receiver.
• They add a Secret to configure authentication against the remote service.

Additional resources

• https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage
• https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write

Feature Overview

• This Section:* High-Level description of the feature ie: Executive Summary

• Note: A Feature is a capability or a well defined set of functionality that delivers business value. Features can include additions or changes to existing functionality. Features can easily span multiple teams, and multiple releases.

Goals

• This Section:* Provide high-level goal statement, providing user context and expected user outcome(s) for this feature

• …

Requirements

• This Section:* A list of specific needs or objectives that a Feature must deliver to satisfy the Feature.. Some requirements will be flagged as MVP. If an MVP gets shifted, the feature shifts. If a non MVP requirement slips, it does not shift the feature.

(Optional) Use Cases

This Section: 

• Main success scenarios - high-level user stories

• Alternate flow/scenarios - high-level user stories

• ...

Questions to answer…

• ...

Out of Scope

• …

Background, and strategic fit

This Section: What does the person writing code, testing, documenting need to know? What context can be provided to frame this feature.

Assumptions

• ...

Customer Considerations

• ...

Documentation Considerations

Questions to be addressed:

• What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?

• Does this feature have doc impact?  

• New Content, Updates to existing content,  Release Note, or No Doc Impact

• If unsure and no Technical Writer is available, please contact Content Strategy.

• What concepts do customers need to understand to be successful in [action]?

• How do we expect customers will use the feature? For what purpose(s)?

• What reference material might a customer want/need to complete [action]?

• Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.

• What is the doc impact (New Content, Updates to existing content, or Release Note)?

Feature Overview

As a OpenShift administrator, I would like a solution that allows me to upgrade from one EUS version to another with very few steps and only minimum disruption to application workloads while still allowing new application services to be deployed.

Goals

4.8

• Spike, Design, and Scope
• Begin foundational development if possible

4.9

• Foundational items delivered and back ported as necessary

4.10

• Remaining delivery artifacts complete
• Documentation and enablement complete
• Full testing complete

Requirements

Functional requirements break down into the following prioritized list:

1. Make serial upgrades safe
   1. Prevent upgrades before the core components are ready (version skewing, incompatible APIs)
   2. Prevent upgrades before operators or ready
      1. Ensure Operators have a way to express max version
      2. Ensure OLM policy is clear on what happens if max version is not specified
   3. Make back pressure items (reasons you cannot upgrade) clear to administrators along with the actions to resolve
   4. CI MUST be running with test automation
   5. Note: Forcing an upgrade is still possible
2. Make updates faster
   1. Optimize where possible to increase speed of upgrade for core components (SDN/Daemonsets)
3. Reduce the amount of workload disruption
   1. Work load disruption is not just reboots it is any disruption to workloads during the upgrade, of which a reboot is likely the worst case scenario.  This may also include things like rescheduling of workloads.
   2. We will not change the model of how components are deployed, changes to the host still require a reboot
   3. Discover and document any necessary guidelines to reduce the number of items that are developed which would cause a reboot between EUS releases where possible (4.8, 4.9).  
   4. As a stretch goal, discover if it is possible to reduce the reboots between 4.6 and 4.7 
4. Should take into consideration clusters with RHEL workers

Non-Functional Requirements

Questions to answer…

• See https://docs.google.com/document/d/17I1Wd7-R1wRxmboyv1jUFHFkqQcBTorJccdGi1ZqjQE/edit# for Q&A

Out of Scope

• It is not intended to support version skews that fall outside the upstream version skew policy
• It is not intended to eliminate all reboots
• It is not intended to skip releases at this time

Documentation Considerations

Questions to be addressed:

• What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
• Does this feature have doc impact?
• New Content, Updates to existing content, Release Note, or No Doc Impact
• If unsure and no Technical Writer is available, please contact Content Strategy.
• What concepts do customers need to understand to be successful in [action]?
• How do we expect customers will use the feature? For what purpose(s)?
• What reference material might a customer want/need to complete [action]?
• Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
• What is the doc impact (New Content, Updates to existing content, or Release Note)?

Feature Overview

We drive OpenShift cross-market customer success and new customer adoption with constant improvements and feature additions to the existing capabilities of our OpenShift Core Networking (SDN and Network Edge). This feature captures that natural progression of the product.

Goals

• Feature enhancements (performance, scale, configuration, UX, ...)
• Modernization (incorporation and productization of new technologies)

Requirements

• Core Networking Stability
• Core Networking Performance and Scale
• Core Neworking Extensibility (Multus CNIs)
• Core Networking UX (Observability)
• Core Networking Security and Compliance

In Scope

• Network Edge (ingress, DNS, LB)
• SDN (CNI plugins, openshift-sdn, OVN, network policy, egressIP, egress Router, ...)
• Networking Observability

Out of Scope

There are definitely grey areas, but in general:

• CNV
• Service Mesh
• CNF

Documentation Considerations

Questions to be addressed:

• What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
• Does this feature have doc impact?
• New Content, Updates to existing content, Release Note, or No Doc Impact
• If unsure and no Technical Writer is available, please contact Content Strategy.
• What concepts do customers need to understand to be successful in [action]?
• How do we expect customers will use the feature? For what purpose(s)?
• What reference material might a customer want/need to complete [action]?
• Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
• What is the doc impact (New Content, Updates to existing content, or Release Note)?

Feature Overview

Plugin teams need a mechanism to extend the OCP console that is decoupled enough so they can deliver at the cadence of their projects and not be forced in to the OCP Console release timelines.

The OCP Console Dynamic Plugin Framework will enable all our plugin teams to do the following:

• Extend the Console
• Deliver UI code with their Operator
• Work in their own git Repo
• Deliver at their own cadence

Goals

• • Operators can deliver console plugins separate from the console image and update plugins when the operator updates.
  • The dynamic plugin API is similar to the static plugin API to ease migration.
  • Plugins can use shared console components such as list and details page components.
  • Shared components from core will be part of a well-defined plugin API.
  • Plugins can use Patternfly 4 components.
  • Cluster admins control what plugins are enabled.
  • Misbehaving plugins should not break console.
  • Existing static plugins are not affected and will continue to work as expected.

Out of Scope

• • Initially we don't plan to make this a public API. The target use is for Red Hat operators. We might reevaluate later when dynamic plugins are more mature.
  • We can't avoid breaking changes in console dependencies such as Patternfly even if we don't break the console plugin API itself. We'll need a way for plugins to declare compatibility.
  • Plugins won't be sandboxed. They will have full JavaScript access to the DOM and network. Plugins won't be enabled by default, however. A cluster admin will need to enable the plugin.
  • This proposal does not cover allowing plugins to contribute backend console endpoints.

Requirements

 
 Documentation Considerations

Questions to be addressed:

• What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?

• Does this feature have doc impact?  

• New Content, Updates to existing content,  Release Note, or No Doc Impact

• If unsure and no Technical Writer is available, please contact Content Strategy.

• What concepts do customers need to understand to be successful in [action]?

• How do we expect customers will use the feature? For what purpose(s)?

• What reference material might a customer want/need to complete [action]?

• Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.

• What is the doc impact (New Content, Updates to existing content, or Release Note)?

Feature Overview

• This Section:* High-Level description of the feature ie: Executive Summary

• Note: A Feature is a capability or a well defined set of functionality that delivers business value. Features can include additions or changes to existing functionality. Features can easily span multiple teams, and multiple releases.

Goals

• This Section:* Provide high-level goal statement, providing user context and expected user outcome(s) for this feature

• …

Requirements

• This Section:* A list of specific needs or objectives that a Feature must deliver to satisfy the Feature.. Some requirements will be flagged as MVP. If an MVP gets shifted, the feature shifts. If a non MVP requirement slips, it does not shift the feature.

(Optional) Use Cases

This Section: 

• Main success scenarios - high-level user stories

• Alternate flow/scenarios - high-level user stories

• ...

Questions to answer…

• ...

Out of Scope

• …

Background, and strategic fit

This Section: What does the person writing code, testing, documenting need to know? What context can be provided to frame this feature.

Assumptions

• ...

Customer Considerations

• ...

Documentation Considerations

Questions to be addressed:

• What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?

• Does this feature have doc impact?  

• New Content, Updates to existing content,  Release Note, or No Doc Impact

• If unsure and no Technical Writer is available, please contact Content Strategy.

• What concepts do customers need to understand to be successful in [action]?

• How do we expect customers will use the feature? For what purpose(s)?

• What reference material might a customer want/need to complete [action]?

• Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.

• What is the doc impact (New Content, Updates to existing content, or Release Note)?

Feature Overview

OpenShift console supports new features and elevated experience for Operator Lifecycle Manager (OLM) Operators and Cluster Operators.

Goal:

OCP Console improves the controls and visibility for managing vendor-provided software in customers’ infrastructure and making these solutions available for customers' internal users.

To achieve this, 

• Operator Lifecycle Manager (OLM) teams have been introducing new features aiming towards simplification and ease of use for both developers and cluster admins.

• On the Cluster Operators side, the console iteratively improves the visibilities to the resources being associated with the Operators to improve the overall managing experience.

We want to make sure OLM’s and Cluster Operators' new features are exposed in the console so admin console users can benefit from them.

Benefits:

• Cluster admin/Operator consumers:
  • Able to see, learn, and interact with OLM managed and/or Cluster Operators associated resources in openShift console.

Requirements

(Optional) Use Cases
<--- Remove this text when creating a Feature in Jira, only for reference --->
* Main success scenarios - high-level user stories
* Alternate flow/scenarios - high-level user stories
* ...

Questions to answer...
How will the user interact with this feature?
Which users will use this and when will they use it?
Is this feature used as part of the current user interface?

Out of Scope
<--- Remove this text when creating a Feature in Jira, only for reference --->
# List of non-requirements or things not included in this feature
# ...

Background, and strategic fit
<--- Remove this text when creating a Feature in Jira, only for reference --->
What does the person writing code, testing, documenting need to know? What context can be provided to frame this feature.

Assumptions
<--- Remove this text when creating a Feature in Jira, only for reference --->
* Are there assumptions being made regarding prerequisites and dependencies?
* Are there assumptions about hardware, software or people resources?
* ...

Customer Considerations
<--- Remove this text when creating a Feature in Jira, only for reference --->
* Are there specific customer environments that need to be considered (such as working with existing h/w and software)?
...

Documentation Considerations
<--- Remove this text when creating a Feature in Jira, only for reference --->
Questions to be addressed:
* What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
* Does this feature have doc impact?
* New Content, Updates to existing content, Release Note, or No Doc Impact
* If unsure and no Technical Writer is available, please contact Content Strategy.
* What concepts do customers need to understand to be successful in [action]?
* How do we expect customers will use the feature? For what purpose(s)?
* What reference material might a customer want/need to complete [action]?
* Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
* What is the doc impact (New Content, Updates to existing content, or Release Note)?

OCP/Telco Definition of Done
Feature Template descriptions and documentation.
Feature Overview

• Connect OpenShift workloads to Google services with Google Workload Identity

• Enable customers to access Google services from workloads on OpenShift clusters using Google Workload Identity (aka WIF)
  https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity 

Goals

• Customers want to be able to manage and operate OpenShift on Google Cloud Platform with workload identity, much like they do with AWS + STS or Azure + workload identity.
• Customers want to be able to manage and operate operators and customer workloads on top of OCP on GCP with workload identity.

Requirements

• Add support to CCO for the Installation and Upgrade using both UPI and IPI methods with GCP workload identity.
• Support install and upgrades for connected and disconnected/restriction environments.
• Support the use of Operators with GCP workload identity with minimal friction.
• Support for HyperShift and non-HyperShift clusters.
• This Section:* A list of specific needs or objectives that a Feature must deliver to satisfy the Feature.. Some requirements will be flagged as MVP. If an MVP gets shifted, the feature shifts. If a non MVP requirement slips, it does not shift the feature.

(Optional) Use Cases

This Section:

• Main success scenarios - high-level user stories
• Alternate flow/scenarios - high-level user stories
• ...

Questions to answer…

• ...

Out of Scope

• …

Background, and strategic fit

This Section: What does the person writing code, testing, documenting need to know? What context can be provided to frame this feature.

Assumptions

• ...

Customer Considerations

• ...

Documentation Considerations

Questions to be addressed:

• What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
• Does this feature have doc impact?
• New Content, Updates to existing content, Release Note, or No Doc Impact
• If unsure and no Technical Writer is available, please contact Content Strategy.
• What concepts do customers need to understand to be successful in [action]?
• How do we expect customers will use the feature? For what purpose(s)?
• What reference material might a customer want/need to complete [action]?
• Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
• What is the doc impact (New Content, Updates to existing content, or Release Note)?

Goal:

Enable and support Multus CNI for microshift.

Background:

Customers with advanced networking requirement need to be able to attach additional networks to a pod, e.g. for high-performance requirements using SR-IOV or complex VLAN setups etc.

Requirements:

1. opt-in approach: customers can add multus if needed, e.g. by installing/adding "microshift-networking-multus" rpm package to their installation.
2. if possible, it would be good to be able to add multus  an existing installation. If that requires a restart/reboot, that is acceptable. If not possible, it has to be clearly documented. 
3. it is acceptable that once multus has been added to an installation, it can not be removed. If removal can be implemented easily, that would be good. If not possible, then it has to be clearly documented.  
4. Regarding additional networks:
   1. As part of the MVP, the Bridge plugin must be fully supported  
   2. As stretch goal,  macvlan and ipvlan plugins  should be supported
   3. Other plugins, esp. host device and sr-iov are out of scope for the MVP, but will be added with a later version.
   4. Multiple additional networks need to be configurable, e.g. two different bridges leading to two different networks, each consumed by different pods.
5. IP V6 with bridge plugin. Secondary NICs passed to a container via the bridge plugin should work with IP V6, if the consuming pod does support V6. See also "out of scope"
6. Regarding IPAM CNI support for IP address provisioning, static and DHCP must be supported.

Documentation:

1. In the existing "networking" book, we need a new chapter "4. Multiple networks". It can re-use a lot content from OCP doc "https://docs.openshift.com/container-platform/4.14/networking/multiple_networks/understanding-multiple-networks.html", but needs an extra chapter in the beginning "Installing support for multiple networks"{}

Testing:

1. A simple "smoke test" that multus can be added, and a 2nd nic added to a pod (e.g. using host device) is sufficient. No need to replicate all the multus tests from OpenShift, as we assume that if it works there, it works with MicroShift.

Customer Considerations:

• This document contains the MVP requirements of a MicroShift EAP customer that need to be considered.

Out of scope:

• Other plugins, esp. host device and sr-iov are out of scope for the MVP, but will be added with a later version.
• IP V6 support with OVN-K. That is scope of  feature OCPSTRAT-385 

• https://drive.google.com/file/d/18O2-TfhryQgd_oOYPjhl-DfPHNzctafB/view

Feature Overview

Enable sharing ConfigMap and Secret across namespaces

Requirements

Questions to answer…

NA

Out of Scope

NA

Background, and strategic fit

Consumption of RHEL entitlements has been a challenge on OCP 4 since it moved to a cluster-based entitlement model compared to the node-based (RHEL subscription manager) entitlement mode. In order to provide a sufficiently similar experience to OCP 3, the entitlement certificates that are made available on the cluster (OCPBU-93) should be shared across namespaces in order to prevent the need for cluster admin to copy these entitlements in each namespace which leads to additional operational challenges for updating and refreshing them. 

Documentation Considerations

Questions to be addressed:
 * What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
 * Does this feature have doc impact?
 * New Content, Updates to existing content, Release Note, or No Doc Impact
 * If unsure and no Technical Writer is available, please contact Content Strategy.
 * What concepts do customers need to understand to be successful in [action]?
 * How do we expect customers will use the feature? For what purpose(s)?
 * What reference material might a customer want/need to complete [action]?
 * Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
 * What is the doc impact (New Content, Updates to existing content, or Release Note)?

Feature Overview

Reduce the OpenShift platform and associated RH provided components to a single physical core on Intel Sapphire Rapids platform for vDU deployments on SingleNode OpenShift.

Goals

• Reduce CaaS platform compute needs so that it can fit within a single physical core with Hyperthreading enabled. (i.e. 2 CPUs)
• Ensure existing DU Profile components fit within reduced compute budget.
• Ensure existing ZTP, TALM, Observability and ACM functionality is not affected.
• Ensure largest partner vDU can run on Single Core OCP.

Requirements

(Optional) Use Cases

• Main success scenario: A telecommunications provider uses ZTP to provision a vDU workload on Single Node OpenShift instance running on an Intel Sapphire Rapids platform. The SNO is managed by an ACM instance and it's lifecycle is managed by TALM.

Questions to answer...

• N/A

Out of Scope

• Core budget reduction on the Remote Worker Node deployment model.

Background, and strategic fit

Assumptions

• The more compute power available for RAN workloads directly translates to the volume of cell coverage that a Far Edge node can support.
• Telecommunications providers want to maximize the cell coverage on Far Edge nodes.
• To provide as much compute power as possible the OpenShift platform must use as little compute power as possible.
• As newer generations of servers are deployed at the Far Edge and the core count increases, no additional cores will be given to the platform for basic operation, all resources will be given to the workloads.

Customer Considerations

• ...

Documentation Considerations

Questions to be addressed:

• What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
  • Administrators must know how to tune their Far Edge nodes to make them as computationally efficient as possible.

• Does this feature have doc impact?
  • Possibly, there should be documentation describing how to tune the Far Edge node such that the platform uses as little compute power as possible.

• New Content, Updates to existing content, Release Note, or No Doc Impact
  • Probably updates to existing content

• If unsure and no Technical Writer is available, please contact Content Strategy. What concepts do customers need to understand to be successful in [action]?
  • Performance Addon Operator, tuned, MCO, Performance Profile Creator

• How do we expect customers will use the feature? For what purpose(s)?
  • Customers will use the Performance Profile Creator to tune their Far Edge nodes. They will use RHACM (ZTP) to provision a Far Edge Single-Node OpenShift deployment with the appropriate Performance Profile.

• What reference material might a customer want/need to complete [action]?
  • Performance Addon Operator, Performance Profile Creator

• Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
  • N/A

• What is the doc impact (New Content, Updates to existing content, or Release Note)?
  • Likely updates to existing content / unsure

Monitoring needs to be reliable and is the very useful when trying to debug clusters in an already degraded state. We want to ensure that metrics scraping can always work if the scraper can reach the target, even if the kube-apiserver is unavailable or unreachable. To do this, we will combine a local authorizer (already merged in many binaries and the rbac-proxy) and client-cert based authentication to have a fully local authentication and authorization path for scraper targets.

If networking (or part of networking) is down and a scraper target cannot reach the kube-apiserver to verify a token and a subjectaccessreview, then the metrics scraper can be rejected. The subjectaccessreview (authorization) is already largely addressed, but service account tokens are still used for scraping targets. Tokens require an external network call that we can avoid by using client certificates. Gathering metrics, especially client metrics, from partially functionally clusters helps narrow the search area between kube-apiserver, etcd, kubelet, and SDN considerably.

In addition, this will significantly reduce the load on the kube-apiserver. We have observed in the CI cluster that token and subject access reviews are a significant percentage of all kube-apiserver traffic.

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.
• ...

Dependencies (internal and external)

1. ...

Previous Work (Optional):

1. …

Open questions::

1. …

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

An epic we can duplicate for each release to ensure we have a place to catch things we ought to be doing regularly but can tend to fall by the wayside.

OCP/Telco Definition of Done
Epic Template descriptions and documentation.

<--- Cut-n-Paste the entire contents of this description into your new Epic --->

Epic Goal

• Rebase OpenShift components to k8s v1.22
• Rebase Jenkins and plugins to latest long term support versions

Why is this important?

• Rebasing ensures components work with the upcoming release of Kubernetes
• Address tech debt related to upstream deprecations and removals.

Scenarios

1. ...

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.
• ...

Dependencies (internal and external)

1. k8s 1.22 release - expected August 4th 2021

Previous Work (Optional):

1. …

Open questions::

1. …

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

Problem:

This epic is mainly focused to track the dev console QE activities for 4.9 Release

Goal:

1. Identify the scenarios for automation
2. Segregate the test Scenarios into smoke, Regression and other user stories
a. Update the https://docs.jboss.org/display/ODC/Automation+Status+Report
3. Align with layered operator teams for updating scripts
3. Work closely with dev team for epic automation
4. Create the automation scripts using cypress
5. Implement CI for nightly builds
6. Execute scripts on sprint basis

Why is it important?

To the track the QE progress at one place

Acceptance criteria:

1. <criteria>

Dependencies (External/Internal):

1. External [For reviewing existing gherkin scripts]
2. Internal [Planned tasks for 4.9 release]

Design Artifacts:

Exploration:

Note:

Problem:

This epic is mainly focused on the 4.10 Release QE activities

Goal:

1. Identify the scenarios for automation
2. Segregate the test Scenarios into smoke, Regression and other user stories
a. Update the https://docs.jboss.org/display/ODC/Automation+Status+Report
3. Align with layered operator teams for updating scripts
3. Work closely with dev team for epic automation
4. Create the automation scripts using cypress
5. Implement CI for nightly builds
6. Execute scripts on sprint basis

Why is it important?

To the track the QE progress at one place in 4.10 Release Confluence page

Use cases:

1. <case>

Acceptance criteria:

1. <criteria>

Dependencies (External/Internal):

Design Artifacts:

Exploration:

Note:

Epic Goal

• Teach the cluster-version operator how to remove in-cluster objects.

Why is this important?

• OpenShift releases can remove components.  For example, two service-catalog operators were removed in 4.5.  Teaching the CVO how to remove manifests will allow us to clean up these resources without leaving cleanup jobs and associated RBAC behind.  We may also benefit from it in some rollback cases, where the presence of a new-in 4.(y+1) object makes the application of 4.y difficult.

Scenarios

1. Born-before-4.5 clusters currently have dangling resources from cleaning up the service-catalog operators (like the openshift-service-catalog-removed namespace in this job).  This enhancement would provide a mechanism for removing them, and any other cruft that we accumulate which lack in-cluster operators.
2. We can add removal manifests to the 4.(y-1) before growing a new component in 4.y, to make 4.y->4.(y-1) rollback more convenient for customers.

Acceptance Criteria

• CVO CI - MUST be running successfully with automated e2e-operator tests covering this functionality.
• Release Technical Enablement - Not required.  This is a purely OCP-internal change.

Dependencies (internal and external)

1. None

Previous Work (Optional):

1. Enhancement from OTA-279 has been accepted.  Remaining work is just implementing the accepted proposal.

Open questions::

1. None

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

Description of problem:

User expects a shadow on the form footer when the add page content is longer then the viewport shows. This was shown in 4.8.

Prerequisites (if any, like setup, operators/versions):

None

Steps to Reproduce

1. Switch to developer perspective
2. Add page
3. Import from Git for example

Actual results:

No shadow at the top of the form footer when the content view is scrollable.

Expected results:

A shadow at the top of the form footer when the content view is scrollable.

Reproducibility (Always/Intermittent/Only Once):

Always

Build Details:

Happen on a cluster (4.9.0-0.nightly-2021-07-07-021823)
and local development (4.9 master, tested with 0588bc0f0b838ae448a68f35c5424f9bbfc65bc9)

Additional info:

None

• https://github.com/openshift/console/pull/9527

Description of problem:

The QuickStart content shows a shadow above and/or below the content when the user can scroll into that direction. This feature is missing now.

Prerequisites (if any, like setup, operators/versions):

None

Steps to Reproduce

1. Open a quick start
2. Reduce the window so that the content of the quick start is scrollable.

Actual results:

No shadow when the user can scroll the content into the content direction.

Expected results:

A shadow when the user can scroll the content into the content direction.

Reproducibility (Always/Intermittent/Only Once):

Always

Build Details:

4.9 (tested on master 2860e58114b4f811ac2ebf6ce34dd99263920e17)

Additional info:

Quick start is now extracted into PatternFly

• https://github.com/openshift/console/pull/9550

Create README based on shared Google doc and add it to OpenShift so we have documentation for i18n work.

• https://github.com/openshift/console/pull/9886

Description

All test scenarios related to add flow should get executed on CI

Acceptance Criteria

• Update the devconsole/package.json
• Verify the scripts on remote cluster

Additional Details:

• https://github.com/openshift/console/pull/9361

Description of problem:

The topology use a PatternFly toolbar component to render its toolbar. It looks like the latest version uses a grid and flex layout which adds an addtional spacing below the toolbar.

Prerequisites (if any, like setup, operators/versions):

None

Steps to Reproduce

1. Open developer console
2. Check topology toolbar design

Actual results:

There is a spacing below the toolbar (Display options, Filter by resource, Filter by name), see attached screenshot.

Expected results:

No additional spacing below the toolbar. (See screenshot of 4.8)

Reproducibility (Always/Intermittent/Only Once):

Always

Build Details:

4.9 master, tested with commit 3c6537eec4f5c165cf214c4100bddeccc104ed44

Additional info:

Maybe this is a patternfly issue, or we should check if the second div in the toolbar should not be rendered. See attached screenshot.

• https://github.com/openshift/console/pull/9453

Description of problem:

.NET builder image is not getting detected

Prerequisites (if any, like setup, operators/versions):

Install Openshift Pipelines Operator

Steps to Reproduce

1. Follow steps of the test case

Actual results:

When git url of a .NET project is provided, the builder image is not getting detected

Expected results:

When git url of a .NET project is provided, the builder image should automatically get detected

Reproducibility (Always/Intermittent/Only Once):

Always

Build Details:

Additional info:

• https://github.com/openshift/console/pull/9493

Description of problem:

When the user opens the topology sidebar for a Deployment with a BuildConfig the string Build #x was complete (and others) are not translated. The browser log shows also an error

Missing i18n key "Build <1></1> was complete" in namespace "public" and language "en."

The strings are translated but are saved as:

Build {link} was complete

Prerequisites (if any, like setup, operators/versions):

None

Steps to Reproduce

1. Open developer perspective
2. Add page
3. Import from Git
4. Select the deployment in the topology graph
5. Switch the language

Actual results:

Build #x ... string is not translated. An error is logged in the browser console.

Expected results:

String is shown in the selected language and no error is logged in the browser console.

Reproducibility (Always/Intermittent/Only Once):

Always

Build Details:

Happen on a cluster (4.9.0-0.nightly-2021-07-07-021823)
and local development (4.9 master, tested with 0588bc0f0b838ae448a68f35c5424f9bbfc65bc9)

Additional info:

None

• https://github.com/openshift/console/pull/9572

Please review the following PR: https://github.com/openshift/containernetworking-plugins/pull/150

The PR has been automatically opened by ART (#forum-ocp-art) team automation and indicates
that the image(s) being used downstream for production builds are not consistent
with the images referenced in this component's github repository.

Differences in upstream and downstream builds impact the fidelity of your CI signal.

If you disagree with the content of this PR, please contact @release-artists
in #forum-ocp-art to discuss the discrepancy.

Closing this issue without addressing the difference will cause the issue to
be reopened automatically.

• https://github.com/openshift/containernetworking-plugins/pull/150

Description of problem:

Following https://issues.redhat.com/browse/CNV-28040
On CNV, when virtual machine, with secondary interfaces connected with bridge CNI, is live migrated we observe disruption at the VM inbound traffic.

The root cause for it is the migration target bridge interface advertise before the migration is completed.

When the migration destination pod is created an IPv6 NS (Neighbor Solicitation)
and NA (Neighbor Advertisement) are sent automatically by the kernel.
The switches at the endpoints (e.g.: migration destination node) tables
get updated and the traffic is forwarded to the migration destination before
the migration is completed [1].

The solution is to have the bridge CNI create the pod interface in "link-down" state [2], the IPv6 NS/NA packets are avoided, CNV in turn, set the pod interface to "link-up" [3].

CNV depends on bridge CNI with [2] bits, which is deployed by cluster-network-operator.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=2186372#c6
[2] https://github.com/kubevirt/kubevirt/pull/11069
[3] https://github.com/containernetworking/plugins/pull/997  

Version-Release number of selected component (if applicable):

4.16.0

How reproducible:

100%

Steps to Reproduce:

    1.
    2.
    3.
    

Actual results:

CNO deploys CNI bridge w/o an option to set the bridge interface down.

Expected results:

CNO to deploy bridge CNI with [1] changes, from release-4.16 branch. 

[1] https://github.com/containernetworking/plugins/pull/997

Additional info:

More https://issues.redhat.com/browse/CNV-28040    

• https://github.com/openshift/containernetworking-plugins/pull/154

Description of problem:

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

    1.
    2.
    3.
    

Actual results:

Expected results:

Additional info:

• https://github.com/openshift/containernetworking-plugins/pull/159

• https://github.com/openshift/containernetworking-plugins/pull/164

Please review the following PR: https://github.com/openshift/containernetworking-plugins/pull/122

The PR has been automatically opened by ART (#forum-ocp-art) team automation and indicates
that the image(s) being used downstream for production builds are not consistent
with the images referenced in this component's github repository.

Differences in upstream and downstream builds impact the fidelity of your CI signal.

If you disagree with the content of this PR, please contact @release-artists
in #forum-ocp-art to discuss the discrepancy.

Closing this issue without addressing the difference will cause the issue to
be reopened automatically.

• https://github.com/openshift/containernetworking-plugins/pull/122

• https://github.com/openshift/containernetworking-plugins/pull/128

Description of problem:

One multus case always fail in QE e2e testing. Using same net-attach-def and pod configure files, testing passed in 4.11 but failed in 4.12 and 4.13

Version-Release number of selected component (if applicable):

4.12 and 4.13

How reproducible:

All the times

Steps to Reproduce:

[weliang@weliang networking]$ oc create -f https://raw.githubusercontent.com/weliang1/verification-tests/master/testdata/networking/multus-cni/NetworkAttachmentDefinitions/runtimeconfig-def-ipandmac.yaml
networkattachmentdefinition.k8s.cni.cncf.io/runtimeconfig-def created
[weliang@weliang networking]$ oc get net-attach-def -o yaml
apiVersion: v1
items:
- apiVersion: k8s.cni.cncf.io/v1
  kind: NetworkAttachmentDefinition
  metadata:
    creationTimestamp: "2023-01-03T16:33:03Z"
    generation: 1
    name: runtimeconfig-def
    namespace: test
    resourceVersion: "64139"
    uid: bb26c08f-adbf-477e-97ab-2aa7461e50c4
  spec:
    config: '{ "cniVersion": "0.3.1", "name": "runtimeconfig-def", "plugins": [{ "type":
      "macvlan", "capabilities": { "ips": true }, "mode": "bridge", "ipam": { "type":
      "static" } }, { "type": "tuning", "capabilities": { "mac": true } }] }'
kind: List
metadata:
  resourceVersion: ""
[weliang@weliang networking]$ oc create -f https://raw.githubusercontent.com/weliang1/verification-tests/master/testdata/networking/multus-cni/Pods/runtimeconfig-pod-ipandmac.yaml
pod/runtimeconfig-pod created
[weliang@weliang networking]$ oc get pod
NAME                READY   STATUS              RESTARTS   AGE
runtimeconfig-pod   0/1     ContainerCreating   0          6s
[weliang@weliang networking]$ oc describe pod runtimeconfig-pod
Name:         runtimeconfig-pod
Namespace:    test
Priority:     0
Node:         weliang-01031-bvxtz-worker-a-qlwz7.c.openshift-qe.internal/10.0.128.4
Start Time:   Tue, 03 Jan 2023 11:33:45 -0500
Labels:       <none>
Annotations:  k8s.v1.cni.cncf.io/networks: [ { "name": "runtimeconfig-def", "ips": [ "192.168.22.2/24" ], "mac": "CA:FE:C0:FF:EE:00" } ]
              openshift.io/scc: anyuid
Status:       Pending
IP:           
IPs:          <none>
Containers:
  runtimeconfig-pod:
    Container ID:   
    Image:          quay.io/openshifttest/hello-sdn@sha256:c89445416459e7adea9a5a416b3365ed3d74f2491beb904d61dc8d1eb89a72a4
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-k5zqd (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-k5zqd:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
    ConfigMapName:           openshift-service-ca.crt
    ConfigMapOptional:       <nil>
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age   From               Message
  ----     ------                  ----  ----               -------
  Normal   Scheduled               26s   default-scheduler  Successfully assigned test/runtimeconfig-pod to weliang-01031-bvxtz-worker-a-qlwz7.c.openshift-qe.internal
  Normal   AddedInterface          24s   multus             Add eth0 [10.128.2.115/23] from openshift-sdn
  Warning  FailedCreatePodSandBox  23s   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_runtimeconfig-pod_test_7d5f3e7a-846d-4cfb-ac78-fd08b27102ae_0(cff792dbd07e8936d04aad31964bd7b626c19a90eb9d92a67736323a1a2303c4): error adding pod test_runtimeconfig-pod to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [test/runtimeconfig-pod/7d5f3e7a-846d-4cfb-ac78-fd08b27102ae:runtimeconfig-def]: error adding container to network "runtimeconfig-def": Interface name contains an invalid character /
  Normal   AddedInterface          7s    multus             Add eth0 [10.128.2.116/23] from openshift-sdn
  Warning  FailedCreatePodSandBox  7s    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_runtimeconfig-pod_test_7d5f3e7a-846d-4cfb-ac78-fd08b27102ae_0(d2456338fa65847d5dc744dea64972912c10b2a32d3450910b0b81cdc9159ca4): error adding pod test_runtimeconfig-pod to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [test/runtimeconfig-pod/7d5f3e7a-846d-4cfb-ac78-fd08b27102ae:runtimeconfig-def]: error adding container to network "runtimeconfig-def": Interface name contains an invalid character /
[weliang@weliang networking]$ 
 

Actual results:

Pod is not running

Expected results:

Pod should be in running state

Additional info:

• https://github.com/openshift/containernetworking-plugins/pull/72
• https://github.com/openshift/containernetworking-plugins/pull/70

• https://github.com/openshift/containernetworking-plugins/pull/165

Currently we see this issue:

 
Aug 28 00:02:20.755103 ip-10-0-131-145 hyperkube[1366]: I0828 00:02:20.755067 1366 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-quorum-guard-588ff9b55d-8lhb7" podUID=5b79def2-9e56-4c93-b8ab-1d04db0f552f containerName="guard" probeResult=failure output=""
then few seconds later
Aug 28 00:02:25.797258 ip-10-0-131-145 hyperkube[1366]: I0828 00:02:25.797231 1366 kubelet.go:2175] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-etcd/etcd-quorum-guard-588ff9b55d-8lhb7"
Try to improve the clustermembercontroller sync loop for health status or just improve to not fail there on probe quard during install at least or scale. Instead of maybe operator status use metrics to track this.

Slack for more context https://coreos.slack.com/archives/C027U68LP/p1630506922034600

AC: 

• come up with a solution which approach we want to take and present in the team meeting
• implement the proposed solution

• https://github.com/openshift/origin/pull/26439

Description of problem:

clusteroperator/network is degraded after running

    FEATURES_ENVIRONMENT="ci" make feature-deploy-on-ci

from openshift-kni/cnf-features-deploy against IPI clusters with OCP 4.13 and 4.14 in CI jobs from Telco 5G DevOps/CI.

Details for a 4.13 job:

https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_release/42141/rehearse-42141-periodic-ci-openshift-release-master-nightly-4.13-e2e-telco5g/1689935408508440576

Details for a 4.14 job:

https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_release/42141/rehearse-42141-periodic-ci-openshift-release-master-nightly-4.14-e2e-telco5g/1689935408541995008

For example, got to artifacts/e2e-telco5g/telco5g-gather-pao/build-log.txt and it will report:

Error from server (BadRequest): container "container-00" in pod "cnfdu5-worker-0-debug" is waiting to start: ContainerCreating
Running gather-pao for T5CI_VERSION=4.13
Running for CNF_BRANCH=master
Running PAO must-gather with tag pao_mg_tag=4.12
[must-gather      ] OUT Using must-gather plug-in image: quay.io/openshift-kni/performance-addon-operator-must-gather:4.12-snapshot
When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information:
ClusterID: 60503edf-ecc6-48f7-b6a6-f4dc34842803
ClusterVersion: Stable at "4.13.0-0.nightly-2023-08-10-021434"
ClusterOperators:
	clusteroperator/network is degraded because DaemonSet "/openshift-multus/dhcp-daemon" rollout is not making progress - pod dhcp-daemon-7lmlq is in CrashLoopBackOff State
DaemonSet "/openshift-multus/dhcp-daemon" rollout is not making progress - pod dhcp-daemon-95tzb is in CrashLoopBackOff State
DaemonSet "/openshift-multus/dhcp-daemon" rollout is not making progress - pod dhcp-daemon-hfxkd is in CrashLoopBackOff State
DaemonSet "/openshift-multus/dhcp-daemon" rollout is not making progress - pod dhcp-daemon-mhwtp is in CrashLoopBackOff State
DaemonSet "/openshift-multus/dhcp-daemon" rollout is not making progress - pod dhcp-daemon-q7gfb is in CrashLoopBackOff State
DaemonSet "/openshift-multus/dhcp-daemon" rollout is not making progress - last change 2023-08-11T10:54:10Z

Version-Release number of selected component (if applicable):

branch release-4.13 from https://github.com/openshift-kni/cnf-features-deploy.git for OCP 4.13
branch master from https://github.com/openshift-kni/cnf-features-deploy.git for OCP 4.14

How reproducible:

Always.

Steps to Reproduce:

1. Install OCP 4.13 or OCP 4.14 with IPI on 3x masters, 2x workers.
2. Clone https://github.com/openshift-kni/cnf-features-deploy.git
3. FEATURES_ENVIRONMENT="ci" make feature-deploy-on-ci
4. oc wait nodes --all --for=condition=Ready=true --timeout=10m
5. oc wait clusteroperators --all --for=condition=Progressing=false --timeout=10m

Actual results:

See above.

Expected results:

All clusteroperators have finished progressing.

Additional info:

Without 'FEATURES_ENVIRONMENT="ci" make feature-deploy-on-ci' the steps to reproduce above work as expected.

• https://github.com/openshift/containernetworking-plugins/pull/116

Please review the following PR: https://github.com/openshift/containernetworking-plugins/pull/142

The PR has been automatically opened by ART (#forum-ocp-art) team automation and indicates
that the image(s) being used downstream for production builds are not consistent
with the images referenced in this component's github repository.

Differences in upstream and downstream builds impact the fidelity of your CI signal.

If you disagree with the content of this PR, please contact @release-artists
in #forum-ocp-art to discuss the discrepancy.

Closing this issue without addressing the difference will cause the issue to
be reopened automatically.

• https://github.com/openshift/containernetworking-plugins/pull/146
• https://github.com/openshift/containernetworking-plugins/pull/145
• https://github.com/openshift/containernetworking-plugins/pull/142

Description of problem:

Multus mac-vlan/ipvlan/vlan cni panics when master interface in container is missing

Version-Release number of selected component (if applicable):

metallb-operator.v4.13.0-202304190216   MetalLB Operator   4.13.0-202304190216 Succeeded

How reproducible:

Create pod with multiple vlan interfaces connected to missing master interface.

Steps to Reproduce:

1. Create pod with multiple vlan interfaces connected to missing master interface in container
2. Make sure that pod stuck in ContainerCreating state 
3. Run oc describe pod PODNAME and read crash message:

 Normal   Scheduled               22s   default-scheduler  Successfully assigned cni-tests/pod-one to worker-0
  Normal   AddedInterface          21s   multus             Add eth0 [10.128.2.231/23] from ovn-kubernetes
  Normal   AddedInterface          21s   multus             Add ext0 [] from cni-tests/tap-one
  Normal   AddedInterface          21s   multus             Add ext0.1 [2001:100::1/64] from cni-tests/mac-vlan-one
  Warning  FailedCreatePodSandBox  18s   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_pod-one_cni-tests_2e831519-effc-4502-8ea7-749eda95bf1d_0(321d7181626b8bbfad062dd7c7cc2ef096f8547e93cb7481a18b7d3613eabffd): error adding pod cni-tests_pod-one to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [cni-tests/pod-one/2e831519-effc-4502-8ea7-749eda95bf1d:mac-vlan]: error adding container to network "mac-vlan": plugin type="macvlan" failed (add): netplugin failed: "panic: runtime error: invalid memory address or nil pointer dereference\n[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x54281a]\n\ngoroutine 1 [running, locked to thread]:\npanic({0x560b00, 0x6979d0})\n\t/usr/lib/golang/src/runtime/panic.go:987 +0x3ba fp=0xc0001ad8f0 sp=0xc0001ad830 pc=0x433d7a\nruntime.panicmem(...)\n\t/usr/lib/golang/src/runtime/panic.go:260\nruntime.sigpanic()\n\t/usr/lib/golang/src/runtime/signal_unix.go:835 +0x2f6 fp=0xc0001ad940 sp=0xc0001ad8f0 pc=0x449cd6\nmain.getMTUByName({0xc00001a978, 0x4}, {0xc00002004a, 0x33}, 0x1)\n\t/usr/src/plugins/plugins/main/macvlan/macvlan.go:167 +0x33a fp=0xc0001ada00 sp=0xc0001ad940 pc=0x54281a\nmain.loadConf(0xc000186770, {0xc00001e009, 0x19e})\n\t/usr/src/plugins/plugins/main/macvlan/macvlan.go:120 +0x155 fp=0xc0001ada80 sp=0xc0001ada00 pc=0x5422d5\nmain.cmdAdd(0xc000186770)\n\t/usr/src/plugins/plugins/main/macvlan/macvlan.go:287 +0x47 fp=0xc0001adcd0 sp=0xc0001ada80 pc=0x543b07\ngithub.com/containernetworking/cni/pkg/skel.(*dispatcher).checkVersionAndCall(0xc0000bdec8, 0xc000186770, {0x5c02b8, 0xc0000e4e40}, 0x592e80)\n\t/usr/src/plugins/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:166 +0x20a fp=0xc0001add60 sp=0xc0001adcd0 pc=0x5371ca\ngithub.com/containernetworking/cni/pkg/skel.(*dispatcher).pluginMain(0xc0000bdec8, 0x698320?, 0xc0000bdeb0?, 0x44ed89?, {0x5c02b8, 0xc0000e4e40}, {0xc0000000f0, 0x22})\n\t/usr/src/plugins/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:219 +0x2ca fp=0xc0001ade68 sp=0xc0001add60 pc=0x53772a\ngithub.com/containernetworking/cni/pkg/skel.PluginMainWithError(...)\n\t/usr/src/plugins/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:273\ngithub.com/containernetworking/cni/pkg/skel.PluginMain(0x588e01?, 0x10?, 0xc0000bdf50?, {0x5c02b8?, 0xc0000e4e40?}, {0xc0000000f0?, 0x0?})\n\t/usr/src/plugins/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:288 +0xd1 fp=0xc0001adf18 sp=0xc0001ade68 pc=0x537d51\nmain.main()\n\t/usr/src/plugins/plugins/main/macvlan/macvlan.go:432 +0xb6 fp=0xc0001adf80 sp=0xc0001adf18 pc=0x544b76\nruntime.main()\n\t/usr/lib/golang/src/runtime/proc.go:250 +0x212 fp=0xc0001adfe0 sp=0xc0001adf80 pc=0x436a12\nruntime.goexit()\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0001adfe8 sp=0xc0001adfe0 pc=0x462fc1\n\ngoroutine 2 [force gc (idle)]:\nruntime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)\n\t/usr/lib/golang/src/runtime/proc.go:363 +0xd6 fp=0xc0000acfb0 sp=0xc0000acf90 pc=0x436dd6\nruntime.goparkunlock(...)\n\t/usr/lib/golang/src/runtime/proc.go:369\nruntime.forcegchelper()\n\t/usr/lib/golang/src/runtime/proc.go:302 +0xad fp=0xc0000acfe0 sp=0xc0000acfb0 pc=0x436c6d\nruntime.goexit()\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000acfe8 sp=0xc0000acfe0 pc=0x462fc1\ncreated by runtime.init.6\n\t/usr/lib/golang/src/runtime/proc.go:290 +0x25\n\ngoroutine 3 [GC sweep wait]:\nruntime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)\n\t/usr/lib/golang/src/runtime/proc.go:363 +0xd6 fp=0xc0000ad790 sp=0xc0000ad770 pc=0x436dd6\nruntime.goparkunlock(...)\n\t/usr/lib/golang/src/runtime/proc.go:369\nruntime.bgsweep(0x0?)\n\t/usr/lib/golang/src/runtime/mgcsweep.go:278 +0x8e fp=0xc0000ad7c8 sp=0xc0000ad790 pc=0x423e4e\nruntime.gcenable.func1()\n\t/usr/lib/golang/src/runtime/mgc.go:178 +0x26 fp=0xc0000ad7e0 sp=0xc0000ad7c8 pc=0x418d06\nruntime.goexit()\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000ad7e8 sp=0xc0000ad7e0 pc=0x462fc1\ncreated by runtime.gcenable\n\t/usr/lib/golang/src/runtime/mgc.go:178 +0x6b\n\ngoroutine 4 [GC scavenge wait]:\nruntime.gopark(0xc0000ca000?, 0x5bf2b8?, 0x1?, 0x0?, 0x0?)\n\t/usr/lib/golang/src/runtime/proc.go:363 +0xd6 fp=0xc0000adf70 sp=0xc0000adf50 pc=0x436dd6\nruntime.goparkunlock(...)\n\t/usr/lib/golang/src/runtime/proc.go:369\nruntime.(*scavengerState).park(0x6a0920)\n\t/usr/lib/golang/src/runtime/mgcscavenge.go:389 +0x53 fp=0xc0000adfa0 sp=0xc0000adf70 pc=0x421ef3\nruntime.bgscavenge(0x0?)\n\t/usr/lib/golang/src/runtime/mgcscavenge.go:617 +0x45 fp=0xc0000adfc8 sp=0xc0000adfa0 pc=0x4224c5\nruntime.gcenable.func2()\n\t/usr/lib/golang/src/runtime/mgc.go:179 +0x26 fp=0xc0000adfe0 sp=0xc0000adfc8 pc=0x418ca6\nruntime.goexit()\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000adfe8 sp=0xc0000adfe0 pc=0x462fc1\ncreated by runtime.gcenable\n\t/usr/lib/golang/src/runtime/mgc.go:179 +0xaa\n\ngoroutine 5 [finalizer wait]:\nruntime.gopark(0x0?, 0xc0000ac670?, 0xab?, 0x61?, 0xc0000ac770?)\n\t/usr/lib/golang/src/runtime/proc.go:363 +0xd6 fp=0xc0000ac628 sp=0xc0000ac608 pc=0x436dd6\nruntime.goparkunlock(...)\n\t/usr/lib/golang/src/runtime/proc.go:369\nruntime.runfinq()\n\t/usr/lib/golang/src/runtime/mfinal.go:180 +0x10f fp=0xc0000ac7e0 sp=0xc0000ac628 pc=0x417e0f\nruntime.goexit()\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000ac7e8 sp=0xc0000ac7e0 pc=0x462fc1\ncreated by runtime.createfing\n\t/usr/lib/golang/src/runtime/mfinal.go:157 +0x45\n"

Actual results:

The readable error message should be provided instead.

Expected results:

We should handle such scenario without crash and The following log should be used instead. 

Error: Failed to create container due to the missing master interface XXX.

Additional info:

• https://github.com/openshift/containernetworking-plugins/pull/98