Chapter 16. Multiple networks

Procedure

1. Optional: For a ClusterUserDefinedNetwork CR that uses a primary network, create a namespace with the k8s.ovn.org/primary-user-defined-network label by entering the following command:

   Copy to Clipboard Toggle word wrap

   $ cat << EOF | oc apply -f -
   apiVersion: v1
   kind: Namespace
   metadata:
     name: <cudn_namespace_name>
     labels:
       k8s.ovn.org/primary-user-defined-network: ""
   EOF

2. Create a request for either a Layer2 or Layer3 topology type cluster-wide user-defined network:

   1. Create a YAML file, such as cluster-layer-two-udn.yaml, to define your request for a Layer2 topology as in the following example:

      Copy to Clipboard Toggle word wrap

      apiVersion: k8s.ovn.org/v1
      kind: ClusterUserDefinedNetwork
      metadata:
        name: <cudn_name> 

      1

      
      spec:
        namespaceSelector: 

      2

      
          matchLabels: 

      3

      
            "<label_1_key>": "<label_1_value>" 

      4

      
            "<label_2_key>": "<label_2_value>" 

      5

      
        network: 

      6

      
          topology: Layer2 

      7

      
          layer2: 

      8

      
            role: Primary 

      9

      
            subnets:
              - "2001:db8::/64"
              - "10.100.0.0/16" 

      10

      1

      Name of your ClusterUserDefinedNetwork CR.

      2

      A label query over the set of namespaces that the cluster UDN CR applies to. Uses the standard Kubernetes MatchLabel selector. Must not point to default or openshift-* namespaces.

      3

      Uses the matchLabels selector type, where terms are evaluated with an AND relationship.

      4 5

      Because the matchLabels selector type is used, provisions namespaces that contain both <label_1_key>=<label_1_value> and <label_2_key>=<label_2_value> labels.

      6

      Describes the network configuration.

      7

      The topology field describes the network configuration; accepted values are Layer2 and Layer3. Specifying a Layer2 topology type creates one logical switch that is shared by all nodes.

      8

      This field specifies the topology configuration. It can be layer2 or layer3.

      9

      Specifies Primary or Secondary. Primary is the only role specification supported in 4.18.

      10

      For Layer2 topology types the following specifies config details for the subnet field:

      • The subnets field is optional.
      • The subnets field is of type string and accepts standard CIDR formats for both IPv4 and IPv6.
      • The subnets field accepts one or two items. For two items, they must be of a different family. For example, subnets values of 10.100.0.0/16 and 2001:db8::/64.
      • Layer2 subnets can be omitted. If omitted, users must configure static IP addresses for the pods. As a consequence, port security only prevents MAC spoofing. For more information, see "Configuring pods with a static IP address".

   2. Create a YAML file, such as cluster-layer-three-udn.yaml, to define your request for a Layer3 topology as in the following example:

      Copy to Clipboard Toggle word wrap

      apiVersion: k8s.ovn.org/v1
      kind: ClusterUserDefinedNetwork
      metadata:
        name: <cudn_name> 

      1

      
      spec:
        namespaceSelector: 

      2

      
          matchExpressions: 

      3

      
          - key: kubernetes.io/metadata.name 

      4

      
            operator: In 

      5

      
            values: ["<example_namespace_one>", "<example_namespace_two>"] 

      6

      
        network: 

      7

      
          topology: Layer3 

      8

      
          layer3: 

      9

      
            role: Primary 

      10

      
            subnets: 

      11

      
              - cidr: 10.100.0.0/16
                hostSubnet: 24

      1

      Name of your ClusterUserDefinedNetwork CR.

      2

      A label query over the set of namespaces that the cluster UDN applies to. Uses the standard Kubernetes MatchLabel selector. Must not point to default or openshift-* namespaces.

      3

      Uses the matchExpressions selector type, where terms are evaluated with an OR relationship.

      4

      Specifies the label key to match.

      5

      Specifies the operator. Valid values include: In, NotIn, Exists, and DoesNotExist.

      6

      Because the matchExpressions type is used, provisions namespaces matching either <example_namespace_one> or <example_namespace_two>.

      7

      Describes the network configuration.

      8

      The topology field describes the network configuration; accepted values are Layer2 and Layer3. Specifying a Layer3 topology type creates a layer 2 segment per node, each with a different subnet. Layer 3 routing is used to interconnect node subnets.

      9

      This field specifies the topology configuration. Valid values are layer2 or layer3.

      10

      Specifies a Primary or Secondary role. Primary is the only role specification supported in 4.18.

      11

      For Layer3 topology types the following specifies config details for the subnet field:

      • The subnets field is mandatory.

      • The type for the subnets field is cidr and hostSubnet:

        • cidr is the cluster subnet and accepts a string value.
        • hostSubnet specifies the nodes subnet prefix that the cluster subnet is split to.
        • For IPv6, only a /64 length is supported for hostSubnet.

3. Apply your request by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc create --validate=true -f <example_cluster_udn>.yaml

   Where <example_cluster_udn>.yaml is the name of your Layer2 or Layer3 configuration file.

4. Verify that your request is successful by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc get clusteruserdefinednetwork <cudn_name> -o yaml

   Where <cudn_name> is the name you created of your cluster-wide user-defined network.

   Example output

   Copy to Clipboard Toggle word wrap

   apiVersion: k8s.ovn.org/v1
   kind: ClusterUserDefinedNetwork
   metadata:
     creationTimestamp: "2024-12-05T15:53:00Z"
     finalizers:
     - k8s.ovn.org/user-defined-network-protection
     generation: 1
     name: my-cudn
     resourceVersion: "47985"
     uid: 16ee0fcf-74d1-4826-a6b7-25c737c1a634
   spec:
     namespaceSelector:
       matchExpressions:
       - key: custom.network.selector
         operator: In
         values:
         - example-namespace-1
         - example-namespace-2
         - example-namespace-3
     network:
       layer3:
         role: Primary
         subnets:
         - cidr: 10.100.0.0/16
       topology: Layer3
   status:
     conditions:
     - lastTransitionTime: "2024-11-19T16:46:34Z"
       message: 'NetworkAttachmentDefinition has been created in following namespaces:
         [example-namespace-1, example-namespace-2, example-namespace-3]'
       reason: NetworkAttachmentDefinitionReady
       status: "True"
       type: NetworkCreated

Procedure

1. From the Administrator perspective, click Networking UserDefinedNetworks.
2. Click ClusterUserDefinedNetwork.
3. In the Name field, specify a name for the cluster-scoped UDN.
4. Specify a value in the Subnet field.
5. In the Project(s) Match Labels field, add the appropriate labels to select namespaces that the cluster UDN applies to.
6. Click Create. The cluster-scoped UDN serves as the default primary network for pods located in namespaces that contain the labels that you specified in step 5.

Procedure

1. Optional: For a UserDefinedNetwork CR that uses a primary network, create a namespace with the k8s.ovn.org/primary-user-defined-network label by entering the following command:

   Copy to Clipboard Toggle word wrap

   $ cat << EOF | oc apply -f -
   apiVersion: v1
   kind: Namespace
   metadata:
     name: <udn_namespace_name>
     labels:
       k8s.ovn.org/primary-user-defined-network: ""
   EOF

2. Create a request for either a Layer2 or Layer3 topology type user-defined network:

   1. Create a YAML file, such as my-layer-two-udn.yaml, to define your request for a Layer2 topology as in the following example:

      Copy to Clipboard Toggle word wrap

      apiVersion: k8s.ovn.org/v1
      kind: UserDefinedNetwork
      metadata:
        name: udn-1 

      1

      
        namespace: <some_custom_namespace>
      spec:
        topology: Layer2 

      2

      
        layer2: 

      3

      
          role: Primary 

      4

      
          subnets:
            - "10.0.0.0/24"
            - "2001:db8::/60" 

      5

      1

      Name of your UserDefinedNetwork resource. This should not be default or duplicate any global namespaces created by the Cluster Network Operator (CNO).

      2

      The topology field describes the network configuration; accepted values are Layer2 and Layer3. Specifying a Layer2 topology type creates one logical switch that is shared by all nodes.

      3

      This field specifies the topology configuration. It can be layer2 or layer3.

      4

      Specifies a Primary or Secondary role.

      5

      For Layer2 topology types the following specifies config details for the subnet field:

      • The subnets field is optional.
      • The subnets field is of type string and accepts standard CIDR formats for both IPv4 and IPv6.
      • The subnets field accepts one or two items. For two items, they must be of a different family. For example, subnets values of 10.100.0.0/16 and 2001:db8::/64.
      • Layer2 subnets can be omitted. If omitted, users must configure IP addresses for the pods. As a consequence, port security only prevents MAC spoofing.
      • The Layer2 subnets field is mandatory when the ipamLifecycle field is specified.

   2. Create a YAML file, such as my-layer-three-udn.yaml, to define your request for a Layer3 topology as in the following example:

      Copy to Clipboard Toggle word wrap

      apiVersion: k8s.ovn.org/v1
      kind: UserDefinedNetwork
      metadata:
        name: udn-2-primary 

      1

      
        namespace: <some_custom_namespace>
      spec:
        topology: Layer3 

      2

      
        layer3: 

      3

      
          role: Primary 

      4

      
          subnets: 

      5

      
            - cidr: 10.150.0.0/16
              hostSubnet: 24
            - cidr: 2001:db8::/60
              hostSubnet: 64
      # ...

      1

      Name of your UserDefinedNetwork resource. This should not be default or duplicate any global namespaces created by the Cluster Network Operator (CNO).

      2

      The topology field describes the network configuration; accepted values are Layer2 and Layer3. Specifying a Layer3 topology type creates a layer 2 segment per node, each with a different subnet. Layer 3 routing is used to interconnect node subnets.

      3

      This field specifies the topology configuration. Valid values are layer2 or layer3.

      4

      Specifies a Primary or Secondary role.

      5

      For Layer3 topology types the following specifies config details for the subnet field:

      • The subnets field is mandatory.

      • The type for the subnets field is cidr and hostSubnet:

        • cidr is equivalent to the clusterNetwork configuration settings of a cluster. The IP addresses in the CIDR are distributed to pods in the user defined network. This parameter accepts a string value.
        • hostSubnet defines the per-node subnet prefix.
        • For IPv6, only a /64 length is supported for hostSubnet.

3. Apply your request by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc apply -f <my_layer_two_udn>.yaml

   Where <my_layer_two_udn>.yaml is the name of your Layer2 or Layer3 configuration file.

4. Verify that your request is successful by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc get userdefinednetworks udn-1 -n <some_custom_namespace> -o yaml

   Where some_custom_namespace is the namespace you created for your user-defined network.

   Example output

   Copy to Clipboard Toggle word wrap

   apiVersion: k8s.ovn.org/v1
   kind: UserDefinedNetwork
   metadata:
     creationTimestamp: "2024-08-28T17:18:47Z"
     finalizers:
     - k8s.ovn.org/user-defined-network-protection
     generation: 1
     name: udn-1
     namespace: some-custom-namespace
     resourceVersion: "53313"
     uid: f483626d-6846-48a1-b88e-6bbeb8bcde8c
   spec:
     layer2:
       role: Primary
       subnets:
       - 10.0.0.0/24
       - 2001:db8::/60
     topology: Layer2
   status:
     conditions:
     - lastTransitionTime: "2024-08-28T17:18:47Z"
       message: NetworkAttachmentDefinition has been created
       reason: NetworkAttachmentDefinitionReady
       status: "True"
       type: NetworkCreated

Procedure

1. From the Administrator perspective, click Networking UserDefinedNetworks.
2. Click Create UserDefinedNetwork.
3. From the Project name list, select the namespace that you previously created.
4. Specify a value in the Subnet field.
5. Click Create. The user-defined network serves as the default primary network for pods that you create in this namespace.

Procedure

1. Create a new YAML file named, such as setsebool-container-use-devices.yaml, with the following details:

   Copy to Clipboard Toggle word wrap

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: 99-worker-setsebool
   spec:
     config:
       ignition:
         version: 3.2.0
       systemd:
         units:
         - enabled: true
           name: setsebool.service
           contents: |
             [Unit]
             Description=Set SELinux boolean for the TAP CNI plugin
             Before=kubelet.service
   
             [Service]
             Type=oneshot
             ExecStart=/usr/sbin/setsebool container_use_devices=on
             RemainAfterExit=true
   
             [Install]
             WantedBy=multi-user.target graphical.target

2. Create the new MachineConfig object by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc apply -f setsebool-container-use-devices.yaml

   Note

   Applying any changes to the MachineConfig object causes all affected nodes to gracefully reboot after the change is applied. This update can take some time to be applied.

3. Verify the change is applied by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc get machineconfigpools

   Expected output

   Copy to Clipboard Toggle word wrap

   NAME        CONFIG                                                UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   master      rendered-master-e5e0c8e8be9194e7c5a882e047379cfa      True      False      False      3              3                   3                     0                      7d2h
   worker      rendered-worker-d6c9ca107fba6cd76cdcbfcedcafa0f2      True      False      False      3              3                   3                     0                      7d

   Note

   All nodes should be in the updated and ready state.

1. Edit the Pod resource definition. If you are editing an existing Pod resource, run the following command to edit its definition in the default editor. Replace <name> with the name of the Pod resource to edit.

   Copy to Clipboard Toggle word wrap

   $ oc edit pod <name>

2. In the Pod resource definition, add the k8s.v1.cni.cncf.io/networks parameter to the pod metadata mapping. The k8s.v1.cni.cncf.io/networks accepts a JSON string of a list of objects that reference the name of NetworkAttachmentDefinition custom resource (CR) names in addition to specifying additional properties.

   Copy to Clipboard Toggle word wrap

   metadata:
     annotations:
       k8s.v1.cni.cncf.io/networks: '[<network>[,<network>,...]]' 

   1

   1

   Replace <network> with a JSON object as shown in the following examples. The single quotes are required.

3. In the following example the annotation specifies which network attachment will have the default route, using the default-route parameter.

   Copy to Clipboard Toggle word wrap

   apiVersion: v1
   kind: Pod
   metadata:
     name: example-pod
     annotations:
       k8s.v1.cni.cncf.io/networks: '[
       {
         "name": "net1"
       },
       {
         "name": "net2", 

   1

   
         "default-route": ["192.0.2.1"] 

   2

   
       }]'
   spec:
     containers:
     - name: example-pod
       command: ["/bin/bash", "-c", "sleep 2000000000000"]
       image: centos/tools

   1

   The name key is the name of the secondary network to associate with the pod.

   2

   The default-route key specifies a value of a gateway for traffic to be routed over if no other routing entry is present in the routing table. If more than one default-route key is specified, this will cause the pod to fail to become active.

1. Edit the CNO CR by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc edit networks.operator.openshift.io cluster

Procedure

1. Edit the Network.operator.openshift.io custom resource (CR) by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc edit network.operator.openshift.io cluster

2. Include the additionalNetworks section shown in this example YAML extract within the spec definition of the custom resource (CR):

   Copy to Clipboard Toggle word wrap

   apiVersion: operator.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   # ...
   spec:
     additionalNetworks:
     - name: whereabouts-shim
       namespace: default
       rawCNIConfig: |-
         {
          "name": "whereabouts-shim",
          "cniVersion": "0.3.1",
          "type": "bridge",
          "ipam": {
            "type": "whereabouts"
          }
         }
       type: Raw
   # ...

3. Save the file and exit the text editor.

4. Verify that the whereabouts-reconciler daemon set deployed successfully by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc get all -n openshift-multus | grep whereabouts-reconciler

   Example output

   Copy to Clipboard Toggle word wrap

   pod/whereabouts-reconciler-jnp6g 1/1 Running 0 6s
   pod/whereabouts-reconciler-k76gg 1/1 Running 0 6s
   pod/whereabouts-reconciler-k86t9 1/1 Running 0 6s
   pod/whereabouts-reconciler-p4sxw 1/1 Running 0 6s
   pod/whereabouts-reconciler-rvfdv 1/1 Running 0 6s
   pod/whereabouts-reconciler-svzw9 1/1 Running 0 6s
   daemonset.apps/whereabouts-reconciler 6 6 6 6 6 kubernetes.io/os=linux 6s

Procedure

1. Run the following command to create a ConfigMap object named whereabouts-config in the openshift-multus namespace with a specific cron expression for the IP reconciler:

   Copy to Clipboard Toggle word wrap

   $ oc create configmap whereabouts-config -n openshift-multus --from-literal=reconciler_cron_expression="*/15 * * * *"

   This cron expression indicates the IP reconciler runs every 15 minutes. Adjust the expression based on your specific requirements.

   Note

   The whereabouts-reconciler daemon set can only consume a cron expression pattern that includes five asterisks. The sixth, which is used to denote seconds, is currently not supported.

2. Retrieve information about resources related to the whereabouts-reconciler daemon set and pods within the openshift-multus namespace by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc get all -n openshift-multus | grep whereabouts-reconciler

   Example output

   Copy to Clipboard Toggle word wrap

   pod/whereabouts-reconciler-2p7hw                   1/1     Running   0             4m14s
   pod/whereabouts-reconciler-76jk7                   1/1     Running   0             4m14s
   pod/whereabouts-reconciler-94zw6                   1/1     Running   0             4m14s
   pod/whereabouts-reconciler-mfh68                   1/1     Running   0             4m14s
   pod/whereabouts-reconciler-pgshz                   1/1     Running   0             4m14s
   pod/whereabouts-reconciler-xn5xz                   1/1     Running   0             4m14s
   daemonset.apps/whereabouts-reconciler          6         6         6       6            6           kubernetes.io/os=linux   4m16s

3. Run the following command to verify that the whereabouts-reconciler pod runs the IP reconciler with the configured interval:

   Copy to Clipboard Toggle word wrap

   $ oc -n openshift-multus logs whereabouts-reconciler-2p7hw

   Example output

   Copy to Clipboard Toggle word wrap

   2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/..2024_02_02_16_33_54.1375928161": CREATE
   2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/..2024_02_02_16_33_54.1375928161": CHMOD
   2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/..data_tmp": RENAME
   2024-02-02T16:33:54Z [verbose] using expression: */15 * * * *
   2024-02-02T16:33:54Z [verbose] configuration updated to file "/cron-schedule/..data". New cron expression: */15 * * * *
   2024-02-02T16:33:54Z [verbose] successfully updated CRON configuration id "00c2d1c9-631d-403f-bb86-73ad104a6817" - new cron expression: */15 * * * *
   2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/config": CREATE
   2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/..2024_02_02_16_26_17.3874177937": REMOVE
   2024-02-02T16:45:00Z [verbose] starting reconciler run
   2024-02-02T16:45:00Z [debug] NewReconcileLooper - inferred connection data
   2024-02-02T16:45:00Z [debug] listing IP pools
   2024-02-02T16:45:00Z [debug] no IP addresses to cleanup
   2024-02-02T16:45:00Z [verbose] reconciler success

Procedure

1. Set type to whereabouts.

2. Use ipRanges to allocate IP addresses as shown in the following example:

   Copy to Clipboard Toggle word wrap

   cniVersion: operator.openshift.io/v1
   kind: Network
   =metadata:
     name: cluster
   spec:
     additionalNetworks:
     - name: whereabouts-shim
       namespace: default
       type: Raw
       rawCNIConfig: |-
         {
          "name": "whereabouts-dual-stack",
          "cniVersion": "0.3.1,
          "type": "bridge",
          "ipam": {
            "type": "whereabouts",
            "ipRanges": [
                     {"range": "192.168.10.0/24"},
                     {"range": "2001:db8::/64"}
                 ]
          }
         }

3. Attach network to a pod. For more information, see "Adding a pod to a secondary network".
4. Verify that all IP addresses are assigned.

5. Run the following command to ensure the IP addresses are assigned as metadata.

   Copy to Clipboard Toggle word wrap

   $ oc exec -it mypod -- ip a

Procedure

1. Create a dedicated container namespace where you want to deploy your pod by using the following command:

   Copy to Clipboard Toggle word wrap

   $ oc new-project test-namespace

2. Create an SR-IOV node policy:

   1. Create an SriovNetworkNodePolicy object, and then save the YAML in the sriov-node-network-policy.yaml file:

      Copy to Clipboard Toggle word wrap

      apiVersion: sriovnetwork.openshift.io/v1
      kind: SriovNetworkNodePolicy
      metadata:
       name: sriovnic
       namespace: openshift-sriov-network-operator
      spec:
       deviceType: netdevice
       isRdma: false
       needVhostNet: true
       nicSelector:
         vendor: "15b3" 

      1

      
         deviceID: "101b" 

      2

      
         rootDevices: ["00:05.0"]
       numVfs: 10
       priority: 99
       resourceName: sriovnic
       nodeSelector:
          feature.node.kubernetes.io/network-sriov.capable: "true"

      Note

      The SR-IOV network node policy configuration example, with the setting deviceType: netdevice, is tailored specifically for Mellanox Network Interface Cards (NICs).

      1

      The vendor hexadecimal code of the SR-IOV network device. The value 15b3 is associated with a Mellanox NIC.

      2

      The device hexadecimal code of the SR-IOV network device.

   2. Apply the YAML by running the following command:

      Copy to Clipboard Toggle word wrap

      $ oc apply -f sriov-node-network-policy.yaml

      Note

      Applying this might take some time due to the node requiring a reboot.

3. Create an SR-IOV network:

   1. Create the SriovNetwork custom resource (CR) for the additional secondary SR-IOV network attachment as in the following example CR. Save the YAML as the file sriov-network-attachment.yaml:

      Copy to Clipboard Toggle word wrap

      apiVersion: sriovnetwork.openshift.io/v1
      kind: SriovNetwork
      metadata:
       name: sriov-network
       namespace: openshift-sriov-network-operator
      spec:
       networkNamespace: test-namespace
       resourceName: sriovnic
       spoofChk: "off"
       trust: "on"

   2. Apply the YAML by running the following command:

      Copy to Clipboard Toggle word wrap

      $ oc apply -f sriov-network-attachment.yaml

4. Create the VLAN secondary network:

   1. Using the following YAML example, create a file named vlan100-additional-network-configuration.yaml:

      Copy to Clipboard Toggle word wrap

      apiVersion: k8s.cni.cncf.io/v1
      kind: NetworkAttachmentDefinition
      metadata:
        name: vlan-100
        namespace: test-namespace
      spec:
        config: |
          {
            "cniVersion": "0.4.0",
            "name": "vlan-100",
            "plugins": [
              {
                "type": "vlan",
                "master": "ext0", 

      1

      
                "mtu": 1500,
                "vlanId": 100,
                "linkInContainer": true, 

      2

      
                "ipam": {"type": "whereabouts", "ipRanges": [{"range": "1.1.1.0/24"}]}
              }
            ]
          }

      1

      The VLAN configuration needs to specify the master name. This can be configured in the pod networks annotation.

      2

      The linkInContainer parameter must be specified.

   2. Apply the YAML file by running the following command:

      Copy to Clipboard Toggle word wrap

      $ oc apply -f vlan100-additional-network-configuration.yaml

5. Create a pod definition by using the earlier specified networks:

   1. Using the following YAML example, create a file named pod-a.yaml file:

      Note

      The manifest below includes 2 resources:

      • Namespace with security labels
      • Pod definition with appropriate network annotation
      Copy to Clipboard Toggle word wrap

      apiVersion: v1
      kind: Namespace
      metadata:
        name: test-namespace
        labels:
          pod-security.kubernetes.io/enforce: privileged
          pod-security.kubernetes.io/audit: privileged
          pod-security.kubernetes.io/warn: privileged
          security.openshift.io/scc.podSecurityLabelSync: "false"
      ---
      apiVersion: v1
      kind: Pod
      metadata:
        name: nginx-pod
        namespace: test-namespace
        annotations:
          k8s.v1.cni.cncf.io/networks: '[
            {
              "name": "sriov-network",
              "namespace": "test-namespace",
              "interface": "ext0" 

      1

      
            },
            {
              "name": "vlan-100",
              "namespace": "test-namespace",
              "interface": "ext0.100"
            }
          ]'
      spec:
        securityContext:
          runAsNonRoot: true
        containers:
          - name: nginx-container
            image: nginxinc/nginx-unprivileged:latest
            securityContext:
              allowPrivilegeEscalation: false
              capabilities:
                drop: ["ALL"]
            ports:
              - containerPort: 80
            seccompProfile:
              type: "RuntimeDefault"

      1

      The name to be used as the master for the VLAN interface.

   2. Apply the YAML file by running the following command:

      Copy to Clipboard Toggle word wrap

      $ oc apply -f pod-a.yaml

6. Get detailed information about the nginx-pod within the test-namespace by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc describe pods nginx-pod -n test-namespace

   Example output

   Copy to Clipboard Toggle word wrap

   Name:         nginx-pod
   Namespace:    test-namespace
   Priority:     0
   Node:         worker-1/10.46.186.105
   Start Time:   Mon, 14 Aug 2023 16:23:13 -0400
   Labels:       <none>
   Annotations:  k8s.ovn.org/pod-networks:
                   {"default":{"ip_addresses":["10.131.0.26/23"],"mac_address":"0a:58:0a:83:00:1a","gateway_ips":["10.131.0.1"],"routes":[{"dest":"10.128.0.0...
                 k8s.v1.cni.cncf.io/network-status:
                   [{
                       "name": "ovn-kubernetes",
                       "interface": "eth0",
                       "ips": [
                           "10.131.0.26"
                       ],
                       "mac": "0a:58:0a:83:00:1a",
                       "default": true,
                       "dns": {}
                   },{
                       "name": "test-namespace/sriov-network",
                       "interface": "ext0",
                       "mac": "6e:a7:5e:3f:49:1b",
                       "dns": {},
                       "device-info": {
                           "type": "pci",
                           "version": "1.0.0",
                           "pci": {
                               "pci-address": "0000:d8:00.2"
                           }
                       }
                   },{
                       "name": "test-namespace/vlan-100",
                       "interface": "ext0.100",
                       "ips": [
                           "1.1.1.1"
                       ],
                       "mac": "6e:a7:5e:3f:49:1b",
                       "dns": {}
                   }]
                 k8s.v1.cni.cncf.io/networks:
                   [ { "name": "sriov-network", "namespace": "test-namespace", "interface": "ext0" }, { "name": "vlan-100", "namespace": "test-namespace", "i...
                 openshift.io/scc: privileged
   Status:       Running
   IP:           10.131.0.26
   IPs:
     IP:  10.131.0.26

Procedure

1. Create a dedicated container namespace where you want to deploy your pod by entering the following command:

   Copy to Clipboard Toggle word wrap

   $ oc new-project test-namespace

2. Using the following YAML example, create a bridge NetworkAttachmentDefinition custom resource definition (CRD) file named bridge-nad.yaml:

   Copy to Clipboard Toggle word wrap

   apiVersion: "k8s.cni.cncf.io/v1"
   kind: NetworkAttachmentDefinition
   metadata:
     name: bridge-network
   spec:
     config: '{
       "cniVersion": "0.4.0",
       "name": "bridge-network",
       "type": "bridge",
       "bridge": "br-001",
       "isGateway": true,
       "ipMasq": true,
       "hairpinMode": true,
       "ipam": {
         "type": "host-local",
         "subnet": "10.0.0.0/24",
         "routes": [{"dst": "0.0.0.0/0"}]
       }
     }'

3. Run the following command to apply the NetworkAttachmentDefinition CRD to your OpenShift Container Platform cluster:

   Copy to Clipboard Toggle word wrap

   $ oc apply -f bridge-nad.yaml

4. Verify that you successfully created a NetworkAttachmentDefinition CRD by entering the following command:

   Copy to Clipboard Toggle word wrap

   $ oc get network-attachment-definitions

   Example output

   Copy to Clipboard Toggle word wrap

   NAME             AGE
   bridge-network   15s

5. Using the following YAML example, create a file named ipvlan-additional-network-configuration.yaml for the IPVLAN secondary network configuration:

   Copy to Clipboard Toggle word wrap

   apiVersion: k8s.cni.cncf.io/v1
   kind: NetworkAttachmentDefinition
   metadata:
     name: ipvlan-net
     namespace: test-namespace
   spec:
     config: '{
       "cniVersion": "0.3.1",
       "name": "ipvlan-net",
       "type": "ipvlan",
       "master": "net1", 

   1

   
       "mode": "l3",
       "linkInContainer": true, 

   2

   
       "ipam": {"type": "whereabouts", "ipRanges": [{"range": "10.0.0.0/24"}]}
     }'

   1

   Specifies the ethernet interface to associate with the network attachment. This is subsequently configured in the pod networks annotation.

   2

   Specifies that the master interface is in the container network namespace.

6. Apply the YAML file by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc apply -f ipvlan-additional-network-configuration.yaml

7. Verify that the NetworkAttachmentDefinition CRD has been created successfully by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc get network-attachment-definitions

   Example output

   Copy to Clipboard Toggle word wrap

   NAME             AGE
   bridge-network   87s
   ipvlan-net       9s

8. Using the following YAML example, create a file named pod-a.yaml for the pod definition:

   Copy to Clipboard Toggle word wrap

   apiVersion: v1
   kind: Pod
   metadata:
     name: pod-a
     namespace: test-namespace
     annotations:
       k8s.v1.cni.cncf.io/networks: '[
         {
           "name": "bridge-network",
           "interface": "net1" 

   1

   
         },
         {
           "name": "ipvlan-net",
           "interface": "net2"
         }
       ]'
   spec:
     securityContext:
       runAsNonRoot: true
       seccompProfile:
         type: RuntimeDefault
     containers:
     - name: test-pod
       image: quay.io/openshifttest/hello-sdn@sha256:c89445416459e7adea9a5a416b3365ed3d74f2491beb904d61dc8d1eb89a72a4
       securityContext:
         allowPrivilegeEscalation: false
         capabilities:
           drop: [ALL]

   1

   Specifies the name to be used as the master for the IPVLAN interface.

9. Apply the YAML file by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc apply -f pod-a.yaml

10. Verify that the pod is running by using the following command:

    Copy to Clipboard Toggle word wrap

    $ oc get pod -n test-namespace

    Example output

    Copy to Clipboard Toggle word wrap

    NAME    READY   STATUS    RESTARTS   AGE
    pod-a   1/1     Running   0          2m36s

11. Show network interface information about the pod-a resource within the test-namespace by running the following command:

    Copy to Clipboard Toggle word wrap

    $ oc exec -n test-namespace pod-a -- ip a

    Example output

    Copy to Clipboard Toggle word wrap

    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host
           valid_lft forever preferred_lft forever
    3: eth0@if105: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default
        link/ether 0a:58:0a:d9:00:5d brd ff:ff:ff:ff:ff:ff link-netnsid 0
        inet 10.217.0.93/23 brd 10.217.1.255 scope global eth0
           valid_lft forever preferred_lft forever
        inet6 fe80::488b:91ff:fe84:a94b/64 scope link
           valid_lft forever preferred_lft forever
    4: net1@if107: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
        link/ether be:da:bd:7e:f4:37 brd ff:ff:ff:ff:ff:ff link-netnsid 0
        inet 10.0.0.2/24 brd 10.0.0.255 scope global net1
           valid_lft forever preferred_lft forever
        inet6 fe80::bcda:bdff:fe7e:f437/64 scope link
           valid_lft forever preferred_lft forever
    5: net2@net1: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
        link/ether be:da:bd:7e:f4:37 brd ff:ff:ff:ff:ff:ff
        inet 10.0.0.1/24 brd 10.0.0.255 scope global net2
           valid_lft forever preferred_lft forever
        inet6 fe80::beda:bd00:17e:f437/64 scope link
           valid_lft forever preferred_lft forever

    This output shows that the network interface net2 is associated with the physical interface net1.