DPDK applications that "consume" NICs from kernel-space affect Fuel in unexpected ways

Bug #1582898 reported by Pavel Basov
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel Contrail plugin
Fix Released
Medium
Illia Polliul

Bug Description

Detailed bug description:
 DPDK application that rebinds NIC drivers and detaches interfaces from kernel space (such as Contrail vrouter in DPDK mode) will fail some tasks deployment if they are executed after the DPDK process starts.

 - Prevents running netconfig task again, since interfaces "consumed" by a DPDK process are no longer present in kernel space, but are still mentioned in network_scheme.
 - Prevents running configure_default_route, since it is using generate_network_config function.
 - If node fails to deploy for any reason and it's state is set to "error", but by that time DPDK application has already started, nailgun-agent will report that interfaces are no longer present on the node (even though they are present in lspci), which breaks bonding configuration and network-to-interface mapping in nailgun by removing interfaces from node configuration. See attached screenshot.

Steps to reproduce:
 1. Use Fuel 9.
 2. Install Contrail plugin.
 3. Deploy environment with DPDK computes.
 4. Ensure that vrouter-dpdk process is running.
 5. Re-deploy the whole graph on the DPDK compute node (or just run netconfig or configure_default_route tasks).

Expected results:
 Re-deployment of the node is successful.
Actual result:
 Any task that uses network_scheme to change networking configuration will fail to deploy.
 Example error:

/Stage[main]/Osnailyfacter::Netconfig::Netconfig/L23network::L2::Bond[bond1]/L23network::L2::Bond_interface[enp134s0f1]/L23network::L2::Port[enp134s0f1]/L2_port[enp134s0f1]/ensure: change from absent to present failed: Command 'ip link add link name enp134s0f1 type vlan id absent' has been failed with exit_code=1:
Cannot find device "name"

Reproducibility:
 Reproduces every time if dpdk process is running.
Workaround:
 Stop the DPDK application and return NICs to kernel space, before running any puppet tasks that affect networking configuration. Alternatively - cleanup network_scheme from any mentions of "consumed" interfaces"

Description of the environment:
 Operation system: Ubuntu 14.04
 Versions of components: Fuel 9, Contrail 3.0
 Network model: Contrail networking with DPDK vrouters (tunneling segmentation), LACP bonding on compute nodes.
 Related projects installed: fuel-contrail-plugin
Additional information:
 This issue does not affect DPDK-enabled openvswitch implemented by Fuel 9, just the custom DPDK applications.

Revision history for this message
Pavel Basov (pbasov) wrote :
Changed in fuel:
importance: Undecided → High
milestone: none → 10.0
tags: added: feature-dpdk
tags: added: team-network
Pavel Basov (pbasov)
tags: added: customer-found
Dmitry Klenov (dklenov)
Changed in fuel:
assignee: nobody → Networking (l23-network)
status: New → Confirmed
tags: added: area-library
Revision history for this message
Atsuko Ito (yottatsa) wrote :

Could you please show network transformation for this.

Revision history for this message
Dmitry Klenov (dklenov) wrote :

Alexander, please check if it is Fuel-related or plugin-related issue.

Changed in fuel:
assignee: Networking (l23-network) → Oleksandr Martsyniuk (omartsyniuk)
Revision history for this message
Pavel Basov (pbasov) wrote :

Vladimir, posting transformations hash:

"transformations"=>
  [{"action"=>"add-br", "name"=>"br-fw-admin"},
   {"action"=>"add-br", "name"=>"br-mgmt"},
   {"action"=>"add-br", "name"=>"br-storage"},
   {"action"=>"add-br", "name"=>"br-mesh"},
   {"action"=>"add-port", "bridge"=>"br-fw-admin", "name"=>"eno1"},
   {"action"=>"add-bond",
    "bond_properties"=>
     {"lacp_rate"=>"fast", "mode"=>"802.3ad", "xmit_hash_policy"=>"encap3+4"},
    "interface_properties"=>{"vendor_specific"=>{"disable_offloading"=>true}},
    "interfaces"=>["enp132s0f0", "enp134s0f0"],
    "name"=>"bond0"},
   {"action"=>"add-port", "bridge"=>"br-mgmt", "name"=>"bond0.2030"},
   {"action"=>"add-port", "bridge"=>"br-storage", "name"=>"bond0.2003"},
   {"action"=>"add-bond",
    "bond_properties"=>
     {"lacp_rate"=>"fast", "mode"=>"802.3ad", "xmit_hash_policy"=>"encap3+4"},
    "interface_properties"=>{"vendor_specific"=>{"disable_offloading"=>true}},
    "interfaces"=>["enp132s0f1", "enp134s0f1"],
    "name"=>"bond1"},
   {"action"=>"add-port", "bridge"=>"br-mesh", "name"=>"bond1.2004"}],

Nothing special, just a couple of bonds.

Revision history for this message
Atsuko Ito (yottatsa) wrote :

I see nothing DPDK-related. I insist on the removal of the tag

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-library (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/321067

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-library (stable/mitaka)

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/321194

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/321067
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=667812dc68e6a7e9b3650bdb63c24a19a334bbaa
Submitter: Jenkins
Branch: master

commit 667812dc68e6a7e9b3650bdb63c24a19a334bbaa
Author: Sergey Vasilenko <email address hidden>
Date: Wed May 25 18:20:46 2016 +0300

    Add ability for override transformations into network_scheme.

    New type of transformation 'override' was added, because hiera_hash()
    function does not allow to override arrays. Arrays merged instead replace.

    also fix sanitize_transformation for 'noop' transformations

    Change-Id: Ic4c9c6c34d9e3a57392446b534d997666578d110
    Related-bug: #1582898

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/321194
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=7e3c9dd1df023a22bd2a220e48ba5038eb542e9e
Submitter: Jenkins
Branch: stable/mitaka

commit 7e3c9dd1df023a22bd2a220e48ba5038eb542e9e
Author: Sergey Vasilenko <email address hidden>
Date: Wed May 25 18:20:46 2016 +0300

    Add ability for override transformations into network_scheme.

    New type of transformation 'override' was added, because hiera_hash()
    function does not allow to override arrays. Arrays merged instead replace.

    also fix sanitize_transformation for 'noop' transformations

    Change-Id: Ic4c9c6c34d9e3a57392446b534d997666578d110
    Related-bug: #1582898

tags: added: in-stable-mitaka
Atsuko Ito (yottatsa)
tags: added: l23network
removed: feature-dpdk
Changed in fuel:
assignee: Oleksandr Martsyniuk (omartsyniuk) → Sergey Vasilenko (xenolog)
tags: removed: area-library l23network team-network
Changed in fuel:
status: Confirmed → Invalid
assignee: Sergey Vasilenko (xenolog) → Fuel Plugin Contrail (fuel-plugin-contrail)
Revision history for this message
Aleksandr Didenko (adidenko) wrote :

This exists only for contrail plugin, which is not a part of Fuel core. We've merged patches to library that can allow to workaround this issue via plugin. There's nothing else we can do in library at this point. So I've removed all the fuel library/network tags for this bug and to contrail plugin developers.

Changed in fuel:
status: Invalid → Confirmed
Illia Polliul (ipolliul)
affects: fuel → fuel-plugin-contrail
Changed in fuel-plugin-contrail:
milestone: 10.0 → none
no longer affects: fuel/mitaka
Changed in fuel-plugin-contrail:
assignee: Fuel Plugin Contrail (fuel-plugin-contrail) → Illia Polliul (ipolliul)
Andrian Noga (anoga)
Changed in fuel-plugin-contrail:
milestone: none → 4.0.0
Revision history for this message
Pavel Basov (pbasov) wrote :

Changes to l23network should cover our use-cases, but I have yet to test them on baremetal lab though, will update this ticket, once I'm done with verifying.

Illia Polliul (ipolliul)
Changed in fuel-plugin-contrail:
status: Confirmed → In Progress
importance: High → Medium
Illia Polliul (ipolliul)
Changed in fuel-plugin-contrail:
status: In Progress → Fix Released
milestone: 4.0.0 → 4.0.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.