Juniper Openstack

No space left of device error is being reported by vRouter agent during flow deletion

Bug #1575322 reported by Dilip Sundarraj
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R2.20
Fix Committed
High
Prabhjot Singh Sethi
Juniper Openstack r2.23 "r2.23"
R2.21.x
Fix Committed
High
Prabhjot Singh Sethi
Juniper Openstack r2.21.2
R2.22.x
Fix Committed
High
Prabhjot Singh Sethi
Juniper Openstack r2.22.2
R3.0
Fix Committed
High
Prabhjot Singh Sethi
Juniper Openstack r3.0.2.0
Trunk
Fix Committed
High
Prabhjot Singh Sethi
Juniper Openstack r3.1.0.0-fcs

Bug Description

We've observed this in a customer production deployment running R2.21.2 Build 33.

There are atleast 3 vRouters which are reporting this issue currently. Please find their corresponding agent logs here

10.84.5.31:/users/dsundarraj/ge/apr26

VRouter [SYS_ERR]: VRouterError: VRouter operation failed. Error < 28 : No space left on device >. Object < Flow : 4294967295 with Source IP: 10.35.16.13 Source port: 4222 Destination IP: 10.35.49.24 Destination port: 38374 Protocol 6 >. Operation < Deletion >.
Message number : 73243201 controller/src/vnsw/agent/vrouter/ksync/flowtable_ksync.cc 497

Tags: vmware vrouter
Revision history for this message
gagandeeps@juniper.net (gagandeeps) wrote :

@Hari Prasad Killi: Can you please update the status of bug. Customer is asking for status.

Hari Prasad Killi (haripk)
Changed in juniperopenstack:
assignee: Hari Prasad Killi (haripk) → Prabhjot Singh Sethi (prabhjot)
Revision history for this message
Prabhjot Singh Sethi (prabhjot) wrote :

this is a cosmetic issue, ENOSPC error was returned for flow Add/Change operation, however before processing the Ack (with ENOSPC error), there was a delete operation triggered, while logging this message it wrongly correlated it to delete operation instead of Add/Change

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/19804
Submitter: Prabhjot Singh Sethi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/19804
Committed: http://github.org/Juniper/contrail-controller/commit/b488b36b6737bb7558dcc5963d2a314cf153fa0f
Submitter: Zuul
Branch: master

commit b488b36b6737bb7558dcc5963d2a314cf153fa0f
Author: Prabhjot Singh Sethi <email address hidden>
Date: Mon May 2 16:23:55 2016 +0530

Fix Error Log for KSync Error Handler

Issue:
------
Currently Ksync Error Handler tries to get operation string
from the current state which may not be same as the
operation for which the ack was received and ends up
reporting wrong error

Fix:
----
Use Ack event to get the operation string instead of state

Closes-Bug: 1575322
Change-Id: I77f928c28ee8afaa8a0bbe6585edd98d7cd06646

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/19832
Submitter: Prabhjot Singh Sethi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.20

Review in progress for https://review.opencontrail.org/19834
Submitter: Prabhjot Singh Sethi (<email address hidden>)

Prabhjot Singh Sethi (prabhjot)
tags: added: vrouter
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.21.x

Review in progress for https://review.opencontrail.org/19840
Submitter: Prabhjot Singh Sethi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.22.x

Review in progress for https://review.opencontrail.org/19841
Submitter: Prabhjot Singh Sethi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/19832
Committed: http://github.org/Juniper/contrail-controller/commit/f588b2ab00acddb39a8c0378e6aaf0a5719ca6f2
Submitter: Zuul
Branch: R3.0

commit f588b2ab00acddb39a8c0378e6aaf0a5719ca6f2
Author: Prabhjot Singh Sethi <email address hidden>
Date: Mon May 2 16:23:55 2016 +0530

Fix Error Log for KSync Error Handler

Issue:
------
Currently Ksync Error Handler tries to get operation string
from the current state which may not be same as the
operation for which the ack was received and ends up
reporting wrong error

Fix:
----
Use Ack event to get the operation string instead of state

Closes-Bug: 1575322
Change-Id: I77f928c28ee8afaa8a0bbe6585edd98d7cd06646
(cherry picked from commit b488b36b6737bb7558dcc5963d2a314cf153fa0f)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/19834
Committed: http://github.org/Juniper/contrail-controller/commit/cccdfd188c87d9e864052f4d46931377a797d759
Submitter: Zuul
Branch: R2.20

commit cccdfd188c87d9e864052f4d46931377a797d759
Author: Prabhjot Singh Sethi <email address hidden>
Date: Mon May 2 16:23:55 2016 +0530

Fix Error Log for KSync Error Handler

Issue:
------
Currently Ksync Error Handler tries to get operation string
from the current state which may not be same as the
operation for which the ack was received and ends up
reporting wrong error

Fix:
----
Use Ack event to get the operation string instead of state

Conflicts:
 src/ksync/ksync_object.cc
 src/ksync/ksync_sock.cc

Closes-Bug: 1575322
Change-Id: I77f928c28ee8afaa8a0bbe6585edd98d7cd06646
(cherry picked from commit b488b36b6737bb7558dcc5963d2a314cf153fa0f)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/19841
Committed: http://github.org/Juniper/contrail-controller/commit/37ae5e0bc5b7b53773263bb50a85a98ebf520413
Submitter: Zuul
Branch: R2.22.x

commit 37ae5e0bc5b7b53773263bb50a85a98ebf520413
Author: Prabhjot Singh Sethi <email address hidden>
Date: Mon May 2 16:23:55 2016 +0530

Fix Error Log for KSync Error Handler

Issue:
------
Currently Ksync Error Handler tries to get operation string
from the current state which may not be same as the
operation for which the ack was received and ends up
reporting wrong error

Fix:
----
Use Ack event to get the operation string instead of state

Conflicts:
 src/ksync/ksync_sock.cc

Closes-Bug: 1575322
Change-Id: I77f928c28ee8afaa8a0bbe6585edd98d7cd06646
(cherry picked from commit b488b36b6737bb7558dcc5963d2a314cf153fa0f)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/19840
Committed: http://github.org/Juniper/contrail-controller/commit/c211a4ca508312dc74df69d91d9ec9ca8464cb26
Submitter: Zuul
Branch: R2.21.x

commit c211a4ca508312dc74df69d91d9ec9ca8464cb26
Author: Prabhjot Singh Sethi <email address hidden>
Date: Mon May 2 16:23:55 2016 +0530

Fix Error Log for KSync Error Handler

Issue:
------
Currently Ksync Error Handler tries to get operation string
from the current state which may not be same as the
operation for which the ack was received and ends up
reporting wrong error

Fix:
----
Use Ack event to get the operation string instead of state

Conflicts:
 src/ksync/ksync_object.cc
 src/ksync/ksync_sock.cc
 src/vnsw/agent/vrouter/ksync/flowtable_ksync.cc
 src/vnsw/agent/vrouter/ksync/flowtable_ksync.h

Closes-Bug: 1575322
Change-Id: I77f928c28ee8afaa8a0bbe6585edd98d7cd06646
(cherry picked from commit b488b36b6737bb7558dcc5963d2a314cf153fa0f)

Ashish Ranjan (aranjan-n)
information type: Proprietary → Public
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.