Exception AssertionError: continuously logged in nodemgr log file

bug update...

I'm hitting this error as well. contrail-analytics-nodemgr-stderr.log file is flooded with this error:
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored

contrail-analytics 3.0.0.0-2725

HA setup with 3 nodes.

Me too facing this issue. Can we get any details on why are we seeing this issue. Have below questions in my mind.
+ Do we need to provide any remedy for this.
+ If log file /var/log/contrail/contrail-analytics-nodemgr-stderr.log is flooded with this error, and as per the state of this bug "wont fix" we can ignore this error. Will it impact space?

Please help me to understand. Thanks in advance.

I also meeting this error, but could not know why it is ?

Hi Hari,

Could we have a fix to filter these being ignored AssertionError error logs? Customer opened JTAC case to ask to filter this error log in contrail-*nodemgr-stderr.log.

JTAC local testbed Contrail controller node and compute nodes also see the error:
CONT202: 10.85.19.202 root/Juniper
COMP204: 10.85.19.204 root/Juniper

root@cont202:/var/log/contrail# ls -alF *nodemgr-stderr*
-rw-r--r-- 1 root root 37815238 Dec 27 09:58 contrail-analytics-nodemgr-stderr.log
-rw-r--r-- 1 root root 2095893 Dec 27 09:58 contrail-config-nodemgr-stderr.log
-rw-r--r-- 1 root root 52428835 Dec 25 20:58 contrail-config-nodemgr-stderr.log.1
-rw-r--r-- 1 root root 35972584 Dec 27 09:58 contrail-control-nodemgr-stderr.log
-rw-r--r-- 1 root root 22118098 Dec 27 09:58 contrail-database-nodemgr-stderr.log
-rw-r--r-- 1 root root 52428846 Dec 16 00:18 contrail-database-nodemgr-stderr.log.1
root@cont202:/var/log/contrail#
root@cont202:/var/log/contrail#
root@cont202:/var/log/contrail# grep -rn AssertionError contrail-config-nodemgr-stderr.log | less
2:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
3:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
4:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
5:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
6:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
7:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
8:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
9:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
11:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
12:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
13:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
14:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
15:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
16:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
17:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
18:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
20:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
21:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
22:Exception AssertionError: Assertion...

Hi Jack, please advise on the plan to develop the fix. Thank you

LP1466483 fix planned in Contrail 5.0.

Some info on reasons for error:
https://stackoverflow.com/questions/8774958/keyerror-in-module-threading-after-a-successful-py-test-run

More info: http://blog.codepainters.com/2012/11/20/gevent-monkey-patching-versus-sniffer-nose/

=============== traceback begin =======
File "/usr/bin/contrail-nodemgr", line 9, in <module>
    load_entry_point('nodemgr==0.1dev', 'console_scripts', 'contrail-nodemgr')()
File "/usr/lib/python2.7/dist-packages/nodemgr/main.py", line 239, in main
    for line in traceback.format_stack():
=============== traceback end =======

On 5.0 the following commands do not show the above exception:
journalctl -u contrail-analytics-nodemgr.service
journalctl -u contrail-analytics-nodemgr.service -b
systemctl status contrail-analytics-nodemgr.service

root@cont202:/etc/contrail# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-analytics 3.0.3.3-22 22

Hi Jack,

So you are saying Contrail 5.0 do not see these continuous "Exception AssertionError" any more?

And hence no need any fix in Contrail 5.0, right?

Thanks, Jiang Lu

Hi Jack,

Can we have the fix plan for this LP1466483 here?

Thanks, Jiang Lu

Hi Sundar,

Now I see this LP being assigned to you. Could you have a rough fix plan for this LP1466483?

Thanks, Jiang Lu

Can we reassign this LP as previous engineer Sundar being on leave?

Hi Santosh,

Can we have a fix plan for this LP1466483?

Understand it is a low priority LP, but AT&T still need the fix in 3.2.x.

Thanks, Jiang Lu

https://bugs.launchpad.net/juniperopenstack/+bug/1466483

Corresponding jira:
https://contrail-jws.atlassian.net/browse/JCB-173997

Thanks,
- Himanshu B.