Ubuntu
python-netaddr package

[2.1] netaddr assumes MAC OUI is ascii

  1. Artful (17.10)
  2. Bug #1628761
Bug #1628761 reported by Mike Pontillo
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
High
Mike Pontillo
MAAS 2.1.0
python-netaddr (Ubuntu)
Fix Released
Medium
Unassigned
Xenial
Triaged
Medium
Unassigned
Zesty
Triaged
Medium
Unassigned
Artful
Triaged
Medium
Unassigned
Bionic
Fix Released
Medium
Unassigned

Bug Description

This causes a traceback when we try to get the MAC organization, as follows:

    --- <exception caught here> ---
      File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 246, in inContext
        result = inContext.theWork()
      File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 262, in <lambda>
        inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
      File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 118, in callWithContext
        return self.currentContext().callWithContext(ctx, func, *args, **kw)
      File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 81, in callWithContext
        return func(*args,**kw)
      File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 827, in callInContext
        return func(*args, **kwargs)
      File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 603, in call_within_transaction
        return func_outside_txn(*args, **kwargs)
      File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 422, in retrier
        return func(*args, **kwargs)
      File "/usr/lib/python3.5/contextlib.py", line 30, in inner
        return func(*args, **kwds)
      File "/usr/lib/python3/dist-packages/maasserver/websockets/base.py", line 356, in list
        for obj in objs
      File "/usr/lib/python3/dist-packages/maasserver/websockets/base.py", line 356, in <listcomp>
        for obj in objs
      File "/usr/lib/python3/dist-packages/maasserver/websockets/base.py", line 205, in full_dehydrate
        return self.dehydrate(obj, data, for_list=for_list)
      File "/usr/lib/python3/dist-packages/maasserver/websockets/handlers/discovery.py", line 38, in dehydrate
        data["mac_organization"] = obj.mac_organization
      File "/usr/lib/python3/dist-packages/maasserver/models/discovery.py", line 228, in mac_organization
        return get_mac_organization(str(self.mac_address))
      File "/usr/lib/python3/dist-packages/provisioningserver/utils/network.py", line 817, in get_mac_organization
        return get_eui_organization(EUI(mac))
      File "/usr/lib/python3/dist-packages/provisioningserver/utils/network.py", line 802, in get_eui_organization
        registration = eui.oui.registration()
      File "/usr/lib/python3/dist-packages/netaddr/eui/__init__.py", line 478, in oui
        return OUI(self.value >> 24)
      File "/usr/lib/python3/dist-packages/netaddr/eui/__init__.py", line 97, in __init__
        data = fh.read(size)
      File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
        return codecs.ascii_decode(input, self.errors)[0]
    builtins.UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2420: ordinal not in range(128)

Related branches

lp://qastaging/~mpontillo/maas/netaddr-assumes-mac-oui-ascii--bug-1628761
Blake Rouse (community): Approve
Diff: 42 lines (+14/-0)
2 files modified
src/provisioningserver/utils/network.py (+5/-0)
src/provisioningserver/utils/tests/test_network.py (+9/-0)
Revision history for this message
Mike Pontillo (mpontillo) wrote :

This seems to be a bug in netaddr and/or a side effect of corrupt entries, in oui.txt, depending on how you approach the problem.

In '/usr/lib/python3/dist-packages/netaddr/eui/oui.txt' there are entries such as:

  00-22-59 (hex) Guangzhou New Postcom Equipment Co.,Ltd.
  002259 (base 16) Guangzhou New Postcom Equipment Co.,Ltd.
                                No.3¡¡Guangpuxi Road,Guangzhou Science City,
                                Guangzhou Guangdong 510663
                                CHINA

Note the non-ASCII characters. (They don't look correct, so I doubt the entries in this file were imported with a consistent source encoding.) There are numerous examples of characters like this if you search through the file for characters with an 8th bit set.

Mike Pontillo (mpontillo)
summary: - netaddr assumes MAC OUI is ascii
+ [2.1] netaddr assumes MAC OUI is ascii
Changed in maas:
status: New → Triaged
importance: Undecided → High
Mike Pontillo (mpontillo)
Changed in maas:
assignee: nobody → Mike Pontillo (mpontillo)
milestone: none → 2.1.0
Revision history for this message
Gavin Panella (allenap) wrote :

FWIW, http://paste.ubuntu.com/23275209/ (a short Python 3 script) makes me fairly sure that everything in netaddr's oui.txt is UTF-8. The script tries to guess the encoding of each entry using chardet, ignores those guessed as ASCII or UTF-8, then prints each entry after being decoded using the guessed encoding and again using UTF-8. The UTF-8 result always appears to be the better choice: http://paste.ubuntu.com/23275206/.

I mention this because it means that an upstream fix is probably easy, and because we may be able to monkeypatch something in the meantime. I do think the defensive approach you've taken in lp:~mpontillo/maas/netaddr-assumes-mac-oui-ascii--bug-1628761 is the right one for now though.

Of course, the example in the bug description is an exception to the rule: I can't find anything to decode that and produce something reasonable, though UTF-8 does at least offer a non-crashing option. I suspect it's been mangled, maybe by copy-and-paste at some indeterminate time in the past, and the original is now as good as unrecoverable from oui.txt. Similar corruption appears in the oui.txt that the ieee-data package installs.

MAAS Lander (maas-lander)
Changed in maas:
status: Triaged → Fix Committed
Andres Rodriguez (andreserl)
Changed in maas:
status: Fix Committed → Fix Released
Revision history for this message
LaMont Jones (lamont) wrote :

Likewise, as of 0.7.18-2, the mac prefixes 00:22:59, 00:23:89, and 5c:15:15 generate IndexError, because unicode is treated as ascii.

Revision history for this message
James Page (james-page) wrote :

Only impacts py3; if you force the open of the file to use UTF-8 encoding everything is OK.

Changed in python-netaddr (Ubuntu):
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
James Page (james-page) wrote :

Fixed under:

  https://github.com/drkjam/netaddr/commit/af145601df4329a4dc55ef0ce9ce5f8645f09d4f

Changed in python-netaddr (Ubuntu):
status: Confirmed → Triaged
Revision history for this message
James Page (james-page) wrote :

That's included in the next patch release .19 (we're on .18), which we'll get as soon as bionic syncs start happening.

Revision history for this message
James Page (james-page) wrote :

$ rmadison -u debian python-netaddr
python-netaddr | 0.7.7-1 | oldoldstable | source, all
python-netaddr | 0.7.12-2 | oldstable | source, all
python-netaddr | 0.7.12-2 | oldstable-kfreebsd | source, all
python-netaddr | 0.7.18-1~bpo8+1 | jessie-backports | source, all
python-netaddr | 0.7.18-2 | stable | source, all
python-netaddr | 0.7.18-2 | testing | source, all
python-netaddr | 0.7.19-1 | unstable | source, all

Changed in python-netaddr (Ubuntu Bionic):
status: Triaged → Fix Released
Changed in python-netaddr (Ubuntu Xenial):
status: New → Triaged
importance: Undecided → Medium
Changed in python-netaddr (Ubuntu Zesty):
status: New → Triaged
Changed in python-netaddr (Ubuntu Artful):
status: New → Triaged
Changed in python-netaddr (Ubuntu Zesty):
importance: Undecided → Medium
Changed in python-netaddr (Ubuntu Artful):
importance: Undecided → Medium
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.