ascii codec can't decode byte 0xe2

This seems to be charmhelpers.core.hookenv.metadata not properly encoding/decoding for python3.

When yaml.safe_load is called the encoding is off.

This is likely due to Bug #1334482 , where the yaml package is decoding using the system encoding, and there isn't a system encoding so it falls back to something stupid.

Ran into the same issue.

It is possible to reproduce this in debug-hooks by unsetting LC_* variables:

unset "${!LC_@}"

env | grep LC_ ; echo $?
1

./hooks/upgrade-charm

# ...

  File "/var/lib/juju/agents/unit-keystone-saml-mellon-1/charm/hooks/relations/keystone-fid-service-provider/provides.py", line 27, in <module>
    class KeystoneFIDServiceProvides(Endpoint):
  File "/var/lib/juju/agents/unit-keystone-saml-mellon-1/charm/hooks/relations/keystone-fid-service-provider/provides.py", line 47, in KeystoneFIDServiceProvides
    @when('endpoint.{endpoint_name}.joined')
  File "/var/lib/juju/agents/unit-keystone-saml-mellon-1/.venv/lib/python3.6/site-packages/charms/reactive/decorators.py", line 85, in _when_decorator
    endpoint_names = _get_endpoint_names(action)
  File "/var/lib/juju/agents/unit-keystone-saml-mellon-1/.venv/lib/python3.6/site-packages/charms/reactive/decorators.py", line 283, in _get_endpoint_names
    endpoint_names = hookenv.role_and_interface_to_relations(role, interface)
  File "/var/lib/juju/agents/unit-keystone-saml-mellon-1/.venv/lib/python3.6/site-packages/charmhelpers/core/hookenv.py", line 82, in wrapper
    res = func(*args, **kwargs)
  File "/var/lib/juju/agents/unit-keystone-saml-mellon-1/.venv/lib/python3.6/site-packages/charmhelpers/core/hookenv.py", line 700, in role_and_interface_to_relations
    _metadata = metadata()
  File "/var/lib/juju/agents/unit-keystone-saml-mellon-1/.venv/lib/python3.6/site-packages/charmhelpers/core/hookenv.py", line 82, in wrapper
    res = func(*args, **kwargs)
  File "/var/lib/juju/agents/unit-keystone-saml-mellon-1/.venv/lib/python3.6/site-packages/charmhelpers/core/hookenv.py", line 619, in metadata
    return yaml.safe_load(md)
  File "/var/lib/juju/agents/unit-keystone-saml-mellon-1/.venv/lib/python3.6/site-packages/yaml/__init__.py", line 162, in safe_load
    return load(stream, SafeLoader)
  File "/var/lib/juju/agents/unit-keystone-saml-mellon-1/.venv/lib/python3.6/site-packages/yaml/__init__.py", line 112, in load
    loader = Loader(stream)
  File "/var/lib/juju/agents/unit-keystone-saml-mellon-1/.venv/lib/python3.6/site-packages/yaml/loader.py", line 34, in __init__
    Reader.__init__(self, stream)
  File "/var/lib/juju/agents/unit-keystone-saml-mellon-1/.venv/lib/python3.6/site-packages/yaml/reader.py", line 85, in __init__
    self.determine_encoding()
  File "/var/lib/juju/agents/unit-keystone-saml-mellon-1/.venv/lib/python3.6/site-packages/yaml/reader.py", line 124, in determine_encoding
    self.update_raw()
  File "/var/lib/juju/agents/unit-keystone-saml-mellon-1/.venv/lib/python3.6/site-packages/yaml/reader.py", line 178, in update_raw
    data = self.stream.read(size)
  File "/var/lib/juju/agents/unit-keystone-saml-mellon-1/.venv/lib/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 2725: ordinal not in range(128)

To add to #3.

Unsetting "LANG" is also necessary.

> /var/lib/juju/agents/unit-keystone-saml-mellon-1/.venv/lib/python3.6/site-packages/charmhelpers/core/hookenv.py(620)metadata()
    619 ipdb.set_trace()
--> 620 with open(os.path.join(charm_dir(), 'metadata.yaml')) as md:
    621 return yaml.safe_load(md)

ipdb> n
> /var/lib/juju/agents/unit-keystone-saml-mellon-1/.venv/lib/python3.6/site-packages/charmhelpers/core/hookenv.py(621)metadata()
    620 with open(os.path.join(charm_dir(), 'metadata.yaml')) as md:
--> 621 return yaml.safe_load(md)
    622

ipdb> md
<_io.TextIOWrapper name='/var/lib/juju/agents/unit-keystone-saml-mellon-1/charm/metadata.yaml' mode='r' encoding='ANSI_X3.4-1968'>
ipdb> os.path.join(charm_dir(), 'metadata.yaml')
'/var/lib/juju/agents/unit-keystone-saml-mellon-1/charm/metadata.yaml'

hexdump metadata.yaml | grep e2
0000000 6e22 6d61 2265 203a 6b22 7965 7473 6e6f
00000f0 6e20 6365 7365 6173 7972 6320 6e6f 6966
0000890 726f 5320 4d41 2d4c 656c 6576 206c 6e28
0000aa0 6574 6d72 e220 9c80 6973 6e67 6e69 2067 # <---- third column
0000ab0 6563 7472 6669 6369 7461 e265 9d80 6920
0000b20 6369 7461 6f69 2c6e 6e20 746f 7320 6769
0000b80 7420 6968 2073 6163 6573 2e29 4920 206e

In [1]: f = open('metadata.yaml', encoding='utf-8')

In [2]: content = f.read()

In [3]: print(content[2725])
“

In [4]: print(content[2725:2750])
“signing certificate” is

Not sure how those unicode quotation marks got there originally - we can replace them with ASCII ones.

Juju should really make sure locale settings are configured though to use UTF-8.

Review: https://review.opendev.org/#/c/665766

Reviewed: https://review.opendev.org/665766
Committed: https://git.openstack.org/cgit/openstack/charm-keystone-saml-mellon/commit/?id=dde20964eba7fd5a425f5e4e9085c839d0434ff9
Submitter: Zuul
Branch: master

commit dde20964eba7fd5a425f5e4e9085c839d0434ff9
Author: Dmitrii Shcherbakov <email address hidden>
Date: Tue Jun 18 00:23:46 2019 +0300

    Remove non-ASCII quotes in metadata.yaml

    LP: #1334482 and LP: #1832605 summarize the issue.

    Until the larger problem is fixed we can fix the immediate one by not
    using Unicode quotes in metadata.yaml.

    Change-Id: I21249679ac9aa107c0bede96a07e5d6e0b7218a8
    Closes-Bug: #1832605