authenticate ldap binary fields fail when converting fields to utf8

This fix is not working for me. Though it did change the error slightly:

before:
'utf8' codec can't decode byte 0x9d in position 13: invalid start byte

after:
[-] 'utf8' codec can't decode byte 0xf5 in position 2: invalid start byte

Aimon: is your id attribute a binary field?

Almon
I fear I may have mis-labeled this bug. My case was an issue simply with authenticating. The fix I show above should allow someone who is authenticating (and passes through the _id_to_dn code) to narrow the scope of attributes for that call only.

However, there may be other locations where ldap attributes are converted to/from utf8. The following effort was added to keystone to better handle utf-8: https://github.com/openstack/keystone/commit/cbf805161b84f13f459a19bfd46220c4f298b264

Given the scope of this, there could likely other locations where binary fields are converted.

Take a look at the following configuration fields in your keystone.conf:
user_objectclass
user_domain_id_attribute
user_id_attribute
user_name_attribute
user_mail_attribute
user_pass_attribute
user_enabled_attribute
(possibly others)

Some questions to ask yourself when tackling this issue:
* Are any of the fields listed above binary for your ldap?
* Do you have them all configured?
* If you have debug enabled, what search parameters does it output? (i.e. is attrlist specified or not?)
  - If no attrlist is specified, it will default to returning them all (from what I've found).
  - If no attrlist, what was the code path to get there (should hopefully be in a log trace)

Thank you for your reply. More info on my issue can be found here along with full config and log trace: https://ask.openstack.org/en/question/44834/401-on-authenticated-keystone-calls-when-using-ad-on-icehouse/ . I checked all fields and attempted to exclude any non text via tenant_attribute_ignore. It did not help. The 2008 server R2 that I am using is a clean install used only for openstack testing. No users were added or edited other then the openstack users. Let me know of I should turn that ask.openstack.org post into a separate bug.

So your fix does work. However it fails again because there is a spot with no attrlist in assignment/backends/ldap.py:

    def get_role_assignments(self, tenant_dn):
        conn = self.get_connection()
        query = '(objectClass=%s)' % self.object_class

        try:
            roles = conn.search_s(tenant_dn, ldap.SCOPE_ONELEVEL, query)

Turns out that the behavior for "no attribute list" is to return all attributes. Which means that we need to always specify the set of attributes a query should return

Re comment #3

There is no solution other than knowing the data type of an attribute you're querying. You either only retreive attributes known a priori and enforce the fact they are strings. Or you can hardcode a set of attributes which are known to be binary (hardcoded lists are problematic). Or you retreive the schema from the LDAP server and use the schema to lookup the attribute type.

The later (schema lookups) is the robust correct way to do it. It always seemed to me a deficiencey of the Keystone LDAP code that it didn't load the schema.

Fix proposed to branch: master
Review: https://review.openstack.org/119457

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/119578

Reviewed: https://review.openstack.org/119457
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=940551c6bdd1477574c1dee165efe75366343bd7
Submitter: Jenkins
Branch: master

commit 940551c6bdd1477574c1dee165efe75366343bd7
Author: Nathan Kinder <email address hidden>
Date: Fri Sep 5 07:25:21 2014 -0700

    Avoid conversion of binary LDAP values

    A few of the LDAP searches performed in the identity and assignment
    LDAP drivers do not specify the list of attributes to return, which
    causes all attributes present in the LDAP entry to be returned. We
    attempt to convery these values from utf8 to unicode, which fails
    when we encounter a binary value. This patch addresses this problem
    in two ways.

    The first is to only request the attributes that we actually need
    returned from the LDAP server. This avoids potential binary values
    that we do not even need to be concerned with.

    The second thing this patch does is to make our conversion code
    more tolerant by ignoring attributes that contain binary values. If
    a binary value is present, we simply skip over that attribute in
    the conversion and log a debug message. We should not be encounting
    binary values in a typical Keystone deployment, but this gives us
    some protection just in case we do encounter a binary value.

    Closes-bug: #1355489
    Change-Id: I62325ab9c75bac67cca329fb7bd3c74aea7d2867

Reviewed: https://review.openstack.org/119578
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=25cbcf5c8ab68c2805d6cf4f2ddaa77ac0f2ca22
Submitter: Jenkins
Branch: stable/icehouse

commit 25cbcf5c8ab68c2805d6cf4f2ddaa77ac0f2ca22
Author: Nathan Kinder <email address hidden>
Date: Fri Sep 5 07:25:21 2014 -0700

    Avoid conversion of binary LDAP values

    A few of the LDAP searches performed in the identity and assignment
    LDAP drivers do not specify the list of attributes to return, which
    causes all attributes present in the LDAP entry to be returned. We
    attempt to convery these values from utf8 to unicode, which fails
    when we encounter a binary value. This patch addresses this problem
    in two ways.

    The first is to only request the attributes that we actually need
    returned from the LDAP server. This avoids potential binary values
    that we do not even need to be concerned with.

    The second thing this patch does is to make our conversion code
    more tolerant by ignoring attributes that contain binary values. If
    a binary value is present, we simply skip over that attribute in
    the conversion and log a debug message. We should not be encounting
    binary values in a typical Keystone deployment, but this gives us
    some protection just in case we do encounter a binary value.

    Conflicts:
            keystone/assignment/backends/ldap.py
            keystone/identity/backends/ldap.py

    Closes-bug: #1355489
    Change-Id: I62325ab9c75bac67cca329fb7bd3c74aea7d2867
    (cherry picked from commit 940551c6bdd1477574c1dee165efe75366343bd7)

Fix proposed to branch: master
Review: https://review.openstack.org/121711

Change abandoned by Ryan Hsu (<email address hidden>) on branch: master
Review: https://review.openstack.org/121711
Reason: Testing