OpenStack Identity (keystone)

Size limit exceeded when querying AD with more than 1000 entries

Bug #1083463 reported by Jose Castro Leon
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Fix Released
Undecided
Jose Castro Leon
OpenStack Identity (keystone) 2013.1 "grizzly"

Bug Description

When the objects retrieved via LDAP module surpass 1000, Active Directory refuses the query operation with an exception 'size limit exceeded'.

2012-11-27 08:10:11 ERROR [root] {'desc': 'Size limit exceeded'}
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/keystone/common/wsgi.py", line 204, in __call__
    result = method(context, **params)
  File "/usr/lib/python2.6/site-packages/keystone/identity/core.py", line 467, in get_users
    return {'users': self.identity_api.list_users(context)}
  File "/usr/lib/python2.6/site-packages/keystone/common/manager.py", line 47, in _wrapper
    return f(*args, **kw)
  File "/usr/lib/python2.6/site-packages/keystone/identity/backends/ldap/core.py", line 138, in list_users
    return self.user.get_all()
  File "/usr/lib/python2.6/site-packages/keystone/common/ldap/core.py", line 217, in get_all
    for x in self._ldap_get_all(filter)]
  File "/usr/lib/python2.6/site-packages/keystone/common/ldap/core.py", line 204, in _ldap_get_all
    return conn.search_s(self.tree_dn, ldap.SCOPE_ONELEVEL, query)
  File "/usr/lib/python2.6/site-packages/keystone/common/ldap/core.py", line 318, in search_s
    res = self.conn.search_s(dn, scope, query)
  File "/usr/lib64/python2.6/site-packages/ldap/ldapobject.py", line 516, in search_s
    return self.search_ext_s(base,scope,filterstr,attrlist,attrsonly,None,None,timeout=self.timeout)
  File "/usr/lib64/python2.6/site-packages/ldap/ldapobject.py", line 510, in search_ext_s
    return self.result(msgid,all=1,timeout=timeout)[1]
  File "/usr/lib64/python2.6/site-packages/ldap/ldapobject.py", line 436, in result
    res_type,res_data,res_msgid = self.result2(msgid,all,timeout)
  File "/usr/lib64/python2.6/site-packages/ldap/ldapobject.py", line 440, in result2
    res_type, res_data, res_msgid, srv_ctrls = self.result3(msgid,all,timeout)
  File "/usr/lib64/python2.6/site-packages/ldap/ldapobject.py", line 446, in result3
    ldap_result = self._ldap_call(self._l.result3,msgid,all,timeout)
  File "/usr/lib64/python2.6/site-packages/ldap/ldapobject.py", line 96, in _ldap_call
    result = func(*args,**kwargs)
SIZELIMIT_EXCEEDED: {'desc': 'Size limit exceeded'}
2012-11-27 08:10:11 DEBUG [keystone.common.wsgi] ******************** RESPONSE HEADERS ********************
2012-11-27 08:10:11 DEBUG [keystone.common.wsgi] Vary = X-Auth-Token
2012-11-27 08:10:11 DEBUG [keystone.common.wsgi] Content-Type = application/json
2012-11-27 08:10:11 DEBUG [keystone.common.wsgi] Content-Length = 175
2012-11-27 08:10:11 DEBUG [keystone.common.wsgi]
2012-11-27 08:10:11 DEBUG [keystone.common.wsgi] ******************** RESPONSE BODY ********************
2012-11-27 08:10:11 DEBUG [keystone.common.wsgi] {"error": {"message": "An unexpected error prevented the server from fulfilling your request. {'desc': 'Size limit exceeded'}", "code": 500, "title": "Internal Server Error"}}

Revision history for this message
Jose Castro Leon (jose-castro-leon) wrote :

It seems than a paged search on the LDAP server should work, but paged controls are only available with LDAP v3...

Revision history for this message
Wu Wenxiang (wu-wenxiang) wrote :

Should we use "search_ext + result3" instead of "search_s" to avoid SIZELIMIT_EXCEEDED exception?

affects: python-keystoneclient → keystone
Changed in keystone:
assignee: nobody → Wu Wenxiang (wu-wenxiang)
Revision history for this message
Jose Castro Leon (jose-castro-leon) wrote :

The only problem is that query is specific of LDAPv3 servers, not all of LDAP servers implement such thing.

Revision history for this message
Wu Wenxiang (wu-wenxiang) wrote :

Agree with you

Changed in keystone:
assignee: Wu Wenxiang (wu-wenxiang) → nobody
Revision history for this message
Jose Castro Leon (jose-castro-leon) wrote :

We could try to address it using configuration specific properties like 'enable_pagination' and 'pagination size'.
Then modify (if it is enabled) the behaviour of the search query on the LDAP core backend (search_s) because it is applicable to all ldap queries.

Changed in keystone:
assignee: nobody → Jose Castro Leon (jose-castro-leon)
Adam Young (ayoung)
Changed in keystone:
status: New → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/20330

Changed in keystone:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (master)

Reviewed: https://review.openstack.org/20330
Committed: http://github.com/openstack/keystone/commit/83d08cfc7bec00f2f82cff2a5101e63f52b1402b
Submitter: Jenkins
Branch: master

commit 83d08cfc7bec00f2f82cff2a5101e63f52b1402b
Author: Jose Castro Leon <email address hidden>
Date: Tue Feb 19 08:30:38 2013 +0100

    Enable a parameters on ldap to allow paged_search of ldap queries
    This fixes bug 1083463

    Change-Id: Ie7ec7f2214b51766d3108a4557c096d9e6989b6b

Changed in keystone:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in keystone:
milestone: none → grizzly-rc1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in keystone:
milestone: grizzly-rc1 → 2013.1
Revision history for this message
Jean-Sebastien Mouret (js-mouret) wrote :

Whenever I enable page_size, keystone crash with:

    ERROR keystone.common.wsgi [-] 'module' object has no attribute 'LDAP_CONTROL_PAGE_OID'
    TRACE keystone.common.wsgi Traceback (most recent call last):
    TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 207, in __call__
    TRACE keystone.common.wsgi result = method(context, **params)
    TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/identity/controllers.py", line 112, in get_users
    TRACE keystone.common.wsgi user_list = self.identity_api.list_users()
    TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/manager.py", line 47, in wrapper
    TRACE keystone.common.wsgi return f(self, *args, **kwargs)
    TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 189, in wrapper
    TRACE keystone.common.wsgi return f(self, *args, **kwargs)
    TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 328, in list_users
    TRACE keystone.common.wsgi ref_list = driver.list_users(hints or driver_hints.Hints())
    TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/identity/backends/ldap.py", line 81, in list_users
    TRACE keystone.common.wsgi return self.user.get_all_filtered()
    TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/identity/backends/ldap.py", line 245, in get_all_filtered
    TRACE keystone.common.wsgi return [identity.filter_user(user) for user in self.get_all()]
    TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/ldap/core.py", line 786, in get_all
    TRACE keystone.common.wsgi return super(EnabledEmuMixIn, self).get_all(ldap_filter)
    TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/ldap/core.py", line 418, in get_all
    TRACE keystone.common.wsgi for x in self._ldap_get_all(ldap_filter)]
    TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/ldap/core.py", line 394, in _ldap_get_all
    TRACE keystone.common.wsgi self.attribute_mapping.values())
    TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/ldap/core.py", line 594, in search_s
    TRACE keystone.common.wsgi res = self.paged_search_s(dn, scope, query, attrlist)
    TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/ldap/core.py", line 618, in paged_search_s
    TRACE keystone.common.wsgi controlType=ldap.LDAP_CONTROL_PAGE_OID,
    TRACE keystone.common.wsgi AttributeError: 'module' object has no attribute 'LDAP_CONTROL_PAGE_OID'

Version: keystone 1:2014.1+git201404071543~trusty-0ubuntu1

Revision history for this message
Eric Brown (ericwb) wrote :
Download full text (4.8 KiB)

I'm getting the same exception as Jean-Sebastien Mouret. I'm using Icehouse, with page_size=100. Also, after taking away the page_size conf setting, now I can't login. Seems my admin user is no longer assigned to a project or role. Probably a result of the paging code failing, it automatically removed some reslationships.

I think this bug should be reopened.

Here is my ldap version.

root@controller01:~# dpkg --list |grep ldap
ii libaprutil1-ldap:amd64 1.5.3-1 amd64 Apache Portable Runtime Utility Library - LDAP Driver
ii libldap-2.4-2:amd64 2.4.31-1+nmu2ubuntu8 amd64 OpenLDAP libraries
ii python-ldap 2.4.10-1build1 amd64 LDAP interface module for Python

Traceback:

2015-01-29 02:47:49.528 14775 DEBUG keystone.common.ldap.core [-] LDAP unbind unbind_s /usr/lib/python2.7/dist-packages/keystone/common/ldap/cor
e.py:789
2015-01-29 02:47:49.529 14775 ERROR keystone.common.wsgi [-] 'module' object has no attribute 'LDAP_CONTROL_PAGE_OID'
2015-01-29 02:47:49.529 14775 TRACE keystone.common.wsgi Traceback (most recent call last):
2015-01-29 02:47:49.529 14775 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 212, in __call_
_
2015-01-29 02:47:49.529 14775 TRACE keystone.common.wsgi result = method(context, **params)
2015-01-29 02:47:49.529 14775 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py", line 326, in au
thenticate_for_token
2015-01-29 02:47:49.529 14775 TRACE keystone.common.wsgi self._check_and_set_default_scoping(auth_info, auth_context)
2015-01-29 02:47:49.529 14775 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py", line 364, in _c
heck_and_set_default_scoping
2015-01-29 02:47:49.529 14775 TRACE keystone.common.wsgi user_ref = self.identity_api.get_user(auth_context['user_id'])
2015-01-29 02:47:49.529 14775 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 189, in wrapp
er
2015-01-29 02:47:49.529 14775 TRACE keystone.common.wsgi return f(self, *args, **kwargs)
2015-01-29 02:47:49.529 14775 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 307, in get_u
ser
2015-01-29 02:47:49.529 14775 TRACE keystone.common.wsgi ref = driver.get_user(user_id)
2015-01-29 02:47:49.529 14775 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/identity/backends/ldap.py", line 78,
in get_user
2015-01-29 02:47:49.529 14775 TRACE keystone.common.wsgi return identity.filter_user(self._get_user(user_id))
2015-01-29 02:47:49.529 14775 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/identity/backends/ldap.py", line 75,
in _get_user
2015-01-29 02:47:49.529 14775 TRACE keystone.common.wsgi return self.user.get(user_id)
2015-01-29 02:47:49.529 14775 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/ldap/core.py", line 1006, in g
et
2015-01-29 02:47:49.529 ...

Read more...

Revision history for this message
Eric Brown (ericwb) wrote :

Looks like bug https://bugs.launchpad.net/keystone/+bug/1381768 covers this latest issue.

Revision history for this message
Jose Castro Leon (jose-castro-leon) wrote :

It seems a change between ldap libraries python-ldap 2.3 controls are not compatible with 2.4 ones

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.