OpenStack Identity (keystone)

unicode query string raises UnicodeEncodeError

Bug #1448286 reported by Dolph Mathews on 2015-04-24

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Identity (keystone)	Fix Released	Medium	Dolph Mathews	OpenStack Identity (keystone) 8.0.0 "liberty"
	Kilo	Fix Released	Medium	Dolph Mathews	OpenStack Identity (keystone) 2015.1.3

Bug Description

The logging in keystone.common.wsgi is unable to handle unicode query strings. The simplest example would be:

$ curl http://localhost:35357/?Ϡ

This will fail with a backtrace similar to:

  2015-04-24 19:57:45.860 22255 TRACE keystone.common.wsgi File ".../keystone/keystone/common/wsgi.py", line 234, in __call__
  2015-04-24 19:57:45.860 22255 TRACE keystone.common.wsgi 'params': urllib.urlencode(req.params)})
  2015-04-24 19:57:45.860 22255 TRACE keystone.common.wsgi File "/usr/lib/python2.7/urllib.py", line 1311, in urlencode
  2015-04-24 19:57:45.860 22255 TRACE keystone.common.wsgi k = quote_plus(str(k))
  2015-04-24 19:57:45.860 22255 TRACE keystone.common.wsgi UnicodeEncodeError: 'ascii' codec can't encode character u'\u03e0' in position 0: ordinal not in range(128)

Tags:

Dolph Mathews (dolph) on 2015-04-24

Changed in keystone:
importance:	Undecided → Medium
tags:	added: kilo-backport-potential

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-24: Fix proposed to keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/177427

Changed in keystone:
assignee:	nobody → Dolph Mathews (dolph)
status:	New → In Progress

Lin Hua Cheng (lin-hua-cheng) on 2015-04-27

Changed in keystone:
milestone:	none → liberty-1

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-29: Fix merged to keystone (master)

Reviewed: https://review.openstack.org/177427
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=f2a75866319c249e8ebdfb393172ab5ca8816b22
Submitter: Jenkins
Branch: master

commit f2a75866319c249e8ebdfb393172ab5ca8816b22
Author: Dolph Mathews <email address hidden>
Date: Fri Apr 24 20:05:35 2015 +0000

Allow wsgiref to reconstruct URIs per the WSGI spec

Instead of reimplementing URL reconstruction for logging purposes, and
getting it wrong over and over and over, let's just use wsgiref.util.

Change-Id: Ib64877c35ad856bdef8f64f6df44a054eaa2321b
Closes-Bug: 1448286

Changed in keystone:
status:	In Progress → Fix Committed

Doug Hellmann (doug-hellmann) on 2015-06-23

Changed in keystone:
status:	Fix Committed → Fix Released

Revision history for this message

Dolph Mathews (dolph) wrote on 2015-07-14:

In order for this to be safely backported to kilo, the fix for bug 1454968 needs to be included as well (the fix for this bug revealed the problem fixed in bug 1454968).

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-07-14: Fix proposed to keystone (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/201708

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-07-14: Fix proposed to keystone (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/201726

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-08-05: Change abandoned on keystone (stable/juno)

Change abandoned by Dolph Mathews (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/201726

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-08-29: Fix merged to keystone (stable/kilo)

Reviewed: https://review.openstack.org/201708
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=20fd5af10d1551721b9c95f51fbc166977c016ae
Submitter: Jenkins
Branch: stable/kilo

commit 20fd5af10d1551721b9c95f51fbc166977c016ae
Author: Dolph Mathews <email address hidden>
Date: Fri Apr 24 20:05:35 2015 +0000

Fix WSGI request logging

    This is a backport of two patches that must be carried together. The
    first patch, which is a desirable backport, revealed a problem with
    middleware's handling of SCRIPT_NAME values, and so they must be
    backported together to avoid introducing a regression in a stable
    branch.

The original commit messages are below.

---

Allow wsgiref to reconstruct URIs per the WSGI spec

Instead of reimplementing URL reconstruction for logging purposes, and
getting it wrong over and over and over, let's just use wsgiref.util.

    Change-Id: Ib64877c35ad856bdef8f64f6df44a054eaa2321b
    Closes-Bug: 1448286
    (cherry picked from commit f2a75866319c249e8ebdfb393172ab5ca8816b22)

---

Fix req.environ[SCRIPT_NAME] value.

    The computation done inside
    /usr/local/lib/python2.7/dist-packages/routes/middleware.py
    was leaving req.environ[SCRIPT_NAME] with a wrong value with
    a repeated text block. No code in keystone seems to use the
    value of this variable, hence it has escaped detection until
    https://review.openstack.org/#/c/177427/ which logs req URLs
    by using SCRIPT_NAME in the url construction (via
    wsgiref.util.request_uri()).

    I backtraced the execution flow to the middleware.py file
    from the keystone middleware components. Prepending a '/'
    to the mapper path used in the ExtensionsRouter seems harmless
    as well as seems to fix the problem.

See log files before and after, from sample Jenkins runs.

BEFORE: http://goo.gl/YI89e4
AFTER (for this review): http://goo.gl/I1qVaI

Closes-Bug:#1454968

Change-Id: Ia9bf1e1644c996e5aa71ac4e634ead3c4de7edcf
(cherry picked from commit f7385b70b7556c703c96d86223d73d7e9beabd6b)

Reviewed:  https://review.openstack.org/201708
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=20fd5af10d1551721b9c95f51fbc166977c016ae
Submitter: Jenkins
Branch:    stable/kilo

commit 20fd5af10d1551721b9c95f51fbc166977c016ae
Author: Dolph Mathews <dolph.mathews@gmail.com>
Date:   Fri Apr 24 20:05:35 2015 +0000

Fix WSGI request logging
    
    This is a backport of two patches that must be carried together. The
    first patch, which is a desirable backport, revealed a problem with
    middleware's handling of SCRIPT_NAME values, and so they must be
    backported together to avoid introducing a regression in a stable
    branch.
    
    The original commit messages are below.
    
    ---
    
    Allow wsgiref to reconstruct URIs per the WSGI spec
    
    Instead of reimplementing URL reconstruction for logging purposes, and
    getting it wrong over and over and over, let's just use wsgiref.util.
    
    Change-Id: Ib64877c35ad856bdef8f64f6df44a054eaa2321b
    Closes-Bug: 1448286
    (cherry picked from commit f2a75866319c249e8ebdfb393172ab5ca8816b22)
    
    ---
    
    Fix req.environ[SCRIPT_NAME] value.
    
    The computation done inside
    /usr/local/lib/python2.7/dist-packages/routes/middleware.py
    was leaving req.environ[SCRIPT_NAME] with a wrong value with
    a repeated text block. No code in keystone seems to use the
    value of this variable, hence it has escaped detection until
    https://review.openstack.org/#/c/177427/ which logs req URLs
    by using SCRIPT_NAME in the url construction (via
    wsgiref.util.request_uri()).
    
    I backtraced the execution flow to the middleware.py file
    from the keystone middleware components. Prepending a '/'
    to the mapper path used in the ExtensionsRouter seems harmless
    as well as seems to fix the problem.
    
    See log files before and after, from sample Jenkins runs.
    
    BEFORE: http://goo.gl/YI89e4
    AFTER (for this review): http://goo.gl/I1qVaI
    
    Closes-Bug:#1454968
    
    Change-Id: Ia9bf1e1644c996e5aa71ac4e634ead3c4de7edcf
    (cherry picked from commit f7385b70b7556c703c96d86223d73d7e9beabd6b)

Thierry Carrez (ttx) on 2015-10-15

Changed in keystone:
milestone:	liberty-1 → 8.0.0

Report a bug

This report contains Public information

Everyone can see this information.

Duplicates of this bug

Bug #1474162

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.