Swift broken when headers exceed 8k

Bug #1470664 reported by Nikita Koshikov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Alex Schultz
puppet-swift
Fix Released
Undecided
Matthew Mosesohn

Bug Description

This is related to deployments that using PKI/PKIZ for tokens and have more than 1 region.

Due to token becomes large (> 8k) swift can't process such requests and breaks with:

EBUG:iso8601.iso8601:Got u'55' for 'second' with default None
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 10.109.64.3
DEBUG:urllib3.connectionpool:Setting read timeout to <object object at 0x7f7b2e0af090>
DEBUG:urllib3.connectionpool:"HEAD /v1/AUTH_e46f34918675465e888114f30753b180 HTTP/1.1" 400 0
INFO:swiftclient:REQ: curl -i http://10.109.64.3:8080/v1/AUTH_e46f34918675465e888114f30753b180 -I -H "X-Auth-Token: PKIZXXX"
INFO:swiftclient:RESP STATUS: 400 Header Line Too Long
INFO:swiftclient:RESP HEADERS: [('date', 'Wed, 01 Jul 2015 22:53:31 GMT'), ('content-length', '0'), ('content-type', 'text/html; charset=UTF-8'), ('x-trans-id', 'tx596786fa2e8145839e447-0055946f6b')]
ERROR:swiftclient:Account HEAD failed: http://10.109.64.3:8080/v1/AUTH_e46f34918675465e888114f30753b180 400 Header Line Too Long
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1243, in _retry
    rv = func(self.url, self.token, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 528, in head_account
    http_response_content=body)
ClientException: Account HEAD failed: http://10.109.64.3:8080/v1/AUTH_e46f34918675465e888114f30753b180 400 Header Line Too Long
Traceback (most recent call last):
  File "/usr/bin/swift", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/dist-packages/swiftclient/shell.py", line 1287, in main
    globals()['st_%s' % args[0]](parser, argv[1:], output)
  File "/usr/lib/python2.7/dist-packages/swiftclient/shell.py", line 492, in st_stat
    stat_result = swift.stat()
  File "/usr/lib/python2.7/dist-packages/swiftclient/service.py", line 443, in stat
    raise SwiftError('Account not found', exc=err)
swiftclient.service.SwiftError: 'Account not found'

This can be fixed by adding to swift.conf:

[swift-constraints]
max_header_size=<number>

And restarting services.

Changed in fuel:
milestone: none → 7.0
assignee: nobody → Fuel Library Team (fuel-library)
importance: Undecided → High
status: New → Triaged
tags: added: low-hanging-fruit
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Nikita Koshikov (nkoshikov)
status: Triaged → In Progress
Changed in fuel:
assignee: Nikita Koshikov (nkoshikov) → Matthew Mosesohn (raytrac3r)
Changed in puppet-swift:
assignee: nobody → Matthew Mosesohn (raytrac3r)
status: New → In Progress
Revision history for this message
Alex Schultz (alex-schultz) wrote :

puppet-swift was resolved as part of https://review.openstack.org/#/c/198057/

Changed in puppet-swift:
status: In Progress → Fix Committed
Revision history for this message
Alex Schultz (alex-schultz) wrote :

fuel-library fix is part of https://review.openstack.org/#/c/198894/

Changed in fuel:
assignee: Matthew Mosesohn (raytrac3r) → Alex Schultz (alex-schultz)
Mathieu Gagné (mgagne)
Changed in puppet-swift:
milestone: none → 6.0.0
Mathieu Gagné (mgagne)
Changed in puppet-swift:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/198894
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=251f3db523d9d659cdb8237603a1c5c4df207e9f
Submitter: Jenkins
Branch: master

commit 251f3db523d9d659cdb8237603a1c5c4df207e9f
Author: Mykyta Koshykov <email address hidden>
Date: Mon Jul 6 16:15:17 2015 -0500

    Increase swift max header size

    In order to support multiple regions, we need a larger max header
    size for swift. The default max size of 8192 is too small for
    multiregion PKI tokens. This change adds the ability to configure
    the max size and sets the limit to 32768.

    Related-Bug: 1470664
    Related blueprint: detach-components-from-controllers
    Co-Authored-By: Alex Schultz <email address hidden>
    Co-Authored-By: Matthew Mosesohn <email address hidden>

    Change-Id: I6a264a6463bfdfe7cc31d6a350cf71dd45c89f2e

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (master)

Change abandoned by Fuel DevOps Robot (<email address hidden>) on branch: master
Review: https://review.openstack.org/198032
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

tags: added: on-verification
Revision history for this message
Sergey Vilgelm (sergey.vilgelm) wrote :

Verified on ISO#292

Changed in fuel:
status: Fix Committed → Fix Released
tags: removed: on-verification
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.