zaqar

perf: _next_marker() is creating a bottleneck

Bug #1218602 reported by Amit Gandhi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
zaqar
Fix Released
High
Kurt Griffiths
zaqar havana-3 "h3"

Bug Description

the _next_marker() function in https://github.com/stackforge/marconi/blob/master/marconi/storage/mongodb/messages.py#L118 calls find_one() in mongodb.

The performance of this call on average currently takes about 6ms and is about 35% of the total request time.

new relic graph: http://e22b252f78f59e4cc52d-ba5a27421dda25762a9ad645827a0fe0.r68.cf1.rackcdn.com/Screen%20Shot%202013-08-29%20at%204.32.17%20PM.png

Optimizing this call should make a considerable improvement to performance.

Revision history for this message
Kurt Griffiths (kgriffs) wrote :

https://review.openstack.org/#/c/44560/

Changed in marconi:
status: New → Confirmed
status: Confirmed → In Progress
assignee: nobody → Kurt Griffiths (kgriffs)
importance: Undecided → Medium
milestone: none → havana-3
Revision history for this message
Kurt Griffiths (kgriffs) wrote :

Superficial patch above did not help. Moving to a side counter may be our best bet:

https://etherpad.openstack.org/queuing-marker

Changed in marconi:
importance: Medium → High
Kurt Griffiths (kgriffs)
summary: - Performance of find_one() to get next marker should be optimized
+ perf: _next_marker() is creating a bottleneck
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to marconi (master)

Fix proposed to branch: master
Review: https://review.openstack.org/47242

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/47272

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to marconi (master)

Reviewed: https://review.openstack.org/47272
Committed: http://github.com/openstack/marconi/commit/1512db43ec29e9afa6d05e292cdc7cfce5da2029
Submitter: Jenkins
Branch: master

commit 1512db43ec29e9afa6d05e292cdc7cfce5da2029
Author: kgriffs <email address hidden>
Date: Mon Sep 16 18:19:14 2013 -0500

    fix(mongodb): Marker generation creates a bottleneck under heavy load

    This patch changes markers so that they are generated using a per-queue
    side counter. A heuristic is used to mitigate a race condition. Due to the
    new semantics, partial inserts are no longer possible due to collisions,
    which ended up simplifying the retry logic for posting messages.

    As a consequence of this patch, the last message posted no longer needs
    to remain in the queue indefinitely, rendering marconi-gc unnecessary,
    and so it has been removed.

    Also, since the mongod GC worker runs once a minute, the queries no longer
    filter out expired-but-not-yet-gc'd messages; on average, a message may
    live more than 30 seconds passed it's expected lifetime, but I do not
    think that this will harm or complicate any application building on top of
    Marconi, practically speaking. That being said, it is worth calling out
    in documentation.

    Closes-Bug: #1218602
    Change-Id: I34e24e7dd7c4e017c84eb5929ce37ad4c9e5266a

Changed in marconi:
status: In Progress → Fix Committed
Kurt Griffiths (kgriffs)
Changed in marconi:
assignee: Kurt Griffiths (kgriffs) → nobody
assignee: nobody → Kurt Griffiths (kgriffs)
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.