hockeypuck

Support PostgreSQL bulk loading techniques

Bug #1208295 reported by Casey Marshall
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
hockeypuck
Fix Committed
Medium
Casey Marshall
hockeypuck 1.0 "bitquantum of solace"

Bug Description

Loading an SKS dump into Hockeypuck+PostgreSQL has some problems -- insert performance takes a dive -- from 60k keys/hr down to 1-4k/hr as the size grew from 0-500k keys.

The OpenPGP schema is full of foreign key references, other constraints and indexes. Could support disabling/enabling these, perhaps using a special loader utility. Or, direct offline loading with COPY.

http://www.postgresql.org/docs/current/interactive/populate.html

Casey Marshall (cmars)
Changed in hockeypuck:
milestone: none → 1.0
Revision history for this message
Casey Marshall (cmars) wrote :

Creating a utility that inserts directly from SKS dumps with dropped constraints to build an initial database.

Changed in hockeypuck:
status: Triaged → In Progress
Revision history for this message
Casey Marshall (cmars) wrote :

Need to add support for rebuilding the ptree database as well.

Revision history for this message
Casey Marshall (cmars) wrote :

hockeypuck-import now inserts in large transactions with dropped constraints, then de-duplicates and adds constraints.

Changed in hockeypuck:
status: In Progress → Fix Committed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.