Drizzledump import from MySQL crashing slave replication

Bug #929918 reported by Alex Reid
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Drizzle
New
Undecided
Patrick Crews

Bug Description

I'm importing data from a MySQL database and applying it to a new Drizzle database server using --destination-type=database.

This drizzle database server is part of a master-master pair, and as such is adding transactions to sys_replication.queue .. One of these very early on is apparently failing to insert into the queue and crashing the import.

(SQLSTATE 00000) You have an error in your SQL syntax; check the manual that corresponds to your Drizzle server version for the right syntax to use near 's iPad\"
      insert_value: \"1\"
      insert_value: \"0\"
      insert_value:' at line 1
Failure while executing:
INSERT INTO `sys_replication`.`queue` (`master_id`, `trx_id`, `seg_id`, `commit_order`, `originating_server_uuid`, `originating_commit_id`, `msg`) VALUES (1, 28694, 1, 24, 'd64ec01b-2755-42cd-8922-c04c28ca0536' , 24, 'transaction_context {
  server_id: 1
  transaction_id: 28694
  start_timestamp: 1328637584310604
  end_timestamp: 1328637584362243
}
...
It appears to get 12 records correctly, but crashes on the 13th. Here's the source row from MySQL:

'13', '10000015', 'e02c7b2dd2edeb1664770610c22bf06b1a524daa', 'ios', 'Seby''s iPhone 4', '1', '0', '2011-08-10 11:14:45', '2011-08-10 11:14:45'

Here's the part of the log output where things go sour.

record {
      insert_value: \"13\"
      insert_value: \"10000015\"
      insert_value: \"e02c7b2dd2edeb1664770610c22bf06b1a524daa\"
      insert_value: \"ios\"
      insert_value: \"Seby\\\\\\'s iPhone 4\"
      insert_value: \"1\"
      insert_value: \"0\"
      insert_valu <THERE IS BINARY GARBAGE HERE THAT I DELETED TO LET THE PASTE GET UPLOADED SUCCESSFULLY>
errmsg plugin 'Error_message_stderr' errmsg() failedReplication slave: Unable to insert into queue.

You can see all of the log for the box I'm doing the insert into at http://paste.drizzle.org/show/598/ .. minus the binary data that it started spewing after that point. (Which was keeping me from using ANY paste service..) I've attached the log including the binary data as well.

I'm on CentOS 5.6, using the RPM installation of Drizzle drizzle-rel.2012.01.30-rpm . I'm hosted on Amazon, my EC2 node type is m2.2xlarge.

I'm happy to provide any configuration details you'd like to see.

Revision history for this message
Alex Reid (0k53dmx9ci-alex) wrote :
Revision history for this message
Patrick Crews (patrick-crews) wrote :

Very interesting bug...would this drizzledump option perhaps be of use ? (just a shot in the dark)
If this doesn't help, if you could provide a limited dump of your MySQL data (the first 13 - 14 records) + your drizzled configs (+ slave configs), I can see about creating a test case for this. Also, is there anything in the logs of the other server of note?

Thanks for the report.

--my-data-is-mangled

    If your data is UTF8 but has been stored in a latin1 table using a latin1 connection then corruption is likely and drizzledump by default will retrieve mangled data. This is because MySQL will convert the data to UTF8 on the way out to drizzledump and you effectively get a double-conversion to UTF8.

    This typically happens with PHP apps that do not use SET NAMES.

    In these cases setting this option will retrieve the data as you see it in your application.

    New in version Drizzle7: 2011-01-31

Revision history for this message
Alex Reid (0k53dmx9ci-alex) wrote :

I don't think this is the case -- this is a UTF-8 table.

For the record, I'll call it out again -- I'm attempting to have two drizzle servers replicate from each other a la master-master with MySQL. The other thing I suspect is going on here (and may be playing into this issue) is that the slave plugin for drizzle may not have the equivalent of replicate-same-server-id=0 (and may be trying to apply its own updates to itself)

Server 1:

/etc/drizzle/drizzled.cnf
server-id=1
pid-file = /var/run/drizzle/drizzled.pid
datadir = /mnt/drizzle/data
auto-increment-increment = 2
auto-increment-offset = 1
drizzle-protocol.bind-address = 0.0.0.0

/etc/drizzle/slave.cnf
ignore-errors

[master2]
master-host = ec2-50-18-94-202.us-west-1.compute.amazonaws.com
master-port = 3306
master-user = root
master-pass = cheese

Server 2:

/etc/drizzle/drizzled.cnf
server-id=2
pid-file = /var/run/drizzle/drizzled.pid
datadir = /mnt/drizzle/data
auto-increment-increment = 2
auto-increment-offset = 2
drizzle-protocol.bind-address = 0.0.0.0

/etc/drizzle/slave.cnf
ignore-errors

[master1]
master-host = ec2-50-18-228-13.us-west-1.compute.amazonaws.com
master-port = 3306
master-user = root
master-pass = cheese

Schema:

delimiter $$

CREATE TABLE `device` (
  `device_id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
  `user_id` bigint(20) unsigned NOT NULL,
  `device_string` varchar(64) NOT NULL,
  `device_os` enum('ios','android') NOT NULL DEFAULT 'ios',
  `name` varchar(128) NOT NULL,
  `enabled` tinyint(1) NOT NULL DEFAULT '1',
  `is_deleted` tinyint(1) NOT NULL DEFAULT '0',
  `created_on` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
  `updated_on` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (`device_id`),
  KEY `user_id` (`user_id`),
  KEY `device_idx` (`device_os`,`device_string`)
) ENGINE=InnoDB AUTO_INCREMENT=522 DEFAULT CHARSET=utf8$$

Attached is the first 13 rows, including the offending one.

Changed in drizzle:
assignee: nobody → Patrick Crews (patrick-crews)
Revision history for this message
Patrick Crews (patrick-crews) wrote :

Can't blame a guy for hoping it was something simple ; )
My thinking was also leaning in your direction: master-master = dupe-applying updates. (see the aforementioned wishes for the easy fix <g>)
For some reason, I had thought we had addressed this issue, but I cannot find reference to it.
Will work on a test case for this tomorrow. Thanks for the great / quick information

Revision history for this message
Stewart Smith (stewart) wrote : Re: [Bug 929918] Re: Drizzledump import from MySQL crashing slave replication

On Fri, 10 Feb 2012 01:49:42 -0000, Patrick Crews <email address hidden> wrote:
> Can't blame a guy for hoping it was something simple ; )
> My thinking was also leaning in your direction: master-master = dupe-applying updates. (see the aforementioned wishes for the easy fix <g>)
> For some reason, I had thought we had addressed this issue, but I cannot find reference to it.
> Will work on a test case for this tomorrow. Thanks for the great /
> quick information

there should be per server uuid - but I can't remember which version
these came in...

--
Stewart Smith

Revision history for this message
Alex Reid (0k53dmx9ci-alex) wrote :

There are server uuids in this build. It doesn't seem to be solving this particular issue. :)

tags: added: slave-plugin
removed: slave
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.