cinder-backup service randomly dies in ceph jobs
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Cinder |
Confirmed
|
Undecided
|
Unassigned |
Bug Description
As seen here:
2016-11-17 12:51:38.859 ERROR cinder.service [req-68988b27-
2016-11-17 12:51:38.859 3181 ERROR cinder.service Traceback (most recent call last):
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/opt/stack/
2016-11-17 12:51:38.859 3181 ERROR cinder.service Service.service_id)
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/opt/stack/
2016-11-17 12:51:38.859 3181 ERROR cinder.service orm_obj = db.get_
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/opt/stack/
2016-11-17 12:51:38.859 3181 ERROR cinder.service return IMPL.get_
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/opt/stack/
2016-11-17 12:51:38.859 3181 ERROR cinder.service return f(*args, **kwargs)
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/opt/stack/
2016-11-17 12:51:38.859 3181 ERROR cinder.service return _GET_METHODS[
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/opt/stack/
2016-11-17 12:51:38.859 3181 ERROR cinder.service return f(*args, **kwargs)
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/opt/stack/
2016-11-17 12:51:38.859 3181 ERROR cinder.service service = None if not query else query.first()
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/usr/local/
2016-11-17 12:51:38.859 3181 ERROR cinder.service ret = list(self[0:1])
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/usr/local/
2016-11-17 12:51:38.859 3181 ERROR cinder.service return list(res)
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/usr/local/
2016-11-17 12:51:38.859 3181 ERROR cinder.service return self._execute_
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/usr/local/
2016-11-17 12:51:38.859 3181 ERROR cinder.service close_with_
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/usr/local/
2016-11-17 12:51:38.859 3181 ERROR cinder.service **kw)
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/usr/local/
2016-11-17 12:51:38.859 3181 ERROR cinder.service execution_
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/usr/local/
2016-11-17 12:51:38.859 3181 ERROR cinder.service conn = engine.
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/usr/local/
2016-11-17 12:51:38.859 3181 ERROR cinder.service self._wrap_
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/usr/local/
2016-11-17 12:51:38.859 3181 ERROR cinder.service e, dialect, self)
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/usr/local/
2016-11-17 12:51:38.859 3181 ERROR cinder.service util.raise_
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/usr/local/
2016-11-17 12:51:38.859 3181 ERROR cinder.service reraise(
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/usr/local/
2016-11-17 12:51:38.859 3181 ERROR cinder.service return fn()
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/usr/local/
2016-11-17 12:51:38.859 3181 ERROR cinder.service return _ConnectionFair
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/usr/local/
2016-11-17 12:51:38.859 3181 ERROR cinder.service fairy = _ConnectionReco
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/usr/local/
2016-11-17 12:51:38.859 3181 ERROR cinder.service rec = pool._do_get()
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/usr/local/
2016-11-17 12:51:38.859 3181 ERROR cinder.service self._dec_
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/usr/local/
2016-11-17 12:51:38.859 3181 ERROR cinder.service compat.
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/usr/local/
2016-11-17 12:51:38.859 3181 ERROR cinder.service return self._create_
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/usr/local/
2016-11-17 12:51:38.859 3181 ERROR cinder.service return _ConnectionReco
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/usr/local/
2016-11-17 12:51:38.859 3181 ERROR cinder.service self.connection = self.__connect()
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/usr/local/
2016-11-17 12:51:38.859 3181 ERROR cinder.service connection = self.__
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/usr/local/
2016-11-17 12:51:38.859 3181 ERROR cinder.service return dialect.
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/usr/local/
2016-11-17 12:51:38.859 3181 ERROR cinder.service return self.dbapi.
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/usr/local/
2016-11-17 12:51:38.859 3181 ERROR cinder.service return Connection(*args, **kwargs)
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/usr/local/
2016-11-17 12:51:38.859 3181 ERROR cinder.service self.connect()
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/usr/local/
2016-11-17 12:51:38.859 3181 ERROR cinder.service self._request_
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/usr/local/
2016-11-17 12:51:38.859 3181 ERROR cinder.service auth_packet = self._read_packet()
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/usr/local/
2016-11-17 12:51:38.859 3181 ERROR cinder.service packet_header = self._read_bytes(4)
2016-11-17 12:51:38.859 3181 ERROR cinder.service File "/usr/local/
2016-11-17 12:51:38.859 3181 ERROR cinder.service 2013, "Lost connection to MySQL server during query")
2016-11-17 12:51:38.859 3181 ERROR cinder.service DBConnectionError: (pymysql.
2016-11-17 12:51:38.859 3181 ERROR cinder.service
50 hits in 7 days, check and gate, all ceph jobs, but not all failures, so it must be a random timing issue when it does and if tempest is running a volume backup test at the time that it dies only then do we have a test failure.
req-68988b27- 1b9a-45b3- b7d1-ce0ee22eae 44 is the request ID from the backup create POST, and that's what failed above, so that's definitely the issue for this failure.
2016-11-17 12:51:19,219 8649 INFO [tempest. lib.common. rest_client] Request (VolumesBackups AdminV1Test: test_volume_ backup_ export_ import) : 201 POST http:// 127.0.0. 1:8776/ v1/92a8db454386 4610b83b54a9964 5f36b/backups/ import_ record 0.037s lib.common. rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'} backup. drivers. ceph", "backup_url": "eyJvYmplY3RfY2 91bnQiOiAwLCAic 25hcHNob3RfaWQi OiBudWxsLCAicGF yZW50X2lkIjogbn VsbCwgInVzZXJfa WQiOiAiM2JlMGIz ZmE1NjI4NDkyZTh lZjNhYzRhMjdiNT RmNjYiLCAicHJva mVjdF9pZCI6ICI5 MmE4ZGI0NTQzODY 0NjEwYjgzYjU0YT k5NjQ1ZjM2YiIsI CJjb250YWluZXIi OiAiYmFja3VwcyI sICJ0ZW1wX3ZvbH VtZV9pZCI6IG51b GwsICJkaXNwbGF5 X25hbWUiOiAidGV tcGVzdC1Wb2x1bW VzQmFja3Vwc0Fkb WluVjFUZXN0LUJh Y2t1cC0xMzc3NDg 1OTQiLCAic2Vydm ljZSI6ICJjaW5kZ XIuYmFja3VwLmRy aXZlcnMuY2VwaCI sICJjcmVhdGVkX2 F0IjogIjIwMTYtM TEtMTdUMTI6NTA6 NDdaIiwgInVwZGF 0ZWRfYXQiOiAiMj AxNi0xMS0xN1QxM jo1MToxOFoiLCAi ZGlzcGxheV9kZXN jcmlwdGlvbiI6IG 51bGwsICJkYXRhX 3RpbWVzdGFtcCI6 ICIyMDE2LTExLTE 3VDEyOjUwOjQ3Wi IsICJzZXJ2aWNlX 21ldGFkYXRhIjog bnVsbCwgImF2YWl sYWJpbGl0eV96b2 5lIjogIm5vdmEiL CAiaWQiOiAiYzYz NDZiYzQtMDhkYy0 0MmY2LWI3OTgtM2 FmNmFkNzU5OTUxI iwgInJlc3RvcmVf dm9sdW1lX2lkIjo gbnVsbCwgInNpem UiOiAxLCAidGVtc F9zbmFwc2hvdF9p ZCI6IG51bGwsICJ zdGF0dXMiOiAiYX ZhaWxhYmxlIiwgI nZvbHVtZV9pZCI6 ICJiYzU2OGJjYy0 2OTZkLTRkNzUtYj EzMy0zNTZhZTM2N DJjZTIiLCAiZGVs ZXRlZF9hdCI6IG5 1bGwsICJmYWlsX3 JlYXNvbiI6IG51b GwsICJudW1fZGVw ZW5kZW50X2JhY2t 1cHMiOiAwLCAiaG 9zdCI6ICJ1YnVud HUteGVuaWFsLWlu ZnJhY2xvdWQtY2h vY29sYXRlLTU0Mj QzMjIiLCAiZHJpd mVyX2luZm8iOiB7 fSwgImRlbGV0ZWQ iOiBmYWxzZX0= "}} request- id': 'req-68988b27- 1b9a-45b3- b7d1-ce0ee22eae 44', 'x-openstack- request- id': 'req-68988b27- 1b9a-45b3- b7d1-ce0ee22eae 44', 'status': '201', 'connection': 'close', 'date': 'Thu, 17 Nov 2016 12:51:19 GMT', 'content-length': '346', 'content-type': 'application/json', 'content-location': 'http:// 127.0.0. 1:8776/ v1/92a8db454386 4610b83b54a9964 5f36b/backups/ import_ record'} 08dc-42f6- b798-3af6ad7599 51", "links": [{"href": "http:// 127.0.0. 1:8776/ v1/92a8db454386 4610b83b54a9964 5f36b/backups/ c6346bc4- 08dc-42f6- b798-3af6ad7599 51", "rel": "self"}, {"href": "http:// 127.0.0. 1:8776/ 92a8db454386461 0b83b54a99645f3 6b/backups/ c6346bc4- 08dc-42f6- b798-3af6ad7599 51", "rel": "bookmark"}], "name": null}}
2016-11-17 13:20:28.892457 | 2016-11-17 12:51:19,219 8649 DEBUG [tempest.
2016-11-17 13:20:28.892863 | Body: {"backup-record": {"backup_service": "cinder.
2016-11-17 13:20:28.893282 | Response - Headers: {'x-compute-
2016-11-17 13:20:28.893518 | Body: {"backup": {"id": "c6346bc4-