Could not start Service[neutron-server]

Get the same problem, logs are attached

need a reproducer on the newer ISO.

Reproduced on ISO # 336:

{"build_id": "2014-06-02_18-24-56", "mirantis": "yes", "build_number": "336", "nailgun_sha": "a828d6b7610f872980d5a2113774f1cda6f6810b", "ostf_sha": "c959aa55f83fe2555cf2d382559271c7a9b17467", "fuelmain_sha": "8ed4269439efd09027e1e4e89d49e366604375ff", "astute_sha": "55df06b2e84fa5d71a1cc0e78dbccab5db29d968", "release": "4.1B", "fuellib_sha": "7a0bc0c733afdc02f30dc848579c8cce12b64ebd"}

Here is the part of puppet logs (node-1):

<27>Jun 2 19:05:30 node-1 puppet-apply[30524]: (/Stage[main]/Neutron::Server/Service[neutron-server]/ensure) change from stopped to running failed: Could not start Service[neutron-server]: Execution of '/sbin/service neutron-server start' returned 6: at /etc/puppet/modules/neutron/manifests/server.pp:93

and commands outputs:

[root@fuelweb ~]# ssh node-1 "/sbin/service neutron-server start; echo $?"

neutron-db-check: ERROR: neutron database does not have version information. You must run the "neutron-db-manage stamp" command before starting neutron services.
6

[root@fuelweb ~]# ssh node-1 "neutron-db-manage --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini stamp head"

No handlers could be found for logger "neutron.common.legacy"
INFO [alembic.migration] Context impl MySQLImpl.
INFO [alembic.migration] Will assume non-transactional DDL.

[root@fuelweb ~]# ssh node-1 "/sbin/service neutron-server start; echo $?"

Starting neutron: [ OK ]
0

neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade head

That command needs to be run on an empty neutron database prior to starting neutron-server. This should be done in puppet like in upstream. When the database is empty when upgrade runs it sets up the version correctly. stamp is only needed when trying to run upgrade with existing tables.

Marking as confirmed for Havana, but we need info from Fuel Hardening team, whether this bug affects current master and Icehouse branch.

Fix proposed to branch: stable/4.1
Review: https://review.openstack.org/97395

Fix proposed to branch: master
Review: https://review.openstack.org/97397

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/97398

This problem only exists in 4.1 starting with Havana 2013.2.3, and was never reproduced on Icehouse so far. Marking as Incomplete for 5.0 and 5.1.

Correction: we're likely to encounter this problem in 5.0.1 as well, a non-zero DB schema version number is required to reproduce.

Reviewed: https://review.openstack.org/97395
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=5d87d59db50a64a73c0846b97e55d3498075d5c4
Submitter: Jenkins
Branch: stable/4.1

commit 5d87d59db50a64a73c0846b97e55d3498075d5c4
Author: Vladimir Kuklin <email address hidden>
Date: Tue Jun 3 09:36:10 2014 +0400

    Run database migration before neutron-server starts

    Co-authored-by: Ryan Moe <email address hidden>
    Change-Id: Iafce36930c1c73e3e64e34d94ca1e52f850a1421
    Closes-bug: #1324859

verified {"build_id": "2014-06-04_09-16-08", "mirantis": "yes", "build_number": "341", "nailgun_sha": "a828d6b7610f872980d5a2113774f1cda6f6810b", "ostf_sha": "c959aa55f83fe2555cf2d382559271c7a9b17467", "fuelmain_sha": "7ed0f85acc0bab4b9157703a618b8cc9fd7de3e1", "astute_sha": "55df06b2e84fa5d71a1cc0e78dbccab5db29d968", "release": "4.1B", "fuellib_sha": "0e96fc5a340cd57f75c454ea8536471379299494"}

this bug should be confirmed for 2014.1.1, I guess. Marking as Confirmed for a while. Recheck needed.

it looks like, the fix is related not only to db migrations but also to default plugin ini config file.

Reviewed: https://review.openstack.org/97398
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=f94feb46c15e3e2f8af6d227803808902723bc68
Submitter: Jenkins
Branch: stable/5.0

commit f94feb46c15e3e2f8af6d227803808902723bc68
Author: Vladimir Kuklin <email address hidden>
Date: Tue Jun 3 09:36:10 2014 +0400

    Run database migration before neutron-server starts

    Change-Id: Iafce36930c1c73e3e64e34d94ca1e52f850a1421
    Closes-bug: #1324859

Reviewed: https://review.openstack.org/97397
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=c104235821c917395732c9dca7f1542102fcb6a6
Submitter: Jenkins
Branch: master

commit c104235821c917395732c9dca7f1542102fcb6a6
Author: Vladimir Kuklin <email address hidden>
Date: Tue Jun 3 09:36:10 2014 +0400

    Run database migration before neutron-server

    - Depends on merge of 2014.1.1 (or newer)
    - neutron with alembic migrations

    Closes-bug: #1324859
    Change-Id: Iafce36930c1c73e3e64e34d94ca1e52f850a1421

{
build_id: "2014-08-14_19-52-36",
mirantis: "yes",
build_number: "170",
ostf_sha: "09b6bccf7d476771ac859bb3c76c9ebec9da9e1f",
nailgun_sha: "af3d1922bfc21345f81be3454115ab6139675c35",
production: "docker",
api: "1.0",
fuelmain_sha: "fd58828f404e4298ed338e8f44c6a326cebd31de",
astute_sha: "6db5f5031b74e67b92fcac1f7998eaa296d68025",
release: "5.0.1",
fuellib_sha: "a31dbac8fff9cf6bc4cd0d23459670e34b27a9ab"
}