OpenStack Cinder Charm

apache2 does not start for cinder DFS api wsgi configuration

Bug #1671703 reported by Yuki Yamashita on 2017-03-10

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Cinder Charm	Triaged	Low	Unassigned

Bug Description

I tried cinder deploy use openstack-origin-git.
As a result, I found a problem that apache2 does not start.

My environment is following.
juju: 2.0.2-xenial-amd64
Deploy OS series: trusty
##################
uname -a
Linux juju-91e1d2-0-lxd-9 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
##################

Deploy OpenStack: Mitaka

Charm configuration:
##################
cat cinder_dev.yaml
series: trusty

services:
  cinder:
    charm: cs:trusty/cinder
    options:
      openstack-origin-git:
        "repositories: [ {name: requirements, repository: \'git://github.com/openstack/requirements\', branch: stable/mitaka}, {name: cinder, repository: \'git://github.com/openstack/cinder\', branch: stable/mitaka} ] \nrelease: stable/mitaka"
      glance-api-version: 2
##################

charm-cinder rev: 260

Juju status log:
##################
juju status
please enter password for admin on juju-ctrl1:
Model Controller Cloud/Region Version
y-sandbox1 juju-ctrl1 maas 2.0.2

App Version Status Scale Charm Store Rev OS Notes
cinder stable/mitaka blocked 1 cinder jujucharms 260 ubuntu
glance stable/mitaka active 1 glance jujucharms 256 ubuntu
keystone stable/mitaka active 1 keystone jujucharms 0 ubuntu
mysql 5.5.37-25.10... active 1 percona-cluster jujucharms 250 ubuntu
nova-cloud-controller 2016.1 active 1 nova-cloud-controller jujucharms 294 ubuntu
nova-compute stable/mitaka active 1 nova-compute jujucharms 264 ubuntu
rabbitmq-server 3.5.7 active 1 rabbitmq-server jujucharms 61 ubuntu

Unit Workload Agent Machine Public address Ports Message
cinder/2* blocked idle 0/lxd/9 10.0.11.73 8776/tcp Services not running that should be: apache2 <---- apache does not start!!!
glance/1* active idle 0/lxd/5 10.0.10.77 9292/tcp Unit is ready
keystone/0* active idle 0/lxd/4 10.0.10.20 5000/tcp Unit is ready
mysql/0* active idle 0/lxd/2 10.0.11.236 Unit is ready
nova-cloud-controller/0* active idle 0/lxd/1 10.0.11.170 3333/tcp,8773/tcp,8774/tcp Unit is ready
nova-compute/0* active idle 1 10.0.11.124 Unit is ready
rabbitmq-server/0* active executing 0/lxd/3 10.0.10.76 5672/tcp (update-status) Unit is ready

Machine State DNS Inst id Series AZ
0 started 10.0.10.75 aqgxqs trusty default
0/lxd/1 started 10.0.11.170 juju-91e1d2-0-lxd-1 trusty
0/lxd/2 started 10.0.11.236 juju-91e1d2-0-lxd-2 trusty
0/lxd/3 started 10.0.10.76 juju-91e1d2-0-lxd-3 trusty
0/lxd/4 started 10.0.10.20 juju-91e1d2-0-lxd-4 trusty
0/lxd/5 started 10.0.10.77 juju-91e1d2-0-lxd-5 trusty
0/lxd/9 started 10.0.11.73 juju-91e1d2-0-lxd-9 trusty
1 started 10.0.11.124 db4xnt trusty default

Relation Provides Consumes Type
cluster cinder cinder peer
identity-service cinder keystone regular
shared-db cinder mysql regular
amqp cinder rabbitmq-server regular
cluster glance glance peer
identity-service glance keystone regular
shared-db glance mysql regular
image-service glance nova-cloud-controller regular
image-service glance nova-compute regular
cluster keystone keystone peer
shared-db keystone mysql regular
identity-service keystone nova-cloud-controller regular
cluster mysql mysql peer
shared-db mysql nova-cloud-controller regular
cluster nova-cloud-controller nova-cloud-controller peer
cloud-compute nova-cloud-controller nova-compute regular
amqp nova-cloud-controller rabbitmq-server regular
compute-peer nova-compute nova-compute peer
amqp nova-compute rabbitmq-server regular
cluster rabbitmq-server rabbitmq-server peer
##################

/var/log/juju/unit-cinder-2.log:
##################
2017-03-10 04:34:39 INFO shared-db-relation-changed * Restarting web server apache2
2017-03-10 04:34:39 INFO shared-db-relation-changed ...fail!
2017-03-10 04:34:39 INFO shared-db-relation-changed * The apache2 configtest failed.
2017-03-10 04:34:39 INFO shared-db-relation-changed Output of config test was:
2017-03-10 04:34:39 INFO shared-db-relation-changed AH00526: Syntax error on line 7 of /etc/apache2/sites-enabled/wsgi-openstack-api.conf:
2017-03-10 04:34:39 INFO shared-db-relation-changed Invalid command 'WSGIDaemonProcess', perhaps misspelled or defined by a module not included in the server configuration
2017-03-10 04:34:39 INFO shared-db-relation-changed Action 'configtest' failed.
2017-03-10 04:34:39 INFO shared-db-relation-changed The Apache error log may have more information.
##################

I tried examining the problem to following procedure.

sudo service apache2 start
* Starting web server apache2 *
* The apache2 configtest failed.
Output of config test was:
AH00526: Syntax error on line 7 of /etc/apache2/sites-enabled/wsgi-openstack-api.conf:
Invalid command 'WSGIDaemonProcess', perhaps misspelled or defined by a module not included in the server configuration
Action 'configtest' failed.
The Apache error log may have more information.

I tried install libapache2-mod-wsgi.

sudo apt-get install libapache2-mod-wsgi
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
libfreetype6 os-prober
Use 'apt-get autoremove' to remove them.
The following NEW packages will be installed:
libapache2-mod-wsgi
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 67.4 kB of archives.
After this operation, 248 kB of additional disk space will be used.
Get:1 http://10.0.10.10/ubuntu/ trusty-updates/main libapache2-mod-wsgi amd64 3.4-4ubuntu2.1.14.04.2 [67.4 kB]
Fetched 67.4 kB in 0s (0 B/s)
Selecting previously unselected package libapache2-mod-wsgi.
(Reading database ... 35410 files and directories currently installed.)
Preparing to unpack .../libapache2-mod-wsgi_3.4-4ubuntu2.1.14.04.2_amd64.deb ...
Unpacking libapache2-mod-wsgi (3.4-4ubuntu2.1.14.04.2) ...
Setting up libapache2-mod-wsgi (3.4-4ubuntu2.1.14.04.2) ...
apache2_invoke: Enable module wsgi
* Restarting web server apache2

ubuntu@juju-91e1d2-0-lxd-9:~$ sudo service apache2 status
* apache2 is running

Apache2 started, but Juju still could not recognize apache2.
##################
cinder/2* blocked idle 0/lxd/9 10.0.11.73 8776/tcp Services not running that should be: apache2 <---- apache does not start!!!
##################

I noticed that wsgi is binding ##################
netstat -tna
Active Internet connections Proto Recv-Q Send-Q Local Address tcp 0 0 0.0.0.0:8776 tcp 0 0 0.0.0.0:22 tcp 0 0 127.0.0.1:8888 tcp 0 0 10.0.11.73:59307 tcp 0 0 10.0.11.73:38112 tcp 0 0 10.0.11.73:38118 tcp 0 0 10.0.11.73:59330 tcp 0 0 10.0.11.73:59331 tcp 0 0 10.0.11.73:38117 tcp 0 0 10.0.20.96:22 tcp 0 0 10.0.11.73:43729 tcp 0 0 10.0.11.73:44078 tcp 0 0 10.0.11.73:38111 tcp6 0 0 ::1:11211 tcp6 0 0 :::80 tcp6 0 0 :::22 tcp6 0 0 :::8766 ################## with ipv6.
(servers and established)
Foreign Address State
0.0.0.0:* LISTEN
0.0.0.0:* LISTEN
0.0.0.0:* LISTEN
10.0.10.76:5672 ESTABLISHED
10.0.10.38:17070 ESTABLISHED
10.0.10.38:17070 ESTABLISHED
10.0.10.76:5672 ESTABLISHED
10.0.10.76:5672 ESTABLISHED
10.0.10.38:17070 ESTABLISHED
10.0.20.1:52658 ESTABLISHED
10.0.11.236:3306 ESTABLISHED
10.0.11.236:3306 ESTABLISHED
10.0.10.38:17070 ESTABLISHED
:::* LISTEN
:::* LISTEN
:::* LISTEN
:::* LISTEN

I corrected it as follows and the problem was solved.

ubuntu@juju-91e1d2-0-lxd-9:~$ diff -u wsgi-openstack-api.conf.orig /etc/apache2/sites-enabled/wsgi-openstack-api.conf
--- wsgi-openstack-api.conf.orig 2017-03-10 04:34:20.375854449 +0000
+++ /etc/apache2/sites-enabled/wsgi-openstack-api.conf 2017-03-10 05:18:58.339894144 +0000
@@ -1,6 +1,6 @@
# Configuration file maintained by Juju. Local changes may be overwritten.

-Listen 8766
+Listen 0.0.0.0:8766
<VirtualHost *:8766>
WSGIDaemonProcess cinder processes=48 threads=1 user=cinder group=cinder \
python-path=/mnt/openstack-git/venv/lib/python2.7/site-packages \

I tried fix charm-cinder.
https://github.com/openstack/charm-cinder/compare/master...konono:fix_apache2_fail

Yuki Yamashita (kono) on 2017-03-10

Changed in charm-cinder:
assignee:	nobody → Yuki Yamashita (kono)

Revision history for this message

Frode Nordahl (fnordahl) wrote on 2017-03-16:

Thank you for your bug report.

1)
charm-cinder should not attempt to configure WSGI when not running on OpenStack Ocata or newer. Have a look at run_in_apache() function in hooks/cinder_utils.py

So there might be an issue with how this is detected when deploying from Git.

2)
When deploying from the binary packages from Ubuntu Cloud Archive for xenial-ocata the dependency is not missing.

3)
The note about apache binding IPv6-only is incorrect. netstat displays the LISTEN line in this way when a daemon is listening for both IPv4 and IPv6.

OpenStack contributions follow a separate workflow, have a look here: https://docs.openstack.org/infra/manual/developers.html

Revision history for this message

Corey Bryant (corey.bryant) wrote on 2017-03-16:

Hi Yuki,

Thanks for using the charms and the thorough bug report. Any changes need to be made through gerrit rather than pull requests. If you'd like to propose a change via openstack gerrit we could get some eyes on it and hopefully get your fix landed. https://docs.openstack.org/infra/manual/developers.html

Please do note though from the 17.02 charm release notes (https://docs.openstack.org/developer/charm-guide/1702.html): "Deploy from Source (DFS) support is under review for sustainability and may be removed during the Pike development cycle."

The code has not been getting the attention it needs.

Corey

summary:	- apache2 does not start for cinder api wsgi configuration + apache2 does not start for cinder DFS api wsgi configuration
Changed in charm-cinder:
status:	New → Triaged
importance:	Undecided → Low

Yuki Yamashita (kono) on 2017-03-23

Changed in charm-cinder:
assignee:	Yuki Yamashita (kono) → nobody

Revision history for this message

Yuki Yamashita (kono) wrote on 2017-03-23:

Hi Corey and Frode.
Thank you for your reply.

Understood that using gerrit to fix it.

> 3)
> The note about apache binding IPv6-only is incorrect. netstat displays the LISTEN line in this way when a daemon is listening for both IPv4 and IPv6.
Sorry, I misunderstood.

I talked with the Canonical support team.
Because it has "canonical" to take over the modifications, I unassigned me.

Thank you!!

Revision history for this message

Frode Nordahl (fnordahl) wrote on 2017-03-23:

os_release in charm-helpers return the wrong value when deploying from Git:
2017-03-23 08:48:01 INFO juju-log [cinder] DEBUG: os_release >= ocata: 'True' os_release = 'stable/mitaka'

For reference, when deploying from packages it looks like this:
2017-03-23 08:45:15 INFO juju-log [cinder] DEBUG: os_release >= ocata: 'False' os_release = 'mitaka'

Revision history for this message

Frode Nordahl (fnordahl) wrote on 2017-03-23:

Actually, if you update your cinder_dev.yaml to contain 'release: mitaka' and not 'release: stable/mitaka' this will work as it should. branch: should still be 'stable/mitaka'

I do agree that the documentation might not be explicit enough about how to use this.

Revision history for this message

Yuki Yamashita (kono) wrote on 2017-03-24:

Hi Frode,
Thank you your comment.

I carefully read and understood.
1080:def run_in_apache():
1081- """Return true if cinder API is run under apache2 with mod_wsgi in
1082- this release.
1083- """
1084- return os_release('cinder-common') >= 'ocata' <-----The release of OpenStack is in alphabetical order.It is wrongly determined in "stable/mitaka"

I tried deployment use following bundle.yaml,and so deployment succeeded!!!
#######
cat cinder.yaml
series: trusty
services:
  cinder:
    charm: ./trusty/charm-cinder
    options:
      openstack-origin-git: |
        repositories:
        - {name: requirements,
           repository: 'git://github.com/openstack/requirements',
           branch: 'master',
           depth: '5'}
        - {name: cinder,
           repository: 'git://github.com/openstack/cinder',
           branch: '8.1.1',
           depth: '5'}
        release: "mitaka"

I'm sorry, misunderstood it.
I should have known that wsgi support is from the ocata release.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.