dhcp-agent does always provide IP address for instances with re-cycled IP addresses.

The message is received by the agent:

2013-06-11 09:28:30 DEBUG [quantum.openstack.common.rpc.amqp] received {u'_context_roles': [u'Member', u'_member_', u'admin'], u'_context_read_deleted': u'no', u'_context_tenant_id': u'670fedaf933a43d2b2a64148d781839f', u'args': {u'payload': {u'port': {u'status': u'DOWN', u'name': u'', u'admin_state_up': True, u'network_id': u'864facd7-0179-40fe-8928-48c3993b8c00', u'tenant_id': u'670fedaf933a43d2b2a64148d781839f', u'device_owner': u'compute:None', u'mac_address': u'fa:16:3e:da:41:45', u'fixed_ips': [{u'subnet_id': u'2128dbee-d153-47a1-86e4-5e2611233d58', u'ip_address': u'10.5.0.2'}], u'id': u'76672204-43b8-4407-b8f5-bcb847cde39a', u'security_groups': [u'b0dbbcf1-0512-429f-b40b-c24b0f6e08c7', u'b98efe39-08ac-45d3-925d-83c9f1307456'], u'device_id': u'a1e1d055-68c7-4c76-a0c1-47b1e116a8a7'}}}, u'_unique_id': u'e989617aad374a5e94284c174394377f', u'_context_is_admin': False, u'version': u'1.0', u'_context_project_id': u'670fedaf933a43d2b2a64148d781839f', u'_context_timestamp': u'2013-06-11 13:28:30.197746', u'_context_user_id': u'9b118873367f427e950790a0b1504881', u'method': u'port_create_end'}
2013-06-11 09:28:30 DEBUG [quantum.openstack.common.rpc.amqp] unpacked context: {'user_id': u'9b118873367f427e950790a0b1504881', 'roles': [u'Member', u'_member_', u'admin'], 'tenant_id': u'670fedaf933a43d2b2a64148d781839f', 'is_admin': False, 'timestamp': u'2013-06-11 13:28:30.197746', 'project_id': u'670fedaf933a43d2b2a64148d781839f', 'read_deleted': u'no'}
2013-06-11 09:28:30 DEBUG [quantum.openstack.common.lockutils] Got semaphore "agent" for method "port_update_end"...
2013-06-11 09:28:30 DEBUG [quantum.agent.linux.utils] Running command: ['sudo', '/usr/bin/quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'netns', 'exec', 'qdhcp-864facd7-0179-40fe-8928-48c3993b8c00', 'kill', '-HUP', '10083']
2013-06-11 09:28:30 DEBUG [quantum.agent.linux.utils]
Command: ['sudo', '/usr/bin/quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'netns', 'exec', 'qdhcp-864facd7-0179-40fe-8928-48c3993b8c00', 'kill', '-HUP', '10083']
Exit code: 0

And a nice message from dnsmasq:

Jun 11 09:28:30 d7m88-cofgod dnsmasq[10083]: cleared cache
Jun 11 09:28:30 d7m88-cofgod dnsmasq[10083]: duplicate dhcp-host IP address 10.5.0.2 at line 7 of /var/lib/quantum/dhcp/864facd7-0179-40fe-8928-48c3993b8c00/host
Jun 11 09:28:30 d7m88-cofgod dnsmasq-dhcp[10083]: read /var/lib/quantum/dhcp/864facd7-0179-40fe-8928-48c3993b8c00/host
Jun 11 09:28:30 d7m88-cofgod dnsmasq-dhcp[10083]: read /var/lib/quantum/dhcp/864facd7-0179-40fe-8928-48c3993b8c00/opts

I guess the previously allocated IP address is already present so the new one gets ignored.

Similar issues being dicussed on openstack ML; might be a dnsmasq issue:

http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2013q2/007212.html

Raising task for dnsmasq as well.

Status changed to 'Confirmed' because the bug affects multiple users.

According to message "duplicate dhcp-host IP address 10.5.0.2", there should be multiple entries of the same IP 10.5.0.2 in file /var/lib/quantum/dhcp/864facd7-0179-40fe-8928-48c3993b8c00/host, while MAC addresses of the entries may be different.

It seems that in the case dnsmasq server will report error and doesn't allocate IP address for the DHCP request.

In my environment, the problem occurred when I tried to start 20 instances in the same time with the following command.

nova boot --image IMAGE --num-instances 20 test

Some of the instances couldn't get dhcp IP address.

I found the following warning message in /var/log/quantum/server.log.

2013-07-04 18:25:41 WARNING [quantum.scheduler.dhcp_agent_scheduler] No active DHCP agents

And after checking the related code in file ocpcc1:/usr/lib/python2.6/site-packages/quantum/scheduler/dhcp_agent_scheduler.py

71 def is_agent_down(cls, heart_beat_time):
72 LOG.debug("now: %s, heart_beat_time: %s, agent_down_time: %d" % (str(timeutils.utcnow()), heart_beat_time, cfg.CONF.agent_down_time))
73 return timeutils.is_older_than(heart_beat_time,
74 cfg.CONF.agent_down_time)

The line 72 is what I inserted to debug.

After checking the output of line 72, I found the agent_down_time is too short. The default value is 5 seconds.

I added the agent_down_time to /etc/quantum/quantum.conf, and set it to 15 seconds, then all things worked fine.

Hope what I did can help you.

met the same issue !

The same problems， who can reproduce the problem！

Fix proposed to branch: master
Review: https://review.openstack.org/40360

Fix proposed to branch: master
Review: https://review.openstack.org/40361

i think neutron did not delete the port asap，maybe we can set dnsmasq lease time 3600 and wait for neutron deleting the port！

The problem is the lease database is not updated properly. This is being addressed by: https://review.openstack.org/#/c/37580/

FWIW I still see this (or at least an issue with the same symptoms and effects) in Havana RC2.

The dnsmasq host file contains what I think is an old mac->IP mapping; the IP has been recycle for a new instance which appears lower down in the file but dnsmasq discards this record.

The mac address for the old instance is not present anywhere in the deployment (i checked ports across all tenants).

Oct 16 13:07:22 cofgod dnsmasq[14049]: cleared cache
Oct 16 13:07:22 cofgod dnsmasq-dhcp[14049]: read /var/lib/neutron/dhcp/bab112ff-46c3-4034-8d27-936d7bb1ecd5/host
Oct 16 13:07:22 cofgod dnsmasq-dhcp[14049]: read /var/lib/neutron/dhcp/bab112ff-46c3-4034-8d27-936d7bb1ecd5/opts
Oct 16 13:07:22 cofgod dnsmasq-dhcp[14049]: duplicate IP address 10.5.0.45 in .

...

Oct 16 13:13:38 cofgod dnsmasq-dhcp[14049]: DHCPDISCOVER(ns-169187f7-9c) fa:16:3e:f5:a3:2d no address available

Possibly a new vector, I have IPv6 as well as IPv4 on a network. I see this segment of the (rather large) host file:

fa:16:3e:9d:de:21,host-192-168-2-103.openstacklocal,192.168.2.103
fa:16:3e:9d:de:21,host-2607-f088-0-2--1339.openstacklocal,2607:f088:0:2::1339

When I remove the second line, and kill -HUP the daemon, IPv4 starts working. I tried running kill -HUP without removing the line and did not get a response.

Hopefully, whatever the problem is will be considered and fixed for both IPv4 and IPv6.

I am running dnsmasq from ubuntu packages: 2.66-4ubuntu1

It seems this has been fixed with https://review.openstack.org/#/c/37580/

quantum is no longer found in any supported Ubuntu release