cannot create a service with kuryr network in swarm mode

Bug #1705967 reported by Hyunsun Moon
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kuryr-libnetwork
Confirmed
Undecided
Liping Mao

Bug Description

Steps to reproduce.

1. install docker and set up swarm mode
2. create kuryr-libnetwork plugin with global scope
[DEFAULT]
bindir = /usr/libexec/kuryr
capability_scope = global
log_dir = /var/log/kuryr
log_file = kuryr.log
log_level = DEBUG

[neutron]
project_name = admin
username = admin
password = nova
auth_url = http://10.1.1.161:35357/v2.0
auth_type = password

3. create network with kuryr driver
sudo docker network create --driver=kuryr/libnetwork2:latest --ipam-driver=kuryr/libnetwork2:latest --subnet=10.10.10.0/24 --gateway=10.10.10.1 kuryr-ne

4. create a service with the network
sudo docker service create --name redis --network kuryr-net redis:3.0.6

5. fails with the following Docker errors

[Docker daemon error]
DEBU[1559] Calling POST /v1.30/services/create
DEBU[1559] form data: {"EndpointSpec":{"Mode":"vip"},"Labels":{},"Mode":{"Replicated":{}},"Name":"redis-kuryr","TaskTemplate":{"ContainerSpec":{"DNSConfig":{},"Image":"redis:3.0.6@sha256:6a692a76c2081888b589e26e6ec835743119fe453d67ecf03df7de5b73d69842"},"ForceUpdate":0,"Networks":[{"Target":"kuryr-net"}],"Placement":{"Platforms":[{"Architecture":"amd64","OS":"linux"}]},"Resources":{"Limits":{},"Reservations":{}}}}
DEBU[1559] Service mz49ao8vlvki3ywkq4qkivmcq was scaled up from 0 to 1 instances module=node node.id=i58gldd9jzr8evqc0emvzo8p4
INFO[1560] Traceback (most recent call last): plugin=0a1a95e68639cf89ea6790a5f445769fe4061ffb74cb75f4152a5a87a520cdf8
INFO[1560] File "/usr/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request plugin=0a1a95e68639cf89ea6790a5f445769fe4061ffb74cb75f4152a5a87a520cdf8
INFO[1560] rv = self.dispatch_request() plugin=0a1a95e68639cf89ea6790a5f445769fe4061ffb74cb75f4152a5a87a520cdf8
INFO[1560] File "/usr/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request plugin=0a1a95e68639cf89ea6790a5f445769fe4061ffb74cb75f4152a5a87a520cdf8
INFO[1560] return self.view_functions[rule.endpoint](**req.view_args) plugin=0a1a95e68639cf89ea6790a5f445769fe4061ffb74cb75f4152a5a87a520cdf8
INFO[1560] File "/usr/lib/python2.7/site-packages/kuryr_libnetwork/controllers.py", line 1562, in ipam_request_address plugin=0a1a95e68639cf89ea6790a5f445769fe4061ffb74cb75f4152a5a87a520cdf8
INFO[1560] ("Subnet with pool {0} does not exist.").format(pool_id)) plugin=0a1a95e68639cf89ea6790a5f445769fe4061ffb74cb75f4152a5a87a520cdf8
INFO[1560] KuryrException: Subnet with pool a43c906d-17bc-4be5-a976-55ca361ce165 does not exist. plugin=0a1a95e68639cf89ea6790a5f445769fe4061ffb74cb75f4152a5a87a520cdf8
INFO[1560] [pid: 12|app: 0|req: 7/11] 0.0.0.0 () {26 vars in 359 bytes} [Mon Jul 24 01:44:29 2017] POST /IpamDriver.RequestAddress => generated 85 bytes in 341 msecs (HTTP/1.1 500) 2 headers in 127 bytes (3 switches on core 0) plugin=0a1a95e68639cf89ea6790a5f445769fe4061ffb74cb75f4152a5a87a520cdf8
ERRO[1560] Failed allocation for service mz49ao8vlvki3ywkq4qkivmcq error="could not allocate VIP from IPAM: IpamDriver.RequestAddress: Subnet with pool a43c906d-17bc-4be5-a976-55ca361ce165 does not exist." module=node node.id=i58gldd9jzr8evqc0emvzo8p4

[Kuryr error]
2017-07-24 01:44:29.456 12 INFO kuryr_libnetwork.controllers [-] Received JSON data {u'Address': u'', u'Options': None, u'PoolID': u'a43c906d-17bc-4be5-a976-55ca361ce165'} for /IpamDriver.RequestAddress
2017-07-24 01:44:29.631 12 ERROR kuryr_libnetwork.utils [-] Unexpected error happened: Subnet with pool a43c906d-17bc-4be5-a976-55ca361ce165 does not exist.: KuryrException: Subnet with pool a43c906d-17bc-4be5-a976-55ca361ce165 does not exist.

Liping Mao (limao)
Changed in kuryr-libnetwork:
assignee: nobody → Liping Mao (limao)
status: New → Confirmed
Revision history for this message
Liping Mao (limao) wrote :

Hi Hyunsun Moon,

Sorry for the delay.
I use the latest version(Your two patches has merged, thanks for helping on improve kuryr-libnetwork) to test this. I can reproduce this problem in my local, I can see same error.

And see REPLICAS is 0/1:
root@devstack:/opt/stack/kuryr-libnetwork# docker service list
ID NAME MODE REPLICAS IMAGE PORTS
0dgd74ckz2jl redis replicated 0/1 redis:3.0.6

This may related with the comments in following issue:
https://github.com/moby/moby/issues/32604

Still need to do further debug right now.

Regards,
Liping Mao

Revision history for this message
Liping Mao (limao) wrote :

Hi Hyunsun Moon,

I find one thing strange for me:

When docker libnetwork call kuryr-libnetwork IpamDriver.RequestAddress, you can see the debug log, the Address is null, Options are None:
2017-08-07 07:11:09.630 13 DEBUG kuryr_libnetwork.controllers [-] Received JSON data {u'Address': u'', u'Options': None, u'PoolID': u'644aca38-1334-4e47-964d-7d38f1ef1263'} for /IpamDriver.RequestAddress ipam_request_address /usr/lib/python2.7/site-packages/kuryr_libnetwork/controllers.py:1513

This is unexpected in kuryr-libnetwork. Let me double check why we have this kind of call from libnetwork.

Regards,
Liping Mao

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.