Comment 3 for bug 1270651

Investigating

On Jan 19, 2014, at 3:14 PM, Rob Day <email address hidden> wrote:

> https://bugs.launchpad.net/libmemcached/+bug/1251482 seems to be fixing
> a similar issue, but I think my fix has the advantage that if a server
> is genuinely down (e.g. the attempt to reconnect over TCP fails),
> backoff logic will still kick in and protect it, rather than being
> bypassed entirely.
>
> --
> You received this bug notification because you are subscribed to
> libmemcached.
> https://bugs.launchpad.net/bugs/1270651
>
> Title:
> [patch included] libmemcached exhibits odd behaviour on TCP connection
> close
>
> Status in libmemcached - A C and C++ client library for memcached:
> New
>
> Bug description:
> I created a libmemcached client connection to a memcached server over
> TCP. This connection was long-lived and infrequently used. At some
> point, my memcached server was restarted, which caused it to close the
> libmemcached connection with a TCP FIN, which was ACKed.
>
> A short while later (when the memcached server was back up and
> running), I tried to send a request over the libmemcached client
> connection. libmemcached tried to use the previous connection that had
> been closed with a FIN, which resulted in a TCP RST from the other
> end. This failure caused libmemcached to set a backoff timeout on the
> connection, so that even though the TCP FIN had arrived several
> minutes ago, the memcached server was running again and the
> reconnection attempt succeeded, libmemcached wouldn't let me make any
> more requests on that connection for a short while.
>
> I also noticed that the behaviour varied depending on what type of
> request I was making - if I tried to store data in memcached, the
> error code was "MEMCACHED_CONNECTION_FAILED" (which was useful as I
> could simply notice that and retry the request) but when retrieving
> data from memcached the error code was "MEMCACHED_NOT_FOUND" which was
> much less useful.
>
> The attached patch (against the v1.0 trunk, though I was using libmemcached 1.0.10 when I saw this behaviour) fixes both these issues:
> * for TCP connections, an failure to send data will never set a backoff timeout on the server - only a failure to connect will do that. This protects against the case where a remote server drops the connection, but this is not noticed for a short while, during which the server recovers. The request to send data will still fail, but if the request is retried, libmemcached will successfully re-establish the connection and send the data rather than rejecting the request due to backoff logic. (The behaviour for UDP is unchanged.)
>
> * for get/gets requests, if there is no MEMCACHED_SUCCESS result and
> one or more servers returned a MEMCACHED_CONNECTION_FAILED error, the
> return code will be MEMCACHED_CONNECTION_FAILED rather than
> MEMCACHED_NOT_FOUND. This lets the client make informed decisions
> about whether to retry.
>
> I'm afraid my editor has stripped trailing spaces in a couple of
> places - hopefully that's not too confusing.
>
> I have permission from my employer to contribute this back under
> libmemcached's 3-clause BSD license.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/libmemcached/+bug/1270651/+subscriptions