Comment 8 for bug 1905965

Hello,

Today I submitted changes related to John's proposition (c.f PS3 https://review.opendev.org/c/openstack/oslo.messaging/+/764776/3 ).

These changes are still "WIP" to allow us to discuss about the related details of the implementation.

The first that we could discuss about is can we drop the `direct_mandatory_flag` in favor of `at_least_once`.

I think it doesn't really make sense to keep the both params flags in parallel.

Either we want to re-raise the `MessageUndelivered` exception or to loop over a timeout, but I don't think we want to stop to use the mandatory flag option, isn't? For this reason I think we can simply define the `at_least_once` option as did in PS3 and simply abandon the `direct_mandatory_flag` option.

Also, notice that I tried to simulate the `MessageUndelivered` scenario by using oslo.messaging's simulator, unfortunatelly I couldn't reproduce the MessageUndeliverable exception.

Find bellow the followed process, maybe you will have an idea oon how to simulate that.

1. Run a rabbitmq server and allow to access to the dashboard to monitor things:

```
$ sudo podman run -d -e RABBITMQ_NODENAME=rabbitmq \
     -p 5672:5672 \
     -p 15672:15672 \
     --name rabbitmq rabbitmq:3-management
```

2. Prepare some configuration to use with our simulator:

```
# tuto.conf
[DEFAULT]
transport_url = rabbit://localhost/
at_least_once = False
```

3. Run the RPC server

```
$ tox -e venv -- python tools/simulator.py --config-file ./tuto.conf -d rpc-server -w 40
```

Notice that I asked to the server to wait before answer to allow me to manually delete the reply queue on the rabbitmq server side.

4. Run the RPC client

```
$ tox -e venv -- python tools/simulator.py --config-file ./tuto.conf -d rpc-client
```

5. Delete the corresponding reply queue

```
$ sudo podman exec -it rabbitmq rabbitmqctl delete_queue reply_<id>
```

Unfortunatelly I can't fall into the expected situation by following this process...

I suppose that some other mechanisms catch that something went wrong with this scenario and it could explain why I didn't reach my expected scenario.

Please let me know if these changes fit our expectations and if they make sense to you or if they needs changes.