Z39.50 Remote Search Should Log Connection Failures

Evergreen version: 3.5.3
OpenSRF Version: 3.2.2
PostgreSQL: 10
Servers: Ubuntu 18.04

We recently had an issue with network routing for our Evergreen bricks and they were unable to connect to remote sites, though for the most part, Evergreen was working just fine. During this period staff were unable to search OCLC (or any other site) using Z39.50 from the staff client.

The real problem was that the Z39.50 client code was unable to connect to the remote server. However, this was not obvious from the Evergreen logs. As far as we could tell, we were connecting, doing a search, and getting 0 results back:

2022-01-27 11:54:48 bd2-bh1 open-ils.search: [INFO:1625:Z3950.pm:377:1643302488111020] z3950: connecting to server zcat.oclc.org:210:OLUCWorldCat as
2022-01-27 11:54:48 bd2-bh1 open-ils.search: [INFO:1625:Z3950.pm:398:1643302488111020] z3950: query => @attr 1=12 @attr 4=1 @attr 5=0 "1285712761"
2022-01-27 11:54:48 bd2-bh1 open-ils.search: [INFO:1625:Z3950.pm:410:1643302488111020] z3950: search [@attr 1=12 @attr 4=1 @attr 5=0 "1285712761" ] took 0 seconds
2022-01-27 11:55:08 bd2-bh1 open-ils.search: [INFO:1625:Z3950.pm:439:1643302488111020] z3950: using record format 'F1' and transmission format 'usmarc'
2022-01-27 11:55:08 bd2-bh1 open-ils.search: [INFO:1625:Z3950.pm:445:1643302488111020] z3950: 'oclc' search returned 0 hits

It was not until sometime later when Stripe was logging failures to connect to the API gateway that we realized the actual problem.

It would be better for Z39.50 to log failures to connect to the remote server. Our issues could have been resolved much sooner.