Timeout server should be set in frontend and backend

Scenario: Indico application has a few long-running operations that require more time to finish properly so we need to change the timeout.

Environment:
Content Cache stable/167

How to reproduce (assuming that Indico is properly set, up and running):

1) Set the sites
juju config content-cache sites="indico.local:
  locations:
    /:
      backend-check-method: GET
      backend-check-path: /health
      backend-inter-time: 10s
      backend-maxconn: 50
      backends: ['1.1.1.1:80']
      backend-options: ['http-request set-header X-Forwarded-For %[src]','timeout server 300s']
      extra-config:
        - client_max_body_size 15m
        - proxy_cache_bypass \$cookie_indico_session
        - proxy_read_timeout 300s
"

Command here: https://pastebin.canonical.com/p/B5Qvsx7mh6/

2) Trigger a long-running operation (import of registrants)
curl 'http://indico.local/event/4/manage/registration/7/registrations/import' -H "Authorization: Bearer ..." -X 'POST' -H 'accept: application/json' -H 'cache-control: no-cache' -F source_file=@/tmp/test.csv

HAProxy log:
Mar 8 16:11:12 juju-07387c-2 haproxy[13741]: 10.42.91.1:36208 [08/Mar/2023:16:10:22.433] cached-indico-local backend-cached-indico-local/server_1 0/0/0/-1/50003 504 195 - - sH-- 2/1/0/0/0 0/0 {} "POST /event/4/manage/registration/5/registrations/import HTTP/1.1" 0A2A5B01:8D70_0A2A5BF7:0050_6408B36E_01C2:35AD

Mar 8 16:12:09 juju-07387c-2 haproxy[13741]: 127.0.0.1:42670 [08/Mar/2023:16:10:22.433] indico-local backend-indico-local/server_1 0/0/0/107407/107407 200 737 - - ---- 1/1/0/0/0 0/0 {0A2A5B01:8D70_0A2A5BF7:0050_6408B36E_01C2:35AD} "POST /event/4/manage/registration/5/registrations/import HTTP/1.1" 7F000001:A6AE_7F000001:1F90_6408B36E_01C3:35AD

NGINX log:
10.42.91.1 - - [08/Mar/2023:16:11:12 +0000] "POST /event/4/manage/registration/5/registrations/import HTTP/1.1" 499 0 "-" "curl/7.85.0" 50.003 - - 0A2A5B01:8D70_0A2A5BF7:0050_6408B36E_01C2:35AD

Fix:

1) Manually change /etc/haproxy/haproxy.cfg adding "timeout server 300s" to "backend-cached-indico-local"

backend backend-cached-indico-local
    option httpchk GET /_status/content-cache-check HTTP/1.1\r\nHost:\ indico.local\r\nUser-Agent:\ haproxy/httpchk
    http-request set-header Host indico.local
    http-request set-header X-Forwarded-For %[src]
    balance leastconn
    timeout server 300s
    server server_1 127.0.0.1:6080 check inter 2s rise 2 fall 60 maxconn 50

backend backend-indico-local
    option httpchk GET /health HTTP/1.1\r\nHost:\ indico.local\r\nUser-Agent:\ haproxy/httpchk
    http-request set-header Host indico.local
    http-request set-header X-Forwarded-For %[src]
    timeout server 300s
    option redispatch 1
    balance leastconn
    server server_1 192.168.68.57:80 check inter 10s rise 2 fall 5 maxconn 50

Entire config here: https://pastebin.canonical.com/p/N7v8WQQzcv/

Result (success):

HAProxy log:
Mar 8 16:29:50 juju-07387c-2 haproxy[15402]: 127.0.0.1:57576 [08/Mar/2023:16:28:05.872] indico-local backend-indico-local/server_1 0/0/0/104757/104757 200 737 - - ---- 2/1/0/0/0 0/0 {0A2A5B01:AF8C_0A2A5BF7:0050_6408B795_0034:3C2A} "POST /event/4/manage/registration/7/registrations/import HTTP/1.1" 7F000001:E0E8_7F000001:1F90_6408B795_0035:3C2A
Mar 8 16:29:50 juju-07387c-2 haproxy[15402]: 10.42.91.1:44940 [08/Mar/2023:16:28:05.871] cached-indico-local backend-cached-indico-local/server_1 0/0/0/104758/104759 200 834 - - ---- 1/1/0/1/0 0/0 {} "POST /event/4/manage/registration/7/registrations/import HTTP/1.1" 0A2A5B01:AF8C_0A2A5BF7:0050_6408B795_0034:3C2A

NGINX log:
10.42.91.1 - - [08/Mar/2023:16:29:50 +0000] "POST /event/4/manage/registration/7/registrations/import HTTP/1.1" 200 834 "-" "curl/7.85.0" 104.757 - 104.761 0A2A5B01:AF8C_0A2A5BF7:0050_6408B795_0034:3C2A