Random Error 503 Service Unavailable on Varnish; its log shows no fetcherror

Randomly our production site encounters a 503 error; each time after a varnish service restart, the requested site comes up normally.

I looked at the varnish log and all I see is "Service Unavailable" but nothing else.

I'm posting a portion of the varnishlog, the default.vcl and the varnish.params

• << BeReq >> 857875

• Begin bereq 857874 fetch

• Timestamp Start: 1553896092.649343 0.000000 0.000000

• BereqMethod GET

• BereqURL /huf_en_us/search/ajax/suggest/?_=1553896063503&q=pot+leaf+sock

• BereqProtocol HTTP/1.0

• BereqHeader X-Real-IP: 172.16.204.112

• BereqHeader X-Forwarded-Proto: https

• BereqHeader SSL-OFFLOADED: https

• BereqHeader X-Forwarded-Port: 443

• BereqHeader Host: www.hufworldwide.com

• BereqHeader CF-IPCountry: US

• BereqHeader CF-RAY: 4bf50bf2cb163852-LAX

• BereqHeader CF-Visitor: {"scheme":"https"}

• BereqHeader accept: application/json, text/javascript, /; q=0.01

• BereqHeader x-requested-with: XMLHttpRequest

• BereqHeader accept-language: en-us

• BereqHeader listrak-listening: 1

• BereqHeader referer: hufworldwide.com/huf_en_us/triple-triangle-pullover-hoodie-pf00100

• BereqHeader dnt: 1

• BereqHeader user-agent: Mozilla/5.0 (iPhone; CPU iPhone OS 12_1_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1

• BereqHeader x-newrelic-id: VQcDV1JVABAEUVJVDgkOXw==

• BereqHeader cookie: store=huf_en_us; _vuid=5759a66b-8175-43cc-a614-51a32c0d7a08; PHPSESSID=los3vfdipo3ako096qoo830m05; form_key=QY53qr4aLFMypLqi; private_content_version=927393d221b9f95d0d23c147b7a38d6b; section_data_ids=%7B%22directory-data%22%3A1553894394%2C%22cus

• BereqHeader CF-Connecting-IP: 2606:a000:1216:8689:a5a4:dde1:69bd:10ac

• BereqHeader CF-Pseudo-IPv4: 252.241.164.167

• BereqHeader True-Client-IP: 2606:a000:1216:8689:a5a4:dde1:69bd:10ac

• BereqHeader CDN-Loop: cloudflare

• BereqHeader X-Forwarded-For: 172.16.204.112, 127.0.0.1

• BereqHeader Accept-Encoding: gzip

• BereqProtocol HTTP/1.1

• BereqHeader X-Varnish: 857875

• VCL_call BACKEND_FETCH

• VCL_return fetch

• BackendOpen 54 web02(172.16.204.102,,80) 172.16.204.104 60836

• Backend 54 d web02(172.16.204.102,,80)

• Timestamp Bereq: 1553896092.649819 0.000476 0.000476

• Timestamp Beresp: 1553896093.563128 0.913785 0.913309

• BerespProtocol HTTP/1.1

• BerespStatus 503

• BerespReason Service Unavailable

• BerespHeader Date: Fri, 29 Mar 2019 21:48:12 GMT

• BerespHeader Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_fcgid/2.3.9 PHP/7.0.30

• BerespHeader X-Powered-By: PHP/7.0.30

• BerespHeader X-Content-Type-Options: nosniff

• BerespHeader X-XSS-Protection: 1; mode=block

• BerespHeader X-Frame-Options: SAMEORIGIN

• BerespHeader Expires: Thu, 19 Nov 1981 08:52:00 GMT

• BerespHeader Cache-Control: no-store, no-cache, must-revalidate

• BerespHeader Pragma: no-cache

• BerespHeader Set-Cookie: PHPSESSID=los3vfdipo3ako096qoo830m05; expires=Sun, 28-Apr-2019 21:48:13 GMT; Max-Age=2592000; path=/; domain=www.hufworldwide.com; secure; HttpOnly

• BerespHeader X-NewRelic-App-Data: PxQGUFJVDAUIR1VXBAYOXl0IFB9AMQYAZBBZDEtZV0ZaClc9HiBQFg1ZWT1JJUpcXhAiDVlFRQkIXVNBPkkuA1cHVhZXZGh0QQRUBEYOQQk4anYRFj9kdUILDxZ0XlkSFl5aXwcUPz55DEwHSltRQkcKS0MdUR1SVAYHUUpTFgoAXVRQGxwGSkYCAwZbUltQB1oMWw0MBQVWRxUHUA1ABzk=

• BerespHeader Connection: close

• BerespHeader Transfer-Encoding: chunked

• BerespHeader Content-Type: text/html; charset=UTF-8

• TTL RFC -1 -1 -1 1553896094 1553896094 1553896092 375007920 0

• VCL_call BACKEND_RESPONSE

• TTL VCL 0 10 0 1553896094

• VCL_return deliver

• BerespHeader Content-Encoding: gzip

• Storage malloc Transient

• ObjProtocol HTTP/1.1

• ObjStatus 503

• ObjReason Service Unavailable

• ObjHeader Date: Fri, 29 Mar 2019 21:48:12 GMT

• ObjHeader Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_fcgid/2.3.9 PHP/7.0.30

• ObjHeader X-Powered-By: PHP/7.0.30

• ObjHeader X-Content-Type-Options: nosniff

• ObjHeader X-XSS-Protection: 1; mode=block

• ObjHeader X-Frame-Options: SAMEORIGIN

• ObjHeader Expires: Thu, 19 Nov 1981 08:52:00 GMT

• ObjHeader Cache-Control: no-store, no-cache, must-revalidate

• ObjHeader Pragma: no-cache

• ObjHeader Set-Cookie: PHPSESSID=los3vfdipo3ako096qoo830m05; expires=Sun, 28-Apr-2019 21:48:13 GMT; Max-Age=2592000; path=/; domain=www.hufworldwide.com; secure; HttpOnly

• ObjHeader X-NewRelic-App-Data: PxQGUFJVDAUIR1VXBAYOXl0IFB9AMQYAZBBZDEtZV0ZaClc9HiBQFg1ZWT1JJUpcXhAiDVlFRQkIXVNBPkkuA1cHVhZXZGh0QQRUBEYOQQk4anYRFj9kdUILDxZ0XlkSFl5aXwcUPz55DEwHSltRQkcKS0MdUR1SVAYHUUpTFgoAXVRQGxwGSkYCAwZbUltQB1oMWw0MBQVWRxUHUA1ABzk=

• ObjHeader Content-Type: text/html; charset=UTF-8

• ObjHeader Content-Encoding: gzip

• Fetch_Body 2 chunked -

• Gzip G F E 16777 6581 80 52568 52578

• Timestamp BerespBody: 1553896093.572716 0.923373 0.009588

• BackendClose 54 web02(172.16.204.102,,80)

• Length 6581

• BereqAcct 3058 0 3058 869 16789 17658

• End

default.vcl

vcl 4.0;

import std; import directors;

REM The minimal Varnish version is 4.0 For SSL offloading, pass the following header in your proxy server or load balancer: 'SSL-OFFLOADED: https'

backend web01 { .host = "172.16.204.101"; .port = "80"; .connect_timeout = 1600s; .first_byte_timeout = 1600s; .between_bytes_timeout = 1600s; .max_connections = 20000; }

backend web02 { .host = "172.16.204.102"; .port = "80"; .connect_timeout = 1600s; .first_byte_timeout = 1600s; .between_bytes_timeout = 1600s; .max_connections = 20000; }

backend web03 { .host = "172.16.204.120"; .port = "80"; .connect_timeout = 1600s; .first_byte_timeout = 1600s; .between_bytes_timeout = 1600s; .max_connections = 20000; }

backend web04 { .host = "172.16.204.121"; .port = "80"; .connect_timeout = 1600s; .first_byte_timeout = 1600s; .between_bytes_timeout = 1600s; .max_connections = 20000; }

varnish.params

REM Varnish environment configuration description. This was derived from REM the old style sysconfig/defaults settings

REM Set this to 1 to make systemd reload try to switch VCL without restart. RELOAD_VCL=1

REM Set WARMUP_TIME to force a delay in reload-vcl between vcl.load and vcl.use REM This is useful when backend probe definitions need some time before declaring REM configured backends healthy, to avoid routing traffic to a non-healthy backend. REM WARMUP_TIME=0

REM Main configuration file. You probably want to change it. VARNISH_VCL_CONF=/etc/varnish/default.vcl

REM Default address and port to bind to. Blank address means all IPv4 REM and IPv6 interfaces, otherwise specify a host name, an IPv4 dotted REM quad, or an IPv6 address in brackets. REM VARNISH_LISTEN_ADDRESS=192.168.1.5 VARNISH_LISTEN_PORT=8080

REM Admin interface listen address and port VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1 VARNISH_ADMIN_LISTEN_PORT=6082

REM Shared secret file for admin interface VARNISH_SECRET_FILE=/etc/varnish/secret

REM Backend storage specification, see Storage Types in the varnishd(5) REM man page for details. VARNISH_STORAGE="malloc,24G"

REM User and group for the varnishd worker processes VARNISH_USER=builder VARNISH_GROUP=builder

REM Other options, see the man page varnishd(1) DAEMON_OPTS="-p thread_pool_min=200 -p thread_pool_max=2000 -p thread_pool_timeout=600 • p http_resp_hdr_len=90000 -p http_max_hdr=200"