Cloudflare returning 520 due to empty server response from Heroku

Question

My Rails app which has been working great for years suddenly started returning Cloudflare 520 errors. Specifically, api.exampleapp.com backend calls return the 520 whereas hits to the frontend www.exampleapp.com subdomain are working just fine.

The hard part about this is nothing has changed in either my configuration, or code at all. Cloudflare believes this is happening as the Heroku server is returning an empty response.

> GET / HTTP/1.1
> Host: api.exampleapp.com
> Accept: */*
> Accept-Encoding: deflate, gzip
> 
{ [5 bytes data]
* TLSv1.2 (IN), TLS alert, close notify (256):
{ [2 bytes data]
* Empty reply from server
* Connection #0 to host ORIGIN_IP  left intact
curl: (52) Empty reply from server
error: exit status 52

On the Heroku end, my logs don't even seem to register the request when I hit any of these urls. I also double-checked my SSL setup (Origin certificate created at Cloudflare installed on Heroku), just in case, and it seems to be correct and is not expired.

The app is down for a couple of days now, users are complaining, and no response from either customer care teams despite being a paid customer. My dev ops knowledge is fairly limited.

Answer 1

Welcome to the club: https://community.cloudflare.com/t/sometimes-a-cf-520-error/288733 It seems to be a Cloudflare issue introduced in late July affecting hundreds of sites running very different configurations. It's been almost a month since the issue was first reported, Cloudflare "fixed" it twice, but it's still there. Very frustrating.

Answer 2

Change your webserver logs to a info state and see if your application is not exceeding some HTTP/2 directive while processing the connection.

If this is the case, try to increase the directive size:

#nginx
server {
    ...
    http2_max_field_size 64k;
    http2_max_header_size 64k;
}