Jeffrey I. Schiller
Networking, Security and Cavies

Beware Chrome and HTTP/2 Debugging

It's been a while since I updated this blog, but I thought this might be of interest…

TL;DR

HTTP/1 specifies that the Cookie header contains all the cookies being sent, separated by semicolons. However, HTTP/2 permits multiple cookie headers.

If Chrome makes a connection to an HTTP/2 capable web server, and the connection is protected by TLS, it will use HTTP/2.

If you look at a transaction in the Network panel of the developer tools, you will see one Cookie header, with multiple cookies. However, what is sent is actually two (or more?) headers, each with one cookie. I suspect this discrepancy is the result of the conversion from one header to multiple headers happening at a lower level in the Chrome networking code. This bit me!

The Gory Details

So, I have a service written in Rust that uses the warp web server (which I do like!). For those not familiar with Rust, it generates very efficient and memory safe code, without needing a garbage collector. It also has a steep learning curve, which I was still climbing at the time this situation occurred. Warp also supports HTTP/2.

I decided that I wanted to add some admin functions to the site. Whereas the prime function of the site needs to handle significant traffic (one of the reasons I chose to use Rust and Warp), the admin traffic would be minimal (basically just me checking things).

I'm also a big fan of the Django python framework. So, I decided to write the admin interface in Django. I would then add a reverse proxy to the Warp written side that would redirect the admin links appropriately. This was actually pretty easy to do, as there is already a warp reverse proxy Rust "crate" that I could just use.

I wrote the code, debugged it, all was good. Then I go and stage it on a test server, and the Django code doesn't work. Upon looking at things on the Django side, I see that instead of several cookies being delivered to Django, there is just the first cookie, with all other cookies appended, separated by commas. I.e., it was as if something replacing the semicolons in the Cookie header with commas!

This only happened in a production like environment. Examination with the developer tools showed a correctly formatted Cookie header. So, what gives? At one point I figured maybe (not sure how) the use of TLS had something to do with it, so I set up a reverse tunnel (stunnel on Linux). But all worked. Only when connecting with Chrome (or Firefox for that matter) did the problem surface. And, of course, looking at the actual network traffic would be difficult, as it is encrypted!

Ultimately, I added debugging code to the Rust server that displayed the headers that it saw, and lo and behold, there were more than one Cookie header. Django, for its part, wasn't expecting this and managed to merge them together, separated by commas. A little more poking around, and I was able to determine that the connection was coming in over HTTP/2, which permits this.

Of course, once I knew what was happening, the fix was easy. I just had may code turn the multiple cookie headers back into one Cookie header before calling the reverse proxy code. This is important because the connection between the reverse proxy and the Django code, was an HTTP/1 connection (it isn't encrypted, it is all in one docker swarm, whose network layer is already encrypted).