Scaling Django to 8 Billion Page Views
https://blog.disqus.com/scaling-django-to-8-billion-page-views
Posted byMatt Robenolton September 24, 2013
As we’re approaching 8 billion page views per month and 45k requests per second, we’ve learned a couple things about delivering comments to a lot of different people. Disqus is very well known for usingDjangofor almost all of our web traffic, and that continues to be a thing today. As with any web framework, there are inherent trade-offs: rapid development vs performance, familiarity for new developers vs something custom, etc. Disqus likes to lean towards rapid development and familiarity over performance, and something fine tuned for our exact needs.
So, why is a web framework slow?
On the surface, the first impression is that a web framework is slow because there is a lot of boiler plate and unnecessary code that is not needed for your application, and that is a valid impression. In practice, slowness is usually not a product of your framework’s bloat or the language choice. Slowness is likely a result of the fact that your request is communicating with other services across your network. In our case, these other services arePostgreSQL,Redis,Cassandra, andMemcached, just to name a few. Slow database queries and network latency generally outweigh the performance overhead of a robust framework such as Django.
To get around these latencies, people use various forms of caching. The most tangible approach would be to use the built-inDjango cache library.
The common pattern for application level caching is such:
data = cache.get('stuff')
if data is None:
data = list(Stuff.objects.all())
cache.set('stuff', data)
return data
If you are familiar with Django, this should be a pretty familiar pattern. This form of caching is simple and straightforward, and works really well for most things. Paired with Memcached, things are fast enough, but there is still a lot of work still being done to serve a request.
Dealing with 45k requests per second
We’ve cached our “slow” things. There is still a lot of unnecessary work that needs to be done at rate of 45 thousand times per second. We’re probably rendering some JSON, or rendering an HTML template, or simply parsing HTML and executing our Django middleware. The point is, we want to be able to short-circuit all of this work, and leave Django to do what it does best: serve unique data only.
Out of 45k requests per second, how many are truly unique? How many of those responses are actually different from one response to the next? Do we really need to keep doing the same work over and over again when the result is always the same? We really want to cache whole responses and skip all of the other work.
Introducing Varnish
What even isVarnish? Varnish is a piece of software that sits between our load balancers and our Django backends and acts as an HTTP caching layer. What this means is that it can cache the entire HTTP response without even hitting a Django server, if we know that request won’t be unique.
Previously, Varnish was a bit of a black box to us. We installed it and configuration was very minimal, and honestly, this worked very well. But I thought we could do more.
I spent some time learning more about Varnish and some tricks that we could use. Over time, we were able to shave off several thousands of requests per second from ever hitting our Django servers. Today, out of the 45k inbound requests every second, only about 15k or so actually hit our app servers. The rest are absorbed by Varnish and served to the user very quickly and efficiently.
Since this has been very useful for us and a good learning experience, this topic has been the subject of a few recent talks of mine.
Most recently, I spoke atDjangoCon USin Chicago. This talk was aimed toward people who weren’t familiar with Varnish, with the hopes of inspiring and motivating them to learn more. For me, I was excited to give this talk because it’s a topic that isn’t explained very often to application developers. It’s a talk that I’d really liked to have heard a few years ago, and hopefully bridges the gap in understanding how HTTP really works and how you can manipulate it to interact with tools such as Varnish.
Prior to that, I presented atVUG7 (Varnish User Group)in New York, and went into details about some of the exact tricks that we use to help overcome some of our problems. This talk goes into a lot of detail about the specificVCLsthat we use for each endpoint needed to deliver our embed.
tl;dr
Check outVarnish. It won’t solve all of your problems, but it’s something worth investing the time into learning about and evaluating.
