shuts down

We were sad to hear about the sudden departure of our frenemies* at Nitrous.

Whenever something like this happens, there's a conversation about the reliability of SaaS services in the face of VC pressure -- do you really want to trust your business to a company that has VCs breathing down their neck setting deadlines for exits, and ready to pull the plug if things take a dip?

So we want to reassure you that we haven't taken on any big VC money. And although we do occasionally wonder, from our cramped little London office above a Burrito shop**, what the lives of developers in startups with 7-figure investments and swanky San Francisco offices with their table-tennis tables and elaborate trousers, we mostly don't resent it much. Considering we're now cashflow-positive and our investor (Hi Robert!) has just guaranteed the sole cash injection we need to achieve actual-for-reals-profitability, and that that investment is two orders of magnitude less than what Nitrous took on, well, let's say we hope we're a reasonably low-risk bet as far as our customers are concerned, and we plan to be around for a while.

Anyways, so much to say that, if you were using Nitrous to code Python, and you still fancy having a place in THE CLOUD to store your stuff, head on over to our site, and we'll be happy to see you.

So, in honour of our defunct friends at Nitrous, why not enjoy this naughties dark DnB classic by the unmistakeable Bad Company. Rinseout!

* yes, "frenemies", its a thing. if you're cringing, rest assured that, being British, I cringed the first time I heard this word too. So. American. But like all things Silicon Valley, there's a nugget of truth in it, and more than that. competitors can be friends (someone should remind politicians of this), and particularly when you're out creating new markets, a competitor doing a good job can genuinely increase the size of your own customer pool. even when they don't die!.

** actually it's now a Vietnamese. Nice Banh-mi, delicious roast belly pork, could do with being a little more generous on the meat portions though

Today's upgrade: improving websites, better security

This morning's system update went smoothly, and we've made a couple of great changes :-)

Improved website routing

This one should be pretty much transparent to you, but we've revamped the way we route requests for the websites that we host; this should speed things up for some people.

Noisy neighbours always cause problems, in the real world and on the Internet. When someone writes a website that hogs system resources on PythonAnywhere, sometimes it can impact other people who happen to be on the same server. Naturally, we monitor the system, and when we find a particularly badly-behaved website we notify its owner by email and ask them to fix it -- or in extreme cases, if it's causing serious problems, we shut it down. But that's far from ideal.

Today's update makes that all a lot better. We've given ourselves, the system administrators, fine-grained control over where websites run. So now, if we see a website that's causing slowdowns for other users, as well as notifying the owner so that they can fix it, we can move it right away onto a server where it won't impact other people. We're calling it "putting them in the sin bin"...

Security is important people have reminded us frequently in suspiciously-similar Tweets. And they're right! So we've implemented two-factor authentication, using Google Authenticator (or any other TOTP app). It's currently going through a short internal-only testing process (in other words, we've switched it on for our own accounts to see if it breaks anything) and if all is well, we'll provide it as an option for everyone next week.

On the subject of security, we've also fixed a couple of bugs: Nikhil Mittal reported a CSRF issue on PythonAnywhere that would have allowed an attacker who knew both your username and the internal database ID of one of your scheduled tasks to delete that task, if they tricked you into visiting a web page that they controlled while you were logged in to PythonAnywhere. It wouldn't have given the attacker access to any of your data, but it could have been really irritating, and we're glad it was reported so that we could fix it. Bug: fixed. Bug bounty: paid. Nikhil also reported some issues around our email confirmation system, which we've also fixed.

...and the rest

As always, we've put in a number of user interface tweaks, including fixing the print preview on IPython notebooks.

That's it!

Thanks for reading, and for using PythonAnywhere :-)

The PythonAnywhere newsletter, September 2016

We try to get a newsletter out every month, but in August we were all too busy working on our latest and greatest features to manage it. It wasn't that we were all out sunning ourselves, honest :-)

Here's what we were up to:

A new system image

This is actually quite a big deal, even if it might seem a bit dull at first glance. Accounts that have been created since 8 September have more recent versions of all of our batteries included. For example, while older accounts have Django 1.3.7 installed for Python 2.7, newer accounts get Django 1.10. The old image is called "classic" because we couldn't think of a better name, and the new one is called "dangermouse" because we remember the children's television of the UK in the 1980s much too well.

You can see how the two images differ by trying different filters on the batteries included page (which is now a shiny, JavaScript-driven, filterable list rather than a simple static page). If you have an older account and want to switch to the new system image, just drop us a line at Be warned: if you have old code that depends on the old Python packages, then it might stop working. So check your code first.

So, why did this take so long? And why do we think it's a big deal? Well, one of the development practices we follow here is what we call "Embarrassment-Oriented Project Management". In a nutshell, this says "if you're trying to work out what you need to work on next, ask yourself what makes you blush when you have to explain it to someone"... One of the things that was a tad embarassing was that we only supported Django 1.3.7 for Python 2.7. Sure, we supported later Django versions for later Python versions, but still.

But we had a problem. Until now, everyone on our site had to use the same system image, with the same versions of everything. We could add new things -- which is why, when Python 3.5 was installed, it had recent versions of all the system packages. But we couldn't upgrade anything for older Python versions, because it would have meant breaking any code that people had running that depended on the old system models. If you had a Django 1.3.7 website, and we upgraded to 1.10, then your site would suddenly stop working, and you'd have to change the code to make it work again. That would have been rather rude of us, so we didn't do it.

So what we needed to do was develop a way for different people to have different system images. For a while, we thought that Docker was the solution -- but unfortunately (for reasons related to scalability in our specific environment) it turned out not to be. So we've extended our virtualization system to support multiple system images.

More images will come in the future, once we've thought of an appropriately-irreverent name starting with "E" for the next one.

Right, enough of that! What else?

Let's Encrypt

Let's Encrypt is a project supported by the Linux Foundation to help secure the web by providing free HTTPS certificates for any domain you own. Its certificates are just as good as ones that you pay for in almost every way -- the only downside is that you have to renew them once every few months rather than once a year. (That's also a security feature, of course, because if someone steals your private key then they can only impersonate you for a few months rather than a year. But it is a little inconvenient.)

On PythonAnywhere, you already get free HTTPS for your websites that are subdomains of -- you don't need to do anything -- but if you have a custom domain, you need to give us a certificate to install. We've been honing our process to make this as easy as possible, and we're now pretty confident in them. So now there's no excuse to not secure your website -- just follow the instructions here.

The inside scoop from our forums

New modules

Although you can install Python packages on PythonAnywhere yourself, we like to make sure that we have plenty of batteries included.

Because of the new system image, we've not actually added any new modules this time around -- but we've updated everything we could to the latest version for the new "dangermouse" image. You can compare and contrast the different version in each image over on the new, shiny, interactive batteries included page.

New whitelisted sites

Paying PythonAnywhere customers get unrestricted Internet access, but if you're a free PythonAnywhere user, you may have hit problems when writing code that tries to access sites elsewhere on the Internet. We have to restrict you to sites on a whitelist to stop hackers from creating dummy accounts to hide their identities when breaking into other people's websites.

But we really do encourage you to suggest new sites that should be on the whitelist. Our rule is, if it's got an official public API, which means that the site's owners are encouraging automated access to their server, then we'll whitelist it. Just drop us a line with a link to the API docs.

Here are some sites we've added since our last newsletter:

  • -- Neo4j-as-a-service
  • -- a friendly to-do list
  • -- and another to-do list!
  • -- The American Association of Variable Star Observers.
  • -- a marketplace for algorithms.
  • -- who's playing near you?
  • -- desktop-wallpapers-as-a-service :-S
  • -- support the poorest schools in the United States
  • -- an Indian SMS provider.
  • -- an email provider
  • -- an open database for television fans
  • -- an exchange site for virtual world currencies
  • -- real-time weather information
  • -- the aviation weather REST API.
  • -- open data from the city of Madrid, Spain.
  • -- the developer portal for a UK supermarket (!)
  • -- a recipe site.
  • -- pretty much what you'd expect from the name...
  • -- open data from the BBC
  • -- open data from MapQuest.
  • -- open data from the city of Barcelona, Spain
  • -- get Open Graph data about websites using an API -- even for sites that don't explicitly support it.
  • -- free classic eBooks
  • -- RSS feeds from the Internet Movie Database
  • -- a collection of bioinformatics web services.
  • -- an online game similar to Risk.
  • -- US real-estate listings.

And that's it

Thanks for reading our newsletter! Tune in the same time next month (ish) for more news from PythonAnywhere.

Latest deploy: Some nice new features and a surprise

Rename web apps

Yes, we know it's been a long time coming, but now you can rename your web apps (and, as a result change the domain they're served from) right on the web app setup page. Look for the little edit pencil icon next to your web app address.

Students can share with teacher

We've made it easier for students to share their consoles with their teacher.

List invoices on accounts page

For those of you that may be wondering how much of your hard-earned money you've spent on PythonAnywhere, we've added a list of all of your invoices to the Account page.

PDF export for Jupyter notebooks works

A helpful user pointed out that "Download as PDF" wasn't working in Jupyter notebooks on PythonAnywhere. So we fixed it.

"bash console here" on editor page

If you're ever editing a file and want to open a Bash console in the same directory as the file, now you can.

General security, usability and stability fixes

As usual. This is usually where we put all the fixes for bugs that are too embarrassing to list.

Something great that we're not telling you anything about

until we've tested it ourselves.

Back to school tips for teachers, from PythonAnywhere

Dear teachers,

If there's one thing we know, it's that teachers (and students) love, it's being reminded that the holidays is that the holidays are coming to an end. Hooray!

Here's a few ideas and pointers for some of the things that we hope will make your life, as a teacher, easier.

1. PythonAnywhere means all your students have exactly the same setup

If you're already a PythonAnywhere user, you'll already know this. But, any teacher that's struggled with getting several different students, with different hardware and operating systems, all set up and working for a basic programming class, know how much time is easily lost.

With PythonAnywhere, everyone has the same kind of terminal, the same operating system, the same versions of Python, and the same standard installation. And you can jump on and view exactly what your students are doing.

2. PythonAnywhere makes it easy to go beyond the basics, and use external libraries

We have hundreds of packages preinstalled, and it's easy to install new ones. Here's a few examples of packages, and the associated topic which you could build a lesson around:

  • Scraping with requests and beautiful soup
  • Accessing the google maps api with google-api-python-client
  • Number crunching with numpy/pandas/scipy/scikit-learn
  • Use sqlalchemy to access our provided mysql/postgres databases
  • Best practices such as version control and isolated environments with git/mercurial/virtualenv
  • Python standard library modules such as sqlite3, unittest, re, pickle, pdb

Here's our full list of preinstalled "batteries included"

And here are our instructions for installing new modules

3. PythonAnywhere makes it easy to distribute and collect assignments from your students.

Because all your students' home directories are available to you in a Bash console, it's easy to copy files from your accounts to theirs, and back again. Check out the worked example on our help page.

Once you've got the basic idea, your imagination is the only constraint on what you could achieve by building little automated scripts to help your classes. Off the top of our non-teacher heads, how about:

  • Checking last updated times, and counting line numbers, in assignments files, to see who's been working and who hasn't

  • Writing a scheduled task to copy assignments across to your own folder at a pre-specified time, thus automating the "deadline" for the assignment

  • Automatically running student scripts and emailing yourself the results, possibly even grading assignments automatically.

If any of that piques your curiosity but you're not quite sure how to go about it, don't hesitate to get in touch, and we'll happily point you in the right direction.

4. PythonAnywhere makes it really easy for your students to build a real, publicly available website

You can use python web frameworks such as django, flask, bottle or web2py while we take care of the infrastructure behind it. No need to worry about server configuration, load balancing, or keeping the operating system patched - we take care of all that for you.

5. We recently introduced Jupyter Notebook support (aka IPython notebook).

These are interactive "notebooks" that let you run bits of code step by step and get instant feedback. It allows you to include explanatory text and data visualizations directly in the notebook and is used by university researchers worldwide.

Many teachers have told us they love the interactive nature of notebooks, with the easy ability to "go back", change things, and see the effect filter down -- something that really helps students with exploratory coding.

Unfortunately notebooks are currently a premium feature, but (read on...)

6. We can create a custom "teacher" plan for you, if you need some paid features for your students.

Currently, free accounts have certain restrictions:

  • Internet access is restricted to a whitelist of sites (

  • You cannot use SSH, and your disk & CPU quota is limited

  • You cannot use Jupyter Notebooks

  • You cannot host applications on custom domains (only

However, we have started offering special pricing for teachers, whereby we can lift one or many of the above restrictions, for $1/student/month. Get in touch with us and we'll help you out.

7. You can create your own step-by-step tutorial, integrated into the PythonAnywhere UI.

Ever used one of our little green "tutorials", the step-by-step instructions that sit in green boxes at the top of a pythonanywhere screen?

They're actually open-source, and you can write your own, for your students to use, and we'll integrate it into PythonAnywhere. Get in touch for more information.

That's our top seven, but there's many more little things that (we hope) will make life for teachers easier:

  • You can work on the go from your tablet/ipad
  • We can do bulk creation of accounts for all the students in your class in one go.
  • We can also set up special "exam" accounts, for you to use in test situations, or controlled assessments.

We're always keen to go the extra mile to make sure teachers and students are getting the best out of PythonAnywhere, so don't hesitate to email us, and let us know what you're doing, and how we can help.

Latest deploy: new stylings, editor fixes, and our API beta

Morning all! A lovely day for leave-ing an old server image behind and welcoming in a new, independent, codebase. #brupgrade #brelease #breployment.

Style tweaks, improvements to responsiveness

On a bit of a whim we decided to upgrade to bootstrap 3, so you'll notice slightly different stylings. Flat buttons! Oh-so-3-years-ago. But also, there are some improvements to the way the site displays on mobile and smaller displays, which is nice.

We also upgraded to the latest version of the ace editor, which should bring a few little improvements too, like better vim keybindings, and better support for ipads (you can now scroll, yay!)

API beta

It's not ready for prime-time yet, but we've started work on a PythonAnywhere API. It may end up not being something we publish for general use, and just something for us to use behind the scenes, but if you're keen to take a look, get in touch, and we'll switch it on for you. Currently the API allows you to do stuff to your web apps, namely:

  • create new web app
  • reload web app
  • update webapp settings: virtualenv, static files.

You should bear in mind that anything you build using that api will probably break when we next do a release, we're making no guarantees about backward-compatibility, or that the api will even work as it is. So really it's just for playing around or for the curious for now. Still, email us if you're interested, we'd love to hear from you.

Other changes

  • There's now a UI for updating the working directory and source files location for your web apps.
  • The editor now gives you a useful warning when you try and save if you're over your quota (and it no longer deletes your whole file, which some consider a bonus)
  • We've made some changes to try and make it easier for us to deal with forum spam. ugh.
  • and a few assorted minor bugfixes.

Your comments and suggestions are always welcome. Enjoy!

The PythonAnywhere newsletter, June 2016

Summer is just around the corner in the northern hemisphere -- here in London we can tell because the rain is just that little bit warmer. We're taking full advantage of the lighter and later grey in the sky by working even longer hours to make PythonAnywhere better. How? Just read on...

Better databases!

Over a year ago, we started supporting Postgres databases. Since then, we've found that we've been able to optimise the service so that it scales better than we thought, so we're pleased to pass on the cost savings to you.

Postgres is now $7/month instead of $15, so if you've ever thought about checking it out but been put off by the price, now's the time to take a look!

We've also applied the new price to existing Postgres users for their next bill. So that's more moolah in your pockets dear users, don't spend it all at once ;)

Also -- the way we handle MySQL databases needed some work to make it faster, more efficient, and more scalable. Over the last month we've learned more about obscure aspects of database administration than we ever wanted to know... but the end result for you should be that your database access is faster and smoother. Let us know how it's going!

Useful tips and interesting ideas

The inside scoop from our forums

New modules

Although you can install Python packages on PythonAnywhere yourself, we like to make sure that we have plenty of batteries included.

We haven't added any new ones since our last newsletter :-( -- but we've got a bunch in the pipeline -- more next time!

New whitelisted sites

Paying PythonAnywhere customers get unrestricted Internet access, but if you're a free PythonAnywhere user, you may have hit problems when writing code that tries to access sites elsewhere on the Internet. We have to restrict you to sites on a whitelist to stop hackers from creating dummy accounts to hide their identities when breaking into other people's websites.

But we really do encourage you to suggest new sites that should be on the whitelist. Our rule is, if it's got an official public API, which means that the site's owners are encouraging automated access to their server, then we'll whitelist it.

Here are some sites we've added since our last newsletter:

  • and -- the Internet Archive!
  • and -- If This Then That
  • * -- "the easiest way to map and analyze your location data"
  • -- Blockchain web services
  • -- more Bitcoin goodness
  • -- a geographical database
  • and -- APIs for human resources applications
  • -- enterprise messaging
  • -- an API for digging up information about IP addresses
  • -- a GoDaddy service for posting your details to aggregators like Google, Yelp, and so on.
  • -- parcel tracking
  • -- phone and text APIs
  • -- a recipe site
  • -- Chicago Transit Authority

And that's it

Thanks for reading our newsletter! Tune in the same time next month (ish) for more news from PythonAnywhere.

Today's upgrade - postgres price drop, mysql scaling improvements

Nothing earth-shattering to report today, but some good news:

Postgres is cheaper

It's been over a year since we first tentatively launched our postgres service, and we've found that we're able to optimise the service so that it scales better than we thought, so we're pleased to pass on the cost savings to you.

Postgres is now $7/month instead of $15. That price will apply to all new plans and upgrades, and we'll also start applying the new price to existing users for their next bill. So that more moolah in your pockets dear users, don't spend it all at once ;)

MySQL infrastructure changes

These changes won't really be very visible from the user point of view, so this isn't very interesting to you, beloved readers, per se, but it took us loads of time and effort so we have to say something to make it all feel worthwhile and satisfy our own egos. Anyways, we made some changes to the way we shard users amongst MySQL servers in our clusters, which mean it's now much easier for us to add extra MySQL capacity whenever we want to.

For the curious, did you know that (depending on your OS and config), filesystem limits on the number of hard links in a single directory might limit you to a maximum of 32,000 databases on a single mysql instance? Not that we ever came anywhere near that, but still, good to know. #tipsforpaasproviders. console now Python 3.5

Our live consoles on the front page are now Python 3.5 instead of 3.4. We've also made them "regular" Python consoles instead of IPython (which was always a slightly weird decision, even though IPython is all awesome and everything, but a regular Python console is what new users are most likely to see, and ours do have tab-completion switched on you know?)

Onwards and upwards folks! In our next iteration we hope to be able to release a first beta of an API for PythonAnywhere. Watch this space :)

Scaling a startup from side project to 20 million hits/month - an interview with creator Kaustubh

We recently wished farewell to a customer who had been with us for about 18 months, during which time he saw some incredible growth in what was originally just a side project. We spoke to him about how he found the experience of scaling on PythonAnywhere, and why he decided to move on. stats
Project started:October 2014
Requests:20 million / month
Active users:1000+

What's your background? How long have you been programming?

I am currently pursuing Bachelor's in Computer Science & Engineering. I have been programming from school days but RailwayAPI was the first substantial project I did.

Can you describe what does? What first gave you the idea to build a site like this?

It all started with an idea to build an app which let Train travelers in India find the best available route between two stations. It's very difficult to get confirmed bookings in India and so I thought it would be great if there was an app which helps people break their journey up, using multiple trains to reach their destination in minimum time.

While working on that idea I realized that I needed train data to make such a thing possible. There wasn't a reliable API for Indian railways and I realized that several developers would be facing the same problem. And hence RailwayAPI was born.

It is a collection of APIs which let developers access all kinds of Railway data like Seat Availability, Train Route, Live Train status etc in easy-to-use JSON formats.

Why did you choose Python and Flask? Are you happy with your choice?

Python was the language I already knew well and there were several great libraries available for it so the decision was very easy to make for me.

Since the exposed part of the API was just going to be URL views which capture GET requests and return the corresponding JSON, Flask was the most suitable with its minimality which is enough for what I was doing.

What made you choose PythonAnywhere?

This was my first webapp and initially I had very little experience in setting up such an environment. After some research I stumbled upon PythonAnywhere which was extremely easy to set up, with a nice and clean UI. It also lets you instantly scale up your app by just sliding the number of workers you are going to need. After that I needn't go anywhere else!

Your site quickly became one of our busiest sites -- when did you realise it was getting big? How was the experience of scaling up on PythonAnywhere?

I realized that it was getting big when one of the user complained about load balancer errors they were getting. But it wasn't an issue as I quickly scaled up the number of workers and site was back to normal functioning in seconds.

What kind of traffic did you have last month, for example?

The site got about 19.6 million requests last month! And for the month before that it was about 16 million requests.

You've now decided to move on -- why is that, where did you move to, and was it easy to make the transition?

Yes, I have moved to a VPS now at Digital Ocean. It wasn't an easy decision to make for me and it was done only after considerable thought. I loved PythonAnywhere and I continue to love it but I needed more flexible configurations so a VPS was required at the current stage.

It was quite difficult to move to a VPS because I have grown accustomed with the ease of use at PythonAnywhere where everything is pre set up and you can just focus on writing your code instead of writing configurations.

In general, what do you think are the pros and cons of PythonAnywhere?

I think I have already mentioned a lot of Pros about PythonAnywhere. But in short if you just want to focus on building your app rather than setting up environment, which is what ideally it should be than there are very few places like PythonAnywhere out there. Also developers should know that setting up an environment is not a one off process, it requires constant monitoring and changes so that any modification in the code doesn't break the configuration and vice versa. Its quite a time consuming process but PythonAnywhere takes care of all of that.

I would have liked it if PA supported asynchronous workers. In fact this was the main reason for moving out because my application was Network I/O bound and async workers were better suited for such a task, and that wasn't available here, at least for now.

Also it would be great if Redis and Web Sockets were also supported at some point.

It sounds like the site was a big success. What's next for

The goal is to give developers access to Indian Railways data without hassle. There are some other ideas I am working on like making available Train and Seat Availability prediction/analytics through API.

Do you have any advice for other aspiring web developers?

I am still learning a lot myself!

Although I could say from my experience that the most important thing I learned while developing the API was that engineering your App to be scalable is one the most challenging tasks, which is often overlooked by people in the beginning.

I had to re-code several of the modules several times because they resembled 'hacky' code but as the pressure on the site grew they started to break. So it would be good if developers also think about how their app would respond to such scenarios in the future.

Thanks again Kaustubh, and best of luck with the future of the project!

Anatomy of a bug

We recently fixed a problem in our website hosting code that was causing weird errors under very specific and rare circumstances. The problem had been there for several years, and -- while we knew that odd stuff happened every now and then -- we'd never been able to reproduce it reliably enough to debug it. But a lucky coincidence of circumstances, when two people tripped over the bug in quick succession, clarified the issues and let us work out the solution. It's an interesting tale, so we thought we'd share it.

The problem

Every now and then -- say, once every six months -- someone would report that they had a Flask website which would inexplicably start returning CSRF errors for every request. The apps in question generally had CSRF protection disabled. The error messages were very generic -- there was no indication of where in the stack they came from. And the problem would go away if the web app was reloaded.

There was no obvious pattern that we could see, apart from the fact that it was always Flask apps. No bugs had been reported against Flask's CSRF protection plugins.

We couldn't for the life of us work out what was causing the problem. Our normal request-processing stack (two layers of nginx for loadbalancing and basic HTTP(S), then uWSGI) doesn't do any CSRF-related stuff, and it sends all headers and requests through unchanged. But no-one else was seeing this problem on other hosts, so it seemed likely that there was something odd going on with our service specifically.

The lucky coincidence

Two things happened: we recently upgraded the version of Django that we use to host the PythonAnywhere website itself to 1.9 (from an embarassingly old version that I won't mention). And then, two people reported the weird CSRF problems in quick succession. One was using Flask, but the other was using Django. So maybe the problem wasn't Flask-specific.

Even more interestingly, they were both getting exactly the same error message -- and it was slightly different to the one that had been coming back before. Furthermore it was one that, on Googling, we could only find in newer versions of the Django codebase. This sounded suspiciously like it was coming from somewhere in our infrastructure -- it had changed when we upgraded, and it was a Django message even for the Flask app. But why would our code -- the Django code that makes up the PythonAnywhere website -- be called when a request was being made to one of our customers' sites?

Website wakeup

PythonAnywhere's web hosting platform is made up of a large number of web servers. Each web server is capable of serving any of the tens of thousands of websites we host. But, of course, the code for only a subset of those sites is running on any given server at any given time. We load-balance across the web servers, sending the traffic for each hosted site to a specific server. If we add another web server when things are busy, it's a simple job to tell the loadbalancer to spread the traffic out differently, so that the new server takes over handling some of the sites.

If one of our webservers gets a request for a website that isn't already running on it, the request gets redirected internally to a view on a PythonAnywhere web app, which checks to make sure that the site is one that we're actually meant to be hosting (rather than some random domain that someone happens to have pointed at one of our IP addresses). If it is, it starts the site up and redirects the incoming request back round again internally to the original URL. Once that's happened, the requester gets the response from the freshly-started website.

This means that when we add a new server, which will be running no websites initially, it will start them up one by one as it receives the first request for each from the loadbalancer.


Once we'd realised that the error that was coming back for our customers' sites was from Django, and had changed when we upgraded Django, it was clear that this "wake up" view in our own code was the only possible culprit. It's the only bit of our code that would ever be run in response to a request for a customer site.

We took a look at the view -- and, it turned out, it wasn't marked as CSRF exempt. Why does that matter?

We use CSRF heavily across our site (to mitigate the risk of a malicious person putting up a website that posted a "delete my web app" request to PythonAnywhere when someone visited it), and CSRF protection is the default -- each view must explicitly opt-out.

Now, "wake up" requests to this view would come in without any CSRF tokens or other related data. CSRF is site-specific, and the wake-up requests were not really requests for our site at all -- they were requests meant for another site that had been temporarily passed over to us so that we could start up the code to handle them.

So it was definitely a bug that the view wasn't marked CSRF exempt.

Importantly, though, CSRF protection is only relevant to POST requests. GET requests don't need it, as they are (so long as you're doing things properly) not state-changing operations.

So we'd worked out part of the problem -- if a website wasn't running on a given server, and a request for that site was sent to the server, then only a GET request would start the site correctly. A POST request would get a spurious CSRF error, and the site would not be started up.

But our customers who had reported the problem were saying that when their websites got into the CSRF-error state, they'd stay in that state regardless of how much traffic they got. And also, what was the Flask connection? While we now had one person seeing the problem with Django, every other report had been against Flask.

Finally, one of our customers who was having the problem told us something that made it all clear. He explained that his site was collecting data from other computers elsewhere on the Internet, which were only ever POSTing to it. He had discovered that when it got into the state when it was always responding with CSRF errors, he didn't need to reload it to fix it. He just needed to log in to the admin page.

Suddenly it all became clear. The lack of the @csrf_exempt decorator on our "wake up the website" view meant that it would always respond to a POST request with a CSRF error. But if it received a GET request (for example, a hit from a browser on the admin login page, or a manual reload from the "Web" tab on PythonAnywhere), it would work properly and start the website. Once that had happened, then all further POST requests would work.

Types of websites

This taught us something interesting about the kind of sites people build. There are two main kinds of sites on PythonAnywhere (and we think on the Internet at large) -- human-readable and computer-readable.

Human-readable sites get a lot of GET requests as people hit them and browse around looking at pictures of cats. They get the occasional POST as someone comments on a cat picture, or sends a message to the cat or its owner, but most hits are GETs.

Computer-readable sites are basically web APIs. They also get quite a few GETs. But sometimes, they're designed to get mostly POSTs. The data-collection site that our customer was building, for example. Or sites that exist entirely to receive webhook messages.

Most of the sites on PythonAnywhere are human-readable, but there are a fair number of computer-readable ones. Of the computer-readable ones, most get a decent number of GET requests. A tiny proportion are APIs that get predominantly POSTs. So while every site on PythonAnywhere would have the problem where it couldn't be woken up by a POST request, only a very small subset of them would be seriously affected -- the ones that got almost nothing but POSTs.

The problem was only affecting that tiny proportion of sites. But the fact that their sites were only handling POSTs wasn't something that was obviously relevant information to the customers who were reporting the problem, so they didn't mention it. And because we didn't know what the cause was, we never knew that it was something we should ask about.

And the Flask connection? Well, if you're building a website that handles lots of POST requests, and you don't need to worry about user management, administration, and user interface design, which framework are you most likely to pick? Flask is almost certainly the most popular framework for people building simple web APIs. So it wasn't quite a red herring that the problem seemed originally to be Flask-specific -- there was a very strong correlation there -- but that correlation really did mislead us into thinking it was some weird interaction with Flask specifically.

The fix

Once we knew what the problem was, it was trivial to fix -- just adding a @csrf_exempt on the correct view -- and we had it patched within an hour, even including testing.

But it was a long and confusing year puzzling over the issue! Hopefully the explanation gives some flavour of the sleuthing process.

Page 3 of 16.

« More recent postsOlder posts »

PythonAnywhere is a Python development and hosting environment that displays in your web browser and runs on our servers. They're already set up with everything you need. It's easy to use, fast, and powerful. There's even a useful free plan.

You can sign up here.