Today's upgrade included a few new packages in the standard server image:
We also improved the default "Unhandled Exception" page, which is shown when a users' web app allows an exception to bubble up to our part of the stack. We now include a slightly friendlier message, explaining to any of the users' users that there's an error, and explaining to the user where they can find their log files and look for debug info.
And in the background, we've deployed a bunch of infrastructure changes related to postgres support. We're getting there, slowly slowly!
Oh yes, and we've enabled dynamic previews in the forums, so you get an idea of how the markdown syntax will translate. It actually uses the same library as stackoverflow, it's called pagedown. Hope you find 'em useful!
Our founder, Giles Thomas, gave a high-level introduction to our load-balancing system as a talk at this summer's EuroPython. There's a video up on PyVideo: An HTTP request's journey through a platform-as-a-service. And here are the slides [PDF].
Fancy helping to build the Python world's favourite PaaS (probably)? We're looking for a "junior" programmer with plenty of smarts to come and join the team, learn the stuff we do, and inject some new ideas...
Here's some stuff you'd be doing:
Working in an Extreme Programming (XP) shop, pair programming and TDD all day, woo*.
Devops! Or what we take it to mean, which is that you deploy to and administer the servers as well as write code for them. Lots of Linux command-line goodness in SSH sessions, and automated deployment stuff.
Sexy CV-padding technologies! Like Docker, nginx, websockets, Django, copy-on-write filesystems, Ansible, GNU/Linux (Ubuntu), virtualbox, vagrant, continuous integration, AWS, redis, Postgres, and even Windows XP! (although we're phasing that last one out, to our great chagrin). Don't worry, you don't have to know any of these before you show up, you'll get to learn them all on the job...
Learn vim (if you want to) much faster than you would on your own by being forced to pair program with vim cultists all too happy to explain the abstruse keyboard shortcuts they're using...
Get involved in the nonprogramming aspects of the business too (we all do!), like customer support and marketing. Honestly, that can be fun too.
Work near enough to Silicon Roundabout that you can walk to the Hacker News meetups, but not so near that you're forced to overhear bad startup ideas being pitched in every coffee shop
* The pair programming thing is an unbelievably good deal for new developers btw, there's just no faster way of learning than sitting down next to someone that knows and being able to ask them questions all day, and they're not allowed to get annoyed.
Here's the kind of person we'd like
Here's some stuff we don't care about:
Send us an email telling us why you'd like to work here, and a current CV, to email@example.com
Here's what's new in the latest version of PythonAnywhere that we released this morning:
After a lengthy outage last night, we want to let you know about the events that led up to it and how we can improve our outage responses to reduce or eliminate downtime when things go wrong.
As usual with these things, there is no single cause to be blamed. It was a confluence of a number of things happening together:
It is a fact of life that our machines run on physical hardware. As much as the cloud, in general, and AWS, in particular, try to insulate us from that fact. Hardware fails and we need to deal with it when it does. In fact, we believe that a large part of the long-term value of PythonAnywhere is that we deal with it so you don't have to.
Since our early days, we have been finding and eliminating single points of failure to increase the robustness of our service, but there are still a few left and we have plans to eliminate them, too. One of the remaining ones is the file server and that's the machine that suffered the hardware failure last night.
The purpose of the file server is to make your private file storage available to all of the web, console, and scheduled task servers. It does this by owning a set of Elastic Block Storage devices, arranged in a RAID cluster, and sharing them out over NFS. This means that we can easily upgrade the file server hardware, and simply move the data storage volumes over from the old hardware to the new.
Under normal operation, we have a backup server that has a live, constantly updated copy of the data from the file server, so in the event of either a file server outage, or a hardware problem with the file server's attached storage, we can switch across to using that instead. However, yesterday, we upgraded the storage space on the backup server and switched to SSDs. This meant that, instead of starting off with a hot backup, we had a period where the backup server disks were empty and were syncing with the file server. So we had no fallback when the file server died. Just to be completely clear -- the data storage volumes themselves were unaffected. But the file server that was connected to them, and which serves them out via NFS, crashed hard.
With all of that in mind, we decided to try to resurrect the file server. On the advice of AWS support, we tried to stop the machine and restart it so it would spin up on new hardware. The stop ended up taking an enormous amount of time and then restarting it took a long time, too. After trying several times and poring over boot logs, we determined that the boot disk of the instance had been corrupted by the hardware failure. Now the only path we could see to a working cluster was to create an entirely new one which could take over and use the storage disks from the dead file server. So we kicked off the build (which takes about 20min) and waited. After re-attaching the disks, we checked that they were intact and switched over to the new cluster.
We have learned important lessons from this outage and we'll be using them to improve our service. We would like to extend a big thank you and a grovelling apology to all our loyal customers who were extremely patient with us.
This morning we upgraded PythonAnywhere, and the upgrade process took much longer than expected. Here's what happened.
The main visible feature in this new version was Python 3.4. But a secondary reason in doing it was to move from one Amazon "Availablity Zone" to another. PythonAnywhere runs on Amazon Web Services, and availability zones are essentially different Amazon data centers.
We needed to move from the zone us-east-1a to us-east-1c. This is because the 1a zone does not support Amazon's latest generation of servers, called m3. m3 instances are faster than the m1 instances we currently have, and also have local SSD storage. We expect that moving from m1 to m3 instances will make PythonAnywhere -- our site, and sites that are hosted with us -- significantly faster for everyone.
Moving from one availability zone to another is difficult for us, because we use EBS -- essentially network-attached storage volumes -- to hold our customers' data. EBS volumes exist inside a specific availability zone, and machines in one zone can't connect to volumes in another. So, we worked out some clever tricks to move the data across beforehand.
We use a tool called DRBD to keep data safe. DRBD basically allows us to associate a backup server with each of our file servers. Every time you write to your private file storage in PythonAnywhere, it's written to a file server, which stores it on an EBS volume but also then also sends the data over to its associated backup server, which writes it to another EBS volume. This means that if the EBS volume on the file server goes wrong (possible, but unlikely) we always have a backup to switch over to.
So, in our previous release of PythonAnywhere last month, we moved all of the backup servers to us-east-1c. Over the course of a few hours after that move, all of our customers' data was copied over to 1c, and it was then kept in sync on an ongoing basis over the following weeks.
When we pushed our update today, we essentially started a new PythonAnywhere cluster in 1c, but the EBS volumes that were previously attached to the old cluster's backup servers were attached to the new cluster's file servers (after making sure that all pending updates had synced across). Because all updates had been replicated from the old file servers in 1a to these disks in 1c, this meant that we'd transparently migrated everything from 1a to 1c with minimal downtime.
That was the theory. And as far as it went, it worked flawlessly.
But we'd forgotten one thing. Not all customer data is on file servers that are backed up this way. The Dropbox storage, and storage for web app logs, is stored in a different way. So while we'd migrated everyone's normal files -- the stuff in /home/USERNAME, /tmp, and so on -- we'd not migrated the logs or the Dropbox shares. These were stuck in 1a, and we needed to move them to 1c so that they could be attached to the new servers there.
This was a problem. Without the migration trick using DRBD, the best way to move an EBS volume from one availability zone to another is to take a "snapshot" of it (which creates a copy that isn't associated with any specific zone) and then create a fresh volume from the snapshot in the appropriate zone. This is not a quick process. And the problem only became apparent to us when we were committed enough to the move to us-east-1c that undoing the migration would have been risky and slow.
So, 25 minutes into our upgrade, we started the snapshot process. We hoped it would be quick.
After 10 minutes of the snapshots of the Dropbox and the log storage data running, we noticed something worrying. The log storage snapshot was running at a reasonable speed. But the Dropbox storage snapshot hadn't even reached 1% yet. This is when we started talking to Amazon's tech support team. Unfortunately, after much discussion with them, it was determined that there was essentially nothing that could be done to speed up either of the snapshots.
After discussion internally we came to the conclusion that while the system couldn't run safely without the log storage having been migrated, we could run it without the Dropbox storage. We've deprecated Dropbox support recently due to problems with our connection to Dropbox themselves, so we don't think anyone's relying on it. So, we waited until the log storage snapshot completed (which took about 90 minutes), created a new EBS volume in us-east-1c, and brought the system back up.
Our apologies for the outage.
We released a new version of PythonAnywhere this morning. There were some nasty problems with the go-live (more about that later) but here's what we added:
Thanks to gregdelozier, Malcolm, robert, aaronzimmerman, Cartroo, barnsey, andmalc, corvax, giorgostzampanakis, dominochinese, stablum, algoqueue for the suggestions.
A minor release today, which included:
Happy coding everyone!
It's been about 6 months since we last delivered a state-of-the-PythonAnywhere address and, looking at everything that's happened since the last one, it's long-overdue.
Following the extremely ... mixed ... reaction to our upworthy/buzzfeed spoof report, we decided to gauge the reaction if we went in totally the opposite direction. So let's get straight into PythonAnywhere's very first newsletter of 2014 — the "style is for wimps" edition!
One of the biggest tech news items of the year so far is the discovery of the Heartbleed bug. Even though we (like almost every other Linux-based host) were vulnerable to it, we patched our servers just hours after it was announced to the public (for some reason we weren't included in the early disclosure group with Google and Facebook). Our users were safe from Heartbleed and they didn't have to do anything!
The reaction to our customer stories was so great last time that we've got 2 for you this time! You can read about
Yes, we heard your cries and you can now pay us using your credit card without involving PayPal in any part of the transaction.
Dismayed by the huge gulf between our $12 web developer accounts and our $99 startup accounts? We've got you covered — you can now design a pricing plan that exactly fits your unique requirements.
That is all — I did warn you that this would be a no-frills newsletter.
Some of our frenemies in the PaaS world, who shall remain nameless, offer a "git push" model for deployment. People are fond of it, and sometimes ask us whether they could do that on PythonAnywhere too.
The answer is: you totally can! Because PythonAnywhere is, at heart, just a web-based UI wrapped around a fully-featured Linux server environment, you can do lots and lots of things.
Here are the ingredients:
Here are the steps in detail:
(This guide assumes you already have a repo containing a web app which you want to deploy).
We create a directory, and make it into a "bare" git repo, ie, one that be
git push'd to. Let's say you've been working in a local repository called
mysite; you'd do this:
# From a Bash console on PythonAnywhere: mkdir -p ~/bare-repos/mysite.git cd !$ git init --bare ls # should show HEAD branches config...
(Technically there's no reason why the repo on PythonAnywhere needs to be your other repository's name with ".git" on the end, but we'll do that to keep things simple. You can call it something else if you want to, but it should end with ".git" -- that's the convention for bare repos.)
Next, navigate to your bare repository (in the file browser, or using a console-based editor like vim if you prefer), and create a new file at ~/bare-repos/mysite.git/hooks/post-receive. The name matters for this file!
#!/bin/bash mkdir -p /var/www/sites/mysite GIT_WORK_TREE=/var/www/sites/mysite git checkout -f
A bare repo doesn't have a working tree, it just stores your repository
in the magical git database format, with all the hashes and other voodoo.
git checkout -f along with the
GIT_WORK_TREE variable will tell
git to checkout an actual working tree with real code, to the specified
location. You can put this wherever you like..…
Now we just need to make the hook executable:
# In a Bash console on PythonAnywhere: chmod +x ~/bare-repos/mysite.git/hooks/post-receive
Back on your own PC, in the git repo for your web app, adding the remote is a single command, and then we can do our first push to it to make sure everything works:
# In your local code repo. Substitute in your own username + path git remote add pythonanywhere firstname.lastname@example.org:/home/myusername/bare-repos/mysite.git git push -u pythonanywhere master # output should end with "Branch master set up to track remote branch master from pythonanywhere."
You may get asked for your password at this point. If so, I strongly recommending setting up private key authentication instead, with a passphrase-encrypted private key on your machine, and adding your public key to ~/.ssh/authorized_keys on the server. There's a good guide here.
# From a Bash console on PythonAnywhere: ls /var/www/sites/mysite # should show your code!
If you want to use your own domain for this step, I'll assume you already have it set up on your registrar with a CNAME pointing at yourusername.pythonanywhere.com. More info.
We've now got our code on the server, let's set it up as a PythonAnywere web app. This should be a one-off operation.
Next, edit your WSGI file, and make it point at the wsgi app in your code. I was using a Django app, so mine looked like this:
# /var/www/www_mydomain_com_wsgi.py import os import sys path = '/var/www/sites/mysite' if path not in sys.path: sys.path.append(path) os.environ['DJANGO_SETTINGS_MODULE'] = 'superlists.settings' import django.core.handlers.wsgi application = django.core.handlers.wsgi.WSGIHandler()
Go back to the web app tab, hit "Reload", and make sure your site works. It took me a couple of goes to get it right!
NB: There are some subtleties here, like getting static files and your database config working, which I'll gloss over for now. See the example at the end for some inspiration though -- you can definitely get almost anything to work!
This is the last step -- we use a sekrit, undocumented feature of
PythonAnywhere, which is that our web workers actually watch the
WSGI file for changes, so if you
touch it, your web app will reload
So, back in ~/bare-repos/mysite.git/hooks/post-receive:
#!/bin/bash GIT_WORK_TREE=/var/www/sites/mysite git checkout -f touch /var/www/www_mydomain_com_wsgi.py
Substitute in the path to your own WSGI file.
Back on your own PC, make a trivial but visible change to your app, and then:
git commit -am"test change to see if push to pa works" git push pythonanywhere
It should just take a few seconds (although sometimes it takes as long as a minute) for the web worker to notice the touch to your web app, and then reload your site in your web browser... You should see your changes!
You can do pretty much anything you like in the post-receive, so for example:
#!/bin/bash set -e # exit if anything fails # run unit tests in a temp folder mkdir -p /tmp/testcheckout GIT_WORK_TREE=/tmp/testcheckout git checkout -f python3 /tmp/testcheckout/manage.py test lists rm -rf /tmp/testcheckout # checkout new code mkdir -p /var/www/sites/mysite GIT_WORK_TREE=/var/www/sites/mysite git checkout -f # update static files and database python3 /var/www/sites/mysite/manage.py collectstatic -y python3 /var/www/sites/mysite/manage.py migrate -y # bounce web app touch /var/www/test3_ottg_eu_wsgi.py
And so on. When you get it working, why not let us know! If this turns out to be popular, we may look to automate some of the steps as features...
Image credit: Human Pictogram 2.0 at pictogram2.com
Page 1 of 9.