We're experimenting with different ways of balancing resource usage between our users. One problem we've been experiencing is that individual users could quite easily suck up almost all of the CPU power on a console server.

cgroups to the rescue!

We've set up some cgroups at:

/cgroup/cpuacct/users/$username

and

/cgroup/cpu/user_types/anon (150)
/cgroup/cpu/user_types/free  (300)
/cgroup/cpu/user_types/paying (700)
/cgroup/cpu/user_types/tarpit (2)

The cpuacct cgroup lets us keep track of how much CPU time each user is using. Then we use the cpu cgroup by setting a value called cpu.shares, the figure in brackets above, which defines what relative proportion of the processor the user is allowed, at busy times.

We then set a daily allowance for each user type -- currently, 100 seconds for free users, 5000 seconds for Hacker users, and 20,000 seconds for Web-Developer users. When a user goes over their daily limit, we move the into the tarpit. We then reset the counters once a day (at different times for each user).

The tarpit is used by the scheduler when the server gets busy -- so, if no-one else is using the server, you still get 100% of the CPU. But, once there is contention, your jobs will start to run much more slowly.

It's not a perfect solution, and sometimes we find ourselves needing to straight-out kill user processes when they're causing trouble for others (that only happens rarely though). But at least now, users will have some kind of visible warning that they're exceeding their resource allocation.

There's more info at www.pythonanywhere.com/tarpit/

We're keen on feedback! What do you think of this system? How would you improve it? What is it like, from the user's point of view?

Using Vim inside a PythonAnywhere console could be a bit annoying. A number of keyboard shortcuts are caught by the browser before they ever hit our site. Notably for Vim these include Ctrl-W and Ctrl-T.

But there is a better way! And it involves making Chrome open PythonAnywhere in it's special application mode. In this mode absolutely none of the keyboard shortcuts are captured. It's neat. Which is my reserved way of saying that it fixes EVERY problem I've had with PythonAnywhere and Vim.

I also imagine that those intrepid Emacs users will be happy to be able to send all those torturous escape sequences as well. Bless them.

Anyway, it looks like this:

Notice that all of Chrome's chrome is gone. It looks a bit like a native application. To get into this mode you have a couple of options. Firstly you can start chrome with the command line switch

"--app=https://www.pythonanywhere.com"

Or, even easier you can create a desktop shortcut via Chrome itself.

Simply

1. Visit http://www.pythonanywhere.com
2. Open the Chrome menu and select: Tools / Create Application Shortcuts...
3. Tick the locations you want the short cuts to appear

And there you have it. Now you can start an appified version of PythonAnywhere from your Start / Application menu and or desktop shortcut.

Try it out, it's made vim in a PythonAnywhere console indistinguishable from using it in a local console for me.

ps: If you are wondering what the hell that decorator is actually doing in the screenshot. It's something we use for testing. It's a decorator that decorates decorators and provides a set of the decorators used to decorate the function. It means we can easily assert that a function has been decorated by a particular decorator. Clear?

We've got several infrastructure stability + reliability improvements in this release, but more visible are some of the new binaries and packages.

Someone (I wonder who?) demanded a whole AsciiDoc toolchain to be able to write books with. Aside from that, you'll find:

• pyCurl (thanks jquacinella)
• pyquery (thanks chnrxn)
• PyPDF2 (thanks madsquint)
• pyunrar2 (thanks Brian)
• django-openid-auth (thanks ferens_dima)
• epydoc (thanks mdipierro + rochacbruno)
• bashplotlib (we found that one ourselves - v cool!)
• census (thanks Brett)
• sqlalchemy-migrate (thanks )
• Whoosh and flask-whooshalchemy (thanks )

You'll also find Twitter's Bower browser / front-end web-dev package management tool.

Other minor fixes include:

• fixing an issue with restricted internet access in SSH
• the "reload" button for web apps is now clearly labeled with the domain it reloads
• Trailing spaces no longer break static files mappings
• And the forums are finally paginated!

Let us know if everything looks OK!

Static files are the bits of your web site/application that are not created on the fly by Python. They might be images you want to embed, css files for styling, or even a PDF that you want people to be able to download.

If you use PythonAnywhere's Django quickstart then we already provide some sensible defaults which will get things working without you having to do anything. If however you are migrating an existing site or want to set things up manually then I will walk you through the process.

Out of the box Django needs static files for the admin interface. If you visit www.yoursite.com/admin and it doesn't have any styling then this is the problem. Django looks for these static files at www.yoursite.com/static/admin. So under the web tab of your PythonAnywhere dashboard you need to select the tab for your site and add your first entry for static files.

The url should be:/static/admin/

The directory (or path) should be: /usr/local/lib/python2.7/site-packages/django/contrib/admin/media

Next you probably want to serve some css or images out of a folder called static in your project folder. If your username was eliza and your Django project was called suitcase the following setup would work for you.

The url should be: /static/

The directory (or path) should be: /home/eliza/suitcase/static.

You resulting static files settings should now look something like this. Of course your username and project name should be different. That's the "eliza" and "suitcase" part of the path.

Any sub folders are also searched. So you could put your css files in a folder at /home/eliza/suitcase/static/css and a visitor going to http://www.yoursite.com/static/css/my.css would be sent the file /home/eliza/suitcase/static/css/my.css.

As always, remember that you will need to reload your web app before these changes actually take affect.

Happy Pythoning.

Giles and I are just winding down following our early morning deploy -- it was a 3AM start for the two of us. Our best case was 20 minutes of downtime, our worst case prediction was about 3.5 hours. We came in at just under an hour and a half in the end. Definitely glad we did it so early in the morning.

The main reason for the deploy was an infrastructure upgrade, the fileserver has been upgraded from Debian to Ubuntu. It's been in testing for 3 weeks, and it should fix an intermittent issue with user quotas. Will follow up with more highly nerdy technical detail about XFS and NFS, later.

We still managed to slip in a couple of tiny improvements:

• envsubst now works in chroot jails, which was otherwise causing some worrying messages for anyone trying a git rebase. But, to be honest, git rebase is terrifying at the best of times.
• a small bugfix to the PythonAnywhere static files UI, where users with multiple webapps had duplicate "Add new entry" elements
• And a fix to our Bottle.py quick-deploy wizard, which means templates now work out of the box

Anyone else had some fun with early starts recently? I don't think Giles even went to bed to be honest...

Image credit: London sunrise by A nosa disco necesítanos, on Flickr

Last Thursday, the Python Software Foundation announced that a UK-based company had applied for the exclusive European-wide trademark on the word Python as applied to software, web hosting, computer services, and so on.

Obviously we're a little concerned about this, and we recommend that anyone else who's using Python in a business in the EU should go to the PSF post linked above and see if there's anything they can do to help.

The company who's applied for the trademark (whose name is surprisingly hard to track down, but we'll call them POBox Hosting) provide a fairly standard kind of VPS solution, which they call "Python cloud servers". These servers may or may not contain Python-the-language as part of their setup, but it's also fairly obvious that support for Python-the-language is not a major selling point for them, and isn't the reason for their use of the name "Python". Instead, it looks like they chose that name because from time to time they'd used it for various other products in the past. They've owned the domain python.co.uk for at least the last ten years.

There are two ways in which their trademark is problematic.

Firstly, it is likely to confuse people -- it's easy to imagine someone hearing about Python the language, getting a "Python cloud server", and then wondering where all this great stuff they've heard about is. This could be a huge problem for the Python community as a whole -- if lots of people are using your trademark for reasons unrelated to the mark, then it is weakened and comes to mean almost nothing. The PSF have rules about how the Python trademark can be used, and they have to enforce them to make sure that "Python" in a computer software context continues to have a specific meaning. (As an aside, we use "Python" in our company name and elsewhere by agreement of the PSF; they're fine with it because (a) we asked and (b) it is "a permitted nominative use unlikely to cause confusion".)

Secondly, and much more immediately worryingly for us at PythonAnywhere -- and for any other EU-based business based around Python -- is the exclusivity of the trademark. Imagine if we had to describe what we did without using the word "Python"? "SomethingAnywhere is a development and hosting environment for a programming language that we can't name, that displays in your web browser and runs on our servers." Changing the company's name would be the least of our problems.

So, we strongly agree with the PSF's opposition to this trademark application. In the world of computing, "Python" invariably means Python-the-language. Any other use is confusing, and exclusive trademark on it to mean anything else would be extremely harmful to the Python community.

Just for anyone who's interested in a little more detail, here's the letter we sent the PSF expressing our support, which hopefully will be of use to them in their opposition to the trademark.

To whom it may concern,

We are writing to oppose the application by POBox Hosting for a trademark for the use of the word “Python” as it relates to computer software, servers and web hosting, CTM application number 010848208. In the computing world, the word “Python” is overwhelmingly used to refer to the Python programming language, and any other interpretation is hard to credit in that context.

We are PythonAnywhere LLP, a company based in the United Kingdom, offering computer software, servers and web hosting services worldwide to programmers who use the Python programming language. We use the word “Python” in our company name and when describing our products under license from the Python Software Foundation. A service that provides a “Python” service that is not specifically designed to incorporate the Python programming language harms our business by causing confusion amongst customers. An exclusive mark that prevented us from using the word “Python” to describe our business would essentially make us and other similar companies unable to operate in the EU.

Some background: the PythonAnywhere service was created by a company called Resolver Systems Ltd, which was founded in 2005, and it was acquired by PythonAnywhere LLP in a corporate restructure in late 2012 (the people are still the same). Members of the team behind Resolver Systems and PythonAnywhere first heard of the Python programming language in 1997, and have used it professionally in the European Union since that time.

When we founded Resolver Systems, our aim was to create a new, highly programmable spreadsheet. The Python programming language was the obvious choice for this, and when we released our first product, Resolver One, in 2007, we marketed it as “the Pythonic spreadsheet”. It has been downloaded by approximately 50,000 people worldwide, and approximately 40% of sales were in the EU.

PythonAnywhere, which we announced in March 2011, is a website where programmers who use the Python programming language can easily work on programming projects and host websites without needing to leave their web browser or install software. The customer signs up, clicks a button, and starts typing Python programs in their browser. All server management, maintenance and, for web developers, scalability, is handled by the system. This is extremely useful for Python developers, trainers and educators (we are used by a number of university courses), and for people developing and hosting their own Internet services and websites.

Since PythonAnywhere's launch, it has been visited by over 174,000 people worldwide, approximately 35% of whom are in the European Union. Around 15,000 of the people who have visited the site have registered to use it – again, about 35% of these are from the European Union.

We believe that when people look for “Python web hosting” or similar terms, they are looking for services like ours and like those of our competitors – that is, they are looking for a service that specialises in providing web hosting (or software, or servers) for people who use the Python programming language. On that basis we believe that POBox Hosting's “Python” product is confusing to consumers of Web hosting and software, and any trademark they gained over the term would damage the Web hosting sector in the EU as a whole.

It is worth noting that the Wikipedia page for the word “Python”, in English, French, German and Dutch, offers the Python programming language as one of the top possible meanings, but does not refer to POBox Hosting or their products. The Spanish, Italian, and Portuguese pages for the word Python are all specifically about the Python programming language. In the minds of Wikipedia's editors (which, the site being editable by anyone, is a good approximation to the minds of each language's speakers) Python in the context of computers clearly refers to the Python programming language. The customers for Web/Internet hosting and related services are computer specialists, and will therefore be confused by a trademark that does not relate to this language.

Furthermore, if web hosting and related companies that provide services that help developers who use the Python programming language specifically are prevented from using the word “Python” to describe their services by the applied-for trademark, they will be unable to operate.

A situation where a company who use Python in a sense different to the computing world at large have the exclusive right to use Python in the EU is clearly untenable.

We're proud to be sponsoring some of the rewards for Michael Herman's Kickstarter book project: Real Python for Web Development, featuring web2py. Michael has a great plan and solid experience, and web2py creator Massimo Di Pierro and Fletcher Heisler, the author of Real Python, are helping out as co-editors.

Michael's blown through his initial funding goal, raising $2,500 in less than 24 hours -- so now he's got some stretch goals. If he raises $10,000 then the book will also cover Flask (a widely-requested extra). And if it gets up to $15,000 or even $20,000, he'll add something else -- and he wants suggestions as to what. So even if you don't want to help fund the project, why not get involved and suggest something?

The big news for this release is proper static files support. Until now users either had to serve files via Python, using their web app framework, or use an undocumented and unsupported feature of hacking things into /var/www/static. Slightly embarassing.

So, we're pleased to present to you our first attempt at proper support for static files. Check out the Web Tab, and you'll find a new section allowing you to map URLs to folders in your patch -- these can be anywhere you like, they don't have to be in /var/www.

Let us know how you like it!

Bonus, secret update -- check out the editor. If you don't see anything new, that's probably good news, but if you do notice anything, it's probably a little warning message from PyFlakes, which should automatically detect some potential problems with your Python code, like syntax errors, unused imports and so on. Hope you find it helpful! (we all use it).

Thanks to everyone that's been chasing us to get static files in, including njr, x, econpy, kgullion, sterling312 and little_dood.

Last weekend, we had two outages, both in the early hours (UK time) of the morning; we were finding it hard to diagnose the cause, because our outage alert system had failed and we only discovered it in each case when someone happened to check their email. On Monday we fixed the outage alert system.

Yesterday we had another, every phone in the office started beeping loudly, and we logged in right away -- so we know what made it happen. While we're not 100% sure that the cause of this problem was the same as the one on the weekend, the symptoms were similar enough that we're pretty sure that it was.

The symptom in all of these cases was that the main PythonAnywhere website was down, giving "502 Bad Gateway" errors. Over the weekend, all of the customer websites we checked were OK, but when it happened yesterday, we noticed that one specific customer website was extremely busy, and getting the same 502 error. Further investigation showed that it had been linked from the front page of Reddit and was getting about 100 hits/second. So that would explain why it was so busy -- and the site's author hadn't expected so much traffic so soon, so it was not super-optimised, which would explain why it couldn't handle the load.

But the question was, how was it affecting the main PythonAnywhere site? We have a lot of stuff in place to stop web applications from affecting each other, which is why the other user websites on the same server were up and running.

Explaining what happened requires a little background, both about how nginx works and about how we use it and uwsgi to serve large numbers of Python web applications.

nginx 50x errors

Our web servers run nginx as a front-end server. It delegates web requests to uwsgi instances that actually run the Python web applications. These requests are channeled between nginx and uwsgi via Unix domain sockets; each web application has a socket that is accessible both from outside our users' sandboxes (where the nginx server runs) and inside (where the uwsgi apps that serve each user's website live).

When a request comes in but the associated uwsgi server has a problem there are a number of possible errors. Firstly, the uwsgi app might be up and running, accept the incoming request, but then crash or take an unreasonably long time to process it. This leads to an error number 504, "Gateway timeout". This is what we normally expect to happen when a user's web app is under too much load to handle incoming requests. Another happens when the user's web app is just not up and running, so when nginx tries to put the request onto the socket for uwsgi, it can't. This causes a 502 "Bad gateway" error.

What we hadn't realised, or at least fully appreciated, is that there's a second kind of situation where you can get a 502 error. It's possible for a uwsgi server to be up and running, but so overloaded that from nginx's perspective it looks like it's down. These cause 502 errors too.

In this last case, you get an nginx error like this:

2013/01/31 15:15:42 [error] 4919#0: *1947447 connect() to unix:/var/sockets/www.example.com/socket failed (11: Resource temporarily unavailable) while connecting to upstream, client: XXX.XXX.XXX.XXX, server: www.example.com, request: "GET /something HTTP/1.1", upstream: "uwsgi://unix:/var/sockets/www.example.com/socket:", host: "www.example.com", referrer: "https://www.example.com/"

Now, if you google for that error you'll see lots of hints suggesting you increase the nginx proxy_connect_timeout setting, or the Linux net.core.netdev_max_backlog or net.core.somaxconn values. These can possibly help; essentially if the error is being caused by a very short-term spike in traffic, those changes will help smooth it out by letting more stuff queue up on the connection between the uwsgi server and nginx. But if the uwsgi server is just getting so much traffic that there's no way it can handle it, all changing the settings will do is make the queue take longer to fill up and cause the problem.

How we manage user web applications

Right, so by now it's probably pretty clear that the outage was caused by heavy traffic overloading a uwsgi server somewhere. But how did that take out our site?

Well, the problem is in how we start web applications. Each of our web servers can potentially be responsible for hundreds of web apps. We do not start uwsgi handlers for all of them when the server comes up, as this would bring the machine to its knees. So what we do is start them on demand; when a request comes in for a web application that isn't running, we detect that it's not running and use an "internal redirect" to push the request over to a part of the main PythonAnywhere web application that starts it up, then does another redirect back so that the requester sees the right page. Here's the nginx config for that:

server {

    listen      80;
    server_name  ~(?<domain>.+)$;
    location / {
        # various bits of config here, elided for clarity

        # if you get a 502 error, go to "fallback".
        error_page 502 = @fallback;
    }
    location @fallback {
        # In the fallback triggered by a 502, restart the user's web app.
        # FOR THE LOVE OF GOD, DON'T DO THIS!
        proxy_pass https://www.pythonanywhere.com/run_initialize_web_application_code/$scheme/$domain/$uri?$query_string;
    }
}

Sharp-eyed readers have probably worked out the problem by now. We were relying on the 502 "Bad gateway" errors to identify unstarted web applications; we'd assumed that unresponsive ones would always give us a 504 "Gateway timeout". This was a mistake. The customer website that was getting all of the traffic was causing 502s; and this meant that our setup was constantly telling the main PythonAnywhere app to restart the associated uswgi server. This was happening on the same server as handles our own site, dozens of times a second -- which, of course, meant that our site went down.

Lessons learned

The main lesson we've learned here is that we cannot use 502 errors to decide whether or not a user's uwsgi server is up and running. We'll blog about the fix for that later.

Another thing we may well look at is moving the "start a user's web application" code out of the main PythonAnywhere app. This would mean that even if our fix doesn't work in every case, it will not take down our site and the many people who use PythonAnywhere for running and developing non-web Python code in the cloud will be blissfully unaware of the problem.

So, any thoughts from our readers here? Is there anything we're missing in this analysis?

[UPDATE, 1 Feb 2013 @ 17:36]

We've now tested and pushed what we think is a solid fix.

Some background first. The uwsgi processes on a given server are all managed for us by a uwsgi master application. If a "vassal" file is is present for a web application (in /etc/uwsgi/vassals) then the application will be started by the uwsgi master, and if the web application crashes then the uwsgi master can be trusted to see this and re-start it.

So that means that we only need to start a user's web app if it gives a 502 error and there is no vassal file for it. If it gives a 502 and there is a vassal file, then the web app can be assumed to be running but having problems.

Which in turn means that all we needed to do was put a check in our nginx configuration to check for the vassal file, and only do the internal redirect to start the app if the vassal file isn't there. If the vassal file is there, it can just raise a 502 error as normal. The config looks like this:

server {

    listen      80;
    server_name  ~(?<domain>.+)$;
    location / {
        # various bits of config here, elided for clarity

        # if you get a 502 error, go to "fallback".
        error_page 502 = @fallback;
    }
    location @fallback {
        # In the fallback triggered by a 502, restart the user's web app
        # if and only if there is no vassal file for it.
        if (-f /etc/uwsgi/vassals/$domain.ini) {
            return 502;
        }
        proxy_pass https://www.pythonanywhere.com/run_initialize_web_application_code/$scheme/$domain/$uri?$query_string;
    }
}

Again, any thoughts much appreciated!

We've just released a new version of PythonAnywhere. This is a scheduled release of a set of new features, and is not related to the weekend's outages (which are still under investigation, hopefully we'll have more information soon!).

The new stuff is:

• We've upgraded our Python 3 version from 3.2 to 3.3.
• Bugfix: if you quick-start a web2py app in a directory that is not ~/web2py, the admin password will now be correctly set.
• Improved UI on iPads -- you can now hide the space that we reserve for the on-screen keyboard, which is useful if you're using a Bluetooth keyboard, or are just monitoring a log file.
• New packages: R, RPy, gettext, geventhttpclient, npm (yes, that's right -- node.js on PythonAnywhere -- you can't run a node server yet though), PyStemmer, mysql.connector, asciidoc.

Thanks to all of the people who suggested these fixes, including marladarla7, pigeonflight, pwoolcoc, gebloom, djfinton, chanin, markstocks, jpic, ezamr, and Kernie_xvid,