PythonAnywhere News

The story of brer programmer and the tar-pit

harry — Mon, 13 May 2013 16:10:55 +0000

We're experimenting with different ways of balancing resource usage between our users. One problem we've been experiencing is that individual users could quite easily suck up almost all of the CPU power on a console server.

cgroups to the rescue!

We've set up some cgroups at:

/cgroup/cpuacct/users/$username

and

/cgroup/cpu/user_types/anon (150)
/cgroup/cpu/user_types/free  (300)
/cgroup/cpu/user_types/paying (700)
/cgroup/cpu/user_types/tarpit (2)

The cpuacct cgroup lets us keep track of how much CPU time each user is using. Then we use the cpu cgroup by setting a value called cpu.shares, the figure in brackets above, which defines what relative proportion of the processor the user is allowed, at busy times.

We then set a daily allowance for each user type -- currently, 100 seconds for free users, 5000 seconds for Hacker users, and 20,000 seconds for Web-Developer users. When a user goes over their daily limit, we move the into the tarpit. We then reset the counters once a day (at different times for each user).

The tarpit is used by the scheduler when the server gets busy -- so, if no-one else is using the server, you still get 100% of the CPU. But, once there is contention, your jobs will start to run much more slowly.

It's not a perfect solution, and sometimes we find ourselves needing to straight-out kill user processes when they're causing trouble for others (that only happens rarely though). But at least now, users will have some kind of visible warning that they're exceeding their resource allocation.

There's more info at www.pythonanywhere.com/tarpit/

We're keen on feedback! What do you think of this system? How would you improve it? What is it like, from the user's point of view?

Using PythonAnywhere in Chrome's App mode

hansel — Fri, 19 Apr 2013 12:11:12 +0000

Using Vim inside a PythonAnywhere console could be a bit annoying. A number of keyboard shortcuts are caught by the browser before they ever hit our site. Notably for Vim these include Ctrl-W and Ctrl-T.

But there is a better way! And it involves making Chrome open PythonAnywhere in it's special application mode. In this mode absolutely none of the keyboard shortcuts are captured. It's neat. Which is my reserved way of saying that it fixes EVERY problem I've had with PythonAnywhere and Vim.

I also imagine that those intrepid Emacs users will be happy to be able to send all those torturous escape sequences as well. Bless them.

Anyway, it looks like this:

Notice that all of Chrome's chrome is gone. It looks a bit like a native application. To get into this mode you have a couple of options. Firstly you can start chrome with the command line switch

"--app=https://www.pythonanywhere.com"

Or, even easier you can create a desktop shortcut via Chrome itself.

Simply

Visit http://www.pythonanywhere.com
Open the Chrome menu and select: Tools / Create Application Shortcuts...
Tick the locations you want the short cuts to appear

And there you have it. Now you can start an appified version of PythonAnywhere from your Start / Application menu and or desktop shortcut.

Try it out, it's made vim in a PythonAnywhere console indistinguishable from using it in a local console for me.

ps: If you are wondering what the hell that decorator is actually doing in the screenshot. It's something we use for testing. It's a decorator that decorates decorators and provides a set of the decorators used to decorate the function. It means we can easily assert that a function has been decorated by a particular decorator. Clear?

Loads more packages, AsciiDoc, plus paginated forums, finally!

harry — Thu, 21 Mar 2013 13:38:05 +0000

We've got several infrastructure stability + reliability improvements in this release, but more visible are some of the new binaries and packages.

Someone (I wonder who?) demanded a whole AsciiDoc toolchain to be able to write books with. Aside from that, you'll find:

pyCurl (thanks jquacinella)
pyquery (thanks chnrxn)
PyPDF2 (thanks madsquint)
pyunrar2 (thanks Brian)
django-openid-auth (thanks ferens_dima)
epydoc (thanks mdipierro + rochacbruno)
bashplotlib (we found that one ourselves - v cool!)
census (thanks Brett)
sqlalchemy-migrate (thanks )
Whoosh and flask-whooshalchemy (thanks )

You'll also find Twitter's Bower browser / front-end web-dev package management tool.

Other minor fixes include:

fixing an issue with restricted internet access in SSH
the "reload" button for web apps is now clearly labeled with the domain it reloads
Trailing spaces no longer break static files mappings
And the forums are finally paginated!

Let us know if everything looks OK!

How do I serve static files using django & PythonAnywhere?

hansel — Fri, 01 Mar 2013 15:56:17 +0000

Static files are the bits of your web site/application that are not created on the fly by Python. They might be images you want to embed, css files for styling, or even a PDF that you want people to be able to download.

If you use PythonAnywhere's Django quickstart then we already provide some sensible defaults which will get things working without you having to do anything. If however you are migrating an existing site or want to set things up manually then I will walk you through the process.

Out of the box Django needs static files for the admin interface. If you visit www.yoursite.com/admin and it doesn't have any styling then this is the problem. Django looks for these static files at www.yoursite.com/static/admin. So under the web tab of your PythonAnywhere dashboard you need to select the tab for your site and add your first entry for static files.

The url should be:/static/admin/

The directory (or path) should be: /usr/local/lib/python2.7/site-packages/django/contrib/admin/media

Next you probably want to serve some css or images out of a folder called static in your project folder. If your username was eliza and your Django project was called suitcase the following setup would work for you.

The url should be: /static/

The directory (or path) should be: /home/eliza/suitcase/static.

You resulting static files settings should now look something like this. Of course your username and project name should be different. That's the "eliza" and "suitcase" part of the path.

Any sub folders are also searched. So you could put your css files in a folder at /home/eliza/suitcase/static/css and a visitor going to http://www.yoursite.com/static/css/my.css would be sent the file /home/eliza/suitcase/static/css/my.css.

As always, remember that you will need to reload your web app before these changes actually take affect.

Happy Pythoning.

New release - fileserver fixes and a couple of tweaks

harry — Thu, 28 Feb 2013 05:22:53 +0000

Giles and I are just winding down following our early morning deploy -- it was a 3AM start for the two of us. Our best case was 20 minutes of downtime, our worst case prediction was about 3.5 hours. We came in at just under an hour and a half in the end. Definitely glad we did it so early in the morning.

The main reason for the deploy was an infrastructure upgrade, the fileserver has been upgraded from Debian to Ubuntu. It's been in testing for 3 weeks, and it should fix an intermittent issue with user quotas. Will follow up with more highly nerdy technical detail about XFS and NFS, later.

We still managed to slip in a couple of tiny improvements:

envsubst now works in chroot jails, which was otherwise causing some worrying messages for anyone trying a git rebase. But, to be honest, git rebase is terrifying at the best of times.
a small bugfix to the PythonAnywhere static files UI, where users with multiple webapps had duplicate "Add new entry" elements
And a fix to our Bottle.py quick-deploy wizard, which means templates now work out of the box

Anyone else had some fun with early starts recently? I don't think Giles even went to bed to be honest...

Image credit: London sunrise by A nosa disco necesítanos, on Flickr

The Python trademark in Europe

giles — Mon, 18 Feb 2013 15:19:57 +0000

Last Thursday, the Python Software Foundation announced that a UK-based company had applied for the exclusive European-wide trademark on the word Python as applied to software, web hosting, computer services, and so on.

Obviously we're a little concerned about this, and we recommend that anyone else who's using Python in a business in the EU should go to the PSF post linked above and see if there's anything they can do to help.

The company who's applied for the trademark (whose name is surprisingly hard to track down, but we'll call them POBox Hosting) provide a fairly standard kind of VPS solution, which they call "Python cloud servers". These servers may or may not contain Python-the-language as part of their setup, but it's also fairly obvious that support for Python-the-language is not a major selling point for them, and isn't the reason for their use of the name "Python". Instead, it looks like they chose that name because from time to time they'd used it for various other products in the past. They've owned the domain python.co.uk for at least the last ten years.

There are two ways in which their trademark is problematic.

Firstly, it is likely to confuse people -- it's easy to imagine someone hearing about Python the language, getting a "Python cloud server", and then wondering where all this great stuff they've heard about is. This could be a huge problem for the Python community as a whole -- if lots of people are using your trademark for reasons unrelated to the mark, then it is weakened and comes to mean almost nothing. The PSF have rules about how the Python trademark can be used, and they have to enforce them to make sure that "Python" in a computer software context continues to have a specific meaning. (As an aside, we use "Python" in our company name and elsewhere by agreement of the PSF; they're fine with it because (a) we asked and (b) it is "a permitted nominative use unlikely to cause confusion".)

Secondly, and much more immediately worryingly for us at PythonAnywhere -- and for any other EU-based business based around Python -- is the exclusivity of the trademark. Imagine if we had to describe what we did without using the word "Python"? "SomethingAnywhere is a development and hosting environment for a programming language that we can't name, that displays in your web browser and runs on our servers." Changing the company's name would be the least of our problems.

So, we strongly agree with the PSF's opposition to this trademark application. In the world of computing, "Python" invariably means Python-the-language. Any other use is confusing, and exclusive trademark on it to mean anything else would be extremely harmful to the Python community.

Just for anyone who's interested in a little more detail, here's the letter we sent the PSF expressing our support, which hopefully will be of use to them in their opposition to the trademark.

To whom it may concern,

We are writing to oppose the application by POBox Hosting for a trademark for the use of the word “Python” as it relates to computer software, servers and web hosting, CTM application number 010848208. In the computing world, the word “Python” is overwhelmingly used to refer to the Python programming language, and any other interpretation is hard to credit in that context.

We are PythonAnywhere LLP, a company based in the United Kingdom, offering computer software, servers and web hosting services worldwide to programmers who use the Python programming language. We use the word “Python” in our company name and when describing our products under license from the Python Software Foundation. A service that provides a “Python” service that is not specifically designed to incorporate the Python programming language harms our business by causing confusion amongst customers. An exclusive mark that prevented us from using the word “Python” to describe our business would essentially make us and other similar companies unable to operate in the EU.

Some background: the PythonAnywhere service was created by a company called Resolver Systems Ltd, which was founded in 2005, and it was acquired by PythonAnywhere LLP in a corporate restructure in late 2012 (the people are still the same). Members of the team behind Resolver Systems and PythonAnywhere first heard of the Python programming language in 1997, and have used it professionally in the European Union since that time.

When we founded Resolver Systems, our aim was to create a new, highly programmable spreadsheet. The Python programming language was the obvious choice for this, and when we released our first product, Resolver One, in 2007, we marketed it as “the Pythonic spreadsheet”. It has been downloaded by approximately 50,000 people worldwide, and approximately 40% of sales were in the EU.

PythonAnywhere, which we announced in March 2011, is a website where programmers who use the Python programming language can easily work on programming projects and host websites without needing to leave their web browser or install software. The customer signs up, clicks a button, and starts typing Python programs in their browser. All server management, maintenance and, for web developers, scalability, is handled by the system. This is extremely useful for Python developers, trainers and educators (we are used by a number of university courses), and for people developing and hosting their own Internet services and websites.

Since PythonAnywhere's launch, it has been visited by over 174,000 people worldwide, approximately 35% of whom are in the European Union. Around 15,000 of the people who have visited the site have registered to use it – again, about 35% of these are from the European Union.

We believe that when people look for “Python web hosting” or similar terms, they are looking for services like ours and like those of our competitors – that is, they are looking for a service that specialises in providing web hosting (or software, or servers) for people who use the Python programming language. On that basis we believe that POBox Hosting's “Python” product is confusing to consumers of Web hosting and software, and any trademark they gained over the term would damage the Web hosting sector in the EU as a whole.

It is worth noting that the Wikipedia page for the word “Python”, in English, French, German and Dutch, offers the Python programming language as one of the top possible meanings, but does not refer to POBox Hosting or their products. The Spanish, Italian, and Portuguese pages for the word Python are all specifically about the Python programming language. In the minds of Wikipedia's editors (which, the site being editable by anyone, is a good approximation to the minds of each language's speakers) Python in the context of computers clearly refers to the Python programming language. The customers for Web/Internet hosting and related services are computer specialists, and will therefore be confused by a trademark that does not relate to this language.

Furthermore, if web hosting and related companies that provide services that help developers who use the Python programming language specifically are prevented from using the word “Python” to describe their services by the applied-for trademark, they will be unable to operate.

A situation where a company who use Python in a sense different to the computing world at large have the exclusive right to use Python in the EU is clearly untenable.

A kickstarter book project

giles — Thu, 07 Feb 2013 17:48:31 +0000

We're proud to be sponsoring some of the rewards for Michael Herman's Kickstarter book project: Real Python for Web Development, featuring web2py. Michael has a great plan and solid experience, and web2py creator Massimo Di Pierro and Fletcher Heisler, the author of Real Python, are helping out as co-editors.

Michael's blown through his initial funding goal, raising $2,500 in less than 24 hours -- so now he's got some stretch goals. If he raises $10,000 then the book will also cover Flask (a widely-requested extra). And if it gets up to $15,000 or even $20,000, he'll add something else -- and he wants suggestions as to what. So even if you don't want to help fund the project, why not get involved and suggest something?

Static files -- finally!

harry — Thu, 07 Feb 2013 12:15:08 +0000

The big news for this release is proper static files support. Until now users either had to serve files via Python, using their web app framework, or use an undocumented and unsupported feature of hacking things into /var/www/static. Slightly embarassing.

So, we're pleased to present to you our first attempt at proper support for static files. Check out the Web Tab, and you'll find a new section allowing you to map URLs to folders in your patch -- these can be anywhere you like, they don't have to be in /var/www.

Let us know how you like it!

Bonus, secret update -- check out the editor. If you don't see anything new, that's probably good news, but if you do notice anything, it's probably a little warning message from PyFlakes, which should automatically detect some potential problems with your Python code, like syntax errors, unused imports and so on. Hope you find it helpful! (we all use it).

Thanks to everyone that's been chasing us to get static files in, including njr, x, econpy, kgullion, sterling312 and little_dood.

Outage report; or, the perils of using 502s to trigger web application initialisation

giles — Fri, 01 Feb 2013 11:22:19 +0000

Last weekend, we had two outages, both in the early hours (UK time) of the morning; we were finding it hard to diagnose the cause, because our outage alert system had failed and we only discovered it in each case when someone happened to check their email. On Monday we fixed the outage alert system.

Yesterday we had another, every phone in the office started beeping loudly, and we logged in right away -- so we know what made it happen. While we're not 100% sure that the cause of this problem was the same as the one on the weekend, the symptoms were similar enough that we're pretty sure that it was.

The symptom in all of these cases was that the main PythonAnywhere website was down, giving "502 Bad Gateway" errors. Over the weekend, all of the customer websites we checked were OK, but when it happened yesterday, we noticed that one specific customer website was extremely busy, and getting the same 502 error. Further investigation showed that it had been linked from the front page of Reddit and was getting about 100 hits/second. So that would explain why it was so busy -- and the site's author hadn't expected so much traffic so soon, so it was not super-optimised, which would explain why it couldn't handle the load.

But the question was, how was it affecting the main PythonAnywhere site? We have a lot of stuff in place to stop web applications from affecting each other, which is why the other user websites on the same server were up and running.

Explaining what happened requires a little background, both about how nginx works and about how we use it and uwsgi to serve large numbers of Python web applications.

nginx 50x errors

Our web servers run nginx as a front-end server. It delegates web requests to uwsgi instances that actually run the Python web applications. These requests are channeled between nginx and uwsgi via Unix domain sockets; each web application has a socket that is accessible both from outside our users' sandboxes (where the nginx server runs) and inside (where the uwsgi apps that serve each user's website live).

When a request comes in but the associated uwsgi server has a problem there are a number of possible errors. Firstly, the uwsgi app might be up and running, accept the incoming request, but then crash or take an unreasonably long time to process it. This leads to an error number 504, "Gateway timeout". This is what we normally expect to happen when a user's web app is under too much load to handle incoming requests. Another happens when the user's web app is just not up and running, so when nginx tries to put the request onto the socket for uwsgi, it can't. This causes a 502 "Bad gateway" error.

What we hadn't realised, or at least fully appreciated, is that there's a second kind of situation where you can get a 502 error. It's possible for a uwsgi server to be up and running, but so overloaded that from nginx's perspective it looks like it's down. These cause 502 errors too.

In this last case, you get an nginx error like this:

2013/01/31 15:15:42 [error] 4919#0: *1947447 connect() to unix:/var/sockets/www.example.com/socket failed (11: Resource temporarily unavailable) while connecting to upstream, client: XXX.XXX.XXX.XXX, server: www.example.com, request: "GET /something HTTP/1.1", upstream: "uwsgi://unix:/var/sockets/www.example.com/socket:", host: "www.example.com", referrer: "https://www.example.com/"

Now, if you google for that error you'll see lots of hints suggesting you increase the nginx proxy_connect_timeout setting, or the Linux net.core.netdev_max_backlog or net.core.somaxconn values. These can possibly help; essentially if the error is being caused by a very short-term spike in traffic, those changes will help smooth it out by letting more stuff queue up on the connection between the uwsgi server and nginx. But if the uwsgi server is just getting so much traffic that there's no way it can handle it, all changing the settings will do is make the queue take longer to fill up and cause the problem.

How we manage user web applications

Right, so by now it's probably pretty clear that the outage was caused by heavy traffic overloading a uwsgi server somewhere. But how did that take out our site?

Well, the problem is in how we start web applications. Each of our web servers can potentially be responsible for hundreds of web apps. We do not start uwsgi handlers for all of them when the server comes up, as this would bring the machine to its knees. So what we do is start them on demand; when a request comes in for a web application that isn't running, we detect that it's not running and use an "internal redirect" to push the request over to a part of the main PythonAnywhere web application that starts it up, then does another redirect back so that the requester sees the right page. Here's the nginx config for that:

server {

    listen      80;
    server_name  ~(?<domain>.+)$;
    location / {
        # various bits of config here, elided for clarity

        # if you get a 502 error, go to "fallback".
        error_page 502 = @fallback;
    }
    location @fallback {
        # In the fallback triggered by a 502, restart the user's web app.
        # FOR THE LOVE OF GOD, DON'T DO THIS!
        proxy_pass https://www.pythonanywhere.com/run_initialize_web_application_code/$scheme/$domain/$uri?$query_string;
    }
}

Sharp-eyed readers have probably worked out the problem by now. We were relying on the 502 "Bad gateway" errors to identify unstarted web applications; we'd assumed that unresponsive ones would always give us a 504 "Gateway timeout". This was a mistake. The customer website that was getting all of the traffic was causing 502s; and this meant that our setup was constantly telling the main PythonAnywhere app to restart the associated uswgi server. This was happening on the same server as handles our own site, dozens of times a second -- which, of course, meant that our site went down.

Lessons learned

The main lesson we've learned here is that we cannot use 502 errors to decide whether or not a user's uwsgi server is up and running. We'll blog about the fix for that later.

Another thing we may well look at is moving the "start a user's web application" code out of the main PythonAnywhere app. This would mean that even if our fix doesn't work in every case, it will not take down our site and the many people who use PythonAnywhere for running and developing non-web Python code in the cloud will be blissfully unaware of the problem.

So, any thoughts from our readers here? Is there anything we're missing in this analysis?

[UPDATE, 1 Feb 2013 @ 17:36]

We've now tested and pushed what we think is a solid fix.

Some background first. The uwsgi processes on a given server are all managed for us by a uwsgi master application. If a "vassal" file is is present for a web application (in /etc/uwsgi/vassals) then the application will be started by the uwsgi master, and if the web application crashes then the uwsgi master can be trusted to see this and re-start it.

So that means that we only need to start a user's web app if it gives a 502 error and there is no vassal file for it. If it gives a 502 and there is a vassal file, then the web app can be assumed to be running but having problems.

Which in turn means that all we needed to do was put a check in our nginx configuration to check for the vassal file, and only do the internal redirect to start the app if the vassal file isn't there. If the vassal file is there, it can just raise a 502 error as normal. The config looks like this:

server {

    listen      80;
    server_name  ~(?<domain>.+)$;
    location / {
        # various bits of config here, elided for clarity

        # if you get a 502 error, go to "fallback".
        error_page 502 = @fallback;
    }
    location @fallback {
        # In the fallback triggered by a 502, restart the user's web app
        # if and only if there is no vassal file for it.
        if (-f /etc/uwsgi/vassals/$domain.ini) {
            return 502;
        }
        proxy_pass https://www.pythonanywhere.com/run_initialize_web_application_code/$scheme/$domain/$uri?$query_string;
    }
}

Again, any thoughts much appreciated!

New release :-)

giles — Mon, 28 Jan 2013 13:45:17 +0000

We've just released a new version of PythonAnywhere. This is a scheduled release of a set of new features, and is not related to the weekend's outages (which are still under investigation, hopefully we'll have more information soon!).

The new stuff is:

We've upgraded our Python 3 version from 3.2 to 3.3.
Bugfix: if you quick-start a web2py app in a directory that is not ~/web2py, the admin password will now be correctly set.
Improved UI on iPads -- you can now hide the space that we reserve for the on-screen keyboard, which is useful if you're using a Bluetooth keyboard, or are just monitoring a log file.
New packages: R, RPy, gettext, geventhttpclient, npm (yes, that's right -- node.js on PythonAnywhere -- you can't run a node server yet though), PyStemmer, mysql.connector, asciidoc.

Thanks to all of the people who suggested these fixes, including marladarla7, pigeonflight, pwoolcoc, gebloom, djfinton, chanin, markstocks, jpic, ezamr, and Kernie_xvid,

Outages over the weekend

giles — Mon, 28 Jan 2013 10:33:14 +0000

Over the last weekend we had three separate outages; it appears that something on one of our servers was rapidly increasing the memory it was using and took down the main PythonAnywhere website. The majority of our users websites were unaffected.

We're investigating the problem and will post again with an update.

New release - more console stability improvements, unicode fix

harry — Wed, 09 Jan 2013 13:56:52 +0000

Today's new release features:

a fix to the file browser unicode bug. This was a regression whereby non-ascii characters in filenames caused problems. Shame on us! fixed now.
I'm counting chickens here, but we may have fixed one of our resource leaks, which means we should need to restart console servers less often. That should mean that your console sessions can stay live for longer
Improvements to load-balancing between console servers.
and finally, a bugfix that was causing errors in "Reload Web app".

Thanks to everyone that's been pestering/encouraging us to get these fixes and features in, and people who've reported bugs, including a2j, in4paschenko, btccharts, web2pyslices, lmsmodule , nim4n, SignumSol, viralto, hardsoft, xdanyelo, afinney, rudi, tanay009 + sansoftaus

Keep in touch!

Harry + the PythonAnywhere team.

WEBAPP ALL THE THINGS!

harry — Thu, 20 Dec 2012 15:14:59 +0000

Well, it's been weeks and weeks and weeks, but we've finally (finally) managed to get a version of MULTIPLE WEBAPPS out.

This is a soft launch, so all you avid blog and forum readers are finding out about it first. We'd love your feedback! Is it working OK? Can you break it? What should we improve?

Is it working OK? Can you break it? What should we improve?

Head on over to your Web Tab and take a look. You'll spot your web apps listed on the left - at this point, if you're a Hosting user (which we're now calling Web Developer) you'll probably have two: one for your <username>.pythonanywhere.com subdomain, and one for your own domain.

These are now totally independent, so you can delete your subdomain app if you want to, or replace it with something else. You can also start adding any other domains you own, using a little wizard we've built (with much sweating and cursing at JavaScript I might add).

Also, what do you think of our new pricing table? Do you think we've got the balance of features right, esp. re storage space + number of web apps? What do you think of our new account names (warning: only the answer "super-cool" will be accepted as valid feedback to that last.).

[image credit -- "map of the Internet" by Steve Jurveston]

Calling all LaTeX experts...

harry — Fri, 07 Dec 2012 12:22:57 +0000

We've just released a new version of PythonAnywhere which features a full suite of LaTex executables, available for you to run, in a browser, in our Bash consoles.

We've been experimenting with this for a couple of weeks, and we've had people in floods of gratitude, saying "YES! Never again will I have to install LaTeX! Thank you thank you thank you!".

So, perhaps it's of use to other people out there, even outside the world of Python? If you're a big LaTeX user, come check it out. Just sign up for a free account, start a Bash console and see what you can do. You can use dropbox, github or just a simple upload (in the Files menu) to get your documents uploaded.

We've basically installed a combination of the Debian texlive and ghostscript packages. If you're a Python fan, there are a few libraries to help as well, like texcaller and pdflatex which you may find useful too.

We crave feedback like prehistoric mammals crave an asteroid, so keep it coming!

Love, the PythonAnywhere team.

Faster file creation and saving

hansel — Wed, 28 Nov 2012 12:23:41 +0000

Hi all,

We deployed a new version this morning. The visible changes are

a speed up of file creation and file saving. This means that all of the most pressing issues around the file browser performance have now been fixed
The Python mimetypes module now works correctly

Progress on multiple web apps is looking good. We are rewriting the last couple of 'Quickstarts' right now.

Cheers

Even-even-fasterer file browsing and editing, woo

harry — Wed, 21 Nov 2012 12:16:10 +0000

We've just released a new version with some more improvements:

browsing files a tad faster
opening files for viewing + editing much faster
saving files still slow

We've also added a few packages:

Orange (thanks menemosyne1212)
flask-mail
Hg-Git (thanks kitsu_eb)
PyDataLog (thanks Pierre)
s3cmd (thanks basketcase)

More improvements on their way - speeding up saving files is next, and we're back to work on finishing multiple webapps, so hopefully that won't be long either.

Keep your feedback and suggestions coming!

Faster file viewing

giles — Mon, 19 Nov 2012 14:08:48 +0000

We've just pushed an updated version of PythonAnywhere! The biggest change this time around is that browsing through directories in the "Files" tab is much quicker.

Some background: in the older version of PythonAnywhere, every time you browsed a directory, launched a new process that ran inside your sandbox. This gave us great security pretty much for free -- as the file view was running in your sandbox, it was certain that you couldn't see anyone else's files. The problem was that as more and more people started using PythonAnywhere (yay!) more and more processes were being launched, and it got slower and slower (boo!). So we've had to bite the bullet and code our own secure sandboxed file browser. The first cut of this, which just uses the new system for browsing directories, is what we've just released. We're also working on using the same technique for viewing, saving and editing files -- that's coming soon.

There have also been a couple of minor tweaks: head and ping are now available from Bash consoles, and user web applications can now accept requests of 32kB, up from 4kB (which was too small for, in particular, web apps that wanted to use OpenID).

New release - a first crack at scaling issues

harry — Fri, 09 Nov 2012 13:39:04 +0000

Hi all,

As many of you know, we've been having some scaling issues in the last week or so. This new release is our first crack at addressing them.

Here's what we've managed:

running scheduled tasks on a different server type from consoles
implement more async / batch job processing of, eg, quota updates using beanstalk
optimise our console launchers and file browser views to remove blocking calls to cgroups binaries (these seemed to be real processor hogs)
more load balancing in uwsgi, including max limits on memory usage (1/2 a gig per worker) and request time (10 minutes)

It's been a fairly hectic couple of weeks! We're very conscious of unacceptable performance, since we use PythonAnywhere every day - in fact, there's a bit of a Catch-22, since PythonAnywhere being slow slows down our development of fixes to that slowness... Anyways, we've got there in the end, and we look forward to seeing how those improvements pan out. We're keeping a close eye on things for the next few days!

Lots of stuff still to do - we're half-way through some more major speedups to the file browser, and we've got lots more planned. And, of course, we're looking forward to getting back to work on the multiple-web-apps / Pro plans, which are tantalisingly 90% finished -- so we've only got the remaining 90% to do ;-)

Performance issues: What the problem is and what we are doing about it

hansel — Tue, 06 Nov 2012 12:00:56 +0000

Over the last couple of months our users have been noticing increasing performance issues. I think the most noticeable one is browsing directories and viewing files. Though the other problem is a slow down of console performance.

The file browser is currently slow because it launches a new user process which is chrooted and then inspects a user's files. This made for very easy and simple security and when a server is under light load it was actually quite quick. Under heavy load it grinds to a halt. Process start times balloon making file operations take forever for the user.

We are currently rewriting the code so that it no longer launches a sub-process. As of today we are about halfway through the total rewrite and plan on releasing it in stages once each segment passes through our testing loop.

The other main performance issue are slow consoles. By this I mean slow start times and laggy response to typing. Part of this problem is that scheduled tasks are run on the same server type as consoles. Scheduled tasks are very spiky so we've split those out to their own server type. Something that will help with latency while also making it easier to scale horizontally.

The slow start times for consoles is related to the file browser problem. Starting a console would launch a number of sub-processes to perform some admin tasks before actually launching the Python or Bash process. We've rewritten these functions so that this doesn't happen. We should be able to release these fixes soon as they are currently in testing.

Thanks to everyone for bearing with us, we appreciate the support. Fixing the performance problems have been our top priority and we will not stop working on it until it's fast for everyone.

All the best

From the PythonAnywhere team

New minor release

giles — Fri, 26 Oct 2012 11:45:31 +0000

We've just upgraded PythonAnywhere with a bunch of stability fixes. One nice new thing: if you create a new web2py app, you'll get version 2.0.9 -- and even better, the build-in "upgrade" button in the web2py admin view now works -- just click it, click OK, then once it's done, use the "Reload web app" button on the Web tab of your PythonAnywhere dashboard.