The PythonAnywhere newsletter, September 2018

Well, our last "monthly" newsletter was in September 2017. We must have shifted the bits in the period left one, or something like that :-)

Anyway, welcome to the September 2018 PythonAnywhere newsletter :-) Here's what we've been up to.

Python 3.7 support

We recently added support for Python 3.7. If you signed up since 28 August, you'll have it available on your account -- you can use it just like any other Python version.

If you signed up before then, it's a little more complicated, but we can update your account to provide it -- there's more information in this blog post.

Self-installation of HTTPS certificates

We've also been working on making setting up HTTPS on your website a bit more streamlined. Previously you had to get the certificate and the private key, and then email us asking for them to be installed, which could take up to 24 hours. Now you can cut our support team out of the loop and install it all yourself. Check out this blog post for the details.

There will be more improvements to HTTPS support coming soon...

Force HTTPS

Another shiny new feature: built-in support for forcing people who visit your site to use HTTPS instead of non-secure HTTP, without the need to change your code! Once again, there's more info on the blog.

PythonAnywhere metrics

We're wondering if it would be interesting for you to hear a bit about some of the metrics we monitor internally to see what's happening in our systems. Here's a random grab-bag of some numbers for this month:

  • Web requests: we're processing on average about 225 hits/second through our systems (across all websites) with spikes at busy times of up to 350/second. For comparison -- apparently that's about what Stack Overflow have to deal with. But there's a difference; they're just one site, but for us...
  • That's across about 28,000 websites. Of course the number of hits sites get is very much spread over a long tail distribution -- many of those sites are ones that people set up as part of tutorials (like the excellent Django Girls), so they only get hits from their owners, while on the other hand the busiest websites might be processing 40 hits/second at their peak times
  • By contrast, there are only 10,000 scheduled tasks :-S
  • Our live system currently comprises 51 separate machines on Amazon AWS.

Let us know whether those are the metrics you'd like to see, whether you'd like to see more, or if you think it's completely uninteresting :-)

GDPR

Like every tech business in the world, we spent a lot of time late last year and early this year working on ensuring that we were compliant with the GDPR. This was doubly-important to us -- most companies are "data controllers" in GDPR terminology, which means that they store data about people, but we are also "data processors", which means that we run computers and programs that other people use in their role as data controllers. Or, to make that a bit more concrete -- if you have personal data about people on a website that you host with us, you're a data controller, and you're delegating the data processing to us.

This of course, meant that it was more than twice as much work for us as it was for most people, but we got it all done a week before the deadline :-)

One interesting side-effect of all of this was that we realised that these newsletters sometime say things like "hey, we've added this cool new feature (for paid accounts only)" -- and when they do say something like that, it's not unreasonable to see them as marketing. We had to make super-sure that all marketing messages from us were opt-in only, so we unsubscribed everyone from the newsletter and put up a banner on login so that people would know about it.

That in turn means that this newsletter is going to about ten times fewer people than the one before -- so if you're reading it over email, thanks for choosing to receive it :-) Of course, you can always unsubscribe using the link at the bottom of the message, or from the email settings tab on the Account page.

New modules

Although you can install Python packages on PythonAnywhere yourself, we like to make sure that we have plenty of batteries included.

Everything got updated for the new system image that provides access to Python 3.7, so if you're using that image, you should have the most recent (or at least a very recent) version of everything :-)

New whitelisted sites

Paying PythonAnywhere customers get unrestricted Internet access, but if you're a free PythonAnywhere user, you may have hit problems when writing code that tries to access sites elsewhere on the Internet. We have to restrict you to sites on a whitelist to stop hackers from creating dummy accounts to hide their identities when breaking into other people's websites.

But we really do encourage you to suggest new sites that should be on the whitelist. Our rule is, if it's got an official public API, which means that the site's owners are encouraging automated access to their server, then we'll whitelist it. Just drop us a line with a link to the API docs.

We've added too many sites to list since our last newsletter to list them all -- but please keep them coming!

That's all for now

That's all we've got this time around. We have some big new features in the pipeline, so keep tuned! Maybe we'll even get our next newsletter out in October 2018. Or at least sometime in 2018...


Force HTTPS on your website

One smaller feature we added in our last system update was the ability to force HTTPS without needing to change your code. Here's a bit more information about why you might want to do that, and how it works.

Why force HTTPS?

If you're using your default yourusername.pythonanywhere.com website on PythonAnywhere, people can access it using a non-secure connection via http://yourusername.pythonanywhere.com, or using a secure connection via https://yourusername.pythonanywhere.com. Likewise, if you have a paid account and are using a custom domain, they can access it non-securely via http://www.yourdomain.com or -- if you've set up an HTTPS certificate -- they can access it securely via https://www.yourdomain.com.

Having two ways to access the site is fine, but these days there's a general move across the Internet to using HTTPS for everything. Newer versions of Chrome explicitly put the words "Not secure" in the browser's address bar when you access a non-HTTPS site, and Google also apparently rank HTTPS sites higher in search results.

So often, what you want to do is set things up so that people can only use HTTPS to access your site -- never simple non-secure HTTP. This is called "forcing HTTPS", and works by sending back a redirect when people try to access the non-secure URL. So if someone goes to http://www.yourdomain.com, they're just bounced straight over to https://www.yourdomain.com, and likewise if they go to http://www.yourdomain.com/something they're redirected to https://www.yourdomain.com/something.

How it used to work

Because forcing HTTPS is a common requirement, most of the main Python web frameworks have support for it build in -- Django has a SECURE_SSL_REDIRECT setting, for example, and there's a Flask extension called flask-sslify. In the past, we suggested that people add the appropriate configuration to their code to use those features, and it all worked fine.

There were just two problems:

  • If you use the static files system on the "Web" tab, it picks up any requests before they get anywhere near your code. So while any requests that went to your view code were redirected to HTTPS, your static assets would still be served over plain HTTP if that was the kind of request made by the browser.
  • Putting this kind of protocol-level stuff in the Python code just feels like it's happening at the wrong layer of the application. In particular, if you're testing locally then perhaps you don't want HTTPS to be forced there -- and while the framework plugins generally have support for only forcing it in non-debug environments, it can be a little fiddly.

How it works now

We made a bunch of changes to the way our loadbalancers work as part of our recent system update, primarily in order to improve the way we handle HTTPS certificates. Because we were working in that part of the code, it was easy for us to add the capability to force HTTPS to our own infrastructure. So now, in order to force HTTPS for your website, just go to the "Web" page inside PythonAnywhere, select the site on the left, scroll down to the "Security" section, and toggle it on:

You'll need to reload the site using the button at the top of the page to activate it.

Caveats

There are a couple of things to keep in mind when using this feature.

  • Most importantly, if you have a custom domain, you'll need to make sure you keep your HTTPS certificate updated and that it does not expire. This is because every person coming to your site will be sent to the secure version -- and if the cert has expired, they'll get a security warning, which won't look good!
  • A redirect message from a server only includes the URL to redirect to -- not, for example, any POSTed data. This means that we can't do a force-HTTPS redirect in response to a POST request, because the data would get lost. (It also wouldn't be super-valuable -- if the user has POSTed data, then it's already gone across the Internet unencrypted, so sending it again in encrypted form wouldn't really help matters.)

Any questions?

Hopefully that was all pretty clear, but if you do have any questions, just drop us a line at support@pythonanywhere.com or leave a comment below.


Python 3.7 now available!

If you signed up since 28 August, you'll have Python 3.7 available on your account -- you can use it just like any other Python version.

If you signed up before then, it's a little more complicated, because adding Python 3.7 to your account requires changing your system image. Each account has an associated system image, which determines which Python versions, Python packages, operating system packages, and so on are available. The new image is called "earlgrey" (after the previous system images, "classic" and "dangermouse").

What this means is that if we change your system image, the pre-installed Python packages will all get upgraded, which means that any code you have that depends on them might stop working if it's not compatible with the new versions.

For previous upgrades, from classic to dangermouse, we said that all was OK if you were using a virtualenv, which (of course) removes the dependency on the pre-installed Python packages. However, this new image not only adds Python 3.7 and upgrades packages, it also upgrades the older Python versions -- for example, from the antediluvian 2.7.6 to 2.7.12, and from 3.6.0 to 3.6.6. This can break virtualenvs in some cases.

So, long story short -- we can switch your account over to the new system image, but you may need to rebuild your virtualenvs afterwards if you're using them -- and you may need to update your code to handle newer pre-installed Python packages if you're not using virtualenvs.

There are more details about exactly which package versions are included in which system image on the batteries included page. And if you'd like to switch your account over to earlgrey, just drop us a line using the "Send feedback" button. (If you've read all of the above, and understand that you may have to make code/virtualenv changes, mention that you have in the feedback message as otherwise we'll respond by basically repeating all of the stuff we just said, and asking "are you sure?")


New feature: self-installation of SSL certificates!

Our system update last week added on an API to let you install HTTP certificates yourself instead of having to email us. We've been beta-testing it over the last seven days, and it's now ready to go live :-)

You can either use it by accessing the API directly from your code, or by using our helper scripts (which you can pip install).

This is the first step towards a much improved system for HTTPS -- watch this space for more information.

What this is all about

We supply an HTTPS certificate for all websites that are subdomains of pythonanywhere.com -- so your website at yourusername.pythonanywhere.com handles HTTPS by default. But if you create a website with a custom domain, you need to get a certificate. This is because only the owner of a domain can create a certificate for it, to stop people from (say) creating certs for www.google.com.

Luckily, it's easy enough to create a certificate for your site, either by using the excellent Let's Encrypt (for which we have detailed instructions here) or from one of the many commercial providers.

But once you've created your certificate, you need to get it installed. In the past, the way we asked you to do that was to store the certificate and the private key somewhere in your PythonAnywhere file storage, then email us. The system worked well, but it did have one problem -- you needed to wait for us to install the certificate, which meant that in the worst case you could wait for up to 24 hours for it to be installed -- maybe not a huge issue when you're just getting started, but a big deal if you accidentally let the certificate expire and need a renewed one installed ASAP.

How to use it: the simple way

The easiest way to install your certificate is to use the PythonAnywhere helper scripts. The first step to do that is to make sure you have an API token set up for your account; go to the "Account" page and then click the "API token" tab. If you see this:

...then you're all set. If, however, you see this:

...then you need to click the button to generate a key.

Once you've done that, start a new Bash console, and run this to install the PythonAnywhere helper scripts:

pip3.6 install --user --upgrade pythonanywhere

(If you're on our "classic" image and don't have Python 3.6 available, you can use pip3.5 instead.)

Now you can run the script to install the certificate. If you've used our instructions for Let's Encrypt then the certificate and the key are in a well-known place, so you can just do this:

pa_install_webapp_letsencrypt_ssl.py www.yourdomain.com

If you've got a certificate from a different provider, you can specify the combined certificate and the key location by using a different command:

pa_install_webapp_ssl.py www.yourdomain.com /home/yourusername/something/combined-cert.pem /home/yourusername/something/private-key.pem

...adjusting the paths to point to the appropriate files.

If all goes well, you'll see output like this:

< Setting up SSL for www.yourdomain.com via API >
   \
    ~<:>>>>>>>>>
< Reloading www.yourdomain.com via API >
   \
    ~<:>>>>>>>>>
  _________________________________________________________________
/                                                                   \
| That's all set up now :-) Your new certificate will expire         |
| on 12 November 2018, so shortly before then you should             |
| renew it (see https://help.pythonanywhere.com/pages/LetsEncrypt/)  |
| and install the new certificate.                                   |
\                                                                   /
  -----------------------------------------------------------------
   \
    ~<:>>>>>>>>>

If you're not using Let's Encrypt, it will look slightly different, of course. If you get an error and can't work out what to do, please do email us at support@pythonanywhere.com.

There's one important thing to notice there -- the renewal date. You'll still need to renew the certificate and install a new one when it expires.

How to use it: scripting your own stuff

If you're an API kind of person, you can use the new ssl endpoint under the webapps URL to directly set your SSL certificate and private key with a POST request; you can also get SSL information (when the cert will expire, who issued it, and so on) by using the GET method. More information in the API documentation.

What's next?

An API and some command line scripts for this stuff is all very well, but we're far from done with improving the way SSL certs work on PythonAnywhere. Our long-term goal is to make this even easier -- watch this space for more information.

Any questions?

Hopefully all of that is pretty clear! But if you have any questions, please do let us know.


System update this morning

We deployed a new version of PythonAnywhere this morning. Everything went pretty smoothly; there were a few problems with some hosted websites shortly afterwards (an error in a load-distribution algorithm put too many websites on some servers, and not enough on others) but some sharp-eye customers spotted the problem and let us know, and we were able to rebalance things and fix the issue quickly.

There are a couple of great new features in the new system, but we're doing some last-minute testing before making them live -- watch this space for more information :-)


Blocked in Russia

We've heard reports from some Russian users that PythonAnywhere, and sites that we host, are blocked by their ISPs. The specific error message that they get when visiting the sites in Chrome is ERR_CONNECTION_TIMED_OUT -- Firefox has a similar one.

This appears to be the fallout from the Russian government's ongoing attempts to block the use of the Telegram instant messaging app. Because Telegram use various cloud hosting providers, including AWS and Google, the government has responded by blocking very large ranges of IPs owned by those providers. We're included in those ranges.

Right now, the situation is developing quite rapidly; the government appears to be rethinking its large-scale bans and some people who previously couldn't see our site can see it again now. The system we use to monitor our IPs' block status (which ironically enough is a Telegram bot) is still saying that we're blocked, though.

We're monitoring the situation. We can reasonably easily move everything over to new IPs, but we're wary of doing that in a situation where the new ones might get blocked the next day. When things have settled down a bit, if we're still blocked, we'll fix that.

[update 2018-05-16] Things seem to be settling down now, with no new large IP ranges being blocked, but it looks like some ISPs in Russia still aren't allowing access to our site and hosted sites. We've put up a test server on an IP address that we don't think is blocked. If you're in Russia and still seeing problems, please let us know what you see if you click here.


Terms and conditions and privacy policy update (yes, it's another GDPR thing)

Like a lot of companies, we're updating our terms and conditions and privacy and cookies policy in order to comply with the GDPR. The GDPR is a large regulatory change from the European Union, and is mostly about people's personal data and how it is shared.

The new T&Cs and PP will come into effect on 10 May 2018 and if you carry on using PythonAnywhere after that date, you'll be agreeing to them, so we figured it would be a good idea to post an explanation about the highlights of the changes.

If you just want to see what the new documents contain, here they are:

If you want to know a a bit more about what changed, and why, read on for a (non-exhaustive) overview. It's worth saying that this is just the changes that we think are important -- if you want to know exactly what the new terms and privacy policy say, you should read them.

The specifically GDPR-related stuff

There are two ways the GDPR impacts PythonAnywhere: data that we collect about you so that you can have an account on our site, and data that you collect about other people and use PythonAnywhere to process (eg. if you collect email addresses or the like on a website you run on PythonAnywhere).

Data that we collect about you

Like all websites that collect information about people from the EU, we now need to explain exactly what we do with any personal data we collect from you, and why we collect it. That's what our own privacy and cookies policy is about, though there are a few things in the terms and conditions too.

This is a pretty simple one for us. Because we don't make money from advertising, we don't collect any more data about you than we need to run the site -- your email address and so on, some payment-related stuff if you're a paying customer, etc., and standard website analytics. To be perfectly honest, the less we know, the happier we are -- it makes us a much less attractive target for data thieves :-)

Of course, you can provide us with more information -- maybe your name is Jane Smith and you chose the name "JaneSmith" as your username, or you posted something in the forums saying "My name is Jane Smith and this is my code", or you run a website that mentions your name, address, telephone number, and favourite colour.

Our new privacy and cookies policy covers all of this, along with mentions of the fact that we keep website access logs (that's 4.3, if you want to know how access logs are described in legal terms) and lots of stuff about cookies. So much stuff about cookies.

Data that you "control" and we "process"

"Data controller" and "data processor" are terms used by the GDPR. They can mean different things in different circumstances, but relating to PythonAnywhere, if you're storing personal data about yourself or other people on our systems, you are a data controller and we are a data processor acting for you. Let's imagine that you're hosting a website with us, and on your website you collect personal information about people -- maybe email addresses or names. In the GDPR's terminology, this makes you a "data controller" because you control the data that you've collected. But because we're providing you with the computers that the data is being stored and manipulated on, we're a "data processor".

The GDPR requires data controllers and data processors to have a contract in place that essentially says "this is what we each do" -- a data processing agreement (DPA). It doesn't need to be super-specific, but it does need to meet certain legal criteria.

We've noticed that some companies are sending out separate DPA contracts to cover this in addition to their normal terms and conditions. But in our lawyers' opinion, that's not necessary. When you agree to our terms and conditions and we accept you as a user of our site, that is a contract being formed between you and us. And if that contract includes all the appropriate stuff for the contract between a data processor and a data controller, then all is well.

So, Appendix 1 in the new terms and conditions covers all of that; if you're working towards GDPR compliance yourself, and you need a copy of the DPA, just use that. It mentions the essential stuff -- what we will do and not do, who processes data as a subcontractor (Amazon AWS, who own the servers PythonAnywhere runs on), and whether or not data goes out of the EU (it does, but that's OK because AWS is covered under the EU-US Privacy Shield Framework).

There are also some other GDPR-related changes dotted around the terms and conditions -- basically, making it clear that you can't use PythonAnywhere to breach the GDPR (so no spamming, please) and so on.

That's about it for the GDPR changes!

Other stuff

While we were modifying the T&Cs to be GDPR compliant, our legal advisors took the time to put in a few other updates to make sure that we're doing everything properly. Most of these are pretty minor (things like moving the details of the company from the top of the document to the bottom, or replacing the legalese "natural person" with the more, um, natural word "individual"), but a few are worth noting:

  • The T&Cs are now explicit about the fact that you agreeing to the terms and conditions (and us letting you use the site) form a contract. This was already the case, but it's good to be clear about these things.
  • If you're a consumer (that is, not a business) and you don't like any future changes to the terms and conditions, then under the new terms you'll have the legal right to cancel your account and -- if you have a paid account -- get a refund for any unused time left from your last payment. Of course, technically this only applies when we update our terms and conditions the next time, as this wording wasn't in the old T&Cs... but we won't be difficult if you decide that you don't like these new ones specifically and want to cancel your account.
  • "Cooling off" (section 7). We offer a 30-day no-questions-asked money-back guarantee. But not all businesses do, so there is EU-wide legislation that creates something similar (but not as good) for EU consumers. When you sign up for a service, you get 14 days to change your mind. The company in question is not obliged to provide you with anything during those 14 days, but you can request that they do. If you don't ask them to provide you with the service during those 14 days, then you can get any money you paid them back by cancelling any time over those 14 days; if you do ask them to provide you with the service, then you can at least get back a pro-rated refund. If you've signed up for a new electricity contract or something similar in the EU recently, you'll probably remember that they asked you if you were willing for them to start supplying you immediately -- this is why. Anyway, given that our own guarantee has a longer timeline (30 days instead of 14), is more generous (we'll refund everything without pro-rating), and covers more people (everyone in the world rather than just consumers in the EU), then you can probably regard section 7 (and the associated "model cancellation form") as boilerplate. If you want to cancel a paid subscription within the first 30 days and get a refund, just cancel it on the "Account" page and drop us an email.
  • Finally, and perhaps most importantly, we no longer accept faxes as a way to send and receive information about PythonAnywhere accounts. Which is probably a good thing, because we've never had a fax machine...

That's it!

That's all we though it was important to highlight in the new terms; if you have any questions then please leave a comment below. Thanks for reading!


Introducing the beta for always-on tasks

Update 8 January 2018 -- the beta is temporarily on hold while we sort out some issues that cropped up in the first phase. We'll announce here when it's reopened for new users.

Today we're starting the beta for a new paid feature on PythonAnywhere: always-on tasks. If you have a paid account and would like to try them out, drop us a line at support@pythonanywhere.com.

With always-on tasks, you can keep a script constantly running. Once the feature's been switched on for your account, they can be set up in the "Tasks" tab -- you specify a bash command (eg. "python3.6 /home/username/myscript.py"), and the system will start running it. If it exits -- say, if it crashes, or if the server has a problem -- it will be restarted quickly.

We'd be really grateful for any and all feedback people have about this. We've been testing it internally for some months, and a private beta has been running for a while -- so we're reasonably certain it's solid and reliable, but please do remember that it is currently beta, and there may be bugs we don't know about.


New look and feel!

We updated PythonAnywhere this morning :-)

The most visible change is that we've refreshed the look and feel -- we'd love to know what you think, especially about the new dashboard page that you get when you log in or click the PythonAnywhere logo. Is it useful? Is there other stuff we should add there, or are there things we should remove? Just drop us a line using the "Feedback" link.

There have also been some big changes under the hood. The most important ones are part of a project we're not quite ready to announce (coming soon, we promise!) but more visibly, we've improved scalability for Jupyter/IPython notebooks, so hopefully you'll notice a speedup if you using them.

We've also fixed a couple of minor bugs:

  • A few issues with editing scheduled tasks have been fixed.
  • You used to get an internal server error if you tried to start a MySQL console when you were out of disk quota. Now you don't.
  • Similarly, starting a Jupyter notebook with non-ASCII characters in its filename works properly now.

That's all for today's update. Any feedback much appreciated!


Response times now available in webapp logs

we deployed a new version this morning. Most of the work was on a sekrit hidden feature that we can't talk about yet (oooo) but there is one small thing we've added that we hope you find useful: we've added response times to your webapp access logs:

you can see them at the end of each line, in the format response_time=N.NNN.

We hope that you'll find these useful if you ever have to chase down performance issues. Let us know what you think!


Page 1 of 16.

Older posts »

PythonAnywhere is a Python development and hosting environment that displays in your web browser and runs on our servers. They're already set up with everything you need. It's easy to use, fast, and powerful. There's even a useful free plan.

You can sign up here.