Outage report: 20, 21 and 22 July 2017

We had several outages over the last few days. The problem appears to be fixed now, but investigations into the underlying cause are still underway. This post is a summary of what happened, and what we know so far. Once we've got a better understanding of the issue, we'll post more.

It's worth saying at the outset that while the problems related to the way we manage our users' files, those files themselves were always safe. While availability problems are clearly a big issue, we regard data integrity as more important.

20 July: the system update

On Thursday 20 July, at 05:00 UTC, we released a new system update for PythonAnywhere. This was largely an infrastructural update. In particular, we updated our file servers from Ubuntu 14.04 to 16.04, as part of a general upgrade of all servers in our cluster.

File servers are, of course, the servers that manage the files in your PythonAnywhere disk storage. Each server handles the data for a specific set of users, and serves the files up to the other servers in the cluster that need them -- the "execution" servers where your websites, your scheduled tasks, and your consoles run. The files themselves are stored on network-attached storage (and mirrored in realtime to redundant disks on a separate set of backup servers); the file servers simply act as NFS servers and manage a few simple things like disk quotas.

While the system update took a little longer than we'd planned, once everything was up and running, the system looked stable and all monitoring looked good.

20 July: initial problems

At 12:07 UTC our monitoring system showed a very brief issue. From some of our web servers, it appeared that access to one of our file servers, file-2, had very briefly slowed right down -- it was taking more than 30 seconds to list the specific directory that is monitored. The problem cleared up after about 60 seconds. Other file servers were completely unaffected. We did some investigations, but couldn't find anything, so we chalked it up as a glitch and kept an eye out for further problems.

At 14:12 UTC it happened again, and then over the course of the afternoon, the "glitches" became more frequent and started lasting longer. We discovered that the symptom from the file server's side was that all of the NFS daemons -- the processes that together make up an NFS server -- would all become busy; system load would rise from about 1.5 to 64 or so. They were all waiting uninterruptably on what we think was disk I/O (status "D" in top).

The problem only affected file-2 -- other file servers were all fine. Given that every file server had been upgraded to an identical system image, our initial suspicion was that there might be some kind of hardware problem. At 17:20 UTC we got in touch with AWS to discuss whether this was likely.

By 19:10 our discussions with AWS had revealed nothing of interest. The "glitches" had become a noticeable problem for users, and we decided that while there was no obvious sign of hardware problems, it would be good to at least eliminate that as a possible cause, so we took a snapshot of all disks containing user data (for safety), then migrated the server to new hardware, causing a 20-minute outage for users on that file server (who were already seeing a serious slowdown anyway), and a 5-minute outage for everyone else, the latter because we had to reboot the execution servers

After this move, at 19:57 UTC, everything seemed OK. Our monitoring was clear, and the users we were in touch with confirmed that everything was looking good.

21 July: the problem continues

At 14:31 UTC on 21 July, we saw another glitch on our monitoring. Again, the problem cleared up quickly, but we started looking again into what could possibly be the cause. There were further glitches at 15:17 and 16:51, but then the problem seemed to clear up.

Unfortunately at 22:44 it flared up again. Again, the issues started happening more frequently, and lasting longer each time, until they became very noticeable for our users at around 00:30 UTC. At 00:55 UTC we decided to move the server to different hardware again -- there's no official way to force a move to new hardware on AWS; stopping and starting an instance usually does it, but there's a small chance you'd end up on the same physical host again, so a second attempt seemed worth-while. If nothing else, it would hopefully at least clear things up for another 12 hours or so and buy us time to work out what was really going wrong.

This time, things didn't go according to plan. The file server failed to come up on the new hardware, and trying to move again did not help. We decided that we were going to need to provision a completely fresh file server, and move the disks across. While we have processes in place for replacing file servers as part of a normal system update, and for moving them to new hardware without changing (for example) their IP address, replacing one under these circumstances is not a procedure we've done before. Luckily, it went as well as could be expected under the circumstances. At 01:23 UTC we'd worked out what to do and started the new file server. By 01:50 we'd started the migration, and by 02:20 UTC everything was moved over. There were a few remaining glitches, but these were cleared up by 02:45 UTC.

23 July -- more problems -- and a resolution?

We did not expect the fix we'd put in to be a permanent solution -- though we did have a faint hope that perhaps the problem had been caused by some configuration issue on file-2, which might have been remediated by our having provisioned a new server rather than simply moving the old one. This was never a particularly strong hope, however, and when the problems started again at 12:16 UTC we weren't entirely surprised.

We had generated two new hypotheses about the possible cause of these issues by now:

  • The number of NFS daemons running on the machine was either too low or too high. Before our upgrade, we'd been running 8 daemons, and we'd moved to running 64 -- which we believed was the right number given the size of the machines in question, but of course we could have been wrong.
  • The upgrade from Ubuntu 14.04 to 16.04 could have introduced some kind of bug at the NFS level.

The problem with both of these hypotheses was that only one of our file servers was affected. All file servers had the same number of workers, and all had been upgraded to 16.04.

Still, it was worth a try, we thought. We decided to try changing the number of daemon processes first, as it we believed it would cause minimal downtime; however, we started up a new file server on 14.04 so that it would be ready just in case.

At 14:41 UTC we reduced the number of workers down to eight. We were happy to see that this was picked up across the cluster without any need to reboot anything, so there was no downtime.

Unfortunately, at 15:04, we saw another problem. We decided to spend more time investigating a few ideas that had occurred to us before taking the system down again. At 19:00 we tried increasing the number of NFS processes to 128, but that didn't help. At 19:23 we decided to go ahead with switching over to the 14.04 server we'd prepared earlier. We kicked off some backup snapshots of the user data, just in case there were problems, and at 19:38 we started the migration over.

This completed at 19:46, but required a restart of all of the execution servers in order to pick up the new file server. We started this process immediately, and web servers came back online at 19:48, consoles at 19:50, and scheduled tasks at 19:55.

By 20:00 we were satisfied that everything looked good, and so we went back to monitoring.

Where we are now

Since the update on Saturday, there were no monitoring glitches at all on Sunday, but we did see one potential problem on Monday at 12:03. However this blip was only noticed from one of our web servers (previous issues affected at least three at a time, and sometimes as many as nine), and the problem has not been followed by any subsequent outages in the 4 hours since, which is somewhat reassuring.

We're continuing to monitor closely, and are brainstorming hypotheses to explain what might have happened (or, perhaps still be happening). Of particular interest is the fact that this issue only affected one of our file servers, despite all of them having been upgraded. One possibility we're considering is that the correlation in timing with the upgrade is simply a red herring -- that instead there's some kind of access pattern, some particular pattern of reads/writes to the storage, which only started at around midday on Thursday after the system update. We're planning possible ways to investigate that should the problem occur again.

Either way, whether the problem is solved now or not, we clearly have much more investigation to do. We'll post again when we have more information.


Using the PythonAnywhere API: an (open source) helper script to create a Django webapp with a virtualenv

With the beta launch of our API, we want to start making it possible for people to do more scripted things with PythonAnywhere.

Our starter for 10 was this: our web-based UI has some helpers for creating new web apps for common web frameworks (Django, Flask, web2py, etc), but they pin you to the system-installed version of those packages. Using a virtualenv would give the user more flexibility, but currently that means using the more complicated "manual config" option.

The API means it's now possible to build a single command-line tool that you can run from a PythonAnywhere console to create, from scratch, a new Django project, with a virtualenv, all in one go.

The command-line tool in a Bash console

The script is called pa_start_django_webapp_with_virtualenv.py and it's available by default in any PythonAnywhere Bash console. Here's its command-line help:

$ pa_start_django_webapp_with_virtualenv.py -h
Create a new Django webapp with a virtualenv.  Defaults to
your free domain, the latest version of Django and Python 3.6

Usage:
  pa_start_django_webapp_with_virtualenv.py [--domain=<domain> --django=<django-version> --python=<python-version>] [--nuke]
Options:
  --domain=<domain>         Domain name, eg www.mydomain.com   [default: your-username.pythonanywhere.com]
  --django=<django-version> Django version, eg "1.8.4"  [default: latest]
  --python=<python-version> Python version, eg "2.7"    [default: 3.6]
  --nuke                    *Irrevocably* delete any existing web app config on this domain. Irrevocably.

Seeing it in action

If we launch the script and accept the defaults, we can watch it automatically going through all the steps you'd normally have to go through manually to create a new web app on PythonAnywhere:

screenshot of helper script building virtualenv etc

  • Creating a virtualenv
  • Installing Django into it
  • Creating a folder for your django site and running manage.py startproject
  • Creating a web app configuration on the PythonAnywhere Web tab
  • Editing your WSGI file to import your django app (and setting DJANGO_SETTINGS_MODULE correctly)
  • Creating a static files mapping on the web tab so that /static/ and /media/ urls work
  • updating settings.py to match those, and to set ALLOWED_HOSTS
  • runs manage.py collectstatic to get your CSS up and running
  • reloads your webapp to make it all live
screenshot of helper script building virtualenv etc Lovely progress reports! [1]

Once it's run, you can see your web app is immediately live.

screenshot of django it worked page

And your code is ready to go in a folder predictably named after the site's domain name:

17:08 ~ $ tree -L 3 commanderkeen.pythonanywhere.com/                                                            
commanderkeen.pythonanywhere.com/
├── manage.py
├── mysite
│   ├── __init__.py
│   ├── __pycache__
│   │   ├── __init__.cpython-36.pyc
│   │   ├── settings.cpython-36.pyc
│   │   └── urls.cpython-36.pyc
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
└── static
    └── admin
        ├── css
        ├── fonts
        ├── img
        └── js

Plans for the future -- it's open source!

The helper script is online here: github.com/pythonanywhere/helper_scripts. Issues, pull requests and suggestions are gratefully accepted.

The audacious vision of the future would be a script called something like pa_autoconfigure.py which takes the URL to a GitHub repo, and:

  • pulls down the repo
  • detects a requirements.txt and automatically creates a virtualenv for it
  • detects common web frameworks like Django, Flask, etc
  • automatically creates a PythonAnywhere WSGI config for them
  • automatically creates a web app configuration as well

It's ambitious but it shouldn't be too hard to get working. We'd love to get you, the user, involved in shaping the way such a tool might work though.

(and of course, there's nothing preventing you from just writing your own script to do these things, and to hell with our own corporate versions! Feel free to let us know if you do that. Or keep it as your own little secret).

[1] That's right, forget cowsay, on PythonAnywhere we have snakesay


The PythonAnywhere API: beta now available for all users

We've been slowly developing an API for PythonAnywhere, and we've now enabled it so that all users can try it out if they wish. Head over to your accounts page and find the "API Token" tab to get started.

The API is still very much in beta, and it's by no means complete! We've started out with a few endpoints that we thought we ourselves would find useful, and some that we needed internally.

screenshot of api page Yes, I have revoked that token :)

Probably the most interesting functionality that it exposes at the moment is the ability to create, modify, and reload a webapp remotely, but there's a few other things in there as well, like file and console sharing. You can find a full list of endpoints here: http://help.pythonanywhere.com/pages/API

We're keen to hear your thoughts and suggestions!


System update this morning

This morning's system update went well :-)

There aren't any major visible new features in the new system -- it was primarily an infrastructural change. The operating system on our underlying servers has been upgraded from Ubuntu 14.04 to 16.04 (so we had to rewrite all of our upstart system jobs as systemd ones, which was... fun). The sandboxes where your code runs have been kept as Ubuntu 14.04, so that your code doesn't break due to the system it runs on changing unexpectedly.

A future update, hopefully soon, will enable a 16.04-based system image that you'll be able to opt in to use when it's convenient to you. There's also a big new feature that we're working on that required the OS upgrade -- more about that another time...

There were a few minor changes beyond that:

  • We've increased the server capacity available for scheduled tasks.
  • We've significantly reworked the system for logging of error messages for web apps, so they should appear pretty much instantly in your error logs rather than being delayed -- previously they could sometimes take 30 seconds to arrive, which could make debugging frustrating.
  • We've been working on improving the API; there's some documentation here. The API is still in early alpha, and may change without warning, but if you'd like to have a play, just let us know and we can switch it on for your account.
  • A couple of security fixes -- mostly rate-limiting things that send email so that they can't be abused by trolls trying to fill up your inbox with junk. A shout-out to akash c for reporting that.

Tomorrow's system update cancelled

At the last minute, we discovered a bug in the version of PythonAnywhere we were planning to deploy tomorrow. We're pretty certain about the fix, but rather than rush code into the live system without thorough testing, we're delaying the system update. We hope it'll all be ready to go early next week.


New release! Python 3.6 :-)

Today's system update was mostly infrastructure changes and security fixes, but there's one big visible change -- we now support Python 3.6!

The new version is only available for accounts using the new dangermouse image, so if you'd like to be switched over and get not just the latest Python, but also new shiny versions of all of our installed system images, just send us a message using the "Send feedback" link.


Blocked in Russia

A week ago, one of the sites we were hosting was reported to us by the Russian authorities (specifically, the Federal Service for Supervision in the Sphere of Telecom, Information Technologies and Mass Communications [ROSKOMNADZOR]) for hosting illegal content. They said that we must take it down, or risk having the associated IP address blocked in Russia.

In the current political climate, it's very important to note that the site in question was offering services that are illegal not just in Russia, but in the UK, the United States, and almost every other country we can think of. Our terms and conditions do not allow sites engaging in "any activities that are illegal". We were glad to have it brought to our attention on that basis. If the situation had been different, and (say) we'd been told to take down a website expressing political views by the government of a country hostile to those views, our response would have been very different.

But in this case, the situation was clear-cut. We took the site down, notified its owner that we'd done so and explained why, and responded to the original notification saying that we had done this.

Last night, we discovered that the site had been re-created, and we took it down again and blocked all access for the user. But it seems that this was too late; today a different Russian customer contacted us and said that they could not access their own site, or PythonAnywhere. Further investigation determined that the IP address in question had indeed been blocked by the Russian government. It's accessible from everywhere else in the world -- but not from Russia. This is a problem, because the address in question serves our own site, and a large portion of our customers'. And we have a lot of customers in Russia!

We do have the capability to move sites from IP address to IP address (that's part of why we strongly recommend that customers with custom domains use CNAMEs instead of A records to set up their websites), but this was not designed to be done under emergency circumstances, so it takes time. We felt that in this case, a rapid response was required.

As such, we've implemented a secondary IP address that will work for any PythonAnywhere-hosted website. Over the course of this weekend, we will be testing it. If it is essential that your website be visible in Russia, please contact us on support@pythonanywhere.com and we can move you across during this testing period. If it all looks good early next week, we'll move all other sites on the old IP address over, including our own.

[update] As of 17:15 on 2017-03-07, we've migrated all websites over to the new, unblocked IP address. The only sites that should still be affected are those configured via A records (or similar) to route directly to the old address (a small number). We'll be scanning the sites we host and emailing the affected customers tomorrow.


New release! File sharing, and some nice little fixes and improvements

Honestly, it's strange. We'll work on a bunch of features, care about them deeply for a few days or weeks, commit them to the CI server, and then when we come to deploy them a little while later, we'll have almost forgotten about them. Take today, when Glenn and I were discussing writing the blog post for the release

-- "Not much user-visible stuff in this one was there? Just infrastructure I think..."

-- "Let's have a look. Oh yes, we fixed the ipad text editor. And we did the disable-webapps-on-downgrade thing. Oh yeah, and change-webapp-python-version, people have been asking for that. Oh, wow, and shared files! I'd almost totally forgotten!"

So actually, dear users, lots of nice things to show you.

File sharing

People have been asking us since forever about whether they could use PythonAnywhere to share code with their friends. or to show off that famous text-based guessing game we've all made early on in our programming careers. And, after years of saying "I keep telling people there's no demand for it", we've finally managed to make a start.

If you open up the Editor on PythonAnywhere you'll see a new button marked Share. screenshot of share button

You'll be able to get a link that you can share with your friends, who'll then be able to view your code, and, if they dare, copy it into their own accounts and run it.

screenshot of share menu

We're keen to know what you think, so do send feedback!

Change Python version for a web app

Another feature request, more minor this time; you'll also see a new button that'll let you change the version of Python for an existing web app. Sadly the button won't magically convert all your code from Python 2 to Python 3 though, so that's still up to you...

screenshot of change python ui

More debugging info on the "Unhandled Exception" page.

When there's a bug, or your code raises an exception for whatever reason, your site will return our standard "Unhandled Exception" page. We've now enhanced it so that, if it notices you're currently logged into PythonAnywhere and are the owner of the site, it will show you some extra debugging info, that's not visible to other users.

screenshot of new error page

Why not introduce some bugs into your code and see for yourself?

ipad fix, and other bits and pieces

We finally gave up on using the fully-featured syntax-highlighting editor on ipads (it seemed like it worked but it really didn't, once you tried to do anything remotely complicated) and have reverted to using a simple textarea.

If you're trying to use PythonAnywhere on a mobile device and notice any problems with the editor, do let us know, and we'll see if we can do the same for your platform.

Other than that, nothing major! A small improvement to the workflow for people who downgrade and re-upgrade their accounts, a fix to a bug with __init__.py in django projects,

Keep your suggestions and comments coming, thanks for being our users and customers, and speak soon!

Harry + the team.


Building a simple Telegram bot using PythonAnywhere

There's an explosion of chat apps and bots at the moment, and it's easy to see why. They're a useful new way of interacting with computer systems, they're interesting to code, and they're actually surprisingly easy to create.

This blog post shows how you can get a simple bot up and running, using Telegram. Telegram isn't as popular a messaging platform as WhatsApp or Skype, but it's much easier to build bots for. You'll need a normal computer and also a phone on which you can install the Telegram app. When you've finished working through the steps here, you'll have a bot that can have an almost-plausible conversation with you.

It uses PythonAnywhere, which probably isn't very surprising given the name of this blog ;-) You can do everything in here using a free PythonAnywhere account, and the bot you wind up with will be fully-functional. You'll only need a paid-for account if your bot starts getting lots of users -- of the order of thousands of messages a day.

So, without further ado, let's get started!

Creating a bot

The first thing you need to do is tell Telegram that you want to create a bot. For this, you'll need a Telegram account -- install their app on your phone, and get it set up.

Next, start a conversation with the "BotFather". This is a bot that Telegram themselves run, and it controls the creation and registration of bots on their platform. On the Android version of their app, here's what you do (other platforms are similar)

  • Tap on the start conversation button in the bottom right to start a new conversation.
  • Tap the magnifying glass "Search" icon near the top right.
  • Type "botfather".
  • Tap on the "@BotFather" that appears. Make sure it has a blue checkmark next to it
  • It will display a welcome message.
  • Click the "Start" button.
  • Send it a message "/newbot"
  • It will ask for a name for your bot. I'm going to call mine "GTDemo" but you should choose a name that's unique to you.
  • Next it will ask for a username; I'll use "GTDemoBot"
  • If all goes well, it will print out a message telling you that the bot was created. There's one important bit of information in there that you'll need for later: the token to access the HTTP API. It will be a long string of alphanumeric characters, maybe with a colon in it. To keep this for later, I copied the message on my phone and then emailed it to myself -- not super-secure, but probably safe enough if you're not going to be putting anything secret into your bot.

Right, so let's check that your bot is created, even if it's currently not very talkative. Start a conversation with it, using the same method to start a chat as you did with the BotFather. Hopefully you'll be able to find it and start a chat, but when you click the "Start" button, nothing will happen.

No big surprise there. Let's make it do something.

A first simple bot

On your computer:

  • I recommend you create a new PythonAnywhere account for this tutorial -- we'll be creating a website later, and if you already have a website, we don't want to get confused between the two. You can have multiple free accounts, even using the same email address, so there's no harm in signing up for a new one. Just create a free "Beginner" one.
  • Start a "Bash" console
  • In there, run

    pip3.5 install --user telepot
    

    this will install (for your own PythonAnywhere account) the excellent telepot Python library, which hides some of the complexities of talking to Telegram's API. Wait for the process to complete.

  • Next, click the PythonAnywhere logo to the top left to go back to the PythonAnywhere dashboard.

  • Go to the "Files" tab.
  • In the "Enter new file name", type a filename ending with ".py" for your bot's code -- say, firstsimplebot.py -- and click the "New file" button.
  • Enter the following code, replacing "YOUR_AUTHORIZATION_TOKEN" with the token that the BotFather gave you earlier:

    import telepot
    import time
    import urllib3
    
    # You can leave this bit out if you're using a paid PythonAnywhere account
    proxy_url = "http://proxy.server:3128"
    telepot.api._pools = {
        'default': urllib3.ProxyManager(proxy_url=proxy_url, num_pools=3, maxsize=10, retries=False, timeout=30),
    }
    telepot.api._onetime_pool_spec = (urllib3.ProxyManager, dict(proxy_url=proxy_url, num_pools=1, maxsize=1, retries=False, timeout=30))
    # end of the stuff that's only needed for free accounts
    
    bot = telepot.Bot('YOUR_AUTHORIZATION_TOKEN')
    
    def handle(msg):
        content_type, chat_type, chat_id = telepot.glance(msg)
        print(content_type, chat_type, chat_id)
    
        if content_type == 'text':
            bot.sendMessage(chat_id, "You said '{}'".format(msg["text"]))
    
    bot.message_loop(handle)
    
    print ('Listening ...')
    
    # Keep the program running.
    while 1:
        time.sleep(10)
    
  • Click the ">>> Run this file" button at the bottom of the page.

  • A console will appear at the bottom of the page. After a little while, it should display "Listening ..."

Now go back to your phone. In the chat with your bot, type "Hello". It should almost immediately reply "You said 'Hello'".

If you take another look at the console on PythonAnywhere, you'll see that it will have printed out some information about the message -- probably something like

text private 321518746

Woo! A working bot :-)

Let's work through that code bit by bit.

    import telepot
    import time
    import urllib3

This bit just imports the Python modules that we're going to use.

    # You can leave this bit out if you're using a paid PythonAnywhere account
    proxy_url = "http://proxy.server:3128"
    telepot.api._pools = {
        'default': urllib3.ProxyManager(proxy_url=proxy_url, num_pools=3, maxsize=10, retries=False, timeout=30),
    }
    telepot.api._onetime_pool_spec = (urllib3.ProxyManager, dict(proxy_url=proxy_url, num_pools=1, maxsize=1, retries=False, timeout=30))
    # end of the stuff that's only needed for free accounts

Like the comments say, this stuff is only needed if you're using a free "Beginner" PythonAnywhere account -- we are, of course, for this tutorial, but you can remove it if you want to reuse the code in a paid account later. It's there because free accounts can only connect outwards to particular external websites, and those connections have to go through a proxy server. Many APIs pick up the details of the proxy server automatically from their system environment when they're running, but telepot doesn't. It's not a problem, it just means we have to be a bit more explicit and say "use this proxy over here".

    bot = telepot.Bot('YOUR_AUTHORIZATION_TOKEN')

Now we get to the core of the code. This line uses telepot to connect to Telegram's server.

Next, we define a function that knows how to handle messages from Telepot.

    def handle(msg):
        content_type, chat_type, chat_id = telepot.glance(msg)

The first thing we do is pull the useful information out of the message, using telepot's glance utility function.

        print(content_type, chat_type, chat_id)

...we print out some of the information, just for debugging purposes.

        if content_type == 'text':
            bot.sendMessage(chat_id, "You said '{}'".format(msg["text"]))

We only handle text messages for the time being; speech recognition is a bit outside the bounds of this tutorial... When we get a text message, we simply reply back telling the person what they said.

So that's the end of the message-handler function. Back to the main code:

    bot.message_loop(handle)

This tells telepot to start running a message loop. This is a background thread that will keep running until the program exits; it listens on the connection that was opened to Telegram and waits for incoming messages. When they come in, it calls our handle function with the details.

    print ('Listening ...')

So we print out a message to our own console to show that we're up and running...

    # Keep the program running.
    while 1:
        time.sleep(10)

And then we wait forever. Like I said, the telepot message loop will only keep running until our program exits, so we want to stop it from exiting.

So now we have a working bot and we know how it works. Let's make it better.

Moving to webhooks

The bot that you have right now is just running inside the console underneath your editor. It will actually keep running for quite a while, but if PythonAnywhere do any system maintenance work that requires restarting the server it's on, it will stop and not restart. That's obviously not much good for a bot, so let's fix it.

What we'll use is Telegram's "webhooks" API. Webhooks are a different way of connecting to Telegram. Our previous code made an out-bound connection from PythonAnywhere to Telegram, then relied on Telegram sending messages down that connection for processing. With webhooks, things are reversed. We essentially tell Telegram, "when my bot receives a message, connect to PythonAnywhere and pass on the message". And the "connect to PythonAnywhere" bit is done by creating a web application to run inside your PythonAnywhere account that will serve a really simple API.

If any of that sounds daunting, don't worry. It's actually pretty simple, and the instructions are detailed :-)

  • Click on the PythonAnywhere logo to go back to the PythonAnywhere dashboard.
  • On the "Consoles" tab, click on the small "X" next to the "firstsimplebot.py" console. This is important -- it will kill the running bot that we've already created so that it doesn't interfere with the the new one we're about to create.
  • Go to the "Web" tab.
  • Click the "Add a new web app" button.
  • The first page will just tell you that the web app will be hosted at your-pythonanywhere-username.pythonanywhere.com. Click next.
  • On the next page, choose the "Flask" web framework. Flask is a great choice for simple websites that are designed for APIs.
  • On the next step, choose "Python 3.5". That's the version we installed telepot for.
  • On the next page, just accept the default location for your Flask app. It will be something like /home/your-pythonanywhere-username/mysite/flask_app.py
  • After a short wait, you'll see an "All done!" message and your website will be set up. There will be a link to it -- follow the link and you should see a message saying "Hello from Flask!"

So now you have a simple website running that just displays one message. What we need to do next is configure it so that instead, it's running an API that Telegram can connect to. And we also need to tell Telegram that it's there, and which bot it's there to handle.

  • Click your browser's "Back" button to go back to the "Web" tab.
  • Look down the page a bit, and you'll see the "Code" section.
  • In that section, open the "Go to directory" link for the "Source code" in a new browser tab. (It'll be useful to keep the "Web" tab around for later.)
  • In the new tab, you'll see the "Files" page. One file will be called "flask_app.py"; click on it to go to the editor.

Enter the following code. Don't worry about what it does yet, we'll go through that in a second. But don't forget to replace YOUR_AUTHORIZATION_TOKEN with your Telegram HTTP API token, and YOUR_PYTHONANYWHERE_USERNAME with your PythonAnywhere username. Also replace A_SECRET_NUMBER with a number that only you know; a good way to get one that's properly random is to go to this online GUID generator, which will generate a unique number like "c04a4995-a7e2-4bf5-b8ab-d7599105d1d1".

from flask import Flask, request
import telepot
import urllib3

proxy_url = "http://proxy.server:3128"
telepot.api._pools = {
    'default': urllib3.ProxyManager(proxy_url=proxy_url, num_pools=3, maxsize=10, retries=False, timeout=30),
}
telepot.api._onetime_pool_spec = (urllib3.ProxyManager, dict(proxy_url=proxy_url, num_pools=1, maxsize=1, retries=False, timeout=30))

secret = "A_SECRET_NUMBER"
bot = telepot.Bot('YOUR_AUTHORIZATION_TOKEN')
bot.setWebhook("https://YOUR_PYTHONANYWHERE_USERNAME.pythonanywhere.com/{}".format(secret), max_connections=1)

app = Flask(__name__)

@app.route('/{}'.format(secret), methods=["POST"])
def telegram_webhook():
    update = request.get_json()
    if "message" in update:
        text = update["message"]["text"]
        chat_id = update["message"]["chat"]["id"]
        bot.sendMessage(chat_id, "From the web: you said '{}'".format(text))
    return "OK"

Once you've entered the code and made sure you've made the three substitutions:

  • Save the file
  • Switch to the browser tab with your web app setup in it.
  • Click the green "Reload" button near the top.
  • Wait for the "spinner" to finish.

Back on your phone, send another message. This time you should get a message back saying clearly that it came from the web. So now we have a bot using webhooks!

Let's work through the code now:

from flask import Flask, request
import telepot
import urllib3

So again, we import some Python modules. This time as well as the telepot and the urllib3 stuff that we need to talk to Telegram, we use some stuff from Flask.

proxy_url = "http://proxy.server:3128"
telepot.api._pools = {
    'default': urllib3.ProxyManager(proxy_url=proxy_url, num_pools=3, maxsize=10, retries=False, timeout=30),
}
telepot.api._onetime_pool_spec = (urllib3.ProxyManager, dict(proxy_url=proxy_url, num_pools=1, maxsize=1, retries=False, timeout=30))

Once again, the stuff we need to access Telegram from a free PythonAnywhere account.

secret = "A_SECRET_NUMBER"

Now, this is a bit of best-practice for Telegram bots using webhooks. Your bot is running as a publicly-accessible website. Anyone in the world could connect to it. And of course we really don't want random people to be able to connect, pretending to be Telegram, and make it say inappropriate things... so, we're going to say that the website only serves up one page, and the URL for that page is unguessable. This should make things reasonably safe. You'll see the code for that in a moment.

bot = telepot.Bot('YOUR_AUTHORIZATION_TOKEN')

We connect to Telegram using telepot, just like we did before.

bot.setWebhook("https://YOUR_PYTHONANYWHERE_USERNAME.pythonanywhere.com/{}".format(secret), max_connections=1)

We use telepot to send a message to Telegram saying "when my bot gets a message, this is the URL to send stuff to". This, of course, not only contains the host name for your website with your PythonAnywhere username, it also includes the hopefully-unguessable secret that we defined earlier. It's also worth noting that it uses secure HTTPS rather than HTTP -- all websites on PythonAnywhere, even free ones, get HTTPS by default, and Telegram (quite sensibly) will only send webhooks over HTTPS.

app = Flask(__name__)

Now we create a Flask application to handle requests.

@app.route('/{}'.format(secret), methods=["POST"])
def telegram_webhook():

This is some Flask code to say "when you get a POST request on the secret URL, run the following function". If you want to learn more about how Flask works, we have a tutorial on that too.

    update = request.get_json()

Telegram sends stuff to bots using JSON encoding, so we decode it to get a Python dictionary.

    if "message" in update:

If the thing we received from Telegran was a message...

        text = update["message"]["text"]
        chat_id = update["message"]["chat"]["id"]

...extract the text of the message, and the ID of the chat session which it forms a part of...

        bot.sendMessage(chat_id, "From the web: you said '{}'".format(text))

...then send the reply back using telepot...

    return "OK"

...and return something to Telegram to say that all is OK.

So now we have, and hopefully understand, a simple Telegram bot that will keep running pretty much forever! Websites on PythonAnywhere free accounts last for three months, and then you can extend them for another three months -- and three months later you can extend again, and so on, as many times as you like. So as long as you're willing to log in to PythonAnywhere four times a year, you're all set :-)

But the bot is pretty boring at the moment. Let's make it a little more interesting.

Introducing Eliza

Sorry, Hamilton fans, not Angelica and Peggy's sister. Eliza is an early natural language processing system, and the normal implementation simulates a Rogerian psychotherapist -- a kind of therapist who simply turns every question back on the patient. That makes it an easy one to implement and use in a bot like this.

Doubly conveniently, the Python nltk package provides an implementation of Eliza, so we don't even need to code it ourselves.

Let's check out how it works. Go to the PythonAnywhere dashboard, and start a new Bash console. In it, try out Eliza in a Python 3.5 interpreter like this (the answers it gives you may vary):

19:20 ~ $ python3.5
Python 3.5.2 (default, Jul 17 2016, 00:00:00) 
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from nltk.chat.eliza import eliza_chatbot
>>> eliza_chatbot.respond("Hello")
'Hi there... how are you today?'
>>> eliza_chatbot.respond("I'm well")
"Why do you think you're well?"
>>> eliza_chatbot.respond("I'm not sure")
'How does being not sure make you feel?'
>>> eliza_chatbot.respond("A little confused, to be honest")
'Very interesting.'
>>> eliza_chatbot.respond("Is it?")
'Why do you ask that?'

OK, that should give you the feel of how it works. Let's code it up.

  • Go back to the dashboard, to the "Files" tab, then to your Flask app's code (probably inside mysite/flask_app.py).
  • Add a new import to the top:

    from nltk.chat.eliza import eliza_chatbot
    
  • Inside the telegram_webhook function, replace this line:

    bot.sendMessage(chat_id, "From the web: you said '{}'".format(text))
    

    with this:

    if text == "/start":
        bot.sendMessage(chat_id, "Hello, I'm the therapist.  How can I help?")
    else:
        bot.sendMessage(chat_id, eliza_chatbot.respond(text))
    

    When someone first connects to a Telegram bot, the app sends you a text message saying "/start", so we have a special case for that so that Eliza doesn't say something weird like "Why do you say that /start?". But all other messages we simply send to Eliza for processing, then return.

  • Go to the "Web" tab and hit the green "Reload" button.

Back on your phone, let's start a new session so that we can chat with Eliza afresh.

  • In the old session, click the "Hamburger" menu to the top right and select "Delete chat"
  • Click on the button to start a chat again, then search for your bot again, click it.
  • Start the chat by clicking the "Start" button.

And now you should be able to talk to your chatbot! Many happy hours of not-very-useful therapy to be had :-)

That's it!

That's all for this tutorial. If you hit any problems, leave a comment below. And if you have any thoughts on how we could extend it, just let us know. Have fun, and happy botting!


The PythonAnywhere Newsletter, December 2016

Welcome to our Christmas newsletter! Featuring a selection of the Internet's very worst Christmas-themed animated Gifs!

snow with christmas tree in foreground

Here's what we've been up to:

Is it called "Reindeer Shaving" at this time of year?

dancing reindeer. sigh.

One change you might have noticed is that the Consoles page now has little Xs for killing consoles. The original idea was just to change them from being links that cause a page refresh to being ajax calls, which would let you kill multiple consoles at the same time. Somehow though, that small user interface tweak turned into the whole office deciding to treat that How it feels to learn JavaScript in 2016 comedy blog post as if it were an instruction manual, and we have now spent several days knee deep in React, ES6, promises, webpack, npm, Enzyme, fetch, promises, promises, and many, many more. Still, by the end of it, it all worked, and we have to conclude that ES6 is much nicer to work with than horrible old javascript.

Deep Tarpit

santa throwing a sack around

Santa needs some names for his "naughty or nice" list, and we've got some ideas for him!

Our tarpit has always been a soft limit, so going over your CPU allowance isn't the end of the world, it just slows your processes down, so that people who are within quota get first dibs on the precious cpu-cycles. Some people are taking the mick though, and we've now built some tools to detect these "deep tarpit" users, and kill their processes. So, more CPU-time for the good users, and naughty users now get the lowest priority of all, "kill".

The inside scoop from the forums

animated xmas tree

New modules

Santa on a reindeer

Although you can install Python packages on PythonAnywhere yourself, we like to make sure that we have plenty of batteries included.

  • We have been switching people over to the new dangermouse image! Send us an email if you want to switch.

  • You may have noticed a InsecurePlatformWarning when using pip > 9.0. This is a bug/regression in the pip code for python versions 2.7.9 or lower. We are keeping track of the issue and here are some ways to work around it for now.

New whitelisted sites

santa spinning on a rugby ball or something??

Paying PythonAnywhere customers get unrestricted Internet access, but if you're a free PythonAnywhere user, you may have hit problems when writing code that tries to access sites elsewhere on the Internet. We have to restrict you to sites on a whitelist to stop hackers from creating dummy accounts to hide their identities when breaking into other people's websites.

But we encourage you to suggest new sites that should be on the whitelist! Our rule is, if it's got an official public API (which means that the site's owners are encouraging automated access to their server) then we'll whitelist it. Just drop us a line with a link to the API docs.

Here are some sites we've added since our last newsletter:

.okta.com
.unwiredlabs.com
app.datadoghq.com
api.discogs.com
api.test.netbanx.com
api.test.paysafe.com
api.tinychat.com
apis.vworld.kr
cps.combain.com
epayment.bbs.no
epayment-test.bbs.no
repo.continuum.io
services.arcgisonline.com
steamcommunity.com
timesofindia.indiatimes.com
vizier.u-strasbg.fr
www.datapowerlearning.com
www.nbrb.by
www.rediff.com

What's coming up next year?

Over Christmas we're working on a few small tweaks and improvements, and for early next year, we're looking to start work on a feature that many of you have been requesting for a long time. But shush! No spoilers ;)

Thanks for reading our newsletter! Tune in the same time next month (ish) for more news from PythonAnywhere.


Page 1 of 15.

Older posts »

PythonAnywhere is a Python development and hosting environment that displays in your web browser and runs on our servers. They're already set up with everything you need. It's easy to use, fast, and powerful. There's even a useful free plan.

You can sign up here.