The PythonAnywhere newsletter, March 2016

Well, it's been nine months since our last newsletter and we've got a lot to tell you... Let's get started.

Cool new stuff part 1: Jupyter/IPython notebooks

Since the end of last year, all paid PythonAnywhere accounts have supported Jupyter/IPython notebooks. If you go to the "Files" tab, you can run existing notebooks, or create new ones. If you do anything involving data analysis, or exploratory interactive coding, they're a must-see.

We're still working out how to provide some kind of access for free users (without breaking the bank with our own server costs) so stay tuned...

Cool new stuff part 2: Education

Do you teach programming to a class of students? Do you have a coach who's helping you learn to code? Or do you just have a bunch of PythonAnywhere accounts and would like to access them all from one login?

With our new education feature, when you're logged into PythonAnywhere you can go to the "Account" tab and then to the "Teacher" section, and enter the username of another account. The person who's logged in to that other account can then "switch modes" so that they can use the site as if they are you. They can see your files, look at your consoles, and so on.

So -- if you're a teacher, next time you're running a class, ask your students to nominate you as their teacher, and you'll be able to help them without having to shoulder-surf. Super-useful for remote classes. (If you want us to bulk-create a bunch of accounts for your students beforehand, just get in touch on support@pythonanywhere.com.)

If you're a student, just nominate your coach as your teacher, and they can help you with your coding questions quickly and easily.

And if you have a bunch of PythonAnywhere accounts that you want to access while logged in as a "superuser" account, just log in to each of the other accounts in turn and nominate your main account as the teacher. If you're a web developer using PythonAnywhere to host your customers' sites -- in separate accounts to make billing easier -- then you never need to wonder what the login details for each of the customers are.

There's more information here.

More cool new stuff! Part 3, custom consoles

Do you often need to start a console running a particular script? Maybe there's a metrics script you run once a day to find out how much money your wildly-successful website has made over the last 24 hours :-) Or maybe you just want to download some data to update your site.

On the "Consoles" tab, you can add a custom console to do whatever you want. Click on the little "+" icon next to "Custom", and you can enter a name (for you to recognise it by) and a bash command, or a path to a script. Click the checkmark, and you'll have a new custom script on your "Consoles" tab. From now on, every time you want to launch your script, it's just a click away.

Give it a go -- you'll be surprised how many helpful scripts you wind up adding.

More about custom consoles in this blog post.

Yet more cool new stuff! Part 4, web app hit counting

How busy is your website? If you have a free account, you can now go to the "Web" tab and see how many hits you've had in the last hour, day or month -- and comparable numbers for last month. And if you're a paying customer, you get pretty live charts you can zoom into and analyse in depth :-)

Pretty pictures here.

Even more cool new stuff! Part 5, a better editor

We've made our in-browser editor (the one you get if you click on a .py file in the "Files" tab) much better. Many thanks for everyone for the suggestions! There are two really noticeable changes:

  • The console that shows the results of your code when you click the "Save and run" button is no longer in a popup tab -- it's right there in the editor. No more problems with popup blockers, or having to click back and forth between tabs.
  • "Save as" -- it's kind of silly that our editor didn't have this. Now it does :-)

Stuff that isn't really very cool but you probably need to know!

  • Since day one, we've provided a MySQL database for everyone. The hostname we suggested for accessing it was simply mysql.server. (We have stuff in place so that address can point to different places for different people.) That wasn't working too well, unfortunately -- basically, the stuff to make the same address go to different servers for different people made it kind of slow -- so we've changed the address. If you go to the "Databases" tab and look at the top, in the "Connecting" section, you'll see a "Database host address", which is the one you should use now. That address isn't accessible from outside PythonAnywhere (for security reasons) but it will work inside.

    We are removing support for the mysql.server address in the very near future (probably about a month), so update your config accordingly.

  • Website CNAMEs. If you have a paid account and are using a custom domain, we used to tell you to point a CNAME at yourusername.pythonanywhere.com. This really confused lots of people, because you can also have a website at yourusername.pythonanywhere.com, and that might be showing a completely different site to the one on your custom domain. We've revamped that, and now you can specify a different numeric CNAME value that doesn't host a site at all. We recommend you change over -- though, again, we'll continue to support the old-style CNAME for the time being.

From the forums and our blog

New modules

Although you can install Python packages on PythonAnywhere yourself, we like to make sure that we have plenty of batteries included. Here's what we've added since the last newsletter:

Python 2.7

plotly (1.9.3)
Upgraded tweepy to 3.5.0 (so that it works with Twitter's latest API)

Python 3.3 and 3.4

We added around 150 packages so that Python 3 packages match (as much as possible) the packages that are available for Python 2.

Here are some highlights, but you can visit the complete list for more details.

GitPython (1.0.1)
google-api-python-client (1.4.1)
pycurl (7.19.5.1)
pyflakes (0.9.2)
pyspotify (2.0.0)
tweepy (3.3.0)
xlrd (0.9.3)
xlwt (1.0.0)

New whitelisted sites

If you're a free PythonAnywhere user, you may have hit problems when writing code that tries to access sites elsewhere on the Internet. We have to restrict you to sites on a whitelist -- ones with an official public API -- to stop hackers from using us as a one-stop-shop to hide their identities when doing nefarious things. But we keep adding stuff to the whitelist; since our last newsletter, we've added over 100 new sites, but here are some that you may recognise:

api.coursera.org
api.dropboxapi.com
*.federalreserve.gov
api.pushover.net
api.stripe.com
data.sparkfun.com
soundcloud.com

And that's it!

Thanks for reading, and tune in next month for another exciting newsletter from PythonAnywhere.


Webapps and scheduled task expiries

tl;dr: for free accounts, web apps and scheduled tasks will stop running after a while if you don't log in. We'll email you a warning before this happens. Here's why:

Loads of people create free Python websites on PythonAnywhere and this is a really cool thing. Some of these websites are active ones where people are hosting their personal stuff, doing academic things, etc., and they want to keep them running. This is awesome! We want people to do that and we're happy to host this stuff for free.

For other use cases, some people may setup a webapp to try out a new web framework. Their owners may not intend to keep them running forever. That's fine too! We're glad to help people learn.

The problem for us is that we can't tell which is which.

Search engines such as Google and Baidu, and other web crawlers continuously index and hit all of the free websites, so they all look active, even the ones that nobody is using. We also didn't have any mechanism to tell which scheduled tasks were still important to their owners.

We don't want to reduce the service that we offer for free. We think it's an important way to give back to the community (and of course lots of free users start paying us after a while -- the record is a free user who started paying us after three years! -- so we have an incentive there too ;-)

On the other hand, PythonAnywhere is also very focused on offering a long term, sustainable service. In order to be a long term solution for you, we need a way to avoid accumulating dead code that will just grow and grow with no way of reducing it.

So, we've added a way for people to say "I'm still interested in keeping this web app/task running". Every three months (for web apps) or four weeks (for scheduled tasks) we'll email you to check. If you want to keep it up and running, there's a link in the email to click that will make sure it's all good for another three months or four weeks as appropriate. If you don't, you can just ignore the email.

If you miss the expiry email, all is not lost -- we won't delete anything. All your files, webapp setup, tasks and task logs will still be kept, but they just won't be actively running. You just need to login and click a button to re-enable them, and they will be extended by 3 months/4 weeks again.

We hope this works out OK, and helps us avoid running stuff that nobody wants anymore, which adds unnecessary congestion to the servers and ultimately increases the costs we have to charge our paying customers. Hopefully, it is a good balance between the two goals of being long term sustainable and making sure that we can continue to host stuff that people actually want for free.

Reach out to us if you have any thoughts!


Deprecation warning: "mysql.server" hostname being retired

Relax everyone! We're not switching off our mysql service, just switching off one of the old names it was available under. You'll still be able to access it, and more reliably, under the new name, with no downtime. Details follow...

Action is required if you set up your mysql instance over a year ago

The old mysql.server proxy service is being shut down

We originally set up a local mysql-proxy instance on each server, which would forward traffic to our actual database servers. It was available locally under the hostname mysql.server, which we inject into people's hosts files. We decided this service wasn't as reliable as we'd like, and have been using custom DNS routing instead.

For the last 12 months or so we've been telling people to use the new hostnames, so you only need to worry if you set up your mysql connection a long time ago (maximum respect to our OG users by the way!)

Use the new myusername.mysql.pythonanywhere-services.com address from your Databases tab

Head on over to your Databases tab (available from the dashboard) and you'll find the new hostname you should be using. Then, scrobble away in your settings.py, or DAL.py, or wherever it is that you store your mysql settings, bounce your web app or application, kick the tyres, and you're good to go.

You have 15 seconds to comply

Get it done soon! We expect to retire the mysql.server service at our next-but-one deployment, which could be as little as two weeks away, so get it done. Why not do it now? Just drop us a line via support@pythonanywhere.com if you need any help.


Jupyter notebooks finance demo

ipython-demo

The goal of this demo is to show how ipython notebooks can be used in conjunction with different datasources (eg: Quandl) and useful python libraries (eg: pandas) to do financial analysis. It will try to slowly introduce new and useful functions for the new python user.

Since oil-equity corr has been all the talk these days (this demo was written in Jan 2016), let's take a look at it!

In [1]:
# PythonAnywhere comes pre-installed with Quandl, so you just need to import it
import Quandl

# first, go to quandl.com and search for the ticker symbol that you want # let's say we want to look at (continuous) front month crude vs e-mini S&Ps

cl = Quandl.get('CHRIS/CME_CL1') es = Quandl.get('CHRIS/CME_ES1')

In [2]:
# Quandl.get() returns a pandas dataframe, so you can use all the pandas goodies
# For example, you can use tail to look at the most recent data, just like the unix tail binary!
es.tail()

Out[2]:
Open High Low Last Change Settle Volume Open Interest
Date
2016-03-02 1976.25 1984.75 1966.25 1982.25 5.5 1983.5 1814091 3008297
2016-03-03 1981.75 1992.50 1974.75 1991.50 7.0 1990.5 1541249 3008594
2016-03-04 1991.00 2007.50 1984.00 1994.75 4.5 1995.0 2232860 3018684
2016-03-07 1994.50 2004.50 1984.50 1999.75 4.0 1999.0 1623905 3012243
2016-03-08 1999.00 2000.25 1976.00 1982.50 18.0 1981.0 1928239 3005808

In [3]:
# you can also get statistics
es.describe()

Out[3]:
Open High Low Last Change Settle Volume Open Interest
count 4723.000000 4736.000000 4738.000000 4738.000000 512.000000 4738.000000 4738.000000 4738.000000
mean 1313.871427 1325.847867 1305.396581 1316.367885 13.185547 1316.353480 1076569.155129 1451989.689743
std 310.180613 312.158070 311.821112 312.201487 12.239507 312.167612 956268.606849 1159894.330244
min 674.750000 694.750000 665.750000 676.000000 0.250000 676.000000 0.000000 0.000000
25% 1109.250000 1117.000000 1102.500000 1109.750000 4.000000 1109.750000 182620.250000 193737.500000
50% 1268.500000 1277.875000 1259.500000 1269.500000 9.500000 1269.500000 857774.500000 1351206.000000
75% 1437.000000 1449.500000 1429.000000 1438.687500 19.562500 1438.687500 1728705.250000 2678325.750000
max 2129.250000 2134.000000 2122.750000 2128.750000 100.250000 2128.000000 6285917.000000 3594453.000000

But wait!

What do we have here? Did you notice that the count is different for the different columns?

Let's take a look at what the missing values are:

In [4]:
# select the rows where Open has missing data points
es[es['Open'].isnull()].head()

Out[4]:
Open High Low Last Change Settle Volume Open Interest
Date
2015-11-17 NaN 2063.50 2041.50 2049.75 1.00 2049.0 1610071 2803541
2015-11-27 NaN 2098.25 2081.50 2090.50 2.00 2090.0 653079 2761335
2015-12-01 NaN 2101.50 2083.50 2099.25 20.25 2100.0 1479676 2764688
2015-12-02 NaN 2105.00 2075.00 2083.50 18.50 2081.5 1709808 2759024
2015-12-09 NaN 2079.75 2034.25 2045.25 16.75 2042.0 2660114 2624311

Hmmm. Time to spend money and buy good data?

Eh. We really only need the daily close here anyways (ie. the settle column). Let's zoom in on that.

In [5]:
es_close = es.Settle  # WHAT IS THIS SORCERY? Attribute access!
es_close.head()

Out[5]:
Date
1997-09-09    934
1997-09-10    915
1997-09-11    908
1997-09-12    924
1997-09-15    922
Name: Settle, dtype: float64

In [6]:
print(type(es))
print(type(es_close))

<class 'pandas.core.frame.DataFrame'>
<class 'pandas.core.series.Series'>

Oh ok. A column of a DataFrame is a Series (also a pandas object).

Note that it is still linked to the DataFrame (ie. changing the Series will change the DataFrame as well)

In [7]:
# Okay- time to quickly check the crude time series as well
cl.describe()

Out[7]:
Open High Low Last Change Settle Volume Open Interest
count 8273.000000 8275.000000 8275.000000 8275.000000 518.000000 8275.000000 8275.000000 8274.000000
mean 41.777084 42.333422 41.186375 41.777544 1.025753 41.777361 112520.504411 125494.688301
std 29.568643 29.946338 29.134537 29.559547 0.871786 29.559366 123411.769025 110104.481053
min 10.000000 11.020000 9.750000 10.420000 0.010000 10.420000 0.000000 0.000000
25% 19.600000 19.790000 19.400000 19.610000 0.400000 19.610000 30618.000000 46379.500000
50% 28.300000 28.650000 27.970000 28.320000 0.810000 28.320000 58659.000000 87553.000000
75% 61.290000 62.150000 60.485000 61.365000 1.470000 61.365000 166439.000000 176041.500000
max 145.190000 147.270000 143.220000 145.290000 7.540000 145.290000 824242.000000 608831.000000

In [8]:
# Hmm. That's a lot more counts. Does the crude time series start earlier than e-mini's?
cl.head()

Out[8]:
Open High Low Last Change Settle Volume Open Interest
Date
1983-03-30 29.01 29.56 29.01 29.40 NaN 29.40 949 470
1983-03-31 29.40 29.60 29.25 29.29 NaN 29.29 521 523
1983-04-04 29.30 29.70 29.29 29.44 NaN 29.44 156 583
1983-04-05 29.50 29.80 29.50 29.71 NaN 29.71 175 623
1983-04-06 29.90 29.92 29.65 29.90 NaN 29.90 392 640

In [9]:
earliest_es_date = es.index[0]

# at first glance, you could just do cl[earliest_es_date:].head()

Out[9]:
Open High Low Last Change Settle Volume Open Interest
Date
1997-09-09 19.43 19.61 19.37 19.42 NaN 19.42 32299 88070
1997-09-10 19.57 19.57 19.35 19.42 NaN 19.42 41858 86872
1997-09-11 19.49 19.72 19.30 19.37 NaN 19.37 52342 80434
1997-09-12 19.42 19.47 19.27 19.32 NaN 19.32 28540 80440
1997-09-15 19.29 19.38 19.23 19.27 NaN 19.27 31610 76590

In [10]:
# but just in case there is no matching precise date, we can also take the closest date:
closest_row = cl.index.searchsorted(earliest_es_date)
cl_close = cl.iloc[closest_row:].Settle
cl_close.head()

Out[10]:
Date
1997-09-09    19.42
1997-09-10    19.42
1997-09-11    19.37
1997-09-12    19.32
1997-09-15    19.27
Name: Settle, dtype: float64

In [11]:
# ok lets just plot this guy
import matplotlib
import matplotlib.pyplot as plt
# use new pretty plots
matplotlib.style.use('ggplot')
# get ipython notebook to show graphs
%pylab inline

es_close.plot()

Populating the interactive namespace from numpy and matplotlib
Out[11]:
<matplotlib.axes._subplots.AxesSubplot at 0x7fe1ef10cf28>

That was satisfying- our all too familar S&P chart. Let's try to plot both S&P and oil in the same graph.

In [12]:
plt.figure()
es_close.plot()
cl_close.plot()
plt.yscale('log')

Meh. Okay. LET's ACTUALLY DO SOME MATH!

(... ahem. stats)

In [13]:
es['Settle'].corr(cl['Settle'])

Out[13]:
0.34046181767351186

okay... MOAR GRAPHS! I hear you say

In [14]:
import pandas as pd
pd.rolling_corr(es_close, cl_close, window=252).dropna()
# why 252? because that's the number of trading days in a year

Out[14]:
Date
2014-11-21   -0.369646
2014-11-24   -0.386641
2014-11-25   -0.404232
2014-11-26   -0.421947
2014-11-28   -0.441608
2014-12-01   -0.457250
2014-12-02   -0.473815
2014-12-03   -0.488969
2014-12-04   -0.502630
2014-12-05   -0.516329
2014-12-08   -0.526947
2014-12-09   -0.536849
2014-12-10   -0.541803
2014-12-11   -0.548337
2014-12-12   -0.549632
2014-12-15   -0.549979
2014-12-16   -0.546815
2014-12-17   -0.551038
2014-12-18   -0.560348
2014-12-19   -0.569132
2014-12-22   -0.577833
2014-12-23   -0.586649
2014-12-24   -0.594898
2014-12-26   -0.603125
2014-12-29   -0.610925
2014-12-30   -0.617783
2014-12-31   -0.622196
2015-01-02   -0.626952
2015-01-05   -0.628422
2015-01-06   -0.627138
                ...   
2016-01-26    0.613887
2016-01-27    0.620853
2016-01-28    0.626633
2016-01-29    0.630797
2016-02-01    0.636777
2016-02-02    0.644177
2016-02-03    0.650006
2016-02-04    0.655308
2016-02-05    0.661402
2016-02-08    0.668422
2016-02-09    0.676663
2016-02-10    0.684008
2016-02-11    0.692094
2016-02-12    0.697270
2016-02-16    0.701433
2016-02-17    0.703887
2016-02-18    0.706600
2016-02-19    0.709578
2016-02-22    0.711551
2016-02-23    0.714437
2016-02-24    0.717001
2016-02-25    0.718281
2016-02-26    0.720613
2016-02-29    0.722511
2016-03-01    0.723214
2016-03-02    0.723336
2016-03-03    0.722921
2016-03-04    0.722644
2016-03-07    0.722566
2016-03-08    0.722830
Name: Settle, dtype: float64

That's weird. You'd expect the first year to drop out (because the rolling correlation window starts after the first year), but it should have started after Sept 1998. Instead it is starting in 2014...

In [15]:
print(len(cl_close))
print(len(es_close))

4646
4738

In [16]:
merged = pd.concat({'es': es_close, 'cl': cl_close}, axis=1)
# maybe this is the culprit?
merged[merged['cl'].isnull()].head()

Out[16]:
cl es
Date
1997-11-27 NaN 959.50
1997-11-28 NaN 955.00
1998-01-19 NaN 972.25
1998-02-16 NaN 1019.00
1998-05-25 NaN 1116.50

In [17]:
merged.dropna(how='any', inplace=True)
# BAD DATA BEGONE!
merged[merged['cl'].isnull()]

Out[17]:
cl es
Date

In [18]:
pd.rolling_corr(merged.es, merged.cl, window=252).dropna().plot()
plt.axhline(0, color='k')

Out[18]:
<matplotlib.lines.Line2D at 0x7fe1e2e5fef0>

Brilliant! But this is still quite inconclusive in terms of equity/crude corr. Why? Well we are forgetting about one HUGE HUGE factor affecting correlation here.

In [19]:
# D'oh
import numpy as np
print('Autocorrelation for a random series is {:.3f}'.format(
    pd.Series(np.random.randn(100000)).autocorr())
)
print('But, autocorrelation for S&P is {:3f}'.format(es_close.autocorr()))

Autocorrelation for a random series is -0.003
But, autocorrelation for S&P is 0.998803

So that's why we should look at %-change instead of $-close or $-change...

In [20]:
daily_returns = merged.pct_change()
rolling_correlation = pd.rolling_corr(daily_returns.es, daily_returns.cl, window=252).dropna()
rolling_correlation.plot()
plt.axhline(0, color='k')
title('Rolling 1 yr correlation between Oil and S&P')

Out[20]:
<matplotlib.text.Text at 0x7fe1e2c89ba8>

Great. Now this is much more interesting. It is quite clear that the period of higher correlation in oil prices came after 2009. Qualitatively, we know (if you worked in finance back then) that this was the case: previously, extreme high oil prices (over $100/bbl) were seen as a drag on the economy. Nowadays, extreme low oil prices are seen as an indication of weakness in global demand, with oil prices, equity, credit etc all selling off hand in hand when there is risk off sentiment.

Let's plot some pretty graphs to show what we know qualitatively, and make sure our memory was correct.

In [21]:
# vertically split into two subplots, and align x-axis
fig, (ax1, ax2) = plt.subplots(2, 1, sharex=True)
fig.suptitle('Checking our intuition about correlation', fontsize=14, fontweight='bold')
# make space for the title
fig.subplots_adjust(top=0.85)

rolling_correlation.plot(ax=ax1) ax1.set_title('Rolling correlation of WTI returns vs S&P returns') ax1.axhline(0, color='k') ax1.tick_params( which='both', # both major and minor ticks bottom='off', top='off', right='off', labelbottom='off' # labels along the bottom edge are off )

cl_close.plot(ax=ax2) ax2.set_title('Price of front month WTI crude') ax2.tick_params(which='both', top='off', right='off') ax2.tick_params(which='minor', bottom='off') ax2.yaxis.set_major_locator(MaxNLocator(5)) # how many ticks

Alright, fine. So we can distinctly see the regime change starting from the European debt crisis, when oil came back down from $150/bbl. Traders no longer saw high oil prices as a drag on the economy, and instead focused on their intention on global demand instead as we entered a period of slow growth.

Also, all the recent talk about equity oil correlation, we have actually seen higher correlations in the 2011-2013 period.

So this is an interesting observation. But as data scientists, we must test this hypothesis! If the cause of this recent spike in equity/crude corr is really driven by risk off sentiment, let's see if there is also much stronger cross asset correlation in other risk assets. Stay tuned for the next part of this series!


Quickstart: TensorFlow-Examples on PythonAnywhere

Aymeric Damien's "TensorFlow Examples" repository popped up on Hacker News today, and I decided to take a look. TensorFlow is an Open Source library Machine Intelligence, built by Google, and Aymeric's examples are not only pretty neat, but they also have IPython notebook versions.

Here's how I got it all running on a PythonAnywhere account, from a bash console:

$ pip install --user --upgrade https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.7.1-cp27-none-linux_x86_64.whl
$ git clone git@github.com:aymericdamien/TensorFlow-Examples.git
$ cd TensorFlow-Examples/examples/1\ -\ Introduction/
$ python helloworld.py

That printed out Hello, TensorFlow!, so stuff has clearly installed properly. Let's train and run a neural net.

$ cd ../3\ -\ Neural\ Networks/
$ python multilayer_perceptron.py

That downloads some test data (a standard set of images of digits), and trains the net to recognise them.

Now, as I'm a paying PythonAnywhere customer, I can run IPython Notebooks. So, on the "Files" tab, I navigated to the TensorFlow subdirectory of my home directory, then went into the notebooks subdirectory, then down into 3 - Neural Networks. I clicked on the multilayer_perceptron.ipynb file, and got a notebook. It told me that it couldn't find a kernel called "IPython (Python 2.7)", but gave me a list of alternatives -- I just picked "Python 2.7" and clicked OK.

Next, I tried to run the notebook ("Cell" menu, "Run all" option). It failed, saying that it couldn't import input_data. That was easy to fix -- it looks like that module (which is the one that downloads the training dataset) is in the repository's examples subdirectory, but not in the notebooks one. Back to the bash console in a different tab:

$ cp ~/TensorFlow-Examples/examples/3\ -\ Neural\ Networks/input_data.py ~/TensorFlow-Examples/notebooks/3\ -\ Neural\ Networks/

...then back to the notebook, and run all again -- and it starts training my network again :-)

Now, the next step -- to try to understand what all this stuff actually does, and how it works. I suspect that will be the difficult part.


Security update applied for CVE-2016-0728

Yesterday, Yevgeny Pats of Perception Point security announced publicly a local privilege escalation vulnerability in the Linux kernel, which has been given the code CVE-2016-0728. Most Linux systems had this vulnerability.

Using his code, an attacker who could run programs on a vulnerable system as a non-root user could either crash it, or become the root user. Letting other people -- you, our users -- run code on our servers is what PythonAnywhere is all about :-) But, of course, when you run code, it's not as the root user. If someone managed to become root on a PythonAnywhere server, there's a possibility that they would be able to see other people's stuff -- though, because of our sandboxing system, they'd need to make use of further vulnerabilities to do that, and we're not aware of any such vulnerabilities. (To put it another way -- your code is running in a sandbox. If you'd managed to become root, you'd still be in the sandbox. We don't think that even root could escape our sandbox.)

Perception Point had practiced responsible disclosure of this vulnerability, so when they published their notes on it publicly, the various Linux distributions had known about it for a few days, and had patches available. We immediately applied these patches, and as of 10pm UTC yesterday, all PythonAnywhere servers on which you can execute code -- the ones used for consoles, web apps, and for scheduled tasks -- were running kernel version 3.13.0-76, which Ubuntu released to patch this specific problem.

As an aside, we've also attempted to exploit this vulnerability in our own test instances of PythonAnywhere, and on a number of virtual machines, in order to understand it better. We were not able to use it to become root, although we were able to crash some of the VMs. This makes a certain amount of sense -- we believe that the vulnerability does not always work, and when it fails, it can crash the kernel. As we were able to crash VMs several times, but not get to root, it seems that crashes outnumber privilege escalation. So we'd expect that if someone had been trying to use it on PythonAnywhere before it was announced, we'd have seen a number of inexplicable crashes of our console servers' operating systems (which would be the most likely place for someone to run it). We haven't noticed anything like that recently. So while that doesn't prove anything definitively, it's a comforting indicator.


New stuff: UI changes, new packages, and limited Java

A big infrastructure update this morning, but we managed to fit in some nice new features too!

  • A popular request, especially from people using PythonAnywhere in education: when you run a Python file from inside our in-browser code editor, it now shows the output of the file in a pane underneath the editor, instead of trying to pop up a new browser tab.
  • Plotly is a popular package for generating interactive charts, especially in IPython notebooks. We now have it installed by default.
  • For anyone who's keen on using external databases that need ODBC to connect, we now have all of the operating system packages installed to support it on our site. Just run pip2.7 install --user pyodbc (adjusting the Python version as required) to get the Python package.
  • We have no plans to become JavaAnywhere, but Java can be useful for some purposes, even in a Python program. We've started the process towards making it available; if you have Docker consoles enabled for your account, the Java command-line program will work. It doesn't currently work in scheduled tasks or web applications. We'll be extending support for Java in the future, but if all you want to do is run it from the command line, get in touch and we'll enable it for your account. Feedback will definitely be appreciated!

Seasonal notebooks!

We've got a present for all of our paying customers -- if you celebrate Christmas, you can call it a Christmas present, and if you don't you can just call it a present :-)

IPython notebooks are now available on all paid accounts on PythonAnywhere. Give them a go! You can start a new one, or run one that you've saved or uploaded, from the "Files" tab. It's a new feature, so we're particularly keen on hearing any feedback you'd like to send us.

If you're not a paying customer yet, you don't need to feel left out -- we'll be supporting them (perhaps with a couple of limitations) on free accounts in the New Year.

Happy holidays to everyone, and we look forward to seeing you in the New Year.


Security advisory: please change your PythonAnywhere MySQL password

tl;dr: On 19 November we were notified of a security vulnerability on PythonAnywhere. It could not have been used to access files on your PythonAnywhere storage, including your code, nor was any personally identifiable data in our databases exposed. We also have no indication that it was ever exploited. It was fixed within two hours of notification. However, as a precautionary measure, we are recommending that if you use MySQL on PythonAnywhere, you should change your MySQL password.

Here are the details.

On 19 November a user notified us of a security issue. In certain specific circumstances, a paying PythonAnywhere user could get a list of running processes on one of our console servers. We identified the problem and pushed a fix live 90 minutes later. We have analysed our logs, and cannot see any evidence of anyone having exploited this vulnerability, but clearly we cannot rule out the possibility that someone did exploit it in the past, or did so in a way we can't detect.

If someone did exploit it, they would have been able to see the usernames of all people logged in to that console server, and the full command lines of any processes they were running. They would not have had access to any user file storage, or to any of the databases where we store any personally identifiable information (for example, names and addresses). We do not store credit card numbers on our servers.

However, because full command lines for running processes were visible, any private information that you had on command lines could have been seen by such an attacker. This includes any MySQL passwords, as these are visible in the command line if you happen to have a MySQL console (started from the "Databases" tab) running on the server. Postgres passwords are not at risk, as these are never specified on a command line (unless you explicitly put them on a command line yourself).

As an example of a non-MySQL credential that might have been potentially exposed, imagine you're using a third-party service that sends text messages. Normally you would interact with this using Python, but if you're just playing around, you might use curl from a bash console like this:

curl https://api.provider.com/send-message -u '[YOUR ACCOUNT CREDENTIALS]' -d "message=Hello" -d "number=+15551234567"

You can see that if an attacker had used the vulnerability at exactly the right time, they might have seen those credentials.

Once again, we have no evidence to suggest that anyone has exploited this security hole, and it is now fixed. However, we strongly advise that as a precautionary security measure, you change any MySQL password you have set on the "Databases" tab within PythonAnywhere, and that you also change any other login credentials that might have been exposed on a command line.

If you have any questions, please email us at support@pythonanywhere.com.


A beginner's guide to building a simple database-backed Flask website on PythonAnywhere

It's really easy to get started with Flask on PythonAnywhere, but if it's the first database-backed website you've ever built, it can feel a little daunting. Here are some step-by-step instructions. We'll build a really simple website -- just a page where anyone can leave a comment, with the comments stored in a database so that they last forever. We'll also password-protect it so that it doesn't fill up with spam. Here's what it will look like:

We assume that you've got a little bit of basic Python and HTML knowledge -- for example, that you've done an online course in both of them. Everything else we'll explain as we go along. Let's get started!

First steps

Firstly, create a PythonAnywhere account if you haven't already. A free "Beginner" account is enough for this tutorial.

Once you've signed up, you'll be taken to the "Consoles" tab:

You can dismiss the green welcome section at the top (use the "X" in its top right) -- everything there is accessible later from the "Help" link if you want to see it.

Now, click on the "Web" tab, and you'll be taken to a page where you can create a website:

Click on the "Add a new web app" button to the left. This will pop up a "Wizard" which allows you to configure your site. If you have a free account, it will look like this:

If you decided to go for a paid account (thanks :-), then it will be a bit different:

What we're doing on this page is specifying the host name in the URL that people will enter to see your website. Free accounts can have one website, and it must be at yourusername.pythonanywhere.com. Paid accounts have the option of using their own custom host names in their URLs.

Once a website has been created, it's on that particular host name forever. There are easy ways to create new websites that use the same code as an existing one, so if you create a website at yourusername.pythonanywhere.com and then later want to move it to www.yourowndomain.com then that's pretty simple.

For now, we'll stick to the free option. If you have a free account, just click the "Next" button, and if you have a paid one, click the checkbox next to the yourusername.pythonanywhere.com, then click "Next". This will take you on to the next page in the wizard.

This page is where we select the web framework we want to use. This tutorial is for Flask, so click that one to go on to the next page.

PythonAnywhere has various versions of Python installed, and each version has its associated version of Flask. You can use different Flask versions to the ones we supply by default, but it's a little more tricky (you need to use a thing called a virtualenv), so for this tutorial we'll create a site using Python 3.4, with the default Flask version. Click the option, and you'll be taken to the next page:

This page is asking you where you want to put your code. Code on PythonAnywhere is stored in your home directory, /home/yourusername, and in its subdirectories. Now, Flask is a particularly lighweight framework, and you can write a simple Flask app in a single file. PythonAnywhere is asking you where it should create a directory and put a single file with a really really simple website. The default should be fine; it will create a subdirectory of your home directory called mysite and then will put the Flask code into a file called flask_app.py inside that directory.

(It will overwrite any other file with the same name, so if you're not using a new PythonAnywhere account, make sure that the file that it's got in the "Path" input box isn't one of your existing files.)

Once you're sure you're OK with the filename, click "Next". There will be a brief pause while PythonAnywhere sets up the website, and then you'll be taken to the configuration page for the site:

You can see that the host name for the site is on the left-hand side, along with the "Add a new web app" button. If you had multiple websites in your PythonAnywhere account, they would appear there too. But the one that's currently selected is the one you just created, and if you scroll down a bit you can see all of its settings. We'll ignore most of these for the moment.

Before we do any coding, let's check out the site that PythonAnywhere has generated for us by default. Right-click the host name, just after the words "Configuration for", and select the "Open in new tab" option; this will (of course) open your site in a new tab, which is useful when you're developing -- you can keep the site open in one tab and the code and other stuff in another, so it's easier to check out the effects of the changes you make.

Here's what it should look like.

OK, it's pretty simple, but it's a start. Let's take a look at the code! Go back to the tab showing the website configuration (keeping the one showing your site open), and click on the directory titled "Source code" under the "Code" section:

You'll be taken to a different page, showing the contents of the subdirectory of your home directory where your website's code lives:

Click on the flask_app.py file, and you'll see the (really really simple) code that defines your Flask app. It looks like this:

It's worth working through this line-by-line:

from flask import Flask

As you'd expect, this loads the Flask framework so that you can use it.

app = Flask(__name__)

This creates a Flask application to run your code.

@app.route('/')

This decorator specifies that the following method defines what happens when someone goes to the location "/" on your site -- eg. if they go to http://yourusername.pythonanywhere.com/. If you wanted to define what happens when they go to http://yourusername.pythonanywhere.com/foo then you'd use @app.route('/foo') instead.

def hello_world():
    return 'Hello from Flask!'

This simple function just says that when someone goes to the location, they get back the (unformatted) text "Hello from Flask".

Try changing it -- for example, to "This is my new shiny Flask app". Once you've made the change, click the "Save" button at the top to save the file to PythonAnywhere:

...then the reload button (to the far right, looking like two curved arrows making a circle), which stops your website and then starts it again with the fresh code.

A "spinner" will appear next to the button to tell you that PythonAnywhere is working. Once it has disappeared, go to the tab showing the website again, hit the page refresh button, and you'll see that it has changed as you'd expect.

Keeping our code under control

OK, so we've seen the code, and we've changed it. Now, professional developers don't change code without it being under some kind of source code control, so let's get that set up before we change anything else.

(Coding without source-code control is basically trying to program with one hand tied behind your back. It's possible, but you're making stuff unnecessarily hard for yourself. With a source-code control system like git, you can "commit" your code at any point to make a place that you can easily get back to. So, before you embark on major changes, you can make sure that if you mess up, you can get back to where you were before you started: you'll be able to roll back to the last working version. You can also do stuff like maintaining multiple parallel versions of your code -- say, in-development and live -- and, later on, you can easily collaborate with others. git is actually quite simple to use for the basic stuff, though it can get daunting, if not terrifying, once you get into the more advanced features. This tutorial will give the simple git commands you need to use for the basic checkpoint-rollback stuff, which is probably the most important bit.)

Back in the tab where you were editing the source code file, go back to the page showing the directory listing (by clicking the browser's "Back" button, or clicking the second-last item, mysite, in the "breadcrumb" listing separated by "/" characters just above the editor). In the directory listing page, click the "Open bash console here" link near the top right.

This will start an in-browser command line where you can enter bash commands -- like git ones.

Type in the ls command to list the contents of the directory:

Right, let's get that under source code control. Firstly, run the following two commands to configure git (replacing the stuff in the quotes appropriately):

git config --global user.name "Your Name"
git config --global user.email "you@example.com"

You won't need to run that again on PythonAnywhere. The next step is to create a source-code control "repository" in this directory, which is another simple command:

git init

It should print out something like "Initialized empty Git repository in /home/yourusername/mysite/.git/"

Now, git is going to track all changes to files in this directory, but there's some stuff we want it to ignore -- temporary files created by Python and that kind of thing. You do this by creating a file called ".gitignore". Let's create one with some appropriate stuff for a Flask project in it; first, type this command:

cat > .gitignore

Anything you type into the console from now on will be added to the .gitignore file, so enter the following items, each on its own line:

*.pyc
__pycache__

Once you've entered the last one, and hit return, then type Control-D. This will go back to the bash "$" prompt.

Now we can ask git which files it can see. Type

git status

You'll see something like this:

On branch master

Initial commit

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        .gitignore
        flask_app.py

nothing added to commit but untracked files present (use "git add" to track)

So, it's saying that it doesn't see any changes to files that it is tracking -- which makes sense, because it hasn't been told to track any files yet. But it can see two files that it isn't tracking yet -- our source file, and the .gitignore file we just created. We want it to track those files, so we type:

git add .gitignore flask_app.py

...to add them both. Run git status again to see what it thinks of that, and you'll see something like this:

On branch master

Initial commit

Changes to be committed:
  (use "git rm --cached <file>..." to unstage)

        new file:   .gitignore
        new file:   flask_app.py

So, now it knows about the files. We want to make a commit now -- that is, we want to store the current state of the files in the repository so that we can get back to this state easily in the future. Run this command to make a commit with an appropriate comment (the bit in the quotes):

git commit -m"First commit of a really simple web app, with a .gitignore file"

It will print out something like this:

[master (root-commit) 962d9c7] First commit of a really simple web app, with a .gitignore file
 2 files changed, 13 insertions(+)
 create mode 100644 .gitignore
 create mode 100644 flask_app.py

Now we can see what the status is after that by running git status again:

On branch master
nothing to commit, working directory clean

This means that as far as git is concerned, everything is in order.

Let's make some changes so that we can show how to roll them back; go back to the code editing window, by clicking "Back" to get to the directory listing, then clicking on the source file again. Let's add a new "view" -- that is, a new URL that your web app will respond to. Right now, it will only handle requests to http://yourusername.pythonanywhere.com/ -- it will respond with a "not found" error if, for example, you go to http://yourusername.pythonanywhere.com/wibble. (Check that in the tab where you're viewing your website.)

So let's add some code to handle the "/wibble" version. Just put this at the bottom of the source code file:

@app.route('/wibble')
def wibble():
    return 'This is my pointless new page'

Click the "Save" button again, then the reload button, and check out your site on the tab you kept open earlier. Now, both http://yourusername.pythonanywhere.com/ and http://yourusername.pythonanywhere.com/wibble will work, and will return different text.

So that was our first change, but it was pretty pointless. Let's use git to undo it. Click on the PythonAnywhere logo at the top right of the editor page, and you'll be taken to the main PythonAnywhere "Consoles" list. You'll see that under "Your consoles" there's a thing called something like "Bash console 1904962".

When we opened the bash window earlier, it created a persistent console that will carry on running on PythonAnywhere, even when you're not viewing it. (It may be reset due to server maintenance on our side, but in general it will keep running for a long time.) So if you click on it, it will take you back to where you were before:

Let's see what git thinks is going on, by running git status again. You'll get something like this:

On branch master

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   flask_app.py

no changes added to commit (use "git add" and/or "git commit -a")

So it can see that we changed the flask_app.py file. That's the change we want to undo. Usefully, it even tells us what to do to roll back a particular file. Run this:

git checkout -- flask_app.py

Run git status again, and you'll see that it no longer sees that file as changed:

On branch master
nothing to commit, working directory clean

Let's confirm that our change really was un-done. Click the PythonAnywhere logo again, then click on the "Files" tab next to the "Consoles" one. That will take you to a directory listing for your home directory. From there, click on the directory containing the code for your website (mysite if you stuck with the default) and you'll get a directory listing for that, which should contain the flask_app.py file. Click on that, and you'll be taken to the editor again, and you'll see that the pointless change has been backed out for us.

Hopefully that was at least instructive in showing how you can easily back out changes if you make a mess of your program while developing :-)

Right, now let's start working on our app.

A first cut with dummy data

A web app in Flask, like most frameworks, consists of two kinds of file: source code, which is written in Python, and templates, which are written in an extended version of HTML. Basically, the source code says what the web app should do, and the templates say how it should be displayed.

Our website is just going to be a bunch of comments, one after another, with a text box at the bottom so that people can add new ones. So we'll start off by writing a template that displays some dummy data, and a view in the Python code that renders (that is, displays) the template.

The template first. Click on the back button to get to your web app's source code directory listing page. Templates, by convention, go in a subdirectory of your source code directory, inventively called templates. So we can create that by typing "templates" into the "Enter new directory name" input near the top of the page, then clicking the "New" button next to it. That will take us to the directory listing for the new directory, which of course is empty. We'll create our template file in there; type "main_page.html" into the "Enter new file name" input, and click the "New" button next to it. This will take you to the editor, which will of course be empty.

Type the following HTML into the editor:

<html>
    <head>
        <title>My scratchboard page</title>
    </head>

    <body>

        <div>
            This is the first dummy comment.
        </div>

        <div>
            This is the the second dummy comment.  It's no more interesting
            than the first.
        </div>

        <div>
            This is the third dummy comment.  It's actually quite exciting!
        </div>

        <div>
            <form action="." method="POST">
                <textarea name="contents" placeholder="Enter a comment"></textarea>
                <input type="submit" value="Post comment">
            </form>
        </div>

    </body>
</html>

Save it, and go back to the source code directory (you can use the breadcrumbs at the top of the page -- remember, they're the things separated by "/"s) and edit your source code file. We need to change it so that it renders that template. To do that, replace the existing view, which looks like this:

@app.route('/')
def hello_world():
    return 'This is my new shiny Flask app'

...with one that looks like this:

@app.route("/")
def index():
    return render_template("main_page.html")

You also need to change the line at the top that says

from flask import Flask

to import the render_template function too, like this:

from flask import Flask, render_template

Also, just to help in case you've made a typo in the code somewhere, add this line just after the line that says app = Flask(__name__)

app.config["DEBUG"] = True

Now, save the file, reload the website, and refresh it in your other tab. It'll look like this:

That's pretty ugly, but it has our stuff from the template -- it all worked :-)

Let's take another checkpoint using git. This time, to save us from having to click around all the time, we'll open up our bash console in a new browser tab. Go back to your source code editor tab, right-click the PythonAnywhere logo, and select the option to open the link in a new tab. In the new tab, click on the "Bash console" link under "Your consoles". You should now have three tabs open -- the one where you're editing your source code, the one with the bash console, and the one viewing your site.

In the bash console, type git status. It will say something like this:

On branch master

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   flask_app.py

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        templates/

no changes added to commit (use "git add" and/or "git commit -a")

This means that it can see that we've changed flask_app.py, and also that we've added a directory called templates. It doesn't say anything about what's inside the templates directory.

We want to add both of these in one commit; taken together, the addition of a template and the code to render it make a single consistent change to our app, so it makes sense to bundle the two together. So, add the new directory

git add templates/

...and add the file

git add flask_app.py

Run git status to see what the situation now is:

On branch master
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

        modified:   flask_app.py
        new file:   templates/main_page.html

Now it's telling us about the contents of the templates directory. (If we'd put more files into it, then the git add templates/ would have added all of them.)

So now we commit those changes to the repository in one, by running this command:

git commit -m"A first cut at the template"

It will print out something like this:

[master 4e8b505] A first cut at the template
 2 files changed, 34 insertions(+), 4 deletions(-)
 create mode 100644 templates/main_page.html

Now, it's useful to know what's happened in a repository in the past; so far we've done two commits. What do they look like? Run the command git log -- it will print out the revision history, something like this:

commit 4e8b50560bb4ec32a273e3ad68faf1a8cb87fafa
Author: A PythonAnywhere dev <support@pythonanywhere.com>
Date:   Fri Oct 30 17:39:34 2015 +0000

    A first cut at the template

commit 962d9c7115a7169bcd6204b6c08911447cfd2e13
Author: A PythonAnywhere dev <support@pythonanywhere.com>
Date:   Fri Oct 30 16:24:54 2015 +0000

    First commit of a really simple web app, with a .gitignore file

You can see that there are two commits shown, with the most recent one at the top. Those complicated hexadecimal numbers just after the word "commit" are unique identifiers for the specific checkpoints you've made to your code. More advanced usage of git often involves using those identifiers to undo specific changes, or to pass sets of changes to other developers. But we won't go into that here...

A less ugly site

So our change is committed. Let's make the site look a bit prettier. Go back to your source code tab. Now we want to edit the template. We could do that here, but let's open yet another tab so that we can switch between editing the different files easily. Right-click on the source code directory in the breadcrumb trail, and open it in a new tab. In the new tab, go to the templates subdirectory, then click on your main_page.html template file.

Now you have four tabs:

  • The source code, flask_app.py
  • The template, main_page.html
  • The bash console for git
  • The web site itself.

To make the site prettier, we'll use the popular Bootstrap framework. Inside the <head> section of your HTML, before the title, add this:

<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">

<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css" integrity="sha512-dTfge/zgoMYpP7QbHy4gWMEGsbsdZeCXz7irItjcC3sPUFtf0kuFbDz/ixG7ArTxmDjLXDmezHubeNikyKGVyQ==" crossorigin="anonymous">

Save the file, then go to the tab that's showing your site, and hit the page refresh button there. You'll see that the font has become a bit nicer, but it's still not super-pretty.

(Eagle-eyed readers will have spotted that this time we didn't have to click the "Reload" button in the editor. For arcane technical reasons, you have to reload the web site every time you change Python code, but not when you change templates.)

Let's improve things a bit more. In the template editor tab, add this at the start of the "Body" section:

<nav class="navbar navbar-inverse">
  <div class="container">
    <div class="navbar-header">
      <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar">
        <span class="sr-only">Toggle navigation</span>
        <span class="icon-bar"></span>
        <span class="icon-bar"></span>
        <span class="icon-bar"></span>
      </button>
      <a class="navbar-brand" href="#">My scratchpad</a>
    </div>
  </div>
</nav>


<div class="container">

...then, at the end of the body, just before the </body> tag, add this:

</div><!-- /.container -->

Next, for each of the

<div>

lines containing our dummy comments, add class="row" before the close >, like this:

<div class="row">

Also add the same to the <div> that contains the <form>.

For the <textarea> tag, add class="form-control".

Go to the tab showing your page, and refresh -- it should look like this:

If it looks weird and ugly (or, at least, weirder and more ugly than that screenshot), check your HTML template and see if there are any obvious errors. It's really worth spending some time trying to fix the errors yourself, as that's the best way to learn how things work, but if you really find yourself struggling, here's a complete copy of what should be in the template file so that you can copy and paste it into the editor:

<html>
    <head>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">
        <meta name="viewport" content="width=device-width, initial-scale=1">

        <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css" integrity="sha512-dTfge/zgoMYpP7QbHy4gWMEGsbsdZeCXz7irItjcC3sPUFtf0kuFbDz/ixG7ArTxmDjLXDmezHubeNikyKGVyQ==" crossorigin="anonymous">

        <title>My scratchboard page</title>
    </head>

    <body>
        <nav class="navbar navbar-inverse">
          <div class="container">
            <div class="navbar-header">
              <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar">
                <span class="sr-only">Toggle navigation</span>
                <span class="icon-bar"></span>
                <span class="icon-bar"></span>
                <span class="icon-bar"></span>
              </button>
              <a class="navbar-brand" href="#">My scratchpad</a>
            </div>
          </div>
        </nav>

        <div class="container">

            <div class="row">
                This is the first dummy comment.
            </div>

            <div class="row">
                This is the the second dummy comment.  It's no more interesting
                than the first.
            </div>

            <div class="row">
                This is the third dummy comment.  It's actually quite exciting!
            </div>

            <div class="row">
                <form action="." method="POST">
                    <textarea class="form-control" name="contents" placeholder="Enter a comment"></textarea>
                    <input type="submit" value="Post comment">
                </form>
            </div>

        </div>

    </body>
</html>

OK, so we have a reasonably pretty page. Let's get that committed into our git repository! Go to your Bash console tab, and run git status:

On branch master
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   templates/main_page.html

no changes added to commit (use "git add" and/or "git commit -a")

It can see that we've just got changes to our template, so we add that file to the list of things to commit...

git add templates/main_page.html

...then we commit it.

git commit -m"Made the site prettier."

Run git status again to see that everything's properly committed, and git log to see that the new commit has been added to the start of the list.

Sending and receiving data

Let's go back to the tab showing our site. Right now, if we type something into the text area for entering new comments, and click on the "Post comment" page, we'll get a weird error -- which makes perfect sense, as we haven't written any code to store anything. Try it! You'll get an error like this:

What this error means is that we tried to send data to the page, but it's not configured to receive data, only to send it. When a browser accesses a page, it sends a message which contains an item saying what "method" it wants to use to access it. By convention, this is "GET" to view data, or "POST" to send data -- that's why the <form> tag in our template says method="POST". (Like everything in programming, it's actually a bit more complicated than that, but that's a good start :-)

Right now, our view only handles "GET" methods, because that's what Flask views do by default. Let's fix that; it's really simple. Go to the tab showing your Python code, and replace

@app.route("/")

...with this:

@app.route("/", methods=["GET", "POST"])

Save the file, hit the reload button in the editor, then go to the tab showing your page; click back to get away from the error page if it's still showing, then try to post a comment again.

Now you won't get an error, but it will just take you back to the page you were viewing before, and completely forget your comment. Definitely a step forward, but not a very big one.

Let's start storing the comments that people send us. We'll do it an easy way first, without using a database, then we'll add the database in later.

The easy way to store comments is just to do it in a variable inside the Python code. This is not great for the long run, because every time we reload the website, it will stop the program serving the site and then start it again, so all of the comments will be lost. Also, if you're on a paid plan, you might have multiple processes running your code to handle your site (free plans have only one process for the whole site because they're free, paid plans can have lots so that they can handle more simultaneous viewers) and each process will have different sets of comments. So we can't use the variable-based version for the long run, but as a step forward it's definitely worthwhile :-)

First, let's define the variable. In the source code tab, add this in between the code that switches debug mode on, and the @app.route decorator:

comments = []

Next, change the contents of the index function so that it looks like this:

@app.route("/", methods=["GET", "POST"])
def index():
    if request.method == "GET":
        return render_template("main_page.html", comments=comments)

    comments.append(request.form["contents"])
    return redirect(url_for('index'))

Line-by-line, this means:

@app.route("/", methods=["GET", "POST"])

Specifies that the following function defines the view for the "/" URL, and that it accepts both "GET" and "POST" requests.

def index():

The name of the view function, of course

    if request.method == "GET":
        return render_template("main_page.html", comments=comments)

If the request we're currently processing is a "GET" one, the viewer just wants to see the page, so we render the template. We've added an extra parameter to the call to render_template -- the comments=comments bit. This means that the list of comments that we've defined as a variable further up will be available inside our template; we'll update that to use the list in a moment.

If the request isn't a "GET" (and so, we know it's a "POST" -- someone's clicked the "Post comment" button) then the next bit is executed:

    comments.append(request.form["contents"])

This extracts the stuff that was typed into the textarea in the page from the browser's request; we know it's in a thing called contents because that was the name we gave the textarea in the template:

<textarea class="form-control" name="contents" placeholder="Enter a comment"></textarea>

Once we've extracted the comment contents from the request, we add it to the list. Finally, once that's been stored, we send a message back to the browser saying "Please request this page again, this time using a 'GET' method", so that the user can see the results of their post:

    return redirect(url_for('index'))

We just need to make one more change in our code: we've used a bunch of new Flask functions and so we need to import them. Change

from flask import Flask, render_template

to this:

from flask import Flask, redirect, render_template, request, url_for

Save the file, then hit the reload button in the editor. Go to the tab showing your site, and have a play... unfortunately it still won't do anything, even when you post a comment! This is because although we've added code to accept incoming comments, we're not doing anything in the template to display them. So let's fix that.

Go to the tab where you're editing your template, and get rid of the three dummy comments:

<div class="row">
    This is the first dummy comment.
</div>

<div class="row">
    This is the the second dummy comment.  It's no more interesting
    than the first.
</div>

<div class="row">
    This is the third dummy comment.  It's actually quite exciting!
</div>

In their place, we'll put a loop in the Flask template language that will iterate over all of the comments that we passed in to it in the render_template code in our Python code, and put in a <div> for each of them:

{% for comment in comments %}
    <div class="row">
        {{ comment }}
    </div>
{% endfor %}

This should be pretty clear. Commands in the Flask template language, like for and endfor go inside {%..%}, and then you can also use {{..}} and a variable to insert the value of the variable at the current point.

Now, save the file, and go to the tab showing your site. Hit the page refresh button -- you'll see that the dummy comments have gone, and if you posted some comments while you were trying it out before you updated the template, you'll see those have been added instead! Try adding a new new comments; they'll appear as soon as you post them.

If things don't work, try to debug it. Check for typos in your code, and see if there are any errors displayed. If the worst comes to the worst and you really can't work out what you mis-typed, here's the code that you should have:

Python:

from flask import Flask, redirect, render_template, request, url_for

app = Flask(__name__)
app.config["DEBUG"] = True

comments = []

@app.route("/", methods=["GET", "POST"])
def index():
    if request.method == "GET":
        return render_template("main_page.html", comments=comments)

    comments.append(request.form["contents"])
    return redirect(url_for('index'))

Template:

<html>
    <head>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">
        <meta name="viewport" content="width=device-width, initial-scale=1">

        <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css" integrity="sha512-dTfge/zgoMYpP7QbHy4gWMEGsbsdZeCXz7irItjcC3sPUFtf0kuFbDz/ixG7ArTxmDjLXDmezHubeNikyKGVyQ==" crossorigin="anonymous">

        <title>My scratchboard page</title>
    </head>

    <body>
        <nav class="navbar navbar-inverse">
          <div class="container">
            <div class="navbar-header">
              <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar">
                <span class="sr-only">Toggle navigation</span>
                <span class="icon-bar"></span>
                <span class="icon-bar"></span>
                <span class="icon-bar"></span>
              </button>
              <a class="navbar-brand" href="#">My scratchpad</a>
            </div>
          </div>
        </nav>

        <div class="container">

            {% for comment in comments %}
                <div class="row">
                    {{ comment }}
                </div>
            {% endfor %}

            <div class="row">
                <form action="." method="POST">
                    <textarea class="form-control" name="contents" placeholder="Enter a comment"></textarea>
                    <input type="submit" value="Post comment">
                </form>
            </div>

        </div>

    </body>
</html>

So, if everything's working OK, it's probably time to commit the changes. Over to the Bash console tab, and run git status yet again:

On branch master
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   flask_app.py
        modified:   templates/main_page.html

no changes added to commit (use "git add" and/or "git commit -a")

This time, just for a change, let's add our files to the list of files to commit, and commit them in one go. This doesn't work if you've created a new file, but as we've only modified existing ones, it's an option. We just add an a before the m in the git commit command to say "add files":

git commit -am"Replaced dummy comments with some in-memory storage"

Once again, run git status to confirm that everything's now properly synchronized up, and git log to see your new commit in the history for the repository.

Password protection

So, we have a page where random people on the Internet can add comments. We all know what the Internet is like, and this is going to fill up with ads for fake Rolexes and dodgy pharmaceuticals pretty rapidly if we leave it open to the world. So let's add some password protection. PythonAnywhere provides this for us.

To activate it, go to any of your tabs, right-click the PythonAnywhere to open the consoles page in a new browser tab, then click on the "Web" tab. Make sure your website is selected on the left-hand side, then scroll to the bottom of the page. Just above the big red "delete" button there's a section titled "Password protection". Enter a username and a password here, and switch the toggle above those fields to "Enabled". Scroll up to the top of the page again, and click the big green "Reload" button.

Now head over to the tab where you're viewing your website, and refresh the page. This time you should be prompted for a password. You only need to enter it this one time for this browser session, so it won't get in our way later on -- but it will keep the spammers at bay... Once this is all done, you can close the browser tab where you set the password.

Bring on the database

Now it's time to add a database in to the mix! You've probably already noticed the problem that adding one will solve; when you reloaded your website as part of enabling the password protection, all of your comments disappeared. So right now we have a site that will lose all of its data every time you reload it -- which means, of course, every time you change the code. Not ideal.

We could store the comments in a simple file -- and that would work fine for a basic site like this, but would become increasingly hard to maintain as we added new stuff in the future. Databases were designed to keep structured data in an easy-to-maintain, easy-to manage fashion, and that's why we're going to use one.

There are many database programs used for web apps; some examples are:

  • SQLite, a simple lightweight database that keeps everything in a specially-formatted file on your disk. Not super-high-performance, but popular because it's so simple.
  • MySQL, which runs as a separate server process to which programs running on other machines can connect, storing data on the disks of the machine where it's running. Fairly simple, pretty powerful. Probably the most popular option.
  • PostgreSQL (for historical reasons, normally just called Postgres), which runs as a server, like MySQL, and is better with larger datasets. Growing in popularity, but needs a bit more server power to run.
  • MongoDB, which allows you to store your data in a less-structured way than SQL databases like the previous three. Also server-based. New, slightly oddball -- the hipster option.

SQLite is pretty slow on PythonAnywhere, and doesn't scale well for larger sites anyway. Postgres is a paid feature on PythonAnywhere because of its larger server power requirements, and we don't have built-in support for MongoDB. So we're going to use MySQL.

One convenient thing about modern database programming, however, is that to a certain degree you don't need to worry about which database you're actually using. There are database interface libraries called Object-Relational Mappers (ORMs), which make it possible (most of the time) to use the same code regardless of the underlying database. We're going to use one called SQLAlchemy, which works well with Flask.

Right, enough background. Let's create a MySQL database! In your tab where you're editing the Python source code, right-click on the PythonAnywhere logo and open the page in a new tab. On the "Consoles" page that appears, click on the "Databases" tab to the far right. You'll wind up here:

At the top, under "Connecting", you'll see the details you'll need to connect to the MySQL server associated with your account. Next is a list of MySQL databases, which, as you haven't created any, is of course empty. And at the bottom there's a space to enter a MySQL password. The password is needed to connect to the MySQL server so that other people can't connect to your one, and initially it's unset. Enter a new password here; make it something different to the one you use to log in to PythonAnywhere, because this database password will have to exist in your Python code (so that it can connect to your server), so it's more secure to use a different one. Make a note of the password you've chosen, and once you've entered it twice in the appropriate fields, click the "Set MySQL password" button. You'll come back to the "Databases" tab, but now some extra options will have appeared:

So now we have a database called yourusername$default, and an option to create a database. In general, it's good practice to have a separate database for each website you're working on to keep everything nicely separated. Go to the "Create database" section in the middle of the page, and create one, giving it the name "comments":

Once you've clicked the "Create" button, you'll see it's been added to the list that previously showed the yourusername$default one:

You can see that the database name is slightly different to the simple comments that you specified -- it's got your username and a dollar before it. (If your username is longer than 16 characters, it will be the username truncated to 16 characters.) This is simply to make sure that the name doesn't clash with any other users' databases that happen to be on the same server.

OK, so now we have a database. Let's add some code to connect to it! Leave the databases page open, and switch back to the tab where you have your Python code. Just after the lines that say:

app = Flask(__name__)
app.config["DEBUG"] = True

...add the following code to configure the database connection:

SQLALCHEMY_DATABASE_URI = "mysql+mysqlconnector://{username}:{password}@{hostname}/{databasename}".format(
    username="<the username from the 'Databases' tab>",
    password="<the password you set on the 'Databases' tab>",
    hostname="<the database host address from the 'Databases' tab>",
    databasename="<the database name you chose, probably yourusername$comments>",
)
app.config["SQLALCHEMY_DATABASE_URI"] = SQLALCHEMY_DATABASE_URI
app.config["SQLALCHEMY_POOL_RECYCLE"] = 299

You need to change the values we set for username, hostname and databasename to the values from the "Databases" tab, and change the password to the one you set there earlier. (All of these are without the <> angle brackets, of course -- those are just there in the code above to highlight things you need to change.)

The first of these commands sets a variable to the specifically-formatted string SQLAlchemy needs to connect to your database. The second stashes that away in Flask's application configuration settings.

The third sets an importand database connection configuration parameter, which you don't need to know the details of (though the following sidebar explains it in case you're interested :-)

Sidebar: connection timeouts. Opening a connection from your website code to a MySQL server takes a small amount of time. If you opened one for every hit on your website, it would be slightly slower. If your site was really busy, the aggregate of all of the small amounts of time opening connections could add up to quite a slowdown. To avoid this, SQLAlchemy operates a "connection pool". It keeps a set of connections to the database, and re-uses them. When you want a connection, it gives you one from the pool, creating a new one if the pool is empty, and when you're done with it, the connection is returned to the pool for future reuse. However, in order to stop people from hogging database connections, MySQL servers close unused connections after a particular amount of time. On PythonAnywhere, this timeout is set to 300 seconds. If your site is busy and your connections are always busy, this doesn't matter. But if it's not, a connection in the pool might be closed by the server because it wasn't being used. The SQLALCHEMY_POOL_RECYCLE variable is set to 299 to tell SQLAlchemy that it should throw away connections that haven't been used for 299 seconds, so that it doesn't give them to you and cause your code to crash because it's trying to use a connection that has already been closed by the server.

So that's the code that configures our database connection; now we need some code to actually connect to it! Put this just after the code you just added:

db = SQLAlchemy(app)

This simply creates a SQLAlchemy object using the connections we put into the Flask app's config. We need to add an extra import statement to the start of our code to make this work, just after the other import that imports stuff from Flask:

from flask.ext.sqlalchemy import SQLAlchemy

Right, we've added some code to connect to the database. Let's sanity-check it and make sure we haven't made any silly typos; hit the button to reload your website, and refresh the page in the other tab where you're viewing it. If all is OK, you should get a site that behaves just like it did before. If there's a problem, Flask should tell you what it is.

Once you've got everything working, go to the console where you've been doing git stuff, and use git commit to save a checkpoint.

Next, we need to add a model. A model is a Python class that specifies the stuff that you want to store in the database; SQLAlchemy handles all of the complexities of loading stuff from and storing stuff in MySQL; the price is that you have to specify in detail exactly what you want. Here's the class definition for our model, which you should put in just above the line that says comments = []:

class Comment(db.Model):

    __tablename__ = "comments"

    id = db.Column(db.Integer, primary_key=True)
    content = db.Column(db.String(4096))

An SQL database can seen as something similar to a spreadsheet; you have a number of tables, just like a spreadsheet has a number of sheets inside it. Each table can be thought of as a place to store a specific kind of thing; on the Python side, each one would be represented by a separate class. We have only one kind of thing to store, so we are going to have just one table, called "comments". Each table is formed of rows and columns. Each row corresponds to an entry in the database -- in our simple example, each comment would be a row. On the Python side, this basically means one row per object stored. The columns are the attributes of the class, and are pretty much fixed; for our database, we're specifying two columns: an integer id, which we'll keep for housekeeping purposes later on, and a string content, which holds, of course, the contents of the comment. If we wanted to add more stuff to comments in the future -- say, the author's name -- we'd need to add extra columns for those things. (Changing the MySQL database structure to add more columns can be tricky; the process is called migrating the database. I'll link to useful stuff about that at the end of this tutorial.)

Now that we've defined our model, we need to get SQLAlchemy to create the database tables. This is a one-off operation -- once we've created the database's structure, it's done. Head over to the bash console we've been using so far for git stuff, and run ipython3.4 to start a Python interpreter.

Once it's running, you need to import the database manager from your code:

from flask_app import db

Now, we want to create the tables. This is really easy:

db.create_all()

Once you've done that, the table has been created in the database. Let's confirm that. Go to the tab you kept open that's showing the "Databases" PythonAnywhere tab, and click on your database name (yourusername$comments). This will start a new console, running the MySQL command line. Once it has loaded (it will show a mysql> prompt), run the command:

show tables;

(Don't forget the semicolon at the end!)

This should show you that you have a table called comments. To inspect it and make sure that it has the stuff you'd expect, run the command:

describe comments;

You'll see that we have two columns, one called id, and one called content -- just as you'd expect from the class definition in our Python code!

Let's do another checkpoint. Go back to the tab where you previously did the git stuff, and more recently used the Python interpreter to create the tables. Exit the Python interpreter, using Control-D or exit(). Now let's try out a new git command:

git diff

You'll see that git prints out all of the changes to your code in green, with a bit of code around it in white for context:

That's often useful when you've not done a commit for a while and need a reminder of what you've changed. For now, just do a git add and a git commit to save your changes.

So now we have code in our Flask app to connect to the database, a Python definition of what we want to store in the database, and tables created in the database on the MySQL server to store the data. Let's add the code to actually use all of that!

Firstly, we can get rid of the line where we create the old in-memory storage for the comments, the one like this:

comments = []

Once that's deleted, we can change the code that uses it in the index view. Firstly, let's write code to load all of the comments from the database and pass them to the template rather than use the in-memory variable. Replace this line:

    return render_template("main_page.html", comments=comments)

...with this:

    return render_template("main_page.html", comments=Comment.query.all())

The query attribute is something that SQLAlchemy added to your Comment class allowing you to make queries to the database, and its all method, like you'd expect, just gets all comments.

Now, we need to change the template. Previously, our in-memory list of comments was just a list of strings. Now that we've changed it to query the database, the template is getting a list (or something that will act like a list) of Comment objects. We want to get their content attribute, and the change is pretty much like you'd expect. In the editor for the template, change the bit inside the for loop where we insert the comment:

                {{ comment }}

...to this:

                {{ comment.content }}

Now we've implented the code to display comments, but we still need to write code to add them. Back in the editor for the Python code, replace this:

comments.append(request.form["contents"])

...with this:

comment = Comment(content=request.form["contents"])
db.session.add(comment)
db.session.commit()

The first line creates the Python object that represents the comment, but doesn't store it in the database. The second sends the command to the database to store it, but leaves a transaction open. Transactions allow you to batch up a bunch of changes into one, for efficiency and also so that if an error occurs you can easily abort and have none of them happen; basically it's a way of batching things up into a set of small changes that, taken together, make one logically coherent larger change. As we're only making one change -- adding one comment -- then we can commit immediately, which closes the transaction and stores everything. (You can probably see a similarity between this "prepare things and when you've got a bunch of things ready, commit to store it all" and the git add followed by git commit commands we've been using for git. That's no coincidence! It's a common pattern across lots of areas of programming.)

OK, so we have code to load comments from the database and code to store them in the database. We're done! Save the file, and hit the button at the top of the editor to reload the website, and go over to the tab where you have it open. Refresh the page; it should be the normal page with no comments yet. Type one in, and click the "Post comment" button. It will appear.

Now, the acid test to prove that it's being stored in the database and can survive a website reload: go to the editor window again, and click the button to reload your web app. Wait until it's done, then go back to the website. Refresh the page.... and your comment will still be there!

Ta-dah!

Let's make one final sanity check; we can inspect the database directly and see what's in there. Go back to the MySQL command window that you opened earlier, and run this command:

select * from comments;

In lovely 1970s-style ASCII-art table graphics, you'll get the contents of your database -- one shiny new row (or more if you've entered multiple comments):

+----+-------------------------------------------+
| id | content                                   |
+----+-------------------------------------------+
|  1 | This is my first comment for the database |
+----+-------------------------------------------+
1 row in set (0.00 sec)

So now we have a website that will allow people to enter random comments, and will remember them even if the website is reloaded. Everything has been coded in Flask, and the data is stored in MySQL using SQLAlchemy. One last thing before we can call this done, though -- go over to the console where you've been running git, and do one last git commit:

git commit -am"Yay, it's all finished :-D"

You've built a simple database-backed Flask website on PythonAnywhere.

I hope it all went smoothly for you, and if you had any problems, just post a comment below!

If you'd like to study further, and find out more about Flask, I recommend these tutorials:

If you're thinking of developing this app further and adding new stuff to the database, you'll need to learn about database migrations; here are the docs

If you think we should do a follow-up to this tutorial, just post a comment with your suggestion below :-)

Thanks for reading!


Page 3 of 15.

« More recent postsOlder posts »

PythonAnywhere is a Python development and hosting environment that displays in your web browser and runs on our servers. They're already set up with everything you need. It's easy to use, fast, and powerful. There's even a useful free plan.

You can sign up here.