Developing PythonAnywhere on PythonAnywhere - Dogfooding part deux.

We’ve been “dogfooding” PythonAnywhere for about 6 months now - it always seemed right that, if we were building an online development environment, we should probably actually use it to build itself. Pleasingly recursive, to say nothing of helping to focus our dev process.

Until now though, we were using a special, dedicated cluster that replicates the live environment, but that allowed us to do certain things like use NFS to edit code on our dev VMs. But last week we made the switch to actually using the live, production servers.

So we’re now using exactly the same environment as our users - and it’s been quite an eye-opener. It’s made us bump certain things right up our priority queue.

Like, has anyone else noticed that consoles sometimes seem to hang for a few seconds, once every few minutes? Well, it turns out there was blocking call somewhere in our ioloop, but we hadn’t really noticed because we were off in our own little cluster. Boy is it annoying on live though! So we’ve spent a couple of days tracking down the problem, and we’ve now got a fix going through our integration loop.

Although our previous dogfooding was good, this is much better - we truly are feeling the same pain as our users, and we’re properly motivated to get fixing all those little annoyances!

That’s the basic message, for the curious there’s some technical details to follow.

SSHFS as a way of sharing code with Dev VMs

If you’ll forgive me for re-using an old illustration:

Dogfood setup

What you see there is basically the same setup as what we use now - just replace “Dogfood cluster” with “Live Pythonanywhere environment” and “shared via NFS” with “shared via SSHFS”.

We develop using “normal” PythonAnywhere - there’s nothing special about our accounts, other than their being paid-for accounts: we do need several of the premium features, like SSH, unrestricted internet and the ability to start more than 2 consoles!

We start in a Bash console, checking out the PythonAnywhere source code from Github into our personal file storage areas.

Then we use the same Bash consoles to spin up our dev clusters - they’re basically a set of Amazon EC2 micro-instances, which we can control in scripts using boto and fabric, or manually by logging in with SSH

During server builds, we get our servers to mount the source code from our PythonAnywhere file storage over SSHFS. It’s not a very fast filesystem, but it works for our purposes, since the servers only read source code from disk infrequently, working mainly from RAM.

(SSHFS is an absolute nightmare for other purposes. Don’t try and run git against a folder that’s mounted remotely over SSHFS. You’ll be waiting a long time.)

We did look at other options - the idea of having our dev servers hooked into github somehow, and pull down source code automatically whenever they saw new commits - but we decided that would be too much overhead in our dev process. We don’t need individual commits for every single tweak to some bit of HTML/CSS!

For those interested, here’s how we get our dev servers to mount some folders on our personal PythonAnywhere accounts:

def mount_source_tree_via_sshfs():
    run("ssh-keygen -N '' -f ~/.ssh/id_rsa")
    get("/root/.ssh/", "/tmp/pubkey.tmp")
    local("cat /tmp/pubkey.tmp >> ~/.ssh/authorized_keys")
    local("chmod 600 ~/.ssh/authorized_keys")
    local("rm /tmp/pubkey.tmp")
    append("/root/.ssh/config", "ServerAliveInterval 60")
    run("ssh-keyscan >> ~/.ssh/known_hosts")

    username = pwd.getpwuid(os.getuid()).pw_name
    path_to_source = os.path.join('/home', username, 'PythonAnywhere', 'server')
    run('mkdir -p /home/anywhere')
    run('sshfs /home/anywhere -o reconnect -C -o workaround=all -o allow_other' % (username, path_to_source))
    append('/etc/fstab', ' /home/anywhere fuse rw,nosuid,nodev,allow_other,max_read=65536 0 0' % (
        username, path_to_source, )

We run unit tests on the dev servers, and we usually run selenium test from our local PCs. Although you can run selenium tests from PythonAnywhere itself using xvfb , it’s still a bit slow (we are looking into it), and besides it’s nice to actually see the browsers pop up in front of you and go through the tests.

So that’s that - PythonAnywhere is a step closer to being entirely developed on PythonAnywhere :-)

Thoughts, comments, suggestions?

comments powered by Disqus