Django Notes: tests, setUp method and db data

This won’t be a full post, but just a quick note (probably the first one of a serie) about development with Django.

When we write a TestCase test, if we have defined a setUp method, it will be called before the execution of each test. One could think that the database is completely reset after each test, but this is not true (not like I was thinking). After each test, whatever we wrote on the database is rolled back. If we create a “Client” row (assuming we have a model called Clients) in our setUp, when we call it the second time the ID won’t be 1 as someone (me included) could expect. It will be 2 instead, because the database has not completely deleted and created from scratch.

This means that we can’t assume that our Client ID will always be 1 for each test and we should rather reference to it in a dinamic way like: self.client.id

This could be a trivial thing for many people but I was not 100% sure about this so I asked for a confirmation on #django IRC room and people (expecially apollo13) was kind enough to explain me how it works.

Soma.fm + Spotify + import.io + Python mashup: automatically create a Spotify playlist with Soma.fm tracks

I’m a big fan of Soma.fm (a 25+ channels streaming radio based in San Francisco) and during the years I’ve been writing clients for this radio for different mobile platforms (Maemo, MeeGo, Harmattan, Windows Phone, BlackBerry 10, Jolla). I love in particular their “Indie Pop Rock” channel that during these years made me discover some very good artists.

When Spotify finally was available in Italy (I’m still using it right now that I live in the UK), something that I always missed was a radio with the same good music. Why not just listening to Soma.fm? Because I like to listen to the music while I commute and in the London Underground it’s nearly impossible to have signal.

So I was thinking: it would be nice to have a Spotify playlist with Soma.fm tracks. Wait a moment…. I can do it!

Frankenstein_Jr_Mel_Brooks_1974

Soma.fm publishes the tracks history with all the tracks streamed during the last hour http://somafm.com/indiepop/songhistory.html so I just needed something to parse this list for me and return me a well formatted version.

Thanks to import.io (it’s a service that takes a web page as input, parse the data and generates a RESTful API to access this data) I was able to easily get the data I needed. At this point I only needed to be able to loop through the list, search each track on Spotify and add it to my playlist.

The source code is fully available here https://github.com/andreagrandi/spotisoma

Note: you can’t just get the code and run it. You will need to get your own import.io api key, generate your import.io api url, get a Spotify application key (the old/deprecated one, since it was nearly impossible for me to use oauth in a simple Python script due to the fact I didn’t have an endpoint to receive the token back. You can get more informations here: https://pyspotify.mopidy.com/en/latest/quickstart/#application-keys ) and set your env variables with your Spotify username and password. Last but not least: the old Spotify library only works with Premium accounts.

Goenv – Go Environment Manager

To briefly explain what Goenv is, I will assume you have previously worked with Python. Basically it’s what Virtualenv is for Python. Goenv (and it’s wrapper goof) creates a folder for a new project and set the $GOPATH env variable to that folder path. At this point every time you do go get, the libraries will be installed in that specific $GOPATH.

It’s very important to use separate $GOPATH for each project, because this allow us to use different library versions for each project and avoid version conflicts.

Installation

Goenv is now installed, we will now install its wrapper goof:

Edit .bashrc (or .zshrc if you use zsh) and append these lines:

How to use it

To create a new go environment use make:

To exit the go environment use deactivate:

To use an environment use workon:

To show available environments use show:

Goenv itself is not enough to manage Go packages. It would be like using Virtualenv only and not using pip and requirements. In a future post I will explain how to use Godep.

 

 

Go: defining methods on struct types

In Go it’s possible to define methods on struct types. The syntax needed for it can be a bit strange for people that are used to define classes and methods in Java, C# etc… but once you learn it it’s quite easy to use.

In my case for example I needed something that could contain a Timer object, a string and a method that could start the timer and call a method at the end of the Timer execution. I implemented it in this way:

The key point is row 6func (timer DeviceTimer) startTimer() { … }” where I defined a method called startTimer and I specify timer DeviceTimer inside the func definition. This basically “extends” the struct DeviceTimer adding that method to it. This means that I can call that method in this way:

This is all you need to do. If you want to read more about this subject, I can suggest to read these two articles:

Note: I’m not a Go expert and these are just my personal notes I’m taking during my learning experience. I’m very keen to share my notes with everyone, but please don’t take them as notes from an expert Go developer.

How to create a Docker image for PostgreSQL and persist data

Before I start, let me confirm to you that official Docker images for PostgreSQL already exist and are available here: https://registry.hub.docker.com/_/postgres/ so this howto wants to be a guide to explain how to create these images and talk about some of the Docker features.

I will assume that you have already installed Docker on your machine. I have tested these instructions both on Ubuntu Linux and OSX (OSX users will need to install boot2docker, instructions are not available in this guide).

Dockerfile

To create a Docker image we need to create a text file named Dockerfile and use the available commands and syntax to declare how the image will be built. At the beginning of the file we need to specify the base image we are going to use and our contact informations:

In our case we are using Ubuntu 14.04 as base image. After these instructions we need to add PostgreSQL package repository and GnuPG public key:

then we need to update the packages available in Ubuntu and install PostgreSQL:

We are installing version 9.3 of PostgreSQL, instructions would be very similar for any other version of the database.

Note: it’s important to have apt-get update and apt-get install commands in the same RUN line, else they would be considered two different layers by Docker and in case an updated package is available it won’t be installed when the image is rebuilt.

At this point we switch to postgres user to execute the next commands:

We switch to root user and we complete the configuration:

We expose the port where PostgreSQL will listen to:

We setup the data and shared folders that we will use later:

Finally we switch again to the postgres user and we define the entry command for this image:

The full Dockerfile is available here https://github.com/andreagrandi/postgresql-docker/blob/master/Dockerfile

Building Docker image

Once the Dockerfile is ready, we need to build the image before running it in a container. Please customize the tag name using your own docker.io hub account (or you won’t be able to push it to the hub):

Running the PostgreSQL Docker container

To run the container, once the image is built, you just need to use this command:

Testing the running PostgreSQL

To test the running container we can use any client, even the commandline one:

When you are prompted for password, type: pguser
Please note that localhost is only valid if you are running Docker on Ubuntu. If you are an OSX user, you need to discover the correct ip using: boot2docker ip

Persisting data

You may have noticed that once you stop the container, if you previously wrote some data on the DB, that data is lost. This is because by default Docker containers are not persistent. We can resolve this problem using a data container. My only suggestion is not to do it manually and use a tool like fig to orchestrate this. Fig is a tool to orchestrate containers and its features are being rewritten in Go language and integrated into Docker itself. So if you prepare a fig.yml configuration file now, you will be able, hopefully, to reuse it once this feature will be integrated into Docker. Please refer to fig website for the instructions to install it (briefly: under Ubuntu you can use pip install fig and under OSX you can use brew install fig).

Save this file as fig.yml in the same folder of the Dockerfile and spin up the container using this command: fig up

If you try to write some data on the database and then you stop (CTRL+C) the running containers and spin up them again, you will see that your data is still there.

Conclusion

This is just an example of how to prepare a Docker container for a specific service. The difficoult part is when you have to spin up multiple services (for example a Django web application using PostgreSQL, RabbitMQ, MongoDB etc…), connect them all together and orchestrate the solution. I will maybe talk about this in one of the next posts. You can find the full source code of my PostgreSQL Docker image, including the fig.yml file in this repository https://github.com/andreagrandi/postgresql-docker