Introduction

In this period of COVID-19 emergency, many countries are publishing COVID related data that is being used by many existing projects and researchers.

The main problem with these data is that they are being released in CSV format on some GitHub repository. While we fully appreciate the opennes of this format, unfortunataly it can introduce an additional work to be done (downloading the data, cleaning it, importing the data into a database, keeping it updated etc...) before someone can consume and analyse the data.

covid-api

covid-api project is a free and open source API service which automatically imports the data from various sources (at the moment we support the John Hopkins CSSE data source) and makes it available as a REST API.

The service is still under development, but an initial version (with regularly updated data) is already available at https://api.covid19data.cloud.

How to use the data

To consume the API you don't need an account nor you need to authenticate in any way. You just need to request the right endpoint using the supported parameters.

Here is an example for Python language:

In [1]: import requests

In [2]: response = requests.get('https://api.covid19data.cloud/v1/jh/daily-reports?last_update_from=2020-04-01&last_update_to=2020-04-03&country=Italy')

In [3]: response.json()
Out[3]:
[{'id': 35343,
'country_region': 'Italy',
'province_state': None,
'fips': None,
'admin2': None,
'last_update': '2020-04-01T21:58:34',
'confirmed': 110574,
'deaths': 13155,
'recovered': 16847},
{'id': 37895,
'country_region': 'Italy',
'province_state': None,
'fips': None,
'admin2': None,
'last_update': '2020-04-02T23:25:14',
'confirmed': 115242,
'deaths': 13915,
'recovered': 18278}]

Further API documentation is available at https://api.covid19data.cloud/docs

Next steps

While we keep polishing the code and improving the existing data import procedure, we are planning to support additional data sources. The next one we are going to support is the Italian Protezione Civile.

If you are aware of an additional data source that you would like to see covered, please let us know (creating a new Issue on GitHub) or send us a pull request.

Contribute to the project

If you are a Python developer and would like to contribute to the project, my advice is to first have a look at the main documentation available in the README.

Then I suggest to have a look at the existing Issues and see where help is needed or in alternative you can open a new Issue or send a pull request with fixes and improvements.

I also recommend to become familiar with our Code of Conduct before sending any contribution.

Sponsors and Thanks

I want to thank Heroku for accepting to sponsor the hosting of this service.

I also want to thank all the volunteers involved in the project for their help and contributions.

Disclaimer

We are doing our best to keep the available data updated, clean (removing duplicates), and to provide a reliable service, but we are not in any way responsible for the accuracy of the data nor for the availability of the service itself. Please use it at your own risk.

Abuse notice: we are currently not requiring any registration or authentication to use this service because we would like to keep it as simple as possible. Please do not abuse the service or you will force us to require a registration (subject to approval) to continue using it.

In the previous part of the tutorial we implemented details management, relations between models, nested APIs and a different level of permissions. Our API is basically complete but it is working properly? Is the source code free of bugs? Would you feel confident to refactor the code without breaking something? The answer to all our question is probably no. I can't be sure if the code behaves properly nor I would feel confident refactoring anything without having some tests coverage.

As I mentioned previously, we should have written tests since the beginning, but I really didn't want to mix too many concepts together and I wanted to let the user concentrate on the Rest Framework instead.

Test structure and configuration

Before beginning the fourth part of this tutorial, make sure you have grabbed the latest source code from https://github.com/andreagrandi/drf-tutorial and you have checked out the previous git tag:

git checkout tutorial-1.14

Django has an integrated test runner but my personal choice is to use pytest, so as first thing let's install the needed libraries:

pip install pytest pytest-django

As long as we respect a minimum of conventions (test files must start with test_ prefix), tests can be placed anywhere in the code. My advice is to put them all together in a separate folder and divide them according to app names. In our case we are going to create a folder named "tests" at the same level of manage.py file. Inside this folder we need to create a __init__.py file and another folder called catalog with an additional __init__.py inside. Now, still at the same level of manage.py create a file called pytest.ini with this content:

[pytest]
DJANGO_SETTINGS_MODULE=drftutorial.settings

Are you feeling confused? No problem. You can checkout the source code containing these changes.

git checkout tutorial-1.15

You can check if you have done everything correctly going inside the drftutorial folder (the one containing manage.py) and launching pytest. If you see something like this, you did your changes correctly:

(drf-tutorial) ➜  drftutorial git:(master) pytest
============================================================================================================================= test session starts ==============================================================================================================================
platform darwin -- Python 2.7.13, pytest-3.0.6, py-1.4.32, pluggy-0.4.0
Django settings: drftutorial.settings (from ini file)
rootdir: /Users/andrea/Projects/drf-tutorial/drftutorial, inifile: pytest.ini
plugins: django-3.1.2
collected 0 items

========================================================================================================================= no tests ran in 0.01 seconds =========================================================================================================================
(drf-tutorial) ➜  drftutorial git:(master)

Writing the first test

To begin with, I will show you how to write a simple test that will verify if the API can return the products list. If you remember we implemented this API in the first part of the tutorial. First of all create a file called test_views.py under the folder drftutorial/tests/catalog/ and add this code:

import pytest
from django.urls import reverse
from rest_framework import status
from rest_framework.test import APITestCase


class TestProductList(APITestCase):
    @pytest.mark.django_db
    def test_can_get_product_list(self):
        url = reverse('product-list')
        response = self.client.get(url)
        self.assertEqual(response.status_code, status.HTTP_200_OK)
        self.assertEqual(len(response.json()), 8)

before being able to run this test we need to change a little thing in the catalog/urls.py file, something we should have done since the beginning. Please change the first url in this way, adding the name parameter:

urlpatterns = [
    url(r'^products/$', views.ProductList.as_view(), name='product-list'),
    ...

at this point we are able to run our test suite again and verify the test is passing:

(drf-tutorial) ➜  drftutorial git:(test-productlist) ✗ pytest -v
============================================================================================================================= test session starts ==============================================================================================================================
platform darwin -- Python 2.7.13, pytest-3.0.6, py-1.4.32, pluggy-0.4.0 -- /Users/andrea/.virtualenvs/drf-tutorial/bin/python2.7
cachedir: .cache
Django settings: drftutorial.settings (from ini file)
rootdir: /Users/andrea/Projects/drf-tutorial/drftutorial, inifile: pytest.ini
plugins: django-3.1.2
collected 1 items

tests/catalog/test_views.py::TestProductList::test_can_get_product_list PASSED

=========================================================================================================================== 1 passed in 0.98 seconds ===========================================================================================================================

To checkout the source code at this point:

git checkout tutorial-1.16

Explaining the test code

When we implement a test, the first thing to do is to create a test_* file and import the minimum necessary to write a test class and method. Each test class must inherit from APITestCase and have a name that start with Test, like TestProductList. Since we use pytest, we need to mark our method with @pytest.mark.django_db decorator, to tell the test suite our code will explicitly access the database. We are going to use the client object that is integrated in APITestCase to perform the request. Before doing that we first get the local url using Django's reverse function. At this point we do the call using the client:

response = self.client.get(url)

and then we assert a couple of things that we expect to be true:

self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(len(response.json()), 8)

We check that our API returns the 200 status code and that in the returned JSON there are 8 elements.

It's normally a good practice to create test data inside the tests, but in our case we previously created a data migration that creates test data. Migrations are run every time we run tests so when we call our API, the data will be already there.

Wrapping up

I've written a few tests for all the views we have implemented until now and they are available if you checkout this version of the code:

git checkout tutorial-1.17

I've only tested the views but it would be nice to test even the permission class, for example. Please remember to write your tests first, if possible: implementing the code will be much more natural once the tests are already in place.

In the previous part we implemented authentication, permissions and the possibility to POST new products for admins. In this new episode we will see how to implement details management, relations between models, nested APIs and a different level of permissions.

If you haven't completed the previous parts or if you want to begin from this one, checkout the right code first:

git checkout tutorial-1.10

Handling Product Details

Our current API methods allow us to list all the products we have in our catalog and to create a new one (if we have admin permissions), but what if we wanted to delete or update a single one? What if we wanted to get only a specific product? We need to handle details.

As first thing we need to change the ProductSerializer to return the id of the product. Edit catalog/serializers.py and change the class in this way:

class ProductSerializer(serializers.ModelSerializer):
    class Meta:
        model = Product
        fields = ('id', 'name', 'description', 'price')

After changing the serializer we need to implement a new view called ProductDetail. Edit catalog/views.py and add the following imports:

from django.http import Http404
from rest_framework.response import Response
from rest_framework.views import APIView
from rest_framework import status

and the following class:

class ProductDetail(APIView):
    def get_object(self, pk):
        try:
            return Product.objects.get(pk=pk)
        except Product.DoesNotExist:
            raise Http404

    def get(self, request, pk, format=None):
        product = self.get_object(pk)
        serializer = ProductSerializer(product)
        return Response(serializer.data)

    def put(self, request, pk, format=None):
        product = self.get_object(pk)
        serializer = ProductSerializer(product, data=request.data)
        if serializer.is_valid():
            serializer.save()
            return Response(serializer.data)
        return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)

    def delete(self, request, pk, format=None):
        product = self.get_object(pk)
        product.delete()
        return Response(status=status.HTTP_204_NO_CONTENT)

let's connect the new view to the urls, editing catalog/urls.py and changing the code in this way:

urlpatterns = [
    url(r'^products/$', views.ProductList.as_view()),
    url(r'^products/(?P<pk>[0-9]+)/$', views.ProductDetail.as_view()),
]

If we try to PUT, DELETE or GET a product like /products/1/ we can now update, delete or retrieve an existing item, but there is a little problem: we haven't set any permission on this class, so anyone can do it. The previous view was also more compact, why don't we use a generic view to perform these basic operations? Let's refactor ProductDetail with a RetrieveUpdateDestroyAPIView generic class. Open catalog/views.py and change the class code in this way:

class ProductDetail(generics.RetrieveUpdateDestroyAPIView):
    queryset = Product.objects.all()
    serializer_class = ProductSerializer
    permission_classes = (IsAdminOrReadOnly, )

That's it! With just three lines of code we have now implemented the same feature of the previous class, plus we have set the correct permissions.

To checkout the code at this point:

git checkout tutorial-1.12

Reviews - Relations between models

As many online catalogs already have, it would be nice if our API had an endpoint where it is possible to leave a review for a product and get a list of reviews for a specific product. To implement this feature we need to add a new model to our application. Edit catalog/models.py adding this import:

from django.contrib.auth.models import User

and this Django model:

class Review(models.Model):
    product = models.ForeignKey(Product, related_name='reviews')
    title = models.CharField(max_length=255)
    review = models.TextField()
    rating = models.IntegerField()
    created_by = models.ForeignKey(User)

after creating the model, please remember to create the related DB migration:

$ ./manage.py makemigrations catalog

When the model is ready, we have to do some changes to the serializers. First of all we need to write a new one, for our new Review model. Then we have to change our ProductSerializer so that it will return its related reviews. Each Product can have multiple Review. And each Review will be always linked to a specific Product. Edit catalog/serializers.py and change it in this way:

from .models import Product, Review
from rest_framework import serializers


class ReviewSerializer(serializers.ModelSerializer):
    created_by = serializers.ReadOnlyField(source='created_by.username')

    class Meta:
        model = Review
        fields = ('id', 'title', 'review', 'rating', 'created_by')


class ProductSerializer(serializers.ModelSerializer):
    reviews = ReviewSerializer(many=True, read_only=True)

    class Meta:
        model = Product
        fields = ('id', 'name', 'description', 'price', 'reviews')

Note: in ReviewSerializer when we serialise the user contained in created_by field, return the username instead of the id (to make it more human readable). Another important thing to notice is that the value of the related_name we have set in the Review model must match with the field name we have added in ProductSerializer fields property. In this case we have set it to reviews.

At this point we need to add a new view. Edit catalog/views.py and add the following imports:

from rest_framework.permissions import IsAuthenticatedOrReadOnly
from .models import Product, Review
from .serializers import ProductSerializer, ReviewSerializer

then add this class:

class ReviewList(generics.ListCreateAPIView):
    queryset = Review.objects.all()
    serializer_class = ReviewSerializer
    permission_classes = (IsAuthenticatedOrReadOnly, )

    def perform_create(self, serializer):
        serializer.save(
            created_by=self.request.user,
            product_id=self.kwargs['pk'])

As you can notice, I had to customise the perform_create method because the default one doesn't know anything about the fact we want to set the created_by and product_id fields. Finally we need to bind this new view to a specific url, so we need to edit catalog/urls.py and add this:

...
    url(r'^products/(?P<pk>[0-9]+)/reviews/$', views.ReviewList.as_view()),
]

At this point any authenticated user should be able to POST a review for a product and anyone should be able to get the list of reviews for each product. If you have any problem with the code and want to move to this point, please checkout this:

git checkout tutorial-1.13

Nested APIs details

To complete our API endpoints for Review, we need to add an additional feature that will let users to edit/delete their own review. Before implementing the new view, we need a little bit of refactoring and a new permission class. Edit catalog/permissions.py and add this new class:

class IsOwnerOrReadOnly(BasePermission):
    def has_object_permission(self, request, view, obj):
        if request.method in SAFE_METHODS:
            return True

        return obj.created_by == request.user

Basically this will permit changes to the review only to its author. Now we are going to add new urls and doing some refactoring at the same time. Edit catalog/urls.py and change the urls in this way:

urlpatterns = [
    url(r'^products/$', views.ProductList.as_view()),
    url(r'^products/(?P<product_id>[0-9]+)/$', views.ProductDetail.as_view()),
    url(
        r'^products/(?P<product_id>[0-9]+)/reviews/$',
        views.ReviewList.as_view()
    ),
    url(
        r'^products/(?P<product_id>[0-9]+)/reviews/(?P<review_id>[0-9]+)/$',
        views.ReviewDetail.as_view()
    ),
]

You may have noticed that I substituted pk with product_id. In the latest url I added, we need to be able to identify two primary keys: the one for the product and the one for the review. I renamed the previous ones for consistency. Now it's time to add the new view for Review details. Edit catalog/view.py and add this class:

class ReviewDetail(generics.RetrieveUpdateDestroyAPIView):
    serializer_class = ReviewSerializer
    permission_classes = (IsAuthenticatedOrReadOnly, IsOwnerOrReadOnly)
    lookup_url_kwarg = 'review_id'

    def get_queryset(self):
        review = self.kwargs['review_id']
        return Review.objects.filter(id=review)

What are we doing here? You may have noticed that we set a new property called lookup_url_kwarg. That property is being used to determine the keyword in urls.py to be used for the primary key lookup.

You will also need to do some refactoring to the other views, to adapt them to the changes we just did to the urls. I suggest you to have a look at the diffs here: https://github.com/andreagrandi/drf-tutorial/compare/tutorial-1.13...tutorial-1.14 or you can have a look at the whole file here https://github.com/andreagrandi/drf-tutorial/blob/541bf31c11fd1dbf2bcc1d31312086995e3e5b48/drftutorial/catalog/views.py

In alternative, you can fetch the whole source code at this point:

git checkout tutorial-1.14

Wrapping Up

In this third part of the tutorial you learned how to handle model details in the API and how relations between different model work. In the next part of the tutorial we will do something we should have done since the beginning: adding tests to our code and learn how to properly test the API.

Feedback Please

If you enjoyed this tutorial, please leave me some feedback! I really want to improve my work, based on the users feedback so any little advice will be appreciated, thanks!

In the first part of this tutorial we have seen how to create a basic API using Django Rest Framework. This second part will explain how to implement POST methods and add different levels of permissions and authentication. If you are starting from part 2, you may want to checkout the source code at this exact point:

git checkout tutorial-1.4

A step back

Before showing how easy it is to implement a POST method for our existing API, I want to do a step back and show you the "manual way", using just the APIView class. Edit the file catalog/views.py and change the code in this way:

from django.http import HttpResponse
from rest_framework.response import Response
from rest_framework.views import APIView
from .models import Product
from .serializers import ProductSerializer


class ProductList(APIView):
    def get(self, request, format=None):
        products = Product.objects.all()
        serializer = ProductSerializer(products, many=True)
        return Response(serializer.data)

If we try to use the API again (from the browser of from the http client), it will still work in the same way. The difference here is that we are using the very basic APIView class and we have explicitly defined the GET method for it.

Implementing a POST method with APIView

An API is not being used at its full potential if it's read only. We are going to implement a POST method for the existing view and testing it with httpie client again. First of all we need to add an import to catalog/views.py

from rest_framework import status

then we add this method to our ProductList class:

def post(self, request, format=None):
    serializer = ProductSerializer(data=request.data)
    if serializer.is_valid():
        serializer.save()
        return Response(serializer.data, status=status.HTTP_201_CREATED)
    return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)

Now let's test our POST method we just implemented:

$ http --json POST http://127.0.0.1:8000/products/ name="Salamino" description="Salamino Piccante" price="10.50"
HTTP/1.0 201 Created
Allow: GET, POST, HEAD, OPTIONS
Content-Type: application/json
Date: Thu, 29 Sep 2016 11:48:48 GMT
Server: WSGIServer/0.1 Python/2.7.10
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "description": "Salamino Piccante",
    "name": "Salamino",
    "price": "10.50"
}

It works! In case something doesn't work, try to fetch the source code at this point:

git checkout tutorial-1.7

Implementing a POST method with ListCreateAPIView

Do you remember when I mentioned at the beginning that there is an easy way to do the same thing? I wasn't cheating. Let's change again our old code in catalog/views.py but this time we will use a different base class:

from django.http import HttpResponse
from rest_framework import generics
from rest_framework.response import Response
from .models import Product
from .serializers import ProductSerializer


class ProductList(generics.ListCreateAPIView):
    queryset = Product.objects.all()
    serializer_class = ProductSerializer

let's test this again with httpie:

$ http --json POST http://127.0.0.1:8000/products/ name="Pecorino" description="Pecorino Sardo" price="7.00"
HTTP/1.0 201 Created
Allow: GET, POST, HEAD, OPTIONS
Content-Type: application/json
Date: Thu, 29 Sep 2016 15:21:20 GMT
Server: WSGIServer/0.1 Python/2.7.10
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "description": "Pecorino Sardo",
    "name": "Pecorino",
    "price": "7.00"
}

We just POSTed some data on the API! How can it work? Well, we have changed the base class from ListAPIView to ListCreateAPIView. This particular class implements a generic POST method that will accept and validate all the fields through the specified serializer.

Authentication

Now our API let us add products to the catalog, amazing! But... is it exactly what we want? In a real scenario we don't want any random user to be able to add products in our database, so we are going to protect the POST method allowing only Admin users.

Before digging into Django Rest Framework permissions, we need to setup an authentication system. For simplicity we will implement TokenAuthentication. As first step we need to edit settings.py and insert rest_framework.authtoken in the INSTALLED_APPS:

    ...
    'rest_framework',
    'rest_framework.authtoken',
    'catalog',
]

after this, we need to add TokenAuthentication as default authentication class (append this in settings.py at the end):

REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'rest_framework.authentication.TokenAuthentication',
    )
}

Finally we need to add a particular URL to the project so that clients will be able to call an endpoint passing username and password to get a token back. Edit drftutorial/urls.py and make it's like this:

from django.conf.urls import url, include
from django.contrib import admin
from rest_framework.authtoken.views import obtain_auth_token

urlpatterns = [
    url(r'^admin/', admin.site.urls),
    url(r'^', include('catalog.urls')),
    url(r'^api-token-auth/', obtain_auth_token),
]

Don't forget to re-run the migrations, because TokenAuthorization needs to change a couple of tables:

$ ./manage.py migrate
Operations to perform:
    Apply all migrations: admin, auth, authtoken, catalog, contenttypes, sessions
Running migrations:
    Applying authtoken.0001_initial... OK
    Applying authtoken.0002_auto_20160226_1747... OK

In case you had any problem changing the code up to this point, you can always fetch the related git tag:

git checkout tutorial-1.9

Testing the Authentication

Before testing the authentication, make sure you created at least the Django superuser with:

$ ./manage.py createsuperuser

now let's try to obtain the token we will need later for our API calls:

$ http --json POST http://127.0.0.1:8000/api-token-auth/ username="yourusername" password="yourpassword"
HTTP/1.0 200 OK
Allow: POST, OPTIONS
Content-Type: application/json
Date: Fri, 30 Sep 2016 08:55:07 GMT
Server: WSGIServer/0.1 Python/2.7.11
X-Frame-Options: SAMEORIGIN

{
    "token": "bc9514f0892368cfd0ea792a977aff55d53e3634"
}

We will need to pass this token in every API call we want to be authenticated. The token is being passed through the "Authentication" header parameter.

API Permissions

Authentication is something that identify the user with a particular system. Permissions instead are the level of things that are allowed or not allowed for a particular user. In our case we said we want to let Admin users to be able to POST new products and we want to let even anonymous users to GET the product list.

Django Rest Framework has some built-in classes that we can apply to our views to define the level of permissions. We could have used the IsAdminUser class, but it would not allow anonymous users to perform the GET request. Or we could have used IsAuthenticatedOrReadOnly class, but this would allow any registered user to add products (and we want to let only admins).

Or...we can define our own permission class and have exactly what we want. Create a new file catalog/permissions.py

from rest_framework.permissions import BasePermission, SAFE_METHODS


class IsAdminOrReadOnly(BasePermission):
    def has_permission(self, request, view):
        if request.method in SAFE_METHODS:
            return True
        else:
            return request.user.is_staff

Just as a side note, SAFE_METHODS are GET, HEAD and OPTIONS. These method are considered "safe" because they don't change any existing data. Open catalog/views.py again, import this at the beginning:

from .permissions import IsAdminOrReadOnly

and set this as permission_classes to ProductList:

...
serializer_class = ProductSerializer
permission_classes = (IsAdminOrReadOnly, )

Let's now try to add a new product using the token we got before (you will have to use your own token of course, mine only works on my local db):

$ http --json POST http://127.0.0.1:8000/products/ name="Lardo" description="Lardo di Colonnata" price="8.50" 'Authorization: Token bc9514f0892368cfd0ea792a977aff55d53e3634'
HTTP/1.0 201 Created
Allow: GET, POST, HEAD, OPTIONS
Content-Type: application/json
Date: Fri, 30 Sep 2016 13:04:13 GMT
Server: WSGIServer/0.1 Python/2.7.11
Vary: Accept
X-Frame-Options: SAMEORIGIN

{
    "description": "Lardo di Colonnata",
    "name": "Lardo",
    "price": "8.50"
}

It worked! We have now protected our API so that not admin people can't create any product. If you have any problem with the code, you can check it out with this tag:

git checkout tutorial-1.10

Wrapping Up

We have now implemented the POST method to add new products to our catalog. In the next episode we will see how to implement endpoints to get a single product, to update or delete products and finally we will allow registered users to send a review for a specific product.

Feedback Please

I know, this blog doesn't have any "comment" feature (I was tired of dealing with spam), but if you want to provide some feedback you can still do it by email. Just visit my About page, you will find my email there.