covid-api - a free and open source API service for COVID-19 data

Posted on Fri 10 April 2020 in Development • Tagged with covid, api, rest, data, covid-19, service, free, open, source

Introduction

In this period of COVID-19 emergency, many countries are publishing COVID related data that is being used by many existing projects and researchers.

The main problem with these data is that they are being released in CSV format on some GitHub repository. While we fully appreciate the opennes of this format, unfortunataly it can introduce an additional work to be done (downloading the data, cleaning it, importing the data into a database, keeping it updated etc...) before someone can consume and analyse the data.

covid-api

covid-api project is a free and open source API service which automatically imports the data from various sources (at the moment we support the John Hopkins CSSE data source) and makes it available as a REST API.

The service is still under development, but an initial version (with regularly updated data) is already available at https://api.covid19data.cloud.

How to use the data

To consume the API you don't need an account nor you need to authenticate in any way. You just need to request the right endpoint using the supported parameters.

Here is an example for Python language:

In [1]: import requests

In [2]: response = requests.get('https://api.covid19data.cloud/v1/jh/daily-reports?last_update_from=2020-04-01&last_update_to=2020-04-03&country=Italy')

In [3]: response.json()
Out[3]:
[{'id': 35343,
'country_region': 'Italy',
'province_state': None,
'fips': None,
'admin2': None,
'last_update': '2020-04-01T21:58:34',
'confirmed': 110574,
'deaths': 13155,
'recovered': 16847},
{'id': 37895,
'country_region': 'Italy',
'province_state': None,
'fips': None,
'admin2': None,
'last_update': '2020-04-02T23:25:14',
'confirmed': 115242,
'deaths': 13915,
'recovered': 18278}]

Further API documentation is available at https://api.covid19data.cloud/docs

Next steps

While we keep polishing the code and improving the existing data import procedure, we are planning to support additional data sources. The next one we are going to support is the Italian Protezione Civile.

If you are aware of an additional data source that you would like to see covered, please let us know (creating a new Issue on GitHub) or send us a pull request.

Contribute to the project

If you are a Python developer and would like to contribute to the project, my advice is to first have a look at the main documentation available in the README.

Then I suggest to have a look at the existing Issues and see where help is needed or in alternative you can open a new Issue or send a pull request with fixes and improvements.

I also recommend to become familiar with our Code of Conduct before sending any contribution.

Sponsors and Thanks

I want to thank Heroku for accepting to sponsor the hosting of this service.

I also want to thank all the volunteers involved in the project for their help and contributions.

Disclaimer

We are doing our best to keep the available data updated, clean (removing duplicates), and to provide a reliable service, but we are not in any way responsible for the accuracy of the data nor for the availability of the service itself. Please use it at your own risk.

Abuse notice: we are currently not requiring any registration or authentication to use this service because we would like to keep it as simple as possible. Please do not abuse the service or you will force us to require a registration (subject to approval) to continue using it.


Google is moving some EU citizens data to US

Posted on Tue 25 February 2020 in Privacy • Tagged with privacy, google, gdpr, eu, brexit, citizens, data, controller

What is going on

As you may have heard already, because of brexit, Google is moving UK citizens data from the Northern Ireland data controller to the US one (Google LLC). Leaving the EU, UK citizens are not protected anymore by GDPR, and while this may be unfair, Google is legally allowed to do it.

The problem

Even if I'm an Italian citizen and I live in Italy, a few days ago I received this email from them:

google data controller

What's wrong with it?

The point is that I'm an Italian citizen, living in Italy. I have nothing to do with UK (even if I lived there for a few years in the past, my account was created from Italy).

Why do they mention "UK leaving EU" to me, if I don't live in UK?

I tried to contact them multiple times on their @Google account on Twitter, but I got no reply at all. I tried to search online and it looks like I'm not alone, they are doing this to many other people: https://support.google.com/accounts/thread/29317992?hl=en&authuser=1

Looking for help

What should I do? Is this legally allowed?

If there was an easy way to complain with them, I would have done it already, but I've tried to search on their website (even googling it... no pun intended) but I couldn't find a single contact form to report this issue and of course they are ignoring both Twitter and that forum I linked previously.

Should I report them to the Privacy Authority? If yes, how?

Full text of the email

Here is the full text of the email I received:

We’re improving our Terms of Service and making them easier for you to understand. 
The changes will take effect on 31 March 2020, and they won’t impact the way that you use 
Google services. And, because the United Kingdom (UK) is leaving the European Union (EU), 
Google LLC will now be the service provider and the data controller responsible for your 
information and for complying with applicable privacy laws for UK consumer users.

For more details, we’ve provided a summary of the key changes and Frequently asked questions. 
And the next time that you visit Google, you’ll have the chance to review and accept the new Terms. 
At a glance, here’s what this update means for you:

•   Improved readability: While our Terms remain a legal document, we’ve done our best to make them 
    easier to understand, including by adding links to useful information and providing definitions.
•   Better communication: We’ve clearly explained when we’ll make changes to our services 
    (like adding or removing a feature) and when we’ll restrict or end a user’s access. 
    And we’ll do more to notify you when a change negatively impacts your experience on our services.
•   Adding Google Chrome, Google Chrome OS and Google Drive to the Terms: Our improved Terms now 
    cover Google Chrome, Google Chrome OS and Google Drive, which also have service-specific terms 
    and policies to help you understand what’s unique to those services.
•   Your service provider and data controller is now Google LLC: Because the UK is leaving the EU, 
    we’ve updated our Terms so that a United States-based company, Google LLC, is now your service 
    provider instead of Google Ireland Limited. Google LLC will also become the data controller 
    responsible for your information and complying with applicable privacy laws. 
    We’re making similar changes to the Terms of Service for YouTube, YouTube Paid Services and 
    Google Play. These changes to our Terms and privacy policy don’t affect your privacy settings 
    or the way that we treat your information (see the privacy policy for details). 
    As a reminder, you can always visit your Google Account to review your privacy settings and 
    manage how your data is used.

If you’re the guardian of a child under the age required to manage their own Google Account and 
you use Family Link to manage their use of Google services, please note that when you accept 
our new Terms, you do so on their behalf as well, and you may want to discuss these changes with them.

And of course, if you don’t agree to our new Terms and what we can expect from each other as you 
use our services, you can find more information about your options in our Frequently asked questions.

Thank you for using Google’s services.
Your Google team

How to deploy a static website to Github Pages using CircleCI

Posted on Sun 24 February 2019 in Web • Tagged with ci, circleci, github, static, website, deploy

Since I created my blog with a static pages generator, I've been using TravisCI to automate the pages build and deployment. My desire to learn something new (we are using CircleCI at work, but I never configured it from scratch) and the recent news about TravisCI acquisition and employees layoff, led me to think about moving to a different service.

Github Pages

Every account on Github can use a special repository to publish static pages. In my case, since I have github.com/andreagrandi, my special repository is named github.com/andreagrandi.github.io. Once I publish my pages there, they will be accessible from https://andreagrandi.github.io.

You will need to use the master branch of the special repository directly and not the gh-pages branch which is available to each repository.

CircleCI

CircleCI is a very flexible and powerful continuous integration tool, which is also free for open source projects. As long as your static website is located on a public repository on Github, you won't have to pay anything to use it. In my case, the surce code of this website is available at https://github.com/andreagrandi/andreagrandi.it

Configuration

You can find the complete configuration at this address. The only value you won't find is GH_TOKEN. You need to generate this token on Github, at this address: https://github.com/settings/tokens. Give it a nice description like "CircleCI deployment token", select repo scope and finally click Generate token button. This token will be used to git push... your pages once they are built. Please remember to keep this token secret and not to publish it anywhere.

In my configuration you may notice that I'm using Pelican static websites generator, but apart from a few changes, the structure of the configuration should be very similar even if you use Jekill, Hugo etc... it doesn't really matter how you generate the pages, the deployment phase will be the same.

Deployment script

You will notice that there is a complete bash script embedded in the CircleCI configuration. This script configures git, fetches the existing andreagrandi.github.io repository, and sync the built pages with the existing ones (this avoid creating a commit which contains all the pages so it will contain just the added content). Once the commit is made, the script will finally push the changes to the repository.

Please note: regardless of CircleCI settings, the deployment will only happens if we are pushing (or merging a pull request) to master (if [ "${CIRCLE_BRANCH}" = "master" ]; then) and it will actually commit and push pages only if there is something new to commit (if git commit -m "CircleCI build $CIRCLE_BUILD_NUM pushed to Github Pages" ; then). For example if I'm just updating something in the CircleCI configuration, which doesn't change anything in the content, the pages won't be deployed again.

Conclusion

My first impression of CircleCI is that is faster than TravisCI and this means that I can publish my content more quickly. The possibility of using Docker containers as base image is really powerful and in more complex scenarios we can reproduce the building environment locally on our machine. If you have any advices about how to improve my build script, feel free to leave a comment here.


Skipping tests depending on the Python version

Posted on Thu 21 February 2019 in Python • Tagged with python, test, programming, software, development, testing

Sometimes we want to run certain tests only on a specific version of Python.

Suppose you are migrating a large project from Python 2 to Python 3 and you know in advance that certain tests won't run under Python 3.

Chances are that during the migration you are already using the six library. The six libraries have two boolean properties which are initialised to True depending on the Python version which is being used: PY2 when running under Python 2 and PY3 when running under Python 3.

This library, combined with the skipIf method of unittest library can be used to easily skip tests when using Python 3:

import six
import unittest


class MyTestCase(unittest.TestCase):


    @unittest.skipIf(six.PY3, "not compatible with Python 3")
    def test_example(self):
        # This test won't run under Python 3
        pass

Credits

Thanks to my colleague Nicola for giving me the inspiration to write this post.