Pablo Iranzo Gómez's blog

nov 05, 2016

Unit testing for stampy

Since my prior post on Contributing to OpenStack, I liked the idea of using some automated tests to validate functionality and specifically, the corner cases that could arise when playing with the code.

Most of the errors fixed so far on stampy, were related with some pieces of the code not properly handling UTF or some information returned, etc and still it has improved, the idea of ensuring that prior errors were not put back into the code when some other changes were performed, started to arise to be a priority.

For implementing them, I made use of nose, which can be executed with nosetests and are available on Fedora as 'python-nose' and to provide further automation, I've also relied on tox also inspired n what OpenStack does.

Let's start with tox: once installed, a new configuration file is created for it, defining the different environments and configuration in a similar way to:

minversion = 2.0
envlist = py27,pep8
skipsdist = True

passenv = CI TRAVIS TRAVIS_*
deps = -r{toxinidir}/requirements.txt
commands =
    /usr/bin/find . -type f -name "*.pyc" -delete
    nosetests \
commands = flake8

commands = {posargs}

commands =
  coverage report

show-source = True

This file, defines two environments, one for validating pep8 for the python formatting and another one for validating python 2.7.

The environment definition for the tests, also performs some commands like executed the forementioned nosetests to run the defined unit tests.

Above tox.ini also mentions requirements.txt and test-requirements.txt, which define the python packages required to validate the program, that will be automatically installed by tox on a virtualenv, so the alternate versions being used, doesn't interfere with the system-wide ones we're using.

About the tests themselves, as nosetests does automatic discovery of tests to perform, I've created a new folder named tests/ and placed there some files in alphabetically order:

ls -l tests
total 28
-rw-r--r--. 1 iranzo iranzo  709 nov  5 16:58
-rw-r--r--. 1 iranzo iranzo  739 nov  3 09:56
-rw-r--r--. 1 iranzo iranzo  456 nov  3 23:53
-rw-r--r--. 1 iranzo iranzo  581 nov  3 09:56
-rw-r--r--. 1 iranzo iranzo 3544 nov  5 18:19
-rw-r--r--. 1 iranzo iranzo  477 nov  3 23:15
-rw-r--r--. 1 iranzo iranzo  230 nov  3 09:56

First one test_00-setup takes the required commands to define the enviroment, as on each validation run of tox, a new environment should be available not to mask errors that could be overlooked.

#!/usr/bin/env python
# encoding: utf-8

from unittest import TestCase

from stampy.stampy import config, setconfig, createdb, dbsql

# Precreate DB for other operations to work

# Define configuration for tests
setconfig('token', '279488369:AAFqGVesZ-81n9sFafLQxUUCVO8_8L3JNEU')
setconfig('owner', 'iranzo')
setconfig('url', '')
setconfig('verbosity', 'DEBUG')

# Empty karma database in case it contained some leftover
dbsql('DELETE from karma')
dbsql('DELETE from quote')

class TestStampy(TestCase):
    def test_owner(self):
        self.assertEqual(config('owner'), 'iranzo')

This file creates the database if none is existing and defines some sample values, like DEBUG level, url for contacting telegram API servers, or even a token that can be used to test the functionality for sending messages.

Also, if the database is already existing, empties the karma table, quotes (and sets sequence to 0 to simulate TRUNCATE which is not available on sqlite)

An unittest is specified under the class inherited from TestCase imported from unittest, there for each one of the tests we want to performed, a new 'definition' is created and after it an assert is used, for example assertEqual validates that the function call returns the value provided as secondary argument, failing otherwise.

From that point, the tests are performed again in alphabetically order, so be careful in the naming of each tests or define a sequence number to use a top-to-bottom approach that will be probably easier to understand.

For example, for karma changes we've:

#!/usr/bin/env python
# encoding: utf-8

from unittest import TestCase

from stampy.stampy import getkarma, updatekarma, putkarma

class TestStampy(TestCase):
    def test_putkarma(self):
        putkarma('patata', 0)
        self.assertEqual(getkarma('patata'), 0)

    def test_getkarma(self):
        self.assertEqual(getkarma('patata'), 0)

    def test_updatekarmaplus(self):
        updatekarma('patata', 2)
        self.assertEqual(getkarma('patata'), 2)

    def test_updatekarmarem(self):
        updatekarma('patata', -1)
        self.assertEqual(getkarma('patata'), 1)

Which starts by putting a known karma on a word, validating, verifying the query, update the value by a positive number and later, decrease it with a negative one.

For the aliases, we use a similar aproach, as we also play with the karma changes when an alias is defined:

#!/usr/bin/env python
# encoding: utf-8

from unittest import TestCase

from stampy.stampy import getkarma, putkarma, createalias, getalias, deletealias

class TestStampy(TestCase):

    def test_createalias(self):
        createalias('patata', 'creilla')
        self.assertEqual(getalias('patata'), 'creilla')

    def test_getalias(self):
        self.assertEqual(getalias('patata'), 'creilla')

    def test_increasealiaskarma(self):
        updatekarma('patata', 1)
        self.assertEqual(getkarma('patata'), 1)

        # Alias doesn't get increased as the 'aliases' modifications are in
        # process, not in the individual functions
        self.assertEqual(getkarma('creilla'), 0)

    def test_removealias(self):
        self.assertEqual(getkarma('creilla'), 0)

    def test_removekarma(self):
        putkarma('patata', 0)
        self.assertEqual(getkarma('patata'), 0)

Where an alias is created, verified, karma in creased on the word with an alias, and then the aliased value.

As noted in the above example, the individual function for the karma doesn't take into consideration the aliases so this must be handled by processing a message set via process(messages) which has been also modified as well as other functions to allow the implementation of individual tests for them.

This will for sure end up with some more code rewriting so the functions can be fully tested individually and as a whole, to ensure that the bot behaves as intended... and many more tests to come to the code.

As an end, an example of the execution of tox and the results raised:

py27 installed: coverage==4.2,nose==1.3.7,prettytable==0.7.2
py27 runtests: PYTHONHASHSEED='604985980'
py27 runtests: commands[0] | /usr/bin/find . -type f -name *.pyc -delete
py27 runtests: commands[1] | nosetests
Ran 18 tests in 14.996s

pep8 installed: coverage==4.2,nose==1.3.7,prettytable==0.7.2
pep8 runtests: PYTHONHASHSEED='604985980'
pep8 runtests: commands[0] | flake8
WARNING:test command found but not installed in testenv
  cmd: /usr/bin/flake8
  env: /home/iranzo/DEVEL/private/stampython/.tox/pep8
Maybe you forgot to specify a dependency? See also the whitelist_externals envconfig setting.
__________________________________________________________________________ summary ___________________________________________________________________________
  py27: commands succeeded
  pep8: commands succeeded
  congratulations :)

If you're using a CI system, like 'Travis', which is also available to repos, a .travis.yml can be added to the repo to ensure those tests are performed automatically on each code push:

language: python
    - 2.7

    email: false

    - pip install pep8
    - pip install misspellings
    - pip install nose

    # Run pep8 on all .py files in all subfolders
    # (I ignore "E402: module level import not at top of file"
    # because of use case sys.path.append('..'); import <module>)
    - find . -name \*.py -exec pep8 --ignore=E402,E501 {} +
    - find . -name '*.py' | misspellings -f -
    - nosetests