Tag Archives: python

I use nosetests when running my python tests. It is a really neat little tool that will automatically discover and run all tests in the project with one single command line command - nosetests.

But it has more neatness then this. With a simple option it will also print test coverage for the project: --with-coverage, and if coverage should be calculated base on specific packages then use --cover-package which takes a list of packages to calculate coverage on. This is useful when there are library packages for a virtual env or similar within the same directory as you run nose.

I don't like to leave lose ends so to ensure that there are no lingering compiled *.pyc files sitting around I run the following script before commit:

Where <source-dir> should be replaced with directories where source files are. This is a space delimited list. And <package> is replaced with packages which should have coverage statistics calculated.

DRY (as in Don't Repeat Yourself) is one of the corner stones of a clean and agile code. This is especially so with the production code but also with test code where finding good abstractions will greatly increase speed and lower maintenance.

When working with python I have found a very nice tool called Clone Digger that helps finding repetitions in code. It is easy to set up and work with and the report is clear and easy to read.

Install it with pip:

pip install clonedigger

When installed it is automatically added to the tool chain. To run it use the following command:

clonedigger package

Package is the directory where the python code is to be found. Clone digger takes a list of packages separated with space. --help will show help on options and configuration.

Clone digger will work it's way through all files in each package and compare them, both within each package and across package listed in the same invocation. If there is code, such as auto generated code, that is not modifiable, it needs to be removed from the packages added in order for the tool not to analyse them.

It will generate a report looking like the one below to output.html (this is configurable).

Clone Digger output

Working with tools like this makes it easy to spot where new abstractions should be added to remove code duplication. Both in test and production code. Where ever repetition is found which does not fill an important documenting function (which is sometimes the case in tests) it is easy to spot where and how a method should be added to pull the functionality together. It will also help in naming such methods. There are of cause some false positives. Especially in the test code where method bodies are often descriptive rather then dry. But even here it helps to find common assertions which can be pulled up and made into new descriptive methods. This becomes especially valuable when working with given-when-then in tests.

I have recently been looking how to create a clean Domain Driven Design architecture using SQLAlchemy and specifically how to map domain classes to tables. This post will discuss the two techniques available and why to choose one over the other.

Lets start with looking at the two strategies available. SQLAlchemy offers a Data Mapper out of the box. It is also possible to use an Active Record approach by using the Declarative Base ORM extension.

The Active Record approach is a very attractive way of working. It adds metadata to the domain objects that tells the engine how to save them to the database. It's quick and easy to use and avoids extra classes or configurations that handles the database mapping. It's a perfect fit for most web applications. The main drawback is that storage logic is tightly coupled with the domain objects. This can make the domain difficult to work with over time. It also means that any changes required in storage logic will force changes into the domain breaking the Single Responsibility Principle. Active Record will also prove very costly if the project finds that a change of storage technology would be beneficial. A not to unlikely event with such a variation in industry strength storage technologies available.

Data Mappers provides high decoupling by keeping all meta data for mapping objects to tables separate from the domain. The mapping can be placed in its own module which makes it easy to divide the domain into coherent modules with no need to cater for storage concerns. It also makes it easy to model more advanced relational concepts in a large database. Since it allows for complete separation of concerns for domain and data access it is also easy to change or swap out the data storage technology completely without any needs to change the domain. It does however increase the complexity of the application since it adds an extra module/package.

If the project in question is going to use a clean implementation of Domain Driven Design only the Data Mapper works. It is also the only possible choice if it is likely that the project will be changing storage technology.

If, however, the project is a traditional web application with a domain model that is simple or close to non-existing then Active Record is a good choice. It will provide more speed of development. Making a clear distinction between domain logic tests and data access tests will make it easier to later extract data access out into Data Mappers if needed.

Most of us have at least heard about the DRY principle. But how is it applied in a pragmatic way in software development?

I will take you though my thoughts as I work with an application to illustrate how, when and why I apply it as well as where it may be better not to.

The code is in python. It is not a fully working application since it isn't open source.

The example application is a simple stock management application. It has a warehouses. The warehouse has sections and each section contains SKUs (Stock Keeping Units). It also has a catalogue with categories and products. Each product is bound to one or more SKUs.

I am using SQLAlchemy as ORM and this also maintains the database schema using declarative_base. In the tests I am using fixture to create data fixtures that can be injected to the database when executing the tests. Since the application is so simple SQLite is a sufficient database engine.

The first requirement is to create a function that will find a warehouse using a warehouse ID. The first step is to start the database and inject data into it. This is done in the setUp() method in the test like so:

The WarehouseFixture and SectionFixture are fixture.DataSet classes, Warehouse and Section are domain classes.

This makes it possible to query data through the DOA and then assert the results using the fixtures.

After completing the other tiers needed in the application to expose this as a JSon front end I need to create some functional tests. They will need to inject the data into the database in the same way that the DAO test did. Then the functional tests will validate the JSon response from the server using the fixtures.

I could, and I know some would, C&P the function calls above into a new setUp() method in the functional test. But doing so will disconnect the fixtures and DAO tests in the functional tests with the tests used in the DAO. Since the DAO tests and fixtures will evolve with the data model and database we don't want the functional tests lose that connection. Instead I pull up some functions from the DAO test case into stand alone functions. This can then be used from the functional tests as well as the DAO tests.

We now have generic functions that can be called in different, unrelated, tests. And the code is DRY.

Next it is time to add some new DAO tests. This DAO will be in a new module which handles the catalogue. Since the previous refactoring did not require decoupling from the warehouse, both DAO and functional tests test the warehouse, it took place within the warehouse dataaccess test module. Since this will create a new decoupled module the test modules should not be coupled. To enable this we create a utility module called data_fixtures with a new class called DataFixtureTestCase like so:

Since python supports multiple inheritance it is possible to subclass this class and AsyncHTTPTestCase when creating functional tests. In the test cases setUp() function setupDatabase() is called to set up the database and teardownDatabase() is called in tearDown() to tear it down. The setupDatabase() also ensures that the required properties have been set before setting up the database.

To keep the connection from functional tests to dataaccess the functional test module imports the engine, env and fixtures properties from dataaccess.

The above refactorings helps keep the code DRY whilst easy to understand. It also creates a place where common assertions bound to data fixtures can live.

It should be noted that when working with test DRY is not always your friend. If the tests loses any of it's expressiveness as "executing documentation" (tests is the first stop for understanding production code) expressiveness wins any day. This is actually reversed from production code which must be kept DRY at all times.

My last entry walked through the Android application for the C2DM PoC. This one will look closer at the server side.

The web server is written in Python and sits on top of the tornado framework. To get started you will first need to check it out from github.

Then there are two things that needs changing for the serve to run. First of, tornado is included as a git sub module. In order to get the tornado server you need to run the below commands, when in the project folder, in the terminal:

This will pull down tornado from their github repository.

Secondly you will need to change the following two lines (lines 36 and 37) of code in c2dm/c2dm_service_facade.py:

Replace the Google account with the account you have registered to use with C2DM and the password with the password for said account.

You should now be ready to go. To launch the server on port 8888 execute the following terminal command:

There isn't much more to it. As I have said before not all requirements for C2DM is met by this application but it will work as a prof of concept.