Chris Lamb

Swordfish

BlogProjectsAbout

View all my projects »

TokyoCabinet-based RESTful NoSQL database.


Swordfish is a high-level persistent key-value database with a RESTful HTTP interface.

It was designed to complement a relational database by providing a means to denormalise—or partly denormalise—the results of expensive JOIN operations, particularly where standard techniques such as caching would be insufficient.

Swordfish's primary data structure is a dictionary with ordered keys and provides functionality for atomically manipulating and querying these structures, particularly set-based operations such as intersection and difference. Swordfish also supports counter types, multiple databases and an efficient means of populating a large number of trees in a broadcast (or "map") manner.

Swordfish is especially effective in web applications with "friends" or similar concepts, as the queries typically required are not only time-consuming to execute, the results cannot be shared with other users by caching. In addition, the use of JOINs prevents the implementation of sharding and other optimisations.

Clients

Clients may be trivially constructed for all languages that boast HTTP and JSON libraries.

However, Swordfish ships with an optional wrapper for projects using the Django web-development framework - a Swordfish-backed QuerySet object is provided that allows Swordfish queries to satisfy the interface of common Django utilities such as django.core.paginator.Paginator as well as the usual QuerySet properties such as laziness, splicing and local caching. This quasi-QuerySet additionally provides an interface to move from values stored in Swordfish (typically database IDs) to instances of Django models using a single database query.

Documentation

The source tarball includes a fully-working demonstration of a Twitter clone based on Django and Swordfish which acts as an educational tool for learning about and experimenting with Swordfish.

(It is not a recommendation that a such as system could have this architecture, it merely provides a shared vocabulary and an overview of Swordfish's functionality.)

Technology

Swordfish currently uses Tokyo Cabinet as the storage backend and libevent for low-level event notification.

You can browse the source tree, create or file bugs or download the code from Git:

$ git clone https://github.com/lamby/swordfish

Swordfish is released under the GNU General Public license version 3 or later (server) and the BSD license (client libraries).

View all my projects »