About

The Duke DataAccessioner was built out of the need for a simple GUI interface to allow technical services staff an easy way of migrating data off disks and onto a file server for basic preservation, further appraisal, arrangement, & description. It also provides a way to integrate common metadata tools at the time of migration rather than after the fact. With a simplified interface and being written in Java it is intended to be easily adopted by smaller institutions with little or no IT staff support. Existing tools fulfill different parts of the idea but never completed it. (National Library of Australia system is another option that recently came to light and appears to be very good although it has higher infrastructure requirements.) The very first version of the tool was written in the course of a week in early 2008 and, although usable, it was more of a proof-of-concept. For nearly a year there was no active development except to tune the metadata output and fix some bugs. In January 2009 the Data Accessioner was revisited with a revised architecture. Also, the metadata tool adapters and the custom metadata manager where extracted to be used as plugins.

If you have any questions please contact the Electronic Records Archivist.

Accessioner

The accessioner is comprised of four main components: the migrator, metadata manager, graphical interface, and optional tool adapters.

The migrator recursively navigates a file tree and creates a copy of the tree in a given destination with the option of skipping specified files and directories (nothing novel here). The biggest difference is that it creates an MD5 for the file before copying it and then, after the copy creates another on the new copy and compares the two. If there is any difference it creates an error and will notify the user. Modern systems typically do error checking on transfer but this application is intended to be as exact as possible to the point of paranoia. It also sets the last modified date of the copy to that of the original. The migrator also instructs adapter plugins to do their work and sends the resulting metadata to the metadata manager.

The metadata manager is in charge of constructing the metadata structure filling it out and handling the metadata passed by the adapter plugins. Two default metadata managers are built in: default and none. The default is a simple file/folder structure inside a collection/accession root that will take whatever metadata is passed to it. Anyone can also write their own metadata manager plugin to use instead. The one created for Duke RBMSCL/UA is below. The DataAccessioner could also be used as an ingest interface to repository systems such as Fedora via a custom metadata manager.

The graphical interface is a single window. It is in reality not necessary to run the migrator which can be started (although with limited options) via the command line.

Plugins

There are two types of plugins: adapters and metadata managers. Adapters can either wrap existing functionality such as Droid and Jhove (an anti-virus plugin will probably be next) or introduce completely new (Java) code. The existing Jhove and Droid plugins run on each file (the new copy) individually and return their results. Metadata Managers simply override the existing metadata managers to modify the default schema or use a completely different one such as METS. Plugins are used via the Java Plugin Framework (JPF). To use plugins simply place the plugin in a "plugins" directory that resides next to the DataAccessioner jar. New plugins should conform to the JPF specification and require the "core" plugin which defines extension points. Future plugins will likely include a virus checker and a handle assigner. (The handle assigner could be built into the metadata manager as well.) NOTE: JPF plugins do not work with Apple's OS X (although they should). The errors are comming from the internals of JPF and I have not been able to find a fix.

Downloads

Below you can find the Data Accessioner, plugins, and a very short guide for in-house use. The source code is now hosted on Github.

Data Accessioner

Version 0.3.0 (Stable): last updated 2011-09-02

Version 0.4.0 (Development):

Plugins