American Song Sheets, another new digital collection we published in October, includes approximately 1,800 broadsides and song sheets from nineteenth-century America. For this collection, we provide the song sheet images, as well as the searchable full text of the song lyrics. Will processed the full text to generate a collection-level “term cloud” based on commonly occurring words within the lyrics. This technique has proven useful for other collections, such as the Sidney Gamble Photographs of China term clouds (in two languages!) and the Americans in the Land of Lenin collection term cloud.

For the Song Sheets, Will also used full-text processing to enhance the metadata for each item with “more frequent words” and “less frequent words.” These approaches allow us to support additional browsing pathways for our users without the costs of hand-crafted metadata.

Continuing the introductions … I’m Rich Murray, and I’m one of two Metadata Librarians in the Digital Collections Program. I’m based in the Cataloging & Metadata Services Department, and I work with Noah Huffman (in the Rare Book, Manuscript, & Special Collections Library) to plan and create metadata for our digital collections.

What does that mean, exactly? Basically, Noah and I –- and the rest of the metadata team –- work to describe, organize, and allow users to discover the cool stuff in our digital collections. Metadata is “data about data,” and without it, a 5000-item digital collection is like 5000 photographs thrown into a big pile. You might be able to find what you want by going through them all one at a time, but it will probably take forever, and you may get to the end and discover that what you were looking for wasn’t in the big pile anyway.

With good metadata, though, you can find what you’re looking for much more efficiently and painlessly. We group objects into categories based on subject, format, time period, or anything else that makes sense. We apply captions to images, keywords to advertisements, plot summaries to videos, and anything else we think will help you find what you’re looking for. And if we’re doing our job right, the metadata we provide might even lead you to really, really cool stuff that you didn’t even know you were looking for.

Fortunately, Noah and I don’t have to do all this on our own. We work with a great group of folks, including the rest of the Digital Collections Implementation Team, the Metadata Advisory Group, and other staff throughout the libraries. It’s a team effort, and as our Digital Collections Program grows, more and more of us are involved in making it happen.

The other part of my job, which may sound completely unrelated at first but really isn’t, is serving as the Catalog Librarian for Spanish & Portuguese (and Catalan and Galician) Languages. Both parts of my job involve describing, arranging, and providing access to the library’s collections so you can find what you need.

Metadata isn’t something new –- it’s what librarians and archivists have been doing all along, even if we called it something else. Connecting people and ideas is what do best, and as part of the Digital Collections Program, I get to spend my days bringing some truly remarkable resources to an audience around the world. It doesn’t get much better than that.

About a year ago, an OCLC report on a survey of 18 member institutions, “RLG Programs Descriptive Metadata Practices Survey Results,” supported that basic conclusion.  When asked about the tools that they used to “create, edit and store metadata descrptions” of digital and physical resources, a sizable majority responded “customized” or “homegrown” tool.

Since my initial inquiry, we launched a new installation of our digital collections.  Yet we still lack a full-featured software platform for capturing descriptive metadata.

We did our own informal survey of peer institutions building digital collections, which further reinforced that familiar conclusion — there are lots of Excel spreadsheets, Access and FileMaker databases, etc.,
out there, but no enterprise-level solution.

We also articulated a detailed series of specifications for a metadata tool.  The library has committed to hiring two programmers each to a two-year appointment for this purpose.  The job description is here, and there are two openings for it.

When we thought through through the specifications for a metadata tool, we went on and envisioned a full-featured platform to support digital collections, including digitization, asset management and a front end.   I’m going to list in some detail the specifications from what we refer to as the Digitization and Description use cases.

Digitization

* Supports identification – an itemized list representing each scan in the collection. This list includes 1st pass descriptive metadata, structural metadata (how the components of an item are related to  one
another) and location metadata (box #, folder #, album, chapter, etc) identifiers, item type and dimensions.

* Check out systems – The ability to create an arbitrary grouping of consecutive components that can be “checked out” by a scanner operator during the scanning/QC process. These units include information about the physical location of the materials and any pertinent information needed to scan. Needs to enable multiple users at the same time.

* Reporting interface – In conjunction with the checkout system above the system needs to be able to report statistics such as total number of items in a collection, number of items left to scan and/or qc, the average amount of time it take to scan a “unit.”

* Pulls technical metadata from the image header – Must be able to extract technical metadata from image files.

* Student worker login access – Restricted read/write access through login credentials

* Generates image derivatives according to dimension specification – Generation of derivatives via batch processing with the option to change dimensions to fit changing web displays. Must use color profiles and have options to control compression quality.

* Generates checksums – Creates checksums of files that move through the system to ensure that the files have not become altered.

Description

* Supports Duke Core (multiple metadata schema) metadata creation – Duke Core, a modified version of qualified Dublin Core, is the standard metadata schema developed for digital collections at the Duke libraries by the Metadata Advisory Group.

*  Authority lists — including sharing authority lists between similar projects, setting default dropdowns for all projects/items (e.g. Type), etc.

* Set mandatory fields and cardinality constraints.

* Assign values in mass to every item in a collection – Collections often have particular metadata that needs to be applied to every item in the collection (e.g., subject terms, creator, etc.).

* Find and edit existing records easily.

* Integrates with digitization workflow.

* See digital object while editing metadata. Users should be able to see the digital object while they are creating or editing the corresponding metadata.  Does not have to be the highest-resolution image, but a working version.

* Displays record status – Allows catalogers to specify the state of a record.  System should allow catalogers to specify this status and list records in a way that provides at-a-glance overview of work remaining within a collection.

* Handle item-level metadata-only records – Some of our digital collections are metadata only. The tool must allow users to create and edit metadata records that do not have an attached digital object.

* User Interface simple and intuitive, distributed system. Could be web-based – The interface should be simple and intuitive, and should allow multiple users to work at the same time, though they should not be able to edit the same record at the same time.  A web-based tool would allow users to work on digital collections from anywhere and would not require them to use a computer with particular software installed.

* Supports UTF-8 universal character sets – Metadata for Duke’s digital collections often includes special characters (diacritics, non-Roman characters, etc.). The tool must accommodate UTF-8 character sets.

When I took this position six years ago my job title was the only “metadata” anything in the library.  Now we have two Metadata Librarians, a committee called the Metadata Advisory Group, and an internal metadata “standard” named Duke Core (derived from that other core).  What used to be our Cataloging Department is now known as Cataloging & Metadata Services.  Yes (rubs hands together, cackles) my plan is working beautifully.  Next I will change my job title to “Toll House Cookies for Everyone Analyst.”

Anyway, the first time I told a friend of mine outside of the library field about my job, she said, “Huh?  Megadata?  What’s megadata?”  This particular friend was in law school at the time, so now when people ask her, “What do you do?” she says “I’m a lawyer.”  I gave up answering that question directly; now I just say I’m a computer programmer (partially true) or a librarian (nominally untrue, though true in the sense of “a person who works in a library on library stuff”).

But at least now I have to explain my job less when I’m IN the library than I did six years ago.

Anyway, this Word Press thingie for digital collections has been sitting up on cinder blocks on the side of the house for a while, and I thought I’d take it for a spin.  Wheeee!  Before I move on to a subject other than “me me me” I’ll add that I contribute 6-to-8 hundred words of topical observation to the Chapel Hill News‘ “My View” feature every 7 weeks or so.  My most recent column attempts to make issues relating to “megadata” and library technology seem like the kind of thing you talk about in a newspaper.

Coming soon … a post about metadata.