| 1. Personal Introduction
How North American Women's Letters and Diaries came to be
first conceived the idea for North American Women's Letters and
Diaries early in 2000, while researching the microfilm collections
at the Library of Congress. As I pored through these grainy and
difficult-to-read manuscripts of early diaries, it occurred to me that
this kind of material - more than most - would benefit from being in
electronic form.
One of the problems with the microfilm was that the colored inks of
the original documents had faded and leached through the paper. The
originals would have been hard enough to read, and when photographed in
black and white many were almost unintelligible. Even with these
difficulties, I found items of great interest. One passage - in a clean,
cursive script that had obviously been dictated - showed me what a
schoolgirl had learned in 1793. Presumably her teacher had wandered up
and down the classroom dictating the geography lesson. It struck me that
this snapshot was the closest one could get today to experiencing a
class of that time.
When I moved on to the published diaries, I found more easily legible
and consistently interesting material. A search of bibliographies led me
to Joyce Goodfriend's wonderful, annotated bibliography, which in turn
pointed me to many items of interest. Mary Almy's account of sighting a
fleet off Newport, Rhode Island, in July 1778 is one notable example.
Her diary gives an hour-by-hour account of the reactions of the town as
a fleet anchors off shore. At first the townspeople think that it's the
British fleet and are happy. They then discover it's the French and
become concerned. Finally, Almy describes how the shopkeepers lock their
doors, concerned about a foraging expedition by the crews of the eleven
warships. For me, reading these passages was like watching the CNN crews
reporting that the Marines were coming ashore during the Gulf War. The
immediacy and emotion were there as if the events were happening today.
Such is the value of diaries and letters. Unlike memoirs, they
present the raw moment without the distortions of hindsight. With
letters and diaries, there is no time lag between the event and the
writing, time in which to correct attitudes or facts that may have
become unpopular or erroneous.
The more research I did, the more aware I became of the deficiencies
of print in handling this kind of material. Firstly, there is the
problem of organization. The editor of a print collection of diaries and
letters has to choose a single way to arrange the collection - typically
by theme or by author. But for readers who are interested in topics that
cut across authors and themes, that leaves no way of accessing the
relevant material. Then there is the problem of size. Sometimes an
editor has to discard materials because of the need to conserve printing
costs, or because he thinks that the average reader will find a passage
too dull.
If done right, the electronic medium has no such barriers. The size
of a database can be almost unlimited, and the materials can be
organized with multiple threads, so that the reader can quickly go to
just the materials that interest her. Someone focusing on a particular
month of history does not need to pore over numerous volumes, each
written by a different author. Instead she can see the month organized
as it actually happened - materials from many sources, arranged
chronologically. She can view multiple perspectives on the same event
with ease. I saw that scholars frequently and painstakingly would
combine sources this way in paper form, and I knew that in electronic
form they could do it more thoroughly and in seconds.
As I thought about all of this, I realized that it would be possible
to create a historical document of unprecedented utility. In fact, with
a carefully constructed thesaurus of controlled terms, we could take the
experiences described in these primary materials and create a series of
virtual documents, each representing the collective consciousness of
many individuals. Instead of reading the view of a single individual
about the death of her child, for example, we could examine the views of
hundreds of individuals over time.
Thinking about the next steps, I realized that print is a
surprisingly tolerant medium, microfilm even more so. Errors in
cataloging or in terminology are at worst an annoyance in these media. A
reader who scrolls through a book will sooner or later find what he's
looking for. In contrast, the electronic medium is one of the least
tolerant media invented - a single typo can make it impossible to find
something. Poorly chosen subject headings can lead a user to miss
critical documents. The absence of a linear path through the data can
make navigation impossible.
Turning the idea into a design
In July 2000, I was joined by Eileen Lawrence and Pat Lawry.
Pat Lawry came to Alexander Street Press with many years of
professional indexing experience. She quickly understood the concept and
began developing fields and controlled vocabularies. I think both of us
were surprised when we realized just how powerful these fields would be.
One benchmark was to make it possible to get all letters, sent by women
under the age of 20, who mentioned the world 'gold' in 1849. Such
indexing would enable wholly new discoveries to be made.
Eileen Lawrence immediately began presenting the idea to librarians
and academics at leading universities. Over the next six months, she
made more than fifty site visits and countless phone calls. In airports,
on planes and in hotel rooms, she would write up what she'd learned,
sending a steady stream of email full of comments and suggestions. Back
at the office we would meet to reshape the contents and specifications
of the database, based on all the valuable customer feedback we were
receiving. For example, University of Chicago encouraged us to retain
page numbers so that scholars could cite materials easily; Emory
suggested we drop our socio-economic status field because it was too
subjective; and UC Irvine suggested we include a field for recipient
gender to facilitate analysis of how gender affects the language of
letters - just to list a few of the many ideas libraries offered. We
kept visiting, phoning, listening, shaping and reshaping the product. By
September 2000, we'd settled on some 50 additional fields that
collectively would transform the material and make it of use to a wide
range of scholars and students. Little did we know how many more fields
were yet to come.
Along the way, we found more bibliographies and additional sources.
Our combined bibliography was now twice the size of Goodfriend's and
included many rare materials, including extracts from journals, some
manuscript materials, and books.
With continuing research and feedback, by December we had more than
80 different fields in the database design. The product was sufficiently
well developed that many leading institutions had already purchased it.
Editorial Policies
Our editorial policy was now established. For our initial selection
of materials we used Goodfriend, supported by a number of other
bibliographies. Ms. Goodfriend was kind enough to license us her
annotations, so providing an additional level of context. The University
of Wisconsin suggested that we use the Handbook of Women Writers
to ensure that we weren't missing any important individuals. So we used
this and other sources to identify omissions, then we searched for those
sources.
Our primary goal was to identify items of special interest for
Women's Studies, Social Studies, and History. We searched for materials
that illustrated the role and status of women, personal attitudes and
reactions to specific historical or personal events or individuals, and
descriptions that conveyed the life and times of the past.
Because we wanted a database of women's thoughts at the time they
thought them, we decided to exclude memoirs, focusing instead on
contemporaneous accounts only, and omitting correspondence from men.
Building the interface and the engine
A canned prototype of the interface had been in place since August
2000. From the outset, our goal was to enable users quickly to find what
they were looking for, with the fewest possible clicks. At the same
time, we aimed not to sacrifice any of the power made possible by the
indexing. Blending these two goals - power and simplicity - was the
challenge.
As I looked at search engines, I noticed that many of them had
sacrificed searching power in an effort for simplicity. This trend had
become extreme. Search engines like Lycos and AltaVista were devoting
less than 3% of their opening screens' space to searching. Even
library-oriented services had reduced the number of questions that their
databases could answer. We decided that one of our design principles
would be to enable patrons to take advantage of all the search power. We
would provide a simple search screen for novices, but also unprecedented
search power for advanced users.
Another design issue related to passive versus active searching. A
book's table of contents allows for passive searching - it doesn't
require the user to do or know anything, but rather lets the users
simply browse and see everything in the book. And because a table of
contents is sequenced in a particular order, it lets the user know
what's missing as well as what's present. Recognizing the ease that
table-of-contents browsing provides, we decided to create the same for
the database. The only difference was this: thanks to our indexing and
our controlled vocabularies, we could have numerous tables instead of
the customary one, giving the user multiple entry choices. So today,
North American Women's Letters and Diaries can be viewed in its entirety by author, by source, by year, by
personal event, and by historical event. Within two clicks and without
keying anything, a user can see whether the database contains what he's
looking for.
Besides checking the table of contents of a book, a person will also
search for a specific terms in the index at the back - active searching.
But unlike a book, where the user looks up one index item at a time, a
well-constructed electronic database allows searching by multiple and
combined fields. With our 80 index fields, we knew that the power of the
searching would be tremendous if we could find the right search engine.
In the fall of 2000, after an in-depth search, we decided on the
PhiloLogic search engine at the University of Chicago. I had worked with
Catherine Mardikes, Mark Olsen and Robert Morrissey before, and I knew
that PhiloLogic had the ability to integrate full-text searching with
fielded searching. Moreover, it had the potential to analyze texts using
word frequencies. This feature allows users to see when and where words
appear and how frequently. Now North American Women's Letters and Diaries would be of use beyond
Women's Studies and History - it would also be a powerful English and
Linguistics tool.
By now the project was in full swing. Laura Gosling, Assistant
Editor, had built an excellent thesaurus, indexing records were being
generated, and we were working hard to build the live prototype.
A live prototype is released
A live database with 1,000 pages of material was finished and loaded
a few days before the January 2001 American Library Association
Midwinter Conference. I think I speak for all of us when I say that we
were nervous. I had seen only one library database before that had
nearly as many fields as we'd put into our file. In theory the search
power would allow us to ask questions that no one had ever been able to
ask before. Now was the time to see if it actually worked!
At the conference, we presented to more than 100 individuals in
meetings, as well as countless booth visitors. The response was
wonderful. With only 1% of the data loaded, North American Women's Letters and Diaries was already fun
to use and producing fascinating search results. One of the first
searches yielded a letter from a woman in May 1849 to her husband, who
had followed his dreams to California. "Do not imagine that I want to
come, for I do not, and think that you will be a real goose to stay, for
all the gold in the mines; but if you do stay, I am coming, or else I
will get a divorce. You know that they are trying to make a law here,
that if a man and his wife are separated two years, they can get a
divorce; so if you wish to get rid of me, you know what to do."
During the next months we will be adding new material at an average
rate of 5,000 pages per month, including new, previously unpublished
manuscript material, as well as licensed in-copyright materials. Working
closely with you, our customers, our partners, and other users of the
database, we'll continue to listen to your comments and make
improvements. We welcome your input.
I'd like to finish by expressing my gratitude to all of you who have
helped make this happen. In particular I'd like to thank our charter
customers, who had faith in our vision, and whose suggestions helped
make a good idea into a great one. I'd also like to acknowledge
Catherine Mardikes and Mark Olsen who worked so hard on the project. I
hope you'll get as much fun and learning out of the database as I expect
to.
Stephen Rhind-Tutt, President
Alexander Street Press
May 2001 |
