LCSH as linked data: beyond "dash-dash"

The SKOS version of LCSH developed by LC has made some choices in how LCSH would be presented in a linked-data format. One of these choices is that the complex headings (which is the vast majority of them) are treated as a single string:

Italy--History--1492-1559--Fiction

While this might fit appropriately as a SKOS vocabulary, in my opinion it does not work as linked data. I'm going to try to explain why, although it's quite complex. Part of that complexity is that LCSH is itself complex, primarly because there are many exceptions to any pattern that you might care to describe. (For more on this, I suggest Lois Mai Chan's Library of Congress Subject Headings, 4th edition, the chapter on geographic subject headings, pp. 67-89)

Taking the heading above, as I mentioned in my previous post, the geographic term Italy is not in LCSH even though it can indeed be used as a subject heading. Instead, Italy is defined as a name heading in the LC name authorities file. In that file, and only in the name file, alternate forms of the name are included (altLabels, in SKOS terminology):

451 __ |a Repubblica italiana (1946- )
451 __ |a Italian Republic (1946- )
451 __ |a Wlochy
451 __ |a Regno d’Italia (1861-1946)
451 __ |a It?alyah
451 __ |a Italia
451 __ |a Italie
451 __ |a Italien
451 __ |a Italii?a?
451 __ |a Kgl. Italienische Regierung
451 __ |a Ko¨nigliche Italienische Regierung

There are no altLabels in the LCSH entry for Italy--etc. And because the term Italy is buried in an undifferentiated string, there is no linked data way to say that the Italy in Italy--History--1492-1559--Fiction is the same as http://id.loc.gov/authorities/n79021783, which will presumably be the URI for the name.

It is assumed in LC authorities that the altLabels for a name term that appears in a subject heading apply to both the name used as a name and the name used as a subject heading. In the card catalog, where the name alone would appear first in the alphabetical browse of the cards, it was only necessary to make references to that "head" of the list, which would, in our case, be Italy alone. This has caused great problems in online catalogs where searching is by keyword, not a linear alphabetical search. Some systems manage to get around this by doing a string compare to the same subfields in name headings and subject headings, and then transferring the altLabel forms to the related subject headings.

$a Shakespeare, William, $d 1564-1616
$a Shakespeare, William, $d 1564-1616 $v Adaptations $v Periodicals

In this case, the $a and $d subfields represent the same authoritative entity. The rules say that they are, and must be, the same authoritative entity. If they don't match exactly then someone has done something wrong. They are both instances of a name identified as "n 78095332", and which will presumably be given the URI http://id.loc.gov/authorities/n78095332. There is no question about that.

There is also no question that when the name is used in a subject heading it has the full meaning that it is given in the name heading record, including alternate forms of the name and the many notes fields provided by the catalogers that created the authority record. That these don't appear in the LCSH file does not mean that it is not the case: it means only that the LCSH record assumes that the name record exists and provides that information, and that the information is applied to the name in the subject entry through the linear nature of the dictionary catalog.

We musn't confuse the form with the meaning. That LCSH has a rather arrested form is unfortunate, but it was never intended to be used outside of the context of the full set of authorities that gives full treatment to those things that have "proper names." (c.f. Chan, chapter 4)

If we wish for the LC authorities to be used in a linked data environment, then we have to make sure that the linking capabilities are there. Although I agree that each LCSH record has an identifier, and that identifier should be used, I don't agree that what is expressed in the LCSH record is a dumb, undifferentiated string. In this post I have addressed the relation to name headings, but there are other uses of controlled vocabularies within the subject headings that I haven't fully investigated yet.

LCSH as linked data: what is an LC Subject Heading?

The Library of Congress Subject Headings have been placed online in SKOS. You can search within the set or download the entire thing in RDF/XML or a n-triples. This is a welcome development.

I must say that I would also welcome some documentation on the decisions that were made, as viewing the actual data has left me with a number of questions. I'm going to begin my comments with a question about scope, and some confusion that is causing me as I think about how I would want to use this data.

What's an LC Subject Heading?

It appears that the LCSH file that is online represents those authority records whose LC control number begin with "sh", as in: sh 00009880. (Numbering 342,684 records.) However, if you do a Subject Authority Headings search in the LC authorities database you will retrieve any authority record that can be used as a subject. This means that you will retrieve personal names, corporate names, and geographic entities that can be used as subjects. (Note, this is probably a large portion of the name authority file.) This is a mixture of records with LCCNs that begin "n" (for name file) and those that begin "sh" (for subject heading file). I'm at a loss to explain/understand what determines whether a heading has an LCCN beginning with "sh" and would love to get an explanation.

The result is that a search in the LCSH file on the word "Italy" brings up 3,516 headings, with the word somewhere in the heading. However, the heading "Italy" alone is not included. You do have:

Italy, Central
Italy, Northern
Italy, Southern

and you have:

Italy, Northern--Civilization
Italy, Northern--Civilization--Germanic influences
etc.

But not "Italy."

A search in the name heading database on LC's online authority file yields a name heading entry for "Italy." That database (whose response is in the form of a browse list) has innumerable pages for corporate names under the initial term "Italy":

Italy.
Italy. Ambasciata (India)
Italy. Confederazione fascista degli industriali.

It also includes "Italy, Southern" with its LC control number "sh 85069035".

The upshot is that the LC Subject heading file at http://id.loc.gov is not the same as a subject heading search in the online authorities database. It also isn't always logical which file headings fall into. The "Italy. Ambasciata (India)" is in the name heading file as a corporate name, but "Palazzo Dell'Ambasciata di Spagna (Rome, Italy)" is in the subject heading file as a corporate name. There undoubtedly is a set of rules that explains all of this, but it seems to me that a separation of the subject file and the name files creates a split between headings that will not be mirrored in actual use.

This may not matter if the files are combined in the end, and the URI makes it look like all authorities will have ids that directly follow "/authorities/" in the URI. However, although they are both coded as corporate names, the "Palazzo... " record gets the "cool URI" http://id.loc.gov/authorities/sh2002000509#concept. Note the ending in "concept". I don't know what hash ending will be given to entries from the names file, but I do find it odd that corporate names ccould have two different hash endings, depending on which file they are from. To be frank, especially since the division into different files doesn't seem terribly logical, and that many items in the name file can also be used as concepts, I would prefer that the "#" indicate the type of heading (personal name, corporate name, conference, geographical name, topic) rather than the file that it comes from. That is, that the "#" would reflect the MARC tag - 100, 110, 111, 150, 151.

Walt Crawford should read the document

In his March, 2009 Cites & Insites, Walt Crawford does a roundup of comments on the Google/AAP settlement, and gets very agitated when reviewing some of my posts. I'm used to that. But agitation tends to cancel out reason, and Walt gets some things wrong that he might have understood better if he had kept a clear head.

In response to my criticism that Google is digitizing without regard to collection building, Walt says:

"I don’t know of any big academic library or public library that’s a single disciplinary collection—or, realistically, a set of well-curated collections. "

I'd like to hear from academic librarians on this one. My understanding was that an academic library is INDEED a set of well-curated collections.

Walt:

"I don’t remember public universities admitting to substantial costs in cooperating with Google."

What's the cost? Dan Greenstein estimated $1-2 per book. Cheap, but still considerable for a library scanning millions of books. The cost is primarily in staff time, shelving and reshelving books. Under this agreement, there is also the cost of meeting the security requirements that are imposed. (That's in Appendix D) These requirements, which are possibly quite reasonable, will have a greater cost than what most libraries do today for digital materials, and will be one of the primary reasons why some libraries do not contract to receive copies of the digitized items. (Note that some of the potential library partners are working hard to collaborate on the Hathi Trust, which does appear to meet the standards of the agreement; others, however, have decided that they will not attempt to store digital copies.)

In a post I argued that had libraries gone ahead and digitized their own collections (for the purposes of indexing and searching), that this probably would have been considered fair use.

Walt:

"Well…this is not a judicial finding. I find it unfortunate that Google didn’t fight the good fight, and I think it will make things much harder for another commercial entity to attempt similar digitization and use—but I don’t see that library use of “their own materials” has changed in any way."

Not of their hard copy materials, but legal minds think that this changes the landscape for digitization and the use of digitized materials, even closing some options that might have been available before.

"The proposed settlement agreement would give Google a monopoly on the largest digital library of books in the world. It and BRR, which will also be a monopoly, will have considerable freedom to set prices and terms and conditions for Book Search’s commercial services.... If asked, the authors of orphan books in major research libraries might well prefer for their books to be available under Creative Commons licenses or put in the public domain so that fellow researchers could have greater access to them. The BRR will have an institutional bias against encouraging this or considering what terms of access most authors of books in the corpus would want." Pam Samuelson

And to my statement:

"The digitization of books by Google is a massive project that will result in the privatization of a public good: the contents of libraries. While the libraries will still be there, Google will have a de facto monopoly on the online version of their contents."

Walt first prefaces it with:

"I take issue with the very first sentence, as I’ve taken issue consistently with the same claim by others with even higher profiles than Coyle (who are even less likely to ever admit they could be mistaken)."

Well, it would have been nice if he had said who they are. But thanks for letting me know that you consider me a "lower profile" person, Walt. He goes on to say:

"Nonsense. Sheer, utter nonsense. The libraries and contents will still be there. OCA will still be there. I’m sorry, but this one just drives me nuts: It’s demonization of the worst kind and an abuse of the language."

Well, I'm not sure how this abuses language, but there is general agreement that Google gets a monopoly... at least on out-of-print books, which is the vast majority of books in libraries. (Not on public domain books, which is what the OCA digitizes, but anyone can digitize public domain books.) So although the libraries and their contents will still be there, and can be used in hard copy as they are today, no one but Google can digitize the in-copyright works without incurring liability. So "monopoly on online version of their contents" is a factual statement, if you understand that public domain is public domain. (Note, this settlement agreement is extremely complex, with some real zingers hidden in its 134 pages. It's not possible to cover it all in a blog post, so anyone who is interested really needs to read the document itself, painful as that process is.)

In terms of preservation and longevity concerns, Walt asks:

"Won’t the fully-participating libraries have digital copies? I can’t think of institutions with better longevity."

To begin with, only fully participating libraries will have digital copies, and we don't yet know how many libraries will choose that option. Other libraries, even those that are only allowing Google to digitize public domain books, do not get to keep copies of the digital files. (Not only that, public domain libraries that have been cooperating with Google have to delete all of their copies of the files that they hold today, as per this agreement. See Appendix B-3.) The only party with copies of all of the files will be Google.

There are statements in the settlement about what happens if Google "fails to meet the Require Library Services Requirement" or simply decides not to continue. I refer you to page 84 of the settlement, and hope that someone can make sense out of it. The way I read it, libraries can then engage a third-party provider, who will receive the files from Google.

The key thing here is that even in the event of the failure of Google, libraries are not allowed to make uses of their own scans, such as those that are permitted to Google by this settlement. The restriction to "computational uses" and some other minor uses stands, even in that eventuality.

When I say:

"Google should be required to carry all digital Books without discrimination and without liability."

Walt replies:

"You mean “all digital books that Google’s scanned”? I suspect Google wouldn’t argue with this."

That is exactly what I mean, and Google does indeed argue with it. As a matter of fact, the settlement only obligates Google to provide access to at least 85% of the books it scans. That "access" refers to the subscription service that will be available to libraries and other institutions. The settlement says:

"Google may, at its discretion, exclude particular Books from one or more Display Uses for editorial or non-editorial reasons." p.36

That's followed by an affirmation of the "value of the principle of freedom of expression," which I must say rings a bit hollow in this context. Google has to notify the Registry if it has excluded a book, and to provide a digital copy of that book to the Registry. The Registry can then seek out a third party to provide services for excluded books. Here, however, is James Grimmelmann's concern on that front:

"The second is that no one besides the Registry might ever find out that Google has chosen to de-list a book. If the Registry doesn’t or can’t engage a replacement for Google, the book would genuinely vanish from this new Library of Alexandria. Perhaps that should happen for some books, but decisions like that shouldn’t be made in secret. When Google choses to exclude a book for editorial reasons, it should be [R13] required to inform the copyright owner and the general public, not just the Registry. "

What might Google exclude? Perhaps very little, but at the ALA panel in Denver in January, 2009, Dan Clancy of Google gave an off-the-cuff remark that, as I recall, had the word "pornography" in it. Given the recent embarassment of Amazon when it had to face the fact that many of its best sellers are rather salacious in nature, I can imagine Google also developing concern about the visibility of the texts that make us uncomfortable.

There are a lot of legitimate reasons for concern about this proposed settlement. And I don't think that anything that I have said is "nonsense."

Updates on Google/AAP Settlement

There have been some developments (and some non-developments) in the saga of the settlement between Google and the Author's Guild/Association of American Publishers.

First, the Internet Archive requested to intervene in the suit, saying that it, too, was digitizing books and therefore should be given the same status as Google. Judge Denny Chin, the judge deciding whether the settlement will be granted, denied the Archive's request with no additional information as to his reasoning.

Next, a group of authors and author representatives asked for an extension on the comment/intervention period, saying that they needed more time to study the agreement and how it affects their rights. Judge Chin granted a four-month extension, which now delays the opt-out period until September of 2009. This, of course, means four more months in which Google and the AAP are in limbo waiting to hear if the settlement will pass the court's test. Meanwhile, Google is going forward as if the settlement will be granted, re-negotiating contracts with the participating libraries and digitizing the files at the Copyright Office. (This latter I know from comments by Robert Kasunic of the Copyright Office at a meeting in Berkeley on the settlement.)

There is no question that it is very difficult to understand the implications of this suit. I myself thought about various books and articles that I have written that have been published and could not determine if I would be a member of the class of rights holders that the AAP and the Author's Guild claim to represent. Items of mine originally published on the Net have been reproduced in books and journals, and some items have been translated and republished. While a living popular author may have kept some control over his or her work, there are many millions more of us who have had some truck with the published word but in less clear circumstances. Anyone without an intellectual property lawyer on retainer will be hard-pressed to make a determination of their status.

The third development, and the most important, is that thanks to the efforts of some hard-working people (including the Internet Archive and Consumer Watchdog), the Justice Department has decided to take a look at antitrust implications of the settlement agreement. It's been widely noted that the agreement gives Google a monopoly over the vast number of books that are orphaned. It has also been noted that the settlement contains a "most favored nation" clause -- that is, that it contains wording that would require that Google always be afforded the most favorable terms that anyone else receives. This is another way of preventing competition, thus, the anti-trust question.

Numerous amicus briefs, originally intended to meet the May 5 deadline before the extension, are appearing. A good place to follow this is on James Grimmelmann's blog, Laboratorium.

E-books rise

The IDPF and AAP keep some statistics on ebook sales. Their latest press release says:

Trade eBook sales were $8,800,000 for January, a very significant 173.6% increase over January 2008. Just a reminder these are wholesale revenues reported from 13 participating Trade Publishers.

E-book sales are still a very small portion of overall book sales, but while sales of hard copies are falling, e-books are rising. The Kindle may have had some influence here.

On a slightly different Kindle note, I received an email from "Monica GoldenbergOn behalf of the National Federation of the Blind":

I wanted to let you know that tomorrow, on April 7^th in New York City, the Reading Rights Coalition, representing millions of disabled people who cannot read print, will protest the threatened removal of the text-to-speech function from e-books for the Amazon Kindle 2 which promised for the first time easy, mainstream access to over 255,000 books. Hundreds of disabled Americans (the blind and people with dyslexia, learning difficulties, spinal cord injuries, seniors losing vision, stroke survivors) will assemble to demand that the Authors Guild reverse its decision.

(Protest covered by CNET.)

Organizations serving and representing people with text disabilities have been very active in the development of e-book standards and technologies because the e-book promises to greatly increase the access that people with disabilities have to print resources. As e-books and e-book readers become mainstream, more books and cheaper devices become available to everyone, including disabled readers, who are otherwise a niche market with no profit possibilities for companies serving them.

The Author's Guild is missing the point here, restricting access rather than allowing e-books a larger audience. And that audience isn't only the legally blind -- it is all of us who are aging, who have tired eyes, who commute by car, or who just want to take a walk and read the newspaper at the same time. The ability to turn on text-to-speech, with its monotone voice and odd pronunciation, is not the main reason to purchase e-books, but it has its moments for all of us, and is essential for many.

LC discovers infinity

If you were at ALA Midwinter in Denver (January, 2009) you may have been in one of the meetings where the Library of Congress announced its intention to atone for the lcsh.info fiasco. In case you missed that, Ed Summers of LC created an online version of the Library of Congress Subject Heading authority records, re-organized as a SKOS vocabulary and available for linking on the open Web. After being available for about six months (beginning in May of 2008), Ed was asked by his employer to take down the site on December 18, 2008. This was in spite of the fact that the data had been out there long enough to have a number of users, and that the removal broke existing systems that had developed around the data.

[Note, lcsh.info has been re-born as http://lcsubjects.org/, hosted by Talis.]

The outcry in the community was strong, including a reply to Ed's lcsh.info blog post by Sir Web himself, Tim Berners-Lee. Library of Congress must have been suitably embarassed.

Thus the announcement at Midwinter that LC not only understands the value of linked open access to LCSH, but that all of the vocabularies managed by LC -- from the name authorities to the lists of document types, languages, locations, etc., -- need to be openly available in a format suitable for inclusion in Web services. LC has created a web site to host these vocabularies: id.loc.gov. On that site they say:

Initially, within 6 to 8 weeks, the Library of Congress will release its first offering: the Library of Congress Subject Headings. This will be an almost verbatim re-release of the system and content once found at the popular prototype lcsh.info service.

They also say:

We aim to make resources available on this site within 6-8 weeks. Check this site regularly for more updates as we continue to develop this service!

The page is dated 1/22/09. My calculations show that 9 weeks have passed. OK, that's only one week over their stated deadline. But nothing on the page has changed. No resources have been made available. An "almost verbatim" release of lcsh.info should not be too hard given that Ed had code written that he has made publicly available.

But even today, the promised service is 6-8 weeks away. It may stay that way for a long time. Maybe even forever.

Why does this matter? It matters because the availability of these vocabularies is essential for the library world to move forward. Some of us have been asking LC to put the vocabularies online in a machine-actionable format for a very long time. The Dublin Core community worked with LC to create a machine-actionable and URI-identified version of the MARC role terms as early as 2005. You can't find this linked from any of the MARC documentation. Some of us brought up the topic ad nauseum at MARBI meetings, but to no avail. Now LC seems to have "gotten it" conceptually but they have yet to show us that they can deliver.

I may seem to be undeservedly impatient on this score, but it's not that we have been waiting for this for 9 weeks: we've been waiting for years. And quite honestly, this is not rocket science, nor does LC have no guidance for how to manage this data. In fact, they could use the NSDL Metadata Registry, or, if they insist on hosting this themselves, the Registry's source code is available. Quite frankly, if LC does not prove to us soon that it can perform this necessary function, I feel that we are quite justified in going forward without them, registering the vocabularies where they can be used and managed by anyone who needs them, and going forward with a transformation of library data that will meet 21st century needs.

Un-uniform titles

The Open Library will soon be revealing its first attempt to bring together all of the many published books that represent the same work. It's been a fascinating exercise; sometimes very satisfying and other times terribly frustrating. If I hadn't already been convinced that we will need to change our data practices if we want to implement FRBR, this experience would have convinced me.

One of the problems that we ran into was one that Thom Hickey at OCLC had already reported in his blog post: that uniform titles (MARC 240) are both necessary for the identification of works, and a hindrance. The uniform title, which is being called the 'work title' in RDA (see Chapter 6) actually serves two (possibly three) different functions, and unfortunately this is not being fixed in RDA.

The first function of the work title is to bring together the different expressions of a work. This is mostly obvious for works that have been issued with different titles (the various Hamlets over time) and for works that have been translated (which also includes Hamlet). In this case, the work gets a 'work title' that is unifying, and this work title helps create work views in bibliographic databases.

Shakespeare, William, 1564-1616
Hamlet.
The tragedy of Hamlet Prince of Denmark, as is now acted by Her majesties Servants.

Shakespeare, William, 1564-1616
Hamlet.
The tragicall historie of Hamlet, Prince of Denmarke.

Shakespeare, William, 1564-1616
Hamlet.
William Shakespeare's Hamlet, Prince of Denmark.

The second function performed by the uniform title field is to give works a collective title. These are titles like 'Essays' or 'Works.' This is a title given to a grouping of works, not a single work. It's kind of a superset of works, and the same work title can be given to a different selection of an author's works. This uniform title does not help gather and display the FRBR work level, and in particular it isn't useful for user displays because the grouping title is so broad and vague. It probably would be useful as a genre for retrieval, but it's not great as an organization for works. In particular, you wouldn't want to present these to users as the same work:

Bacon, Francis, 1561-1626.
Essays
The essayes or counsels, ciuill and morall, of Francis Lo. Verulam, Viscount St. Alban

Bacon, Francis, 1561-1626.
Essays
The essaies of Sr Francis Bacon knight, the Kings Atturney Generall. His Religious meditations. Places of perswasion and disswasion. Seene and allowed

These are of the genre essays, and a genre data element is commonly used as a facet in systems that have that functionality. But the genre should not be confused with the work title, as it is here.

The function that may or may not be a third function relates to the additions to the uniform title, which really should be handled elsewhere in the record. Thus:

Hamlet. French
Hamlet. German
Hamlet. Italian

Languages and dates get all mixed in with the title of the work when the title is a 'heading' in the bibliographic record. Like the use of the uniform title for genre, systems today can provide this kind of organization, if it is desired, from data in the record, and can use it for a variety of purposes such as selection or grouping. There is absolutely no need to tack this data onto the work title now that records are no longer being placed in linear catalogs.

Note that I'm aware that I haven't expounded on the uses of uniform titles in music cataloging. The uniform titles in music cataloging are fascinating constructs for the arrangement of musical pieces, but they are not work titles. I haven't had any experience trying to create work views of music, however I'm sure that's a very interesting problem; one that I hope someone else will solve and share with the rest of us.

We need a work title if we are to follow the bibliographic concepts in FRBR. One of the big problems with the data we create today is that so many data elements are performing multiple functions that may be clear to humans but aren't coded in such a way to be clear for machine processing. That this same mistake is being made in RDA, which is supposed to be based on FRBR, shows that we still aren't designing our data for machine processing. In this day and age, that is pretty sad.