Troubleshooting eXist

For the past few months I’ve been experiencing some mysterious (to me at any rate) problems with eXist crashing. Generally speaking the problems seemed to be due to corrupted index files, and/or issues with temporary fragments. The problems seem to intensify the more the metadata editor is used, and I noticed when I deleted a collection the database would invariable crash, usually the following night. I’ve been limping along by surreptitiously restarting and re-indexing the database, but have had several instances where the database could not be restored in this way and had to restore from backup. This meant that the site was down and completely non-functional while I restored the files.

I think the issue is pressing enough that I’m going to have to put a lot of other things on hold to sort it out. I’ve found a few threads (1,2,3) on the mailing list that seem to address the issue, and it looks like some of it can be solved by an improvement of my xqueries, and that the issue of temporary fragments is on the minds of the developers.

I think I’ll start going over my xqueries to eleminate the creation of temporary fragments where I can. This will be a useful exercise anyway, as many of the queries were written before I had a through understanding of xquery. Hopefully this will significantly improve performance. Another option could be to move the metadata processing to the development server, and have a master/slave configuration to update the live site every night. Since the biggest problems with crashing/corrupted indexes seems to happen when metadata creation work is highest, this could be a way to cut down interruptions in service on the public side.

I like that UVM has given me the freedom to experiment and use technologies that are not being used elsewhere in the library, however with the freedom have come some headaches. I do not have colleagues to fall back on for troubleshooting. I understand why small digital projects go with systems like ContentDM, the trade off in flexibility may be well worth the time saved in other areas.

For every solution a new problem

It seems like everytime I solve a problem I create a new one.

As I’ve mentioned several times before, we are using MADS (largely downloaded from OCLC as MARC and transformed to MADS) for authority control. This is great for insuring a consistency during cataloging, particularly as we have moved to a slightly more distributed cataloging workflow. We now have 6 part-time people rather than 2 part-time people working on cataloging. Having more consistent cataloging in terms of authorities has in turn has helped with standardizing the faceted browsing of the site.

However I find that I’m now facing two new issues:
1) I now need MADS form that allows the catalogers to create local records, and make changes to existing records. It would also be useful to have an upload feature that allows someone other than me to add new records from OCLC to the database. I knew this would be a need, but for every feature I add to the admin side of the application, time is taken away from the other parts of the project that I should also be working on, including: project management, collection development, metadata workflow management, interface refinement and new feature implementation. So it is a balancing act of what needs to be done the most urgently, and since it only takes me a minute or two to add or change a record, this may fall pretty far down on my list even though the catalogers are requesting it.

2) I’m also struggling with how to take advantage the data in these records. Currently we store them in eXist, the catalogers can search the terms, and then select the appropriate term, and have it added to their record via the nifty subject suggest utility. It is great for the catalogers, but doesn’t really do much for the users of the website.

MADS records are interesting, they have related terms, variants, broader and narrower terms, etc. I’m wondering should all or at least part of these records be indexed with the items? For example, if I have a record cataloged with “Automobiles”, for the best search results, shouldn’t I also index this record with the variants of this term like “auto”, “car” and “Motorcars?” Do OPACS generally do this? I’m unsure, as I really don’t spend much quality time with our OPAC. To me adding the variants and related terms makes a certain amount of sense, but broader and narrower terms probably don’t.

Are there other ways I can take advantage of these broader and narrower terms? Perhaps I could have have a little icon next to each term on the browse collections pages indicating that variations/broader/narrower terms are available?

While thinking about the above, I’ve noticed some eccentricities in the MADS data that might make some taking full advantage of the MADS records a little tricky. For example:

<mads version=”beta”>
<authority>
<name type=”corporate” authority=”naf”>
<namePart>Burlington (Vt.)</namePart>
<namePart>Dept. of Streets.</namePart>
</name>
</authority>
<variant type=”other”>
<name type=”corporate”>
<namePart>Burlington (Vt.)</namePart>
<namePart>Streets, Dept. of</namePart>
</name>
</variant>
<variant type=”other”>
<name type=”corporate”>
<namePart>Burlington (Vt.)</namePart>
<namePart>Engineering Division.</namePart>
<namePart>Dept. of Streets</namePart>
</name>
</variant>

Nowhere in the above record is Department spelled out. I would think that would be a useful variant to have for a keyword search. Also, I find a lot of corporate names that include geographic subdivisions as part of the name:

<name type=”corporate” authority=”local”>
<namePart>Edmunds Elementary School (Burlington, Vt.)</namePart>
</name>

wouldn’t it make more sense like this?

<name type=”corporate” authority=”local”>
<namePart>Edmunds Elementary School</namePart>
</name>
<geographic>Vermont</geographic>
<geographic>Burlington</geographic>

Then you could potentially break out the geographic sub divisions for faceting. I guess that gets back to the argument of pre or post coordinated headings. All in all I find authority records baffling, but I’m still hoping to make some use of them. I’d be interested in any work that others are doing utilizing authority records to improve access.

Quick update

I have a steadily growing list of things I need to get finished, but I finally managed to get around to adding RSS (actually atom) feeds for tracking search results and also for tracking collections. This will be particularly handy for a collection like the McAllister Photographs which will be a work in progress for quite some time. (It is currently growing at a rate of about 50-100 photos a week, but I imagine this will slow down once the cataloging staff has caught up to our part time scanning tech.)

In the process of adding the feeds I also took a the opportunity to upgrade to Solr 1.2, which was a pretty painless upgrade, with some nice additional functionality. I hope to get a chance to install 1.3 on my development machine next week to explore the MoreLikeThis functionality. I’d like to use this feature on the item pages, allowing users to get some immediate related items, in addition to using the subject and geographic headings to get related items.