One step forward, one step back

eXist crashed on Wednesday. Actually crash is probably the wrong word, it seemed to be running fine but then failed to restart when I restarted Tomcat. We have backups, run early every morning, but that doesn’t help for the data that was entered during the day on Wednesday. More worrisome is that it is still unclear to me what caused the corruption in the database. I found a few discussions on the exist mailing list that seemed to be about similar problems, but without any satisfactory answers as to why the corruption occurred.

http://thread.gmane.org/gmane.text.xml.exist/5254/focus=5254

http://thread.gmane.org/gmane.text.xml.exist/7161/focus=7248

After a day and a half of trying to figure out what went wrong I caved and wrote to the list. I try to put that off for as long as possible, because while the list is very active, and generally helpful, I hate asking a question and then figuring out the answer myself later (or, I’ll be honest, getting an answer back that makes me feel stupid). I ‘ve been unable to reproduce the error after replacing the corrupt instance with the one from the backup. I have a feeling it was something I was working on during the morning on Wednesday, which means either my search xquery (which was outputting some java exceptions), or perhaps some of the xupdates I was using to add new elements to a few hundred documents at once.

Now that we are back up and running I’m returning to the question of my search xqueries, which I think need to be a little more sophisticated, the heart of which looks like this:

let $results :=
for $hits in collection('/db/mets')/mets:mets/mets:dmdSec[@ID='dmdDC']
    //descendant::dc:dc/child::* [self::* |= $_query]let $type := 
$hits/ancestor::mets:mets/@TYPElet $title := $hits/parent::*/dc:title[1]
let $id := $hits/parent::*/dc:identifier
return
<item id="{string($id)}" type="{string($type)}">
  <title>{string($title)}</title>
  {$hits}
</item>

It is problematic because it returns multiple hits for a single document. This is a pretty easy fix to make, but I also ran into a problem with this query when I had over 1000 hits, I encountered a java error (as noted above), so I will need rework this. I can limit the number of results returned, or I could use this search to only search collection level records, not item level records, most likely the first option. I have also had a request to include the author/creator field in the results which is a minor fix.

Update: My answer from the eXist list about the database corruption:

I fixed your issue. It wasn’t a “real” corruption, just removing the .lck files would have helped. As the exception shows, the lock files were damaged.

> org.exist.storage.lock.FileLock.read(FileLock.java:208)
> at
> org.exist.storage.lock.FileLock.tryLock(FileLock.java:108)
> at
> org.exist.storage.BrokerPool.canReadDataDir(BrokerPool.java:596)

Anyway, the startup process should handle this. After a database crash, the file locks might be incomplete. eXist will now check this.

So, that is good to know. Also eXist 1.0 and 1.1 final have just been released, I may take some time this week to upgrade to 1.1 final.

Advertisements

%d bloggers like this: