Archive for the ‘eXist’ Category

Session Solutions

March 12, 2008

So I came up with two possible solutions to my sessions problem, I’m sure there are others as well.

1) I could store all the xqueries that build my metadata admin interface in eXist with restricted read permissions so that uses would be presented with a login from the server when they arrive at the page, this would initiate the REST session, which would presumably perpetuate as long as the window was active (and thus hopefully by-pass the random logout issue.)

I haven’t actually tried this one, I just assume it would work. I don’t particularly like the solution because I don’t store most of my xqueries in eXist. However, I haven’t ruled it out.

2) Solution 2 is the one I worked on all day yesterday. First I’ve gone back to submitting the data via a POST which sends the data to an xquery, this query then uses the xmldb:store() function to save the data to eXist. This xquery also checks to see if there is a current session using session:exists(), if not it will send back “Not logged in!” if there is a current session the xquery will attempt to save the record to the database, if successful the xmldb:store() function returns the path to the resource, if it fails it returns an empty string.

My XForm now uses an event observer to check if the submit action was successful, if not it will warn the user that they are either not logged in, or have insufficient privileges to edit the resource and toggles them to a login form located at the top of their current page. Users can then login, a separate event is fired that changes or creates a new session and the user can save the data in the form, all without refreshing or leaving the form.

After much tweaking this seems to work. The nice thing is that having the login on the form saves the user from losing the data that they have been working on.

Here are what the additions the form look like:

Two new instances, the first to check what sort of error the xquery sends back if it is not successful:

<xforms:instance id="submitError">
  <dummy xmlns="">

The second to hold the login information:

<xforms:instance id="login">
  <dummy xmlns="">

Additions to the submission action:

<xforms:submission id="submit" method="post" ref="instance('metadata')"
 replace="text" instance="submitError" action="xqueryGoesHere.xql">
   <xforms:action ev:event="xforms-submit-done" ev:observer="submit">
     <xforms:message level="modal">Item Saved!</xforms:message>
   <xforms:action ev:event="xforms-submit-error" ev:observer="submit">
     <xforms:message level="modal" 
       ref="instance('submitError')/child::*[. = '']">You do not have permission 
       to edit this resource! Please log in.</xforms:message>
     <xforms:message level="modal"  
       ref="instance('submitError')/child::*[. = 'Not logged in!']">You are no longer 
       logged in! Please log in.</xforms:message>
     <xforms:toggle case="case-login"/>

I’ve then added a xforms:switch to the top of the MODS form that toggles between a login form and summary information about the resource. As you can see in the submission, if there is an error, the top part of the form will toggle to display a login form for the users, case-login. Once the user logs in with the new name and password, they click login which fires this submission action:

<xforms:submission id="do-login" method="get" replace="instance" 
 instance="login" separator="&" ref="instance('login')" action="login.xql">
  <xforms:action ev:event="xforms-submit-done" ev:observer="do-login">
    <xforms:message level="modal">You are now logged in. Please re-save your data.
    <xforms:toggle case="ready"/>
  <xforms:action ev:event="xforms-submit-error" ev:observer="do-login">
    <xforms:message level="modal">Incorrect login information</xforms:message>
    <xforms:toggle case="case-login"/>

The login.xql looks very similar to this example from the eXist website. If the login is successful the form toggles from the login info to the MODS form info. The user is then prompted to re-save their data as the new user.

I’m still a little nervous putting the catalogers back to work with these forms because I haven’t been able to replicate some of the problems they had reported, but  I’m hoping even if they do get logged out this new form will allow them to log back in without losing any data. I’ve also finally made it possible to page from one record to the next in the queue, maybe that will buy me a little extra goodwill.


Struggling with sessions

March 10, 2008

Right before I left for Code4lib 2008 we had a serious db crash, this was an eXist issue, which I’m hoping was partially solved by upgrading to 1.2 as well as some re-writing of xqueries to eliminate most of our temporary fragments creation. We haven’t had any problems with crashing since the upgrade, but a by product of the crash was some odd behavior that was reported to me by one of the catalogers. She was getting logged out in the middle of editing a record. Because of the way my forms had been set up she wasn’t getting any reliable messages about the status of the record (if it had been successfully saved or not), so she assumed a record had been saved, but it wasn’t, several records were lost, and worse, cataloger good will is now a bit more shaky than it was.

I had been submitting the data from our XForms to an xquery via post, the xquery then did some post processing and submitted the data to eXist using xupdate. Thinking it would be more straight forward I switched to using the RESTservlet, which allowed me to do a simple PUT from my XForm.  However,  I can’t seem to persist my session through the PUT request. My metadata administrative interface is a series of xqueries, the user must log in on the fist page using the eXist xmldb:login function, they then can browse through records in process, pull up completed records to edit and create new collections. However now when the user presses the save button on a record they are asked to login again. Once they have logged in from an XForm, they can save as many records as they like, so essentially it creates another session.

After some back and forth on the eXist mailing list it appears that sessions persist across xqueries, but that the RESTserver does not reuse this session information. The suggested solution was to post from my XForm to an xquery… which is what I had been doing. The real problem I’m having is how to get reliable information back to the user on whether or not the save was successful, and if not (due to an expired session) allow them to log back in without navigating away from the form they are on.

I’m wondering if I can use an XForms alert that is tied to an xforms-submit-error response. I haven’t found any examples of this so far, but I’m still looking.  Also, I seem to be getting an error message back from my form even when the data has submitted successfully.

I’m racking my brains for a workaround solution, because I’d like to get metadata work back underway, so I have a few weeks to monitor it. Any suggestions, or examples you know of would be greatly appreciated. Otherwise I’ll just carry on testing various solutions until I run across one that works.

Fixing temprorary fragments

February 12, 2008

Surprise! Or at least to me… My XForms are not the source of the majority of my temporary fragments, at least as far as I can tell. This is good news because the XForms constitute the most complicated part of my web app, and would take the longest amount of time to troubleshoot and fix.

I’ve follwed the trail of temprorary fragments on my development machine by tracking the exist logs. After every query/page request I check the logs to see if a temprary fragment has been created. In addition  I’ve set up a little xquery that pulls the results out of /db/system/temp so I can see excatly what the temprary fragment is, allowing me to pin point the problems in my xqueries.

The results supprised me, although after a little more reading on eXist and temporary fragments, they make sense. In particular this thread on the eXist mailing list was enlightening. I use the doc() function to return the xml response from Solr and then transform it with XSL stored in eXist. For some reason I was calling the results like this:  doc(‘solrResults’)/child::*  As noted in the above thread, applying an xpath to the returned fragment causes it to be stored as a temporary fragment. The fix has been easy enough, I simply removed the child::* operator and adjusted my XSL.

I have a few other queries that were also creating temporary fragments, and for the most part the issues are very similar, the use of xpath on a returned fragment, rather than using xsl, or even creating variables in the xquery to get the values. For the most part it has been pretty simple to rewrite these queries, and has also given me a chance to clean up some of my code.  In addition I’ve allocated more memory to Tomcat. Hopefully these two adjustments will alleviate the issues we have been having the past few weeks.

Troubleshooting eXist

January 28, 2008

For the past few months I’ve been experiencing some mysterious (to me at any rate) problems with eXist crashing. Generally speaking the problems seemed to be due to corrupted index files, and/or issues with temporary fragments. The problems seem to intensify the more the metadata editor is used, and I noticed when I deleted a collection the database would invariable crash, usually the following night. I’ve been limping along by surreptitiously restarting and re-indexing the database, but have had several instances where the database could not be restored in this way and had to restore from backup. This meant that the site was down and completely non-functional while I restored the files.

I think the issue is pressing enough that I’m going to have to put a lot of other things on hold to sort it out. I’ve found a few threads (1,2,3) on the mailing list that seem to address the issue, and it looks like some of it can be solved by an improvement of my xqueries, and that the issue of temporary fragments is on the minds of the developers.

I think I’ll start going over my xqueries to eleminate the creation of temporary fragments where I can. This will be a useful exercise anyway, as many of the queries were written before I had a through understanding of xquery. Hopefully this will significantly improve performance. Another option could be to move the metadata processing to the development server, and have a master/slave configuration to update the live site every night. Since the biggest problems with crashing/corrupted indexes seems to happen when metadata creation work is highest, this could be a way to cut down interruptions in service on the public side.

I like that UVM has given me the freedom to experiment and use technologies that are not being used elsewhere in the library, however with the freedom have come some headaches. I do not have colleagues to fall back on for troubleshooting. I understand why small digital projects go with systems like ContentDM, the trade off in flexibility may be well worth the time saved in other areas.

Update on CDI XForms

June 14, 2007

Well, the new version of the Firefox XForms extension came out last week, and all my forms are working again. I am working on a version for Orbeon as well and I have a lot of thoughts about Orbeon vrs. Firefox, but will save that for a later post.

I have posted a version of my metadata “processing pages” on xforms@code4lib. This version is in xquery and includes the new tabbed version of the DC forms and also a version of my Solr form. I have also included a sample record. If you want to take it for a test drive you can install eXist and Solr in tomcat and add the sample record to eXist under /db/mets. You may have to make some changes to the addIndex.xql to specify your Solr instance.

Solr revisited

April 23, 2007

Pretty much everything I wrote in my previous post about Solr is now obsolete. Up until last Sunday evening I had Solr running with Cocoon. However I had all sorts of problems with Cocoon, some stemming from my complete inability to go back to using XSLT 1.0 (which I needed to do in order to take advantage of daisy-chaining), and some stemming from bad (non HTML) characters in our metadata, most likely from pasting from Word documents.

At the same time I was struggling with Cocoon, this conversation was happening on the eXist listserv, which reminded me that I could use the eXist doc() function to send Solr requests, and transform the resulting response. I’m blaming being overworked as the reason I wasted so much time with Cocoon when I already use this function for retrieving XSL stylesheets for doing transformations in nearly every XQuery that I write.

So now my requests to Solr are sent via an xquery that looks like this:

xquery version "1.0";

declare namespace util="";

declare namespace request="";

declare namespace x="";

declare namespace xlink = "";

declare namespace xslt="";

declare namespace bh = "";
(:Fields for limiting search : )

declare variable $field1 {request:request-parameter('field1', 'ft')};

declare variable $field2 {request:request-parameter('field2', 'ft')};

declare variable $field3 {request:request-parameter('field3', 'ft')};(:Search terms:)

declare variable $term1 {replace(request:request-parameter('term1', ''), "'", '"')};

declare variable $term2 {replace(request:request-parameter('term2', ''), "'", '"')};

declare variable $term3 {replace(request:request-parameter('term3', ''), "'", '"')};
(:Boolean operators: )

declare variable $bool1 {request:request-parameter('bool1', 'and')};

declare variable $bool2 {request:request-parameter('bool2', 'and')};
(:Variables for paging through results: )

declare variable $start {request:request-parameter('start', 0) cast as xs:integer};

declare variable $rows {request:request-parameter('rows', 25) cast as xs:integer};
(:Filters applied to search results: )

declare variable $filter {request:request-parameter('filter', '')};
(: Applies correct Solr field for fielded searching : )
declare function bh:field($field as xs:string) as xs:string {
   if ($field = "au") then
   else if ($field = "ti") then
   else if ($field = "ab") then
   else if ($field = "su") then
    else ''
(: Builds query parameters as a string : )
declare function bh:build-query()as xs:string{
let $queryString :=
        if ($term1 != '') then
          concat(bh:field($field1), $term1)
        else '',
        if ($term2 != '') then
            if ($bool1 = 'and' and $term1 != '') then ' AND '
            else if ($bool1 = 'or' and $term1 != '') then ' OR '
            else if($bool1 = 'not' and $term1 !='') then ' NOT '
            else ' ',bh:field($field2), $term2)
            else '',
         if ($term3 != '') then
             if ($bool2 = 'and' and ($term1 != '' or $term2 != '')) then ' AND '
             else if ($bool2 = 'or' and $term1 != '' or $term2 != '') then ' OR '
             else if ($bool2 = 'not' and $term1 != '' or $term2 != '') then ' NOT '
             else ' ',bh:field($field3), $term3)
             else '',
         if ($term1 = '' and $term2 = '' and $term3 = '') then
            concat('/no-search-terms',' ')
         else '' )
  return encode-for-uri($queryString)
declare function bh:filter(){
 if($filter != '') then
    encode-for-uri(concat(' ',translate($filter,';',' ')))
 else ''
declare function bh:fullQuery(){
let $searchPath :=
return  $searchPath
(:Stylesheet used for dispay: )
let $xsl := doc('/path/search.xsl')
(:Format results : )
let $results :=
<query-results term1="{$term1}" field1="{$field1}" bool1="{$bool1}"
   term2="{$term2}"field2="{$field2}" bool2="{$bool2}"
   term3="{$term3}" field3="{$field3}" filter="{bh:filter()}">
         if((exists($term1) and $term1 = '') and (exists(term2) and $term2 = '')
           and (exists(term3) and $term3 = '') ) then
             <response hits="0">Your search returned 0 results</response>
         else    doc(bh:fullQuery())/child::*
return xslt:stream-transform($results, $xsl, () )

The results are transformed using XSLT (2.0).

I find this works pretty well, but I’m also very interested in exploring this new HTTP extension model which is pretty much what I was hoping for back when I started exploring the Solr/eXist combination. (Which just demonstrates once again what a great community of developers eXist has.)

Documents are still added to the index using a combination of XQuery and XForms. Next week I’ll be refining our editor to make submitting completed records to the index a one (maybe two) button process. I’m pretty pleased with Solr and have gotten a very positive response to the browsing and limiting features. I still have some features to work on, for example, while my users can add filters to search results, they can not remove them. This seems like a pretty easy javascript fix, but I haven’t really had the time to implement it yet.

A Good Year… for XML

February 16, 2007

Elliotte Rusty Harold over at IBM developerWorks is predicting an exciting year for XML (Ten predictions for XML in 2007) , including a nod at XQuery, native XML databases, and XForms, all of which I’m rather heavily invested in. I’ve been pretty comfortable with the choices I made for the CDI, still, Harold’s predictions are reassuring, it is nice to hear someone else is predicting a bright future for the limb I’ve climbed out onto.

Getting xforms and eXist talking

October 23, 2006

I used three different methods for implementing XForms with eXist (I am currently only using #3).

  1. Orbeon Presentation Server
  2. An XQuery passed through a cocoon pipeline which transformed the XML results using an XSL stylesheet. The edited documents were saved to eXist using REST put.
  3. An XQuery that first authenticates the user and then outputs an xform using the retrieved document as the XForms instance and using REST put for submission.

I looked at Orbeon about 6 months ago, it seems like a great product, especially in light of the poor browser support for XForms. If I needed to make a commercially viable XForms application I would probably spend some more time playing with Orbeon. However, I felt that Orbeon added yet another layer of complication to my application and it was not necessary in our situation because our XForms are being used in a controlled environment (and I don’t have more time). I can be assured that everyone using the editing interface will be using Firefox, not a great solution, but so far so good.

I generally prefer XSL to XQuery, so my XQueries have, in the past been pretty simple, returning very raw results to an XSL stylesheet. My first attempt at XForms used XQuery to call the XML document to be edited (or to create a new document). Then I had a cocoon pipeline  to specify the XSL stylesheet for transformation and the serialization option.

<map:match pattern="getDC.xq">
  <map:generate src="getDC.xq" type="xquery"/>
  <map:transform src="stylesheets/dcForms.xsl"/>
  <map:serialize type="xhtml11"/>

In the main sitemap.xmap file (included with exist), xhtml11 is the serializer that will output mime-type=”application/xhtml+xml”.

The XSL stylesheet then selects the node I’m interested in and uses xsl:copy to populate the XForms instance. The submission element uses put to replace the existing document with the edited document.

  <xf:instance id="metadata">
    <xsl:copy-of select="/child::*"/>

  <xf:submission id="submit" method="put" replace="all">
    <xsl:attribute name="action">
      <xsl:value-of   select="concat


One of the features that I found the most useful in XForms is the ability to repeat elements as needed. Here is an example from a very simple form that we used to edit Dublin Core records that uses XForms repeat. This field populates the dc:subject element and adds a type attribute selected from a drop down menu. New subject elements are added or deleted when the users click the add or delete buttons created by xf:trigger. I use xf:setvalue to create each new element as a blank element, otherwise the element will simply copy the data from the first instance of the element.

<xf:repeat id="repeat.subject" nodeset="//dc:subject">
  <xf:input ref=".">
   <xf:label>Subject Headings:</xf:label>

  <xf:select1 ref="@type">
   <xf:label>Type: </xf:label>
 <xf:trigger class="delete" appearance="minimal">
    <xf:action ev:event="DOMActivate">
     <xf:delete nodeset="instance('metadata')//dc:subject"
 <xf:trigger class="add" appearance="minimal">
    <xf:label>Add a subject field</xf:label>
    <xf:action ev:event="DOMActivate">
      <xf:insert nodeset="instance('metadata')//dc:subject"
           at="index('repeat.subject')" position="after"/>
      <xf:setvalue ref="instance('metadata')//dc:subject[last()]" value=""/>

This method works fine, but I as stated in an earlier post I wanted to route all my metadata processing XQueries through a password authenticating XQuery. This query calls a series of metadata administrative tasks, including the XForms.

I added the following namespaces for XForms to my xquery:

declare namespace xf="";
declare namespace ev="";

And the exist:serialize option:

declare option exist:serialize "method=xhtml media-type=application/xhtml+xml";

The XForm is contained in a function called by an authenticating function. I made very few changes to the XForm code I had been using in my XSL stylesheets to get it working when called by the XQuery.

Here is a round-up of some of the resources I have found to be most useful in building these forms:

  1. eXist mailing list – search the archives for XForms
  2. eXist – XQuery examples
  3. Cocoon website (pipelines)
  4. XForms tutorial – Adrian de Jonge’s blog
  5. XForms – Tutorials and Cookbook – Wikibooks
  6. XForms for HTML Authors – W3C
  7. O’Reilly XForms Essentials by Micah Dubinko

One step forward, one step back

October 13, 2006

eXist crashed on Wednesday. Actually crash is probably the wrong word, it seemed to be running fine but then failed to restart when I restarted Tomcat. We have backups, run early every morning, but that doesn’t help for the data that was entered during the day on Wednesday. More worrisome is that it is still unclear to me what caused the corruption in the database. I found a few discussions on the exist mailing list that seemed to be about similar problems, but without any satisfactory answers as to why the corruption occurred.

After a day and a half of trying to figure out what went wrong I caved and wrote to the list. I try to put that off for as long as possible, because while the list is very active, and generally helpful, I hate asking a question and then figuring out the answer myself later (or, I’ll be honest, getting an answer back that makes me feel stupid). I ‘ve been unable to reproduce the error after replacing the corrupt instance with the one from the backup. I have a feeling it was something I was working on during the morning on Wednesday, which means either my search xquery (which was outputting some java exceptions), or perhaps some of the xupdates I was using to add new elements to a few hundred documents at once.

Now that we are back up and running I’m returning to the question of my search xqueries, which I think need to be a little more sophisticated, the heart of which looks like this:

let $results :=
for $hits in collection('/db/mets')/mets:mets/mets:dmdSec[@ID='dmdDC']
    //descendant::dc:dc/child::* [self::* |= $_query]let $type := 
$hits/ancestor::mets:mets/@TYPElet $title := $hits/parent::*/dc:title[1]
let $id := $hits/parent::*/dc:identifier
<item id="{string($id)}" type="{string($type)}">

It is problematic because it returns multiple hits for a single document. This is a pretty easy fix to make, but I also ran into a problem with this query when I had over 1000 hits, I encountered a java error (as noted above), so I will need rework this. I can limit the number of results returned, or I could use this search to only search collection level records, not item level records, most likely the first option. I have also had a request to include the author/creator field in the results which is a minor fix.

Update: My answer from the eXist list about the database corruption:

I fixed your issue. It wasn’t a “real” corruption, just removing the .lck files would have helped. As the exception shows, the lock files were damaged.

> at
> at

Anyway, the startup process should handle this. After a database crash, the file locks might be incomplete. eXist will now check this.

So, that is good to know. Also eXist 1.0 and 1.1 final have just been released, I may take some time this week to upgrade to 1.1 final.

Recovered and moving on

October 5, 2006

Well, the weekend was enough time for me to recover from the extreme frustration I experienced on Friday, and I’m happy to say that my xforms are now being called by a password protected xquery.

As mentioned in my last post the big hang up was the mime type (declared in the HTTP header) for the xforms, it has to be application/xhtml+xml (which by the way IE 6 does not support). I had already written a very simple xquery and then transformed the data with an XSL stylesheet. However when I tried the transform:stream-transform using this xquery and stylesheet I had no luck forcing the HTTP header to output the correct mime type. So, much to my disappointment, I ended up having to write the entire xform in xquery. There really isn’t any drawback to this other than that I’m much more comfortable coding with XSL than xquery. The forms are working and are now password protected, so I’m feeling pretty good about that. I can continue research into the problematic mime types at a later date if I really want to go back to using XSL.

So what I have ended up with is one long xquery that authenticates the user and then initiates the various pieces of the metadata processing interface. All portions of the interface are called by different functions in this xquery, which has allowed me to learn user defined xquery functions without to much pain, as I had originaly written all of these as separate xqueries. For this xquery I defined each action (for example: view metadata queue, create new collection and edit records) as a separate function and then call them from a main function which first tests for session authentication and if it doesn’t find it presents the user with a log-on screen.

I’m feeling pretty good about my new understanding of user created functions, and may be ready to try something a little more complex, like recursive functions.

Here is the short list of resources I found most useful for learning functions, and also for writing and troubleshooting my xforms:

Xquery Functions (

A “how to” for xforms (

WC3 intro to xforms (

And the eXist listserv archives (actually there seems to be a lot of talk on the listserv just in the past few weeks about eXist and xforms).

I also have a rudimentary page turning application up and running, more about that later.

Update:  I just wanted to clarify where the problem with the mime-type was cropping up. As one commenter noted it is possible to change the mime type in eXist with declare option exist:serialize “method=xhtml media-type=application/xhtml+xml”; which is what I ended up using for my xquery based xform. I had originally tried to transform the output from this xquery this with an xsl stylesheet, and this is where I “lost” the application/xhtml+xml mime type. I also had an xsl:output statement in my xsl stylesheet that looked like this: <xsl:output omit-xml-declaration=”yes” encoding=”UTF-8″ method=”xhtml” indent=”yes” media-type=”application/xhtml+xml”>. I retrieved two different results. When I used transform:transform which goes through eXist I returned the correct mime type, but could not get my namspace declarations into the html tag and when I used transform:stream-transform, which bypasses eXist, I had all the correct namespaces in my html tags but the mime type was text/html. I’m guessing the problem lies in a default output for eXist and perhaps Tomcat.