XQuery in eXist-db

XQuery Test Suite compliance

The eXist-db XQuery implementation is tested against the official XML Query Test Suite (XQTS). This suite, which focuses on XQuery 1.0, contains more than 14,000 tests, and eXist-db passes more than 99.4% of the tests. We are continuously trying to improve these results. The XQuery 3.0 Test Suite (QT3) has just become available and has been integrated into our test harness, though the results are not reliable yet.

Supported Optional Features

In addition to the standard features, eXist-db provides extended support for modules and implements the full axis feature, which means you can use the optional axes: ancestor, ancestor-or-self, following, following-sibling, preceding, and preceding-sibling. (The only optional axis not supported is the namespace axis.)

Unsupported features

eXist-db implements all features described in the XQuery 1.0 specification, with the exception of the following:

• Schema-related Features (validate and import schema). eXist-db's XQuery processor does currently not support the schema import and schema validation features defined as optional in the XQuery specification. eXist-db provides extension functions to perform XML validation. The database does not store type information along with the nodes. It therefore cannot know the typed value of a node and has to assume xs:untypedAtomic. This is the behaviour defined by the XQuery specification.

• eXist-db does not support specifying a data type in an element or attribute test. The node test element(test-node) is supported, but the test element(test-node, xs:integer) will result in a syntax error.

eXist-db does nevertheless support strong typing whenever the expected type of an expression, a function argument or function return value is explicitly specified or can be known otherwise. eXist-db is not lax about type checks.

XQuery 3.0 Support

eXist-db implements the following features of the "XQuery 3.0" Working Draft

• Higher Order Functions: eXist-db completely supports higher-order functions, including features like inline functions, closures, and partial function application. For more information, see the article on the eXist-db blog, Higher-Order Functions in XQuery 3.0

• Group by clause in FLWOR expressions: "group by" provides an efficient way to group the sequences generated in a FLWOR expression. For example,

  xquery version "3.0"; for $speechBySpeaker in //SPEECH[ft:query(., "king")] group by $speaker := $speechBySpeaker/SPEAKER order by $speaker return <speaker name="{$speaker}"> { $speechBySpeaker } </speaker>

  queries the Shakespeare plays and groups the result by speaker.

• Try/Catch: The try/catch expression provides error handling for dynamic errors and type errors. For example,

  try { 'a' + 7 } catch * { concat($err:code, ": ", $err:description) }
  returns the full error (excerpted here):

  err:XPTY0004: It is a type error if...

  For more information, see the article on the eXist-db blog, Higher-Order Functions in XQuery 3.0 and the specification.

• The new String Concatenation expression: A convenient alternative to the concat() function. Strings can be joined with

  ||

  . For example,
  "Hello " || $world || "!"
  is equivalent to
  concat("Hello", $world, "!")
  . See the original announcement on exist-open and the specification.

• The Simple Map Operator: Can be used to replace short "for" loops, providing performance benefits due to its simpler processing compared to a full "for" statement. In an example like

  ("red", "blue", "green") ! string-length() ! (. * 2)
  , the right-hand expression is evaluated once for each item in the sequence to the left of "!". The example results in

  6, 8, 10

  . See the original announcement on exist-open and the specification.

• Switch expression: Eliminates the need for long conditional chains for string values. See the article on the eXist-db blog, Switch Expression, and the specification.

• New functions: XQuery 3.0 adds a number of functions, some previously available only in XSLT are now available in XQuery 3.0. See, for example, the specification on date/time formatting functions like fn:format:date(), formatting numbers like fn:format-number(), and fn:analyze-string().

• Private functions and function annotations: Annotations declare properties associated with functions and variables. For example, functions can now be declared as %private or %public. See the specification.

• Serialization Parameters: XQuery 3.0 version of the language provides a standard way to set serialization parameters. The old, non-standard approach in eXist-db was to use a single option with all parameters in the value:

  declare option exist:serialize "method=json media-type=application/json";
  The standard way of doing this in XQuery 3.0 would be:
  declare namespace output = "http://www.w3.org/2010/xslt-xquery-serialization"; declare option output:method "json"; declare option output:media-type "application/json";
  See the original announcement on exist-open and the specification

To make use of these features of XQuery 3.0, use the proper version declaration in the prolog of your queries:

xquery version "3.0";

Missing XQuery 3.0 Features

eXist-db 2.0 does not support some of the less frequently used XQuery 3.0 constructs, mainly because we did not encounter many use cases for them yet. This may certainly change though:

• "tumbling" and "sliding window" in FLWOR expressions

• "count" clause in FLWOR expressions

• "allowing empty" in FLWOR clause

The following functions from the XQuery 3.0 function specification are missing as well, but most of them are in fact just replacements for functions which have been available in eXist-db since a long time:

XQuery 3.1

The specification for XQuery 3.1 is still subject to change. The main additions are maps and arrays.

Other Related Specifications

• Full Text Search: eXist-db has an implementation-specific Full Text Search facility, built on the Lucene library (among several methods of indexing). It does not currently support the syntax in the W3C XQuery and XPath Full Text 1.0 Recommendation.

• XQuery Update: eXist-db has an implementation-specific XQuery Update syntax. It does not currently support the syntax in the W3C XQuery Update Facility 1.0 Recommendation. The main difference is that the eXist-db implementation supports in-place updates. Switching to the W3C recommendation would break backwards compatibility. We postponed this step to after 2.0.

Using Relative URIs

If the location specified in an import statement is a relative URI, the query engine will try to load the module relatively to the current module load path. The module load path is determined as follows:

1. if the main XQuery was retrieved from the file system, the module load path points to that directory. This applies to queries executed through the XQueryServlet, XQueryGenerator or the Java admin client.

2. if the main XQuery was loaded from a database collection, the module load path is the URI of that collection.

   For example, if you access an XQuery via the REST server:

   http://localhost:8080/exist/servlet/db/modules/test.xq

   All relative module paths will be resolved relative to the /db/modules collection.

Preloaded Modules

Preloaded modules do not need to be explicitly imported or declared in the prolog of queries. The <builtin-modules> element in conf.xml lists the namespaces and the corresponding Java class that implements the module of all modules to be preloaded:

Creating Modules in XQuery

You can declare an XQuery file as a module and import it using the import module directive. The XQuery engine imports each module only once during compilation. The compiled module is then made available through the static XQuery context.

Creating Modules in Java

To register a Java-based XQuery modules, eXist-db requires a namespace URI by which the module is identified, and the list of functions it supplies. For this, you need only to pass a driver class to the XQuery engine, and this class should implement the interface org.exist.xpath.InternalModule.

Moreover, the class org.exist.xpath.AbstractInternalModule already provides an implementation skeleton. The class constructor expects an array of function definitions for all functions that should be registered. A function definition (class FunctionDef) has two properties: the static signature of the function (as an instance of FunctionSignature), and the Java Class that implements the function.

A function is a class extending org.exist.xquery.Function or org.exist.xquery.BasicFunction. Functions without special requirements (e.g. overloading) should subclass BasicFunction. To illustrate, the following is a simple function definition:

In looking at this sample, first note that every function class has to provide a function signature. The function signature defines the QName by which the function is identified, a documentation string, the sequence types of all arguments, and the sequence type of the returned value. In the example above, we accept a single argument named "text" of type xs:string and a cardinality of ZERO_OR_MORE with the description "The text to echo". In other words, we accept any sequence of strings containing zero or more items. The return value is of type xs:string and a cardinality of ZERO_OR_MORE with the description "the echoed text". Note: The parameter description should be normal sentence starting with a capital letter. The return value description is always prepended with "Returns ", so have the text to match.

Next, the subclass overwrites the eval method, which has two arguments: the first contains the values of all arguments passed to the function, the second passes the current context sequence (which might be null). Note that the argument values in the array args have already been checked to match the sequence types defined in the function signature. We therefore do not have to recheck the length of the array: if more or less than one argument were passed to the function, an exception would have been thrown before eval gets called.

In XQuery, all values are passed as sequences. A sequence consists of one or more items, and every item is either an atomic value or a node. Furthermore, a single item is also a sequence. The function signature specifies that any sequence containing zero or more strings is acceptable for our method. We therefore have to check if the empty sequence has been passed. In this case, the function call returns immediately. Otherwise, we iterate through each item in the sequence, prepend echo:" to its string value, and add it to the result sequence.

In the next step, we want to add the function to a new module, and therefore provide a driver class. The driver class defines a namespace URI and a default prefix for the module. Functions are registered by passing an array of FunctionDef to the constructor. The following is an example driver class definition:

Finally, we are able to use this newly created module in an XQuery script:

The query engine recognizes the java: prefix in the location URI, and treats the remaining part (in this case, org.exist.examples.xquery.ExampleModule) as a fully qualified class name leading to the driver class of the module.

Serialization in XQuery 1.0

In XQuery 1.0, serialization parameters can be set by declare option statement in the query prolog. In declare option, the serialization parameters can be specified as follows:

declare option exist:serialize "method=xhtml media-type=application/xhtml+html";

Here, single options are specified within the string literal, separated by a whitespace. Note also that the option QName must be exist:serialize, where the exist prefix is bound to the namespace http://exist.sourceforge.net/NS/exist, which is declared by default and need not be specified explicitly.

Note that these same options can be passed using the XPathQueryService.setProperty() and XQueryService.setProperty() methods in Java. These methods are defined in javax.xml.transform.OutputKeys and EXistOutputKeys. The latter eXist-specific options include the following:

expand-xincludes= yes | no

should the serializer expand XInclude elements?

highlight-matches= both | elements | attributes | none

when querying text with the full text or n-gram extensions, the query engine tracks the exact position of all matches inside text content. The serializer can later use this information to mark those matches by wrapping them into an element <exist:match>.

Setting highlight-matches=both will enable this feature for every kind of indexable node.

process-xsl-pi= yes | no

if a document is serialized and it has an XSL processing instruction, eXist-db can try to load the referenced stylesheet and apply it to the document.

add-exist-id= element | all | none

shows the internal node ids of an element by adding an attribute exist:id="internal-node-id". Setting add-exist-id to "element" will only show the node id of the top-level element, "all" will show the ids of all elements.

The general options include the following:

method= xml | xhtml | json | text

determines the serialization method. Should be one of "xml", "xhtml", "json", or "text". The "xhtml" method makes sure that XHTML elements with an empty content model are serialized in the minimized form, i.e. img will be output as <img/>.

jsonp= myFunctionName

Only relevant when the serialization method is set to 'json'. Causes the JSON output to be wrapped in the named JSONP function.

media-type

The MIME content type of the output. It will mainly be used to set the HTTP Content-Type header (if the query is running in an HTTP context).

encoding

specifies the character encoding to be used for outputting the instance of the data model

doctype-public

a doctype declaration will be output if doctype-public and/or doctype-system are set. The corresponding identifier is taken from the value of the parameter.

doctype-system

a doctype declaration will be output if doctype-public and/or doctype-system are set. The corresponding identifier is taken from the value of the parameter.

indent= yes | no

indent the document to make it easier to read. Indenting adds whitespace characters to element nodes, restricted by the rules given in the XQuery serialization spec.

omit-xml-declaration= yes | no

output an XML declaration if the parameter is set to "no"

For example, to disable XInclude expansion, and indent the output, you can use the following syntax:

declare option exist:serialize "expand-xincludes=no";

For the output method parameter, eXist-db currently recognizes three methods: xml, xhtml and text. Note that unlike the xml method, the xhtml setting uses only the short form for elements that are declared empty in the xhtml DTD. For example, the br tag is always returned as <br/>. On the other hand, the text method only returns the contents of elements - for instance, <A>Content</A> is returned as Content. However, attribute values, processing instructions, comments, etc. are all ignored.

Serialization in XQuery 3.0

XQuery 3.0 version of the language provides a standard way to set serialization parameters. The old, non-standard approach in eXist-db was to use a single option with all parameters in the value:

The old approach is still supported for backwards compatibility. The parameter names remain the same as well.

exist:timer

Provides a simple way to measure the time for executing a given expression. For example:

(# exist:timer #) { //some/path/expression }

creates a timer for the expression enclosed in curly braces and prints timing information to the trace logger. Please note that trace needs to be enabled in log4j.xml:

exist:batch-transaction

Currently only for XQuery Update Extensions. Provides a method for batching updates on the database into a single Transaction, allowing a set of updates to be atomically guaranteed. Also for each affected document or collection, any configured Triggers will only be called once, the prepare() method will be fired before the first update to the configured resource and the finish() method fired after the last update to the configured resource.

(# exist:batch-transaction #) {
	            	update value //some/path/expressionA width "valueA",
            		update value //some/path/expressionB width "valueB"
	             }

Uses a single Transaction and Trigger events for the expressions enclosed in curly braces.

exist:force-index-use

For debugging purposes. An expression that can be assisted by indexes: comparisons, fn:matches()... Will raise an error if, for any reason, this assistance can not be performed.

This can help to check whether the indexes are correctly defined or not.

(# exist:force-index-use #) {
	            	//group[. = "dba"]
	             }

Raises an error (currently XPDYxxxx since this kind of dynamic error is not yet defined by the XQuery specifications) if the general comparison doesn't use a range or a QName index.

exist:no-index

This prevents the query engine to use the index in expressions that can be assisted by indexes: comparisons, fn:matches()... Useful if the searched value isn't very selective or if it is cheaper to traverse the previous step of a path expression than querying the index.

(# exist:no-index #) {
                    //group[. = "dba"]
                 }

exist:optimize

For testing only. This pragma is normally inserted automatically by the query rewriter (if enabled) to optimize an expression that implements the org.exist.xquery.Optimizable interface.

//((#exist:optimize#) { item[stock = 10] })