Database Backup and Restore

Abstract

This section discusses eXist-db's database backup/restore procedures. eXist-db provides different methods for creating backups, which will be explained below.

Backup Format

During backup, eXist-db exports the contents of its database (as standard XML files) to a hierarchy of directories on the hard drive. This hierarchy is organized according to the organization of collections in the database.

Other files stored during backup include index configuration files and user settings. Resource and collection metadata is exported to a special XML file, called __contents__.xml, which lists information including the resource type, owner, modification date and/or the permissions assigned to a resource. You will find one __contents__.xml file in each directory created by the backup. This descriptor file is required to restore the backup.

Since eXist-db uses an open XML format rather than a proprietary format for its database files, users can manually modify files in the backup directories without requiring special software. Any changes made to these files are reflected in the database with a restore or once the data is imported to another database system.

It is even possible to directly edit user data and permissions stored in the file /db/system/users.xml. This is particularly useful when making global changes to the user database. For example, to reset the passwords for all your users, you can simply edit the file users.xml by removing the password attribute, or set it to a default value and restore the document.

Backup Methods

There are two main methods for creating a backup:

Client-side

You can use the Java admin client or a small command line utility to create a backup of the data on the server. In this case, the client controls the backup process. The server is not blocked and continues to accept requests from other clients. Other users can modify the db while the backup is running, so logical dependencies between different resources may not be preserved correctly.

Client-side backups are not safe. The client uses the XML:DB API to access the db. This means that it cannot backup documents or collections if they are damaged in any way.

Server-side

Server-side backups are usually run through eXist-db's job scheduler, though they can also be triggered via the web interface. This type of backup extracts the data directly from the low-level database files. It is thus much faster then a client-side backup. It also supports incremental backups.

It is guaranteed that the database is in a consistent state while the backup is running. Possible corruptions in the db will be detected and the backup process will try to work around them. In many cases, damaged resources can at least be partially recovered.

The format of the generated backup archives will be the same for both backup methods. They can all be restored via the standard Java client.

Server-side Backup

This is now the recommended backup method. To guarantee consistency, server-side backups are always executed as system tasks, which means that the database will be switched to a protected service mode before the backup starts. eXist-db will wait for all pending transactions to complete before it enters protected mode. A database checkpoint will be performed and the backup task is executed. While the system task is running, no new transactions will be allowed. Concurrent requests by other clients will be blocked and added to the internal queue. Once the backup is complete, the database will switch back to normal service and all locks will be released.

You can trigger the backup via the Dashboard:

Clicking the Trigger button will schedule a single backup task. The server will wait for all running transactions to return before it executes the task. You can click on Refresh to update the view, which lists all backup archives currently available within the standard backup directory. Click on the name of an archive to download it.

Beginning with version 1.2.5, eXist-db can also create incremental backups. Only resources which were modified since the last backup are archived.

All backups will be stored in ZIP format into a directory export/ below the main data directory, which means webapp/WEB-INF/data/export/ by default (unless you configured a different data directory in conf.xml).

Automatic Consistency Check

The system task runs a consistency and sanity check on the database before backing it up. Reports of this check are written into the same directory as the backup archive. The last report can also be viewed via JMX (see below).

The consistency check will first check the collection hierarchy, then scan through the stored node tree of every document in the db, testing node properties like the node's id, child count, attribute count and node relationships. Contrary to normal database operations, the different dbx files are checked independently. This means that even if a collection is no longer readable, the tool will still be able to scan the documents in the damaged collection.

The backup task uses the information provided by the consistency check to work around damages in the db. It tries to export as much data as possible, even if parts of the collection hierarchy are corrupted or documents are damaged:

• Descendant collections will be exported properly even if their ancestor collection is corrupted

• Documents which are intact but belong to a destroyed collection will be stored into a special collection /db/lost_and_found

• Damaged documents are detected and are removed from the backup

<job type="system" class="org.exist.storage.ConsistencyCheckTask"
    cron-trigger="0 0 0/12 * * ?">
    <!-- the output directory. paths are relative to the data dir -->
    <parameter name="output" value="export"/>
    <parameter name="zip" value="yes"/>
    <parameter name="backup" value="yes"/>
    <parameter name="incremental" value="yes"/>
	<parameter name="incremental-check" value="no"/>
</job>

Emergency Export Tool

eXist provides a graphical interface to the consistency check and backup utilities which can be used in case of an emergency, in particular if the database does not start up properly anymore. The tool needs direct access to the database files, so any running database instance has to be stopped before launching the GUI.

Use the following command line to start the utility:

java -jar start.jar org.exist.backup.ExportGUI

If you installed the eXist distribution using the installer, a shortcut to this should have been placed into the start menu, so you don't need to type above command.

On a headless system you can use the command-line version instead:

java -jar start.jar org.exist.backup.ExportMain

Call it with parameter -h to get a list of possible options.

For every check run, an error report will be written into the directory specified in Output Directory. If you clicked on Check Export, the utility will also export the database into a zip file in the same directory. This backup can be restored via the standard backup/restore tools.

Using JMX to View Check Reports

If Java Management Extensions (JMX) are enabled in the Java VM that is running eXist, you can use a JMX client to see the latest consistency check reports. The screenshot shows jconsole, which is included with the Java 5 and 6 JDKs.

eXist also includes a command-line JMX client. Call it with parameter -s to see the latest consistency report:

java -jar start.jar org.exist.management.client.JMXClient -s

This may produce output as shown below:

Sanity report
-----------------------------------------------
                Status: FAIL
      Last check start: Thu May 08 21:40:00 CEST 2008
        Last check end: Thu May 08 21:40:00 CEST 2008
            Check took: 594ms
            Error code: RESOURCE_ACCESS_FAILED
            Description: 32

You can also subscribe to the notifications made available by the SanityReport MBean to be informed of sanity check results. Please consult the documentation on how to configure JMX.

Client-side Backup

You can either use the Java-based Admin Client, or the backup command line utility.

If you are using the Admin Client, do the following:

If you are using the command-line utility for the backup/restore, do the following:

To launch the utility, do ONE of the following:

• start either the bin/backup.sh (Unix), OR the bin/backup.bat (Windows/DOS) script file

• OR enter on the command-line:

  java -jar start.jar backup 
                          -u [admin_username] 
                          -p [admin_password] 
                          -b [collection_path] 
                          -d [target_path]
                          -ouri=[xml_uri]

  To view the all of the available options for this command, use the -h parameter.

Use the -b parameter to indicate the collection path, and the -d parameter to indicate the target directory on your system. You can also specify the current admin username using the -u parameter, and the admin password using the -p parameter. For example, to backup the entire database on a Unix system to the target directory /var/backup/hd060501, you would enter the following:

bin/backup.sh -u admin -p admin-pass -b /db -d /var/backup/hd060501

By default, the utility connects to the database at the URI: xmldb:exist://localhost:8080/exist/xmlrpc. If you want to backup a database at a different location, specify its XML:DB URI (excluding any collection path) using the -ouri parameter. For example, the following backup on a Unix Tomcat system running on port 80 specifies the database URI xmldb:exist://192.168.1.2:80/xmlrpc

bin/backup.sh -u admin -p admin-pass -b /db -d /var/backup/hd060501
                -ouri=xmldb:exist://192.168.1.2:80/xmlrpc

Restoring the Database

bin/backup.sh -u admin -p admin-pass -P backup-pass 
                        -r /var/backup/hd060501/db/__contents__.xml

Backing Up Single Apps

Creating a backup of a single expath application or library package is rather straightforward within eXide. Either use:

• the synchronize feature to write the package contents to a directory on disk, or

• call Application / Download App to retrieve a .xar package which can be deployed into another eXist-db instance

Standalone Backup/Restore

backrest