Backing-up/Restoring Jenkins

Having good backups of your Jenkins instance is critically important. Backups are used for:

  • Disaster recovery.

  • Recovering an older configuration (an accidental configuration change may not be discovered for some time).

  • Recovering a file that is corrupted or was deleted accidentally.

This page discusses the following:

  • How to create a backup

  • Files that should be backed up

  • How to validate a backup to ensure that it is usable

Creating a Backup

Various schemes can be used to create backups. These are discussed in this section:

  • Filesystem snapshots

  • Plugins for backup

  • Write a shell script that backs up the Jenkins instance

Filesystem snapshots

Filesystem snapshots provide maximum consistency for backups. They also run faster than live backups, reducing the possibility of copying different data at different time points. They are supported by:

  • The Linux Logical Volume Manager (LVM)

  • Linux btrfs

  • Solaris ZFS (which also supports incremental backups)

  • FreeBSD ZFS

  • OpenZFS on Linux

  • Some other file system architectures

  • Many cloud providers

  • Some separate storage devices also let you create snapshots at the storage level.

Plugins for backup

Several plugins are available for backup. From the main menu select Manage Jenkins, then go to Manage Plugins>Available and search for backup. Note that none of the open source plugins are currently being maintained. You can try these plugins but you may have problems with them.

Writing a shell script for backups

You can write your own shell script that copies the appropriate files and directories to a backup location. Use cron to schedule when the backup script runs.

The shell script should create a directory such as /mnt/backup to which the backup will be written; be sure that you have write permissions to that directory. Consider creating /mnt/backup as a separate filesystem with its own mount point. An alternative method is to create a subdirectory in /var. Note that if you use this method, you might need to use the sudo command to execute the restore operation. Backing up to /tmp is not advised because /tmp may be cleaned on reboot.

Create a unique identifier for each backup (use a timestamp, for example) to ensure that today’s backup does not overwrite yesterday’s backup.

Writing files to a local file system is the fastest way to take the backup. Consider copying the completed backup to a remote backup server or device for long term storage.

Back up the Controller Key Separately

Never include the controller key in your Jenkins backup!

The controller key is used to encrypt data in the secrets directory that secures credentials. It is stored in the $JENKINS_HOME/secrets/hudson.util.Secret file in the $JENKINS_HOME/secrets/ directory and encrypted with master.key. If you need to restore a system from a backup, you will need this file. And, if someone else accesses your backups and has this key, they have full access to all your information.

You should treat your controller key like you treat your SSH private key and NEVER include it in a regular backup. Instead, back up the master.key file separately and store it in a very secure location away from your other backups. It is a very file small that is seldom changed. If you need to do a full system restore, you will need to restore the rest of the system and then apply the backup of the master.key file separately.

Which Files Should Be Backed Up?

The number of files you back up can affect both the time required to run the backup and the size of the resulting backup. It also impacts the complexity of restoring the system from the backup. Here we discuss why various files should be backed up and list some files that could safely be excluded from at least some backups.

$JENKINS_HOME

Backing up the entire $JENKINS_HOME directory preserves the entire Jenkins instance. To restore the system, just copy the entire backup to the new system.

Note, however, that JENKINS_HOME includes a number of files that do not really need to be backed up. Selecting specific directories and files to back up yields smaller backups but may require a greater effort to restore a system. One approach is to back up different directories on different schedules.

Configuration files

Configuration files are stored directly in the $JENKINS_HOME directory. ./config.xml is the main Jenkins configuration file. Other configuration files also have the .xml suffix. Specify $JENKINS_HOME/*.xml to back up all configuration files.

Configuration files can also be stored in an SCM repository. This keeps copies of all previous versions of each file that can be retrieved using standard SCM facilities.

./jobs Subdirectory

The $JENKINS_HOME/jobs directory contains information related to all the jobs you create in Jenkins.

  • ./builds — Contains build records

  • ./builds/archive — Contains archived artifacts

    • Back this up if it is important to retain these artifacts long-term

    • These can be very large and may make your backups very large

  • ./workspace — Contains files checked out from the SCM

    • It is usually not necessary to back up these files. You can perform a clean checkout after restoring the system.

  • ./plugins/*.hpi — Plugin packages with specific versions used on your system

  • ./plugins/*.jpi — Plugin packages with specific versions used on your system

What may not need to be backed up

The following files and directories do not usually need to be included in every routine backup because you can download the latest version when you are restoring a system. However, some disaster recovery experts recommend against doing any upgrades while restoring the system, to avoid delays caused by compatibility issues that might arise. If your disaster recovery plan specifies that you restore the system using the same software versions that were previously running, you can make an infrequent backup of the system and all downloaded tools and use that to restore the system..

  • ./war — Exploded war file

    • To restore a system, download the latest war file.

  • ./cache — Downloaded tools

    • To restore a system, download the current version of the tools.

  • ./tools — Extracted tools

    • To restore a system, extract the tools again.

  • ./plugins/xxx — Subdirectories of installed plugins

    • These will be automatically populated on the next restart.

Validating a backup

Your backup strategy should include validation of each backup. You do not want to learn that your backup is no good when you need it!

A simple way to validate a full backup is to restore it to a temporary location. Create a directory for the test validation (such as /mnt/backup-test) and restore the backup to that directory.

Set $JENKINS_HOME to point to this directory, specifying a random HTTP port so you do not collide with the real Jenkins instance:

export JENKINS_HOME=/mnt/backup-test

Now execute the restored Jenkins instance:

java -jar jenkins.war ---httpPort=9999

Summary

  • Making backups is a Jenkins best practice.

  • Backups are critical for disaster recovery.

  • Always set up a backup policy that defines:

    • The configurations and records that need to be saved from the controller

    • How often backups should be taken

    • Where backups should be stored

  • Validate your backups.

    • You should periodically check whether your backups are intact and can be used to meet your recovery objectives.

Going further

Some recommended readings on this subject: