– Disaster Recovery

Jb5: Disaster Recovery Guide

Mon, 01 Jan 0001 00:00:00 +0000

For video tutorials, please click here:

Guide

Backups generated in v5.2.11 are not compatible in v5.3+. You will need to install v5.2.11 in order to complete a Disaster Recovery with those backups. Please follow the steps HERE to install v5.2.11.

JetBackup Config Backup Job

JetBackup 5 will backup it's internal databases only when JB Config is enabled on your configured backup destinations. In order to do so, click on “Export JB Config” on your preferred Destination. Keep your DR destination details and Master Encryption Key to be able to restore from DR.

Job Configuration

The JetBackup Config Backup Job is configured to be executed every night at 01:00 AM Server Time with a total schedule retention of 3. The JetBackup Config Backup Job is an Encrypted Compressed Backup that will dump the entire JetBackup MongoDB and Jetindexd Database in order to allow for Disaster Recovery.

Backup Structure: Compressed
Encrypted: True
Schedule: Daily - 01:00 AM Server Time
Retention: 3 Total Backups

Since the JetBackup Config Backup Job is encrypted, please make sure to keep a copy of your Master Encryption Key in a secure remote location. If the original server is lost, there is no other way to retrieve the Master Encryption Key. However, you may still install a Fresh JetBackup 5 instance and restore accounts using a Read-Only Destination. For more details see How to perform Disaster Recovery without Master Encryption Key?.

Disaster Recovery Best Practices

Consistent Environment Configurations

We recommend ensuring that the new environment you will be restoring to is similar to the source environment as much as possible to help limit incompatibilities between environments that may result in issues during restoration. This may include but is not limited to your Server Hardware (CPU, RAM, Hardrive), Operating System (distribution and version), web server (apache, Nginx, etc.), database (MySQL, PostgreSQL, MongoDB), Configurations and Settings, Addons and/or Extensions.

Perform Routine Recovery Testing

Testing your Disaster Recovery Plan regularly may help ensure that you have an effective and well-designed plan. Simulating Disaster Recovery scenarios may help identify potential points of failure in your current plan which can then be mitigated and/or addressed early on.

Upgrade/Migration Tips

When utilizing Disaster Recovery for Upgrade/Migration purposes, we highly recommend keeping the original server available for rollback until all accounts are restored and systems and features are verified to be functioning properly on the new server.

Recover from Disaster

Getting Started

JetBackup has the ability to restore from a disaster by storing our JetBackup internal database on your preferred destination. Performing a Disaster Recovery will restore the JetBackup Configuration that was backed up as well as give the option to restore all the accounts that have backups located on the configured destinations.

If for any reason you cannot perform a full Disaster Recovery, you can still restore your accounts manually from the destinations by following these instructions: How to perform Disaster Recovery without Master Encryption Key?

Important information before starting Disaster Recovery!

JetBackup will backup it's internal DB only when “Export JB Config” is enabled on your backup destination. In order to do so, click on "Export JB Config" on your preferred Destination. Keep your DR destination details to be able to restore from DR. The JB config backup will be executed every night and will keep a max of 3 snapshots.
Please make sure to keep your Master Encryption Key in a secure remote location.
There must be at least 1 User Package and/or Reseller Package (if applicable to your Panel) on the new server for JetBackup to assign the accounts to during a DR restore. Please ensure that you specify a “Default Package” in JetBackup's Panel Settings to use when the accounts backed up package is not found on the server.
The Disaster Recovery process is not intended for cross-panel migrations. Please take a look at the “Clone Jobs” feature, if you'd like to restore accounts across different control panels.
Restoring your JetBackup Configurations may take some time depending the number of backups indexed by JetBackup at the time the JB Config Export was generated. Please allow enough time for the JB Configurations Restore to complete to begin the Disaster Recovery process.

Initial Installation and Setup

With an entire data loss, your first step will be to install JetBackup on a supported platform that meets our system requirements. See Installation.

After Installation, you will be prompted to choose whether you want to Recover from Disaster, continue with existing data or start with a fresh install in the Disaster Recovery Wizard.

Choose "Recover from Disaster" to continue with disaster recovery.

Now, we need to restore JetBackup's internal database, so JetBackup will have all the information necessary to restore your accounts.

Please have the following information available for the next steps!

Destination Details and Backups Path for the destination where Export JB Config was enabled on your previous configuration. JetBackup will add this destination as a Read-Only Destination to pull your JetBackup Config information stored on that destination.
Master Encryption Key This encryption key is used to decrypt your JetBackup Config Backup and is REQUIRED to restore your previous JB configuration.

Restoring JB Configurations

The next step requires you to select your Destination Type. Any destination configured during Disaster Recovery will be added as a Read-Only Destination.

The "Local" destination is selected in the "Destination Type" dropdown by default. When restoring from a remote destination, the related destination plugin will need to be installed from the JetApps Repository first, and JetBackup will guide you through this process.

To restore JB Config, please select your destination from the "Destination Type" dropdown.

You will see the "Install Now" prompt when selecting a remote destination type.

Select "Install Now" to install the destination plugin.

Then enter the Destination details that you gathered earlier:

Choose the database backup (JetBackup settings) that you want to restore from (usually you will want the most recent backup that was created).

As our database is encrypted, you will need to specify the Master Encryption Key from your previous installation/server.

Enter the Master Encryption Key in the Encryption Key field:

If the key is valid, you will see the "Creating destination and reindexing JB configurations backups" after clicking "OK".

This may take some time to complete, depending on the size of your JetBackup database when the JB Config backup was generated and the number of snapshots.

Please do not exit or refresh the page while the JB Configurations restore is in progress.

Once completed, you will be able to restore your accounts from the destination.

(Optional) You can set "Delete orphan backups older than" option to "Disabled" in your JetBackup Settings to prevent orphan backups from being untentionally removed.

Restoring Accounts

Now we are ready to start restoring your accounts! Simply choose the accounts to restore through the Account Restore Filters and hit the "Finish & login to panel" to continue to your JetBackup Admin Panel while accounts are being restored in the background.

NOTE: JetBackup prioritizes restores by the account type out of the box. Along with the Queue Priority Groups setting, this ensures that parent accounts gets restored before their corresponding child accounts. For more Information regarding JetBackup 5's priority systems, please CLICK HERE.

Your accounts are now in the accounts restoration queue and may take some time to complete while the selected accounts are being restored. You can click "Finish and login to Panel", then visit the queue to check the status.

NOTE: The Disaster Recovery Wizard only lists backups from accounts that do not exist on the current server and were previously active on the source server when JetBackup generated the JB Config Backup.

Encrypted Backups

In order to restore an Encrypted Backup, it is REQUIRED to have both the JB Config and your Master Encryption Key. If the Master Encryption Key is missing, then the associated Encrypted Backups are PERMANENTLY lost in the event of a Disaster Recovery.

If a user has allowed Only Encrypted Backups, then their Encryption Key is stored locally on the server and will be tied to the JB Config. The user and admin will both be able to perform restores with the Encypted Backups.

The user can choose to store their key remotely where they will be shown the key once and be responisble for storing it in a safe place. The admin will not be able to perform any action on the Encrypted Backups. The user will have to perform restores on their end or provide the admin with the key so that they may perform it on the user's behalf.

NOTE: The JetBackup Config backup job runs daily at 1:00 AM server time. If performing a test, it MUST be after the JB Config job runs.

Post Disaster Recovery Checks

Re-Enable Export JB Config - Don't forget to re-enable "Export JB Config" on your preferred destinations! By default, all destinations will have the Export JB Config setting disabled post Disaster Recovery. You will need to re-enable them under Destination Settings.

Re-Enable Backup Jobs - Don't forget to re-enable your backup jobs! By default, backup jobs that are imported from your JB Config will be disabled. You will need to re-enable them under Backup Jobs.

Disabling Backup Jobs and Destinations on Original Server - In cases where you are migrating from a source server to a new server, please either make sure you uninstall JetBackup OR disable all Destinations and Backup Jobs on the source server to prevent potential conflicts with multiple instances of the same backup job running on 2 (or more) servers concurrently.

Jb5: Manually Restore from Destination

Mon, 01 Jan 0001 00:00:00 +0000

How to perform Disaster Recovery without Master Encryption Key?

When you have a disaster on your server and you have lost your data, if you didn't enable the JB config backup feature or have lost the Master Encryption Key, you can still restore your accounts from your backup destination. In order to restore it, you will have to go through a few steps:

Set up your server.
Install your supported control panel.
Install JetBackup. For more information visit Installing the JetApps Repository

After the JetBackup installation is over, go to the JetBackup Dashboard and you will see the Disaster Recovery page. Click on New Installation.

NOTE: When performing Disaster Recovery without the Master Encryption Key, you will be limited to exclusively restoring from read-only destinations. All previous JetBackup Configurations (Destinations, Backup Jobs, etc.) will not be restored and JetBackup will not be able to continue backups from where it left off. This means that JetBackup will have to start generating new backups from scratch. To see how to perform a standard Disaster Recovery, visit Recover from Disaster.

Go to General Setting and change the [Days to keep orphan backups]{.title-ref} value to 0, so JetBackup will not auto remove these files.
Create New Destination, point the Backup Directory path to the same location where your backups are stored, and Enable Read-Only Mode. Note: If you do not want to add the destination as Read-Only please see the KnowledgeBase Article: Using the same backup destination for multiple servers
The Destination will automatically reindex after creation.

If the path is set correctly this should index all your snapshots.

Go to JetBackup -> Restore&Download and select 'orphan accounts' option on the Accounts Type. Select the accounts and restore.

Jb5: Disaster Recovery BMR

Mon, 01 Jan 0001 00:00:00 +0000

The Bare-Metal Restore (BMR) feature is available in JetBackup 5 v5.3.0

The Bare Metal Restore feature for JetBackup 5 allows for a backup job to be created that contains operating system, panel, panel accounts, and server files/configuration data. This information is stored in an ISO format that can be used for a server-level restore.

JetBackup's BMR Guide utilizes RHEL's Relax and Recover (ReaR) to perform a complete system restore. Please take note of the restrictions and limitations below.

ReaR Known Limitations:

ReaR supports recovery to new hardware, however the hardware of the original server may not be compatible with the hardware in the new server. Please check and confirm hardware compatibility before performing the restore.
Recovery to new hardware may also require manual adjustments to:
- modify disklayout.conf (adjust amount of disks, controllers, and any other custom configs)
- edit partition/LV sizes in the case of restoring to a smaller storage.
- configure network interfaces

Note: As of v5.3.17+, supported BMR destinations are limited to the following destinations: S3-Compatible, SFTP, and FTP.

To prevent license file errors, please ensure there is an active control panel license BEFORE starting the recovery process. JetBackup 5 cannot restore accounts if the associated control panel is not properly licensed.

The JetBackup 5 Disaster Recovery BMR Process is split into 2 Separate Backup Jobs:

A Disaster Recovery Backup Job that backs up data required for the server-level restore (OS, disks, Panel, Server files, and configuration) and generates a bootable ISO to execute the server-level restoration.
A Standard Full Account Backup Job that stores all data at the account level which is then restored after the server-level restore is complete.

Here are the steps to take when performing a Disaster Recovery BMR:

Create a Server Backup (BMR) Backup Job and Full Accounts Backup Job - Ensure that the backup jobs are successfully completed before proceeding to the next step.

We suggest adding these recommended files and folders to exclude

By default, the following files/directories are excluded from the BMR backup job:

/proc
/sys
/home/virtfs/\*
/home/{linux_users}
/usr/share/cagefs-skeleton/\*
JetBackup 5 Workspace directory 
JetBackup 5 /lib/mongod/* directory 
JetBackup 5 tmp folder

Download the Bootable Recovery ISO

You can download the ISO from the Restore & Download page in JetBackup 5
Or directly via your configured Backup Destination /PATH/TO/DESTINATION/jetbackup_4_1_{DR_BACKUP_JOB_ID}}/snap.1

Boot up the ISO and run the recovery

Initial Boot Configuration

On initial boot, the new server will still have the original IPs that were set on the source server.

To prevent any conflicts, update your IP and network configuration. You are able to update the IPs by editing your server's network configuration with the following command:

ifconfig \<interface_name\> \<ip_address\> netmask \<netmask_address\>

You can verify your changes by executing ifconfig

Execute chroot /mnt/local and make any necessary IP changes to /etc/sysconfig/network-scripts.

Once you have completed all of the IP/network configurations run rear recover in the CLI. This will automatically configure the recovery process to fetch server level data, operating system, disks, server and Panel files and configurations, etc. When finished, reboot the server.

We recommend running a system update and additional post-BMR checks/updates depending on your server environment (OS/Panel).

You have now completed the server-level restore. When JetBackup 5 is launched, it will begin the Disaster Recovery process to restore accounts and JetBackup configurations.

Common BMR FAQs and Errors

This section lists the common questions and errors you may encounter during the BMR process.

libsystemd-shared-252.so => not found Error:

The error indicates that the BMR process is attempting to load the shared library (libsystemd-shared-252.so), however, it cannot be found.

To resolve this issue, verify which package provides the missing library and ensure it is installed:

CentOS / AlmaLinux / RHEL / CloudLinux:

yum provides */libsystemd-shared-252.so

Debian / Ubuntu:

dpkg -S libsystemd-shared-252.so

If the file isn’t found, you may need to reinstall the appropriate version of the systemd package:

CentOS / AlmaLinux / RHEL / CloudLinux:

yum reinstall systemd

Debian / Ubuntu:

apt update
apt install --reinstall systemd

grub2-mkstandalone: error: /usr/lib/grub/x86_64-efi/modinfo.sh doesn't exist / type: grub-probe: not found / no /usr/*/grub*/x86_64-efi/moddep.lst file / ERROR: Failed to make bootable EFI image of GRUB2 (error during grub2-mkstandalone of grub2-mkstandalone may fail to make a bootable EFI image of GRUB2 (no /usr/*/grub*/x86_64-efi/moddep.lst file)

This error indicate that your system is missing essential GRUB2 EFI components required to build a bootable standalone EFI image using grub2-mkstandalone.

To resolve, ensure the required GRUB packages are installed on your system.

RHEL / AlmaLinux / Rocky / CentOS:

yum install grub2-efi-x64-modules grub2-tools

Debian / Ubuntu:

apt update
apt install grub-efi-amd64 grub-efi-amd64-bin grub2-common

Broken symlink '/boot/grub2-bkp/grubenv.rpmnew' in recovery system because 'readlink' cannot determine its link target

This error is caused by the server admin creating a “backup” of their original grub configuration, however those symlinks are now invalid.

This can be resolved by adding “Directories and Files to Exclude” list to the BMR backup job or clearing up the broken symlinks.

Adjusting ReaR Config options

JetBackup 5 allows you to adjust the configuration options used during Rescue ISO generation by creating a custom configuration file at:

/usr/local/jetapps/etc/jetbackup5/rear.default.conf

This file overrides the default settings used by the ReaR tool during the ISO creation process.

Example:

To disable serial console output in the generated ISO, add the following line to the config file:

USE_SERIAL_CONSOLE="no"