Break Glass #07: Hardened Repository Lockout: Restoring Access to a Disconnected Veeam Linux Repository

Why This Happens

The Hardened Linux Repository uses single-use credentials. When you add the repository to VBR, you supply a Linux account username and password one time. VBR uses those credentials to deploy the Veeam Data Mover service on the Linux host, then stops storing them. The Linux host then has SSH disabled, the account removed from the sudo group, and remote access locked down. After that, the Veeam transport service on the Linux host runs under that account continuously, and VBR communicates with it directly without needing credentials again under normal operation.

The lockout happens when that trust relationship breaks. A VBR server rebuild is the most common trigger. The new VBR has no record of the prior connection and cannot reconnect. A component upgrade triggered from the VBR console requires new single-use credentials to redeploy the updated transport service, and if SSH is disabled on the Linux host and credentials are not prepared in advance, the upgrade fails and the connection breaks. Someone deleting or modifying the service account during a security audit also severs the connection because the transport service can no longer start.

Note: v13 introduced certificate-based authentication for hardened repositories deployed via the Veeam Infrastructure Appliance (VIA). If your hardened repo was deployed using the VHR ISO and added to VBR through the Infrastructure Appliance, it uses certificate-based authentication rather than single-use credentials. The VIA path does not require the SSH and credential dance described below. This article covers the BYO Linux path with single-use credentials, which is the most common deployment.

The backup data and immutability flags on disk are never affected by a connectivity failure. The files are there. Only the VBR-to-repo connection needs to be restored.

Triage

1. 1In the VBR console, go to Backup Infrastructure, Managed Servers. Find the Linux host. Right-click and select Connection Test. Note the exact error. "Unable to deploy the Veeam Deployer service due to SSH credentials not specified" means the single-use credential process needs to run. A transport service error means the service account issue is on the Linux side.
2. 2Confirm you can reach the Linux host. If SSH was disabled as part of the hardening setup, you need physical console access or out-of-band management (IPMI, iDRAC, iLO, or hypervisor console) to re-enable it. Confirm you have a path to the host before proceeding.
3. 3Verify the Veeam service account still exists on the Linux host:
   id veeamrepo
   Replace veeamrepo with the account name used during initial setup. If the account does not exist, the service account was deleted. Go to Recovery Path B.
4. 4Check whether the veeamtransport service is running:
   sudo systemctl status veeamtransport
   If it is stopped, that confirms a service account or installation problem rather than a pure credentials issue.
5. 5Verify the backup files are intact. Navigate to the backup directory and confirm VBK and VIB files are present. Check immutability flags on a sample of files:
   lsattr /path/to/backup/directory/ | head -20
   Files with the 'i' attribute are immutable. If the files are present and flagged, your backup data is safe.

Recovery Path A: Re-Establishing the Connection

The Linux host is healthy. VBR lost the connection record. The fix is re-running the single-use credential setup with SSH temporarily re-enabled.

1. 1On the Linux host, temporarily re-enable SSH and add the service account back to the sudo group. Both are required because VBR needs to authenticate and deploy components. On Debian and Ubuntu, the group is called sudo. On RHEL, Rocky, and other Red Hat derivatives, the group is called wheel:
   # Debian / Ubuntu sudo systemctl start sshd sudo usermod -aG sudo veeamrepo # RHEL / Rocky sudo systemctl start sshd sudo usermod -aG wheel veeamrepo
2. 2In the VBR console, go to Backup Infrastructure, Managed Servers. Right-click the Linux server and select Properties. Work through the wizard to the SSH Connection step.
3. 3At the Credentials step, click Add and select Single-use credentials for hardened repository. Enter the service account username and current password. Do not use the standard Linux account credential type. It is rejected with an error about hardened repository roles requiring single-use credentials.
4. 4Complete the wizard. VBR re-authenticates, redeploys the transport components, and restores the connection. Confirm the connection test passes before closing the wizard.
5. 5Immediately re-apply the hardening steps on the Linux host. Do not wait. The window where SSH is enabled is a security exposure:
   # Debian / Ubuntu sudo deluser veeamrepo sudo sudo systemctl stop sshd sudo systemctl disable sshd sudo ufw deny ssh # RHEL / Rocky sudo gpasswd -d veeamrepo wheel sudo systemctl stop sshd sudo systemctl disable sshd sudo firewall-cmd --remove-service=ssh --permanent sudo firewall-cmd --reload
   Confirm the VBR console shows the repository as accessible before you close SSH.
6. 6Run a backup job targeting the repository. Confirm the job completes and backup chains are visible under the repository in VBR.

Recovery Path B: Service Account Deleted or Transport Service Not Starting

1. 1Re-create the service account with the same username used during original setup:
   sudo useradd -m -s /bin/bash veeamrepo sudo passwd veeamrepo
2. 2Follow Recovery Path A steps 1 through 5 to re-establish the VBR connection.
3. 3If you are using a different account name than the original, the backup files on disk need ownership updated. Changing the account that VBR uses to connect to the hardened repo requires removing immutability flags from the backup files, running chown to reassign ownership to the new account, and re-applying the immutability flags. The full procedure is in Veeam KB4348. Do not attempt this without reading that KB first. Running chown on immutable files fails silently and removing the immutability flags without a plan to re-apply them is a security risk.
4. 4After the VBR connection is restored, confirm the transport service starts and stays running:
   sudo systemctl restart veeamtransport sudo systemctl status veeamtransport
5. 5Run a test restore from the repository to confirm the backup data is readable end to end.

Recovery Path C: VHR Appliance (Managed Hardened Repository) Lockout

The Veeam Managed Hardened Repository appliance deploys Rocky Linux with the DISA STIG security profile. This profile uses faillock to lock user accounts after 3 incorrect password attempts. Once faillock triggers on the veeamsvc or vhradmin account, you cannot log in. Root is disabled. SSH is disabled by default. Console access is your only path in.

1. 1You need physical or hypervisor console access to the appliance. Disable all backup jobs targeting this repository before proceeding.
2. 2Reboot the appliance with the console visible. When the GRUB boot menu appears, press e to edit the boot entry. Find the line beginning with linux. Append the following to the end of that line:
   rd.lvm.lv=systemvol/var rd.lvm.lv=systemvol/varlog init=/bin/sh
   Press Ctrl+X to boot with those parameters.
3. 3At the shell prompt, remount the filesystem read-write:
   mount -o remount,rw /
4. 4Reset the faillock state for the locked account:
   faillock --user veeamsvc --reset
   If the locked account is vhradmin instead, substitute that name. Reset both if you are not sure which one triggered the lockout.
5. 5Reboot the appliance. The account should be accessible again. Use the Veeam Host Management web UI to re-establish the VBR connection through Backup Infrastructure. The full procedure for the VHR appliance is documented in Veeam KB4663.

Gotchas

Prevention Checklist

• Document the service account name and password in the shared team vault on the day you set up the hardened repo. It must be retrievable by someone who has never worked on this system before.
• Write a formal runbook for component upgrades that includes the SSH enable, credential supply, and re-hardening steps. Treat every VBR update that touches Linux host components as a planned maintenance event for the hardened repo.
• Explicitly tag the service account in Linux user management to exclude it from any account cleanup, deprovisioning, or security hardening scripts.
• Keep out-of-band access to the hardened repo host documented and tested. If SSH is disabled and you have no console path, you have no recovery path without physical access to the machine.
• Test the full reconnection procedure in a lab before you need it in production. The single-use credential workflow is straightforward once you have done it once. The first time under pressure is not when you want to figure it out.