Veeam and Linux Hardened Repository Troubleshooting

Veeam and Linux Hardened Repository Troubleshooting

Veeam v13 Series | Component: Hardened Repository | Audience: Infrastructure Engineers, Backup Administrators, Security Engineers

1. How the Hardened Repository Actually Works

Before diving into break-fix, you need to understand what the hardened repository is doing under the hood. The hardened repository is a Linux server with an XFS file system where Veeam sets the Linux immutable attribute (chattr +i) on backup files. That is the entire mechanism. It is not a proprietary lock. It is the standard Linux extended file attribute that prevents any modification or deletion of the file, even by root, until the attribute is removed.

For each backup file, VBR creates a .veeam.N.lock file that contains immutability period metadata. The xattr attribute with the immutable time period is also set on each backup file. The veeamimmureposvc service monitors these files and manages the immutability lifecycle. The veeamtransport service handles data transfer on port 6162 (management) and ports 2500-3300 (data transfer).

The single-use credential model means VBR does not store the SSH credentials for the repository server. They are used once during initial setup to deploy the Veeam Data Mover components, and then communication happens through the deployed services using certificates. After setup, SSH can be disabled entirely.

Almost every hardened repository troubleshooting scenario comes down to one of these: XFS configuration, file permissions, credential or certificate state, time synchronization, or service lifecycle.

2. "File System Behind the Provided Path Is Not XFS"

This is the most common error during initial repository creation. VBR queries the file system type and gets back something other than XFS, or it cannot determine the file system type at all.

Root Causes

The volume is not mounted. This is the number one cause. The XFS partition exists and is formatted correctly, but it is not mounted when VBR queries it. VBR can browse to the directory path (because the mount point directory exists), but the underlying file system is whatever the parent mount is (usually ext4 on the root volume). Verify the mount is active:

If this does not show XFS as the type, the volume is not mounted. Check /etc/fstab and run mount -a.

XFS formatted without reflink and CRC. Veeam requires XFS with reflink enabled for fast cloning. The correct format command is:

If the volume was formatted with default XFS options, reflink may not be enabled (it depends on the distribution and xfsprogs version). Verify:

If it says reflink=0, you need to reformat. If there is existing data, back it up first.

PATH issue for the Veeam service user. VBR runs file system detection commands through the service user's shell. If the user's .bashrc does not include /sbin and /usr/sbin in the PATH, the detection commands may fail silently. Add this to the Veeam user's .bashrc:

3. Permission Denied: Failed to Create Directory

The error Agent failed to process method {FileSystem.DirectoryCreate} means the Veeam Data Mover service cannot create subdirectories in the repository path. This is a file ownership or permission problem.

Fix

The Veeam service user must own the repository directory and have rwx permissions on it. Replace locveeam with your actual service user name:

This error also reappears after VBR component upgrades. The upgrade process can alter permissions on the repository directory. If backups start failing after an upgrade with this error, re-run the ownership and permission commands above.

4. Single-Use Credential Lockouts

Single-use credentials are used once during the managed server setup wizard and are not stored in the VBR configuration database. After setup, Veeam communicates with the repository using certificates deployed during the initial connection. This is secure but creates a specific failure mode: if the certificate trust breaks, you need SSH access to re-run the wizard.

Common Scenarios

Need to edit the managed server properties but SSH is disabled. You cannot get past the credentials step in the properties wizard without a valid SSH connection. To change even the description field, you need to temporarily re-enable SSH, provide the single-use credentials, complete the wizard, then disable SSH again.

Repository went offline after VBR upgrade. Re-save the single-use credentials. Navigate to Backup Infrastructure > Managed Servers > Linux, right-click the repository server, select Properties, and walk through the wizard providing the single-use credentials again. This re-establishes the certificate trust. Multiple community reports confirm this resolves the post-upgrade connectivity issue.

The service user was deleted. Do not delete the user account that was used as the single-use credential. The veeamtransport service runs in the context of this user. If the user is deleted, the service cannot start after a reboot. Re-create the user with the same name and UID to restore service functionality.

5. Repository Time Shift Detected

The warning "repository time shift detected, immutability flag cannot be set" means the immutability service detected a clock discrepancy that exceeds its tolerance threshold. Backups complete but the immutability attribute is not applied to the backup files. This is a security feature introduced in v12.1 to prevent immutability manipulation through clock tampering.

How the Detection Works

When the immutable service starts, it creates a timeLog file in /etc/veeam/immureposvc/. Every 10 minutes, it writes the current UTC time and hardware clock time to this file. It calculates the drift between updates and accumulates it in a moveTime parameter. If the accumulated drift exceeds 24 hours, the service creates an immutable retainLock file that blocks all future immutability assignments.

Common Triggers

Extended server shutdown (the server was powered off over a weekend). Stopping the VeeamTransport service for maintenance. Intentional clock changes by an administrator. NTP not configured, causing gradual drift. Re-adding a previously used hardened repository that already has a timeLog file.

Fix for Customer-Built Repositories

1. 1Log into the hardened repository console (physical access or re-enable SSH temporarily).
2. 2Remove the immutable attribute from the lock file:
   sudo chattr -i /etc/veeam/immureposvc/retainLock
3. 3Delete the lock file:
   sudo rm /etc/veeam/immureposvc/retainLock
4. 4Restart the VeeamTransport service:
   sudo systemctl restart veeamtransport
5. 5Run the backup job. A new timeLog is created and immutability resumes.

For VHR appliance-based repositories (built from the Veeam Hardened Repository ISO), use the "Reset time lock" option in the Configurator Main Menu instead of manual file manipulation.

6. veeamtransport Service Will Not Start After Reboot

The veeamtransport service is the core data mover on the repository server. If it does not start, VBR cannot reach the repository at all.

Diagnostic

Common Causes

Service user was deleted. As described in Section 4, the transport service runs under the Veeam service user. If that user no longer exists, the service fails on startup. Re-create the user.

Repository volume not mounted at boot. If the XFS volume mount fails during boot (bad fstab entry, disk not detected), the transport service may start but VBR will report the repository as offline or show path errors. Check mount | grep xfs and verify the repository path is accessible.

Port conflict on 6162. Another service is binding to TCP 6162 before veeamtransport starts. Check with ss -tlnp | grep 6162.

7. SSH Key Mismatches After OS Patching

When VBR first connects to the Linux server, it saves a fingerprint of the SSH host key in its configuration database. If the SSH host keys change (OS reinstall, sshd reconfiguration, certain security patches), VBR detects the mismatch and refuses to connect to prevent a man-in-the-middle attack.

Fix

In the VBR console, go to Backup Infrastructure > Managed Servers > Linux, right-click the server, select Properties, and walk through the wizard. VBR will show a warning about the changed fingerprint. Accept the new fingerprint to update the stored value. This requires SSH to be enabled temporarily.

If you are patching the OS regularly (which you should be, even on a hardened repository), plan for this. After major kernel or OpenSSH updates, verify VBR connectivity and re-accept the fingerprint if needed.

8. Upgrading VBR Components on a Locked-Down Repository

When you upgrade VBR, the Veeam components on the Linux repository also need to be updated. This is the most operationally disruptive event for a hardened repository because the upgrade process requires temporary sudo access and SSH connectivity.

Upgrade Procedure

1. 1Re-enable SSH on the hardened repository (physical console access required if SSH is fully disabled).
2. 2Temporarily add the Veeam service user back to the sudo group:
   sudo usermod -aG sudo locveeam
3. 3In VBR, go to Managed Servers > Linux, right-click the repository server, select Properties, and re-enter the single-use credentials. VBR will detect the version mismatch and upgrade the components.
4. 4After the upgrade completes, remove the user from sudo:
   sudo deluser locveeam sudo
5. 5Re-verify ownership and permissions on the repository directory:
   sudo chown -R locveeam:locveeam /mnt/veeamrepo
   sudo chmod 700 /mnt/veeamrepo
6. 6Disable SSH again.
7. 7Run a test backup to confirm the repository is functioning correctly with the updated components.

This is a manual process. There is no way to automate VBR component upgrades on a properly locked-down hardened repository because the entire security model is built around preventing remote privileged access.

9. SGID Bit Conflicts with Immutability Verification

KB4631 documents a specific issue where the SGID (Set Group ID) bit on the repository directory causes files created within it to inherit the directory's group ownership instead of the creating user's group. VBR's immutability verification checks that .veeam.lock files are owned by root:root. If the SGID bit causes them to inherit a different group, the verification fails and immutability cannot be updated or restore points cannot be removed after expiration.

Diagnostic

Fix

Then fix existing lock file ownership if needed. This is a subtle issue that typically only surfaces when the directory was created with non-default permissions or inherited from a parent directory with SGID set.

10. Immutability and Backup Chain Compatibility

Not every backup chain type is compatible with immutability. Because immutable files cannot be modified or deleted during the immutability period, the backup chain must be one that only creates new files without changing existing ones.

VBR will block you from configuring incompatible chain types when targeting an immutable repository. But if you change the repository assignment on an existing job, verify the chain type matches before running the job.

The immutability period starts from the moment the last restore point in the active backup chain is created, not from the creation time of each individual file. The immutability period is extended for existing backup files each time a new restore point is added to the chain. This means backup files in an active chain remain immutable as long as the chain is active and new restore points are being added.

11. Disaster Recovery: Reconnecting a Hardened Repository to a New VBR Server

If the VBR server is destroyed and you need to add the existing hardened repository to a brand new VBR installation, the process is straightforward but has a credential nuance.

1. 1Re-enable SSH on the hardened repository (physical console access).
2. 2Temporarily add the Veeam service user to the sudo group.
3. 3In the new VBR console, add the repository server as a managed Linux server using the single-use credentials option. VBR deploys fresh Veeam components.
4. 4Add the repository path. VBR detects the existing backup files and imports them.
5. 5Remove sudo, disable SSH, verify backups.

A community-reported nuance: when reconnecting to a new VBR server, the single-use credential option may fail with an authentication error even though the credentials are correct. In this case, use the standard "Linux account" credential option instead (not single-use) to establish the initial connection. After the managed server is added and components are deployed, you can re-secure the environment normally. The backup data remains intact and immutable throughout this process.