Break Glass #02: Lost Encryption Password
Break Glass // Scenario 02
Someone needs a restore. The backup is encrypted. Nobody can find the password. The engineer who set it left six months ago. Your backups exist. You cannot read them. Here is what you do next and what you do when there is nothing left to do.
Why This Happens
Backup encryption passwords get lost in predictable ways. An engineer sets encryption on a job, stores the password in a personal vault, and then leaves the company. The shared vault has an entry but it was updated during a rotation and the old backups still need the password that was current when they were written. A hint that made perfect sense 18 months ago is completely opaque to everyone in the room today.
In MSP environments it is common for encryption to be enabled job by job over time, with different passwords per client, managed by different technicians. There is no central audit. The first time anyone notices the gap is when a restore fails and the client is waiting.
There is also the VBR rebuild scenario. When a VBR server is rebuilt from an unencrypted configuration backup, the encryption key hashes are not preserved. Jobs that were working perfectly before the rebuild now appear locked. The data is fine. VBR just lost its record of the key.
Veeam gives you two legitimate paths out of this. First path: you find the password and enter it. Second path: you have Veeam Backup Enterprise Manager deployed, Loss Protection was enabled before this backup was created, and the VBR server was connected to VBEM at the time. Both conditions on that second path have to be true, not just one.
If neither path is available, I am going to tell you that clearly and not make you read twelve more steps to find out. The data is not recoverable. AES-256 with no key has no back door.
Triage
- 1Open the VBR console. Navigate to Home and find the backup in question. Try to initiate any restore operation -- right-click a restore point and start a file-level or full VM restore. If VBR prompts you for a password, the encryption keys are not in the local configuration database. You need the password.
- 2Check whether the backup keys might already be in the database from a different angle. From the main menu, select Credentials and Passwords, then Encryption Passwords. Find the entry for this job. The hint is visible here. This is your first lead -- even a bad hint is something to work from.
- 3Before anything else, exhaust the password search. Check the team password manager, any personal vaults from former employees (request IT recovery of those accounts), email history if passwords were communicated by email, physical DR documentation, and any runbooks stored offline. Do not skip this step in a rush to try technical recovery paths.
- 4Determine whether Veeam Backup Enterprise Manager is deployed in your environment. Open a browser and try to reach the VBEM web interface -- typically at https://your-vbem-server/. If VBEM exists, log in as an administrator.
- 5In VBEM, go to Configuration, then Key Management. Confirm two things: Loss Protection is shown as enabled, and the VBR server that owns this backup is listed and connected. Both must be true. If Loss Protection is disabled, or if this VBR server was never connected to VBEM, Path B is not available to you.
Decision Point
Found the password: go to Recovery Path A. VBEM is deployed, Loss Protection was enabled before the backup was created, and the VBR server was connected to VBEM: go to Recovery Path B. Neither condition is met: go to The Hard Stop at the end of this article.
Recovery Path A -- You Have the Password
- 1In the VBR console, go to Home. Find the backup. If it appears under Backups as encrypted or is prompting for a password, right-click it and select Decrypt backup. Alternatively, click Specify Password on the ribbon with the backup selected.
- 2Enter the password. If the chain has had multiple password rotations, use the most recent password. The current password unlocks all restore points in the chain, including ones created under older passwords. You do not need a history of every password used.
- 3If the current password does not work, try the previous password. This can happen if the backup chain has not had a new full backup since the last rotation -- older increments may still require the password that was active when they were written.
- 4Once the password is accepted, proceed with the restore normally. Veeam decrypts the restore point metadata and you can select specific VMs and restore points.
- 5After completing the restore, fix the record. From the main menu, select Credentials and Passwords, then Encryption Passwords. Select the entry for this job and click Edit. Update the hint to something unambiguous. Update the stored password in the team vault. Confirm that at minimum two people know where to find it.
- 6Run a manual configuration backup with encryption enabled immediately. This captures the current encryption key hashes so a future VBR rebuild does not lose them again.
Recovery Path B -- Enterprise Manager Loss Protection
Three conditions must all be true for this path to work. VBEM is deployed. Loss Protection was enabled in VBEM before the backup was created. The VBR server was connected to VBEM at the time the backup ran. If any one of the three is false, this path is closed.
- 1In the VBR console, go to Home. Find the encrypted backup under Backups. Right-click it and select Decrypt backup, or select it and click Specify Password on the ribbon.
- 2In the Specify Password window, click the link that says "I have lost the password." This launches the Encryption Key Restore wizard on the VBR server.
- 3At the Request step of the wizard, VBR generates a request message -- a block of encrypted text specific to this backup and this server. Copy it exactly to the clipboard or save it to a text file. Do not modify or truncate it.
- 4Open a browser and log into the VBEM web interface as an administrator.
- 5In VBEM, go to Configuration, then Key Management. Click Password Recovery. The Password Recovery wizard opens. Paste the request text from step 3.
- 6VBEM searches its database for a matching public backup server key. If it finds one, it decrypts the storage keys using the VBEM private key and generates a response -- another block of encrypted text. Copy the response exactly.
- 7Return to the VBR console. Back in the Encryption Key Restore wizard, paste the response at the Response step. Click Next and complete the wizard. Veeam processes the response, retrieves the storage keys, and unlocks the backup.
- 8Verify the backup is accessible. Right-click a restore point and confirm you can browse it without a password prompt.
- 9Update the encryption password record. From the main menu, select Credentials and Passwords, then Encryption Passwords. Edit the entry for this job. If the original password is unrecoverable, run a new active full backup to associate a known, documented password with the chain going forward.
The Hard Stop
No Recovery Path Available
No password and no pre-configured VBEM Loss Protection means the data in those backup files is not accessible. This is not a Veeam limitation -- this is AES-256 working exactly as intended. There is no support tool, no Veeam Engineering escalation, and no undocumented method that recovers backup data without the encryption key. The data exists on disk. It cannot be read. Your options from here are: find another copy of the data (replication target, tape, cloud copy), identify what the RPO impact is, and restore from that source. If no other copy exists, the data is gone. Document what happened and use it to drive the prevention work that comes next.
Gotchas
VBEM Loss Protection Must Exist Before the Backup Is Created
Enabling Loss Protection in VBEM today does not protect any backup that was created before you enabled it. The VBEM public key gets embedded in backup files at write time. If VBEM was not configured when the backup ran, that backup cannot use the VBEM recovery path -- ever. This is the most common reason environments with VBEM still lose access to older backups. You gain Loss Protection for everything going forward. Older chains remain locked until you rotate and run a new full.
Unencrypted Config Backup Loses All Key Hashes
VBR explicitly warns you about this in the configuration backup settings dialog. If your config backup is not encrypted, encryption key hashes are not included. After a VBR rebuild from an unencrypted config backup, every encrypted job requires manual password re-entry before the first restore can run. There is also a harder consequence: if the Password Manager has at least one encryption password entry and you have config backup encryption disabled, Veeam disables the configuration backup entirely. This is not a soft warning. It stops running. Enable config backup encryption.
Current Password Unlocks the Entire Chain
When a password rotation happens, Veeam re-encrypts the metadata using the new password going forward. But you only need the current (most recent) password to decrypt any restore point in the chain -- including those written under older passwords. You do not need a log of every password ever used on a job. What this means practically: if you find any version of the password that post-dates the last full backup, try it. It may unlock the entire chain.
The VBEM Response Is Specific to One Server
The response VBEM generates is encrypted for the specific VBR server that created the request. You cannot generate the request on VBR Server A and apply the response to VBR Server B. If you rebuilt your VBR server and are trying to recover backups from before the rebuild, the request must come from the rebuilt server after the original configuration has been restored onto it. The request-response pair is also effectively single-use -- do not close the Encryption Key Restore wizard before you complete the exchange, or you will need to generate a new request.
Loss Protection Requires a Paid License
Password Loss Protection via Enterprise Manager is not available on free or community editions of Veeam. It requires a paid Veeam Data Platform license. If your environment is running on free licenses, this path does not exist regardless of whether VBEM is deployed. Check your license tier in the VBR console before assuming Loss Protection is available to you.
The Password Verify Function in VBR
From the main menu, go to Credentials and Passwords, then Encryption Passwords. Select any password entry and click Verify. Enter the password you have stored for that entry. VBR tells you immediately whether it matches. This is the right way to confirm your stored passwords are correct before you need them under pressure. Build this into a quarterly maintenance task across every VBR server in your estate.
Prevention Checklist
- Deploy Veeam Backup Enterprise Manager and connect every VBR server to it. Enable Loss Protection in Configuration, Key Management before creating any encrypted backup jobs. Do it in that order.
- Enable configuration backup encryption on every VBR server. This is what keeps encryption key hashes alive through a rebuild. The warning in the UI about what gets excluded without it is accurate.
- Run the password Verify function in Credentials and Passwords, Encryption Passwords quarterly. Confirm that what is stored in your vault actually opens the entry in VBR. Document the results.
- Use a consistent, organization-wide encryption password where your compliance requirements allow it, rather than unique passwords per job or per client. Fewer passwords mean fewer opportunities for loss.
- After any staff departure, identify and rotate every encryption password that person created or managed. Do it while you still can -- before you need a restore from those backups.
- Run a test restore from each encrypted job at least quarterly. A restore that requires a password proves the password is correct. A restore that works silently proves the key is in the database. Either way you learn something useful.
- If you rotate an encryption password, run a manual configuration backup with encryption enabled immediately afterward. This captures the updated key hash before anything else can go wrong.
Break Glass Recap
- Two recovery paths: you have the password, or VBEM Loss Protection was pre-configured
- No password plus no VBEM Loss Protection is a hard stop -- data is not recoverable
- VBEM Loss Protection must be enabled before the backup is created -- it does not backfill
- Key restore request path: right-click encrypted backup, Decrypt backup, click "I have lost the password"
- VBEM response is VBR-server-specific -- cannot be applied to a different host
- Current password unlocks the entire chain including older restore points
- Unencrypted config backup loses all encryption key hashes -- fix this now
- Password Verify: Main Menu > Credentials and Passwords > Encryption Passwords > Verify
- Paid license required for VBEM Loss Protection
- After a VBR rebuild from unencrypted config backup, all encrypted jobs need manual password re-entry