SQL HADR Triage

Always On availability groups · DR-exercise failure triage & T-SQL library

Runs entirely in this browser. No network calls, nothing leaves this page. Works on restricted networks and opened straight from a local file or USB. Queries are read-only diagnostics unless marked otherwise; anything destructive is flagged.

Triage Path

Mass audit. Everything that doesn't fail over

One read-only script, run on the new primary after failover (or on the DR replica before the exercise). Each section returns a labeled result set. Empty sections are good news. Compare CONFIG / LINKED_SERVER / CREDENTIAL sections against the same run on prod.

/* =====================================================
   HADR DR AUDIT. Run on the NEW PRIMARY
   Read-only. Each section returns a labeled result set.
   ===================================================== */
SET NOCOUNT ON;

/* 1 · Replica + database sync state */
SELECT 'AG_STATE' AS chk, ag.name AS ag,
       ar.replica_server_name AS replica, ars.role_desc AS role_now,
       DB_NAME(drs.database_id) AS db,
       drs.synchronization_state_desc AS sync_state,
       drs.suspend_reason_desc AS suspend_reason,
       drs.log_send_queue_size AS send_kb,
       drs.redo_queue_size AS redo_kb
FROM sys.dm_hadr_database_replica_states drs
JOIN sys.availability_replicas ar ON drs.replica_id = ar.replica_id
JOIN sys.availability_groups ag ON drs.group_id = ag.group_id
JOIN sys.dm_hadr_availability_replica_states ars
     ON drs.replica_id = ars.replica_id;

/* 2 · Orphaned users in every writable user db */
DECLARE @sql nvarchar(max) = N'';
SELECT @sql += N'USE ' + QUOTENAME(name) + N';
SELECT ''ORPHANED_USER'' AS chk, DB_NAME() AS db,
       dp.name AS db_user, dp.type_desc
FROM sys.database_principals dp
LEFT JOIN sys.server_principals sp ON dp.sid = sp.sid
WHERE sp.sid IS NULL AND dp.type IN (''S'',''U'',''G'')
  AND dp.name NOT IN
      (''guest'',''dbo'',''sys'',''INFORMATION_SCHEMA'');
'
FROM sys.databases
WHERE database_id > 4 AND state_desc = 'ONLINE'
  AND DATABASEPROPERTYEX(name,'Updateability') = 'READ_WRITE';
EXEC sys.sp_executesql @sql;

/* 3 · Log truncation blockers */
SELECT 'LOG_REUSE' AS chk, name AS db,
       log_reuse_wait_desc, recovery_model_desc
FROM sys.databases
WHERE database_id > 4
  AND log_reuse_wait_desc <> 'NOTHING';

/* 4 · Backup gaps (FULL-recovery dbs: log 1h+, full 7d+) */
SELECT 'BACKUP_GAP' AS chk, d.name AS db,
  MAX(CASE WHEN b.type='D' THEN b.backup_finish_date END) AS last_full,
  MAX(CASE WHEN b.type='L' THEN b.backup_finish_date END) AS last_log
FROM sys.databases d
LEFT JOIN msdb.dbo.backupset b ON b.database_name = d.name
WHERE d.database_id > 4 AND d.recovery_model_desc = 'FULL'
GROUP BY d.name
HAVING MAX(CASE WHEN b.type='L' THEN b.backup_finish_date END)
         < DATEADD(HOUR,-1,GETDATE())
    OR MAX(CASE WHEN b.type='D' THEN b.backup_finish_date END)
         < DATEADD(DAY,-7,GETDATE())
    OR MAX(b.backup_finish_date) IS NULL;

/* 5 · Services + disabled Agent jobs */
SELECT 'SERVICE' AS chk, servicename, status_desc, startup_type_desc
FROM sys.dm_server_services;
SELECT 'JOB_DISABLED' AS chk, name AS job
FROM msdb.dbo.sysjobs WHERE enabled = 0;

/* 6 · Backup preference. Do backups run HERE now? */
SELECT 'BACKUP_PREF' AS chk, ag.name AS ag,
       ag.automated_backup_preference_desc,
       adc.database_name,
       sys.fn_hadr_backup_is_preferred_replica(adc.database_name)
         AS backs_up_here
FROM sys.availability_groups ag
JOIN sys.availability_databases_cluster adc
     ON ag.group_id = adc.group_id;

/* 7 · TDE state + certificates present on this server */
SELECT 'TDE_DB' AS chk, DB_NAME(database_id) AS db,
  CASE encryption_state WHEN 3 THEN 'ENCRYPTED'
       WHEN 2 THEN 'IN PROGRESS' WHEN 1 THEN 'UNENCRYPTED'
       ELSE CONVERT(varchar(10),encryption_state) END AS tde_state
FROM sys.dm_database_encryption_keys;
SELECT 'CERT' AS chk, name, expiry_date
FROM master.sys.certificates WHERE name NOT LIKE '##%';

/* 8 · Server-level objects apps depend on */
SELECT 'LINKED_SERVER' AS chk, name, data_source
FROM sys.servers WHERE is_linked = 1;
SELECT 'CREDENTIAL' AS chk, name, credential_identity
FROM sys.credentials;
SELECT 'AGENT_PROXY' AS chk, name FROM msdb.dbo.sysproxies;

/* 9 · Config fingerprint. Diff by eye vs prod */
SELECT 'CONFIG' AS chk, name, value_in_use
FROM sys.configurations
WHERE name IN ('max server memory (MB)',
  'max degree of parallelism','cost threshold for parallelism',
  'optimize for ad hoc workloads','backup compression default',
  'remote admin connections');

Login transfer. Generate CREATE LOGIN scripts

Run on the prod primary. Output = ready-to-run statements for the DR replica. SQL logins carry their original SID + password hash, so db users bind automatically. No orphans, no password resets. Review output before running (skip accounts that shouldn't exist in DR).

/* SQL logins. Same SID + hashed password */
SELECT 'CREATE LOGIN ' + QUOTENAME(p.name)
 + ' WITH PASSWORD = '
 + CONVERT(varchar(512), l.password_hash, 1) + ' HASHED'
 + ', SID = ' + CONVERT(varchar(100), p.sid, 1)
 + ', DEFAULT_DATABASE = ' + QUOTENAME(p.default_database_name)
 + ', CHECK_POLICY = '
 + CASE WHEN l.is_policy_checked = 1 THEN 'ON' ELSE 'OFF' END
 + ', CHECK_EXPIRATION = '
 + CASE WHEN l.is_expiration_checked = 1 THEN 'ON' ELSE 'OFF' END
 + ';' AS stmt
FROM sys.server_principals p
JOIN sys.sql_logins l ON p.principal_id = l.principal_id
WHERE p.type = 'S' AND p.name NOT LIKE '##%' AND p.name <> 'sa'

UNION ALL
/* Windows logins / groups */
SELECT 'CREATE LOGIN ' + QUOTENAME(name)
 + ' FROM WINDOWS WITH DEFAULT_DATABASE = '
 + QUOTENAME(default_database_name) + ';'
FROM sys.server_principals
WHERE type IN ('U','G') AND name NOT LIKE 'NT %';

/* Server role memberships */
SELECT 'ALTER SERVER ROLE ' + QUOTENAME(r.name)
 + ' ADD MEMBER ' + QUOTENAME(m.name) + ';' AS stmt
FROM sys.server_role_members rm
JOIN sys.server_principals r
     ON rm.role_principal_id = r.principal_id
JOIN sys.server_principals m
     ON rm.member_principal_id = m.principal_id
WHERE m.name NOT LIKE '##%' AND m.name <> 'sa';

/* Explicit server-level grants, REVIEW before running */
SELECT 'GRANT ' + sp.permission_name
 + ' TO ' + QUOTENAME(p.name) + ';' AS stmt
FROM sys.server_permissions sp
JOIN sys.server_principals p
     ON sp.grantee_principal_id = p.principal_id
WHERE sp.state = 'G' AND sp.class = 100
  AND p.name NOT LIKE '##%'
  AND p.name NOT IN ('sa','public');

Rebuild linked servers on DR

Run on prod: first the inventory (what exists + how logins map), then the generator. Remote passwords can never be scripted out. Fill those from your vault.

/* inventory. Servers + login mappings */
SELECT s.name, s.data_source, s.provider, s.product,
       ll.uses_self_credential AS self_map,
       ll.remote_name AS remote_login,
       sp.name AS for_local_login
FROM sys.servers s
LEFT JOIN sys.linked_logins ll ON s.server_id = ll.server_id
LEFT JOIN sys.server_principals sp
       ON ll.local_principal_id = sp.principal_id
WHERE s.is_linked = 1;

/* generator. Recreate on DR */
SELECT 'EXEC sp_addlinkedserver @server=N''' + name
 + ''', @srvproduct=N''' + ISNULL(product,'')
 + ''', @provider=N''' + provider
 + ''', @datasrc=N''' + ISNULL(data_source,'') + ''';' AS stmt
FROM sys.servers WHERE is_linked = 1;

/* then per server. Map the login (password from vault): */
-- EXEC sp_addlinkedsrvlogin @rmtsrvname = N'LINKEDNAME',
--   @useself = 'FALSE', @locallogin = NULL,
--   @rmtuser = N'remote_user', @rmtpassword = '<vault>';

/* verify from DR: */
-- EXEC sp_testlinkedserver N'LINKEDNAME';

DR gotcha: even recreated perfectly, a linked server pointing at a prod-side hostname is dead if that site is down. Check whether it should point at the target's own DR partner instead.

Rebuild credentials & Agent proxies on DR

/* generator. Secrets can't be extracted, pull from vault */
SELECT 'CREATE CREDENTIAL ' + QUOTENAME(name)
 + ' WITH IDENTITY = N''' + credential_identity
 + ''', SECRET = N''<vault>'';' AS stmt
FROM sys.credentials;

/* proxies that ride on those credentials */
SELECT 'EXEC msdb.dbo.sp_add_proxy @proxy_name=N''' + p.name
 + ''', @credential_name=N''' + c.name
 + ''', @enabled=1;' AS stmt
FROM msdb.dbo.sysproxies p
JOIN sys.credentials c ON p.credential_id = c.credential_id;

/* which subsystems each proxy is granted (redo on DR) */
SELECT p.name AS proxy, s.subsystem
FROM msdb.dbo.sysproxysubsystem ps
JOIN msdb.dbo.sysproxies p ON ps.proxy_id = p.proxy_id
JOIN msdb.dbo.syssubsystems s
     ON ps.subsystem_id = s.subsystem_id;

Server-level permissions: the grants generator is in the login-transfer section above. Easy button when tooling is allowed: Copy-DbaLogin -Source PROD -Destination DR (dbatools) moves logins, SIDs, passwords, and roles in one shot. Agent jobs: Copy-DbaAgentJob.

Fleet sweep, PowerShell mass-runner

Runs every section above against a whole server list via Invoke-Sqlcmd. One CSV per section plus summary.csv, findings table at the end, connect failures flagged per server. Read-only. Copy the script below and save it as Invoke-HadrAudit.ps1.

# usage
.\Invoke-HadrAudit.ps1 -Servers 'DRSQL01','DRSQL02\INST1'
.\Invoke-HadrAudit.ps1 -ServerList .\servers.txt -OutDir C:\Temp\hadr-audit
.\Invoke-HadrAudit.ps1 -Servers DRSQL01 -SqlCredential (Get-Credential)

# servers.txt = one instance per line, # comments allowed

<#
.SYNOPSIS
  HADR DR fleet audit. Runs the read-only "things that don't fail over" checks
  against a list of SQL Server instances and writes per-section CSVs plus a
  findings summary. Companion to the SQL HADR Triage page (Detect tab).

.DESCRIPTION
  Read-only. Requires Invoke-Sqlcmd (SqlServer module, or legacy SQLPS).
  Windows auth by default; pass -SqlCredential for SQL auth.
  Works on Windows PowerShell 5.1 and PowerShell 7+.

.EXAMPLE
  .\Invoke-HadrAudit.ps1 -Servers 'DRSQL01','DRSQL02\INST1'

.EXAMPLE
  .\Invoke-HadrAudit.ps1 -ServerList .\servers.txt -OutDir C:\Temp\hadr-audit

.EXAMPLE
  .\Invoke-HadrAudit.ps1 -Servers DRSQL01 -SqlCredential (Get-Credential)
#>
[CmdletBinding()]
param(
  [string[]]$Servers,
  [string]$ServerList,               # text file, one instance per line, # = comment
  [string]$OutDir = ".\hadr-audit",
  [pscredential]$SqlCredential,
  [int]$QueryTimeout = 60
)

$ErrorActionPreference = 'Stop'

# ---------- resolve server list ----------
if ($ServerList) {
  if (-not (Test-Path $ServerList)) { throw "Server list not found: $ServerList" }
  $Servers = @($Servers) + @(Get-Content $ServerList | Where-Object { $_ -and $_ -notmatch '^\s*#' })
}
$Servers = @($Servers | Where-Object { $_ } | ForEach-Object { $_.Trim() } | Select-Object -Unique)
if ($Servers.Count -eq 0) { throw "No servers given. Use -Servers or -ServerList." }

# ---------- ensure Invoke-Sqlcmd ----------
if (-not (Get-Command Invoke-Sqlcmd -ErrorAction SilentlyContinue)) {
  $loaded = $false
  foreach ($m in 'SqlServer','SQLPS') {
    try { Import-Module $m -DisableNameChecking -ErrorAction Stop; $loaded = $true; break } catch { }
  }
  if (-not $loaded) {
    throw "Invoke-Sqlcmd not available. Run: Install-Module SqlServer -Scope CurrentUser"
  }
}
$sqlcmdParams = (Get-Command Invoke-Sqlcmd).Parameters

# ---------- audit sections (all read-only) ----------
$Sections = [ordered]@{

AG_STATE = @'
SELECT ag.name AS ag, ar.replica_server_name AS replica,
       ars.role_desc AS role_now, DB_NAME(drs.database_id) AS db,
       drs.synchronization_state_desc AS sync_state,
       drs.suspend_reason_desc AS suspend_reason,
       drs.log_send_queue_size AS send_kb, drs.redo_queue_size AS redo_kb
FROM sys.dm_hadr_database_replica_states drs
JOIN sys.availability_replicas ar ON drs.replica_id = ar.replica_id
JOIN sys.availability_groups ag ON drs.group_id = ag.group_id
JOIN sys.dm_hadr_availability_replica_states ars ON drs.replica_id = ars.replica_id;
'@

SEEDING_MODE = @'
SELECT ag.name AS ag, ar.replica_server_name AS replica,
       ar.seeding_mode_desc, ar.availability_mode_desc, ar.failover_mode_desc
FROM sys.availability_replicas ar
JOIN sys.availability_groups ag ON ar.group_id = ag.group_id;
'@

ORPHANED_USER = @'
SET NOCOUNT ON;
DECLARE @sql nvarchar(max) = N'';
SELECT @sql += N'USE ' + QUOTENAME(name) + N';
SELECT DB_NAME() AS db, dp.name AS db_user, dp.type_desc
FROM sys.database_principals dp
LEFT JOIN sys.server_principals sp ON dp.sid = sp.sid
WHERE sp.sid IS NULL AND dp.type IN (''S'',''U'',''G'')
  AND dp.name NOT IN (''guest'',''dbo'',''sys'',''INFORMATION_SCHEMA'');
'
FROM sys.databases
WHERE database_id > 4 AND state_desc = 'ONLINE'
  AND DATABASEPROPERTYEX(name,'Updateability') = 'READ_WRITE';
EXEC sys.sp_executesql @sql;
'@

LOG_REUSE = @'
SELECT name AS db, log_reuse_wait_desc, recovery_model_desc
FROM sys.databases
WHERE database_id > 4 AND log_reuse_wait_desc <> 'NOTHING';
'@

BACKUP_GAP = @'
SELECT d.name AS db,
  MAX(CASE WHEN b.type='D' THEN b.backup_finish_date END) AS last_full,
  MAX(CASE WHEN b.type='L' THEN b.backup_finish_date END) AS last_log
FROM sys.databases d
LEFT JOIN msdb.dbo.backupset b ON b.database_name = d.name
WHERE d.database_id > 4 AND d.recovery_model_desc = 'FULL'
GROUP BY d.name
HAVING MAX(CASE WHEN b.type='L' THEN b.backup_finish_date END) < DATEADD(HOUR,-1,GETDATE())
    OR MAX(CASE WHEN b.type='D' THEN b.backup_finish_date END) < DATEADD(DAY,-7,GETDATE())
    OR MAX(b.backup_finish_date) IS NULL;
'@

SERVICE = @'
SELECT servicename, status_desc, startup_type_desc
FROM sys.dm_server_services;
'@

JOB_DISABLED = @'
SELECT name AS job FROM msdb.dbo.sysjobs WHERE enabled = 0;
'@

BACKUP_PREF = @'
SELECT ag.name AS ag, ag.automated_backup_preference_desc,
       adc.database_name,
       sys.fn_hadr_backup_is_preferred_replica(adc.database_name) AS backs_up_here
FROM sys.availability_groups ag
JOIN sys.availability_databases_cluster adc ON ag.group_id = adc.group_id;
'@

TDE_DB = @'
SELECT DB_NAME(database_id) AS db,
  CASE encryption_state WHEN 3 THEN 'ENCRYPTED' WHEN 2 THEN 'IN PROGRESS'
       WHEN 1 THEN 'UNENCRYPTED' ELSE CONVERT(varchar(10),encryption_state) END AS tde_state
FROM sys.dm_database_encryption_keys;
'@

CERT = @'
SELECT name, expiry_date FROM master.sys.certificates WHERE name NOT LIKE '##%';
'@

LINKED_SERVER = @'
SELECT name, data_source FROM sys.servers WHERE is_linked = 1;
'@

CREDENTIAL = @'
SELECT name, credential_identity FROM sys.credentials;
'@

AGENT_PROXY = @'
SELECT name FROM msdb.dbo.sysproxies;
'@

CONFIG = @'
SELECT name, CONVERT(bigint, value_in_use) AS value_in_use
FROM sys.configurations
WHERE name IN ('max server memory (MB)','max degree of parallelism',
  'cost threshold for parallelism','optimize for ad hoc workloads',
  'backup compression default','remote admin connections');
'@
}

# ---------- which rows count as ISSUES (vs informational) ----------
$IssueFilters = @{
  AG_STATE      = { ($_.suspend_reason -isnot [System.DBNull] -and $_.suspend_reason) -or
                    ($_.sync_state -notin @('SYNCHRONIZED','SYNCHRONIZING')) }
  ORPHANED_USER = { $true }
  LOG_REUSE     = { $true }
  BACKUP_GAP    = { $true }
  SERVICE       = { $_.status_desc -ne 'Running' -and $_.startup_type_desc -ne 'Disabled' }
  TDE_DB        = { $_.tde_state -eq 'IN PROGRESS' }
}

$noise = 'RowError','RowState','Table','ItemArray','HasErrors'   # DataRow baggage

function Invoke-Audit {
  param([string]$Server,[string]$Query)
  $p = @{ ServerInstance = $Server; Query = $Query; QueryTimeout = $QueryTimeout; ErrorAction = 'Stop' }
  if ($SqlCredential) {
    if ($sqlcmdParams.ContainsKey('Credential')) { $p.Credential = $SqlCredential }
    else {
      $p.Username = $SqlCredential.UserName
      $p.Password = $SqlCredential.GetNetworkCredential().Password
    }
  }
  if ($sqlcmdParams.ContainsKey('TrustServerCertificate')) { $p.TrustServerCertificate = $true }
  if ($sqlcmdParams.ContainsKey('Encrypt')) { $p.Encrypt = 'Optional' }
  Invoke-Sqlcmd @p
}

# ---------- run ----------
$stamp  = Get-Date -Format 'yyyyMMdd-HHmmss'
$runDir = Join-Path $OutDir $stamp
New-Item -ItemType Directory -Path $runDir -Force | Out-Null
$summary = New-Object System.Collections.Generic.List[object]

foreach ($srv in $Servers) {
  Write-Host "`n== $srv ==" -ForegroundColor Cyan
  try { Invoke-Audit -Server $srv -Query 'SELECT @@SERVERNAME AS s;' | Out-Null }
  catch {
    Write-Warning "  CONNECT FAIL: $($_.Exception.Message)"
    $summary.Add([pscustomobject]@{ Server=$srv; Section='CONNECT'; Rows=0; Issues=1; Note=$_.Exception.Message })
    continue
  }

  foreach ($name in $Sections.Keys) {
    try {
      $rows = @(Invoke-Audit -Server $srv -Query $Sections[$name])
      $issues = 0
      if ($IssueFilters.ContainsKey($name) -and $rows.Count -gt 0) {
        $issues = @($rows | Where-Object $IssueFilters[$name]).Count
      }
      if ($rows.Count -gt 0) {
        $rows |
          Select-Object -Property @{n='Server';e={$srv}},* -ExcludeProperty $noise |
          Export-Csv -Path (Join-Path $runDir "$name.csv") -NoTypeInformation -Append
      }
      $summary.Add([pscustomobject]@{ Server=$srv; Section=$name; Rows=$rows.Count; Issues=$issues; Note='' })
      $color = 'DarkGray'
      if ($issues -gt 0) { $color = 'Yellow' }
      Write-Host ("  {0,-14} rows: {1,4}  issues: {2}" -f $name, $rows.Count, $issues) -ForegroundColor $color
    }
    catch {
      # Section can fail legitimately (e.g. no AG on this instance). Record and move on
      $summary.Add([pscustomobject]@{ Server=$srv; Section=$name; Rows=0; Issues=0; Note=$_.Exception.Message })
      Write-Host ("  {0,-14} skipped: {1}" -f $name, $_.Exception.Message.Split("`n")[0]) -ForegroundColor DarkYellow
    }
  }
}

# ---------- summary ----------
$summary | Export-Csv -Path (Join-Path $runDir 'summary.csv') -NoTypeInformation
$flagged = @($summary | Where-Object { $_.Issues -gt 0 })

Write-Host "`n================ FINDINGS ================" -ForegroundColor Cyan
if ($flagged.Count -eq 0) {
  Write-Host "No issues flagged across $($Servers.Count) server(s). CSVs in $runDir" -ForegroundColor Green
} else {
  $flagged | Format-Table Server, Section, Rows, Issues, Note -AutoSize
  Write-Host "Details per section in: $runDir" -ForegroundColor Yellow
}

Step 0. Is automatic seeding even configured?

SELECT ag.name AS ag, ar.replica_server_name,
       ar.seeding_mode_desc, ar.availability_mode_desc,
       ar.failover_mode_desc
FROM sys.availability_replicas ar
JOIN sys.availability_groups ag ON ar.group_id = ag.group_id
ORDER BY ag.name, ar.replica_server_name;

MANUAL = no auto-seeding; databases must arrive by backup/restore (below). AUTOMATIC = SQL streams the db itself. Monitor with the seeding DMVs in the Queries tab.

Enable automatic seeding

-- on the PRIMARY: switch the DR replica to auto-seed
ALTER AVAILABILITY GROUP [YourAG]
  MODIFY REPLICA ON 'DRSERVER'
  WITH (SEEDING_MODE = AUTOMATIC);

-- on the SECONDARY: let the AG create databases
ALTER AVAILABILITY GROUP [YourAG] GRANT CREATE ANY DATABASE;

Needs matching data/log file paths on both replicas (or SQL 2017+ for some flexibility), and the db added to the AG on the primary. Seeding a big db over the DR link is slow. Check the ETA DMV before waiting on it.

Manual seeding. Backup/restore runbook

The reliable path when auto-seeding is off, blocked, or too slow. Everything restores WITH NORECOVERY until the join.

/* TDE database? Move the certificate FIRST or the
   restore fails. On PROD: */
BACKUP CERTIFICATE YourTDECert
  TO FILE = '\\share\YourTDECert.cer'
  WITH PRIVATE KEY (FILE = '\\share\YourTDECert.pvk',
       ENCRYPTION BY PASSWORD = '<strong-password>');

/* On DR (create a master key first if none): */
CREATE MASTER KEY ENCRYPTION BY PASSWORD = '<strong-password>';
CREATE CERTIFICATE YourTDECert
  FROM FILE = '\\share\YourTDECert.cer'
  WITH PRIVATE KEY (FILE = '\\share\YourTDECert.pvk',
       DECRYPTION BY PASSWORD = '<strong-password>');

/* STEP 1. On PRIMARY: copy-only full backup
   (COPY_ONLY leaves the diff chain alone) */
BACKUP DATABASE [YourDb]
  TO DISK = '\\share\YourDb_seed.bak'
  WITH COPY_ONLY, COMPRESSION, CHECKSUM, STATS = 5;

/* STEP 2. On DR: check logical names / paths */
RESTORE FILELISTONLY FROM DISK = '\\share\YourDb_seed.bak';

/* STEP 3. On DR: restore, MOVE if paths differ */
RESTORE DATABASE [YourDb]
  FROM DISK = '\\share\YourDb_seed.bak'
  WITH NORECOVERY, CHECKSUM, STATS = 5,
  MOVE 'YourDb_Data' TO 'D:\Data\YourDb.mdf',
  MOVE 'YourDb_Log'  TO 'L:\Log\YourDb_log.ldf';

/* STEP 4. On PRIMARY: fresh log backup */
BACKUP LOG [YourDb]
  TO DISK = '\\share\YourDb_seed.trn'
  WITH COMPRESSION, CHECKSUM;

/* STEP 5. On DR: restore the log, still NORECOVERY */
RESTORE LOG [YourDb]
  FROM DISK = '\\share\YourDb_seed.trn'
  WITH NORECOVERY;

/* STEP 6. Join the AG.
   If this replica was never joined (run on DR once): */
ALTER AVAILABILITY GROUP [YourAG] JOIN;
/* then per database (on DR): */
ALTER DATABASE [YourDb]
  SET HADR AVAILABILITY GROUP = [YourAG];

/* STEP 7. Verify: AG health query (Queries tab)
   → state should go SYNCHRONIZING, queues draining */

Join error	Move
"log in this backup set begins at LSN..."	A log backup is missing from the middle. Restore them in order, or take/restore one more fresh log backup
Restored copy too old to join	One more log backup on primary → restore NORECOVERY → join again
Unknown log backups splitting the chain	Old prod backup jobs still firing. Find them (msdb backupset query, Queries tab) and stop the extra job
Db stuck "Restoring..."	That's correct pre-join. Do NOT restore WITH RECOVERY; join the AG instead

⚡ All-systems triage. Run this first on any host/monitor alert

One paste-and-run, read-only. Gives a GREEN/RED verdict, then every database's state, AG replica roles with hot (synchronous, zero-loss) vs cold (async DR), sync health, suspended movement, redo/send queue, listener, cluster nodes, and last backups. Changes nothing. Companion PowerShell runner for a whole server list: Invoke-HadrAllSystemsCheck.ps1 (an unreachable box becomes its own RED line).

/*===========================================================
  HADR ALL-SYSTEMS TRIAGE  .  read-only  .  safe on production
  Run on any replica you can still reach. Read top-down.
===========================================================*/
SET NOCOUNT ON;

/*-- 0. WHERE AM I -----------------------------------------*/
SELECT [check]='0 . This instance', server=@@SERVERNAME,
   is_up='YES (you are connected)',
   role_here=ISNULL((SELECT TOP 1 role_desc
        FROM sys.dm_hadr_availability_replica_states WHERE is_local=1),'NOT IN AN AG'),
   sql_start_time=(SELECT sqlserver_start_time FROM sys.dm_os_sys_info),
   version=LEFT(CONVERT(varchar(200),SERVERPROPERTY('ProductVersion')),20);

/*-- 1. ROLL-UP VERDICT (read this first) ------------------*/
DECLARE @dbBad int,@replBad int,@moveBad int,@redoHot int;
SELECT @dbBad=COUNT(*) FROM sys.databases
   WHERE state_desc<>'ONLINE' AND database_id>4;
SELECT @replBad=COUNT(*) FROM sys.dm_hadr_availability_replica_states
   WHERE synchronization_health_desc<>'HEALTHY';
SELECT @moveBad=COUNT(*) FROM sys.dm_hadr_database_replica_states
   WHERE is_suspended=1
      OR synchronization_state_desc IN ('NOT SYNCHRONIZING','NOT SYNCHRONIZED');
SELECT @redoHot=COUNT(*) FROM sys.dm_hadr_database_replica_states
   WHERE redo_queue_size>51200;   -- >50 MB behind on redo
SELECT [check]='1 . VERDICT',
   verdict=CASE WHEN ISNULL(@dbBad,0)+ISNULL(@replBad,0)+ISNULL(@moveBad,0)=0
                THEN 'ALL GREEN . systems online and synchronizing'
                ELSE 'ATTENTION . see flagged sections below' END,
   dbs_not_online=ISNULL(@dbBad,0), replicas_not_healthy=ISNULL(@replBad,0),
   db_movement_problems=ISNULL(@moveBad,0), replicas_behind_on_redo=ISNULL(@redoHot,0);

/*-- 2. EVERY DATABASE ONLINE? -----------------------------*/
SELECT [check]='2 . Database states', database_name=name, state=state_desc,
   flag=CASE WHEN state_desc='ONLINE' THEN 'ok' ELSE 'NOT ONLINE' END,
   recovery_model=recovery_model_desc, log_reuse_wait=log_reuse_wait_desc
FROM sys.databases WHERE database_id>4
ORDER BY CASE WHEN state_desc='ONLINE' THEN 1 ELSE 0 END, name;

/*-- 3. AG REPLICAS . hot (sync) vs cold (async DR) --------*/
SELECT [check]='3 . AG replicas', ag_name=ag.name,
   replica_server=ar.replica_server_name, role=hars.role_desc,
   temperature=CASE ar.availability_mode_desc
        WHEN 'SYNCHRONOUS_COMMIT'  THEN 'HOT  (sync, zero-loss)'
        WHEN 'ASYNCHRONOUS_COMMIT' THEN 'COLD (async DR)'
        ELSE ar.availability_mode_desc END,
   failover=ar.failover_mode_desc, connected=hars.connected_state_desc,
   sync_health=hars.synchronization_health_desc, op_state=hars.operational_state_desc,
   readable=ar.secondary_role_allow_connections_desc
FROM sys.availability_replicas ar
JOIN sys.availability_groups ag ON ag.group_id=ar.group_id
JOIN sys.dm_hadr_availability_replica_states hars ON hars.replica_id=ar.replica_id
ORDER BY ag.name, CASE hars.role_desc WHEN 'PRIMARY' THEN 0 ELSE 1 END,
   ar.availability_mode_desc;

/*-- 4. PER-DB SYNC . the failover-readiness gauge ---------*/
SELECT [check]='4 . DB sync state', ag_name=ag.name,
   replica_server=ar.replica_server_name, database_name=DB_NAME(drs.database_id),
   sync_state=drs.synchronization_state_desc,
   suspended=CASE drs.is_suspended WHEN 1 THEN 'SUSPENDED' ELSE 'no' END,
   suspend_reason=drs.suspend_reason_desc,
   log_send_queue_kb=drs.log_send_queue_size, redo_queue_kb=drs.redo_queue_size,
   redo_rate_kb_s=drs.redo_rate,
   flag=CASE WHEN drs.is_suspended=1 THEN 'movement suspended'
        WHEN drs.synchronization_state_desc<>'SYNCHRONIZED'
             AND ar.availability_mode_desc='SYNCHRONOUS_COMMIT' THEN 'sync replica not caught up'
        WHEN drs.redo_queue_size>51200 THEN '>50MB behind on redo'
        ELSE 'ok' END
FROM sys.dm_hadr_database_replica_states drs
JOIN sys.availability_replicas ar ON ar.replica_id=drs.replica_id
JOIN sys.availability_groups ag ON ag.group_id=ar.group_id
ORDER BY ag.name, DB_NAME(drs.database_id), ar.replica_server_name;

/*-- 5. LISTENER (the name apps connect to) ----------------*/
SELECT [check]='5 . Listener', ag_name=ag.name, listener=agl.dns_name,
   port=agl.port, ip=aglip.ip_address, ip_state=aglip.state_desc
FROM sys.availability_group_listeners agl
JOIN sys.availability_groups ag ON ag.group_id=agl.group_id
LEFT JOIN sys.availability_group_listener_ip_addresses aglip
     ON aglip.listener_id=agl.listener_id
ORDER BY ag.name;

/*-- 6. CLUSTER NODES (WSFC quorum view) -------------------*/
SELECT [check]='6 . Cluster members', member_name=member_name,
   member_type=member_type_desc, member_state=member_state_desc,
   quorum_votes=number_of_quorum_votes
FROM sys.dm_hadr_cluster_members ORDER BY member_state_desc, member_name;

/*-- 7. LAST BACKUPS (recoverability) ----------------------*/
SELECT [check]='7 . Last backups', database_name=d.name,
   last_full=MAX(CASE WHEN b.type='D' THEN b.backup_finish_date END),
   last_diff=MAX(CASE WHEN b.type='I' THEN b.backup_finish_date END),
   last_log =MAX(CASE WHEN b.type='L' THEN b.backup_finish_date END),
   mins_since_log=DATEDIFF(minute,
        MAX(CASE WHEN b.type='L' THEN b.backup_finish_date END),GETDATE())
FROM sys.databases d
LEFT JOIN msdb.dbo.backupset b ON b.database_name=d.name
WHERE d.database_id>4 GROUP BY d.name ORDER BY d.name;

AG health. One-glance overview

SELECT ag.name AS ag_name, ar.replica_server_name,
       ars.role_desc, DB_NAME(drs.database_id) AS db,
       drs.synchronization_state_desc,
       drs.synchronization_health_desc,
       drs.suspend_reason_desc,
       drs.log_send_queue_size, drs.redo_queue_size
FROM sys.dm_hadr_database_replica_states drs
JOIN sys.availability_replicas ar
     ON drs.replica_id = ar.replica_id
JOIN sys.availability_groups ag
     ON drs.group_id = ag.group_id
JOIN sys.dm_hadr_availability_replica_states ars
     ON drs.replica_id = ars.replica_id
ORDER BY ag.name, ar.replica_server_name, db;

Why can't the log truncate?

SELECT name, log_reuse_wait_desc,
       recovery_model_desc, state_desc
FROM sys.databases
ORDER BY name;

Orphaned users on new primary

-- run in each user database
SELECT dp.name AS db_user, dp.type_desc
FROM sys.database_principals dp
LEFT JOIN sys.server_principals sp ON dp.sid = sp.sid
WHERE sp.sid IS NULL
  AND dp.type IN ('S','U','G')
  AND dp.name NOT IN ('guest','dbo','sys','INFORMATION_SCHEMA');

-- fix (per user, once matching login exists):
-- ALTER USER [username] WITH LOGIN = [loginname];

Resume suspended data movement

-- on the replica where the db shows SUSPENDED
ALTER DATABASE [YourDb] SET HADR RESUME;

Automatic seeding progress

SELECT ag.name, ds.database_name,
       ds.current_state, ds.failure_state_desc,
       ds.start_time, ds.completion_time
FROM sys.dm_hadr_automatic_seeding ds
JOIN sys.availability_groups ag ON ds.ag_id = ag.group_id
ORDER BY ds.start_time DESC;

SELECT database_name, transfer_rate_bytes_per_second,
       transferred_size_bytes, database_size_bytes,
       start_time_utc, estimate_time_complete_utc
FROM sys.dm_hadr_physical_seeding_stats;

Listener config as SQL sees it

SELECT l.dns_name, l.port, ip.ip_address,
       ip.state_desc, ip.network_subnet_ip
FROM sys.availability_group_listeners l
JOIN sys.availability_group_listener_ip_addresses ip
     ON l.listener_id = ip.listener_id;

Cluster / DNS checks (PowerShell, on a node)

# resolve what clients actually get
nslookup your-listener-name

# cluster resources behind the listener
Get-ClusterResource |
  Where-Object ResourceType -in 'Network Name','IP Address' |
  Format-Table Name, State, OwnerGroup

# listener DNS behavior (run per Network Name resource)
Get-ClusterResource "AG_ListenerName" |
  Get-ClusterParameter RegisterAllProvidersIP, HostRecordTTL

# flush stale DNS on an app/client box
ipconfig /flushdns

Last backups per database

SELECT d.name,
  MAX(CASE WHEN b.type='D' THEN b.backup_finish_date END) AS last_full,
  MAX(CASE WHEN b.type='I' THEN b.backup_finish_date END) AS last_diff,
  MAX(CASE WHEN b.type='L' THEN b.backup_finish_date END) AS last_log
FROM sys.databases d
LEFT JOIN msdb.dbo.backupset b ON b.database_name = d.name
GROUP BY d.name ORDER BY d.name;

-- is THIS replica the preferred backup replica?
SELECT sys.fn_hadr_backup_is_preferred_replica('YourDb');

Network path / IP connectivity checks (general)

Run from the CLIENT/app box (PowerShell/cmd), working outside-in. These prove whether it's DNS, IP, port, or auth before you touch SQL.

# 1 · name resolution. What IP does the client actually get?
nslookup your-sql-name
# multi-subnet AG may return several IPs; note which is the live node

# 2 · IP + port reachability (THE key test)
Test-NetConnection your-sql-name -Port 1433
# TcpTestSucceeded : True  = network path open (login issue if it still fails)
#                    False = firewall or nothing listening on that port

# 3 · named instance? the SQL Browser uses UDP 1434
Test-NetConnection your-sql-host -Port 1434 -InformationLevel Detailed

# 4 · path / where it dies
tracert your-sql-name
# 5 · flush a stale client DNS cache after a failover
ipconfig /flushdns

-- run ON the SQL host: what port/IP is SQL actually listening on?
--   (or read the errorlog line "Server is listening on")
SELECT local_net_address, local_tcp_port, client_net_address
FROM sys.dm_exec_connections WHERE session_id = @@SPID;
EXEC xp_readerrorlog 0, 1, N'Server is listening on';

# run ON the SQL host: is the port open in the Windows firewall?
Get-NetFirewallRule -Enabled True |
  Where-Object DisplayName -match 'SQL' |
  Format-Table DisplayName, Direction, Action
# quick listening-ports view
netstat -ano | findstr :1433

Backup health & agent checks (general)

Run these when a backup alert fires (agent retransmission/crash-loop) or any time you need to prove you're protected. Comments say where to run each.

-- run on the SQL instance: anything with NO backup in the last 24h
SELECT d.name, d.recovery_model_desc,
       MAX(b.backup_finish_date) AS last_backup,
       DATEDIFF(HOUR, MAX(b.backup_finish_date), GETDATE()) AS hours_ago
FROM sys.databases d
LEFT JOIN msdb.dbo.backupset b ON b.database_name = d.name
WHERE d.database_id > 4
GROUP BY d.name, d.recovery_model_desc
HAVING MAX(b.backup_finish_date) < DATEADD(HOUR,-24,GETDATE())
    OR MAX(b.backup_finish_date) IS NULL
ORDER BY hours_ago DESC;

-- run on the SQL instance: backup jobs that FAILED recently (msdb)
SELECT j.name AS job, h.run_date, h.run_time, h.message
FROM msdb.dbo.sysjobhistory h
JOIN msdb.dbo.sysjobs j ON h.job_id = j.job_id
WHERE h.run_status = 0            -- 0 = failed
  AND (j.name LIKE '%backup%' OR j.name LIKE '%cohesity%' OR h.message LIKE '%backup%')
ORDER BY h.run_date DESC, h.run_time DESC;

-- run on the SQL instance: recent backup THROUGHPUT (slow = link/agent problem)
SELECT TOP 20 database_name, type,
       backup_finish_date,
       CAST(backup_size/1048576.0 AS decimal(10,1)) AS mb,
       DATEDIFF(SECOND, backup_start_date, backup_finish_date) AS secs,
       CAST(backup_size/1048576.0 /
            NULLIF(DATEDIFF(SECOND, backup_start_date, backup_finish_date),0)
            AS decimal(10,1)) AS mb_per_sec
FROM msdb.dbo.backupset
ORDER BY backup_finish_date DESC;

:: run on the HOST (cmd/PowerShell), NOT in SQL:
:: VSS writers Cohesity/agent-based backups depend on - any FAILED = the cause
vssadmin list writers
:: look for the SQL Server VSS Writer + others; State should be [1] Stable, no error

⚠ Forced failover (data-loss risk)

-- ONLY with incident-command approval, primary unrecoverable.
-- On the target secondary:
ALTER AVAILABILITY GROUP [YourAG]
  FORCE_FAILOVER_ALLOW_DATA_LOSS;

-- afterwards, every other replica's dbs are SUSPENDED.
-- On each, after validating:
ALTER DATABASE [YourDb] SET HADR RESUME;

Error code lookup

Type an error number (18456, 1418, 9002…) or a word from the message. Exact/contains matching here. The Ops Console has the full fuzzy search plus the wait-type board.

What does NOT fail over with the AG

The AG moves databases only. Everything server-level lives outside it and must already exist on the DR replica. The Detect tab's mass audit checks every row of this table in one run:

Object	Gotcha after failover
Logins (SQL + Windows)	SQL logins must be created with the same SID or db users orphan
Agent jobs & operators	Backup/maintenance jobs may not exist. Or run on the wrong replica
Linked servers	Missing or pointing at prod-side names
Credentials / proxies	Jobs fail with auth errors
Server-level permissions & roles	App works but admin/monitoring accounts lose rights
TDE certificates / master key	Encrypted DB won't come online without the cert restored
Instance config (max memory, trace flags, tempdb)	Performance surprises on the DR side

Listener & DNS quick facts

Setting	Why it bites in DR
`RegisterAllProvidersIP = 1`	All subnet IPs registered; legacy clients without `MultiSubnetFailover=True` may try the dead IP and hang
`RegisterAllProvidersIP = 0`	Only the online IP registers. But clients wait on `HostRecordTTL` (default 1200s!) for DNS to catch up
`MultiSubnetFailover=True`	Connection string tries all IPs in parallel. The right answer for multi-subnet AGs
AD computer object (CNO/VCO)	Listener creation/online fails if the cluster object can't update DNS / create the VCO in AD

log_reuse_wait_desc decoder

Value	Meaning · move
`LOG_BACKUP`	Take a log backup. Check backup jobs actually run on the current primary / preferred replica
`AVAILABILITY_REPLICA`	A secondary is suspended/behind. Log can't truncate past what it hasn't received. Fix or resume the secondary; if it's gone for good, remove it from the AG
`ACTIVE_TRANSACTION`	Long-running/open txn. Find it (`DBCC OPENTRAN`) before the disk fills
`DATABASE_MIRRORING`/`REPLICATION`	Other HA features holding the log. Investigate that feature

Sync vs async. Failover rules

Synchronous-commit replica in SYNCHRONIZED state → planned failover, no data loss. Asynchronous (typical for DR sites) → planned failover is blocked; you either switch it to sync and wait for SYNCHRONIZED, or take a forced failover and accept possible data loss. After any forced failover, all other replicas suspend and need review + HADR RESUME.

Database state decoder. When to act

You see	Normal when…	Act when / what to do
Restoring…	seeding or mid restore chain; required before AG join	Never RESTORE WITH RECOVERY if it's joining an AG. Run the JOIN. Unexpected = someone SET HADR OFF
Recovering	right after startup/failover	Stuck for long = redo backlog or errorlog issue. Check DB STARTUP progress
Recovery Pending	never	Errorlog first: missing file (17204), TDE cert (33111), permissions (5120). Fix cause, SET ONLINE
Suspect	never	Recovery failed. Restore from backup; EMERGENCY only for salvage with approval
Offline / Single_user	deliberate maintenance	Find who/why before flipping back to ONLINE / MULTI_USER
Synchronized	sync replica, healthy	Nothing. Safe no-loss failover target
Synchronizing	always, on async replicas	On a SYNC replica: wait for Synchronized before failover; never arriving = check queues/network
Not Synchronizing	never	Movement stopped. Get suspend_reason, fix, HADR RESUME (suspended secondary blocks log truncation!)
Reverting	after failover (undo phase)	Wait. Interrupting makes it longer
Initializing / Seeding	auto-seeding in flight	Stalled at 0 = path/permission problem, Seed tab

Searchable versions with queries: type any state into the Ops Console search, or tap its DB states chip.

Version & edition. Check first, it changes the answer

Availability-group features differ sharply by SQL Server version AND edition (Enterprise vs Standard). Run this before trusting any AG step or DMV below.

SELECT SERVERPROPERTY('ProductVersion')  AS version,      -- e.g. 15.0.x = 2019, 16.0.x = 2022
       SERVERPROPERTY('ProductLevel')    AS patch_level,  -- RTM / SPn / CUn
       SERVERPROPERTY('Edition')         AS edition,       -- Enterprise / Standard / Developer
       SERVERPROPERTY('EngineEdition')   AS engine_edition,-- 3 = Enterprise-features, 2 = Standard
       SERVERPROPERTY('IsHadrEnabled')   AS hadr_enabled;  -- 1 = Always On turned on

Feature	Needs	Watch out
Always On AGs (full)	Enterprise, 2012+	Not on Standard at all
Basic Availability Groups	Standard, 2016+	2 replicas, 1 db, NO readable secondary, NO backups on secondary
Readable secondaries / read-only routing	Enterprise, 2012+	Enterprise only; improved 2016+
Automatic seeding	2016+	2012/2014 = manual backup/restore seeding ONLY (the seeding DMVs return nothing)
3 automatic-failover replicas	2016+	2012/2014 cap at 2 sync-auto-failover
Distributed AGs	Enterprise, 2016+	,
Read-scale AG (no WSFC)	2017+	Clusterless; no automatic failover
Contained AGs (own master/msdb)	2022	Newest; logins/jobs travel WITH the AG

Queries here that use sys.dm_hadr_automatic_seeding / physical_seeding_stats assume 2016+. On 2012/2014, use the manual backup/restore runbook (Seed tab). Auto-seeding simply doesn't exist. Standard Edition: expect Basic AG limits, not the full toolset.

Post-failover smoke test

#	Check
1	All DBs on new primary: SYNCHRONIZED/SYNCHRONIZING, none SUSPENDED or NOT SYNCHRONIZING
2	Listener resolves to the new primary's IP from an app server, not just the DB node
3	App service accounts can log in. No orphaned users
4	Log backups running on the new primary; `log_reuse_wait_desc` clean
5	Agent jobs enabled/appropriate for this replica's role
6	Old primary (if reachable) rejoined as secondary and resumed

Triage log

Timestamped record of what you saw and did. Feeds the DR-exercise report. Stored only in this browser.

Personal decision-support template for SQL Server Always On DR exercises. Not official policy of any institution. Defer to your site's approved runbook and DBA/incident-command guidance where they differ. Anything marked with ⚠ can lose data; get approval first.

Triage Path

Edit decision tree (JSON)