diff --git a/demos-archive/VMFS-VMDK Snapshot/README - VMFS-VMDK Snapshot - V1.md b/demos-archive/VMFS-VMDK Snapshot/README - VMFS-VMDK Snapshot - V1.md new file mode 100644 index 0000000..c80bd67 --- /dev/null +++ b/demos-archive/VMFS-VMDK Snapshot/README - VMFS-VMDK Snapshot - V1.md @@ -0,0 +1,46 @@ +**VMFS/VMDK + SQL Server Snapshot Scripts** +

+This folder contains VMFS/VMDK + SQL Server example snapshot scripts. + +**Files:** +- VMFS-VMDK Snapshot.ps1 + + +
+ + +**Scenario:** +
This example script shows steps to snapshot a VMFS datastore that contains data & log VMDKs for a SQL Server. The overall scenario is taking a snapshot of a production SQL Server's underlying datastore, and cloning the datastore to then overlay a pre-existing non-production datastore for a non-production SQL Server. + +All references to a "target" refer to the non-production side (VM, datastore, etc). + +**Prerequisites:** +1. The production datastore must already be cloned and presented once, to the non-production side. +2. This script assumes the database(s) are already attached on the target, non-production SQL Server. + +**Important Usage Notes:** +
You must pre-setup the target VM with a cloned datastore from the source already. You will ONLY be utilizing the specific VMDK(s) that contain the data/log files of interest, from the cloned datastore. Also note that the VMFS datastore does not need to only exclusively contain VMDKs for the SQL Server in question. If other VMDKs are present in the datastore, used by the either the source SQL Server VM or other VMs, they do not need to be deleted or otherwise manipulated during this cloning process. Remember FlashArray deduplicates data, thus a clone's set of additional, unused VMDKs will not have a negative impact. + +For the cloned datastore pre-setup, you can use subsets of the code below to clone the source datastore, present it to the target server, then attach the VMDK(s) containing the production databases that will be re-cloned with this script. Once "staged," you can then use this script fully to refresh the data files in the cloned datastore that is attached to the target server. + +When cloning, note that the target datastore is dropped and replaced entirely. This is because when cloning a datastore, it must be resignatured and the datastore will be renamed with a non-deterministic naming scheme (snap-[[GUID chars]]-[[original DS name]]). Thus it is not possible to know what the new datastore name will be until the resignature step is executed. + +This script also assumes that all database files (data and log) are on the same volume/single VMDK. If multiple volumes/VMDKs are being used, you will have to adjust the code (ex: add additional foreach loops for manipulating multiple VMDKs). + + +
+ + +**Disclaimer:** +
+This example script is provided AS-IS and meant to be a building block to be adapted to fit an individual organization's infrastructure. +
+
+ +We encourage the modification and expansion of these scripts by the community. Although not necessary, please issue a Pull Request (PR) if you wish to request merging your modified code in to this repository. + + +
+ + +_The contents of the repository are intended as examples only and should be modified to work in your individual environments. No script examples should be used in a production environment without fully testing them in a development or lab environment. There are no expressed or implied warranties or liability for the use of these example scripts and templates presented by Pure Storage and/or their creators._ diff --git a/demos-archive/VMFS-VMDK Snapshot/VMFS-VMDK Snapshot - V1.ps1 b/demos-archive/VMFS-VMDK Snapshot/VMFS-VMDK Snapshot - V1.ps1 new file mode 100644 index 0000000..882c09d --- /dev/null +++ b/demos-archive/VMFS-VMDK Snapshot/VMFS-VMDK Snapshot - V1.ps1 @@ -0,0 +1,171 @@ +############################################################################################################################## +# Refresh VMFS VMDK with Snapshot Demo +# +# +# Scenario: +# Snapshot and clone a "production" VMDK in a VMFS datastore, then present it to a "non-production" server. +# +# This example has two databases: ExampleDb1, ExampleDb2, whose data and log files both reside on a single disk/VMDK. +# +# +# Usage Notes: +# +# You must pre-setup the target VM with a cloned datastore from the source already. You will ONLY be utilizing +# the SPECIFIC VMDK(s) that contain the data/log files of interest, from the cloned datastore. Other VMDKs can safely be +# ignored since they are deduped on FlashArray. +# +# For the cloned datastore pre-setup, you can use subsets of the code below to clone the source datastore, present it to +# the target server, then attach the VMDK(s) containing the production databases that will be re-cloned with this script. +# Once "staged," you can then use this script fully to refresh the data files in the cloned datastore that is attached +# to the target server. +# +# This script also assumes that all database files (data and log) are on the same volume/single VMDK. If multiple +# volumes/VMDKs are being used, adjust the code to add additional foreach loops when manipulating the VMDKs. +# +# 2025/12/22: AYun - Renamed to "VMFS-VMDK Snapshot - V1.ps1" and migrated to archive in +# PureStorage-OpenConnect\sqlserver-scripts\demos-archive\VMFS-VMDK Snapshot +# +# Disclaimer: +# This example script is provided AS-IS and meant to be a building block to be adapted to fit an individual +# organization's infrastructure. +############################################################################################################################## + + + +# Import powershell modules +Import-Module PureStoragePowerShellSDK2 +Import-Module VMware.VimAutomation.Core +Import-Module SqlServer + + + +# Declare variables +$TargetVM = 'SqlServer1' # Name of target VM +$Databases = @('ExampleDb1','ExampleDb2') # Array of database names +$TargetDiskSerialNumber = '6000c02022cb876dcd321example01b' # Target Disk Serial Number +$VIServerName = 'vcenter.example.com' # vCenter FQDN +$ClusterName = 'WorkloadCluster1' # VMware Cluster +$SourceDatastoreName = 'vmware_sql_datastore' # VMware datastore name +$SourceVMDKPath = 'SqlServer1_1/SqlServer1.vmdk' # VMDK path inside the VMFS datastore +$ArrayName = 'flasharray1.example.com' # FlashArray FQDN +$SourceVolumeName = 'sql_volume_1' # Source volume name on FlashArray (may be same as your datastore name) +$TargetVolumeName = 'sql_volume_2' # Target volume name on FlashArray (may be same as your datastore name) + + + +# Set Credential - this assumes the same credential for the target VM and vCenter +$Credential = Get-Credential + + + +# Create a Powershell session against the target VM +$TargetVMSession = New-PSSession -ComputerName $TargetVM -Credential $Credential + + + +# Connect to vCenter +$VIServer = Connect-VIServer -Server $VIServerName -Protocol https -Credential $Credential + + + +# Offline the target database(s) by looping through $Databases array +foreach ($Database in $Databases) { + $Query = "ALTER DATABASE [$Database] SET OFFLINE WITH ROLLBACK IMMEDIATE" + Invoke-Sqlcmd -ServerInstance $TargetVM -Database master -Query $Query +} + + + +# Offline the volumes that have SQL data +Invoke-Command -Session $TargetVMSession -ScriptBlock { Get-Disk | Where-Object { $_.SerialNumber -eq $using:TargetDiskSerialNumber } | Set-Disk -IsOffline $True } + + + +# Prepare to remove the VMDK from the VM +$VM = Get-VM -Server $VIServer -Name $TargetVM +$HardDisk = Get-HardDisk -VM $VM | Where-Object { $_.FileName -match $SourceVMDKPath } + + + +# Remove the VMDK from the VM +Remove-HardDisk -HardDisk $HardDisk -Confirm:$false + + + +# Prepare to remove the stale datastore +$DataStore = $HardDisk.Filename.Substring(1, ($HardDisk.Filename.LastIndexOf(']') - 1)) +$Hosts = Get-Cluster $ClusterName | Get-VMHost | Where-Object { ($_.ConnectionState -eq 'Connected') } + + + +# Guest hard disk removed, now remove the stale datastore - this can take a min or two +Get-Datastore $DataStore | Remove-Datastore -VMHost $Hosts[0] -Confirm:$False + + + +# Connect to the array, authenticate. Remember disclaimer at the top! +$FlashArray = Connect-Pfa2Array -Endpoint $ArrayName -Credential ($Credential) -IgnoreCertificateError + + + +# Perform the volume overwrite (no intermediate snapshot needed!) +New-Pfa2Volume -Array $FlashArray -Name $TargetVolumeName -SourceName $SourceVolumeName -Overwrite $True + + + +# Rescan storage on each ESX host in the $Hosts array +foreach ($VmHost in $Hosts) { + Get-VMHostStorage -RescanAllHba -RescanVmfs -VMHost $VmHost | Out-Null +} + + + +# Connect to EsxCli +$esxcli = Get-EsxCli -VMHost $Hosts[0] + + + +# Resignature the cloned datastore +$EsxCli.Storage.Vmfs.Snapshot.Resignature($SourceDatastoreName) + + + +# Find the assigned datastore name, this may take a few seconds +# NOTE: when a datastore comes back, it's name will be "snap-[GUID chars]-[original DS name]" +# This is why the wildcard match below is needed. +$DataStore = (Get-Datastore | Where-Object { $_.Name -match 'snap' -and $_.Name -match $SourceDatastoreName }) + + + +# Rescan storage again to make sure all hosts can see the new datastore +foreach ($VmHost in $Hosts) { + Get-VMHostStorage -RescanAllHba -RescanVmfs -VMHost $VmHost | Out-Null +} + + + +# Attach the VMDK from the newly cloned datastore back to the target VM +New-HardDisk -VM $VM -DiskPath "[$DataStore] $SourceVMDKPath" + + + +# Online the volume on the target VM +Invoke-Command -Session $TargetVMSession -ScriptBlock { Get-Disk | Where-Object { $_.SerialNumber -eq $using:TargetDiskSerialNumber } | Set-Disk -IsOffline $False } + + + +# Volume might be read-only, ensure it's read/write +Invoke-Command -Session $TargetVMSession -ScriptBlock { Get-Disk | Where-Object { $_.SerialNumber -eq $using:TargetDiskSerialNumber } | Set-Disk -IsReadOnly $False } + + + +# Online the target database(s) by looping through $Databases array +foreach ($Database in $Databases) { + $Query = "ALTER DATABASE [$Database] SET ONLINE WITH ROLLBACK IMMEDIATE" + Invoke-Sqlcmd -ServerInstance $TargetVM -Database master -Query $Query +} + + + +# Remove powershell session +Remove-PSSession $TargetVMSession diff --git a/demos-sdk2/Point in Time Recovery - VMFS/Point in Time Recovery - VMFS.ps1 b/demos-sdk2/Point in Time Recovery - VMFS/Point in Time Recovery - VMFS.ps1 new file mode 100644 index 0000000..e2ad494 --- /dev/null +++ b/demos-sdk2/Point in Time Recovery - VMFS/Point in Time Recovery - VMFS.ps1 @@ -0,0 +1,350 @@ +############################################################################################################################## +# Point In Time Recovery - Using SQL Server 2022's T-SQL Snapshot Backup feature w. VMFS/VMDK datastore/files. +# +# Scenario: +# Perform a point in time restore using SQL Server 2022's T-SQL Snapshot Backup +# feature. This uses a FlashArray snapshot as the base of the restore, then restores +# a log backup. +# +# IMPORTANT NOTE: +# This example script is built for 1 database spanned across two VMDK files/volumes +# from a single datastore. +# +# The granularity or unit of work for this workflow is a VMDK file(s) and the entirety +# of its contents. Therefore, everything in the VMDK file(s) including files for other +# databases will be impacted/overwritten. +# +# This example will need to be adapted if you wish to support multiple databases on +# the same set of VMDK(s). +# +# Prerequisites: +# 1. PowerShell Modules: dbatools & PureStoragePowerShellSDK2 +# +# Usage Notes: +# * Each section of the script is meant to be run individually, one after another. +# * The script is NOT meant to be executed all at once. +# +# Disclaimer: +# This example script is provided AS-IS and is meant to be a building +# block to be adapted to fit an individual organization's +# infrastructure. +############################################################################################################################## + + + +# Import powershell modules +Import-Module dbatools +Import-Module PureStoragePowerShellSDK2 + + + +# Declare all variables +# VMware variables +$VIServerName = 'vcenter.example.com' +$SourceDatastoreName = 'source_sql_datastore' +$SourceVMDKPaths = @('source_vm/sqldata.vmdk','source_vm/sqllog.vmdk') + + + +# FlashArray variables +$ArrayName = 'flasharray1.example.com' # FlashArray FQDN +$FAHostGroupName = 'FAHostGroupName' # HostGroup Name on FlashArray for the ESXi cluster +$SourceVolumeName = 'volume_name' # Volume name on FlashArray containing database files +$PGroupName = 'protection_group' # Name of the Protection Group on FlashArray1 + + + +# Windows/SQL Server variables +$TargetSQLServer = 'target_sqlserver.example.com' # SQL Server Instance FQDN +$TargetVM = 'target_sqlserver' # SQL Server VM name in VCenter +$DbName = 'AdventureWorks' # Name of database +$BackupShare = '\\flashblade1.example.com\backups' # File system location to write the backup metadata file +$TargetDisks = @('1234c29689bc0888d32dcd2919a67z89', '1234c299721c4ba4a937552fb298a76') # The serial numbers of the Windows volume containing database files; use get-disk + + + +# Build a PowerShell Remoting Session to the Server +$SqlServerSession = New-PSSession -ComputerName $TargetSQLServer + + + +# Build a persistent SMO connection +$SqlInstance = Connect-DbaInstance -SqlInstance $TargetSQLServer -TrustServerCertificate -NonPooledConnection + + + +# Let's get some information about our database, take note of the size +Get-DbaDatabase -SqlInstance $SqlInstance -Database $DbName | + Select-Object Name, SizeMB + + + +# Connect to the FlashArray's REST API +$Credential = Get-Credential -UserName "$env:USERNAME" -Message 'Enter your credential information...' +$FlashArray = Connect-Pfa2Array –EndPoint $ArrayName -Credential $Credential -IgnoreCertificateError + + + +#### +# Execute our backup + +# Freeze the database +$Query = "ALTER DATABASE $DbName SET SUSPEND_FOR_SNAPSHOT_BACKUP = ON" +Invoke-DbaQuery -SqlInstance $SqlInstance -Query $Query -Verbose + + + +# Take a snapshot of the Protection Group while the database is frozen +$Snapshot = New-Pfa2ProtectionGroupSnapshot -Array $FlashArray -SourceName $PGroupName + + + +# Take a metadata backup of the database, this will automatically unfreeze +# if successful +# We'll use MEDIADESCRIPTION to hold some information about our snapshot and +# the flasharray its held on +$BackupFile = "$BackupShare\$DbName-$(Get-Date -Format FileDateTime).bkm" + +$Query = "BACKUP DATABASE $DbName + TO DISK='$BackupFile' + WITH METADATA_ONLY, MEDIADESCRIPTION='$($Snapshot.Name)|$($FlashArray.ArrayName)'" +Invoke-DbaQuery -SqlInstance $SqlInstance -Query $Query -Verbose + +### +# Backup completed + + + +### +# Backup Verification + +# Let's check out the error log to see what SQL Server thinks happened +Get-DbaErrorLog -SqlInstance $SqlInstance -LogNumber 0 -After (Get-Date).AddMinutes(-15) | Format-Table + + + +# The backup is recorded in MSDB as a Full backup with snapshot +$BackupHistory = Get-DbaDbBackupHistory -SqlInstance $SqlInstance -Database $DbName -Last +$BackupHistory + + + +# Let's explore the stuff in the backup header... +# Remember, VDI is just a contract saying what's in the backup matches what SQL Server thinks is in the backup. +Read-DbaBackupHeader -SqlInstance $SqlInstance -Path $BackupFile + + + +### +# Take a Transaction Log backup +# +# NOTE: If you are testing this with a database in SIMPLE RECOVERY, there seems to be an occasional bug in +# Backup-DbaDatabase that keeps a DataReader connection open. Subsequent dbatools cmdlet steps may fail. +# Skip this step if your database in SIMPLE RECOVERY. +$LogBackup = Backup-DbaDatabase -SqlInstance $SqlInstance -Database $DbName -Type Log -Path $BackupShare -CompressBackup + + + +### +# DEMO - Delete a table +Invoke-DbaQuery -SqlInstance $SqlInstance -Database $DbName -Query "SELECT TOP 10 * FROM Sales.Customer" + + + +# Delete a table +Invoke-DbaQuery -SqlInstance $SqlInstance -Database $DbName -Query "DROP TABLE Sales.Customer" + + + +# Confirm it is gone +Invoke-DbaQuery -SqlInstance $SqlInstance -Database $DbName -Query "SELECT TOP 10 * FROM Sales.Customer" + + + +### +# Review State of Database and backup + +# Let's check out the state of the database, size, last full and last log +Get-DbaDatabase -SqlInstance $SqlInstance -Database $DbName | + Select-Object Name, Size, LastFullBackup, LastLogBackup + + + +# We can get the snapshot name from the $Snapshot variable above, but what if we didn't know this ahead of time? +# We can also get the snapshot name from the MEDIADESCRIPTION in the backup file. +$Query = "RESTORE LABELONLY FROM DISK = '$BackupFile'" +$Labels = Invoke-DbaQuery -SqlInstance $SqlInstance -Query $Query -Verbose +$SnapshotName = (($Labels | Select-Object MediaDescription -ExpandProperty MediaDescription).Split('|'))[0] +$ArrayName = (($Labels | Select-Object MediaDescription -ExpandProperty MediaDescription).Split('|'))[1] + + + +### +# Start the Restore Process + +# Connect to vCenter +$VIServer = Connect-VIServer -Server $VIServerName -Protocol https -Credential $Credential +$TargetSQLServerVM = Get-VM -Server $VIServer -Name $TargetVM +$VMESXiHost = Get-VMhost -VM $TargetSQLServerVM + + + +# Create a new volume from the selected snapshot of the source +$SnapshotSuffix = (Get-Date).ToString("yyyyMMdd-HHmmss") +$NewClonedVolumeName = "$($SourceVolumeName)-clone-$($SnapshotSuffix)" +$SnapshotSourceVolumeName = $SnapshotName + ".$SourceVolumeName" +New-Pfa2Volume -Array $FlashArray -Name $NewClonedVolumeName -SourceName $SnapshotSourceVolumeName -Overwrite $true + + + +# Present the new volume to the ESXi host group +New-Pfa2Connection -Array $FlashArray -HostGroupName $FAHostGroupName -VolumeName $NewClonedVolumeName + + + +# ESXi host must now rescan storage +Get-VMHostStorage -RescanAllHba -RescanVmfs -VMHost $VMESXiHost + + + +# Connect to EsxCli +$EsxCli = Get-EsxCli -VMHost $VMESXiHost + + + +### Diagnostic +# Retrieve a list of the snapshots that have been presented to the host (our cloned volume should be present) +# $snapInfo = $EsxCli.storage.vmfs.snapshot.list() +# $snapInfo | where-object { ($_.VolumeName -match $SourceDatastoreName) } +# $snapInfo + + + +# Resignature the cloned datastore +$EsxCli.storage.vmfs.snapshot.resignature($SourceDatastoreName) + + + +# Find the newly resignatured datastore name +# NOTE: +# After a datastore is resignatured, its name will be "snap-[GUID chars]-[original DS name]" +# This is why the wildcard match below is needed. +$clonedDatastore = (Get-Datastore | ? { $_.name -match 'snap' -and $_.name -match $SourceDatastoreName }) + +while ($clonedDatastore -eq $null) { + # We may have to wait a little bit before the datastore is fully operational + Start-Sleep -Seconds 5 + $clonedDatastore = (Get-Datastore | Where-Object { $_.name -match 'snap' -and $_.name -match $SourceDatastoreName }) +} + + + +# Must rescan storage again so ESXi hosts(s) can see the new cloned datastore +Get-VMHostStorage -RescanAllHba -RescanVmfs -VMHost $VMESXiHost + + + +######################################## +# Prepare SQL Server & Windows for the +# snapshot overlay operation +######################################## +# Offline the database, which we'd have to do anyway if we were restoring a full backup +$Query = "ALTER DATABASE $DbName SET OFFLINE WITH ROLLBACK IMMEDIATE" +Invoke-DbaQuery -SqlInstance $SqlInstance -Database master -Query $Query + + + +# Offline the volume(s) in Windows +Foreach ($TargetDisk in $TargetDisks) { + Invoke-Command -Session $SqlServerSession -ScriptBlock { Get-Disk | Where-Object { $_.SerialNumber -eq $using:TargetDisk } | Set-Disk -IsOffline $True } +} + + + +# Remove the original VMDK(s), within the original datastore +Foreach ($SourceVMDKPath in $SourceVMDKPaths) { + $harddisk = Get-HardDisk -VM $TargetSQLServerVM | ? { $_.FileName -match $SourceVMDKPath } + Remove-HardDisk -HardDisk $harddisk -Confirm:$false -DeletePermanently +} + + + +# Attach the new VMDK(s) from the newly cloned datastore back to the target VM +Foreach ($SourceVMDKPath in $SourceVMDKPaths) { + $newlyAttachedDisk = New-HardDisk -VM $TargetSQLServerVM -DiskPath "[$($clonedDatastore.Name)] $SourceVMDKPath" +} + + + +# Online the volume(s) in Windows +Foreach ($TargetDisk in $TargetDisks) { + Invoke-Command -Session $SqlServerSession -ScriptBlock { Get-Disk | Where-Object { $_.SerialNumber -eq $using:TargetDisk } | Set-Disk -IsOffline $False } +} + + + +# Restore the database with no recovery, which means we can restore LOG native SQL Server backups +$Query = "RESTORE DATABASE $DbName FROM DISK = '$BackupFile' WITH METADATA_ONLY, REPLACE, NORECOVERY" +Invoke-DbaQuery -SqlInstance $SqlInstance -Database master -Query $Query -Verbose + + + +# Let's check the current state of the database...its RESTORING +Get-DbaDbState -SqlInstance $SqlInstance -Database $DbName + + + +# Restore the log backup. +Restore-DbaDatabase -SqlInstance $SqlInstance -Database $DbName -Path $LogBackup.BackupPath -NoRecovery -Continue + + + +# Online the database +$Query = "RESTORE DATABASE $DbName WITH RECOVERY" +Invoke-DbaQuery -SqlInstance $SqlInstance -Database master -Query $Query + + + +# Verify Restore +Invoke-DbaQuery -SqlInstance $SqlInstance -Database $DbName -Query "SELECT TOP 10 * FROM dbo.Recipes" + + + +######################### +# Begin Clean Up Steps +######################### +$destinationDatastore = Get-Datastore -Name $SourceDatastoreName + + + +# Perform Storage vMotion to move the new VMDK disk(s) to the original source datastore. Should be fast +# thanks to XCOPY +Foreach ($SourceVMDKPath in $SourceVMDKPaths) { + $newlyAttachedDisk = Get-HardDisk -VM $TargetSQLServerVM | ? { $_.FileName -match $SourceVMDKPath } + Move-HardDisk -HardDisk $newlyAttachedDisk -Datastore $destinationDatastore -Confirm:$false +} + + + +# Now that the VMDKs have been moved back to the primary datastore, we can remove the temporary cloned +# datastore - this can take a min or two. +# First, removing from VCenter +Remove-Datastore -Datastore $clonedDatastore -VMHost $VMESXiHost -Confirm:$false + + + +# On FlashArray, disconnect the cloned volume from the ESXi cluster +Remove-Pfa2Connection -Array $FlashArray -HostGroupName $FAHostGroupName -VolumeName $NewClonedVolumeName + + + +# On FlashArray, destroy the cloned volume +Remove-Pfa2Volume -Array $FlashArray -Name $NewClonedVolumeName + + + +# Clean up +Remove-PSSession $SqlServerSession + + + diff --git a/demos-sdk2/Point in Time Recovery - VMFS/README.md b/demos-sdk2/Point in Time Recovery - VMFS/README.md new file mode 100644 index 0000000..ecbc7d2 --- /dev/null +++ b/demos-sdk2/Point in Time Recovery - VMFS/README.md @@ -0,0 +1,40 @@ +# Point In Time Recovery - Using SQL Server 2022's T-SQL Snapshot Backup feature w. VMFS/VMDK datastore/files. + + + +
+ + +# Scenario: +Perform a point in time restore using SQL Server 2022's T-SQL Snapshot Backup feature. This uses a FlashArray snapshot as the base of the restore, then restores a log backup. + +# IMPORTANT NOTE: +This example script is built for 1 database spanned across two VMDK files/volumes from a single datastore. + +The granularity or unit of work for this workflow is a VMDK file(s) and the entirety of its contents. Therefore, everything in the VMDK file(s) including files for other databases will be impacted/overwritten. + +This example will need to be adapted if you wish to support multiple databases on the same set of VMDK(s). + +# Prerequisites: +1. PowerShell Modules: dbatools & PureStoragePowerShellSDK2 + +# Usage Notes: +Each section of the script is meant to be run one after the other. The script is not meant to be executed all at once. + + +
+ + +# Disclaimer: +This example script is provided AS-IS and is meant to be a building block to be adapted to fit an individual organization's infrastructure. +

+_PLEASE_ do not save your passwords in cleartext here. +Use NTFS secured, encrypted files or whatever else -- never cleartext! +

+We encourage the modification and expansion of these scripts by the community. Although not necessary, please issue a Pull Request (PR) if you wish to request merging your modified code in to this repository. + + +
+ + +_The contents of the repository are intended as examples only and should be modified to work in your individual environments. No script examples should be used in a production environment without fully testing them in a development or lab environment. There are no expressed or implied warranties or liability for the use of these example scripts and templates presented by Pure Storage and/or their creators._ diff --git a/demos-sdk2/VMFS-VMDK Snapshot/README.md b/demos-sdk2/VMFS-VMDK Snapshot/README.md index e67b0dd..c66f32d 100644 --- a/demos-sdk2/VMFS-VMDK Snapshot/README.md +++ b/demos-sdk2/VMFS-VMDK Snapshot/README.md @@ -1,46 +1,39 @@ -**VMFS/VMDK + SQL Server Snapshot Scripts** -

-This folder contains VMFS/VMDK + SQL Server example snapshot scripts. - -**Files:** -- VMFS-VMDK Snapshot.ps1 - +# Refresh VMFS VMDK(s) with Snapshot Demo
-**Scenario:** -
This example script shows steps to snapshot a VMFS datastore that contains data & log VMDKs for a SQL Server. The overall scenario is taking a snapshot of a production SQL Server's underlying datastore, and cloning the datastore to then overlay a pre-existing non-production datastore for a non-production SQL Server. - -All references to a "target" refer to the non-production side (VM, datastore, etc). - -**Prerequisites:** -1. The production datastore must already be cloned and presented once, to the non-production side. -2. This script assumes the database(s) are already attached on the target, non-production SQL Server. - -**Important Usage Notes:** -
You must pre-setup the target VM with a cloned datastore from the source already. You will ONLY be utilizing the specific VMDK(s) that contain the data/log files of interest, from the cloned datastore. Also note that the VMFS datastore does not need to only exclusively contain VMDKs for the SQL Server in question. If other VMDKs are present in the datastore, used by the either the source SQL Server VM or other VMs, they do not need to be deleted or otherwise manipulated during this cloning process. Remember FlashArray deduplicates data, thus a clone's set of additional, unused VMDKs will not have a negative impact. - -For the cloned datastore pre-setup, you can use subsets of the code below to clone the source datastore, present it to the target server, then attach the VMDK(s) containing the production databases that will be re-cloned with this script. Once "staged," you can then use this script fully to refresh the data files in the cloned datastore that is attached to the target server. - -When cloning, note that the target datastore is dropped and replaced entirely. This is because when cloning a datastore, it must be resignatured and the datastore will be renamed with a non-deterministic naming scheme (snap-[[GUID chars]]-[[original DS name]]). Thus it is not possible to know what the new datastore name will be until the resignature step is executed. - -This script also assumes that all database files (data and log) are on the same volume/single VMDK. If multiple volumes/VMDKs are being used, you will have to adjust the code (ex: add additional foreach loops for manipulating multiple VMDKs). +# Scenario: +Production SQL Server & database(s) reside on a VMFS datastore. Non-production SQL Server resides on +a different VMFS datastore. User database(s) data and log files reside on two different VMDK disks +in each datastore. +

+Each datastore also resides on a different FlashArray, to demonstrate use of async snapshot replication. +

+This example is for a repeatable refresh scenario, such as a nightly refresh of a production database on +another non-production SQL Server. +

+This example's workflow takes an on-demand snapshot of the Production datastore and async replicates it to +the second FlashArray. Then the snapshot is cloned as a new temporary volume/datastore. The VMDKs with the +production database files, residing on the temporary cloned datastore are attached to the target SQL Server, +replacing the prior VMDKs that stored the database files previously. Finally Storage vMotion is used to +migrate the VMDKs to the non-production datastore, then the temporary cloned datastore is discarded. +

+This workflow is intended to only be impact select Windows Disks/VMDKs that contain user databases. + +# Disclaimer: +This example script is provided AS-IS and is meant to be a building block to be adapted to fit an individual organization's infrastructure. +

+_PLEASE_ do not save your passwords in cleartext here. +Use NTFS secured, encrypted files or whatever else -- never cleartext! +

+We encourage the modification and expansion of these scripts by the community. Although not necessary, please issue a Pull Request (PR) if you wish to request merging your modified code in to this repository.
-**Disclaimer:** -
-This example script is provided AS-IS and meant to be a building block to be adapted to fit an individual organization's infrastructure. -
-
+_The contents of the repository are intended as examples only and should be modified to work in your individual environments. No script examples should be used in a production environment without fully testing them in a development or lab environment. There are no expressed or implied warranties or liability for the use of these example scripts and templates presented by Pure Storage and/or their creators._ -We encourage the modification and expansion of these scripts by the community. Although not necessary, please issue a Pull Request (PR) if you wish to request merging your modified code in to this repository. - -
- -_The contents of the repository are intended as examples only and should be modified to work in your individual environments. No script examples should be used in a production environment without fully testing them in a development or lab environment. There are no expressed or implied warranties or liability for the use of these example scripts and templates presented by Pure Storage and/or their creators._ diff --git a/demos-sdk2/VMFS-VMDK Snapshot/VMFS-VMDK Snapshot.ps1 b/demos-sdk2/VMFS-VMDK Snapshot/VMFS-VMDK Snapshot.ps1 index a1b05c7..797491b 100644 --- a/demos-sdk2/VMFS-VMDK Snapshot/VMFS-VMDK Snapshot.ps1 +++ b/demos-sdk2/VMFS-VMDK Snapshot/VMFS-VMDK Snapshot.ps1 @@ -1,57 +1,91 @@ ############################################################################################################################## -# Refresh VMFS VMDK with Snapshot Demo +# Refresh VMFS VMDK(s) with Snapshot Demo # -# -# Scenario: -# Snapshot and clone a "production" VMDK in a VMFS datastore, then present it to a "non-production" server. +# Example Scenario: +# Production SQL Server & database(s) reside on a VMFS datastore. Non-production SQL Server resides on +# a different VMFS datastore. User database(s) data and log files reside on two different VMDK disks +# in each datastore. # -# This example has two databases: ExampleDb1, ExampleDb2, whose data and log files both reside on a single disk/VMDK. -# -# -# Usage Notes: -# -# You must pre-setup the target VM with a cloned datastore from the source already. You will ONLY be utilizing -# the SPECIFIC VMDK(s) that contain the data/log files of interest, from the cloned datastore. Other VMDKs can safely be -# ignored since they are deduped on FlashArray. -# -# For the cloned datastore pre-setup, you can use subsets of the code below to clone the source datastore, present it to -# the target server, then attach the VMDK(s) containing the production databases that will be re-cloned with this script. -# Once "staged," you can then use this script fully to refresh the data files in the cloned datastore that is attached -# to the target server. +# Each datastore also resides on a different FlashArray, to demonstrate use of async snapshot replication. # -# This script also assumes that all database files (data and log) are on the same volume/single VMDK. If multiple -# volumes/VMDKs are being used, adjust the code to add additional foreach loops when manipulating the VMDKs. +# This example is for a repeatable refresh scenario, such as a nightly refresh of a production database on +# another non-production SQL Server. # +# This example's workflow takes an on-demand snapshot of the Production datastore and async replicates it to +# the second FlashArray. Then the snapshot is cloned as a new temporary volume/datastore. The VMDKs with the +# production database files, residing on the temporary cloned datastore are attached to the target SQL Server, +# replacing the prior VMDKs that stored the database files previously. Finally Storage vMotion is used to +# migrate the VMDKs to the non-production datastore, then the temporary cloned datastore is discarded. +# +# This workflow is intended to only be impact select Windows Disks/VMDKs that contain user databases. +# # Disclaimer: # This example script is provided AS-IS and meant to be a building block to be adapted to fit an individual # organization's infrastructure. +# +# _PLEASE_ do not save your passwords in cleartext here. +# Use NTFS secured, encrypted files or whatever else -- never cleartext! +# ############################################################################################################################## # Import powershell modules -Import-Module PureStoragePowerShellSDK2 Import-Module VMware.VimAutomation.Core -Import-Module SqlServer +Import-Module PureStoragePowerShellSDK2 + + + +# Declare all variables +# VMware variables +$VIServerName = 'vcenter.example.com' +$ClusterName = 'WorkloadCluster1' +$SourceDatastoreName = 'source_sql_datastore' +$TargetDatastoreName = 'target_sql_datastore' +$SourceVMDKPaths = @('source_vm/sqldata.vmdk','source_vm/sqllog.vmdk') +$TargetVMDKPaths = @('target_vm/sqldata.vmdk','target_vm/sqllog.vmdk') + +# FlashArray variables +$SourceArrayName = 'flasharray1.example.com' # FlashArray FQDN +$SourceArrayShortName = 'flasharray1' +$TargetArrayName = 'flasharray2.example.com' # FlashArray FQDN +$FAHostGroupName = 'FAHostGroupName' # HostGroup Name on FlashArray for the ESXi cluster +$SourceVolumeName = 'volume_name' +$SourceProtectionGroup = 'protection_group' +$TargetVolumeName = 'target_volume_name' +$TargetProtectionGroup = "$($SourceArrayShortName):$($SourceProtectionGroup)" # [source array name (not FQDN)]:[source protection group name] + +# Windows/SQL Server variables +$SourceVM = 'source_vm' # Not FQDN +$TargetVM = 'target_vm' # Not FQDN +$Databases = @('AdventureWorks','WideWorldImporters') +$TargetDevices = @('1234c29689bc0888d32dcd2919a67z89', '1234c299721c4ba4a937552fb298a76') # The serial numbers of the Windows volume containing database files; use get-disk + +# Get Credentials - this demo example assumes the same credential for the target VM and vCenter +$Credential = Get-Credential -UserName "$env:USERNAME" -Message 'Enter your credential information...' -# Declare variables -$TargetVM = 'SqlServer1' # Name of target VM -$Databases = @('ExampleDb1','ExampleDb2') # Array of database names -$TargetDiskSerialNumber = '6000c02022cb876dcd321example01b' # Target Disk Serial Number -$VIServerName = 'vcenter.example.com' # vCenter FQDN -$ClusterName = 'WorkloadCluster1' # VMware Cluster -$SourceDatastoreName = 'vmware_sql_datastore' # VMware datastore name -$SourceVMDKPath = 'SqlServer1_1/SqlServer1.vmdk' # VMDK path inside the VMFS datastore -$ArrayName = 'flasharray1.example.com' # FlashArray FQDN -$SourceVolumeName = 'sql_volume_1' # Source volume name on FlashArray (may be same as your datastore name) -$TargetVolumeName = 'sql_volume_2' # Target volume name on FlashArray (may be same as your datastore name) +# Connect to the source array +$FlashArray = Connect-Pfa2Array -Endpoint $SourceArrayName -Credential ($Credential) -IgnoreCertificateError -# Set Credential - this assumes the same credential for the target VM and vCenter -$Credential = Get-Credential + + +# Create an on-demand Protection Group snapshot +# NOTE: +# This example uses async replication to generate a snapshot on $SourceArrayName and +# replicate it to $TargetArrayName. Remove -Replication flag if snapshots are only +# being used on local array. +# Alternatively, you may substitute other code to select an existing snapshot here +$MostRecentSnapshot = New-Pfa2ProtectionGroupSnapshot -Array $FlashArray -SourceNames $SourceProtectionGroup -ApplyRetention $true -ReplicateNow $true +$MostRecentSnapshot + + + +### +# Prepare for snapshot overlay @@ -60,109 +94,193 @@ $TargetVMSession = New-PSSession -ComputerName $TargetVM -Credential $Credential +# Import the SQLPS module so SQL commands are available +Import-Module SQLPS -PSSession $TargetVMSession -DisableNameChecking + + + # Connect to vCenter $VIServer = Connect-VIServer -Server $VIServerName -Protocol https -Credential $Credential +$TargetSQLServerVM = Get-VM -Server $VIServer -Name $TargetVM +$VMESXiHost = Get-VMhost -VM $TargetSQLServerVM -# Offline the target database(s) by looping through $Databases array -foreach ($Database in $Databases) { - $Query = "ALTER DATABASE [$Database] SET OFFLINE WITH ROLLBACK IMMEDIATE" - Invoke-Sqlcmd -ServerInstance $TargetVM -Database master -Query $Query -} +# Get discrete hosts connected to the ESXi cluster +$Hosts = Get-Cluster $ClusterName | Get-VMHost | where-object { ($_.ConnectionState -eq 'Connected') } -# Offline the volumes that have SQL data -Invoke-Command -Session $TargetVMSession -ScriptBlock { Get-Disk | Where-Object { $_.SerialNumber -eq $using:TargetDiskSerialNumber } | Set-Disk -IsOffline $True } +# Connect to the target array, authenticate. Remember disclaimer at the top! +$FlashArray = Connect-Pfa2Array -Endpoint $TargetArrayName -Credential ($Credential) -IgnoreCertificateError -# Prepare to remove the VMDK from the VM -$VM = Get-VM -Server $VIServer -Name $TargetVM -$HardDisk = Get-HardDisk -VM $VM | Where-Object { $_.FileName -match $SourceVMDKPath } +# Get the most recent snapshot +# NOTE: +# This next segment may be simplified if async snapshot replication is not being used. +# Alternatively, substitute code to take a new protection group snapshot or use +# one created prior. +$MostRecentSnapshots = Get-Pfa2ProtectionGroupSnapshot -Array $FlashArray -Name $TargetProtectionGroup | Sort-Object created -Descending | Select-Object -Property name -First 5 -# Remove the VMDK from the VM -Remove-HardDisk -HardDisk $HardDisk -Confirm:$false +# Check that the last snapshot has been fully replicated +$FirstSnapStatus = Get-Pfa2ProtectionGroupSnapshotTransfer -Array $FlashArray -Name $MostRecentSnapshots[0].name +if ($FirstSnapStatus.completed -ne $null) { # If $FirstSnapStatus.completed, then it hasn't been fully replicated + $MostRecentSnapshot = $MostRecentSnapshots[0].name +} +else { + # Use prior snapshot instead + $MostRecentSnapshot = $MostRecentSnapshots[1].name +} -# Prepare to remove the stale datastore -$DataStore = $HardDisk.Filename.Substring(1, ($HardDisk.Filename.LastIndexOf(']') - 1)) -$Hosts = Get-Cluster $ClusterName | Get-VMHost | Where-Object { ($_.ConnectionState -eq 'Connected') } +# Create a new volume from the selected snapshot of the source +$SnapshotSuffix = (Get-Date).ToString("yyyyMMdd-HHmmss") +$NewClonedVolumeName = "$($SourceVolumeName)-repl-clone-$($SnapshotSuffix)" +$ReplicatedSourceVolumeName = "$($MostRecentSnapshot).$($SourceVolumeName)" +New-Pfa2Volume -Array $FlashArray -Name $NewClonedVolumeName -SourceName $ReplicatedSourceVolumeName -Overwrite $true -# Guest hard disk removed, now remove the stale datastore - this can take a min or two -Get-Datastore $DataStore | Remove-Datastore -VMHost $Hosts[0] -Confirm:$False + + +# Present the new volume to the ESXi host group. +New-Pfa2Connection -Array $FlashArray -HostGroupName $FAHostGroupName -VolumeName $NewClonedVolumeName + + + +# ESXi host(s) must now rescan storage +Get-VMHostStorage -RescanAllHba -RescanVmfs -VMHost $VMESXiHost -# Connect to the array, authenticate. Remember disclaimer at the top! -$FlashArray = Connect-Pfa2Array -Endpoint $ArrayName -Credential ($Credential) -IgnoreCertificateError +# Connect to EsxCli +$esxcli = Get-EsxCli -VMHost $Hosts[0] + +### Diagnostic +# Retrieve a list of the snapshots that have been presented to the host (our cloned volume should be present) +# $snapInfo = $esxcli.storage.vmfs.snapshot.list() +# $snapInfo | where-object { ($_.VolumeName -match $SourceDatastoreName) } +# $snapInfo -# Perform the volume overwrite (no intermediate snapshot needed!) -New-Pfa2Volume -Array $FlashArray -Name $TargetVolumeName -SourceName $SourceVolumeName -Overwrite $True +# Resignature the cloned datastore +$esxcli.storage.vmfs.snapshot.resignature($SourceDatastoreName) + -# Rescan storage on each ESX host in the $Hosts array -foreach ($VmHost in $Hosts) { - Get-VMHostStorage -RescanAllHba -RescanVmfs -VMHost $VmHost | Out-Null + +# Find the newly resignatured datastore name +# NOTE: +# After a datastore is resignatured, its name will be "snap-[GUID chars]-[original DS name]" +# This is why the wildcard match below is needed. +$clonedDatastore = (Get-Datastore | ? { $_.name -match 'snap' -and $_.name -match $SourceDatastoreName }) + +while ($clonedDatastore -eq $null) { + # We may have to wait a little bit before the datastore is fully operational + Start-Sleep -Seconds 5 + $clonedDatastore = (Get-Datastore | Where-Object { $_.name -match 'snap' -and $_.name -match $SourceDatastoreName }) } +# $clonedDatastore -# Connect to EsxCli -$esxcli = Get-EsxCli -VMHost $Hosts[0] +# Must rescan storage again so ESXi hosts(s) can see the new cloned datastore +Get-VMHostStorage -RescanAllHba -RescanVmfs -VMHost $VMESXiHost -# Resignature the cloned datastore -$EsxCli.Storage.Vmfs.Snapshot.Resignature($SourceDatastoreName) +### +# Prep SQL & Windows for VMDK overlay + +# Offline the target database(s) in SQL Server by looping through $Databases array +Foreach ($database in $Databases) { + # Offline the database + $Query = "ALTER DATABASE " + $($database) + " SET OFFLINE WITH ROLLBACK IMMEDIATE" + Invoke-Command -Session $TargetVMSession -ScriptBlock {Param($querytask) Invoke-Sqlcmd -ServerInstance . -Database master -Query $querytask} -ArgumentList ($Query) +} -# Find the assigned datastore name, this may take a few seconds -# NOTE: when a datastore comes back, it's name will be "snap-[GUID chars]-[original DS name]" -# This is why the wildcard match below is needed. -$DataStore = (Get-Datastore | Where-Object { $_.Name -match 'snap' -and $_.Name -match $SourceDatastoreName }) +# Offline the volumes that have SQL data in Windows by looping through $TargetDevices array +Foreach ($targetdevice in $TargetDevices) { + Invoke-Command -Session $TargetVMSession -ScriptBlock {Param($currentdisk) Get-Disk | ? { $_.SerialNumber -eq $($currentdisk) } | Set-Disk -IsOffline $True } -ArgumentList ($targetdevice) +} -# Rescan storage again to make sure all hosts can see the new datastore -foreach ($VmHost in $Hosts) { - Get-VMHostStorage -RescanAllHba -RescanVmfs -VMHost $VmHost | Out-Null +# Remove the VMDK(s) with stale database files from the VM +Foreach ($TargetVMDKPath in $TargetVMDKPaths) { + $harddisk = Get-HardDisk -VM $TargetSQLServerVM | ? { $_.FileName -match $TargetVMDKPath } + Remove-HardDisk -HardDisk $harddisk -Confirm:$false -DeletePermanently } # Attach the VMDK from the newly cloned datastore back to the target VM -New-HardDisk -VM $VM -DiskPath "[$DataStore] $SourceVMDKPath" +Foreach ($SourceVMDKPath in $SourceVMDKPaths) { + $newlyAttachedDisk = New-HardDisk -VM $TargetSQLServerVM -DiskPath "[$($clonedDatastore.Name)] $SourceVMDKPath" +} -# Online the volume on the target VM -Invoke-Command -Session $TargetVMSession -ScriptBlock { Get-Disk | Where-Object { $_.SerialNumber -eq $using:TargetDiskSerialNumber } | Set-Disk -IsOffline $False } +# Online the volume(s) on the target VM by looping through $TargetDevices array +Foreach ($targetdevice in $TargetDevices) { + Invoke-Command -Session $TargetVMSession -ScriptBlock {Param($currentdisk) Get-Disk | Where-Object { $_.SerialNumber -eq $($currentdisk) } | Set-Disk -IsOffline $False } -ArgumentList ($targetdevice) +} # Volume might be read-only, ensure it's read/write -Invoke-Command -Session $TargetVMSession -ScriptBlock { Get-Disk | Where-Object { $_.SerialNumber -eq $using:TargetDiskSerialNumber } | Set-Disk -IsReadOnly $False } +Foreach ($targetdevice in $TargetDevices) { + Invoke-Command -Session $TargetVMSession -ScriptBlock {Param($currentdisk) Get-Disk | Where-Object { $_.SerialNumber -eq $($currentdisk) } | Set-Disk -IsReadOnly $False } -ArgumentList ($targetdevice) +} # Online the target database(s) by looping through $Databases array -foreach ($Database in $Databases) { - $Query = "ALTER DATABASE [$Database] SET ONLINE WITH ROLLBACK IMMEDIATE" - Invoke-Sqlcmd -ServerInstance $TargetVM -Database master -Query $Query +Foreach ($database in $databases) { + $Query = "ALTER DATABASE " + $($database) + " SET ONLINE WITH ROLLBACK IMMEDIATE" + Invoke-Command -Session $TargetVMSession -ScriptBlock {Param($querytask) Invoke-Sqlcmd -ServerInstance . -Database master -Query $querytask} -ArgumentList ($Query) +} + + + +### +# Databases should now be online and usable +# Start cleanup next + + + +# Perform Storage vMotion to move the new VMDK disk(s) to the original source datastore. +$destinationDatastore = Get-Datastore -Name $TargetDatastoreName + +Foreach ($SourceVMDKPath in $SourceVMDKPaths) { + $newlyAttachedDisk = Get-HardDisk -VM $TargetSQLServerVM | ? { $_.FileName -match $SourceVMDKPath } + Move-HardDisk -HardDisk $newlyAttachedDisk -Datastore $destinationDatastore -Confirm:$false } -# Remove powershell session +# Guest hard disk removed, now remove the stale datastore - this can take a min or two +Remove-Datastore -Datastore $clonedDatastore -VMHost $Hosts[0] -Confirm:$false + + + +# On FlashArray, disconnect the cloned volume from the ESXi cluster +Remove-Pfa2Connection -Array $FlashArray -HostGroupName $FAHostGroupName -VolumeName $NewClonedVolumeName + + + +# On FlashArray, destroy the cloned volume +Remove-Pfa2Volume -Array $FlashArray -Name $NewClonedVolumeName + + + +# Clean up Remove-PSSession $TargetVMSession