Application Consistency with Pure Storage VSS Provider

With the release of the Purity Operating Environment 4.0 comes the new Pure Storage Volume Shadow Copy Service Provider (Pure VSS Provider). The Pure VSS Provider is a new plugin we are providing to take application consistent snapshots for Microsoft applications such as SQL Server, SharePoint and Exchange. It is important to set a foundation on the different types of backups before delving into the inner workings of the Pure VSS Provider.

Inconsistent Backup
This is probably the oldest type of backup. The process is simple, backup software starts at the beginning of a file structure and copies all of the data until it reaches the end, resulting in a backup. What can make this backup inconsistent is that if any user added/modified after a backup but before it completed that would result in an inconsistent backup; the files in the backup are not consistent. For databases these types of backups do not provide adequate protection because they only capture what is on disk. Example, with SQL Server there are a several different files, primary (MDF), log (LDF) and secondary (NDF) files and there could be I/O (transactions) that are still in memory. The inconsistent backup method only captures what is on disk and in the case of a database there may be transactions still in memory.

Crash-Consistent Backup
A crash-consistent backup is where all data is captured at the same point in time. For all things not database related this method of backup should suffice for most recovery situations. A crash-consistent backup, as with an inconsistent backup, does not capture any data that is in memory or any pending I/O transactions. The term “crash” is used because if the data is recovered using this method it would be equivalent to restoring to the exact moment that the server had crashed. Crash-consistent backups can be used but it is important to perform the correct operational tasks to ensure you are restoring properly. For example with SQL Server you may need to replay transaction logs to get the database back to a consistent transactional state.

Application-Consistent Backup
Application-consistent backups provide the highest level of protection and consistency, it captures what is on disk and memory. For Microsoft Windows this is achieved using the Volume Shadow Copy Service (VSS) which freezes I/O, flushes everything to disk and takes a block-level snapshot of the volume. VSS is provided as a framework in the Windows operating system that performs these operations by coordinating between Requester, Writer and Provider. When a VSS request is initiated a VSS Writer (Eg. SqlServerWriter) will flush all I/O to ensure that the database is in a consistent state. Then the VSS (Hardware) Provider will take a block-level snapshot of the volume. Once this is completed the Provider notifies the Writer to resume operations.

Today the Purity Operating Environment takes crash consistent snapshots, but now with the introduction of the Pure VSS Provider we can now provide Application-Consistent Backups for Microsoft server products like SQL Server.

The Pure VSS Provider uses Diskshadow.exe as our out-of-box VSS Requester. Other requesters can be used as long as they adhere to the implementation guidelines of the VSS SDK; for example Symantec NetBackup which we have tested in our labs.

The process by which the Volume Shadow Copy Service works is illustrated below:

VSS_workflow

Steps in a VSS workflow* are as follows:

  1. The requester asks the Volume Shadow Copy Service to enumerate the writers, gather the writer metadata, and prepare for shadow copy creation.
  2. Each writer creates an XML description of the components and data stores that need to be backed up and provides it to the Volume Shadow Copy Service. The writer also defines a restore method, which is used for all components. The Volume Shadow Copy Service provides the writer’s description to the requester, which selects the components that will be backed up.
  3. The Volume Shadow Copy Service notifies all the writers to prepare their data for making a shadow copy.
  4. Each writer prepares the data as appropriate, such as completing all open transactions, rolling transaction logs, and flushing caches. When the data is ready to be shadow-copied, the writer notifies the Volume Shadow Copy Service.
  5. The Volume Shadow Copy Service tells the writers to temporarily freeze application write I/O requests (read I/O requests are still possible) for the few seconds that are required to create the shadow copy of the volume or volumes. The application freeze is not allowed to take longer than 60 seconds. The Volume Shadow Copy Service flushes the file system buffers and then freezes the file system, which ensures that the file system metadata is recorded correctly and the data to be shadow-copied is written in a consistent order.
  6. The Volume Shadow Copy Service tells the provider to create the shadow copy. The shadow copy creation period lasts no more than 10 seconds, during which all write I/O requests to the file system remain frozen.
  7. The Volume Shadow Copy Service releases file system write I/O requests.
  8. VSS tells the writers to thaw application write I/O requests. At this point applications are free to resume writing data to the disk that is being shadow-copied.
  9. The requester can retry the process (go back to step 1) or notify the administrator to retry at a later time.
  10. If the shadow copy is successfully created, the Volume Shadow Copy Service returns the location information for the shadow copy to the requester.

The new Pure VSS Provider can be installed from the Web Management GUI > System > Plugins > VSS. Simply download the 1.0.0 package and install on the individual Microsoft Window Server hosts. It is also possible to use other software deployment tools to install on a large volume of systems.

VSS_System_Plugins

Once the Pure VSS Provider has been installed use Diskshadow.exe from an elevated Administrator prompt to ensure the provider is visible.

DISKSHADOW > list providers

diskshadow

All of the commands can be run directly from Diskshadow but seeing that I have this obsession with Windows PowerShell I have created the following scripts that automates Diskshadow.

New-PureVolShadowCopy.ps1

[CmdletBinding()]
Param(
    [Parameter(Mandatory=$True)][string]$Volume,
    [Parameter(Mandatory=$True)][string]$ScriptName,
    [Parameter(Mandatory=$True)][string]$MetadataFile,
    [Parameter(Mandatory=$True)][string]$ShadowCopyAlias,
    [Parameter(Mandatory=$True)][string]$VerboseMode = "ON"
    )
Remove-Item DEMO.dsh -ErrorAction SilentlyContinue
Remove-Item DEMO.CAB -ErrorAction SilentlyContinue
$dsh = "./$ScriptName.dsh"
"RESET",
"SET CONTEXT PERSISTENT",
"SET OPTION TRANSPORTABLE",
"SET METADATA $MetadataFile",
"SET VERBOSE $VerboseMode",
"BEGIN BACKUP",
"ADD VOLUME $Volume ALIAS $ShadowCopyAlias PROVIDER {781c006a-5829-4a25-81e3-d5e43bd005ab}",
"CREATE",
"END BACKUP" | Set-Content $dsh
DISKSHADOW /s $dsh
Remove-Item $dsh

Example:

.\New-PureVolShadowCopy.ps1 -Volume G: -ScriptName DEMO -MetadataFile DEMO -ShadowCopyAlias DEMO -VerboseMode ON

VSS_PowerShell_Example

Get-PureVolShadowCopy.ps1

[CmdletBinding()]
Param(
    [Parameter(Mandatory=$True)][string]$MetadataFile,
    [Parameter(Mandatory=$True)][string]$ShadowCopyAlias,
    [Parameter(Mandatory=$True)][string]$ExposeAs
    )

$dsh = "./$ScriptName.dsh"
"RESET",
"LOAD METADATA $MetadataFile",
"IMPORT",
"EXPOSE %$ShadowCopyAlias% $ExposeAs",
"EXIT" | Set-Content $dsh
DISKSHADOW /s $dsh
Remove-Item $dsh

Example:

.\Get-PureVolShadowCopy.ps1 -MetadataFile DEMO1 -ShadowCopyAlias DEMO1 -ExposeAs O:

VSS_PowerShell_Example2

References

Questions or comments message me @themsftdude.

Happy snapping!
barkz

  • Jorge Salinas

    Do you think that maybe in the future, the PureStorage Array could shoot those scripts automatically for me? Or ate you Guys planning to build some backup features that support this kinda of functionality?

    • Hi Jorge – There are ways of automating the scripts on a per host basis because VSS is a host based technology. Install VSS whether that be a Hardware Provider, Requester and/or specific Writer and then automate with basic cmd scripts, PowerShell, etc. We offer the Hardware Provider today and we are always evaluating new feature based on working with our customers. To do what you are referring to our FlashArray would need to be the “Requester” and we don’t support that model today. Hopefully that answers the question.

      Thanks,
      barkz

  • Jorge Salinas

    What about Windows VMs running on top of a hypervisor? Lets suppose I have a SQL Server VM running on TOP of VMware ESXi….. Do I have to run the script from inside the VM? Is there any difference if the hypervisor is Hyper-V, should I have to run the scripts inside each VMs or would it be enough If I ran the scripts from the Host in order to get an APP Consistent Backup?

    • Hi Jorge –

      There are a few options here:
      (1) vSphere — Taking an Application Consistent backup with the Pure Storage VSS HW Provider requires that the attached volume be a Raw Device Map (RDM) so that we can quiesce and snapshot the volume vs a datastore. If you want Application Consistency from VMware you would be using their software provider and it would be quiescing the entire VM.
      (2) Hyper-V — Similar to vSphere a connected volume to a Hyper-V VM would need to be a Pass-Through-Disk (PTD) in order to quiesce and snapshot the volume vs VHD/VHDX.

      Both options above require that the VSS Hardware Provider needs to be installed within the Guest OS (aka Guest VM). Once that is done there is a Pure Storage configuration utility to setup a connection to a FlashArray and then diskshadow (comes with Windows) can be used to take VSS snapshots.

      Hope that helps,
      barkz

  • Pingback: Using Protection Groups with VSS Snapshots | Ps//PowerShellGuy()

  • Stefano

    Hi Jorge, Can we mount the contents of the snap to another server for backup?
    We need to move the load of backup to a dedicate server. Thanks

    • Barkz

      A snapshot can be mounted to a different server and then used for backup. An example would be to take a crash consistent or application consistent snapshot of a volume that contains a SQL Server database and then create a new volume from the snapshot then connect to a different host and run a SQL Server backup against that copy.

      Hope that helps,
      Barkz