Pete Hinchley: How to Avoid Re-Caching an Outlook OST Over a WAN

Imagine you have a shit-ton of users working out of a remote office on the other side of a high-latency, low-bandwidth WAN link. These hypothetical users are currently using Outlook 2010 in "cached mode" to access an Exchange server hosted out of Head Office. Over the course of several years, each user has accumulated a mail cache with gigabytes of data. And now you've been asked to deploy Windows 10 to every computer, and in the process, upgrade from Outlook 2010 to Outlook 2016.

You quickly realise that it isn't possible to migrate existing mail caches (OSTs) to Outlook 2016, and the prospect of every user in the office needing to recache their entire mailbox... well, it scares the shit out of you. The WAN link would grind to a halt for months. It would be hell. So, what to do?

Configuring Outlook in "online mode", or suggesting users switch to OWA, isn't going to cut it. The slider feature in Outlook 2016 could be used to limit the size of each user's cache (e.g. only download the last month's worth of content), but this really just kicks the ball down the road. And besides, there are other issues with this approach, such as not being able to access content older than the configured cut-off period when using a shared mailbox.

Someone, somewhere, mentioned something about getting every user to suck the entire content of their mailbox into a PST, and then reversing the process after upgrading to Outlook 2016, but that doesn't really solve the problem. Instead of pulling gigabytes of data from Exchange to Outlook, you are now just pushing it the other way. The end result is the same: a busted WAN.

What to do? Well, what if you used an Outlook 2016 client in Head Office to create a new mail cache for each user. You could transfer the pre-generated OST files to the remote office on portable media, and prior to each user launching Outlook after their computer is re-imaged, inject the relevant OST into their profile. Hmmm. Putting aside the crazy-ass logistics of such an approach, the idea has merit. So how to make it work...

Before I get into the details, I'll throw one more cat into the bag. Outlook will not allow you to re-use the same OST across multiple computers. You can take an OST from computer 1 and reuse it on computer 2, but if you then try and use the same OST on computer 3, or if you re-open Outlook back on computer 1, the OST will be flagged as corrupt.

Now, in the context of our hypothetical scenario, we have several users that roam between computers, and we want to ensure that these users aren't forced to re-download their entire mailbox when they log onto a second or third computer. This means the process we adopt for creating the OST files at Head Office will need to cater for generating multiple files for each user. And you thought this was going to be easy :)

To get the ball rolling, you will need a Windows 10 computer at Head Office with Office 2016 installed (and configured to use "cache mode"). You should then:

The script performs the following tasks for each mailbox within the input file (for the requested number of iterations of each mailbox).

In the course of developing the script, I originally intended to use the SynObject.SyncEnd event (accessed using the "outlook.application" COM object) to detect when Outlook had finished caching a user's mailbox. Unfortunately this event is only triggered following a manual synchronisation. Looking further afield, I decided to use the Microsoft Windows Automation API.

I use the API to read text from the Outlook status bar, and close the client when the message "All folders are up to date" is detected. This code is encapsulated within the get-syncstatus function.

I should point out that the cachemail.ps1 script can also be called using a switch of -refresh. In this mode, in addition to creating new OST files, the script will update files from a previous execution of the script. For example, if two OST files were previously created for the mailbox megan@lab.hinchley.net, rerunning the script with the refresh switch will ensure that both files are resynchronised (i.e. new content added). If the input file was modified prior to using the refresh switch, and three instead of two cache files were requested, both existing cache files would be updated, and one new cache added. Similarly, if the input file was updated to only request a single OST, calling the script with the refresh switch would result in only one of the OST files being updated.

Anyway, here is the code. Even if you don't need to pre-cache a truck-load of Exchange mailboxes, the script should still demonstrate the power of the Windows Automation API.

param([switch]$refresh)

# usage:
# 1. create new mail cache for users in input file.
#   powershell -file cachemail.ps1
# 2. refresh mail cache for users in input file.
#   powershell -file cachemail.ps1 -refresh
# notes:
# - code is dependent on existence of c:\windows\profiler.exe
#   refer to http://hinchley.net/2016/03/25/create-an-outlook-2016-mapi-profile-with-preconfigured-ost/
# - input file is in the format: <email address>,<number of cache files to create>
#   for example:
#   peter@lab.hinchley.net,1
#   megan@lab.hinchley.net,2

# initialisation.
[void][System.Reflection.Assembly]::LoadWithPartialName("UIAutomationClient")
[void][System.Reflection.Assembly]::LoadWithPartialName("UIAutomationTypes")
[void][System.Reflection.Assembly]::LoadWithPartialName("UIAutomationProvider")
[void][System.Reflection.Assembly]::LoadWithPartialName("UIAutomationClientsideProviders")

# input file.
$input = "users.txt"

# output folder.
$output = "C:\OSTs"

# log file.
$log = "audit.log"

# polling interval (seconds).
$poll = 30

# check if input file exists.
if (! (test-path $input)) { write-host "Input file does not exist."; return }

# create output folder.
if (! (test-path $output)) { new-item -type directory $output -ea silentlycontinue }
if (! (test-path $output)) { write-host "Output folder could not be created."; return }

# logging.
function log($message) {
  "{0}`t{1}" -f (get-date -Format yyyy.MM.dd-HH.mm), $message | out-file -append $log
}

# get sync status. return true if synchronised, null if not, and false on error.
function get-syncstatus($id) {
  $root = [Windows.Automation.AutomationElement]::RootElement
  $condition = New-Object Windows.Automation.PropertyCondition([Windows.Automation.AutomationElement]::ProcessIdProperty, $id)
  $element = $root.FindFirst([Windows.Automation.TreeScope]::Children, $condition)

  $condition1 = New-Object Windows.Automation.PropertyCondition([Windows.Automation.AutomationElement]::ClassNameProperty, "NetUInetpane")
  $condition2 = New-Object Windows.Automation.PropertyCondition([Windows.Automation.AutomationElement]::NameProperty, "Status Bar")

  $condition = New-Object Windows.Automation.AndCondition($condition1, $condition2)
  $statusbar = $element.FindFirst([Windows.Automation.TreeScope]::Descendants, $condition)

  $condition = New-Object Windows.Automation.PropertyCondition([Windows.Automation.AutomationElement]::ClassNameProperty, "NetUISimpleButton")
  $elements  = $statusbar.FindAll([Windows.Automation.TreeScope]::Descendants, $condition)

  if ($elements.count -le 0) { return $false }

  $elements | %{ if ($_.current.name -eq 'Progress All folders are up to date.') { return $true } }
}

function clean-profile() {
  $local = "{0}\Microsoft\Outlook" -f $env:localappdata
  $roaming = "{0}\Microsoft\Outlook" -f $env:appdata
  remove-item $local -recurse -force -ea silentlycontinue
  remove-item $roaming -recurse -force -ea silentlycontinue
  remove-item HKCU:SOFTWARE\Microsoft\Office\16.0\Outlook\Profiles\Outlook -recurse -force -ea silentlycontinue
}

function cache-mail($email, $file) {
  # get qualified path to ost file.
  $path = join-path $global:output $file

  # create the empty mapi profile.
  & C:\Windows\profiler.exe Outlook $email $path | out-null

  # start outlook.
  $process = [Diagnostics.Process]::Start("C:\Program Files (x86)\Microsoft Office\Office16\outlook.exe")

  while ($true) {
    # check the synchronisation status.
    start-sleep -seconds $poll
    $status = get-syncstatus $process.id

    # not finished... will check again.
    if ($status -eq $null) { continue }

    # finished... log status.
    switch ($status) {
      $true  { log "Synchronised mail for $email." }
      $false { log "Failed to synchronise mail for $email." }
    }

    # attempt to exit cleanly.
    [void]$process.closemainwindow()
    start-sleep -seconds 2

    # kill outlook if still running.
    if (get-process -id $process.id -ea silentlycontinue) { [void]$process.kill() }

    # wait for outlook to release handles.
    start-sleep -seconds 2

    return
  }
}

get-content $input | %{
  # parse input file.
  $token = $_.split(',')
  $email = $token[0].trim()
  $count = @(1, [int]$token[1])[$token.count -gt 1]

  $files = @()

  # get existing ost file names for the user.
  if ($refresh) { $files = get-childitem -path $output -filter "$email*.ost" | select -expand name }

  # additional files to create after refresh of existing files.
  $count -= $files.count

  # create/refresh ost files for the user.
  while ($count-- -gt 0) { $files += "{0}-{1}.ost" -f $email, [string]([guid]::newguid().guid) }

  # start caching...
  $files | %{ clean-profile; cache-mail $email $_ }
}

At this point you have a mountain of pre-staged OST files. Now what?

The script has a few prerequisites. Firstly, you will need to modify the value of the $backup variable to point to the file share containing the OST files. Secondly, the script assumes the profiler.exe utility exists on each computer under C:\Windows (e.g. you've included the utility in the Windows 10 image). Of course, you can reference the program from another location if necessary.

So how does it work? When the script executes, it checks to see if the current user has an existing MAPI profile. Assuming a profile doesn't exist (i.e the user hasn't previously logged onto the computer, at least hasn't launched Outlook), the script will then check to see if there are any OST files for the user within the configured network share. If there are, the script will call profiler.exe to create a new MAPI profile for the user, and then move one of the OST files from the network share into the user's local profile. When the user launches Outlook, the profile will be preconfigured to use the OST, and only the content delivered to the mailbox since the mail cache was originally created will need to be synchronised. This process is repeated each time a user logs onto a new computer, at least until all pre-created OST files are exhausted.

Anyway, here is the "client-side" code:

param([switch]$replace)

# usage:
# 1. restore a previously created mail cache for the current user.
#    will do nothing if an existing mail cache exists.
#   powershell -file restoremail.ps1
# 2. restore a previously created mail cache for the current user.
#    will overwrite an existing mail profile.
#   powershell -file restoremail.ps1 -replace

# path to mapi profile in registry.
$mapi = "SOFTWARE\Microsoft\Office\16.0\Outlook\Profiles\Outlook"

# path to cache files. change me!
$backup = "\\server\ost"

# local outlook folder in profile.
$local = "{0}\Microsoft\Outlook" -f $env:localappdata

# roaming outlook folder in profile.
$roaming = "{0}\Microsoft\Outlook" -f $env:appdata

function get-email() {
  try {
    if ($user = ([adsisearcher]"samaccountname=$($env:username)").findone().getdirectoryentry()) { return $user.mail }
  } catch {}
}

function get-cache($email) {
  if ($caches = @(get-childitem $global:backup -filter "$email*.ost" -ea silentlycontinue | select-object -expandproperty fullname)) { return $caches[0] }
}

function clean-profile() {
  remove-item $global:local -recurse -force -ea silentlycontinue
  remove-item $global:roaming -recurse -force -ea silentlycontinue
  remove-item HKCU:$global:mapi -recurse -force -ea silentlycontinue
}

# get user's email address.
if (! ($email = get-email)) { return }

# do nothing if there are no existing cache files.
if (! ($cache = get-cache)) { return }

# clean the existing profile if using "replace" mode.
if ($replace) { clean-profile }

# path to the new ost.
$ost = [io.path]::combine($local, $mail, '.ost')

# create the empty mapi profile.
& C:\Windows\profiler.exe Outlook $email $ost | out-null

# move the cache.
move-item $cache $ost