Frequent MSExchangeTransport 15004 events on Ex2k13 Sp1

Exchange 2013 Sp1 comes with built-in overload failsafe functionality that will either temporarily slow down mail flow (implement “tar pits”) or temporarily halt inbound and outbound mail delivery when certain key performance indicators get above the “Normal” levels (i.e. reach “Medium” or “High” levels).  This is to avoid an outage scenario (or DOS attacks) where the server is overrun by a sudden massive spike of delivery requests (f.x. mass mailing a large attachment to several hundred users).

Exchange will then proceed to process the items it already has in the queue and remove the tarpitting or resume the halted mailflow once the performance counters drop down to “Normal” or “Medium” levels.
At “Medium” you will mostly be seeing external delivery and reception being affected – at “High” both internal and external delivery and reception will be affected.

If you’re seeing multiple MSExchangeTransport 15004 and 15005 events on your Exchange 2013 Sp1 system with the component reporting an increase above “Normal” being Version Buckets then you should consider running the Fixit tool from KB2938053 (which essentially contains a Powershell script that makes format changes rather than being a hotfix binary) and monitor the I/O levels of the disk(s) that host the transport queues and logs.

Note: the formatting change that running the Fixit tool implements has a scope beyond what is indicated in the KB – Exchange itself uses the same formatting functionality in .NET that third-party transport agents use (which is presumably also why this KB is listed at the top of the Exchange 2013 SP1 download link on http://support.microsoft.com/kb/2926248)

 Ex2k13Sp1-postfix

Download Fixit tool on http://support.microsoft.com/kb/2938053/en-gb

 

Back Pressure

 

Back Pressure [and Tarpitting explained]
http://technet.microsoft.com/en-us/library/bb201658(v=exchg.150).aspx

Version Buckets Explained
http://blogs.technet.com/b/exchange/archive/2006/04/19/425722.aspx

Troubleshooting MSExchangeTransport Service Events
http://technet.microsoft.com/en-us/library/bb397220(v=EXCHG.80).aspx

 

Symptoms:

15004 – Exchange automatically halts mail transport when performance indicators increase to “High”:

TimeCreated  : 5/8/2014 3:50:34 PM
ProviderName : MSExchangeTransport
Id           : 15004
Message      : The resource pressure increased from Normal to High.

The following resources are under pressure:
Version buckets = 205 [High] [Normal=80 Medium=120 High=200]

               The following components are disabled due to back pressure:
               Inbound mail submission from Hub Transport servers
               Inbound mail submission from the Internet
               Mail submission from Pickup directory
               Mail submission from Replay directory
               Mail submission from Mailbox server
               Mail delivery to remote domains
               Content aggregation
               Mail resubmission from the Message Resubmission component.
               Mail resubmission from the Shadow Redundancy Component

The following resources are in normal state:
Queue database and disk space (“C:\Program Files\Microsoft\Exchange
Server\V15\TransportRoles\data\Queue\mail.que”) = 45% [Normal] [Normal=95% Medium=97% High=99%]
Queue database logging disk space (“C:\Program Files\Microsoft\Exchange
Server\V15\TransportRoles\data\Queue\”) = 45% [Normal] [Normal=95% Medium=97% High=99%]
Private bytes = 4% [Normal] [Normal=71% Medium=73% High=75%]
Physical memory load = 65% [limit is 94% to start dehydrating messages.]
Submission Queue = 0 [Normal] [Normal=2000 Medium=4000 High=10000]
Temporary Storage disk space (“C:\Program Files\Microsoft\Exchange
Server\V15\TransportRoles\data\Temp”) = 45% [Normal] [Normal=95% Medium=97% High=99%]

15005 – Exchange automatically resumes mail transport after performance indicators drop down to “Normal” ~4 minutes after increasing to “High”:

TimeCreated  : 5/8/2014 3:54:05 PM
ProviderName : MSExchangeTransport
Id           : 15005
Message      : The resource pressure decreased from High to Normal.

No components disabled due to back pressure.
The following resources are in normal state:
Queue database and disk space (“C:\Program Files\Microsoft\Exchange
Server\V15\TransportRoles\data\Queue\mail.que”) = 45% [Normal] [Normal=95% Medium=97% High=99%]
Queue database logging disk space (“C:\Program Files\Microsoft\Exchange
Server\V15\TransportRoles\data\Queue\”) = 45% [Normal] [Normal=95% Medium=97% High=99%]
Version buckets = 2 [Normal] [Normal=80 Medium=120 High=200]
Private bytes = 4% [Normal] [Normal=71% Medium=73% High=75%]
Physical memory load = 63% [limit is 94% to start dehydrating messages.]
Submission Queue = 0 [Normal] [Normal=2000 Medium=4000 High=10000]
Temporary Storage disk space (“C:\Program Files\Microsoft\Exchange
Server\V15\TransportRoles\data\Temp”) = 45% [Normal] [Normal=95% Medium=97% High=99%]

Powershell parsing of archived event logs

Problem:

You want to find specific events from multiple archived event logs.

I wonder why he put a gorilla picture in here?

I wonder why he put a gorilla picture in here?

Solution:

EventScrape.ps1 – uses Powershell Get-WinEvent to parse offline event logs and sort them into chronological order.

[array]$TotalSearch=””
# *App* is targeting archived Application logs – change to suit your needs
Get-ChildItem -include *App*.evt,*App*.evtx -Path E:\EventLogs\Winevt\logs\ -recurse |

ForEach-Object {“Parsing $($_.fullname)`r`n”

Try {

$TotalSearch+=Get-WinEvent -FilterHashtable @{

Path=$_.fullname
Id=15004;
#StartTime=”1/14/2011″ ; #in case you want to limit the search to a certain time range
#EndTime=”1/15/2016″
} -EA Stop
$TotalSearch+=Get-WinEvent -FilterHashtable @{

Path=$_.fullname
Id=15005;
#StartTime=”1/14/2011″ ; #in case you want to limit the search to a certain time range
#EndTime=”1/15/2016″
} -EA Stop

} Catch [System.Exception] {“Done”}}

$TotalSearch| sort-object TimeCreated|fl|out-file “BackPressure.txt”
$searchCSV=$TotalSearch| sort-object TimeCreated|convertto-csv -useculture
$searchCSV|out-file “SearchCSV.csv”

gc .\BackPressure.txt |select-string “resource pressure” -context 5,7 |out-file Presures.txt
invoke-item .\Presures.txt

 

Details:

Use PowerShell to Parse Saved Event Logs for Errors
http://blogs.technet.com/b/heyscriptingguy/archive/2011/01/25/use-powershell-to-parse-saved-event-logs-for-errors.aspx