Mail queued up in Hub transport server – Back pressure

Overview

Although email is not always the best way to share files, the method is frequently used. As an administrator, you probably have to allow messages to be sent with attachments. Sometimes these attachments are relatively large. But you also have to balance this business requirement with making sure that your server hardware does not become overly utilized or that some users are denied service while others are processing super large messages.

In Customer Support Services we see a lot of critical server unresponsive type issues caused by someone trying to attach a really large file, say perhaps someone trying to share a DVD home video .ISO with their friends and coworkers.

Although we’ve attempted to harden Exchange out of the box, there are still a few things that you should consider doing to further limit the possibility of something like this happening.

Back Pressure

Exchange 2007 introduces a concept within Transport servers called Back Pressure. You can read all about it here. Suffice it to say, if your server becomes too busy, it will stop accepting new messages, and allow itself time to gracefully recover. It does this to protect itself from the extreme cases.

In short, Back Pressure is Exchange 2007’s way of monitoring available disk space, memory and uncommitted messages. When any of those resources exceed their corresponding thresholds for a sustained period the HUB server stops accepting anonymous submissions (medium threshold) or all submissions (high threshold). For example:

Event Type: Warning
Event Source: MSExchangeTransport
Event Category: ResourceManager
Event ID: 15004
Description:
Resource pressure increased from Medium to High.

Resource utilization of the following resources exceed the normal level:

Version buckets = 213 [High] [Normal=80 Medium=120 High=200]

Back pressure caused the following components to be disabled:
Inbound mail submission from Hub Transport servers
Inbound mail submission from the Internet
Mail submission from the Pickup directory
Mail submission from the Replay directory
Mail submission from Mailbox servers
Mail delivery to remote domains

With large messages, you have the possibility that a database transaction to commit the message into the database will take some time to complete. During that time, the database is tracking the commit with what is called version buckets or version store. So with large messages, you can guess that version buckets will often be the measure of how the mail queue database is keeping up. A few seconds of back pressure a few times per day is fine, but if your server(s) spend a lot of time in back pressure, then there’s the possibility that other messages aren’t being processed in a timely fashion.

Best Practices

An ounce of prevention is worth a pound of cure. So here are the best practices we recommend to protect your server(s) from large messages that might cause outages.

  • Install SP1 RU8. This rollup update contains an extremely important fix that should not be missed. KB 960775 is the fix that you need, particularly if you allow Outlook 2003 clients prior to SP3 to connect to your server. These clients will not ask for the maximum limits before synching and submitting a large message to the server. This can easily cause transaction log file growth and performance problems on the Mailbox server. But, worse, the store-generated DSN messages are then submitted to Transport and the problem can spread. This fix eliminates the possibility of Hub servers being affected. Regardless of this fix, it may still be a best practice to update your clients to SP3 and block legacy (unsupported) clients to limit the damage that can happen on the Mailbox server.
  • Run ExBPA. Although BPA does not know what’s reasonable for your organization, it can make sure that at least size limits are in place. ExBPA can check all of your servers quickly.
  • Set reasonable size limits for your organization based on planning. See above section for commonly missed size limits. You can use the detail output from the BPA to make certain the limits are where you think that they are.
  • Particularly if you’re supporting anything larger than the default 10MB message size, make sure that you’ve updated your edgetransport.exe.config file to the latest guidance for your version of Exchange. At the time of this publishing, the Exchange 2007 guidance when running the latest service pack is as follows:
  • The ESE cache size should be 512MB on any server with more than 4GB of RAM – An easy example:

    DatabaseMaxCacheSize” value=”536870912″ />

    For servers with 8GB of RAM or more, particularly if they are dedicated Hub role with transport dumpster enabled, you can set the value as high as 1GB:

    DatabaseMaxCacheSize” value=”1073741824″

    The version bucket thresholds should be as follows –

    DatabaseCheckPointDepthMax” value=”20971520″ /> to DatabaseCheckPointDepthMax” value=”536870912″

    The checkpoint depth should be approximately half of the DatabaseMaxCacheSize –
    QueueDatabaseLoggingBufferSize” value=”524288″ to QueueDatabaseLoggingBufferSize” value=”5242880″


    QueueDatabaseLoggingFileSize” value=”5242880″ /> to QueueDatabaseLoggingFileSize” value=”31457280″

  • Consider hardening and isolating Internet-facing receive connectors such that spam processing and virus scanning processes for inbound “unclean” message streams are not impacting the rest of mail flow. Set reasonable receive connector limits. This obviously transcends the large message discussion, but this is especially true if you allow larger messages.
  • Make sure that proper exclusions are set for file-based Antivirus software and that temporary locations are also located on drives with adequate space and speed. Temporary files can be created while converting large messages. Scanning these temporary files can cause problems – use proper Exchange Antivirus for protecting the messages and file-level scanning to protect the server(s).

Conclusion

Message size limits will protect your servers and make sure they stay happily running, but there is not any “one-size-fits-all” guidance. Nevertheless, setting reasonable message limits and following the best practices can save you a great deal of trouble.


Ref: http://msexchangeteam.com/archive/2009/07/07/451737.aspx


Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s