Last week we released Exchange Server 2010 Service Pack 1. It has received some great feedback and reviews from customers, experts, analysts, and the Exchange community.

The starting point for SP1 setup/upgrade should be the What's New in SP1, SP1 Release Notes, and Prerequisites docs. As with any new release, there are some frequently asked deployment questions, and known issues, or issues reported by some customers. You may not face these in your environment, but we're posting these here along with some workarounds so you're aware of them as you test and deploy SP1.

1. Upgrade order

   The order of upgrade from Exchange 2010 RTM to SP1 hasn’t changed from what was done in Exchange 2007. Upgrade server roles in the following order:

   1. Client Access server
   2. Hub Transport server
   3. Unified Messaging server
   4. Mailbox server

   The Edge Transport server role can be upgraded at any time; however, we recommend upgrading Edge Transport either before all other server roles have been upgraded or after all other server roles have been upgraded. For more details, see Upgrade from Exchange 2010 RTM to Exchange 2010 SP1 in the documenation.

2. SP1 Prerequisites

   SP1 requires the installation of 4-5 hotfixes, depending on the operating system – Windows Server 2008, or Windows Server 2008 R2. To install the Exchange 2010 SP1 administration tools on Windows 7 and Windows Vista, you requires 2 hotfixes.

   Note: Due to the shared code base for these updates, Windows Server 2008 and Windows Vista share the same updates. Similarly, Windows Server 2008 R2 and Windows 7 share the same updates. Make sure you select the x64 versions of each update to be installed on your Exchange 2010 servers.

   Here’s a matrix of the updates required, including download locations and file names.

   Hotfix	Download	Windows Server 2008	Windows Server 2008 R2	Windows 7 & Windows Vista
   979744 A .NET Framework 2.0-based Multi-AppDomain application stops responding when you run the application	MSDN or Microsoft Connect	Windows6.0-KB979744-x64.msu (CBS: Vista/Win2K8)	Windows6.1-KB979744-x64.msu (CBS: Win7/Win2K8 R2)	N. A.
   983440 An ASP.NET 2.0 hotfix rollup package is available for Windows 7 and for Windows Server 2008 R2	Request from CSS	Yes	Yes	N.A.
   977624 AD RMS clients do not authenticate federated identity providers in Windows Server 2008 or in Windows Vista. Without this update, Active Directory Rights Management Services (AD RMS) features may stop working	Request from CSS using the “View and request hotfix downloads” link in the KBA | US-English	Select the download for Windows Vista for the x64 platform.	N.A.	N.A.
   979917 Two issues occur when you deploy an ASP.NET 2.0-based application on a server that is running IIS 7.0 or IIS 7.5 in Integrated mode	Request from CSS using the Hotfix Request Web Submission Form or by phone (no charge)	Yes	N. A.	N. A.
   973136, FIX: ArgumentNullException exception error message when a .NET Framework 2.0 SP2-based application tries to process a response with zero-length content to an asynchronous ASP.NET Web service request: "Value cannot be null".	Microsoft Connect	Windows6.0-KB973136-x64.msu	N.A.	N. A.
   977592 RPC over HTTP clients cannot connect to the Windows Server 2008 RPC over HTTP servers that have RPC load balancing enabled.	Request from CSS	Select the download for Windows Vista (x64)	N.A.	N. A.
   979099 An update is available to remove the application manifest expiry feature from AD RMS clients.	Download Center	N. A.	Windows6.1-KB979099-x64.msu	N. A.
   982867 WCF services that are hosted by computers together with a NLB fail in .NET Framework 3.5 SP1	MSDN	N. A.	Windows6.1-KB982867-v2-x64.msu (Win7)	X86: Windows6.1-KB982867-v2-x86.msu (Win7) x64: Windows6.1-KB982867-v2-x64.msu (Win7)
   977020 FIX: An application that is based on the Microsoft .NET Framework 2.0 Service Pack 2 and that invokes a Web service call asynchronously throws an exception on a computer that is running Windows 7.	Microsoft Connect	N. A.	N. A.	x64: Windows6.1-KB977020-v2-x64.msu X86: Windows6.1-KB977020-v2-x86.msu

   Some of the hotfixes would have been rolled up in a Windows update or service pack. Given that the Exchange team released SP1 earlier than what was planned and announced earlier, it did not align with some of the work with the Windows platform. As a result, some hotfixes are available from MSDN/Connect, and some require that you request them online using the links in the corresponding KBAs. The administrator experience when initially downloading these hotfixes may be a little odd. However, once you download the hotfixes, and receive two of the hotfixes from CSS, you can use the same for subsequent installs on other servers. In due course, all these updates may become available on the Download Center, and also through Windows Update.

   These hotfixes have been tested extensively as part of Exchange 2010 SP1 deployments within Microsoft and by our TAP customers. They are fully supported by Microsoft.

3. Prerequisite download pages linked from SP1 Setup are unavailable

   When installing Exchange Server 2010 SP1 the prereq check may turn up some required hotfixes to install. The message will include a link to click for help. Clicking this link redirects you to a page saying that the content does not exist.

   We're working to update the linked content.

   Meanwhile, please refer to the TechNet article Exchange 2010 Prerequisites to download and install the prerequisites required for your server version (the hotfixes are linked to in the above table, but you'll still need to install the usual prerequisites such as .Net Framework 3.5 SP1, Windows Remote Management (WinRM) 2.0, and the required OS components).

4. The Missing Exchange Management Shell Shortcut

   Some customers have reported that after upgrading an Exchange Server 2010 server to Exchange 2010 SP1, the Exchange Management Shell shortcut is missing from program options. Additionally, the .ps1 script files associated with the EMS may also be missing.

   We’re actively investigating this issue. Meanwhile, here’s a workaround:

   1. Verify that the ConnectFunctions.ps1, commonconnectfunctions.ps1 and RemoteExchange.ps1 files are present in the %ExchangeInstallPath%\bin directory.

      NOTE: If these files are missing, you can copy the files from the Exchange Server 2010 Service Pack 1 installation media to the %ExchangeInstallPath%\bin directory. These files are present in the \setup\serverroles\common folder.

   2. Click Start -> AdmiinistrativeTools ->, right-click Windows PowerShell Modules, select Send to -> Desktop (as shortcut)
   3. Go to the Properties of the shortcut and on Target replace the path to C:\WINDOWS\system32\WindowsPowerShell\v1.0\powershell.exe -noexit -command ". 'C:\Program Files\Exchange Server\V14\bin\RemoteExchange.ps1'; Connect-ExchangeServer -auto"

      Note: if the Exchange installation folder or drive name is different than the default, you need to change the path accordingly.

5. Upgrading Edge Transport on Forefront Threat Management Gateway (TMG) and Forefront Protection for Exchange 2010

   If you upgrade a server with the Edge Transport server role running with ForeFront Threat Management Gateway (TMG) and ForeFront Protection for Exchange (FPE) enabled for SMTP protection, the ForeFront TMG Managed Control Service may fail to start and E-mail policy configuration settings cannot be applied.

   The TMG team is working on this issue. See Problems when installing Exchange 2010 Service Pack 1 on a TMG configured for Mail protection on the ForeFront TMG (ISA) Team Blog. Exchange 2010 SP1 Release Notes has been updated with the above information.

6. Static Address Book Service Port Configuration Changes

   The location for setting the port the address book service should use has changed in SP1. In Exchange 2010 RTM you had to edit the Microsoft.exchange.addressbook.service.exe.config to configure the service port. In SP1 you must use the following registry key:
   Path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\MSExchangeAB\Parameters
   Value name: RpcTcpPort
   Type: REG_SZ (String)

   
   

   When you apply SP1 to a machine where you had previously configured a static port by editing the Microsoft.exchange.addressbook.service.exe.config file, the upgrade process will not carry forward your static port assignments. Following a restart, the Address Book Service will revert to using a dynamic port instead of a static port specified in the config file. This may cause interruptions in service.

   As with all upgrades where servers are in load balanced pools, we recommend you perform a rolling upgrade — removing servers from the pool, updating them and then moving the pool to the newly upgraded machines. Alternatively, we recommend that you upgrade an array of servers by draining connections from any one machine before you upgrade it.

   There are times when these approaches may not be possible. You can maintain your static port configuration, and have it take effect the moment the address book service starts for the first time following the application of the service pack, by creating the registry key BEFORE you apply SP1 to your server. The registry key has no impact pre SP1, and so by configuring it before you apply the Service Pack you can avoid the need to make changes to set the port post install, and avoid any service interruptions.

7. iPhone, OWA Premium and POP3 & IMAP4 issues due to invalid accepted domain

   After applying E2010 SP1:

   1. iPhone users may not be able to view the content of incoming messages in their Inboxes, and when they try to open a message, they get an error saying:

      This message has not been downloaded from the server.

      Admins may see the following event logged in the Application Event Log on Exchange 2010 CAS Server:

      Watson report about to be sent for process id: 1234, with parameters: E12, c-RTL-AMD64, 14.01.0218.011, AirSync, MSExchange ActiveSync, Microsoft.Exchange.Data.Storage.InboundConversionOptions.CheckImceaDomain, UnexpectedCondition:ArgumentException, 4321, 14.01.0218.015.

   2. OWA Premium users may not be able to reply or forward a message. They may see the following error in OWA:

      An unexpected error occurred and your request couldn't be handled. Exception type: System.ArgumentException, Exception message: imceaDomain must be a valid domain name.

   3. POP3 & IMAP4 users may also not be able to retrieve incoming mail and Admins will see the following event logged in Event Log:

      ERR Server Unavailable. 21; RpcC=6; Excpt=imceaDomain must be a valid domain name.

   Resolution

   Please run the following command under Exchange Management Shell and verify that there is one domain marked as ‘Default’ and it's DomainName & Name values are valid domain names. We were able to reproduce the issue by setting a domain name with a space in it, like "aa bb"

   Get-AcceptedDomain | fl

   If you also have an invalid domain name there (for example, a domain name with a space in it), then removing the space and restarting the server will fix the EAS (iPhone), OWA, POP3 & IMAP4 issues as mentioned above.

   Command to run under EMS would be:

   Set-AcceptedDomain –Identity <value> -Name “ValidSMTPDomainName”

   Thes examples update the Name parameter of the "My Company" and "ABC Local" accepted domains (the space is removed from both):

   Set-AcceptedDomain –Identity “My Company” –Name “MyCompany.Com”
   Set-AcceptedDomain –Identity “ABC Local” –Name “ABC.Local”

8. Error when adding or removing a mailbox database copy

   If a server running Exchange 2010 RTM (or Exchange 2010 SP1 Beta) is upgraded to Exchange 2010 SP1, administrators may experience an error when using the Add-MailboxdDatabaseCopy or Remove-MailboxDatabaseCopy cmdlets to add or remove DAG members.

   When you try to add a DAG member, you may see the following error:

   Add-MailboxDatabaseCopy DAG-DB0 -MailboxServer DAG-2

   The result:

   WARNING: An unexpected error has occurred and a Watson dump is being generated: Registry key has subkeys and recursive removes are not supported by this method.
    
   Registry key has subkeys and recursive removes are not supported by this method.
   + CategoryInfo : NotSpecified: (:) [Add-MailboxDatabaseCopy], InvalidOperationException
   + FullyQualifiedErrorId : System.InvalidOperationException,Microsoft.Exchange.Management.SystemConfigurationTasks.
   AddMailboxDatabaseCopy

   The command is not successful in adding the copy or updating Active Directory to show the copy was added. This happens due to presence of the DumpsterInfo registry key.

   Workaround: Delete the DumpsterInfo key, as shown below.

   1. Identify the GUID of the database that is being added using this command:

      Get-MailboxDatabase DAG-DB0 | fl name,GUID

      The result:

      Name : DAG-DB0
      Guid : 8d3a9778-851c-40a4-91af-65a2c487b4cc

   2. On the server specified in the add command, using the database GUID identified, remove the following registry key:
      HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v14\Replay\State\<DB-GUID>\DumpsterInfo

      The GUID identified in this case is 8d3a9778-851c-40a4-91af-65a2c487b4cc. With this information you can now export and delete the DumpsterInfo key on the server where you are attempting to add the mailbox database copy. This can be easily done using the registry editor, but if you have more than a handful of DAG members, this is best automated using the Shell.

      This example removes the DumpsterInfo key from the 8d3a9778-851c-40a4-91af-65a2c487b4cc key:

      Remove-Item HKLM:\Software\Microsoft\ExchangeServer\Replay\State\8d3a9778-851c-40a4-91af-65a2c487b4cc\DumpsterInfo

      To automate this across all servers in your organization, use the DeleteDumpsterRegKey.ps1 script.

      File: deletedumpsterregkey_ps1.txt
      Description: The DeleteDumpsterRegkey.ps1 script can be used to delete the offending DumpsterInfo registry keys that can cause this problem on all mailbox servers in the organization. Rename the file to DeleteDumpsterRegkey.ps1 (remove the .txt extension).

      For more info, see Tim McMichael’s blog post Exchange 2010 SP1: Error when adding or removing a mailbox database copy.

Thanks to all the folks in CSS and Exchange teams who helped identify, validate and provide workarounds for some of the issues mentioned above, and to the Exchange community and MVPs for their feedback.

Bharat Suneja, Nino Bilic
M. Amir Haque, Greg Taylor,
& Tim McMichael

When will Exchange Server 2010 support FIPS compliance?
Exchange Server 2010 SP1 provides for the ability to disable algorithms which are not FIPS 140-2 compliant. These algorithms are used for encryption, hashing, and signing within the Windows Server 2008 and Windows Server 2008 R2 operating systems that support Exchange Server 2010. When the System cryptography: Use FIPS compliant algorithms for encryption, hashing, and signing setting is enabled in a Group Policy or Local Policy, it disables the use of non-FIPS compliant algorithms such as RC-4. In Exchange 2010 RTM, it caused certain functions to fail. The most notable issue was in Outlook Web App (OWA), as documented in Microsoft Knowledge Base Article KB977961, and in the web-based Exchange Control Panel (ECP).

What is FIPS?
Federal Information Processing Standards (FIPS) are standards utilized to define security and interoperability requirements for cryptographic algorithms that the US Government uses. The FIPS 140-2 publication and standard (Security Requirements for Cryptographic Modules - PDF) defines the cryptographic algorithms as well as standards for key generation and key management. There are several FIPS publications which define how to further secure information systems and provide a standard upon which systems can be evaluated.

For more information on how Microsoft products and libraries comply with FIPS 140, see FIPS 140 Evaluation.

The importance of FIPS compliance to specific customers
Within the United States our customers utilize several guidelines, checklists, and requirements for securing systems, all which call for this policy setting to be enabled on the application’s host operating system (OS). In addition we have customers that do business with the US Government or work in industries where there is significant government oversight.

This policy setting ensures that the host OS, Windows Server 2008 SP2 or greater and Windows Server 2008 R2 or greater, in this case, only utilizes cryptographic algorithms that have passed the Cryptographic Module Validation Program and have been certified by the National Institute for Standards and Technology. Try saying that really fast three times.

The Windows Server OS, specifically the Windows Cryptographic Service Provider (CSP) is responsible for leveraging FIPS compliant algorithms for cipher, hashing, signing and encryption and we don’t actually need to enable anything within Exchange Server 2010. Exchange 2010 does have to know how to process the information provided via the OS, OS components such as Internet Information Server (IIS), and the Windows CSP. Exchange 2010 was released without support for servers which had this setting enabled, but had support and testing aligned for release with Exchange 2010 SP1.

What happens when this policy setting is enabled?
In Exchange 2010 RTM, when the policy setting System cryptography: Use FIPS compliant algorithms for encryption, hashing, and signing is enabled on the Windows Server 2008 or Windows Server 2008 R2 OS, the Schannel Security Provider (SSP) disables Secure Sockets Layer (SSL) protocols which are not part of the FIPS 140 standard. When this policy setting is enabled only FIPS 140-2 approved cryptographic algorithms are utilized. Examples of FIPS 140-2 compliant algorithms are the Triple Data Encryption Standard (3DES) and Triple Data Encryption Algorithm (TDEA) cipher, Advanced Encryption Standard (AES) algorithm and the Secure Hashing Algorithm (SHA) for hashing. In addition only the Transport Layer Security for Secure Sockets Layer (TLS/SSL) protocols will be utilized.

For those of you who have enabled the System cryptography: Use FIPS compliant algorithms for encryption, hashing, and signing setting on an Exchange 2010 server, you may have discovered two distinct issues. The first is that Outlook Web App on your Client Access Servers (CAS) appears to work but generates errors once the customer provides their username and password or smartcard PIN.

For those of us that have customers using Kerberos constrained delegation (KCD), OWA errors out with:
! An unexpected error occurred and your request couldn’t be handled

Expanding the Show Details link provides additional detail, specifically an exception message stating:
The type initializer for ‘Microsoft.Exchange.Data.Storage.GccUtils’ threw an exception

Additionally, an error event (Event ID 4999, Source: MSExchange Common) will be logged in the Application event log on the Exchange CAS.

The second issue is near-identical where the web-based ECP functionality, also provided by the CAS, will fail.

How will this be fixed?
In Exchange 2010 SP1, changes have been made to the code base, tested and verified, to support this setting. Exchange 2010 SP1 operates with support for FIPS 140-2 algorithms if the Windows Server 2008 SP2 and Windows Server 2008 R2 operating systems are configured to utilize the FIPS 140-2 algorithms for system cryptography.

My agency/organization/customer/co-worker asked about this support yesterday. When will Exchange Server 2010 SP1 be released?
Exchange 2010 SP1 has been released and can be downloaded here.

Thanks for your time and my customers and I look forward to it as well!

Bob Christian II

Exchange 2010 features a new resource protection mechanism - user throttling. This feature is designed to limit the amount of resources a single user or application can take up on a CAS to prevent poorly written applications from causing denial of service (DoS) to the rest of the users. You can read about throttling in Understanding Client Throttling Policies. If any of the terminology in this post sounds unfamiliar, please refer to this documentation.

While Exchange 2010 RTM shipped with user throttling "off" by default (most limits were set to infinite), after more testing in Exchange 2010 SP1, we've come up with a tighter set of limits for the throttling policies, and have thus turned user throttling on by default.

We have also changed what happens when users exceed their budget in some cases. In Exchange 2010 RTM version, Exchange rejected any Exchange Web Service (EWS), Exchange ActiveSync (EAS) and Outlook Web App (OWA) requests made by users who exceeded their budget. We've improved on this idea in SP1 in the EWS and ActiveSync protocols, by instead delaying the call just enough for the budget to recharge back into the positive and then execute the request. This means that end users will generally see fewer errors from the ActiveSync client or EWS application. In some rare conditions, such as if the caller is exceeding max number of connections or subscriptions in EWS, we'll still reject the request.

The longest a single request can be delayed is a minute, but this would be an extreme case and one that would signify that something is out of place either on the server, or with the caller. Typically, users and applications will not encounter throttling (except maybe if the user is doing a sync of the whole mailbox). However, some resource-heavy applications may start to get throttled in SP1. If throttling does kick in, the delays will be short enough that users won't notice any effect. However, we've provided ways to gain an insight into what is the user's experience is like due to throttling.

There are two main ways to monitor throttling - by monitoring perf counters and by looking at IIS logs. First, SP1 offers the following useful perf counters (instance is per CAS process) to monitor throttling under the MSExchange Throttling category on a CAS:

• Max Delay Per Minute - this value represents the longest amount of time in msec that anyone was delayed due to throttling in the past minute.
• Max Effective Time In * - this set of counters say that if the throttling policy was set to the counter values, then all requests that have been encountered in the past minute would all go through unthrottled.
• Users Delayed X Milliseconds - the number of users who saw delays greater than "X" (see Delay Time Threshold) milliseconds in the past minute.
• Users X Times OverBudget - the number of users whose requests were rejected more than "X" times in the past minute (see OverBudgetThreshold).
• OverBudgetThreshold - the "X" value for the "Users X Times OverBudget" counter.
• Delay Time Threshold - the "X" value for the "Users Delayed X Milliseconds" counter.
• Total Unique Budgets - number of unique budgets (ie callers/users) seen in the past minute
• Unique Budgets OverBudget - number of unique budgets that went over budget in the past minute

The general rule is that if the "Unique Budgets OverBudget" counter is graphing a line that's close to the "Total Unique Budgets" line, then most of the users in your system are getting throttled. You can further refine that by checking how many users are seeing rejections vs how many are getting delayed by viewing the appropriate "Users X times ..." counter. Finally, you can see if and how much users are delayed by viewing the "Max Delay Per Minute" counter. Also, all of these counters are saved off to SCOM once every minute.

If you do determine that many of your users are getting throttled, you may further try to understand why by digging into IIS logs. As of SP1, only ActiveSync, OWA and EWS log throttling info to IIS. By searching IIS for users or the string "overbudget", you can view which requests they have been making and which have been going over budget. You can refer to Budget Snapshots in the IIS Logs for a breakdown of the different parts of the budget.

If you do determine that your users or applications are throttled too much by your standards and their scenarios are in fact legitimate, then you can tweak the throttling settings to reflect your environment's use by:

1. Turning throttling off
2. Running your regular traffic through Exchange
3. Watching what the "Max Effective Time In *" counters report over the course of a few days
4. Setting the throttling policies to that value. To do this, call Get-ThrottlingPolicy ?| { $_.IsDefault} | Set-ThrottlingPolicy <new param values>

Alternatively, if it is an EWS application using a service account that becomes throttled, and you determine that it is not resource intensive to the Exchange server, you should create a new, custom throttling policy for it. To do this:

1. Call New-ThrottlingPolicy and set the proper parameters (refer to Exchange documentation at the top of the document for explanation of the parameters)
2. Call Get-Mailbox <mailbox of service account that the app is using) | Set-Mailbox -ThrottlingPolicy:<your policy that you just created>

The changes will be picked up within 15 minutes, or immediately after you recycle the EWS app pool in IIS. Please note that custom policies are meant as one-off solutions when a few applications or users are getting throttled and the load they are putting on the system is actually legitimate. You shouldn't update everyone's link to a custom policy - if you need to change throttling settings for the majority of your users, edit the default policy. For more information on throttling please refer to the official documentation linked at the top of this article.

Andrew Salamatov