Abs of Steel and My DKIM Body Hash Won’t Verify. Help Me Dr. Phil!

If you are applying an *inbound* disclaimer with a mail flow rule in Office 365, you may be surprised to see a DKIM body hash failure in the header of the message. ( and if you have never noticed this, well, that’s understandable!)

Example:

 

Message sent from Gmail with disclaimer rule:

 

Authentication-Results spf=pass (sender IP is 209.85.216.180) smtp.mailfrom=gmail.com; contoso.com; dkim=pass (signature was verified) header.d=gmail.com;contoso.com; dmarc=pass action=none header.from=gmail.com;contoso.com; dkim=fail (body hash did not verify) header.d=gmail.com;

 

 

 

 

Message sent from Gmail w/o disclaimer rule:

 

Authentication-Results spf=pass (sender IP is 209.85.220.173) smtp.mailfrom=gmail.com; contoso.com; dkim=pass (signature was verified) header.d=gmail.com;contoso.com; dmarc=pass action=none header.from=gmail.com;contoso.com; dkim=pass (signature was verified) header.d=gmail.com;

 

This brings up some interesting questions. Is the DKIM check *after* mail flow rules are processed? And does this mean I have lost the ability to check for DKIM failures for those messages?

Thankfully, no on both. I have confirmed that you can ignore the failure and have been assured that mail flow rules are evaluated after DKIM verification.

As you can see in my examples, in both cases, the first DKIM check passes and just as importantly, DMARC passes and that is what you should be hanging your hat on.

Stay safe out there!

Designing My Office 365 Tenant for On-Premises

With the new Exchange Online storage limits, I got to thinking what that would look if I were to do this on-prem.

Let’s see, 2GB mailboxes to start… 100GB max sixe – no archive ( Why would I need an archive?)…16,000 mailboxes. Follow the Preferred Architecture otherwise: DAG, 4 copies of the databases, one lagged, JBOD, etc…

Run that through the latest mailbox calculator and wait for the big reveal.

 

Twelve Mailboxes per Database!

 

144 Databases in the DAG. I can handle that. Yes, 12 mailboxes in each. Got it.

Much less RAM than you might think. I have never considered using a “Lagged Copy Server” though.

Ouch. These giant mailboxes will need a bunch of storage! Not sure I am comfortable with databases that size. I am beginning to lose my nerve.

I think I will take the third option and leave the 100GB mailboxes to the Exchange Online professionals. You know who you are.

 

Auto-Forwarding is not dead. It’s very happy!

Auto-Forwarding from Exchange. Now there’s a subject that has been beat to death. We all know how it works and it has certainly been documented enough don’t you think?

Posts from Tim and others are worthy reads. My goal here is to simply put all of this together and maybe point out some things you may not know.

There have traditionally been two options to forward from Exchange: Outlook Rules or Administrator-enabled forwarding.

  1. Outlook Rules: End users have 3 options as illustrated in the image below. Of course, only one can be chosen, but I have checked all three to point them out. Additionally, you can select an existing user from the GAL or contacts or enter a SMTP address ad-hoc.  Regardless of which option in the rule is selected, the messages will be delivered to the user’s inbox.

 

 

 

2. Administrator enabled Auto-Forwarding:

This is done using EAC or Exchange Powershell.

In EAC, under Delivery options for the Mailbox.

Note here that the recipient has to exist in the Address Book.

With Powershell and set-mailbox, you have additional options:

The ForwardingAddress parameter specifies a forwarding address for messages that are sent to this mailbox. A valid value for this parameter is a recipient in your organization. You can use any value that uniquely identifies the recipient.

The ForwardingSmtpAddress parameter specifies a forwarding SMTP address for messages that are sent to this mailbox. Typically, you use this parameter to specify external email addresses that aren’t validated.

Set-Mailbox -Identity "John Woods" -DeliverToMailboxAndForward $true -ForwardingSMTPAddress manuel@contoso.com
 Note the important difference and the ability to include *or* exclude delivery to the mailbox.

How messages are delivered and forwarded is controlled by the DeliverToMailboxAndForward parameter.

  • DeliverToMailboxAndForward is $true   Messages are delivered to this mailbox and forwarded to the specified recipient.
  • DeliverToMailboxAndForward is $false   Messages are only forwarded to the specified recipient. Messages aren’t delivered to this mailbox.

Office 365 Outlook Web Access adds an additional wrinkle here. End-users have access to a trimmed down version of the administrator forwarding option.

This is not an Outlook rule, but similar to :
Set-Mailbox -Identity “John Woods” -DeliverToMailboxAndForward $true -ForwardingSMTPAddress manuel@contoso.com

 Of course, Office 365 users can still create rules for forward in Outlook as well.

 

Why you may not want to forward to external recipients

  1. Data leaks, typically unnoticed, to mailboxes you do not control.
  2. A forwarding rule could create a mail loop between your org and another.
  3. Forwarded messages could land your sending IP addresses on block lists.
  4. Forwarding could bypass your data retention requirements. *

 

Things you may not know about forwarding

*Outlook forwarding rules allow the message to bypass the sent items. Yep, that’s right. The rule is server-based and handled at the transport level. The messages will be in their inbox, but not the sent items folder . You can verify this with a message trace. The source context of the forwarded messages will be Transport Rule Agent. The exception to this is if the rule is run manually against existing messages. Those forwarded messages will be in the sent items. The header of each will look similar to this

If you are a Office 365 customer or run Exchange 2016 on-premises, you can mitigate this loophole.

 

A forward rule and a redirect rule do essentially the same thing except the forwarded message will not have a FW: in the header to indicate to the recipient that the message was forwarded. It will appear to have come directly from the original sender. Well, that’s at least what the official documentation says. From my experience, that is not entirely true. The FW: may not be in the header, but a recipient will be able to see the message was routed through another mail system. It may show “on behalf” or “via” your organization ( Google does this). And of course, if they check the internet headers, the real path will be revealed. Some recipient mail systems may even reject messages forwarded like this.  If the administrator sets forwarding at the mailbox level or a 365 user sets via OWA, it is essentially a redirect.

 

Preventing user auto-forwarding

  1. Block at the remote domain level: Set-RemoteDomain -Identity ExternalDomain -AutoForwardEnabled:$FALSE. This will stop the Outlook forwarding rules.
  2. For granular control, use a transport rule to prevent by group or user. Otherwise, block the default remote domain for auto-forwarding as above. This will also stop Outlook forwarding rules.
  3. Remove the ability for end-users to auto-forward in Office 365 OWA.

So there you have it. A compilation of the best of forwarding articles with some auto-tuning from me .

My recommendation: Block all auto-forwarding at all levels for users. Leave it to the professionals – your messaging administrators who can enable auto-forwarding at the server level.

I can’t think of too many business requirements for auto-forwarding, but I am sure there are out there.  I hope this little update helps you understand it some more.

 

Why can’t I point my clients to the DAG Cluster IP?

I was reminded today of a question I used to see a lot in the forums. Not so much anymore, but perhaps a refresher is in order.

Granted, it seems almost brilliant to simply configure all the URLs and connection points to the DAG IP. And after all it does say its use is “Cluster and Client”  😛

capture

and if that means there is no need to worry about load balancing and let Exchange handle it, then why not?

Here’s Why:.

  1. There is no Exchange dependency on the Cluster IP being online. Both Exchange 2013 and Exchange 2016 support IP-Less Database Availability Groups. The cluster IP can go offline and Exchange will run just fine. The only real reason to assign a Cluster IP address is if you are using backup software or another 3rd party application that requires it. If you run Exchange with the Preferred Architecture recommendations, you won’t be doing backups anyway!
  2. If the Cluster name goes offline and the IP with it, Managed Availability won’t attempt to bring it online. That requires manual intervention. Yuck.
  3. The Cluster IP is held by a specific mailbox server in the DAG at any one time – meaning all client connections will go through that multi-role server and no others.
  4. If the quorum owner moves to another server, there is no guarantee that the clients will handle that gracefully.
  5. The only way to prevent a server from end-user client access in this scenario is to pause or stop the cluster service on the affected server.
  6. IT’S NOT SUPPORTED!

 

 

 

 

 

What really happened to the cast of “Leave it to Beaver” (and a reminder about the DAG Replay Manager)

If you are using lagged copies, you have hopefully also enabled the Replay Manager as well. Once you do so, be aware of the implications. Most notably:

“consider an environment where a given database has 4 copies (3 highly available copies and 1 lagged copy), and the default setting is used for ReplayLagManagerNumAvailableCopies. If a non-lagged copy is out-of-service for any reason (for example, it is suspended, etc.) then the lagged copy will automatically play down its log files in 24 hours.”

To repeat: By default, if a non-lagged copy is out of service for more than a day, the lagged copy of that database will play down its logs and essentially become a HA copy.

So consider this scenario: The servers have a mix of HA and lagged copies on the same drives. One of them encounters some hardware issue, so you suspend all the databases on it and block activation until you can fix the problem, but that’s ok  – there are 3 healthy copies of the databases on other servers. But here is the catch. They have to be 3 HA copies. If it’s two HA copies and one lagged, then log play-down will kick off on those lagged copies after 24 hours if you haven’t changed the default and there goes the suspenders you counting on in case the belt fails.

Sounds obvious, but something that could bite you if you aren’t paying attention and you suddenly realize 2 days later that all the replay queue lengths of the affected databases are at zero, so stay safe out there.

moreyouknow

 

 

Note that in 2016 CU1, Replay Manager is enabled by default and other goodies!

As for what happened to the cast of “Leave it to Beaver”, well, not much really.

leaveittobeaver

 

 

My Top 5 Exchange Experts to Follow and 2 I Wish I Could

In the spirit of making meaningless lists , I thought I would put together my own compilation. These are in no particular order or rank.

Five to Follow

  1. Paul Cunningham: Paul is my go-to, how-to guy. His blog posts are informative, easy to read and hit the mark. He is the only Australian I know. That counts for something.
  2. Tony Redmond: No explanation needed here. I have followed Tony since my 5.5 days, and believe me, it makes him nervous. I was there when he announced that he had passed the “Clap” to the Exchange Product Group. I think I should get a t-shirt for that.
  3. Andrew S Higginbotham: I love his blog posts. A lot of common-sense fixes for those annoying issues we all run into. He’s younger than me and that pisses me off.
  4. Jeff Guillet: Jeff has the uncanny ability to always have a blog post ready just when its needed. And don’t forget to read his ADFS stuff as well! You will typically find Jeff at Ignite sessions propped up against a wall near the front.
  5. Paul Robichaux: Probably the best dressed MVP. I love listening to Paul talk. He has a very reassuring  manner and tone. We all know how good he is, no explanation needed for his inclusion here either.

Two I Wish I Could Follow

  1. Ed Crowley: Ed has been doing this stuff a long time so I’m sure he has no desire to be followed by anyone. I would never physically follow him however, that will only lead to some bus that takes 5 hours to get to the conference just to save a few bucks.
  2. Rich Matheisen: The original Exchange NewsGroup King, Rich has retired from both work and MVP-dom. I learned more about the SMTP RFCs from him than I can ever thank him for. Enjoy your retirement, Richard.

 

I left a lot of people off this list of course, including myself. 😛

It’s safe to say that all the Exchange MVPs I know and love are worth following and listening to, well, except a few. That list is only viewable at Joey’s in Bellevue, WA.

 

 

Sanity Checking Lagged Copies – To SIR* With Love

I seem to recall a presenter posing a question about lagged copies at a recent MEC conference, or maybe it was last year at Ignite. Anyway, the speaker asked for a show of hands if one was using Exchange lagged copies in their org and the number was, well… you could count them on your hands. Hopefully that has increased since then. Personally I don’t see why you wouldn’t use lagged copies if you are going to go the HA route. I’ll concede that a nice wizard to activate the lagged copy would be optimal, but nonetheless with documentation and defined procedures, an experienced admin can get over any fear they may have going backup-less. (Is that a word?)

If you decide to use lagged copies, there are already a number of good tutorials out there. I like my friend Paul’s easy to read article: http://exchangeserverpro.com/exchange-server-2013-lagged-database-copies-action/

Once you are setup, you will hopefully never need to look at them again, but if you aren’t so lucky and experience any sort of event that requires a lagged copy activation or log replay either through admin intervention or by Exchange itself ** – or you just want to periodically ensure things are level-set things, here some things to check post-outage/problem/log play-down/just because:

 

1. Get-MailboxDatabase * | ? {$_.CircularLoggingEnabled -eq $false}

Should return no results. I assume you are lagging for a reason right? Hopefully To get rid of backups. No backups, no log truncation. So you need to enable circular logging.

 

2. Get-MailboxDatabaseCopyStatus * | ? {$_.ActivationPreference -eq “4”} | select Name, Status, *queuelength*, LastInspectedLogTime, ContentIndexState, ReplayLagStatus,ActivationSuspended,ActionInitiator,ActiveCopy | OGV

Output this to a sortable grid view for a quick and easy check. Note: _.ActivationPreference -eq “4”. The assumption here is that you are running 4 copies. 3 HA, 1 lagged. If not, check based upon whatever activation level your lagged copies are set to.

You should see something like the image below. It nice and sortable and allows for quick verification.

 

OGVLaggedCopy-2

What to look for:

Status: Healthy

CopyQueueLength: 0 or close to it

ReplayQueueLength: above 0. Remember, you are checking just the lagged copies here, so each should have a replay queue length.

ContentIndexState: Healthy

ReplayLagStatus: Enabled:True; PlayDownReason:None; Percentage:100; Configured:8.00:00:00 (Actual: Equal or above the Configured – in this example, lag relay is set to 8 days). If you see a copy with a PlayDown reason, it’s time to investigate.

ActivationSuspended: True (assuming you have blocked automatic activation on the lagged copies)

ActionInitiator: Administrator (assuming you have blocked automatic activation on the lagged copies)

ActiveCopy: False.

If any copies are not set correctly to your desired settings. Correct them!

Examples:

 

Set lagged replay on 4th Preference DB to 7 days: Set-MailboxDatabaseCopy <DB>\<Server> -ActivationPreference 4 -ReplayLagTime  7.0:0:0

Disable Automatic activation for lagged database copy: Get-MailboxDatabaseCopyStatus <DB>\<Server> | Suspend-MailboxDatabaseCopy -ActivationOnly

Enable Circular Logging on the Database: Set-MailboxDatabase <DB> -CircularLoggingEnabled $true

 

* SIR= Single Item Retention. Recommended that you enable this for all mailboxes in a lagged environment running w/o backups. Belts and Suspenders.

** The Replay Lag Manager should be enabled in your environment. Be aware that under certain conditions, Exchange may automatically play down the lagged copies.

 

 

 

SSL 3.0 enabled after an Exchange update – Fixed in 2013 CU13

If you have been vigilant, you disabled SSL 3.0 a long time ago on your servers. You may be surprised to find it enabled again after you apply an Exchange Update.

NOTE: This appears to be fixed in CU13 for Exchange 2013. You should still verify after applying any CU however!

From the CU13 setup log:

 

New-Item -path $keyPathRoot”\SSL 3.0″ -ItemType key -Name “Server” -Force;
}
Set-ItemProperty -path $keyPath -name “Enabled” -value 0x0 -Type DWORD -Force;

 

 

 

Now, back to the original issue:

A little history: SSL 3.0 has some well-documented security issues and with a reg tweak and reboot, it’s no longer advertised. You can easily test this with my favorite “sanity-check” site:

https://www.digicert.com/help/

Capture4

 

Enter the server name and click “Check for common vulnerabilities”.

Hopefully it shows green:

Capture5

Great!

Until you apply an Exchange update. So on goes 2013 CU12 for example, and like all good admins you check the certificate one more time against  https://www.digicert.com/help/

Capture2

Doh!

Well, luckily it’s easy enough to fix of course. Reapply that registry setting and reboot.

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 3.0\Server]
“Enabled”=dword:00000000

Capture5

Whew. So, what’s going on here? Well, take a look at the ExchangeSetup.log file under the ExchangeSetupLogs directory at the root of the system drive:

 

04/26/2016 17:27:46.0177] [1] Executing:
$keyPathRoot = “HKLM:\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols”;
$keyPath = $keyPathRoot + “\SSL 2.0\Server”;
if (!(Test-Path $keyPath))
{
New-Item -path $keyPathRoot”\SSL 2.0″ -ItemType key -Name “Server” -Force;
}
Set-ItemProperty -path $keyPath -name “Enabled” -value 0x0 -Type DWORD -Force;

$keyPath = $keyPathRoot + “\SSL 3.0\Server”;
if (!(Test-Path $keyPath))
{
New-Item -path $keyPathRoot”\SSL 3.0″ -ItemType key -Name “Server” -Force;
}
Set-ItemProperty -path $keyPath -name “Enabled” -value 0x1 -Type DWORD -Force;

As you can see, Exchange Setup happily sets that key and enables SSL 3.0.

Just something to put on your post upgrade checklist!

 

Error when accessing a resource mailbox: “The value ” is already present in the collection”

When accessing a resource mailbox in Exchange 2013 EAC, you may encounter an error that you prevents you from viewing or editing the room mailbox properties:

Capture

 

Powershell is no good either!

 

Get-CalendarProcessing <Room>
WARNING: An unexpected error has occurred and a Watson dump is being generated: The value ” is already present in the
collection.
The value ” is already present in the collection.
    + CategoryInfo          : NotSpecified: (:) [Get-CalendarProcessing], InvalidOperationException
    + FullyQualifiedErrorId : System.InvalidOperationException,Microsoft.Exchange.Management.StoreTasks.GetCalendarProcessing

 

Cause: Typically this is because there is a disabled mailbox listed in the RequestInPolicy or BookInPolicy attributes for the room.

Solution: Run the following in Exchange Powershell to clear the values. Example:  Set-CalendarProcessing <Room> -BookInPolicy $null  

Once done, you should be able to access the room via Powershell or EAC and re-add any required requesters to the room policy. Alternatively, if you have Exchange 2010 still around, you can simply remove the disabled mailbox via the 2010 EMC.