Why can’t I point my clients to the DAG Cluster IP?

I was reminded today of a question I used to see a lot in the forums. Not so much anymore, but perhaps a refresher is in order.

Granted, it seems almost brilliant to simply configure all the URLs and connection points to the DAG IP. And after all it does say its use is “Cluster and Client”¬† ūüėõ

capture

and if that means there is no need to worry about load balancing and let Exchange handle it, then why not?

Here’s Why:.

  1. There is no Exchange dependency on the Cluster IP being online. Both Exchange 2013 and Exchange 2016 support IP-Less Database Availability Groups.¬†The cluster IP can go offline and¬†Exchange will run just fine.¬†The only real reason to assign a¬†Cluster IP address is if you are using backup software or another 3rd party application that requires it. If you run Exchange with the Preferred Architecture recommendations, you¬†won’t be doing backups anyway!
  2. If the Cluster name goes offline and the IP with it, Managed Availability won’t attempt to bring it online. That¬†requires manual intervention. Yuck.
  3. The Cluster IP is held by a specific mailbox server in the DAG at any one time – meaning all client connections will go through that multi-role server and no others.
  4. If the quorum owner moves to another server, there is no guarantee that the clients will handle that gracefully.
  5. The only way to prevent a server from end-user client access in this scenario is to pause or stop the cluster service on the affected server.
  6. IT’S NOT SUPPORTED!

 

 

 

 

 

What really happened to the cast of “Leave it to Beaver” (and a reminder about the DAG Replay Manager)

If you are using lagged copies, you have hopefully also enabled the Replay Manager as well. Once you do so, be aware of the implications. Most notably:

“consider an environment where a given database has 4 copies (3 highly available copies and 1 lagged copy), and the default setting is used for ReplayLagManagerNumAvailableCopies. If a non-lagged copy is out-of-service for any reason (for example, it is suspended, etc.) then the lagged copy will automatically play down its log files in 24 hours.”

To repeat: By default, if a non-lagged copy is out of service for more than a day, the lagged copy of that database will play down its logs and essentially become a HA copy.

So consider this scenario:¬†The¬†servers have¬†a mix of HA and lagged copies on the same¬†drives.¬†One of them¬†encounters some¬†hardware issue, so you suspend all the databases on it¬†and block activation until you can fix the problem, but that’s ok¬† – there are 3 healthy copies of the databases on other servers.¬†But here is the catch. They have to be 3 HA copies. If it’s two HA copies and one lagged, then log play-down will kick off on those lagged copies after 24 hours if you haven’t changed the default and there goes the suspenders you counting on in case the belt fails.

Sounds obvious, but something that could bite you if you aren’t paying attention and you suddenly realize 2 days later that all the replay queue lengths of the affected databases are at zero, so stay safe out there.

moreyouknow

 

 

Note that in 2016 CU1, Replay Manager is enabled by default and other goodies!

As for what happened to the cast of “Leave it to Beaver”, well, not much really.

leaveittobeaver

 

 

My Top 5 Exchange Experts to Follow and 2 I Wish I Could

In the spirit of making meaningless lists , I thought I would put together my own compilation. These are in no particular order or rank.

Five to Follow

  1. Paul Cunningham: Paul is my go-to, how-to guy. His blog posts are informative, easy to read and hit the mark. He is the only Australian I know. That counts for something.
  2. Tony Redmond: No explanation needed here. I have followed Tony since my 5.5 days, and believe me, it makes him nervous. I was there when he announced that he¬†had passed the “Clap” to the Exchange Product Group. I think I should get a t-shirt for that.
  3. Andrew S Higginbotham: I love his blog posts. A lot of common-sense fixes for those annoying issues we all run into. He’s younger than me and that pisses me off.
  4. Jeff Guillet: Jeff has the uncanny ability to always have a blog post ready just when its needed. And don’t forget to read his ADFS stuff as well! You will typically find Jeff at Ignite sessions propped up against a wall near the front.
  5. Paul Robichaux: Probably the best dressed MVP. I love listening to Paul talk. He has a very reassuring  manner and tone. We all know how good he is, no explanation needed for his inclusion here either.

Two I Wish I Could Follow

  1. Ed Crowley: Ed has been doing this stuff a long time so I’m sure he has no desire to be followed by anyone. I would never physically follow him however, that will only lead to some bus that takes 5 hours to get to the conference just to save a few bucks.
  2. Rich Matheisen: The original Exchange NewsGroup King, Rich has retired from both work and MVP-dom. I learned more about the SMTP RFCs from him than I can ever thank him for. Enjoy your retirement, Richard.

 

I left a lot of people off this list of course, including myself. ūüėõ

It’s safe to say that all the Exchange MVPs I know and love are worth following and listening to, well, except a few. That list is only viewable at Joey’s in Bellevue, WA.

 

 

Sanity Checking Lagged Copies – To SIR* With Love

I seem to recall a¬†presenter posing a question about lagged copies¬†at a recent MEC conference, or maybe it was last year at Ignite. Anyway, the speaker asked for a show of hands if one was using Exchange lagged copies in their org and the number was, well… you could count them on your hands. Hopefully that has increased since then. Personally I don’t see why you wouldn’t use lagged copies if you are going to go the HA route. I’ll concede that a nice wizard to activate the lagged copy would be optimal, but nonetheless with documentation and defined procedures, an experienced admin¬†can get¬†over any fear they may have going backup-less. (Is that a word?)

If you decide to use lagged copies, there are already a number of good tutorials out there. I like my friend Paul’s easy to read article: http://exchangeserverpro.com/exchange-server-2013-lagged-database-copies-action/

Once you are setup, you will hopefully never need to look at them again, but if you aren’t so lucky and experience any sort of¬†event that requires a lagged copy activation or log replay either through admin intervention or by Exchange itself¬†** –¬†or you just¬†want to periodically ensure things are level-set things,¬†here some things to check post-outage/problem/log play-down/just because:

 

1. Get-MailboxDatabase * | ? {$_.CircularLoggingEnabled -eq $false}

Should return no results. I assume you are lagging for a reason right? Hopefully To get rid of backups. No backups, no log truncation. So you need to enable circular logging.

 

2. Get-MailboxDatabaseCopyStatus * | ? {$_.ActivationPreference -eq “4”} | select Name, Status, *queuelength*, LastInspectedLogTime, ContentIndexState, ReplayLagStatus,ActivationSuspended,ActionInitiator,ActiveCopy | OGV

Output this to a sortable¬†grid view for a quick and easy check. Note: _.ActivationPreference -eq “4”. The assumption here is that you are running 4 copies. 3 HA, 1 lagged. If not, check based upon whatever activation level¬†your lagged copies are set to.

You should see something like the image below. It nice and sortable and allows for quick verification.

 

OGVLaggedCopy-2

What to look for:

Status: Healthy

CopyQueueLength: 0 or close to it

ReplayQueueLength: above 0. Remember, you are checking just the lagged copies here, so each should have a replay queue length.

ContentIndexState: Healthy

ReplayLagStatus: Enabled:True; PlayDownReason:None; Percentage:100; Configured:8.00:00:00 (Actual: Equal or above the Configured – in this example, lag relay is set to 8 days). If you see a copy with a PlayDown reason, it’s time to¬†investigate.

ActivationSuspended: True (assuming you have blocked automatic activation on the lagged copies)

ActionInitiator: Administrator (assuming you have blocked automatic activation on the lagged copies)

ActiveCopy: False.

If any copies are not set correctly to your desired settings. Correct them!

Examples:

 

Set lagged replay on 4th Preference DB to 7 days: Set-MailboxDatabaseCopy <DB>\<Server> -ActivationPreference 4 -ReplayLagTime  7.0:0:0

Disable Automatic activation for lagged database copy: Get-MailboxDatabaseCopyStatus <DB>\<Server> | Suspend-MailboxDatabaseCopy -ActivationOnly

Enable Circular Logging on the Database: Set-MailboxDatabase <DB> -CircularLoggingEnabled $true

 

* SIR= Single Item Retention. Recommended that you enable this for all mailboxes in a lagged environment running w/o backups. Belts and Suspenders.

** The Replay Lag Manager should be enabled in your environment. Be aware that under certain conditions, Exchange may automatically play down the lagged copies.

 

 

 

SSL 3.0 enabled after an Exchange update – Fixed in 2013 CU13

If you have been vigilant, you disabled SSL 3.0 a long time ago on your servers. You may be surprised to find it enabled again after you apply an Exchange Update.

NOTE: This appears to be fixed in CU13 for Exchange 2013. You should still verify after applying any CU however!

From the CU13 setup log:

 

New-Item -path $keyPathRoot”\SSL 3.0″ -ItemType key -Name “Server” -Force;
}
Set-ItemProperty -path $keyPath -name “Enabled” -value 0x0 -Type DWORD -Force;

 

 

 

Now, back to the original issue:

A little history: SSL 3.0 has some well-documented security issues and with a reg tweak and reboot, it’s no longer advertised. You can easily test this with my favorite “sanity-check” site:

https://www.digicert.com/help/

Capture4

 

Enter the server name and click “Check for common vulnerabilities”.

Hopefully it shows green:

Capture5

Great!

Until you apply an Exchange update. So on goes 2013 CU12 for example, and like all good admins you check the certificate one more time against  https://www.digicert.com/help/

Capture2

Doh!

Well, luckily it’s easy enough to fix of course. Reapply that registry setting and reboot.

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 3.0\Server]
“Enabled”=dword:00000000

Capture5

Whew. So, what’s going on here? Well, take a look at the ExchangeSetup.log file¬†under the ExchangeSetupLogs directory at the root of the system drive:

 

04/26/2016 17:27:46.0177] [1] Executing:
$keyPathRoot = “HKLM:\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols”;
$keyPath = $keyPathRoot + “\SSL 2.0\Server”;
if (!(Test-Path $keyPath))
{
New-Item -path $keyPathRoot”\SSL 2.0″ -ItemType key -Name “Server” -Force;
}
Set-ItemProperty -path $keyPath -name “Enabled” -value 0x0 -Type DWORD -Force;

$keyPath = $keyPathRoot + “\SSL 3.0\Server”;
if (!(Test-Path $keyPath))
{
New-Item -path $keyPathRoot”\SSL 3.0″ -ItemType key -Name “Server” -Force;
}
Set-ItemProperty -path $keyPath -name “Enabled” -value 0x1 -Type DWORD -Force;

As you can see, Exchange Setup happily sets that key and enables SSL 3.0.

Just something to put on your post upgrade checklist!

 

Error when accessing a resource mailbox: “The value ” is already present in the collection”

When accessing a resource mailbox in Exchange 2013 EAC, you may encounter an error that you prevents you from viewing or editing the room mailbox properties:

Capture

 

Powershell is no good either!

 

Get-CalendarProcessing <Room>
WARNING: An unexpected error has occurred and a Watson dump is being generated: The value ” is already present in the
collection.
The value ” is already present in the collection.
    + CategoryInfo          : NotSpecified: (:) [Get-CalendarProcessing], InvalidOperationException
    + FullyQualifiedErrorId : System.InvalidOperationException,Microsoft.Exchange.Management.StoreTasks.GetCalendarProcessing

 

Cause: Typically this is because there is a disabled mailbox listed in the RequestInPolicy or BookInPolicy attributes for the room.

Solution: Run the following in Exchange Powershell to clear the values. Example:  Set-CalendarProcessing <Room> -BookInPolicy $null  

Once done, you should be able to access the room via Powershell or EAC and re-add any required requesters to the room policy. Alternatively, if you have Exchange 2010 still around, you can simply remove the disabled mailbox via the 2010 EMC.

 

The Trinity of Email Protection: Lessons Learned using DMARC, DKIM and SPF in Office 365

I am a big fan of DMARC and its ability to easily determine which messages sent¬†as, and from,¬†your SMTP domains are using authenticated and authorized servers. The A in DMARC actually stands for “authentication”, but it’s really a little of both.

Regardless, it’s something I highly recommend and if you are using Office365/EOP, there is really no reason you should not be using it, after all it’s already enabled. In fact, if you choose not to set up¬†any of the email protection trinity, EOP will absolutely mark¬†any message sent from the internet as your domain to your domain as SPAM.

Since you are reading this, I assume you are using it or wanting to use it. To that end, here are some tips and lessons learned from my own experiences:

  1. Start small.¬† Your first DMARC record in DNS ( _dmarc.contoso.com) should do nothing but report. At its very minimum, it should look like this: “v=DMARC1; p=none;rua=mailto:DMARC@contoso.com;”¬†¬†All this does is tell recipient servers that they should send DMARC failure reports to DMARC@contoso.com ( This could be a mail-enabled Public Folder).¬† The p=none does not affect the way messages are delivered. With p=none, you are not suggesting to a recipients mail server that they change the way they handle the messages that fail DMARC checks.¬† I would recommend you always use p=none, unless you have no 3rd party vendors that send mail as your SMTP domain. In that case, p=quarantine is probably¬†worthwhile once you have vetted out your DMARC policy sufficiently.¬†I would never use p=reject¬† – unlike a few ISPs out there.
  2. Get your SPF records in order.¬†If you have 3rd party vendors that send as you, this is especially important. It doesn’t take much to exceed the 10 Lookup Max on SPF records. Heck, if you are using EOP, spf.protection.outlook.com¬†requires three lookups alone! Exceeding that amount can cause failures because recipient domains may ignore your SPF completely.
  3. Don’t forget DKIM! SPF seems to get all the attention, but setting up DKIM could not be easier in Office 365. You can even configure it in the portal now under the Protection menu in EAC. And once setup, there is nothing cooler than seeing that DKIM-Signature in the headers.
  4. There are options if you simply can not get those SPF lookups below 10. The SPF RFC supports macros. Mind. Blown. No really. Imagine a SPF record of “contoso.com IN TXT “v=spf1 include:%{i}._ip.{%h}._ehlo.{%d}._spf.contoso.com ‚Äďall”.¬† Ok, I am not aware of anyone that does this, but it’s a possibility. Another option is not use SPF records and depend on the DKIM record. This is great if you have no or cooperative 3rd party vendors that send as your domain. DKIM is not limited to that pesky 10 max record lookup and as mentioned above, DMARC messages need to pass either the DKIM or SPF and, not both. If you support multiple SMTP domains, you may want to also consider setting up distinct SPF records for each zone rather then using the include option to keep each domain under that 10 limit.
  5. Don’t forget the alignment requirement! As mentioned above, DMARC will check the sender’s valid SPF or DKIM records in DNS. If one or both of those exist and pass, then alignment must pass as well. This¬†is the magic sauce in DMARC. Alignment simply means the header FROM: matches the Domain “from” (i.e. the MAIL FROM: /Return-PATH) of the message. This is an important consideration because even if the message passes the SPF or DKIM check, it can still fail DMARC if for some reason you have processes that set the FROM: in the header to a completely different domain from the RETURN-PATH or MAIL FROM:. I have seen it happen!
  6. Leverage DMARC inbound. The value-add for DMARC is pretty obvious for messages sent out from your org. But the really cool part is using it inbound to stop those endless spoofing attempts without the need for clunky transport rules. You can create nifty rules to check for failures as illustrated in a previous blog post. This is another reason you want those SPF records to be below 10 lookups. If they exceed allowed amount, you may find that your inbound rules do not correctly detect the dmarc=fail or dmarc=pass.
  7. Whitelisting is your friend. If you are going to create inbound DMARC rules,¬†it is very¬†important to remember that SPF lists mail servers that are authorized to send messages on behalf of your SMTP domain. That’s not necessarily the same list of IPs that will sending as your domain directly to internal users. If you are using WorkDay, SalesForce or any of the multitude of SAAS¬†cloud vendors, you can be pretty darn sure they are sending spoofed messages to your domain, but not to external recipients as you, ¬†so they wont be listed in your SPF. Any whitelisting rules MUST be above the DMARC rule. Seems obvious, but don’t overlook it!
  8. Add a check for dmarc=temperror in your inbound transport rule. If the domain in the Mail FROM: doesn’t exist, you may see a spf=temperror or¬†dmarc=temperror in the Authentication-Results header of the message. If your rule isn’t testing on that, it could slip past your defenses. I have actually seen a number of these from images spun up in 3rd party hosted solutions. I prefer to tag these as FAILs, but that is up to you if you want to let them pass.
  9. Set the inbound rule to only notify you initially. As with DMARC in general, start small. Don’t block anything until you feel confident that only the truly unwanted spoofed messages are trapped. Your initial rule should simply check for a DMARC failure, and send an incident report to a mailbox that you monitor. An example rule is found here. And that leads me to¬†the next tip once you ready to take action on DMARC failures.
  10. Quarantine the message, do not simply set it to a high SCL. You really have two options¬† ( I would not delete the message) on how to deal with illegitimate spoofed mail, you can force it the quarantine, or¬†set it at a SCL high enough to mark it as SPAM. I recommend forcing it the quarantine so that end-user safe sender lists can not trump your rule.¬† If you have a user that has added a GAL object to their contacts ( A very common scenario), and has also checked “Also trust e-mail from my contacts”¬†in Safe Senders, the spoofing rule will be defeated.
  11. Add a custom x-header to your rule. This helps you, the help desk and the end-user easily identify why this message was quarantined.

So there are some tips. This is by no means a comprehensive list. I plan to add more or revise these as time permits. If you want to read more, including how to setup DKIM in Office 365,  I would suggest following the blogs of Terry Zink  and Andrew Stobart.

 

 

You are unable to choose the OU in EAC when creating a new Mailbox or Groups…

If you have more than 500 Organization Units¬†in your AD forest, you may..no…you will,¬†run into this issue in the Exchange 2013 EAC when creating a new mailbox or group and want to create the object¬†in a different OU other than the default “Users” container.

Upon accessing the OU Dialog box:

OUBrowse

 

You will see this lovely message:

OU

Unfortunately, this is a known issue. There is no fix yet.

I would recommend you simply create the mailboxes and groups in Powershell if you want to specify the OU.

The work-around for EAC:

  • ¬†Edit the web.config file on the MAILBOX server¬†under

     \\Program Files \ Microsoft \ Exchange Server \ V15 \ ClientAccess \ ecp \

    add the following under the appsettings section of the file.

    <add key=”GetListDefaultResultSize” value=”<number more than OUs in your forest” />

Recycle ECP app pool.

Note that you will need to do this after each Cumulative Update.

 

P.S. If you do not know how many OUs your forest has:

Get-OrganizationalUnit -ResultSize unlimited | Measure-Object

How to create an Outlook Profile for a Hidden Mailbox

Suppose you wanted to create an Outlook profile for a hidden mailbox and, for whatever reason, you do not want to unhide it from the Address Book just long enough to create it. All hope is not lost! ( Unless you are using Outlook 2016). You can do it using the LegacyExchangeDN.

1. Use adsiedit or your favorite LDP viewer/query tool and copy the LegacyExchangeDN of the hidden mailbox. ( I still prefer adfind to this day).

The LegacyExchangeDN value is a property of the user’s object in AD and will be in the form of: ¬†/o=Contoso/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=UserA

2. Create the Outlook profile. It will fail and you will be at the dialog box that shows server name and the users’ mailbox.

Capture

3.¬†Remove the “=SMTP: User@contoso.com” ¬†value of the¬† “Mailbox:” and paste the LegacyExchangeDN value you copied from Step1.

Capture5a

4. Hit Check Name and it should resolve and allow you to create the profile and access the mailbox.

Capture3a

This is with Exchange/Outlook 2013. This does not work with Outlook 2016 from what I have seen:

Capture4a