Office 365 Recommended Configuration Analyzer

In this article, you will learn how to improve security in Exchange Online – Office 365. We will use the Office 365 Recommended Configuration Analyzer (ORCA) tool and create an ORCA report. We will also look at the configuration analyzer in Microsoft 365 security center, but is it the same?

When you have Exchange Online or Office 365, most think they don’t have to do anything anymore. Everything is in the cloud, and Microsoft will do the rest. We are sorry to tell you, but that’s completely wrong. Let’s find out more.

Office 365 Recommended Configuration Analyzer (ORCA)

ORCA is a report that you can run in your environment, highlighting known configuration issues and improvements that can impact your experience with Microsoft Defender for Office 365 (formerly Office 365 Advanced Threat Protection).

The configuration analyzer analyzes the following types of policies:

• Exchange Online Protection (EOP) policies: This includes Microsoft 365 organizations with Exchange Online mailboxes and standalone EOP organizations without Exchange Online mailboxes.

• Microsoft Defender for Office 365 policies: This includes organizations with Microsoft 365 E5 or Defender for Office 365 add-on subscriptions.

You can run the ORCA report without Microsoft Defender for Office 365, but there will be fewer checks. At the moment of writing, the latest ORCA version is 1.9.11.

Before we can use ORCA, we have to connect to Exchange Online PowerShell.

Connect to Exchange Online PowerShell

Start Windows PowerShell as administrator. Install and connect to Exchange Online PowerShell.

PS C:\> Connect-ExchangeOnline

Install ORCA module

Run the Install-Module ORCA cmdlet to install the ORCA PowerShell module.

PS C:\> Install-Module ORCA

You will get a warning. Click on Yes to all.

Verify that you successfully installed the ORCA module with the Get-InstalledModule cmdlet.

PS C:\> Get-InstalledModule | ft -AutoSize

Version Name                     Repository Description
------- ----                     ---------- -----------
2.0.3   ExchangeOnlineManagement PSGallery  This is a General Availability (GA) release of Exchange Online PowerShell V2 module. ...
2.1     GlobalFunctions          PSGallery  This module provides centralized file logging capabilities and other helpful stuff
1.9.11  ORCA                     PSGallery  Office 365 Advanced Threat Protection Recommended Configuration Analyzer (ORCA)
1.4.7   PackageManagement        PSGallery  PackageManagement (a.k.a. OneGet) is a new way to discover and install software packages from around the web....

Get ORCA report

Run Get-ORCAReport cmdlet and let it analyze the tenant and go through the recommendation checks.

PS C:\> Get-ORCAReport
03/08/2021 21:35:10 Performing ORCA Version check...
03/08/2021 21:35:16 Getting Anti-Spam Settings
03/08/2021 21:35:17 Getting Tenant Settings
03/08/2021 21:35:18 Getting Anti-Malware Settings
03/08/2021 21:35:18 Getting Transport Rules
03/08/2021 21:35:19 Getting Accepted Domains
03/08/2021 21:35:19 Getting DKIM Configuration
03/08/2021 21:35:19 Getting Connectors
03/08/2021 21:35:19 Getting MX Reports for all domains
03/08/2021 21:35:55 Analysis - Anti-Spam Policies - Anti-Spam Policy Rules
03/08/2021 21:35:55 Analysis - Anti-Spam Policies - Safety Tips
03/08/2021 21:35:55 Analysis - Anti-Spam Policies - Phish Action
03/08/2021 21:35:56 Analysis - Anti-Spam Policies - Bulk Complaint Level
03/08/2021 21:35:56 Analysis - Anti-Spam Policies - High Confidence Spam Action
03/08/2021 21:35:56 Analysis - Anti-Spam Policies - Allowed Senders
03/08/2021 21:35:56 Analysis - Anti-Spam Policies - Bulk Action
03/08/2021 21:35:56 Analysis - Anti-Spam Policies - IP Allow Lists
03/08/2021 21:35:56 Analysis - Anti-Spam Policies - Domain Whitelisting
03/08/2021 21:35:56 Analysis - Anti-Spam Policies - Quarantine retention period
03/08/2021 21:35:56 Analysis - Anti-Spam Policies - Outbound spam filter policy settings
03/08/2021 21:35:56 Analysis - Anti-Spam Policies - High Confidence Phish Action
03/08/2021 21:35:56 Analysis - Anti-Spam Policies - Mark Bulk as Spam
03/08/2021 21:35:56 Analysis - Anti-Spam Policies - Spam Action
03/08/2021 21:35:56 Analysis - Anti-Spam Policies - Advanced Spam Filter (ASF)
03/08/2021 21:35:56 Analysis - Anti-Spam Policies - End-user Spam notifications
03/08/2021 21:35:56 Analysis - Connectors - Domains
03/08/2021 21:35:56 Analysis - Connectors - Enhanced Filtering Configuration
03/08/2021 21:35:56 Analysis - DKIM - Signing Configuration
03/08/2021 21:35:56 Analysis - DKIM - DNS Records
03/08/2021 21:35:56 Analysis - Malware Filter Policy - Malware Filter Policy Policy Rules
03/08/2021 21:35:56 Analysis - Malware Filter Policy - Internal Sender Notifications
03/08/2021 21:35:56 Analysis - Malware Filter Policy - Common Attachment Type Filter
03/08/2021 21:35:56 Analysis - Malware Filter Policy - External Sender Notifications
03/08/2021 21:35:56 Analysis - Tenant Settings - Unified Audit Log
03/08/2021 21:35:56 Analysis - Transport Rules - Domain Whitelisting
03/08/2021 21:35:56 Analysis - Zero Hour Autopurge - Zero Hour Autopurge Enabled for Malware
03/08/2021 21:35:56 Analysis - Zero Hour Autopurge - Zero Hour Autopurge Enabled for Phish
03/08/2021 21:35:56 Analysis - Zero Hour Autopurge - Zero Hour Autopurge Enabled for Spam
03/08/2021 21:35:56 Analysis - Zero Hour Autopurge - Supported filter policy action
03/08/2021 21:35:56 Generating Output
03/08/2021 21:35:56 Output - HTML
03/08/2021 21:35:57 Complete! Output is in C:\Users\administrator.EXOIP\AppData\Local\Microsoft\ORCA\ORCA-exoip365-202103082135.html

After the above checks, an HTML report is generated and exported to the AppData folder. In the next step, we will look at the ORCA HTML report.

ORCA report details

By default, the HTML report will open in your default browser. The first thing that we did notice is the red block at the top. You will see that if you don’t have Office Advanced Threat Protection (ATP) in the tenant. If you don’t see it, it means you have ATP, and ORCA performed extra checks.

Use Office 365 Advanced Threat Protection (ATP) in your Microsoft 365 tenant for maximum security.

Scroll down to the Summary and check the sections that the ORCA report analyzed.

Scroll to one of the sections that need improvement. In this example, we will look at the Anti-Spam policies.

Microsoft recommends to Set the Bulk Complaint Level threshold to be 6. At the moment, the threshold is set on 7, which is not recommended.

Under each section, you find more information about the recommended settings. Clicking the links will open the Microsoft Docs page. The last link will take you straight to the settings to configure. That’s a great addition!

In the next step, we will look at the Office 365 recommendation configuration analyzer, but this time in the security portal. That’s because Microsoft did add ORCA to the Microsoft 365 security center.

Microsoft 365 security configuration analyzer

Sign in to Microsoft 365 security center. Go to Policies & rules > Threat policies > Configuration analyzer.

Standard recommendations

The default setting is the standard recommendations, and in our tenant, it shows 6 recommendations. Click on the View Strict recommendations.

The Standard and Strict policy setting values used as baselines are described in Recommended settings for EOP and Microsoft Defender for Office 365 security.

Strict recommendations

The strict recommendations do show a total of 9 recommendations. In the column recommendations, click on adopt in each row.

The recommendations are successfully adopted. Click the refresh button.

All settings follow Strict recommendations.

Rerun ORCA report

Let’s run an ORCA report to check if everything shows as OK (green color).

PS C:\> Get-ORCAReport

The ORCA report shows that there are 4 recommendations left. But, the configuration analyzer in the security portal shows that everything is set. Why is it not similar? That’s because Microsoft is still working on integrating ORCA in the security admin center.

Follow the last ORCA report recommendations, and you’re all set. Did the configuration analyzer tool recommendations help you to secure Office 365?

Read more: Microsoft Exchange Server vulnerability check »

Conclusion

In this article, you learned how to check and configure Office 365 security recommendations with the configuration analyzer. The ORCA report will give you more recommendations than in the security portal. Also, the report will give easier access to the Microsoft Documentations page.

As the ORCA gets integrated in the Microsoft 365 security center, I recommend using both for now. Adopt the change through the security center if you only want to push a button. Do you specifically want to adjust the setting, go through the steps outlined in the Microsoft documentation, which you find in the ORCA report under each section.

Did you enjoy this article? You may also like Fix Winmail.dat attachment in Office 365. Don’t forget to follow us and share this article.