We have a project going on, and a lot of user mailboxes need to be…
Exchange mail flow breaks (Disable AntiMalwareScanning)
Mail flow seems not to be working and the mail queue is getting bigger. If you see these issues, disable AntiMalWareScanning for now on the on-premises Exchange Servers and restart the Microsoft Exchange Server Transport Service.
Update: Microsoft releases a patch. See the part Solution to Exchange mail flow breaks.
Table of contents
Introduction
The mail flow is not working and you can check the error in Event Viewer.
Error
The FIP-FS “Microsoft” Scan Engine failed to load. PID: 9244, Error Code: 0x80004005. Error Description: Can’t convert “2201010005” to long.
Get mail queue
Run Exchange Management Shell as administrator and check the queue.
[PS] C:\>Get-ExchangeServer | Get-QueueMICROSOFT RELEASED A PATCH FOR THE EMAIL STUCK IN TRANSPORT QUEUE. SKIP TO THE SOLUTION DOWN BELOW!!
Fix for Exchange mail flow breaks
There are two ways to fix the Exchange mail flow:
- Disable anti malware scanning (1a+2a)
- Bypass anti malware filtering (2a+2b)
Both methods will work. Choose one of these options and follow the steps below.
1a. Disable AntiMalWareScanning
In Exchange Management Shell, run the command to disable AntiMalware Scanning on the Exchange Server on-premises.
Note: This command can take a minute or more. Leave it running till it’s successfully run.
[PS] C:\>& $env:ExchangeInstallPath\Scripts\Disable-AntimalwareScanning.ps1
WARNING: The following service restart is required for the change(s) to take effect : MSExchangeTransport
Anti-malware scanning is successfully disabled. Please restart MSExchangeTransport for the changes to take effect.Note: To re-enable malware filtering, use Enable-Antimalwarescanning.ps1 instead of Disable-Antimalwarescanning.ps1.
You can find the script Disable-AntiMalWareScanning.ps1 in File Explorer.
1b. Bypass anti malware filtering
In Exchange Management Shell, run the command to bypass anti malware filtering on the Exchange Servers on-premises.
[PS] C:\>Get-ExchangeServer | % {Set-MalwareFilteringServer -BypassFiltering $true -Identity $_.Name}2a. Verify AntiMalwareScanning is disabled
To verify that malware filtering is disabled, run the Get-TransportAgent cmdlet and confirm that it returns a value of False.
[PS] C:\>Get-ExchangeServer | % {Get-TransportAgent "Malware Agent"}
Identity Enabled Priority
-------- ------- --------
Malware Agent False 52b. Verify bypass filtering is disabled
To verify that malware bypass filtering is disabled, run the Get-MalwareFilteringServer cmdlet and confirm that it returns a value of True.
[PS] C:\>Get-ExchangeServer | % {Get-MalwareFilteringServer}
Name ForceRescan BypassFiltering DeferWaitTime DeferAttempts UpdateFrequency PrimaryUpdatePath
---- ----------- --------------- ------------- ------------- --------------- -----------------
EX01-2019 False True 5 3 30 http://amupdatedl.microsoft.com/server/amupdate3. Restart Exchange Server Transport Service
Restart the Microsoft Exchange Server Transport service.
[PS] C:\>Restart-Service MSExchangeTransportRestart the Microsoft Exchange Server Transport service on all Exchange Servers.
[PS] C:\>Get-ExchangeServer | % {Get-Service -ServiceName MSExchangeTransport | Restart-Service -Force}Read more: Restart Exchange Server services through PowerShell »
The queue will start to process the mail.
Solution to Exchange mail flow breaks
Microsoft did release a patch and you can run that in your environment.
Implementation of the solution requires customer actions. These actions can be automated with the scan engine reset script from https://aka.ms/ResetScanEngineVersion or they can be performed manually.
Important: Whether you perform the steps automatically or manually, they must be performed on every Exchange 2016 and Exchange 2019 server in your organization.
If you use the automated script, you can run it on multiple servers in parallel. We recommend starting with the servers in your environment that have the largest queues. You can use Get-Queue to view queue size and determine which queues are the largest.
Note: If you previously disabled or bypassed antimalware scanning as a mitigation for this issue, we recommend that you re-enable it after performing the steps below.
1a. Automated Solution
Download the ResetScanEngineVersion.ps1 PowerShell script and place it on the Exchange Server C:\scripts folder. If you don’t have a scripts folder, create one.
- Run Exchange Management Shell as administrator
- Run Set-ExecutionPolicy -ExecutionPolicy RemoteSigned
- Change the path to the scripts folder
- Run the script .\ResetScanEngineVersion.ps1
Important: Run the .\ResetScanEngineVersion.ps1 PowerShell script on each Exchange Server.
Run the script on each Exchange server in your organization. You can run this script on multiple servers in parallel. After the script has completed, you will see the following output:
[PS] C:\>Set-ExecutionPolicy -ExecutionPolicy RemoteSigned
[PS] C:\>cd C:\scripts
[PS] C:\scripts>.\Reset-ScanEngineVersion.ps1
EX01-2019 Stopping services...
EX01-2019 Removing Microsoft engine folder...
EX01-2019 Emptying metadata folder...
EX01-2019 Starting services...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Exchange Transport (MSExchangeTransport)' to start...
EX01-2019 Starting engine update...
Running as EXOIP\administrator.
--------
Connecting to EX01-2019.exoip.local.
Dispatched remote command. Start-EngineUpdate -UpdatePath http://amupdatedl.microsoft.com/server/amupdate
--------Note: Do you get the error “Cannot stop process” in relation to the updateservice.exe located in the FIP-FS? Ensure that the account you sign in to the Exchange Server has Debug programs user rights under the Local Group Policy. Otherwise, the process will not terminate, and the script fails to run.
1b. Manual Solution
You can can also manually perform steps to resolve the issue and restore service. To manually resolve this issue, you must perform the following steps on each Exchange server in your organization:
Remove existing engine and metadata
- Stop the Microsoft Filtering Management service. When prompted to also stop the Microsoft Exchange Transport service, click Yes.
- Use Task Manager to ensure that updateservice.exe is not running.
- Delete the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\amd64\Microsoft.
- Remove all files from the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\metadata.
Update to latest engine
- Start the Microsoft Filtering Management service and the Microsoft Exchange Transport service.
- Open the Exchange Management Shell.
- Navigate to the Scripts folder (%ProgramFiles%\Microsoft\Exchange Server\V15\Scripts).
- Run .\Update-MalwareFilteringServer.ps1 <server FQDN>
[PS] C:\>cd "$env:ProgramFiles\Microsoft\Exchange Server\V15\Scripts"
[PS] C:\Program Files\Microsoft\Exchange Server\V15\Scripts>.\Update-MalwareFilteringServer.ps1 EX01-2019
Running as EXOIP\administrator.
--------
Connecting to EX01-2019.
Dispatched remote command. Start-EngineUpdate -UpdatePath http://amupdatedl.microsoft.com/server/amupdate
--------2. Verify engine update info
- Start Exchange Management Shell.
- Run Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell.
- Run Get-EngineUpdateInformation and verify the UpdateVersion information is 2112330001 or higher.
Note: Run the Get-EngineUpdateInformation cmdlet and verify the UpdateVersion information is 2112330001 or higher.
[PS] C:\>Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell
[PS] C:\>Get-EngineUpdateInformation
Engine : Microsoft
LastChecked : 01/02/2022 08:22:33 AM +01:00
LastUpdated : 01/02/2022 08:22:41 AM +01:00
EngineVersion : 1.1.18800.4
SignatureVersion : 1.355.1227.0
SignatureDateTime : 01/01/2022 12:29:06 PM +01:00
UpdateVersion : 2112330001
UpdateStatus : UpdateAttemptSuccessfulFilter the Application log on the Event sources: FIPFS.
3. Re-enable anti malware scanning
If you did apply the fix before that Microsoft published a solution, enable anti malware scanning. If you did not apply any fix and you immediately applied the solution, you don’t have to re-enable anti malware scanning because it’s enabled by default.
Did you disable anti malware scanning? Run the command below and restart the Microsoft Exchange Transport Service.
[PS] C:\>& $env:ExchangeInstallPath\Scripts\Enable-AntimalwareScanning.ps1
Anti-malware engines are updating. This may take a few minutes.
Checking for engines updated after 12/26/2021 8:31:11 AM.
Updating Microsoft. Last updated : 1/2/2022 8:22:41 AM
WARNING: The following service restart is required for the change(s) to take effect : MSExchangeTransport
Anti-malware scanning is successfully enabled. Please restart MSExchangeTransport for the changes to take effect.Did you bypass anti malware scanning? Run the command below and restart the Microsoft Exchange Transport Service.
[PS] C:\>Get-ExchangeServer | % {Set-MalwareFilteringServer -BypassFiltering $false -Identity $_.Name}4. Verify mail flow and FIPFS error events not present
After updating the engine, we also recommend that you verify that mail flow is working and that FIPFS error events are not present in the Application event log.
Test and verify that mail flow works:
- Test internal mail flow in Exchange Server
- Test inbound mail flow in Exchange Server
- Test outbound mail flow in Exchange Server
Important: Do you see FIPFS errors in the event log and the email are still stuck in the queue/mail flow is not working after running the ResetScanEngineVersion.ps1 PowerShell script? RESTART THE EXCHANGE SERVER.
Latest updates
Update: Microsoft pushed an agent update, but it’s not yet working. So keep anti-malware scanning disabled.
Update1: Updated screenshot with how to check the event viewer for the issue.
Update2: Downloaded the latest MS Filtering Engine Update: Update Version:2201010006, it’s still not working.
MS Filtering Engine Update process has successfully committed and handed off updates for Microsoft
Last Checked:2022-01-01T12:05:10Z
Last Updated:2022-01-01T12:05:18Z
Engine Version:1.1.18800.4
Signature Version:”1.355.1234.0″
Update Version:2201010006
Last Definition Update:2022-01-01T05:20:29.000Z
Update Path:http://amupdatedl.microsoft.com/server/amupdate
Update3: Added both the options on how to disable anti malware scanning or bypass anti malware filtering.
Update4: MS Filtering Engine update will not fix this issue. You don’t have to check for engine updates. Implement this fix and wait for a proper fix release by Microsoft.
Update5: Microsoft Product Manager sent me a message, and they are working on a fix. The fix will be released with how-to instructions. Stay tuned.
Update6: You can wait for the Microsoft patch if you have enough disk space and don’t mind that Exchange Server will deliver the messages later. If not, apply the fix and patch later when Microsoft releases.
Update7: Microsoft Exchange Server expiration message is by default 48 hours (2 days). If a message remains in the queue for longer than this period of time, the message is returned to the sender as a permanent failure.
You can change that value between 1-90 days. The problem is that the longer days you set, the more messages will stay, the larger the mail queue will become. It means you have to keep adding more disk space.
I recommend running the above fix. But, if you do not want to disable the anti malware scanning, it’s better to expand the message expiration timeout in the Transport Server. Just to be safe.
In the below example, it’s 2 days.
[PS] C:\>Get-TransportServer | select Name,MessageExpirationTimeout
Name MessageExpirationTimeout
---- ------------------------
EX01-2019 2.00:00:00Change it to 4 days (or another value).
[PS] C:\>Set-TransportServer -Identity EX01-2019 -MessageExpirationTimeout 4.00:00:00Update8: Microsoft patch is OUT! Run the automated or manual solution. I recommend the automated solution first.
Update9: If you applied the fix before the solution, don’t forget to re-enable antimalware scanning and restart the Microsoft Exchange Transport Service.
Update10: Updated screenshot with how to check the event viewer after applying the solution.
Update11: The mail.que database got larger in size, and it will not shrink automatically. This happened because the Exchange Server kept the mail in the queue, and the messages didn’t get released before the fix/solution. It can take a lot of space, and I recommend making it smaller by removing the mail.que database. After that, Exchange Server will automatically create a new mail.que database.
Read more: Mail.que file large in size Exchange Server »
Update12: If you didn’t have enough disk space on the OS drive or ran out of disk space because of the mail.que database was filling up, it’s now an excellent time to move the mail.que database to another drive. This way, the OS drive will stay active and not give you performance issues.
Read more: Move mail queue to another location in Exchange Server »
Update13: Added articles to test mail flow internal/inbound/outbound.
Update14: Some admins mention that an Exchange Server reboot is required after applying the Solution. After that, the mail queue gets empty/mail flow starts working.
Update15: Running the Update-MalwareFilteringServer.ps1 cmdlet shows: Error starting the anti-malware engine update. To fix this error, run the below steps:
- Run Exchange Management Shell as administrator
- Run Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell
- Run Start-EngineUpdate
- Run Get-EngineUpdateInformation
Update 16: Only Exchange Server 2016 and Exchange Server 2019 are affected. The Exchange Edge Transport server is not affected by this Microsoft Exchange Y2K22 bug.
Update 17: UpdateVersion is now on 2112330003. Added to the article that it needs to show UpdateVersion 2112330001 or higher.
Update 18: Some admins mention that Exchange Server 2013 is also affected and some say it’s not. This is not true and the mail transport will work. But, you have to run the same script automatically or manually.
If your Exchange 2013 server took the antimalware update and it is now on version starting with “22…” you should use the automated or manual steps in this blog post to get your server on an engine version “21…” to continue getting the antimalware updates. Without taking action your server will not get any future antimalware updates.
Update19: Added to run Set-ExecutionPolicy -ExecutionPolicy RemoteSigned in Exchange Management Shell before running the script.
Update20: The definition download takes 30-45 minutes (maybe slower depending on Internet speed), so allow for some time.
Update21: Added solution to updateservice.exe access denied error when running the script.
Subscribe to the newsletter and stay up to date with the latest IT tips and tricks.
Did this article help you? Share this article with your network and help them too.
What Others Are Reading
Revisiting this after a while and my ExchangeHealthChecker is flagging me as this issue is being detected: “Detected problematic FIP-FS version 2201010006 directory Although it should not cause any problems, we recommend performing a FIP-FS reset.”
I run .\Reset-ScanEngineVersion.ps1 and get told: “EXCHANGE16 UpdateVersion: 2112332256
EXCHANGE16 This server is not impacted. Add -Force to proceed anyway.”
I then run Get-EngineUpdateInformation to verify and sure as shootin:
Engine : Microsoft
LastChecked : 11/15/2022 09:50:17 AM -05:00
LastUpdated : 11/15/2022 09:37:42 AM -05:00
EngineVersion : 1.1.19800.4
SignatureVersion : 1.379.386.0
SignatureDateTime : 11/15/2022 01:38:25 AM -05:00
UpdateVersion : 2112332256
UpdateStatus : UpdateAttemptNoUpdate
Sooooo, should I try forcing it or is there something wrong with the HealthChecker script?
Appreciate any thoughts,
Steve
Hi Ali, I have never seen such a great website. It’s maintained and easy to follow. You always help me and other colleagues out. Please do never stop otherwise we will really lose the best. Much thanks!!
I tried to run the script to “ResetScanEngineVersion.ps1” and the process updateservice.exe did not stop, I tried to stop this process and receive “Access Denied”.
What I need to do to sto this process and update the Scan Engine?
Hi,
I have the same problem… did you by any chance manage to stop it?
Thank you in advance,
I ran into the exact same problem and the solution was, the account you are logged in as on the Exchange server, make sure it has ‘Debug Programs’ permission (https://www.stigviewer.com/stig/windows_server_2016/2017-05-18/finding/V-73755). Once I added the account to it, I was able to kill/end the process successfully and the script worked as expected.
Thanks Tabish.
It solved my problem.
We found your article yesterday and it fixed our 2016 Hybrid Exchange Server within minutes.
Thank you Ali!
Matt
Thanks ALI!! My 2019 server had this issue right away and couldn’t get what was going on until I found your article. Thanks for the Great Article. I manage several exchange servers, so thank you.
When a message in the Submission queue reach MessageExpirationTimeout, is this logged somewhere?
I would very much like to find out what messages were lost.
Works for me, but after I rebooted the Exchange 2016 server
Thanks for this post, i will follow, i have a client with DAG, for now the oficcial script FIX not work for us, only disabling the Antimalware services is fixing the issue.
Thanks Ali
Thank you so much Ali…
I just used the MS patch and it was fine.
Exchange 2016
Exchange Sever 2016 CU22,
Just update to 2112330013, but still have Event ID: 1106,
Done with 1a, 1b, and reboot, hmmmm same , Stuck emails,
any idea?
———————————————–
Engine : Microsoft
LastChecked : 01/04/2022 04:17:26 PM +08:00
LastUpdated : 01/04/2022 04:10:30 PM +08:00
EngineVersion : 1.1.18800.4
SignatureVersion : 1.355.1377.0
SignatureDateTime : 01/04/2022 05:36:20 AM +08:00
UpdateVersion : 2112330013
UpdateStatus : UpdateAttemptNoUpdate
——————————
Log Name: Application
Source: Microsoft-Filtering-FIPFS
Date: 1/4/2022 5:58:35 PM
Event ID: 1106
Task Category: None
Level: Error
Keywords:
User: SYSTEM
Computer: XXX
Description:
The FIP-FS Scan Process failed initialization. Error: 0x80004005. Error Details: Unspecified error
Event Xml:
1106
0
2
0
0
0x8000000000000000
32922329
Application
XXX
0x80004005
Unspecified error
Dear Ali,
I had this issue in my Exchange, so I bypassed the filtering. Now, I want to apply the fix released my Microsoft and enable the filtering again but the problem is I do not have an internet connection on my server to download the latest antimalware engine update. what should be done in this case ?
Exchange 2013 did not break the mailflow, but antimalware updates are not coming through anymore. In our case 2201010009 was the last one.
The automatic or manual solution has to be applied to 2013 Exchange servers as well if they installed the 220101* engine update, otherwise the 2013 server will not get new antimalwareupdates.
Thank you a lot. It’s works. Great article!!!
Thanks! This fixed our issue.
This procedure also works for Exchange 2016. Note: I received FIPFS errors until the update status changed to completed.
hi, I am getting this error both manually running or automatic solution
D:\exchangeserver\scripts\Update-MalwareFilteringServer.ps1: An error occurred when starting the anti-malware engine update.
Location Line: 1 Character: 1
+ .\Update-MalwareFilteringServer.ps1 ex02.domain.com
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo: NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId: Microsoft.PowerShell.Commands.WriteErrorException,Update-MalwareFilteringServer.ps1
anyone can help please. Thanks
Hello,
i have the same issue did you already got an solution for this?
BR
Thanks for your post about this mail flow. It helped us in an efficient way.
By the way, many thanks to your tweet which was my first read about this problem,
So, Thanks !
Happy New Year and keep safe.
Ber
Hello,
I ran the script. The engines are successfully updated. But when enable the AntimalwareScanning, the FIPFS error, appear again.
A FIP-FS Scan process returned error 0x84004003 PID: 19816 Msg: Scanning Process caught exception:
Stream ID:
ScanID: {FA185D1D-7B0E-483C-8F23-CBF4FC81F2A7}
(0x84004003) Unknown error 2214608899. Failed to meet engine bias criteria (Available) for filter type (Malware):
Selected engine(s): Microsoft
Available engine(s):
Offline engine(s): ID: {fa185d1d-7b0e-483c-8f23-cbf4fc81f2a7}
After rebooting the server, tried to enable AntimalwareScanning again and the error is not appear.
So, it’s look like, sometimes, just restarting the service is not enough.
Same for me, after running the MS fix and re-enabling the Antimalware Scanner, the submission queue started to grow again. Rebooting the server resolved this second issue: Unknown error 2214608899. Failed to meet engine bias criteria (Available) for filter type (Malware)
I have tried the Script for Update and disabled Malware and enabled Bypass Filtering. But my Mails stay in outgoing connector. What should i do? Incoming Mails are working.
Hi,
I tried with the fix from MS, but don’t work. Need to ByPass… The update -> Failed.
Maybe MS servers are down…
after 4 reboot from Exchange server still not work.
Tried manually too -> no way
Will wait until MS release a KB or SU for Echange…
hi there , thanks for you.
I have done as mentioned in your article but when i am not able to update the engine. I tired manually updating but that also does not work. how to fix this ?
Engine : Microsoft
LastChecked : 01-03-2022 12:18:40 PM +03:00
LastUpdated : 05-22-2020 03:42:35 PM +03:00
EngineVersion : 1.1.17000.7
SignatureVersion : 1.315.1197.0
SignatureDateTime : 05-22-2020 10:21:15 AM +03:00
UpdateVersion : 2005220003
UpdateStatus : UpdateAttemptFailed
Same here buddy,
Tried the script (automatic) also manual. I do also get the UpdateAttemptFailed.
Exchange 2016
Have the same. Can not download scan engines :/ Maybe Ms servers overload?
Did you try to restart the Exchange Server?
No, i haven’t restarted exchange server. Will do that and check.
I have tired restarting and the same error i am , not able to update.
It worked now.
I did the following:
1.Stop de Microsoft Filtering Management Service
2.Stop de Microsoft Exchange Transport Service
3.Stop de Background Intelligent Transfer Service
4.Execute the Fix script.
5.The version did not update to 2112330001 zoals beschreven it still said 2112330003
But no more UpdateAttemptFailed anymore.
Glad to hear!
The UpdateVersion version should be 2112330001 or higher. So the version that you see in the output is good.
It looks like you were never getting the updates to begin with.
LastUpdated : 05-22-2020 03:42:35 PM +03:00
The version of the engine is very old as well
2005220003
I am guessing your servers cannot access the internet or the update site. If you are not seeing the queuing messages you are probably ok.
Thank your for your detailed blog. It worked like a charm on 2016 and 2019 exchange servers. As always: be patient with your exchange servers, download takes a little time. And don’t forget to activate your transport rules after this, if you had some for attachments.
Many thanks after applying this patch issue resolved all mails in queue got cleared instantly make sure to reboot and apply on all exchange nodes.
i have exchange 2013 cu23 in a DAG setup and i have this exact problem. So i think 2013 could be affected by this too
Mail didn’t work and there was panic. The first thing I did was to visit your website alitajran.com because I knew that you will have the answer.
Everything works fine and mail flows. Microsoft should kiss you because you help so many!!
Thanks and happy new year!
Ali, Thanks for your post.
1. Run Enable-AntimalwareScanning.ps1 PowerShell script
2. Verify that antimalware scanning is enabled
3. Reboot the Exchange Server (Important)
4. Check mail queue messages
5. Test mail flow
That worked for me.
Step 3 was very very important.
Had a great break and didn’t look at any emails … When I did noticed no emails since 31st .. Checked other customers – same result
Found your blog post Ali !
What a result .. mine and customers servers back up and running delivering and sending email.
Thanks MS for screwing up mine and many other’s breaks !!
After updating to version 2112330001, it still didn’t work even after a reboot. However, this morning after updating to version 2112330002 and REBOOTING, we’re all back to normal.
Thanks for this post.
Hi,
I trying to use the automatic script but I have problem with download scan engines :/ Goes to max 84932811 / 179030563 and stuck 🙁 anyone has the same to?
The definition download can take up to 30-45 minutes. Give it time.
I know that it takes some time, but it definitely stucking in diffrent moments of downloading. Sometimes 100mb sometimes 800mb but never end download of full 179030563 :/
Important caveat: if you enable .\Enable-AntimalwareScanning.ps1 before running the update scripts like I did – then you may find that the queues don’t empty even though you will appear to be updated, you have to run it again so you get the following line” updating Microsoft” then restart the transport service – only then did the queue empty on a few of my servers – hope that helps someone
[PS] E:\Program Files\Microsoft\Exchange Server\V15\Scripts>.\Enable-AntimalwareScanning.ps1
Anti-malware engines are updating. This may take a few minutes.
Checking for engines updated after 2021-12-26 4:33:14 PM.
Updating Microsoft. Last updated : 2022-01-02 4:19:13 PM <<< here
WARNING: The following service restart is required for the change(s) to take effect : MSExchangeTransport
Many thanks for your excellent work Ali!
Thanks Ali,
I had to reboot exchange 2016 to get mail flowing. Now I get the following error running Get-EngineUpdateInformation The term ‘Get-EngineUpdateInformation’ is not recognized as the name of a cmdlet”.
Any Ideas? It worked after running script, re-enabling antimalware by script, then command stopped working after reboot.
I already have added this in the article:
1. Run Exchange Management Shell as administrator
2. Run the command: Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell
3. Run the command: Get-EngineUpdateInformation
You get this message because you didn’t add the PSSnapin, which is required to run the Get-EngineUpdateInformation cmdlet. The script did add that, and that’s why it worked for you after running the script.
Genius, that worked perfectly.
My other issue since the fix is an eventlog error #74: Process w3wp.exe Connection leak detected for key xyz.local/../../xyzadmin in Microsoft.Exchange.Configuration.Authorization.WSManBudgetManager class. Leaked Value 1. Do you think this has anything to do with the Microsoft script?
Thanks
Hi Ali,
I just closed Exchange Management shell and reopened as Administrator and had to add the snapin again to run Get-EngineUpdateInformation. I Suspect the same happened after I rebooted.
Whew! I arrived at your site after searching for this issue online and found you explanations concise and clear and the fixes were well laid out and accurate. You saved the day for me! I will be following this site for updates. Happy New Year!
Hey Ali, thanks for the info. It´s happening right now in a customer with 2016 but this 2016 was only introduced for the future use of an Hybrid Configuration. They have 2010. One question, I think the antimalware engine was enabled by error when we setup the server so we don´t need it. If I understand correctly, if we just disable the Antimalware Engine, will be fine? Or even with that we need to apply the patch? Thanks!
I recommend patching the Exchange Server, and if you want to keep malware scanning disabled, that’s okay and up to you.
Fixed by running these command in this order (without need to restart the server)
1. .\Reset-ScanEngineVersion.ps1
2. & $env:ExchangeInstallPath\Scripts\Enable-AntimalwareScanning.ps1
3. Stop-Service MSExchangeTransport
4. Restart-Service FMS
5. Start-Service MSExchangeTransport
Thanks guys, have any of you noticed that the emails stop coming in and you have to restart the transport service again?
I’m glad our Exchange Servers 2013 were not affected but I would like to find informations about WHY
I saw that Microsoft “switched off” the PrimaryUpdatePath around 2022-01-01 17:30 UTC at
http://forefrontdl.microsoft.com/server/scanengineupdate
my Servers made the Update before BUT no Mails stuck
as you can see update version is one starting with 22
luckily no errors but I have the feeling I should do something to prevent future problems
[PS] C:\Windows\system32>Get-EngineUpdateInformation
Engine : Microsoft
LastChecked : 01.02.2022 09:13:25 +01:00
LastUpdated : 01.01.2022 06:11:25 +01:00
EngineVersion : 1.1.18800.4
SignatureVersion : 1.355.1247.0
SignatureDateTime : 01.01.2022 12:29:06 +01:00
UpdateVersion : 2201010009
UpdateStatus : UpdateAttemptFailed
@GreySlater,
I am also exactly on the same situation. We have Exchange 2013 on DAG setup and transport looped to third-party email filtering appliance. But still the BypassFiltering is Disabled and AntiMalwareScanning is Enabled. Our mail flow is not affected but the EngineUpdateInformaiton is exacely the same as yours.
I am not sure I should do anyting at this moment. Did you take any further action, and if so what is the status?
Thanks!
I disabled the scan with Disable-Antimalwarescanning.ps1 yesterday as workaround, then the mailflow restored.
After applying the Reset-ScanEngineVersion.ps1 script the update version is 2112330001 which is correct as mentioned in the post.
When i re-enable malware scan using $env:ExchangeInstallPath\Scripts\Enable-Antimalwarescanning.ps1 and restarting transport with Restart-Service MSExchangeTransport, new mail are stuck again.
Anyone else experienced this already? I tested on 2 different environments with the same result.
Event viewer shows event 2203 FIPFS:
A FIP-FS Scan process returned error 0x84004003 PID: 5480 Msg: Scanning Process caught exception:
Stream ID:
ScanID: {D6758B2F-F79A-400D-B5D7-3CD7065C1735}
(0x84004003) Unknown error 2214608899. Failed to meet engine bias criteria (Available) for filter type (Malware):
Selected engine(s): Microsoft
Available engine(s):
Offline engine(s): ID: {d6758b2f-f79a-400d-b5d7-3cd7065c1735}
1. Run Enable-AntimalwareScanning.ps1 PowerShell script
2. Verify that antimalware scanning is enabled
3. Reboot the Exchange Server (Important)
4. Check mail queue messages
5. Test mail flow
Let us know if that worked.
No it did not work.
Had to turn off antimalware again for mail to flow.
Thank you,
Jason
Just noticed the IMPORTANT reboot.
After reboot email is flowing both directions with antimalware enabled.
Many thanks!
Jason
Silly, yes, a full reboot of the server resolves this after the fix and re-enable.
Thanks for all your effort on this one Ali!
Just run “Restart-Service FMS -Force” instead of rebooting the server.
Vincent
Thanks Vincent
I can confirm this – restarting just the ‘Microsoft Filtering Management Service’ is enough
Chris
Hey Mike,
Make sure you reboot the server once antimalware has been enabled and the transport service has been restarted.
Cheers!
Jason
You should finally restart your Exchange Servers. Without restarting server, my Exchange Server was stuck without mail flow. Many thanks to Ali.
EXCHANGE 2019 15.02.0986.005 – Followed each step;
1.) Disabled antimalware
2.) Restarted Transport
3.) Confirmed antimalware state was False
4.) Email is now sending and receiving properly
5.) Ran rest antimalware script (Took 10 Minutes on a Gigabit connection to download) finished successfully.
6.) Verified version is 2112330001
7.) Changed timeout from 2 days to 4 days
8.) Enabled antimalware
9.) Restarted Transport
10.) Sent test email message.
11.) Message stuck in queue
12.) Disabled antimalware
13.) Restarted Transport
14.) Email test immediately was relapsed from queue and deliver successfully and reply was received.
Summary: Stopping the antimalware solves the send and receive issue. Despite updating the scan engine the problem still exists when antimale is enabled. Will wait for a stable fix before I re-enable antimalware. Fortunately we have a frontend advanced threat protection in place.
Thank you for all the info on this great start to 2022!
Jason Z
Thanks for letting us know the steps you have taken. Perhaps a complete Exchange Server restart will help.
Same here after update:
Enable-malware script run
Set-MalwareFilteringServer -BypassFiltering $false
Mail stuck again in queue and event (each time message comes in queue):
A FIP-FS Scan process returned error 0x84004003 PID: 22260 Msg: Scanning Process caught exception:
Stream ID:
ScanID: {ADA3CC38-9DFF-41E2-8033-71E4016F6CDB}
(0x84004003) Unknown error 2214608899. Failed to meet engine bias criteria (Available) for filter type (Malware):
Selected engine(s): Microsoft
Available engine(s):
Offline engine(s): ID: {ada3cc38-9dff-41e2-8033-71e4016f6cdb}
Run script to disable malware scanning, and there is mailflow again (but with Set-MalwareFilteringServer -BypassFiltering $false)
I can confirm as mentioned, that reboot of server is solving all issues after applying the update update script. Run the enable malware script, reboot the server, and all working fine now.
Thx for your updates on this website, will bookmark it for future!
I’m getting:
E:\Program Files\Microsoft\Exchange Server\V15\Scripts\Update-MalwareFilteringServer.ps1 : Error starting the anti-malware engine update.
Anyone else having this issue?
Getting same error on one of my servers. have not figured it out yet
Same here.
Start-Engine cmdlet seems to work and the update path is http://amupdatedl.microsoft.com/server/amupdate if you check via get-malwarefilteringserver
The update is still in progress
ExShell is telling me Start-EngineUpdate isn’t a recognized commandlet
Can you try these steps:
1. Run Exchange Management Shell as administrator
2. Run Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell
3. Run Start-EngineUpdate
4. Run Get-EngineUpdateInformation
Ignore that. I forgot to start the snap-in.
I am. Can’t see a way around it as yet. Am up to 1/4 million queue now.
I’m getting the same error. All paths appear to be correct when called directly.
Anyone able to get past this?
It seems the command work remotely when calling Update-MalwareFilteringServer.ps1. Run the command from another server and put in your remote destination server.
Alternatively run it manually
Add-PSSnapin microsoft.forefront.filtering.management.powershell
Get-EngineUpdateInformation
Start-EngineUpdate
Get-EngineUpdateInformation
Thanks for this, can confirm this worked. I re-enabled malware filtering after and rebooted.
Wow! I came upon this post of yours after I exhausted several hours and decided to actually disable the Antimalware scanning.
Then I saw this. Just wow that you captured it Ali. Thanks!
Microsoft patch is out. Apply the automatic solution or manual solution as mentioned in the article. Don’t forget to re-enable anti-malware scanning.
Thanks Ali, very informative and up to date post. Keep us updated!
Microsoft patch is out. Apply the automatic solution or manual solution as mentioned in the article. Don’t forget to re-enable anti-malware scanning.
Thank you Ali Tajran for post and help, it is solved our problem.
Thanks and Regards,
It worked like a champ for me. Mail flow is working now. Since malware scanning is disabled, what are the risk involved ?
…yes, thank you for the information and the dedication to your craft. I’m sure many people have arrived at your site, as well as others, when researching this issue and breathed a sigh of relief. Relief at having a solution to make things work, but also being able to say, “Hey! It wasn’t me. Microsoft really messed up on this one.”
has anyone noticed that Submission queue is still stacking up?
Dear Ali,
This is a life savior article. Thanks a lot.
I ended up disabling the Antimalware agent and email flow working.
Thanks a lot, but after disabling malware agent, server will in risk? Any advice please
Is there a way to disable this via the registry?
Ali bey merhaba,
Aşağıda ki gibi bir uyarı alıyoruz, sebebi nedir?
[PS] C:\Windows\system32>Set-MalwareFilteringServer excserver -BypassFiltering $true
WARNING: The command completed successfully but no settings of ‘EXCSERVER’ have been modified.
Thanks, I was going crazy trying to figure out why mail was not flowing. Really appreciate your posting. Happy new year!
Thank you. We had the same issue today and I came across this article that provided the fix! We bypassed ours for now to get smtp working again. Figures MS would release something with a bug. Please keep us posted on an update to fix it and how. Thanks again!
I will update the article when Microsoft releases the patch to fix this bug. Stay tuned.
Which versions of Exchange does this apply to?
Thanks for a great article.
Only Exchange Server 2016 and Exchange Server 2019 are affected by this bug, and the fix applies to both these Exchange Server versions.
We also ran into the same issue and thank you for this article. Our SMTP stopped working and by bypassing it, fixed the issue for now till MS fixes their bug.
You’re a life saver.
Watching my queues empty now.
Great blog.
Now maybe I can enjoy the rest of my holiday!
WTF Microsoft??
Does anyone know if setting the BypassFiltering flag to true will also disable 3rd party malware scanners (e.g. ESET Mail Security) running on the transport server?
I only needed to disable any Transport rules that may check mails for Credit card numbers, attachments as they use the same engine, disable these and mail will work… did not have to disable malware scanning completely.
yeah, scratch that… I had to bypass filtering…
Thanks brother. I was like WTF this morning.
Your solution was linked on another forum and I appreciate it.
I used “Disable anti malware scanning (1a+2a)”
Thanks so much for this– it probably saved me some time and some hair that I can’t afford to lose at my age.
Is anyone else seeing messages piled up in their Poison queue from this? I was able to resubmit a bunch of them, but there are quite a few still stuck in there that I can’t get to resubmit (all were received after 1/1/2022 00:00:00).
Thanks a lot for this article ! 🙏🙏🙏
Thank you Ali, worked for me as well. Official microsoft attack 🙂
Is there an official Microsoft page that we can use to track this issue and share it with leadership.
Thank you for keeping us updated.
No, there is no official Microsoft channel yet. However, I have already sent messages to some of the Exchange product leaders that work for Microsoft Exchange Server.
Thank god I migrated to google workspace.
Exchange Online (Office 365/Microsoft 365) does not have this issue. It’s only happening for on-premises Exchange Server.
Hello. Thanks for the article. However, this hasn’t resolved out issue. Any other updates please?
Me has salvado la vida, use la opción de Omitir el filtrado de antimalware y funcionó
Thanks for the blog post, but I don’t think step 2a is correct. E.g. if you have 10 servers then it will just repeat the same command 10 times rather than doing 10 different commands; it’s not using the output from Get-ExchangeServer as an implicit parameter. Putting that another way, I can’t see a way to run Get-TransportAgent for a specific server. I think that you need to run the command once on each server, i.e. log into each server in turn and run the command locally.
The command is correct, and it will check all the Exchange Servers.
Thanks.
You rock Ali! I was troubleshooting this last night until about 4AM with no luck, this article was a huge help! Thanks for all of your Exchange blog posts!
Thank you so much
We ran into this on our tenant about 7pm cst last night and had to figure it out from the logs. After looking at the log files it appeared FIP-FS Scan engine was failing to load because it couldn’t convert the “2201010002” as it considered it to long. Thanks 2022. This caused the it to fail the scan and since transport service is dependent on the antimalware scan it would just give up. Disabling the anti-malware service allowed the transport service to bypass it (after restarting the service) and gives us a temporary fix. Our ticket with Microsoft sat for hours without a response so yeah doesn’t look like they are helping any time soon. Guess we will see what they say on Monday when they actually answer.
Ali – always appreciate all your posts.
Does this need to be run for EXO services also?> We have a DAG and Hybrid setup with few mailboxes online. I still see the queues pending on hybrid server, other DAG nodes are cleared now
but I do too many policies on EXO to disable and not sure which one is the breaking flow
You should also run this on the Exchange Hybrid (on-premises server). After that, run the command to verify that anti-malware is disabled/bypassed successfully. Finally, restart the Microsoft Exchange Transport Service on all the Exchange Servers.
Thanks for the post, awesome. One question tho: what’s the difference between 1a and 1b? I have Transport Rules which do not work anymore after disabling antimalwarescanning described in 1a. I have not tested 1b yet.
In the end, it’s the same.
Thank you!
Just another ‘thank you’ Ali, as this impacted me and your post was very timely and helpful. Much appreciated!
thanks for the great article and specially the updates!
This is a life savior article. Thanks a lot.
I ended up disabling the Antimalware agent, I believe Microsoft is still sleeping and didn’t think we’ll make it through to 2022.
Now we have to wait for them to wake up and maybe release a patch to resolve this urgently as think is somehow putting our servers at risk whether we disable or bypass the MalwareFiltering.
antispam engine below does not work either
Engine Version:1.1.18800.4
Signature Version: 1.355.1247.0
Manual updates of the Antimalware is run with following command line. May require EMS in administrator. $env:ExchangeInstallPath\Scripts\Update-MalwareFilteringServer.ps1 -Identity
Update Details in Event Viewer in Application under ID 6033
The updates probably won’t work until they either change the date format, or they reprogram it to not use the integer type that it currently uses.
I resolved this problem another way ….
Set-MalwareFilteringServer EXserver -BypassFiltering $true
and then restart transport service
get-MalwareFilteringServer EXserver
Name ForceRescan BypassFiltering DeferWaitTime DeferAttempts UpdateFrequency PrimaryUpdatePath
—- ———– ————— ————- ————- ————— —————–
EXserver False True 5 3 30 http://amupdatedl.microsoft.com/server/amupdate
That will work. I updated the article with both methods.
Me salvaste el año nuevo, muchas gracias!!!
I also appreciated your post! This helped us get back up as well.
Microsoft is sleeping. That’s why we have your blog. The commands worked perfect and mail queue is empty.
Have you seen any official Microsoft announcements about this somewhere?
There is nothing yet from Microsoft.
Thank you Ali!!!
download takes forever….
Thanks for your post, it helped os solve the issue today 🙂
You’re welcome. Keep an eye on this article for further updates.