When Microsoft
Exchange have problem cannot receive any email from other party, there will be
a lot of things can be happened. But make sure you pin point the exact issue by
looking at the server event logs for exchange. Also if possible to get the NDR
report from the sending party so we can know exactly what exchange spit out
when error happening. Usually a mature mail systems will have specific error so
administrator can resolve the problem.
On this case, we get
the error message from the sender which are
452-4-3-1-insufficient-system-resources. What we got from this was :
- Server is reachable
- SMTP service is up
- When complete the sending, it
fail with insufficient system resources
On Exchange
documentation, this error related to storage, so the situation was when
exchange going to store the receive message to disk, its spit error because
insufficient resources.
Microsoft Exchange Transport is rejecting message submissions because the available disk space has dropped below the configured threshold.The following resources are under pressure:
Queue database logging path (“C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\data\Queue\”) = 96% [High] [Normal=92% Medium=94% High=96%]
Physical memory load = 90% [limit is 94% to start dehydrating messages.]
The following components are disabled due to back pressure:
Inbound mail submission from Hub Transport servers
Inbound mail submission from the Internet
Mail submission from Pickup directory
Mail submission from Replay directory
Mail submission from Mailbox server
Content aggregation
The following resources are in normal state:
Queue database path (“C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\data\Queue\mail.que”) = 95% [Normal] [Normal=95% Medium=97% High=99%]
Version buckets = 0 [Normal] [Normal=80 Medium=120 High=200]
Private bytes = 14% [Normal] [Normal=71% Medium=73% High=75%]
Batch Point = 0 [Normal] [Normal=2000 Medium=4000 High=8000]
Submission Queue = 0 [Normal] [Normal=1000 Medium=2000 High=4000]
With Microsoft Exchange, it have a monitoring component which monitor available resources - Back Pressure, which is also tracking free space on a disk, where the Exchange Transport service queue are located.
Checking on the disk
resources on servers, we have plenty disk 1.4TB with free 100GB of disk. This
was strange.
Some behavior to note when the threshold limit exceeded, Exchange can :
- Medium (90%) threshold - Stop receiving mail over SMTP from external senders (MAPI client e-mails are yet processed)
- High (99%) threshold - The mailflow stops to be processed completely
Dig down more on the
event log, we found that the drive of Microsoft Exchange transport was because
the available disk space has dropped below the configured threshold. And we
read the documentation in Microsoft, it states that Microsoft Exchange transport
service will need retain a min 10% free disk space where the transport Role
folder resides.
So in this case the
size of 1.4TB will need minimum 10% to be free, which are 140GB, and in this
case not enough as it left only 100GB. This is by design to prevent disk full
and Microsoft Exchange crash.
The solution to this
was easy, just add more disk with expanding the drive, or just move transport
role folder to another disk drive if you cannot expand it. After the disk size
threshold surpassed, restart Microsoft exchange transport service and all will
be running again.
If you want to move the transport queue to another disk , you can edit the config in exchange install located in
$env:exchangeinstallpath\bin\EdgeTransport.exe.config wiith the following changes
<add key=”QueueDatabasePath” value=”C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\data\Queue” />
<add key=”QueueDatabaseLoggingPath” value=”C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\data\Queue” />
When change the trasnport queue path, leave the content and just restart transport service as Exchange will recreate the folder automatically.
Note : Old directory can safely be removed
In my case, we just extend the disk without restart the servers.