poniedziałek, 25 sierpnia 2014

Chasing email displaying problem (or How The World Was Saved)

A few days ago I was involved in investigation why an email that is sent by our application is displayed incorrectly by an application on recipient side. The problem seems trivial, and the root cause was indeed hilarious, but it took a couple of days to troubleshoot...

Our application (Java-based) sends emails using Postfix SMTP server running locally, which relays the email to a corporate mail server, which in turn delivers messages to intended recipients.


This setup has worked very well for us, but some time ago one of the third party recipients started to complain that emails from our production system are displayed incorrectly in their application. Their custom application consumes received emails from their mailbox and displays the content in a text window. The problem was that the email body was displayed garbled, like all line breaks were missing. They were surprised because during tests in UAT environment everything looked fine.
Troubleshooting was not made any easier by the fact that this third party is a huge financial corporation, with significant red tape and inertia. The people that observed the problem had little chance of getting proper IT support on their side, so the burden of investigating the issue was mostly on us - with mild pressure from the client as this issue was a blocker for business plans the client had with that third party.

Initially there was a suspicion that our production and UAT environments were configured differently and one sent emails with Linux-style line breaks (\r) and the other one with Windows-style endings (\r\n). Wrong format could confuse the application on the recipient side...We spent a couple of days comparing UAT and production configuration - no differences found. We went as far as to capture SMTP traffic in both environments and compare it in Wireshark to make sure that what leaves our estate is identical for production and preproduction environments. Nothing - both environments sent identical emails, barring recipient list (to/cc/bcc fields were populated differently for both environments).

Then the attention moved to the corporate mail server, based on Mimecast. The suspicion now was that there were some policies set up that would process preproduction and production emails differently. So we sent tens of emails from both environments trying to identify the reason, using different application configurations and different Mimecast policy settings. Again no conclusive outcome was seen - but interestingly the third party reported that production emails started to look fine at some point. The only difference between these emails and the emails that caused problems a few days back was that one 'cc' address was not present. That address was something like img@company.com.

As it turned out, the reason why perfectly good emails were displayed incorrectly was that the string "img" on recipient address list was interpreted as a start of HTML img tag by the third party application. Upon seeing this string the application switched to some other parsing mode, which caused email body to be interpreted and displayed incorrectly.

I must admit I was shocked to hear that the root cause was such a silly bug. Fortunately that img@company.com address belonged to our client and not to another third party involved in the business process. Therefore a workaround could be deployed quickly - the client set up a mailbox on their side with a different name, and we could get rid of that img address. World saved...

Brak komentarzy:

Prześlij komentarz