All my new received emails are in the same conversation

[Gumbraise]

Once, I received a automatic email from mailer-daemon@setebos.zianet.com and since that all my received emails are in this conversation. Even if I close it it re-open everytime.
What kin of information do you need to fix it?

[abhinavxd]

Hey!
Libredesk automatically drops auto replies so this shouldn't happen.

What version are you on?

[Gumbraise]

Hey!
Libredesk automatically drops auto replies so this shouldn't happen.

What version are you on?

The last one 0.7.0.

[abhinavxd]

Is it possible to share the raw email source with sensitive fields redacted? that would help me debug the issue.

In the meantime, you can fix it immediately by deleting the specific conversation from the database. Since LibreDesk skips auto-replies, this should resolve it going forward.

Let me know, thanks!

[mntmn]

We are also experiencing this problem now in production while running on version 0.5. I've upgraded to 0.7.1 but it didn't change the behavior. This is happening since 2 days, and yesterday I shut down the system to avoid chaos in our support and to be able to investigate the code and the database. I've made a full dump and am looking at the data locally. What I can already see is that since a certain point, all new messages are erraneously added to the same conversation with id 658:

[mntmn]

Another finding that may be close to the issue: when the problem started, the sender used the "cc" field of their email to send a copy to themselves. This cc field wasn't filled in before in the conversation_messages list as I can see. Also, the "source_id" field of this message is an empty string, which seems unusual. Other messages have either NULL or a UUID there.

[mntmn]

I found another thread where someone used the CC field with an actual value, but the problem did not occur there. But I filtered all conversation_messages and the problematic conversation is the only one that has an empty string as the source_id. Maybe this is the problem, that an empty string can get in there but is not handled properly in some query for message to conversation assigment?

[mntmn]

Luckily we were running in debug mode all the time and the logs have the moment where this message with empty source_id was inserted. It happened here where it just says message_id= (screenshot clipped for privacy regarding the message content):

[mntmn]

So I think this is an interplay of 2 bugs, but I'm not 100% sure yet:

1. The IMAP code can somehow yield an empty string message_id and this gets inserted into the DB (no validation).
2. This conversation with empty string source_id somehow matches as the existing conversation for any subsequent incoming message. So there must also be something slightly wrong with the code (maybe the SQL query) that finds an existing conversation for a new message.

[mntmn]

I investigated further and the root of Problem 1 is that go-imap v2 cannot parse some Message IDs and returns an empty string in this case. The email in this case has a Message ID that contains two @ signs (so the message ID is something like <message123@username@example.org>, which I believe is out of spec, but it is not uncommon. When googling for this issue, one can find some hits about Outlook generating such message IDs (at least in the past).

[mntmn]

The root cause is here emersion/go-message#154

@abhinavxd here's also a possible workaround emersion/go-message#154 (comment)

While waiting for a fix, I'm wondering how I could clean up our production DB. I can of course put the correct message ID manually into the DB row of the conversation. But how could I get some important messages that arrived in the meantime correctly fed into new conversations? Could I delete the converstation_messages for these new mails and would they be recreated on the next run (imap scan)?

[abhinavxd]

Hey,
Thanks for the detailed report and suggested fixes, let me check will patch this asap.

Yes, messages will be recreated and remapped once you set the correct message-id and delete any incorrectly merged messages.

You can do this (take DB backup first)

• Disable the inbox so all IMAP scanning stops.
• Delete the incorrectly merged messages.
• Set Scan Inbox Since to 72 hrs (for example) if you want to rescan all messages from the last 3 days ( or more )

In edit inbox form

[abhinavxd]

I've opened a PR #127 for this