FUDforum
Fast Uncompromising Discussions. FUDforum will get your users talking.

Home » FUDforum » How To » Mailing list integration question
Show: Today's Messages :: Unread Messages :: Show Polls :: Message Navigator
| Subscribe to topic | Bookmark topic 
Switch to threaded view of this topic Create a new topic Submit Reply
Mailing list integration question [message #158742] Tue, 17 March 2009 08:51 Go to next message
Jim O is currently offline  Jim O   United States
Messages: 4
Registered: March 2009
Karma: 0
Junior Member
add to buddy list
ignore all messages by this user
I am in the process of setting up a new forum part of which I want to integrate with a mailing list.

My question regards the headers sent out by FUDforum. Specifically will they include the "In-Reply-To:" and/or "References:" headers so that they are properly sorted by mailing list archives and MUA's?


Jim
Re: Mailing list integration question [message #158743 is a reply to message #158742] Tue, 17 March 2009 10:10 Go to previous messageGo to next message
naudefj is currently offline  naudefj   South Africa
Messages: 3632
Registered: December 2004
Karma: 17
Senior Member
Administrator
Core Developer
add to buddy list
ignore all messages by this user
Hi Jim,

I haven't tried it myself. However, from "mlist_post.inc" it looks like it would use an "In-Reply-To":

if ($reply_to) {
        $header .= "In-Reply-To: <".$reply_to.">\n";
}


Best regards.

Frank
Re: Mailing list integration question [message #158777 is a reply to message #158743] Thu, 19 March 2009 21:01 Go to previous messageGo to next message
Jim O is currently offline  Jim O   United States
Messages: 4
Registered: March 2009
Karma: 0
Junior Member
add to buddy list
ignore all messages by this user
Thanks Frank.

In doing some testing I found that all messages in a thread seem to default to "in reply to" the original entry, no the most recent one. That could affect threading on list archives and in MUA's.

Also, how hard would it be to add an extra field for references headers and include them? This is a request of list members.


Jim
Re: Mailing list integration question [message #158779 is a reply to message #158777] Fri, 20 March 2009 11:39 Go to previous messageGo to next message
naudefj is currently offline  naudefj   South Africa
Messages: 3632
Registered: December 2004
Karma: 17
Senior Member
Administrator
Core Developer
add to buddy list
ignore all messages by this user
I shouldn't be too difficult. However, I don't have time to do it right now (unless you can somehow motivate me). If you are willing to take it on, I can guide you through the high-level steps.

Best regards.

Frank
Re: Mailing list integration question [message #167484 is a reply to message #158779] Sat, 30 June 2012 23:57 Go to previous messageGo to next message
cpreston is currently offline  cpreston   United States
Messages: 160
Registered: June 2012
Location: Oceanside
Karma: 6
Senior Member
add to buddy list
ignore all messages by this user
Like to bump this thread. I also would like for the mailing list integration piece to include these headers. Naudefj, I'm prepared to motivate you. Send me a PM on what would do that. Wink

There's also one other feature I'd like to add to the mailing list module, and I'd be glad to pay for the dev time. On my current phpbb/mail2forum setup, I have moderation turned on. But I've changed the default behavior so that once you've been successfully moderated, you're no longer moderated. Basically I read the person's first post(s). If they're not spam, then I approve their posts. Once I approve their post, their user id is put in a safe_users table, and they're never moderated again. Possible?
Re: Mailing list integration question [message #168785 is a reply to message #167484] Thu, 25 July 2013 00:41 Go to previous messageGo to next message
cpreston is currently offline  cpreston   United States
Messages: 160
Registered: June 2012
Location: Oceanside
Karma: 6
Senior Member
add to buddy list
ignore all messages by this user
Frank,

I really need this now and am prepared to pay for it. Please send me a PM or email to curtis - at - backupcentral.com. TIA
Re: Mailing list integration question [message #168815 is a reply to message #168785] Fri, 26 July 2013 20:40 Go to previous messageGo to next message
cpreston is currently offline  cpreston   United States
Messages: 160
Registered: June 2012
Location: Oceanside
Karma: 6
Senior Member
add to buddy list
ignore all messages by this user
Frank,

I know you have a real job. Wink So unless I hear a "yo!" real soon, I'm going to get my developers started on this. Here's what I've come up with for a plan. It would be great if you could take a look and see what you think.

I plan to edit the core FUDforum code so that it includes "In-Reply-To" and "References" headers when it sends a reply to an email.

As I understand it, to do this, I need to do the following:
1. Get the Message-ID of each message
- Grab the Message-ID from any incoming emails
- Create and/or get the Message-ID for any outgoing emails

2. Store the Message-ID of each post, associated with the post.

3. Insert "In-Reply-To" and "References" headers into any outgoing emails:

(In the following, "Previous-Message-ID" is the Message-ID of message currently being replied TO, AKA the Message-ID of the previous post in the thread -- not the message-ID of the outbound message.)

<code>In-Reply-To: <Previous-Message-ID>
References: <First-message-ID-of-current-thread> <next-message-ID-in-thread> <next-message-ID-in-thread> <etc> <Previous-message-ID></code>

(If there is only one previous message in thread, there is only one message-ID in the References line. If there are more than ten previous message IDs, you are supposed to drop the second one until you have ten or fewer. Always start with the first-message-ID and end with the Previous-Message-ID.)

I will submit diffs from the core code. It seems like adding a field to the posts table would be the most likely place to store the Message-ID for each post, but that does mean adding a field to a core table. What do you think about that, Frank?

Just in case I read it wrong, the details of how to do In-reply-to and References headers is based on the info in this web page:
http://cr.yp.to/immhf/thread.html
Re: Mailing list integration question [message #168823 is a reply to message #168815] Sat, 27 July 2013 09:50 Go to previous messageGo to next message
naudefj is currently offline  naudefj   
Messages: 3632
Registered: December 2004
Karma: 17
Senior Member
Administrator
Core Developer
add to buddy list
ignore all messages by this user
Some comments:

FUDforum doesn't generate Message-ID's, but will grab and store it if the message is returned to it from the list (fud30_msg.mlist_msg_id column). So, 1 & 2 are already implemented. The obvious problem with this approach is that users can reply to a message on the forum before its Message-ID is captured. Current solution is to sync frequently (every few seconds).

Function mail_list_post() in mlist_post.inc will add 'In-Reply-To:' headers to mails. Next step would be to get someone to provide code to add the 'References:' header as well.

Best regards.
Frank
Re: Mailing list integration question [message #168828 is a reply to message #168823] Sat, 27 July 2013 23:38 Go to previous messageGo to next message
cpreston is currently offline  cpreston   United States
Messages: 160
Registered: June 2012
Location: Oceanside
Karma: 6
Senior Member
add to buddy list
ignore all messages by this user
You're using PHPMailer, right? It is generating the Message-ID, but it's making a random one because you're not specifying it as an environment variable MessageID. That is according to this:
http://www.tig12.net/downloads/apidocs/wp/wp-includes/PHPMailer.class.html# det_fields_MessageID

What message-ID are you grabbing if you're getting it from the one the list returns? Are you grabbing the message ID of the incoming email or the Message-ID of the email you sent that it is replying to, which would be the one you should specify in the "In-Reply-To" field. Whichever one you're grabbing, we end up with not enough info for a complete "References" tag. If you're grabbing the Message-ID of incoming message that is a reply to a forum post, you have the latest message-ID but not the first one, which is what you're supposed to put in the References tag. If you're getting the previous one, then you don't have the current one....

It seems more proper to generate the Message-ID for any outgoing messages, and then pass them on to PHPMailer (pretty easy to generate one http://www.jwz.org/doc/mid.html), then store it with the message in the mlist_msg_id column. Then you do what you're already doing and grab any Message-IDs of any incoming messages and store them in the same column. Then we've got Message-IDs for every message.

You're already doing the right thing with mail_list_post with the "In-Reply-To:" tag. Since we would have Message-IDs for all messages, then it's just a matter of a database query to get the first, last, and other Message-IDs so we can generate the References header.

What do you think?
Re: Mailing list integration question [message #168830 is a reply to message #168828] Sun, 28 July 2013 02:38 Go to previous messageGo to next message
cpreston is currently offline  cpreston   United States
Messages: 160
Registered: June 2012
Location: Oceanside
Karma: 6
Senior Member
add to buddy list
ignore all messages by this user
I've been poking around in the code, and I have a few questions.

Lines 45-49 of mlist_post.inc seems to be creating our own Message ID, updating mlist_msg_id w/the value, and making it part of the email before it is sent. So isn't the code already doing what I'm suggesting regarding Message IDs?

If my understanding is correct, all messages that have been sent out via email should have a messageid in their database entry. That means all I really need to do to accomplish what I want is the proper database query and formatting of a References header. It would seem that all this could be put right after line 41 in moist_post.inc.

This one could be my misunderstanding of PHP, but doesn't line 12 of moist_post.inc set the reply_to to null no matter what is passed to the function? Lines 40-42 would then not do anything because reply-to will always be null? I see similar behavior in the send_email function in remail.php, where $header is set to '' no matter what headers are sent to it. This would suggest that even if the Reply-To header was set, it wouldn't be used.

it appears that the send_email function (with the class fur_smtp) in remail.php sends the actual email, so you're not using PHPMailer. is that correct?
Re: Mailing list integration question [message #168839 is a reply to message #168830] Sun, 28 July 2013 12:20 Go to previous messageGo to next message
naudefj is currently offline  naudefj   
Messages: 3632
Registered: December 2004
Karma: 17
Senior Member
Administrator
Core Developer
add to buddy list
ignore all messages by this user
cpreston wrote on Sun, 28 July 2013 08:38
Lines 45-49 of mlist_post.inc seems to be creating our own Message ID, updating mlist_msg_id w/the value, and making it part of the email before it is sent. So isn't the code already doing what I'm suggesting regarding Message IDs?

Yes, you're right! Just remember that the mailing list may assign a different Message ID, and we will grab and store the list's ID as soon as we spot the message on the list.

cpreston wrote on Sun, 28 July 2013 08:38
This one could be my misunderstanding of PHP, but doesn't line 12 of moist_post.inc set the reply_to to null no matter what is passed to the function? Lines 40-42 would then not do anything because reply-to will always be null?

It sets the default value to NULL. So, if no value is supplied, we will start with nothing.

cpreston wrote on Sun, 28 July 2013 08:38
...it appears that the send_email function (with the class fur_smtp) in remail.php sends the actual email, so you're not using PHPMailer. is that correct?

Correct, we're most definitely not using PHPMailer. Should we?
Re: Mailing list integration question [message #168841 is a reply to message #168839] Sun, 28 July 2013 12:57 Go to previous messageGo to next message
cpreston is currently offline  cpreston   United States
Messages: 160
Registered: June 2012
Location: Oceanside
Karma: 6
Senior Member
add to buddy list
ignore all messages by this user
Quote:
Yes, you're right! Just remember that the mailing list may assign a different Message ID, and we will grab and store the list's ID as soon as we spot the message on the list.


That's not how Message-ID is SUPPOSED to work. Each message should have a message-id and should not be changed. Some mailing lists (like Mailman) will give an email a message-id if it doesn't have one, because it should have one for proper threading. I've verified that mailman and listserv will only generate a message-id if the incoming message doesn't have one, or if IT is the one sending the message (e.g. a reminder or something).

Quote:
we're most definitely not using PHPMailer. Should we?


Everybody else sending email in PHP sure does. Not sure how big of a deal it is to change that, though, or what benefit you would get by changing it.

[Updated on: Sun, 28 July 2013 15:37]

Report message to a moderator

Re: Mailing list integration question [message #168843 is a reply to message #168841] Sun, 28 July 2013 16:32 Go to previous messageGo to next message
cpreston is currently offline  cpreston   United States
Messages: 160
Registered: June 2012
Location: Oceanside
Karma: 6
Senior Member
add to buddy list
ignore all messages by this user
OK, I tested the functionality we're talking about. I viewed the Message-IDs and the In-Reply-To references in Outlook by looking at the email headers. I viewed what FUDforum was placing in the database by monitoring the mlist_msg_id column for the associated messages.

I verified:
1. FUDforum creates a Message-ID for original forum messages that are sent to a mailing list and it stores that info in the mlist_msg_id column
2. Neither FUDforum nor Mailman changes a Message-ID of an incoming message if it has one (which it should if it was sent by any decent MTA), but Mailman will give it one if it doesn't have one
3. FUDforum does properly grab the Message-ID of the incoming message and stores that in the mlist_msg_id column
4. Most importantly, FUDforum uses the values in that column to properly create the In-Reply-To header. (I did several posts back and forth from email and the forum, and each time FUDforum assigned the previous Message-ID as the In-Reply-To header.

So really all that's left is to create a database query that would get all the Message-IDs for a given thread, then create a References line based on that. Awesome!
Re: Mailing list integration question [message #168844 is a reply to message #168843] Sun, 28 July 2013 17:08 Go to previous messageGo to next message
cpreston is currently offline  cpreston   United States
Messages: 160
Registered: June 2012
Location: Oceanside
Karma: 6
Senior Member
add to buddy list
ignore all messages by this user
I'm guessing about line 48 in mlist_post.inc we do something like this. I'm reaching here since I barely understand the syntax, but I thought I'd give it a shot. It can probably done with one query and a join, right? I don't get joins. Wink I stole the db_saq string from another .inc file. I'm guessing it's what i need here. Wink

#Get the thread_id this message is in
db_saq('SELECT m.thread_id FROM $GLOBALS['DBHOST_TBL_PREFIX'] .'msg WHERE m.msg_id='. $msg_id .' AND m.subject='$subject');

#Get all the msg_ids from this thread, ordered by time.
db_saq('SELECT m.msg_id, m.post_stamp FROM $GLOBALS['DBHOST_TBL_PREFIX'] .'msg WHERE m.thread_id='. $thread_id .' ORDER BY m.post_stamp');

This should go into an array that we can then grab them from. Does that sound about right?
Re: Mailing list integration question [message #168868 is a reply to message #168844] Wed, 31 July 2013 13:18 Go to previous message
cpreston is currently offline  cpreston   United States
Messages: 160
Registered: June 2012
Location: Oceanside
Karma: 6
Senior Member
add to buddy list
ignore all messages by this user
I have completed the code to make this happen, but am going to post it in the hacks forum. Feel free to put it in 3.0.5! Once we figured out how to do it, it didn't take that many lines of code.
Quick Reply
Formatting Tools:   
  Switch to threaded view of this topic Create a new topic
Previous Topic: Not getting any emails from FUDforum
Next Topic: Post shows up above Sticky and Announcement Threads
Goto Forum:
  

-=] Back to Top [=-
[ Syndicate this forum (XML) ] [ RSS ]

Current Time: Mon Dec 18 17:32:05 EST 2017

Total time taken to generate the page: 0.00865 seconds