FUDforum
Fast Uncompromising Discussions. FUDforum will get your users talking.

Home » FUDforum Development » Bug Reports » nntp charset problems
Show: Today's Messages :: Unread Messages :: Show Polls :: Message Navigator
| Subscribe to topic | Bookmark topic 
Switch to threaded view of this topic Create a new topic Submit Reply
nntp charset problems [message #158908] Tue, 14 April 2009 05:33 Go to next message
webmaster_at_naszojciec is currently offline  webmaster_at_naszojciec   Poland
Messages: 4
Registered: April 2009
Karma: 0
Junior Member
add to buddy list
ignore all messages by this user
Changed handle of multi-line headers,
Changed regexp for charset detect
Added check for header 'content-transfer-encoding' == 'base64'
Convert send subject with iconv_mime_encode() but PHP5 only
Added charset convert in decode_header_value()


scripts_common.inc.orig 1.36
nntp.inc.orig 1.77

<?php
--- nntp.inc.orig       2009-04-14 10:46:11.000000000 +0200
+++ nntp.inc    2009-04-14 11:19:08.000000000 +0200
@@ -162,+162,@@
        function 
format_headers()
        {
                
/* convert to unix line endings and handle multi-line headers */
-               $this->headers str_replace(array("\r\n""\n "), array("\n",""), $this->headers);
+               
$this->headers str_replace("\r\n","\n"$this->headers);
+               
$this->headers preg_replace("/\n(\t| )+/"' '$this->headers);

                
$hdr explode("\n"trim($this->headers));
                
$this->headers = array();
@@ -
402,12 +403,15 @@
                                }
                        }

-                       if (isset(
$this->headers['content-type']) && preg_match('!charset="([^"]+)"!'$this->headers['content-type'], $m)) {
+                       if (isset(
$this->headers['content-type']) && preg_match('!charset="?([^"]+?)"?(;|\s|$)!'$this->headers['content-type'], $m)) {
                                
$charset $m[1];
                        } else {
                                
$charset $GLOBALS['CHARSET'];
                        }
-                       if (isset(
$this->headers['content-transfer-encoding']) && $this->headers['content-transfer-encoding'] == 'quoted-printable') {
+                       if (isset(
$this->headers['content-transfer-encoding']) &&
+                               (
$this->headers['content-transfer-encoding'] == 'quoted-printable'
+                                       || $this->headers['content-transfer-encoding'] == 'base64')
+                               ) {
                                
$enc $this->headers['content-transfer-encoding'];
                        } else {
                                
$enc '';
@@ -
420,+424,@@
                        }

                        
$msg_post->subject apply_custom_replace($this->subject);
+                       if ( !
preg_match('!(.*?)(=\?([^?]+)\?(Q|B)\?([^?]*)\?=)[[:space:]]*([^=]*)!i'$msg_post->subject)) {
+                               
$msg_post->subject decode_string($msg_post->subject,"",$charset);
+                       }
                        
$msg_post->body decode_string($this->body$enc$charset);
                        if ( !
strlen($msg_post->subject) ) {
                                
$msg_post->subject "(no subject)";
@@ -
546,+553,@@

                
fputs($this->fs"From: $from\r\n");
                
fputs($this->fs"Newsgroups: $this->newsgroup\r\n");
-               
fputs($this->fs"Subject: $subject\r\n");
+               
fputs($this->fsiconv_mime_encode("Subject:"$subject,
+                               array(
'scheme'=>'Q'"input-charset"=>$GLOBALS['CHARSET'],"output-charset"=>$GLOBALS['CHARSET']))
+                               .
"\r\n");

                if (
$GLOBALS['FORUM_TITLE']) {
                        
fputs($this->fs"Organization: ".$GLOBALS['FORUM_TITLE']."\r\n");
?>


<?php
--- scripts_common.inc.orig 2009-04-14 11:24:09.000000000 +0200
+++ scripts_common.inc  2009-04-14 11:27:26.000000000 +0200
@@ -147,+147,@@
                        
$ec_type strtolower($m[4][$i]);

                        if (
$ec_type == 'q') {
-                               
$newval .= decode_string(str_replace('_'' '$m[5][$i]), 'quoted-printable');
+                               
$newval .= decode_string(str_replace('_'' '$m[5][$i]), 'quoted-printable',$m[3][$i]);
                        } else if (
$ec_type == 'b') {
-                               
$newval .= decode_string($m[5][$i], 'base64');
+                               
$newval .= decode_string($m[5][$i], 'base64',$m[3][$i]);
                        }

                        if (!empty(
$m[5][$i])) {
?>
Re: nntp charset problems [message #158909 is a reply to message #158908] Tue, 14 April 2009 07:21 Go to previous messageGo to next message
naudefj is currently offline  naudefj   South Africa
Messages: 3624
Registered: December 2004
Karma: 17
Senior Member
Administrator
Core Developer
remove from buddy list
ignore all messages by this user
Great job! Thanks. I'll test it later and commit it to the CVS repository.

BTW: FUDforum still supports both PHP4 and 5 (I know it's time to drop PHP4, but that will probably only happen in the FUDforum 3.x range). To support both we should add an if-check around the "iconv_mime_encode" function.

Best regards.

Frank
Re: nntp charset problems [message #158910 is a reply to message #158909] Tue, 14 April 2009 07:50 Go to previous messageGo to next message
webmaster_at_naszojciec is currently offline  webmaster_at_naszojciec   Poland
Messages: 4
Registered: April 2009
Karma: 0
Junior Member
add to buddy list
ignore all messages by this user
This part won't work, remove it.

@@ -420,6 +424,9 @@
}

$msg_post->subject = apply_custom_replace($this->subject);
+ if ( !preg_match('!(.*?)(=\?([^?]+)\?(Q|B)\?([^?]*)\?=)[[:space:]]*([^=]*)!i', $msg_post->subject)) {
+ $msg_post->subject = decode_string($msg_post->subject,"",$charset);
+ }
$msg_post->body = decode_string($this->body, $enc, $charset);
if ( !strlen($msg_post->subject) ) {
$msg_post->subject = "(no subject)";
Re: nntp charset problems [message #158929 is a reply to message #158910] Thu, 16 April 2009 11:09 Go to previous messageGo to next message
naudefj is currently offline  naudefj   South Africa
Messages: 3624
Registered: December 2004
Karma: 17
Senior Member
Administrator
Core Developer
remove from buddy list
ignore all messages by this user
If you don't mind, can you please implement the above suggestions and repost the patch?
Re: nntp charset problems [message #158969 is a reply to message #158929] Mon, 20 April 2009 04:59 Go to previous messageGo to next message
webmaster_at_naszojciec is currently offline  webmaster_at_naszojciec   Poland
Messages: 4
Registered: April 2009
Karma: 0
Junior Member
add to buddy list
ignore all messages by this user
Here it is, nntp.inc only:

<?php
--- nntp.inc.1.79       2009-04-20 10:49:14.000000000 +0200
+++ nntp.inc    2009-04-20 10:56:27.000000000 +0200
@@ -162,+162,@@
        function 
format_headers()
        {
                
/* convert to unix line endings and handle multi-line headers */
-               $this->headers str_replace(array("\r\n""\n "), array("\n",""), $this->headers);
+               
$this->headers str_replace("\r\n","\n"$this->headers);
+               
$this->headers preg_replace("/\n(\t| )+/"' '$this->headers);

                
$hdr explode("\n"trim($this->headers));
                
$this->headers = array();
@@ -
402,12 +403,15 @@
                                }
                        }

-                       if (isset(
$this->headers['content-type']) && preg_match('!charset="([^"]+)"!'$this->headers['content-type'], $m)) {
+                       if (isset(
$this->headers['content-type']) && preg_match('!charset="?([^"]+?)"?(;|\s|$)!'$this->headers['content-type'], $m)) {
                                
$charset $m[1];
                        } else {
                                
$charset $GLOBALS['CHARSET'];
                        }
-                       if (isset(
$this->headers['content-transfer-encoding']) && $this->headers['content-transfer-encoding'] == 'quoted-printable') {
+                       if (isset(
$this->headers['content-transfer-encoding']) &&
+                               (
$this->headers['content-transfer-encoding'] == 'quoted-printable'
+                                       || $this->headers['content-transfer-encoding'] == 'base64')
+                               ) {
                                
$enc $this->headers['content-transfer-encoding'];
                        } else {
                                
$enc '';
@@ -
546,+550,@@

                
fputs($this->fs"From: $from\r\n");
                
fputs($this->fs"Newsgroups: $this->newsgroup\r\n");
-               
fputs($this->fs"Subject: $subject\r\n");
+               
fputs($this->fsiconv_mime_encode("Subject"$subject,
+                               array(
'scheme'=>'Q'"input-charset"=>$GLOBALS['CHARSET'],"output-charset"=>$GLOBALS['CHARSET']))
+                               .
"\r\n");

                if (
$GLOBALS['FORUM_TITLE']) {
                        
fputs($this->fs"Organization: ".$GLOBALS['FORUM_TITLE']."\r\n");
?>

[Updated on: Mon, 20 April 2009 04:59]

Report message to a moderator

Re: nntp charset problems [message #159087 is a reply to message #158969] Sun, 03 May 2009 14:45 Go to previous messageGo to next message
naudefj is currently offline  naudefj   South Africa
Messages: 3624
Registered: December 2004
Karma: 17
Senior Member
Administrator
Core Developer
remove from buddy list
ignore all messages by this user
A variation of your patch was committed. For details, see http://cvs.prohost.org/c/index.cgi/FUDforum/chngview?cn=11920
Re: nntp charset problems [message #159095 is a reply to message #159087] Mon, 04 May 2009 04:42 Go to previous messageGo to next message
webmaster_at_naszojciec is currently offline  webmaster_at_naszojciec   Poland
Messages: 4
Registered: April 2009
Karma: 0
Junior Member
add to buddy list
ignore all messages by this user
Unfortunetly, in commited version is my first diff which is buggy.
There is posbility of double decode for subject.
Subject is decoded by decode_header_value.
Here is small patch for version 1.80 of nntp.inc.


<?php
--- nntp.inc.1.80       2009-05-04 10:36:19.000000000 +0200
+++ nntp.inc    2009-05-04 10:40:38.000000000 +0200
@@ -424,+424,@@
                        }

                        
$msg_post->subject apply_custom_replace($this->subject);
-                       if ( !
preg_match('!(.*?)(=\?([^?]+)\?(Q|B)\?([^?]*)\?=)([^=]*)!i'$msg_post->subject)) {
-                               
$msg_post->subject decode_string($msg_post->subject''$charset);
-                       }
                        
$msg_post->body decode_string($this->body$enc$charset);
                        if ( !
strlen($msg_post->subject) ) {
                                
$msg_post->subject "(no subject)";
?>

[Updated on: Mon, 04 May 2009 04:42]

Report message to a moderator

Re: nntp charset problems [message #159148 is a reply to message #159095] Sat, 09 May 2009 01:41 Go to previous messageGo to next message
naudefj is currently offline  naudefj   South Africa
Messages: 3624
Registered: December 2004
Karma: 17
Senior Member
Administrator
Core Developer
remove from buddy list
ignore all messages by this user
Thanks for the update. Committed with http://cvs.prohost.org/c/index.cgi/FUDforum/chngview?cn=11922
Re: nntp charset problems [message #160134 is a reply to message #158908] Tue, 28 July 2009 13:24 Go to previous messageGo to next message
ANiko is currently offline  ANiko   United States
Messages: 30
Registered: May 2009
Karma: 0
Member
add to buddy list
ignore all messages by this user
The results of this patch look odd now that they're incorporated into FUDforum 2.8.1 for English (utf-8). When posted to a newsgroup, the subject line starts with a colon, followed by a space, followed by the subject line.

I deleted the added iconv_mime_encode code in the nntp.inc file, putting the code back as it was for earlier versions of FUDforum. The subject line now looks like it should, but is there a correct way to fix this for the English speakers?

Thanks.
Re: nntp charset problems [message #160136 is a reply to message #160134] Tue, 28 July 2009 15:32 Go to previous messageGo to next message
naudefj is currently offline  naudefj   South Africa
Messages: 3624
Registered: December 2004
Karma: 17
Senior Member
Administrator
Core Developer
remove from buddy list
ignore all messages by this user
It's already fixed in CVS, see http://cvs.prohost.org/c/index.cgi/FUDforum/chngview?cn=11946 (last hunk).
Re: nntp charset problems [message #160138 is a reply to message #160136] Tue, 28 July 2009 15:52 Go to previous message
ANiko is currently offline  ANiko   United States
Messages: 30
Registered: May 2009
Karma: 0
Member
add to buddy list
ignore all messages by this user
Oh. That was simple. Thanks!
Quick Reply
Formatting Tools:   
  Switch to threaded view of this topic Create a new topic
Previous Topic: HELP! Cannot login with exact username and password
Next Topic: Fud API Error
Goto Forum:
  

-=] Back to Top [=-
[ Syndicate this forum (XML) ] [ RSS ]

Current Time: Thu Oct 19 10:47:02 EDT 2017

Total time taken to generate the page: 0.00802 seconds