FUDforum
Fast Uncompromising Discussions. FUDforum will get your users talking.

Home » FUDforum » FUDforum Suggestions » Newsgroup synchronizing
Show: Today's Messages :: Unread Messages :: Show Polls :: Message Navigator
| Subscribe to topic | Bookmark topic 
Switch to threaded view of this topic Create a new topic Submit Reply
Newsgroup synchronizing [message #159118] Wed, 06 May 2009 13:47 Go to next message
wahono is currently offline  wahono   Indonesia
Messages: 10
Registered: April 2009
Karma: 0
Junior Member
add to buddy list
ignore all messages by this user
- First cron start from 100 to 200, but I got error on 150rd message.
- I don't know if my cron process got error
- system run next cron at other time, and this second cron process start synchronizing from 100rd message again. So the message is duplicated.
- If I make cron job every 15 minutes and I don't know this error until 3 hours, than messages 100rd to 150rd is duplicate 12 times.

My suggestion, if cron of nntp.php got error, it is safe to ignoring this error message and continue to next step (next message).

I think this is serious problem for newsgroup integration.

Thanks
Wahono
Re: Newsgroup synchronizing [message #159119 is a reply to message #159118] Wed, 06 May 2009 13:50 Go to previous messageGo to next message
wahono is currently offline  wahono   Indonesia
Messages: 10
Registered: April 2009
Karma: 0
Junior Member
add to buddy list
ignore all messages by this user
And I got error while synchronizing only in message with attachments files. Some of this attached file is in big size file but some in small files. I don't know what is the reasons.
Re: Newsgroup synchronizing [message #159123 is a reply to message #159119] Wed, 06 May 2009 14:24 Go to previous messageGo to next message
kerryg is currently offline  kerryg   Canada
Messages: 157
Registered: September 2008
Karma: 0
Senior Member
add to buddy list
ignore all messages by this user
How are the attachments encoded? MIME? yEnc? FUD gave me similar problems with that, it can't deal with those kind of attachments yet.

Here's a couple of things you might want to try. First, find your php.ini and raise its memory limits. Try syncing again and see if that solves it.

The "manual" approach that I took involved bypassing a number of problem posts. Shut off your cron, delete all the sync'd posts out of that subforum, compact messages and rebuild your search index. Then sync again, shutting down your cron before you get to the problem post (you may need to adjust the "number of posts per sync")

When you're sure you've identified the problem post, go into your server and find your forum data folder. Find /errors/.nntp/ and inside it will be two files named for the newsgroup you're syncing. Ignore the one that says "lock", open the other one for editing. It contains the post number of the last post sync'd, so increment that number by one and it'll bypass the problem post.

[Updated on: Wed, 06 May 2009 14:25]

Report message to a moderator

Re: Newsgroup synchronizing [message #159125 is a reply to message #159123] Wed, 06 May 2009 14:33 Go to previous messageGo to next message
naudefj is currently offline  naudefj   South Africa
Messages: 3632
Registered: December 2004
Karma: 17
Senior Member
Administrator
Core Developer
add to buddy list
ignore all messages by this user
Can you please post the error?

If you want us to try and reproduce the error, please post your NNTP server and newsgroup name.

Best of luck!

Frank
Re: Newsgroup synchronizing [message #159127 is a reply to message #159125] Thu, 07 May 2009 07:05 Go to previous messageGo to next message
wahono is currently offline  wahono   Indonesia
Messages: 10
Registered: April 2009
Karma: 0
Junior Member
add to buddy list
ignore all messages by this user
naudefj wrote on Wed, 06 May 2009 14:33
Can you please post the error?

If you want us to try and reproduce the error, please post your NNTP server and newsgroup name.

Best of luck!

Frank



The error result is broken text, I can't upload here. And in FUDforum this message is :

message in fudforum
This is a multi-part message in MIME format.
--------------000408070804020005050104
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Martin Schreiber escribió:
> Julio Jiménez Borreguero wrote:
>
>> I tried the last svn trunk and found a weird issue with tdbstringgrid. I
>> see only the first row is shown (the others are in the screen but not
>> rendered) ie. I have a master detail relation and when I click when a
>> row is supposed to be the detail is shown.
>>
>> If I execute the query again (set tmsesqlquery active after closing the
>> transaction), all is fine. It only happens the first time the
>> tmsesqlquery is active.
>>
> Please send a simple testcase.

Here is the test

Compile and run the program, press the button, you'll see only one row
(left side) but if you click in the tdbstringgrid you'll see how the
ticket in the right side will be shown. (so the rows exists)
if you press the left button (it executes the query again) all is fine.

Thanks


Julio


--------------000408070804020005050104
Content-Type: application/x-bzip;
name="test.tar.bz2"
Content-Transfer-Encoding: base64
Content-Disposition: inline;
filename="test.tar.bz2"

QlpoOTFBWSZTWYeniPMFeNT///////////////////////////////////// ////////5FU4
+ngAWgGihIEXB6K4IAB7xu77B3sADDvds8AAAAHoAEPAAWj6n0cNeGNal26A A+ezaZ8Zyfez
qO31H2Y++zA+19wAHQe+ZMYOs3bbXaTS7Gj7tPYWKo92V6au3p2bLW9g7pg0 oHtjr2rK9b0r
....
....
long binary text==========>
....
....
20zUuNPdWWHv+n7K+f2fW6B9zVy0XJshiUXn9su7q7J2WevPP3fPwx5fLj39 Pa+n9v/Xv+B+
0n6G9B+2WFVasqv1TJ/Cln5EOLPy7oPnaOHoQ/9jDoHgZsYKp9Mtn2zCfmPv XodMPpsRWRFI
sGIxnWDXAes9sWHSIgKWRFSRFT/IBDPFbE//xdyRThQkLEFcA8A=
--------------000408070804020005050104--


File attachment doesn't show as attached file but inline message.

You can try with my newsgroup :
server : news.grid-sky.com
user and password =''
group : public.mseide-msegui.talk
message 15130-15134

My PHP conf is like this :

memory_limit 512M 512M
open_basedir no value no value
output_buffering no value no value
output_handler no value no value
post_max_size 80M 80M
precision 12 12
realpath_cache_size 16K 16K
realpath_cache_ttl 120 120
register_argc_argv On On
register_globals On On
register_long_arrays On On
report_memleaks On On
report_zend_debug On On
safe_mode Off Off
safe_mode_exec_dir no value no value
safe_mode_gid Off Off
safe_mode_include_dir no value no value
sendmail_from no value no value
sendmail_path no value no value
serialize_precision 100 100
short_open_tag On On
SMTP localhost localhost
smtp_port 25 25
sql.safe_mode Off Off
track_errors Off Off
unserialize_callback_func no value no value
upload_max_filesize 200M 200M

Thanks in advance
Wahono

[Updated on: Thu, 07 May 2009 07:16]

Report message to a moderator

Re: Newsgroup synchronizing [message #159128 is a reply to message #159123] Thu, 07 May 2009 07:43 Go to previous messageGo to next message
wahono is currently offline  wahono   Indonesia
Messages: 10
Registered: April 2009
Karma: 0
Junior Member
add to buddy list
ignore all messages by this user
kerryg wrote on Wed, 06 May 2009 14:24
How are the attachments encoded? MIME? yEnc? FUD gave me similar problems with that, it can't deal with those kind of attachments yet.


Yes, MIME.

kerryg wrote on Wed, 06 May 2009 14:24
Here's a couple of things you might want to try. First, find your php.ini and raise its memory limits. Try syncing again and see if that solves it.

The "manual" approach that I took involved bypassing a number of problem posts. Shut off your cron, delete all the sync'd posts out of that subforum, compact messages and rebuild your search index. Then sync again, shutting down your cron before you get to the problem post (you may need to adjust the "number of posts per sync")

When you're sure you've identified the problem post, go into your server and find your forum data folder. Find /errors/.nntp/ and inside it will be two files named for the newsgroup you're syncing. Ignore the one that says "lock", open the other one for editing. It contains the post number of the last post sync'd, so increment that number by one and it'll bypass the problem post.


My PHP memory limit is 512MB and max upload file is 200MB.
I can do fix syn with your way. But I think that rather than have duplicated messages, it is better to messages that can not be synchronized is ignoring this message.

Thanks
Wahono
Re: Newsgroup synchronizing [message #159132 is a reply to message #159128] Thu, 07 May 2009 14:34 Go to previous messageGo to next message
kerryg is currently offline  kerryg   Canada
Messages: 157
Registered: September 2008
Karma: 0
Senior Member
add to buddy list
ignore all messages by this user
That's consistent with my own symptoms, detailed in this thread - http://fudforum.org/forum/index.php?t=msg&th=118143&start=0& - so I'm pretty sure we've identified the problem anyway.

Basically, as it was the last time I checked, the NNTP script doesn't understand MIME (and yEnc) encoded attachments. It stalls when it hits some; it can also break and go into a loop, endlessly duplicating the last import of messages, when it hits some particular attachments; and if it imports them at all, it does so as a block of text inline with the message.

What version of FUDForum are you running? Some of the looping/duplicating issues were looked at.

It would indeed be better if FUDforum could understand MIME-encoded attachments, but a full fix would involve creating code to permit FUDforum to import them. Frank is aware of the issue, but he has a lot of stuff on his priority list so this may have to wait until a user writes code to support it.

[Updated on: Thu, 07 May 2009 14:39]

Report message to a moderator

Re: Newsgroup synchronizing [message #159133 is a reply to message #159132] Thu, 07 May 2009 15:19 Go to previous messageGo to next message
wahono is currently offline  wahono   Indonesia
Messages: 10
Registered: April 2009
Karma: 0
Junior Member
add to buddy list
ignore all messages by this user
I use versi 2.8.1 RC1.

I do synchronize from my PC (localhost webserver).
I do that because it seems in my hosting provider has limited memory PHP. But I turns out the server to my PC, the error still appears.
Re: Newsgroup synchronizing [message #159134 is a reply to message #159118] Thu, 07 May 2009 15:23 Go to previous messageGo to next message
wahono is currently offline  wahono   Indonesia
Messages: 10
Registered: April 2009
Karma: 0
Junior Member
add to buddy list
ignore all messages by this user
Because of this problem (file attachment in the message stick), then table fud28_index and fud28_search become very large. Because each letter of the binary attachment file is considered to be 1 word.
Re: Newsgroup synchronizing [message #159135 is a reply to message #159132] Thu, 07 May 2009 15:38 Go to previous messageGo to next message
naudefj is currently offline  naudefj   South Africa
Messages: 3632
Registered: December 2004
Karma: 17
Senior Member
Administrator
Core Developer
add to buddy list
ignore all messages by this user
OK, I can simulate the problem:

Importing public.mseide-msegui.talk message 15133

PHP Fatal error:  Allowed memory size of 67108864 bytes exhausted (tried to allocate 40 bytes) in /lampstack/apache2/htdocs/forum/data/include/theme/default/wordwrap.inc on line 49


From the above it looks like the import is choking on a large post. You will need to increase your memory limit or skip this message.

Best regards.

Frank
Re: Newsgroup synchronizing [message #159136 is a reply to message #159134] Thu, 07 May 2009 16:02 Go to previous messageGo to next message
kerryg is currently offline  kerryg   Canada
Messages: 157
Registered: September 2008
Karma: 0
Senior Member
add to buddy list
ignore all messages by this user
wahono wrote on Thu, 07 May 2009 12:23
Because of this problem (file attachment in the message stick), then table fud28_index and fud28_search become very large. Because each letter of the binary attachment file is considered to be 1 word.


Ahah! Good to know. That might be at the root of some of my own problems.
Re: Newsgroup synchronizing [message #159137 is a reply to message #159135] Thu, 07 May 2009 16:08 Go to previous messageGo to next message
wahono is currently offline  wahono   Indonesia
Messages: 10
Registered: April 2009
Karma: 0
Junior Member
add to buddy list
ignore all messages by this user
naudefj wrote on Thu, 07 May 2009 15:38
OK, I can simulate the problem:

Importing public.mseide-msegui.talk message 15133

PHP Fatal error:  Allowed memory size of 67108864 bytes exhausted (tried to allocate 40 bytes) in /lampstack/apache2/htdocs/forum/data/include/theme/default/wordwrap.inc on line 49


From the above it looks like the import is choking on a large post. You will need to increase your memory limit or skip this message.

Best regards.

Frank



I don't use some hosting provider. I do synchronize in my pc with my own webserver. My PHP memory limit is 512MB and max upload file is 200MB. I don't get the message such as you got. But I got message as broken text. I think that is binay file attachment parsing.

My plan is if synchronizing finished in my local PC, I'll upload to my sites and continue get new messages with cron.

I'm confused with this problem.
Re: Newsgroup synchronizing [message #159138 is a reply to message #159137] Thu, 07 May 2009 16:22 Go to previous messageGo to next message
naudefj is currently offline  naudefj   South Africa
Messages: 3632
Registered: December 2004
Karma: 17
Senior Member
Administrator
Core Developer
add to buddy list
ignore all messages by this user
OK, so you already increased the memory limit and can import the message, but when you look at it you see binary characters. The reason is that the post has a very large MIME file attached to it and FUDforum doesn't recognize it as an attachment (only UUENCODED attachments are supported at this stage). As a result the attachment is included in the message body.
Re: Newsgroup synchronizing [message #159139 is a reply to message #159138] Thu, 07 May 2009 16:28 Go to previous messageGo to next message
wahono is currently offline  wahono   Indonesia
Messages: 10
Registered: April 2009
Karma: 0
Junior Member
add to buddy list
ignore all messages by this user
But this problem not on big file attachment only. In message 16042rd (thread "display trouble on ubuntu 9.04"), in this thread only PNG file with size 299KB. But when I import it, attached files is included to message body too.

Thanks
Re: Newsgroup synchronizing [message #159143 is a reply to message #159139] Fri, 08 May 2009 02:14 Go to previous messageGo to next message
naudefj is currently offline  naudefj   South Africa
Messages: 3632
Registered: December 2004
Karma: 17
Senior Member
Administrator
Core Developer
add to buddy list
ignore all messages by this user
Right, all MIME encoded files will be displayed in the message body. You are welcome to update the script to support MIME attachments and contribute it back to the project.
Re: Newsgroup synchronizing [message #159186 is a reply to message #159143] Mon, 11 May 2009 12:54 Go to previous messageGo to next message
wahono is currently offline  wahono   Indonesia
Messages: 10
Registered: April 2009
Karma: 0
Junior Member
add to buddy list
ignore all messages by this user
Which files should I check? I'm not web programmer, I'm FreePascal programmer, but I'll try it and I hope I can.

Thanks
Re: Newsgroup synchronizing [message #159188 is a reply to message #159186] Mon, 11 May 2009 13:13 Go to previous messageGo to next message
naudefj is currently offline  naudefj   South Africa
Messages: 3632
Registered: December 2004
Karma: 17
Senior Member
Administrator
Core Developer
add to buddy list
ignore all messages by this user
See parse_attachments() in include/nntp.inc

Regards.

Frank
Re: Newsgroup synchronizing [message #159207 is a reply to message #159188] Tue, 12 May 2009 13:39 Go to previous message
kerryg is currently offline  kerryg   Canada
Messages: 157
Registered: September 2008
Karma: 0
Senior Member
add to buddy list
ignore all messages by this user
I'm not a coder so I can't be of direct help, but I'll gladly help test.
Quick Reply
Formatting Tools:   
  Switch to threaded view of this topic Create a new topic
Previous Topic: Whats in a name?
Next Topic: Automatic Image Resizing
Goto Forum:
  

-=] Back to Top [=-
[ Syndicate this forum (XML) ] [ RSS ]

Current Time: Tue Dec 12 14:55:15 EST 2017

Total time taken to generate the page: 0.00924 seconds