| USENET sync breaks and goes into endless cycle when it encounters attachment [message #158149] |
Thu, 05 February 2009 07:13  |
kerryg
Messages: 157
Registered: September 2008
Karma: 0
|
Senior Member |
|
|
I have three newsgroups I'm trying to sync to FUDForum. The first has no attachments and sync'd well. The second had a "null subject" in a post that was preventing sync, thanks to Frank this is now resolved (see separate thread).
The third group contains over 45,000 posts; when syncing to FUDForum, things go well right until it hits a specific post and then goes into an endless loop, cycling back the previous posts. The post it's hitting is the first post in that group with an attachment.
My (uneducated) suspicion is that the sync script sees the attachment as malformed, and is trying to read the attachment as part of the message body.
I thought this was fixed here > http://cvs.prohost.org/c/index.cgi/FUDforum/chngview?cn=11557 - but if this is indeed the same issue it seems to not be caught by that fix.
Thanks!
PS: new Admin control panel looks great!
- K
[Updated on: Thu, 05 February 2009 07:15] Report message to a moderator |
|
|
|
|
|
| Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158166 is a reply to message #158155] |
Fri, 06 February 2009 01:02  |
kerryg
Messages: 157
Registered: September 2008
Karma: 0
|
Senior Member |
|
|
AHA! I posted this to you privately as well, Frank, but just to have all the info in one place -
I just managed to *break* sync in one of the previously functioning forums by attaching a 700k+ PDF to a post, and posting it to the newsgroup it's synced with. Later I posted a followup message in the same newsgroup.
Sure enough, in the forum I've synced the newsgroup to, the forum is filling up with copies of that post with the PDF attachment again and again - and not moving past it to the text post I made after that. Same symptoms as the other groups. When I look at the message on the forum, it's not an attachment but a big lump of code inserted in the message body as text.
Seems that FUDForum is refusing to parse the attachments as attachments and is treating them as message body. The bigger problem of course is that somewhere in this process it's cycling back endlessly trying again and again - rather than simply failing and moving on to sync the next post.
Suggestions?
[Updated on: Fri, 06 February 2009 01:16] Report message to a moderator |
|
|
|
|
|
| Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158194 is a reply to message #158192] |
Sun, 08 February 2009 19:11 |
kerryg
Messages: 157
Registered: September 2008
Karma: 0
|
Senior Member |
|
|
Ahhh! That would explain it; there might be quite a few MIME encoded attachments in the forums.
I thought MIME encoding was the most common encoding for attachments - wouldn't the lack of support of MIME encoding break a *lot* of peoples' forums?
I wish I could do the coding on this myself; unfortunately I can't - it's's not that I'm unwilling to invest the energy, I simply don't know how to code and would have to teach myself php from scratch from first principles in order to do this. Unfortunately from the sounds of this, it sounds like the forum will remain non-syncable, thus completely dead in the water, until FUDForum can deal with MIME attachments.
Any suggestions?
[Updated on: Sun, 08 February 2009 19:14] Report message to a moderator |
|
|
|
| Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158195 is a reply to message #158194] |
Sun, 08 February 2009 20:39 |
Marticus
Messages: 272
Registered: June 2002
Karma: 1
|
Senior Member |
|
|
uuencoding has been around much longer and is still the dominant encoding type for usenet. Interestingly, something called yenc has been slowing making its way into the scene, but I think some server are flagging it as spam and it is still young. From what I recall mime was not well designed for transporting binaries especially multi-part. Some popular decoders/news readers have poorly implemented mime support.
Granted my information is based on my limited and slightly dated exposure to usenet, it may still be accurate and might explain why MIME support was not added when usenet support was added a few years ago.
[Updated on: Sun, 08 February 2009 20:50] Report message to a moderator |
|
|
|
| Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158196 is a reply to message #158195] |
Sun, 08 February 2009 20:46 |
kerryg
Messages: 157
Registered: September 2008
Karma: 0
|
Senior Member |
|
|
Thanks for the reply, Marticus.
This is likely going to crop up again, and it is going to *break forums* when it happens - it is a complete show-stopper over here, I might as well delete my forum since the entire point of trying to get it working for nearly six months now has been to get a USENET group onto the Web.
Until we have a proper fix, maybe FUD could include a "kill switch" to delete/ignore them rather than just breaking - or even some sort of optional "fail-safe" feature to truncate messages whose body exceeds a certain length?
[Updated on: Sun, 08 February 2009 20:50] Report message to a moderator |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158251 is a reply to message #158243] |
Thu, 12 February 2009 17:58 |
kerryg
Messages: 157
Registered: September 2008
Karma: 0
|
Senior Member |
|
|
Upgraded to 2.8.0 RC2 - same behaviour. Sync gets to the same post and then doubles back and re-syncs the 20 last posts (I set max import to 20, this is a small test forum) before in an endless loop.
Cron sends me these errors every sync attempt (one min)-
Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 262144 bytes) in /home/kerrygal/FUDforum/include/theme/default/isearch.inc on line 64
and this after five min:
Warning: mysql_fetch_row(): supplied argument is not a valid MySQL result resource in /home/kerrygal/FUDforum/include/theme/default/db.inc on line 27
(/home/kerrygal/FUDforum/include/theme/default/db.inc:72<br />
/home/kerrygal/FUDforum/include/nntp.inc:387<br />
/home/kerrygal/FUDforum/scripts/nntp.php:91<br />
) 2006: MySQL server has gone away<br />
Query: UPDATE fud26_msg SET mlist_msg_id='48f4ca40@linux' WHERE id=2378 AND mlist_msg_id IS NULL<br />
Server Version: <br />
(just FYI, around that time there might have been user activity, I was deleting one of the messages to check for results).
I've not modified the FUDFOrum theme, I'll just leave it "default" for now. I've disabled sync once again until this is resolved.
I'm going to contact Hostmonster and find out if there's anything non-standard on their end.
[Updated on: Thu, 12 February 2009 18:19] Report message to a moderator |
|
|
|
|
|
| Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158254 is a reply to message #158253] |
Thu, 12 February 2009 23:12 |
kerryg
Messages: 157
Registered: September 2008
Karma: 0
|
Senior Member |
|
|
OK, thanks, done. I deleted all posts in the forum; then set the number of posts to sync at 1 in that script. I "cleared tracker"; checked forum consistency, rebuild search index and compacted posts. Then I set the cron to run again.
The last post to sync successfully was this post, which had two attachments, a JPG and a PDF (that is - although the attachments didn't get parsed and were included in the body text, the message was otherwise successfully synced):
From: KerryGalloway <kerrygalloway(at)telus(dot)net>
Newsgroups: IDEA.emuensoniqparis-theproject
Date: Thu, 5 Feb 2009 14:14:34 -0800
Message-ID: <498b69ed@linux>
References: <C5B0A115(dot)19418%kg(at)kerrygalloway(dot)com> <C5B0A2D0(dot)1941B%kg(at)kerrygalloway(dot)com> <498b686b@linux> <498b691b@linux> <498b6949@linux>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="--------------108178411374671789"
Subject: Re: Attachment test BIG, don't open
User-Agent: Unison/1.7.9
NNTP-Posting-Host: d207-6-240-90.bchsia.telus.net
X-Trace: linux 1233873405 207.6.240.90 (6 Feb 2009 09:36:45 +1000)
Lines: 10669
Path: linux!not-for-mail
Xref: linux IDEA.emuensoniqparis-theproject:589
http://kerrygalloway.com/ParisForums/index.php?t=msg&th=729&start=0 &
The next post, the one that ISN'T successfully syncing, is a repost of the same PDF, but I had instead used the "attach file" feature of that newsreader app (which apparently yEnc encoded it and split it into two parts):
From: KerryGalloway <kerrygalloway(at)telus(dot)net>
Newsgroups: IDEA.emuensoniqparis-theproject
Date: Thu, 5 Feb 2009 14:16:11 -0800
Message-ID: <498b6a4d@linux>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit
Subject: A BIG FILE TEST - "8in Manual.pdf" yEnc (1/2)
User-Agent: Unison/1.7.9
NNTP-Posting-Host: d207-6-240-90.bchsia.telus.net
X-Trace: linux 1233873511 207.6.240.90 (6 Feb 2009 09:38:31 +1000)
Lines: 2781
Path: linux!not-for-mail
Xref: linux IDEA.emuensoniqparis-theproject:590
(fwiw the second part is -
From: KerryGalloway <kerrygalloway(at)telus(dot)net>
Newsgroups: IDEA.emuensoniqparis-theproject
Date: Thu, 5 Feb 2009 14:16:36 -0800
Message-ID: <498b6a67@linux>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit
Subject: A BIG FILE TEST - "8in Manual.pdf" yEnc (2/2)
User-Agent: Unison/1.7.9
NNTP-Posting-Host: d207-6-240-90.bchsia.telus.net
X-Trace: linux 1233873514 207.6.240.90 (6 Feb 2009 09:38:34 +1000)
Lines: 1387
Path: linux!not-for-mail
Xref: linux IDEA.emuensoniqparis-theproject:591
A further anomaly, if this is helpful, is that all posts are now duplicated exactly once in this forum, and the last TWO posts seem to be multiplying endlessly. I just deleted everything, zeroed everything, applied the patch in the other thread and tried another sync. Some posts are coming in duplicated one time (with max posts = 1) [UPDATE: appears to be all posts duplicated].
http://kerrygalloway.com/ParisForums/index.php?t=thread&frm_id=2&
[Updated on: Thu, 12 February 2009 23:51] Report message to a moderator |
|
|
|
|
|
| Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158267 is a reply to message #158265] |
Sat, 14 February 2009 02:40 |
kerryg
Messages: 157
Registered: September 2008
Karma: 0
|
Senior Member |
|
|
Thanks Frank -
OK, applied patch; deleted test forum and all messages; checked forum consistency; rebuilt search index; rebuilt all themes.
Created new forum; set up new newsgroup script. Set up and ran new cron.
Forum is now syncing the very first (earliest) post in the newsgroup only, in an endless loop - http://kerrygalloway.com/ParisForums/index.php?t=thread&frm_id=3&
I cleared everything and reapplied the patch, seems to have been applied correctly. Since my install is fresh and unchanged except for the patches, is there anything in the way of persistent data elsewhere on my server you think might be helpful for me to try clearing or eliminating - say deleting and reinstalling php.ini or deleting the sql database for FUDForum and creating a new one...?
[Updated on: Sat, 14 February 2009 03:09] Report message to a moderator |
|
|
|
|
|
| Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158278 is a reply to message #158277] |
Sat, 14 February 2009 18:02 |
kerryg
Messages: 157
Registered: September 2008
Karma: 0
|
Senior Member |
|
|
Quick progress report - applied patch and I've got the sync in progress again; you've beaten the "duplicate" thing that cropped up and we're half way through the sync now.
Within fifteen or twenty minutes it should be up to the "attachment issue" post; I'll add observations to this post when it gets there.
Update - the "theproject" group is now syncing correctly! The post with the encoded attachment as body text (as you mentioned) finally came across properly,sync carried on normally afterwards without cycling, and I just made a test post which synced normally! Woohoo!
Next step - I'm now attempting to sync the next-largest group (about 1700 posts if I recall) to see how that goes. After that I'll try the biggest with 47,000 posts. More as it develops.
I guess this particular USENET server, with its peculiar ways, has been a real "torture test" for FUDForum - if you get this one synced solidly, it's going to be hard to break 
UPDATE: Unbelievable - still keeping my fingers crossed, but the FORSALE group is now coming across. I can't tell you what a relief this is to see after nearly a year of trying.
Thank you Frank!
[Updated on: Sat, 14 February 2009 18:53] Report message to a moderator |
|
|
|
| Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158279 is a reply to message #158278] |
Sat, 14 February 2009 19:29 |
kerryg
Messages: 157
Registered: September 2008
Karma: 0
|
Senior Member |
|
|
OK, original reported situation was that out of three original newsgroups, one sync'd fine, two did not. I managed to break the third with a test attachment, so that made three non-sync'ers. The issues were 1) posts with null subjects and 2) attachments that choked FUDForum's sync.
Current status: newly broken group is now fully sync'd and functioning normally. Second group is now fully sync'd and functioning normally, and the third is part way through what looks like a normal sync.
So at this time this issue looks to be very close to RESOLVED (I'll re-title the thread as such when the main group is synced)! Killer work, Frank!
[Updated on: Sat, 14 February 2009 21:21] Report message to a moderator |
|
|
|
| Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158281 is a reply to message #158279] |
Sat, 14 February 2009 20:42 |
kerryg
Messages: 157
Registered: September 2008
Karma: 0
|
Senior Member |
|
|
OK - nearly there, I think. A few small issues: I set the "max messages" to = 100 so we could chew through to any errors fast; the sync on the main group went fine for nearly 5000 posts but choked in November '05 (I assume you're still getting cc'd on these errors, so you've got the post number?). Disabled the cron, deleted the dupes (it did do the "duplication of posts" thing on cycling back), set max messages to "1" and restarting sync from 20 - 30 posts before it, we'll see what happens when it hits it again (update: OK, now it just stops at that post, but at least there's no "cycling back and duplicating" now, it just refuses to proceed).
Rebuilding Search Index is now throwing this error:
Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 71 bytes) in /home/kerrygal/FUDforum/include/theme/default/isearch.inc on line 64
The other two groups seem to be good now - and this is *much* further into syncing the third (main) group than I've ever got, so you're definitely on the right track.
[Updated on: Sat, 14 February 2009 21:20] Report message to a moderator |
|
|
|
|
|
| Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158294 is a reply to message #158284] |
Sun, 15 February 2009 08:19 |
kerryg
Messages: 157
Registered: September 2008
Karma: 0
|
Senior Member |
|
|
Thanks Frank - OK, memory doubled to 128M in nntp.php; theme rebuilt; consistency checked; search index not rebuilt (see note at bottom of post); messages compacted. sync is still hanging up on the same post - cron sends me this error message on sync, seems to be specific to one particular message/user -
Importing IDEA.EMUEnsoniqPARIS message 170926
(/home/kerrygal/FUDforum/include/theme/default/db.inc:72<br />
/home/kerrygal/FUDforum/include/theme/default/db.inc:176<br />
/home/kerrygal/FUDforum/include/theme/default/users_reg.inc:164<br />
/home/kerrygal/FUDforum/include/scripts_common.inc:69<br />
/home/kerrygal/FUDforum/include/scripts_common.inc:27<br />
/home/kerrygal/FUDforum/include/nntp.inc:416<br />
/home/kerrygal/FUDforum/scripts/nntp.php:92<br />
) 1062: Duplicate entry '&quot;Mike Audet&amp;quot' for key 4<br />
Query: INSERT INTO
fud26_users (
login,
alias,
passwd,
name,
email,
icq,
aim,
yahoo,
msnm,
jabber,
affero,
google,
skype,
posts_ppg,
time_zone,
bday,
last_visit,
conf_key,
user_image,
join_date,
location,
theme,
occupation,
interests,
referer_id,
last_read,
sig,
home_page,
bio,
users_opt,
reg_ip,
topics_per_page,
flag_cc,
flag_country
) VALUES (
'\"Mike Audet&quot[1]',
'&amp;quot;Mike Audet&amp;amp;quot',
'1262ae11c8bfff19836228404efdc058',
'',
'\"Mike Audet\" jglgdf;oaaifhvsoz.com',
NULL,
NULL,
NULL,
NULL,
NULL,
NULL,
NULL,
NULL,
40,
'America/Vancouver',
0,
1234684862,
'',
NULL,
1131716975,
NULL,
1,
NULL,
NULL,
0,
1234684862,
NULL,
NULL,
NULL,
4357012,
2130706433,
40,
NULL,
NULL
)
<br />
Server Version: 5.0.67-community-log<br />
Could it be anything to do with the bogus email address he inserted (jglgdf;oaaifhvsoz.com) being a malformed email address? Or is it having difficulty adding him as a user because he's already been added as a user from another post in another subforum with another email address?
I'll separate out this other possibly unrelated issue for clarity - on "Rebuild Search Index", FUDForum's still giving me an out of memory error:
Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 71 bytes) in /home/kerrygal/FUDforum/include/theme/default/isearch.inc on line 64
[Updated on: Sun, 15 February 2009 19:32] Report message to a moderator |
|
|
|
|
|
| Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158300 is a reply to message #158299] |
Sun, 15 February 2009 20:28 |
kerryg
Messages: 157
Registered: September 2008
Karma: 0
|
Senior Member |
|
|
OK, applied, rebuilt theme - success, it's moved past that post!
[Total side issue, but - a user that's used a spoofed or gibberish email return address as an anti-spam measure will get added as a member as many times as he spoofed his address. I can certainly live with that for the moment, but is there a way to retroactively reassign such messages to valid user accounts? Actually, since in some old-style USENET communities this isn't an uncommon dodge to avoid spam - is there any way to merge *accounts* retroactively?]
[Updated on: Sun, 15 February 2009 20:32] Report message to a moderator |
|
|
|
|
|
|
|
| Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158303 is a reply to message #158302] |
Mon, 16 February 2009 03:18 |
kerryg
Messages: 157
Registered: September 2008
Karma: 0
|
Senior Member |
|
|
Just an update - nearly 25,000 USENET posts brought across now out of over 45,000 and still working. Once it makes it all the way through, I'm considering deleting the forum and reloading the whole thing from scratch.
I've got it set to import 100 messages at a time. It still chokes on the occasional post (it's happened a few times) and cycles back and duplicates the previous posts. At that point I stop sync, go back, manually clear out any duplicates, set maximum import to 1 post and restart sync. Eventually it chews past the problem post and at that point I boot it back up to 100 (since even at 100 posts per sync it requires constant watching for eight hours to sync the forums, do you think there's a memory setting or a timeout that I can increase that might help it pull this sync off reliably at 100 posts all the way through?)
Here's one of the posts it choked on:
Importing IDEA.EMUEnsoniqPARIS message 189955
Warning: mysql_fetch_row(): supplied argument is not a valid MySQL result resource in /home/kerrygal/FUDforum/include/theme/default/db.inc on line 27
(/home/kerrygal/FUDforum/include/theme/default/db.inc:72<br />
/home/kerrygal/FUDforum/include/theme/default/db.inc:132<br />
/home/kerrygal/FUDforum/include/scripts_common.inc:15<br />
/home/kerrygal/FUDforum/include/nntp.inc:416<br />
/home/kerrygal/FUDforum/scripts/nntp.php:92<br />
) 2006: MySQL server has gone away<br />
Query: SELECT id FROM fud26_users WHERE email='musurgio(at)otenet(dot)gr'<br />
Server Version: <br />
However - I'd have to say that even having to nursemaid the sync along like this represents an *enormous* step forward for this situation. Way to go, Frank!
[Updated on: Mon, 16 February 2009 03:19] Report message to a moderator |
|
|
|
| Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158304 is a reply to message #158303] |
Mon, 16 February 2009 04:24 |
kerryg
Messages: 157
Registered: September 2008
Karma: 0
|
Senior Member |
|
|
OK, LOTS of progress to date: got 28,000 posts in so far, but got up to this one (on 16th April '07) and stalled and cycled back, and can't get past it at maximum posts=1 either. It has an 837k .rar attachment.
Importing IDEA.EMUEnsoniqPARIS message 193976
Warning: mysql_fetch_row(): supplied argument is not a valid MySQL result resource in /home/kerrygal/FUDforum/include/theme/default/db.inc on line 27
(/home/kerrygal/FUDforum/include/theme/default/db.inc:72<br />
/home/kerrygal/FUDforum/include/theme/default/db.inc:132<br />
/home/kerrygal/FUDforum/include/scripts_common.inc:15<br />
/home/kerrygal/FUDforum/include/nntp.inc:416<br />
/home/kerrygal/FUDforum/scripts/nntp.php:92<br />
) 2006: MySQL server has gone away<br />
Query: SELECT id FROM fud26_users WHERE email='musurgioNOSPAM(at)otenet(dot)gr'<br />
Server Version: <br />
http://kerrygalloway.com/ParisForums/index.php?t=thread&frm_id=4&
[Updated on: Mon, 16 February 2009 05:14] Report message to a moderator |
|
|
|
|
|
|
|
| Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158307 is a reply to message #158306] |
Mon, 16 February 2009 21:22 |
kerryg
Messages: 157
Registered: September 2008
Karma: 0
|
Senior Member |
|
|
So close, but something seems to have bogged down last night at the very end. It's almost as if it thinks the sync is finished, but it's only made it to Sept 2007 - (UPDATE: I was wrong, it's throwing errors, I missed them when I reconfigured cron - it's been stalling on one post for 24 hours)
http://kerrygalloway.com/ParisForums/index.php?t=thread&frm_id=4&
UPDATED to add info - have rebuilt everything I could reach, no luck yet. Will keep trying to think of something.
PS: apologies if my cron error forwarding was spamming you!
Further update: this is the current error, slightly different than previous -
Importing IDEA.EMUEnsoniqPARIS message 201175<br />
Warning: mysql_fetch_row(): supplied argument is not a valid MySQL result resource in /home/kerrygal/FUDforum/include/theme/default/db.inc on line 27
(/home/kerrygal/FUDforum/include/theme/default/db.inc:72<br />
/home/kerrygal/FUDforum/include/theme/default/db.inc:168<br />
/home/kerrygal/FUDforum/include/scripts_common.inc:82<br />
/home/kerrygal/FUDforum/include/nntp.inc:453<br />
/home/kerrygal/FUDforum/scripts/nntp.php:92<br />
) 2006: MySQL server has gone away<br />
Query: SELECT m.id, m.thread_id FROM fud26_msg m INNER JOIN fud26_thread t ON m.thread_id=t.id WHERE m.mlist_msg_id='46f94d5d$1@linux' AND t.forum_id=4 ORDER BY m.post_stamp DESC LIMIT 1<br />
Server Version: <br />
[Updated on: Tue, 17 February 2009 06:21] Report message to a moderator |
|
|
|
|
|
|
|
|
|
Calendar
Search
Help
Members
Register
Login
Home
]