FUDforum: Bug Reports » USENET sync breaks and goes into endless cycle when it encounters attachment

Home » FUDforum Development » Bug Reports » USENET sync breaks and goes into endless cycle when it encounters attachment

Show: Today's Messages :: Polls :: Message Navigator

USENET sync breaks and goes into endless cycle when it encounters attachment [message #158149]

Thu, 05 February 2009 07:13

kerryg

Messages: 157
Registered: September 2008

Karma: 0

Senior Member

I have three newsgroups I'm trying to sync to FUDForum. The first has no attachments and sync'd well. The second had a "null subject" in a post that was preventing sync, thanks to Frank this is now resolved (see separate thread).

The third group contains over 45,000 posts; when syncing to FUDForum, things go well right until it hits a specific post and then goes into an endless loop, cycling back the previous posts. The post it's hitting is the first post in that group with an attachment.

My (uneducated) suspicion is that the sync script sees the attachment as malformed, and is trying to read the attachment as part of the message body.

I thought this was fixed here > http://cvs.prohost.org/c/index.cgi/FUDforum/chngview?cn=11557 - but if this is indeed the same issue it seems to not be caught by that fix.

Thanks!

PS: new Admin control panel looks great!

- K

[Updated on: Thu, 05 February 2009 07:15]

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158155 is a reply to message #158149]

Thu, 05 February 2009 10:53

naudefj

Messages: 3771
Registered: December 2004

Karma: 28

Senior Member
Administrator
Core Developer

Yes, that fix is part of 2.8 and wasn't overwritten by subsequent changes. So, I guess it must be a new issue. Can you please post (or mail) me the group name and message number?

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158166 is a reply to message #158155]

Fri, 06 February 2009 01:02

kerryg

Messages: 157
Registered: September 2008

Karma: 0

Senior Member

AHA! I posted this to you privately as well, Frank, but just to have all the info in one place -

I just managed to *break* sync in one of the previously functioning forums by attaching a 700k+ PDF to a post, and posting it to the newsgroup it's synced with. Later I posted a followup message in the same newsgroup.

Sure enough, in the forum I've synced the newsgroup to, the forum is filling up with copies of that post with the PDF attachment again and again - and not moving past it to the text post I made after that. Same symptoms as the other groups. When I look at the message on the forum, it's not an attachment but a big lump of code inserted in the message body as text.

Seems that FUDForum is refusing to parse the attachments as attachments and is treating them as message body. The bigger problem of course is that somewhere in this process it's cycling back endlessly trying again and again - rather than simply failing and moving on to sync the next post.

Suggestions?

[Updated on: Fri, 06 February 2009 01:16]

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158192 is a reply to message #158166]

Sun, 08 February 2009 16:39

naudefj

Messages: 3771
Registered: December 2004

Karma: 28

Senior Member
Administrator
Core Developer

Some feedback: I was able to simulate the attachment problem, but not the looping problem.

The post you are loading has a MIME encoded attachment. However, it seems that FUDforum only supports UUENCODED attachments. It won't be too difficult to add, but someone will need to do the coding.

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158194 is a reply to message #158192]

Sun, 08 February 2009 19:11

kerryg

Messages: 157
Registered: September 2008

Karma: 0

Senior Member

Ahhh! That would explain it; there might be quite a few MIME encoded attachments in the forums.

I thought MIME encoding was the most common encoding for attachments - wouldn't the lack of support of MIME encoding break a *lot* of peoples' forums?

I wish I could do the coding on this myself; unfortunately I can't - it's's not that I'm unwilling to invest the energy, I simply don't know how to code and would have to teach myself php from scratch from first principles in order to do this. Unfortunately from the sounds of this, it sounds like the forum will remain non-syncable, thus completely dead in the water, until FUDForum can deal with MIME attachments.

Any suggestions?

[Updated on: Sun, 08 February 2009 19:14]

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158195 is a reply to message #158194]

Sun, 08 February 2009 20:39

Marticus

Messages: 272
Registered: June 2002

Karma: 1

Senior Member

uuencoding has been around much longer and is still the dominant encoding type for usenet. Interestingly, something called yenc has been slowing making its way into the scene, but I think some server are flagging it as spam and it is still young. From what I recall mime was not well designed for transporting binaries especially multi-part. Some popular decoders/news readers have poorly implemented mime support.

Granted my information is based on my limited and slightly dated exposure to usenet, it may still be accurate and might explain why MIME support was not added when usenet support was added a few years ago.

[Updated on: Sun, 08 February 2009 20:50]

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158196 is a reply to message #158195]

Sun, 08 February 2009 20:46

kerryg

Messages: 157
Registered: September 2008

Karma: 0

Senior Member

Thanks for the reply, Marticus.

This is likely going to crop up again, and it is going to *break forums* when it happens - it is a complete show-stopper over here, I might as well delete my forum since the entire point of trying to get it working for nearly six months now has been to get a USENET group onto the Web.

Until we have a proper fix, maybe FUD could include a "kill switch" to delete/ignore them rather than just breaking - or even some sort of optional "fail-safe" feature to truncate messages whose body exceeds a certain length?

[Updated on: Sun, 08 February 2009 20:50]

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158197 is a reply to message #158196]

Sun, 08 February 2009 20:51

Marticus

Messages: 272
Registered: June 2002

Karma: 1

Senior Member

Note I edited my post to clarify that my information may or may not apply to current usenet standards as it was based on my dated exposure and that it could explain why the support was not included when usenet support was added to fudforum.

I wouldn't be opposed to looking into it a bit to see how difficult it would be to implement.

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158198 is a reply to message #158197]

Sun, 08 February 2009 21:08

kerryg

Messages: 157
Registered: September 2008

Karma: 0

Senior Member

If you can look into this, that would be excellent, Marticus! Thanks much!

Edited to add: here are some related threads in case they shed any light.

This first one outlines and "fixes" the duplication issue - perhaps it just needs a tweak?
http://fudforum.org/forum/index.php?t=msg&th=7727&start=0&

http://fudforum.org/forum/index.php?t=msg&th=6924&start=0&

http://fudforum.org/forum/index.php?t=msg&goto=14222&#msg_14222

http://fudforum.org/forum/index.php?t=msg&goto=21364&&srch=NNTP +attachment#msg_21364

http://fudforum.org/forum/index.php?t=msg&goto=32348&&srch=NNTP +attachment#msg_32348

http://fudforum.org/forum/index.php?t=msg&goto=33220&&srch=NNTP +attachment#msg_33220

[Updated on: Mon, 09 February 2009 00:54]

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158199 is a reply to message #158195]

Mon, 09 February 2009 01:30

kerryg

Messages: 157
Registered: September 2008

Karma: 0

Senior Member

By the way, Marticus - the file that broke sync might have been a yEnc file. Hard to tell which one's breaking it right now.

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158203 is a reply to message #158199]

Mon, 09 February 2009 16:31

naudefj

Messages: 3771
Registered: December 2004

Karma: 28

Senior Member
Administrator
Core Developer

Well, the test case you've send me was definitely a MIME attachment that loaded as a block of strange characters in the message body. However, here is proof that it won't cycle (at least not with the 2.8 code):

$ vi ../errors/.nntp/news.parisnewsgroup.com-IDEA.EMUEnsoniqPARIS
$ cat ../errors/.nntp/news.parisnewsgroup.com-IDEA.EMUEnsoniqPARIS
166028
$ php ./nntp.php 1
$ cat ../errors/.nntp/news.parisnewsgroup.com-IDEA.EMUEnsoniqPARIS
166029

As you can see, we start with message 166028 and it is correctly incremented. So, maybe you have a different problem?

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158204 is a reply to message #158203]

Mon, 09 February 2009 22:03

kerryg

Messages: 157
Registered: September 2008

Karma: 0

Senior Member

VERY strange, but thanks Frank - now there's been an update it's probably a good time for me to back up the data and reinstall the forum from scratch once again and see what comes out on the other side.

Besides the attachment failure, are you able to achieve otherwise-successful sync with all three forums?

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158206 is a reply to message #158204]

Tue, 10 February 2009 05:05

naudefj

Messages: 3771
Registered: December 2004

Karma: 28

Senior Member
Administrator
Core Developer

No, unfortunately I don't have that much free time. If you have very specific and reproducible problems, I'm willing to look into them.

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158208 is a reply to message #158206]

Tue, 10 February 2009 06:14

kerryg

Messages: 157
Registered: September 2008

Karma: 0

Senior Member

OK Frank - if you reckon the "cycling" thing is anomalous I'll try a full reinstall, which I'd hope should zero everything, and see what happens. Whatever the outcome of this I greatly appreciate the efforts you've already put in solving the registration and "null subject" issues.

- k

[Updated on: Tue, 10 February 2009 06:14]

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158210 is a reply to message #158208]

Tue, 10 February 2009 08:13

kerryg

Messages: 157
Registered: September 2008

Karma: 0

Senior Member

Uninstalled FUDForum (I manually deleted the forum data root folder and the forum WWW folder rather than using the uninstaller - does that make a difference?), reinstalled from scratch, synced. No luck, still breaks in exactly the same place on the same post.

Any suggestions of things to try?

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158241 is a reply to message #158210]

Thu, 12 February 2009 02:44

Marticus

Messages: 272
Registered: June 2002

Karma: 1

Senior Member

Teach your usenet users to use uuencoded attachments until a solution presents itself.

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158243 is a reply to message #158241]

Thu, 12 February 2009 03:19

kerryg

Messages: 157
Registered: September 2008

Karma: 0

Senior Member

Marticus wrote on Wed, 11 February 2009 18:44

Teach your usenet users to use uuencoded attachments until a solution presents itself.

Unfortunately, I'm not the admin of the newsgroup (just the forum), so I don't have the authority to do that. Probably more to the point, it's an existing server and the ones with MIME attachments that are breaking the sync (if that is indeed the issue) were posted between 2005 and today.

[Updated on: Thu, 12 February 2009 03:20]

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158251 is a reply to message #158243]

Thu, 12 February 2009 17:58

kerryg

Messages: 157
Registered: September 2008

Karma: 0

Senior Member

Upgraded to 2.8.0 RC2 - same behaviour. Sync gets to the same post and then doubles back and re-syncs the 20 last posts (I set max import to 20, this is a small test forum) before in an endless loop.

Cron sends me these errors every sync attempt (one min)-

Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 262144 bytes) in /home/kerrygal/FUDforum/include/theme/default/isearch.inc on line 64

and this after five min:

Warning: mysql_fetch_row(): supplied argument is not a valid MySQL result resource in /home/kerrygal/FUDforum/include/theme/default/db.inc on line 27
(/home/kerrygal/FUDforum/include/theme/default/db.inc:72<br />
/home/kerrygal/FUDforum/include/nntp.inc:387<br />
/home/kerrygal/FUDforum/scripts/nntp.php:91<br />
) 2006: MySQL server has gone away<br />
Query: UPDATE fud26_msg SET mlist_msg_id='48f4ca40@linux' WHERE id=2378 AND mlist_msg_id IS NULL<br />
Server Version: <br />

(just FYI, around that time there might have been user activity, I was deleting one of the messages to check for results).

I've not modified the FUDFOrum theme, I'll just leave it "default" for now. I've disabled sync once again until this is resolved.

I'm going to contact Hostmonster and find out if there's anything non-standard on their end.

[Updated on: Thu, 12 February 2009 18:19]

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158253 is a reply to message #158251]

Thu, 12 February 2009 20:45

naudefj

Messages: 3771
Registered: December 2004

Karma: 28

Senior Member
Administrator
Core Developer

Quote:

I set max import to 20,...

Please set this to 1 while testing. You've reported a problem with message 166028, while the error may be anywhere from 166029 to 166048.

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158254 is a reply to message #158253]

Thu, 12 February 2009 23:12

kerryg

Messages: 157
Registered: September 2008

Karma: 0

Senior Member

OK, thanks, done. I deleted all posts in the forum; then set the number of posts to sync at 1 in that script. I "cleared tracker"; checked forum consistency, rebuild search index and compacted posts. Then I set the cron to run again.

The last post to sync successfully was this post, which had two attachments, a JPG and a PDF (that is - although the attachments didn't get parsed and were included in the body text, the message was otherwise successfully synced):

From: KerryGalloway <kerrygalloway(at)telus(dot)net>
Newsgroups: IDEA.emuensoniqparis-theproject
Date: Thu, 5 Feb 2009 14:14:34 -0800
Message-ID: <498b69ed@linux>
References: <C5B0A115(dot)19418%kg(at)kerrygalloway(dot)com> <C5B0A2D0(dot)1941B%kg(at)kerrygalloway(dot)com> <498b686b@linux> <498b691b@linux> <498b6949@linux>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="--------------108178411374671789"
Subject: Re: Attachment test BIG, don't open
User-Agent: Unison/1.7.9
NNTP-Posting-Host: d207-6-240-90.bchsia.telus.net
X-Trace: linux 1233873405 207.6.240.90 (6 Feb 2009 09:36:45 +1000)
Lines: 10669
Path: linux!not-for-mail
Xref: linux IDEA.emuensoniqparis-theproject:589

http://kerrygalloway.com/ParisForums/index.php?t=msg&th=729&start=0 &

The next post, the one that ISN'T successfully syncing, is a repost of the same PDF, but I had instead used the "attach file" feature of that newsreader app (which apparently yEnc encoded it and split it into two parts):

From: KerryGalloway <kerrygalloway(at)telus(dot)net>
Newsgroups: IDEA.emuensoniqparis-theproject
Date: Thu, 5 Feb 2009 14:16:11 -0800
Message-ID: <498b6a4d@linux>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit
Subject: A BIG FILE TEST - "8in Manual.pdf" yEnc (1/2)
User-Agent: Unison/1.7.9
NNTP-Posting-Host: d207-6-240-90.bchsia.telus.net
X-Trace: linux 1233873511 207.6.240.90 (6 Feb 2009 09:38:31 +1000)
Lines: 2781
Path: linux!not-for-mail
Xref: linux IDEA.emuensoniqparis-theproject:590

(fwiw the second part is -

From: KerryGalloway <kerrygalloway(at)telus(dot)net>
Newsgroups: IDEA.emuensoniqparis-theproject
Date: Thu, 5 Feb 2009 14:16:36 -0800
Message-ID: <498b6a67@linux>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit
Subject: A BIG FILE TEST - "8in Manual.pdf" yEnc (2/2)
User-Agent: Unison/1.7.9
NNTP-Posting-Host: d207-6-240-90.bchsia.telus.net
X-Trace: linux 1233873514 207.6.240.90 (6 Feb 2009 09:38:34 +1000)
Lines: 1387
Path: linux!not-for-mail
Xref: linux IDEA.emuensoniqparis-theproject:591

A further anomaly, if this is helpful, is that all posts are now duplicated exactly once in this forum, and the last TWO posts seem to be multiplying endlessly. I just deleted everything, zeroed everything, applied the patch in the other thread and tried another sync. Some posts are coming in duplicated one time (with max posts = 1) [UPDATE: appears to be all posts duplicated].

http://kerrygalloway.com/ParisForums/index.php?t=thread&frm_id=2&

[Updated on: Thu, 12 February 2009 23:51]

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158265 is a reply to message #158254]

Fri, 13 February 2009 21:21

naudefj

Messages: 3771
Registered: December 2004

Karma: 28

Senior Member
Administrator
Core Developer

Please apply this patch: http://cvs.prohost.org/c/index.cgi/FUDforum/chngview?cn=11862

It should fix the post duplication and memory limit problems. However, attachments will still load as binary characters in the post body.

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158267 is a reply to message #158265]

Sat, 14 February 2009 02:40

kerryg

Messages: 157
Registered: September 2008

Karma: 0

Senior Member

Thanks Frank -

OK, applied patch; deleted test forum and all messages; checked forum consistency; rebuilt search index; rebuilt all themes.

Created new forum; set up new newsgroup script. Set up and ran new cron.

Forum is now syncing the very first (earliest) post in the newsgroup only, in an endless loop - http://kerrygalloway.com/ParisForums/index.php?t=thread&frm_id=3&

I cleared everything and reapplied the patch, seems to have been applied correctly. Since my install is fresh and unchanged except for the patches, is there anything in the way of persistent data elsewhere on my server you think might be helpful for me to try clearing or eliminating - say deleting and reinstalling php.ini or deleting the sql database for FUDForum and creating a new one...?

[Updated on: Sat, 14 February 2009 03:09]

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158277 is a reply to message #158267]

Sat, 14 February 2009 15:09

naudefj

Messages: 3771
Registered: December 2004

Karma: 28

Senior Member
Administrator
Core Developer

Can you please apply this patch and test again: http://cvs.prohost.org/c/index.cgi/FUDforum/chngview?cn=11865

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158278 is a reply to message #158277]

Sat, 14 February 2009 18:02

kerryg

Messages: 157
Registered: September 2008

Karma: 0

Senior Member

Quick progress report - applied patch and I've got the sync in progress again; you've beaten the "duplicate" thing that cropped up and we're half way through the sync now.

Within fifteen or twenty minutes it should be up to the "attachment issue" post; I'll add observations to this post when it gets there.

Update - the "theproject" group is now syncing correctly! The post with the encoded attachment as body text (as you mentioned) finally came across properly,sync carried on normally afterwards without cycling, and I just made a test post which synced normally! Woohoo!

Next step - I'm now attempting to sync the next-largest group (about 1700 posts if I recall) to see how that goes. After that I'll try the biggest with 47,000 posts. More as it develops.

I guess this particular USENET server, with its peculiar ways, has been a real "torture test" for FUDForum - if you get this one synced solidly, it's going to be hard to break Very Happy

Very Happy

UPDATE: Unbelievable - still keeping my fingers crossed, but the FORSALE group is now coming across. I can't tell you what a relief this is to see after nearly a year of trying.

Thank you Frank!

[Updated on: Sat, 14 February 2009 18:53]

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158279 is a reply to message #158278]

Sat, 14 February 2009 19:29

kerryg

Messages: 157
Registered: September 2008

Karma: 0

Senior Member

OK, original reported situation was that out of three original newsgroups, one sync'd fine, two did not. I managed to break the third with a test attachment, so that made three non-sync'ers. The issues were 1) posts with null subjects and 2) attachments that choked FUDForum's sync.

Current status: newly broken group is now fully sync'd and functioning normally. Second group is now fully sync'd and functioning normally, and the third is part way through what looks like a normal sync.

So at this time this issue looks to be very close to RESOLVED (I'll re-title the thread as such when the main group is synced)! Killer work, Frank!

[Updated on: Sat, 14 February 2009 21:21]

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158281 is a reply to message #158279]

Sat, 14 February 2009 20:42

kerryg

Messages: 157
Registered: September 2008

Karma: 0

Senior Member

OK - nearly there, I think. A few small issues: I set the "max messages" to = 100 so we could chew through to any errors fast; the sync on the main group went fine for nearly 5000 posts but choked in November '05 (I assume you're still getting cc'd on these errors, so you've got the post number?). Disabled the cron, deleted the dupes (it did do the "duplication of posts" thing on cycling back), set max messages to "1" and restarting sync from 20 - 30 posts before it, we'll see what happens when it hits it again (update: OK, now it just stops at that post, but at least there's no "cycling back and duplicating" now, it just refuses to proceed).

Rebuilding Search Index is now throwing this error:

Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 71 bytes) in /home/kerrygal/FUDforum/include/theme/default/isearch.inc on line 64

The other two groups seem to be good now - and this is *much* further into syncing the third (main) group than I've ever got, so you're definitely on the right track.

[Updated on: Sat, 14 February 2009 21:20]

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158284 is a reply to message #158281]

Sun, 15 February 2009 04:32

naudefj

Messages: 3771
Registered: December 2004

Karma: 28

Senior Member
Administrator
Core Developer

Please increase the memory limit in data/scripts/nntp.php (line 14) and try again.

ini_set("memory_limit","128M");

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158294 is a reply to message #158284]

Sun, 15 February 2009 08:19

kerryg

Messages: 157
Registered: September 2008

Karma: 0

Senior Member

Thanks Frank - OK, memory doubled to 128M in nntp.php; theme rebuilt; consistency checked; search index not rebuilt (see note at bottom of post); messages compacted. sync is still hanging up on the same post - cron sends me this error message on sync, seems to be specific to one particular message/user -

Importing IDEA.EMUEnsoniqPARIS message 170926
(/home/kerrygal/FUDforum/include/theme/default/db.inc:72<br />
/home/kerrygal/FUDforum/include/theme/default/db.inc:176<br />
/home/kerrygal/FUDforum/include/theme/default/users_reg.inc:164<br />
/home/kerrygal/FUDforum/include/scripts_common.inc:69<br />
/home/kerrygal/FUDforum/include/scripts_common.inc:27<br />
/home/kerrygal/FUDforum/include/nntp.inc:416<br />
/home/kerrygal/FUDforum/scripts/nntp.php:92<br />
) 1062: Duplicate entry '&amp;quot;Mike Audet&amp;amp;quot' for key 4<br />
Query: INSERT INTO
			fud26_users (
				login,
				alias,
				passwd,
				name,
				email,
				icq,
				aim,
				yahoo,
				msnm,
				jabber,
				affero,
				google,
				skype,
				posts_ppg,
				time_zone,
				bday,
				last_visit,
				conf_key,
				user_image,
				join_date,
				location,
				theme,
				occupation,
				interests,
				referer_id,
				last_read,
				sig,
				home_page,
				bio,
				users_opt,
				reg_ip,
				topics_per_page,
				flag_cc,
				flag_country
			) VALUES (
				'\&quot;Mike Audet&amp;quot[1]',
				'&amp;amp;quot;Mike Audet&amp;amp;amp;quot',
				'1262ae11c8bfff19836228404efdc058',
				'',
				'\&quot;Mike Audet\&quot; jglgdf;oaaifhvsoz.com',
				NULL,
				NULL,
				NULL,
				NULL,
				NULL,
				NULL,
				NULL,
				NULL,
				40,
				'America/Vancouver',
				0,
				1234684862,
				'',
				NULL,
				1131716975,
				NULL,
				1,
				NULL,
				NULL,
				0,
				1234684862,
				NULL,
				NULL,
				NULL,
				4357012,
				2130706433,
				40,
				NULL,
				NULL
			)
		<br />
Server Version: 5.0.67-community-log<br />

Could it be anything to do with the bogus email address he inserted (jglgdf;oaaifhvsoz.com) being a malformed email address? Or is it having difficulty adding him as a user because he's already been added as a user from another post in another subforum with another email address?

I'll separate out this other possibly unrelated issue for clarity - on "Rebuild Search Index", FUDForum's still giving me an out of memory error:

Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 71 bytes) in /home/kerrygal/FUDforum/include/theme/default/isearch.inc on line 64

[Updated on: Sun, 15 February 2009 19:32]

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158299 is a reply to message #158294]

Sun, 15 February 2009 20:03

naudefj

Messages: 3771
Registered: December 2004

Karma: 28

Senior Member
Administrator
Core Developer

Here's another patch for you to try: http://cvs.prohost.org/c/index.cgi/FUDforum/chngview?cn=11866

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158300 is a reply to message #158299]

Sun, 15 February 2009 20:28

kerryg

Messages: 157
Registered: September 2008

Karma: 0

Senior Member

OK, applied, rebuilt theme - success, it's moved past that post!

[Total side issue, but - a user that's used a spoofed or gibberish email return address as an anti-spam measure will get added as a member as many times as he spoofed his address. I can certainly live with that for the moment, but is there a way to retroactively reassign such messages to valid user accounts? Actually, since in some old-style USENET communities this isn't an uncommon dodge to avoid spam - is there any way to merge *accounts* retroactively?]

[Updated on: Sun, 15 February 2009 20:32]

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158301 is a reply to message #158300]

Sun, 15 February 2009 20:46

naudefj

Messages: 3771
Registered: December 2004

Karma: 28

Senior Member
Administrator
Core Developer

Unfortunately not. If you need to merge logins, you will have to write a SQL script to manipulate the database directly.

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158302 is a reply to message #158301]

Sun, 15 February 2009 20:53

kerryg

Messages: 157
Registered: September 2008

Karma: 0

Senior Member

Cool. We're about 1000 posts further in the sync and no problems so far, BTW, except for this error that crops up now and then -

mporting IDEA.EMUEnsoniqPARIS message 172441

Warning: mysql_fetch_row(): supplied argument is not a valid MySQL result resource in /home/kerrygal/FUDforum/include/theme/default/db.inc on line 27
(/home/kerrygal/FUDforum/include/theme/default/db.inc:72<br />
/home/kerrygal/FUDforum/include/theme/default/db.inc:132<br />
/home/kerrygal/FUDforum/include/scripts_common.inc:15<br />
/home/kerrygal/FUDforum/include/nntp.inc:416<br />
/home/kerrygal/FUDforum/scripts/nntp.php:92<br />
) 2006: MySQL server has gone away<br />
Query: SELECT id FROM fud26_users WHERE email='rlincoln(at)nospam(dot)kc(dot)rr(dot)com'<br />
Server Version: <br />

Is it anything I should be concerned about?

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158303 is a reply to message #158302]

Mon, 16 February 2009 03:18

kerryg

Messages: 157
Registered: September 2008

Karma: 0

Senior Member

Just an update - nearly 25,000 USENET posts brought across now out of over 45,000 and still working. Once it makes it all the way through, I'm considering deleting the forum and reloading the whole thing from scratch.

I've got it set to import 100 messages at a time. It still chokes on the occasional post (it's happened a few times) and cycles back and duplicates the previous posts. At that point I stop sync, go back, manually clear out any duplicates, set maximum import to 1 post and restart sync. Eventually it chews past the problem post and at that point I boot it back up to 100 (since even at 100 posts per sync it requires constant watching for eight hours to sync the forums, do you think there's a memory setting or a timeout that I can increase that might help it pull this sync off reliably at 100 posts all the way through?)

Here's one of the posts it choked on:

Importing IDEA.EMUEnsoniqPARIS message 189955

Warning: mysql_fetch_row(): supplied argument is not a valid MySQL result resource in /home/kerrygal/FUDforum/include/theme/default/db.inc on line 27
(/home/kerrygal/FUDforum/include/theme/default/db.inc:72<br />
/home/kerrygal/FUDforum/include/theme/default/db.inc:132<br />
/home/kerrygal/FUDforum/include/scripts_common.inc:15<br />
/home/kerrygal/FUDforum/include/nntp.inc:416<br />
/home/kerrygal/FUDforum/scripts/nntp.php:92<br />
) 2006: MySQL server has gone away<br />
Query: SELECT id FROM fud26_users WHERE email='musurgio(at)otenet(dot)gr'<br />
Server Version: <br />

However - I'd have to say that even having to nursemaid the sync along like this represents an *enormous* step forward for this situation. Way to go, Frank!

[Updated on: Mon, 16 February 2009 03:19]

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158304 is a reply to message #158303]

Mon, 16 February 2009 04:24

kerryg

Messages: 157
Registered: September 2008

Karma: 0

Senior Member

OK, LOTS of progress to date: got 28,000 posts in so far, but got up to this one (on 16th April '07) and stalled and cycled back, and can't get past it at maximum posts=1 either. It has an 837k .rar attachment.

Importing IDEA.EMUEnsoniqPARIS message 193976

Warning: mysql_fetch_row(): supplied argument is not a valid MySQL result resource in /home/kerrygal/FUDforum/include/theme/default/db.inc on line 27
(/home/kerrygal/FUDforum/include/theme/default/db.inc:72<br />
/home/kerrygal/FUDforum/include/theme/default/db.inc:132<br />
/home/kerrygal/FUDforum/include/scripts_common.inc:15<br />
/home/kerrygal/FUDforum/include/nntp.inc:416<br />
/home/kerrygal/FUDforum/scripts/nntp.php:92<br />
) 2006: MySQL server has gone away<br />
Query: SELECT id FROM fud26_users WHERE email='musurgioNOSPAM(at)otenet(dot)gr'<br />
Server Version: <br />

http://kerrygalloway.com/ParisForums/index.php?t=thread&frm_id=4&

[Updated on: Mon, 16 February 2009 05:14]

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158305 is a reply to message #158304]

Mon, 16 February 2009 05:50

naudefj

Messages: 3771
Registered: December 2004

Karma: 28

Senior Member
Administrator
Core Developer

The "MySQL server has gone away" error (previous 3 posts) probably means that you are loading too fast for the database to keep up. If the problem persists, restart the DB and check your MySQL log for errors.

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158306 is a reply to message #158305]

Mon, 16 February 2009 06:47

kerryg

Messages: 157
Registered: September 2008

Karma: 0

Senior Member

Yep, you were right; I left it for a while (set to maximum posts = 1) and eventually it chewed through that post too. Looking good!

Update: 34,000 posts across and still going strong. Fingers crossed.

[Updated on: Tue, 17 February 2009 06:22]

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158307 is a reply to message #158306]

Mon, 16 February 2009 21:22

kerryg

Messages: 157
Registered: September 2008

Karma: 0

Senior Member

So close, but something seems to have bogged down last night at the very end. It's almost as if it thinks the sync is finished, but it's only made it to Sept 2007 - (UPDATE: I was wrong, it's throwing errors, I missed them when I reconfigured cron - it's been stalling on one post for 24 hours)

http://kerrygalloway.com/ParisForums/index.php?t=thread&frm_id=4&

UPDATED to add info - have rebuilt everything I could reach, no luck yet. Will keep trying to think of something.

PS: apologies if my cron error forwarding was spamming you!

Further update: this is the current error, slightly different than previous -

Importing IDEA.EMUEnsoniqPARIS message 201175<br />
Warning: mysql_fetch_row(): supplied argument is not a valid MySQL result resource in /home/kerrygal/FUDforum/include/theme/default/db.inc on line 27
(/home/kerrygal/FUDforum/include/theme/default/db.inc:72<br />
/home/kerrygal/FUDforum/include/theme/default/db.inc:168<br />
/home/kerrygal/FUDforum/include/scripts_common.inc:82<br />
/home/kerrygal/FUDforum/include/nntp.inc:453<br />
/home/kerrygal/FUDforum/scripts/nntp.php:92<br />
) 2006: MySQL server has gone away<br />
Query: SELECT m.id, m.thread_id FROM fud26_msg m INNER JOIN fud26_thread t ON m.thread_id=t.id WHERE m.mlist_msg_id='46f94d5d$1@linux' AND t.forum_id=4 ORDER BY m.post_stamp DESC LIMIT 1<br />
Server Version: <br />

[Updated on: Tue, 17 February 2009 06:21]

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158319 is a reply to message #158307]

Tue, 17 February 2009 18:56

kerryg

Messages: 157
Registered: September 2008

Karma: 0

Senior Member

Thanks Frank - still stuck here - how do I restart the db?

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158320 is a reply to message #158319]

Tue, 17 February 2009 20:08

naudefj

Messages: 3771
Registered: December 2004

Karma: 28

Senior Member
Administrator
Core Developer

It should be something like:

# /etc/init.d/mysqld restart

Also check for errors in MySQL's log file. You may need to repair some tables.

Best of luck!

Frank

Report message to a moderator

Re: USENET sync breaks and goes into endless cycle when it encounters attachment [message #158321 is a reply to message #158320]

Tue, 17 February 2009 21:02

kerryg

Messages: 157
Registered: September 2008

Karma: 0

Senior Member

Hi Frank; I'll get in touch with my server for some assistance on restarting the db; I've verified and repaired the tables several times in the last 48 hours using MySQLAdmin; let me know if you reckon that's not adequate or if more is required.

Report message to a moderator

Pages (2): [1 2 › »]

Previous Topic:	Small bug - "Members" list does not sort by join date.
Next Topic:	Small issues probably related to the new search box

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Fri Apr 26 22:22:04 GMT 2024

Total time taken to generate the page: 0.03302 seconds