FUDforum: FUDforum Installation Issues » German characters in old posts after upgrade to 3.0.0

Home » FUDforum » FUDforum Installation Issues » German characters in old posts after upgrade to 3.0.0

Show: Today's Messages :: Polls :: Message Navigator

German characters in old posts after upgrade to 3.0.0 [message #160907]

Fri, 06 November 2009 03:43

jm2morri

Messages: 225
Registered: January 2009
Location: Stratford, Ontario, Canad...

Karma: 0

Senior Member

Hello everyone,

I have mostly completed my move to 3.0.0 and it seems to be working very well--thank you very much to Frank et al for all the work.

I've confirmed that the charset appears to be set right and German characters get to the nntp server correctly. This works for new messages.

However, all the old messages that had German characters in it I see the black diamond with question mark instead. I ran the mysql charset converter utility but that hasn't fixed it.

Any ideas?

James.

Report message to a moderator

Re: German characters in old posts after upgrade to 3.0.0 [message #160941 is a reply to message #160907]

Sat, 07 November 2009 10:29

naudefj

Messages: 3771
Registered: December 2004

Karma: 28

Senior Member
Administrator
Core Developer

You may need to use the Compact Messages ACP to convert your message files to UTF-8. Please read the documentation and remember to backup before trying it.

Report message to a moderator

Re: German characters in old posts after upgrade to 3.0.0 [message #160961 is a reply to message #160941]

Sat, 07 November 2009 18:39

jm2morri

Messages: 225
Registered: January 2009
Location: Stratford, Ontario, Canad...

Karma: 0

Senior Member

naudefj wrote on Sat, 07 November 2009 05:29

You may need to use the Compact Messages ACP to convert your message files to UTF-8. Please read the documentation and remember to backup before trying it.

How do I confirm what encoding is currently used? I can't remember which one was used last.

James.

Report message to a moderator

Re: German characters in old posts after upgrade to 3.0.0 [message #160964 is a reply to message #160961]

Sat, 07 November 2009 19:01

The Witcher

Messages: 675
Registered: May 2009
Location: USA

Karma: 3

Senior Member

jm2morri wrote on Sat, 07 November 2009 12:39

How do I confirm what encoding is currently used? I can't remember which one was used last.

James.

I believe the coding is listed in the theme manager utility along with language and the base template set.

"I'm a Witcher, I solve human problems; not always using a sword!"

Report message to a moderator

Re: German characters in old posts after upgrade to 3.0.0 [message #160965 is a reply to message #160941]

Sat, 07 November 2009 19:05

The Witcher

Messages: 675
Registered: May 2009
Location: USA

Karma: 3

Senior Member

naudefj wrote on Sat, 07 November 2009 04:29

You may need to use the Compact Messages ACP to convert your message files to UTF-8. Please read the documentation and remember to backup before trying it.

Is it possible that this, and the quotes, are presenting problems because they no longer acknowledge the theme that was originally use?

Like the problem wild cat had?

"I'm a Witcher, I solve human problems; not always using a sword!"

Report message to a moderator

Re: German characters in old posts after upgrade to 3.0.0 [message #160970 is a reply to message #160965]

Sun, 08 November 2009 10:00

naudefj

Messages: 3771
Registered: December 2004

Karma: 28

Senior Member
Administrator
Core Developer

To get the discussion back on track, this is not caused by themes, templates or locales.

The data is encoded with the original character set (probably Latin1), but by rendering it as UTF-8, it cannot be displayed properly. This means that a conversion from ISO-8859-1 (an educated guess) to UTF-8 will probably fix it.

The problem is that new data is now added as UTF-8. Which means that the conversion will corrupt the new data. A catch-22.

Report message to a moderator

Re: German characters in old posts after upgrade to 3.0.0 [message #160971 is a reply to message #160970]

Sun, 08 November 2009 16:05

jm2morri

Messages: 225
Registered: January 2009
Location: Stratford, Ontario, Canad...

Karma: 0

Senior Member

Thanks for the explanation Frank.

This only effects a few forums and I can re-import them, that should be no problem.

Can I create a new forum and sync it with a news group that is synced with a current news group? Is this going to mess anything up? Or do I need to delete the current forum and create a new one?

Cheers,

James.

Report message to a moderator

Re: German characters in old posts after upgrade to 3.0.0 [message #160974 is a reply to message #160971]

Sun, 08 November 2009 19:24

naudefj

Messages: 3771
Registered: December 2004

Karma: 28

Senior Member
Administrator
Core Developer

jm2morri wrote on Sun, 08 November 2009 18:05

Can I create a new forum and sync it with a news group that is synced with a current news group?

It should work. Create a new forum, hide it from users and load it. When done, delete the old forum and make the new one visible.

Report message to a moderator

Re: German characters in old posts after upgrade to 3.0.0 [message #160977 is a reply to message #160974]

Sun, 08 November 2009 20:42

naudefj

Messages: 3771
Registered: December 2004

Karma: 28

Senior Member
Administrator
Core Developer

Just for the fun of it, go to the Replacement and Censorship System and configure:

Perl Regex
/([\x80-\xFF])/e
chr(0xC0|ord('\\1')>>6).chr(0x80|ord('\\1')&0x3F)

You need to edit it again to properly set the /e modifier.

It's a long shot, but it might just fix the bad chars.

Report message to a moderator

Re: German characters in old posts after upgrade to 3.0.0 [message #160995 is a reply to message #160977]

Mon, 09 November 2009 22:35

jm2morri

Messages: 225
Registered: January 2009
Location: Stratford, Ontario, Canad...

Karma: 0

Senior Member

I tried the replacement and censorship but couldn't make it work. Re-importing is OK anyways.

So I tried that. I can't get any new messages to show up. I tried adding a new forum and linking it to a newsgroup but it wouldn't import anything. When I ran the sync script it said it had 3 messages (which wasn't enough, there should have been something around 10).

But even then, nothing shows up.

So I tried deleting the old forum and newsgroup rule but same results. I tried clearing the tracker for the nntp sync but that didn't help either.

The forums are properly syncing with groups on the same nntp server.

Any ideas?

James.

Report message to a moderator

Re: German characters in old posts after upgrade to 3.0.0 [message #161002 is a reply to message #160995]

Tue, 10 November 2009 06:41

naudefj

Messages: 3771
Registered: December 2004

Karma: 28

Senior Member
Administrator
Core Developer

I see we only have one tracker per newsgroup name. This will make it impossible to load the same group into multiple forums. Definitely something we need to fix before the next release.

Report message to a moderator

Re: German characters in old posts after upgrade to 3.0.0 [message #161020 is a reply to message #161002]

Tue, 10 November 2009 15:17

jm2morri

Messages: 225
Registered: January 2009
Location: Stratford, Ontario, Canad...

Karma: 0

Senior Member

OK, but now I've deleted both the original forum (that had non-utf-8 characaters) and the new one that I tried to sync to the same news group. Then I added a new one and linked it to the same news group. But now it won't update either, even if I clear the tracker.

Is there some cache or setting I need to clear out? It looks like it thinks the forum is up to date when it isn't?

I can delete the forum again if required--there are only a few messages. Is there a process I should go through to clear out the various internal settings correctly?

James.

Report message to a moderator

Re: German characters in old posts after upgrade to 3.0.0 [message #161021 is a reply to message #161020]

Tue, 10 November 2009 15:36

naudefj

Messages: 3771
Registered: December 2004

Karma: 28

Senior Member
Administrator
Core Developer

Please delete all files in the forum's FUDdata/errors/.nntp directory and try again.

Report message to a moderator

Re: German characters in old posts after upgrade to 3.0.0 [message #161023 is a reply to message #161021]

Tue, 10 November 2009 18:18

jm2morri

Messages: 225
Registered: January 2009
Location: Stratford, Ontario, Canad...

Karma: 0

Senior Member

naudefj wrote on Tue, 10 November 2009 10:36

Please delete all files in the forum's FUDdata/errors/.nntp directory and try again.

I deleted all the files and reran the sync script--no change. No new messages were reported as being imported and none showed up in the forum.

I then cleared the tracker again. The sync script then reported that 3 messages were imported. But again, none show up in the forum.

James.

Report message to a moderator

Re: German characters in old posts after upgrade to 3.0.0 [message #161024 is a reply to message #161023]

Tue, 10 November 2009 19:29

naudefj

Messages: 3771
Registered: December 2004

Karma: 28

Senior Member
Administrator
Core Developer

Can you send me the group, server, etc. so I can try to reproduce the problem?

Report message to a moderator

Re: German characters in old posts after upgrade to 3.0.0 [message #161028 is a reply to message #161024]

Wed, 11 November 2009 00:04

jm2morri

Messages: 225
Registered: January 2009
Location: Stratford, Ontario, Canad...

Karma: 0

Senior Member

Sent details via pm.

James.

Report message to a moderator

Re: German characters in old posts after upgrade to 3.0.0 [message #161036 is a reply to message #161028]

Wed, 11 November 2009 18:01

naudefj

Messages: 3771
Registered: December 2004

Karma: 28

Senior Member
Administrator
Core Developer

Output after making the script a bit more verbose:

Unable to Fetch Article #190, NTTP Msg: 423 Bad article number
Unable to Fetch Article #191, NTTP Msg: 423 Bad article number
Unable to Fetch Article #192, NTTP Msg: 423 Bad article number

From the above, it looks like some kind of Usenet problem. Can you please ask your admin/provider to have a look?

Report message to a moderator

Re: German characters in old posts after upgrade to 3.0.0 [message #161039 is a reply to message #161036]

Wed, 11 November 2009 21:06

jm2morri

Messages: 225
Registered: January 2009
Location: Stratford, Ontario, Canad...

Karma: 0

Senior Member

Thanks for the info Frank. It turns out, that newsgroup was pruned of its old messages.

Tried another, it's importing as we speak and it shows up well.

So I think I'm on the way to finalising this transition.

James.

Report message to a moderator

Aw: German characters in old posts after upgrade to 3.0.0 [message #161828 is a reply to message #160907]

Thu, 11 February 2010 20:02

alopezie

Messages: 106
Registered: September 2003

Karma: 1

Senior Member

There seems to be a quite simple reason.

Fudforum < 3 has html encoding charset=ISO-8859-15 in the html-header (defined in thm/default/i18n/german), while 3.0 has utf-8

So

1) running the conversion database via upgrade.php AND
2) text files change from ISO to UTF via "compact messages" in ACM AND
3) changing html-header in thm AND
4) rebuildung theme

made it work for me

Alopezie.de - das Forum zum Thema Haarausfall

Report message to a moderator

Previous Topic:	Problems following upgrade to v3
Next Topic:	Memory issues when installing

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Sat Nov 23 20:57:43 GMT 2024

Total time taken to generate the page: 0.02661 seconds