FUDforum
Fast Uncompromising Discussions. FUDforum will get your users talking.

Home » FUDforum » FUDforum Installation Issues » German characters in old posts after upgrade to 3.0.0
Show: Today's Messages :: Unread Messages :: Show Polls :: Message Navigator
| Subscribe to topic | Bookmark topic 
Switch to threaded view of this topic Create a new topic Submit Reply
German characters in old posts after upgrade to 3.0.0 [message #160907] Thu, 05 November 2009 22:43 Go to next message
jm2morri is currently offline  jm2morri   Canada
Messages: 225
Registered: January 2009
Location: Stratford, Ontario, Canad...
Karma: 0
Senior Member
add to buddy list
ignore all messages by this user
Hello everyone,

I have mostly completed my move to 3.0.0 and it seems to be working very well--thank you very much to Frank et al for all the work.

I've confirmed that the charset appears to be set right and German characters get to the nntp server correctly. This works for new messages.

However, all the old messages that had German characters in it I see the black diamond with question mark instead. I ran the mysql charset converter utility but that hasn't fixed it.

Any ideas?

James.
Re: German characters in old posts after upgrade to 3.0.0 [message #160941 is a reply to message #160907] Sat, 07 November 2009 05:29 Go to previous messageGo to next message
naudefj is currently offline  naudefj   South Africa
Messages: 3687
Registered: December 2004
Karma: 19
Senior Member
Administrator
Core Developer
add to buddy list
ignore all messages by this user
You may need to use the Compact Messages ACP to convert your message files to UTF-8. Please read the documentation and remember to backup before trying it.
Re: German characters in old posts after upgrade to 3.0.0 [message #160961 is a reply to message #160941] Sat, 07 November 2009 13:39 Go to previous messageGo to next message
jm2morri is currently offline  jm2morri   Canada
Messages: 225
Registered: January 2009
Location: Stratford, Ontario, Canad...
Karma: 0
Senior Member
add to buddy list
ignore all messages by this user
naudefj wrote on Sat, 07 November 2009 05:29
You may need to use the Compact Messages ACP to convert your message files to UTF-8. Please read the documentation and remember to backup before trying it.


How do I confirm what encoding is currently used? I can't remember which one was used last.

James.
icon6.gif  Re: German characters in old posts after upgrade to 3.0.0 [message #160964 is a reply to message #160961] Sat, 07 November 2009 14:01 Go to previous messageGo to next message
The Witcher is currently offline  The Witcher   United States
Messages: 662
Registered: May 2009
Location: USA
Karma: 2
Senior Member
add to buddy list
ignore all messages by this user
jm2morri wrote on Sat, 07 November 2009 12:39
How do I confirm what encoding is currently used? I can't remember which one was used last.

James.


I believe the coding is listed in the theme manager utility along with language and the base template set.


"I'm a Witcher, I solve human problems; not always using a sword!"
icon6.gif  Re: German characters in old posts after upgrade to 3.0.0 [message #160965 is a reply to message #160941] Sat, 07 November 2009 14:05 Go to previous messageGo to next message
The Witcher is currently offline  The Witcher   United States
Messages: 662
Registered: May 2009
Location: USA
Karma: 2
Senior Member
add to buddy list
ignore all messages by this user
naudefj wrote on Sat, 07 November 2009 04:29
You may need to use the Compact Messages ACP to convert your message files to UTF-8. Please read the documentation and remember to backup before trying it.


Is it possible that this, and the quotes, are presenting problems because they no longer acknowledge the theme that was originally use?

Like the problem wild cat had?


"I'm a Witcher, I solve human problems; not always using a sword!"
Re: German characters in old posts after upgrade to 3.0.0 [message #160970 is a reply to message #160965] Sun, 08 November 2009 05:00 Go to previous messageGo to next message
naudefj is currently offline  naudefj   South Africa
Messages: 3687
Registered: December 2004
Karma: 19
Senior Member
Administrator
Core Developer
add to buddy list
ignore all messages by this user
To get the discussion back on track, this is not caused by themes, templates or locales.

The data is encoded with the original character set (probably Latin1), but by rendering it as UTF-8, it cannot be displayed properly. This means that a conversion from ISO-8859-1 (an educated guess) to UTF-8 will probably fix it.

The problem is that new data is now added as UTF-8. Which means that the conversion will corrupt the new data. A catch-22.
Re: German characters in old posts after upgrade to 3.0.0 [message #160971 is a reply to message #160970] Sun, 08 November 2009 11:05 Go to previous messageGo to next message
jm2morri is currently offline  jm2morri   Canada
Messages: 225
Registered: January 2009
Location: Stratford, Ontario, Canad...
Karma: 0
Senior Member
add to buddy list
ignore all messages by this user
Thanks for the explanation Frank.

This only effects a few forums and I can re-import them, that should be no problem.

Can I create a new forum and sync it with a news group that is synced with a current news group? Is this going to mess anything up? Or do I need to delete the current forum and create a new one?

Cheers,

James.
Re: German characters in old posts after upgrade to 3.0.0 [message #160974 is a reply to message #160971] Sun, 08 November 2009 14:24 Go to previous messageGo to next message
naudefj is currently offline  naudefj   South Africa
Messages: 3687
Registered: December 2004
Karma: 19
Senior Member
Administrator
Core Developer
add to buddy list
ignore all messages by this user
jm2morri wrote on Sun, 08 November 2009 18:05
Can I create a new forum and sync it with a news group that is synced with a current news group?


It should work. Create a new forum, hide it from users and load it. When done, delete the old forum and make the new one visible.
Re: German characters in old posts after upgrade to 3.0.0 [message #160977 is a reply to message #160974] Sun, 08 November 2009 15:42 Go to previous messageGo to next message
naudefj is currently offline  naudefj   South Africa
Messages: 3687
Registered: December 2004
Karma: 19
Senior Member
Administrator
Core Developer
add to buddy list
ignore all messages by this user
Just for the fun of it, go to the Replacement and Censorship System and configure:

Perl Regex
/([\x80-\xFF])/e
chr(0xC0|ord('\\1')>>6).chr(0x80|ord('\\1')&0x3F)

You need to edit it again to properly set the /e modifier.

It's a long shot, but it might just fix the bad chars.
Re: German characters in old posts after upgrade to 3.0.0 [message #160995 is a reply to message #160977] Mon, 09 November 2009 17:35 Go to previous messageGo to next message
jm2morri is currently offline  jm2morri   Canada
Messages: 225
Registered: January 2009
Location: Stratford, Ontario, Canad...
Karma: 0
Senior Member
add to buddy list
ignore all messages by this user
I tried the replacement and censorship but couldn't make it work. Re-importing is OK anyways.

So I tried that. I can't get any new messages to show up. I tried adding a new forum and linking it to a newsgroup but it wouldn't import anything. When I ran the sync script it said it had 3 messages (which wasn't enough, there should have been something around 10).

But even then, nothing shows up.

So I tried deleting the old forum and newsgroup rule but same results. I tried clearing the tracker for the nntp sync but that didn't help either.

The forums are properly syncing with groups on the same nntp server.

Any ideas?

James.
Re: German characters in old posts after upgrade to 3.0.0 [message #161002 is a reply to message #160995] Tue, 10 November 2009 01:41 Go to previous messageGo to next message
naudefj is currently offline  naudefj   South Africa
Messages: 3687
Registered: December 2004
Karma: 19
Senior Member
Administrator
Core Developer
add to buddy list
ignore all messages by this user
I see we only have one tracker per newsgroup name. This will make it impossible to load the same group into multiple forums. Definitely something we need to fix before the next release.
Re: German characters in old posts after upgrade to 3.0.0 [message #161020 is a reply to message #161002] Tue, 10 November 2009 10:17 Go to previous messageGo to next message
jm2morri is currently offline  jm2morri   Canada
Messages: 225
Registered: January 2009
Location: Stratford, Ontario, Canad...
Karma: 0
Senior Member
add to buddy list
ignore all messages by this user
OK, but now I've deleted both the original forum (that had non-utf-8 characaters) and the new one that I tried to sync to the same news group. Then I added a new one and linked it to the same news group. But now it won't update either, even if I clear the tracker.

Is there some cache or setting I need to clear out? It looks like it thinks the forum is up to date when it isn't?

I can delete the forum again if required--there are only a few messages. Is there a process I should go through to clear out the various internal settings correctly?

James.
Re: German characters in old posts after upgrade to 3.0.0 [message #161021 is a reply to message #161020] Tue, 10 November 2009 10:36 Go to previous messageGo to next message
naudefj is currently offline  naudefj   South Africa
Messages: 3687
Registered: December 2004
Karma: 19
Senior Member
Administrator
Core Developer
add to buddy list
ignore all messages by this user
Please delete all files in the forum's FUDdata/errors/.nntp directory and try again.
Re: German characters in old posts after upgrade to 3.0.0 [message #161023 is a reply to message #161021] Tue, 10 November 2009 13:18 Go to previous messageGo to next message
jm2morri is currently offline  jm2morri   Canada
Messages: 225
Registered: January 2009
Location: Stratford, Ontario, Canad...
Karma: 0
Senior Member
add to buddy list
ignore all messages by this user
naudefj wrote on Tue, 10 November 2009 10:36
Please delete all files in the forum's FUDdata/errors/.nntp directory and try again.


I deleted all the files and reran the sync script--no change. No new messages were reported as being imported and none showed up in the forum.

I then cleared the tracker again. The sync script then reported that 3 messages were imported. But again, none show up in the forum.

James.
Re: German characters in old posts after upgrade to 3.0.0 [message #161024 is a reply to message #161023] Tue, 10 November 2009 14:29 Go to previous messageGo to next message
naudefj is currently offline  naudefj   South Africa
Messages: 3687
Registered: December 2004
Karma: 19
Senior Member
Administrator
Core Developer
add to buddy list
ignore all messages by this user
Can you send me the group, server, etc. so I can try to reproduce the problem?
Re: German characters in old posts after upgrade to 3.0.0 [message #161028 is a reply to message #161024] Tue, 10 November 2009 19:04 Go to previous messageGo to next message
jm2morri is currently offline  jm2morri   Canada
Messages: 225
Registered: January 2009
Location: Stratford, Ontario, Canad...
Karma: 0
Senior Member
add to buddy list
ignore all messages by this user
Sent details via pm.

James.
Re: German characters in old posts after upgrade to 3.0.0 [message #161036 is a reply to message #161028] Wed, 11 November 2009 13:01 Go to previous messageGo to next message
naudefj is currently offline  naudefj   South Africa
Messages: 3687
Registered: December 2004
Karma: 19
Senior Member
Administrator
Core Developer
add to buddy list
ignore all messages by this user
Output after making the script a bit more verbose:

Unable to Fetch Article #190, NTTP Msg: 423 Bad article number
Unable to Fetch Article #191, NTTP Msg: 423 Bad article number
Unable to Fetch Article #192, NTTP Msg: 423 Bad article number


From the above, it looks like some kind of Usenet problem. Can you please ask your admin/provider to have a look?
Re: German characters in old posts after upgrade to 3.0.0 [message #161039 is a reply to message #161036] Wed, 11 November 2009 16:06 Go to previous messageGo to next message
jm2morri is currently offline  jm2morri   Canada
Messages: 225
Registered: January 2009
Location: Stratford, Ontario, Canad...
Karma: 0
Senior Member
add to buddy list
ignore all messages by this user
Thanks for the info Frank. It turns out, that newsgroup was pruned of its old messages.

Tried another, it's importing as we speak and it shows up well.

So I think I'm on the way to finalising this transition.

James.
Aw: German characters in old posts after upgrade to 3.0.0 [message #161828 is a reply to message #160907] Thu, 11 February 2010 15:02 Go to previous message
alopezie is currently offline  alopezie   Germany
Messages: 98
Registered: September 2003
Karma: 1
Member
add to buddy list
ignore all messages by this user
There seems to be a quite simple reason.

Fudforum < 3 has html encoding charset=ISO-8859-15 in the html-header (defined in thm/default/i18n/german), while 3.0 has utf-8

So

1) running the conversion database via upgrade.php AND
2) text files change from ISO to UTF via "compact messages" in ACM AND
3) changing html-header in thm AND
4) rebuildung theme

made it work for me


Alopezie.de - das Forum zum Thema Haarausfall
Quick Reply
Formatting Tools:   
  Switch to threaded view of this topic Create a new topic
Previous Topic: Problems following upgrade to v3
Next Topic: Memory issues when installing
Goto Forum:
  

-=] Back to Top [=-
[ Syndicate this forum (XML) ] [ RSS ]

Current Time: Sun Dec 16 15:49:29 EST 2018

Total time taken to generate the page: 0.00941 seconds