FUDforum
Fast Uncompromising Discussions. FUDforum will get your users talking.

Home » FUDforum » FUDforum Installation Issues » Backup import woes (Trying to migrate a large-ish forum, and it isn't working)
Show: Today's Messages :: Unread Messages :: Show Polls :: Message Navigator
| Subscribe to topic | Bookmark topic 
Switch to threaded view of this topic Create a new topic Submit Reply
Backup import woes [message #157984] Sun, 18 January 2009 01:23 Go to next message
otherbird is currently offline  otherbird   
Messages: 25
Registered: September 2006
Karma: 0
Junior Member
add to buddy list
ignore all messages by this user
Hi,

I had a fairly busy forum running FUDforum 2.7.6. Now I'm switching to a new hosting provider and decided to upgrade everything in sight during the migration.

So far it's not going too well, f.e. I'm still stuck with PHP 5.1.6 *grumble* and have no way to upgrade that (blame: Plesk). But FUDforum shouldn't care about that, right? 2.7.6 didn't have any problems with 5.1.6.

So I've wgetted and installed FUD about a million times, or so it feels, and the install always goes perfectly whether it's 2.7.7 or 2.7.6, but the data import fails even though I've made all the timeouts ludicrously high in php.ini. It doesn't seem to matter if I change my table prefix from the original or not, and it doesn't seem to matter which version of FUD I install. On the last attempt I had max_execution_time set to 2360 seconds and memory_limit at 256 MB, and the import script still died after about 20 minutes.

If I'm lucky I'm only locked out and unable to enter the admin page because my password isn't known to the forum db. Last time around I didn't even get that much, just totally blank pages all round.

Any bright ideas for getting around this? It all seems to run OK, in that tables are being populated up to a certain point, it's just that it dies before it can import all the data.

Re: Backup import woes [message #157985 is a reply to message #157984] Sun, 18 January 2009 01:49 Go to previous messageGo to next message
naudefj is currently offline  naudefj   South Africa
Messages: 3687
Registered: December 2004
Karma: 19
Senior Member
Administrator
Core Developer
add to buddy list
ignore all messages by this user
Maybe it's not PHP, but your webserver or you browser that is timing out. Try to also increase your webservers' timeout parameter and if your browser quits, see if it continues to run on the back end.

If all else fails, import the database from the command line:
$ mysql -ufudforum -p.... fuddb <forum.dump

Best of luck!

Frank
Re: Backup import woes [message #157986 is a reply to message #157985] Sun, 18 January 2009 01:57 Go to previous messageGo to next message
otherbird is currently offline  otherbird   
Messages: 25
Registered: September 2006
Karma: 0
Junior Member
add to buddy list
ignore all messages by this user
Thanks Frank.

I actually don't have the 'direct' route as an option (an admin freaked out over Gaza and deleted most of 2 years' history a couple of days back - this is a mid-east forum). So I'll hope the server thing works.

The only other thing is that the gzipped 'fud' file actually contains an sqlite3 db, so it should be possible to interrogate it somehow...

- Steph

ps Don't you ever sleep? Smile

Re: Backup import woes [message #157987 is a reply to message #157986] Sun, 18 January 2009 03:23 Go to previous messageGo to next message
naudefj is currently offline  naudefj   South Africa
Messages: 3687
Registered: December 2004
Karma: 19
Senior Member
Administrator
Core Developer
add to buddy list
ignore all messages by this user
I've never worked with sqlite before. However, from http://www.sqlite.org/sqlite.html it looks like the command to import might be something like:

$ zcat forum.dump.gz | sqlite3 fuddb
Re: Backup import woes [message #157988 is a reply to message #157987] Sun, 18 January 2009 04:02 Go to previous messageGo to next message
otherbird is currently offline  otherbird   
Messages: 25
Registered: September 2006
Karma: 0
Junior Member
add to buddy list
ignore all messages by this user
I _think_ that just makes a sqlite sql schema... but there's stuff around on the net about conversion to MySQL, it's just going to be a bit hit and miss.

I've got a bit further now, and it looks like we're both right. The process runs in the background until PHP's max_execution_time (or possibly the socket timeout, since I'm keeping those in sync) is reached. So I've gone up to 40 minutes now, which at least got me through the fud_index barrier if not quite to fud_msg.

Fiddling with httpd.conf didn't appear to have any effect, except that I got the 'failing' admin login page instead of a blank one this time.

I'll report back when I've made it to fud_users. This assuming I ever do.

- Steph
Re: Backup import woes - mystery solved, and a feature request [message #157999 is a reply to message #157988] Sun, 18 January 2009 19:56 Go to previous messageGo to next message
otherbird is currently offline  otherbird   
Messages: 25
Registered: September 2006
Karma: 0
Junior Member
add to buddy list
ignore all messages by this user
OK, so it's taken forever to figure it out but I'm there now.

The import is slow *anyway*, in that the page in the browser goes blank because it times out even though the process is still running in the background. This means you don't get to see error messages. Or at least, I don't. So it wasn't until I gunzipped the backup file and went through all 500 megs of it with a fine-toothed comb that I got a clue.

Two years ago, when our ME forum started up, we had problems attracting Arab contributors. One of the reasons - we thought - was that it can be physically dangerous for Arabs living in the Middle East to talk openly to Israelis. With this in mind, I tweaked the FUDforum code to allow anonymous sign-in with a username of choice, rather than 'Anonymous Coward'. (I also set up a random list of default names while I was in there, just for laughs.) For any of this to work I needed to add a new field to the fud_msg table, anon_user.

Obviously I did this manually, so the backup doesn't know anything about it; the table's created without the field, and then the backup script tries to populate it with my extra data. Kerboom. But only after an hour or so of updating nicely in silence and blankness.

I'm trying to remember now if I also touched the db when I added the 'search by message id' facility to our forum. I sincerely hope not (!) 'cos that's quite a useful feature.

It turns out that the fud.gz file isn't actually an sqlite db, it just includes one in the form of forum.db.php, which itself contains the table schema for FUDforum in sqlite3 format. This file isn't updated during a backup, so it 'knows' nothing about my db changes. Basically if I want to keep my custom field I'll need to make the same change in the newly generated db and then prevent the backup overwriting the table structure somehow. (I'll also need to do this when upgrading, natch.) The easier option would be to delete the extra field of data prior to upgrade, since obviously I changed a bunch of scripts 2 years ago too and I'd otherwise need to figure all that again to retain the existing behaviour. I'm moving toward the easier option; we don't have anonymous sign-in any more these days, so only our archives would be affected by the disappearance of anon_name.

One thing that came out of this experience (apart from 'steph don't mess with application code') is the sheer slowness of a FUD migration, without which debugging would've been relatively plain sailing; not having to wait over an hour every time to see if it worked would've been sweet! With that in mind, would it be possible to leave the geoip table out of the backup script in future FUDforum releases, or to offer the option to do so? It's relatively trivial to update geoip directly from the ip2country site, and excluding that table data should speed up both backup and migration quite considerably.

Cheers anon,

- Steph
Re: Backup import woes - mystery solved, and a feature request [message #158004 is a reply to message #157999] Mon, 19 January 2009 02:35 Go to previous messageGo to next message
naudefj is currently offline  naudefj   South Africa
Messages: 3687
Registered: December 2004
Karma: 19
Senior Member
Administrator
Core Developer
add to buddy list
ignore all messages by this user
I'm glad you've managed to restore your forum!

Quote:

would it be possible to leave the geoip table out of the backup script in future FUDforum releases, or to offer the option to do so?


I'm not sure how to respond to your feature request. It will certainly requires a lot of extra coding, for which I don't have time right now. So, if you feel strongly about it, you are welcome to write a patch. Personally I would just update the documentation to truncate the geoip table before migrations.
Re: Backup import woes - mystery solved, and a feature request [message #158024 is a reply to message #158004] Mon, 19 January 2009 10:36 Go to previous message
otherbird is currently offline  otherbird   
Messages: 25
Registered: September 2006
Karma: 0
Junior Member
add to buddy list
ignore all messages by this user
If I'd known I was going to be using _this_ backup for a migration I'd have done that anyway, but that's a fair point Smile

I'm not there yet, still trying to prevent the schema overwrite. I've gunzipped the dumpfile and am hoping that altering admimport.php (+ reported size) in there will be enough, 'cos I've no idea how forum.db.php is created or whether it's used during import. If it is I'll need to prevent it.

- Steph
Quick Reply
Formatting Tools:   
  Switch to threaded view of this topic Create a new topic
Previous Topic: NNTP Authentication does not work
Next Topic: NNTP posts work for a while, then stop
Goto Forum:
  

-=] Back to Top [=-
[ Syndicate this forum (XML) ] [ RSS ]

Current Time: Sun Dec 16 12:34:30 EST 2018

Total time taken to generate the page: 0.00776 seconds