FUDforum: How To » Data dump hangs

Home » FUDforum » How To » Data dump hangs

Show: Today's Messages :: Polls :: Message Navigator

Data dump hangs [message #34111]

Fri, 06 October 2006 05:09

mdbrown

Messages: 18
Registered: December 2005

Karma: 0

Junior Member

On my FUDforum 2.7.3 installation a forum datadump apparently hangs while processing the index table. I've let it cook all night, but in the morning it's still stuck on the fud26_index. I'm getting a little nervous about not having a current backup, so I hope someone has a solution. Here's the datadump screen display for your perusal:

Compressing forum datafiles
Processing directory: /home/rvconnection/FUDforum
Processing directory: /home/rvconnection/FUDforum/thm
Processing directory: /home/rvconnection/FUDforum/scripts
Processing directory: /home/rvconnection/FUDforum/include
Processing directory: /home/rvconnection/FUDforum/messages
Processing directory: /home/rvconnection/FUDforum/files
Processing directory: /home/rvconnection/FUDforum/sql
Processing directory: /home/rvconnection/FUDforum/src
Processing directory: /home/rvconnection/FUDforum/errors
Processing directory: /home/rvconnection/FUDforum/cache
Processing directory: /home/rvconnection/FUDforum/template
Processing directory: /home/rvconnection/FUDforum/thm/default
Processing directory: /home/rvconnection/FUDforum/thm/path_info
Processing directory: /home/rvconnection/FUDforum/sql/mysql
Processing directory: /home/rvconnection/FUDforum/sql/pdo
Processing directory: /home/rvconnection/FUDforum/sql/pgsql
Processing directory: /home/rvconnection/FUDforum/errors/.nntp
Processing directory: /home/rvconnection/FUDforum/errors/.mlist
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n
Processing directory: /home/rvconnection/FUDforum/thm/default/tmpl
Processing directory: /home/rvconnection/FUDforum/thm/path_info/tmpl
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/chinese_big5
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/spanish
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/english
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/bulgarian
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/finnish
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/russian-1251
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/italian
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/portuguese_br
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/german
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/swedish
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/norwegian
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/turkish
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/czech
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/chinese
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/dutch
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/russian
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/latvian
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/korean
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/romanian
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/slovak
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/french
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/lithuanian
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/portuguese
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/japanese
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/polish
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/indonesian
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/chinese_big5/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/chinese_big5/help
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/spanish/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/english/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/english/help
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/bulgarian/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/bulgarian/help
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/finnish/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/russian-1251/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/russian-1251/help
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/italian/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/italian/help
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/portuguese_br/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/german/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/german/help
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/swedish/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/norwegian/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/turkish/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/turkish/help
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/czech/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/chinese/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/chinese/help
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/dutch/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/dutch/help
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/russian/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/russian/help
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/latvian/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/romanian/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/slovak/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/french/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/french/help
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/lithuanian/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/portuguese/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/polish/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/polish/help
Processing directory: /home/rvconnection/public_html/images
Processing directory: /home/rvconnection/public_html/images/smiley_icons
Processing directory: /home/rvconnection/public_html/images/message_icons
Processing directory: /home/rvconnection/public_html/images/mime
Processing directory: /home/rvconnection/public_html/images/avatars
Processing directory: /home/rvconnection/public_html/images/forum_icons
Processing directory: /home/rvconnection/public_html/images/custom_avatars
Processing directory: /home/rvconnection/public_html/adm
Processing table: fud26_action_log (1212) .... DONE
Processing table: fud26_ann_forums (0) .... DONE
Processing table: fud26_announce (1) .... DONE
Processing table: fud26_attach (1035) .... DONE
Processing table: fud26_avatar (0) .... DONE
Processing table: fud26_blocked_logins (0) .... DONE
Processing table: fud26_buddy (76) .... DONE
Processing table: fud26_cat (4) .... DONE
Processing table: fud26_custom_tags (1) .... DONE
Processing table: fud26_email_block (0) .... DONE
Processing table: fud26_ext_block (0) .... DONE
Processing table: fud26_fc_view (11) .... DONE
Processing table: fud26_forum (11) .... DONE
Processing table: fud26_forum_notify (5) .... DONE
Processing table: fud26_forum_read (330) .... DONE
Processing table: fud26_group_cache (22) .... DONE
Processing table: fud26_group_members (22) .... DONE
Processing table: fud26_group_resources (13) .... DONE
Processing table: fud26_groups (13) .... DONE
Processing table: fud26_index (793793) ....

Report message to a moderator

Re: Data dump hangs [message #34119 is a reply to message #34111]

Fri, 06 October 2006 15:05

Ilia

Messages: 13241
Registered: January 2002

Karma: 0

Senior Member
Administrator
Core Developer

It is a very large table as you can tell almost 800,000 records, so it will take some time to dump to disk. I suspect your browser has simply timed out before the request had a chance to complete. I'd recommend running the script via command line for a large forum.

FUDforum Core Developer

Report message to a moderator

Re: Data dump hangs [message #34130 is a reply to message #34119]

Fri, 06 October 2006 18:14

mdbrown

Messages: 18
Registered: December 2005

Karma: 0

Junior Member

Ilia wrote on Fri, 06 October 2006 11:05

I don't suppose optimizing the index is going to speed up the dump...

Report message to a moderator

Re: Data dump hangs [message #34132 is a reply to message #34130]

Sat, 07 October 2006 00:10

Ilia

Messages: 13241
Registered: January 2002

Karma: 0

Senior Member
Administrator
Core Developer

Fraid not, it just a big write of lots of data.

FUDforum Core Developer

Report message to a moderator

Re: Data dump hangs [message #34139 is a reply to message #34119]

Sat, 07 October 2006 21:29

mdbrown

Messages: 18
Registered: December 2005

Karma: 0

Junior Member

Ilia wrote on Fri, 06 October 2006 11:05

As I feared, the datadump done via the Admin Control Panel didn't finish completely:

-rw-r--r-- 1 root nobody 120651776 Oct 5 00:02 FUDforum_05_10_2006_12_00.fud.gz

bash-2.05b# gzip -t FUDforum_05_10_2006_12_00.fud.gz

gzip: FUDforum_05_10_2006_12_00.fud.gz: unexpected end of file

Which command-line script are you referring to, specifically? I don't see any likely candidates in the /FUDforum/scripts directory.

Report message to a moderator

Re: Data dump hangs [message #34141 is a reply to message #34139]

Sun, 08 October 2006 05:56

mdbrown

Messages: 18
Registered: December 2005

Karma: 0

Junior Member

mdbrown wrote on Sat, 07 October 2006 17:29

Which command-line script are you referring to, specifically? I don't see any likely candidates in the /FUDforum/scripts directory.

Nevermind, I had overlooked adm/admdump.php.

Actually, I ended up making a copy named "clidump.php" and then uncommented the "fud_use('adm_cli.inc', 1)" and "cli_execute('')" lines in it.

Executing the script from the command line, the datadump finished in (ha!) record time -- and the key word is "finished." Furthermore, it passed gzip's integrity check. I'm going to look at the code to see how tough it would be to suppress all the unneeded HTML code that's generated by the script.

But right now I'm in the process of importing the datadump into a new FUDforum installation. It's been chugging away for about twenty minutes, which makes me nervous about what will happen to the import process if my browser times out. I examined admimport.php to see if it could also be executed from the command line, but guess not. Anyway, I'm keeping my fingers crossed that the import finishes without incident.

BTW, have I mentioned how much I love FUDforum? If not, let me correct the oversight here and now.

Report message to a moderator

Re: Data dump hangs [message #34142 is a reply to message #34111]

Sun, 08 October 2006 07:33

mdbrown

Messages: 18
Registered: December 2005

Karma: 0

Junior Member

UPDATE: The data import has been cooking for a few hours now. I see that the user table has been modified, but none of the others; the messages and files directories seem to be receiving some files; and df indicates FUDforum's file system is growing, albeit sloooowly. The browser displays a page that's blank except for the Admin Control Panel's menu on the left, but at least the browser hasn't timed out yet. Sure wish I could execute admimport.php from the command line instead; it would be less likely to cause an aneurysm.

Report message to a moderator

Re: Data dump hangs [message #34151 is a reply to message #34111]

Mon, 09 October 2006 00:20

mdbrown

Messages: 18
Registered: December 2005

Karma: 0

Junior Member

To heck with using the browser interface to import. The command line is the way to go. I copied admimport.php to cliimport.php and modified it with the --

define('recovery_mode', 1);
$_POST['path'] = '/path/to/dumpfile';

-- method, and fired if off in background mode. Now we're getting somewhere! So far, it has processed 350,000 queries in about an hour and a quarter, so it's not quite half-way finished yet, but it's processing to beat the band. I'm guessing it will take about three hours or so to import the entire 800,000 record dump.

At some point, if no one else has done so, I'll modify both clidump.php and cliimport.php to output non-HTML text to the terminal.

For large, time-consuming operations, running scripts via the command line can't be beat.

Report message to a moderator

Re: Data import blows up [message #34152 is a reply to message #34111]

Mon, 09 October 2006 01:09

mdbrown

Messages: 18
Registered: December 2005

Karma: 0

Junior Member

D'OH again!

php ./cliimport.php &
Processed 10000 queries Processed 20000 queries Processed 30000 queries Processed 40000 queries Processed 50000 queries Processed 60000 queries Processed 70000 queries Processed 80000 queries Processed 90000 queries Processed 100000 queries Processed 110000 queries Processed 120000 queries Processed 130000 queries Processed 140000 queries Processed 150000 queries Processed 160000 queries Processed 170000 queries Processed 180000 queries Processed 190000 queries Processed 200000 queries Processed 210000 queries Processed 220000 queries Processed 230000 queries Processed 240000 queries Processed 250000 queries Processed 260000 queries Processed 270000 queries Processed 280000 queries Processed 290000 queries Processed 300000 queries Processed 310000 queries Processed 320000 queries Processed 330000 queries Processed 340000 queries Processed 350000 queries Processed 360000 queries Processed 370000 queries Processed 380000 queries Processed 390000 queries Processed 400000 queries Processed 410000 queries Processed 420000 queries Processed 430000 queries Processed 440000 queries Processed 450000 queries Processed 460000 queries Processed 470000 queries Processed 480000 queries Processed 490000 queries 
Fatal error: Maximum execution time of 6000 seconds exceeded in /www/assimil8ed/adm/cliimport.php on line 198

[1] Exit 255 php ./cliimport.php

Now what? I'm about 310,000 queries short. If no one has a solution, I guess I'm screwed, blued, and tatooed.

[Updated on: Mon, 09 October 2006 01:29]

Report message to a moderator

Re: Data import blows up [message #34153 is a reply to message #34111]

Mon, 09 October 2006 01:28

mdbrown

Messages: 18
Registered: December 2005

Karma: 0

Junior Member

I increased max_execution_time to 28800 seconds in php.ini. Hope that will solve the problem. But now I have to hack cliimport.php to force it to resume the import at 490000 so as to not duplicate the effort. I guess I'm only screwed and blued, but not tatooed just yet.

[Updated on: Mon, 09 October 2006 01:29]

Report message to a moderator

Re: Data import blows up [message #34155 is a reply to message #34153]

Mon, 09 October 2006 03:32

mdbrown

Messages: 18
Registered: December 2005

Karma: 0

Junior Member

mdbrown wrote on Sun, 08 October 2006 21:28

But now I have to hack cliimport.php to force it to resume the import at 490000 so as to not duplicate the effort.

In regard to resuming the import where it left off, if possible, here's fud26_index's last record:

+---------+---------+--------+
| id | word_id | msg_id |
+---------+---------+--------+
| 1302073 | 22 | 9354 |
+---------+---------+--------+

I've been pretty much flying solo in this thread, but if anyone has any help to offer it would be welcomed.

Report message to a moderator

Re: Data import blows up [message #34157 is a reply to message #34155]

Mon, 09 October 2006 13:49

Ilia

Messages: 13241
Registered: January 2002

Karma: 0

Senior Member
Administrator
Core Developer

Well, increasing the execution time in the script via set_time_limit(0) will probably ensure a complete run. If it still does not work you can forgo the forum's built-in data dump and simply use mysql dump for database info and tar for forum's files.

FUDforum Core Developer

Report message to a moderator

Re: Data import blows up [message #34169 is a reply to message #34157]

Mon, 09 October 2006 20:36

mdbrown

Messages: 18
Registered: December 2005

Karma: 0

Junior Member

Ilia wrote on Mon, 09 October 2006 09:49

Thanks, I'll try that. But first, I'll restore the SQL tables and FUDforum/files and FUDforum/messages directories to their pristine, pre-import state...

Okay, ready to execute...

ono-sendai# php ./cliimport.php &
[1] 9718
ono-sendai# <html>
<head>
<link rel="StyleSheet" href="adm.css" type="text/css">
<meta http-equiv="Content-Type" content="text/html; charset=PHP Notice: Undefined variable: usr in /www/rvconnection/adm/admpanel.php on line 5
PHP Notice: Undefined variable: usr in /www/rvconnection/adm/admpanel.php on line 5
PHP Notice: Undefined variable: usr in /www/rvconnection/adm/admpanel.php on line 7
us-ascii">
</head>
<body>
<table class="maintable">
<tr>
<td class="linkdata">
<table class="linktable">
<tr>
<td>
<a href="../PHP Notice: Use of undefined constant __fud_index_name__ - assumed '__fud_index_name__' in /www/rvconnection/adm/admpanel.php on line 21
PHP Notice: Use of undefined constant __adm_rsidl - assumed '__adm_rsidl' in /www/rvconnection/adm/admpanel.php on line 21
__fud_index_name__?__adm_rsidl">Return To Forum</a>
</td></tr>
<tr><td>
Admin Control Panel</td></tr>
<tr>
<td nowrap>
Checks/Consistency 
<a href="consist.php?PHP Notice: Use of undefined constant __adm_rsidl - assumed '__adm_rsidl' in /www/rvconnection/adm/admpanel.php on line 28
__adm_rsidl">Forum Consistency</a> 
<a href="indexdb.php?PHP Notice: Use of undefined constant __adm_rsidl - assumed '__adm_rsidl' in /www/rvconnection/adm/admpanel.php on line 29
__adm_rsidl">Rebuild Search Index</a> 
<a href="compact.php?PHP Notice: Use of undefined constant __adm_rsidl - assumed '__adm_rsidl' in /www/rvconnection/adm/admpanel.php on line 30
__adm_rsidl">Compact Messages</a> 
.
.
.

And there are a bunch more like that. Hope those are benign messages. The funny thing is, the earlier run that timed out at 6000 seconds didn't have those errors. Here's the output from that run:

ono-sendai# php ./cliimport.php &
<html>
<head>
<link rel="StyleSheet" href="adm.css" type="text/css">
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body>
<table class="maintable">
<tr>
<td class="linkdata">
<table class="linktable">
<tr>
<td>
<a href="../__fud_index_name__?__adm_rsidl">Return To Forum</a>
</td></tr>
<tr><td>
Admin Control Panel</td></tr>
<tr>
<td nowrap>
Checks/Consistency 
<a href="consist.php?__adm_rsidl">Forum Consistency</a> 
<a href="indexdb.php?__adm_rsidl">Rebuild Search Index</a> 
<a href="compact.php?__adm_rsidl">Compact Messages</a> 

General Management 
<a href="admglobal.php?__adm_rsidl">Global Settings Manager</a> 
<a href="admreplace.php?__adm_rsidl">Replacement & Censorship System</a> 
<a href="admmime.php?__adm_rsidl">MIME Managment System</a> 
<a href="admrdf.php?__adm_rsidl">RDF Feed Managment</a> 
<a href="admpdf.php?__adm_rsidl">PDF Generation Managment</a> 
<a href="admspell.php?__adm_rsidl">Custom Dictionary Spell Checker</a> <a href="admbrowse.php?__adm_rsidl">File Manager</a> <a href="admlock.php?__adm_rsidl">Lock/Unlock Forum's Files</a> <a href="admstats.php?__adm_rsidl">Forum Statistics</a> 
<a href="admlog.php?__adm_rsidl">Action Log Viewer</a> 
<a href="admerr.php?__adm_rsidl">Error Log Viewer</a> 
<a href="admsysinfo.php?__adm_rsidl">System Info</a> 

Forum Management 
<a href="admcat.php?__adm_rsidl">Category & Forum Management</a> 
<a href="admdelfrm.php?__adm_rsidl">Deleted Forums</a> 
<a href="admannounce.php?__adm_rsidl">Announcement Manager</a> 
<a href="admprune.php?__adm_rsidl">Topic Pruning</a> 
<a href="admaprune.php?__adm_rsidl">Attachment Pruning</a> 
<a href="admmlist.php?__adm_rsidl">Mailing List Manager</a> 
<a href="admnntp.php?__adm_rsidl">Newsgroup Manager</a> 
.
.
.

BTW, I'm using MySQL 4.0.15 and PHP 4.4.1 on a FreeBSD 4.11 server.

I appreciate your assistance.

Report message to a moderator

Re: Data import blows up [message #34176 is a reply to message #34169]

Tue, 10 October 2006 13:27

Ilia

Messages: 13241
Registered: January 2002

Karma: 0

Senior Member
Administrator
Core Developer

The warnings you are seeing can be ignored, they are caused by the fact that you are bypassing authentication system and running the script on the command line.

No progress indicator from import? You may want to add flush() calls to force PHP to dump output to screen right away and not buffer it in memory.

FUDforum Core Developer

Report message to a moderator

Previous Topic:	Attachment problem
Next Topic:	how view one image in a post

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Wed Nov 27 19:25:05 GMT 2024

Total time taken to generate the page: 0.04650 seconds