| Forum Benchmark [message #356] |
Thu, 07 February 2002 22:25  |
Ilia
Messages: 13241
Registered: January 2002
Karma: 0
|
Senior Member
Administrator
Core Developer |
|
|
The purpose of this benchmark is to examine the various features offered by various forums and compare how all these forums perform under similar loads. Hopefully, this will give you an idea of the benefits of each forum software, so that you may choose the one which suits your needs the best, whether it be speed, features or something in between.
The idea originally came from the benchmarks done by Aurora developers and published at http://ikonboard.com/aurora/brochure/bench.html
unfortunately, those benchmarks tell virtually nothing about the forums tested and what kinds of data sets were present in each forum and were they equal.
ie. 100 posts in Aurora & 1000 posts in VBulletin.
We intend to examine most of the commonly used forum packages available today. All of the benchmarks will be performed on the same system and as similar conditions as possible. The conditions cannot be absolutely identical because various forums offer different features on their pages, which may not be available on other forums.
There will be 3 common sets of data: small, medium & large these will be generated. This data will be arranged in an easy to read/parse format to allow anyone to easily import it in to a forum of their choice & try this benchmark for themselves.
The benchmarking tool that we will use is called "httperf", which is written by David Mosberger and Tai Jin of Hewlett-Packard Research Labs and released under the GPL. This tool can be obtained from: http://www.hpl.hp.com/personal/David_Mosberger/httperf.html
Here is the list of intended benchmarks that would be run on the forums:
* Every one of those benchmarks will be run in 3 separate conditions based on the user
- New User coming to a page ( no session or cookie )
- Anonymous user ( a user who has visited the forum before, but is not registered )
- Registered user who is logged in
- Forum's index page generation
- Small Forum ( 3 categories & 3 forums per category )
- Medium Forum ( 6 categories & 5 forums per category )
- Large Forum ( 10 categories & 10 forums per category )
- Forum thread list ( 40 threads per page )
- Small Forum ( 100 threads in a forum )
- Medium Forum ( 1000 thread in a forum )
- Large Forum ( 20000 threads in a forum )
- Message display (flat view)
- Small Forum ( 10 messages in a thread )
- Medium Forum ( 100 messages in a thread )
- Large Forum ( 500 messages in a thread )
- Message display (threaded view)
- Small Forum ( 10 messages in a thread )
- Medium Forum ( 100 messages in a thread )
- Large Forum ( 500 messages in a thread )
- Search Engine Messages (if available)
* A test will be done searching 1st in just the subjects and the in the entire message
- Small Forum ( 9000 messages )
- Medium Forum ( 300000 messages )
- Large Forum ( 1000000 messages )
- Search Engine Users (if available)
* One search benchmark will go through login names & second will go through email address'
- Small Forum ( 100 users )
- Medium Forum ( 1000 users )
- Large Forum ( 10000 users )
In addition to the benchmark there would be a complete list of information that the forum provides for each page, such as login indicators, icons, user options, etc...
We are currently looking for suggestion for other benchmarks & comparisons that can be run between the various forum packages.
If you have such suggestions, please let us know by replying to this message or sending an email to forum(at)prohost(dot)org.
FUDforum Core Developer
|
|
|
|
| Re: Forum Benchmark [message #357 is a reply to message #356] |
Fri, 08 February 2002 01:04  |
mattm
Messages: 2
Registered: February 2002
Karma: 0
|
Junior Member |
|
|
About the Aurora benchmarks:
| Quote: |
unfortunately, those benchmarks tell virtually nothing about the forums tested and what kinds of data sets were present in each forum and were they equal.
ie. 100 posts in Aurora & 1000 posts in VBulletin.
|
From the aurora benchmarks page:
| Quote: |
All tests were conducted on the same system with the same background processes running each time. Both Apache and MySQL were given a complete restart after each series of tests to ensure fairness. All bulletin board systems we tested had the same number of posts, replies, members, and forums.
|
Each bulletin board had the same number of posts, forums and members.
Each bulletin board was populated with the same information, the same data set was converted into the boards native format. mySQL and HTTPD was restarted after the population, and inbetween each bulletin board test.
The bulletin board was then set to run and the average number of pages it rendered was calulated, using the same formula, etc for each test.
By all means, make your own tests but I am confident that Aurora will still come out on top.
There has been much discussion about the validity of the test as Aurora isn't finished. The tests were conducted on the board index, forum listing and topic view pages - all three sections that are complete on Aurora. Infact, there is some optimization still to be performed - so testing the final version will probably be to Auroras advantage.
I'm not trying to start a war here, I'm just clarifying the hear-say.
|
|
|
|
| Re: Forum Benchmark [message #358 is a reply to message #357] |
Fri, 08 February 2002 01:08 |
Ilia
Messages: 13241
Registered: January 2002
Karma: 0
|
Senior Member
Administrator
Core Developer |
|
|
I am not putting your results in questions, I am merely questioning the usefulness of those tests because no one can replicate those benchmarks with the data you've provided.
That is why I am suggesting some sort of a standard benchmark data set, which can be used to compare various forums. Consider the benchmarks done by http://www.spec.org and the amount of information that can be gleamed from every benchmark.
FUDforum Core Developer
|
|
|
|
| Re: Forum Benchmark [message #359 is a reply to message #357] |
Fri, 08 February 2002 01:13 |
hackie
Messages: 177
Registered: January 2002
Karma: 0
|
Senior Member
Core Developer |
|
|
| Quote: |
Each bulletin board had the same number of posts, forums and members.
Each bulletin board was populated with the same information, the same data set was converted into the boards native format. mySQL and HTTPD was restarted after the population, and inbetween each bulletin board test.
The bulletin board was then set to run and the average number of pages it rendered was calulated, using the same formula, etc for each test.
By all means, make your own tests but I am confident that Aurora will still come out on top.
There has been much discussion about the validity of the test as Aurora isn't finished. The tests were conducted on the board index, forum listing and topic view pages - all three sections that are complete on Aurora. Infact, there is some optimization still to be performed - so testing the final version will probably be to Auroras advantage.
I'm not trying to start a war here, I'm just clarifying the hear-say.
|
Okie, but as was stated, when you present benchmarks full disclosure is in order, full disclosure involves the hardware benchmarks were run on (this should go into quite detail! Down the version of the network card and driver used for it).
This information should be enough for anyone that anyone desiring to do so CAN replicate your benchmarks to confirm them.
Aurora most certainly didn't provide the required information to allow anyone interested replicate the benchmarks.
cc intelligence.c -o intelligence
$ ./intelligence
Segmentation fault
|
|
|
|
|
|
| Re: Forum Benchmark [message #361 is a reply to message #360] |
Fri, 08 February 2002 01:23 |
hackie
Messages: 177
Registered: January 2002
Karma: 0
|
Senior Member
Core Developer |
|
|
| mattm wrote on Thu, 07 February 2002 8:18 PM |
The benchmarks were presented in an easy to digest format. The data is unbiased and the equipment used was not intended to give any one system an advantage.
The brochure is designed as a marketing tool - the "average' webmaster will not be able to understand reams f server / process and script debug information.
As previously stated, I'm not against anyone carrying out their own tests, I'm all for it and I'll do all that I can to assist.
|
Heh, okie, well, we came on a bit hard, that's understandable I hope, but anyhow... As I said, what we're trying to do should be interesting to you guys also. That is we will develop a standard set of benchmarks to run on a forum and thereby judge it's speed (we here doesn't mean US, i.e. the developers of FUDforum, this means you guys also).
Some questions/benchmarks suggested are listed in the root message. If you have anything to add please do so. For example, some benchmark that we missed, or seems unfair or maybe pointless.
If you have any suggestions PLEASE make them. We don't wunna be alone on this, and we don't want to be the sole people designing this.
cc intelligence.c -o intelligence
$ ./intelligence
Segmentation fault
[Updated on: Fri, 08 February 2002 01:24] Report message to a moderator |
|
|
|
Calendar
Search
Help
Members
Register
Login
Home
]