Subscribe to the feed.
(An improved benchmark is being developed. A link to it will be here when it is completed.)
When it comes to a browser benchmark people act like this:
The problem with browser benchmarks today is that they are extremely synthetic. A majority of them are Javascript only because its easier to test.
Why Do We Need a Better Benchmark?
- Sunspider – Javascript only.
- V8 - Javascript only.
- Dromaeo – Mostly Javascript, some CSS.
- Acid Test – Random web standards. No one really cares unless you get close to 100 or 0.
- Peacekeeper – It runs Java and then allows itself to download and run an EXE. It spends most of the time doing a 3D benchmark on your browser as though you were running 3dMark.
Peacekeeper: You must click the smaller link at the bottom to avoid running an EXE
(Note: Pasi of Futuremark has clarified that Peacekeeper only runs the executable when you run without a system scan. It is also noted that the 3d looking tests are actually canvas tests.
Maybe 2 more FPS on my browser if I overclock?
As you can see, Javascript benchmarks dominate the landscape. There is a complete lack of benchmarks with anything remotely applying to real life situations.
Thus there are many who completely dismiss these benchmarks, stating that they don’t really measure the “true speed” of a browser. Maybe they are right.
How to Make a Better Benchmark?
According to Microsoft:
Knowing the top speed of a car doesn’t tell you how fast you can drive in rush hour. To actually see the difference in page loads between all three browsers, you need slow-motion video. This one’s also a tie.
Using slow-motion video? What a great idea. Maybe we can benchmark operating systems like that.
However there is a built-in timer which can let us know when the browser finishes loading a page. The hard part is getting it to load in a new page while using websites that people actually use in real life. Details explained later.
Setup
Core 2 Duo 1.8 GHZ 2GB Mem Vista 32bit Chrome 3.0.189.0 Firefox 3.5 RC 2 Internet Explorer 8.0.6001.18702 Opera 10.00 Beta Build 1551 Safari 4.0 (530.17) 0 Extensions.
Websites
Here are the websites used in this test.
- Baidu.com – Chinese Google
- Blogger.com – Popular blogsite
- Facebook.com – Popular social site
- Google.com – Popular search Engine
- Havenworks.com – A poorly designed site weighing 820 kb with over 280 images
- Live.com – Microsoft search engine
- Myspace.com/tom – Typical Myspace profile
- Reddit.com – Link aggregator
- Wikipedia.org – Online encyclopedia
All websites were saved to disk and opened beforehand into the browser to minimize the impact of the few remaining external links and hard drive seek times.
Results: Average Time to Load 1 Page – 10 Repetitions (in ms)
Chrome
- Baidu – 229
- Blogger – 38.6
- Facebook – 146.1
- Google – 11
- Havenworks – 482.2
- Live – 42.3
- Myspace – 536.5
- Reddit – 141.4
- Wikipedia – 222.8
- Total Average – 205.5
Firefox
- Baidu – 37.3
- Blogger – 93.9
- Facebook – 263.9
- Google – 56.4
- Havenworks – 1377.8
- Live – 76.9
- Myspace – 1386.4
- Reddit – 337.3
- Wikipedia – 114.1
- Total Average – 416
Internet Explorer
- Baidu – 32.6
- Blogger – 92.2
- Facebook – 238.7
- Google – 34.4
- Havenworks – 2722.2
- Live – 107.7
- Myspace – 922.6
- Reddit – 451
- Wikipedia – 410.2
- Total Average – 556.8
Opera
- Baidu – 24
- Blogger – 77.5
- Facebook – 214
- Google – 27.5
- Havenworks – 584
- Live – 47
- Myspace – 500
- Reddit – 279
- Wikipedia – 83
- Total Average – 204
Safari
- Baidu – 238.3
- Blogger – 58.9
- Facebook – 203.4
- Google – 22.4
- Havenworks – 438.3
- Live – 47
- Myspace – 442.8
- Reddit – 161.8
- Wikipedia – 233.6
- Total Average – 205.2
Conclusion
Average Loading Times (in ms)
Opera 204 Safari 205.2 Chrome 205.5 Firefox 416 Internet Explorer 556.8
Opera is the winner, but not by far. Interestingly, the browsers all perform differently on different websites.
Before You Complain
Download
Download the benchmark here and use your own websites if you like. Follow the instructions in the source code.
Frequently Asked Questions
Q: According to some outdated website, webkit fires onload way too early. Therefore your benchmark is flawed.
A: It was fixed 2 years ago.
Q: Your statistics suck. You should have used a geometric mean with n degrees of freedom and 95th percentile measurements with a chi-squared test to calculate the deviation from a Weibull distribution … etc…
A: I have yet to hear a convincing reason why a simple average is not good enough. The only complaints I have so far is from people saying that the Havenworks bench is unfair. Why should I weigh it any differently than the other sites? Run your own tests.
Q: Can you include my favorite X browser and X platform so your benchmark doesn’t suck?
A: I don’t have infinite time, money, computers, and incentives. If you can offer any of those, maybe I will oblige.
Q: Why save websites to disk? Why not just load them online?
A: Because we’re testing loading speed, not your connection and how slow the website is.
Q: But what about all the super duper network code that my browser has?
A: The only feature that would make any appreciable difference would be concurrent connections (which I believe all modern browsers have). Other features like DNS look-ahead (likely cached) or different networking engines probably costs microseconds compared to the milliseconds it takes for a page to load.
Q: Why isn’t my favorite website there?
A: Because I couldn’t find a way to reliably download and adapt them. Wget refuses to work correctly, and some websites have some Javascript that messes with the script. So I just used Firefox to download. (Yahoo, Slashdot, and Digg don’t play nicely)
Q: Some of your test sites still have external links.
A: I tried HTTrack, wget, and the browsers for saving pages, and none of them work 100%. You can fix it by hand editing, but I don’t have the time, money, and incentive to do so completely. I have removed some by hand, but websites nowadays like to obfuscate their javascript making it a hassle.
Q: Why didn’t you use DOMContentLoaded etc..?
A: This was what I wanted to use. But it doesn’t work on IE8, and 90% of the time, the browser is stuck and useless in that stage (especially when loading from hard drive instead of online).
Q: You have to paste that tiny script into every webpage you want to benchmark. That’s lame.
A: My first version did not require that. Blame IE8 for being the only browser to break it.
Q: My browser is still the best.
A: Ok
Q: Your benchmark is worthless because it doesn’t show the benefit of extensions or how it matches the color of my eyes, I will never consider using another browser.
A: Ok
Related posts:
Q: Why save websites to disk? Why not just load them online?
A: Because we’re testing loading speed, not your connection and how slow the website is.
This doesn’t take into account serialized vs parallel JS file loading, which has a disproportionally high impact on performance.
I forgot to mention that I had already pre-opened each site in each browser to minimize the impact of hard drive seek time and the few remaining external links I couldn’t find and remove.
This was a great post on loading time of pages. Though loading time of a page shouldn’t be the only thing considered when running a browser benchmark.
“Though loading time of a page shouldn’t be the only thing considered when running a browser benchmark.”
I’m not sure about you but that’s what I use my browser for… to load webpages. So if one can do it faster than another, that saves me time.
ps that quote by microsoft is priceless. Basically them saying we know our browser isn’t by any means the fastest but most sites are quick enough that we’ll call it a tie.
If I see correctly, Opera only outperforms Safari and Chrome on the baidu and wikipedia pages. (maybe google and live, but that seems less relevant)
It would be interesting to see how/why. For baidu, I’d venture maybe something to do with handling of unicode characters? And if you’re testing on the homepage of wikipedia (definitely not your average webpage), I’d venture the same.
Any idea ?
Even though you got lucky with the results (Opera is indeed the fastest), this is pseudoscience.
Bong: HELLO CAPTAIN OBVIOUS
Google’s browser is the fastest at loading Google, but MSN’s is the slowest at loading Live. These things happen, I guess
The fact that the Firefox load times for Havenworks and Myspace are so close to eachother is interesting. I wonder if there’s something attempting to load on those pages which causes Firefox to wait for a ~1.3 second timeout.
I’m not convinced your distribution was normal either. You should give more information. average load times can be skewed heavily. Try a boxplot.
They looked normal to me. The timing was pretty much the same across the 10 repetitions because I preloaded the pages. If you don’t believe me, try it out yourself.
Are you loading the Facebook homepage or a profile page for testing?
Homepage. You need to sign up to access the profiles so I didn’t test a profile.
Using total averages seems a really twisted way to compare browsers. Since havenworks/myspace is so much bigger than the rest of the sites mentioned, when you say “total average” you are essentially saying “how it loads havenworks and myspace”. The site by site comparisons are useful even if your test is not exactly rigorous.
.
I thought about that before I published the results.
But it just seemed unfair to isolate each page if one browser totally bombs havenworks and myspace and zips through the others.
Take a look at IE, it beats both Chrome and Safari on Baidu, but it does not deserve to be near the top spot if for example you only compare Baidu, Blogger, and Havenworks with relative scores.
I’m curious as to what affect some of those sites have in relation to their invalid markup or javascript. MySpace, for example, is a coding nightmare and you can’t account for how a browser will react to such pages gone horribly wrong.
So is one browser faster with a 100% standards compliant page? It should be faster but is one always faster than the others? Do they, then, all run about the same? Should we back pat a browser for handling bad code faster? Does that say anything, really, about that browser if it can?
Can you try Chrome 2.0.172.3x and see if it’s any better than Chrome 3.0.x when it comes to loading Baidu?
Your chart does not display the data optimally. Here are a couple of tables that show differences from the average for each site:
http://dmccabe.org/07/browser-benchmarks/browsers.html
I suspect your test is flawed, because some browsers fire their onload event before the page has really finished loaded and rendering. Notably Safari (and I presume Chrome, which is also Webkit based)… http://ajaxian.com/archives/safari-3-onload-firing-and-bad-timing
Yes this question has been raised, but the bug was already fixed a long time ago.
https://bugs.webkit.org/show_bug.cgi?id=13241
It seems that the problem maybe still exist, e.g.
https://bugs.webkit.org/show_bug.cgi?id=15505
https://bugs.webkit.org/show_bug.cgi?id=17761
Hi
Thanks for your work. It is appreciated.
But why are you making so much emphasis about people complaining about the results. Let them complain. A fact is a fact. Numbers can speak for themselves.
BR
paines
Regarding your overall average. It was mentioned before that the large webpages skew the average badly, which it kind of does. Can you figure out how much memory is downloaded when viewing a site and do a weighted average of the sum of the loadtimes divided by the size of the website?
It would be interesting to compare the percentage difference between the browsers online and load-from-disk. You may need more samples to remove kinks in the online versions.
Overall it’s good work. It doesn’t matter if people whine about fairness or otherwise. If you post the methodology and the data, You’ve added something constructive. With those, people can reach their own conclusions using their own criteria of what is important.
This benchmark sorta confirms mu supposition that even with all the speed of Chrome and Safari on the JS side, browsing speed doesn’t improve proportionally.
Also, the last two questions in the FAQ are pure awesome!
Thank you for taking the time to do this!
Regarding our Peacekeeper benchmark, contrary to what your saying Peacekeeper doesn’t run the benchmark from the .exe, the installed component is only provided for system information detection, so we can match scores to system details.
You can verify this yourself by running Peacekeeper without the system scan from this link: http://service.futuremark.com/peacekeeper/run.action
No installing, no .exes.
The test you have a screen capture of actually displays a video taken from 3DMark Vantage and adds a ripple effect using the canvas element from html5, so unlike 3DMark it doesn’t render any 3D, but rather tests how fast the browser is in rendering graphics with canvas.
It’s still true that Peacekeeper runs mostly JavaScript stuff in
addition to some rendering tests, but claiming that it mostly runs 3DMark.exe in your brower is just incorrect.
Cheers,
Pasi Virtanen
Site Manager
Futuremark
Thanks for the information Pasi. I’ve updated.
First of all thanks for testing.Resuls are same as I wait
Opera is the fastest everytime and also this is good decision to make this test on local drive to see exact and wright result webpages…
But also,if you make different test as online it would be better..We are also waiting your online test results..
Thanks again.
Thanks for this great new testing. As you say, pure javascript speed is not always the most interesting factor.
However, to detail that even more, I would say that the typical day of a websurfer consists of many, many sites, and some of the sites you use more than others. Furthermore, I would not be surprised to see that a specific site favours a specific browser because of their internal politics (or external, for that matter). For instance, I would be only half surprised if Facebook were better optimized for IE than Firefox, deep in the code.
Anyway, it’s very good work, thanks !
PS: If I were evil, I would optimize my sites to slow down browsers I do not like.
some quick calculations i did to get the standard deviation (something i don’t think is unnecessary since it’ll tell you which browser is more consistent http://en.wikipedia.org/wiki/Standard_deviation)
turns out chrome and safari is more consistent than opera and ie is a joke for consistency.
> let chrome = [229, 38.6, 146.1, 11, 482.2, 42.3, 536.5, 141.4, 222.8]
> sd chrome
178.5536589818379
> let ff = [37.9, 93.9, 263.9, 56.4, 1377.8, 76.9, 1386.4, 337.3, 114.1]
> sd ff
524.7450237972724
> let ie8 = [32.6, 92.2, 238.7, 34.4, 2722.2, 107.7, 922.6, 451, 410.2]
> sd ie8
811.16315403816
> let opera = [24, 77.5, 214, 27.5, 584, 47, 500, 279, 83]
> sd opera
199.04926804967434
> let safari = [238.3, 58.9, 203.4, 22.4, 438.3, 47, 442.8, 161.8, 233.6]
> sd safari
146.9788722534259
yay for haskell…also i should’ve done it with R, but i couldn’t be bothered installing it just for this quick calculation.
Hello Mark thanks for the information, but it doesn’t make too much sense to calculate the standard deviation on completely different webpages because it is expected that they will have largely different loading times.
The loading times also doesn’t follow a normal curve when you calculate it across pages like that.
In the next version of the benchmark which is about 70% complete,it will automatically calculate the std-dev of the same websites and overall averages to satisfy the statistics geeks. So stay tuned. It may pop up in a couple days.
You have to NORMALIZE your numbers before doing average.
BASIC procedure for usable “average”:
1) average the times for Baidu
2) write down every result as percentage of the average from 1)
3) repeat this for every website
4) with the numbers from 2) and 3) you can do SUM (and optionally divide by number of websites) for “weighted average” numbers
This is not the 100% right method but shall suffice for any meaningfull comparisons.
Whithou that normalization, you shouls just publish the sumarized graph and forget the averages. You will save yourself the disgrace.
BTW, this normalization would not bee needed provided your mix of websites represented exactly the mix of every web user.
That is not likely to happen so normalization is a MUST.
Without normalization, you are giving too big a weight to (your specific) sites which load slowly (are more complex).
BTW, doing deviation _across_ sites is a complete madness. For it to have ANY value you would require several hundred sites normalized by their real market share …
Mino, I thought I had already made myself clear, but this is why normalization does not add much value to the tests:
If I visit 10 different websites, and it takes each browser around 10 seconds to load them, I will have wasted 10 seconds.
Let’s say that 1 website takes around 95% of the time. Do I care if the 9 other websites take only take milliseconds to load? No. I will still have wasted 10 or more seconds.
Now let’s suppose I did exactly what you and many others wanted. Relative scoring:
Site 1:
Browser A: 1 ms
Browser B: 2 ms
Site 2:
Browser A: 1 ms
Browser B: 2 ms
Site 3:
Browser A: 50 seconds
Browser B: 25 seconds
According to you and every other person that complained, Browser A should be the winner because it won 2 out of 3 websites with a 1:2 ratio.
However, according to my tests, Browser B would win overall. If you still think that browser A should win, then I don’t know what to tell you. Not every histogram should be normalized willy nilly.
Mino: “this normalization would not bee needed provided your mix of websites represented exactly the mix of every web user.”
Its not possible to represent the exact mix of every web user but 8 out of 9 of the websites I picked were in the top 20 of Alexa’s most visited websites.
This is why I released the source code to allow people to test their own websites.
On a side note, the second version of this benchmark should be out soon.
Yes, I have mentioned this point also.
The problem with that is you are assuming that you website selection/distribution is representative.
It is not. Will never be.
Please, do not confuse “representative” with average. You selection may be representative for only several users even though it is average a whole country.
There is alway question how much normalization you want to introduce but some you shall include always.
Plain average is not very useful if data points are as diverse as we see here.
On the other hand fully normalized do not include the psychological importance of slow sites.
I would go with normalized average while commenting out the summary graph to take care of the slow/complex sites part.
Anyway, this is you show so just my 2 cents. Good luck
Sorry for the wording, three o’clock here
Ugh man, when i started reading your ranting i went blind for 2 seconds, your point and the arguments behind it are completely worthless (2 cents after all aint that much right?)
“Is not, it will never be”
Who cares all that crap? We are talking about real world stuff here, 20 top most visited sites, which browser runs faster, “This browser will be the fastest browser ever loading all websites in 0.1 sec… starting year 2202″
Haha no thanks, my granny doesn’t give a crap about your blah blah.
Well not quite. When I open 10 websites in my browser, even if just one of them lags terribly, I will start reading / logging on / whatever I wanted to be doing, at any of the other 9 tabs. So I will not waste all that time. That’s what tabbed browsing is about. We generally don’t browse internet in a linear way anymore.
I agree with Mino that some degree of normalization is needed for such benchmark.
Can you tell me if you performed the tests with browsers with addons disabled ?
So for instance for IE using:
run IEXLORE.EXE -extoff
Else thrid party addons like flash or java could influence the test.
It is very strange that you included the havenworks page as it is a both a terrible webpage and it has a ridiculous impact on your teststats.
I only disabled extensions, not npapi/activex plugins like flash. I didn’t have Java on. I left flash on because it is an important part of the overall browser speed. They were all using the same version of flash, so the speed differences shouldn’t be because one browser had a faster flash than another but because of interface and rendering schemes.
I included Havenworks because I thought it would be a good stress test, and some websites do happen to look like that. Amazingly the Myspace page is not too far off in terms of loading times, so it is not totally unreasonable.
Interesting..
Makes me glad to be using Opera 10! [full release] build 1750.
Good article,
\o/
Soooo… is this Real Websites benchmark still forthcoming, or did development on this stop?
It would be a pretty cool tool for web developers who are working on optimizing a site’s performance/loading time.
-Steve
Steve, its sitting here 70% completed.
I lost my motivation to complete it because it really only serves to appease the statistics fanatics, and I earned only ~50 cents in advertisements even there have been 23k visitors to this page to date.
If you have little need for the standard deviation and other statistics, there isn’t a problem with adapting this one.
nice post, great article, I especially like the Q&As towards the end. BTW my browser really is the best…
Are you guys talking about wasting time here!? Or developing a true Browser Benchmarking scheme for “real world” browser use and analysis?
The overall idea here, as I understand it, was to perform a set of benchmarks on loading times for common/popular pages on common/popular, fresh, default browsers because there are so few “real world” benchmarks out there. It was NOT, from my understanding, stating that the data from the the said benchmarks is the END all and BE all of browser speed. I use Opera@USB on MANY different computers with MANY different connection types. I use portable firefox with a max of 62 connections per site. I can understand most of how this benchmark works but not all. Is the default # of CPP (connections per page) the same on all browsers? What kind of bandwidth do you have available? Thus, I would have to agree with some of the arguments on here.
This data and Benchmarking Scheme is a GREAT way for obtaining generalized load time on “default browsers”. Especially for web developers to get an idea of how fast their page will load on 10% of the web browsers out there without 6 tool bars and 30 add-ons.
Now a days you can so heavily modify most browsers, I just don’t know. For example my results would be heavily skewed because of the changes I’ve made to my browsers and the fact I have 2X(21MB)/sec cable connections bridged, one for up and one for down with strict load balancing.
I wouldn’t say your data/results are irrelevent because there are those out there that run a clean, mostly add-on free browser(s). These figures matter in that sense.
But if you where to tell a general user that, “this browser is faster at loading these sites.” I can guarantee you it won’t be for long, and if they install ANY new browser it will be faster then the one they “were” using.
“Opera is the winner, but not by far. Interestingly, the browsers all perform differently on different websites.”
THE BROWSERS ALL PERFORM DIFFERENTLY ON DIFFERENT WEBSITES.
This will always be the case and one browser will never truly be faster then another. We should stop looking at the results as “what browser you should use” and start looking at the results as what can you change to make the browser you like, are developing, are coding your page for, faster.
Love the work and this post was VERY will done. There are just SO many variables out there.
Every user would have to install and use (on a daily basis) each browser, as they normally would, THEN run the benchmark. Take those benchmark results and run them through and averager ( I know it’s not a word), then you might find the fastest browser, for YOU.
My suggestion would be to make an add-on for the popular browsers that saves the load time statistical values (ms to load, amount of data downloaded, speed of download/upload, flash blocker? Ad blocker? that is less data to download) and upload said data anonymously to an averager, then output real time averages to the add-on visually. That would give you a better idea of “real world” browser speed. And the geeks here know how much general users LOVE add-ons and junk. You could probably get a pretty awesome following with some great statistical data.
Anyways, I’m only really posting this to practice for my online class discussions for school. So what I say here really is worth 2 cents, maybe even a hay penny.
Keep up the good work!
BTW. I love opera and it is my default, but for some reason the built in spell checker considers “a” in the middle of “a” sentence as mis-spelled. It may be faster, but that doesn’t mean it’s better.
I completely agree. I recently decided that I wanted to find which suited ME best, so I downloaded Chrome and Opera. I’ve been using Firefox for ages (I quit using IE back when it was still IE 5 or something like that) so I thought I’d try out something new, just to see. And given the fact that browsing experience is very personal, and therefore extremely subjective and full of variables, let me give you one more. The actual browsing experience, while giving the impression of being the most visible metric, isn’t necessarily the most important. There is something to be said for PC (or MAC) security. Due to rampant adware, malware, virusi, trojans, etc., ad nauseum, having a browser that helps mitigate these varying threats seems to me to be as important, if not more so, as page load time. If I have a super-duper-lightning-faster-than-everything-else-in-the-universe browser, but even the most elementary of malicious code is allowed through and within a week I have an essentially useless computer (assuming, among other things, 0-day threats that my security suite fails to stop, at least initially), how good does that actually make the browser?
Obviously, having proper firewall and anti-virus protections in place is a must, even with the most sophisticated of internet browsers, however, completely ignoring the security issues of certain software seems obtuse when the last statistic I heard (this is purely anecdotal, so you science purist, cover your ears) was that a fresh install of Windows without any updates (I want to say it was SP1), no firewall or antivirus protection, connected to the internet with the integrated, non-updated IE was completely incapacitated due to malicious code within minutes. Again, completely anecdotal and heresay, but it’s a stat that I wouldn’t be surprised at, were it to be true.
I’ve seen some interesting blogs concerning security amongst browsers, but being the psuedo-scientist that I am, I don’t have links for you. However, my point is not to bash this browser or that on security, but to raise the issue of, where’s the benchmark for that. It would be interesting to see a similar benchmark as is exhibited here, but include metrics for malicious code. I don’t know how the various security suites determine the malicious code counts that I’ve seen them use for marketing purposes, but using similar methodology, you could design a benchmark that tests real-world browsing experience, adding in the various arbitrary compliance tests, adding in security metrics and then throw all the statistical values into a blender, puree for 30 seconds, pour over hot-headed fanatics and let stew for several months. See what happens. My 60 Russian kopeks.