Review sites and reliability - The truth

[Gouhan]

So here's something that's become increasingly annoying with review sites. The reviews are carried out with little to no concern for measurement precision or accuracy. This doesn't apply to all the sites of course, some are still as reliable as ever and should be read and soaked as always. Others however I've seen them deteriorate and while they may have been acceptable before, they are downright ridiculous right now.
Here's how this whole thing shakes out from a financial point of view if you weren't aware.

Websites, need ad revenue to survive. Since nobody pays to read the Tech Report, Guru3D etc for example. They generate revenue and keep the site going through the adverts. Without these adverts the sites would shut down and that's how the entire industry is structured. This is no different form normal magazines that you buy from a store shelf. Advertisers pay the publications and websites, and the websites and publications should in theory write free and unbiased content. Obviously you can see where there may be a conflict of interest despite some sites claiming that editorial and advertorial agreements are mutually exclusive. On paper they may be but in reality they are not.
The deterioration of reliable sources of information or distortion of the truth however has very little to do with this above structure. It is flawed for it's own reasons, but that is not why I'm writing this today. What I'm writing about is technical journalism that is lazy. I understand when one writes for a particular demographic that may not be interested in the details or the technical side of things. Perhaps if you're writing to an audience that is technical but you're not writing a technical piece but rather an overview of your experience with the product within a technical community. Those are not to be confused with what I am talking about here. Specifically I'm referring to technical journalists who (by the way I would rather say writers as journalism is something that is to be studied which nobody in the industry has as far as I'm aware) by their very nature engage a pseudo technical or perhaps knowledgeable audience, but they themselves are lacking in objectivity and technical know how. the former of the two is actually more troublesome especially because it stems from laziness and complacency.

Let me make you the following example.
We have just gone through the entire NVIDIA GTX 970 debacle. I'm not here to re-hash what has been said already, it's done and hopefully everyone is in a better place now than before. The problem here is in how sites handled this entire issue. There is one site that is guilty of all I'm writing about here in particular and I'll name it at the end but for now consider the following. NVIDIA doesn't spend money on publication advertorial. NVIDIA is not a public facing company, they do not have us the end users as customers, but rather their add in board partners etc. Short of the NVIDIA Shield and all related consoles, they are vendor and semiconductor or compute firm. This applies to the graphics division at AMD as well. ASUS, Club3D etc they are the customers for these vendors, not the end user.
Now that this is clear, why does it matter? Well, it matters because AMD (for other reasons) much like NVIDIA do not place advertorial on websites, therefor anyone site or publication can say anything about the company without consequence, objectivity or evidence. If we have a review site here on Carbonite, we can trash NVIDIA and AMD pretty much as well as we like. Reason being, there is no consequence to it as they don't pay the bills. Can we do the same to ASUS, GALAX, MSI etc? Nope, no we can't because if we do then there will be a fallout from that .
As such the issue with the GTX 970 received so much attention, flared by several tech sites, yet in reality do you know how many end users returned cards, called in to any vendor to complain etc?
One major NVIDIA vendor, largest independent vendor in the USA) had three applications. Two of which didn't follow through with the R.M.A procedure, thus only one actually had the GTX 970 exchanged. In percentages for this particular vendor it was less than 0.9% of the 970s they shipped.

Another GPU partner who I will not name, even larger told their distributors that performance is identical and no such correspondence will be entered into about the issue, let alone granting any R.M.A despite NVIDIA stating they would reimburse all returned stock. The said GPU partner does advertise on one of the inflammatory sites and they had this information and they said nothing to the forum dwellers, Facebook followers etc. Nothing, yet they called out NVIDIA on the 970 issue. This same website, has previously had difference with the partner I mentioned earlier that had a return rate of 1 card for the 970 and even then, the situation became explosive through sensationalist journalism.

Now here's another situation that took place last year at the launch of the X99 motherboards.
I'm in a position where I am able to get a hold of upcoming components for the desktop space usually before retail availability in most places of the world. This isn't always the case but it does happen every so often. Many times I am able to go through several revisions of components long before they are even announced or at least are available for distributors to order. By the time many tech journalists receive these components I've usually used a fee of them over some time and have seen the various driver and firmware updates. Turning a lemon of a product into a stellar purchase. What I've seen from this very same website, is them give an award to a motherboard in particular that I knew for a fact was in essence non functional. In fact the BIOS they received on their board couldn't even load X.M.P profiles for memory. I know this because I had over a dozen of these BIOS revisions, cooked specifically to fix various issues. I know this because I was there, helping diagnose issues on a daily basis, for weeks on end.
This site, went on to award this X99 motherboard a recommended award, in the state that they received it in where basic stability was compromised, one of the issues being that the board would fail to POST after a regular system shutdown. None of this was in the review, despite it being a review that spans over 12 pages in length.

Then there's an issue regarding graphics card reviews. Where there are graphs with over 12 GPUs in over 10 games and benchmarks.
An admirable amount of work to put in for a review. I know that this is plenty of work and it's annoying to say the least and very time consuming. In fact I've spent over 12 hours in the last 18 hours just running through benchmarks and that's with just four variables, which are two operating systems and two graphics cards. Now consider that this site amongst others are presenting performance figures of 10 graphics cards over 10 benchmarks, then literally translates into well over 100 hours of testing. Then there's the writing, uploading, photography, generating graphs etc. Let alone any anomalies you may find during testing that very often means you must throw out every single result and start again. Given that hardware is given to technical writers, only a week before NDA lifts, and sometimes only days. It's near impossible to see how this can be managed. Let alone write reviews of more than one variation of the same platform or product. What that tells me is that either there are massive teams behind these websites in which case that the author needs to be changed to reflect that. Or more likely something more sinister is going on, like fudged benchmarks results.

To be continued, in a bit.

 

[Gouhan]

Part II
Here is another example of how unreliable and outright misleading several of these sites have become as well. THG used to be a well renowned and respected website for instance and through a series of manipulated and benchmark results on not a single but up to four reviews in succession through 1999 until 2002. The site lost virtually all credibility. Today, they are far more thorough, but that has tarnished the reputation in a way as to cast doubt on virtually all editorial they generate even when it's correct.
A classic example is in how, it was only THG that investigated the AMD Radeon R290 and R290X issue with the shipping BIOS on retail boards versus the media sample BIOS. To this day this discrepancy in performance exists. Right now, a media sample R9 290 and R9 290X still deliver higher performance than that which you would get from a retail sample. It has nothing to do with fan curves, an ubber mode in the BIOS or a normal mode. This was as clear as day when a retail board I was testing at higher clock speeds, resulted in lower performance than the media sample at lower clocks. Even with the retail card clocking in 100MHz on the core higher. This is an outstanding issue today and for every review site that still uses those results, they are invalid as not a single retail card or 3rd party Radeon R9 290/290X performs like these, regardless of drivers (scaling would be linear when using the same driver).

Sticking with the Radeon R9 290X. How odd that, the higher performance of the R9 290X at UHD resolutions was downplayed against the GTX 780Ti. Now it's true that the difference can be diminished and overcome by factory overclocks, but when comparing reference card to reference card, the 290X is ahead. You may think this is immaterial since the GTX 970 and GTX 980 have been released, but this brings us to another issue that is yet again glossed over as if it doesn't exist. The GTX 780Ti is faster than the GTX 980 with all things equal. Take for instance the fastest GTX 780Ti on the market, the EVGA GTX 780Ti K|NGP|N Edition which has a boost clock of 1215MHz. An identical clock to that of the GIGABYTE GTX 780Ti GHz Edition. Compare these graphics cards against the fastest GTX 980 on the market; the EVGA GTX 980 Classified with a boost clock of 1,442MHz. At UHD settings, the K|NGP|N and GHz Edition 780 Ti GPUs are faster. Not by a small margin either. Yet this is represented nowhere. You may have seen UHD "benchmarks" but do take note of when the benchmarks were done for the GTX 780Ti, what drivers were used and compare them with the drivers used for the GTX 980 which came more than a year later. These same results are then stacked against the Media sample Radeon R9 290X and R9 290 results. That directly means nothing in those graphs in respect to each of these graphics cards is true (780Ti, GTX 980 and R9 290X).

Now, please look at the following paying specific attention to the GTX TITAN and the GTX 770 GPUs:
The 1920x1080 results are newer drivers than 1920x1200 results. The 1080P results are on a 4.2GHz 4770K while the 19920x1200 results are on a 3770K @ 4.6GHz. Note that these are GPU bound settings and the CPU makes absolutely no difference here as this discrepancy is repeated again at UHD vs 5760x1080 (where the latter resolution is less demanding of course)



I"m not here talking about Techpowerup, but another site specifically. I'm only using their graphs as an example.

I will share more examples in the third part, but suffice to say I"m sure you get the picture quite vividly. The testing methodology and subsequent data presented is next to useless.
Back to the 290X issue. Now you may be thinking why AMD didn't call out all sites about remaining silent regarding UHD/4K performance?
Well, that would mean all testing had to be redone with retail cards. The very ones that do not offer the same samples as the initial Media sample batch from AMD. Which would then highlight clearly that the performance of these versus retail cards has still not been resolved yet these are the benchmarks they would want to use to highlight the R290X's superiority over then competition at these particular settings. As such, everyone just sits on this, doesn't think about. Some don't even know while others pretend it never happened and isn't happening.

To be continued

 

[NeeekoBeeeko]

Wow, very interesting read. Well done on making all points relevant.

How different are the media GPU's compared to retail GPU's? (Besides clock speed)

I'm all for reviews. I'd used to run my own review site a number of years ago.

Basically, at the end of the day, if you want your site to be well done and effective, you need to go full time. It's almost impossible to manage a full blown site with hundreds/thousands/more of active users while still running a 9-5 job.

 

[crackOlicious]

Wow, very interesting read. Well done on making all points relevant.

How different are the media GPU's compared to retail GPU's? (Besides clock speed)

I'm all for reviews. I'd used to run my own review site a number of years ago.

Basically, at the end of the day, if you want your site to be well done and effective, you need to go full time. It's almost impossible to manage a full blown site with hundreds/thousands/more of active users while still running a 9-5 job.

9-5? You lucky[emoji14]

 

[Oj0]

I've put two and two together, I'm the furthest from a fan of that writer for that site, I feel he is incredibly arrogant. In fact, I'm not a fan of that site in general.

 

[Oj0]

Ha, that site has a few lines of rubbish English directly copy pasted between multiple reviews

 

[NeeekoBeeeko]

9-5? You lucky[emoji14]

Lol, maybe 8-5

 

[NeeekoBeeeko]

I've put two and two together, I'm the furthest from a fan of that writer for that site, I feel he is incredibly arrogant. In fact, I'm not a fan of that site in general.

That site?

 

[Mike]

So what are some sites that can be trusted?

 

[NeeekoBeeeko]

So what are some sites that can be trusted?

Surely TechReport should be one of them, but ShockG recommends not. Though I must point out, they do have an additional stream of revenue, that being the paid subscriber service.

TekSynidcate? They don't look too bad either.

HardOCP seems to also have their heads on the right way.

 

[Oj0]

So what are some sites that can be trusted?

The Inquirer and SemiAccurate

/ducks

 

[NeeekoBeeeko]

Fudzilla can sometimes be good, though they are very anti-Apple and more AMD biased.

 

[blogbytes]

Great article.

One thing though, you said this:

"The GTX 780Ti is faster than the GTX 980 with all things equal. Take for instance the fastest GTX 780Ti on the market, the EVGA GTX 780Ti K|NGP|N Edition, that has a boost clock of 1215MHz. An identical clock to that of the GIGABYTE GTX 780Ti GHz Edition. Compare these graphics cards against the fastest GTX 980 Ti on the market, the EVGA GTX 980 Classified with a boost clock of 1,442MHz. At UHD settings the K|NGP|N and GHz Edition 780 Ti GPUs are faster. not by a small margin either."

So you're saying they are only faster in UHD? How do you measure the speed? In benchmarks? It's funny cause I had a GHz gigabyte 780ti and it was slower in firestrike than the msi 980 4G. I've done 1080p, 1440p and the 4k test. In all 3 tests I had a higher score with the 980?


Sent from my iPhone using Tapatalk

 

[OhiAs_ZA]

Great article.

One thing though, you said this:

"The GTX 780Ti is faster than the GTX 980 with all things equal. Take for instance the fastest GTX 780Ti on the market, the EVGA GTX 780Ti K|NGP|N Edition, that has a boost clock of 1215MHz. An identical clock to that of the GIGABYTE GTX 780Ti GHz Edition. Compare these graphics cards against the fastest GTX 980 Ti on the market, the EVGA GTX 980 Classified with a boost clock of 1,442MHz. At UHD settings the K|NGP|N and GHz Edition 780 Ti GPUs are faster. not by a small margin either."

So you're saying they are only faster in UHD? How do you measure the speed? In benchmarks? It's funny cause I had a GHz gigabyte 780ti and it was slower in firestrike than the msi 980 4G. I've done 1080p, 1440p and the 4k test. In all 3 tests I had a higher score with the 980?


Sent from my iPhone using Tapatalk

Interesting....

 

[el3venth]

One major NVIDIA vendor, largest independent vendor in the USA) had three applications. Two of which didn't follow through with the R.M.A procedure, thus only one actually had the GTX 970 exchanged. In percentages for this particular vendor it was less than 0.9% of the 970s they shipped.

1 person followed through with their RMA procedure, and that accounts for 0.9% of their 970s that was shipped.
so 1/111 = +- 0.9%. So I am guessing this major vendor has sold 111x GTX 970s.

If that is the sales of nVidia, I can see why the Titan series is so expensive

 

[Jack_Bauer_ZA]

I watch youtube reviews (teksyndicate, linustips, etc) not for helping to decide what to recommend to a customer or buy myself, but for the sheer look of delight and comedy the guys deliver. Reviews mean boggerol to me. What ShockG said is absolutely true. So many sites "test" a card and reading the review i think wow this product must be awesoem. Then i buy and try just to see that in my pc it performs no better than the previous model/product.

Like a 970 for me. I had EVGA and Msi models that was hyped in so many reviews and guys what? In my pc they don't perform any better than a 290 el cheapo ref i had. Actually coil wine/downclocking else thay fail in games all issues that i had on diff 970's. Go figure the hype. Try for yourself before you buy.

 

[Gouhan]

1 person followed through with their RMA procedure, and that accounts for 0.9% of their 970s that was shipped.
so 1/111 = +- 0.9%. So I am guessing this major vendor has sold 111x GTX 970s.

If that is the sales of nVidia, I can see why the Titan series is so expensive

Less than. This bit is important and in no way does it mean 0.9%. 0.0009% is still less than 0.9%
[MENTION=887]blogbytes[/MENTION]
The synthetic results in this case show the exact opposite of what the games show. Only HWBOT's Unigine Heaven Xtreme preset is in line with the game performance at UHD. 3DMark Vantage through to FS Extreme are going to show the 980 ahead, regardless of settings. Which is precisely why synthetic benchmarks not based on game engines are more often than not a misrepresentation of performance.

Tomb Raider: 3840x2160 Ultra detail, TressFX disabled
EVGA GTX 780Ti KPE: 52FPS
EVGA GTX 980 Class: 31.4FPS


Alien Isolation: 3840x2160 SMAA T2x, HDAO Maximum detail
EVGA GTX 780Ti KPE: 44.74FPS
EVGA GTX 980 Class: 31.95FPS

Forceware 349.90 and 347.88 used (Win10 10041 and Windows 7 SP1)

 

[Oj0]

[MENTION=64]ShockG[/MENTION] How are you finding Alien Isolation? The preview videos looked great, the reviews don't look so great Sorry I had to dash earlier, I had an unexpected visitor and then figured it's late your side of the world anyway.

 

[blogbytes]

Less than. This bit is important and in no way does it mean 0.9%. 0.0009% is still less than 0.9%
[MENTION=887]blogbytes[/MENTION]
The synthetic results in this case show the exact opposite of what the games show. Only HWBOT's Unigine Heaven Xtreme preset is in line with the game performance at UHD. 3DMark Vantage through to FS Extreme are going to show the 980 ahead, regardless of settings. Which is precisely why synthetic benchmarks not based on game engines are more often than not a misrepresentation of performance.

Tomb Raider: 3840x2160 Ultra detail, TressFX disabled
EVGA GTX 780Ti KPE: 52FPS
EVGA GTX 980 Class: 31.4FPS


Alien Isolation: 3840x2160 SMAA T2x, HDAO Maximum detail
EVGA GTX 780Ti KPE: 44.74FPS
EVGA GTX 980 Class: 31.95FPS

Forceware 349.90 and 347.88 used (Win10 10041 and Windows 7 SP1)

Thanks for the explanation.

Is it just tomb raider where the 780s outrun the 980s or other games to? And only at UHD res? Or could it just be that the Kepler drivers are more matured?

I am happy with my 980. Do I feel a difference compared to the 780ti? Don't be silly


Sent from my iPhone using Tapatalk

 

[NeeekoBeeeko]

I watch youtube reviews (teksyndicate, linustips, etc) not for helping to decide what to recommend to a customer or buy myself, but for the sheer look of delight and comedy the guys deliver. Reviews mean boggerol to me. What ShockG said is absolutely true. So many sites "test" a card and reading the review i think wow this product must be awesoem. Then i buy and try just to see that in my pc it performs no better than the previous model/product.

Like a 970 for me. I had EVGA and Msi models that was hyped in so many reviews and guys what? In my pc they don't perform any better than a 290 el cheapo ref i had. Actually coil wine/downclocking else thay fail in games all issues that i had on diff 970's. Go figure the hype. Try for yourself before you buy.

I'm also starting to prefer these youtube videos. They are fun to watch. But, I still enjoy reading reviews too.

 

[Gouhan]

Video reviews are always great to watch for sure but there's a downside to that which is not intentional but happens passively. These days people who consume components etc or component news are less knowledgeable than before. The buying public has been distilled gradually into expecting a cell phone like review for a graphics card. There isn't something inherently wrong with video reviews, in many ways they are just better than the written word. The issue is that, these video reviews are made to be exciting in presentation not in content. So an audience is told what they need to be concerned about and what they don't need to be concerned about. It is unlike a video review of a game where seeing it in motion is precisely what you want. When dealing with things of a technical nature, it is worth being able to go back to a piece of literature, reading it again and having it click. This isn't a prerequisite for the consumption of any information or how people make purchasing decisions, but it has the side effect of encouraging lazy journalism. end users have virtually no objective analysis of any technology and it's my nvidia vs your amd, but on both camps nothing is understood thoroughly. Where forums used to be a place to find out more about any one technology, learn and engage, they are places where lines are drawn and people are perpetually arguing about whatever side they happen to support. Anecdotal evidence is used as empirical proof or any one opinion or another. An active example is how poorly DirectX 12, 11 and even 10 was communicated across to end users. with each iteration causing confusion and even worse lack of interest.

As you know MS is to release DX12 alongside Windows 10. We can all discuss how much of a shame it is to not release DX12 to Windows 7 users, but .. well that's shameful corporate greediness I am afraid. DirectX 12, both Nvidia and AMD are prepping for it in both software at driver level and of course hardware APUs and graphics cards).

That's from the chief writer at Guru3D. He who has more access to technical personal at all major vendors and developers should know better. The information is there publicly and in private as to why it isn't possible to bring DirectX12 to Windows7. However he will write this on a major website and because of a systematic dulling (not intentional or a focused effort to do so) of the audience, this kind of thing goes unchallenged mostly and worse yet, believed.

 

[Gouhan]

Thanks for the explanation.

Is it just tomb raider where the 780s outrun the 980s or other games to? And only at UHD res? Or could it just be that the Kepler drivers are more matured?

I am happy with my 980. Do I feel a difference compared to the 780ti? Don't be silly


Sent from my iPhone using Tapatalk

This is in every game.
A fundamental problem with many reviews these days is that most of them lack sensibility to what it is GPUs or whatever else they are doing is supposedly used for. The synthetic tests dont' relate to anything meaningful in the way they are represented and there's no trend line or story that's put together to represent performance.

For instance, if you take a GTX 980 Classified and compare it with the 780 Ti Classified exclusively at UHD, you'll find that 780Ti is always faster.
The issue is more websites run the following settings: 3840x2160 4xMSAA, then report the results.
1. at a rudimentary level anti-aliasing, works to solve the problem of not having enough pixel data to represent any one polygon or more. By design, AA methods are interpolation schemes that generate pixel data from a fewer number of pixels.
2. This is a problem that diminishes the higher image resolution becomes. with an 8MP image and depending on the screen size, you actually do not need AA at all. It is possible though to exhaust the memory resources on a 780Ti long before you do so on a 980 simply because it has a 1GiB advantage. since the image is too taxing on the GPU anyway, you end up with the 980 delivering 12fps while the 780Ti delivers 8 or even 7fps. From that the conclusion is made that the 980 is better at UHD gaming.
The truth is that at any setting where the 3GiB frame buffer on the 780Ti becomes a limit to performance, the 980 would not have the power to process that image size anyway so it would be unplayable anyway. The true measure of UHD/4K performance is to set 3840x2160 but with no AA then measure the performance. That way you re not testing the frame buffer size, but actual GPU power.

 

[Oj0]

[MENTION=64]ShockG[/MENTION] you still around? I don't want to message you in case you've hit the hay already.

 

[Gouhan]

Crucial DDR4 2133 MHz 32 GB (4x 8 GB) Review | techPowerUp
Cadaveca strikes again! DDR4 memory review with no overclocking results.

 

[axon1988]

Hey guys, so with all of this in mind, are there some sites that less informed folks like myself can go on to read about hardware reviews that will not mislead me?

I've read through all of this and it's reminded me about how many sites are biased against AMD's cpus. Or at least that is how I understand it. Maybe we can make a list of sites that are trustworthy?

 

[Gouhan]

Hey guys, so with all of this in mind, are there some sites that less informed folks like myself can go on to read about hardware reviews that will not mislead me?

I've read through all of this and it's reminded me about how many sites are biased against AMD's cpus. Or at least that is how I understand it. Maybe we can make a list of sites that are trustworthy?

You will still end up at the regular sites like Anandtech, THG (which has improved tremendously and is actually more reliable now than the competitors).
As for AMd, by their own submission, their CPUs sucked and you need only speak to former employees who will tell you as much and why they left (others we fired etc). Perhaps what speaks to that is how Zen is very much closer to the Phenom II in design than it is bulldozer cores. Much like how Intel gave up on Pentium4 and went back to what made the Pentium-3 great. For AMD however it took 10 years, whereas for Intel it took 5 years to ride out their mistaken architecture bets.

 

[baasgene]

Re: AMD Fiji/Fury more details

Gigabyte GeForce GTX 980 Ti G1 Gaming SOC Review - Introduction

The G1 is another 15% faster than the reference Ti?? Da fug. Now just waiting on the review of the MSI Gaming 6g before I order, screw Pascal that's only next year. The time is now

 

[Gouhan]

Re: AMD Fiji/Fury more details

Heads up on GURU3D's review.
The scores in that review are incorrect!
I've checked and checked again, they do not match up at all.
What is supposedly reference scores are overclocked. something just doesn't wash with those results. I'll be switching to a near identical setup as he is running just to confirm. However it's either GIGABYTE is shipping different BIOS versions or something is wrong with my system which I doubt. (OC results match and exceed his but are in line with clock speed increases)
Even his CPU score isn't right. @ 4.4GHz, I've yet to see a CPU score that high especially with memory at 2133MHz. Something is fishy!

 

[PancakesZA]

Re: AMD Fiji/Fury more details

[MENTION=64]ShockG[/MENTION] Which review are you referring to bud?

 

[Gouhan]

Re: AMD Fiji/Fury more details

The linked one of the GTX 980Ti from GURU3D.
The one linked above by bassgene which actually shouldn't be here as this is a Fijii thread.
Gigabyte GeForce GTX 980 Ti G1 Gaming SOC Review - Introduction

Nevermind.
His clocks are just not stock that's all. He is running a 100MHz higher than stock when he reports stock results. SO Turbo frequency (real) is around 1500MHz!
I wouldn't put any faith in that review as it is. Results are not correct!