As you may now be aware, I recently joined the team here at Lavasoft as CTO. I genuinely look forward to the many challenges that this industry provides, and I'm very excited about the opportunities that lie ahead for Lavasoft and our customers. From time to time, you'll be hearing from me, here in the corporate blog as well as through our Security Center.

One of the things in this industry that really bothers me is the unstructured approach to product testing by 'independent' testing agencies, as well as other reviewers. Some have the best of intentions, but most are reviewing based on outdated information or processes.

We've received several questions from customers asking about various product reviews out there. Yes, product reviews and product testing is important...but the system of testing needs to be challenged. I've got some pretty strong opinions about 'independent' tests of anti-spyware products, and I plan to release more detailed information soon - particularly as we approach various agencies with a call for standardized processes. Meanwhile, here is a peek at my thoughts about 'independent' product testing :

My Comments on the Testing

With numbers over 100,000, and a claim such as "inactive samples" the test was likely run against one or more collections of threats. This is not a real-world test scenario. This is not a "reality check."

It sounds to me like XXX organization is still stuck in the mentality of detecting viruses, which can indeed be found anywhere on a system. They probably did things like scanning a CD, DVD, or separate hard drive full of inactive threats, usually with "safe" renamed file extensions.

If so then this does not reflect reality, because real users have real, active infections. Otherwise, their system is not actually infected.

Today's anti-malware products generally look for infections, specifically where components and processes those infections are actually located on the system.

While it is true that real-world anti-malware testing is especially difficult these days, using shortcuts, like collection scanning, to simplify anti-malware testing is a disservice to both users and developers; because it does not reflect the real world of real users with real infected systems.

If XXX organization is going to test, they need to test reality. In other words, they need to test, primarily, how products do against real-world infected systems; specifically systems that are infected with current and fully functional threats. Only then can the testing be viewed as scientifically valid, truly useful, and thus a real service to real users.

Note: At the time of this writing, I haven't actually seen the "test methodology" or perhaps "test mythology" depending on how well the testing reflects real users' real system connected to the real world of threats on the Internet.