In late February, AVG paired its updated anti-virus engine with a real-time malware scanner that vets search engine results before you click on them. If you search Google, for instance, this LinkScanner automatically visits each address that turns up on Google's results page.
According to the company, more than 20 million people have
downloaded the new AVG 8, and this has caused a huge up-tick in traffic
on sites across the web, including yours truly. Because the
scanner attempts to disguise itself as a real live human click,
webmasters who rely on log files for their traffic numbers may be
unaware their stats are skewed. And others complain that LinkScanner
has added extra dollars to their bandwidth bill.
Daniel Brandt, who runs Wikipedia Watch,
estimates that LinkScanner traffic to the site has outstripped
legitimate clicks by nearly ten times. In this graph, the pink line
represents suspected LinkScanner scans, the blue line legitimate clicks:
When we first told the tale
of AVG's fake traffic earlier this month, we pointed out that if
webmasters were wise to the problem, they could filter LinkScanner
visits from their log files. Each scan left a unique user agent:
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)."
But over the weekend, the company changed this user agent on the for-pay version of AVG 8. It appears that scans now use these agents as well:
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)
Judging from the log files of two separate web sites, including
Wikipedia Watch, the first agent is by far the most common. Which is
bad news for webmasters. That's also the Internet Explorer 6 user
agent. Unlike the other two - and the original "1813" agent - it's a
perfectly valid agent that may turn up with real clicks.
AVG's chief of research Roger Thompson says the for-pay LinkScanner is only
using the IE6 user agent. Presumably, the company believes this is more
likely to fool malware exploits. "There are still ways for concerned
web masters to filter LinkScanner requests out of their statistics," he
told us over email. But he did not acknowledge that this could clip legitimate traffic as well.
Many webmasters may have no choice but to abandon log file analysis,
adopting alternative tools from companies like Google, Yahoo!,
comScore, or Nielsen NetRatings. And these tools have their drawbacks.
comScore's service tends to underestimate traffic from daytime work
machines. And if you go with Google Analytics, you have to tag your
pages with JavaScript - and share your traffic numbers with Google.
Plus, these tools won't solve the bandwidth issue.
In an effort to fix this problem, one web master advocates
redirecting AVG scans back to AVG's site. "Many webmasters simply tell
LinkScanner to scan AVG's site instead, so their site gets marked as
malware free every time - while AVG gets handed the extra bandwidth
cost," says the webmaster of TheSilhouettes.org.
But this assumes that AVG is using a unique agent. And at the
moment, it's not. The send-it-back-to-AVG method may redirect
legitimate clicks as well.
Which gets to the heart of the matter: AVG's security philosophy is
fundamentally at odds with webmaster peace of mind. The company wants
to scan search results, and it wants to scan them in a way that's
difficult to distinguish from real traffic. "In order to detect the
really tricky - and by association, the most important - malicious
content, we need to look just like a browser driven by a human being,"
AVG chief of research Roger Thompson has told us.
And if that causes problems for webmasters, Thompson says, so be it.
"I don't want to sound flip about this, but if you want to make
omelets, you have to break some eggs."
Clearly, the company doesn't fully realize the importance of web
analytics. "Web analytics is about finding trends which can help online
marketers/webmasters improve things for their visitors and their
businesses," says Steve Jackson, co-chair of the International Web
Analytics Association. "It's a big part of the whole online ecosystem
in a fast growing up industry.
"No-one wants spyware or viruses, and AVG does provide a useful
service which is getting better all the time. I wish, however, they
would take business needs into account before launching software that
makes life even more difficult for the people trying to do the
analytics. Web analytics is not easy at the best of times, and this
kind of thing from AVG just compounded the problem.
"In order to make an omelet you have to crack some eggs. But a good
omelet has cheese, ham, peppers, mushrooms and all sorts of other
ingredients which AVG seem to have forgotten about."
But AVG continues to say it's working to solve the problem -
including the bandwidth issue. Referring to LinkScanner's new IE6-like
user agent, Thompson told us, "We intend to leave those in place until
we can find the right balance point which will allow us to continue to
provide the best possible protection for our customers, without
imposing too much extra bandwidth on websites."
AVG disguises fake traffic as IE6