When it comes to internet access, we all want speed. But measuring speed, and, indeed, other indicators of ISP performance, can be fraught. It is also controversial. 

Wellington-based TrueNet has been measuring Kiwi ISPs’ performance since 2010.

The company, which has a contract with the Commerce Commission to provide its data, was formed by telecommunications consultant and former engineering senior executive John Butt. 

Butt teamed up with software company Catalyst, which took a stake in the company in return for writing the software needed. To date, TrueNet has completed more than 450 million tests. 

The company uses probes for its testing. These probes are standard Mikrotik routers with TrueNet software installed on them to run the tests. 

The probes are installed in the homes of more than 400 volunteer panellists, chosen on the basis of their geographic location, technology and ISP. 

While there are almost 500 physical probes in use, Butt says these now test three technologies for broadband including fibre, copper and fixed wireless or mobile.  

Butt says that per head of population, TrueNet’s tests use more probes than any other test organisation globally. He says the tests are “very standard tests on the performance of the internet” and are carried out every hour. They test for speed, latency and jitter. 

A detection test to check which ISP is being used is also run. Panellists must use TrueNet’s wi-fi and router rather than their own, to ensure all traffic goes through the probe. The detection test reveals when another ISP is being used and means TrueNet can stop testing. Panellists may be using another wi-fi service to, for example, download streaming video.

“It’s a really important part of our testing because we don’t want our end results to be biased or faulty because the panellist is using another service and router, then we’re not testing the line, we’re testing what is left of the line.” 

Equally, Butt says TrueNet doesn’t want panellists’ internet experience to be degraded by the testing. To encourage panellists to use the probe’s wi-fi, TrueNet provides double reach wi-fi, which, Butt says, is “much better than what most people have at home”. 

Data is sent back to TrueNet as the tests are done. It’s the only means of communication the probe has. “The probe can’t talk to anyone. It only has one address it can talk to and that’s our server. And we can’t talk to it either,” says Butt.  

So secure is the system that even TrueNet’s engineer can’t contact the probes. Any updates needed can only be done when the probe contacts TrueNet’s server. 

However, the tests are only part of the equation.  

At the end of the month, TrueNet downloads all the data gathered into a pivot table and the process of cleansing begins. With the average probe averaging 720 downloads a month, from Auckland, and also 720 a month from Wellington for the throughput checks alone, there’s a lot of data to check. And it’s all done manually. 

“It would be very difficult to put an algorithm together to do it,” says Butt of cleansing the data. Instead, he relies on an engineer who analyses the tables, looking for patterns and any data that looks suspicious and was likely caused by something that was not the ISP’s fault. 

Volunteers who effectively compete with the test, by not using TrueNet’s wi-fi, for example, can skew results. 

The company uses median per hour results, with a probe returning 30 or 31 results per

month, for 1pm, say, depending on the number of days in the month. The median result is drawn from this. Tests are conducted every hour the probe is turned on. If the sample size from a probe is less than five for any hour across the month – because the probe has only been turned on for the last few days of the month, for example – the probe’s data for that month is excluded. 

A second, more thorough cleansing follows before the data is put into Datawrapper, a software tool that creates charts. 

Butt is scathing of other testing companies, saying only one other has “any legitimacy to its testing”. He says most tests use software on users’ computers, or downloads from a server, and have no idea what connection is being used. 

“You don’t know anything about the remote end, and the results reflect that,” he says. This inevitably leads to inaccuracy, adds Butt.

“Worldwide, all the regulators who do testing like the Commerce Commission does, use our technique, where they have a probe in someone’s house that is wired to the gateway.

“It is the only reasonable way to test, because then we know where our probes are, the connection we’ve got and the ISP being used.” 


* The Commerce Commission declined to comment for this article because it is considering tenders from TrueNet and other firms who are competing to monitor ISP’s performance. 


Wellington-based TrueNet has been measuring Kiwi ISPs' performance since 2010.