When consumers have a greater handle on broadband performance they can make better choices. The Commerce Commission wants this data so it can check service providers are delivering on their promises. Sam Crawford – the Sam behind SamKnows – explains the broadband measuring role his company plays to Bill Bennett

Many organisations monitor New Zealand broadband speeds. They get data in different ways. Google, Akamai (the content delivery network services’ provider) and even Netflix produce reports based on the activity they see on their servers. Speed test company Ookla produces an aggregate number based on data collected by its ‘speedtest’ website.

While these numbers have their uses, they don’t provide a rigorous, timely and independent report of line speeds. Consumers, businesses and the regulator would find it hard to use that data to prove New Zealanders are, or are not, getting the services they are paying for.

Enter SamKnows. In 2018, the Commerce Commission appointed the UK-based company to run its Measuring Broadband New Zealand programme. SamKnows measures the performance of service providers, broadband plans and communications technologies. It provides regular performance reports that are posted for all to see on the Measuring Broadband New Zealand website.

SamKnows works with many telecommunications regulators around the world. Customers include Ofcom in the UK and the Australian Communications and Media Authority. SamKnows CEO Sam Crawford refers to this as the company’s “government work”. He says SamKnows also works with telcos and ISPs (Internet Service Providers). The split between government and commercial work is roughly 50:50.

“We have government clients in Canada, the UK, parts of Europe, Hong Kong, Singapore and Saudi Arabia, as well as New Zealand and Australia. We tend to work with a couple of ISPs in each market. Sometimes more than a couple. The vast majority of ISPs in the US are customers.”

Crawford says SamKnows researched the market before coming to New Zealand, so it knew what to expect. Even so, he says, one thing jumped out in front of him when the company started collecting New Zealand lines’ data.

“I was very impressed by the headline access speeds. There are plenty of people in New Zealand running at 1Gbps. It seems commonplace. Your nearest neighbours, the Australians, are typically running at one tenth of your speeds.

“In general, access speeds are very good, in part because there has been fibre penetration for some time now. After I visited New Zealand, I realised how impressive it was reaching that level of penetration, although I understand there are rural areas that are not served.”

LOCAL TRAFFIC ROUTED OVERSEAS

One issue that emerged from the early data SamKnows collected was a connectivity or peering problem with one of the ISPs.

He says: “As a result, a lot of traffic was going overseas before returning to Auckland or Wellington. This looks very strange coming from Europe where this kind of thing simply would not happen. Peering inconsistencies seem alien. We [Europe] are a small, densely connected market and everyone just peers openly with one another, and it’s at a low cost.

“I can’t thing of the last time we had a situation in the UK where, as a matter of policy, traffic was routed to the US before coming back. We have seen it in Singapore and in parts of Australia.”

He says another surprise was the Commerce Commission’s openness to innovation. “They were good and inventive with some of their advertising to get people to sign up as volunteer panellists. We’ve not worked with any other regulators who have done things like that.”  

Panellists are the volunteers who attach a SamKnows white box to their home or business broadband line. The aim is to get 3000 volunteers and the company is still recruiting.

Crawford says: “We have people on a whole variety of RSPs (Retail Service Providers). We don’t always have a sophisticated enough statistical sample to report on all of them. We can’t report the data on a company in good faith if we only have one or two of their users.”

A medium-sized New Zealand service provider might have 15,000 to 20,000 customers on its fibre plans. Crawford says that isn’t a lot when you are trying to recruit people to install the measuring hardware. “You need maybe a couple of hundred people to get the data. It might only be point five percent of the company’s customer base. But getting to recruit one in a thousand or even one in a couple of hundred is a tall ask.”

This becomes more of an issue when it comes to breaking down customers into different technologies or plan speeds.

This is why Crawford is keen to recruit more volunteers. He says that he’d like to, say, compare the broadband performance of Spark customers using ADSL (Asymmetric Digital Subscriber Line) in rural parts of the North Island with customers using the same service elsewhere. (ADSL is the slower broadband service delivered through the standard copper telecoms network.)

He says: “If we don’t meet our minimum sample size, we can’t do it. The guide size is 45 users for every strand that we want to report on. If, say, we don’t have 45 users on Spark ADSL in rural parts of the North Island we can’t report. Forty-three users is not good enough.”

Crawford says this can get frustrating, but he says that it is important SamKnows’ reports are reliable and defensible. “We don’t want to end up in a shouting match with any of the ISPs, but, if we do, we want to make sure we’re on the right side of the argument.

“We’d love to break out numbers for HFC (Hybrid Fibre Coaxial cable). We’d also like to break out fixed wireless broadband by Skinny, Vodafone and so on, but, frankly, we need more samples to do that. (HFC technology combines optical fibre and coaxial cable.)

“We’re on a recruitment drive ahead of the next report and we are trying to reach as many people as possible. There’s a huge stock of white boxes waiting to go out; we need people to sign up and say they are interested.”

The SamKnows story

Sam Crawford started his business 10 years ago. Not long before, BT announced a programme to roll out broadband in the UK. This was done on a demand basis. If enough people in an area wanted the service, BT would build it there.

Crawford got involved in collecting names in his area and developed tools for automating the registration process. He used a website name given to him as a gift: SamKnows.

He was involved in tracking the availability and roll out of fast broadband services. As this happened, services emerged offering ‘free’ broadband, the catch was there were no guarantees about the service quality. ISPs were using traffic shaping and customers were naive about the market.

Crawford started measuring performance using the first prototype of the device that evolved into today’s white box. He had limited resources but contacted the UK telecommunications regulator to see if there was any interest in the project. There was.

Today, SamKnows has about 40 employees. Most are based in London. Crawford says two-thirds of the employees work on the technical side of the business. They might be hardware engineers, C++ programmers, mobile or web developers. There are support staff and people who look after infrastructure and support systems. The other third of SamKnows’ employees are account managers along with a couple of sales staff.