Windstream's Downstream Throughput


The main section below describes the abysmal situation that prompted the creation of this page in 2010. See the updates for a history, and see how it looked when we started (for how it can always be worse).
This is Windstream's downstream throughput (in Mbps, with a linear regression line) in Arroyo del Agua, New Mexico (87012). At the lowest tier in 2010 they promised 3 Mbps; the highest was 12; I'm now at 25. Here's what they've delivered over the years. The blue grid line is the nominal rate; the regression line is red.
When I subscribed (I had no choice; they're the local telco monopoly and this valley has no cable), I wanted the fastest tier, but I decided to try the lowest tier first—to see if they could deliver on that—before paying for more. They FAILED! I've compared numerous data points with my neighbors, and if you're paying for 12 Mbps, you're really getting burned: higher tiers seem to fare no better than lower tiers at peak times.

J'Accuse!

I believe the cause of this is that their backhaul is radically oversubscribed but they're too cheap and greedy to invest in the infrastructure necessary to provide their customers what they promise in their—essentially fraudulent—sales literature, where I don't remember seeing anything like this:
Expect actual broadband only when you're asleep or at work. When you're home and awake expect something more like dialup. You remember dialup don't you? Welcome back!
In networking, for high quality throughput you must build to peak demand, or at least devise a combination of infrastructure and compensating strategies that adequately approximates that ideal. Windstream doesn't even come close. The problem is that if your ISP does not care to give you what they promise, and is disinclined to be honest about it, they can usually dismiss speed test data as anecdotal or blame it on internal wiring. Before dealing with Windstream, I didn't think a large telecom would have a systematic policy of trying to deceive its customers, but if you call 'tech support', you'll get an amazing runaround; you'll be told to unplug things and restart things by people who obviously have no idea what they're doing, and assume you're too clueless to notice. You might be told that from 'there' your speed is fine, although if they are actually testing something and not just faking it, their test packets must be getting a much better QoS priority than their customers'. A sign that they might actually be testing something is that increasingly I got "Hmmm, yeah, that's not good..." A sign that they're faking is that I've been told to unplug my ethernet cable and then been told "Ah see, the speed JUMPED up!" while watching wireshark across the NIC showing that NO packets were going ANYWHERE. You might be told that they'll send someone out, even though you say that's a waste of a field tech's time because there is nothing to fix. You can point out that your gateway's diagnostics report the Signal/Noise Margin and Attenuation between you and the DSLAM are 25.1dB and 5.8dB respectively, that these are excellent numbers, and that MLab's NDT probe always produces variations on the following. You can point out that all of this testing is moot, because your wiring does not have emotional mood swings that cause radical fluctuations in throughput. When probed, the network equipment claims to be OC-48, which has nominal bandwidth of ~2.35 Gbps. Back in August 2009, when I first signed up, the throughput was fine for a few days; then for a little while it seemed to be reasonable during the day, sometimes peaking above 3 Mbps, becoming really bad only in the evening; by March of 2010 it was horrible all day long, with occasional anomalous peaks at about 80% of what was promised, prompting me to create this page. See updates for what transpired subsequently. If you ask, you will not be told their oversubscription rate. Apparently Windstream thinks it is more cost-effective to cheat and lie to their customers, and hire legions of untrained people to pretend they are tech support than to replace their defective switches or slide some more fiber down that tube. I have fruitlessly argued all of these points with those legions. Here therefore are my data. I challenge Windstream to offer an alternative explanation to mine.

Design Notes

To produce this page, I wrote a Java client/server socket pair that attempts to be as 'generous' as possible to the connection. Its results tend to be about equal to Ookla's, and much more generous than 2wire's. It doesn't include initial latency and socket set-up in its calculations; it only starts the clock after the first (8K) test packet is received, not sent. Its throughput on localhost is ~120 Mbps, on a slow (1.4 GHz) single-core Intel P4, and since the load is normally split between the client and the server, we can pretty safely ignore CPU load as a significant contributor to slow throughput in the sub-3 Mbps range (and indeed CPU usage is empirically negligible). The client runs as a Linux cron job or a Windows scheduled task, but can be invoked manually any time. The server runs on a low-load backbone-connected Linux server. Each run produces a log entry on the client that is then sent to the server. Originally the server's logs were rendered out to this page as PNGs by a PHP page that invoked Google's pretty sweet Image Charts API, but in typical Google fashion they killed that API on March 18, 2019, so now I render the data on the client with javascript and canvas.

Updates

100313.2330 - Added a regression line. We seem to be trending upward! Let's see if it's just the weekend...

100314.1207 - Flattened out.

100316.0100 - Changed the auto-check interval to every hour, then took a few hours off.

100316.2250 - Added the ability to split charts and scroll through them (look for side arrows).

100320.1714 - Today, for the first time, the single-chart mean is above 1 Mbps.

100322.1204 - The client was down for 3 hours last night, so Windstream got a free pass at what looks like a time of abysmal performance.

100326.1538 - Got an email from Mollie at Windstream, offering help...

100329.1640 - ...I said great, as long as it's a fix and not just preferential treatment for my packets.

100330.1400 - Got a call from a 'level 2' person at Windstream; he said it was a 'known issue' that they were working on.

100410.0021 - It still all looks pretty much the same to me.

100420.1501 - Notice the big data gap on 4/18? Lines went completely dead throughout the region at about 00:45 on the 18th. Interestingly, DSL was back up before voice, about 9 hours later at 10:00. Local-only voice was back by 22:00. Long distance was up the next morning. I hope nobody in the area needed 911 for those 21 hours. BTW, the mean is back solidly below 1 Mps, and that is ignoring the dead time on the 18th, which, strictly speaking, should have counted as 9 zeros.

100507.1632 - Well...it's been over a month since Windstream offered to help me with 'my' problem, and nothing has changed. I guess we can safely conclude that they have no answer to my challenge. Have you noticed that every bill is higher than the last? This is the first time I have ever seen a utility do this: inexorably raise the bill every single month, without delivering what they promise, and without explanation. Perhaps it's time to turn over my results to the FTC or PUC, or whoever else is responsible for protecting citizens from this sort of behavior.

100521.1407 - Starting yesterday morning we look fixed! Is it time to retire this page? I'll let the test run a little longer, then, as I promised Mollie on 100329, run it from a neighbor's. If the fix looks real, we might be done! I feel like Ralph Nader or Upton Sinclair. :)

110305.2111 - My neighbor's results were always suspicious-looking, as if the fix had not been real, as if Windstream had somehow been giving special priority to my packets. Now—suddenly—it looks as if Windstream has decided to stop doing that, so we're back! Here, for comparison, are my results in May 2010, when 'the fix was in'—apparently just to shut me up.

110724.1931 - I was in Seattle for few weeks, and when I came back on July 6 Windstream was completely down for all but 5 or so of the next 72 hours. Their downstream throughput seems to have improved though. We'll see...

120227.2320 - That big gap starting 120206 wasn't Windstream's fault; I was away on the beach!

120626.2120 - The gap at the end of May WAS Windstream's fault: four whole days completely down!

131215.1800 - After being reasonable (though not great) for a while, peak-time throughput has degenerated radically down to nearly 0 since December 2, 2013.

140623.0031 - After a few good months, peak-time perfomance started to degrade in May, and now it's complete garbage. With a nominal speed of 12 Mbps, peak-time throughput is often less than 1 Mbps. Normally throughput starts to recover at about midnight, but at midnight tonight it was barely 0.5 Mbps with no load from me.

140705.1507 - Worse than I could possibly have imagined! From about 7:30 in the morning until about 2:30 the next morning every day, with nominal 12 Mbps and approximately no load from me, throughput is around 1 Mbps with lows approaching 0.

140813.1942 - Windstream has a new record! I've had some form of online connection pretty much continuously since the mid 1980s. Windstream had set the record for the single longest outage ever on 100418 (9 hours), but they now have blown past that record DRAMATICALLY! See that blank spot from 140806.0300 to 140811.1300? That's a 130-hour outage!!! Now, life in the country without TV or radio or news or the ability to make a phone call is not without its charms, and our job here is to measure speed, not availability, so we ignore connection failures even if they are Windstream's fault, but if you think 130-hour outages in the 21st Century are unacceptable, you might want to look elsewhere for an ISP.

180225.2108 - Wow, it's been a long time since I've felt inspired to update this page, but something nefarious is afoot. After years of reasonably solid 12 Mbps downstream (we won't talk about the miserable 600 Kbps upstream), Windstream upgraded my neighborhood to 25 Mbps in June of last year. So for the last eight months I've almost had real —if horribly overpriced— broadband, with one weird thing: the trend line was usually somewhere in the mid-20s, but the actual throughput was clustered in two bands about 5 Mbps above and below the trend line (see the charts of the past several months). Then suddenly, starting Feb. 12, performance plunged. Now the trend line is about where it was before the upgrade, and the consistency is much worse. But here's the nefarious part. If I go to speedtest.net, it says my downstream speed is ~50 Mbps, which is a straight-out lie. It's clear that Windstream is gaming speedtest.net (and probably other well-known speed testers). How do I know it's not bad output from the server, you ask? I have another test that pulls a 9 MB tiff from hubblesite, and the same tiff from another server. When I run that test on a reasonably well-connected machine I get something like: THROUGHPUT: 200.22427662071 Mbps. When I run it through my Windstream connection I get something more like: THROUGHPUT: 15.024178056315 Mbps. The range so far has always been between 5 and 20 Mbps, with an average of about 14 —which exactly matches recent charts.

180314.1449 - Recent problems seem to be resolved.

190318.0000 - Google just killed the Image Charts API.

◊ contact me ◊