POSTS
Problems with my Internet Connection - Part 2
See part 1 for a more detailed description of the problem I am experiencing with my internet connection.
Results
Let’s get straight to the results of my download speed measurements. The measurement script, raw data and visualization source code can be found in this github repo. The following graphs present what I’ve measured by downloading a mp4 from reddit and a video snippet from ard (hosted on akamai I believe). For the first one the data was downloaded from a server located in my home network:
Raw data points or the individual lines can be hidden with a click on the legend. This can help, especially for lower resolutions. You can use the tools on the right to zoom into a specific area of the chart. Hover over a data point to see detailed values.
Pretty astonishing results in my opinion. I didn’t think they would be this pronounced. Let’s ignore the absolute numbers for now. I would expect every major site to perform as ard does in the graph:
A pretty even average and some outliers here and there. The data for reddit is in the same vicinity in the morning but it absolutely crashes down each afternoon. Some afternoons/evenings, like on Friday the 15th, have a very stable
low average. The average is taken over 2 hours (12 measurements). To be specific: The average download speed from reddit is below 55KB/s for around 6 hours that day. No wonder even pictures have a hard time loading.
Now you might say: ‘Well yes, this must be a server side issue.’ That’s why I ran the same tests from a Hetzner server I rent, as explained in part 1. That server is also located in Germany, but the internet connection is quite different. Of course it’s a lot faster, but that’s not important for this test. I wanted to see if it had the same drops in download speed for reddit. Here are the results:
The reddit download speed is very stable. Meanwhile ard seems to have some available capacity especially between midnight and noon leading to even higher averages. But, again, we are not talking about the absolute max numbers
here. Both averages don’t go down very far for the complete measurement interval and only a few raw data points are low. Those are not very significant.
Discussion
I think the results above do a good job at showing that there really is an issue here and I’m not hallucinating. An average download speed of far below 100KB/s is awful and I hope this is somehow fixable. But what is the cause? I’m not sure, but everything I found so far leads me to believe that peering is the issue. But I wasn’t aware that the symptoms can be that bad.
I also found this post on net neutrality published by my ISP. It’s in German, but you might be able to translate it with one of the translation services available. That article also explains how ISPs should be interested in good peering performance and contracts, so maybe I’m in luck and they won’t ignore that issue.
Please let me know if you have an idea what’s going on or are experiencing similar issues. I’ll also contact my ISP to find out what they think and if they can fix it. This post will be updated with more results over the next days to cover more days and maybe some comments from the ISP or others. Cheers!
Updates
- Updated Data.
- You can scroll left in the visualizations to see more data than the last 7 days.
- Yesterday I contacted ISP’s support. Monday someone from the “expert team” will call me.
Some interesting comments on this article can be found on this german subreddit:
- User
mici012mentions peeringdb.com. Deutsche Telekom had one 20 GBit connection to DE-CIX for 13 million customers. o2, another DSL provider has two 400 Gbit connections for only 2M customers. - 1und1 has switched some customers to the versatel (DSL provider they bought) backbone. link
In a Telekom support thread concerning slow connections to reddit and imgur in the evening a Telekom employee shared the following statement yesterday (March 22nd 2019, translated by me):
“We are sorry for the caused inconvenience. Several content providers currently send huge amounts of data to customers in our network and use less appropriate channels for that. That leads to overloads. Together with these content providers we analyzed the causes and took first steps that should help improve the customer experience a bit. We work with the content providers to create additional capacity and relieve the exchange nodes of our network. Until additional capacity is created, it can occasionally come to further restrictions throughout the next weeks.”
I will probably add some more thoughts on this statement soon.
Using a vpn to route traffic through a server somewhere else should help eliminate the issues. This might be a functional workaround but is not a satisfactory solution in my opinion.
- Updated data. Tagesschau video file got taken down at 10am on the 23rd. I noticed that night and replaced with a different video snippet from ARD (hoping it will stay up a little longer). You can see when it was offline in both graphs. There is a long straight line. I normalized the key to ´ard´ to keep stuff simple. It’s also technically correct.
- I had a fun time with 1&1 customer support. More on that will probably come in a separate post. Today someone from the “expert team” finally called. He never read this blog post even though I linked it several times in the support process. He said my modem/router is the problem (he seemed to be quite convinced about that). I guess that’s just their procedure. I don’t believe the box has any issues. Next call on tuesday next week. Let’s see how this continues.
- Updated data.
- Installed the new fritzbox router last night. As expected: No change. That would’ve surprised me anyway. In the next phone call a 1&1 support employee will probably suggest to send a telekom technician. That probably won’t help, because it’s not the DSL connection itself, but the routing that’s the problem. If I’m not wrong at least. But I wasn’t able to keep him from wanting to send me a new box even though mine was fine, so I probably won’t be able to convince him that sending a technician to my house won’t make a difference. I had hoped that I could save both sites time and money by preparing measurements beforehand. But until now, no one from the support team has even looked at those. Frustrating…
- Updated data.
- Updated data.
- I was able to convince the support not to send me a technician. I got a measurement script from them that I ran a few times yesterday night and once this morning. In the results it’s pretty obvious that there’s a capacity issue (telekom -> telia -> fastly CDN) at night and everything is fine in the morning. There are high pings and lost packets at night. I sent those results (and, again, a link to this article) to 1und1 and hope an expert will have a look. Next call will be thursday april 11th.
- Updated data.
- Updated data.
- Last week support called. They didn’t see the mail with my measurement results. I also got a mail the same day with a notice that my DSL access will be modernized. No further info. The calling support agent didn’t know about this mail. She told me she will look for my mail and then let me know when she will call next. Nothing since then.
- Updated data. There is some missing data last week, because my home server was off for a while after an electricity outage.
- I had a frustrating call with 1&1 support again. Everyone I talked to until now didn’t really understand the situation. I’ll wait if the DSL modernization in two days will change anything.
- A few minutes ago some positive news started to come at this Telekom support thread. Many users are reporting full speed access to reddit and other fastly hosted sites. Let’s see if it stays this way.
- Updated data.
- After telekom changed their routing I couldn’t measure any more problems. But I didn’t have a lot of time to measure because …
- … yesterday morning 1und1 switched me over to the versatel backbone. Since then, no issues anymore. This will probably be the last update of this article.