Take charge of your routing — or how I got rid of buffering issues with TIDAL
Circumvent intentional Internet Service Provider bottlenecks by using a VPN
Once upon a time, a naive individual signed up for a high-res music streaming service. But soon he discovered, after his supply of musical dose kept being interrupted regularly, that it was not as easy as signing up for a streaming service and living happily ever after. The music kept buffering, cutting out in the middle of songs or just did not start at all. This is the technological deep-dive of that individual after he discovered that this is an Internet Service Provider problem and not one of the streaming service.
So, how did it all begin? I use TIDAL for my music streaming needs, mainly because of their high-resolution offering. Thus I stream to my HIFI system at home via Roon (an in-home music collection and streaming system) which integrates nicely with TIDAL. On my phone when I am out and about, I use the native TIDAL-App.
Somewhere at the end of the last year, the music played via Roon kept stuttering and cutting out/skipping whole tracks when streaming at home.
Naturally I was not very amused about my listening experience being interrupted, so I tried the App on my phone. The result was almost the same except the App on my phone did not skip songs but the stuttering was equally bad. Another thing was songs not starting at all sometimes, instead showing the dreaded, never-ending buffering animation.
I blamed the service for having not enough capacity available (very verbally of course. Plus it was a weekend, so this theory made some sense to me).
Then, when I went to work and left the WiFi range for my commute to work, the phone switched to the mobile carrier and all issues were gone. The same at work, which made me think: This had to be an ISP issue!
Research: The story of greed and peering
Focusing my googling wizardry on the ISP issues I eventually stumbled about something: Hideous peering practices of my ISP (German telecom, better known for product lines T-Online or T-Mobile). The article by hosting company Hetzner describes the phenomenon as follows, dubbing it “Double paid traffic”:
Currently, about two-thirds of our network traffic is directly exchanged with cost-neutral peerings of various partners. Among them are private peerings with large network operators.
For this reason, we have observed with growing concerns that some DSL and cable operators with leading market positions in Germany, the USA and also in Asia, do not have open peering policies, but on the other hand also do not have not having sufficient capacity to other Tier 1 carriers to ensure a fast and smooth access to the global Internet for their customers.
For high-performance internet access for customers, there needs to be sufficient capacities to all other Tier 1 carriers. Further network expansion is then traditionally problem-free, as long as both Tier 1 carries are willing, at no charge, to take more peering with each other. Once one of the parties tries to charge money from other carriers for further network expansion however, such discussions often fail and the much-needed network expansion does not proceed.
Often the DSL providers will argue that they don’t want to bear the costs of expanding network capacity on their own.
There are a few things of note here:
First, let’s clarify what peering means in the context of an ISP. Peering is the voluntary process of exchanging data between two networks, namely that of the ISP and that of another Company (datacenter, other ISP, global carriers etc.)
Second, there are two types of peering:
Open or public peering refers to the practice of exchanging data at a location open to everyone and every network operator is responsible for getting their data to that exchange point. The exchange itself is done without charge (“settlement free”), everybody profits from their own customers and not the exchange. Think of it as a group of people voluntarily meeting in a cafe to speak with each other. Everybody is in charge of getting to the meeting and everyone orders their own coffee.
Private peering on the other hand is the practice of exchanging data in a private location. This can be in the interest of both networks and they agree to not charge for the peering or, and this is the important part, one network provider sees the opportunity to make some quick bucks and lets themself be payed for bringing their data to the exchange — think of it like two people, Person A and B, meeting each other in another cafe because Person B wants to exchange important documents or information (unrelated to both people) with Person A, but for Person A to agree to meet with Person B, he or she makes Person B first agree to pay for transport to the cafe and to pay for the coffee.
As you might already think, the private peering is causing us the issue. To quote the article mentioned before again (DPC stands for “Double paid carrier”):
The customers of the DPCs experience in practice access to the Internet, which is divided into two categories. On the one hand there is content which they can access quickly and with good performance. This content comes from those content providers who pay the DPC for a direct network access to their customers.
On the other side is the rest of the Internet content. In particular between 19 and 22 o’clock the concerned DSL and cable operators hit their capacity limits and therefore prevent the fluid and necessary exchange of data between the individual networks. This always leads to complaints from customers of these providers because their servers are not accessible with the usual performance they are used to from us.
In Essence: This is the reason net neutrality is important. Double paid content essentially creates an internet consisting of two classes: The ones who get to enjoy the paved highway and the others who are left stuck in the mud (like my music).
Furthermore, it is not a phenomenon of one ISP in Germany. There are more examples of this, take a look at this Article on Quartz about Netflix paying Comcast in the U.S. or this one on ArsTechnica and I am certain that with some further digging, you will find more.
Action: Take charge of your routing!
So, now that we know that my ISP (which by the way already charges the highest amount on the German market — yay, monopoly!) wants to be paid not only by me but by the content provider as well, what can be done to reduce the impact so we don’t get stuck in the mud when streaming media?
Well, we could change the Internet Service Provider obviously. But that is not an option in my case, as there is simply no other Provider that supplies the same speeds in my area.
Another thing would be endlessly complaining to TIDAL support hoping they get their CDN provider to pay my ISP. I doubt this solution will get me very far.
Then I came up with the third solution — what if I could route my traffic over the “paved highway” and then back just before the finish line?
That is what I ended up doing, by first determining the CDN provider TIDAL uses and then routing only the traffic to that CDN over a VPN. Sounds complicated? With a bit of light reading it is surprisingly easy.
Determining the CDN provider
To get anywhere to start with, I first needed to know which CDN my music is sent from. I accomplished this by looking at where the TIDAL web player gets the music from using my Browsers Developer Tools. With Google Chrome this is as easy as playing a song using the web player and then right-clicking on for example the play button and then choosing “Inspect this element” in the context menu.
A new area will pop up exposing various developer tools. We are interested in the “Network”-Tab.
After switching to the network tab, we can see all incoming resources for this web page.
You will notice that there is a certain type of entry that will repeat over and over again — a file ending with “.mp4”. These are packages containing chunks of the music currently playing.
To find out about the hostname of the server sending these packages to you, hover over on of the packages and a tooltip will tell you. Mine is sp-ad-fa.audio.tidal.com.
Now that we know the hostname the TIDAL-App receives the music from, we can find out which CDN provider is being used or if one is used at all. To do that, just open up a command prompt on your operating system, such as CMD on Windows or a Terminal on Mac/Linux, and type "nslookup <hostname>" (replacing <hostname> with the result from earlier of course). Hit Enter and you should see the hostname being resolved.
From the result of the nslookup-Command we can see the IP-Address of the server sending the audio files as well as the actual real hostname of that server which is different than what we entered (probably a load-balancing thing or just an alias). This is convenient as we do not have to find out where the IP-Address belongs to, we can already see from the canonical name that TIDAL seems to use the Fastly CDN for my region.
Determining the CDNs IP-Addresses
We know which IP-Address serves the music and that those IP-Address belongs to the Fastly CDN. But who guaranties that the IP does not change intermittendly? Therefore let's see if Fastly provides us with their possible Addresses. A quick Google search for "Fastly CDN ip addresses" reveals that they do for Firewall whitelisting purposes, in fact they even have an API for it so you can grab them programmatically. Neat!
All that is left to do is set up our router to establish a VPN connection and route those IP-Address ranges through it — allowing hopefully all devices in the network to bypass the purposely installed ISP-bottleneck.
Setting up the VPN and routing on a Ubiquiti USG
My router is a Ubiquiti Unifi Security Gateway so obviously the following only applies to devices using the Unifi Controller (but there is probably a method for your router as well if you ask Mr. Google for it).
Start by going into the settings using the gear icon in the bottom left.
Then click on the "Networks"-Tab and choose "Create a new network". You will be greeted with a blank form for a new network definition. Fill it as follows:
- I named my new network "Fastly CDN over PIA" as the purpose is to tunnel traffic to the Fastly CDN via Private Internet Access, the VPN Provider I am using.
- Next set "Purpose" to "VPN-Client" set the checkbox for "Enabled".
- Important: Leave "Default route" unchecked or ALL of your internet traffic will go through the VPN which is probably not what you want for your home network (Speed, latency!).
- For "Route distance" enter the number 5 into the input field
- Now click the little Plus-Icon next to "Remote subnet".
Enter the subnet containing the IP-Address we got from Fastly earlier, in my case, for the IP-Address "126.96.36.199", it is "188.8.131.52/16".
- In the category "DNS" check "Use VPN server-provided DNS servers"
- For "Server" I entered one of the nearest VPN servers (in my case denmark.privateinternetaccess.com or nl.privateinternetaccess.com)
- As this is a PPTP tunnel, I generated the "PPTP/L2TP/SOCKS Username and Password" on the PIA site and entered them into the username and password fields
- Finally check the option "Require Microsoft Point-to-Point Encryption" and click save to submit the form
That's it! The USG will pickup the changes from the controller and go ahead and provision itself with the new settings. In a minute or two the changes should be live and the VPN connection established.
Conclusion: Result & Thoughts
Let's have a look if this did what we wanted. First I am going to check, if the routing actually changes with the VPN enabled by using the "traceroute" command. This will show the path network packets take to the destination.
This is the traceroute output without the VPN connection:
And this is after the VPN connection has been established, traffic being redirected to the VPN server first and then masked:
We can see all paths between the nodes the packages travels along are split between two providers with an exchange point in the middle, my ISP (ending in DTAG.DE) and Telia (the second provider).
Interestingly enough, now that my music plays without a hitch using a VPN server in Amsterdam, the Fastly CDN IP also traces back to a location in Amsterdam. Would it not be ridiculous if they would be in the same datacenter? This would in accordance with the research in the beginning of this article indicate that the CDN IP is specifically rate-limited or throttled somewhere in the ISP chain!
If you made it all this way to the end — thank you for reading and I hope you learnt something about networks and how policies of ISPs affect us end-users. In case you had a similar experience, leave a comment down below and share your story!