An Analysis of the Triple M Brisbane Playlist

When listening to radio in the car I do exactly what I would at home, channel surf.  Generally I listen to Triple J but also switch between Triple M, 4ZZZ, and 612 ABC for something different.  For a couple of months there was a period when every time I switched over to Triple M it seemed that they were playing The Spin Doctors.  Now any normal person would pass that off as mere coincidence and leave it there, but if you've read this blog before you're probably aware that I'm not normal.  I started to wonder exactly what type of songs were played on the station, did they play certain music at different times of the day to cater to a different demographic?  More importantly how often do they actually play The Spin Doctors?  To answer this I needed to log what songs were played, unsurprisingly I chose a solution that required electronics to log the RDS messages the station broadcasts.  The method used was described in a previous post, Using a Raspberry Pi to Log Songs Played on a Radio Station.  They do list recently played songs in their on-line streaming player, but I couldn't figure out a way to easily log them.  Besides, electronics is more fun.

I've since discovered a similar analysis was done by Daniel Nitsche in 2014.

The logger was left to run for two weeks, generating a file with 273902 entries, on average logging what was being played every 4-5 seconds.  Writing software to turn that into something useful was in my opinion going to take longer to do than editing the file by hand as there were intelligent decision to make along the way.  There were small errors that needed correcting as well.  For example, when the daytime presenters played a snippet of the songs to be aired in the next hour, that was also broadcast on the RDS message system.  So it would appear they played the song and then played it again within the hour.  That had to be corrected.  I'm not saying this couldn't be automated, but for a one off, it's not worth it.

After editing, the data showed that there were 2805 songs played over the two week period, about 8 songs an hour.  That sounds about right.  There's probably the occasional error in my data set, but I believe it's accurate enough to show any trends.  There may also be some errors in the release year I've listed for the songs.  Initially I tried to use an on-line music database to get the release date for each song, but that was way to slow and error prone.  In the end I Googled each of the 868 songs to find out when they were released (doesn't take as long as you think, I averaged about 6 a minute).  Then comes the problem of when a song was actually released.  Some songs are released on the album but their release date is when the single comes out.  I choose when they were first publicly available.

So what can we actually learn from this data?  I started with the histogram below to see the distribution of songs played.  The oldest song played was "Sympathy for the Devil" by the Rolling Stones from 1968.  Then there's a  bit of a peak in 1971 before the bulk of songs from about 1980 to 2000.  1991 seems to be a massive year for music and unsurprisingly aligns closely with the active rock format the Triple M subscribes to.  Billboard has a go at explaining 1991.  The bulk of the songs played in 1991 are from the following classic albums.

U2 - Achtung Baby

Red Hot Chili Peppers - Blood Sugar Sex Magik

R.E.M. - Out Of Time

Pearl Jam - Ten

Nirvana - Nevermind

Metallica - Metallica

Guns N' Roses - Use You Illusion I

Guns N' Roses - Use You Illusion II

The next thing to notice is that there aren't many songs from the 2000's, but they do play a lot of recent music from the last 5 years.

Yearly Distribution of Songs Played on Triple M Brisbane

The next plot shows how the release years of songs are distributed over the two week period.  Although you can see the information from the histogram reflected here, there isn't much too much else to see here.

Release Year of Music Played on Triple M Brisbane

The last graph was a bit boring, but by taking the data and showing it differently we can learn some interesting things in the one below.  By plotting all the weekday data on one daily graph, patterns begin to emerge.  For instance, between the hours of 5 and 9 am it appears that there is a blackout on pre 1980 music.  You can also see that songs are less frequently played between 6 and 9 am when the breakfast crew are doing their thing.

When the drive show comes on from 4 to 6 pm you can once again see the drop off in the amount of music played to make way for the presenter.  Between 6 and 7 pm there is a sports show that plays relatively little music as well.  Then between 7 and 9:30 pm the focus seems to be on 80's music, this then changes to 90's music until about midnight.  I assume they have data that says people who like 80's music are in bed by 9:30 pm.

Weekday Daily Distribution of Songs Played on Triple M Brisbane

The data for the weekend shows similar patterns.  After lunch, the sports coverage starts and the music stops.  There's also a relatively dense block of new music between about 6:30 and 8:30 at night, this turns out to be a show on Sunday that plays a lot of new music.

Weekend Daily Distribution of Songs Played on Triple M Brisbane

I also did up a Poincaré plot to see if any other patterns emerged, but the only thing obvious is the predominance of 1991.  If you've never seen one of these before it's a graph of data that plots data point n against (n+1).  They're very useful for finding patterns in binary files as well.

Poincaré plot - Release Year of Songs Played on Triple M Brisbane

So to wrap things up I looked at what the most popular artists were.

Artist	Times Played
Red Hot Chili Peppers	76
INXS	69
Foo Fighters	66
U2	65
Pearl Jam	61
R E M	56
Midnight Oil	55
AC/DC	44
Guns N' Roses	40
Powderfinger	39

I also looked at the most popular songs too.

Song	Artist	Times Played
Georgia	Vance Joy	26
Hold Back The River	James Bray	19
Congregation	Foo Fighters	17
Blame It On Me	George Ezra	17
Every Breaking Wave	U2	17
Crystals	Of Monsters and Men	16
Someone New	Hozier	15
Believe	Mumford & Sons	15
Two Princes	Spin Doctors	14

Ha!!!!!  Proof I'm not insane*.  Two Princes by Spin Doctors is the 9th most played song on Triple M.  Coincidentally released in 1991.

*This blog post probably doesn't help my case.  Hmmm :-/

You can find my original logs here.

The Effect Of Heavy Rain On RDS Message Reception

In a recent post I demonstrated how to log RDS messages from FM radio stations.  As a follow up, I thought I'd share an interesting result I observed while logging data over a two week period.  I set up the equipment as described, and for most of the logging period everything worked as planned.  The receiver decoded around 13 messages per minute, sometimes more, sometimes less, but most importantly the messages were almost error free.

On the last day of logging I checked the progress and noticed that the messages contained a lot of errors, not only that, the number of messages received per minute had dropped to one or two.  I'd like to build suspense, but I knew what was wrong immediately, we were experiencing some of the heaviest rain Brisbane had seen in years.  The rain was degrading the signal from the radio station to a point that still allowed audio to be heard, but prevented decoding of RDS messages.

I thought it'd be cool to create a visual aid to compare historical weather data to the logged RDS messages.  Radar data from www.theweatherchaser.com was added to a graph showing how many messages were received in each 6 minute period over the two week long experiment.  6 minutes was chosen as this is the length of time between radar frames.  The frames for the animation were created using python and matplotlib.

VirtualDub was then used to convert the image sequence into a video.

I wouldn't say that there's a strong correlation between the radar images and the frequency of the received messages.  It's important to remember that the radar only shows rainfall.  There could be cloud cover or lighting strikes effecting the radio transmission that doesn't appear on the radar.  The number Hopefully it's obvious that this isn't a tightly controlled experiment, just a bit of fun. 

The file used to create the above animation can be found here.

si470x_log_processor.py

It can't be used to recreate the animation as the source data is missing.  When learning how to use matplotlib I had trouble finding out how to do certain things.  I though placing the file here might help someone in same situation in the future.

Using a Raspberry Pi to Log Songs Played on a Radio Station

I recently came up with an idea that required me to build a device to log the songs played on a radio station.  There are a few ways to go about this, you could scrape the data from their streaming radio service, you could use something like Shazam to identify the songs and record them, or you could use the RDS data encoded in the radio signal.  I decided to go the RDS way.

Nearly everyone is familiar with the RDS data stream even if they don't know it.  Newer car stereos that display the current song playing get this from the RDS data.  The FM radio station is first demodulated.  In this demodulated signal at 57 kHz there is double side-band suppressed carrier that contains the RDS data.

The initial method I chose to receive this signal didn't work.  I planned to use a software defined radio approach using an RTL-SDR dongle and a Raspberry Pi, but I just couldn't get it to work.  After some Googling I came up with Plan B.  Use an FM radio receiver IC that contained an RDS decoder and connect that to a Raspberry Pi.  Luckily, Sparkfun had a breakout board for the si470x that was perfect for the job.

I didn't want this to be a massive project so I grabbed publicly available code and modified it for my purposes.  The code was tested until it could run for long periods without crashing.  There's no exception handling, I just reduced the errors until the system was stable.  I could have spent another couple days making it "perfect", but what's the point, this isn't a product or a permanent installation, it has to run for a week or so and give me usable data, and it's doing that.  Currently at around 45 hours.

The code for this project is available from github.  I urge you to have a read of the README file there it contains more information about the code.

si470x-RDS_Logger on GitHub

Based on code from

KansasCoder https://www.raspberrypi.org/forums/viewtopic.php?t=28920

Stephen Phillips http://blog.scphillips.com/2013/07/getting-a-python-script-to-run-in-the-background-as-a-service-on-boot/

Andrea Fabrizi https://github.com/andreafabrizi/Dropbox-Uploader

Raspberry Pi and si470x based RDS receiver/logger

The system above looks complicated, but it really isn't.  The white brick in the lower left is a USB power supply.  The Raspberry Pi is the device at the top left, connected to it by jumper wires is the si470x breakout board.  The Pi is also connected to a USB 3 hub that has a Wi-Fi dongle and a USB 2 hub connected to it.  That may sound strange, but the Pi can't recognise USB 1.1 devices plugged into a USB 3 hub, so you need a USB 2 hub in there to allow the connection of the mouse and keyboard.  Yeah, I could have done this via an SSH terminal, but as I was connecting wires directly to the Pi it just seemed easier to be at the device while I did it.

The software is simple as well.  I modified the original code to just constantly poll the si470x receiver, log the RDS messages with timestamps and station frequency, and then rotate the logs hourly.  The script was also configured to run as a background service on start-up.  So if it loses power, it should just automatically start logging again.

I also planned to run the dropbox_uploader utility on a schedule with a cron job to backup the log files.  I wrote the script to do this, but it seemed easy enough to run it manually every now and then.

RDS log file

The RF signal chain is important, you need a good signal to get the RDS data.  Even if you can hear the station clearly you still might not be able to receive song names.  The si470x breakout board uses the headphone wiring as the FM radio antenna, just like most phones, if you want the radio to work you have to use some type of headphone.  The presence on a human body also boosts the signal.  This is a nice solution in most cases, but I couldn't sit there wearing headphones for a week.  Well, I could but I don't think I'd have a job at the end of the project.  I guess that'd be a bad thing, maybe.

To get a good signal I took a folded dipole antenna made of ribbon cable I just happened to have and used that, I put it on a table and put some books on the end to stop it rolling up.

300 ohm Folded dipole ribbon cable antenna

The folded dipole has a 300 ohm balanced output, but I needed a 75 ohm unbalanced signal, so the antenna was connected to a balun.

300 - 75 Ohm Balun

To get the coax signal into the breakout board I used a simple adapter.  In terms of impedance matching it's not ideal, but I did the best I could with the rest of the chain so it turned out OK.

Coax to 3.5mm plug

Connecting the Raspberry Pi to the si470x breakout board is relatively easy, just be careful GPIO 23 is actually pin 16, don't know who came up with that idea.

Add caption

I had a lot of fun with this project. It took a week of coding and fiddling in my spare time to get it working.  I'm pretty happy with that.  I'm usually more self concious about my coding ability, but you know what, who cares.  I made something that does what I want, and if anyone has issues with my code they're more than welcome to improve it and do whatever they want with it.  I'd love that.  That's why it's out there, to help the next person that comes along.

What's The Channel Capacity Of The ADS-B System

If you've seen my twitter feed lately you may have noticed I've been playing around with ADS-B, the system used by commercial planes to report their position and other details.  I find it an interesting topic and a great way to get into software defined radio.  For just $20 you can start to play around with a TV tuner dongle and map air traffic in your local area.  As usual I wanted to get down into the nitty-gritty and find out more about how the protocol works, particularly how many planes it can support at once.

First of all we'll take a quick overview of how the system works.  The ADS-B traffic I'm interested in occurs at a frequency of 1090 MHz and consists of frames/packets that are 112 microseconds long with a 8 microsecond preamble, a total of 120 microseconds.  All of the data a plane transmits can't fit into a single frame, so there are different types transmitted, some are initiated by the plane, some are initiated by ground radar systems, some are required twice a second, some are required every 5 seconds.  We'll get to this later, but for now I want to look at the pure throughput of the channel.  It may seem simple, but as the number of planes/stations transmitting increases, the probability that they'll "talk" over each other increases, and frames start to collide.  If we take this example to an absurd extent and assume there are infinite planes, the throughput of the channel will be zero because the planes will always be talking over the top of each other.  There's a sweet spot that can be found.

Grey frames are ones that have collided

If messages are 120 microseconds long, it should be easy to determine the probability of collisions at certain data rates.  First though, a few assumptions need to be made.  Planes transmit their ADS-B frames whenever they want.  There are communication systems that only transmit at certain time periods (slotted systems), and some listen to the channel and wait for it to be clear before sending (carrier sense multiple access), these methods reduce collisions, increasing the throughput of the channel.  In the case of ADS-B, it's hard to find information on how it's actually done, so I'm going to assume the random transmit method, which is the basis for ALOHAnet, the first public demonstration of a wireless packet data network.  So most of the analysis has been done for us.  In fact the images (with attribution) in this blog have come from the Wikipedia article for ALOHAnet.  This article is going to apply the theory of the ALOHAnet to ADS-B.

Another assumption that needs to be made is the signal strength coming from different stations/planes.  We'll assume that all the signals are received at the same level, and will therefore collide.  In reality a plane transmitting 150 km away, will get lost in the signal of a plane transmitting 500 m away and will not be seen as a collision.

The final assumption to make is that the simple CRC checks in some of the ADS-B messages aren't used to correct the signal.  If that were the case small collisions would be OK, but let's keep things simple.

Frames transmitted between t0-T and t0+T will collide with the yellow test frame

Firstly define the time taken to transmit a frame as T, and also define G as the average number of transmission attempts per frame in Erlangs.  In the above image it can seen that any frames transmitted in the period from t0-T to t0 will collide with the start of the yellow test frame, and any frames transmitted in the period from t0 to t0+T will collide with the end of the yellow frame.  From this we can see the probability of a successful transmission of the yellow test frame is equal to the probability that no other frames are transmitted during the period between t0-T to t0+T.  A period of 2T.

If the arrival distribution is modelled as a poisson process the probability that k frames are sent in a time period of 2T is equal to

By setting k to zero we can see the probability of no other frames sent in this time frame.

This gives a throughput in frames per frame period of

The maximum value of S can be easily found.

So, the maximum throughout is 0.184 frames per frame period, and this occurs at a channel load of 0.5 Erlangs.  This is demonstrated in the graph below for the Pure Aloha system.

Throughput vs load for slotted and pure Aloha

Now to bring this all back to ADS-B is rather simple.  T is simply the 120 microseconds mentioned earlier.  We know the maximum throughput is 0.184 frames per frame period, which in this case is 0.184 frames per 120 microseconds which is equal to 1533 frames per second.

1533 is the number of frames per second supported in this instance, but it's not the number of planes.  Each plane is required to transmit location and velocity frames twice a second.  They are also required to transmit a flight ID frame every 5 seconds.  This means at a bare minimum the plane is transmitting 4.2 frames per second.  I'm unsure if the plane is required to transmit it's position twice a second or position frames twice a second (there is a difference).  ADS-B uses a strange compression method called compact position reporting to transmit the latitude and longitude of a plane.  Two frames are required for a full latitude and longitude position.  I'm going to stick with 4.2 frames a second but it could be 6.2 frames per second.

Taking the number of 4.2 frames per second per plane as a minimum means that there can't be more than 1533/4.2 = 365 planes visible to a receiver at any one time.  Even that's an unlikely scenario, because at that point there are quiet a few frame collisions occurring. Having said that there will always be some collisions.  At maximum capacity of G equal to 0.5, the probability of no collisions in two frame periods is equal to e^(-1) = 0.368, if the load were reduced by a factor of 10 this becomes e^(-0.1) = 0.905, so although the probability of a collision is reduced.  It's still around 10% which is not insignificant.  This load represents a more realistic number of planes, 36 (365/10) , detected by an ADS-B station.  So obviously there's somewhere in the specification for ADS-B that defines an acceptable level of frame collisions, but because it seems to be a propriety standard, we can only guess what that number is.

The point of this article was to get a feel for the capacity of ADS-B.  My numbers are a ballpark figure.  I'm doing some coding and I need to figure out if I can handle the incoming data using the method I plan to use.  I'm pretty sure I can.

Further Reading
http://defenseelectronicsmag.com/site-files/defenseelectronicsmag.com/files/archive/rfdesign.com/mag/512RFDSF3.pdf
http://www.homepages.mcb.net/bones/SBS/Article/Barebones42_Socket_Data.htm

References
http://users.ecs.soton.ac.uk/sqc/EL336/CNL-7.pdf
http://en.wikipedia.org/wiki/ALOHAnet