Merge A Data Set With A Template File To Generate Output Files

For something I'm working on I need to be able to create a large number of files by filling in fields in a template file with entries from a data set. You'd think that would be easy with Linux but I couldn't find a way to do it. (This will be where people tell me a thousand different ways to do it) I didn't think what I wanted was complicated so I wrote SimpleMerge to take care of it. It is a basic Python script that takes data from a tab delimited data file and fills in data fields in a template file.

The first row of the data file are the field identifiers to find and replace and the other rows are just data. This file can be easily generated from a spreadsheet program. The template file contains the structure of the file you intend to create, just with field identifiers in the place of real data.

I haven't done extensive testing on the program but it seems to work fine.  It handles UTF-8 file encoding and maintains the line endings of the template file for both UNIX and Windows systems. The following command generates the two files File1.txt and File2.txt as seen in the block diagram below.

SimpleMerge.py template.txt Data.txt

Simple Merge Block Diagram

You can use this method on any file really, even SVG files.  Hint hint wink wink.  You can go from this template file.....

SVG Template

to this in a matter of minutes. Just by replacing colour and three text fields.

Generated Images

I make no guarantee as to how well this works. So my advice is to back things up before using it. Have fun.

Get The Code!

.

Sorting Subjective Items With Software

Have you ever needed to sort a large number of items that don't have an easily defined order?  Maybe it's what TV to buy or where you're going for a holiday.  Some things only have a specific order to you. So ordering them with software isn't an easy option.  To simplify the process I wrote Sortamajig.  Watch it in action.

For a long time now I've know that the ballot paper for the 2016 Australian senate elections had a chance of being large, and because I'm a little pedantic/broken I like numbering all the boxes. The ballot paper for the Australian senate can get a little complicated.  You have the option of voting for at least 6 parties(sort of) above the line (which automatically selects the candidates under the line in order), or numbering at least 12 individual candidates under the line.  That might not seem hard, but in the 2016 election the ballot paper for Queensland will have about 40 columns and 122 candidates under the line.  This is where Sortamajig helps.

Australian Senate Ballot Paper

All I have to do is feed it a group of images and it will continue asking me to choose between two of them until they're sorted.  If I can't make a choice, there's a button to randomly make a choice for me.  If I make a mistake there are undo and redo buttons as well.  When the sort is complete, I click on generate output and an image grid and text file of the results is generated.

Sortamajig GUI

Now to test out the program I needed something I can sort that's subjective and easily comparable.  So I decided to go with Jelly Beans, particularly Jelly Belly brand.  I love them, and people's favourites vary just as much as their favourite politicians.

Test Jelly Beans - Barcode 07156798554

To generate some data I decided to do three trials on separate days and combine the data into a final result.  Some flavours can overpower the next flavour tasted, to prevent this the tasting order was randomised on each trial by shuffling a 50 numbered pieces of paper and placing them in the compartments on the box tray.  The jelly beans were tasted in this order.

Randomized Sampling Process

Below are the results of the three trials in image form.  The jelly beans are ordered from left to right and then top to bottom.  For instance, in the first image Strawberry Jam is first and Plum is second.

Trial 1

Trial 2

Trial 3

To combine the results I averaged the position of each flavour in the different trials and re-ranked them using the average.  The graph below shows the average position on the horizontal axis and the position and spread of the three trials on the vertical axis.

Aggregated Trial Results

An interesting thing I noticed when processing the results of the 3 trials is that I'm reasonably certain about the flavours I like the most and least.  In the middle of the rankings however, the positions of a particular flavour in each trial has a wider spread and is less certain.

The code is really rough and needs to be cleaned up.  If you have a look at the Github repo at some point in the future I may have improved it.

Get The Code!

Some of the groundwork for this tool can be found in these other posts.

Merge Sort... Sort Of 
Smart TkInter Image Manager for Labels
A Python Sorting Manager With a Human Comparison Operator
Visually Appealing Image Layout Algorithm

Now for the important part.  What were my results?

01.  Lemon Drop
02.  Pomegranate
03.  Tutti Fruitti
04.  Plum
05.  Orange
06.  Raspberry
07.  Lemon
08.  Lemon Lime
09.  Strawberry Jam
10.  Tangerine
11.  Bubble Gum
12.  Orange Sherbet
13.  Pink Grapefruit
14.  Red Apple
15.  Cotton Candy
16.  Green Apple
17.  Wild Blackberry
18.  Blueberry
19.  Grape
20.  Island Punch
21.  Crushed Pineapple
22.  Strawberry Cheesecake
23.  Pina Colada
24.  Strawberry Daiquiri
25.  Buttered Popcorn
26.  Berry Blue
27.  Margarita
28.  Cream Soda
29.  Peach
30.  Caramel Corn
31.  Very Cherry
32.  Cherry Cola
33.  Mango
34.  Kiwi
35.  Watermelon
36.  Toasted Marshmallow
37.  Coconut
38.  Chocolate Pudding
39.  Cantaloupe
40.  Caramel Apple
41.  Vanilla
42.  Juicy Pear
43.  Green Tea
44.  Top Banana
45.  Dark Chocolate
46.  Cafe Latte
47.  Liquorice
48.  Cappuccino
49.  Chili Mango
50.  Sizzling Cinnamon

Bit Reversal Permutation Access

It'll take me a while to get to my point, but stick with me.  Recently while I was waiting for a train, a freight train passed the station and I came up with an interesting thought experiment.  I wondered "where is that train going, what does its journey look like, and how could I find out?"  It's relatively simple to just Google the answer or look up satellite maps of the trains tracks, but that doesn't tell you information about where it was at what time, what speed it was doing, or something a little stranger like weather conditions.

Note: Before I get into things, I should point out that this is just a thought experiment.  Sometimes I like to think of problems and try to solve them just to keep my mind busy when I'm bored.  Sticking a random piece of electronics to a train would be a dumb and potentially dangerous thing to do.  Don't be stupid.  Now, on to the problem at hand.

I wondered could a tracking device be attached to the train to take readings and report back?  Of course it can, there's nothing too complicated about that.  A few sensors, a single board computer, a battery, and some powerful magnets to hold it in place and you're done.  But how does the data get back to you?  At the moment the Raspberry Pi 3 is the new hotness in the maker community and it comes with built in WiFi.  As the train makes its way across the countryside, instead of using the mobile phone network to return the information could the data be transmitted back to the user by latching on to fleeting connections to the internet via open WiFi hotspot, kind of like a monkey swinging from vine to vine in the jungle. #LabouredSimile

Now an arrangement like this where you have unpredictable and limited connection to the internet means you have to transmit the data in the most useful way possible.  Let's say the data is stored in regularly sampled records, what's the best order to transmit those back?  Well, the first thing I want to know is where the train is now.  The next most useful thing to transmit back isn't the next record, because it won't differ that much from the previously transmitted record.  What you really want next is the record half way between the start and the previously transmitted record.  This will fill in the missing data in a way that gives you the most useful information first.  If we continue this pattern you just keep filling in the records that are midway between the ones transmitted previously.

I thought about it and realised that if you have an incrementing binary counter, almost all you need to do is reverse the order of the bits in the counter and access the records in that order.  That gets you most of the way there, unfortunately it gives you the oldest data first.  To fix that, just invert the bits.  That sounds pretty complicated, but hopefully the animation below can hep you out.  It assumes we have 16 records to transmit and demonstrates the "In Order", "Reversed", and "Reversed & Inverted" methods.

Bit Reversal Permutation Access

You can see that the "Reversed & Inverted" method returns the data in the order required.  It returns the newest data first and keeps returning records in between already returned records.  After a bit of Googling I discovered this is called Bit Reversal Permutation, and is used in the process of calculating Fast Fourier Transforms.

There are a lot of assumptions in my scenario, like adequate WiFi strength, and power of two records.  It's just a bit of fun that as it turns out has a real world application.

Improved Computer Assisted Human Based Data Entry

OK, I'm revisiting my post from last week.  I was certain that I could reduce the time it took to enter data with some simple modifications.  The results are at the end. The main change was the removal of the suggestion frame from the right and its replacement with a suggestion label positioned under the box that's currently in focus.  Take a look at the previous version to see the difference.   Another change was that previously pressing enter while in an entry box had no effect.  Now it selects the first suggestion and moves focus to the next box down.

These changes reduce the number of keypresses and the amount of scanning that your eyes need to do of the on screen data.  I was also feeling a little nostalgic, so I've chosen some 4-bit colours for different elements. :-)

Data entry with suggestions

The window is broken into two different frames, divided at the blue line, one on the left for the image to process, and one on the right for the entry boxes and the status label.  The second frame in arranged into 11 rows, divided at the red lines, with the grid manager.

Arrangement of input window

The hard bit however was getting the suggestion label to be placed under the entry box.  First of all, a few things need to be discussed in the image below.

The frame is divided into 3 columns indicated by the blue lines.  This allows small coloured indicators besides the entry box to indicate which one has focus.  This is done because ttk entry boxes don't allow the background colour to be changed.  The small coloured regions are fixed size and the entry box is allowed to resize to fill the rest of the space.

The next step isn't as simple as placing a label under the input box.  First a frame needs to be inserted with the place manager that's underneath the entry row and is the same width.  This frame is outlined in yellow.


self.suggestion_frame.place(relx=0,
                            rely=1,
                            relwidth=1,
                            bordermode='outside',
                            in_=self.entry_rows[row_number].entry)

This then has the suggestion label inserted in the frame.  The label stops at the green line.

self.suggestion_label.grid(in_=self.suggestion_frame,
                           column=0,
                           row=0,
                           sticky='nsw')

I hear you say that's weird, why not just add the label under the entry box, why use a frame?  It's because of how tkinter handles long lines in labels.  I construct a string of text to add to the label that contains carriage return characters (\n).  This usually works for small lines, but for long lines, the string is cropped to make it fit the label.  It doesn't realise that the carriage return characters start a new line.  Fox example, the string 

really long text part one \n really long text part two \n really long text part three

would display correctly on three lines if the label was wide enough.  However if the label was only wide enough to display 4 characters, it would display "real" once and then crop the rest of the text before it gets to the carriage return characters.  What you probably expected to see was:

real

real

real

The way to get around this is to create a frame of the correct width and then place a label in it that has no set width.  The label will then resize to a width to fit all the text, while the frame acts to crop the text limiting what's shown.

Suggestion label

So did the entry speed increase?  Yes it did.  In my last post I noted that typing the text with no assistance took about 90s per image.  My first attempt took 1:25:41 for 76 images or 67s per image.  This new arrangement allowed for a time of 1:07:26 for 76 images or 53s per image.  I'm pretty happy with that.  That's a 40% decrease in the input time from unaided and a 20% decrease from the initial attempt.  Although I'm likely to get faster the more I type the data, the tests were a week apart so I think this effect is minimal.

Get the code

.

Visualizing Periodic Order Schedules

I've been grappling with a problem involving order schedules and thought a visualization may help to understand the problem better.  There just doesn't seem to be an existing graph that satisfies all my criteria.  Before I go to far it may help to explain my situation.  Imagine a system where weekly repeating orders are placed, and each of these orders go through several stages of fulfilment.  For example, here is an order placed every week on Friday.

Friday - Order Placed 

-on order-

Saturday - Order Delivered

-on hold-

Sunday - Stock placed on show for sale

-on show-

Tuesday - Check remaining stock and investigate overstocks

-overs-

This seems simple to understand, the order is placed and is considered "on order", once it's delivered it's put "on hold" until it's checked to make sure the delivery is correct.  Once placed on show for sale it's considered "on sale".  After the next load arrives any remaining stock is considered to be an "over" or overstock.  Not too hard.  Now imagine you have many of these orders during a week and they all overlap.  It starts to become confusing.  A traditional way of displaying this might be in the form of the graph below.  It's certainly clear, but the cyclic nature of the orders isn't displayed.  For example, the order at the top should be entered again starting at the second Friday column in the graph, but this makes it look like you have more orders that you really do.  Is there a better way?

Linear order graph

The cyclic nature of the orders suggest that a circular graph may be useful and with a bit of playing around I came up with the visualization below.  The first event in an order sequence is mark on the outer part of the graph, it then spirals inwards indicating the different stages of the order process with different colours.  It's advantages are that orders only need to be added to the graph once and it allows the user to see the current state of all orders.  The graph below contains 5 orders and is easy to read once you understand how it works.

Circular order graph

Time for an example.  Let's say that on Tuesday at 6 pm you are curious about stock levels.  You start by drawing a line from the centre at this time on the edge of the graph.  You can see that the line crosses a green line, this indicates on show stock, it crosses a yellow line, this indicates on hold stock, and it crosses two red lines, both of these are orders that have been placed and not delivered.  If you want know the time the next load is delivered, rotate the line forward, until one of the red lines turns yellow.  This happens around midday Wednesday.

You may also want to know the history of the stock on show.  That's also easy, just follow the green line backwards.  Events mentioned are labelled with a black dot. You can see that the stock became available for sale on Sunday morning, arrived midday Saturday, and was ordered Thursday morning (that's a pretty long lead time isn't it).  For completeness the blue line indicates overstocks, and as this is the last stage of the order process and doesn't really have an end time, the line will fade out over two days.

Another interesting feature is that you can immediately see when an order for stock was placed by looking at the white grid lines.  Look at where the black line crosses the yellow line.  It's 4 and bit divisions from the edge of the graph.  This means that order was placed 4 and bit days ago.  You can confirm this by following the line back to see that this order was placed on Friday morning.

It's just a prototype and the colours need to be tweaked.  You could also make an axis logarithmic to emphasise certain data.  I like it.  It reminds me of a Smith Chart.

I've put the python code I used to generate this in  gist.  It's not pretty and should just be used to get you started.

Get the code!

Find All Triangles In A Diagram And Add Together The Numbers They Contain

If you're a night owl like me you may have come across those late night game shows where you call in to win money.  I'm not a fan of them.  The rules are very vague, and they're broadcast on channel 74 (4ME) at a highly compressed resolution at 576i, so it can be hard to see some of the details (especially for older people).  You've got to hand it to the hosts though, there's no way I couldn't stand there for hours babbling on a about nothing, that's what I have a blog for :-)

One of the games they've been playing lately intrigued me.  You have to find all the triangles in a diagram like the one below, and then add all the numbers in each triangle together to reach a final value.

Puzzle from Call and Win 15/12/2015

The host suggests a way to find the answer by drawing it out and finding all the triangles manually, and sure it'll work, but where's the fun in that I ask you?  Can we write software to find the answer?  Of course we can.

We'll start by taking a closer look at the puzzle.  They use a mix of Roman and Arabic numeral to be "tricky".  There are also some areas with multiple number in them.  For our analysis these will be simplified manually when the problem is defined in software.

Puzzle

To start with, let's strip all the useless information and number each vertex.  Each region will also be labelled with a number and its value placed in brackets.  Now we can discuss the problem in formal terms.

To define the puzzle in terms that software can understand, it isn't sufficient to just feed it a list of vertices and how they're connected.  For example, in the image below, if vertex 5 is moved into region 8. The connections of the graph are preserved, but its shape is changed.  To make sure that the structure is well defined, it's better to define the network by describing each region by a list of vertices in a consistent clockwise or anticlockwise manner.  This isn't exactly true, but it works for this problem.  (I've had a crash course in graph theory and discovered and learnt some interesting things)  That's great, but you can't find triangles with just a  definition of the regions.  A graph isn't a geometric representation of a structure, it just defines connections.  Adding a list of the straight lines in the graph will enable us to solve the problem.  These are defined by listing all groups of 3 or more vertices that are collinear (2 vertices are trivially collinear so they don't need to be defined).

Structured Network

The above structured network (I'm calling it that because it's a network, but it has some geometric constraints) can be defined by the following 16 lines of code.  The Region objects are defined by supplying an id, a value, and a list of vertices.  The StraightLineSegment objects are defined by a set of vertices.

Defining the structured network

We now have the network defined in software, but how do we actually solve the problem?  The human way to go about it is to find the triangles and then add all the numbers in them.  It's trivial to write a program to find the triangles in the network.  For example pick, vertex 1, it's connected by straight lines to vertices 2, 3, 8, 10, 4, 7, 6, 5.  For each of these vertices you would then repeat the process.  Let's choose vertex 2.  It's connected to vertices (1, 8, 7, 9, 4, 3) by straight lines. You ignore vertex 1 because that's where you came from, but if any of these vertices can be connected back to vertex 1, you've found a triangle.  Out of that last list, vertices (8, 7, 4, 3) connect to vertex 1.  This means you've found 3 triangles, (1,2,8), (1,2,7), (1,2,4).  You ignore vertex 3 because it's a straight line.  There's a subtle problem with that though.  It's hard to find the regions that make up a compound region like the one constrained by the vertices (1, 2, 4).  This means it's hard to add the numbers together.  It's easy for humans, but it's not as simple for computers.

The way I've chosen to solve the problem is to find every possible compound region in the network adding the values as I go.  The triangular regions are then identified and the others are discarded.

Finding compound regions

You may be thinking "Oh, it's just combinations of regions", but hold on.  If I want the region that's a combination of regions 6, 7, and 8.  You need to know how to construct it. (Region 6 + Region 7) + Region 8 makes sense, but (Region 6 + Region 8) + Region 7 doesn't.  Adding Region 6 to 8 doesn't make sense they aren't connected.  This implies that Region addition isn't commutative.

The easiest way to find all compound regions is to start with all the base regions and see how many regions with two base regions you can find.  For example if we select region 6.  It's connected to regions 3 and 7.

Expanding a region

This means you've found two regions with 2 base regions. The new regions are region {6, 3} and region {6, 7}.  Because they've been discovered by looking at connected regions it's easy to find the vertices that describe their edges.  It's also easy to add their values.

Compound region {6,7}

You'll find duplicate regions using this method, but they're easily removed.  Once you've found all the compound regions with 2 base regions in them, you can repeat the process to find all the regions with 3 base regions in them.  For example if region {6, 7} is chosen, you can see that 3 new regions {3, 6, 7} {4, 6, 7} and {6, 7, 8} are found.  This process is repeated until you find a region that contains all base regions.  Once this point is reached, there are no more regions to add.

Expanding region {6,7}

Now triangles need to be identified.  Let's look at region {6, 7, 8}.  It's defined by a vertex list of [1, 8, 10, 4, 6, 7].  If we look at this ordered circular list 3 vertices at a time we can find vertices to remove.  For example vertices [1, 8 ,10] are collinear.  This means we can remove vertex 8 from the list.  Repeating this process until no more vertices can be removed will show if the region is triangular.

For example
[1, 8, 10, 4, 6, 7] -> [1, 10, 4, 6, 7] -> [1, 4, 6, 7] -> [1, 4, 6]

You now have a list containing 3 vertices.  This means it's a triangle.  You then just go through and find the sum of all these elements.  I'll go into some more of the theory and implementation in the next blog post, but that's the basics of it.

If you've read this far I bet you've tried to work out the answer.  Scroll to the bottom of this page and you can see the output of my program with the answer way down the bottom

Get the code!

Base regions ({id} =value= *vertices*)
count = 9

({7} =10= *10* *6* *7* *8*)
({5} =4= *9* *4* *10*)
({3} =5= *1* *2* *8*)
({6} =8= *1* *8* *7*)
({8} =1= *6* *10* *4*)
({4} =2= *10* *8* *2* *9*)
({1} =8= *2* *3* *9*)
({2} =3= *4* *9* *3*)
({9} =9= *5* *6* *4*)


Multi edge straight lines -*vertices*-
count = 7

-*1* *2* *3*-
-*8* *1* *10* *4*-
-*1* *5* *6* *7*-
-*8* *2* *7*-
-*9* *10* *3* *6*-
-*9* *2* *4*-
-*3* *4* *5*-


edge list |*vertices*| -> {connected region id}
count = 18

|*9* *2*| -> {4}{1}
|*10* *6*| -> {7}{8}
|*9* *4*| -> {5}{2}
|*1* *7*| -> {6}
|*3* *4*| -> {2}
|*8* *1*| -> {6}{3}
|*1* *2*| -> {3}
|*6* *7*| -> {7}
|*8* *2*| -> {4}{3}
|*9* *3*| -> {2}{1}
|*8* *7*| -> {6}{7}
|*2* *3*| -> {1}
|*8* *10*| -> {7}{4}
|*4* *6*| -> {8}{9}
|*5* *6*| -> {9}
|*10* *4*| -> {5}{8}
|*9* *10*| -> {4}{5}
|*4* *5*| -> {9}


Compound regions ({id} =value= *vertices*)
count = 174

({4, 6, 7} =20= *6* *7* *1* *8* *2* *9* *10*)
({6, 7} =18= *10* *6* *7* *1* *8*)
({1, 2, 4, 5, 7, 8, 9} =37= *3* *4* *5* *6* *7* *8* *2*)
({1, 4, 7} =20= *2* *3* *9* *10* *6* *7* *8*)
({8, 9} =10= *5* *6* *10* *4*)
({8, 1, 4, 5, 7} =25= *4* *6* *7* *8* *2* *3* *9*)
({1, 2, 3, 4, 5, 6, 8} =31= *1* *2* *3* *4* *6* *10* *8* *7*)
({2, 3, 4, 5, 6, 8} =23= *1* *2* *9* *3* *4* *6* *10* *8* *7*)
({1, 3, 4, 7, 8, 9} =35= *1* *2* *3* *9* *10* *4* *5* *6* *7* *8*)
({1, 3, 4, 5, 7} =29= *4* *10* *6* *7* *8* *1* *2* *3* *9*)
({1, 4, 5, 6, 7, 8} =33= *3* *9* *4* *6* *7* *1* *8* *2*)
({5} =4= *9* *4* *10*)
({1, 3, 4, 6, 7, 8, 9} =43= *5* *6* *7* *1* *2* *3* *9* *10* *4*)
({2, 5, 6, 7, 8, 9} =35= *4* *5* *6* *7* *1* *8* *10* *9* *3*)
({3, 4} =7= *1* *2* *9* *10* *8*)
({8, 9, 6, 7} =28= *5* *6* *7* *1* *8* *10* *4*)
({2, 5} =7= *4* *10* *9* *3*)
({8, 4, 7} =13= *6* *7* *8* *2* *9* *10* *4*)
({1, 4, 5, 7} =24= *6* *7* *8* *2* *3* *9* *4* *10*)
({1, 2, 3, 4, 7, 8, 9} =38= *5* *6* *7* *8* *1* *2* *3* *4* *9* *10* *4*)
({8, 9, 4, 5} =16= *4* *5* *6* *10* *8* *2* *9*)
({8, 9, 4, 5, 1} =24= *4* *5* *6* *10* *8* *2* *3* *9*)
({1, 4, 5, 6, 7} =32= *6* *7* *1* *8* *2* *3* *9* *4* *10*)
({2, 3, 4, 5, 7} =24= *4* *10* *6* *7* *8* *1* *2* *9* *3*)
({1, 3, 4, 6, 7, 8} =34= *7* *1* *2* *3* *9* *10* *4* *6*)
({1, 3, 4} =15= *2* *3* *9* *10* *8* *1*)
({8, 5} =5= *9* *4* *6* *10*)
({1, 2, 5, 6, 7, 8} =34= *3* *4* *6* *7* *1* *8* *10* *9* *2*)
({2, 3, 4, 5, 6} =22= *4* *10* *8* *7* *1* *2* *9* *3*)
({8, 1, 2, 4, 5} =18= *8* *2* *3* *4* *6* *10*)
({8, 3, 4, 5, 6} =20= *1* *2* *9* *4* *6* *10* *8* *7*)
({3, 4, 5} =11= *8* *1* *2* *9* *4* *10*)
({2, 3, 5, 6, 7, 8} =31= *9* *3* *4* *6* *7* *1* *2* *8* *10*)
({8, 1, 4, 5} =15= *8* *2* *3* *9* *4* *6* *10*)
({1, 3, 4, 7} =25= *10* *6* *7* *8* *1* *2* *3* *9*)
({3, 4, 5, 7, 8, 9} =31= *5* *6* *7* *8* *1* *2* *9* *4*)
({6} =8= *1* *8* *7*)
({8, 4, 5, 7} =17= *9* *4* *6* *7* *8* *2*)
({8, 9, 5} =14= *5* *6* *10* *9* *4*)
({8, 2, 5} =8= *4* *6* *10* *9* *3*)
({8, 2, 3, 4, 5} =15= *8* *1* *2* *9* *3* *4* *6* *10*)
({1} =8= *2* *3* *9*)
({1, 2, 3, 4, 5, 6, 7, 8} =41= *7* *1* *2* *3* *4* *6*)
({2, 3, 4, 5, 6, 8, 9} =32= *4* *5* *6* *10* *8* *7* *1* *2* *9* *3*)
({3, 4, 5, 7} =21= *1* *2* *9* *4* *10* *6* *7* *8*)
({8, 9, 2, 4, 5} =19= *4* *5* *6* *10* *8* *2* *9* *3*)
({2, 4, 5, 7} =19= *4* *10* *6* *7* *8* *2* *9* *3*)
({4, 7} =12= *6* *7* *8* *2* *9* *10*)
({1, 2, 5, 6, 7, 8, 9} =43= *3* *4* *5* *6* *7* *1* *8* *10* *9* *2*)
({2, 3, 4, 5} =14= *4* *10* *8* *1* *2* *9* *3*)
({9} =9= *5* *6* *4*)
({1, 4, 6, 7, 8, 9} =38= *3* *9* *10* *4* *5* *6* *7* *1* *8* *2*)
({4, 5, 6, 7} =24= *6* *7* *1* *8* *2* *9* *4* *10*)
({8, 5, 7} =15= *6* *7* *8* *10* *9* *4*)
({1, 2, 3, 4, 5, 6, 7, 8, 9} =50= *1* *2* *3* *4* *5* *6* *7*)
({8, 1, 3, 4, 5} =20= *8* *1* *2* *3* *9* *4* *6* *10*)
({2} =3= *4* *9* *3*)
({8, 9, 5, 7} =24= *5* *6* *7* *8* *10* *9* *4*)
({1, 2, 4, 6, 7, 8} =32= *3* *4* *9* *10* *4* *6* *7* *1* *8* *2*)
({3, 4, 6} =15= *1* *2* *9* *10* *8* *7*)
({3, 4, 7} =17= *1* *2* *9* *10* *6* *7* *8*)
({1, 2, 4, 5, 8, 9} =27= *8* *2* *3* *4* *5* *6* *10*)
({2, 4, 5, 6, 7, 8, 9} =37= *4* *5* *6* *7* *1* *8* *2* *9* *3*)
({1, 3, 4, 5, 7, 8, 9} =39= *4* *5* *6* *7* *8* *1* *2* *3* *9*)
({3, 4, 5, 6} =19= *8* *7* *1* *2* *9* *4* *10*)
({1, 3, 4, 5, 6, 7} =37= *6* *7* *1* *2* *3* *9* *4* *10*)
({8, 9, 2, 5, 1} =25= *3* *4* *5* *6* *10* *9* *2*)
({1, 2, 4, 5, 6, 7, 8} =36= *3* *4* *6* *7* *1* *8* *2*)
({3, 5, 6, 7, 8, 9} =37= *9* *4* *5* *6* *7* *1* *2* *8* *10*)
({8, 9, 3, 6, 7} =33= *4* *5* *6* *7* *1* *2* *8* *10*)
({1, 2, 5, 7, 8, 9} =35= *3* *4* *5* *6* *7* *8* *10* *9* *2*)
({1, 2, 4, 6, 7, 8, 9} =41= *5* *6* *7* *1* *8* *2* *3* *4* *9* *10* *4*)
({1, 2, 3, 4, 6, 7, 8, 9} =46= *5* *6* *7* *1* *2* *3* *4* *9* *10* *4*)
({8, 1, 4, 6, 7} =29= *7* *1* *8* *2* *3* *9* *10* *4* *6*)
({1, 4} =10= *10* *8* *2* *3* *9*)
({2, 4, 5, 6, 7} =27= *4* *10* *6* *7* *1* *8* *2* *9* *3*)
({1, 2, 4, 5, 7} =27= *2* *3* *4* *10* *6* *7* *8*)
({3, 4, 5, 6, 7} =29= *6* *7* *1* *2* *9* *4* *10*)
({1, 2, 4, 7, 8, 9} =33= *5* *6* *7* *8* *2* *3* *4* *9* *10* *4*)
({3, 4, 6, 7, 8, 9} =35= *5* *6* *7* *1* *2* *9* *10* *4*)
({4, 5, 7} =16= *6* *7* *8* *2* *9* *4* *10*)
({3, 4, 5, 6, 7, 8} =30= *7* *1* *2* *9* *4* *6*)
({8, 1, 2, 4, 7} =24= *6* *7* *8* *2* *3* *4* *9* *10* *4*)
({1, 2, 3, 4, 6, 7} =36= *6* *7* *1* *2* *3* *4* *9* *10*)
({8, 2, 5, 6, 7} =26= *4* *6* *7* *1* *8* *10* *9* *3*)
({8, 1, 2, 5} =16= *3* *4* *6* *10* *9* *2*)
({8, 6, 7} =19= *7* *1* *8* *10* *4* *6*)
({1, 2, 3, 5, 6, 7, 8, 9} =48= *3* *4* *5* *6* *7* *1* *2* *8* *10* *9* *2*)
({8, 3, 5, 6, 7} =28= *9* *4* *6* *7* *1* *2* *8* *10*)
({8, 1, 3, 4, 7} =26= *1* *2* *3* *9* *10* *4* *6* *7* *8*)
({2, 4, 5, 6, 7, 8} =28= *7* *1* *8* *2* *9* *3* *4* *6*)
({1, 3, 4, 5, 7, 8} =30= *1* *2* *3* *9* *4* *6* *7* *8*)
({2, 3, 5, 6, 7, 8, 9} =40= *9* *3* *4* *5* *6* *7* *1* *2* *8* *10*)
({1, 2, 4, 5, 6, 7, 8, 9} =45= *3* *4* *5* *6* *7* *1* *8* *2*)
({8, 9, 4, 7} =22= *5* *6* *7* *8* *2* *9* *10* *4*)
({1, 2, 3, 4, 5, 8, 9} =32= *2* *3* *4* *5* *6* *10* *8* *1*)
({8, 2, 4, 5} =10= *4* *6* *10* *8* *2* *9* *3*)
({3} =5= *1* *2* *8*)
({8, 3, 4, 5, 7} =22= *6* *7* *8* *1* *2* *9* *4*)
({2, 3, 4, 5, 7, 8, 9} =34= *1* *2* *9* *3* *4* *5* *6* *7* *8*)
({1, 2, 4, 5, 6, 7} =35= *3* *4* *10* *6* *7* *1* *8* *2*)
({1, 3, 4, 5, 6, 8, 9} =37= *4* *5* *6* *10* *8* *7* *1* *2* *3* *9*)
({8, 9, 2, 5, 7} =27= *3* *4* *5* *6* *7* *8* *10* *9*)
({8, 3, 4, 5} =12= *8* *1* *2* *9* *4* *6* *10*)
({8, 9, 3, 4, 7} =27= *1* *2* *9* *10* *4* *5* *6* *7* *8*)
({2, 3, 4, 5, 6, 7, 8, 9} =42= *4* *5* *6* *7* *1* *2* *9* *3*)
({8, 9, 2, 5} =17= *4* *5* *6* *10* *9* *3*)
({1, 3, 4, 5, 8, 9} =29= *4* *5* *6* *10* *8* *1* *2* *3* *9*)
({1, 2, 3, 4, 5, 6, 8, 9} =40= *1* *2* *3* *4* *5* *6* *10* *8* *7*)
({1, 3, 4, 5, 6, 7, 8, 9} =47= *4* *5* *6* *7* *1* *2* *3* *9*)
({1, 3, 4, 6, 7} =33= *7* *1* *2* *3* *9* *10* *6*)
({1, 3, 4, 5, 6} =27= *4* *10* *8* *7* *1* *2* *3* *9*)
({8, 4, 6, 7} =21= *7* *1* *8* *2* *9* *10* *4* *6*)
({1, 4, 5} =14= *8* *2* *3* *9* *4* *10*)
({2, 3, 4, 5, 7, 8} =25= *4* *6* *7* *8* *1* *2* *9* *3*)
({1, 2, 3, 4, 5, 7, 8} =33= *1* *2* *3* *4* *6* *7* *8*)
({1, 3, 4, 5} =19= *4* *10* *8* *1* *2* *3* *9*)
({8, 9, 3, 4, 5} =21= *2* *9* *4* *5* *6* *10* *8* *1*)
({1, 3, 4, 5, 6, 7, 8} =38= *7* *1* *2* *3* *9* *4* *6*)
({8, 9, 4, 5, 7} =26= *9* *4* *5* *6* *7* *8* *2*)
({8, 3, 4, 7} =18= *7* *8* *1* *2* *9* *10* *4* *6*)
({8, 7} =11= *7* *8* *10* *4* *6*)
({3, 6, 7} =23= *7* *1* *2* *8* *10* *6*)
({1, 3, 4, 5, 6, 8} =28= *1* *2* *3* *9* *4* *6* *10* *8* *7*)
({3, 4, 5, 6, 7, 8, 9} =39= *5* *6* *7* *1* *2* *9* *4*)
({2, 4, 5} =9= *4* *10* *8* *2* *9* *3*)
({8, 9, 7} =20= *4* *5* *6* *7* *8* *10*)
({8, 9, 7, 4, 1} =30= *10* *4* *5* *6* *7* *8* *2* *3* *9*)
({1, 2, 3, 4, 5, 6, 7} =40= *6* *7* *1* *2* *3* *4* *10*)
({3, 6} =13= *1* *2* *8* *7*)
({1, 2, 3, 4} =18= *2* *3* *4* *9* *10* *8* *1*)
({1, 2} =11= *2* *3* *4* *9*)
({8, 4, 5} =7= *4* *6* *10* *8* *2* *9*)
({1, 2, 4} =13= *8* *2* *3* *4* *9* *10*)
({1, 4, 6, 7} =28= *10* *6* *7* *1* *8* *2* *3* *9*)
({8, 9, 4, 6, 7} =30= *9* *10* *4* *5* *6* *7* *1* *8* *2*)
({1, 2, 3, 4, 6} =26= *8* *7* *1* *2* *3* *4* *9* *10*)
({1, 2, 3, 4, 5, 6} =30= *1* *2* *3* *4* *10* *8* *7*)
({1, 2, 4, 6, 7} =31= *6* *7* *1* *8* *2* *3* *4* *9* *10*)
({3, 4, 5, 6, 8, 9} =29= *1* *2* *9* *4* *5* *6* *10* *8* *7*)
({1, 3, 4, 6} =23= *10* *8* *7* *1* *2* *3* *9*)
({7} =10= *10* *6* *7* *8*)
({2, 4, 5, 7, 8, 9} =29= *4* *5* *6* *7* *8* *2* *9* *3*)
({1, 2, 4, 7} =23= *6* *7* *8* *2* *3* *4* *9* *10*)
({3, 4, 6, 7} =25= *10* *6* *7* *1* *2* *9*)
({1, 2, 3, 4, 6, 7, 8} =37= *7* *1* *2* *3* *4* *9* *10* *4* *6*)
({1, 2, 4, 5} =17= *3* *4* *10* *8* *2*)
({1, 2, 3, 4, 5} =22= *2* *3* *4* *10* *8* *1*)
({2, 3, 4, 5, 6, 7, 8} =33= *7* *1* *2* *9* *3* *4* *6*)
({8, 1, 4, 7} =21= *7* *8* *2* *3* *9* *10* *4* *6*)
({1, 2, 3, 4, 5, 8} =23= *8* *1* *2* *3* *4* *6* *10*)
({8, 5, 6, 7} =23= *6* *7* *1* *8* *10* *9* *4*)
({1, 2, 4, 5, 7, 8} =28= *3* *4* *6* *7* *8* *2*)
({8, 3, 4, 6, 7} =26= *7* *1* *2* *9* *10* *4* *6*)
({8} =1= *6* *10* *4*)
({1, 2, 3, 4, 5, 7, 8, 9} =42= *1* *2* *3* *4* *5* *6* *7* *8*)
({1, 2, 3, 5, 6, 7, 8} =39= *3* *4* *6* *7* *1* *2* *8* *10* *9* *2*)
({8, 2, 5, 7} =18= *3* *4* *6* *7* *8* *10* *9*)
({8, 1, 2, 5, 7} =26= *3* *4* *6* *7* *8* *10* *9* *2*)
({8, 9, 5, 6, 7} =32= *5* *6* *7* *1* *8* *10* *9* *4*)
({1, 4, 5, 7, 8, 9} =34= *3* *9* *4* *5* *6* *7* *8* *2*)
({8, 2, 4, 5, 7} =20= *7* *8* *2* *9* *3* *4* *6*)
({2, 3, 4, 5, 6, 7} =32= *4* *10* *6* *7* *1* *2* *9* *3*)
({4} =2= *10* *8* *2* *9*)
({1, 2, 5} =15= *3* *4* *10* *9* *2*)
({2, 3, 4, 5, 8, 9} =24= *2* *9* *3* *4* *5* *6* *10* *8* *1*)
({1, 2, 3, 4, 5, 7} =32= *1* *2* *3* *4* *10* *6* *7* *8*)
({1, 2, 3, 4, 7} =28= *1* *2* *3* *4* *9* *10* *6* *7* *8*)
({4, 5, 6, 7, 8, 9} =34= *9* *4* *5* *6* *7* *1* *8* *2*)
({1, 4, 5, 6, 7, 8, 9} =42= *4* *5* *6* *7* *1* *8* *2* *3* *9*)
({8, 3, 6, 7} =24= *4* *6* *7* *1* *2* *8* *10*)
({1, 2, 3, 4, 7, 8} =29= *1* *2* *3* *4* *9* *10* *4* *6* *7* *8*)
({8, 4, 5, 6, 7} =25= *6* *7* *1* *8* *2* *9* *4*)
({4, 5} =6= *4* *10* *8* *2* *9*)


Triangular Regions ({id} =value= *vertices*)
count = 22

({8, 6, 7} =19= *7* *1* *8* *10* *4* *6*)
({3, 6} =13= *1* *2* *8* *7*)
({1, 2} =11= *2* *3* *4* *9*)
({6, 7} =18= *10* *6* *7* *1* *8*)
({1, 2, 3, 4, 5} =22= *2* *3* *4* *10* *8* *1*)
({5} =4= *9* *4* *10*)
({3} =5= *1* *2* *8*)
({8, 9, 6, 7} =28= *5* *6* *7* *1* *8* *10* *4*)
({2, 5} =7= *4* *10* *9* *3*)
({9} =9= *5* *6* *4*)
({2} =3= *4* *9* *3*)
({8, 9, 2, 5} =17= *4* *5* *6* *10* *9* *3*)
({1, 2, 3, 4, 5, 6, 7, 8, 9} =50= *1* *2* *3* *4* *5* *6* *7*)
({1, 3, 4, 6, 7} =33= *7* *1* *2* *3* *9* *10* *6*)
({8} =1= *6* *10* *4*)
({1, 3, 4} =15= *2* *3* *9* *10* *8* *1*)
({8, 5} =5= *9* *4* *6* *10*)
({8, 2, 5} =8= *4* *6* *10* *9* *3*)
({3, 4, 5} =11= *8* *1* *2* *9* *4* *10*)
({6} =8= *1* *8* *7*)
({1} =8= *2* *3* *9*)
({4, 5} =6= *4* *10* *8* *2* *9*)


Sum of all the numbers in each triangular region = 301

Process an MBOX file with python

Getting snowed under with email?  Before you go and clear it all out, have a read of this because it may prevent the same thing happening in the future.

I'm on holidays, and one of the first things on my to do list was to clean out my inbox.  From what I've heard of other peoples experiences, my 400 email inbox isn't too bad, but it was getting unmanageable.  It's been that way for a while and I eventually got to a point where I didn't delete anything and just let it build up.  I did that for a reason, I wanted to analyse all of the email to see where it was coming from.  That way I could take appropriate action to change email settings on services, unsubscribe from things, or set up some filters.  By investing some time now, I can reduce my ongoing maintenance.

I wasn't sure how I was going to go about this.  I originally thought that I'd have to use something like a google apps script to retrieve data from my email account, but as it turns out Google finally got their act together and now have a way to download an archive of your email.  They even give you the option to select specific labels to include in the downloadable MBOX file.  This is handy as I only want the emails in the inbox.

Email download process

From this point things were easy.  Python is able to parse the MBOX file and extract the required information from the emails.  The subject and sender emails addresses were extracted and processed with the following procedure.

1.  Split the strings at whitespace
2.  Remove everything except alphanumeric characters
3.  Convert the string to lowercase

All the results were combined and then sorted and counted to give a result similar to the below image.

Email word frequency

It's not perfect, but it gives you a quick way to identify problem areas.  Watch out for Unicode problems though, I think I've taken care of it but it's hard to tell.

Github Gist

An Analysis of the Triple M Brisbane Playlist

When listening to radio in the car I do exactly what I would at home, channel surf.  Generally I listen to Triple J but also switch between Triple M, 4ZZZ, and 612 ABC for something different.  For a couple of months there was a period when every time I switched over to Triple M it seemed that they were playing The Spin Doctors.  Now any normal person would pass that off as mere coincidence and leave it there, but if you've read this blog before you're probably aware that I'm not normal.  I started to wonder exactly what type of songs were played on the station, did they play certain music at different times of the day to cater to a different demographic?  More importantly how often do they actually play The Spin Doctors?  To answer this I needed to log what songs were played, unsurprisingly I chose a solution that required electronics to log the RDS messages the station broadcasts.  The method used was described in a previous post, Using a Raspberry Pi to Log Songs Played on a Radio Station.  They do list recently played songs in their on-line streaming player, but I couldn't figure out a way to easily log them.  Besides, electronics is more fun.

I've since discovered a similar analysis was done by Daniel Nitsche in 2014.

The logger was left to run for two weeks, generating a file with 273902 entries, on average logging what was being played every 4-5 seconds.  Writing software to turn that into something useful was in my opinion going to take longer to do than editing the file by hand as there were intelligent decision to make along the way.  There were small errors that needed correcting as well.  For example, when the daytime presenters played a snippet of the songs to be aired in the next hour, that was also broadcast on the RDS message system.  So it would appear they played the song and then played it again within the hour.  That had to be corrected.  I'm not saying this couldn't be automated, but for a one off, it's not worth it.

After editing, the data showed that there were 2805 songs played over the two week period, about 8 songs an hour.  That sounds about right.  There's probably the occasional error in my data set, but I believe it's accurate enough to show any trends.  There may also be some errors in the release year I've listed for the songs.  Initially I tried to use an on-line music database to get the release date for each song, but that was way to slow and error prone.  In the end I Googled each of the 868 songs to find out when they were released (doesn't take as long as you think, I averaged about 6 a minute).  Then comes the problem of when a song was actually released.  Some songs are released on the album but their release date is when the single comes out.  I choose when they were first publicly available.

So what can we actually learn from this data?  I started with the histogram below to see the distribution of songs played.  The oldest song played was "Sympathy for the Devil" by the Rolling Stones from 1968.  Then there's a  bit of a peak in 1971 before the bulk of songs from about 1980 to 2000.  1991 seems to be a massive year for music and unsurprisingly aligns closely with the active rock format the Triple M subscribes to.  Billboard has a go at explaining 1991.  The bulk of the songs played in 1991 are from the following classic albums.

U2 - Achtung Baby

Red Hot Chili Peppers - Blood Sugar Sex Magik

R.E.M. - Out Of Time

Pearl Jam - Ten

Nirvana - Nevermind

Metallica - Metallica

Guns N' Roses - Use You Illusion I

Guns N' Roses - Use You Illusion II

The next thing to notice is that there aren't many songs from the 2000's, but they do play a lot of recent music from the last 5 years.

Yearly Distribution of Songs Played on Triple M Brisbane

The next plot shows how the release years of songs are distributed over the two week period.  Although you can see the information from the histogram reflected here, there isn't much too much else to see here.

Release Year of Music Played on Triple M Brisbane

The last graph was a bit boring, but by taking the data and showing it differently we can learn some interesting things in the one below.  By plotting all the weekday data on one daily graph, patterns begin to emerge.  For instance, between the hours of 5 and 9 am it appears that there is a blackout on pre 1980 music.  You can also see that songs are less frequently played between 6 and 9 am when the breakfast crew are doing their thing.

When the drive show comes on from 4 to 6 pm you can once again see the drop off in the amount of music played to make way for the presenter.  Between 6 and 7 pm there is a sports show that plays relatively little music as well.  Then between 7 and 9:30 pm the focus seems to be on 80's music, this then changes to 90's music until about midnight.  I assume they have data that says people who like 80's music are in bed by 9:30 pm.

Weekday Daily Distribution of Songs Played on Triple M Brisbane

The data for the weekend shows similar patterns.  After lunch, the sports coverage starts and the music stops.  There's also a relatively dense block of new music between about 6:30 and 8:30 at night, this turns out to be a show on Sunday that plays a lot of new music.

Weekend Daily Distribution of Songs Played on Triple M Brisbane

I also did up a Poincaré plot to see if any other patterns emerged, but the only thing obvious is the predominance of 1991.  If you've never seen one of these before it's a graph of data that plots data point n against (n+1).  They're very useful for finding patterns in binary files as well.

Poincaré plot - Release Year of Songs Played on Triple M Brisbane

So to wrap things up I looked at what the most popular artists were.

Artist	Times Played
Red Hot Chili Peppers	76
INXS	69
Foo Fighters	66
U2	65
Pearl Jam	61
R E M	56
Midnight Oil	55
AC/DC	44
Guns N' Roses	40
Powderfinger	39

I also looked at the most popular songs too.

Song	Artist	Times Played
Georgia	Vance Joy	26
Hold Back The River	James Bray	19
Congregation	Foo Fighters	17
Blame It On Me	George Ezra	17
Every Breaking Wave	U2	17
Crystals	Of Monsters and Men	16
Someone New	Hozier	15
Believe	Mumford & Sons	15
Two Princes	Spin Doctors	14

Ha!!!!!  Proof I'm not insane*.  Two Princes by Spin Doctors is the 9th most played song on Triple M.  Coincidentally released in 1991.

*This blog post probably doesn't help my case.  Hmmm :-/

You can find my original logs here.

Attempting to Determine How Audio Data is Stored in Flash Memory

You may have seen that I've recently been analysing a toy that plays animal sounds.  It's nothing complicated, it just plays one of 108 stored sounds when a collector card is scanned though an optical interface.  For some reason I thought it would be fun to see if I could replace the audio data with my own sounds.  From what I can tell, the data is stored in a 2 MiB flash memory IC on the PCB, I'm 99% sure it doesn't hold program memory.  2 MiB is way too much for such a simple task.  Besides, that amount of memory seems the perfect amount to store the sounds the player uses.

To examine the memory I'd have to get a programming adapter to download the contents of the chip.  I did a quick search of ebay and found the EZP2010 for $40.  I know what you're saying, "there are cheaper options available", and although that's true, this had a few things going for it.  It had a faster delivery, but most importantly it came with ZIF SOIC to DIL adapters, which turn a 30 minute job into a 30 second job.

EZP2010 Memory  Programmer

After stuffing around for an hour trying to set-up drivers, I finally got the programmer installed.  There isn't much to it, but it does the job.  It has the ability to copy chips as a standalone device not connected to a computer, but as I didn't need that I didn't bother testing it.

Device Under test in DIL ZIF socket

To read the memory of the IC it was removed from the PCB with a hot air gun and placed in the SOIC ZIF adapter.  Having these made the task so much easier.

Device Under Test in SOIC ZIF scoket to DIL adapter

There's a bit of confusion over what chip I'm actually trying to read.  The image below indicates that the Chip is a 25L1605D, but the programmer detects it as a 25L1635D.  Both have the same memory capacity and both give the same results when used as a setting to read the data form the memory.

Flash Memory IC

Once you have the software set-up it's idiot proof.  Put the IC into the socket as shown in the diagram, detect device or configure it manually, then hit the read button.

Flash Programmer Software

I tried playing the recovered data as audio by importing it as different types of raw data in Audacity and Goldwave, but each time all I got was static.  It would've been unlikely to get the exact format, but I was hoping for some type of recognisable distortion that would help to reveal the nature of the data to me.  No such luck.

I expect to see an area to tell the device how many sounds are in memory and something like a lookup table to indicate the location of each sound byte.

My goal is to see if I can determine the structure of the data, and as a first quick test I checked out the data using the histogram function of HxD.  As you can see from the image below, apart from the spike in the centre, all the bytes seem to be evenly distributed.  Not what I was wanting to see.  It's not a certainty, but If you see an even distribution of bytes it indicates encryption or compression has been used.  I was a little excited to see the spike in the middle though.

Byte Histogram of recovered Data

That excitement was short lived.  It turns out that there is a large block of unused memory at the end of the file containing the character 0x80

Repeated 0x80 at end of file

To get a better idea of what I'm looking for, I had a look through digikey to find a sound playing IC that could be similar to what's used in this toy.  There's no way to know what device has been used as it's a chip on board device, but chip manufactures like to compete on features, and if it's in one companies IC there's a good chance that it's in the others too.

The cheapest device I found was a ISD3800 chip corder and a quick look at the data sheet gives us some important insights.  It supports the type of memory that our toy uses, and shows some of the audio compression algorithms that could be used.  The algorithms used may not observe byte boundaries i.e. 2,3,4,5,6,7,8, 10, 12 bit samples.

ISD3800 sound player IC data

For more analysis the data was opened in Audacity with the spectrograph view turned on.  There are three distinct features visible here.  Two vertical lines and a gap at the end.

Spectrogram of Data when opened as a raw audio file in Audacity

Zooming in on the waveform at the first vertical line shows a couple of triangular shaped waveforms.

Interesting Section of Data in Audacity

The second vertical line indicates this descending step feature in the waveform.

Interesting Section of Data in Audacity

As seen before, the section at the end of the file is a grouping of the 0x80 byte, in this format interpreted as a zero.

Nothing at end of File

I follow +Oona Räisänen on twitter and have seen how useful baudline can be.  So I gave it a try.  I'm still learning the interface, but it will come in handy for a few other tests I want to run.  While in Linux I tried the data in binwalk but got absolutely nothing.  The entropy plot did show the regions noted above though.

Baudline Interface

The bit view window turned out to be not so helpful,  the poincare plot was all black.  I'm not sure if I used it right, did I over saturate it and it just shows everything  as black.  I might do my own in octave.

Statistical Analysis of Data in Baudline

So where am I at?  I have more of an idea of what I'm looking for, but have no leads.  I have an SPI bus protocol analyser coming that will come in handy.  I can play one sound and record what memory addresses it accesses and what data is returned, for some reason they may not use a sequential addressing system. The rate it does this could also reveal that a variable bit rate compression algorithm was used.

To sum up, I don't like my chances, but it's a fun cat and mouse game.  I'm learning some new techniques, while solving a challenging puzzle.