Open Street Map Transit Time Map

I thought that today I would do a quick post about my Open Street Map project I dabbled with a while back.
The basic concept was to create a travel time map for Brisbane that graphically shows via colour mapping the time taken to get to the city via public transport.  Sounds complicated, well it is.  The more that you look at the problem the harder it becomes.  That aside, the hardest part was finding a cohesive source of timetable information for all the different sources in Brisbane, and that was not going to happen.

Ideally I needed a database of stops, and their location, routes between stops, and timetable data.  All this info is out there, but not any easily accessible format.  The timetable and route information could be scraped of websites, but the GPS data required for the stops would just be a nightmare.  The train stations and ferries aren't that bad, but have you got any idea how many bus stops there are? Neither do I, because there are too many for one person to enter into a database.  I noticed recently the Brisbane City Council had a competition for programmers where they released this data to them.  I think it's meant for an app to find the nearest bus stop and things like that.  Anyhow, that data could be handy.

The other major problem I had is that there is only limited connectivity in the network.  By that I mean most bus stops aren't located in the same position as a train station.  So to make the network complete you need to consider ad hoc walking routes between different stops.  There is no data for this, you need to work these out for yourself.  With a bit of blood sweat and tears they could probably be derived from the OSM data set, but the project was becoming a bit too big for one person.  Don't even mention the behaviour of commuters, I walk about 6 km/h and might catch a train, where my sister walks about 4 km/h and might miss it, and because of the inherent granularity of the timetables, being a minute late to the station may mean you get to your destination 20 minutes later.

I did however manage to get an OpenGL viewer in c++ going for OSM tiles so datasets I was hopefully going to enter could be visualised.  Once that was done I called it a day.  I would still like to finish the project one day, but that's for future Grant, he has better coding skills than me, who just codes out of necessity.  It was starting to become a project that was more suited to a team of people not just one person.

Anyway here is some screen captures I recorded as the project progressed. Captions are from YouTube.

Caption - A quick tour around Brisbane in an OpenGL application.  The program uses a custom rendered set of tiles from the Open Street Map project to provide a 3D navigable map.  It's been developed as a framework for another project I have going.  Few problems, the rendering of the tiles by Mapnik had some issues not rendering certain features, and a limited amount of graphics memory (32MB) which can be seen by the less than ideal quality of the map and occasional bluring of adjacent tiles.  Neither are big problems and are easily solved.  The jittery camera movent is an artifact of the process used for the screen capture.  The tour starts in the CBD, heads toward Sherwood and on to Forest Lake.  The camera then travels towards Brisbane Airport demonstrating some of the camera movements along the way.  Nowhere as good as the other OSM-3D project out there, but I needed something simple that I could build on.  UPDATE - I have since started work on an updated version.  The framework is done just need to fill in the details.  This time I am on a new laptop with 512MB of VRAM, using Windows.  Just to make things a bit nicer I have also added a skybox and some satellite imagery from nearmap.  When rendering the tile set, it was done at twice the resolution so it should be a lot clearer.

 

Caption - This is the framework for the newer version of the program that was mentioned in the last clip. This was just a quick play around to show the improvement. It renders at about 200 fps but should be higher. This is because I am using a fragment shader to warp the satellite imagery to make it more realistic. I am currently changing this to use pre-warped images that can be rendered like any other texture. The clip shows the sky box and the two different speeds that the camera is able to move at. Was having issues finding a way to do the screen capture, but stumbled across FRAPS. This was done as half resolution.

Caption - Not a lot different here. Just testing a higher resolution output. Added a base test map. The satellite imagery is now pre-warped to save some time. Should be at 720p. Use the pop out feature or make it full screen for full effect, and select 720p as the resolution. Still doesn't do it justice like a video card does ;-) It's a LOT smoother. Have the fps nudging 300 again with multisampling. (I know I can't render at 300 fps, it's just for benchmarking) Now I have to get down and dirty with threads and loading the sub textures.

Caption - This is a small update, went through tidied up some code. The first stage on rendering tiles is to change the background image. As you zoom in, the details on the map become finer and the map dimensions double. The lowest detail map is 1024x1024, but the background image for the highest detail would be 524288x524288 which is unmanageable. So these large maps of higher detail have been shrunk down to 1024x1024. This allows the detail to increase but produces a blurry map, which can be seen when zooming in. But, as these are background maps this doesn't matter, the goal is to provide the right level of detail in the distance, the blurry imagery close to the camera will be rendered over by higher resolution maps. The imagery in the distance that can be seen on the zoom out is the goal. As these images are used a lot in the rendering, they are loaded on opening of the program and transferred to the graphics card and stay resident for the duration of the program. I also added a nifty little cross hair.

Caption - This is the final result for now. The project is basically a small program to view GIS data in 3D. Higher details tiles swap in and out and work to an extent, but could be done better. The data set loaded is the city train network that was downloaded from the OSM data set and simplified. The network is elevated in this clip. A tour from Ipswich to the city is shown. Now that the program is now at a stage I can use to visualise and check my data sets, I can move onto the next stage. Onward and upward.

Making a DVD with a Menu in Linux

As a follow up to my previous post about transfering VHS tapes to computer, I thought I would show the next part of the process, authoring your own DVD's with menus.

Before doing anything, I had to go through my video files with Virtual Dub and set the frame rate to 25 fps and resync the audio and video.  The sync problem was caused mainly by dropped frames when I did the initial capture.  Because the video was poor quality in a few places, the capture card couldn't record a frame.  This is a dropped frame.  The audio however doesn't come with that problem.  Because the audio is just an analog signal with no structure, it is just recorded as is.  As time goes on, the audio starts to lag the video.  Most of the videos only had a couple of dropped frames, and by luck were only short videos, so the problem wasn't evident.  A couple of clips required a bit more attention.  All I had to do was copy some frames and insert them back into the movie in the static to lengthen the vision and re-sync it with the audio.  Now onto authoring the DVD.

After looking around for a while I settled on Bombono DVD, which is available for Windows and Linux.  At first I tried to use it in Windows, but I didn't really get anywhere, so I just went back to the Linux version.

The interface is pretty simple, you start by adding media on the "Source" tab.  This can be audio, still images or video.  After all media is added, switch to the Menu tab.  From here you can create all the different menus you want and create links to media items and other menus.  The links can be either text, static images or movies.  I did have a little trouble lining up my links and images but I just altered the resulting xml file by hand.

After you set all this up, go to the output tab and select the output method.  I choose to have a disc image created that I could then burn to DVD later.

I had one problem though.  The image quality was pretty crap.  This was because Bombono had to transcode my video files, and without knowing more about them, all it could do was use the default options.  So by doing the transcoding myself I could tailor the options to my video.  This was pretty easy with WinFF.  Set "convert to" to DVD, and "preset" to PAL DVD HQ Fullscreen.  To get the best quality and specify that the video was interlaced, the following line is added to the FFmpeg tab.

-flags +ilme+ildct+alt -top 1

Once the AVI's had been transcoded to MPEG's and used as the media in the Bombono project, the output was perfect.  You may want something a little more flexible and powerful, but for something free it was perfect for my use and I highly recommend it.

Pilfered Power

Sometimes when you look at something you just know it's wrong.  That happened recently when I looked at my sisters power bill.  It was an average of 15 kWh a day which is an average load of 625 Watt, which is not high for some places, but for her house it seemed ridiculous.  For a two person apartment with only the fridge, microwave, and air-conditioner plugged in continuously, something was afoot.

So I went through and recorded the power use of every device I could, either by reading the specs or measuring it myself.  The TV and associated devices were turned off at the wall when not used.  For the 5 hours a day average that they were on, they only drew 70 Watt, giving a daily average of 0.35 kWh.  With the air-conditioner never used and only drawing a 3 Watt standby that only accounts for another 0.072 kWh.  The 1000 Watt microwave was only used for about 15 minutes a day average, giving another 0.25 kWh.  From the specs, the fridge was using only 1.5 kWh a day and her use of the fridge was what I considered average, not leaving the door open too long or opening it too often.  The washing machine averaged a load a day, which was 0.6 kWh. Don't forget the dishwasher, it used 1.5 kWh for its daily load.  These were the only devices that were plugged in at the time and adding it up, they only account for about 4.3 kWh a day.

So there is still another 10.7 kWh a day to account for.  Ahh, the oven and stove I thought, had they been using that a lot?  No, their use would average out to about half an hour a day, at about 4 kW of power, giving another 2 kWh, still 8.7 kWh from the target.

I had another look at the power bill and noticed that there was only one tariff, which meant that the electric hot water system had to be included as well.  I checked the thermostat and it was already set to the lowest legal temperature, so I tried to find out how much power it was using.  After digging around in the Rheem website I found a table which listed the maintenance energy of water heaters.  This is the energy that the system uses to keep the water at a temperature when no water is being used, ie the waste heat.  An approximate value for this is 2 kWh a day.  Now lets say they were to use 50 Litres of hot water a day, which is heated from 20 degrees Celsius to 65 degrees Celsius.  That would require 9.4 MJ or 2.61 kWh.  So, getting closer, with the total 4.61 kWh from the hot water system, there is only about 4.1 kWh to account for.  However, that was it, there were no other power sinks.

This accounted for about three quarters of the power, but what about the rest?  Maybe my measurements and estimates were out a little, but I would have to add 35 percent to my measurements to get what was on the bill.  Not impossible, but my readings did seem a long way off.  So I started taking regular meter readings to get some more data.  That's when things got interesting.  One day when no-one was there, with no-one taking a shower, no cooking, no washing, and no tv, the meter recoreded an energy use of 19 kWh.  WHAT! There had to be something wrong.  I wondered if the hot water system was leaking in to the overflow pipe, or if a pipe had a leak somewhere.  To find out if this was the case, I decided to turn off the hot water for a day and take another measurement.

Being an apartment building, the switch box had about 20 fuses for 4 or 5 apartments.  After looking around I found and turned off what I thought to be the hot water system for my sisters apartment.  Before doing this I checked the voltage at the hot water system with a voltage detector (I love this thing, It's the closest I'll ever get to being a Time Lord and having a sonic screwdriver).  I then checked again afterwards to see if I had turned off the right fuse, which I had.

Crescent Voltage Detector

The next step was to check if for some reason the fridge was on the same circuit.  Leaving the fridge without power for 24 hours would be a really bad thing.  However, while doing this the neighbours appeared at the switch box.  Half their power points were dead.  Bingo.  Turns out that my sister had been supplying power to the neighbours for a few of their appliances, but mainly the heater they were using at night during winter.

After a call to an electrician everything was sorted.  He was at first sceptical, but after testing everything himself he came to the same conclusion.  The solution was easy, move the offending circuit over to the neighbours meter and run a new line to the hot water system.

There's no way to tell who did this or when it happened.  It could have been a dodgy previous tenant or a mistake during installation.  Paying for more than you should is annoying, but more importantly this was a dangerous situation.  Let's say that a tradey comes to replace the hot water system and turns the breaker off, the neighbours think that a breaker has tripped, so they go and turn it back on, the tradey is now working with live wires but they don't know it.  Or an electrician could come to work in the neighbours apartment, shut off the power and not realise that the power points are still live because they are connected to a different apartment.  A seemingly innocuous problem could end with someone being seriously injured or worse.

After it was fixed, the meter readings came back to 10 kWh a day, which is close to what I had calculated from my measurements and estimations.  So, if something doesn't feel right, trust your gut feeling and either find out why your wrong or find out what is wrong with the situation.  It could be more important than you think.

Fluorescent Starter

I recently changed a fluoro starter at my sisters place and was amazed to see it was made in Costa Rica.  Don't get me wrong, I have nothing against Costa Rica, but I had never seen anything made there.  I suppose that I have become accustomed to everything electronic being made in China.  Anyway, that aside, it got me thinking, I must have changed 100 or so of these things in my life and never even wondered how they work.  So I thought I crack it open and have a look.

Fluorescent Starter Innards

I don't know much about them, but I know the newer ones are more complicated and have more electronics, but given the obvious age of this one it had to be the older kind, that I now know is called a glow starter.  There isn't much to it, just a capacitor and what looks to be some sort of neon light.

Schematic

The schematic above give a clear view of how the lamp (A), ballast (G), and starter (C) are all connected, and makes things a bit clearer.  To start with there is no power in the system, then power is applied to the terminals at (B), but the there are two problems, there is no current able to flow through the filaments (F) at each end to "boil" off electrons and create a conduction path for the arc, also there is also not a high enough voltage across the tube to strike an arc.  This is where the little glass tube (D) plays its part.  The glass tube is actually a gas discharge lamp with a bimetallic switch in it.  The full mains voltage is applied across the neon tube, via the filaments and ballast, and causes it to glow and give off heat.  This in turn causes the bimetallic switch to close which allows current to flow thought the filaments and preheat them, causing electrons to be liberated.  Because the bimetallic switch is now closed, the heat associated with the neon glow is no longer there, which then causes the switch to reopen. This causes a rapid change in the current flowing thought the inductive ballast which causes a voltage spike (remember V= L di/dt ?).  This voltage spike is large enough to strike an arc in the tube, turning the light on.

Once the tube is started, everything takes care of itself, the discharge in the fluorescent tube is enough to keep electrons liberated, and the voltage across the operating tube, which is in parallel with the neon tube, is low enough to prevent the tube glowing and re-closing the bimetallic switch.  The capacitor (E) in the starter doesn't have much to do really, it is there as a snubber to protect the bimetallic contacts from arcing when they open.  By doing this, the life of the contacts is extended.

This is a basic look at how a starter works.  Most starters and ballasts are now electronic, and therefore more complicated, but the basic purpose of a starter remains the same.  Firstly, preheat the filaments, and then create a voltage spike for ignition of the lamp.

Finding Duplicate Files With Linux

Recently I needed to to clean up an external USB drive and remove duplicate files.  Since I run Windows, things like this were a pain in the arse.  That changed about two years ago when I started using a virtual Ubuntu machine on my laptop.  It gives you the best of both worlds, things that are annoying on one OS are a breeze on the other.  This was one of those times I knew that Linux would be perfect.

So after a bit of searching I  found the following command by syssyphus over at www.commandlinefu.com.

find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate

Let's break it down and take a closer look at it.  It's made up of a number of commands strung together with pipes, the first being a find command that lists the size of all non empty files in the current directory and its subdirectories. The -printf "%s\n" prints the file size followed by a newline.  Next sort -rn orders that output numerically in reverse order.  If there are duplicate files, the sizes are now beside each other in the output. If you then run uniq -d all unique file sizes are removed from the output and a list of duplicates remains, because if a file has a unique file size, it can't be a duplicate.

That's the easy part, you're now left with a unique list of file sizes that could be duplicates.  The next command, xargs, goes through this list one at a time and uses find to locate all files that size. By using -print0 a null terminator is used to separate the files that are found (this helps if there are unusual characters in the file name).  xargs then runs the files through md5sum to produce an output of a hash value followed by the file name.  The -0 option is used to tell xargs that the incoming list is null terminated.  The list is then sorted and sent to uniq.  As the hash only takes up the first 32 characters of the line, the -w32 option is used to compare lines based only on the hash. The --all-repeated=separate option removes all unique hashes and groups all duplicates together separated by blank lines.

A pretty elegant solution that only fails if you have two files the same size and you get a hash collision between them.  I think there may also be a slight problem in the last sort command if there are newline characters in the file name, but all of these problems are pretty uncommon and since the command doesn't delete anything it doesn't really matter.

One problem I ran into though was that I had about 2000 different file sizes that were listed after the first find command.  This means that xargs has to run the next find command 2000 times and it was taking a while.  So instead of re-finding the files I managed to alter the command to keep the file names from the first find.

find -not -empty -type f -printf "%s %p\n" | awk '{if (x[$1]) { xc[$1]++; print $0; if (xc[$1] == 1) {print x[$1]}} x[$1] = $0}' | sed 's/^[0-9 ]*//' | xargs -d '\n' md5sum | sort | uniq -w32 --all-repeated=separate

The first find command is basically the same as before except it prints the file name after the size.  awk is then used to remove unique file size entries. The if (x[$1]) checks if anything is in the associative array using the first field $1 (file size) as an index.  If nothing is in the array at this index the entire line is then added to this location x[$1] = $0, nothing is printed.

If something is in the array at $1, we now have a duplicate file size.  A counter array is then incremented and the line is printed.  If this is the first time a duplicate has been found the original line that was used for comparison can be printed from the x array, as it is also a duplicate.  This is accomplished by the if (xc[$1] == 1) {print x[$1]} section of the command.   Note that the incoming list of file sizes and names doesn't need to be ordered for this to work.  What is left now is a list of files that have other files the same size, basically unique file sizes removed.

sed is now used to strip off the leading spaces and numbers, ie the file size, this just leaves the file name.  Once again xargs then sends the file names through md5sum, but this time a new line character is used as the delimiter.  As before the output is sorted and unique hashes removed.

Doing it this way means that the find command is run only once, and gave me a fair bit of a speed up.  However, because awk is a line based tool, any files with newlines characters will cause this to fall over.  It was more or less just an intellectual exercise to see if it could be done.

The outputs of these commands were then redirected to a file for checking.  The easiest way I found was to go though the file and place the letter x at the start of any line that contained a file you want deleted.  Then just run the following commands.

infile.txt | grep '^x' | cut -c 36- > outfile.txt

xargs -d '\n' rm < outfile.txt

The first command initially removes lines from the list that don't start with an x, it then cuts 36 characters from the start of the line.  This is 32 for the hash, 1 for the x, and 3 for spaces, leaving only the file name. The second command redirects the file to xargs, and using a newline character as a delimiter then sends it to rm to delete the file.

There are programs like fdupes to do this for you, but I was after a little more control, enabling me to do a little bit of the processing at a time, because at pretty much any point you can break the command up and redirect it to a file for later processing.

It must be said though that if you intend to do anything like this, TEST, TEST, and TEST.  Oh, did I mention TEST.  One of the commands used on your system may be slightly different than mine and you could end up losing a lot of data.