Just a quick post today. I've been working on a piece of python software that's tkinter based that requires images to be displayed. My preferred way of doing this is to use a tkinter.ttk.Label to hold images, but they're aren't very flexible. To overcome this I wrote a subclass of the tkinter.ttk.Label called TkLabelImage. It allows images to be easily loaded and displayed by labels while controlling their quality and behaviour when resized.
In the example below a TkLabelImage object is created, it's image behaviour is set, and the file name is specified.
The first line creates a TkLabelImage object that's a child widget of self.frame1. After that, the behaviour is set on the next line. The first argument refers to the image quality desired when the image is resized. 'low' will resize the image without anti-aliasing, 'high' will use anti-aliasing, and dynamic will resize the image without anti-aliasing, and after a delay in milliseconds defined by the third argument will do a final anti-aliased refresh. This means that windows can be resized smoothly by only rendering lower quality images when the resize is occurring, but as soon as it's complete a higher quality image is refreshed. The second argument specifies if the aspect ratio of the image should be preserved. When the configure event of the main program is called, the following method is called to refresh the image.
self.label1.fill()
The image below is a demonstration of this. The window is split into two frames that each hold a label. Each label displays an image. The first has aspect lock on while the second doesn't.
Comapre the effects of aspect lock
The window below shows the effect of anti-aliasing on the image. The first image uses it while the second doesn't.
Compare the effects of anti-aliasing
I am by no means a python expert I'm sure there are many things wrong about how I've implemented my solution. It works for me though. I'd love if someone who was more skilled took the idea and ran with it.
OK, I'm revisiting my post from last week. I was certain that I could reduce the time it took to enter data with some simple modifications. The results are at the end. The main change was the removal of the suggestion frame from the right and its replacement with a suggestion label positioned under the box that's currently in focus. Take a look at the previous version to see the difference. Another change was that previously pressing enter while in an entry box had no effect. Now it selects the first suggestion and moves focus to the next box down.
These changes reduce the number of keypresses and the amount of scanning that your eyes need to do of the on screen data. I was also feeling a little nostalgic, so I've chosen some 4-bit colours for different elements. :-)
Data entry with suggestions
The window is broken into two different frames, divided at the blue line, one on the left for the image to process, and one on the right for the entry boxes and the status label. The second frame in arranged into 11 rows, divided at the red lines, with the grid manager.
Arrangement of input window
The hard bit however was getting the suggestion label to be placed under the entry box. First of all, a few things need to be discussed in the image below.
The frame is divided into 3 columns indicated by the blue lines. This allows small coloured indicators besides the entry box to indicate which one has focus. This is done because ttk entry boxes don't allow the background colour to be changed. The small coloured regions are fixed size and the entry box is allowed to resize to fill the rest of the space.
The next step isn't as simple as placing a label under the input box. First a frame needs to be inserted with the place manager that's underneath the entry row and is the same width. This frame is outlined in yellow.
I hear you say that's weird, why not just add the label under the entry box, why use a frame? It's because of how tkinter handles long lines in labels. I construct a string of text to add to the label that contains carriage return characters (\n). This usually works for small lines, but for long lines, the string is cropped to make it fit the label. It doesn't realise that the carriage return characters start a new line. Fox example, the string
really long text part one \n really long text part two \n really long text part three
would display correctly on three lines if the label was wide enough. However if the label was only wide enough to display 4 characters, it would display "real" once and then crop the rest of the text before it gets to the carriage return characters. What you probably expected to see was:
real
real
real
The way to get around this is to create a frame of the correct width and then place a label in it that has no set width. The label will then resize to a width to fit all the text, while the frame acts to crop the text limiting what's shown.
Suggestion label
So did the entry speed increase? Yes it did. In my last post I noted that typing the text with no assistance took about 90s per image. My first attempt took 1:25:41 for 76 images or 67s per image. This new arrangement allowed for a time of 1:07:26 for 76 images or 53s per image. I'm pretty happy with that. That's a 40% decrease in the input time from unaided and a 20% decrease from the initial attempt. Although I'm likely to get faster the more I type the data, the tests were a week apart so I think this effect is minimal.
Can software help a human process images faster than a computer when you take into account the time setting up the image processing pipeline? That's the intent of "SoylentOCR - It's made out of people". It's a python-tkinter based program that can help with small data entry or image classifying tasks.
For the last few years several people have tried to predict the results of the Triple J Hottest 100 song poll by analysing images of votes placed on social media. Their predictions are usually pretty good, but when they do get it wrong it's because of some quirk in the optical character recognition (OCR) process. That's not a criticism, OCR is hard. What's that I hear you say? OCR is a solved problem. Yeah, it kind of is, but it all depends on the quality of the source images, the accuracy required, and how you train the software and post process the results. This is definitely a case where the quality of the source images is less than ideal as demonstrated below.
4 different sample ballots found online
In this situation, if you had to process one million images, you'd probably spend time training your software to recognise particular fonts, and writing post processing software to make sure the results are accurate. If you had one image to process you'd probably just type the results in manually. My points is that somewhere between these two extreme cases there is a cross over point. I don't know where that is, but I'd like to explore improving the manual entry option. One of my favourite XKCD comics illustrates this point. Sometimes spending time to make a task more efficient doesn't make sense. It's quicker in the long run to just do it inefficiently.
To improve the process I came up with a program that displays the images in a frame and allows data entry in multiple entry boxes in a second frame. As you type, suggestions for what you are trying to enter appear in the third frame. To move forward and backward though the images to be processed you use tab and shift-tab. Moving between entry boxes is done with the up and down keys. A status box also shows your progress.
When entering data into an entry box, it's split into words at spaces. The first 10 entries containing all of these words are displayed as suggestions. One of these can be selected to fill the entry box by pressing the control key along with the number of the suggestion. You can see in the image below that typing in "el in pa" is enough to make "paper kites the electric indigo" the top suggestion.
soylentOCR user interface
When first started, the database
contains no suggestions at all, but these are slowly built up when data
is entered. Any entry that is not a suggestion becomes ones when you
move to the next image. To further improve performance, the suggestions
are in the order of the most common ones previously entered. Although
it's possible to start with a completely empty database, it helps to "prime" it, by adding a list of suggestions from elsewhere
in a database editor. These can be ignored in the final analysis. The
sqlite3 database contains one table with columns titled:
FILE_NAME, ATTRIBUTE_NUMBER, ATTRIBUTE
There
is a combined primary key over the FILE_NAME and ATTRIBUTE_NUMBER
columns. To help my example I found a list of eligible songs, converted
it to lower case, replaced any character that wasn't a letter, number,
or space with a space (Notepad++ is awesome). I also added approximately 20 songs from a
betting website that were predicted to win. This means that these songs
are now already in the database twice and appear at the top of the
suggestion list. This is done as they are most likely to appear when
entering data. A set of training data that complies with the table format was created in a spreadsheet, saved as a CSV file, and imported into the database. It sounds complicated but it isn't.
Training data imported into SqliteBrowser
The code isn't perfect. For example the directory containing the images is hard coded into the software. It also treats all files in that directory as images. When it encounters one it can't open it shows a red square and places a notification in the status box.
Not an image notification
All that's left now is to sit down and try it out.
To process the 76 images in that directory it took me 1:25:41. That averages about 67 seconds per image. Not as good as I was hoping. Earlier tests of just typing the data in notepad gave results of about 90 seconds per image. So yeah, it's an improvement but not much of one. I did however get a better feel for the problem and have some ideas about how to improve the software.
First of all, 10 suggestions are too many. It turned out to be much easier to type until there were only one or two suggestions and then select one. The other issue was that most of my time was wasted while my eyes were going back and forth between entry and suggestion. My original intent was to have the suggestions appear under the entry boxes, just like Google does when you enter a search term. This means you only need to look in one area. At the time I was a beginner with tkinter and had no idea how to do what I wanted. I think I may know now so I'll give that a try.
To make the results more rigorous this would be better implemented as some sort of web app. Multiple people could log in and be assigned images to process using a shared database. Each image could be processed at least twice by different people and the results compared. If they don't match there's an error
So overall I'm happy with the result. I learnt tkinter, and did achieve a reduction in the time required to enter the data from an image. Unexpectedly it became clear that the program could be used for other purposes. Imagine you were doing some landscaping and you wanted to choose plants for a garden. If you had images of all the plants it would be trivial to go through and rate them 1-10. This is a task that a computer just couldn't do because it's your own personal opinion.
