That is all.
Well, the Hottest 100 happened yesterday, so now it’s time to evaluate how my spoilers list went:
- 147 songs were given a 1% or higher chance of appearing in the Hottest 100 by my bootstrap analysis. 98 of those 147 songs made the Hottest 100.
- 85 of the songs in the Spoiled Top 100 made the Hottest 100.
- The top 72 songs in the spoiled list made the Hottest 100.
- 6 of the 10 songs in the Spoiled Top 10 made the Hottest 100′s Top 10.
- #1 was correctly predicted.
Why wasn’t it more accurate, or as successful as last year?
- We don’t know how good the OCR model was at transcribing votes. We’ll need to do work fitting our OCR model to known transcriptions if we want to do this again next year.
- The sample size was much lower.
- Daft Punk fans don’t post to Instagram enough (inherent bias)?
- The bootstrap resampling technique did not account for taste: particular songs often being selected together.
I’ve got every confidence that this method will be viable for next year, especially since the results were a much less spectacular spoiler than 2012′s Warmest 100. Let’s see if we can make a better model for next year.
You’re probably familiar with Triple J’s Hottest 100. It’s the world’s largest write-in music poll. Last year, Triple J made an easy, shareable link for people to post their votes out on Twitter and Facebook. Alas, these links were easy scraped from the web, and the Warmest 100 (link to 2012 count) was born. The top 10 (but not its order) was revealed, and the top three was guessed perfectly.
This year, voters weren’t given a shareable link, but a few thousand people took photos of their confirmation e-mails and posted them to Instagram. With a tiny bit of OCR work, the Warmest 100 guys posted their predictions for this year. They found about half the number of votes that they did last year through the scraping method, which is no mean feat, given the lack of indexing.
So the question is — how useful are these votes in predicting the Hottest 100? What songs can we be sure will be in the Hottest 100? How certain is that top 10?
Both years, Justin Warren independently replicated their countdown (spoilers), and has written up his methodology for collecting the votes this year. I asked him for his data to do some analysis, happily, he obliged.
He’s since updated his method, and his counts, and written those up, too (spoilers).
Update: he’s updated his method *again* based on some feedback I offered, and has also written that up (spoilers). This is the data my final visualisation runs off.
So, what have I done with the data?
When you have a sample — a small number of votes — from the entire count, you can’t really be certain where each song will actually appear in the count.
In this case, Justin’s data collected 17,000 votes from an estimated 1.5 million votes. That’s a sample of 0.1% of the total estimated vote. It’s a sample, but we have no idea how that compares to the actual count.
If we think that the votes that we have is a representative sample of all of the votes, then what we’d like to know is what would happen if we scale this sample up to the entire count. Where will songs end up if there’s a slight inaccuracy in our sample?
The good news is that computers give us a way to figure that out!
Bootstrap analysis (due to Effron) is a statistical technique that looks at a sample of votes from the whole set of votes, and randomly generates a new set of votes, with about as many votes as the original sample. The trick is that you weight each song by the amount of votes it received in the sample. This means that songs are picked in roughly the same proportion as they appear in the sample. The random sampling based on this weighted data adds noise.
You can think of this sample as a “noisy” version of the original sample. That is, it will be a version of the original sample, but with slight variation.
If you repeat this sampling process several thousand times, and rank the songs each time, you can get a feel for where each song could appear in the rankings.
How do you do that? Well, you can look at all of the rankings a given song gets for each randomised set. Sort this list, and pick the middle 98% of them. Based on that middle 98% of rankings, you can be 98% certain that the song will be at one of those positions. In statistics, this middle 98% is called the 98% confidence interval by bootstrap.
You can repeat this for different confidence levels, by picking a different amount of rankings around the middle.
I’ve used Google Spreadsheets to visualise these confidence intervals. The lightest blues are the 99% confidence intervals. The darkest blue intervals are the 70% confidence interval. The darkest single cell is the median — i.e. the middle of all of the rankings that we collected for that song in the bootstrap process.
The visualisation is up on Google Docs. (spoilers, etc).
I’ve run the same visualisation on Justin’s 2012 data, it’s less of a spoiler than the 2013 version if you care about that. It can inform the rest of the article for you.
First up, a bit on my methodology: Justin’s data didn’t separate votes into their original ballots. So, I had to pick songs individually. To improve accuracy, I selected songs in blocks of 10, where each song in the block of 10 is different — this vaguely resembles the actual voting process.
In my experiments, I ran the sampling and ranking process 10,000 times.
You’ll notice some interesting trends in this visualisation. The first one is that the higher the song is in the countdown, the narrower its blue interval is. Why is this so?
Well, as songs get more popular, the distance between each song in the number of votes received grows. In Justin’s sample of the votes, #100 and #73 were separated by 15 votes. So if one or two votes changed between #73 and #100, that ordering could change spectacularly. Given Justin’s sample is of 17,000 votes, 15 votes represents an 0.1% change in the vote.
So at those low rankings, a tiny change in votes can make for a massive difference in ranking.
At the other end of the count, #1 and #2 are separated by 16 votes. #3 and #4 are separated by 22 votes. #4 and #5 are separated by 51 votes. Down the bottom of the list, where 16 votes could move a song 33 places in our count, you’d need 16 votes to change just to swap positions 1 and 2.
What this means from a statistical perspective is that the closer to the top you are, the more work you need to do to change your position in the count.
You’ll also see this phenomenon in the right-hand side of the intervals. The interval of a given colour on the right-hand side of the interval will generally be longer than the same colour on the left. Once again, this is because lower ranks swap around more easily than higher ranks.
Update: Since writing this article, I ran one more test – how many of the songs in the top 100 of Justin’s raw sample of votes will make it into the actual Hottest 100? Well, bootstrapping helps us here too. For each bootstrap trial, I take the top 100 songs, and see how many of those are in the raw top 100. I reckon, with 98% confidence, that we’ll get 91 songs in the actual Hottest 100. Thanks to David Quach for the suggestion.
In summary: the Warmest 100 approach is statistically a very good indicator of the top 4 songs. The top 4 is almost certainly correct (except that 1&2 and 3&4 might swap around between themselves). Everything up to #7 will probably be in the top 10.
The sampling approach is less accurate at the bottom, but I’m pretty confident everything in the top 70 will be in the actual top 100.
I’m also pretty confident that 91 of the songs in the raw top 100 will appear in the actual top 100.
I’ll be making some notes on how these confidence intervals got borne out in the actual count on Monday. I’m very interested to see how this analysis gives us a better idea of how accurate the Warmest 100 actually is.
The second of my DroidCon India talks introduces developers of mobile apps with the difficulties of designing for mobile networks. It also contains a series of design ideas that developers can take back to their back-end development team, so that the APIs that they produce for accessing their services are less difficult to use in a mobile context.
My first talk from DroidCon India 2013 (November, Bangalore). It’s an exploration into the approach that we’ve taken at AsdeqDocs in producing a properly cross-platform mobile app. We take the approach of separating our core application logic into a C++ codebase, and apply platform-specific user interfaces over that codebase.
This talk covers the software engineering principles that make that work; as well as the benefits, difficulties, and insights that we’ve learned over a few years of doing this. It’s probably the favourite of my mobile dev talks.
Very pleased to say that I’ll, once again, be running an Open Programming Miniconf at Linux.conf.au in January. This time around, the conference will be at the University of Western Australia in Perth.
I’m especially pleased, because after initially being rejected by the conference team, with limited time to assemble a line-up, I’ve put together what I think is the best Programming miniconf lineup in the five years I’ve been running it.
One of the goals of the Open Programming Miniconf is to be a forum for developers to share their craft: ideas for improving the way people code, and topics that are of benefit to people who develop using many open source programming languages. This year, for the first time, I think we’ve filled that remit.
This year’s talks cover everything from low-level mobile programming and driver development, to deployment of web applications, as well as talks about packaging, deployment, and development tools.
We also don’t have a single state-of-the-language talk. Everything’s about topics that can be transferred to any number of languages.
I’m excited! If you’re interested in the miniconf, check out our schedule and all of our abstracts at the conference wiki. See you in Perth!
… Once again, I’ve completely failed to document my travels this year. I need to do better. Here’s my first attempt.
I’m off to Bangalore, India tomorrow to join in with DroidCon India 2013! I’m presenting two talks, and being a panelist on a panel:
- Portable Logic/Native UI – Explores the multi-platform app structure that we use on AsdeqDocs; things I wish we’d known at the start of the project, and things that we’ve learned doing it.
- Panel: The State of Android Development in India – I’m on this panel as an international voice amongst a table of locals. It’ll be interesting to see how my perspective gels with the other panelists.
- Making Mobile Web Services That Don’t Suck – A talk that covers everything a mobile dev needs to know to understand how mobile networks work, and how to work with their back-end team to make an API that doesn’t suck.
I’ll post back here with slides and videos as they become available.
I’ll be in Bangalore until late on Saturday, then coming home via Singapore for a few days. Should be fun!
Lifehacker regularly features a segment where they interview famous people and ask them how they work. Rather than waiting for the e-mail that would never come, my friend Jack Scott decided to answer this set of questions on his own last week, and tapped me to answer them after him. So here’s my answers.
Location: Hobart, Australia
Current gig: By who pays me: Software Developer at Asdeq Labs. By what I love to do: Open Source Community person; general developer conference raconteur.
Current mobile device: Nexus 5 & Nexus 7.
Current computer: The one I directly use? That’d be a 2013-era MacBook Air; 13″ screen, with all of the extra trimmings.
One word that best describes how you work:
(But, if I’m actually passionate about something, that word might well be “obsessively”.)
What apps/software/tools can’t you live without?
Python. It’s what I go to every day when I need to quickly bash out some proof-of-concept code or make some calculations. Even if I don’t use Python in my day job, Python prototypes will often form the genesis of production code I write in another language. Surprisingly often.
Also: Keynote. Or at least version 5 of it, I haven’t tried Version 6 yet. It makes making presentations easy, and I seem to be doing a fair bit of that at the moment. It’s probably the one piece of software that keeps me tied to Mac OS X.
What’s your workplace like?
At work, I have a pretty generic veneered flat-pack style desk, with a 24″ monitor, and a laptop stand so I can put my laptop’s screen parallel with my larger monitor. I also have a Microsoft split keyboard, which I still can’t use properly. If I were planning my own office, I’d probably have an Aeron chair. But I’m not (at the moment, anyway), so I won’t
At home, I’ll sit wherever feels most comfortable to do whatever it is I need to do. Often that seems to be bed, just because I’m writing stuff, and it seems like a good place to do it.
What’s your best time-saving trick/life hack?
If you’re travelling for more than 4 hours, learn to sleep on planes, and fly at night. Waking up in another city is cool, and having a whole extra day to do things on a trip is like generating extra time for free. It’s a productive use of sleep time!
What’s your favourite to-do list manager?
Honestly, I tend not to use them. I’m generally across most of what I have to do in a day. If I have deadlines, I’ll shove them in a calendar. Otherwise, meh.
Besides your phone and computer, what gadget can’t you live without?
A coffee maker. I like coffee of high quality. I have a rather nice espresso machine, which is the high-end model of a low-end brand; when I’m travelling, I carry an AeroPress and Hario Slim grinder, with a supply of high-quality beans. It saves me money, and I don’t complain about the coffee being awful when I’m somewhere I’m unfamiliar with!
What everyday thing are you better at than anyone else? What’s your secret?
It seems to be remembering things. No secret, I just do it. Brains are weird like that.
On a completely different note, I have absolutely no natural pre-conception of how good other people are at things I know how to do. I’ve found that getting good at presenting technical material is great for figuring out what people need to know to know something (ask me about this sometime).
What do you listen to while you work?
If I’m in at the office, not very much. I hate music getting interrupted, so I’ll take my headphones off the moment I sit down.
If I’m at home, and I’m listening to music, pretty much anything in my library. Right now it’s jumping between Rumours by Fleetwood Mac, and Dear Miss Lonelyhearts by Cold War Kids. But that could change any moment.
What are you currently reading?
Python documentation. AppleScript documentation. Mostly so I can figure out how to implement features in my side project (Keynote-as-a-service). More generally it’s things on Wikipedia. I like to know things. Then I can remember them.
Are you more of an introvert or extrovert?
Though, introverts tend to think I’m extroverted. Probably because I can talk to a crowd if I need to. Needless to say, that’s a completely different skill to actually talking to people one-on-one, which I still have no idea how to do.
What’s your sleep routine like?
Pretty regular. I go to sleep sometime between 22:00 and 23:30, and wake up, just before my alarm does, before 7:00. I wake up with disturbing regularity.
Fill in the blank. I’d love to see _____ answer these same questions.
What’s the best advice you’ve ever received?
Life’s too short for bad coffee.
If you don’t like coffee, substitute this for something else you actually like.
Basically, if you’re going out of your way to find something mediocre, or not as good as you can find in the general area, you’re wasting your time. Don’t do it. Be exceptional, and expose yourself to people who are great at what they do. You’ll almost always find some way to apply it to whatever you do.
And yes. Speaking with people who know how to make coffee properly has helped me be a better programmer
Is there anything else you want to add for readers?
Not particularly. I prefer responding to stimulus than coming up with ideas out of thin air.
Errm, so if you want to get an idea from me in the future, ask me something direct, and don’t ask for open-ended ideas.
I love Apple’s presentation tool, Keynote. In fact, if I had to nominate a single piece of software that was keeping me using Mac OS X, Keynote would be it. I haven’t yet found another tool that lets me throw together great-looking slides as quickly as keynote does.
On the other hand, I also really like using Android. And this is a problem, because Apple’s Keynote Remote app only works on iOS. Keynote Remote is an app that allows you to remote control Keynote from your phone. It also sends down a screen preview, presenter notes, and it also allows you to peek ahead to your next slide. Basically, it’s a killer app for people who want to step out from behind the lectern, and still have their notes and be aware of where they’re up to in their presentation.
And it only runs on iOS.
So this is where I introduce my new project: KAAS, or “Keynote-as-a-Service” is a Python-based HTTP server that lives on the same laptop as you’re presenting from, and exposes a JSON API for doing everything that Keynote Remote does, and potentially more. It’s Apache 2.0-licensed, and it already has a reasonable amount of documentation (though it could use a whole lot more).
I’ve thrown together a basic HTML front-end, with a really bad UI, just so you can see it in action.
In parallel, I’m developing an Android-based keynote remote, called Keymote. Once I release the app, I’ll be selling it for a nominal fee through the Play Store. It’s currently in Alpha testing, but if you want to try it out, let me know, and I’ll grant you access.
So how does KAAS work?
Keynote 5.x (iWork 2009) offers a reasonably comprehensive AppleScript interface* to creating and controlling slideshows with Keynote. It also has a remarkable HTML & JSON export format that, with some basic understanding of the JSON format, allows you to reconstruct how the slideshow will look at each stage of build.
Even better, it tells you when builds will be skipped, or when they’ll be auto-played. In concert, you can use this to determine where Keynote will be after you advance the slideshow, and you can build up build previews (lol) based on the commands in the JSON.
What’s best is that exporting such a HTML & JSON package is exposed through the AppleScript bridge, so it’s easy to do automagically.
In combination, you can use these to replicate the back-end functionality of Keynote remote.
So, if you’re interested in testing out Keymote, or if you want to contribute to KAAS, let me know. I’d be grateful for help and happy testers in any form.
(*Yup, this doesn’t work with Keynote 6.0. It’s apparently a substantial re-write, and Apple have removed the AppleScript interface to the new version. According to this support note, AppleScript support will come back. Hopefully there’ll be something resembling the Export format too.)
(Wooo, catch-up blog time!)
I was one of the invited presenters at the second PyCon Canada in Toronto.
My talk, “Android: The Land that Python Forgot?” looked at the state of Python development on the Android platform, and how we can improve things.
As for PyCon Canada itself? Well the conference itself was fantastic — a friendly, enthusiastic organising team, really good talks, and a beautiful host city. I’m really looking forward to returning to Canada next year when the US PyCon moves to Montréal in April.