There must be no barriers for freedom of inquiry. There is no place for dogma in science. The scientist is free, and must be free to ask any question, to doubt any asssertion, to seek for any evidence, to correct any errors.
J. Robert Oppenheimer
There must be no barriers for freedom of inquiry. There is no place for dogma in science. The scientist is free, and must be free to ask any question, to doubt any asssertion, to seek for any evidence, to correct any errors.
J. Robert Oppenheimer
Academia and App Store Reviews
19 March 2011
I’d been thinking about how App Store reviews follow an academic approach for several months, and a pair of tweets by Daniel Jalkut yesterday spurred me to finish it. Daniel wrote:
“App Store approval process is like time travel to the past. We spent all that time earning faster deployment capacity and then we lost it.”
and:
“Imagine if you had to wait two weeks for a blog post or tweet to go live. Love the App Store but it’s challenging to freedom of expression.”
In my other life, I’m a professor in paleobiology. I study ancient marine organisms and rocks they’re buried in, partly because this is essential for finding petroleum, but also because it’s the main way we know about past conditions on the surface of the Earth.
The life of a research scientist in academia follows a predictable script. First, get inspired with a unique insight about how nature works. Write a grant proposal for funding that will let you test your idea. After a few attempts, your proposal hopefully gets funded. Go out collect the data and do the analyses that test your idea. Publish one or more manuscripts that document what you did and what you found out. Do this successfully (that is, find interesting, novel, and useful things about the world) and you get tenure and promoted. If you don’t, you get fired and go into a new line of work.
Most of these steps – writing grants, getting published, getting tenure – hinge on something called peer review. In peer review, your proposal, manuscript, or tenure file gets circulated among a small set of scientists, and they pass judgement on what they see. Three aspects of peer review are significant.
First, the peers are knowledgeable; they’re in your field and generally have the background to judge your work. Botanists aren’t (usually) judging physicists on the merits of their work.
Second, the review is usually anonymous, but one-sided: you may not know who the reviewers are, but they know who you are. When the reviewers make a decision, you often have little or no chance for discussion. This anonymity serves a purpose: it allows reviewers to offer candid judgements without fear of reprisals.
Third, the process takes time, often lots of it. For manuscripts, one might wait a couple of months for the reviews. For grant proposals, the wait is often more like six months. Final word on tenure and promotion can be even longer. Academics are used to this delayed gratification, but Daniel Jalkut is right: it does challenge the freedom of expression, and these long waits kill the joy for many.
Flaws in peer review are common. Not all reviewers are knowledgeable, and worse, some don’t realize or acknowledge their lack of expertise. Anonymity meant to protect reviewers can offer a convenient hiding place from which to damn competitors or shrink from scrutiny. Busy schedules of everyone mean that documents sit on someone’s desk for much of the review period.
Despite all this, the process usually works. Good ideas generally make it through, even if it takes multiple tries. Most bad ideas fall by the wayside. Most talented researchers survive the process, and those whose talents lie elsewhere often move to those jobs. Peer review can be ugly, but it serves science well, and it’s an effective filter.
I come with this background when I put on my developer hat. Consequently, the App Store review process has always seemed natural to me: you work hard on something, submit it for review, and wait for a thumbs up or thumbs down. You get told if there’s a problem, but the system isn’t designed for any sort of real discussion. Nothing about this struck me as unreasonable. You assume that the reviewers are knowledgeable, but they remain anonymous. And the process takes time.
To non-academics, much of this seems ludicrous. Why does the process have to be anonymous? Why does the communication have to be so difficult? And why does it have to take so long? These are great questions, and we’ve not heard good answers.
The parallels to an academic review style seem so strong that I really wonder if, in the days before the App Store, Apple made a call to nearby Stanford and asked them, “Hey, you guys do you reviews all the time. How does it work?“ They got their answer, and the initial incarnations of the App Store review process have a decidedly academic feel to them.
It seems to work much like it does in academia. Malicious code has been kept out of the App Store, unlike like some infamous cases on the more open Android platform. After an explosion of fart apps on the iPhone, Apple’s review process for the Mac App Store now includes quality and utility as review criteria.
It’s also difficult to get real feedback. Gus Mueller, for one, endured a lengthy series of reviews when problems were revealed one by one. Another developer languished in the review process for over a year. When an app gets rejected, it has been difficult to be able to talk with an Apple representative about it. Perhaps even more so than in academic review, the App Store review board has been as communicative as a rock wall.
I’ve been fortunate: I’ve had no problems in the reviews of my apps. The stories I’ve read, though, have been gruesome, and the App Store review process needs attention. Craig Hockenberry wrote a thorough critique last summer, and Apple has taken steps towards being more open, most recently a Resolution Center. My plea for App Store reviews is simple: make the process less academic.
First, open the lines of communication. Make it possible to talk with someone: some problems are too difficult to resolve or too slow to resolve through email. And please, don’t give us a tech-support robot, slogging their way through a binder of scripted questions and responses. Give us an engineer that can speak knowledgeably about our case.
Second, give us a view behind the curtain. When it takes weeks or worse to hear the results of the review, let us know what’s really happening. Is our app actually in review or is it waiting in a queue because volume is so high? Let us see reviewer notes on our app as the review unfolds. Even better, let us post comments or answer questions during the review.
Last, give us a presentation and a forum at WWDC on the App Store review process. Some talks have been given before on this, but Apple’s engineers quickly scooted out before questions could be asked. Many developers don’t get the sense that their voices are heard and a little effort here could substantially improve things.
Anonymous peer review works in the sciences, but it’s not the way to go in the App Store.