Children’s App Reviews: Does Money Talk? Part Two

dollar-signAug. 27, 2015 “I can make or break you” may sound more like a line from Good Fellas than a children’s app review site, but the skullduggery and reticent storytelling seeping out of the industry cracks since I wrote “Part One: Children’s Apps: Does Money Talk?” has nothing on Scorsese.

The last few weeks have been a sophisticated sift and sort game of “perception vs reality” as I try to matrix the players responding to an ‘open call for feedback’ on paid app reviews, sorting out serious dirty laundry that has me covering my ears like the three wise monkeys from my own childhood stint in Japan, ‘Hear no evil, see no evil, speak no evil.’

Being outside of the kids apps sphere, my wide-eyed “say whaaa?” incredulity of “off the record” tidbits has ranged from trying to determine what constitutes disgruntled former employees, favoritism and influence peddling, to stories of lawsuits, IP theft, giveaway fraud, tainted research dollars with corporate tethers and graft and corruption that rivals the robber barons of the wild west frontier…21st century digital edition.

cloak of secrecy 2009 jai parker florida todayMisinformation abounds…

…Like the perception that app reviewers are gleefully rolling in money gleaned from fee-based app reviews, when most indie sites doing thorough assessments barely make enough cash to put in a ‘tip jar,’ once site upkeep and costs of doing business are extracted.

…Or the naïve notion by app reviewers that their self-selection process of determining what’s review-worthy (with pay for play predominance) is NOT blatantly perpetuating an unlevel playing field already dinged with divots and sinkholes that can cripple a talented team that missteps and put them out of the game permanently before even getting to show they’re stuff.

Then there’s the “corporate tech/media doesn’t influence ME” balderdash, which is laughable given the fear-based ‘disease to please’ by all parties, along with new partnerships and mentoring models like apps hooking up with big industry players in early cycles of app creation, or transitions of ad hoc, grassroots collectives like Moms with Apps into formalized policy alliances, since they’re now ACT/The App Association…It brings a whole new meaning to the “Know What’s Inside ® ” transparency motto.

Let’s all just acknowledge bias exists in many shapes and forms and use critical thinking skills to make sense of it, shall we?

For the “TLDR” Crowd, I’ll Start with the Finish


 apps image shutterstockAre paid app reviews happening? Unequivocally.

Becoming normative? Yes, much more so.

Openly disclosed? Rarely per review.

Are paid app reviews ethical? Objective? Effective? Depends on who you ask. (not to mention site reputation, criteria, and content quality)

Do “expedited reviews” increase sales and visibility? Not necessarily. That’s a summary overview from some of the feedback we’ll dig into with voices/views from all lenses in part two.

ShhhhhTipsters asking to be cloaked in anonymity with some of the vested interests in play leave me open to libel if I ‘print what I now know’ on some of the backstories, which stirs the queasies in me, but there are enough repeated patterns in profound commentary to report the basics and enlighten while giving a sweeping brushstroke to a canvas of ‘too fast, too soon’ galactic startup growth in an industry comprised of flame outs and shooting stars.

The good news is, “on the record” names and backgrounds have helped me deconstruct business models to gain a better understanding of both the value and the systemic problems of kids app review money matters that shift faster than you can say ‘expedited review.’

In sum, we’re only scraping the surface of the problematic underbelly of money machinations and media giants dominating the landscape, as fledgling apps struggle to ‘get discovered’ in the current review process which is lopsided in favor of the almighty greenback, even for the best of reasons, like site sustainability. In today’s marketplace of over a million children’s apps in the iTunes store alone, the opportunities and challenges for app creators astound.

digital storytime logoCarisa Kluver of Digital Storytime offers a unique, hybrid perspective and context, given her five years of ongoing insights as a reviewer since the iPad’s very inception, the app developers in her family, and her own current career segue back into teaching, given the massive influx of voices and media and monetization shifts. She summarily cautioned,

“The app market is very crowded today and getting noticed is more of a needle in the haystack issue than it is like getting noticed at a school dance. Chances are, even with a paid review, if your app isn’t exceptional it simply won’t break even, period. This isn’t your grandmother’s app store anymore … it’s hard core…

…For app developers, I recommend paying for expedited reviews on ‘trusted’ sites with solid traffic if they want coverage, but not to expect direct sales from reviews. For parents, my tips are in the article “The Dirty Little Secret About App Review Sites” and my favorite sites are here: (full Q&A interview with Carisa Kluver’s candor about the industry forthcoming)

reading train iconTeacher of 20 years and K-1 Reading Train app developer Libby Curran was even more blunt and cautionary:

“Here’s what I would say to a new app developer: There is no magic bullet. Even getting featured by Apple doesn’t guarantee continued success. A good review on a respected site won’t shift the needle much. Four years ago an innovative app could rise to the top. Not anymore. You probably shouldn’t quit your day job.

It is very hard to make a living as a children’s app developer these days. Even some of the leading app companies are struggling. Discoverability, the culture of free apps, the overwhelming number of apps in the App Store all add to problems. Lack of direct contact with customers is a big issue. Also, Apple does not count ‘educational sales’ towards ranking, so for someone targeting that audience like me in special education, those sales don’t help an app get ranked and discovered…” —Libby Curran (@readingbuds K-1 Special Educator, Grand Prize Winner, Technology and Reading Award, ILA, 2015, a People Magazine Teacher of the Year 2012, Christa McAuliffe Sabbatical Fellow, ’06-07 etc.)

Apps Environs and Tech Titans

Clearly the universal problem of discovery is readily apparent and agreed upon…the solutions and best practices? Not so much.

In part one we discussed the difficulty for parents and educators trying to discern what’s good out there beyond word of mouth and buzz from pals since the FTC interpretation of what constitutes ‘disclosures’ for paid reviews is squishy not to mention loaded with inherent challenges like bias, monitoring, enforcement, fines and big tech, big budget domination.

Today, we’re talking to the creators and reviewers themselves…trying to discern exactly ‘what’s changed and why’ in the short amount of time apps have been in existence and help parents and educators navigate through the hype of ‘content marketing’ and help FIX the flaws in the very foundation of this nascent process to build a better system for all.

nytimes chromebook ipad media ed techMarket research firm IDC reports a $7 billion K-12 and higher ed school sector, so it’s no wonder some of the blood baths among ed tech creators are messy as they joust for position like a Game of Thrones script!

The Bill and Melinda Gates Foundation spending alone brings a whole new meaning to “EdSurge,” as makers of apps vie for attention of teachers to be used (or even swallowed by) tech titans as research and funders co-mingle with opportunities in the education realm, and winners and losers become entangled in the siege.

How do larger hub sites that do reviews keep from becoming ‘lobbyists with apps’ impacted by influential media tethers and tech platforms that can influence, especially in the education sphere?

As it is, Google Play and Apple continue their App Store wars in the media and now the platforms are fueling their fires targeting schools with kids no doubt caught in the crossfire for mindshare.

Beyond the popularity of education apps, the free apps and “freemium” focus dominating the kids app sphere has led to modifications of the review platforms themselves…

Some review hubs have shifted focus from more robust, detailed coverage to watered down “click-throughs for page views” to feature “free app” banners and tons of contests, code giveaways, and app chat noise that ultimately shifts  toward quantity over quality when the sameness of stars and graphics trump helpful analysis and critical thinking with nuanced commentary.

red-hen-clipart-red-henThe more in-depth the coverage, the more one has to ask, “and who’s going to pay for that writing” since we all know those of us with indie sites who do NOT have staffers and are NOT paid via ad revenue are writing our hearts out in a public service labor of love, in-depth research, and personal analysis that is NOT sustainable. (myself included…things have to shift…ideas?)

We’re seeing seismic quakes and shake outs that may signal it’s time for rebuilding a whole new structural foundation that’s stronger and reinforced with logic, fairness, value and disclosure criteria. So, parents and educators:

What are ‘paid app reviews’ and why should we even care?

What’s the difference between an expedited review and a “sponsored” review?

When did this become ‘a thing?’ Here are a few voices and views for better understanding:

The App Developer Lens:


kids apps edit red fbAn “expedited review” claims to fast track an app’s presence into the process of discovery, but that can get pricey fast for small developers trying to debut a worthy app without breaking the bank.

More importantly, this means well-funded, mediocre apps can get a boost to hopscotch over exceptional ones that are languishing “in the tall grass awaiting discovery” (as one creative app developer phrased it!) simply due to their fledgling status. That’s a profound problem for innovation, and has rubbed many startups the wrong way as evidenced by this Facebook dialogue screenshot (closed group/names removed) discussing my last post in part one.

One app creator lamented to me,

“Reviewers who offer an “expedited” review (and there are MANY) are biased toward developers who pay. Even if the reviewer only takes money to review “good quality” apps, they are ignoring other “good quality” apps by developers who weren’t prepared to pay their expedited review fee. In my experience, a “free” review either takes months or, more likely, never materializes at all.

Do the readers know this? I don’t think so.

…My experience has been that a noticeable majority of websites/blogs exclusively doing kids app reviews are asking for some sort of a fee for reviews. I would estimate 70% of the reviewers that got back to me were in this category.”

journalism for saleMei of Mosster Toys validated a similar experience in her comment on part one, “After our experience with trying to get reviews on our kids apps, where we went down the route of paying and non-paying – I am starting to think that a high percentage of the reviews on these sites are paid…They didn’t have much time to look at apps (where) we didn’t pay for reviews”

Jenna Hoffstein of Little Worlds Interactive said, “…It’s not enough to just be good at making games. To give yourself a real chance of financial success you also need to invest significant amounts of time into business and PR/marketing. In a hit-driven business like games there’s never a silver bullet for success, and all you can do it give yourself the best chance you can. As it becomes easier to make games and most of the marketplaces like the App Store become more and more crowded, discoverability becomes a bigger and bigger concern. It’s entirely too easy for an app to be lost among the sea of hundreds of other apps released on the same day.”

Most sites put a caveat in place to pass their own self-ascribed standards to even be considered for a review, free or paid, yet there’s a notable lack of outreach for high quality apps who clearly make the grade, which smacks of skewed objectivity…

Reviewers should be upfront if a “fee for service” review is their “policy” and drop the illusion there’s any ‘unpaid’ option, as this was a MAJOR time sink for indie app developers wasting valuable hours awaiting word and following up while their app collected dust bunnies in the review queue with months and months of lag…IF a “we’re too busy” response was even given at all.

As one award-winning app creator summed:

“ALL of our coverage came from unpaid sites. Interestingly, NOT ONE of the websites that offered an expedited review option has reviewed our app. This would lead me to believe that either these websites are exclusively doing app reviews that have been paid for or they are so inundated with paid review requests that they are unable to get around to doing the free reviews in a reasonable amount of time. ”

Cash vs Cache: Costs, Credibility and Context

To give a general idea of how much review sites are charging, this 2013 AppyNation starter compilation offers an interesting thread of comments I found more helpful than their brief list itself, as it’s important to uncork a living lab (a ‘collaboratory’) to work together on the policies and practices of children’s media to raise the bar for all.

smart apps for kids pricing from site $325 expedited review within 24 hours including video, and multiple tiers of expedited review opportunities in the media kit. – $200 accelerated review; one week turn around for a homepage featured review with links to app/site/and promoted on TWA social media channels/over 100K followers, $55 sponsored post or press release w/media blast

Digital Storytime $150 Premium/full review, $100 Basic/brief review both within 10 business days $250 + $125 follow up of ‘pre-market and app testing’ and ‘fast tracked reviews’ of $25 per app review, plus the cost of your app (Update: note from founder in our comments section) $75 expedited review within 10-14 days, $100 sponsored post – $125 expedited review within one week (this site had pop-ups specifying ETAs for review time) $45 for expedited review within 3-10 working days, $25 for indie package 10-30 working days, not accepting free reviews because of volume $90 fixed rate priority service/social media promotion within 15 days – $125 priority review service within 14days will not publish apps under 4 stars; free reviews 3-6 month wait (their site says they use ads vs paid reviews, but according to one app developer, they were told they “review very few submissions – to improve odds – For $200, we’ll describe your app, link to it on iTunes, and feature it in our email newsletter”)

Anyway, you get the gist. It REALLY adds up cost-wise for an app developer and becomes frustratingly vapid from a valuation standpoint for a parent or educator when sites have vastly different depth in ratings/analysis/graphics/seals and certifications for any form of ‘at a glance’ depiction of what’s worthy.

The App Reviewer Lens:


App reviewers have been very forthcoming (and sometimes unnervingly candid) in discussing just how much things are changing in the apps world, and how we’ve ended up with so many expedited reviews with sheer volume impacting quality of critical content.

They emphasize the imperative of using only sites with vetted, experienced reviewers and remind parents to read the bios/backgrounds, depth of critique and time-stamps of each review, since any one of those elements can skew the focus and validity. As a whole, many I spoke with seem to regard app reviews as a business more than a journalistic endeavor…voicing bias in today’s mainstream media editorial lens as well.

appolearningJill Goodman, adjunct faculty at University of Central Florida Department of Education where her duties include serving on the dissertation committees of students doing their theses on apps in early childhood education, happens to write reviews for AppoLearning. She also has written over 1000 app reviews at other major sites like Smart Apps for Kids where she previously served as Editor.

Jill Goodman shared:

“Most sites with 10,000+ followers or 100,000 or less Alexa scores DO use expedited reviews.

During my tenure with one of these sites, I never saw instances of ratings inflation due to being an advertiser or “friend.” As editor I bumped down by 1/2 point far more scores than I bumped up. In some ways there’s a bigger conflict when an app is reviewed as a favor to a developer.

Paying for an expedited review makes it business and is about more than the review. It comes with substantial visibility and social media promotion.”

teachers with apps termsJayne Clare of the Teachers with Apps site said:

“Reviewing any app is time consuming especially if you want to do field-testing with kids as TWA does. And field-testing is only part of the overall process, after gathering data a review must be written and then it is no small feat posting reviews into the website with images, links, configuring SEO and other tedious tasks… I don’t feel the necessity to state we were given a promo code for an app we review, many are free and the cost is usually minimal.

As for Best Practices, it is most important to be true to one’s standards, we do not review mediocre apps – even if they offer to pay. We only want to represent apps that we feel have educational merit of fine quality.

…TWA is a small operation, we would like to become a non-profit and have corporate sponsors so that we could promote the indie developers and help get some of the underdogs recognition without charging developers. Personally, I do not feel that sponsors should impose their own products on reviewers and reviewers should be strong enough to stand by what they believe is quality.”

digital storytime feesCarisa Kluver of Digital Storytime said:

“I do think expedited reviews are now the norm for any decent sized site that has been around for more than a couple years. To get real recognition, an app either needs to be break-out exceptional (like the 1% or for apps the .01%) or it needs a real marketing budget that includes paying for ‘expedited’ reviews.

Many sites have found a fine-line to straddle that lets them continue to review with an unflinching eye, but others have succumbed to pressure to make paid reviews less negative or to gloss over or omit issues that consumers would otherwise want to know about. We get literally HUNDREDS of requests every month, beyond what anyone could review or even sample.”

kids apps screenshot dust or magic dialogueOne of my favorite resources for sifting through quality conversations about apps and other kids media from a child development standpoint is the Fred Rogers Center (Facebook group here.) 

I hope they tackle this topic with their academic research fellows, meanwhile, parents seeking more context on how to find good content might want to peruse two fabulous Facebook groups useful for anyone interested in the children’s media sphere, Dust or Magic is an open group discussing children’s apps and interactive media (shown here) and Children’s Media Professionals is a closed/professional group by invitation/referral with larger topicality.

Both are invaluable resources for opening up dialogue on important topics like this, though I’ve noticed in keeping with the cliques and tribes of industry dynamics there’s a propensity to ‘lurk’ instead of leap into the conversations, unless you’re ancillary and unaligned as I am. (I’ll post this to both groups and see if any sparks fly) 

One app reviewer summed the general industry consensus,

“Anyone who reads more than five app reviews can easily recognize honest opinion from promotion. And anyway, after reading a review, users land on the App Store page where they will often find more feedback from users to help them make their mind.”

sy_avatar_ningThat said, I’m not personally seeing that level of media literacy discernment for those outside of the industry, especially among parents searching for apps.

Of the dozens of parents I interviewed most didn’t have ANY idea kids app reviews were often paid for, and since many parents also equate ‘education’ with institutional neutrality, many were surprised to hear paid reviews or “fee for service” arrangements are becoming a norm, especially sites involving teachers and research among educators.

Again, payment may be gaining ground and normative for the “ed tech” industry wide, but for many parents there are assumptions made when using the word “education”…Many have come to free associate the word with…well…being free, as in public education.

These days the private sector is becoming more enmeshed with student media in multiple ways, from data privacy and records beyond, so parents need to amp their media literacy, monitor and manage industry transparency (as well as their own bias) and reframe that conversation altogether with more accuracy and savvy.

So Who DOES Offer Unpaid Reviews?

geekswithjuniors logoIt takes a lot of hustle to keep knocking on the door and filter out which sites offer unpaid reviews in a sea of expedited fees. Larger, mainstream press coverage often comes from staffed or syndicated sites like Common Sense Media  (with its entire Graphite apps and ed tech review division) Tech with Kids, or subscription offerings like Children’s Technology Review.

If you follow the media literacy money trail you’ll also find the quality review sites without fee for service arrangements are simply monetized differently, giving indie developers much needed lift without perceived conflict of interest.

For example, Appysmarts uses their YouTube video channel clout and breaks even with affiliate marketing and sidebar ads (policy screenshot from GWJ below) KinderTown (now owned by Demme Learning) offers free reviews in appx 8 weeks, is age specific geared only from ages 3-8, reducing volume.

pappasappar logoGlobally, in Sweden is a heavily trafficked kids app review site using affiliate marketing commissions and Google AdSense (with a policy that developers may not advertise their apps) to keep their extensive collection of kids app reviews free, and clear of conflict of interest. And app developers have also said Turkish site is a reliable resource without review fees. Point is, unpaid reviews  ARE still out there, it just takes time to wade through the tsunami of so many review sites…

kids apps geekswithjuniors screenshot

Ironically, just a few years ago, expedited reviews were called out as contaminated content with lengthy commentary…

Today, I find it’s harder to discover sites who do NOT charge outright with rate cards and cash for cache.

As for bias? We’ve already talked about ed tech tethers a bit with the Gates Foundation, and of course Edutopia is aligned with the George Lucas Educational Foundation, sister company to Lucas Education Research. Ed tech mergers and acquisitions are frankly a revolving door of business interests…

One seasoned reviewer spoke off the record in full candor,

“I don’t know of any review sites completely free of any ties to anybody. Maybe Kindertown but with their former director now at Apple even that could be bias. Educade is AT&T, AppoLearning is owned by a guy from Yahoo News. In the end I’m not sure it matters. There are few outliers. Most apps get consistent ratings across sites. Balefire might be the exception but I find some of those ratings ridiculous. Giving a Dr. Panda app an F is crazy. They are just silly harmless fun.”

Reviewing the Reviewers: Quality, Bias, Insights


dust or magic logo Representatives from Common Sense Media, Teachers with Apps and the Children’s Technology Review convened at the Dust or Magic Children’s AppFest (the day before their AppCamp) an insightful albeit pricey “pay to play” endeavor in itself where startups, leaders and reviewers share best practices and pitch their goods for feedback and collaborative support from colleagues within the industry.

If you’re a fellow media infovore (intrigued but not involved with apps on a granular level) also check out the free offerings of EdCamp a free hub for educators, homeschoolers, reviewers and developers that’s been described as “loaded with armies of sponsors but helpful and free” with giveaways like classroom licenses for BrainPop and other fun finds.

Clearly invaluable conversations are happening here on ethics, privacy, design, best practices, what works and what doesn’t, and ways to be ‘discovered’ in the sea of selections. (Special thanks to industry good guy Scott Traylor of 360Kid recording these video clips to enlighten/inform and raise the bar in the kids media sphere; he’s woofed as a watchdog from both sides of the media fence and has provided very helpful video documentation for newbies like me) 

Listen closely to the discussion around the 17:20-21:00 mark about the ethics behind expedited reviews and the ‘contaminants of objectivity’ cited by Dust or Magic/Children Technology Review’s Warren Buckleitner as they all discuss the app environments on iTunes, Google Play and review hubs…

It’s a very helpful 4-minute clip within the abbreviated video, worth your time to see the elephant in the living room very fast.

Expedited Reviews & Ethics: At appx 17:20-21:00 mark


I’ll leave it there for now, as the video speaks volumes.

Do Expedited Reviews Even Work?

Why Have Kids App Review Sites Changed So Much Recently?

How Can We Build a Better System with Fairness & Frankness?

What Tips Can Industry Pros Give Parents to Find Their Apps?


In Part 1 we focused on expedited review disclosures (as opposed to ‘in-app’ upsell wristslaps by ASA watchdogs this week) and in Part 2 we’ve heard pros/cons from some developers and reviewers directly…

As we segue to Part 3 we’ll be looking at solutions-based ideas, design thinking improvements, and talking tips for PARENTS from industry pros to help navigate the app quagmire. Sound off and we’ll keep compiling the commentary…

Do you feel ‘big media’ will be ‘owning’ the app space as an inevitable outcome? What will that mean for creativity at the onset? Will new models emerge? How can parents figure out who’s playing in who’s sandbox as they try to make sense of paid and unpaid recommendations?

Stay Tuned for more in Part 3

Visual Credits: Jeff Parker, cartoonist 2009 Florida Today/Cloak of Secrecy, screenshot of photo by Gabriella Demczuk for The New York Times, Money talks graphic ’08; Journalist for sale via Tineye at, reused in this eye-popping PK blog post; Little Red Hen clip



  1. Hi Amy-

    I really enjoyed these two pieces and thank you for including Balefire Labs. It particularly gave me the opportunity to clarify our free versus paid services on our own website, because from reading your piece it seemed like we weren’t clear enough that all of our standard reviews are conducted for free. I immediately made some changes to our website, so thank you for providing that insight. We do have an expedited review available, but it does not “bump” anyone else from their spot in the queue. The expedited app is just added to our monthly batch. Our other fee-based developer services are outside of our normal review process and are intended to support developers in their instructional design efforts.

    It is true that there is really no money in edapp review sites. Many of us whose efforts are NOT underwritten by bigger businesses or funded by private foundations got into this to try to make a contribution to education. Most of us are kind mission-oriented and looking to help teachers and kids. But it’s just not clear that very many teachers and parents value the information that we, as a group, provide. We know that, for the most part, they are not willing to pay for our expertise and information. And I don’t think that we, as a group, have figured out a successful business model to do this work. Nobody goes into app reviewing with illusions of getting rich.

    Thanks again for your thoughtful piece.
    Karen Mahon
    President & Founder
    Balefire Labs

  2. Sorry, most of us are “kind of mission-oriented.” Not just “kind.” 😉

  3. For me, review sites give you a chance to hear what others think of your app. The more critical the better. I don’t think of them as sales drivers, but as a way to get feedback and improve my apps.

    As a small indie its hard to justify paying for fast reviews, and yes some sites flat out refuse to review your app otherwise. Of course, every little helps in terms of getting your app out there of course and will result in some direct sales, but perhaps not enough to cover some of the fees.

    Doing reviews is hard and time consuming, so I empathise with those blogs and the issues around making it pay.

    One free review site that you missed is the Swedish site They give good critical reviews, are popular and not reviewing apps that are freemium or full of advertising.

  4. Thank you all for your comments, keep ’em coming and I’ll keep compiling them! I have now been in touch with and updated to include them…(again, not being directly in the industry, my outreach for open calls on Twitter was abysmally America-centric and only scraped the surface; I DO recall now that a few European app devs mentioned Pappasappar so thx for filling that gaping chasm, Chris!)

    Also, Karen, I appreciate your candor and am glad I could help Balefire Labs with any clarifications on the policy front…Your mention of paid reviews NOT bumping free reviews from your queue would be a first, given the volumes of app dev feedback I received on that very practice. (So much so that I felt the need to call it out, as SO many sites SAY they ‘do free reviews as well as paid’ and simply leave folks hanging for months if they ever get a call back at all…

    I’d love to hear more about that and storytelling from app devs who have experienced this fairness. That would be SO refreshing! Fire away folks…give me your pros/cons and firsthand stories 🙂

  5. p.s. Chris, also, I loved your feedback comment “the more critical the better” as that’s how we all learn and grow and raise the creativity bar to put great work out there…I’m wondering if you have any personal insights into the sites that offer ‘analysis’ over reviews? I’m starting to see more of that ‘critique’ component as a fee for service offering…I suppose it would depend on the reputation/credibility/experience of the sites offering the services…Curious.

    Also, can you shed any light on how workshop formats and collaborative events like Dust or Magic and AppCamps and EdCamps and such help create better products…and whether it’s informal/free/collegial feedback or if it’s paid/curated formal consulting? (I’m guessing once a fee is paid to get in, the feedback is the ‘value add’ whereas the free events are hit or miss and/or finding consultants within the tribe? Again…no direct knowledge here so elucidate if you can?)

  6. Thanks for the response, Amy. Forgive me for this long post, but your responses provoke more thoughts on my end:

    I can’t speak for anyone else, but on the Balefire Labs site we publish what month and year we are currently scheduling into for app reviews. When anyone (whether developer, teacher, parent or other) submits an app suggestion for review, we respond and tell them what month we will be reviewing it and the date of its anticipated review publication.

    I can only think of one time (of more than 4,500 completed app reviews) when an app submitted for review fell by the wayside. In that case, the app offered a limited free trial. We tried it, not realizing that the free trial was time-based, not use-based. As a result, the trial elapsed before we completed the review and we never were able to work out with the developer a “reset” of the free trial.

    I am not an app developer, but I find the customer feedback given to developers at workshops and events (or even virtually) not to be very helpful. Most customers say things like they want apps that are “engaging and fun” and that kids are “motivated” to use. If you unpack requests like that and ask what people mean by “engaging,” for example, you find that people mean different things. Some believe “engaging” means that an app will keep a learner busy, working quietly for 15 minutes; others mean that learners will want to use the app again and again. It’s imprecise language and it creates problems for execution when it comes to actually creating an app.

    Often those customers (and often the software developers to whom they are speaking) don’t have the instructional design or educational research experience to know what particular features and functions an app needs to create that user experience. So “engaging” and “motivational” are sort of subjective, bucket terms that are not actionable for engineers.

    For example, in the instructional design community we know that adapting levels of difficulty are more likely to produce high rates of engagement and that mastery-based learning is more motivational than many other approaches. If customers were more informed about that then they could say to developers “we want adaptive difficulty and mastery-based.” A request like that actually IS actionable. But there is a disconnect between what edtech customers *believe* about instruction and what the research supports with data. That is why we structure our rubric the way we do at Balefire Labs. The goal is to give the customers *objective* information about what an app has and does, relative to what instructional research says is important for learning outcomes.

    Yet still we hear things from customers like, “Yes, it might be effective, but is it fun?” because they often don’t understand the relationship between “fun” and “achievement” for learners. The answer to this question is that kids who achieve with an app are likely to experience it as fun. And the most likely way to build an app in which every student can achieve is by building one with adaptive learning paths that adjust to the performance of each individual learner.

    In summary, there is a disconnect between the language that customers use to ask for what they want and what they really NEED in order to effect meaningful student learning outcomes. We try to provide customer education in that regard on our site, but it’s a tough road to hoe. We also try to provide developer education on that’s what we focus on in our consulting services…so you can see how it goes well above and beyond a simple review.

    Okay, getting off the soapbox. Sorry to rattle on and on! Thanks for the opportunity to talk about these issues.

  7. Helpful insights on language lingo, Karen…Though “adaptive difficulty and mastery based” is out of my daily media literacy lexicon, I’d venture to say that with all the edu jargon, multiple rubrics, diff learning targets/yields/desires etc., even words like ‘effective’ are subjective and distorted as each person has a diff lens to view how that outcome (and value) is determined and applied.

    We’ve all got to be mindful of intent and style, since some apps are more about informal learning or participatory/social/collaborative vs achievement and assessment driven…Of course many aren’t even in the ed tech sphere at all, yet there’s still plenty of learning going on. (e.g. using imagination, creativity, the arts, nature, kinetic play, etc)

    And yep, keep on sharing openly as dialogue helps *everyone* learn…Ancora imparo!

  8. Another one that charges is The saddest part about this is that the writer also writes for edutopia.

  9. Amy, ah, there’s the rub. How does one know that “learning” has occurred?

  10. Hi, Amy.

    Wow. This is such a complex thicket of issues it’s hard to know where to dig in. I’ll try to be succinct, but there’s a whole lot to say.

    First let me say, Amy, that I appreciate your posts very much. This is a very important topic and I’m glad to see it being aired.

    My company (Native Brain) develops math curricula delivered via apps. We base our designs on high-quality educational research. Our top-line priority is producing apps and instructional tools that support effective learning, and finding ways to enable teachers and third party researchers with no vested interest in the tools to be able to verify and evaluate the efficacy of the tools and the quality of learning outcomes. We are big on checks and balances to avoid bias in every aspect of the app development and dissemination process. I see that as a way (likely the only way) to improve education substantially going forward.

    Currently, the educational domain suffers from what Howard Gardner calls “massive misalignment.” I think this misalignment might be especially acute in the educational app space, and one place it becomes particularly obvious is in the app review part of the space. The misalignment of information and incentives is strangling people in the ecosystem who want to do and benefit from good work – students, parents, teachers, app reviewers, app developers, and so on.

    Examples of misalignment:
    + Lack of standards for what qualifies as “educational.” Many of the apps that are submitted into the education category and get high marks under one or more rating systems are obviously not even minimally instructional (also see this post by Balefire Labs on this point: If the term “educational” doesn’t actually mean anything specific, why do we have a separate category for it? Education is not the same as free play or fun – education is defined as “the process of giving or receiving systematic instruction.” This term shouldn’t be as wide open as people take it to be if we expect it to be a useful distinction, and it shouldn’t be merely a marketing device. You can’t deliver systematic instruction or evaluate whether the instruction is effective if you haven’t clearly defined what the intended learning outcomes are, just to take one glaring example. This should be table stakes for any app claiming to be educational. Maybe we need a separate category for “educational tools” that support instruction but are not themselves instructional, and another for “edutainment” that is mostly entertaining but people hope students will learn something (not specified).

    + Bias and lack of fairness. Our own experience with app reviewers was similar to what others have reported – many didn’t respond, not even to acknowledge receipt of our submission. Recognizing the conflicts of interest with paid reviews, at first we refused to pay for any reviews, wanting consumers (parents, teachers, and students) to get good, unbiased reviews of our app so they can make good decisions. To your question about fairness, Balefire Labs and Teachers With Apps were the only reviewers who reviewed our app during this time without a fee even though we reached out or submitted it to many. (Note that once we started getting some traction a few others did pick up on it later.) Regarding TWA, not only did they review the app but they put it into the hands of students and observed them using it. I believe this practice helps limit at least one important form of bias – where adults try to imagine children’s experience with the app without giving it to multiple children in the appropriate age range to actually observe their experience. Regarding Balefire Labs, not only did we not pay them anything for the review, but they have a process that involves transparent, objective criteria based on empirical research about the features of effective instructional design, and they have established internal checks and balances to make sure the reviews are objective – in particular, they have multiple trained reviewers apply the review criteria to a subset of their apps and compare those reviews for consistency (this is a check on what is called “inter-rater reliability”). I know of no other review service or site that goes to these lengths to ensure fairness, objectivity, alignment with instructional efficacy, or reliability.

    + Ad hoc, inconsistent, and irrelevant evaluation criteria. I’ve seen reviews that are the equivalent of “my kid didn’t like it so I give it an F.” That’s what I mean by ad hoc – there aren’t really any principled criteria there. In terms of inconsistency, an app can get top marks from one review site and an F from another one – and the reasons for these differences are generally not readily accessible to parents and teachers (but they probably have nothing to do with different evaluations of educational efficacy). In addition, some people use irrelevant – or even counterproductive – criteria for evaluating educational apps. To take one example, one factor people consider when purchasing a game like Angry Birds is how many hours of entertainment it is likely to deliver. More hours translate to higher value. I have heard many reviewers and consumers apply the same criterion to educational apps. But if the purpose of an educational app is to help a person learn something, shouldn’t we value apps that actually get to that same learning outcome in a *shorter* amount of time? Carrying over that criterion is actually detrimental in this case, but it’s done all the time. The same point applies when people over-emphasize how much “fun” they think an app will or should be (in terms of superficial features like graphics, animation, and theme). These design features can actually interfere with or undermine learning efficacy and yet they get a lot of weight in many evaluations of educational apps.

    + Financial misalignment. At the end of the day, both app producers and app reviewers have to pay the bills, just like everyone else. With all of this misalignment in the system, it’s hard to do “good work” in the educational technology space. People don’t seem to realize that when they make a purchasing decision, they are not only receiving the goods or services purchased, but they are voting on which providers should survive and which should be starved out of existence. If they purchase or download apps primarily based on entertainment value or which has the biggest marketing budget behind it and so on, they should understand that they are actively creating a world where their children’s actual learning is an afterthought – if it’s a consideration at all.

  11. Christian: yes, I was actually just given several screenshots of bloggers working both sides of the editorial fence and am revamping part 3 to include the problematic journalistic conundrums of page view clout, currying favor, and blurring the lines between editorial ed tech recommendations and promotional content gleaned through blogosphere back end/monetization methods…”It’s complicated.” (As they say on Facebook…)

  12. Michael: Very helpful add-ons/feedback for everyone here, especially the part about “massive misalignment” and the arbitrary use of the word “educational” which is loosey goosey at best…glad you’re sharing your info aloud and will snag some of it for part 3…thank you for that! 🙂

Speak Your Mind