This is a paper I gave via Skype at the NordiChi 2016 conference in Sweden on the Future of Reading and Human Computer Interaction.
Picturebook Scholarship applied to Story App Design
This study describes how concepts from picturebook scholarship have been used to inform the adaptation of a picturebook to a story app for the iPad. Guidelines for the creation and assessment of story apps emerged from a process of research, adaptation, play design, and classroom testing.
There is little guidance for story app developers on the creation of interactive literary artworks. The aim of this study is to provide a bridge from the world of picturebook publishing and academia to the children’s digital space with a framework derived from practice and theory to help creators improve the quality of their story apps. Two concepts from picturebook scholarship foreground this study:
1. Picturebooks are played multimodally through the combined contributions of words and pictures.
2. Children read picturebooks differently than adults do.
Research and testing
The dynamic of word-picture interanimation is the fundamental principle of picturebook scholarship. In the best picturebooks, words and pictures are not redundant, but relate by mutual enhancement, alternation, andor contradiction or counterpoint. Picturebook reading for children is not linear, but a hermeneutic process of moving from words to pictures and back again, constantly revising their understanding of each; “we interpret the text in terms of the pictures and the pictures in terms of the texts in a potentially never-ending sequence” [Sipe 122)]. Therefore, animations, interactions and sounds should be replayable and child-controlled to allow for each child’s unique process. For children, the picturebook is a puzzle. They look at book illustration carefully to uncover all there is to be seen in the picture and to find “secrets” (Kiefer 36). They see “the smallest details that many illustrators include in their pictures and that we adults often overlook” ( Kiefer 35). Close looking helps them to deduce what action is being suggested by the still pictures; what movements in time and space; and what ironies, implications, and connections exist between the two modes. “[T]he pleasure of picturebooks is not just the stories they tell but also in the game of figuring out what those stories are” [Nodelman and Reimer 298]. This same process can be extended to the story app where children need to discover interactions, figure out how they work, and discern the interrelationship between interactions and the other modes. This process is what Mackey calls serious play [167-170].
My relationship with schools as a children’s book author illustrator allowed me to test The King’s Ears in grade-two (age 7) classrooms. In two sessions conducted when the app was half-finished, ten iPads were shared amongst 30 children. We started with a reading of the codex with the pages projected on a screen. Children then broke into groups to play the app while the app’s two programmers, an intern, and I stood behind the groups with iphones to video the children’s hands and voices. Afterwards, we reconvened to brainstorm how to design the remaining pages of the book as app screens. One boy suggested sophisticated design ideas which included how to change the original book illustration to accommodate his idea of having the letters that spelled out the king’s secret rain down from the sky.
He gave me permission use his idea in the app. When the session ended, excited teachers told me that that boy had reading challenges and refused to participate in reading time. They were amazed at how involved he was in the testing session, and how intelligent his design suggestions were. They pointed out that a number of the most vocal participants in the brainstorming session had reading challenges.
The classroom sessions demonstrated that children worked better in small groups than alone. Knowing from my research that puzzling is an integral part of picturebook reading, The King’s Ears has few hints or instructions. I observed that playing the app in small groups allowed shared problem-solving. Keen involvement, laughter, mimicry of the app sounds, experimentation, playing, and discussion of the story were evident in both sessions, which confirmed my belief that few hints or instructions are necessary in the well-designed story app. Children touched everything on the screen to see what it would do, and spontaneously discussed what the modes meant in terms of the story.
Our final UI design was influenced by seeing children play with the app in two different game engines. Most participants preferred the button page-turn in the Robot Storybuilder template to the custom-coded, swipe page-turn in our Cocos 2d version. Seeing that everyone wanted a turn with some of the interactions led us to add a reload button so turn-taking could take place without leaving the screen.
It was also evident that child-controlled narration suited more needs than auto-narration. Fluent readers tended to read on their own before listening to the narrator, and the less fluent liked to listen to the narration first. Having the choice to tap the narration on and off is a useful feature for both strong and struggling readers.
My research and testing showed that creators of story apps who place a lot of hints might not understand the repetitive, exploratory, experimental, speculative, deductive, and hermeneutic way children play a picturebook. These developers may not have faith that children will find and figure out their interactions. Excessive hinting also suggests a lack of logical interconnection between the narrative contributions to the story made by each of the modes of words, pictures, sound, animation and interaction on each screen.
A set of five guiding questions has emerged from this study:
1. Are the assets to a professional standard? Too often the words and pictures in story apps do not measure up to the quality found in trade picturebooks. Narrators, sounds, and animation should also be of high quality.
2. Are the sounds, interactions, and animations integral [Madej 12], telling the story through enhancement, alternation, and/or contradiction as they do in the best picturebooks? Or are they incidental (ornamental) [Madej 9] and only present to add busyness to the screen?
3. Do the words remain on the screen to allow for print awareness (Diamant-Cohen xiii-xiv) for children who cannot yet read or, for those who can, to allow hermeneutic cycling?
4. Is there judicious use of hinting which leaves room for serious play? Is there a logical connection between an interaction and the story that rewards puzzling?
5. Are animations automatic or child-controlled? If automatic, is there a way to redesign them so they are child-controlled? The connection between the hand and the brain may make an animation triggered by the finger, such as dragging a flying bird around the screen, feel more real and exciting, and encourage embodied empathy.
To produce quality story apps, developers need to learn from picturebook scholarship and publishing, and the codex book world needs to recognize the story app as a children’s literary format, particularly for readers who find imaginative entry into written text difficult. This study will add to the growing body of scholarship on the story app, the effect of which may be to raise its status as a literary artwork, as well as establish standards for creators. Henry Jenkins argues that, “for new media such as video games, serious critical and scholarly analysis is a vehicle for the maturation of the medium, for the training of its practitioners, and for the education of its audience” [qtd. in Bizzocchi and Tanenbaum 270].
Bizzocchi, Jim & Joshua Tanenbaum. 2011. “Well Read: Applying Close Reading Techniques to Gameplay Experience.” Well Played 3.0: 262-90. http://press.etc.cmu.edu/content/well-read-jim-bizzocchi-joshua-tanenbaum
Diamant-Cohen, Betsy, Melanie A. Hetrick, and Celia Yitzhak (illus.). 2013. Transforming Preschool Storytime: A Modern Vision and a Year of Programs. Neal-Schuman–American Library Association.
Jovanovic, Katarina, and Phillipe Béhà. 2008. The King Has Goat Ears. Tradewind Books.
Mackey, Margaret. 2007. Literacies across Media: Playing the Text. Routledge.
Madej, Krystina. October 2003. Towards Digital Narrative for Children: From Education to Entertainment: A Historical Perspective. ACM Computers in Entertainment, Vol. 1, No. 1, Article 03. http://dl.acm.org/citation.cfm?id=950585
Nodelman, Perry and Mavis Reimer. 2003. The pleasures of Children’s Literature. Allyn and Bacon.
Sipe, Lawrence R. 1998. “How Picture Books Work: A Semiotically Framed Theory of Text-Picture Relationships.” Children’s Literature in Education 29.2: 97-107.http://link.springer.com/article/10.1023/A%3A1022459009182