Researchers have been trying to get artificial intelligence (AI) to understand and generate narratives for years. A big problem in their line of research has now seemingly been solved, as researchers from Disney and the University of Massachusetts Boston have developed neural networks that can evaluate short stories and predict how popular they could be with a large audience.
AI that can Evaluate Short Stories
This AI can evaluate these stories but do not try to mimic professional, experienced critics. Instead, they manage to analyze the quality of the narrative and predict its popularity, according to Boyang “Albert” Li, a research scientist at Disney Research:
“Our neural networks had some success in predicting the popularity of stories. You can’t yet use them to pick out winners for your local writing competition, but they can be used to guide future research.”
One of the challenges researchers had to face was the lack of large databases of stories that have been evaluated by humans. They could use these to train the neural networks. To solve the problem, researchers turned to question and answer website Quora, as users on the platform regularly answer questions by describing their own experiences in the form of narratives.
Researchers picked up as many as 55,000 answers from Quora, and classified about 28,000 as relevant narratives, averaging 369 words per answer. Researchers then used upvotes – given by users to helpful answers – to measure a story’s “proxy for narrative quality,” and potential popularity.
To understand the complex semantics of the various analyzed stories and their characters, two different neural networks were developed. One looked at separate parts of each story, including the question that inspired it, while the other one evaluated “how meaning of the events and story regions emerged from the entire story” while looking at the story as a whole. Both AIs made predictions of which texts would be most popular with readers.
The two neural networks outperformed baseline text evaluation system, but the holistic AI showed an 18 percent improvement over the one that focused on separate parts.
Possible Future Uses
This is the first step towards getting AIs to create narratives themselves, as they need to understand what makes a good text before trying to create one themselves. Disney and other leading movies studios could easily use these AIs to, in the future, choose one script over another based on how well the AI says it will perform with the audience.
Markus Gross, vice president at Disney Research, stated:
“The ability to predict narrative quality impacts on both story creation and story understanding. To evaluate quality, the AI needs some level of understanding of the text. And if AIs are to create narratives, they need to be able to judge the quality of what they are producing”.
When matured, these neural networks could even be used to generate narratives for production, meaning one day we could watch movies written by AI that knows exactly what will get us on the edge of our seats.