It takes a trained eye to assess all the different components of a painting, and it’s even more difficult to amass the knowledge needed to put that work in historical context. But what if a computer could do it all for us?
That’s what researchers Ahmed Elgammal and Babak Saleh of Rutgers University are trying to figure out. They’ve created an algorithm that uses machine vision and learning, plus big data, to “look” at thousands of paintings and determine which are the most creative and innovative.
These concepts may seem subjective, but computers have the potential to assess them in a more consistent way than humans. Elgammal and Saleh use a visual classification system that breaks down objects and types of scenes into categories called “classemes.” These groups can be simple (like basic shapes and colors), more complicated (like spotting a barn or the Empire State Building), or complex (a dead body, a figure running away). The algorithm can break a painting down into 2,559 classemes.
The researchers combine this analysis with databases that contain photos of paintings, such as the Wikiart collection, which has 62,000 entries. Such large collections give the algorithm diverse data to learn from. And as the algorithm goes through paintings, it also forms connections (like a web or network) between paintings based on their chronological age and what classemes they include.
This is how the algorithm draws conclusions about creativity and innovation. When a painting component shows up for the first time or in a novel way, it indicates originality. As MIT Technology Review points out, this approach uses network theory in a similar way to tracking epidemics, finding the source of traffic, or tracing popular people in social networks. “In most cases the results of the algorithm are pieces of art that art historians indeed highlight as innovative and influential,” Elgammal and Saleh told Tech Review.
The problem with this type of analysis is defining terms like “creativity” so the algorithm actually produces meaningful results. The researchers say that the method could eventually be refined to analyze other types of art, too, like literature or music. But don’t worry, scholars. Your jobs will be safe until the algorithm can ponder Ceci n’est pas une pipe.
