“I never painted dreams. I painted my own reality,” said Frida Kahlo, Mexico’s woman painter known for her many portraits, self-portraits, and works inspired by the nature.
The same cannot be said for the new artist-in-residence FRIDA at Carnegie Mellon University’s Robotics Institute, whose name is inspired by the Mexican artist. FRIDA is not an artist but a robotic arm equipped with a paintbrush that uses artificial intelligence to collaborate with humans on works of art. Just ask FRIDA to paint a picture, and it gets to work putting brush to canvas.
FRIDA stands for Framework and Robotics Initiative for Developing Arts. The project is led by Schaldenbrand with RI faculty members Jean Oh and Jim McCann and has attracted students and researchers across CMU.
Users can direct FRIDA by entering a text description, providing other artwork to inspire its style, or uploading a photo and asking it to paint a representation of it. The team is also experimenting with other inputs, including audio. They played the ABBA song “Dancing Queen” and asked FRIDA to paint it.
The robot uses AI models similar to those that power tools like OpenAI’s ChatGPT and DALL-E 2, which generate text or an image, respectively, in response to a text description. FRIDA simulates how it would paint a picture with brush strokes and uses machine learning to evaluate its progress as it works.
But what are the results in the end? Impressionistic and whimsical, according to researchers. The brushstrokes are bold and lack the precision so often sought in robotic endeavors. If Frida makes a mistake, it riffs on it by incorporating the errant splotch of paint into the end result.
“FRIDA is a project exploring the intersection of human and robotic creativity,” McCann said. “FRIDA is using the kind of AI models that have been developed to do things like caption images and understand scene content and applying it to this artistic generative problem.”
The robot artist taps into AI and machine learning several times during its artistic process. First, it spends an hour or more learning how to use its paintbrush. It then uses large vision-language models trained on massive data sets that pair text and images scraped from the internet to understand the input. AI systems use these models to generate new text or images based on a prompt.
One of the biggest technical challenges in producing a physical image is to reduce the simulation-to-real gap, the difference between what Frida composes in the simulation and what she paints on the canvas.
Once FRIDA’s human user has specified a concept of the painting he wishes to create, the robot uses machine learning to create its simulation and develop a plan to make a painting to achieve the user’s goals. FRIDA displays a color pallet on a computer screen for a human to mix and provide to the robot.
At this point, armed with a brush and paint, the robot artist will make its first strokes. Every so often, the robot uses an overhead camera to capture an image of the painting. The image helps FRIDA evaluate its progress and refine its plan if needed. The whole process takes hours.
“People wonder if FRIDA is going to take artists’ jobs, but the main goal of the FRIDA project is quite the opposite. We want to really promote human creativity through FRIDA,” Oh said. “For instance, I personally wanted to be an artist. Now, I can actually collaborate with FRIDA to express my ideas in painting.”
AI-powered FRIDA robot collaborates with humans to create art
Source: Tambay News
0 Comments