PandoraBird: Identifying the Types of Music That May Be Favored by Our Avian Co-Inhabitants at DigiHuman Lab Rutgers University

PandoraBird: Identifying the Types of Music That May Be Favored by Our Avian Co-Inhabitants is a site-specific installation that uses computer vision and interactive software to track the music choices made by local feeder birds. The project is in collaboration with the computer scientist Dr. Ahmed Elgammal and the The Art and Artificial Intelligence Laboratory at Rutgers University, which is a platform for the artistic uses of computer vision and machine learning.

Aims and Objectives
Our current project involves creating one outdoor, sound-emitting, interactive sculpture. This sculpture or “listening station” plays music in a dedicated genre: classical, country or rock. Lifted into the air on a 10ft. post, the stations will features a directional sound cone in the form of a hood (PandoraBird design, figure 2). This structure covers an external speaker and a customized mp3 player running a music application that has been specifically created for wild birds. This audio hood is mounted over a tray of wild bird food. A small camera, mounted at the tray level uses computer vision to quantify the number and species of birds feeding during each musical interlude. A solar panel and rechargeable battery pack provid each listening station a contained, renewable power source. The PandoraBird project is a mobile, self-sustaining learning system that can be exhibited at museums, sculpture parks and other outdoor venues that may be visited by wild feeder birds. The computer interfaces with the iPod app, effectively allowing each avian species to identify which tunes it prefers in a given genre, and to build a species-specific database of favored music. The station invites viewer participation on the ground and has a webcam that allows humans (all over the world) to watch and listen to the birds’ musical choices.

Background
In 2007, Demaray collaborated with the videographer James Walsh on Listening Stations for Birds, That Play Human Music (art images, figure 1) Created for the woods of the Abington Art Center in Pennsylvania, a forested sculpture garden that is surrounded by the suburbs of Philadelphia, the piece considered the way that life forms, human and otherwise, may interact in a shared environment. Set along a secluded nature trail, this installation was comprised of a series of four sound-emitting sculptures, each playing a loop of either classical, jazz, country, or rock music and offering a tray of wild bird food. While this early work aimed to see what kind of music birds might like, its primary motive was to get viewers out into the woods, to interact with other species and to consider the impact of our presence on other life forms. Observations on the musical preferences of the feeding birds were noted by park visitors on worksheets. While the piece was popular with park visiter and local birds alike, data collection proved inconclusive, largely due to issues with bird observation and identification.

18_BirdStacolor

Listening Stations for Birds, that play human music, Elizabeth Demaray and James Walsh, Abington Art Center, Jenkintown, PA 2007

For Demaray and Walsh, the most difficult part of this ambitious installation was however having to choose the four or five tunes that were played at each listening station. How does one select what songs a bird might like best? The team ended up choosing works that they felt might be considered human masterpieces in each category. They concluded that if they were going to give a gift to these other life forms—if this was in fact an act of trans-species giving—it should be the best that humans have to offer.

With advents in the field of computer vision, The PandoraBird Project is now be able to create an interactive system that may actually be able to identify which specific tunes individual birds prefer. Elgammal’s work group in the Department of Computer Science specializes in using computer vision for fine-grained recognition, which is the problem of recognizing subordinate categories. In a study titled Write A Classifier: Learning Fine-Grained Visual Classifiers from Text and Images (NSF #1409683), his group is investigating algorithms for automatically recognizing localized body parts. This study supports the PandoraBird system by creating algorithms that automatically recognize bird species from images based on text descriptions of these species

Rationale
There is ample circumstantial evidence that many avian species pay attention to human sound. In the US Mid-Atlantic Region, cat birds and mockingbirds replicate noises made by people. In Australia, the lyre bird even learns human tunes and teaches them to successive generations of its young. Utilizing a computer vision system, PandoraBird may allow us to better understand, and ultimately communicate with, the other life forms in our shared environment.

The PandorBird Project Design, figure 2

PandoraBird

Collaborative Design
The offerings in each genre of music are chosen using standard criteria from web-based “music-discovery services,” using melody, harmony, rhythm, form, and composition. The system will initially begin with a small database of different musical compositions in its defined genre. When a bird feeds during one of these melodies, PandoraBird uses computer vision to record its species and length of stay. If the feeding continues to the end of a piece of music, the system will select another melody with similar qualities. The presence of an individual bird at the feeder is logged by the listening station as a “thumbs up” or “thumbs down”, for the piece of music currently playing and this feedback refine the system’s playlist.

The Significance of the Project in Its Field
PandoraBird may be the first example of a computer vision system dedicated to identifying the musical choices of feeder birds. The novel algorithm for species identification and interactive system that the project represents may be used for a wide range of future purposes. We plant to share the data base of preferred human songs in real-time during the project’s installation. In a more immediate context however, the authors of this project maintain that if we are to bombard other life forms with noise, we should begin to consider which types of noise our companion species might prefer.

Implications for Future Project Design Collaborations
Pandora Radio for Birds may be the first project that utilizes new technology to identify which specific tunes individual birds may like. Using this system as a pilot project, we may ultimately begin to create an interactive system that allows birds to make human-type music choices themselves.

FloraBorg Community Update: 3 IndaPlants Up And Running

The IndaPlant Project: An Act Of Trans-Species Giving—originally beginning as a collaboration between the myself and the engineer Dr. Qingze Zou—is designed to facilitate the free movement and metabolic function of ordinary houseplants. In this effort, we have have successfully created a floraborg, a term we coined to describe an entity that is part plant and part robot. This work has recently led to the creation of a larger team which now includes the biologist Dr. Simeon Kotchoni and the computer scientist Dr. Ahmed Elgammal. Our group is currently working on the creation of a floraborg biocyber interface. Addressing the super sensory capacities of plants, this interface allows humans to decipher plant-based information on ecosystem health, the effects of climate change and air pollution. In this capacity, the IndaPlant may allow us to model and support environments that are able to sustain humans and plants alike. A video of the current project plant community can be viewed at  https://vimeo.com/90457796.

Detail of IndaPlant taken at Rutgers University, June 12, 2014.
Detail of IndaPlant taken at Rutgers University, June 12, 2014.

At the project’s inception, I initially intended to mount the plants on light-seeking Brattenberg vehicles. Originally created through a series of thought exercises by the Italian/Australian Cyberneticist Valentino Brattenberg, these simple vehicles utilize a basic schematic for attraction and avoidance. Once the IndaPlant team began considering the possibilities inherent in the creation of a floraborg however, we realized that we could instead wire the vehicle through an Arduino board. This current configuration not only allows for species-specific programming but also supports simple adaptive behavior, in the form of machine learning. The current IndaPlant community consists of three data-sharing, light-sensing, robotic vehicles, each of which can respond to the needs of a potted plant by moving it around in three-dimensional space in search of sunlight and water. The IndaPlant rides on a three-wheeled triangular carriage. An acrylic shell covers the unit’s base and internal components. Inside the unit’s housing, the Arduino microprocessor and three microcontrollers allow the floraborg to be programmed with the specific needs of the species that it supports. This housing provides a plant docking station at its apex and is externally sided with three solar panels, which the robot uses to re-charge its battery pack when the plant suns itself. Six sonar sensors, used for obstacle detection, are externally mounted the base of the unit.

The IThe IndaPlant Project: An Act Of Trans-Species Giving, Elizabeth Demaray and Dr. Qingze Zou, 2014, utilizes machine learning and robotics to facilitate the free movement and metabolic function of ordinary houseplants.ndaPlant Project: An Act Of Trans-Species Giving—originally beginning as a collaboration between the artist Elizabeth Demaray and the engineer Dr. Qingze Zou—is designed to facilitate the free movement and metabolic function of ordinary houseplants.
The IndaPlant Project: An Act Of Trans-Species Giving, Elizabeth Demaray and Dr. Qingze Zou, 2014, utilizes machine learning and robotics to facilitate the free movement and metabolic function of ordinary houseplants.

As an interactive art installation, the IndaPlant Project was created to be shown in a public exhibition space. The artwork is currently housed in the Engineering Department at Rutgers, where the floraborgs have become part of the daily routine. When Dr. Zou’s comes to work in the morning he is greeted by the three IndaPlants, which jostle with one another to exit his office in search of sun in the adjacent hallway. When an IndaPlant is thirsty, a moisture sensor sends a signal through the unit’s central processor which may decide that its plant species needs water. If so, the unit will locate a water dispenser in the hallway, via an inferred sensor. If a floraborg is in the immediate vicinity of the watering station, passer-buys are invited to give the plant a drink. IndaPlant Project status updates and current videos can be seen at elizbethdemaray.org.

IndaPlant Community Goes Live Each Day From 10:00am to Noon Via Webcam

I am thrilled to report that our current community of three IndaPlants (IP’s) from the IndaPlant ProjectAn Act of Trans-Species Giving went live via a webcam during an exhibition at the Association of Environmental Science Studies in New York this June. Visitors to the gallery at AESS were able to watch the floraborgs (part-plant/part-robot entities that use machine learning to locate sunlight and water) navigate the hallways of the School of Engineering at Rutgers University from 10:00am to noon each day.

One of the interesting aspects of this project is that the IP’s have become part of the daily routine at Rutgers University. When my collaborator on the project, Dr. Qingze Zou, comes to work in the morning he is greeted by the IndaPlants, which jostle with one another to exit his office in search of sun in the adjacent hallway. When an IndaPlant is thirsty, a moisture sensor sends a signal through the unit’s central processor which may decide that its plant species needs water. If so, the unit will locate a water dispenser in the hallway, via an inferred sensor. If a floraborg is in the immediate vicinity of the watering station, passer-buys are invited to give the plant a drink.

My primary interest in creating this art piece lies in the poetic implications of turning an immobile houseplant—which is completely dependent upon human largesse and care—into a free agent. The project has however grown in addressing the relationship of the built to the natural world. The work has led to the synergistic creation of a larger team and what may be a truly significant scientific breakthrough in communicating with plants about the nature of our shared environment. In addition to myself and Dr. Zou, the IndaPlant team now includes the biologist Dr. Simeon Kotchoni and the computer scientist Dr. Ahmed Elgammal. With these joint capabilities our group is now working on the creation of a floraborg biocyber interface. Addressing the super sensory capacities of plants, this interface allows humans to decipher plant-based information on ecosystem health, the effects of climate change and air pollution. In this capacity, a super sensory IndaPlantV2 (IPV2) may allow us to model and support environments that are able to sustain humans and plants alike.

The project is currently up for multiple grants that will allow us to close a positive feed back loop between the plant and its robotic cartage and we have hight expectations for what the future will bring for our floraborgs.

ED

IndaPlant FloraBorg Project Debut French

IndaPlant FloraBorg Project, French debut. The IndaPlant Project had its art debut at CAMAC [art.science.technologie]  Marnay-sur-Seine last night. All my thanks to Laetitia Brion and Carolina Cruz for the French translation work on the wall didactics. And special thanks to Jean Yves, Director of CAMAC for the exhibition space. Below is the French project statement.

CAMAC Center D'Art Marney, Art, Science, Technology, 2013
CAMAC Center D’Art Marnay, Art, Science, Technology, 2013

IndaPlant: Un dispositif qui permet aux plantes de se déplacer pour trouver de la lumière et de l’eau.

Beaucoup de gens ont des plantes d’intérieur chez eux. Le problème est que les plantes

d’intérieures nécessitent un arrosage régulier et ont besoin d’être dans un endroit où elles reçoivent de la lumière.

Surnommé IndaPlant, l’appareil réalisé par le professeur Elizabeth Demaray et le Dr Qingze Zou est à l’écoute des besoins de la plante et recherche de la lumière et de l’eau si celle-ci en a besoin.

Á l’aide d’une carte Arduino et de certains capteurs, ce dispositif est non seulement capable de trouver de la lumière et de l’eau mais aussi de déplacer la plante vers cette zone.

Il est également capable de communiquer avec les autres IndaPlants. Ainsi, si une plante a découvert de l’eau ou de la lumière, le dispositif alerte les autres Indaplants et tous viendront éponger la « soif » d’eau et la « faim » de lumière des plantes.

The IndaPlant FaunaBorg, from the IndaPlant Project, An Act of Trans-Species Giving, 2013
The IndaPlant FaunaBorg, from the IndaPlant Project, An Act of Trans-Species Giving, 2013

En plus de permettre à la plante de s’alimenter, IndaPlant peut aussi recharger sa batterie par le biais de capteurs solaires. IndaPlant a également six capteurs qui lui évitent de heurter des objets dans la maison.

La chose la plus surprenante est qu’il est en mesure de connaître exactement le type de nutriments dont a besoin la plante et en quelle quantité car la carte Arduino est programmée pour connaître les habitudes alimentaires de la plante transportée.

Cette plateforme robotique sera présentée lors du Symposium International des arts électroniques (ISEA) qui aura lieu à Sydney, en Australie, du 7 au 16 juin 2013.