Using Technology to Incite Social Interaction and Affect Mood

Through our technical exploration and interest in public collaboration via music and visuals, we have become more interested in the social aspect of our project;  using technology to instigate social interaction, influence mood, and allow us to study the effects under the conditions of different social spaces.

Full Body Gestures

The basic concept for this would be to place an interface in public spaces, such as pedestrian streets, bus stops, even bathrooms; and use techonology which recognises gestures to create music and visuals based on these gestures.

Hand Gesture Examples

Various gestures would launch different audio samples, such as a drum beat, or bass line, along with visuals which  would play as long as a gesture is held.  The more participants involved, the more music would play, and more visuals created.  For example, one participant may launch drum samples with blue visuals; while the other launches bass samples with yellow visuals.

Through participation, social interaction is instigated, with more participants adding to the whole piece.

To solidify this social interaction, the piece would aim to affect the participants' mood , using specific colours,in a positive manner, and create an association of positive mood, with interaction in public space.  Colour and its affectation of human mood, productivity and social interactivity have been studied in depth, and has been proven to have a large effect on us, with colours such as red making us more likely to be more aggressive, or certain shades of green increasing productivity.

Colour chart and emotions associated with them.

The actual installation itself would use motion tracking technology via the use of readily available hardware such as a standard webcam, (or microsoft kinect, which would allow for a wider variety of gesture tracking). Motion tracking software such as Motion Studio, and Max for Ableton which would allow us to convert gestures into MIDI controls which would in turn launch visuals and audio based on these gestures.

 Visuals displayed via screens or projections, with most of the 'work' being done by the software.  This means that the installation would not require large amount of space and could be placed in a multitude of locations.

Using MIDI control

Over the two weeks our aim has been to control a virtual keyboard using physical input, and also via data taken from the web.

For the physical input, I have been studying the use of MIDI, and using non-standard physical inputs, such as a computer mouse and Xbox 360 controller.

The physical hardware I am using is an Xbox 360 controller, which has standard button inputs, analogue inputs which detect range of motion, and pressure sensitive triggers. This allows a wide range of inputs, which I hope we can apply later in our project, with the aim to be able to use multiple arduino sensors such as heat, light etc. as the final aim.

The software I have been using is Ableton Live 9, a Digital Audio Workstation, which contains an infinite amount of virtual instruments, recognises MIDI input, and is also capable of MIDI mapping, meaning I can map controls to any part of the software (from playing a note on the virtual keyboard, changing volume, filters, eq's etc.)

Ableton has software in-built to recognise actual MIDI instruments, such as the APC 40, electronic pianos etc.,  but the problem I am trying to solve is how to use devices that are not designed to be used for MIDI control.

To bridge this gap, I found GlovePIE, which is a software that is used to map keyboard inputs to controllers for use with video games that do not have controller support as standard, but instead I will be using it to  to map any form of input, into MIDI, which is then put through an internal MIDI port, using LoopBe.

This means that the basic map of input is

Xbox Controller - USB - GlovePIE - LoopBe - Ableton Live.

Below are images of the software interaction.

midi.DeviceOut = 2  is where GlovePIE is sending the MIDI to, in this case channel 2.  LoopBe acts as the bridge, taking the input via channel 2 and sending it to Ableton Live.

midi.channel1.c4 = XInput.A means that when I press the 'A' button on my xbox controller, it activates the midi for the note A4. 

These are the notes where individual sounds, such as a snare, kick or other samples can be mapped.

Below is a screencap of these two interacting.

Below is a video of the controller in action

Data Visualisation, Infographics, and Art

I spent this week looking up various articles and websites in an attempt to discern the differences and similarities between these different formats.  The amount of information available is quite large, and so I have tried to condense these down so that we all have a decent understanding of the formats.  

There is much less information available on data art as it is quite subjective, but previous blog posts have actually covered some different installations and exhibitions of data art.

Data visualisation is viewed by many disciplines as a modern equivalent of visual communication, and is concerned with the visualisation of quantitative data in a way that it can be more easily understood.  It is not owned by any one field, but instead finds different interpretations across mulitple different fields.  It is extremely useful when trying to convey and make sense of large quantities of data.

"Data visualization is both an art and a science. The rate at which data is generated has increased, driven by an increasingly information-based economy. Data created by internet activity and an expanding number of sensors in the environment, such as satellites and traffic cameras, are referred to as "Big Data". Processing, analyzing and communicating this data present a variety of ethical and analytical challenges for data visualization"

This visualisation can be done in various ways across these different fields, from simple bar graphs, to intricate artistic interpretations, whilst still being considered to be under the umbrella of data visualisation. 

This leaves one to try and make the distinction between data art,  visualisation, and simple infographics.

"Infographics are generally created for the purpose of telling or explaining a specific story, and will usually be intended for a specific audience, thus Infographics are subjective. They are self contained and discrete: it is information presented with context; sensitivity designed and presented in a way that is accessible for an audience. The graphic design of an Infographic will be obvious; as it will be designed for that audience, or to fit in with a website or off-line publication's pre-existing style."

Last FM Top Artist Visualisation

Above is a visualisation/infographics of top artist plays from the music streaming website last.fm, from March 2008 to March 2014.  Both top artists and genres are displayed in a creative yet informative fashion.

Mouse Tracking Visualisation

Above is a representation of mouse movements on a screen across a day. Lines trace the path of the mouse across the screen, while circles represent idling time. The images reveal differences in each user's mousing behavior.  

- Popular Science Magazine, January 2015

Data Visualisations are far more likely to be generated automatically through the use of algorithms or computer programs. The process that is used to create them could be applied to another data set. 

They are likely to be presented without context.  The system of organising the data will perhaps be proprietary, and it is this system itself that becomes the interface that the data is accessed through. The data itself becomes its own architecture. The graphic design of it (in the sense of the look and the feel) will likely be deliberately less obvious, and take a back seat to the data itself.

Direction for project

During today's meeting with the group, and after Paul Green's meeting, we had multiple ideas and directions for the project.  

These ideas covered a wide range of topics including wearable technology, advertisement using projection/holographic imagery, methods for vehicular and pedestrian traffic manipulation, as well as musical and other artistic representations of a city and it's atmosphere.

Stemming from the idea of a 'musical square', in which a designated square of the city would take in data such as the amount of people,  wind speed, time of day etc. to play music appropriate to the mood,  we decided on the initial direction for our project.

This direction is the collection and use of data, to be represented as some form of art, be it a painting (digital or otherwise), music, digital artwork, video installation etc.

One example was to take in data from, for example, a foyer of an office building, and have the people in the building rate their mood, and have a digital representation of the mood displayed in say, the lighting of the room.

The scope and direction this could take is still very broad, and so we are spending the next week trying to come up with ideas and filter them into ways that are both original, interesting and also within our own capabilities, and the technology needed to create the product itself.