Trouble at
t'LeedsDataMill:

Oompah.py
Big Data meets Big Brass

Nicholas H.Tollervey / @ntoll
Simon Davy / @bloodearnest

Agenda

  • Oompah.huh..? (Nicholas & Simon)
  • Data analysis. (Simon)
  • Musical mischief. (Nicholas)
  • Questions. (You)
  • Performance! (Enderby Brass Band)

Oompah.huh..?

"We tell stories..."

LDM Hack Day

Can we tell an engaging story about Leeds with data?

http://leedsdatamill.org/dataset/leeds-city-centre-footfall-data

Crazy idea #1

Turn the footfall data into music.

Crazy idea #2

What musical genre epitomizes Leeds?

Our aim was to write Python scripts that turned raw footfall data into music for brass band called "Rhythm of the City".

And so, oompah.py was ironically checked into Github.

Data Analysis Munging

Excel + Messy Data = Aargh!

Please use CSV

Tuesday
name         |     median |   min            histogram               max|
-------------------------------------------------------------------------
00:00        :    33.0 ms [   1.0 ▃▅▆▇▆▇▆▃▄▃▃▂___  _   __  _ _   _ 382.0]
01:00        :    30.0 ms [   4.0 ▃▆▇▇▇▅▄▆▃▂▃________ __ _____ ___ 397.0]
02:00        :    28.0 ms [   5.0 ▃▇▇▆▇▅▅▅▄▃▂▂▂____   __ _ ____  _ 321.0]
03:00        :    23.0 ms [   5.0 ▃▇▇▅▆▅▃▆▃▂▂_______    _  ____ __ 264.0]
04:00        :    12.0 ms [   2.0 ▅▇▇▅▄▃▂▂______ __ _       ___ __ 180.0]
05:00        :    17.0 ms [   1.0 ▄▄▆▄▅▇▅▃▃▂___ ____  __     __ __ 186.0]
06:00        :    64.0 ms [   6.0 ▂▃▃▄▄▂▃▂▄▂▃▂_▃▄▅▇▇▅▄▅▇▅▄▂▂_▂___▂ 163.0]
07:00        :   205.0 ms [   5.0 ▂___ ___▂▃_▂_▂_▃▄▅▇▆▅▇▄▇▃▄▃___ _ 405.0]
08:00        :   470.0 ms [  12.0 _ __ ___▂_▂▂▂▂▃▅▄▄▇▅▄▆▇▄▅▄▂▂____ 745.0]
09:00        :   554.0 ms [  14.0 _____▂_▂▂▆▅▄▅▄▆▇▇▇▆▇▄▄▃▂_____  _ 1096.0]
10:00        :   931.0 ms [  24.0 __ _▂▂▂_▂▃▄▄▄▅▅▇▇▄▄▃▄▂____ ___ _ 1970.0]
11:00        :  1357.0 ms [   0.0 _     _ _____▂▂▄▃▅▅▇▇▆▄▃▃▃______ 2478.0]
12:00        :  1823.0 ms [  56.0 ___ _▂__▂▃_▂▄▆▆▇▅▄▃________    _ 5652.0]
13:00        :  1823.0 ms [  54.0 _________▃▂▂▂▂▃▂▆▇▆▇▅▄▄▃▂▂▂_____ 3014.0]
14:00        :  1534.0 ms [  63.0 ___▂▃▂▂▄▃▅▃▃▅▅▇▇▅▅▅▃▃▂▃▂▂___▂___ 2664.0]
15:00        :  1472.0 ms [  76.0 _ ________▂▃▂▃▄▄▇▇▅▆▄▄▃▅___▂____ 2544.0]
16:00        :  1416.0 ms [  67.0 ___ _▂__▃▂_▂▂▂▄▄▃▄▄▆▇▇▇▅▇▅▅▂▂▃▂_ 2317.0]
17:00        :  1140.0 ms [  42.0 _______▂▂▂▂_▂▃▄▃▃▃▆▄▆▇▄▆▃▃▂___ _ 2081.0]
18:00        :   584.0 ms [  22.0 __▃▃▃▂▂▃▃▂▅▃▄▃▅▇▇▇▇▅▆▆▄▄▃▂▂▂_  _ 1486.0]
19:00        :   291.0 ms [  12.0 ▃▃▂▃▅▂▃_▂▂▃▅▄▅▇▇▆▅▅▂▅▃▃▂▂▂▂▂____ 788.0]
20:00        :   176.0 ms [   2.0 ▄▄▅▃▃▃▃▃▂▄▃▅▄▅▇▆▄▇▅▄▄▅▆▅▃▃▂▂▂___ 497.0]
21:00        :   113.0 ms [   2.0 ▄▆▇▅▂▆▄▂▄▃▅▆▆▆▄▆▃▅▃▅▆▆▂▅▅▂▂▂_▂ _ 442.0]
22:00        :    79.0 ms [   6.0 ▇▅▆▇▅▅▇▃▅▃▅▅▆▄▆▃▆▅▃▅▃▃▄▇▂▃▂▃▂__▂ 344.0]
23:00        :    52.0 ms [   2.0 ▇▆▇▄▃▅▅▃▃▄▅▄▆▄▄▄▅▂▆▃▅▃▃_▃▂_▂ ▂▂_ 345.0]
                 

Time Travel FTW


    WEEK = {}
    WEND = {}
    for f in glob.glob('*.csv'):
        voice = f.split('.')[0].lower().replace(' ', '_')
        WEEK[voice] = defaultdict(list)
        WEND[voice] = defaultdict(list)
        with open(f) as fp:
            for row in csv.DictReader(fp):
                t = row['Hour']
                d = row['WeekDay']
                if d.lower() in ('saturday', 'sunday'):
                    data = WEND
                else:
                    data = WEEK
                data[voice][t].append(int(row['Count']))
                    

Pitch


    def pitch(data, scale):
        """Generate notes from the scale driven by the data."""
        index = 0
        for d in data:
            index = (index + d) % len(scale)
            yield scale[index]
                    

Duration


    def rhythm(data, mn, mx):
        """Produce note durations from provided patterns,
        based on mean intensity
        """
        i = threshold(data)
        pattern = patterns[i]

        # use the data to chose which pattern,
        # so the process is deterministic
        index = 0
        for d in data:
            index = (index + d) % len(pattern)
            for duration in pattern[index]:
                yield duration
            yield None  # indicates a bar has been produced
                    

Voice of the Street


    def voice(data, scale, mn, mx, length=8):
        """Generate a voice from the data, combining pitch and rhythm"""
        bars = 0
        note_iter = pitch(data, scale)
        for duration in rhythm(data, mn, mx):
            if duration is None:
                bars += 1
                yield '|'
                if bars == length:
                    raise StopIteration()
            elif duration[0] == 'r':
                yield duration
            else:
                note = next(note_iter)
                if '%s' in duration:
                    # more complex templated duration (e.g. tuplets)
                    yield duration % note
                else:
                    yield "%s%s" % (note, duration)
                    

Musical
Mischief

BLOOPS!

Charles E.Ives

Music affects because of the effects of sound.

Music engages us to listen.

Music, in some sense, tells a story in sound (there is a discernable narrative).

Can we reveal the story of the data through music?

Elements of Music

  • Pitch: how high or low a note sounds.
  • Rhythm: a sequence of different durations.
  • Melody = Pitch + Rhythm.
  • Timbre: the quality of a sound (e.g. the difference between a flute and violin).
  • Dynamics: loudness.
  • Key: the set of available pitches.
  • Harmony: how pitches sound together as polyphony.
  • Texture: how the harmony is "voiced".
  • Etc, etc...

A Musical Mapping

The ingredients for generating "Rhythm of the City".

  • Each instrument group in the band represents a specific area of the city. Ergo, cameras/locations are differentiated by timbre.
  • The amount of footfall in a location changes the intensity of dynamics and rhythm (more footfall = louder / busier melodies).
  • Use "nice" sounding pentatonic keys.
  • Time of day is indicated by key.
  • Each hour of the day is represented by an eight bar fragment.
  • Tubular bells sound out the hours every eight bars.

The Result!

Does it work..?

IT'S TUBA TIME!

1am, 7am, 12 noon

Questions

Performance

Enderby Brass Band, 2014