Qualifier
Mitja Hmeljak
- Underlying Theme: Music Visualization
- Target exam time: 10:00 AM, Friday July 2, 2004
- Place: LH101
- Focus Topics
- Auditory Perception
- Music Cognition
- Computer Graphics for Sci.Viz. and Virtual Reality
- Sonification
- Committee Members
- Prof. Robert Port (Cog.Sci.) - Topics 1, 2
- Prof. Andrew Hanson (CSCI) - Topics 3, 4
- Prof. Michael Gasser (CSCI) - Topics 1, 2
- Prof. Eric Isaacson (Music) - Topics 1, 2
- Bibliography
Audition and Vision
A. What are the fundamental psychological differences and similarities between audition and vision?
Are there any unbridgeable differences?
Are those differences basically due to the information itself or to our neurocognitive constraints?
B. What sort of goals should a music visualization system try to achieve?
Is there a reason to expect that music, an essentially temporal and auditory phenomenon should be amenable to visualization?
What advantages could be expected if it could be done?
Might visualization of music have any consequences for listener/lookers once they are accustomed to such a transformation?
A: Features of auditory vs. visual perception
- temporal characteristics, or changes over time
- human hearing well designed to discriminate between periodic and
aperiodic events and can detect small changes in the frequency of
continuous signals -> advantage of auditory
over visual displays:
- fast-changing or transient data that might be
blurred or completely missed by visual displays may be easily
detectable in even a primitive, but well-designed auditory display
- auditory perception advantages:
- detection of complex temporal data
- data embedded in other, more static,
signals
- visual vs. auditory spatial field
- visual perception disadvantages:
- visual attention may be diverted from a particular location: visual events may be missed by being outside of visual range -> typical use of auditory monitoring and alarm applications
- visual perception advantages:
- information can be visually 'stacked' in space instead of being 'stacked' in time (which relies on visual memory)
- use of 'small multiples' for multivariant data:
- once the viewer learns one presentation of data, the same
presentation can be repeated for additional data: visual detection of
differences in pattern
- avoids recency bias (i.e. giving more relevance to the most recent sensory input)
- visual information can be augmented by integrating image and explanation (e.g. words/numbers/markers) and multiple cues without masking effects - much more problematic with auditory information
- quantitative visual display:
- (with or without scales, grids, etc) direct quantitative comparison of complex data
B: music visualization:
- music notation is music visualization
- advantages of music visualization:
- music occurs in time: to perceive its structure, transformation into spatial/visual
- simultaneous presentation of past / future events
- several aspects of a composition can be presented in a single (static) visual display
Techniques in Visualizing Music
Summarize techniques found in your readings for visualizing music in
(a) audio (i.e., derived from a performance) and
(b) symbolic (i.e., derived from the score) form.
What are the relative benefits of each?
What are their weak points?
Symbolic visualizations:
- Brinkman/Mesiti:
- graphic representations of musical structure
- extracted from printed score or equivalent information
- note plots
- time,pitch -> x,y
- utilization of acoustic space
- part plots
- include lines i.e. pitches connected in explicit parts
- rhythmic structure visible
- analytic abstractions
- space plots: use of total pitch space
- band plots: high+low+average of used pitches
- aggregate plots: use of pitch classes
- dynamic shape
- etc.
- Malinovski: Music Animation Machine
- Bach Trio Sonata IV, 3rd movement: color used to separate parts:
- Hiraga/Fujishiro, Smith/Williams, etc:
- as above, but in 3D:
- structure?
- Symbolic vs Audio signal visualization:
- Self-similarity (Foote/Cooper)
- short time scale:
- long time scale:
- demo
Visual Display of Rhythm
You are designing a system that will display visually the RHYTHM of an unfamiliar piece of music in real time.
The system should represent how a listener's sense of the rhythmic structure emerges and evolves in time.
You can base it on any theory of rhythm, for example, that of Lerdahl
and Jackendoff (though they won't help with how rhythm emerges in real
time).
Discuss both representation and implementation issues for the system.
- rhytmic structure
- non-overlapping
- with exceptions: units at any level can overlap (possibly but
not commonly including in the overlap entire units at lower levels) -
but this is the only exception to an otherwise non-overlapping structure
- recursive
- allows similar grouping structures to exist at any hierarchical level
- adjacency
- nonadjacent units can not be grouped together at any level
- metrical structure (as from GTTM)
- phenomenal accent := any event at the musical surface
causing emphasis or stress to a moment in the musical flow (attack
points, local stresses, sudden changes in dynamics, timbre, leaps in
pitch, etc...)
- structural accents := any accent caused by melodic or harmonic points of gravity, i.e. cadence, tonal motion
- metric accent := any beat relatively strong in its metrical context
- metric accents:
- created by phenomenal accents, which provide cues with their regularity, or create ambiguities or syncopation by an absence of regularity
- mental contruct inferred from phenomenal accents on the musical surface
- Interaction of grouping and metrical structure
(associational structure := "web" of motivic (timbral, etc)
associations between potentially disjoint groups, not part of the
rhytmic structure of a musical piece)
Beat spectrum visualization (from Foote)
- non real-time example: rhythmic structure of a pop tune, used in a music video
Procedural Animation and Sonification
(1) give a detailed example of a non-trivial procedural animation, and then
(2) indicate what methods, algorithms, and heuristics you would use to produce an appropriate sonification of the animation.
Procedural Animation
- particle set animations
- generate new particles if necessary
- assign new particles initial attributes
- initial position, speed, direction, size, transparency
- shape
- lifetime
- discard exhausted particles
- move all current particles according to their scripts
- render all current particles
- example: sphere eversion by principle of energy minimization
- computations for energy minimization:
- only computed the second half of the eversio
- starts with a surface with the special 4-fold symmetry of the halfway model
- perturbation: following path towards lesser energy
- creation/destruction of triangles:
define facet attribute flat integer; lfac:=.85; tfac:=.9;
flat_tri := {
set facet ff flat max(ff.edge,dihedral)<.18;
delete edges ee where length<.1/tfac and min(ee.face,flat);
refine edges where length>.7*lfac and dihedral>.5;
};
trian := { n lfac; u; l lfac; u; g; w .001/tfac; t .05/tfac; flat_tri; u; g; V; u; };
parambience/sonification
mapping data to specific dimensions of sound:
Last updated: July 2, 2004
mitja (at) indiana . edu
|
