-
Notifications
You must be signed in to change notification settings - Fork 2
/
Copy pathtime.html
458 lines (396 loc) · 41.1 KB
/
time.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Time</title>
<meta name="description" content="">
<meta name="author" content="Graham Wakefield">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="apple-mobile-web-app-capable" content="yes">
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent">
<link rel="stylesheet" href="css/basic.css" type="text/css" />
<link rel="stylesheet" href="css/github.css" type="text/css" />
<style>
img {
max-height: 75vh;
}
td { vertical-align: top;}
header {
background-color:#f5f5f5;
font-size: 75%;
padding: 0.5em;
}
footer {
background-color:#f5f5f5;
font-size: 75%;
padding: 0.5em;
}
</style>
</head>
<body class="centremaxwidth960">
<header><a href="index.html">Foundations of Digital Media</a></header>
<ul>
<li><a href="#time">Time</a><ul>
<li><a href="#making-sense-of-time">Making sense of time</a></li>
<li><a href="#time-lines-and-nested-durations">Time-lines and nested durations</a></li>
<li><a href="#static-and-dynamic-unity-and-change">Static and dynamic (unity and change)</a></li>
<li><a href="#time-waits-for-no-machine">Time waits for no machine</a></li>
<li><a href="#articulations-of-sonic-time">Articulations of sonic time</a></li>
<li><a href="#digital-representation-of-sound">Digital representation of sound</a></li>
<li><a href="#sampling">Sampling</a></li>
<li><a href="#the-great-computer-music-problem">The Great Computer Music Problem</a></li>
<li><a href="#music-iii">Music-III</a></li>
</ul>
</li>
</ul>
<p><a href="http://xkcd.com/1190/"><img src="http://imgs.xkcd.com/comics/time/8eb156cce408df8bb83528382d6a2aa2ce6c74f3c573fd12b058cd1c56420672.png" alt="xkcd"></a></p>
<hr>
<h1 id="time">Time</h1>
<p>Computation is an inherently temporal medium: it comprises information processes that can be described abstractly (through data and code) but these unfold in actuality over time, step by step at the lowest levels. Similarly, time is integral to our perception of, and interaction with the world. If time is so important, how can we represent and reason with it? How can we experience and create with it? Are there parallels in the treatment of time between art and computing? Are there differences we can learn from?</p>
<h2 id="making-sense-of-time">Making sense of time</h2>
<blockquote>
<p>"Music is a secret calculation done by the soul unwaware that it is counting", W. Leibniz.</p>
</blockquote>
<blockquote>
<p>"[Time] is like holding a snowflake in your hands: gradually, as you study it, it melts between your fingers and vanishes" --Carlo Rovelli, The Order of time</p>
</blockquote>
<p>We don't really know how time works. Physics tells us that time is not <em>continuous</em>, <em>directed</em>, nor <em>singular</em> and <em>uniform</em> -- the <em>past</em> is not fixed, the <em>present</em> does not exist. These seem highly incongruous with our perception of time, or our reasoning about it, or our expressions of it in time-based arts.</p>
<ul>
<li><p>To be countable, we measure against a known repeating period (cycle)</p>
<ul>
<li>Breath (of life), pulse, endocrine, sleep, hormonal, menstrual, etc.</li>
<li>Day, tide, moon, season, generation, etc.</li>
<li>Cosmological-astronomical to crystals and atoms, etc.</li>
<li>Complex harmonics of ecosystems, from chirps to lifespans.</li>
</ul>
</li>
<li><p>We can also measure relative rates of non-cyclic times, such as:</p>
<ul>
<li>the "characteristic times" of natural decay, such as the half-life of radioactive elements</li>
<li>attractors of periodic cycles, e.g. predator-prey systems, chaotic systems</li>
</ul>
</li>
</ul>
<h2 id="time-lines-and-nested-durations">Time-lines and nested durations</h2>
<p>A spatial metaphor of time forms a line from past to future. The line could be <strong>ordinal</strong> (a sequence or list, such as the script for a play) or <strong>metric</strong>, in which ecah event has a numeric position and/or each duration has a measurable length, with respect to some measure or clock. This clock is not necessarily 'real time'. </p>
<p><strong>Finite linear time</strong> has a definite beginning (zero time) and end; <strong>infinite linear time</strong> has no definite beginning or end, it just keeps coming. For example, a pre-recorded DVD encodes finite linear time, while the real-time video stream from a CCTV camera has no definite end. </p>
<p><strong>Cyclic</strong> or circular time represents a period that repeats; such as a clock. Within this period, time seems finite. The inverse of the period is the frequency (rate of repetition), with respect to another period/rate. Linear and cyclic time can be combined by representation as a spiral, such as a calendar. Linearizing time suggests the ability to navigate around it: rewind, fast-forward, skipping, scrubbing, scratching.</p>
<p>A <strong>continuous</strong> representation of time can be subdivided without limit. No matter how short the duration, smaller durations can be described within it. This is the representation of time used in understanding analog systems, and the calculus of differential equations (such as function derivatives and integrals). For example, the function <strong>sin(t)</strong> is continuous (where <strong>t</strong> represents a real-valued variable such as time). This is called an <em>implicit</em> function: the value at any position is not given, but can be computed. </p>
<p>A <strong>discrete</strong> representation of time has a lowest temporal resolution below which shorter durations cannot be represented. It describes an ordinal <strong>time series</strong>, a sequence of discrete values. Discrete series are sometimes indicated using square brackets, such as <strong>f[n]</strong> (where <strong>f</strong> is an arbitrary function and <strong>n</strong> is the discrete integer series). It requires a different branch of mathematics: the calculus of difference equations and finite differences. </p>
<blockquote>
<p>We do not know if nature is at root temporally continuous or discrete (it has been debated since at least the time of the early Greek philosophers), however we do know that if time is discrete, it is so on a scale so vastly smaller than what we can perceive, that for practical purposes it may be considered continuous.</p>
</blockquote>
<p>Considering time as a line "spatializes" it, giving a perspective from outside it; this can be very helpful (scripts, scores, etc.) but is not always appropriate, and may lead to the habit of thinking of it as <em>static</em>. </p>
<p>A complemenatary view to time as a line is the view of time as a <strong>set of nested durations of presence</strong>, of which the shortest possible duration is the infinitessmial point of the passing present. Longer durations represent the degrees to which the past endures into the future, as a nested set of widening 'windows' of unity. If I act upon a memory, I effectively persist that memory into a newly made future; in effect, the past of that memory is co-present. </p>
<p>Another view of time, perhaps we can call it <strong>interactive</strong>, is the eternal recurrence of the question: 'what happens next?'</p>
<h2 id="static-and-dynamic-unity-and-change">Static and dynamic (unity and change)</h2>
<p>A simpler distinction we can make separates that which changes from that which does not. Time-based arts (music, film, etc.) can be broken down into "vertical" and "horizontal" structures. The <strong>vertical</strong> structure provides unity: that which remains relatively constant throughout, and thus encompasses the qualities of the whole. Since it influences all parts, vertical structure is often largely outlined in early stages of a work. <strong>Horizontal</strong> structure refers to the temporal form of change: difference, movement, events, repetitions, expectations, surprises, resolutions. </p>
<p>Similarly computations can be broken down into the unchanging <strong>static</strong> and variable <strong>dynamic</strong> components. The rules of a programming <em>language</em> are usually <strong>static</strong>: they are not expected to change during the run-time of the program. On the other hand, the flow of control is partly determined as a program runs, and values in memory can change ("variables") or be allocated and freed as it goes; these are examples of <strong>dynamic</strong> components. </p>
<p>These can be relative divisions; a 'scene' in a film is a relative constant with respect to the frame, but ephemeral (dynamic) with respect to an act. Similarly, a variable may endure for the shortest for loop iteration, or may persist over the entire program's lifetime.</p>
<p><strong>Vertical</strong>:</p>
<ul>
<li>Medum, macroform, frame, style</li>
<li>Materials, technologies, techniques, constraints, rules</li>
<li>Composed by associative, metaphorical, normative and hierarchical relations</li>
<li>Semantics, intentions. Vertical elements of unity may be chosen to best convey the idea, feel, atmosphere, message; or as an experiment to liberate new creativity.</li>
</ul>
<p><strong>Horizontal</strong>:</p>
<ul>
<li>Progressions: beginning as one thing and becoming another.</li>
<li>Also reflections, recollections, repetitions.</li>
<li>Parallel mixtures of speeds (fast/slow) or rates (frequencies high/low), and relative proportions (faster than/slower than).</li>
<li>Bifurcating and coalescing.</li>
<li>Continuous (gradual) or discrete (sudden).</li>
<li>Quantitative (change in size, extent, proportion, measurable, numeric) vs. qualitative (change in kind, nature, tendency, individuality) aspects.</li>
<li>Positive or negative, attractive or repulsive, convergent (affinity) or divergent (contrast).</li>
<li>Effects, causes, intentions, story, destiny, chance.</li>
<li>The subtle energies of dynamic equilibria, homeostatic identity, excitable but quiescent media</li>
</ul>
<!-- Eisenstein believed that “art is always conflict”, the opposition of forces that motivates and shapes action. The opposed forces are dissonance/consonance or tension/release; expectations and their satisfactions or frustrations. But in art the distinctions between horizontal and vertical, and between constrast and affinity, need not be hard and fast. Principles of unity in a work may be guidelines fit to be broken when needed, or a work may pass through phases of different unities or even thread multiple unities into a larger dynamic whole.
### Static and dynamic components in computing
Programming also involves a division of static and dynamic components, usually more explicitly.
The rules of a programming *language* are usually **static**: they are not expected to change during the run-time of the program. On the other hand, the flow of control is partly determined as a program runs, and values in memory can change ("variables") or be allocated and freed as it goes; these are examples of **dynamic** components. Computing inherits from mathematical logic a rigorous attitude toward definitions, thus static and dynamic divisions tend to be more sharply discriminated and pedantically adhered to. However there are times where exceptions are desirable or necessary.
We saw in our first lecture how computing inherited an industrial lineage a linear, task-oriented character: a program having a well-defined job to do defined in advance (as in the analogy of a recipe). From mathematics and cryptography computing inherited the notion that a program's job is to compute an answer. One of the first principles in the theory of computation regards whether a program will or will not terminate with an answer; irrespective of how long that might actually take.
As mainframe computing became established in the 1950's, this evolved into "batch" oriented computing, in which control programs sequentially dispatch to other task oriented "job" programs, for reliability and efficiency. Dumb terminals (in modern terminology, "thin clients") are used to input data for new jobs and monitor output. Gradually this evolved into the UNIX terminal (still present in MacOSX and Linux operating systems), which allows jobs to be created in which the output of one is 'piped' to the input of another.
Although most programs that we consciously interact with today are not linear, task-oriented procedures invoked to compute an answer, they are still built upon these foundations (see the request/response nature of web technologies for example). Time has been absent from many programming languages and interfaces, implicit rather than explicit. That is, we may need to buid our own. For example, Javascript offers only a small number of components for temporality, upon which we will build something more expressive:
```javascript
// returns the number of milliseconds since a program began
performance.now()
// call function `fun` after `period` milliseconds
setTimeout(fun, period)
// call function `fun` after `period` milliseconds, repeatedly
setInterval(fun, period)
// in the browser: call `fun` when the screen is next ready to render (typically about 60 times per second)
requestAnimationFrame(fun)
```
Consider how these could be combined with the various kinds of control flow we encountered in the first lecture:
1. Halt / exit
2. Unconditional jump (label & goto)
3. Choice: conditional jump / if / switch
4. Loops: Repeat / while / for / foreach / break / continue
5. Non-local: subroutines / functions / callbacks / coroutines / continuations / exceptions
6. Parallelism: interrupts / Signals / Multi-threading
7. Dynamic code (linking, plugins, eval)
-->
<h2 id="time-waits-for-no-machine">Time waits for no machine</h2>
<p><img src="img/error.png" alt="error"></p>
<p>Unfortunately, for multimedia, the practical constraints of <strong>real-time</strong> are unavoidable; this is especially sensitive for audio signal processing. To be <strong>timely</strong>, an operation must produce results in less time than the playback of the results requires. Any failure to do so results in a break in the output. </p>
<p>The amount of time it takes for an input event to pass through the computing system and cause experienceable output is the <strong>latency</strong>. Interactive software requires low latency (small fractions of a second) to feel natural. VR and musical applications can require especially low latencies of 5ms or less. </p>
<p>In the conventional view, software development occurs before and after a program runs, that is, outside of <strong>run-time</strong>. But with server programming, in-app scripting, shell scripting, in-game development, live coding etc. this assumption breaks down. The computer music community has been especially active in elevating time to a first-class citizen in programming, through the performance practices of <strong>live coding</strong>. </p>
<p>Further discussion here:</p>
<ul>
<li><a href="http://www.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-30.pdf">Computing needs Time</a></li>
<li><a href="http://impromptu.moso.com.au/extras/sorensen_ow_2010.pdf">Programming With Time</a></li>
</ul>
<h2 id="articulations-of-sonic-time">Articulations of sonic time</h2>
<!--
We have lived in a visually-dominated culture over the last century; it is therefore worthwhile looking at non-visual trajectories for alternate perspectives. For example, how the articulation of sound, particularly music, illuminates the capacities and problematics of time.
-->
<p>Technology has been integral to music history, from luthiers (instrument builders) to composers (techniques) to archives (notational systems) to auditoria (acoustic architecture) to automation (piano rolls) etc. Sound in digital computational media inherited two principal pre-histories, both of which emerged in large part from a musical tradition that was expanding its scope from limited scale systems through serialism and futurism to encompass "any sound whatsoever":</p>
<ul>
<li>Electroacoustic<ul>
<li>manipulations of recorded sound on magnetic tape </li>
<li>that is, time-series data, captured from the real world and re-structured<ul>
<li>speed changes, reversing, fading, mixing & layering, echo delays and tape loops, re-processing through rooms and springs, etc.</li>
</ul>
</li>
</ul>
</li>
<li>Electronic<ul>
<li>construction of raw sound from analog circuits of resistors, capacitors, transistors etc.</li>
<li>that is, systems amenable to mathematical modeling and precise articulation<ul>
<li>oscillators, filters, envelopes, etc.</li>
<li>some research in this area tended toward attempts to replicate realistic sounds of instruments and voices; other research explicitly sought sounds previously unknown</li>
</ul>
</li>
</ul>
</li>
</ul>
<blockquote>
<p>Both these threads remain central to computer music today in the form of samplers and DAWs, modular synthesizers and plugins. </p>
</blockquote>
<p>Composers of the time also began to see all musical parameters as articulations of time at different scales, from macro-structures of an entire composition down to the microstructures of an individual sound event's timbre. </p>
<h2 id="digital-representation-of-sound">Digital representation of sound</h2>
<!--
Sound recording/playback technology requires a mechanism to transform the ephemeral undulations of sound pressure (what we can hear) into a persistent recording medium, such as hardened wax, and another mechanism to the transcribe this record back into waves of sound pressure. In the phonograph (below), introduced in 1877, vibrations cause patterns to be etched into a rotating cylinder.
The modern record player adds an electronic amplifier to drive the movements of a loudspeaker cone, but otherwise follows the same princple. -->
<p>Here are vinyl record grooves under an electron microscope:</p>
<p><img src="https://thevinylfactory.com/wp-content/uploads/2014/10/LongwaysViewElectronMicroscopeImageOfVinlyRecordGroove1-660x400.jpg" alt="http://www.synthgear.com/2010/audio-gear/record-grooves-electron-microscope/"></p>
<p>These grooves are smooth and continuous, since records are an <strong>analog</strong> representation of sound, which is theoretically ideal. Unfortunately it is susceptible to noise and gradual degradation. The <strong>digital</strong> representation of sound on the other hand is completely discrete:</p>
<p><img src="https://external-preview.redd.it/QdN6DWNj93mQSbMHJnUmtAKEGIqBbFTy9tMDVzdFzB8.jpg?auto=webp&s=7891092bf3aeacd97ee804e6060a8b10be1d48bd" alt="A CD under the electron microscope"></p>
<p>At the simplest level, sounds are represented digitally as <strong>a discrete sequence of (binary encoded) numbers</strong>, representing a time series of quantized amplitude values that relate to the variations of compression an expansion in the air pressure waves we hear as sound. </p>
<p>The advent of digital computers promised a kind of unification of the electronic and electroacoustic spaces, since data as a sequence of numbers can be both manipulated as memory (electroacoustic) and generated algorithmically (electronic). That is, the ability to represent sound as data allowed the complete exploration of "any sound whatsoever" to numeric, algorithmic analysis. Composers such as Iannis Xenakis and Herbert Brun attempted to build algorithms that would specify works from individual samples up, disovering entire new realms of noise. </p>
<h2 id="sampling">Sampling</h2>
<!-- This is bound by the sampling theorem and other aspects of [information theory](http://en.wikipedia.org/wiki/Information_theory), just like any other digital representation. -->
<p>Although we can represent continuous <em>functions</em> in the computer (e.g. by name, as in <code>Math.sin</code>), we cannot accurately represent continuous signals they produce, as they would require infinite memory. Instead we <strong>sample</strong> a function, such as a sound being recorded, so rapidly, that we produce a series of values that seem perceptually continuous. Digitized audio a discrete-time, discrete-level signal of an electrical signal. Samples are quantized to a specific bit depth and encoded in series at a specific rate, called the <strong>sampling rate</strong>. </p>
<p><img src="http://upload.wikimedia.org/wikipedia/commons/thumb/a/a5/Analog_digital_series.svg/220px-Analog_digital_series.svg.png" alt="Sampling"></p>
<p>How fast is fast enough? If the function changes continuously but only very slowly, only a few samples per second are enough to reconstruct a function's curve. You do not need to look at the sun every millisecond to see how it moves; checking once per minute would be more than enough. But if we didn't check fast enough, we might miss important information. If we checked it once every 24 hours, it would appear to be stationary. If we checked the sun's position once every 23 hours we would only see a complete cycle every 24 days, and it would make the sun appear to be slowly moving backwards! This phenomenon is called <strong>aliasing</strong>. The Nyquist-Shannon sampling theorem states that the highest frequency that can be accurately represented is one half of the sampling frequency (the <strong>Nyquist frequency</strong>). Anything above this frequency will alias and appear to be a lower frequency. That is, we would have to check the sun at every 12 hours or less to accurately know the rate: enough to capture both night and day (or sunrise and sunset).</p>
<p><img src="http://upload.wikimedia.org/wikipedia/commons/thumb/a/af/CPT-sound-nyquist-thereom-1.5percycle.svg/250px-CPT-sound-nyquist-thereom-1.5percycle.svg.png" alt="Aliasing"></p>
<p>An event that occurs repeatedly, like the sun rising and setting, can be described in terms of its repetition period, or cyclic frequency (the one is the inverse of the other).</p>
<pre><code>period = 1 / frequency
frequency = 1 / period</code></pre>
<p>In units:</p>
<pre><code>seconds = 1 / Hertz
Hertz = 1 / seconds</code></pre>
<img src="img/timescales_of_music.png" />
<p><em>Image taken from <strong>Roads, Curtis. Microsound. MIT press, 2004.</strong></em> </p>
<!--
- The sun's traversal of the Earth's sky has a period of 1 day (approximately 93,600 seconds), which is 0.00001068376 cycles per second (Hz). Long periods imply low frequencies.
- Echoic memory can persist up to a few seconds (demonstration with noise loops).
- The heart beats around 60-100 times a minute (bpm). This also happens to be the typical range of frequencies for musical meter. 60bpm is a frequency of 1 cycle per second (1Hz).
- The lowest frequencies that we perceive as tones begin at around 20 cycles per second (20Hz). That is, every 0.05 seconds (50 milliseconds). The region between rhythm and tone, around 8-20Hz is very interesting: this is where most vibrato modulation is found.
- It also happens to be the region of alpha/beta neural oscillation (brainwaves). 7Hz is approximately 150 ms; 30Hz is approximately 33 ms.
- The persistence of vision effect which allows image sequences to appear as continuous motion begins at around 10-20 frames per second (Hz), depending on content. Cinema traditionally used 24 or 30 frames per second. Rapid movements in modern gaming may demand frame rates of up to 60Hz or more. Roughly speaking, the human visual system can process 10-12 images per second and perceive them individually, while higher rates are perceived as motion. Modulated light (such as a computer display) is perceived as stable at around 50-60fps.
- Much higher rates (and very low latencies) may be needed when integrated into a sensormotor loop: videogame oriented displays may run at 60-120fps, VR displays often use 90 or 120 fps, and musical instrument interfaces may need even higher clock speeds.
- Human singing ranges over fundamental frequencies of about 80Hz to about 1100Hz.
- What we perceive as timbre, or sound color, consists mostly of frequencies above 100Hz well into the thousands of Hz (kHz). For example, we distinguish between different vowel sounds according to proportions of frequencies (called formants) in the range of 240 to 2500 Hz. A high hat cymbal is largely made of a complexity of much higher frequencies.
- Human auditory perception begins to trail off above 10000 - 20000 Hz (10 - 20kHz), depending greatly on age. Sounds above these frequencies are audible to many other species. A high frequency of 20kHz implies a cyclic duration of just 0.00005 seconds. High frequencies imply short durations.
- Standard audio CDs are encoded at a sampling rate of 44100Hz, which means a frequency of 44100 samples per second. That implies they can represent frequencies of up to 22050Hz, close to the limit of human perception. Web audio processing today is also usually sampled at 44100Hz.
-->
<p>The fact that the whole gamut of musical phenomena, from an entire composition of movements, to meter and rhythm, to pitch and finally sound color can be described by a single continuum of time has been noted by composers such as Charles Ives, Henry Cowell, Iannis Xenakis and Karlheinz Stockhausen.</p>
<hr>
<h2 id="the-great-computer-music-problem">The Great Computer Music Problem</h2>
<p>Many of these early electronic and computer music composers were attracted to the medium for the apparently unlimited range of possible timbres it could produce. Rather than being limited to the sounds we can coerce physical objects to emanate, we have the entire spectrum available, every possible variation of placing sample values in a sequence, without limitation. In addition to the potential to find <a href="http://www.youtube.com/watch?v=pQLlR4pvhyg">the new sound</a>, the computer was also attractive to composers as a precise, accurate and moreover obedient performer. Never before could a composer realize such control over every tiny detail of a composition: the computer will perfectly reproduce the commands issued to it. (Of course, both of these trends mirror the industrial era from which they were borne.)</p>
<p>Unfortunately, although the space of possible sounds in just one second of CD-quality audio is practically infinite, the vast majority of them are uninteresting, and quite likely unpleasant.</p>
<blockquote>
<p>This is perhaps emblematic of a more general problem: with so much data, and so much ability to generate & transform it, how to make <em>sense</em> out of it? How to retain that promise of infinity without becoming lost in noise, or resting too much upon what is already known?</p>
</blockquote>
<p>For example, selecting samples randomly results in psychologically indistinguishable white noise. But specifying each sample manually would be beyond tedious. So, with all the infinite possibilities of sounds available, the question is <strong>how should one navigate and determine the space of all possible sample sequences to find things that are interesting</strong>? How should one <strong>organize sound</strong>?</p>
<blockquote>
<p>This is could be a great prompt for making: <strong>to design a function to map from discrete time to an interesting sequence of data as sound</strong>. Without borrowing or leaning too much on things you already know, how would you approach this problem; how would you break it down? For example, what are the salient general features that we perceive in sound, how can they be generated algorithmically, with what kinds of meaningful control?</p>
</blockquote>
<blockquote>
<p>We must also be careful to not let the regular space of digital sound (or indeed pixel space in images) mislead us. The ear is not equally sensitive across frequencies and amplitudes, and there are perceptual effects such as fusion and masking, phantom fundamentals etc. to take into account.</p>
</blockquote>
<p><img src="img/fletcher-munson.png" alt=""> </p>
<ul>
<li><a href="https://en.wikipedia.org/wiki/Hearing_range">Variance between species</a></li>
</ul>
<hr>
<h2 id="music-iii">Music-III</h2>
<p>One of the most influential solutions to this problem was developed by Max Mathews at Bell Labs in the late 1950's, in his <strong>Music-N</strong> series of computer music languages. The influence of his work lives on today both in name (Max in <a href="http://en.wikipedia.org/wiki/Max_(software)">Max/MSP</a> refers to Max Mathews) and in design (the <a href="http://en.wikipedia.org/wiki/Csound">CSound</a> language in particular directly inherits the Music-N design). </p>
<ul>
<li><a href="http://ia601505.us.archive.org/7/items/bstj40-3-677/bstj40-3-677.pdf">Mathews, M. An Acoustic Compiler for Music and Psychological Stimuli</a></li>
</ul>
<p>Mathews' approach exemplifies the "divide and conquer" principle, with inspiration from the Western music tradition, but also a deep concern (that continued throughout his life) with the psychology of listening: what sounds we respond to, and why. In his schema, a computer music composition is composed of:</p>
<ul>
<li><strong>Orchestra</strong>:<ul>
<li>Several <strong>instrument</strong> definitions. <ul>
<li>Defined in terms of several <strong>unit generator</strong> types (see below),<ul>
<li>including configuration parameters,</li>
<li>and the connections between the unit generators.</li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
<li><strong>Score</strong>:<ul>
<li>Some global configuration</li>
<li>Definition of shared, static (timeless) features</li>
<li>Many <strong>note</strong> event data, referring to the orchestra instruments,<ul>
<li>including configuration parameters.</li>
</ul>
</li>
</ul>
</li>
</ul>
<p>Matthews was aware that we perceive time in combinations both discrete events and continuous streams; abstracted into his "note" and "instrument" components respectively. The "score" is able to reflect the fact that we perceive multiple streams simultaneously. </p>
<blockquote>
<p>Is anything missing from this model? If so, how would you represent it?</p>
</blockquote>
<h4 id="unit-generators">Unit Generators</h4>
<p>He noted that streams are mostly formed from largely periodic signals, whose most important properties are period (frequency), duration, amplitude (loudness), and wave shape (timbre), with certain exceptions (such as noise). Furthermore he noted that we are sensitive to modulations of these parameters: vibrato (periodic variation in frequency), tremelo (periodic variation in amplitude), attack and decay characteristics (overall <em>envelope</em> of amplitude). To provide these basic characteristics, without overly constraining the artist, he conceived the highly influential <strong>unit generator</strong> concept. Each unit generator type produces a stream of samples according to a simple underlying algorithm and input parameters. </p>
<p><img src="img/ugen.png" alt="Unit generator"></p>
<p>For example, a pure tone can be produced by taking the <code>sin</code> of a steadily increasing sequence of values (a <code>clock</code> or accumulator), producing a <strong>sine wave</strong>; the parameters may include a multiplier of the clock (for frequency) and a multiplier of the output (for amplitude). </p>
<p>Unit generators (or simply "ugens") can be combined together, by mapping the output of one ugen to an input parameter of another, to create more complex sound generators (e.g. creating tremelo by mapping one sine wave output to the amplitude parameter of another), and ultimately, define the complete instruments of an orchestra. The combination of unit generators is called a <em>directed acyclic <strong>graph</strong></em>, since data flows in one direction and feedback loops are not permitted. </p>
<p>Music-III was a <strong>compiler</strong>, that is, a program that receives code in some form, and produces another program as output. Given <strong>orchestra</strong> specification data, the Music-III compiler produces programs for each of the <strong>instruments</strong>. </p>
<p><img src="img/ugen2.png" alt="Unit generator graph"></p>
<p>Some example unit generators include:</p>
<ul>
<li><strong>Adder</strong>. The output of this ugen is simply the addition of the inputs. In Max/MSP this is [+~]. </li>
<li><strong>Random generator</strong>. Returns a stream of random numbers. In Max/MSP this is called [noise~].</li>
<li><strong>Triangle ramp</strong>. A simple rising signal that wraps in the range [0, 1). In Max/MSP this is called [phasor]. <ul>
<li>In Music-II this is not a bandlimited triangle, and may sound harsh or incorrect at high frequencies. </li>
</ul>
</li>
<li><strong>Wavetable playback</strong>. For the sake of simplicity and efficiency, Music-III also featured the ability to generate <strong>wavetables</strong>, basic wave functions stored in memory that unit generators can refer to. The most commonly used unit generator simply reads from the wavetable according to the input frequency (or rate), looping at the boundaries to create an arbitary periodic waveform. In Max/MSP this could be [cycle<del>] or [groove</del>]. <ul>
<li>In Music-III this does not appear to use interpolation, and thus may suffer aliasing artifacts. </li>
</ul>
</li>
<li><strong>Acoustic output</strong>. Any input to this ugen is mixed to the global acoustic output. This represents the <em>root</em> of the ugen graph. In Max/MSP this is called [dac~]. </li>
</ul>
<p><img src="img/maxmsp.png" alt="MaxMSP"><br><em>Here's what the above instrument looks like (more or less) in Max/MSP.</em></p>
<p>Modern computer music software such as Max, PD, CSound and SuperCollider have hundreds of different unit generators. Although the syntax is very different (and we no longer need to punch holes in card!), the same principles are clearly visible in modern computer music languages, such as the SynthDef construct in <a href="http://en.wikipedia.org/wiki/SuperCollider">SuperCollider</a>:</p>
<pre><code>(
SynthDef(\example, {
Out.ar(0,
SinOsc.ar(rrand(400, 800), 0, 0.2) * Line.kr(1, 0, 1, doneAction: 2)
)
}).play;
)</code></pre>
<h4 id="sequencer">Sequencer</h4>
<p>A separate program, called the <strong>sequencer</strong>, receives score data, and for each note, the sequencer "plays" the corresponding instrument program, with the note parameters to specify frequency, volume, duration etc. The outputs of each note program are mixed into an output stream, which is finally converted into sound. Score events included:</p>
<ul>
<li>Specifying the generation of <em>wavetables</em>, or other static data.</li>
<li>Specifying the time scale / tempo.</li>
<li>Specify note events (with parameters including instrument, duration, and other instrument-specific controls). </li>
<li>Punctuation: <ul>
<li>rests, </li>
<li>synchronization points, </li>
<li>end of the work.</li>
</ul>
</li>
</ul>
<h4 id="acoustic-compiler">Acoustic compiler</h4>
<p>In summary, Max Matthews' <em>Music-III</em> system comprises/requires:</p>
<ul>
<li>A set of pre-defined "unit generators"<ul>
<li>A data-format to define instruments as unit generator graphs</li>
<li>A tool to compile instrument definitions into programs (or functions)</li>
</ul>
</li>
<li>A data format to represent events in a score<ul>
<li>A tool to sequence these note events over time and thus invoke instruments and send the results to audio output</li>
</ul>
</li>
</ul>
<p>The benefits of his approach included both flexibility (a vast diversity of instruments, orchestrae, scores can be produced), efficiency (at $100 per minute!), and suitability to producing interesting sounds. </p>
<p>With modern languages we can overcome many of the limitations in the 1950's programming environment, allowing us to create a much richer and friendlier acoustic compiler and sequencer...</p>
<blockquote>
<p>Is this the only way to structure a composition? Is it the only/best solution to the great computer music problem? What cannot be expressed in this form? What alternatives could be considered?</p>
</blockquote>
<p>How would you implement this today?</p>
<!--
---
# Demystifcation
## Ugens
Let's take a look at how this works today in Max:
- Electronic realm:
- oscillator as a function of time, via an accumulator
- we can start with exploring some functions with [Desmos](https://www.desmos.com/calculator)
- vibrato/tremelo
- envelope, triggers, S&H
- AM, FM, FFM, etc.
- Electroacoustic realm:
- Waveforms playback, reverse,
- looping, wavetables (note circular buffer)
- chopping breaks
- Sampling
- Feedback & delay: a whole spectrum of effects from this
- Spectral effects
- Granular
- OLA via 'tape loop' analogy
What about `sco`-space?
A moment of reflection: remember 'machines & tapes'; that tapes are data are code. To what extent can a signal be a 'tape of instructions'? E.g. a signal stream of triggers to activate an oscillator. Or something more complex, e.g. MIDI data streams; (show MIDI *interpreter* example). Or something more complex (calculator example). So... what kind of interpreter would you write to articulate structure?
- Quick demo: Node for Max player
- Limitations
## A new engine
Although low-dimensional, audio programming can be surprisingly demanding. New data must be generated at reliable, steady rates of forty thousand samples per second or better, and latencies must remain low for musical sensitivity. It occupies a fascinating boundary between the apparently continuous (analog) but actually discrete (digital), and brings software architecture and user-experience issues to the fore in the mapping of perceptual/musical concepts to actual implementations.
Let's try to dig down to low levels of what's really going on here, and see if we can build it back up again. Let's say, we wanted to build a general Music-III style program in Node.js, where we can throw a whole range of Javascript flexibility at it.
First, we can look at what's happening underneath gen~, by sending an `exportcode` message to it, revealing the C++ that it is generating & compiling. The overall structure is of initialization and callback.
Second, we'll look at writing a Node.js [module](nodejs.html#modules) to give us the ability to generate audio in realtime. This still depends on using C++ to access the hardware, so we'll write a native module.
- [Node.js Native C modules quick-start](https://github.com/worldmaking/worldmaking.github.io/wiki/Node.js-native-C-modules)
- [Native modules API](https://nodejs.org/docs/latest/api/n-api.html)
- [Header-only libraries](https://github.com/nothings/single_file_libs) -- and why this is preferable if possible
- E.g. [Miniaudio](https://handmade.network/forums/wip/t/2702-miniaudio__a_c_c++_cross-platform,_single_file,_public_domain_audio_library)
- [Verify with example of sine playback](https://github.com/mackron/miniaudio/blob/master/examples/simple_playback_sine.c)
- Now to generalize this so we can write audio in JS:
- Use a [Float32array](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Float32Array) so that we can share raw writable memory from C++ to JS
- Restructure using a [ringbuffer](https://en.wikipedia.org/wiki/Circular_buffer) to allow continuous updates with a window of flexibility.
<img src="https://upload.wikimedia.org/wikipedia/commons/f/fd/Circular_Buffer_Animation.gif" />
At this point we have abstracted the hardware down to a function to generate the "next sample", possibly as a function of time. Now we don't need to worry about C++ anymore, and it's up to you: how would you articulate a response to the 'computer music problem'?
> E.g. for parallel scheduling ideas, consider temporal recursion and priority queues?
---
---
# Spectral domain
- Spectrogram presents energy distribution at different frequencies
- Can be frequencies of time (e.g. audio), of space (e.g. image), or any other extensive dimension
- Look at spectra of pure, harmonic and complex tones, noise colours, impulse
- Spectrogram = series of spectra over time
> Why is the spectrum of a pure impulse similar to that of white noise?
***Fourier transform***
- Any waveform can be reproduced as a sum of sines (of specific amplitudes, phases, [frequencies])
- Frequency series of "bins" as integer multiples of a fundamental from lowest to highest
- Epicycles
- https://www.youtube.com/watch?v=LznjC4Lo7lE
- https://bl.ocks.org/jinroh/7524988
- https://en.wikipedia.org/wiki/Fourier_series
- FFT (Fast (Discrete) Fourier Transform)
- A clever algorithm to more efficiently compute DFT
- Used everywhere
> Not only sines; wavelet decompositions for example can reconstruct from a wide variety of basis waveforms.
Limitations:
- Windowing (Heisenberg)
- Frequency resolution and exponential distribution
**Heisenberg Uncertainty Principle:** the more accurately you can determine position, the less you can know the conjugate variable of momentum. Similarly for temporal precision vs. frequency precision: a greater frequency resolution (more bins) requires a longer period to produce it, which smears time. The uncertainty principle is inherent in the properties of all wave-like systems, which means, pretty much everything. (Note: not the same as the "observer effect".)
-->
<p><a href="http://www.thisiscolossal.com/2012/05/delightful-paper-pop-ups-by-jenny-chen/"><img src="http://25.media.tumblr.com/b5bbc21c3907802325301007ce31303f/tumblr_mjksjiMKYr1qamt2wo1_500.gif" alt="animation"></a></p>
<footer>DIGM5010 2021-22</footer>
</body>
<script src="js/connect.js"></script>
</html>