Make sure HD is on, and if possible full-screen. It makes a difference.
Here it is on vimeo too, where you can leave comments and likes if you’d like.
Mauricio Pauly asked me to make the videos for two of his works. I thank him for his trust and belief in me, because as we excitedly talked the project over at one of his favorite in-town coffee shops, he didn’t at all hesitate when I requested I be able to approach this however I wanted.
This first piece, Its Fleece Electrostatic, is for violin, and it’s Linda Jankowska’s performance(s) edited together in this video, which I shot over a few hours on a Wednesday in early July of 2014. What follows is how I manipulated that footage over the next several months.
[In this section, I’ve tried to detail some of the technical specifics regarding the making of this video. They are the methods I cobbled together from hours of research on the internet, as well as some tid-bits from some of my excellent programmer friends. (As far as I know there isn’t a more cohesive guide. If there is PLEASE share it with me!) So now that I’ve done the leg work, I want to pass it all along so it’s easier for others to explore video making in this way. Hopefully what follows can help you in some way. I’m just showing you to the digital finger paint–now go create!]
This is the first video where I really dig into datamoshing.
I had been drawn to it for some time, but felt like it was this very mysterious thing just out of reach, just outside the realm of my understanding and technical ability. Some years prior I had found a tutorial on youtube (How To Datamosh by datamosher) and while the process shown is effective and educational, I also found it prohibitively tedious. Oh my god(!) how I could not handle having to set in and out points to manually delete every “I” frame in a length of video using the un-sexiest programs, and then having to convert files back and forth just to see what I had made. There are I frames and there are P frames. I frames are a keyframe or an image whose pixels are referrenced. P frames are frames where the data just refers to the motion of the pixels in the I frames. Datamosher’s youtube videos do a good job of showing you this stuff. So I dropped it and went on to something else. That something else was working on these specific kinds of process drawings. Here’s an example (there are other posts about them).
Above: a strong sense of the pattern (all the colors of sex)
owned by Linda Jankowska
Over the time that I was making these drawings I also saw two videos which inspired me greatly. They are: Welcome To Heartbreak by Kanye West (for it’s use of datamoshing) and Body & Blood by Clipping (for it’s use of greenscreen). Damn it! If I had to do some tedious shit to datamosh, so be it. Welcome To Heartbreak is just too cool. So I braced myself for a massive undertaking.
The time spent making these drawings acclimated me to a certain attitude about process, that I think can be misinterpretted, but best understood, as patience.
So I revisited the idea of datamoshing and after having done some research I came across another way to datamosh using a program called GoldmoshLive. It is basically a Max patch that you feed a length of video into and it fucks it up in some way. The program gives you some options, but they don’t work as described, and what they do do really isn’t that great anyway. Datamoshing means things aren’t working right anyway though, so this is really no matter. Something goes in…something, something…something fucked up comes out. Ding! However, the things you feed into it greatly impact what comes out and the correlations are largely unpredictable. All I’ll say is every adjustment of parameter altered the outcome greatly: opacity, color choices, layers, re-timing of clips, treatment of each layer, keying, filters, clip length etc. But this was a good thing! It encouraged experimentation. And through this experimentation I learned how to ride the digital dragon!
So from this point the central mechanisms for building fleece were the processes of experimentation and selection.
Early into the process however, I discovered Final Cut Pro X couldn’t handle playing a timeline with too many of these Goldmoshed (ie corrupted) clips. One night, while I was knee deep (and I mean baaaalllls deep) and had really found a good groove with it all, I imported the clip that broke the camel’s back and crashed Final Cut. When I was finally able to reopen Final Cut–cuz it had crashed my computer hard–it decided it wasn’t going to recognize the project file at all anymore. EVERYTHING up to that point, GONE. All my greenscreen footage of Linda, seemingly gone (It is not a good feeling thinking you wasted another person’s time and effort). Everything I had constructed across several timelines, gone. This was a good month of work up to this point. Final Cut took its ball and went home. My technical advisor, Rodrigo Constanzo, was away at the time (crushing it at Darmstadt), so that night we performed triage over a long distance phone call.
In the end, I lost everything I had built, but salvaged the initial footage I had shot of Linda in front of the green screen. I didn’t return to the project for a few months because I had to start again…well, not quite from zero…I had learned some things through all the experimentation I had done.
When I started up again, I worked only seconds of footage at a time (really to minimize the risk of another bad crash), in what I perceived to be the moments that make up the whole of the piece. On average building 10 seconds of footage took me 3-4 days. Nothing in the video is automated…or automatically mapped. The decisions are mine. What I mean is, I did not give some program the specification that every time you hear a F# for instance, that the screen goes blue (that never happens anyway, but you get the idea).
While I tried to give each moment in the piece unique consideration, I’ll try to give a very general over-view. These considerations are not sequential and would be repeated throughout
+ Figure out which clips to include and which work well together as layers. Linda and I filmed for the good part of a day so I had a fair share of takes, both of individual sections and some full run-throughs.
+ Think about MOTION and COLOR
+ Consider re-timing sections (slow-motion, speed ramping, f-fwd). Was it appropriate or not? What kind of things to do?
+ Make decisions about color. Really, broad strokes stuff. I’d add layers that would normally not be seen, but datamoshing fucks things up in unexpected ways so I’d add opaque layers on top of opaque layers and experiment with the treatment of each layer just to see what would happen.I can think of this stuff in terms of painting, and I’m just blocking things in. A big brush. A wash.
+ Datamosh the shit out of that stuff. In my case using Ruby scripts or Goldmosh. Changing things so different things happen (as previously mentioned) and then choosing the things I’d feed back into the timeline to then re-export and continue this experimentation with, until the section was what I needed. Each time something got moshed and reimported it felt like I was flattening things down. I don’t mean as in making them less complex. I mean as in cooking them. Rodrigo mentioned that a good parallel might be how Aphex Twin constructs his music. I fucking love Aphex Twin. Also it’s important to note that the first time you import a freshly Goldmoshed clip back into Final Cut it is suuuuuuper corrupted. Sometimes this looked quite cool–but is unpredictable, as I think Final Cut plays/processes that section of video anew each time it drags the playhead across that clip in the timeline. Sometimes that’s [very] good. Sometimes I wanted to lock something down and so I had to import the freshly Goldmoshed clip, export it from Final Cut, then re-import the clip, deleting the first iteration of it. This made that clip more stable. Here are a few screen shots of my timelines (even with sections already flattened–cooked–you can see the beautiful layer-braid-sausage-stacks):
As I continued making the video for fleece, I continued researching. I came across a way to do the initial moshing I had found so tedious years earlier, using a Ruby script. Up until now, I hadn’t gotten around to that technique because Goldmosh was varied enough to keep things interesting.
But then I found AviGlitch – “A Ruby library to destroy your AVI files”.
Perfect! What took me minutes to do before by deleting I frames manually (and god forbid I accidentally delete the first I frame, which would crash Avidemux and I’d have to start all over!), I could do with this script in seconds. I still had to convert my file into an avi file using ffmpegX, and convert the avi file back to an mp4 using Handbrake, but that tedious process that datamosher teaches us about in his youtube video is now automated. The results of doing it manually and tediously are EXACTLY the same as running this script. There are no creative decisions to be made. This is the kind of automation that grants a person freedom. It is what is good about technology. Here is the code for that kind of I frame deletion moshing, which I simply called “mosher”:
#!/usr/bin/ruby require 'aviglitch' avi = AviGlitch.open '/Users/Rodrigo/Desktop/tomosh.avi' avi.glitch(:keyframe) do |data| data.gsub(/\d/, '0') end avi.output '/Users/Rodrigo/Desktop/moshed.avi'
Ok, so here’s how to use Aviglitch: Install Aviglitch: http://www.rubydoc.info/gems/aviglitch/frames Export a clip from Final Cut. The higher the resolution the better, but make sure you don’t export it at a resolution greater than what it was originally filmed because ffmpegX doesn’t like that and it will refuse to convert the clip (or something like that. Rather, if ffmpeg isn’t converting try exporting at different resolutions until it sticks. This is the problem 9 out of 10 times). Convert it to an avi file using ffmpegX. To do that make sure you have these settings:
+ video codec: “XviD[.AVI] (ffmpeg)”
+ audio: un-tick “encode audio”
+ select options: “High Quality”, “Two-pass encoding”, and “Decode with QT” (but UN-tick “Letterbox”) Then change the name of that avi file to “tomosh”.
Then open terminal on your computer and copy this in, making sure your file path is correct…This was mine for example:
cd /Users/Rodrigo/Dropbox/Angie/5\ DATAMOSH/RUBY
In terminal, type the name of the Ruby script you want to run. There’s the one I already typed up above, “./mosher.rb”. Then there are these bloom-ing scripts I used. You can play around and modify the code to change what it does. So this one of my blooming scripts “./bloomer.rb”. This one basically jumps around and blooms a frame referencing just another random point in the same clip. It does this to several frames so it’ll give you a variety of stuff you can then select the good stuff from. So you run, discern, select, reimport. This was my process and I loved how exciting it was because of the different things it’d spit out. You can spend sooooo much time here and it is time well spent:
#!/usr/bin/ruby require 'aviglitch' a = AviGlitch.open '/Users/Rodrigo/Desktop/tomosh.avi' d =  a.frames.each_with_index do |f, i| d.push(i) if f.is_deltaframe? # Collecting non-keyframes indices. end q = a.frames[0, 5] # Keep first key frame. 15.times do # AMOUNT OF TIMES TO RUN x = a.frames[d[rand(d.size)], 1] # Select a certain non-keyframe. q.concat(x * rand(120)) # HOW MANY FRAMES TO BLOOM end o = AviGlitch.open q # New AviGlitch instance using the frames. o.output '/Users/Rodrigo/Desktop/bloomed.avi'
And this is another “.bloomerx1.rb”. This one just blooms the 5th frame of a clip, with the motion of the frames that follow it. Note: The clip used has to be at least 13 frames long. The eight frames that follow the 5th can be from a different sequence, not necessarily the continuation of that same clip. So it’s cool to look at sequences/frames and consider them in terms of the motion they’ll impose on an image/pixels of the 5th frame:
#!/usr/bin/ruby require 'aviglitch' a = AviGlitch.open '/Users/Rodrigo/Desktop/tomosh.avi' d =  a.frames.each_with_index do |f, i| d.push(i) if f.is_deltaframe? end q = a.frames[0, 5] 1.times do x = a.frames[d, 1] q.concat(x * (200)) # HOW MANY FRAMES TO BLOOM end o = AviGlitch.open q # New AviGlitch instance using the frames. o.output '/Users/Rodrigo/Desktop/bloomed.avi'
After you run these things a bunch and they make some building blocks you think you can work with, you convert these avi files back to mp4 files using Handbrake. I found Handbrake sometimes fucked up the aspect ratio during conversion. This is how you fix that….(again, knowledge gained through experimentation and discovery)
+ In toggle presents make sure you have ‘High Profile’ selected
+ Then go to “Picture Settings”
+ Set anamorphic to “None”
+ Un-select “Keep Aspect Ratio”
+ Select Cropping to “Custom”
+ Change all those parameters to zero (one of these will be some other number–this is the problem!)
+ Then manually enter the width and height to match the dimensions of the original video.
WHAT FOLLOWS IS LESS TECHNICAL
Of course I made aesthetic decisions, large and small, all throughout. The thinking that took place on this video as I made it was thick muck. Each section, in addition to being personally deeply significant, also navigates the following relationships in specific ways depending on the section that’s being played:
+ the seen and the heard
+ artificial (recorded/looper) space vs real (air/performer) space
+ co-operative vs disruptive, or even destructive, sounds
+ the silent and the not
+ melody and texture
+ past and future
There is much I can say on each of these points, but I’ll just simply say that I’d invent new puzzles to solve, and try to solve those puzzles the most beautiful ways I could.
However, engaging in the process of making this video was different from engaging in rational, problem-solving thought. Logos. Bios. Technos. Novelty articulated via the modalities thereof. Yes, I believe in serendipity, as well as something like animism which extends itself to media. Mind in media. Life as media.
1 The Past
0.9 The stages of decay
0. 5 A small catastrophe
0.1 Shadow boxing
0 The Present
life is long and beautiful