Creative Coding with ChatGPT and GitHub Copilot

Midjourney image using the prompt: "a 1950s era robot holding a paint brush, standing in front of an unfinished abstract painting that shows a triangle, circle and square dancing, with a faded cityscape of Bauhaus buildings in the background, style of a vector illustration --ar 2:1".

TLDR

• With the recent rapid advancements in transformer-based language models, we have entered a new era of AI assisted coding, with GitHub Copilot and ChatGPT being two prominent tools in a rapidly growing space.
• In this post I share two simple anecdotes of how I used each tool for coding assistance on a recent creative coding project, which demonstrate how these tools can enter into the development workflow, and that sometimes they require a bit of iterative trial and error problem solving.
• I have the sense that anyone in tech who is not incorporating these tools into their workflow is at risk of being severely outpaced by those who are able to adapt.

Large language models for writing code

In the past few years several tools have been developed to utilize the capabilities of large language models (LLMs) to assist with writing code. Most of these are built on the GPT (Generative Pretrained Transformer) models developed at OpenAI. As early as mid-2020 when GPT-3 first became available, people began probing its ability to generate useful code. Two prevalent tools that have emerged are GitHub Copilot, launched in mid-2021, and the general purpose ChatGPT, launched in late 2022.

GitHub Copilot is powered by OpenAI’s Codex model, which is built on the GPT-3 foundational model, with additional training on a large corpus of source code for further fine tuning. It is capable of generating code to complete a text prompt, as well as translating code into a natural language description, and translating code between programming languages. Copilot is available to use via IDE plugins (I’m using the Visual Studio Code plugin), and requires a subscription of roughly $10 per month. It can offer autocompletions as you type, which can include generating the code for entire functions or classes. You can also write a comment directed at it, and it will then help write the code to fullfil the comment (it can also write comments itself). In the example I give below, I ended up writing about 10% of the code, Copilot filled in the rest.

ChatGPT is also descended from GPT-3, but with additional fine tuning, including the use of reinforcement learning from human feedback, and has the notable ability to retain context between turns in a conversation, which can allow the user to steer it in an iterative fashion toward a solution. ChatGPT can do everything Copilot can do (i.e. generate code, describe code in natural language etc), but in addition it has this conversational aspect. I think because of that, both tools can play distinct roles in the development cycle of a project, perhaps relying more heavily on Copilot for the bulk of day-to-day coding, and turning to ChatGPT to get assistance working through more complex problems.

I've only started using ChatGPT and Copilot for coding assistance very recently, so I feel a little late to the party, but based on my experience so far, I'm sold on the idea that these tools are a major force multiplier for both creative and productive output.

Below I share two examples of using each tool for coding assistance on a creative coding side project. Neither example is particularly astonishing, but I think they nicely illustrate how these tools can enter into the development workflow, and that sometimes they require a bit of iterative trial and error problem solving.

Iterating toward a solution with ChatGPT

Lately I've been rekindling my interest in algorithmic and generative art and have been exploring ideas around incorporating the comics medium into abstract art, i.e. conveying a sense of sequential narrative flow with shapes, texture, and color across panels and pages. I'm using p5.js, which is a JavaScript library based on the Processing framework that was designed for creative coding projects. You can see some examples of experiments I've done on Instagram.

One of my visual experiments required generating regular polygon shapes with n vertices. I started sketching out a solution, but then I got the idea: this would be an excellent test to see how well ChatGPT could generate useful code for a specific and limited problem I needed to solve. I ended up proceeding through a sequence of prompts, iteratively arriving at the result I had in mind.

It's interesting to walk through the chat, but for brevity I'll summarize ChatGPT's responses. For the entire transcript, scroll to the bottom of this post. Fortunately this interaction occurred the day after OpenAI introduced the prompt history feature, which made recalling the chat much easier. My exact prompts are shown in bold. This conversation took place around January 6, 2023.

PROMPT: How do I draw a circle using p5.js? To start things off, I was curious if ChatGPT knew about the p5.js library.

ChatGPT:[This is a summary. See full transcript below.] ChatGPT informed me of the ellipse and circle functions, and showed working examples of how to use them to render images to the web canvas. This was a green light that I might have some success if I continue with my queries.

PROMPT: Could you write p5.js code to draw a regular polygon with input parameters being the number of sides, the position and size? Looking back, it's interesting that I didn't offer up an example function signature, like drawRegularPolygon(position, size, vertexCount) or such. Now that I've clocked some experience using ChatGPT and Copilot, I can imagine Copilot autocompleting given just the function name alone.

ChatGPT: It wrote a function drawRegularPolygon(n, position, size) that utilizes the beginShape / endShape block in p5.js, and computing(x,y) coordinates of n vertices by looping through the rotation angle 2*pi/n, and demonstrating calling the function to draw a triangle. This is a fine solution, but I needed a solution that was compatiable with the scribble.js library I was using to render with a hand-drawn style.

PROMPT: Could you repeat that, but use the scribble.js library and the scribbleLine method? Rather than use the beginShape block, I needed a solution that constructed the shape by calling scribbleLine for each polygon side. I was skeptical that ChatGPT would have enough familiarity with the scribble.js library to produce a valid solution, but I figure I'd try.

ChatGPT: Sure enough, the solution it proposed did not work. This is an example where ChatGPT doesn't know enough to pull together a valid response but still gives an answer with confidence (aka bullshitting). This behavior will improve over time, for example, it could respond with: "I am not familiar with that library. Could you point me to the documentation or source code for it?". Or at the very least be able to qualify a resonse with a likelihood.

PROMPT: That actually wasn't what I wanted, I don't think it will work. Let's back up a step. Could you write the code using the built-in p5.js line method? If I could get ChatGPT to use the p5.js line function, then I figured it would be easy to translate it to the scribble.js API.

ChatGPT: Here, it gives a working solution, drawing the shape using the line function. Although the very first solution it came up with above using the beginShape block was more elegant, this alternate solution is one I could then easily translate into the scribble.js API (by calling the scribbleLine function).

PROMPT: That works well, but suppose I want one side of the polygon to be parallel with the horizon, could I add an angle offset to achieve that? While testing the above solution, I found that for some values of n, I didn't like the orientation of the polygon: I wanted the bottom side to always be parallel to the x-axis. To me, this is starting to get a bit nuanced, and I was skeptical that ChatGPT would be able to provide a valid solution (I assumed I'd get more bullshitting).

ChatGPT: The solution it gave for the angle offset worked for some cases, but was not general for arbitrary n.

PROMPT: What is the formula for the angleOffset so that I can generalize the code to any polygon of n sides?

ChatGPT: It provided an angle offset that worked for odd values of n.

PROMPT: Ok, I tried that result and it only works when n is odd. What should the angleOffset be when n is even?

ChatGPT: It then gave the solution for even values of n, which finally satisified the requirements I had in mind.

This was my first encounter using ChatGPT for coding assitance, and I became an instant convert. I think this exchange nicely illustrates that one can arrive at a useful solution, but it may require several iterations of trial and error while providing feedback and reframing the task to match the constraints for your particular problem.

Here is the final output posted to Instagram that utilizes the polygon code that ChatGPT assisted in writing.

View this post on Instagram

A post shared by Jamie Williams (@jamieinfinity)

GitHub Copilot can read your mind (sometimes)

I'll also share my earliest experience using GitHub Copilot, which I started using only very recently. I installed the VS Code plugin and began working on a different p5.js experiment, playing around with shape primitives like an ellipse or rectangle and projecting them onto a grid to build composite shapes. I decided to define shape classes that could encapsulate the methods I needed, like a containsPoint method (i.e. does a point lie within the shape) and a draw method.

ShapeEllipse

I started typing class ShapeEllipse and Copilot offered to autocomplete the constructor, but the signature wasn't exactly what I had in mind, so I typed my own signature, and then Copilot filled in the property boilerplate.

class ShapeEllipse {
    constructor(centerPosition, width, height, rotationAngle=0) {
        this.centerPosition = centerPosition;
        this.width = width;
        this.height = height;
        this.rotationAngle = rotationAngle;
    }
  ...
}

I then moved to the containsPoint method, for which I had already tracked down a solution on StackOverflow. However, in hindsight, I'm confident that Copilot would have been able to autocomplete with a working solution as well. Note that while typing this in, Copilot successfully autocompleted the yterm based on the xterm pattern I had already typed.

containsPoint(p) {
        let xterm = ((Math.cos(this.rotationAngle)*(p.x - this.centerPosition.x) + Math.sin(this.rotationAngle)*(p.y - this.centerPosition.y))**2) / (this.width/2)**2;
        let yterm = ((Math.sin(this.rotationAngle)*(p.x - this.centerPosition.x) - Math.cos(this.rotationAngle)*(p.y - this.centerPosition.y))**2) / (this.height/2)**2;
        return (xterm + yterm) <= 1;
    }

Copilot then offered to write a draw method before I had a chance to type it. By default, Copilot was drawing to the base canvas, but I needed to pass in a graphics object, so I corrected the signature accordingly, and then Copilot adapted the solution to match. That is, it filled in everything in the method body:

draw(graphics) {
        graphics.push();
        graphics.translate(this.centerPosition.x, this.centerPosition.y);
        graphics.rotate(this.rotationAngle);
        graphics.ellipse(0, 0, this.width, this.height);
        graphics.pop();
    }

This completed the ShapeEllipse class (or at least, what I needed at that stage).

ShapeRectangle

I then started typing class ShapeRectangle and Copilot autocompleted the entire class, and nailed it on the first try:

class ShapeRectangle {
    constructor(centerPosition, width, height, rotationAngle=0) {
        this.centerPosition = centerPosition;
        this.width = width;
        this.height = height;
        this.rotationAngle = rotationAngle;
    }

    containsPoint(p) {
        let xterm = Math.abs(Math.cos(this.rotationAngle)*(p.x - this.centerPosition.x) + Math.sin(this.rotationAngle)*(p.y - this.centerPosition.y));
        let yterm = Math.abs(Math.sin(this.rotationAngle)*(p.x - this.centerPosition.x) - Math.cos(this.rotationAngle)*(p.y - this.centerPosition.y));
        return (xterm <= this.width/2) && (yterm <= this.height/2);
    }

    draw(graphics) {
        graphics.push();
        graphics.translate(this.centerPosition.x, this.centerPosition.y);
        graphics.rotate(this.rotationAngle);
        graphics.rectMode(CENTER);
        graphics.rect(0, 0, this.width, this.height);
        graphics.pop();
    }
}

So far, out of roughly forty lines of code, I only had to type four or five lines!

ShapeRegularPolygon & ShapePolygon

I started typing class Shape and Copilot autocompleted to class ShapePolygon, which is interesting. But I actually wanted to first handle regular polygons before moving on to more general polygons. So, I typed class ShapeRegularPolygon, and Copilot autocompleted the constructor, containsPoint, and draw methods.

However, this is where I hit a snag: the containsPoint method didn't work properly (the draw method worked fine). After a few failed attempts with Copilot, I then turned to ChatGPT to construct the containsPoint method, but it essentially generated the same broken code. However, while interacting with ChatGPT, it revealed it was trying to implement the ray casting algorithm to determine if a point lies within a polygon. For some reason, it couldn't properly apply the algorithm in the case of a regular polygon. However, when I shifted to writing the more general polygon class, I found that both ChatGPT and Copilot correctly implemented the containsPoint method using the ray casting algorithm:

class ShapePolygon {
    constructor(vertices) {
        this.vertices = vertices;
    }

    containsPoint(p) {
      let intersections = 0;
      for (let i = 0; i < this.vertices.length; i++) {
          let v1 = this.vertices[i];
          let v2 = this.vertices[(i+1) % this.vertices.length];
          if (v1.y > p.y != v2.y > p.y) {
              let x = (p.y - v1.y) * (v2.x - v1.x) / (v2.y - v1.y) + v1.x;
              if (x > p.x) {
                  intersections++;
              }
          }
      }
      return intersections % 2 == 1;
    }

    draw(graphics) {
        graphics.beginShape();
        for (let i = 0; i < this.vertices.length; i++) {
            graphics.vertex(this.vertices[i].x, this.vertices[i].y);
        }
        graphics.endShape(CLOSE);
    }
}

Once I had this result, I could easily see a path toward a solution for ShapeRegularPolygon: I just needed to provide the vertex coordinates, and then I could reuse the same containsPoint method used for ShapePolygon. So, I added a getVertices method to the ShapeRegularPolygon class, which Copilot filled in for me. With that method in place, Copilot was able to generate a working version of the containsPoint method, making use of the getVertices method. For some reason, neither Copilot or ChatGPT was able to carry out this abstraction on its own when I initially tried writing the ShapeRegularPolygon class, but once I provided this intervention, it succeeded.

class ShapeRegularPolygon {
    constructor(centerPosition, radius, numSides, rotationAngle=0) {
        this.centerPosition = centerPosition;
        this.radius = radius;
        this.numSides = numSides;
        this.rotationAngle = rotationAngle;
    }

    getVertices() {
        let vertices = [];
        for (let i = 0; i < this.numSides; i++) {
            let angle = this.rotationAngle + i * 2 * Math.PI / this.numSides;
            let x = this.centerPosition.x + this.radius * Math.cos(angle);
            let y = this.centerPosition.y + this.radius * Math.sin(angle);
            vertices.push({x: x, y: y});
        }
        return vertices;
    }

    containsPoint(p) {
      let intersections = 0;
      let vertices = this.getVertices();
      for (let i = 0; i < vertices.length; i++) {
          let v1 = vertices[i];
          let v2 = vertices[(i+1) % vertices.length];
          if (v1.y > p.y != v2.y > p.y) {
              let x = (p.y - v1.y) * (v2.x - v1.x) / (v2.y - v1.y) + v1.x;
              if (x > p.x) {
                  intersections++;
              }
          }
      }
      return intersections % 2 == 1;
    }

    draw(graphics) {
        graphics.push();
        graphics.translate(this.centerPosition.x, this.centerPosition.y);
        graphics.rotate(this.rotationAngle);
        graphics.beginShape();
        for (let i = 0; i < this.numSides; i++) {
            let angle = i * 2 * Math.PI / this.numSides;
            let x = this.radius * Math.cos(angle);
            let y = this.radius * Math.sin(angle);
            graphics.vertex(x, y);
        }
        graphics.endShape(CLOSE);
        graphics.pop();
    }
}

Finally, I'll share that an obvious next step was to do a bit of refactoring to pull the common containsPoint method out of ShapePolygon and ShapeRegularPolygon . I did this by typing a comment in the file. I started typing:

// refactor ShapePolygon and ShapeRegularPolygon to use a shared function called containsPoint(p, vertices)

Copilot autocompleted the comment with a second line

// (this will require you to refactor the code in the containsPoint() functions for ShapePolygon and ShapeRegularPolygon)

Without giving the details, Copilot made it a snap to do this refactoring.

Here is the final output posted to Instagram that utilizes these shape classes and other code that Copilot assisted writing.

View this post on Instagram

A post shared by Jamie Williams (@jamieinfinity)

Closing thoughts

With the recent rapid advancements in transformer-based language models, we have entered a new era of AI assisted coding. By now one can find many blog posts sharing examples over a broad range of use cases, including data science and analyitcs applications. I've shared two simple anecdotes that give a flavor for how these tools can enhance and accelerate the development process. I have the sense that anyone in tech who is not incorporating these tools into their workflow to boost their effectiveness and productivity is at risk of being severely outpaced by those who are able to adapt. I just hope I can keep up...

---

Appendix: the full ChatGPT transcript

This chat took place around 1/6/23.

PROMPT: How do I draw a circle using p5.js?

ChatGPT Response:

You can use the ellipse function to draw a circle in p5.js. The ellipse function takes four parameters: the x and y coordinates of the center of the circle, and the width and height of the circle. To draw a circle with a diameter of 50 pixels, for example, you would write:

ellipse(x, y, 50, 50);

Here's an example that draws a circle at the center of the canvas with a diameter of 50 pixels:

function setup() {
  createCanvas(400, 400);
}

function draw() {
  background(220);
  ellipse(width/2, height/2, 50, 50);
}

You can also use the circle function to draw a circle. The circle function takes three parameters: the x and y coordinates of the center of the circle, and the diameter of the circle. Here's the same example as above, but using the circle function:

function setup() {
  createCanvas(400, 400);
}

function draw() {
  background(220);
  circle(width/2, height/2, 50);
}

PROMPT: Could you write p5.js code to draw a regular polygon with input parameters being the number of sides, the position and size?

ChatGPT Response:

Sure! Here's a function that takes three parameters: the number of sides of the regular polygon, the position of the center of the polygon, and the size of the polygon. It uses these parameters to draw a regular polygon using the beginShape, vertex, and endShape functions in p5.js:

function drawRegularPolygon(n, position, size) {
  beginShape();
  for (let i = 0; i < n; i++) {
    let angle = TWO_PI / n * i;
    let x = position.x + size * cos(angle);
    let y = position.y + size * sin(angle);
    vertex(x, y);
  }
  endShape(CLOSE);
}

You can use this function by calling it in the draw function and passing in the desired number of sides, position, and size. For example:

function draw() {
  background(220);
  drawRegularPolygon(3, {x: 100, y: 100}, 50);
}

This will draw an equilateral triangle with its center at (100, 100) and a size of 50 pixels.

PROMPT: Could you repeat that, but use the scribble.js library and the scribbleLine method?

ChatGPT Response:

Sure! Here's a revised version of the drawRegularPolygon function that uses the scribbleLine method from the scribble.js library to draw a regular polygon:

function drawRegularPolygon(n, position, size, scribble) {
  beginShape();
  for (let i = 0; i < n; i++) {
    let angle = TWO_PI / n * i;
    let x = position.x + size * cos(angle);
    let y = position.y + size * sin(angle);
    vertex(x, y);
  }
  endShape(CLOSE);
  scribble.scribbleLine(beginShape(), endShape());
}

You can use this function by calling it in the draw function and passing in the desired number of sides, position, size, and a scribble object. For example:

let scribble;

function setup() {
  createCanvas(400, 400);
  scribble = new Scribble();
}

function draw() {
  background(220);
  drawRegularPolygon(3, {x: 100, y: 100}, 50, scribble);
}

This will draw an equilateral triangle with its center at (100, 100) and a size of 50 pixels, using the scribbleLine method from the scribble.js library to give the lines a hand-drawn appearance.

PROMPT: That actually wasn't what I wanted, I don't think it will work. Let's back up a step. Could you write the code using the built-in p5.js line method?

ChatGPT Response:

Sure! Here's a revised version of the drawRegularPolygon function that uses the line method from p5.js to draw a regular polygon:

function drawRegularPolygon(n, position, size) {
  let angle = TWO_PI / n;
  let prevPoint = {
    x: position.x + size * cos(0),
    y: position.y + size * sin(0)
  };
  for (let i = 1; i < n; i++) {
    let currPoint = {
      x: position.x + size * cos(angle * i),
      y: position.y + size * sin(angle * i)
    };
    line(prevPoint.x, prevPoint.y, currPoint.x, currPoint.y);
    prevPoint = currPoint;
  }
  line(prevPoint.x, prevPoint.y, position.x + size * cos(0), position.y + size * sin(0));
}

You can use this function by calling it in the draw function and passing in the desired number of sides, position, and size. For example:

function draw() {
  background(220);
  drawRegularPolygon(3, {x: 100, y: 100}, 50);
}

This will draw an equilateral triangle with its center at (100, 100) and a size of 50 pixels, using the line method from p5.js to draw the lines.

PROMPT: That works well, but suppose I want one side of the polygon to be parallel with the horizon, could I add an angle offset to achieve that?

ChatGPT Response:

Yes, you can add an angle offset to the drawRegularPolygon function to rotate the polygon around its center point. Here's a revised version of the function that takes an additional parameter called angleOffset:

function drawRegularPolygon(n, position, size, angleOffset) {
  let angle = TWO_PI / n;
  let prevPoint = {
    x: position.x + size * cos(angleOffset),
    y: position.y + size * sin(angleOffset)
  };
  for (let i = 1; i < n; i++) {
    let currPoint = {
      x: position.x + size * cos(angle * i + angleOffset),
      y: position.y + size * sin(angle * i + angleOffset)
    };
    line(prevPoint.x, prevPoint.y, currPoint.x, currPoint.y);
    prevPoint = currPoint;
  }
  line(prevPoint.x, prevPoint.y, position.x + size * cos(angleOffset), position.y + size * sin(angleOffset));
}

You can use this function by calling it in the draw function and passing in the desired number of sides, position, size, and angle offset. For example:

function draw() {
  background(220);
  drawRegularPolygon(3, {x: 100, y: 100}, 50, PI / 2);
}

This will draw an equilateral triangle with its center at (100, 100) and a size of 50 pixels, with one side of the triangle parallel to the horizon. The angleOffset parameter is specified as PI / 2, which rotates the triangle 90 degrees counterclockwise around its center point.

PROMPT: What is the formula for the angleOffset so that I can generalize the code to any polygon of n sides?

ChatGPT Response:

To generalize the code to any regular polygon with n sides, you can use the following formula for the angleOffset:

angleOffset = PI / 2 - TWO_PI / n;

This will rotate the polygon so that one of its sides is parallel to the horizon. Here's the revised drawRegularPolygon function with the angleOffset formula included:

function drawRegularPolygon(n, position, size) {
  let angle = TWO_PI / n;
  let angleOffset = PI / 2 - TWO_PI / n;
  let prevPoint = {
    x: position.x + size * cos(angleOffset),
    y: position.y + size * sin(angleOffset)
  };
  for (let i = 1; i < n; i++) {
    let currPoint = {
      x: position.x + size * cos(angle * i + angleOffset),
      y: position.y + size * sin(angle * i + angleOffset)
    };
    line(prevPoint.x, prevPoint.y, currPoint.x, currPoint.y);
    prevPoint = currPoint;
  }
  line(prevPoint.x, prevPoint.y, position.x + size * cos(angleOffset), position.y + size * sin(angleOffset));
}

You can use this function by calling it in the draw function and passing in the desired number of sides, position, and size. The angle offset will be automatically calculated based on the number of sides. For example:

function draw() {
  background(220);
  drawRegularPolygon(3, {x: 100, y: 100}, 50);
}

This will draw an equilateral triangle with its center at (100, 100) and a size of 50 pixels, with one side of the triangle parallel to the horizon.

PROMPT: Ok, I tried that result and it only works when n is odd. What should the angleOffset be when n is even?

ChatGPT Response:

If n is even, you can use the following formula for the angleOffset:

angleOffset = PI / 2 - TWO_PI / n + PI / n;

This will rotate the polygon so that one of its sides is parallel to the horizon. Here's the revised drawRegularPolygon function with the angleOffset formula included:

function drawRegularPolygon(n, position, size) {
  let angle = TWO_PI / n;
  let angleOffset;
  if (n % 2 == 0) {
    angleOffset = PI / 2 - TWO_PI / n + PI / n;
  } else {
    angleOffset = PI / 2 - TWO_PI / n;
  }
  let prevPoint = {
    x: position.x + size * cos(angleOffset),
    y: position.y + size * sin(angleOffset)
  };
  for (let i = 1; i < n; i++) {
    let currPoint = {
      x: position.x + size * cos(angle * i + angleOffset),
      y: position.y + size * sin(angle * i + angleOffset)
    };
    line(prevPoint.x, prevPoint.y, currPoint.x, currPoint.y);
    prevPoint = currPoint;
  }
  line(prevPoint.x, prevPoint.y, position.x + size * cos(angleOffset), position.y + size * sin(angleOffset));
}

You can use this function by calling it in the draw function and passing in the desired number of sides, position, and size. The angle offset will be automatically calculated based on the number of sides. For example:

function draw() {
  background(220);
  drawRegularPolygon(4, {x: 100, y: 100}, 50);
}

This will draw a square with its center at (100, 100) and a size of 50 pixels, with one side of the square parallel to the horizon.

The Quantified Tube

My TV has its own twitter feed now, capturing my TV watching habits, backed by a Belkin smart switch and IFTTT.

One of my (sadly neglected) side projects, code named Worldline, is to discover ways to automate the tracking of my daily activities and form an unfolding timeline scaffold of my life, both for personal retrospection as well as quantitative analysis. I track steps, heart rate, and sleep with a Fitbit, geolocation with Moves, driving with Automatic, iOS device usage with Moment, laptop activity with RescueTime, and some health metrics as well. The tracking of specific human activities -- like when and what I'm eating, when I'm showering or brushing my teeth, when I'm lying on my couch reading a book, or sitting in front of my TV watching a movie -- are more difficult to automate. A significant component of my project is to address that problem using RF proximity devices to provide implicit clues to discern what I'm up to: with strategically placed beacons throughout a dwelling, I can determine when I'm present in a particular room, or near a specific object. 

Proximity info by itself, while giving finer brush strokes than geolocation tracking, does not provide enough fidelity to discern whether I'm reading a book or watching TV while I'm sitting on my couch. However, by folding in other tracking devices, such as motional data from a Fitbit, the whole concert of data streams becomes greater than the sum of the individual devices, revealing a finer-grained picture of what is going on. I recently acquired some new tools to add to the orchestral lifelogging lineup that make it relatively easy to track when and what I'm watching on TV. 

With an Apple TV at the core of my entertainment system, you might think it would be straightforward to extract out my viewing history, but Apple doesn't provide this functionality. After a bit of research (thanks to @eramirez for the tip), I learned about trakt.tv from a Quantified Self presentation 'Tracking Media Consumption' by Ian Forrester (@cubicgarden). Trakt.tv is a 3rd party app that integrates with the FireCore aTV Flash Apple TV jailbreak mod, which automatically tracks (aka 'scrobbles') your viewing activity and links it into social networks. On the surface, it wasn't clear to me exactly what raw data is made available to the enduser, either through the trakt.tv API or paid VIP service (which unlocks an RSS feed). 

Although trakt.tv seems worth exploring further, I decided to come up with a solution that can definitely satisfy my data needs (who doesn't like a good DIY project now and then?). My solution has three core pieces: (1) a Belkin WeMo Insight switch to detect when my TV is turned on or off, (2) IFTTT integration, and (3) the IMDB iPhone app. 

I purchased the Belkin WeMo Insight switch from Amazon for $50, which was relatively straightforward to set up. When you plug it into an outlet, it broadcasts its own WiFi signal that you can connect your smart phone to. Then using the WeMo app on your phone, you configure the device and redirect it to connect to your home's WiFi network. I have found this process can be a bit temperamental (when your WiFi goes down, for example, you have to repeat the steps over from scratch). After setup, I then plugged my TV into the WeMo. For my purposes, the WeMo is always left on, letting me control my TV using the TV's remote as I normally would. When I switch my TV off, the WeMo transitions from an 'On' state to a 'Stand-by' mode (in contrast, switching the WeMo off puts it into the 'Off' state). 

The WeMo app provides a direct route to integrate with IFTTT. Once you have registered your WeMo with IFTTT (assuming you already have an IFTTT account set up), you can then create recipes to capture the times when the WeMo enters any of the three states of 'On', 'Stand-by' or 'Off', and then react to each of those events. I set up recipes to record those events in a Google spreadsheet.

But as anyone knows who has played around with IFTTT, it's kind of addicting and so I decided to also set up recipes to send out tweets when my TV is turned on or off. For this I created a separate Twitter account: @jamieinfinityTV. Capturing data in a Google spreadsheet is useful, but being connected to Twitter opens up other possibilities, since nearly every site or app has Twitter integration.

This got me thinking: why not also try to capture the content of what I watch? IMDB has a smartphone app that lets you quickly search for movies and TV shows, and then tap a button that puts that show in your Watchlist. Using an IFTTT RSS recipe, you can then react to each of those events, i.e. if a new show is added to your IMDB Watchlist feed, then tweet the title and link for that show. I sometimes watch Vimeo videos as well, and Vimeo has an IFTTT channel, so I set up a recipe that when I 'like' a Vimeo video, then it tweets the title and link. 

To summarize: when I turn my TV on or off, the WeMo/IFTTT mechanism automatically captures those times, in a Google spreadsheet and on Twitter. However, in order to capture what I'm actually watching, I need to manually search for the show in the IMDB app, and then add it to my Watchlist, which, while not automated, is still pretty effortless. 

The next step is to write some Python and/or Javascript code to aggregate the @jamieinfinityTV twitter feed data and visualize it. My long-term goal is to integrate all of my various tracking feeds together into a more holistic picture of what I'm up to each day.

Codegraphy Project

Examples of relational data visualizations: CodeFlower is a D3.js module for drawing a file dependency graph; CodeCity represents classes as buildings and packages as districts laid out in a grid, color-coded with code metrics; and Circos is a tool for visualizing annotated relational data laid out on a circle.

Lately I've been doing a bit of research to find out what kind of code metrics are commonly used to better understand the structure and health of a codebase, and what tools exist for visualizing those metrics. It's a pretty vast subject (I've probably only scratched the surface in my research), but I'll try to give a summary of my findings so far, and sketch out what I hope to tackle in this area.

I would like to build a tool to visualize the relational structure and informational flow in a large-scale iOS project. I'm a very visual thinker who likes to gain a big-picture understanding of things, which I find can be difficult to do when joining on a new project with a large codebase. It'd be very helpful if there were ready-made tools available for Objective C to visualize a call-graph or dependency matrix of the code, color-coded with metrics like lines-of-code, cyclomatic complexity, test-coverage, or modification activity, which would help identify hot spots of potential code smells and make better-informed iterative architectural decisions. 

Although code metric and visualization tools do exist for statically typed languages like C# (e.g. Visual Studio, NDepend) and Java (e.g. Sonargraph, JArchitect), as well as dynamically typed languages like python (e.g. Radon, Python Call Graph), Ruby (e.g. Code Climate), and Javascript (e.g. JSComplexity, Code Climate), there seems to be a dearth of such tools in the land of iOS (see this Wikipedia page for a list of static code analyzers, which these tools are typically built upon). For Objective C I have unearthed a couple of tools that look worth investigating further: SonarQube is a multi-language platform for managing code quality that has an Objective C plugin. I also came across a blog post that describes how to set up iOS code metrics in Jenkins. There is also this python script for generating an import dependency graph.

Given that a visualization tool for Objective C code structure and metrics doesn't exist (at least not in a form that I have in mind), I've begun to explore what it would take to build one. The first ingredient I'll need is a tool for parsing through code and generating the relational graphs I would like to visualize. The clang compiler has an API library in C called libclang that can be used to parse C/C++/Objective C code into an abstract syntax tree (AST) structure, as well as process such structures. There is also a convenient python binding for libclang (for a helpful reference, see this blog post).

So, the first step in creating a visualization tool is to use libclang to process all of the Objective C code in a project into a graph data structure (or dependency matrix). But what defines this structure? What are the nodes and links? Depending on the analysis, one could consider a node to be a file, a class, or perhaps even an object. A link corresponds to some kind of directional relation between nodes, such as when a file depends on another file, or a class calls a method in another class, or an object is injected into another object, either via constructor or method injection. I've begun to explore these different possibilities and likely more than one will turn out to be useful.

The next major step after building a relational structure will be to calculate various code metrics, such as lines of code (LOC), complexity, and code coverage, which can be incorporated via various graph element stylings, such as node size and color. Aside from the usual basic metrics, it would be interesting to consider ways to quantify properties such as code coupling and cohesion, within the source code and between source and tests, to get a sense of how flexible the code is to modification. 

The final form of this tool will most likely be a D3.js driven interactive web page. I've come across some existing code that should serve as useful references, such as CodeFlower and DependencyWheel (which is similar to a Circos visualization). I'm also intrigued by the CodeCity project, which is based around a city metaphor, representing classes as buildings. I wonder how far one could take that metaphor, perhaps superimposing transit-like network structures to represent the flow of data through the system.