Lately, I’ve been doing the exercises in Making Things Talk.

I put some pictures here.

Although I’ve got a DYMO printer, I thought I’d test and see how well Moo stickers would work with the current geohash generator.

The stickers are gorgeous, and perfect for, say, putting in the margins of a book or on a printout.

But they’re a bit small, which means that although my iPhone _can_ read them (with the free NeoReader application, search for it in the app store), it’s still a bit hit-or-miss.

(Would like to see how well the G1 handles it, with its, ahem, superior camera and barcode reader.)

Still, with a bit of tweaking, I can see this working if you wired the Moo API up to the automagic hash generator, and get a pocket full of geohash tags…

What I find compelling is what isn’t visible here: the photo was taken and uploaded with an Eye-Fi and digital camera, without any prompting on my part, except for taking the camera and leaving the camera on. (I got a twitter direct message when the upload was done, which was pretty cool.)

So the photo is pretty high resolution. I just viewed it on full size on flickr, and was able to scan the QRCodes successfully using my iPhone. That means that, conceivably, you could use the Flickr API to trawl your photostream for new photos, check for QRCodes, interpret them, and… what?

That’s an interesting question, isn’t it?

Maybe there’s an existing URL that matches the (unique geohashed) QRCode, and you get that dropped into the photo as a link on a photonote?

Maybe you _own_ that domain name, and you’ve got a CMS or blog running on it. It could create that URL if it doesn’t exist, and put the flickr photo on it.

Maybe there’s text in that photo, now captured at high resolution. If Evernote can trawl through photos to OCR them and make them searchable, then, well, that’s a good proof of concept, right?

(When I mentioned this to Charles Warren, he pointed out that you could run the text against a Google Book Search. That’s another possibility.)

The point, here, is that it’s getting easy to grab image data and put it into a form that’s accessible through a data-centric API. Like Flickr’s (for example). You can do that with a ScanSnap sheet-feed scanner, or a digital camera with eye-fi, or a cameraphone, or a camera mounted above your desk.

That’s my pet obsession: getting unique, tagged data into an API, and then making the transition between the digital shadows and their physical equivalents as frictionless as possible. I’m calling my pet instantiation of the concept “Paper Bits,” and it’s what I’d like to discuss at the PaperCamp that Matt Jones suggested.

I’m not a real programmer: I’d like to create this as an open source software project, and have no clue where to start on that. So, any suggestions are welcome…

There really is too much to summarize in a single post, but here are some of the highlights as I saw them.

People

I had the honor of meeting and befriending some amazing folks. A partial list:

Jennifer Magnolfi, Matt Jones, Timo Arnall, Matt Cottam, Michelle Malott, Chris McCray, Charles Warren, and probably a half-dozen others whose contact data I haven’t pulled out and reviewed yet. (Sorry!)

We managed to spend most of the summit sitting at a table in back, with half of us working on their presentation for the next day while listening to the presentation, and the other half listening to the presentations and exchanging (occasionally snarky) comments on twitter.

The GooglePhone’s internal compass and its ability to do an augmented-reality view with it, using Google Street View, was the technological nerdgasm of the weekend.

PaperCamp

The highlight of the weekend, for me, was meeting Matt and Timo, and getting to talk about paper as prototype spime.

(It was a bit gratifying, when meeting Matt, to be greeted with, “Oh, you’re that guy!” Admittedly, that’s usually what people say when a locally infamous eccentric shows up at a party, but it was fun nonetheless.)

Timo made the insightful comment that the idea of just attaching a URL to paper is kind of broken, and we need richer and more interesting interaction patterns. He and I agreed that the usability of QRCodes is quite poor as they stand, but they are a printable, inexpensive analog for passive RFID and touch interaction in some ways.

Matt percolated a bit, and came up with the idea of having a PaperCamp event for like-minded people to explore what can be done with printable protospimes.

Matt’s suggested topics:

• Way-new Printing
• Protospimes
• Ingestion/Digestion/Representation
• Bionic Sketching
• Folding/Structure
• Paper’s Children

(“What would Paper’s Children be,” I asked. “That’s the point, I’m asking you,” Matt said.)

(I have some ideas about that, but let’s save it for later.)

Memes!

Some ideas that got bounced around, from talks, hallway discussion, and drunken ranting:

• Personal Informatics
• Game Mechanics for motivation and change
• Manatee Rape Waivers – band name, or legal necessity?
• The lack of personal fear in a welfare state (Norway)
• Warren Ellis won’t actually eat your heart if you buy him a beer.
  (Matt claimed this, but I’m still skeptical)
• Obama as the president in Independence Day
• RFID fields as physical objects
• Wii Fit and Nike Plus –- and how you should be able to change and record the messages in both
• Will the FUBAR security fiasco in Mifare RFID chips harm NFC adoption? (Yes.)

So, in other words, we made as much sense as you’d expect.

On the whole, as I said later, it was simply refreshing to show an example of my thought experiments and for people to not look at me like a dog that’s been shown a card trick. Worth the price of admission right there.

Thank-you’s

I really have to thank Dave for inviting me, and of course the summit organizers, Michelle Bauer and Mary Craig for allowing me to attend. Also, everyone else for being generally awesome.

What’s next?

Well, I just got this, from Matt Jones:

@paperbit 15.24, restate your assumptions.

Sounds like a good place to start.

Vladimir Nabokov wrote on index cards, and this enabled him to create wonderfully nonlinear books. One of my favorites is Pale Fire, a book that manages to tell three stories at once, written as the critical analysis of an autobiographical poem.

If that description sounds about as much fun as being repeatedly punched in the balls by circus midgets while alphabetizing census records, I assure you that it is anything but.

In any case, I’d like to take Pale Fire (which is sort of about, and definitely revolves around, an autobiographical poem written in pencil on index cards), and make an edition of it with the poem printed both traditionally (in the first section), and also on index cards. These would be spread through the pages of the book, which is ostensibly a criticism of the poem itself (although it isn’t really, and the book should certainly be read to see why).

What makes the idea seductive to me is that you could easily use semacodes to link the cards (and individual notes on the poem’s stanzas in the “commentary” pages) to an online, networked version. I imagine it as being like a blog, with wiki-style comments.

If done correctly, the result would be a networked book, on paper.

Why?

Well, since there are many possible ways to read the book, and no real authorial word on which is the “true” story, the reader would be invited to join the commentary and argument over the book. This might provide a different way of engaging the story than reading and even discussing the “dumb” paper version.

In addition, the paper itself provides affordances for physical annotation, offline reading, and mixed media, with the additional interest of the poem’s “original” itself being provided along with the printed version.

(Incidentally, this would also fulfill a long-standing geek dream of mine: I’d love to be able to trackback-ping a passage from a paper book I’m reading, from my blog. And see the trackback when I reread the book.)

It would also be a wonderful interaction- and graphic-design problem to solve. With index cards. And data shadows.

The concept is an attractive nuisance. I really can’t let it go, but I really have no idea how to go about getting the rights to do so. Pale Fire is, after all, still under copyright. And I’m not a printer, by any means. But damn, I can’t let the idea go.

This is what happens when I empty out a month or two’s worth of photographs from my camera at once…

Thought I’d put up some links to more ideas along those lines, just while I’m about to head out somewhere…

Pasta and Vinegar linked to an awesome paper about the role played by artifacts in cognition. Which is sort of tangentially related to this idea, if you squint.

Also, if:book has been working over the idea of the networked book, the theory of which they explore in more detail here.

In the comments to the latter entry, Adam Greenfield says:

I’ve become all but unable to think of the objects around me except in terms of Actor-Network theory, as sort of depositions or instantiations of a great deal of matter, energy and information moving through the world. And of course, a book is nothing but a snapshot in that regard; you have to do a lot of extra work if you want to prise out and examine the flows it is a part of, or even those it has set up.

And, because they’re actually brilliant people who do things, rather than blogging dilettantes like myself, if:book is experimenting with a networked book, GAM3R 7H30RY 1.1.

Add in the fact that new Nokia phones come with 2d barcode reading software built in, and it looks like maybe paper books with digital shadows aren’t such a far-fetched idea, after all.

Just a thought.

(Edited to fix horrible sloppy writing… this is why I try to do first drafts on paper.)

In the latest The Ministry, Warren Ellis talks about books with a digital shadow:

On the assumption that in the Western world few people buying comics are too far away from a library or internet cafe at the very least, the potential exists to build an accessible data shadow above any book.

In fact, it could be done in very sophisticated ways. Electronic hyperlinks could be built right into the pages.

…Vertigo books from the ’90s — INVISIBLES, PREACHER, TRANSMETROPOLITAN — were books about ideas. The three of us were writing about our discrete areas of interest, and, in large part, we were telling you about the things we knew. Which isn’t a bad thing. Some people balk at writers having any opinion, interest or intent beyond banging out a neutral yarn, but, you know, fuck that noise. Comics are an educational tool, used for anything from instructional pamphlets for civil disobedience to workplace hygiene. The best fiction, like the best reportage, is about the writer telling the reader where they think they are today, and what they think it looks like.

That would be an interesting way to do a modern comic. One that has its own electronic universe standing behind it, accessible through an URL printed on the front of the book, or multiple URLs seeded throughout the book. The book would not rely on them for its effect and textual integrity, but it would be supported and extended by a directory of information about the book, produced both by the creators and those of its audience who wished to extend their consumption of and involvement with the book.

Let’s explore the possibilities that this opens up.

Let’s say you are a budding comic artist or writer. You read Ellis’s latest epic, which is all papertagged with semacodes or NFC tags, or something else that’s much less clumsy. You start looking through it, and (on the second or third reading, maybe) you follow the hyperlinks.

You find that, in addition to the supplemental creative material that Warren talks about above, the comic has links to a development wiki set up for the purpose of developing the technology that makes the comic project run. It has HOWTO files, source code, downloadable client binaries, the works. The discussion pages have the developers and Warren talking about what the thing should do, and how.

You can see where the dead ends were, and how they were worked around (or not), and a development roadmap for the project. Maybe even a mailing list or separate wiki for it.

There is also a trail of the research and writing Warren did while developing the idea, which already (to an extent) exists: here you see where he bookmarked Semacode with the tag spimeworld; here you see his notes, some links to other peoples’ blog posts on the topic, and the ministry article where he first started publicly talking about the idea.

In short, you see the germ, flowering, and development of the physical object you hold in your hand, both creatively and technically; you have its DNA. The source code. And it can teach you how to do the same.

So, you take that, and you branch off of the wiki, creating your own project, which contains the old, at least in the form of links back to it. You start researching your worldbuilding, and slowly build it with a melange of newspaper articles and sketches and scraps of your own writing.

You write the script, and you and an artist pull it all together, and carefully craft both the work itself, and the links to the source material. Months before the comic is published, the site goes public, and (if you’re lucky), a community forms around it. The paper comic becomes a physical extension of the digital shadow.

More: the comic becomes a coherent structure that takes the big pile of digital mess and cruft and shapes it, and relates it all together in a way that helps tell a story, build a world, and make sense of it, in a way that’s substantially different from the website. It’s a lens to look through.

And the next writer or artist to pick it up can follow the trail back to Warren Ellis’s comic, and the evolution of the system continues. Eventually, you have whole generations and branches of trial and error, and failed experiments, and successful ones.

These become design patterns, embodied in physical objects, which are born from the digital shadow of their source material and the contributions of the artist, writer, fans, critics, geeks, and hackers.

The comic becomes a “theory object.”

Okay, digression time.

Last year, I ran a year-long kind of experiment, where I used index cards exclusively, both to work (i.e. to sketch and to write), and to track the work, using the Getting Things Done “project” paradigm.

It became a sort of meta-meta-project, where I took notes on the process, using the process, and used the cards to brainstorm and sketch better ways to organize the cards, which I then used to organize the sketches and notes. And that testing allowed me to iterate and improve. It was a bootstrap process that was far more fun than you should really legally be able to have while fully clothed and playing with office supplies.

Of course, I didn’t really get much else done, which is a real flaw of any do-it-yourself tool for work tracking. (Most of the hackers I know who have run into this, of course, were doing it on the computer, using python and emacs planner and applescript, and many of those folks ended up using paper. I followed the opposite course, which is itself yet another example of something.)

In any case, I did manage to get some work done, in between experimenting with manila tabs versus coin envelopes for structured card piles, and noticed something interesting as I did so.

That is, when the tool you use to work is itself the tool you use to track your work, the completed work seems to end up as a richer and more meaningful object.

Because it is linked to, or even contains, the sum of the decisions, mistakes, variations, source material, notes, comments, and other metadata that you’ve used to track and manage the project, the thing has a kind of DNA. It’s as though it were food that contains, not only the recipe, but the history of how the recipe was worked out, by trial and error.

I didn’t really have a way of articulating this until I read Julian’s post, Theory Objects and Design Patterns:

The “Theory Object” is a kind of design pattern for design itself. It’s about thinking by creating, and revealing the trail of discovery in the process of making things. And it has this recursive characteristic ‚Äî not just revealing the process in the documentation or the manual or the “write-up” or the “post-mortem”, but actually embedding all of this within the designed Thing itself.

If I’ve got this right, what Julian means, here, is that a “Theory Object” is where you’ve got a prototype, not only in the literal sense, but in the psychological sense of an invariant representation. It holds the set of decisions and design concepts that it embodies within itself, as its digital payload.

Which brings us back to Warren Ellis and his ideas about digital shadows.

This is much like Doug Engelbart’s 1962 vision of an augmented architect’s toolkit (scroll down), where the research and calculations that an architect makes while drawing a home are embedded within the digital blueprint.

The core concept is that the deliverable, itself, is only one (minor) artifact of the creative and critical process, and there is a whole network of theory, concepts, decisions, other possible variations, and other information that, right now, is neither interconnected, nor archived, and is certainly not connected to the deliverable itself.

Thus, the deliverable often needs explanation, or ends up creating a lot of rehashing of the work already done. However, if the deliverable was a theory object, where the documents referenced, history of decisions, discussions, and other elements that created the deliverable, you would have something very much like r0ml’s vision of open source… where the food is served with the recipe that created it.

But in this case, it’s not just the recipe. It’s the lineage of the experimentation and variation that led to the food, which is one physical instantiation of that lineage.

Julian:

The Theory Object is a way of making things ‚Äî of doing design ‚Äî that reveals the process of design, all of Latour’s inscriptions, all the iterations all the trials and failures and wrong turns. Why? To make things public in this way reveals the “made”, “hand-crafted” social character of things. And it’s better, politically and for better futures, to know that something was made this way rather than that way, why such decisions are made, and by whom.

…

Here, finally, is the Theory Object recursion I was talking about: the Theory Object is the design pattern and the instantiation. It is a way to design — a pattern — and the outcome of adhering to that design pattern.

This idea is compelling.

It certainly rubs hard up against some hard-held ideals of design and intellectual property; would a product designer really want her design sketches and notes accessible via the manufactured object? She might (certainly it would extend the reach of her portfolio), but odds are the manufacturer and patent holder might disagree.

What about the prototype, then? Internally, after several years of working with theory objects, a corporation or design firm would have a set of objects surrounded by digital data clouds, living design patterns to be picked up and re-applied when the next revision is due.

Anyway.

Many of the ideas you read about in ubiquitous computing and smart spaces have to do with entertainment: rock videos following you around the house, books that make the room act as sound effects and lighting and things like that.

But there is real value to be had in the idea of objects gaining digital shadows that goes beyond getting people to buy a new stereo so they can play with your latest woo-woo toy. I’m interested to see what else Julian comes up with, and inspired to start developing some of these ideas, too.

As soon as I figure out how, because PalmONE won’t open the camera up on my Treo to use semacode.

(Any tips on where to start from those already playing with this stuff would be much appreciated! :))