File System Infobase Manager

Using the file system for your notes.

I’ve been keeping notes and journals for as long as I can remember. Since I was born long before these cool PC/Mac thing’ies became ubiquitous, my earliest notes were taken on the best technology of the day, paper – but before too long I got with the revolution and moved on to electronics and started typing up all kinds of stuff.

Over the decades technology has been about as fickle as a saloon girl following a round up – just moving on to the next cowboy – and since I’ve used almost every type of system that was rolled out since the green screen VAX I played with in 1983 the resultant collection of my writing notes devolved into an inevitable mish-mash of different file types and structures.

In my notes folders I had files produced on AmiPro, WordStar, WordPerfect, Commence, Ecco Pro, an outliner called Think Tank, an outliner called Outliner, Outlook emails, stuff from an HP 95LX, Lotus Notes, all kinds of electronic exotica…But I carefully saved all these files ‘cause I was sure I’d use them somehow, someday.

In 2007 I became a full time writer and all of a sudden the mining of this hoard was no longer a future todo but a current issue. Someone told me that note taking, journaling and -crucially- retrieving old notes so they could actually be used, was a key writer’s skill. I scrounged for the old data folders and looked for a system to manage them all.

IT Architects like to call these kinds of piles “unstructured infobases” and there are lots of programs around – variously called information managers, PIMs, or Everything Buckets- to help you manage them. Surveying the field I adopted two and demoed a dozen more (probably during times when I should have been writing.)

First I poured all my notes into a fabulous program called Journler. Journler allowed me to think of my infobase as a structured whole and prompted me to habitualize the process of capturing and synthesizing the random bits of data that flowed past me every day. The transition to Journler got me out of all those old file types and into a fairly standardized TXT/RTF/RTFD/HTML set of documents, with PDFs and various image and audio files as accompaniment.

When I outgrew Journler (and you eventually always outgrow these packages) I transitioned to a beast of an application called DEVONThink. DT ultimately showed itself to be both constricting and superfluous. Along the way I played with Evernote, MacJournal, SoHo Notes, Mori, EagleFiler and Yohimbo. I’ve written about these attempts elsewhere. In their own way each one seemed lacking. Each one demanded attention to its own set of quirks that their programers thought of as features. Some were abandoned by their developers (Phil Dow).

So while things were now standardized and so much more usable, they were not stable. I had my notes in a common structure. I could sort and tag, edit and organize them, but I was dependent on one or another of these applications to make sense of it all…kind of like I’d been all along. I finally found a better way.

Now I’m using a system that is stable and sustainable and that seems to fall into the background while I work. It allows me to just refer to my notes, do my writing, create new ideas, synthesize old ones and not wrestle with a system or application while I’m doing it. I think it’s a long term solution, that is platform neutral, and vastly extensible.

It’s called the “file system”. Yep, the files system, that’s all. The very thing we use to run our computer every day. Shocking huh? After all those applications and proprietary file structures who would’a thunk that the best answer to electronic note taking would be the good old file system?

By using consistent file naming conventions and some highly abstract codes I have produced a vastly flexible system that is portable and lets me find just what I want without wasting time in arcane processes or learning the quirks of a program that my well be abandoned in a year or so.

I credit a denizen of the Scrivener discussion boards, amberV, with creating the core of the system. In a series of posts she turned the light on that let me think of how to organize my data in this simple but deeply powerful way. I’ve taken her ideas and modified them but not so very much as to be able to claim any credit for its development; amberV is brilliant and her ability to create vast robustness in a simple design shows this to be true.

Her original discussion are here, and here, but just to be clear the credit for these ideas should go to her, any problems due to my modification are my responsibility alone.

How’s it work?

The system relies on file naming conventions and folders; two things that are as stable, permanent and accessible as anything that will exist in the computing world. To understand this system you have to start with some philosophical ideas about info management.

Every note you take, every article you clip, every email you write is metaphorically just a sheet of paper in your infobase, kind of like a giant 3×5 card, or a single entry in your notebook. They can be complex or simple, but each one is a record, and each record needs to be retrievable in a series of consistent ways.

Most of my notes are RTF files. I like them better than TXTs because I can change fonts, use highlights, add tables and such. Often I’ll add pictures and save the files as RTFDs. Some of the RTFs are stories I’ve written that run a few thousand words, others are ideas that are just a sentence or two. Many are also PDFs of articles I’ve read and annotated. I have thousands of these files altogether.

The organization of these records is based on a index card filling system developed by a neo-ludite from Japan, Noguchi Yukio, who built a filing philosophy to manage his infobase of folders. Others have used it to manage thousands and thousands of index cards.

It’s not that hard to begin thinking of RTF/Ds or TXTs (or DOCs for that mater) as a series of 3×5 cards, only much bigger and more flexible, and when you look closely at Journler or DEVONThink you see that’s really what they are doing, creating a file system for bunch of text, graphic and pdf files.

So the path from card management system to infobase is not that long or arduous.

We all start out this way; and that’s the problem, right? We save a file or two in a folder on our hard drive. Then we begin to do more work and the number of files grows and all of a sudden we can’t see the structure of our work anymore because of all the clutter, so we folder some files and subfolder others. Then the names don’t make sense anymore and the folders don’t help us find stuff. That’s because the names we use for each file are one dimensional, or as the architects like to say, folders and files only have one “axis of information” for retrieval.

Abstract Coding

Most people want to get at their information from different ways at different times. They want to use multiple axis of data identification to work with their notes. Like you all I’ve kept documents in topic folders for years only to be frustrated by the limitations of the one axis categorization of foldering.

This really is the old paper journaling problem transferred to the 21st century. Journals are either perfectly chronological or perfectly topical, but it is very difficult to make them be both chronological and topical at the same time (ie: to use multiple axis of coding),and it is impossible for them to be multi-topical unless you have the transcription habits of a monk, copying and recopying items over and over.

When I used Journler I found that all my notes had other, more abstract, sets of categories beyond the single topic of its resident folder. Besides a note being being about “Art”, or “Productivity”, or “Non-Profit Management”, or “Strategy Formulation” or “Phenomenological Philosophy” (all folder names in in my topics list) they were also either chunks of text I had created, which I called “Thoughts”, or things other people had created, which I called “Notes.”

So, besides file name and topic categorization there was this additional dimension of categorization about who created the information to begin with.

Don’t be fooled by the simplicity of this split between “Thoughts” and “Notes.” This distinction – between what others have produced (inputs to your creative process) and its syntheses (what you do with it) – is really at the core of academic and artistic work. Recently both Merlin Mann and Twyla Tharp have been speaking about this at some length.

In addition to “Thoughts” and “Notes” I also realized that there were other categories at this level of abstraction. Below, when we get to the codes themselves you’ll see them. (And as with any good system development project I found that what I first believed to be categories at this level really weren’t. The idea is keep an equivalent level of abstraction as we work thorough the system. But it was a cool test of the system to see how easy it was to change.)

Chronology

The other idea from amberV was the predominance of chronology to the retrieval of information. Noguchi’s system was based on the importance of recency, which he felt was the best way to find relevant items. I may not remember that the Chomsky article I annotated and saved ended up in “Linguistics” or “Anthropology” or “MIT” or “Chomsky”, but I’m sure going to know that I worked on it in the Fall of 2008.

One of the wonderful things about Journler and to a lesser extent DEVONThink was the ability to flip through your work, to see what you were doing yesterday, or last week or this time last year, and to see it in the context of a set of ideas; what was I reading, what else was going on, was there an art exhibit of conference going on at the same time that influenced my thinking…what was in the news? Incorporating chronology in the system lets you recreate this function.

Tags, full text search on coding systems, reliance on metadata like Spotlight Comments all can be used to get at these dimensions but each is vulnerable to time or error as anyone who has had their “File Creation Date” redefined in a file copy operation can attest, or who lost a set of tags in the hidden OS X DS_ files that did not make the jump to a new folder. So the identification of this information has to be more robust. It also has to be modifiable and definable. A record may well be best dated in December because that was when you first worked on the project, not yesterday when you created the file. Relying on the file modification dates or other meta data means loosing that level of control.

The File Name

So the system has to accessible from multiple axis, one of which needs to be chronological, and it needs to be robust enough to be application independent.

In her system amberV used codes added in MultiMarkdown, a very sophisticated approach. I use the file name. It’s simpler for me and I don’t have to learn MMD. Every file I save has a date, a code, and a unique identifier for the file name.

So this file would be …

090608-W2-File System Philosophy.rtf

Where …

• 090608 is the YYMMDD date of my choosing (might be today, might reference prior dates if the material needs to be fixed in a different time)
• “W” stands for Writing, and “2″ is a sub category for non-fiction, or essays.
• “File System Philosophy” is the unique title, which can be preceded by its own characterization if I desire (I rarely do). The file itself is in a folder called “Dougist” because that is were it was published. But it could have gone in a folder called “Productivity”, or inside a rather extensive tree I have called “Systems”. It doesn’t matter because Spotlight can find the files anywhere.

I like using the file name for these codes because I can use the same system for all my files RTFs, PDFs, JPEGs, Scrivener containers, and the like. And when you get all your file names coded this way they line up in neat perfect columns in Finder so what looks like clutter turns out to be a very good visual reference.

I also like using the file name because you can use a bulk file renamer to do all the coding for you, which would be an insurmountable obstacle for most people (like me) if they have more than a few hundred files. I use A Better File Renamer and it adds the codes like magic. It took me months to structure up my data in Journler, a week to go from Journler to DEVONThink, but less than an hour to go from DT’s export to a fully coded and indexed system using ABFR. And then one day when I decided that I liked “R” (Record) better then “D” (Daily) for daily records, the renaming took 18 seconds. Similarly I re-categorized all the sub-categories of personal and professional development to “4” from “3” and it took about a minute.

And here is a key issue in the system. Rather than putting data into an application and then using the ho-hum functions of the app to work with my ideas, I keep my data separate and then have best in class applications act on it, using their higher levels of functionality.

For example, both Journler and DT (and EagleFiler, and notoriously MacJournal) have anemic text editing functionality. I use Bean and Scrivener (and occasionally Word, OmniOutliner, and WriteRoom) on my Thought and Writing files and get full-functionality.

Similarly, on my PDFs I can use Preview or Skim, or if I’m really out for some major modifications, Adobe. This was the key architectural point that Alex Payne was after in his article about Everything Bucket applications. By using Everything Bucket applications you give up functionality for compactness and eventually that equation works against your creative process. By working in the file system you use best in class apps for each specific purpose.

As an example: From a tagging perspective ABFR is vastly more sophisticated than the internal tags of Together will ever be, and even if the developer goes belly up, you can just move on to the next bulk file renaming utility. The integrity and functionality of your data is not dependent on the existence of the application.

When you start to contemplate the power you get from Word, or Chronosync vs what you give up in say, Evernote, the technical obstacles necessary to setting up a files system based info management system begin to melt.

A fellow Journler user, a brilliant and dedicated supporter of the product named NovaScotian, once commented on this approach, “but all you’ve done is recreate Journler” to which I say, yes, but it’s unbreakable and fully expandable. It will not have file size limits or file type limitations, it can port data into any project and provide information supplies to all my work efforts, and some day when OS X is replaced by ??? I’ll still be in business the next day with all my material.

The Tags

Now whether this coding goes into the file name, as I do it, or the first line of the document as others do, or the last line as a tag, or in the multimatemarkdown text is really less important than getting your head around the actual codes you will use.

Here’s my set.

The six File Name categories I use are:

• Record -R- Just personal recording. Ideas; observations; people watching; basically anything you might put in a diary. AmberV said, “It was liberating to separate thoughts from diary for me. In the past, I’ve had a problem with feeling guilty about keeping a mundane diary. I always felt like I should be doing something of quality in it. This category is not about quality–simply getting the “facts” down. I don’t have to worry about it being filled with eloquence, or using only the nicest inks, nibs, and papers. Just get it all out.”

• 1. Diary
• 2.
• 3. Action (ToDo, Project, etc) (Sub type “P” -R2P- for major projects)
• 4. Development

• Thoughts -T- AmberV described it as, “I draw the line between Record and Thoughts by saying, something that intends to “become” something goes in thoughts. Whether that be a thing that is already taking shape, or just an idea that might expand later. Perhaps creative things that are not attached to any particular project, like a line of prose. If I feel it is going to be become a story, or if it is a list of subjects for the next time I take my camera out, then it goes in Thoughts. This is where I am most liberal about sub-categories. It just makes sense to designate which book something is about, or whatever.” (Was once C=Creative)

• 1. Snips, Fiction
• 2. Observations, Non- Fiction
• 3. My processes and procedures, (Craft processes synthesis T3-W)
• 4. My life ideas, dreams (Goals: T4-G)

• Notes -N- Notes is just that; very similar to Record, except it is material that I have collected as opposed to produced. Everything from research for books, to funny anecdotes. This is also where I store bulk documents downloaded from the web or scanned from paper media. (Was once I=Information = Reference)

• 1. Research
• 2. Book notes (to sort and get my book list)
• 3. Processes (Craft notes N3-W)
• 4.
• 5. Refference
• 6. Quotes (??? Quotes are currently sub-categorised by QUOTE)

• Communications -C- Forums, emails, letters to friends, blog posts, tech support, and other things like that go here. I’ll sub-categorise this one too, if it is a person or forum that I frequently communicate with. (Was once M=coMmunication)

• 1. Private
• 2. Public
• 3. Meetings (Large or small, F2F to conferences, includes phone call notes)
• 4. Work submitted for review

• Writings -W- Thoughts that have grown, matured and been awarded a drivers license. This is my work of creation. Before long writings end up in a Scrivener file, but output of versions are kept as separate files with the name of the recipient as a sub-category.

• 1. Fiction
• 2. Essays and Non-Fiction
• 3. Writing about my writing (The process of my writing, what I am writing about)

• Projects – P- Transformational efforts, that can have notes, thoughts and records. The P is usually affixed to the containing folder. All writing work is project work, but is not included in this category.

• 1. Active Projects
• 2. Finished Projects

The other axis is Contextual and is File Folders

• Journal - Just like a paper one, a chronological list of items. Created from a smart folder that gets every code above.
• Topics - A vast sea of labels in sub-folders, roughly mirroring a library catalogue system or the course offerings at a University, culled based on my interests. Few of these are in current use for a project, but if they were, there would be an alias to them in a Project Folder. When does an item end up in Topics and not Journal? At some level of substance a card will belong in Topics; it’s arbitrary. The parallel question in a paper based system would be, when would you copy out your journal notes and file them with torn out articles in manilla folder.
• Projects – The main difference between Projects and Research is the transformative nature of the work occurring; sequential steps to get something done. Sub-folders are by year, because Projects are (should be) time bound with beginnings and ends. Quite often there is a Scrivener file in a project folder.
• Writing - Writings are different, somewhat timeless and un-categorizable. My writing folder is a special case; a combination of Journal, Research and Project. A purist would have put current writings in Projects and future ideas in Research, but it’s my system, so I have them separate. Groupings by my Fiction vs Essays, WIP vs Published, a few topical smart folders, mostly in support of potential writing projects.
• Organizer – This folder tells me where to go. It is a series of subfolders on my current contexts, like Writing Projects, Current Projects. The idea here these folder holds aliases to the data files in the rest of the database. The key is that these aliases are to current work. I once used flags and labels for this function, but I found that what I wanted to see was that I had a current project called “Develop Community of Writers”, not the 34 files associated with organizing a reading in February.

Other

Labels - I use labels arbitrarily to sort items in large folders. For my WIP, labels connote stage of development, from “goofy uncharacterized thoughts” to “ready to send out”. In class folders they separate administrative stuff, like syllabi, from thing like notes and assignments. The point is they have no global significance, their meaning can change from folder to folder.

X Files – Managing the undone. If I have unfinished work in a file, like I only partially completed a draft, I’ll add “X-” the the file name. I have a saved search that collects all these X files in one place, like a flagging system. Sometimes, if the list gets to long I’ll add ordinals to the X, “X1-” or “X2-” etc so they sort by some priority. I’ve also used labels in this situation too. I keep a little folder over in Organizer called “Administrative Tasks”. If something comes up that I will need to do, and that doesn’t fit anywhere else I’ll make an RTF/TXT here just so I have a file with an “X-” in its file name. It then shows up in the saved search. I’ve tried and tried all the todo list managers. OmniFocus alone devoured a collective month of my life. If you need to manage a list with this level of precision then you are a project manager not a writer. iCal todo’s or a little rtf with tasks work just fine. I could be convinced, maybe to use TaskPaper for this stuff, but I’d manage it the same way, in the file system.

Current Jags Folder – In addition to the Organizer folder, my most used folder is called “Current Jags”. It lives in the Organizer folder. I tend to have a lot of stuff I‘ve pulled down and saved but haven’t gotten around to reading or filing yet. I keep my desktop clean so it can be used as a work space for the current activity I’m on, so all this unprocessed stuff goes into Current Jags. To help you understand it’s purpose I’ll tell you that at one time I called it “Reading”.

Spotlight Comments - I use them sparingly. I have &trips tagged on R1′s about travel so I can search on them and see a history of all my sojourns. Similarly some N1′s are tagged &wrtitersonwriting when I have taken a note where an author speaks about craft. But I go back and forth about adding this text to the rtf itself so it is less susceptible to being washed away in a file copy someday. All my reading notes were once tagged &books but N2 took care of that.

More on file systems, archiving and note taking from Dougist…

Dating DEVONThink

Writing Tools – Journler

The Low Fi Manifesto – Data Architecture, and Journler

Shifting Mediums

WriteRoom and Notational Velocity

• Bookmark on Delicious
• Digg this post
• Recommend on Facebook
• Buzz it up
• Share on Linkedin
• Share with Stumblers
• Tumblr it
• Tweet about it
• Subscribe to the comments on this post
• Tell a friend
• Print for later