Fazal Majid's low-intensity blog

Adobe Acrobat 7 is a necessary evil for some operations such as OCR, rotating or deleting pages (although as an editor PDFPen is cheaper and more useable). Acrobat has grown into a sumo of an application, with incredibly sluggish load times. If you peek under the hood, you will notice it actually has a full copy of MySQL embedded in it. I am not fond of MySQL, but it is not that bad of a database as to account for all the bloat in Acrobat, who knows what else lurks in that package?

At any rate, Acrobat is to be used as a last resort, and most definitely not the viewer of choice. Apple’s own Preview.app is far snappier and more pleasant to use (no annoying Yahoo toolbars or spurious JavaScript warnings when you have the good sense to disable a potential security hole). Acrobat insists on taking over ownership of a file once you save it, however. You could manually change the HFS creator code for each file by hand, but this is where Spotlight’s efficiently indexed metadata database shows synergy with OS X’s underlying UNIX scripting magic. The following one-liner will clear the creator code for all PDF files on your system so they open with Preview instead of Photoshop. It can easily be added to a crontab or launchd script.

mdfind ‘kMDItemCreator==“Adobe Acrobat*”’|tr “\012” “\0”|xargs -0 -n 1 /Developer/Tools/SetFile -c ‘’

If you simply want to strip the creator application from all PDF documents altogether, this script will do the trick:

mdfind ‘kMDItemKind=“Adobe PDF document” and kMDItemCreator!=""’|tr “\012” “\0”|xargs -0 -n 1 /Developer/Tools/SetFile -c ‘’

mdfind queries the Spotlight metadata for all files with a creator that starts with “Adobe Acrobat”. xargs reads arguments from standard input and passes them as command-line arguments to the SetFile utility (part of the OS X Developer Tools). tr replaces line feeds with ASCII 0 NUL characters used as argument separators when xargs is invoked with the -0 option, so embedded spaces in filenames are passed unscathed. I have tested this with Unicode file names, and they are handled correctly as well.

Thanks to Spotlight, the whole process takes only a couple of seconds on my machine, something that would not be possible if it had to scan through 300GB’s worth of files. It also goes to show that Spotlight’s apparent sluggishness when you use it from the GUI stems entirely from the overhead of the GUI struggling to progressively display search results as they are returned by the metadata index, not from the underlying database engine. There is no justification for Google searches of the entire Internet being faster than a local desktop search.

Of course, the technique can be generalized to other ownership changes for contested file formats like HTML, JPEG or TIFF.

I am partial to milk chocolate with high cocoa content. It combines the best of both worlds: the rich flavor of dark chocolate, and the smoothness of milk chocolate. The better grades will be made of cocoa coming from a single region, ideally Venezuelan criollo. The natural candidate would be El Rey, a Venezuelan maker, but I don’t like the milky aftertaste of their Caoba 41% cocoa bar. My favorite, Michel Cluizel, makes superlative bars with 50% cocoa content, but the cocoa is from Java or Madagascar. They even used to have a 60% bar blended with almond cream, unfortunately it seems to have been discontinued.

A new specialty chocolate store, Bittersweet Cafe, opened recently on Fillmore Street in san Francisco. They have a good selection, almost as good as Fog City News, with some brands I haven’t seen before. One of those was this bar made from Venezuelan Chuao criollo beans, with a mix of chopped almonds, hazelnuts, and interestingly, pistachios. It is made by Chuao Chocolatier, a small artisanal company based in San Diego (but we won’t hold their unfortunate choice of location against them).

The bar has a rich, deep flavor, and is not over-sweetened as is unfortunately too often the case with milk chocolates. The cocoa content is not indicated, but I would estimate it at around 40%. The nuts are crunchy and fresh, with no hint of rancidity. This is no small feat, hazelnuts and pistachios spoil easily and are tricky to work with. Fran’s, based in Seattle, won’t even ship some of their Oregon hazelnut confections outside Washington State out of fear they will lose their freshness by the time they arrive. At $6.50 ($6 if you buy direct in packs of 4), this is by no means cheap, but at least they give you an unusual 110 gram portion, none of that wimpy 75 gram size companies like Scharffen-Berger (now a part of the despicable Hershey group) are turning to in order to increase profits.

Merry Christmas to all, hopefully one rich in cacao…

Update (2012-12-13):

They changed the packaging and name. It is now called the “Milk chocolate nut nirvana”.

Update (2013-04-20):

Sadly Bittersweet Cafe’s Fillmore location is no more.

The Story of a Ten-Minute Argument Between Two Great Philosophers

David Edmonds, John Eidinow

Harper Perennial, ISBN 0060936649, Publisher, Buy online

Plato is one of the sacred cows of the philosophy establishment. Whitehead famously wrote: “The safest general characterization of the European philosophical tradition is that it consists of a series of footnotes to Plato”. That said, his political theories are utterly indefensible. His attitude towards the Spartans who defeated and occupied his native Athens (and abolished the democracy the aristocratic Plato hated so much) would in modern terms be classified as that of a Quisling. His proposals are those of a soulmate of Pol Pot: a totalitarian society with a rigid caste system. In Plato’s ideal society, the philosopher-king caste rules through lies and deception¹, children are removed from their parents to be brainwashed by the state, if they are allowed to survive the state’s eugenic culling of the weak in the first place. Dissidents are interned in reeducation camps and killed if they do not eventually recant. Given such moral bankruptcy, it is hard to take anything else he wrote seriously.

One wonders why Plato’s popularity among philosophers remains undimmed. Perhaps this has to do with vanity, as many a philosophy professor toiling in mediocrity no doubt fancies himself an unjustly overlooked candidate for the position of philosopher-king. As related by Plutarch, Plato himself, exiled from Athens by the restored democracy, sought to flatter Denys (Dionysius), the tyrant of Syracuse, by extolling the latter’s putative virtues as a philosopher, in order to induce him to follow the program exposed in The Republic. Denys demonstrated that even tyrants sometimes show wit², dismissed Plato, and even considered enslaving the fawning philosopher so that he may experience a taste of what he advocated for others.

Another explanation may lie in Plato’s polemics against the Sophists. The Athenian democracy did not allow the use of lawyers in judicial proceedings, but the rich found a work-around: they hired specialists, essentially professors of rhetoric, to rehease trials. The Sophists would often boast that they could “prove” something and its opposite. This is a valuable, if somewhat debased skill, but it also has the side effect of destroying the credibility of philosophical methods, something Plato and his later admirers obviously object to. The Sophists were essentially skeptics and relativists, post-modernists over two millennia before post-modernism became fashionable. Plato’s was entirely successful in blackening their name, and the professional philosopher class no doubt owes some atavistic gratitude to him for that masterful propaganda.

Criticism of Plato’s views was mostly timid and limited to an audience of specialists, until Karl Popper revealed it for what it was in his blistering (and justifiably so) magnum opus of 1945, The Open Society and its Enemies: The Spell of Plato, although Bertrand Russell made pretty much the same points in his contemporary History of Western Philosophy. In his book, Popper exposes Plato as the wellspring for both Nazi and Communist totalitarism.

Popper was also a specialist in epistemology, the study of how knowledge is acquired, in science or elsewhere. His criterion of falsifiability (a theory is scientific only if it can be proven false), while not necessarily offering the practical guidance he hoped for, is still the best litmus test for whether a theory belongs to the realm of science (as in Darwin’s theory of Evolution) or not (as with creationism or its dissembling sibling, “Intelligent Design”).

As he believed in the worth of this endeavor, he had little patience for the hollow, superficial brilliance of a Ludwig Wittgenstein (I do not mean in any way to imply equivalence between Wittgenstein and a charlatan like, say, Derrida). Wittgenstein’s Poker retraces the background and history of the sole, explosive encounter between the two Austrians. Wittgenstein allegedly threatened Popper with a red hot fireplace poker, then stormed away in a huff. Debate still rages about exactly what happened, despite no dearth of witnesses, a testimony of sorts to the fallibility of human memory and the limits of the search for knowledge.

The book slowly and methodically lays the background behind the men’s confrontation. Both were born in assimilated Jewish families of Vienna, but the similarity ends there. Wittgenstein was the son of an immensely wealthy industrialist, and his brilliance was widely and immediately acknowledged by his peers. Popper’s middle-class family lost its savings in financial collapse, and he long had to struggle for both a living and recognition. The authors do a reasonably good job at explaining the philosophical differences between the two men in laymen’s terms. While both were refugees from the Nazis, the book’s insistence on their Jewish background as a major factor on their outlook seems a stretch.

After all this preparatory work, the description of the argument itself is a let-down. It is rather hurried, and makes too many unwarranted assumptions on the psychological state of mind of the protagonists, bordering on a fictionalized account.

¹: Interestingly, this concept of the “noble lie” was revived by the so-called neo-cons, many of whom are followers of Leo Strauss, an admirer of Plato. This only goes to show the enduring relevance of Popper’s Open Society.

²: In another instance, Argentina’s Juan Perón “promoted” his critic Jorge Luis Borges to inspector for poultry and rabbits at the Buenos Aires municipal market. Borges reacted with vitriol, but his later approbation of the 1976 junta puts the lie to any grandiose claims of principle.

First in a series:

1. First impressions
2. Asset management
3. Under the hood: file format internals

Cost and hardware requirements

The first thing you notice about Aperture, even before you buy it, is its hefty hardware requirements. I had to upgrade the video card on my PowerMac G5 (dual 2GHz, 5.5GB RAM) to an ATI X800XT, as the stock nVidia 5200FX simply doesn’t make the cut.

Aperture costs $500, not far from the price of the full Photoshop CS2. Clearly, this product is meant for professionals, just like Final Cut Pro. The pricing is not out of line with similar programs like Capture One PRO, but it is rather steep for the advanced amateurs who have flocked to DSLRs and the RAW format. Hopefully, Apple will release a more reasonably priced “Express” version much as they did with Final Cut Express.

File management

Like its sibling iPhoto, Aperture makes the annoying assumption that it will manage your photo collection under its own file hierarchy. It does not play nice and share with other applications, apart from the built-in Photoshop integration, which merely turns Photoshop into an editor, with Aperture calling all the shots and keeping both image files and metadata databases firmly to itself. Photoshop integration does not seem to extend to XMP interoperability, for instance.

This is a major design flaw that will need to be addressed in future versions — most pros use a battery of tools in their workflow, and expect their tools to cooperate using the photographer’s directory structure, not one imposed by the tool. Assuming one Aperture to rule them all is likely going to be too confining, and the roach-motel like nature of Aperture libraries is going to cause major problems down the road. Copying huge picture files around is very inefficient — HFS supports symbolic and hard links, there is no reason to physically copy files. This scheme also renders Aperture close to useless in a networked environment like a magazine or advertising agency, where media files are typically stored on a shared SAN, e.g. on XServe RAID boxes using Fibre Channel.

Fortunately, the file layout is relatively easy to reverse-engineer and it is probably just a question of time until third-party scripts become available to synchronize an Aperture library with regular Pictures folders and XMP sidecar metadata files, or other asset management and metadata databases like Extensis Portfolio or even (shudder) Canto Cumulus. Apple should not make us jump through these hoops – the purpose of workflow is to boost productivity, not hinder it. In any case, Apple is apparently hinting Aperture can output sidecar files, at least according to PDN’s first look article

Performance

Aperture is not the lightning-fast RAW converter we have been dreaming of. Importing RAW files is quite a sluggish affair, taking 2 minutes 15 seconds to import a 665MB folder with 86 Canon Rebel XT CR2 RAW files. In comparison, Bridge takes about a minute to generate thumbnails and previews for the same images. The comparison is not entirely fair, as Aperture’s import process yields high-resolution previews that allow you to magnify the image to see actual pixels with the loupe tool, whereas Bridge’s previews are medium resolution at best. The CPU utilization on my dual G5 is far from pegged however, which suggests the import process was not particularly tuned for SMP or multi-core systems, nor that it even leverages OS X’s multithreading. Aperture will work with other formats like scanned TIFFs as well (import times are even slower, though).

Once import is complete, viewing the files is very smooth and fast. The built-in loupe tool is particularly addictive, and very natural for anyone who has worked with a real loupe on a real light table. A cute visual effect (Quicktime, 6MB) has the loupe flip over as you reach the edges of the screen. The loupe will also magnify the thumbnails, although that will pause execution for the time it takes to read the thumbnail’s preview data into memory.

Workflow innovations

Aperture has two very interesting concept: stacks and versions. Stacks group together multiple images as one. It is very common for a photographer to take several very similar photos. Think of bracketed exposures, or a sports photographer shooting a fast action sequence at 8 frames per second, or a VR photographer making a series of 360-degree shots for use in an immersive panorama. Aperture’s stacks allow you to manage these related images as a single unit, the stack. It is even capable of identifying candidates for a stack automatically using timestamps.

Stacks

This article is work-in-progress, and this section has to be fleshed out

Versions

Versions is a concept clearly drawn from the world of software configuration control systems like CVS or Visual SourceSafe. Aperture does not touch the original image, adjustments like changing the color balance simply record the series of operations to achieve the new version of the image in the metadata database, just like CVS only stores diffs between versions of a file, to save space. This suggests Apple plans future versions of Aperture with shared image repositories, as most modern systems work that way, with a shared central repository, and individual copies for each user, with a check-in/check-out mechanism with conflict resolution.

The parameters for a transform take a trifling amount of memory, and the photographer can experiment to his heart’s content with multiple variants. Photoshop now has equivalent functionality with the introduction of layers comps in CS, but they still feel like bolted-on features rather than integral to the product.

In the early nineties, French firm FITS introduced a groundbreaking program named Live Picture to compete with Photoshop. Its chief claim to fame was that it could deal with huge images very efficiently, because it recorded the operations as a sequence of mathematical transforms rather than the result, in a way eerily reminiscent of PostScript. The transforms were only applied as needed at the display resolution, thus avoiding the need to apply them to the full-resolution image until the final output was required, while still managing to deal with zooming adequately. The software was promising, but the transformation logging technique limited the types of operations that could be performed on images, and due in part to its high price and specialized scope, the product died a slow and painful death. In its current incarnation, it lives on, after a fashion, in the moribund Flashpix image format and a web graphics server program imaginatively named ImageServer.

Chained transforms is a very elegant approach when compared to the brute-force of Photoshop — in Aperture a finished image is represented as a master image and a series of transformations to be applied to it to achieve each version, much like Photoshop keeps a history of changes made to the image in memory (but not in the final disk file). Since Aperture’s transforms are fairly simple, they can be executed in real time by modern graphics cards that support Core Image.

Target market

Keep in mind there are two types of pro photographers: those who divide photographers in two groups, and the others… More seriously:

1. Those who try to build up a portfolio of images over their career, where royalties and residual rights will provide them with financial support when they retire. Most fine art, landscape or nature photographers are in this category, and photojournalists could be assimilated (except their employer owns the rights to the archive in the latter case).
2. Those who do work-for-hire. Wedding photographers, event photographers, product, catalog, advertising and industrial photographers fit in this category.

The first type need a digital asset management database to retrieve and market their images more effectively, and a distribution channel. Most farm out that representation work to the likes of Corbis or Getty Images.

The second type will work intensely on a project, take a lot of frames and show many variants to a client for approval. Once the project is done, it is archived, probably never to be used again, and they move on to the next one. In most cases, the rights to the images remain with those who commissioned them, not the photographer, and thus there is no incentive to spend much effort in organizing them with extensive metadata beyond what is required for the project. These users need a production workflow tool that will streamline the editing and client approval process, the latter mostly performed via email or password-protected websites nowadays.

Aperture’s projects (flat, not nested hierarchically, and thus not all that scalable) and vaults (archives where projects go when they are no longer needed or finished) are a clear indication it is intended mostly for the second type of photographer. Apple did not convey the specialized focus of the product forcefully enough, but the blogosphere’s buzz machine bears much of the blame for raising unwarranted expectations about what the product is about.

Aperture is good for one thing: let wedding photographers and the like go through the editing (as in sorting through slides on a light table, not retouching an individual image) as efficiently as possible. Most wedding pros simply cannot afford the time to individually edit a single picture beyond white balance, tonal adjustments, cropping and sharpening, and Aperture’s built-in tools are perfectly adequate for them.

That is also why there is such a slow import process — this prep work is required to make the actual viewing, side by side comparison, sorting and manipulation as smooth, interactive and responsive as possible to avoid disrupting the photographer’s “flow”. The goal is to have a very smooth virtual light table, not a filing cabinet, where you can move slides around, group them in stacks, toy around with versions, and compare them side by side on dual 30 inch Cinema Displays. The user experience was clearly designed around what the average art director (with his or her loupe almost surgically implanted) is familiar and comfortable with.

The positioning as “iPhoto Pro” obscures the fact that Aperture is sub-optimal for managing or indexing large archives with rich metadata. For the first type of pro photographer (which is also what the average advanced amateur will relate to), a digital asset management database like the excellent Kavasoft Shoebox or the more pedestrian but cross-platform and workgroup-oriented Extensis Portfolio and Canto Cumulus will better fit their needs, with Adobe Bridge being probably sufficient for browsing and reviewing/editing freshly imported photos (when it is not keeling over and crashing, that is).

Conclusion

Aperture is clearly a 1.0 product. It shows promise, but is likely to disappoint as the reality cannot match the hype that developed in the month or so between the announcement and the release. The problem is that there is a Cult of the Mac that raises unrealistic expectations of anything coming out of Cupertino.

The hype around Aperture was certainly immense, I am sure Apple was as surprised as any by how positive the response was (after all, they released Aperture at a relatively obscure pro photo show). They are probably furiously revising plans for the next release right now. I consider Aperture 1.0 more a statement of direction than a finished product.

Last in a series:

1. First impressions
2. Asset management
3. Under the hood: file format internals

This article was never completed because I switched to Lightroom and lost interest. What was done may be of interest to Aperture users, although the data model probably changed since 1.0

Aperture stores its library as a bundle with the extension .aplibrary. This is a concept inherited from NeXTstep, where an entire directory that has the bundle bit set is handled as if it were a single file. A much more elegant system than Mac OS Classic’s data and resource forks.

Inside the bundle, there is a directory Aperture.aplib which contains the metadata for the library in a file Library.apdb. This file is actually a SQLite3 database. SQLite is an excellent, lightweight open-source embedded relational database engine. Sun uses SQLite 2 as the central repository for SMF, the next-generation service management facility that controls booting the Solaris operating system and its automatic fault recovery, a strong vote of confidence by Sun in SQLite’s suitability for mission-critical use. SQLite is also one of the underlying data storage mechanisms used by Apple’s over-engineered Core Data framework.

You don’t have to use Core Data to go through the database, the /usr/bin/sqlite3 command-line utility is perfectly fine for this purpose. Warning: using sqlite3 to access Aperture’s data directly is obviously unsupported by Apple, and should not be done on a mission-critical library. At the very least, make sure Aperture is not running.

ormag ~/Pictures/Aperture Library.aplibrary/Aperture.aplib>sqlite3 Library.apdb
SQLite version 3.1.3
Enter ".help" for instructions
sqlite> .tables
ZRKARCHIVE             ZRKIMAGEADJUSTMENT     ZRKVERSION
ZRKARCHIVERECORD       ZRKMASTER              Z_10VERSIONS
ZRKARCHIVEVOLUME       ZRKPERSISTENTALBUM     Z_METADATA
ZRKFILE                ZRKPROPERTYIDENTIFIER  Z_PRIMARYKEY
ZRKFOLDER              ZRKSEARCHABLEPROPERTY
sqlite> .schema z_metadata
ormag ~/Pictures/Aperture Library.aplibrary/Aperture.aplib>sqlite3 Library.apdb
SQLite version 3.3.7
Enter ".help" for instructions
sqlite> .tables
ZRKARCHIVE             ZRKKEYWORD             ZRKVOLUME
ZRKARCHIVERECORD       ZRKMASTER              Z_11VERSIONS
ZRKARCHIVEVOLUME       ZRKPERSISTENTALBUM     Z_9VERSIONS
ZRKFILE                ZRKPROPERTYIDENTIFIER  Z_METADATA
ZRKFOLDER              ZRKSEARCHABLEPROPERTY  Z_PRIMARYKEY
ZRKIMAGEADJUSTMENT     ZRKVERSION
sqlite> .schema zrkfile
CREATE TABLE ZRKFILE ( Z_ENT INTEGER, Z_PK INTEGER PRIMARY KEY, Z_OPT INTEGER, ZASSHOTNEUTRALY FLOAT, ZFILECREATIONDATE TIMESTAMP, ZIMAGEPATH VARCHAR, ZFILESIZE INTEGER, ZUUID VARCHAR, ZPERMISSIONS INTEGER, ZNAME VARCHAR, ZFILEISREFERENCE INTEGER, ZTYPE VARCHAR, ZFILEMODIFICATIONDATE TIMESTAMP, ZASSHOTNEUTRALX FLOAT, ZFILEALIASDATA BLOB, ZSUBTYPE VARCHAR, ZCHECKSUM VARCHAR, ZPROJECTUUIDCHANGEDATE TIMESTAMP, ZCREATEDATE TIMESTAMP, ZISFILEPROXY INTEGER, ZDATELASTSAVEDINDATABASE TIMESTAMP, ZISMISSING INTEGER, ZVERSIONNAME VARCHAR, ZISTRULYRAW INTEGER, ZPROJECTUUID VARCHAR, ZEXTENSION VARCHAR, ZISORIGINALFILE INTEGER, ZISEXTERNALLYEDITABLE INTEGER, ZFILEVOLUME INTEGER, ZMASTER INTEGER );
CREATE INDEX ZRKFILE_ZCHECKSUM_INDEX ON ZRKFILE (ZCHECKSUM);
CREATE INDEX ZRKFILE_ZCREATEDATE_INDEX ON ZRKFILE (ZCREATEDATE);
CREATE INDEX ZRKFILE_ZFILECREATIONDATE_INDEX ON ZRKFILE (ZFILECREATIONDATE);
CREATE INDEX ZRKFILE_ZFILEMODIFICATIONDATE_INDEX ON ZRKFILE (ZFILEMODIFICATIONDATE);
CREATE INDEX ZRKFILE_ZFILESIZE_INDEX ON ZRKFILE (ZFILESIZE);
CREATE INDEX ZRKFILE_ZFILEVOLUME_INDEX ON ZRKFILE (ZFILEVOLUME);
CREATE INDEX ZRKFILE_ZISEXTERNALLYEDITABLE_INDEX ON ZRKFILE (ZISEXTERNALLYEDITABLE);
CREATE INDEX ZRKFILE_ZMASTER_INDEX ON ZRKFILE (ZMASTER);
CREATE INDEX ZRKFILE_ZNAME_INDEX ON ZRKFILE (ZNAME);
CREATE INDEX ZRKFILE_ZPROJECTUUIDCHANGEDATE_INDEX ON ZRKFILE (ZPROJECTUUIDCHANGEDATE);
CREATE INDEX ZRKFILE_ZUUID_INDEX ON ZRKFILE (ZUUID);
sqlite> .schema z_metadata
CREATE TABLE Z_METADATA (Z_VERSION INTEGER PRIMARY KEY, Z_UUID VARCHAR(255), Z_PLIST BLOB);
sqlite> .schema z_primarykey
CREATE TABLE Z_PRIMARYKEY (Z_ENT INTEGER PRIMARY KEY, Z_NAME VARCHAR, Z_SUPER INTEGER, Z_MAX INTEGER);
sqlite> select * from z_primarykey;
1|RKArchive|0|1
2|RKArchiveRecord|0|0
3|RKArchiveVolume|0|1
4|RKFile|0|2604
5|RKFolder|0|23
6|RKProject|5|0
7|RKProjectSubfolder|5|0
8|RKImageAdjustment|0|1086
9|RKKeyword|0|758
10|RKMaster|0|2604
11|RKPersistentAlbum|0|99
12|RKPropertyIdentifier|0|119
13|RKSearchableProperty|0|84191
14|RKVersion|0|2606
15|RKVolume|0|0
sqlite>

One useful command is .dump, which will dump the entire database in the form of the SQL commands required to recreate it. Even with a single-photo library, this generates many pages of output.

Here is my attempt to reverse engineer the Aperture 1.5.2 data model. CoreData, like all object-relational mappers (ORMs) leaves much to be desired from the relational perspective. The fact SQLite foreign keys constraints are not enforced (and not even set by CoreData) doesn’t help. Click on the diagram below to expand it.

All tables are linked to Z_PRIMARYKEY which implements a form of inheritance using the column Z_ENT to identify classes. The only table that seems to use this today is ZRKFOLDER, where the rows can have a Z_ENT of 5 (Folder), 6 (Project) or 7 (ProjectSubfolder). For clarity, I have omitted the links between all tables and Z_PRIMARYKEY.

ZRKIMAGEADJUSTMENT looks like the table that records the transformations that turn a master image into a version, Live Image style.