Many digital photographers (including myself) prefer shooting in so-called RAW mode. In theory, the camera saves the data exactly as it is read off the sensor, in a proprietary format that can later be processed on a PC or Mac to extract every last drop of performance, dynamic range and detail from the captured image, something the embedded processor on board the camera is hard-pressed to do when it is trying to cook the raw data into a JPEG file in real time.

The debate rages between proponents of JPEG and RAW workflows. What it really reflects is two different approaches to photography, both equally valid.

For people who favor JPEG, the creative moment is when you press the shutter release, and they would rather be out shooting more images than slaving in a darkroom or in front of a computer doing post-processing. This was Henri Cartier-Bresson’s philosophy — he was notoriously ignorant of the details of photographic printing, preferring to rely on a trusted master printmaker. This group also includes professionals like wedding photographers or photojournalists for whom the productivity of a streamlined workflow is an economic necessity (even though the overhead of a RAW workflow diminishes with the right software, it is still there).

Advocates of RAW tend to be perfectionists, almost to the point of becoming image control freaks. In the age of film, they would spend long hours in the darkroom getting their prints just right. This is the approach of Ansel Adams, who used every trick in the book (he invented quite a few of them, like the Zone System) to obtain the creative results he wanted. In his later days, he reprinted many of his most famous photographs in ways that made them darker and filled with foreboding. For RAW aficionados, the RAW file is the negative, and the finished output file, which could well be a JPEG file, the equivalent of a print.

Implicit is the assumption that the RAW files are pristine and have not been tampered with, unlike JPEGs that had post-processing such as white balance or Bayer interpolation applied to them, and certainly no lossy compression. This is why the debate can get emotional when a controversy erupts, such as whether a specific camera’s RAW format is lossless or not.

The new Nikon D70’s predecessor, the D100, had the option of using uncompressed or compressed NEFs. Uncompressed NEFs were about 10MB in size, compressed NEF between 4.5MB and 6MB. In comparison, the Canon 10D lossless CRW format images are around 6MB to 6.5MB in size. In practice, compressed NEFs were not an option as they were simply too slow (the camera would lock up for 20 seconds or so while compressing).

The D70 only offers compressed NEFs as an option, but mercifully they have improved the performance. Ken Rockwell asserts D70 compressed NEFs are lossless, while Thom Hogan claims:

Leaving off Uncompressed NEF is potentially significant–we’ve been limited in our ability to post process highlight detail, since some of it is destroyed in compression.

To find out which one is correct, I read the C language source code for Dave Coffin’s excellent reverse-engineered, open-source RAW converter, dcraw, which supports the D70. The camera has a 12-bit analog to digital converter (ADC) that digitizes the analog signal coming out of the Sony ICX413AQ CCD sensor. In theory a 12-bit sensor should yield up to 2¹² = 4096 possible values, but the RAW conversion reduces these 4096 values into 683 by applying a quantization curve. These 683 values are then encoded using a variable number of bits (1 to 10) with a tree structure similar to the lossless Huffmann or Lempel-Ziv compression schemes used by programs like ZIP.

The decoding curve is embedded in the NEF file (and could thus be changed by a firmware upgrade without having to change NEF converters), I used a D70 NEF file made available by Uwe Steinmuller of Digital Outback Photo.

The quantization discards information by converting 12 bits’ worth of data into into log₂(683) = 9.4 bits’ worth of resolution. The dynamic range is unchanged. This is a fairly common technique – digital telephony encodes 12 bits’ worth of dynamic range in 8 bits using the so-called A-law and mu-law codecs. I modified the program to output the data for the decoding curve (Excel-compatible CSV format), and plotted the curve (PDF) using linear and log-log scales, along with a quadratic regression fit (courtesy of R). The curve resembles a gamma correction curve, linear for values up to 215, then quadratic.

In conclusion, Thom is right – there is some loss of data, mostly in the form of lowered resolution in the highlights.

Does it really matter? You could argue it does not, as most color spaces have gamma correction anyway, but highlights are precisely where digital sensors are weakest, and losing resolution there means less headroom for dynamic range compression in high-contrast scenes. Thom’s argument is that RAW mode may not be able to salvage clipped highlights, but truly lossless RAW could allow recovering detail from marginal highlights. I am not sure how practicable this would be as increasing contrast in the highlights will almost certainly yield noise and posterization. But then again, there are also emotional aspects to the lossless vs. lossy debate…

In any case, simply waving the problem away as “curve shaping” as Rockwell does is not a satisfactory answer. His argument that the cNEF compression gain is not all that high, just as with lossless ZIP compression, is risibly fallacious, and his patronizing tone out of place. Lossless compression entails modest compression ratios, but the converse is definitely not true: if I replace the file with a file that is half the size but all zeroes, I have a 2:1 compression ratio, but 100% data loss. Canon does manage to get the close to the same compression level using lossless compression, but Nikon’s compressed NEF format has the worst of both world – loss of data, without the high compression ratios of JPEG.

Update (2004-05-12):

Franck Bugnet mentioned this technical article by noted astrophotographer Christian Buil. In addition to the quantization I found, it seems that the D70 runs some kind of low-pass filter or median algorithm on the raw sensor data, at least for long exposures, and this is also done for the (not so) RAW format. Apparently, this was done to hide the higher dark current noise and hot pixels in the Nikon’s Sony-sourced CCD sensor compared to the Canon CMOS sensors on the 10D and Digital Rebel/300D, a questionable practice if true. It is not clear if this also applies to normal exposures. The article shows a work-around, but it is too cumbersome for normal usage.

Update (2005-02-15):

Some readers asked whether the loss of data reflected a flaw in dcraw rather than actual loss of data in the NEF itself. I had anticipated that question but never gotten around to publishing the conclusions of my research. Somebody has to vindicate the excellence of Dave Coffin’s software, so here goes.

Dcraw reads raw bits sequentially. All bits read are processed, there is no wastage there. It is conceivable, if highly unlikely, that Nikon would keep the low-order bits elsewhere in the file. If that were the case, however, those bits would still take up space somewhere in the file, even with lossless compression.

In the NEF file I used as a test case, dcraw starts processing the raw data sequentially beginning at an offset of 963,776 bytes from the beginning of the file, and reads in 5.15MB of RAW data, i.e. all the way to the end of the 6.07MB NEF file. The 941K before the offset correspond to the EXIF headers and other metadata, the processing curve parameters and the embedded JPEG (which is usually around 700K in size on a D70). There is no room left elsewhere in the file for the missing 2.5 bits by 6 million pixels (roughly 2MB) of missing low-order sensor data. Even if they were compressed using a LZW or equivalent algorithm the way the raw data is, and assuming a typical 50% compression ratio for nontrivial image data, that would still mean something like 1MB of data that is unaccounted for.

Nikon simply could not have tucked the missing data away anywhere else in the file. The only possible conclusion is that dcraw does indeed extract whatever image data is available in the file.

Update (2005-04-17):

In another disturbing development in Nikon’s RAW formats saga, it seems they are encrypting white balance information in the D2X and D50 NEF format. This is clearly designed to shut out third-party decoders like Adobe Camera RAW or Phase One Capture One, and a decision that is completely unjustifiable on either technical or quality grounds. Needless to say, these shenanigans on Nikon’s part do not inspire respect.

Generally speaking, Nikon’s software is usually somewhat crude and inefficient (just for the record, Canon’s is far worse). For starters, it does not leverage multi-threading or the AltiVec/SSE3 optimizations in modern CPUs. Nikon Scan displays scanned previews at a glacial pace on my dual 2GHz PowerMac G5, and on a modern multi-tasking operating system, there is no reason for the scanning hardware to pause interminably while the previous frame’s data is written to disk.

While Adobe’s promotion of the DNG format is partly self-serving, they do know a thing or two about image processing algorithms. Nikon’s software development kit (SDK) precludes them from implementing those algorithms instead of Nikon’s, and thus disallows Adobe Camera RAW’s advanced features like chromatic aberration or vignetting correction. Attempting to lock out alternative image-processing algorithms is more an admission of (justified) insecurity than anything else.

Another important consideration is the long-term accessibility of the RAW image data. Nikon will not support the D70 for ever — Canon has already discontinued support in its SDK for the RAW files produced by the 2001 vintage D30. I have thousands of photos taken with a D30, and the existence of third-party maintained decoders like Adobe Camera RAW, or yet better open-source ones like Dave Coffin’s is vital for the long-term viability of those images.

Update (2005-06-23):

The quantization applied to NEF files could conceivably be an artifact of the ADC. Paradoxically, most ADCs digitize a signal by using their opposite circuit, a digital to analog converter (DAC). DACs are much easier to build, so many ADCs combine a precision voltage comparator, a DAC and a counter. The counter increases steadily until the corresponding analog voltage matches the signal to digitize.

The quantization curve on the D70 NEF is simple enough that it could be implemented in hardware, by incrementing by 1 until 215, and then incrementing by the value of a counter afterwards. The resulting non-linear voltage ramp would iterate over at most 683 levels instead of a full 4096 before matching the input signal. The factor of nearly 8 speed-up means faster data capture times, and the D70 was clearly designed for speed. If the D70’s ADC (quite possibly one custom-designed for Nikon) is not linear, the quantization of the signal levels would not in itself be lossy as that is indeed the exact data returned by the sensor + ADC combination, but the effect observed by Christian Buil would still mean the D70 NEF format is lossy.