Fazal Majid's low-intensity blog

I am (very) slowly migrating away from the Mac to Ubuntu Linux as my main desktop operating system. The reasons why Apple has lost my confidence are:

• The execrable software quality of recent releases like Catalina (I plan on sticking with Mojave until I have migrated, however long that takes).
• Apple’s increasing locking down of macOS in ways antithetical to software freedom, e.g. SIP or the notarization requirements in Catalina with the denial of service implications
• The fact they no longer even pretend not to price-gouge on the Mac Pro. My days of buying their professional workstations every 5 years have come to an end after 15 years (PowerMac G5, Nehalem Mac Pro, 2013 Mac Pro)
• As the iPhone market is saturating, in their eagerness to come up with a replacement growth engine in “services”, they are pushing app developers towards the despicable and unacceptable subscription licensing model
• The butterfly keyboard fiasco exemplifies the contempt in which the company seems to hold its most loyal customers

On the plus side, ThinkPads have decent keyboards, unlike all Apple laptops since at least 2008, the LG Gram 17 is both lightweight, powerful and its huge screen is a boon to my tired eyes, and I am favorably impressed with the deep level of hardware integration offered by Ubuntu (e.g. displaying the logo and boot status in the UEFI stages of boot), even if I am not enamored of the software bloat or systemd.

One of the tasks in my migration checklist is to find a replacement for my TOTP two-factor authentication solution, which is currently the Lockdown app on iOS, iPadOS and MacOS (based on this recommendation, not to be confused with the Lockdown firewall/VPN app). I don’t trust Authy, they have a record of security failures introduced by their attempts to extend standard TOTP with their proprietary garbage, but I digress…

Thus I need to export Lockdown secrets. The iOS app can print or email a PDF with QR codes as a backup, but that’s not a very usable format for migration.

As I had to add a new TOTP secret to the app recently, that was the impetus to do this as a weekend project. I implemented a small utility called ldexport in Go to decode Lockdown for Mac’s internal file into either JSON or HTML format. Here are some simulated samples:

[
    {
        "Service": "Amazon",
        "Login": "amazon@example.com",
        "Created": "2015-11-18T19:53:34.969532012Z",
        "Modified": "2015-11-18T19:53:34.969532012Z",
        "URL": "otpauth://totp/Amazon%3Aamazon%40example.com?secret=M7IoBWqA2WuzYG27ju82XTWsflPEha3xBafMQ3i9CgwKgp6RdBGh\u0026issuer=Amazon",
        "Favorite": true,
        "Archived": false
    },
    {
        "Service": "PayPal",
        "Login": "ebay@example.com",
        "Created": "2019-11-25T08:46:57.253684043Z",
        "Modified": "2019-11-25T08:46:57.253684043Z",
        "URL": "otpauth://totp/PayPal:ebay@example.com?secret=3gB0VWJFkaYcVIiD\u0026issuer=PayPal",
        "Favorite": false,
        "Archived": false
    },
    {
        "Service": "Reddit",
        "Login": "johndoe",
        "Created": "2020-08-07T19:58:37.930042982+01:00",
        "Modified": "2020-08-07T19:58:37.930042982+01:00",
        "URL": "otpauth://totp/Reddit:johndoe?secret=nDTxDMI6bEgVpHWCViZjDFhXKH1bysRa\u0026issuer=Reddit",
        "Favorite": true,
        "Archived": false
    },
    {
        "Service": "GitHub",
        "Login": "",
        "Created": "2016-05-04T19:04:12.495128989+01:00",
        "Modified": "2017-04-04T06:33:10.641680002+01:00",
        "URL": "otpauth://totp/github.com/johndoe?issuer=GitHub\u0026secret=bXh5qmeTMzcatKKz",
        "Favorite": false,
        "Archived": false
    },
    {
        "Service": "Google",
        "Login": "johndoe@gmail.com",
        "Created": "2015-11-13T05:06:07.103500008Z",
        "Modified": "2015-11-13T05:06:07.103500008Z",
        "URL": "otpauth://totp/Google%3Ajohndoe%40gmail.com?secret=o5MvqdWDt7ZEHHSTuH6rCAUr4M6ozGQD\u0026issuer=Google",
        "Favorite": false,
        "Archived": false
    }
]

TL:DR: If you are using my Temboz feed reader, please update as soon as possible to version 4.4.0 or later.

This is somewhat related to the last security advisory for Temboz. Fields like article title, author or tags, or feed title or description that are supposed to be plain text (not even HTML) were not being sanitized. For XSS. The effect was demonstrated by this article in BoingBoing.

Once again I apologize for potentially exposing you to XSS attacks via malicious feeds, and I would recommend you subscribe to my RSS feed for it so you can get important announcements like this one in the future.

TL:DR: a very solid and trouble-free printer that makes excellent prints, including spectacular panoramics, for a significant fixed price.

Despite striving for the paperless office, and believing photographic prints are mostly a relic, I have a substantial collection of printers (as my daughter points out, it’s 5 printers per person in my household):

• HP OfficeJet Pro X551dw (extremely fast using PageWide fixed head technology, quite economical, huge paper tray capacity, very bulky)
• Epson EcoTank ET-16600 (prints and scans A3, very economical, also very bulky but not considering the print size)
• Brother QL-700, QL-820NWB, QL-1110NWB label printers (can make labels any length you want, the latter two are AirPrint compatible)
• Rollo label printer (will take practically any label stock you can throw at it)
• Fuji Instax SP100 instant photo printer (kids love them)
• Canon Selphy QX10 portable dye-sub sticker printer for my daughter
• two Dai-Nippon Printing DNP DS820A 8" dye-sub printers, one in storage
• An Epson Stylus Photo R2400 in storage
• a couple of Brother TZe label makers
• a Dymo LabelWriter 450 Twin Turbo (unreliable garbage, at least on Mac, avoid)
• A Selpic P1 on the way
• A Prusa i3 MK3S 3D printer (not sure if that counts)

The DNP DS820A replaced my Epson R2400 for two reasons:

• I print seldom enough that inks clogging in the nozzles was a big issue.
• The Epson is a behemoth that is very hard to find a place for, even before I downsized.

The DNP uses dye-sublimation technology to make its prints. You may have encountered one at a drugstore self-service photo kiosk, or at photo events like Macy’s Santa Claus portrait sessions. These printers are designed specifically for these two use cases, and are built like tanks with a steel chassis. Since most events typically gang two or even four printers to maximize throughput, they are also very compact, with a footprint barely larger than an A3 sheet of paper, mine is on a lower shelf in my IKEA FREDDE computer desk.

Until the advent of fine-art photo printers with 6 or more color pigment inks, dye-sub was the top-end digital photo printing technology, thanks to the continous tones it can generate, like photographic processes (e.g. Fuji Frontier or Noritsu QSS digital minilabs, or large-format laser enlargers like the Cymbolic Sciences LightJet or Durst Lambda/Theta). Dye-subs have all but disappeared from the consumer market, however, apart from some Canon Selphy compact printers, and are now largely reserved for professional applications, with a price to match. The DNP DS820A used to cost $1100. They lowered the price to under $1000 a few years ago, but cheaped out by removing the print-catching basket that used to be included in the older package.

You pop off the front panel and install a roll of paper and a reel of dye ribbons in a tray above the paper inside the printer, then pop the front back in. Nothing protrudes and the media is protected from dust, which is really nice. There are two different sizes of media, 8x10 (130 prints) and 8x12 (110 prints). The size is mostly relevant for the dye ribbons that have CMY sections sized in increments of 10 or 12 inches, but a surprising consequence of this is that you cannot switch from 8x10 to 8x12 and vice versa (you can make smaller divisions and the printer will trim them to size using its built-in cutter). The cost per print is about $0.65 for 8x10, $0.80 for 8x12, $1.30 if you get the premium metallic paper. Since the paper and ribbon is consumed no matter the coverage, it’s a constant, unlike the variable costs of an inkjet printer.

The print quality is excellent, as can be expected, as is the color calibration out of the box. It may not quite have the tonal subtlety of an Epson, but there is no visible pixellation. Furthermore, the prints get a clear protective laminate, which makes them smudge-proof and very tough. You can even choose one of four different finishes applied by a roller so no media change required: glossy, luster, matte and fine matte.

One of the marquee features of the DS820A and its little 6" brother the DS620A is the ability to make panoramic prints. Each print is made by combining multiple pages together, with about 2" of overlap wastage, so if your printer is loaded with 8x12 media you can make 8x22 or 8x32 prints, with 8x10 media you can make 8x18 or 8x26. The 8x32 panoramic prints are absolutely spectacular, although finding a suitable frame for them is not a trivial undertaking, that not being a standard print size.

Unfortunately this functionality is not built into the printer driver, but you must use the DNP Hot Folder utility, and while it is available for both Mac and Windows, only the Windows version can make panoramic prints. DNP Hot Folder is meant to use for events where a single PC or server controls multiple printers. You drop the files into a directory per print size (hence the name “Hot Folder”) and the software will automatically print it on the next available printer loaded with the right media. Since the printers run in parallel, even if the print speed is not incredibly fast (about 30 to 60 seconds per print), aggregate throughput is sufficient for a busy event. I have mine on a USB switch (the printer has no network connectivity) to share it between my Mac and my gaming PC.

TL:DR: If you are using my Temboz feed reader with Python 3, please update as soon as possible to version 4.0 or later.

This is because Temboz depends on feedparser, Kurt McKee’s (originally Mark Pilgrim’s) ultra-liberal feed parsing library for Python. One of its responsibilities is to sanitize feed content to strip out potentially dangerous HTML like <script> tags. Unfortunately, I only just realized that on Python 3, due to the absence of the sgmllib module in Python 3 that used to ship with Python 2, feedparser will silently fail and not sanitize the HTML instead of failing safe, e.g. throwing a NotImplementedError. Since this is such a fundamentally flawed approach, I decided no longer to trust feedparser withis responsibility and assign it to Mozilla’s bleach instead. Furthermore, Temboz will now perform a sanity check at startup and refuse to start if <script> tags are not being filtered.

I apologize for potentially exposing you to XSS attacks via malicious feeds. Unfortunately I have no way to reach out to all those who installed Temboz. If you are installing Temboz, I would recommend you subscribe to my RSS feed for it so you can get important announcements like this one in the future.

For a long time I used a service called Shoeboxed to scan and organize my credit card receipts. Basically stuff your receipts in a US prepaid envelope, drop it in the mail, and they scan, OCR and shred them, as well as analyzing the text to extract the information. Unfortunately, since I moved to the UK the service leaves to be desired, and the price has also gone up over time.

I have a couple of Fujitsu ScanSnap document scanners, a S1500M, which is no longer supported on macOS Mojave (but fortunately is by the third-party app ExactScan), and an iX100 which still is supported, as well as a SV600 which is utterly unsuited to dealing with crumpled receipts. The new, dumbed-down ScanSnap Home app that ships with ScanSnaps has a receipt mode. Since I never use the iX100 to scan documents at home given I have a S1500M (it’s a handheld battery=powered simplex scanner that’s mostly intended for mobile use), I dedicated it and ScanSnap Home to scanning receipts.

The basic functionality of scanning, deskewing, OCR-ing and extracting date, amount, vendor and so on mostly works, but otherwise ScanSnap Receipts is an ergonomic disaster. For starters, it never recognizes the currency correctly and always identifies my transactions as being in dollars rather than pounds. Secondly, it inexplicably lacks the ability to batch-edit receipts, e.g. select the date range for my last trip to France and change all of them from dollars to euros. You need to edit them one by one, which is as incredibly tedious as you can imagine.

After a few weeks of this, I decided to take matters in my own hand. It turns out ScanSnap Home uses Core Data backed by an underlying SQLite database. SQLite is the world’s most widely deployed database (every single Android and iOS smartphone includes it, for starters), but the Core Data object-relational mapper above it does a terrific job of obfuscating it and reducing its performance. Nonetheless, after a little bit of digging, I wrote the following script to automate the most repetitive operations:

1. Stop all ScanSnap auxiliary processes, as they have the DB opened even if you quit the app
2. Set the currency for all transactions tagged as Unchecked (ScanSnap Home does this by default) to GBP
3. Normalize all unchecked Waitrose vendor names to Waitrose
4. Same for Tesco
5. Rename M&S to Marks & Spencer
6. Fixes for Superdrug and Sainsbury’s
7. Set the amount of receipts tagged as duplicates to zero
8. Attempt to fix little-endian European format DD/MM/YYYY dates parsed as middle-endian US format MM/DD/YYYY
9. Attempt to fix Euro format DD/MM/YY dates parsed as US format YY/MM/DD
10. Restart ScanSnap Home

The timestamps in the database are in a strange format that seems to be the number of seconds since an epoch of 2001-01-01T00:00:00 UTC. I hope no one tries to scan receipts older than that… You can convert to UNIX timestamps by adding 978307200 and from there to SQLite’s Julian Day format.

One major annoyance thing this script attempts to fix is dates. Because date formats are ambiguous (is 11/3/20 March 11 2020, or November 3 2020 or perversely March 20 2011?) and point-of-sale vendors are neither ISO 8601 nor even Y2100 compliant, parsing dates is a minefield. My assumption is that receipts will be scanned in a reasonably timely manner, and if there is ambiguity, the closest date should win.

#!/bin/sh
pkill -9 -f ScanSnap

sqlite3 "$HOME/Library/Application Support/PFU/ScanSnap Home/Managed/ScanSnapHome.sqlite" << EOF

.mode lines

UPDATE zcontent
SET zcurrencysign=(SELECT z_pk FROM zcurrencysign WHERE zvalue='GBP')
WHERE zdoctype=4 AND z_pk IN (
  SELECT z_4contents
  FROM z_4labels
  JOIN zlabel ON z_15labels=zlabel.z_pk
  WHERE zlabel.zname='Unchecked'
);

UPDATE zcontent
SET zvendor=(SELECT z_pk FROM zvendor WHERE zvalue='Waitrose')
WHERE zdoctype=4 AND z_pk IN (
  SELECT z_4contents
  FROM z_4labels
  JOIN zlabel ON z_15labels=zlabel.z_pk
  WHERE zlabel.zname='Unchecked'
) AND zvendor IN (
  SELECT z_pk FROM zvendor
  WHERE zvalue<>'Waitrose' AND zvalue LIKE '%waitrose%'
);

UPDATE zcontent
SET zvendor=(SELECT z_pk FROM zvendor WHERE zvalue='Tesco')
WHERE zdoctype=4 AND z_pk IN (
  SELECT z_4contents
  FROM z_4labels
  JOIN zlabel ON z_15labels=zlabel.z_pk
  WHERE zlabel.zname='Unchecked'
) AND zvendor IN (
  SELECT z_pk FROM zvendor
  WHERE zvalue<>'Tesco' AND zvalue LIKE '%tesco%'
);

UPDATE zcontent
SET zvendor=(SELECT z_pk FROM zvendor WHERE zvalue='Marks & Spencer')
WHERE zdoctype=4 AND z_pk IN (
  SELECT z_4contents
  FROM z_4labels
  JOIN zlabel ON z_15labels=zlabel.z_pk
  WHERE zlabel.zname='Unchecked'
) AND zvendor IN (
  SELECT z_pk FROM zvendor
  WHERE zvalue LIKE '%M&S%'
);

UPDATE zcontent
SET zvendor=(SELECT z_pk FROM zvendor WHERE zvalue='Superdrug')
WHERE zdoctype=4 AND z_pk IN (
  SELECT z_4contents
  FROM z_4labels
  JOIN zlabel ON z_15labels=zlabel.z_pk
  WHERE zlabel.zname='Unchecked'
) AND zvendor IN (
  SELECT z_pk FROM zvendor
  WHERE zvalue<>'Superdrug' AND zvalue LIKE '%superdrug%'
);

UPDATE zcontent
SET zvendor=(SELECT z_pk FROM zvendor WHERE zvalue='Sainsbury''s')
WHERE zdoctype=4 AND z_pk IN (
  SELECT z_4contents
  FROM z_4labels
  JOIN zlabel ON z_15labels=zlabel.z_pk
  WHERE zlabel.zname='Unchecked'
) AND zvendor IN (
  SELECT z_pk FROM zvendor
  WHERE zvalue<>'Sainsbury''s' AND zvalue LIKE '%Sainsbury%'
);

UPDATE zcontent
SET zamount=0.0
WHERE zdoctype=4 AND z_pk IN (
  SELECT z_4contents
  FROM z_4labels
  JOIN zlabel ON z_15labels=zlabel.z_pk
  WHERE zlabel.zname='Duplicate'
);

UPDATE zcontent
SET zreceiptdate = strftime('%s', 
  strftime('%Y-%d-%m', zreceiptdate+978307200, 'unixepoch', 'localtime'),
  'utc'
)-978307200
WHERE zdoctype=4 AND z_pk IN (
  SELECT z_4contents
  FROM z_4labels
  JOIN zlabel ON z_15labels=zlabel.z_pk
  WHERE zlabel.zname='Unchecked'
)
AND zreceiptdate IS NOT NULL
AND strftime('%s', 
  strftime('%Y-%d-%m', zreceiptdate+978307200, 'unixepoch', 'localtime'),
  'utc'
)-978307200
BETWEEN zreceiptdate AND strftime('%s', 'now')-978307200;

UPDATE zcontent
SET zreceiptdate = strftime('%s',
  strftime('20%d-%m-', zreceiptdate+978307200,
           'unixepoch', 'localtime') ||
  substr(strftime('%Y', zreceiptdate+978307200,
         'unixepoch', 'localtime'), 3),
  'utc'
)-978307200
WHERE zdoctype=4 AND z_pk IN (
  SELECT z_4contents
  FROM z_4labels
  JOIN zlabel ON z_15labels=zlabel.z_pk
  WHERE zlabel.zname='Unchecked'
)
AND zreceiptdate IS NOT NULL
AND strftime('%s',
  strftime('20%d-%m-', zreceiptdate+978307200,
           'unixepoch', 'localtime') ||
  substr(strftime('%Y', zreceiptdate+978307200,
         'unixepoch', 'localtime'), 3),
  'utc'
)-978307200
BETWEEN zreceiptdate AND strftime('%s', 'now')-978307200;

EOF

open /Applications/ScanSnapHomeMain.app

Note that the SQLite database is not used only by expenses (ZCONTENT.ZDOCTYPE=4) but also to store a summary of all documents scanned. Also, the ZCONTENT table has a column ZUNCHECKED that is not what you would expect, it is a constant 1 even if you remove the Unchecked tag from the transaction.

Now, all the usual disclaimers apply, modifying the database directly is not something supported by the app developer, and could have unintended consequences. If you use this script (or more likely modify it for your needs), I disclaim responsibility for any damages or data loss this may cause.

Update (2020-10-01):

Added:

• setting duplicate receipt amounts to 0
• fixing DD/MM/YYYY dates misparsed as MM/DD/YYYY
• fixing DD/MM/YY dates misparsed as YY/MM/DD

I have been shooting stitched panoramas for almost 20 years. I have used manual panorama heads like the Kaidan Kiwi+ and more recently the pocketPANO Compact, robotic heads like the Gigapan EPIC 100, and four successive generations of the Ricoh Theta (Theta, Theta S, Theta V, Theta Z1).

Setting up and iterating a manual head is incredibly tedious. The Gigapan makes it less so, specially when using long lenses (my standard setup is a Leica M typ 240 or M10 with a 90mm f/2 Apo-Summicron-M ASPH). The Theta series was a major breakthrough in that it could produce nearly seamless 360° panoramas with no motion artifacts or ghosting. The Z1 with its large 1″ sensor finally yields image quality that I am happy with.

The viewing situation has also improved. In the early days you needed Java applets or dubious plug-ins. Nowadays, it can all be done in HTML5 with the aid of JavaScript libraries like Panellum. The user experience is still one of scrolling an image through a rectangular viewport in browser window. The experience on mobile is a bit better because it can use the accelerometer so you scroll by panning with your phone or tablet. It’s still not a fully immersive experience.

This Friday Facebook announced a price drop for its Oculus Go VR headset, the entry-level 32GB model being at a near-impulse purchase price of $150, and of course I yielded to the impulse. I had bought the original Oculus Rift to get a sense of what the potential of VR was, but tethered to a beefy PC, it made for impressive demos but not much more.

The Oculus Go changes this completely because it is standalone (it has the guts of a midrange smartphone circa 2018) and affordable. One of the ways they kept costs down is by removing motion tracking: it can detect angular motions of your head, but not when you are walking around, but for purposes of viewing 360° panoramic stills and videos, that is not required.

One of my concerns was how deeply it would be tied to the Facebook privacy-mangling machine. My New Year’s resolution for 2019 was to delete my FB account (my 2020 resolution was to switch all my digital camera clocks to UTC and never again bother with the abomination that is Daylight Saving Time)—underpromise and overdeliver, that’s my motto… Any requirement to have a FB account would be a total deal-breaker for me.

The second concern was how much of a hassle it would be to set up and use with my own photos. Camera-makers are not known for outstanding software and Ricoh is no exception. There is an Oculus third-party app for Theta cameras, but it hasn’t been updated in ages and only lists Theta S compatibility.

I was pleasantly surprised at how smoothly it went. You can avoid the FB account by using an Oculus account (I used mine from the Rift), and no additional apps are required. Just install the Android File Transfer utility if you are on a Mac, copy the files to the headset’s Pictures directory. I would recommend using subfolders because the built-in Gallery app is not smart about caching thumbnails and is very slow at regenerating the view if there are more than about 20 images or so in a folder.

The image quality is not exceptional. Mike Abrash, who worked on the ground-breaking 3D game Quake, and is now Chief Scientist at Oculus, says fully immersive VR requires resolution halfway between 4K and 8K in each eye (vs. 2.5K shared for both eyes on the Go), and is at least a decade away. The immersive nature of the Go does provide that elusive Wow! factor, however, and more than makes up for its designed-to-a-budget shortcomings. The 2560×1440 display with an apparent field of view of 100° yields 3.7MP in the FOV but spherical trigonometry calculations reveal the entire 360° sphere would require a 26MP image to cover it entirely, which is slightly more than the 23MP images the Theta Z1 delivers. Fully immersive VR requires very high resolutions!

It even handles video transparently (you do have to convert Theta videos from the native format to equirectangular projection video with Ricoh’s app, which is excruciatingly slow). Keep in mind that video sizes are large, and with a 32GB model, there are limits to how much you can store on the device. If you plan to view immersive videos, the 64GB model is highly recommended.

The Oculus Go also has a “Cast” feature that will stream what the person wearing the headset is seeing to the phone it is paired with. You can have a friend wear the headset and narrate what they are seeing, I tried this with my architect mother-in-law as I was showing her the sights in Jerusalem, much to her delight (her master’s thesis at SOAS was on the Dome of the Rock). The Go has a unique sound projector developed by Oculus that means the user doesn’t have to wear earbuds, and can hear you speak. I would recommend you change the default display sleep time from the ridiculously short 15 seconds to 3 to 5 minutes, so you can swap the headeet without losing the cast session or resetting the app. Sadly, the battery life is nothing to write home about. I would guesstimate it at 1 to 2 hours, tops.

I still need to figure how to share my 360° VR photos using WebVR so other people can view them from their own Oculus Go (or other headsets).

One essential accessory for the Z1 or another similar 360° camera is a selfie stick or similar implement, otherwise your hands will appear prominently in the final panorama. Ricoh sells three models.

The TM-1 is a very well designed tripod (rumored to be made by Velbon) with a magnetic quick-release mount. It’s easy to deploy with one click, unlike a conventional tripod, and fully extended the camera is at eye height for a natural perspective.

The TM-3 is a short telescopic stick. It’s long enough that your hands no longer appear in the picture but low-profile enoough that the TM-3 itself is invisible. It is well-made, unlike most generic Chinese selfie sticks, unlocks and locks with a simple twist, and the TS-2 case for the Z1 has an opening at the bottom so you don’t need to detach it before putting the camera back in its case, a nice touch.

The TM-2 is a longer version of the TM-3 with an unnecessary swivel head, I haven’t tried it but the swivel head would defeat the invisible factor.

Most travelers pack toiletry bags, known in the US as Dopp kits, to organize their toiletries and prevent them from leaking into their luggage. We need much the same to hold the vast assortment of chargers, cables, dongles and other technological paraphernalia required to function in this day and age. Sadly, the state of quality design in tech dopp kits is sorely lacking at the moment.

My daily loadout includes:

• 12″ MacBook (2015 model)
• 29W Apple USB-C charger
• 2m Apple USB-C charging cable (the third version is the one you want)
• USB-C to Lightning cable
• USB-A to Lightning cable
• 10,000 mAh Xiaomi battery pack
• Etymotic ER-4SR in-ear monitors

When traveling I add:

• Apple USB Type-C Digital AV Multiport Adapter (the latest version with 4Kp60 and HDR support)
• Onkyo DP-X1 high-resolution audio player
• 4-port Anker USB charger
• 3 USB-A to Lightning cables (one for my iPhone, one for my iPad, one for my wife). The USB-A connectors add a surprising amount of bulk, I wish Anker would make a charger with some sort of built-in octopus cable and cable management.
• 2 USB-A to micro-USB cables
• 20,000 mAh Mophie Powerstation XXL USB-C Power Delivery battery pack (can be used to charge the MacBook or iPad Pro or expand their battery life). Much lighter and convenient than the Anker 26,800 mAh lead ingot it replaced.
• Ricoh Theta Z1 360° panoramic camera and TM-3 stick
• Novoflex Mikrostativ mini-tripod
• spare batteries for whatever camera I packed
• Watson travel camera battery charger and whatever interchangeable battery plates are needed
• Samsung T3 256GB USB SSD
• a 18650 flashlight (BLF FW3A or Zebralight SC600FcIV+) and Olight USB-powered 18650 battery charger

Muji hanging toiletry kit

I used this around 2008 when I got the first-generation MacBook Air, to hold its charger and the Sanho Hyperdrive, an early battery pack that had a MagSafe connector until Apple sicced their lawyers at them. It has room for the two and the video dongles, not much else.

Waterfield Design gear pouches

Waterfield Designs was one of the first companies to make iPod cases, and they later added a larger pouch to accommodate an iPod and a bunch of other accessories. They have padded neoprene pockets and are very well made, but the padding adds a tremenoud amount of bulks, and they are also quite large.

Unfortunately they seem to have discontinued these pouches and replaced them with new ones in waxed canvas and leather that might arguably look a little better, but are less practical.

Bond Travel Gear EDC organizer Escapade gear pouch

On paper this is an interesting option. Unfortunately it is quite stiff, which makes it hard to pack, and overengineered with too many small pockets that reduce the effective carrying capacity because there is too much stitching, webbing, zippers, padding and other organizational overhead for the actual contents.

Eagle Creek Etools Organizer Pro

This bag is quite large, it can hold a 10.5″ iPad Pro. The fabric is thin and pliant, and the pockets large. This is a much better option for me as I can put larger items like battery packs, calculators, music players and so on. If it were ever slightly larger, it would also be a great bag to carry necessities on a flight so you don’t have to rummage for them in the overhead bin.

Update (2019-10-17):

It seems this pouch has been discontinued, but it is still readily available as new-old-stock with the usual retailers.

Triple Aught Design OP10

This tacticool pouch replete with mil-spec MOLLE attachments is better designed than the Bond, but still too small and with too many pockets and loops to maximize utility. It’s best used for those who want to carry an assortment of long thin objects like knives, tools, flashlights or cables. I would guesstimate the number of mall ninjas to operators owning this pouch is 10:1.

Incase Nylon accessory organizer

This organizer is recommended by The Wirecutter in their 2019 roundup. The fabric is thin and more flexible than overengineered ones in Cordura or ballistic nylon, which helps keep bulk down and increases the pockets' carrying capacity. The assortment of pockets is very well designed, at least for my needs, and it’s near ideal for a flight (it won’t hold an iPad, though).

Conclusions

The Eagle Creek is the best option. It doesn’t add too much weight and maximizes carrying capacity to space. No one has yet solved the problem of organizing cables so they don’t devolve into a gordian knot, however.

OpenSolaris and Solaris 11 use Apple’s Open-Source Bonjour server mdnsd rather than the Avahi daemon more common  in the Linux world. Unfortunately the OpenSolaris folks never got around to advertising NFS via mdnsd.

obinur ~>dns-sd -R obinur _nfs._tcp. . 2049 path=/export/home/majid
Registering Service obinur._nfs._tcp. port 2049
 path=/export/home/majid
Got a reply for obinur._nfs._tcp.local.: Name now registered and active

Incomplete,never finished, and probably completely out of date after the aftermath of coronavirus. Check out my foodie map instead (although most likely just as out of date).

1. The caramelized onion and sesame baguette at Noe Valley Bakery
2. The chocolate-cherry breakfast bread at Noe Valley Bakery
3. The combination seafood salad at Swan’s Oyster Bar
4. The clam chowder at Ferry Plaza Seafood
5. The mushroom pizza at Delarosa
6. The salted caramel ice cream at Bi-Rite Creamery
7. The “crunchy sticks” (sesame-poppyseed-cheese twists) at Esther’s German Bakery (available at Rainbow Grocery and on Thursdays at the Crocker Galleria)
8. The El Rey chocolate toffee semifreddo at Bix (seasonal)
9. The pistachio macaroons at Boulette’s Larder
10. The burnt caramel and chocolate covered hazelnuts at Michael Recchiuti
11. The chocolate feuilletine cake at Miette
12. The belly buster burger at Mo’s Grille
13. The hazelnut hot chocolate at Christopher Elbow
14. The princess cake at Schubert’s Bakery
15. The ricciarellis (soft bitter-almond macaroons) at Arizmendi
16. The Charlemagne chocolate feuilletine cake at Charles Chocolates
17. The chocolate hazelnut tarts at Tartine
18. The croissant at Tartine
19. The caramelized hazelnut financier at Craftsman and Wolves
20. The bordeaux cherry ice cream at Swensen’s, in a cone and dipped in chocolate (the ice cream itself, not the cone)
21. The flatbread at Universal Cafe
22. The Umami truffle burger
23. Captain Mike’s smoked tuna (Ferry Plaza farmer’s market)
24. The mushroom empañadas at El Porteño
25. The Tcho chocolate liquid nitrogen ice cream at Smitten
26. The macarons at Chantal Guillon
27. Pine nut Bacetti (Howard & 9th)
28. The Atomica pizza at Gialina’s
29. The salted hazelnut and chocolate shortbread at Batter Bakery
30. The chocolate velvet cupcake at Kara’s Cupcakes
31. The lobster roll at Woodhouse Fish Co.
32. The gianduja and pistachio at Coletta Gelato
33. The croissants and pains au chocolat at Arsicault (Arguello & Clement)
34. The lasagna served bubbling hot from the oven at Pazzia
35. The wild mushroom benedict at Mission Beach Cafe
36. The farm egg ravioli at Cotogna
37. The cioppino (spicy tomato-fish stew) at The Tadich Grill
38. The sada (plain) dosa at Dosa Fillmore
39. The French and Fries burger at Roam Artisan Burgers
40. The lasagna at Trattoria di Vittorio

I have been meaning to move away from Wordpress to a static site generator for a very long time, due to:

• The slowness of WP, since every page request makes multiple database calls due to the spaghetti code nature of WP and its plugin architecture. Caching can help somewhat, but it has brittle edge cases.
• Its record of security holes. I mitigated this somewhat by isolating PHP as much as possible.
• It is almost impossible to follow front-end optimization best-practices like minimizing the number of CSS and JS files because each WP plugin has its own

My original plan was to go with Acrylamid, but about a year ago I started experimenting with Hugo. Hugo is blazing fast because it is implemented in Go rather than a slow language like Python or Ruby, and this is game-changing. Nonetheless, it took me over a year to migrate. This post is about the issues I encountered and the workflow I adopted to make it work.

Wordpress content migration

There is a migration tool, but it is far from perfect despite the author’s best efforts, mostly because of the baroque nature of Wordpress itself when combined with plugins and an old site that used several generations of image gallery technology.

Unfortunately, that required rewriting many posts, specially those with photos or embedded code.

Photo galleries

Hugo does not (yet) support image galleries natively. I started looking at the HugoPhotoSwipe project, but got frustrated by bugs in its home-grown YAML parser that broke round-trip editing, and made it very difficult to get galleries with text before and after the gallery proper. The Python-based smartcrop for thumbnails is also excruciatingly slow.

I wrote hugopix to address this. It uses a simpler one-way index file generation method, and the much faster Go smartcrop implementation by Artyom Pervukhin.

Broken asset references

Posts with photo galleries were particularly broken, due to WP’s insistence on replacing photos with links to image pages. I wrote a tool to help me find broken images and other assets, and organize them in a more rational way (e.g. not have PDFs or source code samples be put in static/images).

It also has a mode to identify unused assets, e.g. 1.5GB of images that no longer belong in the hugo tree as their galleries are moving elsewhere.

Password-protected galleries

I used to have galleries of family events on my site, until an incident where some Dutch forum started linking to one of my cousin’s wedding photos and making fun of her. At that point I put a pointed error message for that referrer and controlled access using WP’s protected feature. That said, private family photos do not belong on a public blog and I have other dedicated password-protected galleries with Lightroom integration that make more sense for that use case, so I just removed them from the blog, shaving off 1.5GB of disk in the bargain.

Search

There are systems that can provide search without any server component, e.g. the JavaScript-based search in Sphinx, and I looked at some of the options referenced by the Hugo documentation like the Bleve-based hugoidx but the poor documentation gave me pause, and I’d rather not run Node.js on my server as needed by hugo-lunr.

Having recently implemented full-text search in Temboz using SQLite’s FTS5 extension, I felt more comfortable building my own search server in Go. Because Hugo and fts5index share the same Go template language, this makes a seamless integration in the site’s navigation and page structure easy.

Theme

There is no avoiding this, moving to a new blogging system requires a rewrite of a new theme if you do not want to go with a canned theme. Fortunately, Hugo’s theme system is sane, unlike Wordpress', because it does not have to rely on callbacks and hooks as much as with WP plugins.

RSS GUIDs and permalinks

One pet peeve of mine is when sites change platform with new GUIDs or permalinks in the RSS feeds, causing a flood of old-new articles to appear in my feed reader. Since I believe in showing respect to my readers, I had to avoid this at all costs, and also put in place redirects as needed to avoid 404s for the few pages that did change permalinks (mostly image galleries).

Doing so required copying the embedded RSS template and changing:

<guid>{{ .Permalink }}</guid>

to:

<guid isPermaLink="false">{{ .Params.rss_guid | default .Permalink }}</guid>

The next step was to add rss_guid to the front matter of the last 10 articles in my legacy RSS feed.

I use the Kolor AutoPano Giga panorama-stitching software, recently acquired by GoPro, but I have yet to produce a gigapixel panorama like those they pioneered. This brings up an interesting question: given a camera and lens, what would the pixel size of the largest 360° stitched panorama be?

Wikipedia to the rescue: using the formula for the solid angle of a pyramid, the full panorama size of a camera with m megapixels on a sensor of a x b using a focal length of f would be:

m * π / arctan(ab / 2f / sqrt(4f² + a² + b²))

For single-strip panoramas of height h (usually a or b), the formula would be:

m * π * h / 2f / arctan(ab / 2f / sqrt(4f² + a² + b²))

(this applies only to rectilinear lenses, not fisheyes or other exotics).

Here is a little JavaScript calculator to apply the formula (defaults are for the Sony RX1RII, the highest resolution camera I own):

The only way I can break through the gigapixel barrier with a prime lens is using my 24MP APS-C Fuji X-T2 with a 90mm lens.

Update (2020-01-21):

Now I could reach 10GP with my Nikon Z7 and the Nikkor 500mm f/5.6 PF.

TL:DR—avoid scanners with Contact Image Sensors if you care at all about color fidelity.

After my abortive trial of the Colortrac SmartLF Scan, I did a comparative test of scanning one of my daughter’s A3-sized drawings on a number of scanners I had handy.

I was shocked by the wide variance in the results, as was my wife. This is most obvious in the orange flower on the right.

I scanned a swatch of the orange using a Nix Pro Color Sensor (it’s the orange square in the upper right corner of each scan in the comparison above). When viewed on my freshly calibrated NEC PA302W SpectraView II monitor, the Epson V500 scan is closest, followed by the ScanSnap SV600.

The two scanners using Contact Image Sensor (CIS) technology yielded dismal results. CIS are used in low-end scanners, and they have the benefit of low power usage, which is why the only USB bus-powered scanners available are all CIS models. CIS sensors begat the CMOS sensors used by the vast majority of digital cameras today, superseding CCDs in that application, I would not have expected such a gap in quality.

The digital camera scan was also quite disappointing. I blame the poor quality of the LEDs in the IKEA KVART three-headed lamp I used (pro tip: avoid IKEA LEDs like the plague, they are uniformly horrendous).

I was pleasantly surprised by the excellent performance of the S1500M document scanner. It is meant to be used for scanning sheaves of documents, not artwork, but Fujitsu did not skimp and used a CCD sensor element, and it shows.

Pro tip: a piece of anti-reflective Museum Glass or equivalent can help with curled originals on the ScanSnap SV600. I got mine from scraps at a framing shop. I can’t see a trace of reflections on the scan, unlike on the copy stand.

Update (2018-10-14):

Even more of a pro tip: a Japanese company named Bird Electron makes a series of accessories for the ScanSnap line, including a dust cover for the SV600 and the hilariously Engrish-named PZ-BP600 Book Repressor, essentially a sheet of 3mm anti-reflection coated acrylic with convenient carry handles. They are readily available on eBay from Japanese sellers.

TL:DR summary

Pros:

• Scans very large documents
• Easy to use
• Packs away in a convenient carrying case

Cons:

• So-so color fidelity
• Hard to feed artwork straight
• Dust and debris can easily get on the platen, ruining scans
• Relatively expensive for home use

Review

One thing you do not lack for when your child enters preschool is artwork. They generate prodigious amounts of it, with gusto, and they are often large format pieces on 16×24″ paper (roughly ISO A2). The question is, what do you do with the torrent?

I decided I would scan them, then file them in Ito-Ya Art Profolios, and possibly make annual photobooks for the grandparents. This brings up the logistical challenge of digitizing such large pieces. Most flatbed scanners are limited to 8.5×14″ (US Legal) format. Some like the Epson Expression 11000XL and 12000XL can scan 11×17″ (A3), as can the Fujitsu ScanSnap SV600 book scanner, but that is not fully adequate either. One option would be to fold the artwork up, scan portions then stitch them together in AutoPanoGiga or Photoshop, but that would be extremely cumbersome, specially when you have to do a couple per day. I do not have access to a color copier at my office, and most of these are only A3 anyway.

I purchased a Kaiser RS2 XA copy stand (cheaper to get it direct from Europe on eBay than from the usual suspects like B&H) and got a local framing shop to cut me a scrap of anti-reflective Museum Glass. This goes up to 16×20″ for the price of a midrange flatbed scanner, but it is tricky to set up lights so they don’t induce reflections (no AR coating is perfect), perfectly aligning the camera with the baseboard plane is difficult (I had to shim it using a cut-up credit card), and this still doesn’t solve the problem of the truly large 16×24″ artwork (stands able to handle larger formats are extremely expensive and very bulky).

I then started looking at large-format scanners like those made by Contex or Océ. They are used by architecture firms to scan blueprints and the like, but they are also extremely large, and cost $3000-5000 for entry-level models, along with onerous DRM-encumbered software that requires license dongles and more often than not will not run on a Mac. They are also quite bulky, specially if you get the optional stands.

That is why I was pleasantly surprised to learn British company Colortrac makes a model called the SmartLF Scan! (I will henceforth omit the over-the-top exclamation mark). It is self-contained (can scan to internal memory or a USB stick, although it will also work with a computer over USB or Ethernet, Windows-only, unfortunately), available in 24″ or 36″ wide versions, is very compact compared to its peers, and is even supplied with a nifty custom-fitted wheeled hard case. The price of $2,000 ($2,500 for the 36″ version), while steep for home use, is well within the range of enthusiast photo equipment. I sold a few unused cameras to release funds for one.

Once unpacked, the scanner is surprisingly light. It is quite wide, obviously, to be able to ingest a 24″ wide document (see the CD jewel case in the photo above for scale). There is a LCD control panel and a serviceable keypad-based (not touch) UI. The power supply is of the obnoxious wall-wart type. I wish they used text rather than inscrutable icons in the UI—it is much more informative and usable to see a menu entry for 400dpi resolution rather than checkerboard icons with various pitches.

After selecting your settings (or saving them as defaults), you load paper by feeding it from the front, face up. It is quite hard to feed large-format paper straight, and this is compounded by the lack of guides. On the other hand it is hard to see how Colortrac could have fitted photocopier-style guide rails in such a compact design, and they would be likely to break.

The scanner is simplex, not duplex, unsurprisingly at that price point. The sensor is on top of the feed, which helps control dust and debris sticking to it, but when scanning painted artwork, there will inevitably be crumbs of paint that will detach and stick to the sensor platen. This manifests itself as long dark vertical lines spoiling subsequent scans, something I occasionally also see on my Fujitsu ScanSnap document scanner. Cleaning the Colortrac is way easier than on the ScanSnap, as unfolding rear legs and releasing front catches opens it wide, and a few passes with optical cleaning wipes (I use Zeiss’ single-use ones) will do the trick.

By the manufacturer’s own admission, the scanner is designed to scan technical drawings, not art. It uses a linear contact image sensor (CIS) like lower-end flatbed scanners and document scanners, unlike the higher-fidelity charge-coupled device (CCD) sensors used in higher-end graphics arts and photo scanners. The light source is a row of point light LEDs that casts relatively harsh shadows on the paper. They do make CCD scanners for graphics arts, but they start at $10,000… Contex makes an A2 flatbed CCD scanner, the HD iFlex, but it costs $6,700 (at Jet.com of all places), their iQ Quattro 2490 at $4,500 is the most viable step-up (it uses a CIS, but offers 16-bit color, AdobeRGB and beyond gamut, calibration and magnetic paper guides).

The scanner’s resolution is 600dpi. Scanning 16×24″ originals at that resolution yields a 138MP file that is nearly a gigabyte in size. The 400dpi setting yields a much more reasonable 200MB or so, and compressing them further using tiffcp with zip compression (not an option on the scanner) yields 130-140MB files.

Unfortunately, I ended up returning it. There was a 1cm scratch in the glass platen, which manifested itself as streaks. It takes quite a bit to scratch glass (I don’t think it was Lexan or similar), and I wasn’t scanning sandpaper, so it must have been a factory defect or a customer return. When I looked at the color fidelity of the scans, I was not inclined to order a replacement, and got a Fujitsu ScanSnap SV600 from Japan instead from an Amazon third-party reseller (25% savings over the US price, even if you usually forgo a US warranty on grey-market imports).

The Avery 22807 2-inch circular stickers are a good alternative to Moo, PSPrint et al when you need a small quantity of stickers in a hurry. Unfortunately Avery has not seen it fit to provide usable InDesign templates as they do with some of their other sticker SKUs, only Microsoft Word, which is needless to say inadequate. A search for “Avery 22807 Indesign template” yielded some, but they have issues with missing linked PDF files.

I reverse-engineered the Microsoft template to build one of my own, with dimensions (including the tricky almost-but-not-quite square grid spaced at 5/8″ horizontally but 7/12″ vertically) to simplify “Step and Repeat…”.

I have only tested this with my InDesign CS6, not sure if it will work with older versions.

Avery 22807 2-inch circular labels.indt

https://browser.geekbench.com/v4/cpu/853781

https://browser.geekbench.com/v4/cpu/853704

Big bezel

Better keyboard

Slipcover and USB-A to C adapter included

I never updated my home Mac Pro to El Capitan. To paraphrase Borges, each successive Apple OS release since Snow Leopard makes you long for the previous one. Unfortunately I have no choice but to run the latest OS X release on my work Macs as that is usually required to run the latest Xcode, itself required for the App Store.

I did not realize how bad El Capitan was until I upgraded my work iMac (27-inch 5K model) to Sierra last week. Previously, I would experience a mean time between crashes of around 3 days. I thought it was flaky hardware (the problems started from when I unboxed the computer), but couldn’t find the time to take it to the Genius Bar. I had also experienced the same problem with my old home 2009 Nehalem Mac Pro, which I had taken to the office, in fact that’s why I bought the iMac in the first place (and the first one I ordered had to go back because of defective pins in the RAM expansion slots). The Mac Pro had previously been rock-steady at home.

Since I upgraded to Sierra, I haven’t had a single crash. The only possible conclusion is that El Capitan bugs were to blame. The only thing unusual about this iMac is I upgraded the RAM from OWC, but the memory passes testing using Micromat’s TechTool.

I am not one to look at the Steve Jobs era with rosy-tinted glasses, OS X has never had the same level of stability as Solaris or even Linux, but Apple’s hardware and software quality has really gone to the dogs of late, something Lloyd Chambers dubs Apple Core Rot.

I am now starting to hedge my bets and am testing Ubuntu for my laptop computing needs, first by repurposing my 2008-vintage first-generation MacBook Air that is no longer supported by OS X anyway (works, but painfully slow) and soon with a shiny new HP Spectre on order.

I mentioned previously that I seldom shop for electronics on Amazon.com any more, preferring B&H Photo whenever possible. I now have another reason: avoiding counterfeit goods.

My company boardroom is in an electromagnetic war zone—dozens of competing WiFi access points combined with electronic interference from the US-101 highway just outside make WiFi reception tenuous at best, and unusable more often than not. To work around this, we set up a wired Ethernet switch, and since most of our staff use MacBook Airs, Apple USB Ethernet adapters purchased from Amazon. When I side-graded from my 15″ Retina MacBook Pro to a much more portable 12″ Retina MacBook, I wasn’t able to connect using the dongle, and the name of the device was interspersed with Chinese characters. At first I thought it was an issue with my Satechi USB-C hub, but I experienced the same problems via a genuine Apple USB-C multiport adapter as well.

Eventually I figured out the Ethernet dongles were counterfeit. The packaging, while very similar to Apple’s, was just a tiny bit off, like amateurish margins between the Apple logo and the edges of the card. On the dongles themselves, the side regulatory disclosures sticker was inset, not flush with the body of the adapter.

Counterfeiting is a major problem. By some accounts, one third of all Sandisk memory cards worldwide are counterfeits. In some cases like chargers or batteries, your equipment could be at risk, or even your very life. The counterfeit adapters we purchased from Amazon did not come from Amazon themselves but from a third-party merchant participating in the Amazon marketplace. To Amazon’s credit, we returned them for a prompt, no questions asked refund even though we bought them over six months ago, but it is hard to believe Amazon is unaware of the problem rather than willfully turning a blind eye to it.

My first reaction was to tell our Office Manager to make sure to buy only from Amazon rather than third-party merchants (pro tip: including “amazon” in your Amazon search terms will do that in most cases). Unfortunately, that may not be enough. Amazon has a “fulfilled by Amazon” program for merchants where you ship your goods to them, and they handle warehousing and fulfillment. These “fulfilled by Amazon” items are also more attractive to Prime members. One option Amazon offers is Stickerless, commingled inventory where the items you send are put into a common bin. Amazon still has the ability to trace the provenance of the item through its inventory management, but for purposes of order fulfillment they will be handled just like Amazon’s own stock. Some categories like groceries and beauty products are excluded, but electronics are not.

The implications are huge: even if the vendor is Amazon itself, you cannot be sure that the item is not counterfeit. All the more reason to buy only from trustworthy, single-vendor sites like B&H, even if shipping is a bit slower.

I use Dovecot as my mail server, with maildir format mailboxes. It is very easy to make the Sent folder be the same as the Inbox: just symlink the Sent messages folder’s cur subdirectory to be the same as the Maildir’s top-level new directory. This ensures any email placed in the Sent messages folder is magically a whisked off to the inbox.

starvald ~>ll Maildir/.Sent\ Messages/
total 85
drwxr-xr-x 5 majid engineers 11 Apr 20 13:01 ./
drwxr-xr-x 268 majid engineers 278 Apr 20 13:26 ../
drwxr-xr-x 2 majid engineers 2 Oct 14 2007 courierimapkeywords/
lrwxrwxrwx 1 majid engineers 6 Oct 30 2010 cur -> ../new/
-rw-r--r-- 1 majid engineers 33 Jun 15 2011 dovecot-keywords
-rw-r--r-- 1 majid engineers 54 Apr 20 13:01 dovecot-uidlist
-rw-r--r-- 1 majid engineers 400 Apr 20 08:38 dovecot.index
-rw-r--r-- 1 majid engineers 24K Apr 20 08:39 dovecot.index.cache
-rw-r--r-- 1 majid engineers 26K Apr 20 13:01 dovecot.index.log
drwxr-xr-x 2 majid engineers 2 Apr 19 09:43 new/
drwxr-xr-x 2 majid engineers 2 Apr 20 08:38 tmp/