Fazal Majid's low-intensity blog

In the bad (good?) old days when currency’s worth was established by the amount of gold or silver in coinage, kings would cut corners by debasing currency with lead, which is almost as dense as gold or silver. In the New World, counterfeiters debased gold coins with platinum, which was first smelted by pre-columbian civilizations. Needless to say, the fakes are now worth more than the originals.

The public was not fooled, however, and found ways to test coins for purity, including folkloric ones like biting a coin to see if it is made of malleable gold, rather than harder metals. People would then hoard pure gold coins, and try to rid themselves of debased coins at the earliest opportunity. This led to Gresham’s Law: bad money drives out good money in circulation.

After a year of using Amazon Web Services’ EC2 service at scale for my company (we moved to our own servers at the end of 2011), I conjecture there is a Gresham’s Law of Amazon EC2 instances – bad instances drive out good ones. Let me elaborate:

Amazon EC2 is a good way to launch a service for a startup, without incurring heavy capital expenditures when getting started and prior to securing funding. Unfortunately, EC2 is not a quality service. Instances are unreliable (we used over 80 instances at Amazon, and there was at least one instance failure a week, and sometimes up to 4). Amazon instances have poor disk I/O performance that makes them particularly unsuitable to hosting non-trivial databases (EBS is even worse, and notoriously unreliable).

Performance is also inconsistent—I routinely observed “runt” m1.large instances that performed half as well as the others. We experienced all sorts of failure modes, including disk corruptions, disks that would block forever without timing out, sporadic losses of network connectivity, and many more. Even more puzzling, I would get 50% to 70% failure rate on new instances that would not come up cleanly after being launched.

Some of this is probably due to the fact we use an uncommon OS, OpenSolaris, that is barely supported on EC2, but I suspect a big part of this is that Amazon uses low-end commodity parts, and does not proactively retire failed or flaky hardware from service. Instances that have the bad luck of being assigned to flaky hardware are more likely to fail or perform poorly, and thus more likely to be be destroyed, released and a new one reassigned in the same slot. The inevitable consequence of this is that new instances have a higher likelihood of being runts or otherwise defective than long-running ones.

One work-around is to spin up a large number of instances, test them, and destroy the poor-performing ones. AWS runts are usually correlated with slower CPU clock speeds, as older machines would be running older versions of the Xen hypervisor Amazon uses under the hood, have less cache, slower drives and so on. Iterating through virtual machines as if you are picking melons at a supermarket is a slow and painful job, however, and even their newer machines have their share of runts. We were trying to keep only machines with 2.6 or 2.66GHz processors, but more than 70% of the instances we were getting assigned were 2.2GHz runts, and it would usually take creating 5 or 6 instances on average to get a non-runt.

In the end, we migrated to our own facility in colo, because Amazon’s costs, reliability and performance were just not acceptable, and we had long passed the threshold beyond which it is cheaper to own than rent (I estimate it at $5,000 to $10,000 per month Amazon spend, depending on your workload). It is not as if other cloud providers are any better—before Amazon we had started on Joyent, which supports OpenSolaris natively, and their MTBF was in the order of 2 weeks, apparently because they replaced their original Sun hardware with substandard Dell servers and had issues with power management C-states in the Dell server BIOS.

The dirty secret of cloud services is that there is no reliable source of information on actual performance and reliability of cloud services. This brings out another economic concept, George Akerlof’s famous paper on the market for lemons. In a market where information asymmetry exists, the market will eventually collapse in the absence of guarantees. Until Amazon and others offer SLAs with teeth, you should remain skeptical about their ability to deliver on their promises.

One of the joys challenges of being a first-time parent is being exposed to a bewildering array of gadgets and equipment required to care for the baby, from baby car seats, strollers and diaper pails to 2-axis rocking robots (thanks Rohit!). There is an entire cottage industry of books like Baby Bargains that help you navigate through the confusing and sometimes questionable or outright unnecessary choices.

I have a Withings body weight scale that I really like and I was excited to learn they were going to release a networked video baby monitor. It took a while to get to market in the US, however, so in the interim I purchased a Philips Advent DECT digital baby monitor, which ended up unusable in practice, because its microphone sensitivity is so poor that you can barely hear anything. When the Withings baby monitor finally became available in the US, I immediately ordered it.

Withings is clearly taking design cues from Apple, from the lavishly designed packaging to the glossy white plastic RoundedRect aesthetic and the use of a magnetic clip to attach the baby monitor to the crib. The clip is serviceable, but the magnets are not quite strong enough to hold the unit firmly onto the crib. I would not trust it to keep the monitor from toppling when the baby grows and kicks at the crib. Fortunately they also include a flip-out tab on the base of the unit that can be inserted into a slit on the clip to prevent sliding, although it is not obvious and it took me a while before I discovered this key feature.

The wall wart is a generic black model with swappable AC prongs for international markets, and detracts from the overall package, but since the monitor has a micro-USB input, you can always use another standard AC to USB type A adapter like the iPhone’s, with a USB type A to micro-USB cable. A rechargeable battery is included, with 2 hours’ claimed life, I did not verify that spec.

The initial out of the box experience is good: you connect to the device from your iPhone or iPad using Bluetooth (no messing around with a USB cable as with the Withings scale), enter the WiFi settings in the Withbaby app, and then use WiFi to access the device afterwards. It is as streamlined an experience as you can expect without a keyboard on the unit. There is also an Ethernet jack (it is unclear whether it supports power over Ethernet), but my house was built in 1928 and is not wired upstairs where the baby lives.

Once you enter your credentials into the app, it connects to the monitor and shows you the video and sound. If you put it in the background, you have the option of monitoring audio. Withings will also send you alerts via push notifications if the temperature or humidity is excessive, or if it detects noise or motion. The default settings are way too twitchy, however, and you will find yourself disabling audio notifications as the deluge of alerts is just too much.

The device includes a night light with selectable color, a lullaby player, and the ability to speak to your baby, all controlled through the app. At the front you also have touch controls to turn some of these features on. This is actually a bad idea, as on two occasions I started the lullaby by accident as I was fumbling with it in a dark room, and woke up my baby as a result. Another design flaw is the pulsing blue night light when the unit is rebooting, the Airport Express like amber/green status LED in the back is quite sufficient. Frankly the only one of these features that is useful is the speaker, and the ability to stream from your music collection, such as Dr. Harvey Karp’s white noise selections would be preferable to the canned lullabies.

The video camera is advertised as having a 3 megapixel sensor. It has a wide-angle lens and you can “pan” using the usual iPhone or iPad gestures. The lens is a fixed-focus plastic one, and optical clarity is so-so at best, optimal focus seems to be at 50cm or so. One great feature is the monitor has a normal and night vision mode, similar to the one on some Sony HAD camcorders, with an IR illuminator that provides light for the night vision mode. This means you can watch your baby toss and turn in an otherwise pitch-black room.

You can use the baby monitor from outside your network, and it works fine, even over a 3G connection. Withings allows you up to 15 minutes per day, anything beyond that requires paying them $6 for each 100 minutes. Coming on top of an already expensive device, this seems like a naked money grab from anxious parents. (Updated 2012-09-29: remote monitoring is now free and unlimited).

When the unit works, it is absolutely great: good sound sensitivity and the video feature mostly works as advertised. Unfortunately it frequently does not function, and I find myself performing a hard reboot by removing the battery far more often than I would like. Among the pathologies:

• Once it falsely reported the unit was closed and thus video inaccessible
• Once the camera was in a frozen state, it took a power cycling to get the video moving again.
• Yesterday I could not connect at all, no matter how many times I rebooted my Airport Extreme, the monitor and my wife’s or my iPads. Some detective work using a packet sniffer showed the app was trying to connect to babyws.withings.net using HTTP, which is aliased to s11.withings.net, and that server was down. Some of the documentation suggests you can use the Bluetooth connection to access the monitor, but I was not able to figure out how to do this.

This brings me to a crucial point. The baby monitor is a safety device, and it is utterly unacceptable for its functioning to be dependent on a cloud service, which can and will be a single point of failure. It should use Bonjour or similar discovery methods to work on the LAN, and rely on Withings’ servers only when accessing it from outside the home LAN’s perimeter. I wonder if Withings’ eagerness to nickel-and-dime users by charging for outside monitoring led to this critical design flaw.

The bottom line is the Withings smart baby monitor is a very frustrating device, with its obvious potential marred by failures of execution. If it worked consistently, it would be a top-notch product worthy of its Apple inspiration and lofty price tag, but the general lack of reliability means I cannot recommend it until the bugs are ironed out. Consider it an alpha release at best.

Update (2012-09-17):

Here’s how to make the Withings not-so-smart baby monitor more usable:

• Remove the battery from the unit and hook up the micro-USB power adapter to a Belkin WeMo remote-controlled power switch. This allows you to power-cycle the baby monitor remotely from the same iPhone or iPad you are using the monitor software on.
• Hide the blue led with gaffer’s tape. This prevents the blue light on reboot from waking the baby. Unlike duct tape, gaffer’s tape can be removed without leaving glue residue, although the aesthetics of dark gray gaffer’s tape on the gleaming white unit are questionable at best.
• I haven’t tried covering up the touch controls with gaffer’s tape, which would eliminate the risk of triggering a jingle and waking the baby. The WeMo eliminates the need to enter the room and tinker with the baby monitor.

It’s quite sad to have to pay an extra $50 to work around buggy hardware and software, but it makes a big difference.

Update (2013-05-20):

The micro-USB connector failed and the baby monitor is now essentially a doorstop. Not surprising given how flimsy micro-USB is, compared to mini-USB, for insignificant space savings. Micro-USB was a Nokia design rammed through the USB-IF. In theory it has better insert-remove cycle life than mini-USB, but in practice I’ve never had mini-USB fail, whereas it is a frequent occurrence with micro-USB.

Update (2015-08-17):

USB-C is an improvement over micro-USB, hopefully some future version of the baby monitor will use it. Still nowhere near as robust as Lightning or tip-ring-sleeve, though.

Ansel Adams wrote a celebrated series of photo instructional books. It is organized as a trilogy: The Camera, The Negative and The Print.

Of these, the camera business is still going strong, buoyed by sales of digital cameras and upgraders to DSLRs (although market saturation looms). Negatives are an endangered species as digital photography largely supplanted film. Prints are still going strong, whether using traditional silver halide, inkjet or offset printing, but that probably won’t stave off Kodak’s impending demise.

Last year, I wrote to X-Rite to complain that their software for the Colormunki color calibrator wouldn’t let me profile the full brightness range of my monitor, and limited it to an artificial ceiling designed around the limitations of print. Their response:

The ColorMunki software will not allow a range over 140cd/m luminance value.  This would basically defeat the purpose of using a device to measure luminance when you set the values this high.  While your monitor can be extremely bright however using a value of 300+ would allow your monitor to show you a brightness value that would negatively affect your prints and they would end up coming out so dark that you would be unable to see them.  The industry standard for photographic works is to use the luminance values setup at anywhere from 80 – 120cd/m.  That said, if your ambient light conditions are ranging very high then you should probably be using a value of 120cd/m which is the industry standard and what we recommend.

In the ColorMunki software there is an option to set the luminance according to your ambient light conditions.  This is a pretty high threshold in the software where as you have to read somewhere around 350-400 lux to get a target luminance above the 80 threshold.  If it does go above this value then you certainly will want to stay within the 120 range.  Anything below that will give you a target value of 80.

Implicit in this response is the assumption the only reason why you would want to calibrate your monitor is so you can make more accurate prints. Prints only have a 100:1 or so contrast ratio, compared to LCD monitors for which the static contrast ratio is closer to 1000:1, and plasma TVs go higher yet. This is why images on screen look more vibrant and punchy than prints, as did slides back in the day.

If Apple does indeed come up with the oft-rumored iPad 3 with double the resolution of current models, or 2048×1536 at 260ppi, you would have an easily portable display device with near gallery-grade resolution and massive storage capacity. At that point, prints would lose the last advantage they still hold over digital display technologies.

Image display on screen is the new normal, and X-Rite needs to get with the program. I want to see the full dynamic range and contrast of my images on my screen, not a Hobson’s choice between inaccurate color and crippling them with limitations from legacy print technology.

 

This post may be of use to people experiencing the same problem I had. When I opened PowerPoint slides sent by a coworker, I was getting blank slides. Attempting to switch to outline view would crash PowerPoint. A search for possible solutions yielded inconclusive posts or dead-ends.

After much twiddling, it turns out the problem was with a misconfigured Arial font. Microsoft in its great wisdom installs a copy of Arial that conflicts with the system-supplied one, as explained in Kurt Lang’s must-read article on OS X font management, but that was not the root cause of the problem.

Arial was not appearing at all in the PowerPoint font menu, and opening the slides in Keynote yielded a font warning saying non-existent Arial was replaced by Arial Narrow. Clearly PowerPoint is much less resilient than Keynote to missing fonts (crashing in outline view mode is inexcusable, but that’s Microsoft coding sloppiness for you). If Arial is so vital, it should have been included as a resource in the application bundle itself, but I digress.

Font Book did show Arial as installed, with only a single copy after I purged the conflicting fonts as per Kurt Lang’s recommendations. Validating Arial in Font Book yielded a warning about duplicate fonts, but with no indication of where the duplicate may be, since Font Book itself did not know of any other instance.

It turns out I had an old pre-OpenType copy of Arial in my ~/Library/Fonts folder that was causing a conflict. It was functional enough to be picked up as conflicting and disabling Arial in PowerPoint and Keynote, but not functional enough to be listed by Font Book. I am not sure how it got there, possibly from an older version of Office and transferred over many machine upgrades. Removing the file fixed the problem. of blank slides and the crash in switching to outline view.

Of course, if you care at all about typography, you should use Helvetica rather than an inferior ersatz like Arial, but Arial is so prevalent in the taste-impaired Windows world that one cannot escape its gaucheness entirely.

 

It is strange no one seems to have picked up the news yet, but HP has reissued the legendary HP-15C in a special “30th anniversary limited edition”, and it became available for purchase last week.

The new HP-15C is not strictly speaking a reissue but a replica, as it does not use the original’s Saturn processor, but instead an emulation thereof running on an ARM CPU. Even emulated, it should be much faster than the original 640 kilohertz processor. I ordered two, and received them today.

As expected, the quality is in line with the current HP-12C, i.e. not as good as the 1980s models in terms of key feel, but still leagues ahead of any competing product. The originals used a special 47-point bonding process to ensure the utmost in rigidity and reliability, I doubt the current model had as much attention paid to detail. It is made in China, obviously, the Corvallis facility is long gone. The slipcover fits very poorly (too tight, and the seams are not trimmed properly) and feels thinner and outright cheap compared to the original. The labels on the keys are accurately positioned, at least, unlike the train wreck that was the HP-12C Platinum. The cheat sheet in the back is a garish black on silver as on the 35S, instead of the original’s silver on black. It also uses two 3V CR2032 batteries instead of the 3 button cells in  the original.

Speed-wise, the Limited Edition integrates the normal distribution nearly instantly, when that test that took 34 seconds on the original.

In short: not as good as the original, but still an excellent calculator for those who prize ergonomics.