Fazal Majid's low-intensity blog

Sporadic pontification

Fazal

Amazon wishlist optimizer

I wrote a script several months ago to go through an Amazon wish list and find the combination of items that will best fit within a given budget. Given that the Christmas holiday shopping season seems to have started before Thanksgiving, it seemed topical to release it.

It used the Amazon Web Services API, which is a complete crock (among other failings, it will consistently not return the Amazon.com price for an item, even when explicitly instructed to do so). It does not look like Amazon pays any particular attention to the bug reports I filed. I just gave up on the API and re-implemented it the old-fashioned way, by “scraping” Amazon’s regular (and most definitely not XML-compliant) HTML pages.

It is still very much work in progress, but already somewhat useful. You can use it directly by stuffing your wish list ID in the URL (or using the form below):

Wish list IDAmount

A better way is to drag and drop the highlighted Amazon optimizer bookmarklet link (version 6 as of 2007-05-08) to your browser’s toolbar. You can then browse through Amazon, and once you have found the wish list you are looking for, click on the bookmarklet to open the optimizer in a new window (or tab). By default, it will try and fit a budget of $100 (my decadent tastes are showing, are they not?), but you can change that amount and experiment with different budgets. Surprisingly often, it will find an exact fit. Otherwise, it will try to find the closest match under the budget with as little left over as possible.

There are many caveats. The wishlist optimizer only works for public Amazon.com (US) wish lists. There does not seem to be an easy way to buy multiple items for somebody else’s wish list in one step, although I am working on it, so you will have to go through the wish list and add the items by hand. Shipping costs and wish list priorities are currently not taken into account. Sometimes Amazon will not show a price straight away but instead require you to click on a link, the optimizer will decline to play these marketer’s games and just skip those products.

Be patient – Amazon.com is rather slow right now — it seems they did not learn the lessons of their poor performance towards the end of last year. One of my coworkers ran the optimizer through an acid test with his wife’s 13-page wish list, and it took well over a minute and half to fetch the list, let alone optimize it. One can only imagine how bad it will get when the Christmas shopping season begins in earnest. To mitigate this somewhat, I have added caching – the script will only hit Amazon once per hour for any given wish list. As it works by scraping the web site rather than using the buggy and unreliable Amazon Web Services API, there is a real risk it will stop working if Amazon blocks my server’s IP or if they radically change their wish list UI (they would do better to add additional machines and load-balancers, but that would be too logical).

Update (2005-12-02):

Predictably, Amazon changed their form (they changed the form name from edit-items to editItems) and broke not only the wishlist optimizer, but also the bookmarklet. I fixed this and upgraded to the scraping module BeautifulSoup, but you will need to use the revised bookmarklet above to make it work again.

Update (2010-04-27):

The script has been broken for quite a while, but I fixed it and it should work again.

A book signing with Steven Erikson

Steven EriksonI reviewed the Malazan Book of the Fallen last year — it is one of the very finest Fantasy series, in my opinion. I met Steven Erikson today during a book signing at Borderlands Books in San Francisco. Sadly, there were enough people in the audience who had not read all first five volumes that he read from Memories of Ice rather than from the final manuscript of the sixth volume, The Bonehunters (due out in February 2006) that he carries with him on his Palm PDA.

Tor Books has acquired the rights to the series for the US market. They have already published the first three volumes, and are expected to catch up with the British publishers by the eighth or so. The cover art on the Bantam British edition is better though. The publishing industry has an adage, “mugs sell mags”, and the US covers have more figurative illustrations, sometimes unhappily so, as with the slightly cheesy cover of Memories of Ice

Erikson described the genesis of the series and Malazan universe in a series of literary role-playing games with fellow archeologist Ian Cameron Esslemont, author of Night of Knives, a novel set in the same universe, also the first in a series. He mentioned he is also working on a series of six novelettes featuring the psychopathic necromancers Bauchelain and Korbal Broach (whom he managed to work into Memories of Ice), from the point of view of their long-suffering manservant. The novelettes will be, in Erikson’s own words, “more over the top”. The first two, Blood Follows and The Healthy Dead have already been published (even if Amazon incorrectly claims the latter not available yet), the third one is coming shortly.

When asked whether he was planning on extending the series beyond the planned ten volumes, he mentioned he had the outline of all ten almost from the very beginning (keep in mind it took him 8 years to get The Gardens of the Moon published, and that only happened after he moved to England). There is still a lot of room for spontaneity — as he puts it, if the author is bored when writing the actual books because he put too much effort in preparatory notes, the readers are likely to be bored as well. Erikson also committed to giving “payback” to his readers for sticking with the story (sounds ominous, doesn’t it?), with some snide remarks referring to Robert Jordan’s ever-lengthening Wheel of Time series. The anecdote he mentioned was that of a 75 year old woman who was asking a bookseller when the next installment by Jordan would be published, because she was afraid she might die before that series was completed… In all fairness, Jordan has announced the next volume will be the last, bringing closure to long-suffering fans.

I asked him about the whole extinction of magic as a moral imperative angle, and he indicated the later volumes in the decalogue would bear on the issue. He also said he is in no way endorsing imperialism (Deadhouse Gates is in part inspired from events in the British Empire’s oppression of India and Afghanistan). I also mentioned how difficult I found the abrupt transition introduced by volume 5, Midnight Tides. He agreed, but it was required by the 10-volume story arc, and postponing it would only make things worse. Among other matters, we will read more of the Forkrul Assail, whom he describes as the nastiest of the four founding races.

As a final note, I have been to book signings with Raymond Feist, Robert Jordan and Steven Erikson, and I am always amazed by the inconsiderate people who come with cartons full of books to sign, presumably to make them more collectible and valuable. The value in these events is in meeting the authors and interacting with them, not in giving them tennis elbow for financial gain.

Interesting factoids

Harper’s Magazine, a left-leaning (by American standards) literary gazette, is fairly insipid, but it publishes amusing tidbits in each issue known as Harper’s Index. In a similar vein, here are some surprising bits I have read recently.

  • All 9 members of China’s Politburo are engineers. Source: IEEE Spectrum
  • Western Europe has a population and GDP comparable to the United States, but it has 42% of the world’s WiFi hotspots, compared to 26% in the US. Source: Informa Telecoms and Media.
  • Medical Doctors’ median income in the US is $200,000. Often maligned, median malpractice insurance premiums are only $11,000. Source: Paul von Hippel, Ohio State University.
  • “Administrative costs” represent 19% to 24% of the cost of health care in the US, compared to about 10% in most OECD countries. Source: University of Maine.
  • The French universal medical coverage, despite being rife with abuse and fraud by people who would flunk the means-test for state coverage, costs about 1.4 billion euros per year, slightly under 0.1% of GDP, with approximately 5 million people covered, and health care in general represents about 9% of GDP. Of course, as health insurance is mandatory for all salaried workers, only the unemployed lack coverage in the first place, so the cost of universal coverage in the US would be higher as a proportion of GDP. The French medical system was rated first in the world for general health care by the WHO’s last survey in 2000, so it is not a question of skimping on the quality of care as in the UK.
  • The US spends 15% of its GDP on health care, if that were lowered by 10%, by bringing administrative costs in line with Europe or Canada, the savings would easily cover universal insurance for all Americans.
  • The Philippines and India are respectively ranked No. 4 and 5 destinations for international telephone calls from the US. India hardly registered in 1991. Source: Telegeography.

Palm T|X first impressions

After an abortive experiment with a Nokia Symbian Series 60 smartphone, I bought a Palm T|X on Wednesday, the very day it was announced. I find PDAs superior to fiddly, fragile and cumbersome laptops, and have owned no fewer than 9 Palm compatible handhelds (*) in the last 5 years, which means I upgrade handhelds at least three times more often than my main (desktop) computers. My previous PDA is a Palm Tungsten T3 (I actually bought it after the T5 was announced, so underwhelming the latter is). I even obtained a spare T3 in case the first one broke (since given to my father). I am not entirely sure yet as to whether the T|X is really an upgrade. Here are some first impressions after a few days of use:

Pros:

  • Built-in WiFi. No more fiddling with the easily lost SDIO WiFi card.
  • A better browser. Blazer feels much snappier than Web Pro, specially with the new Fast mode (disables CSS and image loading).
  • More memory, non-volatile if the battery fails.
  • Lighter.
  • Can actually dial and send SMS on a Nokia 6230 via Bluetooth

Cons:

  • Plastic construction feels much less robust (but at least it is not pretending to be metal like the E, E2 or T5, that’s just tacky).
  • No voice recorder, charge LED or vibrating alarm. I seldom use the voice recorder, as I prefer taking notes on 3×5 jotter cards, but the voice recorder works when you have to capture that elusive idea while driving.
  • 20–25% slower processor. Graffiti2 is noticeably slower to respond, for instance.
  • The flip cover with the hinge on the side is less convenient than the one on top, which flips up like a reporter’s notebook, in one fluid motion.
  • The SD slot has a plastic filler card, not a spring-loaded cover.
  • Bigger. Many people complain about the true Tungstens’ slider, but it is very natural to use, and much more convenient than the power switch.
  • The stylus has a nice heft to it, but is not as substantial as the T3’s, and less easy to extract from its slot.
  • Yet another connector design incompatible with previous accessories. The cradle is an expensive option.
  • The home icon on the status bar has disappeared. This is very annoying in daily use
  • The application buttons and the 5-way navigator are less responsive and smaller. The T3 has generally superior haptics (feels much better in the hand).

The only potential deal-breaker is the slower Graffiti performance (there is a visible lag). I will probably keep the T|X due to the convenience of integrated WiFi, but the T3 is a superior device in almost all other respects, in the same class as the Palm V as one of the PDA world’s truly outstanding designs. If Palm were to come out with a new model marrying the WiFi and newer software stack of the T|X with the solid construction and faster processor of the T3, I would definitely upgrade again.

(*): Handspring Visor, Sony Clié T615C, Kyocera QCP-6035, Palm Tungsten T, Sony Clié UX50, Palm Zire 71, Palm Tungsten T3 (x2), and now the Palm T|X.

Update (2010-05-16):

The T|X was the last Palm device I bought. I switched to an iPhone in 2007 and never looked back.

The real story behind the WSIS

There has been much speculation recently about a possible rift in Internet governance. Essentially, many countries resent the US government’s control over the Internet’s policy oversight. They advocate the transfer of those responsibilities to the International Telecommunications Union (ITU), a more multilateral venue. The big news is that the European Union, which previously sat on the fence, came out strongly in favor of this proposal. Unsurprisingly, the US government is hostile to it. More surprisingly, I agree with their unilateralist impulse, obviously for very different reasons. I was planning on writing up a technical explanation as most of the IT press has it completely wrong, as usual, but Eric Rescorla has beaten me to the punch with an excellent summary.

Many commentators have made much hay of the fact the ITU is under the umbrella of the United Nations. The Bush administration is clearly reticent, to say the least, towards the UN, but that is a fairly widespread sentiment among the American policy establishment, by no means limited to Republicans. For some reason, many Americans harbor the absurd fear that somehow the UN is plotting against US sovereignty. Of course, the reality is the UN cannot afford its parking tickets, let alone black helicopters. American hostility towards the UN is curious, as it was the brainchild of a US president, Franklin D. Roosevelt, its charter was signed in San Francisco (at Herbst Theatre, less than a mile from where I live), and it is headquartered in New York.

The UN is ineffective and corrupt, but that is because the powers on the Security Council want it that way. The UN does not have its own army and depends on its member nations, specially those on the Security Council to perform its missions. It is hardly fair to lay the blame for failure in Somalia on the UN’s doorstep. As for corruption, mostly in the form of patronage, it was the way the US and the USSR greased the wheels of diplomacy during the Cold War, buying the votes of tin-pot nations by granting cushy UN jobs to the nephews of their kleptocrats.

A more damning condemnation of the UN is the fact the body does not embody any kind of global democratic representation. The principle is one country, one vote. Just as residents of Wyoming have 60 times more power per capita in the US Senate than Californians, India’s billion inhabitants have as many votes in the General Assembly as those of the tiny Grand Duchy of Liechtenstein. The real action is in the Security Council anyways, but they are not fully represented there either. Had Americans not had a soft spot for Chiang Kai-Shek, China, with its own billion souls, would not have a seat at that table either. That said, the Internet population is spread unevenly across the globe, and the Security Council is probably more representative of it.

In any case, the ITU was established in 1865, long before the UN, and its institutional memory is much different. It is also based in Geneva, like most international organizations, geographically and culturally a world away from New York. In other words, even though it is formally an arm of the UN, the ITU is in practice completely autonomous. The members of the Security Council do not enjoy veto rights in the ITU, and the appointment of its secretary general, while a relatively technocratic and unpoliticized affair, is not subject to US approval, or at least acquiescence, the way the UN secretary-general’s is, or that of more sensitive organizations like the IAEA.

My primary objections to the ITU are not about its political structure, governance or democratic legitimacy, but about its competence, or more precisely the lack of it. The ITU is basically the forum where government PTT monopolies meet incumbent telcos to devise big standards and blow big amounts of hot air. Well into the nineties, they were pushing for a bloated network architecture called OSI, as an alternative to the Internet’s elegant TCP/IP protocol suite. I was not surprised — I used to work at France Télécom’s R&D labs, and had plenty of opportunity to gauge the “caliber” of the incompetent parasites who would go on ITU junkets. Truth be said, those people’s chief competency is bureaucratic wrangling, and like rats leaving a ship, they have since decamped to the greener pastures of the IETF, whose immune system could not prevent a dramatic drop in the quality of its output. The ITU’s institutional bias is towards complex solutions that enshrine the role of legacy telcos, managed scarcity and self-proclaimed intelligent networks that are architected to prevent disruptive change by users on the edge.

When people hyperventilate about Internet governance, they tend to focus on the Domain Name System, even though the real scandal is IPv4 address allocation, like the fact Stanford and MIT each have more IP addresses allocated to them than all of China. Many other hot-button items like the fight against child pornography or pedophiles more properly belongs in criminal-justice organizations like Interpol. But let us humor the pundits and focus on the DNS.

First of all, the country-specific top-level domains like .fr, .cn or the new kid on the block, .eu, are for all practical purposes already under decentralized control. Any government that is afraid the US might tamper with its own country domain (for some reason Brazil is often mentioned in this context) can easily take measures to prevent disruption of domestic traffic by requiring its ISPs to point their DNS servers to authoritative servers under its control for that zone. Thus, the area of contention is really the international generic top-level domains (gTLDs), chief of all .com, the only one that really matters.

What is the threat model for a country that is distrustful of US intentions? The possibility that the US government might delete or redirect a domain it does not like, say, al-qaeda.org? Actually, this happens all the time, not due to the malevolence of the US government, but to the active incompetence of Network Solutions (NSI). You may recall NSI, now a division of Verisign, is the entrenched monopoly that manages the .com top-level domain, and which has so far successfully browbeaten ICANN into prolonging its monopoly, one of its most outrageous claims being that it has intellectual property rights to the .com database. Their security measures, on the other hand, owe more to Keystone Kops, and they routinely allow domain names like sex.com to be hijacked. Breaking the NSI monopoly would be a worthwhile policy objective, but it does not require a change in governance, just the political will to confront Verisign (which, granted, may be more easily found outside the US).

This leads me to believe the root cause for all the hue and cry, apart from the ITU angling for relevance, may well be the question of how the proceeds from domain registration fees are apportioned. Many of the policy decisions concerning the domain name system pertain to the creation of new TLDs like .museum or, more controversially, .xxx. The fact is, nobody wakes up in the middle of the night thinking: “I wish there were a top-level domain .aero so I could reserve a name under it instead of my lame .com domain!”. All these alternative TLDs are at best poor substitutes for .com. Registrars, on the other hand, who provide most of the funding for ICANN, have a vested interest in the proliferation of TLDs, as that gives them more opportunities to collect registration fees.