IT

Like many people, I keep a server running at home 24/7. In my case, it’s an old but relatively quiet Compaq Evo D315 AMD Athlon XP2000 PC with 1GB of RAM, 750GB total disk space and running Solaris 10. It serves as my personal email server (Postfix and Dovecot), to run Temboz, and miscellaneous auxiliary services like DNS, SNMP or being a staging point for off-site backups via rsync. All in all, very light usage, less than 5% average CPU utilization.

I have a Kill-a-Watt power meter measuring the load on that shelf, and the server, along with other devices on standby power, consumes about 160W. At PG&E’s marginal rate of $0.13 per KWh, that comes to $180 a year, or half the cost of the machine itself. I am thinking of upgrading to a machine with 6 750GB or 1TB drives in a 4+2 redundant RAID-Z2 configuration for reliable backups (the current setup runs on ZFS for snapshots but has no provisions for drive failure). I will definitely look at power consumption more closely in my decision process

Update (2007-08-25):

I ended up getting a Sun Ultra 40 M2 dual-core AMD Opteron workstation with 6 additional Seagate 750GB drives. It is remarkably quiet and consumes only 160W, which is pretty good since it does have 7 drives spinning inside. ZFS benchmarks at 160MBps sustained disk I/O…

CVStrac is an amazing productivity booster for any software development group. This simple tool, built around a SQLite database (indeed, by the author of SQLite) combines a bug-tracking database, a CVS browser and a wiki. The three components are fully cross-referenced and build off the strengths of each other. You can handle almost all aspects of the software development process in it, and since it is built on an open database with a radically simple schema, it is trivial to extend. I use CVStrac for Temboz to track bugs, but also to trace changes in the code base to requirements or to bugs, and last but not least, the wiki makes documentation a snap.

For historical reasons, my company uses TWiki for its wiki needs. We configured Apache with mod_rewrite so that the wiki links from CVStrac lead to the corresponding TWiki entry instead of the one in CVStrac itself, which is unused. TWiki is very messy (not surprising, as it is written in Perl), but it has a number of good features like excellent search (it even handles stemming) and a directed graph plug-in that makes it easy to design complex graphs using Bell Labs’ graphviz, without having to deal with the tedious pixel-pushing of GUI tools like Visio or OmniGraffle. The plug-in makes it easy to document UML or E-R graphs, document software dependencies, map process flows and the like.

CVStrac 2.0 introduced extensibility in the wiki syntax via external programs. This allowed me to implement similar functionality in the CVStrac native wiki. To use it, you need to:

1. Download the Python script dot.py and install it somewhere in your path. The sole dependency is graphviz itself, as well as either pysqlite2 or the built-in version bundled with Python 2.5
2. create a custom wiki markup in the CVStrac setup, of type “Program Block”, with the formatter command-line:
   path/dot.py –db CVStrac_database_file –name ‘%m’
   • Insert the graphs using standard dot syntax, bracketed between CVStrac {dot} and {enddot} tags.

{dot}
digraph sw_dependencies {
style=bold;
dpi=72;

temboz [fontcolor=white,style=filled,shape=box,fillcolor=red];
python [fontcolor=white,style=filled,fillcolor=blue];
cheetah [fontcolor=white,style=filled,fillcolor=blue];
sqlite [fontcolor=white,style=filled,fillcolor=blue];

temboz -> cheetah -> python;
temboz -> python -> sqlite -> gawk;
temboz -> cvstrac -> sqlite;
python -> readline;
python -> db4;
python -> openssl;
python -> tk -> tcl;

cvstrac -> "dot.py" -> graphviz -> tk;
"dot.py" -> python;
"dot.py" -> sqlite;
graphviz -> gdpng;
graphviz -> fontconfig -> freetype2;
fontconfig -> expat;
graphviz -> perl;
graphviz -> python;
gdpng -> libpng -> zlib;
gdpng -> freetype2;
}
{enddot}

Another useful plug-in for CVStrac I wrote is one that highlights source code in the CVS browser using the Pygments library. Simply download pygmentize.py, install it Setup/Diff & Filter Programs/File Filter, using the string path_to/pygmentize.py %F. Here is an example of Pygment applied to pygmentize.py itself:

#!/usr/bin/env python
# $Log: pygmentize.py,v $
# Revision 1.3  2007/07/04 19:54:26  majid
# cope with Unicode characters in source
#
# Revision 1.2  2006/12/23 03:51:03  majid
# import pygments.lexers and pygments.formatters explicitly due to Pygments 0.6
#
# Revision 1.1  2006/12/05 20:19:57  majid
# Initial revision
#
"""
CVStrac plugin to Pygmentize source code
"""
import sys, pygments, pygments.lexers, pygments.formatters

def main():
  assert len(sys.argv) == 2
  block = sys.stdin.read()
  try:
    lexer = pygments.lexers.get_lexer_for_filename(sys.argv[1])
    out = pygments.highlight
    block = pygments.highlight(
      block, lexer, pygments.formatters.HtmlFormatter(
      style='colorful', linenos=True, full=True))
  except ValueError:
    pass
  print unicode(block).encode('ascii', 'xmlcharrefreplace')

if __name__ == '__main__':
  main()

Unpaid computer tech support for relatives is not a popular topic among geeks. It is very much a reality, however, specially in Indian communities with extensive extended families like mine. Some of the griping is churlish considering all the favors your family cheerfully does for you, and we probably have it better than MDs who are constantly bombarded with requests for free medical consultations.

At first sight, I would be better off if my relatives had the good sense to ditch Windows and get a Mac instead, but that would in fact compound the problem because I would get even more calls for help from people who are having a hard time dealing with very basic issues on an unfamiliar platform. Mac OS X may be better integrated and secure than Windows, but contrary to popular opinion it is not that much less crash-prone. All computers are unnecessarily hard to use in the first place. I doubt very much the computer industry will mend its ways and put human-centered design first, more likely than not the problem will be “solved” by the progressive eclipse of generations born before widespread computing, the rest of us having perforce adapted to these flawed tools.

A big part of the problem is doing “blind” support over the phone where you don’t see what is going on, and often the person in front of the screen is not technical enough to know what is significant and how to give you a useful and actionable description of what is on screen.

To its credit, Microsoft added remote assistance functionality in Windows XP. Explaining to users how to activate it is a challenge in itself, however, and in any case you need another Windows XP machine to provide the support. I still run Windows 2000 in the sole PC I have (used exclusively for games nowadays) and it makes such a racket I am almost viscerally reluctant to boot it up.

The best solution is to use virtual network computing (VNC), a free, cross-platform remote control protocol originally invented by the former Olivetti-Oracle-AT&T labs in Cambridge, UK. I often use VNC to take control of my home Mac from my office PC or my MacBook Pro. Indeed, VNC is integral to Apple Remote Desktop, Apple’s official remote management product for large Mac installations. There are even VNC clients available for PalmOS and Windows CE so you could remote control your home computer from a Treo. Having VNC running on the ailing PC would allow me to troubleshoot it efficiently from the comfort of my Mac.

Unfortunately, there is still a chicken-and-egg effect. I once tried to get an uncle to set up UltraVNC on his PC and do a reverse SSH forwarding so I could bypass his firewall. It took the better part of an hour, and barely worked. Surely, there has to be a better solution.

One such solution is Copilot, a service from Fog Creek software that repackages VNC in a form that’s easier to use. It is somewhat expensive, however (although that can be seen as a feature if the people calling for help have to pay for it and thus have an incentive to moderate their requests).

Another one that shows some promise is UltraVNC SC, a simplified version of UltraVNC that is designed for help desks (here is a more friendly walkthrough). Unfortunately, it shows a very clunky dialog that makes sense in a corporate help desk setting, but is too confusing for a novice user, and it uses UltraVNC extensions that are not compatible with most other VNC clients like the one I use most, Chicken of the VNC.

In the end, what I ended up doing was to take the source code for the full-featured UltraVNC server, rip out all the user interface and registry settings from it, and hardcode it to open an outgoing connection to my home server alamut.majid.org on TCP port 5500. There isn’t anything on the server listening on port 5500 by default, but I can open a SSH connection to it from anywhere in the world and use SSH reverse port forwarding to connect port 5500 to wherever I am. This neatly sidesteps the problem of firewalls that block incoming connections.

The resulting executable is larger than SC, but still manageable at 500K (vs. 950K for the full version), and requires no input from the user beyond downloading it and running it, thus triggering all sorts of warnings. It’s not good practice to teach users to download and run executables, but presumably they trust me. After the VNC session is finished, the program simply exits (as evidenced by the disappearance of the UltraVNC eye icon from the toolbar

If you want to use a setup like mine, it’s easy enough for a technically inclined person:

1. You could download my executable at majid.org/help, open it in a hex editor (or even Emacs), search for the string alamut.majid.org and overwrite it with the name of the machine you want to use instead (I left plenty of null bytes as padding just in case). Make sure you are overwriting, not inserting new bytes or shrinking the string, as the executable won’t work correctly otherwise.
2. Or you could download the modified source code I used (UltraVNC is a GPL open-source project, so I am bound by the license to release my mods). Edit the string host in winvnc/winvnc/winvnc.cpp (you can also change the reverse VNC port from its default of 5500 if you want), and recompile using the free (as in beer) Visual C++ 2005 Express Edition and the Platform SDK. My Windows programming skills are close to nil, so if I could do it, you probably can as well.

To use the tool, put it up on a website, and when you get a request for help, SSH into the server. On UNIX (including OS X), you would need to issue the command:

ssh -R5500:127.0.0.1:5500 your.server.name.com

Please note I explicitly use 127.0.0.1 rather than localhost, as the former is always an IPv4 address, but on some systems, localhost could bind to the IPv6 equivalent ::1 instead.

On Windows, you will need to set the reverse port forwarding options in PuTTY (or just replace ssh with plink in the command-line above). After that start your VNC client in listen mode (where the VNC client awaits a connection from the server on port 5500 instead of connecting to the server on port 5900). You can then tell the user to download the executable and run it to establish the connection.

Some caveats:

1. The leg of the connection between the PC and the server it is connecting to is not encrypted
2. Depending on XP firewall settings, Windows may ask the user to authorize the program to open a connection
3. At many companies, running a program like this is grounds for dismissal, so make sure whoever is calling you is asking for help on a machine they are authorized to open to the outside.

I hesitated to make this widely available due to the potential for mischief, but crackers have had similar tools like Back Orifice for a very long time, so I am not exactly enhancing their capabilities. On the other hand, this makes life so much easier it’s worth sharing. Helping family deal with Windows will still be a chore, but hopefully a less excruciating one.

Update (2007-03-23):

You can make a customized download of the executable targeting your machine using the form below. Replace example.com with whatever hostname or IP address you have. If you do not have a static IP address, you will need to use a dynamic DNS service like DynDNS or No-IP to map a host name to your dynamic IP address.

Bill Gates announced yesterday he is progressively going to disengage himself from day-to-day participation in Microsoft over the next two years, to concentrate exclusively on his foundation. However questionable Microsoft’s business practices may be, they are no worse than Standard Oil’s. The Rockefellers or Carnegie bought social respectability by endowing institutions for the already comfortable. No matter what the IRS may claim, donating to places like Harvard in exchange for naming rights does not qualify as charity in my book.

In contrast, Gates’ humanitarian work has been remarkable — his money is comforting the truly afflicted of this world, like sufferers of leprosy or malaria. His example is highly unlikely to be emulated by Silicon Valley’s skinflint tycoons (Larry Ellison, Steve Jobs, I’m looking at you). The latter conveniently convinced themselves their wealth is due entirely to their own efforts, never to luck, or government funding in the case of the Internet moguls. This leads to the self-serving belief that they are absolved of any obligation to society or to those less fortunate (in both senses of the term).

This decision is not entirely unexpected. Microsoft has been floundering for the last several years, and has accrued severe managerial bloat, something the ruthless and paranoid Bill Gates circa 1995 would never have allowed to continue. There is remarkable dearth of insightful commentary on the announcement. My take is that the harrowing and humiliating process of the DoJ anti-trust trial proved cathartic and led him to review his priorities, even if the lawsuit itself ended up with an ineffective slap on the wrist.

Some equally interesting reading coming out of Redmond: Broken Windows Theory, an article by a Microsoft project manager on the back story behind the Windows Vista delay, with some really interesting metrics. Apparently Vista takes no less than 24 hours to compile on a fast dual-processor PC. It has 50 levels of dependencies, 50 million lines of code (one metric I personally find meaningless, as you can get more done in one line of Python than in a hundred lines of C/C++). His conclusion is that due to its scale, Vista could simply be structurally unmanageable. Certainly, the supporting infrastructure, as in automation tools, code and dependency analysis, project management et al. ought to be a project in itself of the same scope as, say, Microsoft Word.

When I worked at France Télécom in the late nineties, they were reeling from the near total failure of Frégate, a half-billion dollar billing system of the future project (another interesting metric: two-thirds of billing systems projects worldwide end in failure). The grapevine even devised a unit of measurement, the Potteau, after an eponymous Ingénieur Général (a typically French title with roots in the military engineering side of the civil service) involved in the project. One potteau equals one man-century. It is deemed the unit beyond which any software project is doomed to failure.

Vista involved 2000 developers over 5 years. That’s over 100 potteaux.

I have noticed a trend lately of computers making an annoying whining sound when they are running at low utilization. This happens with my Dual G5 PowerMac, the Dells I ordered 18 months ago for my staff (before we ditched Dell for HP due to the former’s refusal to sell desktops powered with AMD64 chips instead of the inferior Intel parts), and I am starting to notice it with my MacBook Pro when in a really quiet room.

These machines emit an incredibly annoying high-pitched whine when idling, one that disappears when you increase the CPU load (I usually run openssl speed on the Macs). Probably the fan falls in some oscillating pattern because no hysteresis was put into the speed control firmware. It looks like these machines were tested under full load, but not under light load, which just happens to be the most common situation for computers. The short-term work-around is to generate artificial load by running openssl in an infinite loop, or to use a distributed computing program like Folding@Home.

Load-testing is a good thing, but idle-testing is a good idea as well…