Engineer Says Registry Change Bricked 600 PCs, Cost $1 Million in CDs
A former engineer identified only as Oriel says a forced registry change in a small business accounting program broke more than 600 customer PCs. The vendor replaced the machines, then pulled the CD-ROMs from stores, scrapping stock and packaging worth $1 million.
A software engineer identified only as Oriel says a registry change he ordered for a small business accounting program left more than 600 customer PCs unusable and forced his employer to scrap $1 million of CD-ROMs. The dev error bricked 600 machines in the days before a major holiday, he said. The vendor replaced every one.
A fix that backfired
Oriel, whose name is a pseudonym, worked for a vendor selling accounting software for small businesses. The package shipped on CD-ROM and sold in shops. A new version came out every year, always before a holiday built around a giant meal.
The previous year's release had failed to install on some PCs running Windows 95 or Windows ME because of registry corruption. The registry is the database where Windows stores system settings.
To raise install success rates, the team forced updates to registry keys during setup. "A key decision was made – by me – to force the updating of registry keys to enable the install," Oriel said in an email. He added: "What could possibly go wrong?"
Customers lost all their software
Soon after shipping, the support lines filled with calls. Customers said that after installing the accounting package, none of their other software would work. "Installing a new version of our application had borked the customer's machine completely," Oriel wrote.
The company told affected customers to buy a new PC and said it would pay, no questions asked. "In the week prior to the holiday, we replaced over 600 machines," Oriel said.
Hunting for a test machine
Oriel and his colleagues were ordered to fix the bug. They could not reproduce it or find their mistake.
They needed a customer PC with a Windows Registry restore point created before the install. Developers could roll that machine back, install the accounting software and watch what broke. Few customers had made a restore point, so support agents struggled to find a volunteer.
The day before the holiday, one customer offered a machine. The customer lived a few hours away by road. An engineer drove there on the morning of the holiday, handed over a new system and set it up to the customer's specifications.
The job took so long that the engineer was invited to stay for the holiday meal. He then returned to headquarters with the broken PC.
Discs pulled from shelves
Oriel and his colleagues found the problem the next day and built a version that no longer broke PCs. Senior executives then called Oriel into a meeting to explain what had happened.
They decided to pull the faulty CDs from stores. That meant trashing discs and packaging valued at $1 million.
