Automated Install Testing

Audience: Install Developers, Quality Assurance, Beginner to Advanced, Windows Installer (MSI)

I think everyone will agree, once your installation package is built it is a good idea to test it.

Testing installs of any size can be turn out to be tedious and time consuming. There can be many reasons for this, to list a few;

• The install responds differently on various operating systems (like Vista and XP)
• The install needs to support administrative images
• The application supports run from source installations
• There are different code paths taken depending on what is already installed on the system
• The installer upgrades one or several previous versions
• ... The list can go on and on depending on your specific requirements ...

One way is black box testing. Typically this involves nothing more than making a grid of the various scenarios, run them and see what happens. The more requirements the installation must fulfill, the larger the grid will be. This works however, it can be difficult catch every defect and if you are performing each install manually it can eat up a lot of time, which isn’t very cost effective. Most companies leave this strictly up to their quality assurance team, which in turn eats into application testing time.

If you want to save some money, impress your boss and QA team, think about white box testing. The nice thing about white box testing, is scripts can be written that fit into nightly build process which can help uncover known issues. You don’t really have to do anything more than look at some log files and fix the issues... besides write the tests.

If done properly you can uncover defects without even running the installation. These defects can then be fixed before the install even makes it to the QA team, in turn they spend more time testing the application, instead of the install. It’s win, win in my book.

The next few articles will share some handy scripts that I have found useful.

MSI Validation