The survey taker had a government issued laptop to administer the test. It was a large, no-brand name laptop running Windows Vista Basic. After booting into Windows, they run some app that tracks the with the test takers' home addresses, then launches the test. To my surprise, the test itself is within a VMWare Player virtual machine. It booted what looked like some stripped down variant of Debian, then directly into X and the test interface.
The test itself was fairly well designed... obviously some real thought was put into it. It tests your ability to use a basic e-mail client, spreadsheet and understand search engine results. Some tests were reading data from a spreadsheet or chart and calculating answers with a calculator. Some were reading e-mail and inputting values into a spreadsheet.
The test did have some problems.
- Software bugs ... often clicking would fail to work. You would need to figure out workarounds, often involving clicking away then re-entering a part of the UI, to make it work as intended. These bugs often made it frustrating to participate, and probably made me take 25% longer to complete the test. I can see this screwing up their end results as some test takers may just give up due to bugs rather than demonstrate skills that they know.
- I noticed several instances where the on-screen instructions had typos, ambiguously written instructions, or ambiguous text to read and interpret in order to understand the goal of a particular test. This made me wonder if the ambiguity combined with software bugs were INCLUDED INTENTIONALLY as a means to test real-world problem solving skills. If so, this might be a clever design.
- But then I noticed another problem that surely cannot be intended. The survey taker said it is supposed to be administered identically and in English in the 25 countries. I saw several things in the test that were obviously American-centric, and I highly doubt they will be able to successfully test in 25 countries in English.
- The survey administrator complained that the laptops were EXTREMELY SLOW and often the VMWare Player would get stuck during shutdown, preventing her from collecting the results and moving on.