Tuesday, September 19, 2017

Fedora package dependencies to build Bitcoin Core

Updated September 19th, 2017, last tested on Fedora 26

dnf install boost-devel libdb4-devel libdb4-cxx-devel miniupnpc-devel qrencode-devel protobuf-devel autoconf automake libdb4-cxx-devel libtool make openssl-devel libevent-devel qt5-qttools-devel qt5-qtbase-devel

Monday, July 7, 2014

SELinux problems with Erlang on Fedora/RHEL7

couchdb is one of several widely used server applications written in erlang.  I was surprised to find hundreds of Bugzilla tickets for erlang services and selinux-policy.

Problem #1: Erlang is treated by SELinux as part of the RabbitMQ Service
Erlang is a general purpose VM like Java.  However selinux-policy treated core components of Erlang as part of the rabbitmq semodule.

https://github.com/TresysTechnology/refpolicy-contrib/blob/master/rabbitmq.fc
/usr/lib/erlang/erts.*/bin/beam.*       --      gen_context(system_u:object_r:rabbitmq_beam_exec_t,s0)
/usr/lib/erlang/erts.*/bin/epmd --      gen_context(system_u:object_r:rabbitmq_epmd_exec_t,s0)

Apparently someone attempted to make SELinux work only for rabbitmq, erroneously thinking that the generic erlang runtime binaries are part of rabbitmq.

# ls -lZ /usr/lib64/erlang/erts-5.10.4/bin 
-rwxr-xr-x. root root system_u:object_r:rabbitmq_beam_exec_t:s0 beam
-rwxr-xr-x. root root system_u:object_r:rabbitmq_beam_exec_t:s0 beam.smp
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       child_setup
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       ct_run
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       dialyzer
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       dyn_erl
-rwxr-xr-x. root root system_u:object_r:rabbitmq_epmd_exec_t:s0 epmd
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       erl
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       erlc
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       erlexec
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       erl.src
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       escript
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       heart
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       inet_gethost
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       run_erl
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       start
-rw-r--r--. root root system_u:object_r:lib_t:s0       start_erl.src
-rw-r--r--. root root system_u:object_r:lib_t:s0       start.src
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       to_erl
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       typer

Everything in this directory should be context bin_t.

Problem #2: Erlang Services Are Not Using Their Own Semodules!
The many erlang-related tickets filed against selinux-policy are confused because they are not actually running with the expected SELinux context.

# systemctl start  rabbitmq-server.service
# ps auxZ |grep rabbitmq
system_u:system_r:init_t:s0     rabbitmq   359 11.5  0.4 2177032 38972 ?       Ssl  20:18   0:01 /usr/lib64/erlang/erts-5.10.4/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.1.5/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@server -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@server.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@server-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.1.5/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@server-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@server"

The service is running as init_t. This is because  /usr/lib/rabbitmq/bin/rabbitmq-server is not properly labeled with something like rabbitmq_exec_t so it never transitions into its own semodule when launched from systemd.

Similarly couchdb.service launched itself via ExecStart=/usr/bin/erl (symlink /usr/lib64/erlang/erts-5.10.4/bin) and it too was running as init_t.  Lacking execve to anything labeled couchdb_exec_t, erl would fail to transition into  couchdb_t.  couchdb-1.6.0-9+ now runs from systemd with ExecStart=/usr/libexec/couchdb <parameters>.  If the wrapper shell script /usr/libexec/couchdb is labeled as couchdb_exec_t then couchdb service actually runs as couchdb_t.

Note: couchdb-1.6.0-9+ does not use the upstream /bin/couchdb script from systemd because it is rather broken and upstream plans on removing it in the next release.  The way we configure and exeucte couchdb from systemd is close to the future upstream standard way of handling the service.  While we do not use it for the systemd service it remains in the $PATH because users may depend on its behavior for debugging.

Temporary Workaround for CouchDB + SELinux
couchdb-1.6.0-9 may be the first Erlang application in Fedora to properly be confined by its own semodule(?)  We are waiting for selinux-policy to be fixed.  Meanwhile this temporary workaround will allow CouchDB to operate with SELinux enforcing enabled.  These instructions have been tested on Fedora 20 and RHEL 7.

Here are tested packages that I use on RHEL7 + EPEL7.  EPEL7 will soon have these packages.

#### Upgrade to couchdb-1.6.0-9 or later.
# sudo yum install -y selinux-policy-devel git
# git clone https://github.com/wtogami/refpolicy-contrib ||:
# cd refpolicy-contrib
# git checkout couchdb
# git pull
# make -f /usr/share/selinux/devel/Makefile couchdb.pp
# semodule -i couchdb.pp
# sudo restorecon -Rv /usr/lib64/erlang/erts-*/bin/
# sudo restorecon -v /usr/libexec/couchdb 
# sudo restorecon -v /usr/lib64/erlang/lib/couch-*/priv/couchjs
# sudo restorecon -v /usr/lib/systemd/system/couchdb.service

TODO's
Erlang Service Packages
In general erlang packages with their own semodule must execute via a properly labeled wrapper script.  /usr/bin/erl on its own is unable to guess and transition into the proper SELinux context.  See couchdb-1.6.0-9+ /usr/libexec/couchdb for an example.  As noted above, as rabbitmq never was running with the proper context someone who is familiar with rabbitmq will need to ensure the policy continues to operate as expected.

SELinux Policy
https://bugzilla.redhat.com/show_bug.cgi?id=1116014
Generally, selinux-policy must no longer treat core Erlang binaries as part of rabbitmq.  Everything in the /usr/lib*/erlang/erts-*/bin/ directory should be bin_t, perhaps somewhere in the core policies.

Every server application with its own semodule will need the aforementioned wrapper script(s) with context <something>_exec_t.  For example these rabbitmq binaries need to be properly labeled.

ls -Z /usr/lib/rabbitmq/lib/rabbitmq_server-3.1.5/sbin
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       rabbitmqctl
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       rabbitmq-defaults
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       rabbitmq-env
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       rabbitmq-plugins
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       rabbitmq-server

I fixed the couchdb semodule to work with couchdb-1.6.0-8+.  rabbitmq and other packages may need specific fixes to their policies and perhaps also to their packages.

Changelog
July 9th, 2014
  • Updated couchdb policy and improved workaround instructions.
  • couchdb-1.6.0-9.fc20 coming soon for F19 and F20, and EL7.
  • Updated couchdb policy below
    • Fixed all AVC's in diskup
    • Added missing restorecon to workaround instructions.
    • Added unix_dgram_socket create needed for systemd notify

Sunday, June 30, 2013

Transition to new GPG Key

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dear Internet,

My old GPG key is quite old and 1024bit DSA.  So it is time for me to transition
to a new GPG key.  If you know me well, trust me or have signed me in the past,
I need help in re-establishing my Web-of-Trust linkages.

http://pgp.mit.edu:11371/pks/lookup?op=vindex&search=0x6BDDFE8E54A2ACF1
pub   1024D/54A2ACF1 2002-11-25
      Key fingerprint = 785A 304B 08C1 F291 F54F  9A68 6BDD FE8E 54A2 ACF1
uid                  Warren Togami (Linux) <warren@togami.com>
sub   2048g/4AD75982 2002-11-25

http://pgp.mit.edu:11371/pks/lookup?op=vindex&search=0xB1179EB7347DC10D
pub   8192R/347DC10D 2013-06-29
      Key fingerprint = AEC1 8843 9864 7C47 413C  1C3F B117 9EB7 347D C10D
uid                  Warren Togami (2013) <wtogami@gmail.com>
sub   8192R/668709D4 2013-06-29

Warren's OTR Fingerprint
wtogami@gmail.com/ (XMPP): 3DF16C5E 8ABB4BBD 0B43EBF0 52B02ABA A2076B25

Verifying Warren's Identity in Multiple Difficult-to-Fake Ways
==============================================================
* This message is GPG signed by both the old and new key.
* It is available at these permalinks, some with SSL certificates.
    https://togami.com/~warren/warren-gpg-transition-2013.txt.asc
    http://wtogami.fedorapeople.org/warren-gpg-transition-2013.txt.asc
    https://people.apache.org/~wtogami/warren-gpg-transition-2013.txt.asc
* Contact me privately via e-mail, XMPP, and/or phone to verify more.

How to Sign and Upload your GPG Signature
=========================================
gpg --recv-key 347DC10D
gpg --edit-key 347DC10D
sign
save
gpg --send-key 347DC10D

If you trust me, and you are satisfied that my identity is verified,
then I appreciate your signatures.  Please understand though that I
will not counter-sign other people without extensive proof of identity.
If I have reason to know you well from real-life I am more likely to
go through the effort of verifying your identity with whatever means
you are able to provide.

Warren Togami
wtogami@gmail.com
warren@togami.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.13 (GNU/Linux)

iEYEARECAAYFAlHQyggACgkQa93+jlSirPGQ5gCfb2TbLntTKx8Y8w2JQVBdtplc
ZLcAnAzhHfsp5FiGq8Drv06j8K4WjyFEiQQcBAEBAgAGBQJR0MoIAAoJELEXnrc0
fcEN1n0gAL8bv0Fw/In5iyhDagV0/yUnebaUO6Z+/zJrI+Y7ghfprmzNrFvDLDDR
uUh0MsaVbMkWysF7WXLN5HvYr6j5VRm3uzr6uGqhvTlxk8qUFVGhinDgzA66pTPJ
uUc7LZAcayFzpNgDNm96pgzrCuk73CCltVT3bz4lkS6BB58D/UThoRxbSnGpySdN
WcmI5JBAzDj9cM7HPqWqYIAfdI1aQH5nyT+n37+y0XjXdOS49HjN2Z3BHwc/mhRQ
pfYTIhREBf5P7etYZElz/fbFu8K5C8lc3X3zT3EStNCUYqvDZdYnEiZoWp66pW8o
U05Mk32pexFm2o2pWvNHEtT/hUs/F2VdZlpu/UGCaei4M2CDB8KvYEC21WAHuj/V
OiOmA9U11i027Jic9axabZibQWc2Jm+gxQSU+jIOrERVD9jIgVW4Cgx7GMKHDOVN
eSv6og6Y1AUNjnIVrGBIGWp2dpRiaWI7Eqy+QzEfk7hb0+717qmLN9+EmXd6GlNf
P6efUVNde0BUQxPNNrPWevAd6XyXFphi0tC7Eu6YVapcjLEW4kSj6KLlqZGiAmld
CAsexQum1ELIRFVNmjJphdjKa2ZmV0OwiKDyYbKcBub+bPjtrYRYd4BwsrRAa+lG
SQlW1tGQHFxRWrM4btOxGaALg4RN/G/zlGS8A6jThbRO+HzmfYGMRT3xYy1uBzmo
mi0FT3KGuDQ/0aDpE5FoOBa1ldrE48TN5Zt0qQyqGPoYawZ+3UrlBJmLrpHgtQlJ
vzZmvrj6MDK9kRqzmdm2oOGIXWjxdWdbxV4r1JhMUdHcIxrOTYkDNJi7UW1fr9Y8
w4HlHMNGfyRJ5bcAnFrPVz4/0eQ7InQVSyc2/MEt+y98TccT66o66MLOXm8NORiZ
YSqwUh+LnESiaeJrMeX0ZhjI5lCMf4AERHxciOVVzE5Pj8MLggFMcP2YzXXK+br1
SxJ4ePjZI+sgO35IK+kJup8m+ejF4F2oD0oTZqDlpCAfWs5tStutDdflxxR6iXF7
YsILFyjRTjD61bCq/CcKWJJn5M1ATXLOnXHWGlnA/Zj+bQHspt5o6G4/0jjtxcgO
GGaVInEvxlzmp3/uuU/ehbyDETfV/nKnmzWrbK38Fp+T/4ExxLQVsqkAQURODfCM
9v5eKKEaqrncoBmSih7IaOMcqv8w8hsl9W3enZfRM2RvvtlPzKeGeIbp6duYP9KC
Tk9/LriR7BFkL6kCJzjIKoOVHcmDNxaM6xl42ZeWx+bUOWSD4X87Rv127o4DLqPV
ekdq3WCr4sO2wp+D1lYNtZ+9f5w+DM9eA5qg9tUkvHg7eqMdq1GBZ06P1tRRpgi+
XsFsFQlXNDR0nKxS/GBOQdW2OoukcvI=
=hPFO
-----END PGP SIGNATURE-----

Thursday, May 16, 2013

Gitian for Fedora

These RPM packages should allow Gitian VM-based builds to work on Fedora 25.
  • apt-cacher-ng (shipped in Fedora)
  • python-vmbuilder [Link] - updated March 6th for Fedora 25
Updated: March 6th, 2017

Notes:

  • Recently Fedora seems to have locked down /tmp mount permissions with options nodev and nosuid.  To use Gitian's make-base-vm as-is you could temporarily change the /tmp mount permissions.  After you install the base image you no longer need these less protective restrictions on /tmp.
    • Temporary workaround: mount -o remount,dev,suid /tmp
    • Undo workaround: mount -o remount,nodev,nosuid /tmp

Tuesday, February 5, 2013

Planning for LTSP EPEL-6 and Fedora

Joshua Trimm (FAS: enslaver) has joined the K12Linux project, and is currently working on formal integration of LTSP for EL-6. It is our intent for EPEL-6 to eventually contain all components of LTSP. After EPEL-6 is complete, Fedora may be considered. I have largely moved on from this project, but I am helping the transition to new developers. Joshua is doing at least EPEL-6 since his employer relies upon it. In the long-term K12Linux needs more knowledgeable Fedora developers in order to be sustainable.

See this post on epel-devel-list for technical details of what is planned.


Friday, February 1, 2013

SleepyHead for Fedora

SleepyHead is an open source tool that allows visualization of sleep tracking data from  CPAP machines and Oximeters.  This RPM package below is tested and working on Fedora 18.  It needs more work before it can be suitable for Fedora package review.

Download RPM: [Fedora 18]
Fedora RPM TODO
  1. Disable auto version check
  2. .desktop launcher with sheep icon
  3. Clarify GPL version (see README, currently ambiguous)

Wednesday, December 19, 2012

Amazon Instant Video on Fedora


These instructions will fix Adobe Flash Player DRM to allow use of Amazon Instant Video on Fedora 20+ or RHEL7.  This works with 32bit or 64bit flash-plugin-11.2+ from Adobe in Firefox.

Update September 9th, 2014:
fakehal-0.5.14-8 below should now automatically work on Fedora 20 or RHEL7 to allow Firefox to play Amazon Instant Video.

Step 1: Install FakeHAL
Adobe Flash Player for some reason uses HAL for its DRM to function, but HAL was removed from the operating system in Fedora 16.  The fakehal packages below contain a stripped down subset of hal necessary to allow this to work on newer versions of Fedora.  If your firefox and flash-plugin are x86_64, then you will want the corresponding x86_64 fakehal and fakehal-libs packages.

Wednesday, April 11, 2012

KVM with Spice USB Redirection on Fedora 16

After some initial trouble I managed to get Windows 7 32bit working in KVM on Fedora 16 on my Thinkpad T410s.  For the most part it works except for a strange issue where hardware virt works only *once*.  If I shutdown the guest, I need to reboot the laptop for hardware virt to work again.  I then struggled to use USB storage devices in the Windows guest as is often required in school.  It turns out that the USB passthrough in virt-manager is only capable of USB 1.0 and is ultimately unusable due to its fragility.  PCI passthrough for my sdio slot did not work either.

Windows 7 KVM guest with Spice USB Redirection on Fedora 16 host
Fortunately, Spice's USB network redirection seems to work great.  virt-manager is not capable of configuring or using it directly, but following these directions and using the spicy client, I managed to redirect USB 2.0 devices to my KVM guest.  It even has an option to automatically redirect newly plugged devices, and it does so intelligently to the host or guest depending which window has keyboard focus at that moment.

spicy client seems better than spicec or virt-manager, although Shift-F11 to escape full screen mode seems to be broken.  Until this bug is fixed, a temporary workaround is to CTRL-ALT-F2 and kill spicy in order to escape.  UPDATE: It seems Shift-F11 fails to both enter and exit full screen mode.  Full screen can only be entered from the View menu of spicy.  However Shift-F12 to ungrab the mouse does successfully escape from full screen spicy.

Tuesday, March 27, 2012

Windows 7 KVM Trouble on Fedora 16

For the past school year I have been forced to use Windows for various proprietary applications that we are required to use in MBA school. =(  I have been running Fedora within VMWare Player within Windows 7 on my Thinkpad T410s.  Today I wiped my laptop, installed Fedora 16, and I am attempting to install a KVM guest of Windows 7 x64 from an ISO image using these instructions.  Unfortunately, shortly after the ISO boots in KVM it fails with this BSOD.

 
A quick Google search reveals seemingly nothing related to KVM.  On a hunch, I ran the qemu-kvm command that libvirt launched, and it printed this error message: KVM not supported for this target

There were some Fedora 15 era bugs with this error, but nothing more recent.  BIOS has Virtualization enabled, and lsmod shows kvm and kvm_intel loaded.  After reloading kvm_intel, I see this kmsg:
kvm: VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL does not work properly. Using workaround

Oddly enough, the previous KVM not supported for this target message no longer happens.  This however has no effect on the BSOD.

Has anyone successfully installed Windows 7 in Fedora 16 KVM?

UPDATE: Windows 7 x64 install worked in kvm on RHEL6 on a different machine.  For some reason, only Windows 7 x32 works on my Thinkpad T410s with Fedora 16 x86_64.  Makes no sense...

Friday, February 24, 2012

How to fix Android Chrome on CM9

Today Google pushed an update for Android Chrome Beta that broke its ability to run on many CM9 Android 4.0.3 devices.  It says "Chrome is not supported for your version of Android.  Version 4.0 (Ice Cream Sandwich) is the minimal supported version."

The procedure to manually fix Chrome Beta on your CM9 device is beyond the break.

Sunday, February 5, 2012

How to build Android on RHEL6 or Fedora

Google's instructions of how to build Android only detail how to setup your build environment on Ubuntu.  This post describes how to do the equivalent on RHEL6, CentOS 6, Scientific Linux 6 or Fedora 16+.

Monday, January 30, 2012

CM9 Android 4.0.3 for Samsung Epic 4G







Our team released a mostly feature complete Android 4.0.3 build for the Samsung Epic 4G on the Sprint network.  See the Epic CM team blog for more details.

Monday, January 23, 2012

Tools: repoclone, reporeset, repohardlink

I wrote tools repoclone, reporeset and repohardlink to copy and use multiple Android repos in parallel.  By using hardlinks in a git-native manner it clones repos much faster and saves a lot of storage.  [Documentation] [Git]

Tuesday, January 17, 2012

Hawaii Android Automatic Time Zone Bug

This post describes the automatic timezone setting bug that affects Hawaii and many other locations around the globe.  Included below are temporary and permanent workarounds to avoid this issue for existing phones in Hawaii.  I also discuss the proposed location-aware automatic timezone selector that has the greatest chance of fixing this for good for all timezones.

Wednesday, December 28, 2011

Android on Samsung Epic 4G Development Notes

This post contains notes relevant to the development of custom Android firmware for the Samsung Epic 4G on the Sprint network.  Our goal is to make Android 4.0 "Ice Cream Sandwich" fully supported on this phone.  This content has been moved to a dedicated blog here.

    Wednesday, November 30, 2011

    Cartoon - The Starbucks Experience

    Yesterday our MBA marketing team presented our Marketing Plan recommendations to a local business owner.  Part of the presentation was to explain modern marketing concepts including the customer experience and focus on the Job the customer is trying to do.  A frequently cited example is Starbucks.  Apparently McDonalds and Dunkin Donuts beat Starbucks in blind taste tests, yet the general public believes Starbucks to be of premium quality.  Starbucks' success ultimately is not from selling coffee, but rather the overall café experience.  The customer merely is renting space (their third place) as an escape from home or work.
    Thanks to Fedora Project's Máirín Duffy for her cartoon that helped tremendously in conveying this concept.  Our Professor liked it very much and suggested publishing it for use in marketing academia.  Thus she decided to release it under the Creative Commons BY-SA license.  Thanks Mo!

    (I now find myself sitting in Starbucks, renting space, as I write this blog entry.)

    Thursday, September 29, 2011

    Surprise - ISAS uses Linux

    A few days ago I took the International Survey of Adult Skills after my home address was randomly selected to participate.  They apparently want to determine relative levels of education and computer-use skills across households in 25 countries by using a random statistical sample.

    The survey taker had a government issued laptop to administer the test.  It was a large, no-brand name laptop running Windows Vista Basic.  After booting into Windows, they run some app that tracks the with the test takers' home addresses, then launches the test.  To my surprise, the test itself is within a VMWare Player virtual machine.  It booted what looked like some stripped down variant of Debian, then directly into X and the test interface.

    The test itself was fairly well designed... obviously some real thought was put into it.  It tests your ability to use a basic e-mail client, spreadsheet and understand search engine results.  Some tests were reading data from a spreadsheet or chart and calculating answers with a calculator.  Some were reading e-mail and inputting values into a spreadsheet.

    The test did have some problems.
    • Software bugs ... often clicking would fail to work.  You would need to figure out workarounds, often involving clicking away then re-entering a part of the UI, to make it work as intended.  These bugs often made it frustrating to participate, and probably made me take 25% longer to complete the test.  I can see this screwing up their end results as some test takers may just give up due to bugs rather than demonstrate skills that they know.
    • I noticed several instances where the on-screen instructions had typos, ambiguously written instructions, or ambiguous text to read and interpret in order to understand the goal of a particular test.  This made me wonder if the ambiguity combined with software bugs were INCLUDED INTENTIONALLY as a means to test real-world problem solving skills.  If so, this might be a clever design.
    • But then I noticed another problem that surely cannot be intended.  The survey taker said it is supposed to be administered identically and in English in the 25 countries.  I saw several things in the test that were obviously American-centric, and I highly doubt they will be able to successfully test in 25 countries in English.
    • The survey administrator complained that the laptops were EXTREMELY SLOW and often the VMWare Player would get stuck during shutdown, preventing her from collecting the results and moving on.
    It was surprising that this government study used Linux for the test interface.  It seems clear that they used Linux as a method of deploying many identical copies of the survey interface across the world at the lowest cost possible.  It is strange that they would use Linux within Windows, when Windows seemed to be the cause of the extreme slowness.  Between Windows bootup and lots of time wasted fighting the test UI, it seems we wasted ~45 minutes.  That kind of time wasted can really add up to big money when multiplied across many thousands of test takers.  As a whole it seems well thought out, but they could have made the overall program far more effective had they fixed a few of these annoying and time consuming bugs.