Monday, July 7, 2014

SELinux problems with Erlang on Fedora/RHEL7

couchdb is one of several widely used server applications written in erlang.  I was surprised to find hundreds of Bugzilla tickets for erlang services and selinux-policy.

Problem #1: Erlang is treated by SELinux as part of the RabbitMQ Service
Erlang is a general purpose VM like Java.  However selinux-policy treated core components of Erlang as part of the rabbitmq semodule.
/usr/lib/erlang/erts.*/bin/beam.*       --      gen_context(system_u:object_r:rabbitmq_beam_exec_t,s0)
/usr/lib/erlang/erts.*/bin/epmd --      gen_context(system_u:object_r:rabbitmq_epmd_exec_t,s0)

Apparently someone attempted to make SELinux work only for rabbitmq, erroneously thinking that the generic erlang runtime binaries are part of rabbitmq.

# ls -lZ /usr/lib64/erlang/erts-5.10.4/bin 
-rwxr-xr-x. root root system_u:object_r:rabbitmq_beam_exec_t:s0 beam
-rwxr-xr-x. root root system_u:object_r:rabbitmq_beam_exec_t:s0 beam.smp
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       child_setup
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       ct_run
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       dialyzer
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       dyn_erl
-rwxr-xr-x. root root system_u:object_r:rabbitmq_epmd_exec_t:s0 epmd
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       erl
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       erlc
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       erlexec
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       erl.src
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       escript
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       heart
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       inet_gethost
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       run_erl
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       start
-rw-r--r--. root root system_u:object_r:lib_t:s0       start_erl.src
-rw-r--r--. root root system_u:object_r:lib_t:s0       start.src
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       to_erl
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       typer

Everything in this directory should be context bin_t.

Problem #2: Erlang Services Are Not Using Their Own Semodules!
The many erlang-related tickets filed against selinux-policy are confused because they are not actually running with the expected SELinux context.

# systemctl start  rabbitmq-server.service
# ps auxZ |grep rabbitmq
system_u:system_r:init_t:s0     rabbitmq   359 11.5  0.4 2177032 38972 ?       Ssl  20:18   0:01 /usr/lib64/erlang/erts-5.10.4/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.1.5/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@server -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@server.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@server-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.1.5/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@server-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@server"

The service is running as init_t. This is because  /usr/lib/rabbitmq/bin/rabbitmq-server is not properly labeled with something like rabbitmq_exec_t so it never transitions into its own semodule when launched from systemd.

Similarly couchdb.service launched itself via ExecStart=/usr/bin/erl (symlink /usr/lib64/erlang/erts-5.10.4/bin) and it too was running as init_t.  Lacking execve to anything labeled couchdb_exec_t, erl would fail to transition into  couchdb_t.  couchdb-1.6.0-9+ now runs from systemd with ExecStart=/usr/libexec/couchdb <parameters>.  If the wrapper shell script /usr/libexec/couchdb is labeled as couchdb_exec_t then couchdb service actually runs as couchdb_t.

Note: couchdb-1.6.0-9+ does not use the upstream /bin/couchdb script from systemd because it is rather broken and upstream plans on removing it in the next release.  The way we configure and exeucte couchdb from systemd is close to the future upstream standard way of handling the service.  While we do not use it for the systemd service it remains in the $PATH because users may depend on its behavior for debugging.

Temporary Workaround for CouchDB + SELinux
couchdb-1.6.0-9 may be the first Erlang application in Fedora to properly be confined by its own semodule(?)  We are waiting for selinux-policy to be fixed.  Meanwhile this temporary workaround will allow CouchDB to operate with SELinux enforcing enabled.  These instructions have been tested on Fedora 20 and RHEL 7.

Here are tested packages that I use on RHEL7 + EPEL7.  EPEL7 will soon have these packages.

#### Upgrade to couchdb-1.6.0-9 or later.
# sudo yum install -y selinux-policy-devel git
# git clone ||:
# cd refpolicy-contrib
# git checkout couchdb
# git pull
# make -f /usr/share/selinux/devel/Makefile couchdb.pp
# semodule -i couchdb.pp
# sudo restorecon -Rv /usr/lib64/erlang/erts-*/bin/
# sudo restorecon -v /usr/libexec/couchdb 
# sudo restorecon -v /usr/lib64/erlang/lib/couch-*/priv/couchjs
# sudo restorecon -v /usr/lib/systemd/system/couchdb.service

Erlang Service Packages
In general erlang packages with their own semodule must execute via a properly labeled wrapper script.  /usr/bin/erl on its own is unable to guess and transition into the proper SELinux context.  See couchdb-1.6.0-9+ /usr/libexec/couchdb for an example.  As noted above, as rabbitmq never was running with the proper context someone who is familiar with rabbitmq will need to ensure the policy continues to operate as expected.

SELinux Policy
Generally, selinux-policy must no longer treat core Erlang binaries as part of rabbitmq.  Everything in the /usr/lib*/erlang/erts-*/bin/ directory should be bin_t, perhaps somewhere in the core policies.

Every server application with its own semodule will need the aforementioned wrapper script(s) with context <something>_exec_t.  For example these rabbitmq binaries need to be properly labeled.

ls -Z /usr/lib/rabbitmq/lib/rabbitmq_server-3.1.5/sbin
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       rabbitmqctl
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       rabbitmq-defaults
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       rabbitmq-env
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       rabbitmq-plugins
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       rabbitmq-server

I fixed the couchdb semodule to work with couchdb-1.6.0-8+.  rabbitmq and other packages may need specific fixes to their policies and perhaps also to their packages.

July 9th, 2014
  • Updated couchdb policy and improved workaround instructions.
  • couchdb-1.6.0-9.fc20 coming soon for F19 and F20, and EL7.
  • Updated couchdb policy below
    • Fixed all AVC's in diskup
    • Added missing restorecon to workaround instructions.
    • Added unix_dgram_socket create needed for systemd notify

Sunday, June 30, 2013

Transition to new GPG Key

Hash: SHA1

Dear Internet,

My old GPG key is quite old and 1024bit DSA.  So it is time for me to transition
to a new GPG key.  If you know me well, trust me or have signed me in the past,
I need help in re-establishing my Web-of-Trust linkages.
pub   1024D/54A2ACF1 2002-11-25
      Key fingerprint = 785A 304B 08C1 F291 F54F  9A68 6BDD FE8E 54A2 ACF1
uid                  Warren Togami (Linux) <>
sub   2048g/4AD75982 2002-11-25
pub   8192R/347DC10D 2013-06-29
      Key fingerprint = AEC1 8843 9864 7C47 413C  1C3F B117 9EB7 347D C10D
uid                  Warren Togami (2013) <>
sub   8192R/668709D4 2013-06-29

Warren's OTR Fingerprint (XMPP): 3DF16C5E 8ABB4BBD 0B43EBF0 52B02ABA A2076B25

Verifying Warren's Identity in Multiple Difficult-to-Fake Ways
* This message is GPG signed by both the old and new key.
* It is available at these permalinks, some with SSL certificates.
* Contact me privately via e-mail, XMPP, and/or phone to verify more.

How to Sign and Upload your GPG Signature
gpg --recv-key 347DC10D
gpg --edit-key 347DC10D
gpg --send-key 347DC10D

If you trust me, and you are satisfied that my identity is verified,
then I appreciate your signatures.  Please understand though that I
will not counter-sign other people without extensive proof of identity.
If I have reason to know you well from real-life I am more likely to
go through the effort of verifying your identity with whatever means
you are able to provide.

Warren Togami
Version: GnuPG v1.4.13 (GNU/Linux)


Thursday, May 23, 2013

Fedora Build Dependencies for Bitcoin

Updated: March 11th, 2017
boost-devel libdb4-devel libdb4-cxx-devel miniupnpc-devel qrencode-devel protobuf-devel autoconf automake libdb4-cxx-devel libtool make qt5-qttools-devel qt5-qtbase-devel libevent-devel

Thursday, May 16, 2013

Gitian for Fedora

These RPM packages should allow Gitian VM-based builds to work on Fedora 25.
  • apt-cacher-ng (shipped in Fedora)
  • python-vmbuilder [Link] - updated March 6th for Fedora 25
Updated: March 6th, 2017


  • Recently Fedora seems to have locked down /tmp mount permissions with options nodev and nosuid.  To use Gitian's make-base-vm as-is you could temporarily change the /tmp mount permissions.  After you install the base image you no longer need these less protective restrictions on /tmp.
    • Temporary workaround: mount -o remount,dev,suid /tmp
    • Undo workaround: mount -o remount,nodev,nosuid /tmp

Tuesday, February 5, 2013

Planning for LTSP EPEL-6 and Fedora

Joshua Trimm (FAS: enslaver) has joined the K12Linux project, and is currently working on formal integration of LTSP for EL-6. It is our intent for EPEL-6 to eventually contain all components of LTSP. After EPEL-6 is complete, Fedora may be considered. I have largely moved on from this project, but I am helping the transition to new developers. Joshua is doing at least EPEL-6 since his employer relies upon it. In the long-term K12Linux needs more knowledgeable Fedora developers in order to be sustainable.

See this post on epel-devel-list for technical details of what is planned.

Friday, February 1, 2013

SleepyHead for Fedora

SleepyHead is an open source tool that allows visualization of sleep tracking data from  CPAP machines and Oximeters.  This RPM package below is tested and working on Fedora 18.  It needs more work before it can be suitable for Fedora package review.

Download RPM: [Fedora 18]
  1. Disable auto version check
  2. .desktop launcher with sheep icon
  3. Clarify GPL version (see README, currently ambiguous)

Wednesday, December 19, 2012

Amazon Instant Video on Fedora

These instructions will fix Adobe Flash Player DRM to allow use of Amazon Instant Video on Fedora 20+ or RHEL7.  This works with 32bit or 64bit flash-plugin-11.2+ from Adobe in Firefox.

Update September 9th, 2014:
fakehal-0.5.14-8 below should now automatically work on Fedora 20 or RHEL7 to allow Firefox to play Amazon Instant Video.

Step 1: Install FakeHAL
Adobe Flash Player for some reason uses HAL for its DRM to function, but HAL was removed from the operating system in Fedora 16.  The fakehal packages below contain a stripped down subset of hal necessary to allow this to work on newer versions of Fedora.  If your firefox and flash-plugin are x86_64, then you will want the corresponding x86_64 fakehal and fakehal-libs packages.

Wednesday, April 11, 2012

KVM with Spice USB Redirection on Fedora 16

After some initial trouble I managed to get Windows 7 32bit working in KVM on Fedora 16 on my Thinkpad T410s.  For the most part it works except for a strange issue where hardware virt works only *once*.  If I shutdown the guest, I need to reboot the laptop for hardware virt to work again.  I then struggled to use USB storage devices in the Windows guest as is often required in school.  It turns out that the USB passthrough in virt-manager is only capable of USB 1.0 and is ultimately unusable due to its fragility.  PCI passthrough for my sdio slot did not work either.

Windows 7 KVM guest with Spice USB Redirection on Fedora 16 host
Fortunately, Spice's USB network redirection seems to work great.  virt-manager is not capable of configuring or using it directly, but following these directions and using the spicy client, I managed to redirect USB 2.0 devices to my KVM guest.  It even has an option to automatically redirect newly plugged devices, and it does so intelligently to the host or guest depending which window has keyboard focus at that moment.

spicy client seems better than spicec or virt-manager, although Shift-F11 to escape full screen mode seems to be broken.  Until this bug is fixed, a temporary workaround is to CTRL-ALT-F2 and kill spicy in order to escape.  UPDATE: It seems Shift-F11 fails to both enter and exit full screen mode.  Full screen can only be entered from the View menu of spicy.  However Shift-F12 to ungrab the mouse does successfully escape from full screen spicy.

Tuesday, March 27, 2012

Windows 7 KVM Trouble on Fedora 16

For the past school year I have been forced to use Windows for various proprietary applications that we are required to use in MBA school. =(  I have been running Fedora within VMWare Player within Windows 7 on my Thinkpad T410s.  Today I wiped my laptop, installed Fedora 16, and I am attempting to install a KVM guest of Windows 7 x64 from an ISO image using these instructions.  Unfortunately, shortly after the ISO boots in KVM it fails with this BSOD.

A quick Google search reveals seemingly nothing related to KVM.  On a hunch, I ran the qemu-kvm command that libvirt launched, and it printed this error message: KVM not supported for this target

There were some Fedora 15 era bugs with this error, but nothing more recent.  BIOS has Virtualization enabled, and lsmod shows kvm and kvm_intel loaded.  After reloading kvm_intel, I see this kmsg:
kvm: VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL does not work properly. Using workaround

Oddly enough, the previous KVM not supported for this target message no longer happens.  This however has no effect on the BSOD.

Has anyone successfully installed Windows 7 in Fedora 16 KVM?

UPDATE: Windows 7 x64 install worked in kvm on RHEL6 on a different machine.  For some reason, only Windows 7 x32 works on my Thinkpad T410s with Fedora 16 x86_64.  Makes no sense...

Friday, February 24, 2012

How to fix Android Chrome on CM9

Today Google pushed an update for Android Chrome Beta that broke its ability to run on many CM9 Android 4.0.3 devices.  It says "Chrome is not supported for your version of Android.  Version 4.0 (Ice Cream Sandwich) is the minimal supported version."

The procedure to manually fix Chrome Beta on your CM9 device is beyond the break.

Sunday, February 5, 2012

How to build Android on RHEL6 or Fedora

Google's instructions of how to build Android only detail how to setup your build environment on Ubuntu.  This post describes how to do the equivalent on RHEL6, CentOS 6, Scientific Linux 6 or Fedora 16+.

Monday, January 30, 2012

CM9 Android 4.0.3 for Samsung Epic 4G

Our team released a mostly feature complete Android 4.0.3 build for the Samsung Epic 4G on the Sprint network.  See the Epic CM team blog for more details.

Monday, January 23, 2012

Tools: repoclone, reporeset, repohardlink

I wrote tools repoclone, reporeset and repohardlink to copy and use multiple Android repos in parallel.  By using hardlinks in a git-native manner it clones repos much faster and saves a lot of storage.  [Documentation] [Git]

Tuesday, January 17, 2012

Hawaii Android Automatic Time Zone Bug

This post describes the automatic timezone setting bug that affects Hawaii and many other locations around the globe.  Included below are temporary and permanent workarounds to avoid this issue for existing phones in Hawaii.  I also discuss the proposed location-aware automatic timezone selector that has the greatest chance of fixing this for good for all timezones.

Wednesday, December 28, 2011

Android on Samsung Epic 4G Development Notes

This post contains notes relevant to the development of custom Android firmware for the Samsung Epic 4G on the Sprint network.  Our goal is to make Android 4.0 "Ice Cream Sandwich" fully supported on this phone.  This content has been moved to a dedicated blog here.

    Wednesday, November 30, 2011

    Cartoon - The Starbucks Experience

    Yesterday our MBA marketing team presented our Marketing Plan recommendations to a local business owner.  Part of the presentation was to explain modern marketing concepts including the customer experience and focus on the Job the customer is trying to do.  A frequently cited example is Starbucks.  Apparently McDonalds and Dunkin Donuts beat Starbucks in blind taste tests, yet the general public believes Starbucks to be of premium quality.  Starbucks' success ultimately is not from selling coffee, but rather the overall café experience.  The customer merely is renting space (their third place) as an escape from home or work.
    Thanks to Fedora Project's Máirín Duffy for her cartoon that helped tremendously in conveying this concept.  Our Professor liked it very much and suggested publishing it for use in marketing academia.  Thus she decided to release it under the Creative Commons BY-SA license.  Thanks Mo!

    (I now find myself sitting in Starbucks, renting space, as I write this blog entry.)

    Thursday, September 29, 2011

    Surprise - ISAS uses Linux

    A few days ago I took the International Survey of Adult Skills after my home address was randomly selected to participate.  They apparently want to determine relative levels of education and computer-use skills across households in 25 countries by using a random statistical sample.

    The survey taker had a government issued laptop to administer the test.  It was a large, no-brand name laptop running Windows Vista Basic.  After booting into Windows, they run some app that tracks the with the test takers' home addresses, then launches the test.  To my surprise, the test itself is within a VMWare Player virtual machine.  It booted what looked like some stripped down variant of Debian, then directly into X and the test interface.

    The test itself was fairly well designed... obviously some real thought was put into it.  It tests your ability to use a basic e-mail client, spreadsheet and understand search engine results.  Some tests were reading data from a spreadsheet or chart and calculating answers with a calculator.  Some were reading e-mail and inputting values into a spreadsheet.

    The test did have some problems.
    • Software bugs ... often clicking would fail to work.  You would need to figure out workarounds, often involving clicking away then re-entering a part of the UI, to make it work as intended.  These bugs often made it frustrating to participate, and probably made me take 25% longer to complete the test.  I can see this screwing up their end results as some test takers may just give up due to bugs rather than demonstrate skills that they know.
    • I noticed several instances where the on-screen instructions had typos, ambiguously written instructions, or ambiguous text to read and interpret in order to understand the goal of a particular test.  This made me wonder if the ambiguity combined with software bugs were INCLUDED INTENTIONALLY as a means to test real-world problem solving skills.  If so, this might be a clever design.
    • But then I noticed another problem that surely cannot be intended.  The survey taker said it is supposed to be administered identically and in English in the 25 countries.  I saw several things in the test that were obviously American-centric, and I highly doubt they will be able to successfully test in 25 countries in English.
    • The survey administrator complained that the laptops were EXTREMELY SLOW and often the VMWare Player would get stuck during shutdown, preventing her from collecting the results and moving on.
    It was surprising that this government study used Linux for the test interface.  It seems clear that they used Linux as a method of deploying many identical copies of the survey interface across the world at the lowest cost possible.  It is strange that they would use Linux within Windows, when Windows seemed to be the cause of the extreme slowness.  Between Windows bootup and lots of time wasted fighting the test UI, it seems we wasted ~45 minutes.  That kind of time wasted can really add up to big money when multiplied across many thousands of test takers.  As a whole it seems well thought out, but they could have made the overall program far more effective had they fixed a few of these annoying and time consuming bugs.