Problems with Saltstack on FreeBSD

TL;DR: Patch libzmq in your ports tree to fix the socket connection errors.

Saltstack generally works awesome on FreeBSD. But, if you're unlucky enough to actually keep your systems up-to-date, you'll eventually notice that your Salt minion will no longer start. If you start the minion by hand with debugging (-l debug), you'll see something like this:

2015-06-06 21:45:55,189 [salt.cli.daemons ][INFO    ][38573] Setting up the Salt Minion "www.epsilon.maio.me"  
2015-06-06 21:45:55,280 [salt.cli.daemons                         ][INFO    ][38573] The salt minion is shut down  
2015-06-06 21:45:55,282 [salt.cli.daemons                         ][INFO    ][38578] The salt minion is shut down  
2015-06-06 21:45:55,283 [salt.utils.process                       ][DEBUG   ][38579] Created pidfile: /var/run/salt-minion.pid  
2015-06-06 21:45:55,284 [salt.config                              ][DEBUG   ][38579] Reading configuration from /usr/local/etc/salt/minion  
2015-06-06 21:45:55,336 [salt.config                              ][DEBUG   ][38579] Including configuration from '/usr/local/etc/salt/minion.d/_schedule.conf'  
2015-06-06 21:45:55,336 [salt.config                              ][DEBUG   ][38579] Reading configuration from /usr/local/etc/salt/minion.d/_schedule.conf  
2015-06-06 21:45:57,472 [salt.minion                              ][DEBUG   ][38579] Attempting to authenticate with the Salt Master at 10.0.0.12  
2015-06-06 21:45:57,473 [salt.crypt                               ][DEBUG   ][38579] Initializing new SAuth for ('/usr/local/etc/salt/pki/minion', 'www.epsilon.maio.me', 'tcp://10.0.0.12:4506')  
2015-06-06 21:45:57,553 [salt.crypt                               ][DEBUG   ][38579] Decrypting the current master AES key  
2015-06-06 21:45:57,553 [salt.crypt                               ][DEBUG   ][38579] Loaded minion key: /usr/local/etc/salt/pki/minion/minion.pem  
2015-06-06 21:45:57,648 [salt.crypt                               ][DEBUG   ][38579] Decrypting the current master AES key  
2015-06-06 21:45:57,649 [salt.crypt                               ][DEBUG   ][38579] Loaded minion key: /usr/local/etc/salt/pki/minion/minion.pem  
2015-06-06 21:45:57,665 [salt.crypt                               ][DEBUG   ][38579] Loaded minion key: /usr/local/etc/salt/pki/minion/minion.pem  
2015-06-06 21:45:57,673 [salt.crypt                               ][DEBUG   ][38579] Re-using SAuth for ('/usr/local/etc/salt/pki/minion', 'www.epsilon.maio.me', 'tcp://10.0.0.12:4506')  
2015-06-06 21:45:57,688 [salt.crypt                               ][DEBUG   ][38579] Loaded minion key: /usr/local/etc/salt/pki/minion/minion.pem  
2015-06-06 21:45:57,696 [salt.config                              ][DEBUG   ][38579] Reading configuration from /usr/local/etc/salt/minion  
2015-06-06 21:45:57,746 [salt.config                              ][DEBUG   ][38579] Including configuration from '/usr/local/etc/salt/minion.d/_schedule.conf'  
2015-06-06 21:45:57,747 [salt.config                              ][DEBUG   ][38579] Reading configuration from /usr/local/etc/salt/minion.d/_schedule.conf  
2015-06-06 21:45:59,765 [salt.utils.lazy                          ][DEBUG   ][38579] LazyLoaded timezone.get_offset  
2015-06-06 21:45:59,766 [salt.utils.lazy                          ][DEBUG   ][38579] LazyLoaded cmd.run  
2015-06-06 21:45:59,766 [salt.loaded.int.module.cmdmod            ][INFO    ][38579] Executing command 'date +%z' in directory '/'  
2015-06-06 21:45:59,790 [salt.loaded.int.module.cmdmod            ][DEBUG   ][38579] output: -0500  
2015-06-06 21:45:59,795 [salt.utils.lazy                          ][DEBUG   ][38579] LazyLoaded config.merge  
2015-06-06 21:45:59,796 [salt.utils.lazy                          ][DEBUG   ][38579] LazyLoaded mine.update  
2015-06-06 21:45:59,796 [salt.minion                              ][INFO    ][38579] Added mine.update to scheduler  
2015-06-06 21:45:59,796 [salt.utils.schedule                      ][INFO    ][38579] Updating job settings for scheduled job: __mine_interval  
2015-06-06 21:45:59,798 [salt.minion                              ][DEBUG   ][38579] I am www.epsilon.maio.me and I am not supposed to start any proxies. (Likely not a problem)  
2015-06-06 21:45:59,798 [salt.cli.daemons                         ][INFO    ][38579] The salt minion is starting up  
2015-06-06 21:45:59,798 [salt.minion                              ][INFO    ][38579] Minion is starting as user 'root'  
2015-06-06 21:45:59,798 [salt.minion                              ][DEBUG   ][38579] Minion 'www.epsilon.maio.me' trying to tune in  
2015-06-06 21:45:59,799 [salt.minion                              ][DEBUG   ][38579] Minion PUB socket URI: ipc:///var/run/salt/minion/minion_event_1f6967323f_pub.ipc  
2015-06-06 21:45:59,799 [salt.minion                              ][DEBUG   ][38579] Minion PULL socket URI: ipc:///var/run/salt/minion/minion_event_1f6967323f_pull.ipc  
2015-06-06 21:45:59,799 [salt.minion                              ][INFO    ][38579] Starting pub socket on ipc:///var/run/salt/minion/minion_event_1f6967323f_pub.ipc  
2015-06-06 21:45:59,799 [salt.minion                              ][INFO    ][38579] Starting pull socket on ipc:///var/run/salt/minion/minion_event_1f6967323f_pull.ipc  
2015-06-06 21:45:59,800 [salt.minion                              ][DEBUG   ][38579] Generated random reconnect delay between '1000ms' and '11000ms' (8638)  
2015-06-06 21:45:59,800 [salt.minion                              ][DEBUG   ][38579] Setting zmq_reconnect_ivl to '8638ms'  
2015-06-06 21:45:59,800 [salt.minion                              ][DEBUG   ][38579] Setting zmq_reconnect_ivl_max to '11000ms'  
2015-06-06 21:46:03,058 [salt.cli.daemons                         ][INFO    ][38579] The salt minion is shut down  
2015-06-06 21:46:03,058 [salt.scripts                             ][ERROR   ][38579] Invalid argument  
2015-06-06 21:46:03,058 [salt.scripts                             ][WARNING ][38579] ** Restarting minion **  
2015-06-06 21:46:03,058 [salt.scripts                             ][INFO    ][38579] Sleeping random_reauth_delay of 2 seconds  

After research and experimentation, I found a new solution that seems to work.

There's a small bug in libzmq4 that causes the socket handler to fail on IPv6 addresses. On FreeBSD, you can drop this patch into net/libzmq4/files/patch-src_socket__base.cpp in your ports tree and rebuild. Then everything should work beautifully!

Thanks to Barnerd in #freebsd-python on Freenode for pointing me in the right direction for installing the patch!

Sean Johnson

Software developer, music enthusiast, college student, and cat lover. Currently enthralled by the world of networking, security, and cloud technology.

San Antonio, TX https://blog.maio.me

Subscribe to Land of Weird Things

Get the latest posts delivered right to your inbox.

or subscribe via RSS with Feedly!