Telium Support Group

Seeing a status from systemctl that the Asterisk service is dead is not necessarily a problem. Systemd reports on services it starts & stops; and since HAAst does not start/stop Asterisk through systemd it’s normal to see a message like this. More specifically, the Linux distro + FreePBX distro + Asterisk combination will report different status results for individual services. So seeing a systemctl status report that something is dead/not started/started may be misleading.

If you are sure that the Asterisk service should be stopped (based on the status of the peer), then we recommend paring back your environment to a single simple PBX (strip down the layers). This will let you trace the problem down to a single cause:

1. Disable HAAst layer
   1. Power down the remote PBX (peer). Now we’re working with only the local peer (less variables to check)
   2. Disable the haast service and reboot the local peer
   3. After reboot check if Asterisk is running
   4. Check the haast log to ensure it did not start
   5. If HAAst and Asterisk are stopped lets try operating on FreePBX directly. (Proceed to step 2)
   6. If you got here that means you forgot to disable automatic start of Asterisk/FreePBX as outlined in the installation guide.
2. Test FreePBX layer directly
   1. Start FreePBX with ‘fwconsole start’.
   2. Were there any warnings?
   3. Is Asterisk running normally?
   4. Now stop FreePBX with ‘fwconsole stop’
   5. Were there any warnings?
   6. Is Asterisk stopped?
   7. If you see FreePBX errors correct them and reboot the PBX and return to step 2A
   8. if you don’t see any errors then HAAst is having trouble controlling FreePBX. Check the ‘distribution’ setting in the haast.conf file
   9. If haast.conf settings are correct test the Asterisk layer directly
3. Test Asterisk layer directly
   1. Ensure the FreePBX service / start command is disabled as per the HAAst installation guide
   2. Reboot the PBX
   3. If asterisk is started that means you forgot to disable automatic start of Asterisk/FreePBX as outlined in the installation guide.
   4. Start and stop the asterisk service using ‘service’ or ‘systemctl’ commands appropriate for your distribution. Did this show an error?
   5. If you got to this point contact Telium support for assistance through SSH.

Most users find a FreePBX internal problem (eg: ) and upon resolution all works fine again. If you are new to FreePBX you will discover (google) that these types of problems are with FreePBX are well documented. FreePBX may fail to start, fail to stop, etc. which also blocks the Asterisk process from starting/stopping. There is nothing Telium/HAAst can do about this (i.e. FreePBX issue). But if you encounter symptoms as described in the original question then try this procedure to help diagnose the problem.

Telium can offer some suggestions on diagnosing such FreePBX issues but tracking down the cause of FreePBX error messages can be time consuming. Please note that other distributions such as xCALLY Motion do not encounter this type of problem.

Asterisk should not be running after a peer has switched to the standby state. This problem usually occurs if the wrong distribution number is selected in the [asterisk] stanza of haast.conf

Please ensure that you are using the correct distribution number in haast.conf. If you change the distribution number then you must restart the computer (since some poorly written configuration generators remain in an unstable state if not shut down the way they want to).

Yes – we have created tools which use the REST API/web interface of phones (and base stations) to force immediate re-registration following a failover of the cluster. This solution is implementation specific however (make and model of phone, where phone IP data is stored, etc), so please contact Telium professional services who can design a solution for your specific needs.

This tool should be placed into Asterisk post-start event handler (in HAAst), which will cause the nurse phones to immediately reconnect to the active peer following cluster failover.

You don’t mention if you are using a configuration generator (eg: xCally Motion, FreePBX, Elastix, etc), so I’ll provide two answers:

Using A Configuration Generator
If you are using a configuration generator, then the easiest way to setup synchronization is to copy the appropriate sample sync config files into your HAAst configuration directory. For example, if you are using xCALLY Motion then you will need both the basic Asterisk configuration synchronization, as well as the xCALLY Motion configuration synchronization. So copy the following files:

• asteriskconfig.sync.conf
• xcally_motion.sync.conf

from the sample_files/synchronizations directory to the /etc/xdg/telium/haast.conf.d directory and then restart HAAst. (Do so on both peers). Your files and databases will start to synchronize now. (If you are using some other configuration generator, just replace the xCALLY Motion file with the most appropriate sample for your configuration generator).

Using Only Asterisk
If you are not using a configuration generator (as is often the case with large Asterisk installations of 1000 or more phone sets), then the easiest way to setup synchronization is to use one or more sample synchronization files as a starting point. To being with, copy the following file:

• asteriskconfig.sync.conf

from the sample_files/synchronizations directory to the /etc/xdg/telium/haast.conf.d directory (on both peers) for basic synchronization of the /etc/asterisk folder contents, ASTdb, and voicemail. After that, have a look at the sample *.sync.conf files for ideas of what else you can sync.

Customizing The Data Sent To The Standby Peer
If you need to customize content that is synchronized in (eg: to modify the dialplan/trunks/tftp files/etc for the local machine) then have a look at the sample_files/events directory for sample sync.stop.post* files which demonstrate how to perform post-sync customization. You can copy any of these sample files to your /usr/local/haast/events directory to enable them, but be sure to edit each event handler to match your unique system and needs. Note that the name of the event handler file must match the name of the sync job in order to be associated as a related event handler. (e.g. sync job ‘mysql’ would look for a file called ‘sync.stop.post.mysql’ to be linked as the post-sync event handler).

HAAst is load tested to approximately 50 call setups per second (3000 call setups per minute) – so that should not be a problem. The constraint will most likely be the hardware upon which you run Asterisk. HAAst is written entirely in C++ and CPU requirements are generally very low. If in doubt conduct a HAAst trial and monitor CPU load and memory utilization to determine how many setups per minute your hardware can handle.

If you need assistance determining the call volume capacity of your hardware please contact Telium support for assistance.

You are welcome to download the latest software from the Telium website, and overwrite your installation. Please be sure to carefully compare the included config file to check for changes in stanzas, key names, new keys, etc.

If you are running the Free Edition the new software will work immediately. If you have an active maintenance agreement then the upgrade will work as well.

However, if your maintenance agreement has expired then the upgrade will revert to the Free Edition. Since you cannot extend a maintenance agreement that has expired you would have to purchase a new license.

If your maintenance agreement is still active then you can extend it for as long as you like (purchased in 1 year increments from the BUY tab on the Telium web site).

First of all make sure you know which system is generating the error (the host running HAAst, or the host which the post-sync script is connecting to). That may change the answer, and/or where you need to apply changes.

One of these solutions should help:

1. Try adding the “-t” parameter to the ssh command to create a pseudo terminal for the command to run in.
2. Instead of “-t” try “-tt” to force a tty (even if ssh has no local tty).
3. Remove sudo completely from the command in your bash script. Since the HAAst (service) runs as root the processes it forks also run as root (unless the system is further locked down). Assuming error is locally generated.
4. Comment out the line “Defaults requiretty” in the /etc/sudoers file. Then sudo won’t require a TTY anymore for sudo commands. (But undstand security implications).
5. Use the “su –c” command instead of sudo

The best way to communicate programmatically with HAAst is through the socket interface. When haast is running it creates a socket in the directory /var/run. This socket looks and acts just like a regular file; you can write to the file and read from the file to command HAAst and gather status information.

To retrieve the status of the cluster via the socket interface, use the ‘getstatus’ command. The pseudocode to do so is as follows:


Open file /var/run/haast.sock as read+write
Read ‘ready>’ prompt from file
Optionally write command ID and newline to file, e.g. ‘id: 123n’
Write get status command and newline to file, e.g. ‘command:getstatus\n\n’
Read response from file
Close file
Parse response


The ID sent can be any string (to uniquely identify the response) but is optional. In general we recommend sending an auto-incremented counter. Once all required parameters (of this command) have been sent, send a second newline (i.e. 2 sequential newline characters) to tell HAAst the command is complete and ready for processing.

The following is the actual output (read) in red color, and input (written) in blue color, from the above pseudo code:

haast socket interface v1.1
ready>id: 123
command:getstatus

id: 123
result: ok
cluster name: Telium
cluster start time: 1459816998033
cluster start time formatted: Mon Apr 4 20:43:18 2016
cluster run duration: 12404571
cluster run duration formatted: 71 days, 13 hours, 42 minutes, 51 seconds
cluster fail over count: 79
local peer name: PBX1 in Waterloo data center wall
local haast state: 4
local haast state formatted: Active
local peer health state: 1
local peer health state formatted: Normal
local asterisk state: 2
local asterisk state formatted: Started
local asterisk connection state: 7
local asterisk connection state formatted: Logged in
local start time: 1464643147663
local start time formatted: Mon May 30 17:19:07 2016
local haast run duration: 2740022
local haast run duration formatted: 15 days, 17 hours, 7 minutes, 2 seconds
local fail over count: 11
local previous fail over timestamp: 1465102805740
local previous fail over timestamp formatted: Sun Jun 5 01:00:05 2016
local previous fail over cause: 2
local previous fail over cause formatted: Dual standby peer contention detected
local previous haast state: 3
local previous haast state formatted: Standby
local to remote peerlink connection state: 7
local to remote peerlink connection state formatted: Up
remote data available: 1
remote peer name: PBX2 VM in Mississauga data center
remote haast state: 3
remote haast state formatted: Standby
remote peer health state: 1
remote peer health state formatted: Normal
remote asterisk state: 4
remote asterisk state formatted: Stopped
remote asterisk connection state: 4
remote asterisk connection state formatted: Connect failed
remote start time: 1464643407951
remote start time formatted: Mon May 30 17:23:27 2016
remote haast run duration: 2739761
remote haast run duration formatted: 15 days, 17 hours, 2 minutes, 41 seconds
remote fail over count: 8
remote previous fail over timestamp: 1465102804675
remote previous fail over timestamp formatted: Sun Jun 5 01:00:04 2016
remote previous fail over cause: 23
remote previous fail over cause formatted: Local peer automatic demotion request
remote previous haast state: 4
remote previous haast state formatted: Active
remote to local peerlink connection state: 7
remote to local peerlink connection state formatted: Up
ready>

To learn how to use the HAAst socket interface you can use the ‘socat’ command to interact with the HAAst socket from a Bash command line. (You may need to add the socat command / package to your particular Linux installation). The exact syntax for the socat command is as follows:


socat – UNIX-CONNECT:/var/run/haast.sock


You can then send any invalid command (eg: XXXX) and HAAst will respond with a list of valid commands.