Message Concentrator

Truxton uses a message bus proxy to minimize the number of connections to the PostgreSQL database server. PostgreSQL creates a new process for every connection to the server. If you have 20 clients, you will have spawned 20 processes on the server. Since Truxton ETL processes use PostgreSQL as the message queue server, it doubles the number of processes on the server since ETLs use one connection for meta data and another for message bus. Using a concentrator reduces the number of connections to the message bus server to the number of machines connecting to the server. All ETL processes will connect to the local message bus server which will have a single connection to the real message bus server.

How It Works

The concentrator is hosted inside the Truxton Service and activated by setting the mqconcentrator option to true. This will cause Truxton to open a socket on port 2525 and wait for incoming connections. The concentrator will connect to the PostgreSQL message queue server and use this single connection to pass message traffic from all clients connected on port 2525.

Steps

To activate the message bus concentrator, you must tell the Truxton Service to be the concentrator and the ETLs to use it. The mqconcentrator setting tells the Truxton Service to be the server while the mbp setting tells the ETLs which message bus provider to use.

Detailed Steps

1. Stop the Truxton and Les services

net stop les
net stop truxton

2. Edit the TruxtonSettings.xml file in the C:\ProgramData\Truxton\Settings folder.

  <mqconcentrator>Yes</mqconcentrator>
  <mbp>MBCConnie.dll</mbp>

3. Start the Truxton Service

net start truxton

4. Start the Load Status Monitor Service

net start les

Remarks

The protocol spoken over port 2525 is not documented. This allows us to radically change it to meet future needs. Currently, the only message bus server we support is PostgreSQL implemented in the MBCPG.DLL library. However, all message bus functionality is passed through a common interface which contains no implementation details. This will allow us to change message bus providers without having to modify any existing exploitation ETLs. This makes it possible for us to switch messages to ZeroMQ, or Simple Queue Service (SQS), or Service Bus, or Firebase Cloud Messaging, or, well, you get the idea.