IsectMP

Multi-platform, Multi-programming language, Multi-plexing middleware for Windows programmers

IsectMP's history

IsectMP is Isectd (http://sourceforge.net/projects/isectd/) with all the goodies attached that a Windows programmer might need. Although Isectd was originally developed for Unix, I ported it to Windows approximately 10 years ago and have been using it in my Windows projects ever since. Unfortunately, Isectd's lead developer Tom Gagne, appeared to lose interest in his project not long after the Windows port was made and while the source files on SourceForge contain the Windows port the release files (isect-1.6.2.tar.gz) do not. Isectd has lain dormant on SourceForge ever since.

I think a lot more Windows programmers could use a tool like Isectd. You can download it from my Downloads page.

What is Isectd?

Isectd comprises four files:

1. Client - something you code
• The client is your front end application. It could be anything, a GUI, a console app, a Web server. The only thing that matters is that the client sends data (a message) to the message router and then waits for a reply.
2. Message router (Isectd)
• When Isectd is started it is intialised to create a series of queues. These are referred to as services. Client messages are flagged as belonging to a particular service. When Isectd receives them it allocates them to the relevant queue.
3. Remote execution daemon (Execd)
• Execd starts workers on the machine it resides on as child processes. Execd's machine does not have to be the same machine where Isectd resides. After starting a worker Execd's job is complete. However the worker calls back to Isectd and registers itself as belonging to a particular service.
4. Worker – something you code
• As it's name suggests workers do all the heavy lifting in the client server architecture. Examples include database connectors, fax daemons and business rule modules. A collection of identical workers belongs to a queue which in Isectd is referred to as a service.

Concurrency

Isectd handles concurrency using separate processes not threads. A typical client server architecture using Isectd will have multiple clients, multiple workers, multiple Execd's but only one Isectd. Having a single instance of Isectd is not a bottleneck as all Isectd does is pass on small messages from clients to workers and back again. It does not process anything itself. The critical thing is that those messages must be small. As Isectd is single-threaded it's API expects that large messages will chopped into small chunks before sending and being concatenated at the other end.

Isectd's API

Isectd has a very simple API. Just five function calls are used:

Used by workers to attach to Isectd

• socket Attach(const char *, const char *, char *);

Used by clients to attach to Isectd

• socket Login(const char *, const char *, int);

Used by workers and clients

• void Logout(socket);
• bytes_sent Send(socket, char *, isdHeader *);
• char * Recv(socket, isdHeader *);

Send and Recv use a struct called isdHeader:

• typedef struct { /* offsets */
• int32_t len; /* 0 */
• int32_t sequence; /* 4 */
• int16_t reply; /* 8 */
• int16_t error; /* 10 */
• int16_t command; /* 12 */
• int16_t version; /* 14 */
• u_int32_t workerid; /* 16 */
• int16_t more; /* 20 */
• int16_t unused[17];
• double aligner;
• } isdHeader;

Inside isdHeader are three fields that enable Isectd to be clever:

• int32_t len. This tells you the length of the message fragment just received. It is a read-only field.
• int16_t reply. If set to non-zero value, Isectd will forward the reply from the worker back to the client. However, if reply is set to zero Isectd will dump any message returned by the worker, so the client is free to continue with whatever it is doing...I call this fire and forget messaging.
• int16_t more. More is like a long running transaction. As long as the field is non- zero, Isectd ensures that a worker assigned to a client cannot be sent a message by another client. Once the flag is set to zero, Isectd releases the worker back into the free workers pool.
• The remaining fields in isdHeader are not used.

*** Isectd no longer has an API ***

You no longer need the C API to communicate with Isectd. Instead you can send and receive binary strings to and from it's sockets in a similar way to a web server. Each message comprises a 48 byte header (isdHeader re-ordered to make it more efficient) followed by an arbitrary string of any length. The C API stills exists and will be used by C and C++ programmers. For other language users, rolling a few of your own functions in your language of choice is all that is necessary.

Removing the API makes Isectd easily usable by any language that implements sockets (implementing the isdHeader C struct in other languages' Foreign Function Interface (FFI) is hard). More importantly, the blocking problem is bypassed. A number of languages block as soon as any call is made to their FFI. I have dealt with three languages that do this - Python (Zope), Factor and Smalltalk. All of these use 'green' threads rather than OS threads for concurrency and their FFI blocks them.

Use tokenized strings as messages

The way to use Isectd is for your clients and workers to communicate using tokenized strings. A tokenized string can represent anything including binary data (base64 encoding) and that string can be chopped up into arbitrarily small pieces. As long as the pieces are sent and received sequentially they can be reassembled quickly. I use linked lists which are much quicker to iterate over than concatenating the pieces back together.

Isectd's command language

Isectd doesn't use configuration files. Instead it has it's own tiny command language. All aspects of it's configuration are achieved using the supplied console client (isdClient) that sends the commands to Isectd. The supplied console client is capable of reading a configuration file which it then forwards to Isectd's “op” service. It can also send “one-line” commands or interact with Isectd using the console.

Starting Isectd services

To start remote workers, the local console client first tells Isectd to set up a service. Then it tells Isectd the service, remote host address and file path of a worker. Isectd forwards this information to the appropriate Execd, who starts the worker. Once the worker is started it calls back to Isectd to tell it that it is ready. Once all workers are ready (currently Isectd does not keep track of whether all it's workers are ready or not. Instead a reasonable time delay must be incorporated to give the workers time to call back. XYNTService and Sleep.vbs handle the time delays) the local console client tells Isectd to start the service. Now the workers on that service are available to receive messages.

Starting Isectd as a Windows service

XYNTService can be used to start Isectd as a Windows service. XYNTService uses similar commands to those for starting Isectd from a batch file.

Where's the GUI for Isectd?

Check out isdClient. It's a console application rather than a GUI... this is a good thing... really! IsdClient is very powerful. After entering a command enter a new line and type “go” followed by another new line. IsdClient will send your command to Isectd. It also has a number of command line flags, some of which, are undocumented elsewhere:

isdClient.exe [-c command] [-d] [-g consolewidth] [-h host[:port]] [-i inputfile] [-k] [-p priority] [-r][servicename]

-c one line command

-d primitive debugging on isdClient

-g Windows console width

-h host:port

-i input file

-k header.more = 1

-p priority = ?

-r reply = ?

servicename if none is provided client defaults to “op” service, Isectd's command language

Examples of how I've used Isectd

1. I had an application processing application at a finance house written in Visual Basic attached to an Access database that sent out faxed replies. As the workload on the application increased it became apparent that Access could no longer cope with the load. I needed to implement some kind of queuing capability to utilize the multiple fax machines efficiently, which would also work with a client/server database. I chose Isectd and...
• wrote isectFirebird, the FirebirdSQL worker.
• wrote isectFax, a fax worker. isectFax called isectFirebird and populated itself with data from the FirebirdSQL database before sending the fax. Because sending a fax is asynchronous, the users sent a request to isectFax with no reply. Once the fax was sent (or not!) a table was updated in the database that the users could query periodically to see which faxes had failed. So, isectFax was both a worker and a client!
2. I wrote a Zope Python connector which utilised isectFirebird. Unfortunately, Zope's templating language turned out to be too slow for the job at hand. I also discovered that Zope was unable to run free threads to external services. Each time a request was made to the Zope connector, Zope blocked. The project was cancelled.
3. A real-time analyser created lots of data, but the database that accumulated the data was unable to provide remote, real-time reports to the staff. Wrote a polling ODBC data pump in Visual Basic that populated a FirebirdSQL database using isectFirebird located on the same machine as the analyser. Wrote another small Visual Basic graphing application that could be run from anywhere on the network to examine the analyser in real-time minus 5 minutes.
4. Wrote PocketClassmaker in C++. It talks to FirebirdSQL using isectFirebird.

Lessons I've learned

1. Do everything with tokenized strings.
2. Chose the best tool for the job. If you've got middleware in place then you can search around for the best tool available and it doesn't matter what OS or language it's written in. Middleware promotes the philosophy of “what's the quickest solution?” Without it, you're back to “we didn't build it here” mentality.
3. Middleware tools need to stay dumb. Keep complexity OUT of the API. Leave it up to the clients and workers to translate the messages they transmit. The complex API's used by CORBA and COM is why those middleware efforts failed. Conversely HTTP's API is very simple and the end result has been the massive adoption of Web applications everywhere.
4. Middleware speeds up development, since you don't need to keep reloading everything during the code/debug cycle. The previously debugged workers can remain up 24/7 while you test out some feature of a new client and because you create small applications that don't do too much, they remain easy to understand.
5. Middleware written in C with a simple API like Isectd has, can go on for years with minimal maintenance. I could easily imagine I'll still be using Isectd 10 years from now.