theLink 10.0 NHI1 - theKernel - theLink - theConfig - theSq3Lite - theCompiler - theBrain - theGuard
c - tcl - py - rb - jv - cc
Loading...
Searching...
No Matches
HOWTO Filter - The Basics

What is a filter and how is the filter used for.

INTRODUCTION

The filter mode is related to a special usage of the theLink software called a command pipeline. To define a filter create a server with:

Every filter has two context one belongs to the left command and one belongs to the right command:

 <-- left cmd --> <------- filter -------> <-- right cmd -->

                  <-- left --><- right -->
                  <- server -><- client ->
                  <- master -><- slave -->
                  <-context1-><-context2->

  ... command1   @         filter         @   command2 ...

The left context is created on application startup and the right context is created as slave of the left context.

  • if the @ argument is followed by an normal command (server) a local pipeline is created:
    client @ filter @ server
    
  • if the @ argument is followed by an option a non-local pipeline is created:
    <------------ host-1 -------------> <-- network --> <---------- host-2 ----------->
           <---- client arguments ---->                       <--- server arguments -->
                    <-- filter arg. -->
                      <--- options --->
    
    client @ filter @ --tcp --port 7777   ...........   server --tcp --port 7777 --fork
    

BI-DIRECTIONAL FILTER

A bi-directional filter allow a data-flow in both directions and is used in a classical client/server application:

    client ... <--> ... server

as a feature enhancement like a protocol-tunnelling:

    client @ mq2tunnel ... <--> ... tunnel2mq @ server

or to convert the protocol into an other protocol:

    client @ mq2soap ... <--> ... soap-server

To define a bi-directional filter a couple of commands provide support:

  • MqServiceCreate
    • use the token +ALL to add a listener for all services. This feature is used for a tunnel to modify the body at all. (example: aguard)
  • MqServiceProxy
    • use this function to link the left context with the right context identified with the slave-identifier id (default: 0). No data manipulation is performed.
  • MqSlaveGetFilter, MqSlaveGetMaster, MqSlaveGetProxy
    • in a filter service the current context is used to read the data. To send the data an other context, belonging to the other site of the communication, have to be used. This function return the context of the other site.
  • MqServiceTokenGet
    • if the token +ALL is used in ServiceCreate to add a generic service handler the current token is not known. This function return the current token.
  • MqServiceIsTransaction
    • if the token +ALL is used in ServiceCreate to add a generic service handler the current transaction-status is not known. This function return the transaction-status as boolean with true (with-transaction) or false (without-transaction).
    • with-transaction: the package was send with MqSendEND_AND_WAIT or MqSendEND_AND_CALLBACK
    • without-transaction: the package was send with MqSendEND
  • MqDumpExport
    • read and return the read-data-package as MqDumpC. Use this dump to apply a transformation to the data, like encryption (example: aguard), or to save the body in a persistent storage for later use MqDumpImport.
  • MqDumpImport
    • load the MqDumpC into the read-data-package
  • MqProxyForward

ONE-DIRECTIONAL FILTER

A one-directional filter is a special form of a bi-directional filter and allow a data-flow from the left to the right.
This filter is well known from the unix shell to link different commands together:

    command1 | command2 | command3

A theLink command pipeline is created with the special character "@" instead of "|" :

    msgcmd1 @ msgcmd2 @ msgcmd3

To define a theLink filter... create a service handle with MqServiceCreate or MqServiceProxy ... using the token +FTR and +EOF

token description
+FTR required to act on filter data rows. Every filter input data is a list of filter data
rows and every row is a list of filter data columns. Every row is send to the following
filter-command as +FTR service request
+EOF required to act on End-Of-Filter data and is called after all +FTR data was send.
Sometimes the filter data can not be served as +FTR data (example: sorting of the
input rows need to read all rows before the data can be send to the next filter command)
and the+EOF token is used to continue send +FTR data rows.

and send every data item with MqSendEND_AND_WAIT .

BI-DIRECTIONAL FILTER

The following code is based on the JAVA example example/java/Filter3.java:

package example;
import jvmsgque.mqmsgque.*;

create the class, implement the the server interface:

class Filter3 extends MqContextC implements IServerSetup {
public Filter3(MqContextC tmpl) {
super(tmpl);
}
MqContextC(MK_OBJ obj)

define the ServerSetup function required by the interface

public void ServerSetup() {

get the filter-context to implement the proxy later

MqContextC * SlaveGetFilter()

define a Generig-Handler (+ALL) for the current context

ServiceProxy("+ALL");
void ServiceProxy(MQ_TOK token, MQ_SLAVE_ID id=MQ_SLAVE_OTHER)

and the filter context

ftr.ServiceProxy("+ALL");

define the main function, the factory and create the initial context:

public static void main(String[] argv) {
MqContextC srv = MqFactoryC.Add(Filter3.class,"Filter3").New();
MqFactoryC()

process the command-line arguments and start the initial link:

try {
srv.LinkCreate(argv);

finally start the event-loop and wait for an incoming service-request:

srv.ProcessEvent(MqWaitOnEventE.FOREVER);
MqWaitOnEventE

on exit delete the context and finish the process:

} catch (Throwable e) {
srv.ErrorCatch(e);
} finally {
srv.Exit();
}
}
}

ONE-DIRECTIONAL FILTER

The following code is based on the JAVA example example/java/Filter1.java:

package example;
import java.util.ArrayList;
import jvmsgque.mqmsgque.*;

create a class:

class Filter1 extends MqContextC {
private ArrayList<ArrayList<String>> data = new ArrayList<ArrayList<String>>();
public Filter1(MqContextC tmpl) {
super(tmpl);
}

define a service-handler for the +FTR token:

public static class FTR implements MqServiceIF {
public void Callback(MqContextC ctx) {
ArrayList<String> d = new ArrayList<String>();
while (ctx.ReadGetNumItems() != 0) {
d.add("<" + ctx.ReadSTR() + ">");
}
((Filter1)ctx).data.add(d);
ctx.SendRETURN();
}
}

define a service-handler for the +EOF token: The EOF is the end-of-data and is used to finally send out all the collected items at once. This is necessary when filtering the data as a whole, not by row.

public static class EOF implements MqServiceIF {
public void Callback(MqContextC ctx) {

get the filter target (slave 0)

MqContextC ftr = ctx.SlaveGetFilter();

loop over data and send ever item

for (ArrayList<String> d: ((Filter1)ctx).data) {
ftr.SendSTART();
for (String s: d) {
ftr.SendSTR(s);
}
ftr.SendEND_AND_WAIT("+FTR");
}

finally send the EOF to signal \€ end-of-data

ftr.SendSTART();
ftr.SendEND_AND_WAIT("+EOF");
ctx.SendRETURN();
}
}

at the main create initial factory and context:

public static void main(String[] argv) {
MqContextC srv = MqFactoryC.Add(Filter1.class).New();

configure as server:

try {
srv.ConfigSetIsServer(true);

create the link:

srv.LinkCreate(argv);

and setup the both filter-callback:

srv.ServiceCreate("+FTR", new Filter1.FTR());
srv.ServiceCreate("+EOF", new Filter1.EOF());

finally start the event.loop and wait for data:

srv.ProcessEvent(MqWaitOnEventE.FOREVER);

on error catch the message

} catch (Throwable e) {
srv.ErrorCatch(e);

and on exit cleanup the application

} finally {
srv.Exit();
}
}
}

FILTER - SHELL - INTEGRATION

libmqmsgque was designed to act as a glue between different applications. For an overview about the basic concepts we are using the good old shell and using libmqmsgque to extend the usability of the well known pipe '|' syntax.

basic shell behaviour

A shell command-line is a collection of one or more commands linked together using the '|' symbol:

command1 | command2 | command3

command1, command2 and command3 are started by the shell and the stdout of command1 is the stdin of command2 and the stdout of command2 is the stdin of command3. The data send through the pipeline are strings and every command in the pipeline have to parse the string output of the previous command to extract the information's needed.

  • advantage:
    • easy to use and human readable interface based on strings
  • disadvantage:
    • every command has to re-parse the output of the previous command

additional shell behaviour using the libmqmsgque syntax

libmqmsgque is adding an additional link character '@' to the shell and the example from above looks like this:

alfacmd1 @ alfacmd2 @ alfacmd3

Only alfacmd1 is started by the shell and gets '@ alfacmd2 @ alfacmd3' as command-line arguments. libmqmsgque will start the both commands alfacmd2 and alfacmd3 and setup the message-queues:

  1. alfacmd1 -> alfacmd2
  2. alfacmd2 -> alfacmd3

alfacmd2 receiving the output from alfacmd1 and alfacmd3 receiving the output from alfacmd2 without re-parsing the data again.

  • advantage:
    • single parsing of output and direct access to columns and rows
    • the alfacmdX commands are independent processes and only connected by message queues
  • disadvantage:
    • every command using the alfa syntax have to use the libmqmsgque library

interface between shell commands and alfa commands

For full integration of alfa commands into the shell syntax 2 additional interfaces are necessary

  • advantage:
    • direct integration between shell and alfa commands
    • reusing of already available shell commands and adding new features by alfa commands

interface: shellcmd | alfacmd

To connect a shell with an alfa command the special alfa command split is used:

    shellcmd | atool split @ ...

The split command expect input data from stdin and is sending output data as package to an alfa command. For every input data string an output package is created by splitting the input string into output objects using the the delimiter -d.

interface: alfacmd | shellcmd

To connect an alfa with a shell command the special alfa command join is used.
If the libmqmsgque object was created by atool :

    ... @ join | shellcmd

or if the libmqmsgque object was not created by atool :

    ... @ atool join | shellcmd

The join tool expect data from a msgque client as input and create for every input package an stdout output string by joining the objects of the input package together using the delimiter -d .

command pipelines using multiple hosts

starthost: alfacmd1 --tcp --host rmthost --port rmtport
endhost: alfacmd2 --tcp --port rmtport

By default libmqmsgque is using unix-domain sockets (UDS) for communication but inet (TCP) sockets can be used as well. The data-flow is the same as above except that two hosts are involved using libmqmsgque over tcp sockets for connection. The tcp connection is buildup between alfacmd1 and alfacmd2.

  • advantage:
    • multiple hosts can be used

a collection of examples should help to understand the software

example: this is a list of commands already available in this distribution

  • atool - a tool to work like a swiss-knife for command-line operations
  • aexec - a tool to setup a remote function call client and server

example 1 : just the famous hello world example

echo 'hello world with text' | atool split -d " " @ cut -f 0,1 @ join -D ":"

return: hello:world

example 2 : use tcl to create a smart filter

The following tcl code total.tcl does 2 things:

  1. convert the currencies into dollar ($)
  2. calculate the total amount
package require tclmsgque::MqMsgque

set total 0
array set exchange {
  euro    1.3541
  pound   1.9896
  dollar  1
}

proc FTR {ctx} {
  set ftr [$ctx SlaveGetFilter]
  foreach {position amount currency} [$ctx ReadLIST] break
  set amount [expr {$amount * $::exchange($currency)}]
  set currency dollar
  set ::total [expr {$::total + $amount}]
  $ftr Send "W" "+FTR:CD" $position $amount
  $ctx SendRETURN
}

proc EOF {ctx} {
  set ftr [$ctx SlaveGetFilter]
  $ftr Send "W" "+FTR:CD" total $::total
  $ftr Send "W" "+EOF"
  $ctx SendRETURN
}

tclmsgque::MqMsgque Main {
  tclmsgque::MqMsgque::MqContextC create srv
  srv ConfigSetIsServer yes
  srv ConfigSetName total
  try {
    srv LinkCreate {*}$argv
    srv ServiceCreate "+FTR" FTR
    srv ServiceCreate "+EOF" EOF
    srv ProcessEvent MQ_WAIT_FOREVER
  } on error {} {
    srv ErrorCatch
  } finally {
    srv Exit
  }
}

using the following command pipeline:

echo -e "nobody 10 euro\nmanager 1000 dollar\nworker 100 pound" | \
    atool split -d " " @ sort -1 D @ tclsh total.tcl @ \
        atool join -d " : " -0 "%-8s" -1 "%5.2f$"

to create the following result:

nobody   : 13.54$
worker   : 198.96$
manager  : 1000.00$
total    : 1212.50$