Loading...
Searching...
No Matches
MqContextC_Route_PY_API

MqContextC - setup and manage a routing-linkMore...

+ Collaboration diagram for MqContextC_Route_PY_API:

Functions

static OT_ProcRet py_mqmsgque_MqContextC_RouteGetTree (MqContextC_ARGS)
  Python: MkBufferListC ctx.RouteGetTree() C-API
create an overview about all available routing-target and services …
 
static OT_ProcRet py_mqmsgque_MqContextC_RouteCreate (MqContextC_ARGS)
  Python: ctx.RouteCreate(route:string, service:string, ?overwrite:bool=false?) C-API
create/delete a routing-link between context an a service using route
 
static OT_ProcRet py_mqmsgque_MqContextC_RouteDelete (MqContextC_ARGS)
  Python: ctx.RouteDelete(route:string, service:string, ?overwrite:bool=false?) C-API
delete a routing-link created with MqRouteCreate
 
static OT_ProcRet py_mqmsgque_MqContextC_RouteResolve (MqContextC_ARGS)
  Python: [MqContextC...] ctx.RouteResolve(ident:string, ?retnum:int32=-1?) C-API
return a list of all context belonging to ident
 
static OT_ProcRet py_mqmsgque_MqContextC_RouteTraverse (MqContextC_ARGS)
  Python: ctx.RouteTraverse(service:string, ?args:MkBufferListC...=None?) C-API
traverse a tree down and call service if available.
 
static OT_ProcRet py_mqmsgque_MqContextC_RouteGetPath (MqContextC_ARGS)
  Python: string ctx.RouteGetPath() C-API
return the absolut route-connection-string up to the current ctx
 

Detailed Description

MqContextC - setup and manage a routing-link

A routing-link is the connection of two context, the route-source and the route-target, with a unspecifiend number of hub-context in between using a specific service-token.

A single context is identified by the context-identifier as returned by MqClassIdentGet.

A routing-link can be created using Service-Level-Routing or Package-Level-Routing.

The difference between Service-Level-Routing and Package-Level-Routing is the public versa private behaviour.

Summary:

TCP/IP layer model

The TCP/IP model define the following layers (from: https://www.geeksforgeeks.org/tcp-ip-model/) :

Network Access Layer
This layer corresponds to the combination of Data Link Layer and Physical Layer of the OSI model. It looks out for hardware addressing and the protocols present in this layer allows for the physical transmission of data.
Internet Layer
This layer parallels the functions of OSI’s Network layer. It defines the protocols which are responsible for logical transmission of data over the entire network
The main protocols residing at this layer are :
  1. IP – stands for Internet Protocol and it is responsible for delivering packages from the source host to the destination host by looking at the IP addresses in the package headers. IP has 2 versions: IPv4 and IPv6. IPv4 is the one that most of the websites are using currently. But IPv6 is growing as the number of IPv4 addresses are limited in number when compared to the number of users.
  2. ICMP – stands for Internet Control Message Protocol. It is encapsulated within IP datagrams and is responsible for providing hosts with information about network problems.
  3. ARP – stands for Address Resolution Protocol. Its job is to find the hardware address of a host from a known IP address. ARP has several types: Reverse ARP, Proxy ARP, Gratuitous ARP and Inverse ARP.
Host-to-Host Layer
This layer is analogous to the transport layer of the OSI model. It is responsible for end-to-end communication and error-free delivery of data. It shields the upper-layer applications from the complexities of data.
The two main protocols present in this layer are :
  1. Transmission Control Protocol (TCP) – It is known to provide reliable and error-free communication between end systems. It performs sequencing and segmentation of data. It also has acknowledgment feature and controls the flow of the data through flow control mechanism. It is a very effective protocol but has a lot of overhead due to such features. Increased overhead leads to increased cost.
  2. User Datagram Protocol (UDP) – On the other hand does not provide any such features. It is the go-to protocol if your application does not require reliable transport as it is very cost-effective. Unlike TCP, which is connection-oriented protocol, UDP is connectionless.
Application Layer
This layer performs the functions of top three layers of the OSI model: Application, Presentation and Session Layer. It is responsible for node-to-node communication and controls user-interface specifications. Some of the protocols present in this layer are: HTTP, HTTPS, FTP, TFTP, Telnet, SSH, SMTP, SNMP, NTP, DNS, DHCP, NFS, X Window, LPD. Have a look at Protocols in Application Layer for some information about these protocols.
Protocols other than those present in the linked article are :
  1. HTTP and HTTPS – HTTP stands for Hypertext transfer protocol. It is used by the World Wide Web to manage communications between web browsers and servers. HTTPS stands for HTTP-Secure. It is a combination of HTTP with SSL(Secure Socket Layer). It is efficient in cases where the browser need to fill out forms, sign in, authenticate and carry out bank transactions.
  2. SSH – SSH stands for Secure Shell. It is a terminal emulations software similar to Telnet. The reason SSH is more preferred is because of its ability to maintain the encrypted connection. It sets up a secure session over a TCP/IP connection.
  3. NTP – NTP stands for Network Time Protocol. It is used to synchronize the clocks on our computer to one standard time source. It is very useful in situations like bank transactions. Assume the following situation without the presence of NTP. Suppose you carry out a transaction, where your computer reads the time at 2:30 PM while the server records it at 2:28 PM. The server can crash very badly if it’s out of sync.

LibMqMsgque layer model

The libmqmsgque layer model is an extension to the TCP/IP layer model.

TCP/IP Application Layer
This layer is used for the libmqmsgque protocol - The protocol is the "language" of the libmqmsgque
library. The layer is defined as protocol-message-format and as protocol-flow-format.
  1. protocol-message-format - Example from protocoll_mq.h this is the "syntax" of the protocol.
    struct HdrS {
    MK_STRB ID[4] ;
    MK_STRB native ;
    MK_STRB charA ;
    union HdrIU ctxId ;
    MK_STRB charS ;
    union HdrIU bdySize ;
    MK_STRB charO ;
    MK_STRB tok[HDR_TOK_LEN] ;
    MK_STRB charT ;
    union HdrIU transSId ;
    MK_STRB charR ;
    union HdrIU routeId ;
    MK_STRB charC ;
    MK_STRB hs ;
    MK_STRB charE ;
    MK_STRB tt ;
    MK_STRB charF ;
    };
    struct BdyS {
    MK_STRB ID[4] ;
    //MK_STRB charN ; ///< <TT>size: 1 .............. -> total: 4 .. -> character .. -> "+"</TT>
    union HdrIU numItems;
    MK_STRB charE ;
    };
    struct LtrS {
    MK_STRB ID[4] ;
    //MK_STRB charN ; ///< <TT>size: 1 .............. -> total: 4 .. -> character .. -> "+"</TT>
    MK_BINB ltrP[HDR_ID_SIZE] ;
    MK_STRB charE ;
    };
    #define LTR_SIZE (3 + 1 + HDR_ID_SIZE + 1)
    struct RouS {
    MK_STRB ID[4] ;
    //MK_STRB charN ; ///< <TT>size: 1 .............. -> total: 4 .. -> character .. -> "+"</TT>
    MK_BINB rouP[HDR_ID_SIZE] ;
    MK_STRB charE ;
    };
    #define ROU_SIZE (3 + 1 + HDR_ID_SIZE + 1)
    struct PackageS {
    struct HdrS hdr ;
    struct BdyS bdy ;
    };
  2. protocol-flow-format - this is the "grammatic" of the protocol.
LibMqMsgque Service Layer

This layer provide the MqContextC-ServiceApi ontop of the Application Layer.

The master-slave-link is used to create a mesh of nodes defined by different parent-context. The master control the slave.

The master-slave-link is used to perform the following tasks:

  • report error messages from the slave-context to the master-context
  • to create a slave-child-context if a master-child-context is created
  • to delete a slave-context if a master-context is deleted

In difference to the client-server-link the master-slave-link connect two independent parent-context in the same process or thread (e.g. node). This leads to the restriction that only the master-context can be a server-context because only one server-context per node is possible.

   node-0   |           node-1/2        |   node-3/4/5
===================================================================

| <- client/server link -> | <- client/server link -> |

             | <-- master/slave link --> |

                           |- client1-0 -|- server3 ...
             |-  server1  -|
             |             |- client1-1 -|- server4 ...
  client0-0 -|
             |-  server2  -|- client1-2 -|- server5 ...

Definition of the "master-context"

  • the master-context is a parent-context without a child-context available.
  • the master-context is a client-context or a server-context.
  • the link between the master-context and the slave-context is done using MqSlaveWorker or MqSlaveCreate

Definition of the "slave-context"

  • the slave-context is a parent-context without a child-context available.
  • the slave-context is a client-context.
  • the slave-context lifetime is controlled by the master-context.
  • the slave-context report all error-messages to the master-context.
  • the slave-context is identified by a unique-slave-id starting with 0.
  • a special form of a slave-context is a worker-context

Definition of the "worker-context"

  • the worker-context is a slave-context using the image of the master-context self.
  • the master-context can be a server-context or a client-context.
  • the worker-context is created using MqSlaveWorker
  • the worker-context is identified by a unique-slave-id starting with 0.

Definition of the "slave-id"

  • the slave-id is defined at enum MqSlaveE
  • a slave is identified in his master-context by a slave-id
  • the slave-id work like a defined well-known-port-number in /etc/services
  • the slave-id is an integer value with valid values > 0
  • it is a good practice to plan the usage of your slave-id(s)
  • slave
    slave-id value definition
    MQ_SLAVE_MAX 1024 internal: the maximum slave-id … .
    MQ_SLAVE_USER 10 internal: start of user-defined-slave-id .
    MQ_SLAVE_LOOPBACK 0 internal: the loopback-slave-id, (call my own services) .
    MQ_SLAVE_FILTER 1 internal: the filter-slave-id, (on a master get the filter-slave) .
    MQ_SLAVE_MASTER 1 internal: the master-slave-id, (on a slave get the master) .
    MQ_SLAVE_OTHER 1 internal: on the master-ctx get the slave-ctx and on the slave-ctx get the master-ctx .
  • range
    range definition
    0 <= slave-id < MQ_SLAVE_MAX range of valid slave-id's
    0 <= slave-id < MQ_SLAVE_USER internale usage
    MQ_SLAVE_USER <= slave-id < MQ_SLAVE_MAX external usage

Definition of the "LOOPBACK" (0) slave

  •    client   |           server            |
    ===========================================
    
    | <--- client/server --->  | <-- loop --> |
    
                | <------ master/slave -----> |
    
      client -- | -- server -- | -- client -- #
                       ==             ==      #
                     server -- | -- client -- #
  • the loopback has always the slave-id = 0 .
  • the loopback has the same class as the parent, if reflection is not available the MqFactoryInitial is used.
  • the loopback is used to call a service on the same process or thread.
  • the loopback is a special filter without an additional process or thread to be started.
  • the loopback is only internal accessible
  • the service called by the loopback need the same attention as the service called by the filter, the context of the service is the loopback-context.
  • the loopback can call MqServiceCreate to create a new link between a service-token and a service-method, by default all services from the master-context (the owner of the loopback) are also accessible by the loopback.
  • in the service use MqSlaveGetMaster to get the master-context from the loopback-context.
  • Example from MyLoopServer.py create a new loop-server
    import sys
    from pymqmsgque import *
    
    # package-item
    class MyLoopServer(MqContextC):
    
      # factory constructor
      def __init__(self, tmpl=None):
        self.ConfigSetServerSetup(self.ServerSetup)
        # set the "mydata" attribute to the master-context
        self.mydata = "Hello World"
        super().__init__()
    
      # service to serve all EXTERNAL requests for token "HLWO"
      def HLWO_srv(self):
        # get the "loopback" context
        loop = self.SlaveGet(MqSlaveE.LOOPBACK)
        # call the LOOP service on the SAME server
        loop.Send("W","LOOP")
        # answer HLWO with string-return from LOOP
        self.Send("R", "C", loop.ReadSTR())
    
      # service to serve all INTERNAL requests for token "LOOP"
      def LOOP_srv(self):
        # get the "master" context 
        master = self.SlaveGetMaster()
        # answer LOOP with data from MASTER->mydata attribute
        self.Send("R", "C", master.mydata)
    
      # define a service as link between the token "HLWO" and the callback "HLWO_srv"
      def ServerSetup(self):
        # EXTERNAL: link the "HLWO" service with "HLWO_srv"
        self.ServiceCreate("HLWO",MyLoopServer.HLWO_srv)
        # INTERNAL: link the "LOOP" service with "LOOP_srv"
        self.SlaveGet(MqSlaveE.LOOPBACK).ServiceCreate("LOOP", MyLoopServer.LOOP_srv)
    
    # package-main
    if __name__ == "__main__":
    
      # create the "MyLoopServer" factory… and the instance
      srv = MqFactoryC.Add(MyLoopServer).New()
    
      try:
        srv.LinkCreate(sys.argv)
        srv.ProcessEvent(MqWaitOnEventE.FOREVER)
      except Exception as ex:
        srv.ErrorCatch(ex)
      finally:
        srv.Exit()
    

Performance analyse

The performance-test is created with:
  • pipe
    • Nhi1Exec perfclient.c --parent --wrk ? @ perfserver.c
  • spawn, fork, thread
    • Nhi1Exec -r=uds perfserver.c --spawn|fork|thread
    • Nhi1Exec -r=uds perfclient.c --parent --wrk ?
  • the number of workers are set with the –wrk option
  • the cpu is a xeon with 4/8
  • the performance is calculated as worker-context-created / time-in-sec with a ~2sec (default) measurement period.
performance-test code:
  • The test-setup is done as:
      perfclient                                worker                            perfserver
      ==========                                ======                            ==========
      |
      |- loop --wrk x
        |- MqSlaveWorker(...)               ->  worker[1] 
        |- MqSend(worker[1],"E","STR0..")   ->  PerfWorker_I160(...)
                                                |- loop endless
                                                  |- MqContextCreate(...)
                                                  |- MqLinkCreate(...)      <->   MqContextCreate(...)
                                                  |- MqContextDelete(...)   <->   MqContextDelete(...)
      |- sleep x sec
      |- loop --wrk x
        |- MqSend(worker[1],"C"..,"END0")   ->  PerfWorker_END0(...)
        |                                       |- stop loop
        |- "callback" - add number to all   <-  |- return #context 
performance-test results:
  • results generated using the debug environment
    setup –wrk # worker-context performance info
    pipe 1 2500 1000 the pipe start a new worker-context with spawn
    spawn 1 2500 <1000 same as pipe but use network-protocoll
    fork 1 3800 4000 the fork is faster than spawn
    thread 1 16500 9000 the thread is faster than fork
    pipe 4 8000 4500 the worker scale linear up to number of processors
    spawn 4 7600 <4500 -
    fork 4 23200 11500 -
    thread 4 55500 27500 -
    pipe 8 10000 5500 the additional scaling up to the max hyper-threading does not really help
    spawn 8 9100 <5500 -
    fork 8 23200 11500 -
    thread 8 55500 27500 -

example (C++ syntax)

Route-Connection-String

A client-server connection is defined with the route-connection-string as a composition of "identifier" and is build like a UNIX directory tree with the initial client as root.

Example: GUI/Data Frontend

*
|-header
|-GUI-|
frontend-| |-footer
|
|-DB
*

With --ident-from prefix|factory the identifier is defined as prefix-identifier or factory-identifer.

A route-connection-string is the path between TWO locations in the tree using the UNIX syntax:

short syntax direction
dot . current-context
double-dot .. previous-context
slash / root-context (the client)
"string" xxx next-context with name xxx

path between "DB" and "header"

  • relative: ctx.RouteCreate("../GUI/header",service)
  • absolute: ctx.RouteCreate("/frontend/GUI/header",service)

path between "footer" and "DB"

  • relative: ctx.RouteCreate("../../DB",service)
  • absolute: ctx.RouteCreate("/frontend/DB",service)

path between "frontend" and "footer"

  • relative: ctx.RouteCreate("GUI/footer",service)
  • absolute: ctx.RouteCreate("/frontend/GUI/footer",service)

Service-Layer-Routing

The LibMqMsgque-Service-Layer-Routing create proxy-services (MqServiceProxyRoundRobin ...) between frontend and footer. (example c++)

// 1. on "frontend" create a route to the service "MYSV" on the "footer" passing "GUI"
DOING: frontendCtx.RouteCreate("GUI/footer", "MYSV");
// 2. on "frontend" call the route using the "loopback" slave
DOING: frontendCtx.SlaveGet(MQ_SLAVE_LOOPBACK).Send("W", "MYSV:..@..", ..);
// 3. on "frontend/GUI" proxy "footer"
INTERNAL: frontendCtx.ServiceProxyRoundRobin("MYSV", GUICtx);
INTERNAL: GUICtx.ServiceProxyRoundRobin("MYSV", footerCtx);
// 4. on "footer" call the service "MYSV"
SETUP: footerCtx.ServiceCreate("MYSV", ...);
@ MQ_SLAVE_LOOPBACK
internal: the loopback-slave-id, (call my own services)

The MqRouteCreate will take the following action:

  1. create a round-robin-proxy on frontend and on GUI using service MYSV
  2. check if service MYSV is available on footer

Attention:

  1. With the proxy on frontend/GUI no other service with the name MYSV on frontend/GUI is possible and every service-call to frontend or GUI using the service MYSV will be redirected to the service MYSV on footer.
  2. The live-time of the Service-Layer-Routing is up to the next MqRouteDelete call using the same arguments. If the proxy-to-footer becomes invalid on a the GUI-context the routing-link will be recreated.
  3. The MqRouteCreate can be used multiple times with the same Route-Connection-String and service-token and overwrite=true to guarantee that a route is available.

Package-Layer-Routing

...

Function Documentation

◆ py_mqmsgque_MqContextC_RouteCreate()

static OT_ProcRet py_mqmsgque_MqContextC_RouteCreate ( MqContextC_ARGS )
static

Python: ctx.RouteCreate(route:string, service:string, ?overwrite:bool=false?) C-API
create/delete a routing-link between context an a service using route

Definition at line 2384 of file MqContextC_py.c.

2384 {
2386 OT_SETUP_VARARGS(2,3,RouteCreate_doc)
2387 MK_STRN route = 0;
2388 OT_CHECK_REQUIRED(OT_CHECK_STRN (route))
2389 MK_STRN service = 0;
2390 OT_CHECK_REQUIRED(OT_CHECK_STRN (service))
2391 MK_BOOL overwrite = false;
2392 OT_CHECK_OPTIONAL(OT_CHECK_bool (overwrite))
2393 OT_CHECK_NOARGS
2394 MkErrorC_Check(hdl,MqRouteCreate (hdl, route, service, overwrite));
2395 OT_retObj_SET_None
2396 goto end;
2397 error:
2398 OT_retObj_SET_Error
2399 end:
2401}
#define RouteCreate_doc
#define OT_SETUP_hdl
#define OT_retObj_RETURN
#define error
const MK_STRB * MK_STRN
bool MK_BOOL
#define MqRouteCreate(...)

◆ py_mqmsgque_MqContextC_RouteDelete()

static OT_ProcRet py_mqmsgque_MqContextC_RouteDelete ( MqContextC_ARGS )
static

Python: ctx.RouteDelete(route:string, service:string, ?overwrite:bool=false?) C-API
delete a routing-link created with MqRouteCreate

Definition at line 2404 of file MqContextC_py.c.

2404 {
2406 OT_SETUP_VARARGS(2,3,RouteDelete_doc)
2407 MK_STRN route = 0;
2408 OT_CHECK_REQUIRED(OT_CHECK_STRN (route))
2409 MK_STRN service = 0;
2410 OT_CHECK_REQUIRED(OT_CHECK_STRN (service))
2411 MK_BOOL overwrite = false;
2412 OT_CHECK_OPTIONAL(OT_CHECK_bool (overwrite))
2413 OT_CHECK_NOARGS
2414 MkErrorC_Check(hdl,MqRouteDelete (hdl, route, service, overwrite));
2415 OT_retObj_SET_None
2416 goto end;
2417 error:
2418 OT_retObj_SET_Error
2419 end:
2421}
#define RouteDelete_doc
#define MqRouteDelete(...)

◆ py_mqmsgque_MqContextC_RouteGetPath()

static OT_ProcRet py_mqmsgque_MqContextC_RouteGetPath ( MqContextC_ARGS )
static

Python: string ctx.RouteGetPath() C-API
return the absolut route-connection-string up to the current ctx

Definition at line 2465 of file MqContextC_py.c.

2465 {
2467 OT_SETUP_NOARG(RouteGetPath_doc)
2468 OT_CHECK_NOARGS
2469 MK_STRN path_out;
2470 MkErrorC_Check(hdl,MqRouteGetPath (hdl, &path_out));
2471 OT_retObj_SET_STR(path_out)
2472 goto end;
2473 error:
2474 OT_retObj_SET_Error
2475 end:
2477}
#define RouteGetPath_doc
#define MqRouteGetPath(...)

◆ py_mqmsgque_MqContextC_RouteGetTree()

static OT_ProcRet py_mqmsgque_MqContextC_RouteGetTree ( MqContextC_ARGS )
static

Python: MkBufferListC ctx.RouteGetTree() C-API
create an overview about all available routing-target and services …

Definition at line 2367 of file MqContextC_py.c.

2367 {
2369 OT_SETUP_NOARG(RouteGetTree_doc)
2370 OT_CHECK_NOARGS
2371 MK_BFL treeP_out;
2372 MkErrorC_Check(hdl,MqRouteGetTree (hdl, &treeP_out));
2373 OT_retObj_SET_BFL(treeP_out)
2374 goto end;
2375 error:
2376 OT_retObj_SET_Error
2377 end:
2379}
#define RouteGetTree_doc
#define MqRouteGetTree(...)

◆ py_mqmsgque_MqContextC_RouteResolve()

static OT_ProcRet py_mqmsgque_MqContextC_RouteResolve ( MqContextC_ARGS )
static

Python: [MqContextC...] ctx.RouteResolve(ident:string, ?retnum:int32=-1?) C-API
return a list of all context belonging to ident

Definition at line 2424 of file MqContextC_py.c.

2424 {
2426 OT_SETUP_VARARGS(1,2,RouteResolve_doc)
2427 MK_STRN ident = 0;
2428 OT_CHECK_REQUIRED(OT_CHECK_STRN (ident))
2429 MK_NUM retnum = -1;
2430 OT_CHECK_OPTIONAL(OT_CHECK_NI4 (retnum))
2431 OT_CHECK_NOARGS
2432 MQ_CTX_A retVal = MqRouteResolve (hdl, ident, retnum);
2433 OT_retObj_SET_List
2434 for (int i=0; i<retVal.size; i++) {
2435 OT_retObj_APPEND(OT_TMP_CTX_OBJ(retVal.data[i]));
2436 }
2437 goto end;
2438 error:
2439 OT_retObj_SET_Error
2440 end:
2442}
#define RouteResolve_doc
int32_t MK_NUM
#define MqRouteResolve(...)
array of MqContextC instances

◆ py_mqmsgque_MqContextC_RouteTraverse()

static OT_ProcRet py_mqmsgque_MqContextC_RouteTraverse ( MqContextC_ARGS )
static

Python: ctx.RouteTraverse(service:string, ?args:MkBufferListC...=None?) C-API
traverse a tree down and call service if available.

Definition at line 2445 of file MqContextC_py.c.

2445 {
2447 OT_SETUP_VARARGS(1,99,RouteTraverse_doc)
2448 MK_STRN service = 0;
2449 OT_CHECK_REQUIRED(OT_CHECK_STRN (service))
2450 MkBufferListCreateTLS_T(args,10);
2451 OT_CHECK_OPTIONAL(OT_CHECK_BAC (args))
2452 OT_CHECK_NOARGS
2453 MkErrorC_Check(hdl,MqRouteTraverse (hdl, service, args));
2454 OT_retObj_SET_None
2455 goto end;
2456 error:
2457 OT_retObj_SET_Error
2458 end:
2460}
#define RouteTraverse_doc
#define MkBufferListCreateTLS_T(name, num)
#define MqRouteTraverse(...)