theLink 10.0 NHI1 - theKernel - theLink - theConfig - theSq3Lite - theCompiler - theBrain - theGuard
c - tcl - cs - py - rb - jv - cc
Loading...
Searching...
No Matches
MqContextC_Route_PY_API

MqContextC - setup and manage a routing-linkMore...

+ Collaboration diagram for MqContextC_Route_PY_API:

Functions

static OT_ProcRet pymqmsgque_MqContextC_RouteGetTree (MqContextC_ARGS)
  Python: MkBufferListC ctx.RouteGetTree() C-API
create an overview about all available routing-target and services …
 
static OT_ProcRet pymqmsgque_MqContextC_RouteCreate (MqContextC_ARGS)
  Python: ctx.RouteCreate(route:string, service:string, ?overwrite:bool=false?) C-API
create/delete a routing-link between context an a service using route
 
static OT_ProcRet pymqmsgque_MqContextC_RouteDelete (MqContextC_ARGS)
  Python: ctx.RouteDelete(route:string, service:string, ?overwrite:bool=false?) C-API
delete a routing-link created with MqRouteCreate
 
static OT_ProcRet pymqmsgque_MqContextC_RouteResolve (MqContextC_ARGS)
  Python: [MqContextC...] ctx.RouteResolve(ident:string, ?retnum:int32=-1?) C-API
return a list of all context belonging to ident
 
static OT_ProcRet pymqmsgque_MqContextC_RouteTraverse (MqContextC_ARGS)
  Python: ctx.RouteTraverse(service:string, ?args:MkBufferListC...=None?) C-API
traverse a tree down and call service if available.
 
static OT_ProcRet pymqmsgque_MqContextC_RouteGetPath (MqContextC_ARGS)
  Python: string ctx.RouteGetPath() C-API
return the absolut route-connection-string up to the current ctx
 

Detailed Description

MqContextC - setup and manage a routing-link

A routing-link is the connection of two context, the route-source and the route-target, with a unspecifiend number of hub-context in between using a specific service-token.

A single context is identified by the context-identifier as returned by MqClassIdentGet.

A routing-link can be created using Service-Level-Routing or Package-Level-Routing.

The difference between Service-Level-Routing and Package-Level-Routing is the public versa private behaviour.

Summary:

TCP/IP layer model

The TCP/IP model define the following layers (from: https://www.geeksforgeeks.org/tcp-ip-model/) :

Network Access Layer
This layer corresponds to the combination of Data Link Layer and Physical Layer of the OSI model. It looks out for hardware addressing and the protocols present in this layer allows for the physical transmission of data.
Internet Layer
This layer parallels the functions of OSI’s Network layer. It defines the protocols which are responsible for logical transmission of data over the entire network
The main protocols residing at this layer are :
  1. IP – stands for Internet Protocol and it is responsible for delivering packages from the source host to the destination host by looking at the IP addresses in the package headers. IP has 2 versions: IPv4 and IPv6. IPv4 is the one that most of the websites are using currently. But IPv6 is growing as the number of IPv4 addresses are limited in number when compared to the number of users.
  2. ICMP – stands for Internet Control Message Protocol. It is encapsulated within IP datagrams and is responsible for providing hosts with information about network problems.
  3. ARP – stands for Address Resolution Protocol. Its job is to find the hardware address of a host from a known IP address. ARP has several types: Reverse ARP, Proxy ARP, Gratuitous ARP and Inverse ARP.
Host-to-Host Layer
This layer is analogous to the transport layer of the OSI model. It is responsible for end-to-end communication and error-free delivery of data. It shields the upper-layer applications from the complexities of data.
The two main protocols present in this layer are :
  1. Transmission Control Protocol (TCP) – It is known to provide reliable and error-free communication between end systems. It performs sequencing and segmentation of data. It also has acknowledgment feature and controls the flow of the data through flow control mechanism. It is a very effective protocol but has a lot of overhead due to such features. Increased overhead leads to increased cost.
  2. User Datagram Protocol (UDP) – On the other hand does not provide any such features. It is the go-to protocol if your application does not require reliable transport as it is very cost-effective. Unlike TCP, which is connection-oriented protocol, UDP is connectionless.
Application Layer
This layer performs the functions of top three layers of the OSI model: Application, Presentation and Session Layer. It is responsible for node-to-node communication and controls user-interface specifications. Some of the protocols present in this layer are: HTTP, HTTPS, FTP, TFTP, Telnet, SSH, SMTP, SNMP, NTP, DNS, DHCP, NFS, X Window, LPD. Have a look at Protocols in Application Layer for some information about these protocols.
Protocols other than those present in the linked article are :
  1. HTTP and HTTPS – HTTP stands for Hypertext transfer protocol. It is used by the World Wide Web to manage communications between web browsers and servers. HTTPS stands for HTTP-Secure. It is a combination of HTTP with SSL(Secure Socket Layer). It is efficient in cases where the browser need to fill out forms, sign in, authenticate and carry out bank transactions.
  2. SSH – SSH stands for Secure Shell. It is a terminal emulations software similar to Telnet. The reason SSH is more preferred is because of its ability to maintain the encrypted connection. It sets up a secure session over a TCP/IP connection.
  3. NTP – NTP stands for Network Time Protocol. It is used to synchronize the clocks on our computer to one standard time source. It is very useful in situations like bank transactions. Assume the following situation without the presence of NTP. Suppose you carry out a transaction, where your computer reads the time at 2:30 PM while the server records it at 2:28 PM. The server can crash very badly if it’s out of sync.

LibMqMsgque layer model

The libmqmsgque layer model is an extension to the TCP/IP layer model.

TCP/IP Application Layer
This layer is used for the libmqmsgque protocol - The protocol is the "language" of the libmqmsgque
library. The layer is defined as protocol-message-format and as protocol-flow-format.
  1. protocol-message-format - Example from protocoll_mq.h this is the "syntax" of the protocol.
    struct HdrS {
    MK_STRB ID[4] ;
    MK_STRB native ;
    MK_STRB charA ;
    union HdrIU ctxId ;
    MK_STRB charS ;
    union HdrIU bdySize ;
    MK_STRB charO ;
    MK_STRB tok[HDR_TOK_LEN] ;
    MK_STRB charT ;
    union HdrIU transSId ;
    MK_STRB charR ;
    union HdrIU routeId ;
    MK_STRB charC ;
    MK_STRB hs ;
    MK_STRB charE ;
    MK_STRB tt ;
    MK_STRB charF ;
    };
    struct BdyS {
    MK_STRB ID[4] ;
    //MK_STRB charN ; ///< <TT>size: 1 .............. -> total: 4 .. -> character .. -> "+"</TT>
    union HdrIU numItems;
    MK_STRB charE ;
    };
    struct LtrS {
    MK_STRB ID[4] ;
    //MK_STRB charN ; ///< <TT>size: 1 .............. -> total: 4 .. -> character .. -> "+"</TT>
    MK_BINB ltrP[HDR_ID_SIZE] ;
    MK_STRB charE ;
    };
    #define LTR_SIZE (3 + 1 + HDR_ID_SIZE + 1)
    struct RouS {
    MK_STRB ID[4] ;
    //MK_STRB charN ; ///< <TT>size: 1 .............. -> total: 4 .. -> character .. -> "+"</TT>
    MK_BINB rouP[HDR_ID_SIZE] ;
    MK_STRB charE ;
    };
    #define ROU_SIZE (3 + 1 + HDR_ID_SIZE + 1)
    struct PackageS {
    struct HdrS hdr ;
    struct BdyS bdy ;
    };
  2. protocol-flow-format - this is the "grammatic" of the protocol.
LibMqMsgque Service Layer

This layer provide the MqContextC-ServiceApi ontop of the Application Layer.

The master-slave-link is used to create a mesh of nodes defined by different parent-context. The master control the slave.

The master-slave-link is used to perform the following tasks:

  • report error messages from the slave-context to the master-context
  • to create a slave-child-context if a master-child-context is created
  • to delete a slave-context if a master-context is deleted

In difference to the client-server-link the master-slave-link connect two independent parent-context in the same process or thread (e.g. node). This leads to the restriction that only the master-context can be a server-context because only one server-context per node is possible.

   node-0   |           node-1/2        |   node-3/4/5
===================================================================

| <- client/server link -> | <- client/server link -> |

             | <-- master/slave link --> |

                           |- client1-0 -|- server3 ...
             |-  server1  -|
             |             |- client1-1 -|- server4 ...
  client0-0 -|
             |-  server2  -|- client1-2 -|- server5 ...

Definition of the "master-context"

  • the master-context is a parent-context without a child-context available.
  • the master-context is a client-context or a server-context.
  • the link between the master-context and the slave-context is done using MqSlaveWorker or MqSlaveCreate

Definition of the "slave-context"

  • the slave-context is a parent-context without a child-context available.
  • the slave-context is a client-context.
  • the slave-context lifetime is controlled by the master-context.
  • the slave-context report all error-messages to the master-context.
  • the slave-context is identified by a unique-slave-id starting with 0.
  • a special form of a slave-context is a worker-context

Definition of the "worker-context"

  • the worker-context is a slave-context using the image of the master-context self.
  • the master-context can be a server-context or a client-context.
  • the worker-context is created using MqSlaveWorker
  • the worker-context is identified by a unique-slave-id starting with 0.

Definition of the "slave-id"

  • the slave-id is defined at enum MqSlaveE
  • a slave is identified in his master-context by a slave-id
  • the slave-id work like a defined well-known-port-number in /etc/services
  • the slave-id is an integer value with valid values > 0
  • it is a good practice to plan the usage of your slave-id(s)
  • slave
    slave-id value definition
    MQ_SLAVE_MAX 1024 internal: the maximum slave-id … .
    MQ_SLAVE_USER 10 internal: start of user-defined-slave-id .
    MQ_SLAVE_LOOPBACK 0 internal: the loopback-slave-id, (call my own services) .
    MQ_SLAVE_FILTER 1 internal: the filter-slave-id, (on a master get the filter-slave) .
    MQ_SLAVE_MASTER 1 internal: the master-slave-id, (on a slave get the master) .
    MQ_SLAVE_OTHER 1 internal: on the master-ctx get the slave-ctx and on the slave-ctx get the master-ctx .
  • range
    range definition
    0 <= slave-id < MQ_SLAVE_MAX range of valid slave-id's
    0 <= slave-id < MQ_SLAVE_USER internale usage
    MQ_SLAVE_USER <= slave-id < MQ_SLAVE_MAX external usage

Definition of the "LOOPBACK" (0) slave

  •    client   |           server            |
    ===========================================
    
    | <--- client/server --->  | <-- loop --> |
    
                | <------ master/slave -----> |
    
      client -- | -- server -- | -- client -- #
                       ==             ==      #
                     server -- | -- client -- #
  • the loopback has always the slave-id = 0 .
  • the loopback has the same class as the parent, if reflection is not available the MqFactoryInitial is used.
  • the loopback is used to call a service on the same process or thread.
  • the loopback is a special filter without an additional process or thread to be started.
  • the loopback is only internal accessible
  • the service called by the loopback need the same attention as the service called by the filter, the context of the service is the loopback-context.
  • the loopback can call MqServiceCreate to create a new link between a service-token and a service-method, by default all services from the master-context (the owner of the loopback) are also accessible by the loopback.
  • in the service use MqSlaveGetMaster to get the master-context from the loopback-context.
  • Example from MyLoopServer.py create a new loop-server
    import sys
    from pymqmsgque import *
    
    # package-item
    class MyLoopServer(MqContextC):
    
      # factory constructor
      def __init__(self, tmpl=None):
        self.ConfigSetServerSetup(self.ServerSetup)
        # set the "mydata" attribute to the master-context
        self.mydata = "Hello World"
        super().__init__()
    
      # service to serve all EXTERNAL requests for token "HLWO"
      def HLWO_srv(self):
        # get the "loopback" context
        loop = self.SlaveGet(MqSlaveE.LOOPBACK)
        # call the LOOP service on the SAME server
        loop.Send("W","LOOP")
        # answer HLWO with string-return from LOOP
        self.Send("R", "C", loop.ReadSTR())
    
      # service to serve all INTERNAL requests for token "LOOP"
      def LOOP_srv(self):
        # get the "master" context 
        master = self.SlaveGetMaster()
        # answer LOOP with data from MASTER->mydata attribute
        self.Send("R", "C", master.mydata)
    
      # define a service as link between the token "HLWO" and the callback "HLWO_srv"
      def ServerSetup(self):
        # EXTERNAL: link the "HLWO" service with "HLWO_srv"
        self.ServiceCreate("HLWO",MyLoopServer.HLWO_srv)
        # INTERNAL: link the "LOOP" service with "LOOP_srv"
        self.SlaveGet(MqSlaveE.LOOPBACK).ServiceCreate("LOOP", MyLoopServer.LOOP_srv)
    
    # package-main
    if __name__ == "__main__":
    
      # create the "MyLoopServer" factory… and the instance
      srv = MqFactoryC.Add(MyLoopServer).New()
    
      try:
        srv.LinkCreate(sys.argv)
        srv.ProcessEvent(MqWaitOnEventE.FOREVER)
      except Exception as ex:
        srv.ErrorCatch(ex)
      finally:
        srv.Exit()
    

Performance analyse

The performance-test is created with:
  • pipe
    • Nhi1Exec perfclient.c --parent --wrk ? @ perfserver.c
  • spawn, fork, thread
    • Nhi1Exec -r=uds perfserver.c --spawn|fork|thread
    • Nhi1Exec -r=uds perfclient.c --parent --wrk ?
  • the number of workers are set with the –wrk option
  • the cpu is a xeon with 4/8
  • the performance is calculated as worker-context-created / time-in-sec with a ~2sec (default) measurement period.
performance-test code:
  • The test-setup is done as:
      perfclient                                worker                            perfserver
      ==========                                ======                            ==========
      |
      |- loop --wrk x
        |- MqSlaveWorker(...)               ->  worker[1] 
        |- MqSend(worker[1],"E","STR0..")   ->  PerfWorker_I160(...)
                                                |- loop endless
                                                  |- MqContextCreate(...)
                                                  |- MqLinkCreate(...)      <->   MqContextCreate(...)
                                                  |- MqContextDelete(...)   <->   MqContextDelete(...)
      |- sleep x sec
      |- loop --wrk x
        |- MqSend(worker[1],"C"..,"END0")   ->  PerfWorker_END0(...)
        |                                       |- stop loop
        |- "callback" - add number to all   <-  |- return #context 
performance-test results:
  • results generated using the debug environment
    setup –wrk # worker-context performance info
    pipe 1 2500 1000 the pipe start a new worker-context with spawn
    spawn 1 2500 <1000 same as pipe but use network-protocoll
    fork 1 3800 4000 the fork is faster than spawn
    thread 1 16500 9000 the thread is faster than fork
    pipe 4 8000 4500 the worker scale linear up to number of processors
    spawn 4 7600 <4500 -
    fork 4 23200 11500 -
    thread 4 55500 27500 -
    pipe 8 10000 5500 the additional scaling up to the max hyper-threading does not really help
    spawn 8 9100 <5500 -
    fork 8 23200 11500 -
    thread 8 55500 27500 -

example (C++ syntax)

Route-Connection-String

A client-server connection is defined with the route-connection-string as a composition of "identifier" and is build like a UNIX directory tree with the initial client as root.

Example: GUI/Data Frontend

*
|-header
|-GUI-|
frontend-| |-footer
|
|-DB
*

With --ident-from prefix|factory the identifier is defined as prefix-identifier or factory-identifer.

A route-connection-string is the path between TWO locations in the tree using the UNIX syntax:

short syntax direction
dot . current-context
double-dot .. previous-context
slash / root-context (the client)
"string" xxx next-context with name xxx

path between "DB" and "header"

  • relative: ctx.RouteCreate("../GUI/header",service)
  • absolute: ctx.RouteCreate("/frontend/GUI/header",service)

path between "footer" and "DB"

  • relative: ctx.RouteCreate("../../DB",service)
  • absolute: ctx.RouteCreate("/frontend/DB",service)

path between "frontend" and "footer"

  • relative: ctx.RouteCreate("GUI/footer",service)
  • absolute: ctx.RouteCreate("/frontend/GUI/footer",service)

Service-Layer-Routing

The LibMqMsgque-Service-Layer-Routing create proxy-services (MqServiceProxyRoundRobin ...) between frontend and footer. (example c++)

// 1. on "frontend" create a route to the service "MYSV" on the "footer" passing "GUI"
DOING: frontendCtx.RouteCreate("GUI/footer", "MYSV");
// 2. on "frontend" call the route using the "loopback" slave
DOING: frontendCtx.SlaveGet(MQ_SLAVE_LOOPBACK).Send("W", "MYSV:..@..", ..);
// 3. on "frontend/GUI" proxy "footer"
INTERNAL: frontendCtx.ServiceProxyRoundRobin("MYSV", GUICtx);
INTERNAL: GUICtx.ServiceProxyRoundRobin("MYSV", footerCtx);
// 4. on "footer" call the service "MYSV"
SETUP: footerCtx.ServiceCreate("MYSV", ...);
@ MQ_SLAVE_LOOPBACK
internal: the loopback-slave-id, (call my own services)

The MqRouteCreate will take the following action:

  1. create a round-robin-proxy on frontend and on GUI using service MYSV
  2. check if service MYSV is available on footer

Attention:

  1. With the proxy on frontend/GUI no other service with the name MYSV on frontend/GUI is possible and every service-call to frontend or GUI using the service MYSV will be redirected to the service MYSV on footer.
  2. The live-time of the Service-Layer-Routing is up to the next MqRouteDelete call using the same arguments. If the proxy-to-footer becomes invalid on a the GUI-context the routing-link will be recreated.
  3. The MqRouteCreate can be used multiple times with the same Route-Connection-String and service-token and overwrite=true to guarantee that a route is available.

Package-Layer-Routing

...

Function Documentation

◆ pymqmsgque_MqContextC_RouteCreate()

static OT_ProcRet pymqmsgque_MqContextC_RouteCreate ( MqContextC_ARGS )
static

Python: ctx.RouteCreate(route:string, service:string, ?overwrite:bool=false?) C-API
create/delete a routing-link between context an a service using route

Definition at line 2357 of file MqContextC_py.c.

2357 {
2359 OT_SETUP_VARARGS(2,3,RouteCreate_doc)
2360 MK_STRN route = 0;
2361 OT_CHECK_REQUIRED(OT_CHECK_STRN (route))
2362 MK_STRN service = 0;
2363 OT_CHECK_REQUIRED(OT_CHECK_STRN (service))
2364 MK_BOOL overwrite = false;
2365 OT_CHECK_OPTIONAL(OT_CHECK_bool (overwrite))
2366 OT_CHECK_NOARGS
2367 MkErrorC_Check(hdl,MqRouteCreate (hdl, route, service, overwrite));
2368 OT_retObj_SET_None
2369 goto end;
2370 error:
2371 OT_retObj_SET_Error
2372 end:
2374}
#define RouteCreate_doc
#define OT_SETUP_hdl
#define OT_retObj_RETURN
#define error
const MK_STRB * MK_STRN
bool MK_BOOL
#define MqRouteCreate(...)

◆ pymqmsgque_MqContextC_RouteDelete()

static OT_ProcRet pymqmsgque_MqContextC_RouteDelete ( MqContextC_ARGS )
static

Python: ctx.RouteDelete(route:string, service:string, ?overwrite:bool=false?) C-API
delete a routing-link created with MqRouteCreate

Definition at line 2377 of file MqContextC_py.c.

2377 {
2379 OT_SETUP_VARARGS(2,3,RouteDelete_doc)
2380 MK_STRN route = 0;
2381 OT_CHECK_REQUIRED(OT_CHECK_STRN (route))
2382 MK_STRN service = 0;
2383 OT_CHECK_REQUIRED(OT_CHECK_STRN (service))
2384 MK_BOOL overwrite = false;
2385 OT_CHECK_OPTIONAL(OT_CHECK_bool (overwrite))
2386 OT_CHECK_NOARGS
2387 MkErrorC_Check(hdl,MqRouteDelete (hdl, route, service, overwrite));
2388 OT_retObj_SET_None
2389 goto end;
2390 error:
2391 OT_retObj_SET_Error
2392 end:
2394}
#define RouteDelete_doc
#define MqRouteDelete(...)

◆ pymqmsgque_MqContextC_RouteGetPath()

static OT_ProcRet pymqmsgque_MqContextC_RouteGetPath ( MqContextC_ARGS )
static

Python: string ctx.RouteGetPath() C-API
return the absolut route-connection-string up to the current ctx

Definition at line 2438 of file MqContextC_py.c.

2438 {
2440 OT_SETUP_NOARG(RouteGetPath_doc)
2441 OT_CHECK_NOARGS
2442 MK_STRN path_out;
2443 MkErrorC_Check(hdl,MqRouteGetPath (hdl, &path_out));
2444 OT_retObj_SET_STR(path_out)
2445 goto end;
2446 error:
2447 OT_retObj_SET_Error
2448 end:
2450}
#define RouteGetPath_doc
#define MqRouteGetPath(...)

◆ pymqmsgque_MqContextC_RouteGetTree()

static OT_ProcRet pymqmsgque_MqContextC_RouteGetTree ( MqContextC_ARGS )
static

Python: MkBufferListC ctx.RouteGetTree() C-API
create an overview about all available routing-target and services …

Definition at line 2340 of file MqContextC_py.c.

2340 {
2342 OT_SETUP_NOARG(RouteGetTree_doc)
2343 OT_CHECK_NOARGS
2344 MK_BFL treeP_out;
2345 MkErrorC_Check(hdl,MqRouteGetTree (hdl, &treeP_out));
2346 OT_retObj_SET_BFL(treeP_out)
2347 goto end;
2348 error:
2349 OT_retObj_SET_Error
2350 end:
2352}
#define RouteGetTree_doc
#define MqRouteGetTree(...)

◆ pymqmsgque_MqContextC_RouteResolve()

static OT_ProcRet pymqmsgque_MqContextC_RouteResolve ( MqContextC_ARGS )
static

Python: [MqContextC...] ctx.RouteResolve(ident:string, ?retnum:int32=-1?) C-API
return a list of all context belonging to ident

Definition at line 2397 of file MqContextC_py.c.

2397 {
2399 OT_SETUP_VARARGS(1,2,RouteResolve_doc)
2400 MK_STRN ident = 0;
2401 OT_CHECK_REQUIRED(OT_CHECK_STRN (ident))
2402 MK_NUM retnum = -1;
2403 OT_CHECK_OPTIONAL(OT_CHECK_NI4 (retnum))
2404 OT_CHECK_NOARGS
2405 MQ_CTX_A retVal = MqRouteResolve (hdl, ident, retnum);
2406 OT_retObj_SET_List
2407 for (int i=0; i<retVal.size; i++) {
2408 OT_retObj_APPEND(OT_TMP_CTX_OBJ(retVal.data[i]));
2409 }
2410 goto end;
2411 error:
2412 OT_retObj_SET_Error
2413 end:
2415}
#define RouteResolve_doc
int32_t MK_NUM
#define MqRouteResolve(...)
array of MqContextC instances

◆ pymqmsgque_MqContextC_RouteTraverse()

static OT_ProcRet pymqmsgque_MqContextC_RouteTraverse ( MqContextC_ARGS )
static

Python: ctx.RouteTraverse(service:string, ?args:MkBufferListC...=None?) C-API
traverse a tree down and call service if available.

Definition at line 2418 of file MqContextC_py.c.

2418 {
2420 OT_SETUP_VARARGS(1,99,RouteTraverse_doc)
2421 MK_STRN service = 0;
2422 OT_CHECK_REQUIRED(OT_CHECK_STRN (service))
2423 MkBufferListCreateTLS_T(args,10);
2424 OT_CHECK_OPTIONAL(OT_CHECK_BAC (args))
2425 OT_CHECK_NOARGS
2426 MkErrorC_Check(hdl,MqRouteTraverse (hdl, service, args));
2427 OT_retObj_SET_None
2428 goto end;
2429 error:
2430 OT_retObj_SET_Error
2431 end:
2433}
#define RouteTraverse_doc
#define MkBufferListCreateTLS_T(name, num)
#define MqRouteTraverse(...)