MqContextC_SlaveApi_PY_API

MqContextC - create and manage a slave context … More...

Collaboration diagram for MqContextC_SlaveApi_PY_API:

Functions
static OT_ProcRet	pymqmsgque_MqContextC_SlaveGetMaster (MqContextC_ARGS)
	Python: MqContextC ctx.SlaveGetMaster() → C-API opposite function of MqSlaveGetFilter
static OT_ProcRet	pymqmsgque_MqContextC_SlaveGet (MqContextC_ARGS)
	Python: MqContextC ctx.SlaveGet(id:MqSlaveE|int32) → C-API get the slave-context from a master-context …
static OT_ProcRet	pymqmsgque_MqContextC_SlaveGetFilter (MqContextC_ARGS)
	Python: MqContextC ctx.SlaveGetFilter() → C-API get the filter-ctx or the master-ctx …
static OT_ProcRet	pymqmsgque_MqContextC_SlaveGetProxy (MqContextC_ARGS)
	Python: MqContextC ctx.SlaveGetProxy(id:MqSlaveE|int32) → C-API on slave return the master and on master return the slave identified by id.
static OT_ProcRet	pymqmsgque_MqContextC_SlaveCheck (MqContextC_ARGS)
	Python: bool ctx.SlaveCheck(id:MqSlaveE|int32) → C-API check if slave-id is valid
static OT_ProcRet	pymqmsgque_MqContextC_SlaveCreate (MqContextC_ARGS)
	Python: ctx.SlaveCreate(id:MqSlaveE|int32, slave:MqContextC) → C-API create a master/slave link between the master-parent-context and the slave-parent-context …
static OT_ProcRet	pymqmsgque_MqContextC_SlaveDelete (MqContextC_ARGS)
	Python: ctx.SlaveDelete(id:MqSlaveE|int32) → C-API Delete a slave object from a master/slave link identified by id. …
static OT_ProcRet	pymqmsgque_MqContextC_SlaveIs (MqContextC_ARGS)
	Python: bool ctx.SlaveIs() → C-API is the context a slave-context ? …
static OT_ProcRet	pymqmsgque_MqContextC_SlaveWorker (MqContextC_ARGS)
	Python: ctx.SlaveWorker(id:MqSlaveE|int32, ?fct:string="WORKER"?, ?args:MkBufferListC...=None?) → C-API create a master/slave link using the image of the ctx object self. …

Detailed Description

MqContextC - create and manage a slave context …

The master-slave-link is used to create a mesh of nodes defined by different parent-context. The master control the slave.

The master-slave-link is used to perform the following tasks:

• report error messages from the slave-context to the master-context
• to create a slave-child-context if a master-child-context is created
• to delete a slave-context if a master-context is deleted

In difference to the client-server-link the master-slave-link connect two independent parent-context in the same process or thread (e.g. node). This leads to the restriction that only the master-context can be a server-context because only one server-context per node is possible.

   node-0   |           node-1/2        |   node-3/4/5
===================================================================

| <- client/server link -> | <- client/server link -> |

             | <-- master/slave link --> |

                           |- client1-0 -|- server3 ...
             |-  server1  -|
             |             |- client1-1 -|- server4 ...
  client0-0 -|
             |-  server2  -|- client1-2 -|- server5 ...

Definition of the "master-context"

• the master-context is a parent-context without a child-context available.
• the master-context is a client-context or a server-context.
• the link between the master-context and the slave-context is done using MqSlaveWorker or MqSlaveCreate

Definition of the "slave-context"

• the slave-context is a parent-context without a child-context available.
• the slave-context is a client-context.
• the slave-context lifetime is controlled by the master-context.
• the slave-context report all error-messages to the master-context.
• the slave-context is identified by a unique-slave-id starting with 0.
• a special form of a slave-context is a worker-context

Definition of the "worker-context"

• the worker-context is a slave-context using the image of the master-context self.
• the master-context can be a server-context or a client-context.
• the worker-context is created using MqSlaveWorker
• the worker-context is identified by a unique-slave-id starting with 0.

Definition of the "slave-id"

• the slave-id is defined at enum MqSlaveE
• a slave is identified in his master-context by a slave-id
• the slave-id work like a defined well-known-port-number in /etc/services
• the slave-id is an integer value with valid values > 0
• it is a good practice to plan the usage of your slave-id(s)
• slave

  slave-id	value	definition
  MQ_SLAVE_MAX	1024	internal: the maximum slave-id … .
  MQ_SLAVE_USER	10	internal: start of user-defined-slave-id .
  MQ_SLAVE_LOOPBACK	0	internal: the loopback-slave-id, (call my own services) .
  MQ_SLAVE_FILTER	1	internal: the filter-slave-id, (on a master get the filter-slave) .
  MQ_SLAVE_MASTER	1	internal: the master-slave-id, (on a slave get the master) .
  MQ_SLAVE_OTHER	1	internal: on the master-ctx get the slave-ctx and on the slave-ctx get the master-ctx .

• range

  range	definition
  0 <= slave-id < MQ_SLAVE_MAX	range of valid slave-id's
  0 <= slave-id < MQ_SLAVE_USER	internale usage
  MQ_SLAVE_USER <= slave-id < MQ_SLAVE_MAX	external usage

Definition of the "LOOPBACK" (0) slave

•    client   |           server            |
  ===========================================
  
  | <--- client/server --->  | <-- loop --> |
  
              | <------ master/slave -----> |
  
    client -- | -- server -- | -- client -- #
                     ==             ==      #
                   server -- | -- client -- #

• the loopback has always the slave-id = 0 .
• the loopback has the same class as the parent, if reflection is not available the MqFactoryInitial is used.
• the loopback is used to call a service on the same process or thread.
• the loopback is a special filter without an additional process or thread to be started.
• the loopback is only internal accessible
• the service called by the loopback need the same attention as the service called by the filter, the context of the service is the loopback-context.
• the loopback can call MqServiceCreate to create a new link between a service-token and a service-method, by default all services from the master-context (the owner of the loopback) are also accessible by the loopback.
• in the service use MqSlaveGetMaster to get the master-context from the loopback-context.
• Example from MyLoopServer.py → create a new loop-server

  import sys
  from pymqmsgque import *
  
  # package-item
  class MyLoopServer(MqContextC):
  
    # factory constructor
    def __init__(self, tmpl=None):
      self.ConfigSetServerSetup(self.ServerSetup)
      # set the "mydata" attribute to the master-context
      self.mydata = "Hello World"
      super().__init__()
  
    # service to serve all EXTERNAL requests for token "HLWO"
    def HLWO_srv(self):
      # get the "loopback" context
      loop = self.SlaveGet(MqSlaveE.LOOPBACK)
      # call the LOOP service on the SAME server
      loop.Send("W","LOOP")
      # answer HLWO with string-return from LOOP
      self.Send("R", "C", loop.ReadSTR())
  
    # service to serve all INTERNAL requests for token "LOOP"
    def LOOP_srv(self):
      # get the "master" context 
      master = self.SlaveGetMaster()
      # answer LOOP with data from MASTER->mydata attribute
      self.Send("R", "C", master.mydata)
  
    # define a service as link between the token "HLWO" and the callback "HLWO_srv"
    def ServerSetup(self):
      # EXTERNAL: link the "HLWO" service with "HLWO_srv"
      self.ServiceCreate("HLWO",MyLoopServer.HLWO_srv)
      # INTERNAL: link the "LOOP" service with "LOOP_srv"
      self.SlaveGet(MqSlaveE.LOOPBACK).ServiceCreate("LOOP", MyLoopServer.LOOP_srv)
  
  # package-main
  if __name__ == "__main__":
  
    # create the "MyLoopServer" factory… and the instance
    srv = MqFactoryC.Add(MyLoopServer).New()
  
    try:
      srv.LinkCreate(sys.argv)
      srv.ProcessEvent(MqWaitOnEventE.FOREVER)
    except Exception as ex:
      srv.ErrorCatch(ex)
    finally:
      srv.Exit()
  

Performance analyse

The performance-test is created with:

• pipe
  • Nhi1Exec perfclient.c --parent --wrk ? @ perfserver.c
• spawn, fork, thread
  • Nhi1Exec -r=uds perfserver.c --spawn|fork|thread
  • Nhi1Exec -r=uds perfclient.c --parent --wrk ?
• the number of workers are set with the –wrk option
• the cpu is a xeon with 4/8
• the performance is calculated as worker-context-created / time-in-sec with a ~2sec (default) measurement period.

performance-test code:

• The test-setup is done as:

    perfclient                                worker                            perfserver
    ==========                                ======                            ==========
    |
    |- loop --wrk x
      |- MqSlaveWorker(...)               ->  worker[1] 
      |- MqSend(worker[1],"E","STR0..")   ->  PerfWorker_I160(...)
                                              |- loop endless
                                                |- MqContextCreate(...)
                                                |- MqLinkCreate(...)      <->   MqContextCreate(...)
                                                |- MqContextDelete(...)   <->   MqContextDelete(...)
    |- sleep x sec
    |- loop --wrk x
      |- MqSend(worker[1],"C"..,"END0")   ->  PerfWorker_END0(...)
      |                                       |- stop loop
      |- "callback" - add number to all   <-  |- return #context 

• Using the worker-code:

  // asynchronous call
  static enum MkErrorE PerfWorker_STR0( MQ_SERVICE_CALL_ARGS ) {
    MK_BFL largv = NULL;
    MK_BFL pargv = NULL;
    MK_I32 num   = MqReadI32_e(mqctx);
           pargv = MkBufferListDup(MqReadBFL_e(mqctx));
    enum MkErrorE ret = MK_OK;
   
    if (num == -1) num = 999999;  // endless
   
    // start test until "doNext" is "false" or "num" exceeds
    for (int i=0; doNext && i<num; i++) {
      // CREATE the context
      MQ_CTX ctx = MqContextCreate(NULL, mqctx);
      largv = MkBufferListDup(pargv);
      MqLinkCreate_C (ctx, largv) {
        MqContextErrorMove(mqctx,ctx);
        MqContextDelete(ctx);
        goto error;
      }
   
  //MqSend(ctx,"E","MARK:C","start");
   
      MkBufferListDelete(largv);
   
      // DELETE the context
      MqContextDelete(ctx);
   
      // count the loops
      worker_count++;
   
      // give PerfWorker_END0 time to update "doNext"
      MqProcessEvent(mqctx, MQ_WAIT_NO, MK_TIMEOUT_DEFAULT);
    }
   
  end:
  //printV("ret=%i, Delete=%p\n", ret, largv)
    MkBufferListDelete(largv);
    MkBufferListDelete(pargv);
   
    if (doNext == true) {
      // finish the test WITHOUT "END0" -> test on --num, answer required.
      MqSend_E(mqctx, "R", "I", worker_count);
    }
    return ret;
   
  error:
    if (doNext == true) {
      ret = MkErrorStack_1X(mqctx);
    } else {
      ret = MkErrorReset_1X(mqctx);
    }
    goto end;
  }

performance-test results:

• results generated using the debug environment

  setup	–wrk	# worker-context	performance	info
  pipe	1	2500	1000	the pipe start a new worker-context with spawn
  spawn	1	2500	<1000	same as pipe but use network-protocoll
  fork	1	3800	4000	the fork is faster than spawn
  thread	1	16500	9000	the thread is faster than fork
  pipe	4	8000	4500	the worker scale linear up to number of processors
  spawn	4	7600	<4500	-
  fork	4	23200	11500	-
  thread	4	55500	27500	-
  pipe	8	10000	5500	the additional scaling up to the max hyper-threading does not really help
  spawn	8	9100	<5500	-
  fork	8	23200	11500	-
  thread	8	55500	27500	-

Function Documentation

pymqmsgque_MqContextC_SlaveCheck()

static OT_ProcRet pymqmsgque_MqContextC_SlaveCheck ( MqContextC_ARGS )

static

Python: bool ctx.SlaveCheck(id:MqSlaveE|int32) → C-API
check if slave-id is valid

Definition at line 3269 of file MqContextC_py.c.

                                                              {
  OT_SETUP_hdl;
  OT_SETUP_ONEARG(SlaveCheck_doc);
  MQ_SLAVE_ID id = 0;
  OT_CHECK_REQUIRED(OT_CHECK_SLAVE_ID (id))
  OT_CHECK_NOARGS;
  OT_retObj_SET_BOL(MqSlaveCheck (hdl, id));
  goto end;
  error:
  OT_retObj_SET_Error;
  end:
  OT_retObj_RETURN;
}

pymqmsgque_MqContextC_SlaveCreate()

static OT_ProcRet pymqmsgque_MqContextC_SlaveCreate ( MqContextC_ARGS )

static

Python: ctx.SlaveCreate(id:MqSlaveE|int32, slave:MqContextC) → C-API
create a master/slave link between the master-parent-context and the slave-parent-context …

Definition at line 3284 of file MqContextC_py.c.

                                                               {
  OT_SETUP_hdl;
  OT_SETUP_VARARGS(2,2,SlaveCreate_doc);
  MQ_SLAVE_ID id = 0;
  OT_CHECK_REQUIRED(OT_CHECK_SLAVE_ID (id))
  MQ_CTX slave = 0;
  OT_CHECK_REQUIRED(OT_CHECK_CTX (slave,MK_NULL_NO))
  OT_CHECK_NOARGS;
  MkErrorC_Check(hdl,MqSlaveCreate (hdl, id, slave));
  OT_retObj_SET_None;
  goto end;
  error:
  OT_retObj_SET_Error;
  end:
  OT_retObj_RETURN;
}

pymqmsgque_MqContextC_SlaveDelete()

static OT_ProcRet pymqmsgque_MqContextC_SlaveDelete ( MqContextC_ARGS )

static

Python: ctx.SlaveDelete(id:MqSlaveE|int32) → C-API
Delete a slave object from a master/slave link identified by id. …

Definition at line 3302 of file MqContextC_py.c.

                                                               {
  OT_SETUP_hdl;
  OT_SETUP_ONEARG(SlaveDelete_doc);
  MQ_SLAVE_ID id = 0;
  OT_CHECK_REQUIRED(OT_CHECK_SLAVE_ID (id))
  OT_CHECK_NOARGS;
  MkErrorC_Check(hdl,MqSlaveDelete (hdl, id));
  OT_retObj_SET_None;
  goto end;
  error:
  OT_retObj_SET_Error;
  end:
  OT_retObj_RETURN;
}

pymqmsgque_MqContextC_SlaveGet()

static OT_ProcRet pymqmsgque_MqContextC_SlaveGet ( MqContextC_ARGS )

static

Python: MqContextC ctx.SlaveGet(id:MqSlaveE|int32) → C-API
get the slave-context from a master-context …

Definition at line 3218 of file MqContextC_py.c.

                                                            {
  OT_SETUP_hdl;
  OT_SETUP_ONEARG(SlaveGet_doc);
  MQ_SLAVE_ID id = 0;
  OT_CHECK_REQUIRED(OT_CHECK_SLAVE_ID (id))
  OT_CHECK_NOARGS;
  MQ_CTX ctx_out;
  MkErrorC_Check(hdl,MqSlaveGet (hdl, id, &ctx_out));
  OT_retObj_SET_CTX(ctx_out);
  goto end;
  error:
  OT_retObj_SET_Error;
  end:
  OT_retObj_RETURN;
}

pymqmsgque_MqContextC_SlaveGetFilter()

static OT_ProcRet pymqmsgque_MqContextC_SlaveGetFilter ( MqContextC_ARGS )

static

Python: MqContextC ctx.SlaveGetFilter() → C-API
get the filter-ctx or the master-ctx …

Definition at line 3235 of file MqContextC_py.c.

                                                                  {
  OT_SETUP_hdl;
  OT_SETUP_NOARG(SlaveGetFilter_doc);
  OT_CHECK_NOARGS;
  MQ_CTX ctx_out;
  MkErrorC_Check(hdl,MqSlaveGetFilter (hdl, &ctx_out));
  OT_retObj_SET_CTX(ctx_out);
  goto end;
  error:
  OT_retObj_SET_Error;
  end:
  OT_retObj_RETURN;
}

pymqmsgque_MqContextC_SlaveGetMaster()

static OT_ProcRet pymqmsgque_MqContextC_SlaveGetMaster ( MqContextC_ARGS )

static

Python: MqContextC ctx.SlaveGetMaster() → C-API
opposite function of MqSlaveGetFilter

Definition at line 3202 of file MqContextC_py.c.

                                                                  {
  OT_SETUP_hdl;
  OT_SETUP_NOARG(SlaveGetMaster_doc);
  OT_CHECK_NOARGS;
  MQ_CTX retVal = MqSlaveGetMaster (hdl);
  OT_retObj_SET_CTX(retVal);
  goto end;
  error:
  OT_retObj_SET_Error;
  end:
  OT_retObj_RETURN;
}

pymqmsgque_MqContextC_SlaveGetProxy()

static OT_ProcRet pymqmsgque_MqContextC_SlaveGetProxy ( MqContextC_ARGS )

static

Python: MqContextC ctx.SlaveGetProxy(id:MqSlaveE|int32) → C-API
on slave return the master and on master return the slave identified by id.

Definition at line 3250 of file MqContextC_py.c.

                                                                 {
  OT_SETUP_hdl;
  OT_SETUP_ONEARG(SlaveGetProxy_doc);
  MQ_SLAVE_ID id = 0;
  OT_CHECK_REQUIRED(OT_CHECK_SLAVE_ID (id))
  OT_CHECK_NOARGS;
  MQ_CTX ctx_out;
  MkErrorC_Check(hdl,MqSlaveGetProxy (hdl, id, &ctx_out));
  OT_retObj_SET_CTX(ctx_out);
  goto end;
  error:
  OT_retObj_SET_Error;
  end:
  OT_retObj_RETURN;
}

pymqmsgque_MqContextC_SlaveIs()

static OT_ProcRet pymqmsgque_MqContextC_SlaveIs ( MqContextC_ARGS )

static

Python: bool ctx.SlaveIs() → C-API
is the context a slave-context ? …

Definition at line 3318 of file MqContextC_py.c.

                                                           {
  OT_SETUP_hdl;
  OT_SETUP_NOARG(SlaveIs_doc);
  OT_CHECK_NOARGS;
  OT_retObj_SET_BOL(MqSlaveIs (hdl));
  goto end;
  error:
  OT_retObj_SET_Error;
  end:
  OT_retObj_RETURN;
}

pymqmsgque_MqContextC_SlaveWorker()

static OT_ProcRet pymqmsgque_MqContextC_SlaveWorker ( MqContextC_ARGS )

static

Python: ctx.SlaveWorker(id:MqSlaveE|int32, ?fct:string="WORKER"?, ?args:MkBufferListC...=None?) → C-API
create a master/slave link using the image of the ctx object self. …

Definition at line 3331 of file MqContextC_py.c.

                                                               {
  OT_SETUP_hdl;
  OT_SETUP_VARARGS(1,99,SlaveWorker_doc);
  MQ_SLAVE_ID id = 0;
  OT_CHECK_REQUIRED(OT_CHECK_SLAVE_ID (id))
  MK_STRN fct = "WORKER";
  OT_CHECK_OPTIONAL(OT_CHECK_STRN (fct))
  MkBufferListCreateTLS_T(args,10);
  OT_CHECK_OPTIONAL(OT_CHECK_BAC (args))
  OT_CHECK_NOARGS;
  MkErrorC_Check(hdl,MqSlaveWorker (hdl, id, fct, args));
  OT_retObj_SET_None;
  goto end;
  error:
  OT_retObj_SET_Error;
  end:
  OT_retObj_RETURN;
}