Symisc UnQLite

An Embeddable NoSQL Database Engine



UnQLite - Foreign Function Implementation Guide.

The following quick guide is what you do to start experimenting with the Jx9 (Via UnQLite) foreign function mechanism without having to do a lot of tedious reading.


Foreign functions are used to add Jx9 functions or to redefine the behavior of existing Jx9 functions from the outside environment (see below) to the underlying virtual machine. This mechanism is know as “In-process extending”. After successful call to unqlite_create_function(), the installed function is available immediately and can be called from the target Jx9 code.


A foreign function is simply a client function typically implemented in C which is responsible of performing the desired computation. There are many reasons to implement foreign functions, one of them is to perform system calls and gain access to low level resources which is impossible to do using standard Jx9 functions.

Another reason to implement foreign functions is performance. That is, a foreign function implemented in C will typically run 10 times faster than its Jx9 counterpart, this is why most of the built-in Jx9 functions (Over 312 as of this release) such as db_store(), db_create(), json_encode(), db_fetch(), array_merge(), chdir(), fread(), fopen() and many more are implemented in C rather than Jx9.


The signature of the foreign function is as follows:


int (*xFunc)(unqlite_context *pCtx,int argc,unqlite_value **argv)


The C function must accept three parameters. The first parameter is a pointer to a unqlite_context structure. This is the context in which the foreign function executes. In other words, this parameter is the intermediate between the foreign function and the underlying virtual machine.


The application-defined foreign function implementation will pass this pointer through into calls to dozens of interfaces, these includes:

unqlite_context_output(), unqlite_context_throw_error(), unqlite_context_new_scalar(), unqlite_context_user_data(), unqlite_context_alloc_chunk() and many more.


The second parameter is the total number of arguments passed to the foreign function.


The last parameter is an array of pointers to unqlite_value which represents function arguments. The implementation of the foreign functions can extract their contents via one of these interfaces:


unqlite_value_to_int()

unqlite_value_to_bool()

unqlite_value_to_int64()

unqlite_value_to_double()

unqlite_value_to_string()

unqlite_value_to_resource()

unqlite_value_compare()


The computation result (i.e. the return value) of the foreign function can be set via one of these interfaces:


unqlite_result_int()

unqlite_result_int64()

unqlite_result_bool()

unqlite_result_double()

unqlite_result_null()

unqlite_result_string()

unqlite_result_value()

unqlite_result_resource()

unqlite_result_string_format()


If no call is made to one of these interfaces, then a NULL return value is assumed.


The implementation of the foreign function must return UNQLITE_OK on success, but if the callbacks wishes to abort processing (to stop program execution) and thus to emulate the Jx9 die construct, it must return UNQLITE_ABORT instead.

Typical Implementation Of A Foreign Function

A typical implementation of a foreign function would perform the following operations in order:

  1. Check if the given arguments are of the expected numbers and expected types. If the function accepts arguments, this is the first operation it must perform before doing any serious work. For that, a set of interfaces are available:


    unqlite_value_is_int

    unqlite_value_is_float

    unqlite_value_is_bool

    unqlite_value_is_string

    unqlite_value_is_null

    unqlite_value_is_numeric

    unqlite_value_is_callable

    unqlite_value_is_json_array

    unqlite_value_is_scalar

    unqlite_value_is_json_object

    unqlite_value_is_resource

    unqlite_value_is_empty


    To check if the given arguments are of the expected number, simply compare the expected number with the argc parameter (Second parameter the foreign function takes).


  2. If the foreign function accepts arguments, it may need to extract their contents via the following set of interfaces:


    unqlite_value_to_int()

    unqlite_value_to_bool()

    unqlite_value_to_int64()

    unqlite_value_to_double()

    unqlite_value_to_string()

    unqlite_value_to_resource()

    unqlite_value_compare()


  3. Perform the desired computation.


  4. Allocate additional unqlite_value via unqlite_context_new_scalar() and/or unqlite_context_new_array() especially if the foreign function works with JSON arrays or JSON objects (see below for a working example).


  5. If something goes wrong while processing the input, the foreign function can throw an error via the unqlite_context_throw_error() or unqlite_context_throw_error_format() interfaces.


  6. If the foreign function wishes to output a message and thus to emulate the Jx9 print construct, it may call unqlite_context_output() or unqlite_context_output_format().


  7. When done, the computation result (i.e. return value) of the foreign function can be set via one of the these interfaces:


    unqlite_result_int()

    unqlite_result_int64()

    unqlite_result_bool()

    unqlite_result_double()

    unqlite_result_null()

    unqlite_result_string()

    unqlite_result_value()

    unqlite_result_resource()

    unqlite_result_string_format()


  8. And finally, return UNQLITE_OK when done. Note that you do not need to release any allocated resource manually via  unqlite_context_release_value(),   the virtual machine will release any allocated resources for you automatically.

Foreign Function Examples

We will start our examples with the simplest foreign function implemented in C which perform a simple right shift operation on a given number and return the new shifted value as it's computation result (i.e. return value), here is the C code:


  1. int shift_func(

  2.   unqlite_context *pCtx, /* Call Context */

  3.   int argc, /* Total number of arguments passed to the function */

  4.   unqlite_value **argv /* Array of function arguments */

  5. )

  6. {

  7.   int num;

  8.   /* Make sure there is at least one argument and is of the

  9.    * expected type [i.e: numeric].

  10.    */

  11.   if( argc < 1 || !unqlite_value_is_numeric(argv[0]) ){

  12.    /*

  13.     * Missing/Invalid argument,throw a warning and return FALSE.

  14.     * Note that you do not need to log the function name,JX9 will

  15.     * automatically append the function name for you.

  16.     */

  17.    unqlite_context_throw_error(pCtx,UNQLITE_CTX_WARNING,"Missing numeric argument");

  18.   /* Return false */

  19.    unqlite_result_bool(pCtx,0);

  20.    return UNQLITE_OK;

  21.   }

  22.   /* Extract the number */

  23.   num = unqlite_value_to_int(argv[0]);

  24.   /* Shift by 1 */

  25.   num <<= 1;

  26.   /* Return the new value */

  27.   unqlite_result_int(pCtx,num);

  28.   /* All done */

  29.   return UNQLITE_OK;

  30. }


On line 11, we check if there is at least one argument and is numeric (integer or float or a string that looks like a number). If not, we throw an error on line 17 and we return a Boolean false (zero) on line 19 using unqlite_result_bool().


We extract the function argument which is the number to shift on line 23 using unqlite_value_to_int().


We perform the desired computation (right shift the given number by one) on line 25.


And finally, we return the new value as our computation result one line 27 using a call to unqlite_result_int().


Now after installing this function (Compile this C file for a working example), it can be called from your Jx9 script as follows:


   print shift_func(150); //you should see 300.


Another simple foreign function named date_func(). This function does not expect arguments and return the current system date in a string. A typical call to this function would return something like: 2012-23-09 13:53:30. Here is the implementation


  1. int date_func(

  2.   unqlite_context *pCtx, /* Call Context */

  3.   int argc, /* Total number of arguments passed to the function */

  4.   unqlite_value **argv /* Array of function arguments*/

  5. ){

  6.   time_t tt;

  7.   struct tm *pNow;

  8.   /* Get the current time */

  9.   time(&tt);

  10.   pNow = localtime(&tt);

  11.   /*

  12.    * Return the current date.

  13.    */

  14.   unqlite_result_string_format(pCtx,

  15.     "%04d-%02d-%02d %02d:%02d:%02d", /* printf() style format */

  16.      pNow->tm_year + 1900, /* Year */

  17.      pNow->tm_mday, /* Day of the month */

  18.      pNow->tm_mon + 1, /* Month number */

  19.      pNow->tm_hour, /* Hour */

  20.      pNow->tm_min, /* Minutes */

  21.      pNow->tm_sec /* Seconds */

  22.   );

  23.   /* All done */

  24.   return UNQLITE_OK;

  25. }


We get the current date on line 10 using the libc localtime() routine and we return it using  unqlite_result_string_format  on line 14. This function is a work-alike of the "printf()" family of functions from the standard C library which is used to append a formatted string.


Compile this C file for a working version of the function defined above plus others foreign functions.

Working With JSON Arrays/Objects

Working with JSON arrays or objects is relatively easy and involves two or three additional call to Jx9 (Via UnQLite) interfaces. A typical implementation of a foreign function that works with arrays and/or objects would perform the following operations:

  1. Call unqlite_context_new_array() to allocate a fresh unqlite_value of type array or object. The new array/object is empty and need to be populated. This interface is unified for JSON arrays as well JSON objects.

  2. Call unqlite_context_new_scalar() to allocate a new scalar unqlite_value. This value is used to populate array/object entries with the desired value.

  3. Populate the array/object using one or more calls to unqlite_array_add_elem() and its wrappers. Again, these interfaces are unified for JSON arrays and objects.

  4. Finally, return the fresh array/object using the unqlite_result_value() interface.

We'll start our examples by a simple foreign function named array_time_func(). This function does not expects arguments and return the current time in a JSON array. A typical output of this function would look like this:


print(array_time_func());
/* JSON Array */
[14,53,30]

Here is the C code.

  1. int array_time_func(unqlite_context *pCtx,int argc,unqlite_value **argv)

  2. {

  3.   unqlite_value *pArray; /* Our JSON Array */

  4.   unqlite_value *pValue; /* Array entries value */

  5.   time_t tt;

  6.   struct tm *pNow;


  7.   /* Get the current time first */

  8.   time(&tt);

  9.   pNow = localtime(&tt);


  10.   /* Create a new json array */

  11.   pArray = unqlite_context_new_array(pCtx);

  12.   /* Create a worker scalar value */

  13.   pValue = unqlite_context_new_scalar(pCtx);

  14.   if( pArray == 0 || pValue == 0 ){

  15.     unqlite_context_throw_error(pCtx,JX9_CTX_ERR,"Fatal, JX9 is running out of memory");

  16.     /* emulate the die() construct */

  17.     return UNQLITE_ABORT;

  18.   }


  19.   /* Populate the JSON array.

  20.    * Note that we will use the same worker scalar value (pValue) here rather than

  21.    * allocating a new value for each array entry. This is due to the fact

  22.    * that the populated array will make it's own private copy of the inserted

  23.    * key(if available) and it's associated value.

  24.    */


  25.    unqlite_value_int(pValue,pNow->tm_hour); /* Hour */

  26.    /* Insert the hour at the first available index */

  27.    unqlite_array_add_elem(pArray,0,pValue /* Will make it's own copy */);


  28.   /* Overwrite the previous value */

  29.   unqlite_value_int(pValue,pNow->tm_min); /* Minutes */

  30.   /* Insert minutes */

  31.   unqlite_array_add_elem(pArray,0,pValue /* Will make it's own copy */);


  32.   /* Overwrite the previous value */

  33.   unqlite_value_int(pValue,pNow->tm_sec); /* Seconds */

  34.   /* Insert seconds */

  35.   unqlite_array_add_elem(pArray,0,pValue /* Will make it's own copy */);


  36.   /* Return the array as the function return value */

  37.   unqlite_result_value(pCtx,pArray);


  38.   /* All done. Don't worry about freeing memory here, every

  39.    * allocated resource will be released automatically by the engine

  40.    * as soon we return from this foreign function.

  41.    */

  42.   return UNQLITE_OK;

  43. }

Get a Copy of The C File


We allocate a fresh empty JSON array on line 13 using a call to unqlite_context_new_array(). Also, we allocate a fresh scalar value one line 15 using a call to unqlite_context_new_scalar().


We populate the scalar unqlite_value with the desired value, here the current hour on line 29 using a call to unqlite_value_int() and we insert it in the new array using unqlite_array_add_elem() on line 31.


This process is repeated again on line 34, 36, 39 and 41. Note that we use the same scalar value to populate the array with the three entries. This is due to the fact that a call to unqlite_array_add_elem() and its wrappers will make a private copy of the inserted value and so it is safe to use the same scalar value for other insertions.


Finally, we return our array as our computation result one line 44 using unqlite_result_value().

Other useful links

Check out the Introduction To The UnQLite C/C++ Interface for an introductory overview and roadmap to the dozens of UnQLite interface functions.

A separate document, The UnQLite C/C++ Interface, provides detailed specifications for all of the various C/C++ APIs for UnQLite. Once the reader understands the basic principles of operation for UnQLite, that document should be used as a reference guide.

Any questions, visit the Support Page for additional information.



Symisc Systems
Copyright © Symisc Systems