Memory model

As illustrated in the following figure, to use runtime loadable models, a portion of the internal RAMs (read/write AI regions) is fixed or reserved and has absolute addresses. These regions must not be used by the application during inference. A non-fixed executable RAM region is required to install a given model. This read/write region allows to resolve the relocatable references during the installation process at runtime.

(*) This region must be also accessible by the NPU in the case where the epoch controller is enabled.
(**) These regions should be also accessible by the MCU in the case where a operator/epoch is delegated on the MCU.

Limitations

• Only the STM32N6-based series with Neural-ART accelerator is supported.
• Addresses of the used internal/on-chip memory-pools are fixed/absolutes (USEMODE_ABSOLUTE attribute). Only the external addresses relative to the off-chip memories: flash and RAM type, can be relocatable/relative (or USEMODE_RELATIVE attribute)
  
• Only two relocatable/relative memory pools are supported. One for RO regions handling the weights/params and another for an external RW memory region. Note that the external RAM region can be also absolute.
• Secure mode generation and [XIP mode][ref_rel_xip_mode] are not supported with ARM clang/llvm toolchain (cmse is not compatible with ROPI/RWPI).
• Support for the encrypted weights/params throught the ST Edge AI core CLI is not supported.

Embedded compiler options for relocatable mode

To generate relocatable objects, according to the Arm® Embedded toolchain used, two sets of compilation options are considered (see makefiles from the $STEDGEAI_CORE_DIR/scripts/N6_reloc/resources folder):

• For a GCC Arm® Embedded toolchain, all the code, including the network runtime library, is compiled using the -fpic, -msingle-pic-base, and -mno-pic-data-is-text-relative options. The Arm® Cortex®-M r9 register is designated as the platform register for the global offset table (GOT). The primary task of the dynamic/runtime loader is to update the GOT table and the indirect references in the data structure. Both XIP & COPY modes are fully supported.
• For a CLANG/LLVM Arm® Embedded toolchain, all the code is compiled with the options -fropi and -frwpi. The r9 register is also used; however, some constant/text parts must also be updated by the dynamic/runtime loader, which does not allow support for the XIP mode.

Generated files

Note that the memory initializers for the memory regions which are only used for the activations are not requested.

NPU compiler and CLI options

There are no restrictions on the NPU compiler options used for this step, except for the usage of the --all-buffers-info option, which provides detailed information about the intermediate and activation buffers. Note that this additional information is removed during the generation of the runtime loadable model. For the memory-pool descriptor file, the following attributes are mandatory:

• “mode”: “USEMODE_RELATIVE” for external memory pools only. This is mandatory for the external flash part and optional for the external RAM part. Otherwise, use “USEMODE_ABSOLUTE”.
• “fformat”: “FORMAT_RAW” for all memory initializers.

Relocatable options

The -r/--reloc/--relocatable option can be used with the following parameters. These parameters are mainly forwarded to specialized Python scripts.

The parameters can be combined using a comma separator.

$ stedgeai generate -m -m <model-name>.tflite/onnx <gen_options> ---target stm32n6 -st-neural-art ... --reloc split,gen-c-file
 ...
 <output-directory-path>\network_rel.bin
 <output-directory-path>\network_data.bin
 <output-directory-path>\network_img_rel.c
 <output-directory-path>\network_img_rel.h

Memory layout information

Below information is reported in the log and also in a well-formatted file in JSON format: <network>_generate_rel.json.

 ...
 Runtime Loadable Model - Memory Layout (series="stm32n6npu")
 --------------------------------------------------------------------------------
 v8.0 dbg=True async=True sec=True 'Embedded ARM GCC' cpuid=D22 fpu=True float-abi=2

 XIP size             = 3,344      data(2620)+got(708)+bss(16) sections
 COPY size            = 32,544     +ro(29200) sections
 extra sections       = 1,112      got:708(21.2%) header+rel:404(1.4%)
 params size          = 25,713
 acts size            = 56,496
 binary file size     = 32,832
 params file size     = 25,720

 +--------------------------+----------------------------------+------+------------+-------+
 | name (addr)              | flags                            | foff | dst        | size  |
 +--------------------------+----------------------------------+------+------------+-------+
 | xSPI2 (0x20004c64)       | 0x01010500 RELOC.PARAM.0.RCACHED | 0    | 0x00000000 | 25713 |
 | AXISRAM5 (0x20004c6a)    | 0x03020200 RESET.ACTIV.WRITE     | 0    | 0x342e0000 | 56496 |
 | <undefined> (0x00000000) | 0x00000000 UNUSED                | 0    | 0x00000000 | 0     |
 +--------------------------+----------------------------------+------+------------+-------+
  Table: mempool c-descriptors (off=40000a00, 3 entries, from RAM)

 rt_ctx: c_name="network", acts_sz=56,496, params_sz=25,713, ext_ram_sz=0
 rt_ctx: rt_version_desc="atonn-v1.1.3-5-gdabeb3b4d (RELOC.GCC)"

The “mempool c-descriptors” table indicates the memory regions (part of the user memory-pools) and the flags which are considered by the ll_aton_reloc_install() function at runtime.

• The number 0 or 1 indicates the ID of the relocatable memory regions. Currently, only two regions are supported: 0 for a parameters/weights-only region in external flash and 1 for a read/write memory region in the external RAM.
• foff specifies the offset in the parameters/weights section to locate the associated memory initializer when requested.
• dst specifies the destination address. If not equal to zero, the address is an absolute address; otherwise, the region is a relocatable region.
• size specifies the size in bytes.

 +--------------------------+------------------------------+------+------------+-------+
 | name (addr)              | flags                        | foff | dst        | size  |
 +--------------------------+------------------------------+------+------------+-------+
 | AXISRAM6 (0x20004c64)    | 0x03020200 RESET.ACTIV.WRITE | 0    | 0x34350000 | 500   |
 | AXISRAM5 (0x20004c6d)    | 0x02030200 COPY.MIXED.WRITE  | 0    | 0x342e0000 | 33728 |
 | AXISRAM4 (0x20004c76)    | 0x03020200 RESET.ACTIV.WRITE | 0    | 0x34270000 | 40496 |
 | AXISRAM3 (0x20004c7f)    | 0x03020200 RESET.ACTIV.WRITE | 0    | 0x34200000 | 16000 |
 | <undefined> (0x00000000) | 0x00000000 UNUSED            | 0    | 0x00000000 | 0     |
 +--------------------------+------------------------------+------+------------+-------+
  Table: mempool c-descriptors (off=40000428, 5 entries, from RAM)

 +--------------------------+----------------------------------+------+------------+-------+
 | name (addr)              | flags                            | foff | dst        | size  |
 +--------------------------+----------------------------------+------+------------+-------+
 | xSPI1 (0x20004c62)       | 0x01020601 RELOC.ACTIV.1.WCACHED | 0    | 0x00000000 | 64496 |
 | xSPI2 (0x20004c68)       | 0x01010500 RELOC.PARAM.0.RCACHED | 0    | 0x00000000 | 32625 |
 | <undefined> (0x00000000) | 0x00000000 UNUSED                | 0    | 0x00000000 | 0     |
 +--------------------------+----------------------------------+------+------------+-------+
  Table: mempool c-descriptors (off=40001618, 3 entries, from RAM)

XIP execution mode

This execution mode is the privileged mode, where the code and weight sections are stored in the flash memory. Regarding memory placement, this approach is similar to the static method. This mode is efficient in terms of memory usage, only the executable RAM region to store the data/bss/got sections is requested.

COPY execution mode

This mode involves copying the code to a different memory location before execution. This process is useful for optimizing performance or managing memory access speeds. The requested size for the executable RAM region is critical. Along with the data, bss, and GOT sections, it also includes the text and rodata sections.

Memory-pool descriptor files

The "%STEDGEAI_CORE_DIR%/scripts/N6_reloc/test" contains a set of configuration and memory-pool descriptor files which can be used. Here are two main points to consider:

• If nonsecure context is used to execute the deployed model, the base @ of the different memory-pools should be set with a nonsecure address (ex. 0x24350000 instead of 0x34350000 for the AXIRAM6 memory).
• For the memory-pools representing an off-chip device, the "USEMODE_RELATIVE" attribute should be used.

For test purpose with STM32N6570-DK board, different ready-to-use configurations are provided.

For information and test purpose, the *non” reloc memory-pool descriptor files are provided which can be used with a normal deployment flow.

Part of the ./test/mpool/stm32n6_reloc.mpool file:

    ...
    {
        "fname": "AXISRAM6",
        "name":  "npuRAM6",
        "fformat": "FORMAT_RAW",
        "prop":   { "rights": "ACC_WRITE", "throughput": "HIGH", "latency": "LOW",
                    "byteWidth": 8, "freqRatio": 1.25, "read_power": 18.531, "write_power": 16.201 },
        "offset": { "value": "0x34350000", "magnitude":  "BYTES" },
        "size":   { "value": "448",        "magnitude": "KBYTES" }
    },
    {
        "fname": "xSPI1",
        "name":  "hyperRAM",
        "fformat": "FORMAT_RAW",
        "prop":   { "rights": "ACC_WRITE", "throughput": "MID", "latency": "HIGH",
                    "byteWidth": 2, "freqRatio": 5.00, "cacheable": "CACHEABLE_ON",
                    "read_power": 380, "write_power": 340.0, "constants_preferred": "true" },
        "offset": { "value": "0x90000000", "magnitude":  "BYTES" },
        "size":   { "value": "32",         "magnitude": "MBYTES" },
        "mode":   "USEMODE_RELATIVE"
    },
    {
        "fname": "xSPI2",
        "name":  "octoFlash",
        "fformat": "FORMAT_RAW",
        "prop":   { "rights": "ACC_READ",  "throughput": "MID", "latency": "HIGH", 
                    "byteWidth": 1, "freqRatio": 6.00, "cacheable": "CACHEABLE_ON",
                    "read_power": 110, "write_power": 400.0, "constants_preferred": "true" },
        "offset": { "value": "0x70400000", "magnitude":  "BYTES" },
        "size":   { "value": "64",         "magnitude": "MBYTES" },
        "mode":   "USEMODE_RELATIVE"
    }
    ...

“neural_art.json” file

The "%STEDGEAI_CORE_DIR%/scripts/N6_reloc/test" contains two examples of configuration file using the requested memory-pool descriptor files. They provide different profiles using different memory configurations which are aligned with the generic AI test validation.

Part of the ./test/neural_art_reloc.json file:

  ...
   "test" : {
    "memory_pool": "./mpools/stm32n6_reloc.mpool",
      "options": "--native-float --cache-maintenance --Ocache-opt --enable-virtual-mem-pools --Os
                    --optimization 3 --Oauto-sched --all-buffers-info --csv-file network.csv"
   },
   "test-ec" : {
    "memory_pool": "./mpools/stm32n6_reloc.mpool",
        "options": "--native-float --cache-maintenance --Ocache-opt --enable-virtual-mem-pools --Os
                    --optimization 3 --Oauto-sched --all-buffers-info --csv-file network.csv --enable-epoch-controller"
   },
  ...

Description

The ll_aton_reloc_install() function acts as a runtime dynamic loader. It is used to install and to create an instance of a memory-mapped runtime loadable module. By providing the model image pointer (file_ptr), configuration details, and neural network instance, users can set up the model for execution. The function performs compatibility checks, initializes memory pools, and installs/relocates code and data sections as needed.

Parameters

• file_ptr: A uintptr_t value representing the pointer to the image of the runtime loadable model. This parameter specifies the location of the model image to be installed.
• config: A pointer to an ll_aton_reloc_config structure. This parameter provides the configuration details for how the model should be installed, including memory addresses and sizes.
• nn_instance: A pointer to an NN_Instance_TypeDef structure. This parameter is updated to handle the installed model, creating an instance of the neural network.

Return Value

• The function returns an integer value. A return value of 0 typically indicates success, while a nonzero value indicates that an error occurred during the installation process.

Steps Executed

1. Checking step
   • This step checks the compatibility of the binary object against the runtime environment (static part of the firmware). The main points checked include:
     • The version and content of the binary file header.
     • MCU type and whether the FPU (floating-point unit) is enabled (context/setting of the caller is used).
     • Secure or nonsecure context.
     • Whether the binary module has been compiled with the LL_ATON_EB_DBG_INFO or LL_ATON_RT_ASYNC C-defines.
     • Version of the used LL_ATON files.
2. Memory-Pool initialization step
   • If requested, this step initializes the used memory regions for the given model. Specifically:
     • If read/write memory pools handle the params/weights section, the associated memory region is initialized with values from the params/weights section.
     • Optionally, if the AI_RELOC_RT_LOAD_MODE_CLEAR flag is set, the read/write memory region handling the activations is cleared.
3. Code/Data installation and relocation step
   • According to the AI_RELOC_RT_LOAD_MODE_COPY or AI_RELOC_RT_LOAD_MODE_XIP flag:
     • The code/data sections are copied into the executable RAM region.
     • The relocation process is performed to update references.
     • Register the call-backs

This function installs and creates an instance of a runtime loadable model referenced by the file_ptr parameter. The config parameter (of type ll_aton_reloc_config) indicates how to install the model, and the nn_instance (of type NN_Instance_TypeDef) is updated to handle the installed model. This function executes the following steps:

ll_aton_reloc_config C-struct

The purpose of the ll_aton_reloc_config C structure is to provide the parameters which are requested to install a runtime loadable model.

typedef struct _ll_aton_reloc_config {
    uintptr_t exec_ram_addr;  /* base@ of the exec memory region to place the relocatable code/data (8-Bytes aligned) */
    uint32_t exec_ram_size;   /* max size in byte of the exec memory region */
    uintptr_t ext_ram_addr;   /* base@ of the external memory region to place the external pool (if requested) */
    size_t ext_ram_size;      /* max size in byte of the external memory region */
    uintptr_t ext_param_addr; /* base@ of the param memory region (if requested) */
    uint32_t mode;
  } ll_aton_reloc_config;

• 'exec_ram_addr'/'exec_ram_size': These members indicate the base address (8-byte aligned) and the maximum size of the read/write executable RAM memory region. These parameters are mandatory. To determine the required size at runtime, the ll_aton_reloc_get_info function can be used.
• 'ext_ram_addr'/'ext_ram_addr': These members indicate the base address (8-byte aligned) and the maximum size of the read/write external RAM memory region, if requested. To determine the required size at runtime, the ll_aton_reloc_get_info function can be used.
• 'ext_param_addr': This member indicates the base address (8-byte aligned) of the memory region containing the parameters/weights of the deployed model. This option is required when the split option is used; otherwise, it must be set to NULL.
• 'mode': This member indicates the expected execution mode. Or-ed flags can be used. AI_RELOC_RT_LOAD_MODE_XIP or AI_RELOC_RT_LOAD_MODE_CLEAR flag is mandatory. AI_RELOC_RT_LOAD_MODE_CLEAR flag is optional.

Description

The ll_aton_reloc_set_callbacks function is used to overwrite the default registration of the callbacks done in the ll_aton_reloc_install function. This function is optional.

Parameters

• nn_instance: A pointer to the neural network instance (NN_Instance_TypeDef).
• cbs: A pointer to a ll_aton_reloc_callback structure (see ll_aton_reloc_network.h file).

Return Value

• The function returns an integer value. A return value of 0 typically indicates success, while a nonzero value indicates that an error occurred during the installation process.

Description

The ll_aton_reloc_get_info function is used to obtain the main dimensioning information from the image of a runtime loadable model. This information can include details such as the size, memory requirements, and other relevant attributes of the model. By providing a pointer to the model image and a reference to an ll_aton_reloc_info structure, users can retrieve and store the necessary information to properly configure and manage the runtime loadable model.

This function is particularly useful for setting up the memory regions and ensuring that the model can be correctly loaded and executed within the available resources.

Parameters

• file_ptr: A uintptr_t value representing the pointer to the image of the runtime loadable model. This parameter specifies the location of the model image from which the information will be retrieved.
• rt: A pointer to an ll_aton_reloc_info structure. This parameter is used to store the retrieved dimensioning information of the runtime loadable model. The function fills this structure with the relevant details.

Return Value

• The function returns an integer value. A return value of 0 typically indicates success, while a nonzero value indicates that an error occurred during the operation.

ll_aton_reloc_info C-struct

  typedef struct _ll_aton_reloc_info
  {
    const char *c_name;          /* c-name of the model */
    uint32_t variant;            /* 32-b word to handle the reloc rt version,
                                    the used ARM Embedded compiler,
                                    Cortex-Mx (CPUID) and if the FPU is requested */
    uint32_t code_sz;            /* size of the code (header + txt + rodata + data + got + rel sections) */
    uint32_t params_sz;          /* size (in bytes) of the weights */
    uint32_t acts_sz;            /* minimum requested RAM size (in bytes) for the activations buffer */
    uint32_t ext_ram_sz;         /* requested external ram size for the activations (and params) */
    uint32_t rt_ram_xip;         /* minimum requested RAM size to install it, XIP mode */
    uint32_t rt_ram_copy;        /* minimum requested RAM size to install it, COPY mode */
    const char *rt_version_desc; /* rt description */
    uint32_t rt_version;         /* rt version */
    uint32_t rt_version_extra;   /* rt version extra */
  } ll_aton_reloc_info;

Description

The ll_aton_reloc_get_mem_pool_desc function allows to obtain information about the part of the memory-pools which are used for a given model. By providing a pointer to the model image and an index, users can retrieve the necessary information through the returned ll_aton_reloc_mem_pool_desc structure.

The ll_aton_reloc_get_mem_pool_desc function allows users to obtain information about parts of the memory pools used for a given model. By providing a pointer to the model image and an index, users can retrieve the necessary information through the returned ll_aton_reloc_mem_pool_desc structure.

Parameters

• file_ptr: A uintptr_t value representing the pointer to the image of the runtime loadable model. This parameter specifies the location of the model image from which the information will be retrieved.
• index: Index of the requested descriptor.

Return Value

• The function returns a reference of a ll_aton_reloc_mem_pool_desc object. If the specified index is out of range, the function may return NULL.

Example

A typical snippet of code to display memory pool C-descriptors.

  ll_aton_reloc_mem_pool_desc *mem_c_desc;
  int index = 0;

  while ((mem_c_desc = ll_aton_reloc_get_mem_pool_desc((uintptr_t)bin, index)))
  {
    printf(" %d: flags=%x foff=%d dst=%x s=%d\n", index, mem_c_desc->flags,
           mem_c_desc->foff, mem_c_desc->dst, mem_c_desc->size);
    index++;
  }

ll_aton_reloc_mem_pool_desc C-struct

typedef struct _ll_aton_reloc_mem_pool_desc
{
  const char *name; /* name */
  uint32_t flags;   /* type definition: 32b:4x8b <type><data_type><reserved><id> */
  uint32_t foff;    /* offset in the binary file */
  uint32_t dst;     /* dst @ */
  uint32_t size;    /* real size */
} ll_aton_reloc_mem_pool_desc;

AI_RELOC_MPOOL_GET_XXX(flags) macros (see ll_aton_reloc_network.h file) can be used to know the attributes of the memory pool.

Description

The ll_aton_reloc_get_input/output_buffers_info function is used to obtain information about a specific input/output buffer of a neural network instance. This can be useful for understanding the structure and requirements of the input data for the neural network. By providing the neural network instance and the index of the desired input buffer, users can retrieve detailed information about the buffer, such as its size, type, and memory location.

Parameters

• nn_instance: A pointer to the neural network instance (NN_Instance_TypeDef). This parameter specifies the neural network instance for which the input/output buffer information is to be retrieved.
• num: An integer specifying the index of the input/output buffer whose description is to be retrieved. The index is zero-based, meaning that num = 0 refers to the first input buffer, num = 1 refers to the second input buffer, and so on.

Return Value

• The function returns a pointer to a LL_Buffer_InfoTypeDef structure, which contains the description of the specified input/output buffer. If the specified buffer index is out of range, the function may return NULL.

Description

Both ll_aton_reloc_set_input and ll_aton_reloc_set_output functions are used to configure the address of the input and output buffers for a neural network instance, respectively. By providing the neural network instance, buffer index, buffer pointer, and buffer size, users can set up the necessary memory regions for input and output data.

Parameters

• nn_instance: A pointer to the neural network instance (NN_Instance_TypeDef). This parameter specifies the neural network instance for which the output buffer is to be set.
• num: An unsigned integer specifying the index of the output buffer to be set. The index is zero-based.
• buffer: A pointer to the buffer that will hold the output data. This parameter specifies the memory location where the output data is stored.
• size: An unsigned integer specifying the size of the input/output buffer in bytes.

Return Value

• The function returns a value of type LL_ATON_User_IO_Result_t. This return value indicates the result of the operation, such as success or an error code.

Description

Both ll_aton_reloc_get_input and ll_aton_reloc_get_output functions are used to retrieve pointers to the input and output buffers for a neural network instance, respectively. By providing the neural network instance and buffer index, users can obtain direct access to the memory regions used for input and output data.

Parameters

• nn_instance: A pointer to the neural network instance (NN_Instance_TypeDef). This parameter specifies the neural network instance for which the input/output buffer pointer is to be retrieved.
• num: An unsigned integer specifying the index of the output buffer to be retrieved. The index is zero-based.

Return Value

• The function returns a pointer to the output buffer. If the specified buffer index is out of range or an error occurs, the function may return NULL.

“–name/-n” option

The --name/-n option allows to specify/overwrite the expected c-name/file-name of the loadable runtime model. By default, the name of the generated network files is used.

Default behavior.

$ python npu_driver.py -i <gen-dir>/network.c 
...
Generating files...
    creating "build\network_rel.bin" (size=..)

$ python npu_driver.py -i <gen-dir>/my_model.c
...
Generating files...
    creating "build\network_rel.bin" (size=..)

$ python npu_driver.py -i <gen-dir>/network.c -n toto
...
Generating files...
    creating "build\toto_rel.bin" (size=..)