External loader scenario

In this scenario, the memory initializers are exported as separate binary blobs or files. An external loader (e.g., bootloader or firmware component) is used to load the NPU memory initializers. The application, including the model and the AI runtime stack, and the memory initializers are deployed separately.

Application build system scenario

Memory initializers are generated as C objects, which can be directly imported and compiled by the application build process. A simple image is created embedding the entire AI runtime stack and the generated model. The memory contents are embedded into the application binary or linked as part of the firmware. The application itself handles memory initialization during startup.

This approach is suitable when:

• You want a single monolithic firmware image.
• Simplified deployment without external dependencies.

Memory pool descriptor attributes

When generating memory initializers for an NPU system, the memory pool descriptor contains key attributes that guide how memory buffers are positioned, addressed, and output to files.

Management of activation buffers

The required memory regions to handle activations are considered scratch buffers. These memory regions do not need to be filled with zeros before inference. The application must ensure the mapping of these regions by the neural processing unit (NPU) and microcontroller unit (MCU) subsystems during inference. Regarding the weights, the memory pool must be accessible by the MCU to support the software fallback mechanism. In most cases, defining the base addresses as absolute (hardcoded) is recommended to facilitate deployment. To avoid unpredictable situations, memory regions reserved for activations should be excluded from the application memory layout, such as the scatter memory file.

In scenarios where a simple NPU subsystem, such as a unique static random-access memory (SRAM), is shared with the MCU subsystem, a simple memory pool can be defined with a relative address resolved at linking time. In this case, the underlying region is treated as a non-initialized section by the C linker. In C linker terminology, this corresponds to a section like .bss or a custom section marked as NOLOAD. This approach enhances flexibility, making the memory layout more portable and easier to adapt without modifying hardcoded addresses.

Weights and/or activations

In most cases (the nominal or default case), a unique memory initializer is generated containing constant values representing the weights and/or parameters of the entire model. However, in some particular cases, for example, when only the internal NPU RAMs are defined, weights and activations can also be mixed in the same memory pool or multiple memory pools. The placement fully depends on the memory pools descriptor file provided by the user and the underlying heuristics of the NPU compiler.

Weights fetched from flash

The memory initializer is stored in non-volatile memory (or flash). The associated region is memory-mapped into the address space of the NPU and the MCU. During the inference, the weights are directly fetched from its storage region. Base address is absolute, fixed at generation time or linking time. No additional runtime or initialization steps are needed since the weights are directly accessible.

To achieve this, the following typical memory pool descriptor is used. The "prop" key properties is used to indicate that the constants (weights/parameters of the model) should preferably be placed in this RO memory region (with "rights": "ACC_READ" and "constants_preferred" : True).

{ ..
    "prop": { "rights": "ACC_READ",  "constants_preferred": "true", ..},
    "size":   { "value": "64",         "magnitude": "MBYTES" },
.. }

The "mode" and 'offset" attributes allow indicating how to manage the base address. It can be absolute ("mode": "USEMODE_ABSOLUTE") or relative ("mode": "USEMODE_RELATIVE").

{ ..
    "prop": { "rights": "ACC_READ",  "constants_preferred": "true", ..},
    "offset": { "value": "<address_in_flash>", "magnitude":  "BYTES" },
    "size":   { "value": "64",         "magnitude": "MBYTES" },
    "mode": "USEMODE_ABSOLUTE"  // "USEMODE_RELATIVE"
.. }

• In the first case, the "offset" key should be provided and its value is directly used to configure the processing units of the NPU (hardcoded value) as illustrated by the extracted part of code from the generated network.c file.

  ..
  .cache_allocate = 0,
  .addr_base = {(unsigned char *)(0x70000000UL) /* Equivalent hex address = <address_in_flash>UL */}, /* Conv2D_7_weights */
  .offset_start = 1219712,
  ..

• If "USEMODE_RELATIVE" is defined, a specific C-label is generated to reference the base address of the weights/params. The "offset" value is not considered.

  extern unsigned char _mem_pool_<pool_name>_Default[]; /* [1276737]; */
  ..
  .cache_allocate = 0,
  .addr_base = {(unsigned char *)ATON_LIB_VIRTUAL_TO_PHYSICAL_ADDR((uintptr_t)_mem_pool_<pool_name>_Default)}, /* Conv2D_7_weights */
  .offset_start = 1197920,
  ..

To complete the memory pool descriptor, the "fformat" attribute indicates the format of the generated memory initializer file.

{ ..
    "fformat": "FORMAT_RAW", // "FORMAT_C"
    "prop": { "rights": "ACC_READ",  "constants_preferred": "true", ..},
    "offset": { "value": "<address_in_flash>", "magnitude":  "BYTES" },
    "size":   { "value": "64",         "magnitude": "MBYTES" },
    "mode": "USEMODE_ABSOLUTE"  // "USEMODE_RELATIVE"
.. }

Following table defines the way to deploy the memory initializer according to the "mode” value:

Note 1: The configuration 1 is the default setting used by the proposed STM32N6 evaluation environment. The loading and initialization of the memory initializers is fully managed by the "n6_loader" Python script (including the loading of the weights in the internal SRAM). Note 2: Deployment can be facilitate if the symbol is placed at an ‘fixed’ address by a linker directive. Note 3: For absolute placement, no mechanism is in place to verify that the chosen address does not overlap with other critical application sections.

Typical generated c-file containing the C-array as const.

LL_ATON_ALIGNED(64) const uint64_t _mem_pool_<pool_name>_Default[159593] = {
  0xFEF508EAFC98FD05ULL,0xF92742D0E51C39F5ULL,0x09220BE6F428F21CULL,0xD6DF0858DF1CE62BULL,
  ...
  0x4A0E00FD20D7FA94ULL,0x35030B14B04748BEULL,0x0000000000000053ULL,
};

Weights fetched from sram

For the small models or for performance reasons, if the SRAM region is large enough to hold the weights and the activations, it can be interesting to place the weights in a low-latency/high-throughput memory region. Lower power consumption can be also a criterion. To support it, before inference execution, the associated memory initializers must be copied from the storage location (e.g., flash) to the execting location.

The default C-startup code can be used to copy the weights from the flash to the SRAM if the weights are generated as a non-const C-array. To do this, the associated memory pool can be defined as follow:

{ ..
    "fformat": "FORMAT_C",
    "prop": { "rights": "ACC_WRITE",  "constants_preferred": "true", ..},
    "offset": { "value": "<address_in_ram>", "magnitude":  "BYTES" },
    "size":   { "value": "1",         "magnitude": "MBYTES" },
    "mode": "USEMODE_RELATIVE"
.. }

The "constants_preferred" attribute allows to create a single preferable region to handle the weights. As the pool is marked as RW ("ACC_WRITE" attribute), a non-const C-array is created and it will be considered by the embedded target C-toolchain as an initialized data section.

LL_ATON_ALIGNED(64) uint64_t _mem_pool_<pool_name>_Default[15593] = {
  0xFEF508EAFC98FD05ULL,0xF92742D0E51C39F5ULL,0x09220BE6F428F21CULL,0xD6DF0858DF1CE62BULL,
  ...
  0x4A0E00FD20D7FA94ULL,0x35030B14B04748BEULL,0x0000000000000053ULL,
};

Be aware that to use the C-startup, before performing the copying operation (data section initialization), the destination memory should be previously enabled or clocked. If not, custom startup code is required to manage this sequence safely to ensure reliable system initialization and prevent memory access faults during startup or post-init sequence. Otherwise, specific application code should be designed to manage these copying operations before performing the inference.

Weights/activations mixed in sram

This case is similar to the previous case but no preferable memory pool is defined to handle the weights ("constants_preferred" attribute is not used and all memory pools have the "ACC_WRITE" attribute). It is the application’s responsibility to implement the logic to copy the weights according to the used "mode"/"fformat" before to perform the initialization of the model and to execute the inference.

NPU cache consideration

For an NPU subsystem from the STM32N6, the NPU cache can be used to accelerate access to external memories. If the associated memory pool descriptors set the "cacheable" attribute to "CACHEABLE_ON", the end-to-end deployment MUST ensure that the memory initializers are effectively placed in the external memories. Otherwise, placement in internal memory or flash without proper mapping can cause unpredictable inference execution.

RELATIVE mode and virtual memory pool support

When physically contiguous memory regions are mapped to different memory pools, the --enable-virtual-mem-pools option is recommended. This option places large buffers across these pools and optimizes memory placement to improve parallel access performance. Consequently, for these memory pools, "USEMODE_RELATIVE" cannot be used because a single relative base address is is invalid.

RELATIVE mode and Epoch controller

Bitstreams generated for the epoch controller are constant offline-generated objects, they cannot contain indirect or relative addresses directly. With "USEMODE_RELATIVE", the addresses of the referenced objects should be resolved/fixed during the linking phase. To manage these non-constant values, a specific mechanism is in place to update or patch the bitstreams at runtime with the fixed/resolved addresses. To do this, extra SRAM regions (bss section) are created to hold the copied and patched bitstreams. This operation is done (see generated network_ecblobs.h file) during the initialization phase of the model. This operation does not impact inference time, but additional SRAM size must be considered.

Memory pool properties

Using relative mode is inherently more flexible; it allows relaxing the dependency on the generation phase and the integration into the final firmware. However, be aware that in non-homogeneous memory subsystems like the STM32N6, the defined properties (such as ""throughput", ""latency", ""byteWidth", etc.) in the memory pool desciptors are the key factors used by the NPU compiler to optimize the placement of the buffers to guarantee the performance. Consequently, the application build system must ensure that the placement of different memory initializers matches the memory file descriptors used during compilation.