NuttX Initialization Sequence

Overview

At the highest level, the NuttX initialization sequence can be represented in three phases:

  1. The hardware-specific power-on reset initialization,
  2. NuttX RTOS initialization, and
  3. Application Initialization.

This initialization sequence is really quite simple because the system runs in single-thread mode up until the point the that is starts the application. That means that the initialization sequence is just simple, straight-line function calls.

Just before starting the application, the system goes to multi-threaded mode and things can get more complex.

Each of these will be discussed in more detail in the following paragraphs.

Power-On Reset Initialization.

Overview

The software begins execution when the processor is reset. This usually at power-on, but all resets are basically the same whether they occur because of power-on, pressing the reset button, or on a watchdog timer expiration. The software that executes when the processor is reset is unique to the particular CPU architecture and is not a common part of NuttX. The kinds of things that must be done by the architecture-specific reset handling includes:

  1. Putting the processor in its operational state. This may include things like setting CPU modes; initializing co-processors, etc.
  2. Setting up clocking so that the software and peripherals operate as expected,
  3. Setting up the C stack pointer (and other processor registers)
  4. Initializing memory, and
  5. Starting NuttX.

Memory Initialization

In C implementations, there are two general classes of variable storage. First there are the initialized variables. For example, consider the global variable x:

int x = 5;

The C code must be assured that after reset, the variable x has the value 5. Initialized variable of this kind are retained in a special memory section called data (or .data).

Other variables are not initialized. Like the global variable y:

int y;

But the C code will still expect y to have an initial value. That initial value will be zero. All uninitialized variables of this this type have have the value zero. These uninitialized variables are retained in a section called bss (or .bss).

When we say that the reset handling logic initializes memory, we mean two things:

  1. It provides the (initial) values of the initialized variables by copying the values from FLASH into the .data section, and
  2. It resets all of the uninitialized variables to zero. It clears the .bss section.

STM32 F4 Reset

Lets walk through reset sequence of one particular processor. Let's look at the NuttX initialization for the STM32 F4 MCU. This reset logic can be found in two files:

  1. nuttx/arch/arm/src/stm32_vectors.S
  2. nuttx/arch/arm/src/stm32_start.c
nuttx/arch/arm/src/stm32_vectors.S

The roll of stm32_vectors.S in this reset sequence is very small. This file provides all of the STM32 exception vectors and power-on reset is simply another exception vector. Some important things to note about this file:

  1. .section .vectors, “ax”. This pseudo operation will place all of the vectors into a special section call .vectors. On of the STM32 F4 linker scripts is located at nuttx/configs/stm3240g-eval/nsh/ld.script. In that file, you can see that , you can see that the section .vectors is forced to lie at the very beginning of FLASH memory. The STM32 F4 can be configured to boot in different ways via strapping. If it is strapped to boot from FLASH, then the STM32 FLASH memory will be aliased to address 0x0000 0000 when the reset occurs. The is the address of the power-up reset interrupt vector.
  2. The first two 32-bit entries in the vector table represent the power-up exception vector (which we know will be positioned at address 0x0000 0000 when the reset occurs). Those two entries are:
    .word IDLE_STACK /* Vector  0: Reset stack pointer */
    .word __start    /* Vector  1: Reset vector */

The Cortex-M family is unique in the way that is handles the reset vector. Notice that there are two values: the stack pointer for the start-up thread (the IDLE thread), and the entry point in the IDLE thread. When the reset occurs, the the stack pointer is automatically set to the first value and then the processor jumps to reset entry point __start specified in the second entry. This means that the reset exception handling code can be implemented in C rather than assembly language.

nuttx/arch/arm/src/stm32_start.c

The reset vector __start lies in the file stm32_start.c and does the real, low-level architecture-specific initialization. This initialization includes:

  1. stm32_clockconfig(); Initialize the PLLs and peripheral clocking needed by the board.
  2. stm32_fpuconfig(); If the STM32 F4's hardware floating point is initialized, then configure the FPU and enable access to the FPU co-processors.
  3. stm32_lowsetup(); Enable the low-level UART. This is done very early in initialization so that we can get serial debug output to the console as soon as possible. If you are doing a board bring-up this is very important.
  4. stm32_gpioinit(); Perform any GPIO remapping that is needed (this is a stub for the F4, but the F1 family requires this step).
  5. showprogress('A'); This simply outputs the character 'A' on the serial console (only if CONFIG_DEBUG is enabled). If debug is enabled, you will always see the letters ABDE output on the console. That output all comes from this file.
  6. Next the memory is initialized:
    1. The .bss section is set to zero (Letter 'B' is then output if CONFIG_DEBUG is enabled), then
    2. The .data section is set to its initial values (The letter 'C' is output if debug is enabled),
  7. Then board-specific logic is initialized:
    1. stm32_boardinitialize(); This function resides with the board-specific logic. For the case of the STM3240G-EVAL board, this board initialization logic can be found at configs/stm3240g-eval/src/stm32_boot.c.
    2. For the case of the STM3240G-EVAL board, the stm32_boardinitialize() does the following operations:
      1. stm32_spiinitialize(); Initialize SPI chip selects if SPI is enabled.
      2. stm32_selectsram(); Configure the STM32 FSMC to support external SRAM if external SRAM support is enabled.
      3. stm32_autoled_initialize(); Initialize the on-board LEDs if they are used.
  8. When stm32_boardinitialize() returns to __start(), the low-level, architecture-specific initialization is complete and NuttX is started:
    1. os_start(); This is the NuttX entry point. It performs the next phase of RTOS-specific initialization and then brings up the application.

The operations performed by os_start() are discussed in the next paragraph.

NuttX RTOS Initialization

os_start()

When the low-level, architecture-specific initialization is complete and NuttX is started by calling the function os_start(). This function resides in the file nuttx/sched/os_start.c. The operations performed by os_start() are summarized below. Note that many of these features can be disabled from the NuttX configuration file and in that case those operations are not performed:

  1. Initializes some NuttX global data structures,
  2. Initializes the TCB for the IDLE (i.e, the thread that the initialization is performed on),
  3. sem_initialize(); Initialize the POSIX semaphore facilities. This needs to be done first because almost all other OS features depend on POSIX counting semaphores.
  4. kmm_initialize(); Initialize the memory manager (in most configurations, kmm_initialize() is an alias for the common mm_initialize()).
  5. irq_initialize(); Initialize the interrupt handler subsystem. This initializes only data structures; CPU interrupts are still disabled.
  6. wd_initialize(); Initialize the NuttX watchdog timer facility,
  7. clock_initialize(); Initialize the system clock,
  8. timer_initialize(); Initialize the POSIX timer facilities,
  9. sig_initialize(); Initialize the POSIX signal facilities,
  10. mq_initialize(); Initialize the POSIX message queue facilities,
  11. pthread_initialize(); Initialize the POSIX pthread facilities,
  12. fs_initialize(); Initialize file system facilities (currently an empty function),
  13. net_initialize(); Initialize networking facilities,

Up to this point, all of the initialization steps have only been software initializations. Nothing has interacted with the hardware. Rather, all of these steps simply prepared the environment so that things like interrupts and threads can function properly. The next phases depend upon that setup.

  1. up_initialize(); The processor specific details of running the operating system will be handled here. Such things as setting up interrupt service routines and starting the clock are some of the things that are different for each processor and hardware platform. See below for a specific example of the initialization steps performed by the ARM version of this function.
  2. board_initialize(); If CONFIG_BOARD_INITIALIZE is defined in the NuttX configuration, then an additional initialization call is made to a user-provided board_initialize() function. CONFIG_BOARD_INITIALIZE should be defined if there any any board-specific initialization actions that that need to be performed. Note above that there was an earlier, board-specific initialization call (that one to stm32_boardinitialize()). The difference here is the first, low-level initialization call was made before the OS was started. This would be place where low-level hardware configuration would need to be performed such as configuration of GPIO pins. board_initialize(), on the other hand, is called late in the initialization sequence; after the OS has been initialized but before any application tasks have been started. board_initialize() would be an ideal place to do board-specific initialization steps that depend on having an internalized OS such as memory allocations and initialization of device drivers.
  3. lib_initialize(); Initialize the C libraries. This is done last because the libraries may depend on the above.
  4. sched_setupidlefiles(); This is the logic that opens /dev/console and creates stdin, stdout, and stderr for the IDLE thread. All tasks subsequently created by the IDLE thread will inherit these file descriptors.
  5. os_bringup(); Create the initial tasks. This will be described more below.
  6. And finally enter the IDLE loop. After completing the initialization, the roll of the IDLE thread changes. It is now becomes the thread that executes only when there is nothing else to do in the system (hence, the name IDLE thread).

IDLE Thread Activities

As mention, the IDLE thread is the thread that executes only when there is nothing else to do in the system. It has the lowest priority in the system. It always has the priority 0. It is the only thread that is permitted to have the priority 0. And it can never be blocked (otherwise, what would run then?).

As a result, the IDLE thread is always in the g_readytorun list and, in fact, since that list is prioritized, can guaranteed to always be the final entry at the tail of the g_readytorun list.

The IDLE is an an infinite loop. But this does not make it a “CPU hog.” Since it is the lowest priority, the it can be suspended whenever anything else needs to run.

The IDLE thread does two things in this infinite loop:

  1. If the worker was not started (see os_bringup() below), then the IDLE thread will perform memory clean-up. Memory clean is required to handle deferred memory deallocation. Memory allocations must be deferred when the memory is freed in a context where the software does not have access to the heap and, hence, cannot truly free the memory (such as in an interrupt handler). In this case, the memory is simply put into a list of freed memory and, eventually, cleaned up by the IDLE thread. *NOTE*: The worker thread's primary function is as the “bottom half” for extended device driver processing. If the worker thread was started, then it will run at a higher priority than the IDLE thread. In this case, the worker thread will take over responsibility for cleaning up these deferred allocations.
  2. up_idle(); Then the loop calls up_idle(). The operations performed by up_idle() are architecture- and board-specific. In general, this is the location where CPU-specific reduced power operations may be performed.

os_bringup()

This function is called at the very end of the initialization sequence in os_start(), just before entering the IDLE loop. This function is located in nuttx/sched/os_bringup.c. This function starts all of the required threads and tasks needed to bring up the system. This function performed the following specific operations:

  1. If on-demand paging is configured, this function will start the page fill task. This is the task that runs in order to satisfy page faults in processors that have an MMU and in configurations where on-demand paging is enabled.
  2. The, if so configured, this function starts the worker thread. The worker thread may be used to execute any processing deferred to the worker thread via APIs provided in include/nuttx/wqueue.h. The worker thread's primary function is as the “bottom half” for extended device driver processing but can be used for a variety of purposes.
  3. Finally, os_bringup() will start the application task. By default this is the task whose entry has the name user_start(). user_start() is provided by application code and when it runs, it begins the application-specific phase of the initialization sequence as described below.

NOTE: The default user_start() entry point can be changed to use one of the named applications used by NSH. This is a start-up option that is not often used and will not be discussed further here.

STM32 F4 up_initialize()

All ARM-based MCUs share a common up_initialize() implementation provided at nuttx/arch/arm/common/up_initialize.c. The operations perform by this common ARM initialization will, however, call into facilities provided by the particular ARM chip. For the STM32 F4, those facilities would be provided by logic in files as nuttx/arch/arm/src/stm32. The common ARM initialization sequence is:

  1. up_calibratedelay(); One operation that must be performed during a CPU port is the calibration of timing delay loops. If CONFIG_ARCH_CALIBRATION is defined, then up_initialize() will perform some specific operations for the calibration of the delay loop. This, however, is not part of the normal initialization sequence. up_calibratedelay() is implemented within up_initialize.c.
  2. up_addregion(); The basic heap was set up during processing by os_start(). However, if the board supports multiple, discontiguous memory regions, any addition memory regions can be added to the heap by this function. For the STM32 F4, up_addregion() is implemented in nuttx/arch/arm/src/stm32/stm32_allocateheap.c.
  3. up_irqinitialize(); This function initialize the interrupt subsystem. For the STM32 F4, up_irqinitialize() is implemented in nuttx/arch/arm/src/stm32/stm32_irq.c.
  4. up_pminitialize(); If CONFIG_PM is defined, the function must initialize the power management subsystem. This MCU-specific function must be called very early in the intialization sequence before any other device drivers are initialized (since they may attempt to register with the power management subsystem). There is no implementation of up_pminitialize() for any STM32 platform.
  5. up_dmainitialize(); Initialize the DMA subsystem. For the STM32 F4, this DMA initialization can be found in nuttx/arch/arm/src/stm32/stm32_dma.c (which includes nuttx/arch/arm/src/stm32f4xxx_dma.c).
  6. up_timerinit(); Initialize the system timer interrupt. For the STM32 F4, this function initializes the ARM Cortex-M SYSTICK timer and can be found at nuttx/arch/arm/src/stm32/stm32_timerisr.c.
  7. devnull_register(); Registers the standard /dev/null.
  8. Then this function initializes the console device (if any). This means calling one of (1) up_serialinit(); for the standard serial driver (found at nuttx/arch/arm/src/stm32/stm32_serial.c for the STM32 F4), (2) lowconsole_init(); for the low-level, write-only serial console (found at nuttx/drivers/serial/lowconsole_init.c), or (2) ramlog_sysloginit() for the RAM console (found at nuttx/drivers/ramlog.c).
  9. up_netinitialize(); Initialize the network. For the STM32 F4, this function is in nuttx/arch/arm/src/stm32/stm32_eth.c.
  10. up_usbinitialize(); Initialize USB (host or device). For the STM32 F4, this function is in nuttx/arch/arm/src/stm32/stm32_otgfsdev.c.
  11. up_ledon(LED_IRQSENABLED); Finally, up_initialize() illuminates board-specific LEDs to indicate the IRQs are now enabled.

STM32 F4 IDLE thread

The default STM32 F4 IDLE thread is located at nuttx/arch/arm/src/stm32_idle.c. This default version does very little:

  1. It includes a example, “skeleton” function that illustrates that kinds of things that you can do if CONFIG_PM is enabled (this example code is not fully implemented in the default IDLE logic).
  2. The it executes the Cortex-M thumb2 instruction wfi which causes the CPU to sleep until the next interrupt occurs.

Application Initialization

At the conclusion of the OS initialization phase in os_start(), the user application is started by creating a new task at the entry point user_start(). There must be exactly one entry point called user_start() in every application built on top of NuttX. Any additional initialization performed in the user_start() function is purely application dependent.

A Simple Hello World Application

The simplest user application would be the “Hello, World!” example. See apps/examples/hello. Here is the whole example:

int user_start(int argc, char *argv[])
{
  printf("Hello, World!!\n");
  return 0;
}

In this case, no additional application initialization is needed. It just “says hello” and exits.

A Nutt Shell User Application/Command

The NuttShell (NSH) is a simple shell application that may be used with NuttX. It is described here: http://nuttx.org/Documentation/NuttShell.html. It supports a variety of commands and is (very) loosely based on the bash shell and the common utilities used in Unix shell programming.

NSH is implemented as a library that can be found at apps/nshlib. The NSH start-up sequence is very simple. As an example, the the code at apps/examples/nsh/nsh_main.c illustrates how to start NuttX. It simple does the following:

  1. If you have C++ static initializers, it will call your implementation of up_cxxinitialize() which will, in turn, call those static initializers. For the case of the STM3240G-EVAL board, the implementation of up_cxxinitialize() can be found at nuttx/configs/stm3240g-eval/src/up_cxxinitialize.c.
  2. This function then calls nsh_initialize() which initializes the NSH library. nsh_initialize() is described in more detail below.
  3. If the Telnet console is enabled, it calls nsh_telnetstart() which resides in the NSH library. nsh_telnetstart() will start the Telnet daemon that will listen for Telnet connections and start remote NSH sessions.
  4. If a local console is enabled (probably on a serial port), then nsh_consolemain() is called. nsh_consolemain() also resides in the NSH library. nsh_consolemain() does not return so that finished the entire NSH initialization sequence.

nsh_initialize()

The NSH initialization function, nsh_initialize(), be found in apps/nshlib/nsh_init.c. It does only three things:

  • nsh_romfsetc(); If so configured, it executes an NSH start-up script that can be found at /etc/init.d/rcS in the target file system. /etc is the location where a read-only, ROMFS file system is mounted by nsh_romfsetc(). The ROMFS image is, itself, just built into the firmware. By default, this rcS startup script contains the following logic:
    # Create a RAMDISK and mount it at XXXRDMOUNTPOUNTXXX
    
    mkrd -m XXXMKRDMINORXXX -s XXMKRDSECTORSIZEXXX XXMKRDBLOCKSXXX
    mkfatfs /dev/ramXXXMKRDMINORXXX
    mount -t vfat /dev/ramXXXMKRDMINORXXX XXXRDMOUNTPOUNTXXX

Where the XXXX*XXXX strings get replaced in the template when the ROMFS image is created:

  • XXXMKRDMINORXXX will become the RAM device minor number. Default: 0
  • XXMKRDSECTORSIZEXXX will become the RAM device sector size
  • XXMKRDBLOCKSXXX will become the number of sectors in the device.
  • XXXRDMOUNTPOUNTXXX will become the configured mount point. Default: /etc

This script will, then, create a RAMDISK, format a FAT file system on the RAM disk, and then mount the FAT filesystem at a configured mountpoint. This rcS template file can be found at apps/nshlib/rcS.template. The resulting ROMFS file system can be found in apps/nshlib/nsh_romfsimg.h.

* boardctl(); Next any architecture-specific NSH initialization will be performed (if any). The NSH initialization logic will all the non-standard OS interface boardctl() like: (void)boardctl(BOARDIOC_INIT, 0);. The first argument, the command BOARDIOC_INIT, indicates that the boardctl() is being requested to perform application-oriented initialization. In response to this command, boardctl() will call the board-specific implementation of board_app_initialize(). That function is not generally available to application level code in all configurations but can always be accesses via boardctl().

* board_app_initialize(): For the STM3240G-EVAL, this architecture specific initialization can be found at configs/stm3240g-eval/src/stm_nsh.c. This it does things like: (1) Initialize SPI devices, (2) Initialize SDIO, and (3) mount any SD cards that may be inserted.

More about board_initialize()

There are two possibilities then for application startup initialization: (1) In the application itself. For NSH, this means the call to nsh_archinitialize() from nshlib. And (2) from board_initialize(). Each works well in certain contexts but there are also things that I don't like about both possibilities:

(1) From nshlib: The basic problem is that nshlib is part of a user application. But nsh_archinitialize() is often part of the OS. So you may have a direct call from the user application code into non-standard OS functions. In a flat build this works fine and has only aesthetic issues. But it does not work at all in any other kinds of build configurations, that is, in either the protected or the kernel builds configurations. In those build configurations, you will get an undefined reference to nsh_archinitialize() because the kernel is built separately from the applications.

nsh_archinitialize() may do appropriate application kinds of initialization, but it often does inappropriate, internal OS-level initialization like instantiating device drivers. So there are really two levels of application specific initialization: (a) an OS/kernel level application initialization that needs to be performed before the application is started, and (b) a user/application level initialization that can be performed after the application has started.

(2a) board_initialize() is the OS/kernel level application initialization that is performed before the application has started. It usually works quite well and is usually a good replacement for nsh_archinitialize(). But often there is a problem: By default board_initialize() is called on the IDLE thread before the application is started. But the IDLE thread has limitations: It cannot wait for events so it can only be used for simple, straight-line initialization logic. That may be insufficient in some cases.

So for those cases there is yet another twist. First consider the board_initialize() calling sequence: board_initialize() is called directly from do_app_start() in nuttx/sched/init/os_bringup.c just before starting the application task. do_app_start() is, in turn, called from the function os_start_application(). If CONFIG_BOARD_INITTHREAD is not defined, then this is just a normal C function call. But if CONFIG_BOARD_INITTHREAD is defined, then an intermediate, trampoline kernel thread is started. That kernel thread executes do_app_start() and moves the initialization off of the IDLE thread. This works great but is a little more complex than I would like.

(2b) There is also special place for user/application initialization. That is the apps/platform platform directory. This directory should be a mirror of the nutx/configs directory. There should be a board directory in apps/platform for every board that is in nuttx/configs. It is in these apps/platform directories where nsh_archinitialize() should reside. Ideally, all occurrences of nsh_archinitialize() should be moved out of configs/<board>/src and into apps/platform/<board>.

You can see in apps/platform that there is not very much progress toward that. I violate my own architectural vision all of the time (as does everyone else). But such is life.

Customizing NSH Initialization

NSH provides many ways to customize its initialize operations. These are described in Section 4.0 of the NSH documentation.