Linux Process vs NuttX Tasks

You may be used to running programs that are stored in files on Linux or Windows.  If you transition to using NuttX tasks on an MCU with limited resources, you will encounter some behavioral differences. This Wiki page will summarize a few of those differences.

NuttX Build Types

NuttX can be built in several different ways:

  • Kernel Build. The kernal build, selected with CONFIG_BUILD_KERNEL, uses the MCU's Memory Management Unit (MMU) to implement processes very similar to Linux processes. There is no interesting discussion here; NuttX behaves very much like Linux.
  • Flat Build. Most resource-limited MCUs have no MMU and the code is built as a blob that runs in an unprotected, flat address space out of on-chip FLASH memory. This build mode is selected with CONFIG_BUILD_FLAT and is, by far, the most common way that people build NuttX. This is the interesting case to which this Wiki page is directed.
  • Protected Build. Another build option is the protected build. This is essentially the same as the flat build, but uses the MCU's Memory Protection Unit (MPU) to separate unproctect user address ranges from protected system address ranges. The comments of this Wiki page also apply in this case.

Initialization of Global Variables

Linux Behavior

If you are used to writing programs for Linux, then one thing you will notice is that global variables are initialized only once when the system powers up. For example. Consider this tiny program:

bool test = true;

int main(int argc, char **argv)
{
  printf("test: %i\n", test);
  test = false;
  printf("test: %i\n", test);
  return 0;
}

If you build and run this program under Linux, you will always see this output:

test: 1
test: 0

In this case, the global variables are re-initialized each time that you load the file into memory and run it.

NuttX Flat-Build Behavior

But if you build this program into on-chip FLASH and start it as a task (via, say, task_start()) you will see this the first time that you run the program:

test: 1
test: 0

But after that, you will always see:

test: 0
test: 0

The test variable was initialized to true (1) only once at power up, but reset to false (0) each time that the program runs.

If you want the same behavior when the program is built into the common FLASH blob, then you will have modify the code so that global variables are explicitly reset each time the program runs like:

bool test;
  
int main(int argc, char **argv)
{
  test = true;
  printf("test: %i\n", test);
  test = false;
  printf("test: %i\n", test);
  return 0;
}

NuttX Load-able Programs

If you load programs from an file into RAM and execute them, as Linux does, then NuttX will again behave like Linux. Because the flat build NuttX works the same way:  When you execute a NuttX ELF or NxFLAT module in a file, the file is copied into RAM and the global variables are initialized before the program runs.

But code that is built into FLASH works differently.  There is only one set of global variables:  All of the global variables for the blob that is the monolithic FLASH image.  They are all initialized  once at power-up reset.

This is one of the things that makes porting Linux applications into the FLASH blob more complex.  You have to manually initialize each global variable in the main() each time your start the task.

Global Variables and Multiple Task Copies

It is better to avoid the use of global variables in the flat build context whenever possible because that usage adds another limitation:  No more that one copy of the program can run at any given time.  That is because the global variables are shared by each instance (unlike, again, running a program from a file where there is a private copy of each global variable).

One way to support multiple copies of an in-FLASH program is to move all global variables into a structure. If the amount of memory need for global variables is small, then each main() could simply allocate a copy of that structure on the stack. In the simple example above, this might be:

struct my_globals_s
{
  bool test;
}

int main(int argc, char **argv)
{
  struct my_globals_s my_globals = { true };

  printf("test: %i\n", my_globals.test);
  my_globals.test = false;
  printf("test: %i\n", my_globals.test);
  return EXIT_SUCCESS;
}

A pointer to the structure containing the allocated global variables would then have to passed as a parameter to every internal function that needs access to the global variables. So you would change a internal function like:

static void print_value(void)
{
  printf("test: %i\n", test);
}

to

static void print_value(FAR struct my_globals_s *globals)
{
  printf("test: %i\n", globals->test);
}

Then pass a reference to the allocated global data structure each time that the function is called like:

print_value(&my_globals);

If the size of the global variable structure is large, then allocating the instance on the stack might not be such a good idea. In that case, it might be better to allocate the global variable structure using malloc(). But don't forget to free() the allocated variable structure before exiting! (See the following Memory Clean-Up discussion).

struct my_globals_s
{
  bool test;
}

int main(int argc, char **argv)
{
  FAR struct my_globals_s *my_globals;

  my_globals = (FAR struct my_globals_s *)malloc(sizeof(struct my_globals_s));
  if (my_globals = NULL)
    {
      fprintf(stderr, "ERROR: Failed to allocate state structure\n");
      return EXIT_FAILURE;
    }

  my_globals=>test = true;
  printf("test: %i\n", my_globals->test);
  my_globals=>test = false;
  printf("test: %i\n", my_globals->test);

  free(my_globals);
  return EXIT_SUCCESS;
}

Memory Clean-Up

Linux Process Exit

Another, unrelated thing that makes porting Linux programs into the FLASH blob is the memory clean-up.  When a Linux process exits, its entire address environment is destroyed including all of allocated memory. This tiny program will not leak memory if implemented as a Linux process:

int main(int argc, char **argv)
{
  char *buffer = malloc(1024);
  ... do stuff with buffer ...
  return 0;
}

That same program, if ported into the FLASH blob will now have memory leaks because there is no automatic clean-up of allocated memory when the task exits. Instead, you must explicitly clean up all allocated memory by freeing it:

int main(int argc, char **argv)
{
  char *buffer = malloc(1024);
  ... do stuff with buffer ...
  free(buffer);
  return 0;
}

The memory clean-up with the Linux process exits is a consequent of the teardown of the process address environment when the process terminates. Each process contains its own heap; when the process address environment is torndown, that process heap is returned to the OS page allocator. So the memory clean-up basically comes for free.

NuttX Task Exit

But when you run a task in the monolithic, on-chip FLASH blob, you share the same heap with all other tasks. There is no magic clean-up that can find and free your tasks's allocations within the common heap (see “Ways to Free Memory on Task Exit”)

NuttX Process Exit

NOTE that when you run processes on NuttX (with CONFIG_BUILD_KERNEL), NuttX also behaves the same way as Linux:  The address environment is destroyed with the task exits and all of the memory is reclaimed.  But all other cases will leak memory.

Ways to Free Memory on Task Exit

There are ways that you could associate allocated memory with a task so that it could cleaned up when the task exits. That approach has been rejected, however, because (1) it could not be done reliably, and (2) it would add a memory allocation overhead that would not be acceptable in context where memory is constrained. Below is the full text from the top-level TODO list coped on 2016-08-15:

Title:       FREE MEMORY ON TASK EXIT
Description: Add an option to free all memory allocated by a task when the
             task exits. This is probably not be worth the overhead for a
             deeply embedded system.
  
             There would be complexities with this implementation as well
             because often one task allocates memory and then passes the
             memory to another:  The task that "owns" the memory may not
             be the same as the task that allocated the memory.
  
             Update.  From the NuttX forum:
             ...there is a good reason why task A should never delete task B.
             That is because you will strand memory resources. Another feature
             lacking in most flat address space RTOSs is automatic memory
             clean-up when a task exits.
  
             That behavior just comes for free in a process-based OS like Linux:
             Each process has its own heap and when you tear down the process
             environment, you naturally destroy the heap too.
  
             But RTOSs have only a single, shared heap. I have spent some time
             thinking about how you could clean up memory required by a task
             when a task exits. It is not so simple. It is not as simple as
             just keeping memory allocated by a thread in a list then freeing
             the list of allocations when the task exists.
  
             It is not that simple because you don't know how the memory is
             being used. For example, if task A allocates memory that is used
             by task B, then when task A exits, you would not want to free that
             memory needed by task B. In a process-based system, you would
             have to explicitly map shared memory (with reference counting) in
             order to share memory. So the life of shared memory in that
             environment is easily managed.
  
             I have thought that the way that this could be solved in NuttX
             would be: (1) add links and reference counts to all memory allocated
             by a thread. This would increase the memory allocation overhead!
             (2) Keep the list head in the TCB, and (3) extend mmap() and munmap()
             to include the shared memory operations (which would only manage
             the reference counting and the life of the allocation).
  
             Then what about pthreads? Memory should not be freed until the last
             pthread in the group exists. That could be done with an additional
             reference count on the whole allocated memory list (just as streams
             and file descriptors are now shared and persist until the last
             pthread exits).
  
             I think that would work but to me is very unattractive and
             inconsistent with the NuttX "small footprint" objective. ...
  
             Other issues:
             - Memory free time would go up because you would have to remove
               the memory from that list in free().
             - There are special cases inside the RTOS itself.  For example,
               if task A creates task B, then initial memory allocations for
               task B are created by task A.  Some special allocators would
               be required to keep this memory on the correct list (or on
               no list at all).
  
             Updated 2016-06-25:
             For processors with an MMU (Memory Management Unit), NuttX can be
             built in a kernel mode.  In that case, each process will have a
             local copy of its heap (filled with sbrk()) and when the process
             exits, its local heap will be destroyed and the underlying page
             memory is recovered.
  
             So in this case, NuttX work just link Linux or or *nix systems:
             All memory allocated by processes or threads in processes will
             be recovered when the process exists.
  
             But not for the flat memory build.  In that case, the issues
             above do apply.  There is no safe way to recover the memory in
             that case (and even if there were, the additional overhead would
             not be acceptable on most platforms).
  
             This does not prohibit anyone from creating a wrapper for malloc()
             and an atexit() callback that frees memory on task exit.  People
             are free and, in fact, encouraged, to do that.  However, since
             it is inherently unsafe, I would never incorporate anything
             like that into NuttX.
  
Status:      Open.  No changes are planned.
Priority:    Medium/Low, a good feature to prevent memory leaks but would
             have negative impact on memory usage and code size.