Recovering arguments and environment variables from a core dump

The problem

Suppose, not hypothetically, that you have a core dump and need to recover the process’s arguments and environment.

Your first stop is the ELF notes. There is NT_PRPSINFO, and it does carry pr_psargs, but Linux fixes that buffer at 80 bytes in ELF_PRARGSZ. Eighty bytes is not much. Worse, pr_psargs flattens the arguments into a space-delimited string. There is no reliable indication of where one argument ends and the next begins, so an argument that originally contained a space is indistinguishable from two separate arguments.

The environment is worse still: it is not in the ELF notes at all. NT_PRPSINFO does not have it. NT_PRSTATUS does not have it. Nothing does.

If you are using systemd-coredump(8), the journal may include COREDUMP_CMDLINE and COREDUMP_ENVIRON, faithful copies of /proc/[pid]/cmdline and /proc/[pid]/environ captured at crash time. That helps, but only if the journal entry has not been rotated. Core files may outlive the journal, and they might have been moved to another machine. And if you are dealing with a plain kernel core dump, or an Apport crash report with a flat ProcCmdline field instead of the original NUL-delimited argv layout, you are back to lossy metadata.

What the kernel puts on the stack

When the kernel loads an ELF binary, create_elf_tables builds the initial process stack. On the usual downward-growing Linux process stack, the layout looks like this, from high addresses to low addresses:

[string data: argv strings, environ strings, AT_EXECFN string]
AT_PLATFORM string (NUL-terminated, if present)
AT_BASE_PLATFORM string (NUL-terminated, if present)
AT_RANDOM (16 random bytes)
[alignment padding]
auxv[N]  = { AT_NULL, 0 }
auxv[N-1]
...
auxv[0]
NULL               <-- environ terminator
envp[M-1]
...
envp[0]
NULL               <-- argv terminator
argv[argc-1]
...
argv[0]
argc

This is the layout we rely on. The actual NUL-terminated strings for every argument and environment variable sit at the top of the stack, alongside the AT_EXECFN string and, if present, the platform strings referenced by AT_PLATFORM and AT_BASE_PLATFORM. Below them sit the 16 random bytes referenced by AT_RANDOM, then the auxiliary vector, then the pointer arrays envp[] and argv[], each terminated by a NULL, with argc at the very bottom.

So in an ordinary dump, the data is still there. The hard part is finding it. There is no note that says “arguments start here,” so we need an anchor. And because Linux can omit pages for a variety of reasons, we also need to be careful about trusting what we find.

The anchor: AT_RANDOM

The auxiliary vector is the key. Core dumps do preserve it, as an NT_AUXV ELF note. It is an array of type-value pairs: AT_PAGESZ for the page size, AT_ENTRY for the entry point, AT_PHDR for the program headers, and so on. Linux also exposes the same structure for live processes as /proc/[pid]/auxv.

One of those entries is AT_RANDOM, a pointer to 16 bytes of random data that the kernel placed on the stack for the C library’s stack protector. Its value is an address into the stack: specifically, to a spot just above the auxiliary vector itself and just below any optional platform strings.

So AT_RANDOM gives us exactly what we need: an address inside the dumped stack memory, only a few bytes away from the auxiliary vector. That is the anchor.

Scanning down to find the auxiliary vector

From the AT_RANDOM address, we first round down to a word boundary, then scan downward through the stack memory looking for the AT_NULL terminator: two consecutive zero words that mark the end of the auxiliary vector.

But two zero words can appear elsewhere in the data above the auxiliary vector: inside the random bytes, in the optional platform strings, or in padding. A false positive here would poison everything downstream. So we validate each candidate (0, 0) pair: the entry immediately before it must match the last auxiliary-vector entry we already read from the note. We already know the full contents of NT_AUXV; at this stage, we are only locating that array within the dumped stack.

Once we have the address of AT_NULL, we can compute the base of the auxiliary vector:

As a second check, we verify that the two words at auxv_base match the first auxiliary-vector entry. If both checks pass, we know exactly where the auxiliary vector sits in memory. We also cap the scan window so a bad anchor cannot send us wandering arbitrarily through the dump.

Walking backward to arguments and environment variables

The stack layout from here is rigid. One word below auxv_base is the NULL terminator for the envp[] array. We verify that it is zero, then walk backward, collecting pointer-sized words until we hit another zero. That second zero is the NULL terminator of argv[].

At this point we have the envp[] pointer array: every environment variable’s string address, in order.

We then continue walking backward from the argv NULL terminator, collecting pointers. How do we know where argv ends? Right below argv[0] sits argc, and argc must equal the number of argv entries. So we stop when we encounter a word whose value matches the number of pointers collected so far. That is the sentinel. As with the auxiliary vector scan, we put a hard ceiling on how many entries we will accept before declaring the data invalid.

At this point we have two arrays of pointers: one for arguments, one for environment variables. Each pointer is an address into the string data region at the top of the stack.

Resolving the strings

The string pointers all land within a contiguous region near the top of the stack. Rather than reading them one by one, we sort all the pointers, read a bounded chunk covering that whole span, and then, for each pointer, compute its offset into the buffer and scan forward to the NUL terminator.

We have now recovered the original arguments and environment variables intact, from raw stack memory that no structured metadata bothered to keep.

Trying it out

This technique is implemented in ptools, a collection of Linux utilities for inspecting processes and core dumps, inspired by the tools of the same name on Solaris/illumos. The pargs(1) and penv(1) commands use this stack walk to recover arguments and environment variables from core dumps where the usual commands cannot. The source code for the initial stack recovery is a fairly direct implementation of the algorithm described above.