xmalloc crash

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

xmalloc crash

Alfred Baroti
Hi,
I found this from long long time ago.
Is this a serious bug?


[root@n1x ~]#su nix
nix@n1x:/root$ printf "%s\n"
{{a..z},{A..Z},{0..9}}{{a..z},{A..Z},{0..9}}{{a..z},{A..Z},{0..9}}{{a..z},{A..Z},{0..9}}{{a..z},{A..Z},{0..9}}
bash: xmalloc: .././braces.c:793: cannot allocate 7329062664 bytes
(614682624 bytes allocated)
[root@n1x ~]#

GNU bash, version 4.4.12(1)-release (x86_64-pc-linux-gnu)

Distributor ID: Ubuntu
Description:    Ubuntu 17.10
Release:        17.10
Codename:       artful

Thanks,
Alfred
Reply | Threaded
Open this post in threaded view
|

Re: xmalloc crash

Eduardo A. Bustamante López
On Sat, Jan 06, 2018 at 01:42:25AM +0200, Alfred Baroti wrote:
> Hi,
> I found this from long long time ago.
> Is this a serious bug?
[...]

This is not a serious bug at all. It's just a memory allocation failure.

> [root@n1x ~]#su nix
> nix@n1x:/root$ printf "%s\n"
> {{a..z},{A..Z},{0..9}}{{a..z},{A..Z},{0..9}}{{a..z},{A..Z},{0..9}}{{a..z},{A..Z},{0..9}}{{a..z},{A..Z},{0..9}}
> bash: xmalloc: .././braces.c:793: cannot allocate 7329062664 bytes
> (614682624 bytes allocated)

You are performing a brace expansion which generates 62^5 combinations and
requires bash to allocate 7329062664 bytes (6.83 GiB) from the operating
system. The allocation fails at 614682624 bytes (586 MiB).

Once bash is unable to allocate memory, it will crash (xmalloc is a wrapper
around malloc which terminates execution on allocation errors).

Looking at a program execution trace for bash 4.4.12(1)-release (as distributed
by Ubuntu):

    dualbus@ubuntu:~$ LC_ALL=C strace -fo /dev/stdout bash -c ': {{a..z},{A..Z},{0..9}}{{a..z},{A..Z},{0..9}}{{a..z},{A..Z},{0..9}}{{a..z},{A..Z},{0..9}}{{a..z},{A..Z},{0..9}}' | cat -n | tail -n6
    bash: xmalloc: .././braces.c:793: cannot allocate 7329062664 bytes (613113856 bytes allocated)
    116508 28672 brk(0x254ad000)                   = 0x254ad000
    116509 28672 brk(0x254ae000)                   = 0x254ae000
    116510 28672 brk(0x254af000)                   = 0x254af000
    116511 28672 write(2, "bash: xmalloc: .././braces.c:793"..., 95) = 95
    116512 28672 exit_group(2)                     = ?
    116513 28672 +++ exited with 2 +++

You can see that the brk() system call actually succeeds.

So the allocation fails, even if the system has enough memory to handle it.
Upon closer inspection:

    dualbus@ubuntu:~/src/gnu/bash/lib/malloc$ cat -n malloc.c | sed -n '777,781p'
       777  /* Silently reject too-large requests. */
       778  if (nunits >= NBUCKETS) {
       779      abort();
       780    return ((PTR_T) NULL);
       781  }

The internal_malloc function that bash uses on some systems (instead of the
libc provided malloc), including Linux, has a special provision that rejects
memory requests that are too large (NBUCKETS = 30).

So you have two options:

(1) Compile bash by providing `--without-bash-malloc' to the `configure' script
(2) Find another option to generate all these permutations that doesn't require
requesting >580 MiB at once.


Question for Chet: Is there a reason we use bash-malloc in Linux systems by
default, instead of the typical glibc malloc which is available on most of
them?

Reply | Threaded
Open this post in threaded view
|

Re: xmalloc crash

Bob Proulx
Eduardo A. Bustamante López wrote:
> Alfred Baroti wrote:
> > Hi,
> > I found this from long long time ago.
> > Is this a serious bug?
> [...]
>
> This is not a serious bug at all. It's just a memory allocation failure.

Agreed.  Not a bug at all.

> You can see that the brk() system call actually succeeds.

If you are running under the Linux kernel in the default configuration
then memory overcommit is enabled.

  $ sysctl vm.overcommit_memory

With overcommit enabled brk(), and malloc(), will always succeed even
if there isn't really enough memory.  Neither will fork().  It doesn't
mean there actually is that much memory available.  Later if the
kernel runs out of memory to implement the needs then it will fail and
trigger the OOM Out Of Memory Killer to virtually kill -9 processes
until enough memory is freed up to balance the books.  Processes
killed by the OOM have no opportunity to clean up or log anything.

Or vm.overcommit_memory can be configured the traditional Unix way
where enough virtual memory needs to exist or malloc() and fork() will
fail.  IMNHO overcommit off is the best way for enterprise servers.
However overcommit on is convenient for laptops.

Bob

Reply | Threaded
Open this post in threaded view
|

Re: xmalloc crash

Eduardo A. Bustamante López
On Sat, Jan 06, 2018 at 09:13:17PM -0700, Bob Proulx wrote:

> > You can see that the brk() system call actually succeeds.
>
> If you are running under the Linux kernel in the default configuration
> then memory overcommit is enabled.
>
>   $ sysctl vm.overcommit_memory
>
> With overcommit enabled brk(), and malloc(), will always succeed even
> if there isn't really enough memory.  Neither will fork().  It doesn't
> mean there actually is that much memory available.  Later if the
> kernel runs out of memory to implement the needs then it will fail and
> trigger the OOM Out Of Memory Killer to virtually kill -9 processes
> until enough memory is freed up to balance the books.  Processes
> killed by the OOM have no opportunity to clean up or log anything.

This is not related to Linux VM overcommit / OOM killer.


From what I can see, Bash's internal memory allocator (lib/malloc/malloc.c,
internal_malloc) is actually unable to allocate the ~580 MiB of VM, even if
there's enough free physical memory to back it up.

See:
  http://git.savannah.gnu.org/cgit/bash.git/tree/lib/malloc/malloc.c?h=devel&id=ce0469bfbe4176802d4ffd1f272010e14bc4ed81#n777

An easy way to compare is to compile bash with:

 ./configure --with-bash-malloc && make

 vs

 ./configure --without-bash-malloc && make

And then run the reported brace expansion on both.


In my case, the Bash internal allocator fails around 580 MiB. Unlike Bash
compiled to use the system's malloc (in my case, glibc malloc), which allocated
a few GiBs of memory before I had to kill the process (system started to swap).


So there's indeed a limitation in Bash's internal allocator which is enabled by
the configure script in some systems. I don't know if there's a reason for
having that artificial limitation in place though.

Reply | Threaded
Open this post in threaded view
|

Re: xmalloc crash

Chet Ramey
In reply to this post by Alfred Baroti
On 1/5/18 6:42 PM, Alfred Baroti wrote:

> Hi,
> I found this from long long time ago.
> Is this a serious bug?
>
>
> [root@n1x ~]#su nix
> nix@n1x:/root$ printf "%s\n"
> {{a..z},{A..Z},{0..9}}{{a..z},{A..Z},{0..9}}{{a..z},{A..Z},{0..9}}{{a..z},{A..Z},{0..9}}{{a..z},{A..Z},{0..9}}
> bash: xmalloc: .././braces.c:793: cannot allocate 7329062664 bytes
> (614682624 bytes allocated)
> [root@n1x ~]#

You're using the bash malloc, and you've exceeded the maximum permitted
allocation size (2**32 - 1).

The other possible cause is that you've exceeded the resource limit for
the size of the data segment, but that doesn't appear to be the case.

Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
                 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    [hidden email]    http://tiswww.cwru.edu/~chet/

Reply | Threaded
Open this post in threaded view
|

Re: xmalloc crash

Chet Ramey
In reply to this post by Eduardo A. Bustamante López
On 1/6/18 2:40 PM, Eduardo A. Bustamante López wrote:

> So the allocation fails, even if the system has enough memory to handle it.
> Upon closer inspection:
>
>     dualbus@ubuntu:~/src/gnu/bash/lib/malloc$ cat -n malloc.c | sed -n '777,781p'
>        777  /* Silently reject too-large requests. */
>        778  if (nunits >= NBUCKETS) {
>        779      abort();
>        780    return ((PTR_T) NULL);
>        781  }

I'm not sure which version of bash you're looking at, since no version with
that check has ever had abort() there.

> The internal_malloc function that bash uses on some systems (instead of the
> libc provided malloc), including Linux, has a special provision that rejects
> memory requests that are too large (NBUCKETS = 30).

Because otherwise it will not run on 32-bit systems. The maximum allocation
is capped at 2**32 - 1. With the current implementation, you can run the
same binary on 32 and 64-bit systems.

When I converted the bash malloc to run on 64-bit systems (bash-2.05b) --
explicitly checking the request sizes instead of relying on 32-bit integer
overflow -- I left the maximum request size at 2*32-1 (4294967295 bytes).

You could add enough buckets to handle request sizes up to 2**63-1,
conditional on whether or not the compilation environment was for 32 or 64
bits, change NBUCKETS, add the right entries to binsizes[], and probably
have something that will handle requests larger than the current maxmimum.
I haven't done it.

> So you have two options:
>
> (1) Compile bash by providing `--without-bash-malloc' to the `configure' script
> (2) Find another option to generate all these permutations that doesn't require
> requesting >580 MiB at once.

That's the wrong number.

> Question for Chet: Is there a reason we use bash-malloc in Linux systems by
> default, instead of the typical glibc malloc which is available on most of
> them?

It is 1) faster than the glibc malloc, at least when I last checked; 2)
better tuned for bash's allocation pattern; and 3) provides better
debugging info.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
                 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    [hidden email]    http://tiswww.cwru.edu/~chet/

Reply | Threaded
Open this post in threaded view
|

Re: xmalloc crash

Chet Ramey
In reply to this post by Eduardo A. Bustamante López
On 1/7/18 2:27 AM, Eduardo A. Bustamante López wrote:

>
> This is not related to Linux VM overcommit / OOM killer.
>
>
> From what I can see, Bash's internal memory allocator (lib/malloc/malloc.c,
> internal_malloc) is actually unable to allocate the ~580 MiB of VM, even if
> there's enough free physical memory to back it up.

That's the wrong number. The request is for 7329062664 bytes, which is
between 2**32 and 2**33 (around 7 gig) and so exceeds the maximum request
size:

bash: xmalloc: .././braces.c:793: cannot allocate 7329062664 bytes
(614682624 bytes allocated)

The 614682624 is the amount of memory the bash malloc has already obtained
from the kernel (the difference between the break at the first call to
malloc and the call when malloc failed).

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
                 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    [hidden email]    http://tiswww.cwru.edu/~chet/

Reply | Threaded
Open this post in threaded view
|

Re: xmalloc crash

Eduardo A. Bustamante López
In reply to this post by Chet Ramey
On Mon, Jan 8, 2018 at 8:22 AM, Chet Ramey <[hidden email]> wrote:

> On 1/6/18 2:40 PM, Eduardo A. Bustamante López wrote:
>
>> So the allocation fails, even if the system has enough memory to handle it.
>> Upon closer inspection:
>>
>>     dualbus@ubuntu:~/src/gnu/bash/lib/malloc$ cat -n malloc.c | sed -n '777,781p'
>>        777      /* Silently reject too-large requests. */
>>        778      if (nunits >= NBUCKETS) {
>>        779          abort();
>>        780        return ((PTR_T) NULL);
>>        781      }
>
> I'm not sure which version of bash you're looking at, since no version with
> that check has ever had abort() there.

Derp. I added that abort() for debugging purposes and forgot to remove
it. Sorry about that.

Thank you for the explanation by the way!