From: Simon Derr Date: Fri, 5 Aug 2005 02:52:03 +0000 (-0700) Subject: [PATCH] __vm_enough_memory() signedness fix X-Git-Tag: v2.6.13-rc6~36 X-Git-Url: https://err.no/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=2f60f8d3573ff90fe5d75a6d11fd2add1248e7d6;p=linux-2.6 [PATCH] __vm_enough_memory() signedness fix We have found what seems to be a small bug in __vm_enough_memory() when sysctl_overcommit_memory is set to OVERCOMMIT_NEVER. When this bug occurs the systems fails to boot, with /sbin/init whining about fork() returning ENOMEM. We hunted down the problem to this: The deferred update mecanism used in vm_acct_memory(), on a SMP system, allows the vm_committed_space counter to have a negative value. This should not be a problem since this counter is known to be inaccurate. But in __vm_enough_memory() this counter is compared to the `allowed' variable, which is an unsigned long. This comparison is broken since it will consider the negative values of vm_committed_space to be huge positive values, resulting in a memory allocation failure. Signed-off-by: Signed-off-by: Signed-off-by: Andrew Morton Signed-off-by: Linus Torvalds --- diff --git a/mm/mmap.c b/mm/mmap.c index da3fa90a0a..404319477e 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -143,7 +143,11 @@ int __vm_enough_memory(long pages, int cap_sys_admin) leave 3% of the size of this process for other processes */ allowed -= current->mm->total_vm / 32; - if (atomic_read(&vm_committed_space) < allowed) + /* + * cast `allowed' as a signed long because vm_committed_space + * sometimes has a negative value + */ + if (atomic_read(&vm_committed_space) < (long)allowed) return 0; vm_unacct_memory(pages); diff --git a/mm/nommu.c b/mm/nommu.c index ce74452c02..fd4e8df0f0 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -1167,7 +1167,11 @@ int __vm_enough_memory(long pages, int cap_sys_admin) leave 3% of the size of this process for other processes */ allowed -= current->mm->total_vm / 32; - if (atomic_read(&vm_committed_space) < allowed) + /* + * cast `allowed' as a signed long because vm_committed_space + * sometimes has a negative value + */ + if (atomic_read(&vm_committed_space) < (long)allowed) return 0; vm_unacct_memory(pages);