Skip to main content
Welcome guest. | Register | Login | Post

Notification before running out of memory

13 replies [Last post]
libervisco's picture
Offline
Joined: 2006-05-04

It happened twice in a rather short time. First time it was when I had 1GB of real RAM and now I have 1.5GB. The first time culprit was Firefox and the second time I can't be entirely sure as I had both Firefox and Amarok running, but I suspect an AJAX application I've been using.

Running out of RAM and even swap is just disastrous, putting the system to a complete crawl. You can barely kill anything.

Personally I consider this a considerable flaw in a system that advertises itself as user friendly. It should, by default, notify the user before running out of memory. Imagine if this sort of thing happened to someone non-technical! For starts, they wouldn't have a freaking idea about what happened, let alone how to remedy or avoid it in the future.

But no, our beloved OS just goes on like nothing's happening, until it smashes into the wall!

So... I'm looking for a notification program which would notify me when I'm using too much RAM (real or swap or both). If this doesn't exist I'd like to do it manually, with some sort of a bash script perhaps. It should monitor the memory usage and then pop up a message that I'm running out of memory.

Thanks.

tbuitenh's picture
Offline
Joined: 2005-12-21
I think what you actually

I think what you actually want is a way to set a limit on how much memory any program can use, that will interrupt (not kill) a program that goes over the limit and then ask you if you want to kill it or set a higher limit for it and continue.

Part of this would need to be in the kernel (and I bet something not quite unlike it actually is), another part in some daemon, and yet another part would be an extra thread or process belonging to your desktop environment.

The thing is... think of what happens when this daemon and DE extension will be added to the OS: people will complain about two more processes that don't appear to do anything useful Smiling . Also, they will complain about these annoying messages that appear every time they run firefox or some game. I guess as a programmer you just can't win.

The truth is the OS is not the real problem, it has a rough edge in this place but who cares about a rough edge that is only noticed in a situation that shouldn't occur anyway. Applications should not contain memory leaks, and the odd kind of app that might use more memory than a typical ten year old linux box has should take care of what happens when it runs out of memory by itself. After all the only thing an OS can do to stop a process from using more memory is to kill it, and that's usually not what you want.

libervisco's picture
Offline
Joined: 2006-05-04
Well what you describe

Well what you describe might be ideal overall, but I think simply monitoring the amount of memory and the amount of used memory and then notifying the user when the two get too close would be a great improvement. I doubt this would require anything more than an additional small daemon.

That's all I was looking for, not a whole complex system for each application. I think that much really can't be hard to do. If it's going to run into a wall it better tell me so I can stop it myself.

Edit: Just telling me when swap gets close to swamped would really be enough, cause that's what's pending a real disaster. Then it wouldn't show annoying messages in most applications most of the time, just in those rarer cases when something even more annoying than a notification is coming.

tbuitenh's picture
Offline
Joined: 2005-12-21
Maybe one of the many

Maybe one of the many monitoring apps such as gkrellm can do this?

libervisco's picture
Offline
Joined: 2006-05-04
I'm using a system monitor

I'm using a system monitor applet in GNOME, but it doesn't offer such an option. When a memory overload happens I'm usually focused on something else and miss the readings there (well at least this happened the last couple of times).

I'm not sure about gkrellm though. I guess I'll try it, though I don't like its appearance, it's a bit out of place (and time Sticking out tongue) ..

tbuitenh's picture
Offline
Joined: 2005-12-21
That reminds me on ubuntu

That reminds me on ubuntu there is something that warns you when you run out of disk space. I don't know if that's built into gnome or if it's an ubuntu feature. You might send a feature request...

By the way, there is quite a chance that simply monitoring memory (without setting per-process limits) doesn't work. When you're running out, the system is likely to be busy, it might happen that the monitoring process will not be active because another process happens to really need the CPU. I guess that can be solved by using a very liberal (or would that be conservative?) definition of running out of memory: warning when there is still lots left, for example when 3/4 of (ram + swap) has been used.

By the way 2: It's funny this problem happens to you and not to me. You have three times more ram Laughing out loud .

a thing's picture
Offline
Joined: 2005-12-20
Kicker applet

There's a Kicker applet called "Runnaway Process Catcher" that's similar but monitors CPU usage.

libervisco's picture
Offline
Joined: 2006-05-04
Well gkrellm seems to have

Well gkrellm seems to have alerts. Now I just have to figure out what "high alert limit" and "high warn limit" mean and what does a number there actually represent.

tbuitenh wrote:

When you're running out, the system is likely to be busy, it might happen that the monitoring process will not be active because another process happens to really need the CPU.

Not that busy. This usually happens when there is a memory leak which is not necessarily coincided with high CPU usage. Besides, with todays multitasking I can hardly believe a little daemon would somehow be late with its notification.

It's interesting that we're even having this discussion for something as simple as this. IIRC other OSes (not sure if it was windows, might have been an old Mac) had this feature from the beginning.

Anyway 3/4 of RAM sounds decent, or roughly 80 to 90%.

tbuitenh wrote:

By the way 2: It's funny this problem happens to you and not to me. You have three times more ram Laughing out loud

Eh I thought you had more. I guess I forgot. Smiling The two times it happened recently it was apparently due to memory leaks. First time it was probably google maps and second time it was an amarok visualization window that I left running in the background. It's a pity though, those are awesome visualizations.

Apparently when a program is leaking memory, no amount of memory can save you!

tbuitenh's picture
Offline
Joined: 2005-12-21
complexity of real multitasking makes things more difficult
libervisco wrote:
tbuitenh wrote:

When you're running out, the system is likely to be busy, it might happen that the monitoring process will not be active because another process happens to really need the CPU.

Not that busy. This usually happens when there is a memory leak which is not necessarily coincided with high CPU usage. Besides, with todays multitasking I can hardly believe a little daemon would somehow be late with its notification.

Actually todays multitasking is exactly what could make a daemon respond too late. It might have been swapped out or not getting the CPU in time. Linux is not a realtime kernel. To make things worse, there are many process competing for resources.
But I agree that when a process is leaking memory "slowly" (not several M per second) and it's not CPU-heavy, then the notification will appear on time.

---

I remember the out-of-memory error from windows 95. It could be made to report something like "Could not close (some app) because not enough memory is available. Please close some applications."

Offline
Joined: 2007-09-10
I would also like a feature

I would also like a feature like that because I also encountered this problem a few times.

Of course programs shouldn't have memory leaks and take care of lack of memory themselves, but it's not the case for all programs. So instead of letting some program fill up the memory until the system stops responding, it would really be better if it was killed automatically.

If it's possible to know which percentage of memory is used and which program takes up the most memory, it should be possible to kill it and then display a message saying what was killed or log it somewhere.

All I need to do it myself is a way to get this memory info.
I know "ps" could give me the process using the most memory.
But where can I get the used and total memory sizes?

tbuitenh's picture
Offline
Joined: 2005-12-21
It took me a while to look

It took me a while to look this up, and the command to use for it has a man page that says it's something entirely different (limiting written file sizes rather than memory usage), but here's how to have leaky programs get killed when the floor gets too wet Smiling

name this script something like limited-leakyprog, make it executable (chmod 755 limited-leakyprog) and put it in a convenient place such as /usr/local/bin :

#!/bin/bash

ulimit -v max_size_in_kb
leakyprog $@

where max_size_in_kb is a large number without "k" or "kb" or even "_" attached to it.

Note 1: you can limit all kinds of things with ulimit. Run ulimit -a for an overview (including current limits of values).

Note 2: I didn't really test this script (although I played around with ulimit a bit), I expect it should work.

Note 3: Once you set a limit with ulimit, you cannot undo that, you can only make it even more strict. However, once the script ends, you're back in your normal shell without extra limits (if you ran the script from a terminal window or something).

Note 4: The -l and -m options of ulimit are also related to memory, I'm not sure what -l does, but -m does not do what you might expect. It allows a process to use as much memory as it wants, but anything above the -m limit will be swapped out. Of course most of the time this swapping is cached so you won't hear your disk crunch like crazy unless you actually are using all your memory with other programs.

Note 5: You really want to run this from a terminal window a few times before adding it to any menus, so you can see what killed your leaky program, so you can figure out what limit won't kill it all the time but only when it gets out of control.

Note 6: Follow my advice at your own risk.

Offline
Joined: 2007-09-10
My solution: memwatch.sh

ulimit -v max_size_in_kb works very well indeed. Smiling
But it doesn't do exactly what I want.

I have written a script which checks the free RAM every second. If the free RAM (including buffer/cache) goes below a value stored in ~/bin/minmem.txt, it calls the killmaxmemprocess.sh which kills of the most RAM intensive process.
A first attempt is made with killall to terminate the program in a clean way. (usually doesn't work for games like Nexuiz or iteam Sad )
If this doesn't work, kill -9 is used for immediate termination.
After a kill, a message is displayed so the user knows what happened (kill_1="killall" and kill_2="kill -9" with 1 on success).

"kill -9" has the drawback of eventually leaving you in a low resolution (experience a cool RTS desktop effect Smiling ) or a non-working mouse. You can test the effect by launching Nexuiz and doing a kill -9 on it for example.(for the RTS effect, I don't remember which program I used. Needs to leave mouse active after kill.)
I don't know what to do against that yet. (I'll deactivate the script for the tournament anyway since Nexuiz has no memory leak. ^^)

killmaxmemprocess.sh can also be run manually with the following options:
-killmaxmemprocess.sh 0 -> only displays the most RAM intensive process without killing it
-killmaxmemprocess.sh 1 -> displays the most RAM intensive process and kills it

Note:
The script duo writes info about killed programs into a memwatch.log file.

I'm running the memwatch2.sh at startup and already had quite a few kills. The problem is when I install programs with synaptic/apt-get/dpkg. In that case Amarok, Firefox or Tomboy regularly get killed. ^^

Suggestions for improvements or replacements are very welcome. Smiling

Here are the scripts:

killmaxmemprocess.sh

#! /bin/bash
# kill the most memory intensive user process

# Check if all parameters are present
# If no, exit
if [ $# -ne 1 ]
then
        echo
        echo "usage :"
        echo "$0 license_to_kill"
	echo "This shellscript will kill the most memory intensive user process if license_to_kill!=0."
	echo "Otherwise, it just locates it without killing it."
	echo "All missions will be logged in memwatch.log."
        echo
        exit 0
fi

USAGE=`ps -eo %mem,pid,user -o comm= | grep $USER | sort -k1 -n -r | head -1 | awk '{ print $1 } '`
PID=`ps -eo %mem,pid,user -o comm= | grep $USER | sort -k1 -n -r | head -1 | awk '{print $2 }'`
PNAME=`ps -eo %mem,pid,user -o comm= | grep $USER | sort -k1 -n -r | head -1 | awk '{print $4 }'`

echo $(date)
echo "Hostile process detected:"
echo "memory used: $USAGE%"
echo "PID: $PID"
echo "Name: $PNAME"

echo "=========================" >>memwatch.log
echo $(date) >>memwatch.log
echo "Hostile process detected:" >>memwatch.log
echo "memory used: $USAGE%" >>memwatch.log
echo "PID: $PID" >>memwatch.log
echo "Name: $PNAME" >>memwatch.log

if [ $1 -ne 0 ]
then
	echo "killing process..." >>memwatch.log

	kill_1=0
	kill_2=0

	#clean kill
	killall $PNAME
	if [ $? -eq 0 ]
	then
		kill_1=1
	fi

	#messy kill
	kill -9 $PID
	if [ $? -eq 0 ]
	then
		kill_2=1
	fi

	if [ $kill_1 -eq 1 ] || [ $kill_2 -eq 1 ]
	then
		echo "Target successfully eliminated." >>memwatch.log
		echo "kill_1=$kill_1" >>memwatch.log
		echo "kill_2=$kill_2" >>memwatch.log
		zenity --info --text "$PNAME was killed because of excessive RAM usage.\n kill_1=$kill_1 \n kill_2=$kill_2" ||
		kdialog --msgbox "$PNAME was killed because of excessive RAM usage.\n kill_1=$kill_1 \n kill_2=$kill_2" ||
		xmessage -buttons okay -default okay "$PNAME was killed because of excessive RAM usage.\n kill_1=$kill_1 \n kill_2=$kill_2" &
	else
		echo "Mission failed." >>memwatch.log
	fi
	
# 	#messy kill
# 	kill -9 $PID
# 	if [ $? -eq 0 ]
# 	then
# 		echo "Target successfully eliminated." >>memwatch.log
# 		zenity --info --text "$PNAME was killed because of excessive RAM usage." ||
# 		kdialog --msgbox "$PNAME was killed because of excessive RAM usage." ||
# 		xmessage -buttons okay -default okay "$PNAME was killed because of excessive RAM usage."
# 	else
# 		echo "kill -9 failed." >>memwatch.log
# 	fi

fi

memwatch2.sh (first one took min_mem as argument):

#! /bin/bash
# daemon to kill the most memory intensive user process

# Check if all parameters are present
# If no, exit
echo
echo "usage :"
echo "$0"
echo "This shellscript will kill the most memory intensive user process if the free RAM goes beneath the value given in ~/bin/minmem.txt"
echo

#example output of free
#              total       used       free     shared    buffers     cached
# Mem:        450300     420636      29664          0      25172     234092
# -/+ buffers/cache:     161372     288928
# Swap:            0          0          0

while true
do
	sleep 1

# 	Mem:=free | grep 'Mem:' |  awk '{print $1 }'
# 	total=free | grep 'Mem:' |  awk '{print $2 }'
# 	used=free | grep 'Mem:' |  awk '{print $3 }'
	freemem_1=$(free | grep 'Mem:' |  awk '{print $4 }')
# 	shared=free | grep 'Mem:' |  awk '{print $5 }'
# 	buffers=free | grep 'Mem:' |  awk '{print $6 }'
# 	cached=free | grep 'Mem:' |  awk '{print $7 }'
	
# 	-/+=free | grep 'buffers/cache:' |  awk '{print $1 }'
# 	buffers/cache:=free | grep 'buffers/cache:' |  awk '{print $2 }'
# 	used=free | grep 'buffers/cache:' |  awk '{print $3 }'
	freemem_2=$(free | grep 'buffers/cache:' |  awk '{print $4 }')

#	freemem=$(cat /proc/meminfo | head -2 | tail -1 | awk '{print $2 }')
	freemem=$freemem_2;
# 	echo "$freemem_1" >>mem_1.log
# 	echo "$freemem_2" >>mem_2.log

	min_mem=$(cat ~/bin/minmem.txt)

	if test $freemem -lt $min_mem
	then
		echo "=========================" >>memwatch.log
		echo "freemem=$freemem < min_mem" >>memwatch.log
		~/bin/killmaxmemprocess.sh 1
	fi
done

My current minmem.txt file:

Quote:
119149

Here's a fun trivial program to test it with:
RAMfiller.cpp:

//This program will fill up the RAM.
//USE WITH EXTREME CAUTION!!!

int main(void)
{
        while(1)
        {
                new char;
        }
}

Killing efficiency: about 50-70%
When the RAM fills up to fast, my script doesn't have time to react.
I just read man sleep and apparently I can use floating point numbers too. I think this will significantly improve my scripts reaction time. Laughing out loud

Interestingly, the RAMfiller program also gets stopped automatically by GNU/Linux after a while.
Why doesn't it do that with other leaky programs? Sad

I still have to test its effect on Windows. Sticking out tongue

tbuitenh's picture
Offline
Joined: 2005-12-21
You can fix the screen

You can fix the screen resolution with xrandr.

I think checking how much memory is available all the time can slow your computer down a lot if you do it too much. And no matter how often you check, it is not guaranteed to work.

Killing the process that just happens to be using the most memory could be the wrong thing to do. Consider a script like this (not entirely sure of it because I'm half asleep):

$1 &
exec $1

This fills the memory (and processor) with lots of small processes by starting itself twice. Do you want peaceful bigger processes to be killed in situations like this? Eventually the memory monitor would kill itself because it is the biggest process left.

The right action to be taken by a memory monitoring process depends on the particular misbehavior that is causing the use of too much memory.

How about killing the fastest growing process?

Offline
Joined: 2007-09-10
>> Apparently when a

>> Apparently when a program is leaking memory, no amount of memory can save you!

It may not be technically "leaking memory," but some programs seem to have a difficult time releasing their grip on memory.

I want a firefox method to free optional memory or suggest that a certain amount of memory be freed. For example, we should be able to say, release the memory with these 8 tabs I just closed. We should be able to say, release memory associated with these 8 tabs I have open (meaning that bringing any of these tabs to the foreground would require reconstructing the pages from the html that is saved). The html should be saved in the latter case perhaps to follow the http protocol accurately (in some cases anyway), and form information filled in and other things should also be saved; however, the pixels and data structures supporting that need not be saved as they can be rebuilt. We should have the option, and most apps should provide these cabilities (even if it means sending a message through DBUS or sending a specific signal by cli to a nongui app).

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.