Today I discovered something funny while working on a Red Hat Enterprise Linux 3 server. The system had an unusual high CPU load, and one way to figure out what is causing that is the use of the command “top”:
[root@dummy root]# top
...
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
3282 someuser 25 0 195M 195M 50856 S 51.9 9.9 220:57 0 java
3300 someuser 15 0 195M 195M 50856 S 2.9 9.9 39:55 0 java
Well here we can see that the process with PID 3282 is using quite some CPU. One way of getting some more information on that PID is to look it up with the “ps” command:
[root@dummy root]# ps -ef |grep 3282
root 9753 1709 0 15:11 pts/0 00:00:00 grep 3282
[root@dummy root]#That is weird, the process is clearly visible when we use top, but doesn’t seem to exist according to ps. I was quite puzzled by this behavior. How can a process be listed in top, but not appear while using ps?
The answer was surprisingly simple, PID 3282 is not actually a process. It’s a thread.
The ps command doesn’t show threads, and neither does top by default on Red Hat Enterprise Linux 4 and up. I tested my Fedora 8 and Fedora 9 installations, and top by default doesn’t show threads.
On Red Hat Enterprise Linux 3 however, it does show threads by default. Toggling the view of threads using Shift-H while running top makes the “process” disappear.
I found it quite funny how a change to a default setting can get people quite a bit confused.
To make “ps” show threads, you could use the -m option. This will allow “ps” to display threads as well.
I was facing the same doubt and your post has been very useful.
Doing some tests with a simple C program where I create two threads with pthread_create I have seen that the threads themselves and the main program see the same PID (the one corresponding to the main program). Interestingly enough top shows 3 different PIDs (one for the main program and 2 for the new threads with consecutive numbers). These PIDs are not seen with ps but they are somehow reserved because no other process has them.
Finally even if ps does not show this PIDs it is possible to send a SIGKILL signal with kill to this “ghost” PIDs and the whole task (main program and the threads) are killed.
Now I am just wondering what are the reasons behind this behavior and have tried to find more information about NPTL but no much info is available (at least I could not find it).