哪位在Linux上用C＋＋碰到过memory fragmentation吗？ - 未名空间MITBBS历史存档

国际科技财经博客移民网络热点娱乐民生时事公众号

Redian新闻

>未名空间

>Programming - 葵花宝典

哪位在Linux上用C＋＋碰到过memory fragmentation吗？

哪位在Linux上用C＋＋碰到过memory fragmentation吗？# Programming - 葵花宝典

l*u2008-11-29 08:11

1 楼

碰到一个问题，一个C++写的server，run了一天后，用top看占了1.5G，在它想再分配
一块500M的heap时，就out of memory了。
同样的程序，在另一个server上run了一天后，却只占300M，也没有out of memory的问
题。
本人怀疑是memory fragmentation。因为如果是memory leak的话，不管在哪个server
上都会有memory leak。可能两个server的kernel不一样(虽然都是RHEL3)，导致只在一
个server有memory fragmentation？
哪位有这方面经验的能分享一下吗？

g*g2008-11-29 08:11

2 楼

profile it, no better way.

server

【在 l******u 的大作中提到】

: 碰到一个问题，一个C++写的server，run了一天后，用top看占了1.5G，在它想再分配
: 一块500M的heap时，就out of memory了。
: 同样的程序，在另一个server上run了一天后，却只占300M，也没有out of memory的问
: 题。
: 本人怀疑是memory fragmentation。因为如果是memory leak的话，不管在哪个server
: 上都会有memory leak。可能两个server的kernel不一样(虽然都是RHEL3)，导致只在一
: 个server有memory fragmentation？
: 哪位有这方面经验的能分享一下吗？

t*t2008-11-29 08:11

3 楼

用valgrind跑跑看, 就是慢了点

server

【在 l******u 的大作中提到】

c*m2008-11-29 08:11

4 楼

u hv mem leak. mem frag != 1.5G footprint, but == unable to alloc large
chunk.
alloc 500m then other svr u should at least take up >300m. why alloc such
huge size? maybe u should pre-alloc huge mem and self manage, and use
another heap for small allocs and then no frags.

c*m2008-11-29 08:11

5 楼

r u sure its mem usage is 1.5G, not virt size? if 1.5G is virt size then
yes, u more likely hv mem frag.

t*t2008-11-29 08:11

6 楼

actually, glibc default behaviour is to allocate large chunk separately by
mmap.

【在 c**m 的大作中提到】

: u hv mem leak. mem frag != 1.5G footprint, but == unable to alloc large
: chunk.
: alloc 500m then other svr u should at least take up >300m. why alloc such
: huge size? maybe u should pre-alloc huge mem and self manage, and use
: another heap for small allocs and then no frags.

c*m2008-11-29 08:11

7 楼

but if he's frag'ing, then system's i14n may not meet his specific alloc
pattern.

l*u2008-11-29 08:11

8 楼

1.5G is what the columns "SIZE" and "RSS" of the "top" command show.
I need to allocate buffers for incoming socket messages, some of them could
be huge.
The reason I suspect it is memory fragmentation is that the "out of memory"
error:
1) does not always happen on server A, only once every 5-6 days.
2) does not happen on server B at all, even though the cod base and input
are identical.
3) only happens for the largest 1-2 messages. Even after it happened,
small messages still work.
When

【在 c**m 的大作中提到】

: r u sure its mem usage is 1.5G, not virt size? if 1.5G is virt size then
: yes, u more likely hv mem frag.

c*m2008-11-29 08:11

9 楼

I am guessing that when you say "socket messages", you mean your own
application message because buffers for sockets are measured in Ks, not Ms.
So you received huge data through socket and stored them in one chunk of
memory that could be 500M. Without knowing the nature of your application,
my guess is you should process your data and if possible store them to
secondary storage. If your footprint grows to significantly larger than
anticipated my money is on memory leak. I remember that I use

【在 l******u 的大作中提到】

: 1.5G is what the columns "SIZE" and "RSS" of the "top" command show.
: I need to allocate buffers for incoming socket messages, some of them could
: be huge.
: The reason I suspect it is memory fragmentation is that the "out of memory"
: error:
: 1) does not always happen on server A, only once every 5-6 days.
: 2) does not happen on server B at all, even though the cod base and input
: are identical.
: 3) only happens for the largest 1-2 messages. Even after it happened,
: small messages still work.

l*u2008-11-29 08:11

10 楼

The app is to send requests (over the course of 4-5 hours) to a server and
get results back. Some of the results could get big -- I didn't measure
carefully, perhaps not as big as 500M, more likely to be around 100M.
The app isn't particular complex. I've already fixed some major memory
leaks in the app which crashed it every time (the app is > 2 yr old while I'
m 7 months into my tenure here, request/result sizes were getting much
bigger recently) and in testing it runs fine (memory taken at

【在 c**m 的大作中提到】

: I am guessing that when you say "socket messages", you mean your own
: application message because buffers for sockets are measured in Ks, not Ms.
: So you received huge data through socket and stored them in one chunk of
: memory that could be 500M. Without knowing the nature of your application,
: my guess is you should process your data and if possible store them to
: secondary storage. If your footprint grows to significantly larger than
: anticipated my money is on memory leak. I remember that I use

t*t2008-11-29 08:11

11 楼

I'
it
another?
most probably you have memory leak at some place, but it won't be triggered
everytime ---- which still is a memory leak.
problem

【在 l******u 的大作中提到】

: The app is to send requests (over the course of 4-5 hours) to a server and
: get results back. Some of the results could get big -- I didn't measure
: carefully, perhaps not as big as 500M, more likely to be around 100M.
: The app isn't particular complex. I've already fixed some major memory
: leaks in the app which crashed it every time (the app is > 2 yr old while I'
: m 7 months into my tenure here, request/result sizes were getting much
: bigger recently) and in testing it runs fine (memory taken at