How to: Abort DOM/XML loading when memory is low - 未名空间MITBBS历史存档

国际科技财经博客移民网络热点娱乐民生时事公众号

Redian新闻

>未名空间

>Java - 爪哇娇娃

How to: Abort DOM/XML loading when memory is low

How to: Abort DOM/XML loading when memory is low# Java - 爪哇娇娃

r*s2013-07-08 07:07

1 楼

It is using DOM because of POI.
We are not changing the parsers -- too much work.
Assume we have no control of the file size, because the java
vm may be already low in memory, so a small one may blow it.
If it is getting "out of memory error", the whole app is gone.
So the question is whether we could do anything to prevent it
when the memory is low.
We would rather abort the xml loading than halt the whole app.
Any ideas?

o*i2013-07-08 07:07

2 楼

my two cents:
create a thread to try loading the xml file (in try statement and maybe usin
g some "Hedge" techniques)
another thread monitoring the free memory for Runtime, when it lower than a
threshold value and loading still in processing, kill it and clean-up/releas
e the memory.

【在 r*****s 的大作中提到】

: It is using DOM because of POI.
: We are not changing the parsers -- too much work.
: Assume we have no control of the file size, because the java
: vm may be already low in memory, so a small one may blow it.
: If it is getting "out of memory error", the whole app is gone.
: So the question is whether we could do anything to prevent it
: when the memory is low.
: We would rather abort the xml loading than halt the whole app.
: Any ideas?

g*g2013-07-08 07:07

3 楼

You should specify more memory on -Xmx.
You should have a limit for size of the file being processed, and a total
limit of the size of all files being processed concurrently.
You can also call Runtime.freeMemory() before you process the file. But it
may not be reliable.

【在 r*****s 的大作中提到】

F*n2013-07-08 07:07

4 楼

One idea to use SoftReference for your Document.

【在 r*****s 的大作中提到】

F*n2013-07-08 07:07

5 楼

More specifically, you can write a wrapper class for Document that uses a
SoftReference to contains an instance of Document implementation in the XML
API.
public class MyDocument implements Document {
private SoftReference docRef;
// do this for all method implementation
@Override
public Attr createAttribute(String name) {
Document doc = docRef.get();
if (doc == null) abort();
return doc.createAttribute(name);
}
}

【在 F****n 的大作中提到】

: One idea to use SoftReference for your Document.

g*e2013-07-08 07:07

6 楼

soft ref也不能解决load某个文件过大导致OOM错吧

XML

【在 F****n 的大作中提到】

: More specifically, you can write a wrapper class for Document that uses a
: SoftReference to contains an instance of Document implementation in the XML
: API.
: public class MyDocument implements Document {
: private SoftReference docRef;
: // do this for all method implementation
: @Override
: public Attr createAttribute(String name) {
: Document doc = docRef.get();
: if (doc == null) abort();

F*n2013-07-08 07:07

7 楼

It is guaranteed that soft refs will be gc-ed before an OOM exception is
thrown. If the document is gc-ed, the memory is released and the soft ref
will contain null.

【在 g**e 的大作中提到】

: soft ref也不能解决load某个文件过大导致OOM错吧
:
: XML

g*g2013-07-08 07:07

8 楼

while the document is being processed, it's unlikely your soft ref is the
only ref. So it won't be GCed. And if it's not being processed, it should
have no reference in the system. soft ref is for cache, not for this.

【在 F****n 的大作中提到】

: It is guaranteed that soft refs will be gc-ed before an OOM exception is
: thrown. If the document is gc-ed, the memory is released and the soft ref
: will contain null.

F*n2013-07-08 07:07

9 楼

The real document is wrapped inside the outer class.
Only the outer class will be hard referenced during parsing.

~~~~~ Now you learn it can do more.

【在 g*****g 的大作中提到】

: while the document is being processed, it's unlikely your soft ref is the
: only ref. So it won't be GCed. And if it's not being processed, it should
: have no reference in the system. soft ref is for cache, not for this.

g*e2013-07-08 07:07

10 楼

这个我知道，但是这并不能100%保证在OOM之前full GC就kick off了

【在 F****n 的大作中提到】

: It is guaranteed that soft refs will be gc-ed before an OOM exception is
: thrown. If the document is gc-ed, the memory is released and the soft ref
: will contain null.

r*s2013-07-08 07:07

11 楼

主要是这XML Loading是POI handle的，
要是JVM能在OOM之前就Throw a FailToAllocateMemoryError就好了

【在 g**e 的大作中提到】

: 这个我知道，但是这并不能100%保证在OOM之前full GC就kick off了

w*z2013-07-08 07:07

12 楼

You can try to catch java.lang.OutOfMemoryError, but it might be too late at
that time and JVM is dying.
http://stackoverflow.com/questions/2679330/catching-java-lang-o
Before you load xml, can you check the size of the xml file?

【在 r*****s 的大作中提到】

r*s2013-07-08 07:07

13 楼

We did, but
1. 这个File size跟DOM也不是正比的
2. When the jvm free heap is already running low,
even loading a small xml file may blow it up.

at

【在 w**z 的大作中提到】

: You can try to catch java.lang.OutOfMemoryError, but it might be too late at
: that time and JVM is dying.
: http://stackoverflow.com/questions/2679330/catching-java-lang-o
: Before you load xml, can you check the size of the xml file?

w*z2013-07-08 07:07

14 楼

You need to scale up your system to solve the problem. Add more RAM to the
machine, enlarge heap. Or add more machines to your service.

【在 r*****s 的大作中提到】

: We did, but
: 1. 这个File size跟DOM也不是正比的
: 2. When the jvm free heap is already running low,
: even loading a small xml file may blow it up.
:
: at