什么应用需要几百个节点的java呢 - 未名空间MITBBS历史存档

国际科技财经博客移民网络热点娱乐民生时事公众号

Redian新闻

>未名空间

>Java - 爪哇娇娃

什么应用需要几百个节点的java呢

什么应用需要几百个节点的java呢# Java - 爪哇娇娃

i*a2011-11-23 08:11

1 楼

熄灯以后，男生寝室讨论的话题想来大家都明瞭啦。偏偏我们初中时的班主任非要探个
究竟。有一天大家正在兴头时，一个同学打开门去厕所，发现以体育老师出身的班主
任正用标准的引体向上姿势挂在门框上往里看。大家共同惊叫。次日即用报纸把玻璃门
框贴住了。
还有一次，相同的情况，打开门，男生寝室的生活老师正听得美滋滋地。因为猛地一开
门，他顺势栽到了寝室中央。尴尬之余，用无比严厉地声音喝斥到“性，性，一天到
晚就知道谈性。干脆把重庆xx学校改成重庆性学校好了！”

S*h2011-11-23 08:11

2 楼

下周有个面试，（java develop）要求里面有一条
Work with Financial Engineering to design and develop distributed
application (runs using ~500 processors in Linux cluster).
Distributed Computing experience a plus. Experience with DataSynapse a plus.
分布计算没怎么接触过。我见过的分布一般都是科学计算，不知道，大型的java分布计
算可以做什么呢，一般的网站之类，只要几个服务器就可以了，什么应用需要几百个节
点的java呢？

z*32011-11-23 08:11

3 楼

500个processors小意思啦
需要这么大处理量的地方多了
一般随便大一点的金融机构都可以满足这个要求，几个服务器哪里够用
再比如新浪那种流量，几个服务器哪里够用，再比如某些交通运输领域，机票那种
几个服务器哪里够用，地铁那种，才几个服务器哪里够用，电信的我就不算了
以前我管的一个系统，还只是某金融集团的一个小环节
就有500个人每天八小时不间断地刷，用了4个主weblogic外加4个做备份
总共部属在八台服务器上，processor忘记有多少个了

z*32011-11-23 08:11

4 楼

java最强大的就是分布式操作系统了
无所不在，稍微大一点的系统，就有java的存在
不过这玩意估计会扯到集群，说到这个，没经验的够呛

S*h2011-11-23 08:11

5 楼

谢谢回复。没什么经验，见笑了。只是不明白，就是javaEE 一类的服务吗？是不是那
种需要多服务器平衡流量？我只见过在cluster上搞科学计算，一般都是C,C++什么的。
java的分布，只用过简单的RMI。看来还是老老实实的说没见过吧。希望不要是个必须
条件。

y*d2011-11-23 08:11

6 楼

"刷" 是什么样的一个动作？

【在 z*******3 的大作中提到】

: 500个processors小意思啦
: 需要这么大处理量的地方多了
: 一般随便大一点的金融机构都可以满足这个要求，几个服务器哪里够用
: 再比如新浪那种流量，几个服务器哪里够用，再比如某些交通运输领域，机票那种
: 几个服务器哪里够用，地铁那种，才几个服务器哪里够用，电信的我就不算了
: 以前我管的一个系统，还只是某金融集团的一个小环节
: 就有500个人每天八小时不间断地刷，用了4个主weblogic外加4个做备份
: 总共部属在八台服务器上，processor忘记有多少个了

z*e2011-11-23 08:11

7 楼

不不不，你已经做得很好了，这行你在学校里面是没有办法积累经验的
一定要走到社会上去，到工厂公司中去，那才是你大展拳脚的地方
学校里面搞的很少会涉及这些东西，也就是software engineering有说到
但是cs研究方向这么多，software engineering往往是被人为忽视的一个环节
大多数人在cs的核心算法等领域捣腾，固然科学研究需要这些东西
但是工作中，真正需要用到数学的部分其实相当少，不会太过于深入多少
你刚毕业，一般不会让你去负责集群这么大的责任
一般还是定位在senior developer上，面试时候尽管说
三个四个也是集群，又不是只有500个以上才叫集群，管它那么多
工作中肯定有人罩着你，这行阿三特别多
要比他们能吹才行，你肯定不会比阿三更差的

【在 S****h 的大作中提到】

: 谢谢回复。没什么经验，见笑了。只是不明白，就是javaEE 一类的服务吗？是不是那
: 种需要多服务器平衡流量？我只见过在cluster上搞科学计算，一般都是C,C++什么的。
: java的分布，只用过简单的RMI。看来还是老老实实的说没见过吧。希望不要是个必须
: 条件。

S*h2011-11-23 08:11

8 楼

Well. Back from the interview. They told me that they have the calculation
in C++, wrapped with JNI. Each node runs one instance (JNI+C++) and the
parallel is managed by java.

r*s2011-11-23 08:11

9 楼

is one node a machine? or you mean a process or even a thread?
if per machine, you've got to have job scheduler to deploy the
jobs, hudreds of which, to each node, thru job schedulers such as
condor, globus etc. (back to grid computing era.)

calculation

【在 S****h 的大作中提到】

: Well. Back from the interview. They told me that they have the calculation
: in C++, wrapped with JNI. Each node runs one instance (JNI+C++) and the
: parallel is managed by java.

S*h2011-11-23 08:11

10 楼

I think one node means a processor as well as a thread. They need to
simulate a large number of instances relatively independently. Each
instance is not too complicated though (single thread C++ code, wrapped in
JNI). I am not sure what technology they used for grid. But anyway, I
will not need to worry about it unless they extend me an offer. :)

g*g2011-11-23 08:11

11 楼

Typicall one node means one physical box. Or one virtual box
if virtual software is used.

【在 S****h 的大作中提到】

: I think one node means a processor as well as a thread. They need to
: simulate a large number of instances relatively independently. Each
: instance is not too complicated though (single thread C++ code, wrapped in
: JNI). I am not sure what technology they used for grid. But anyway, I
: will not need to worry about it unless they extend me an offer. :)

r*s2011-11-23 08:11

12 楼

是啊，
per thread/process的一般不叫node
用jni+C++也很奇怪
除非legacy code很难重写

【在 g*****g 的大作中提到】

: Typicall one node means one physical box. Or one virtual box
: if virtual software is used.

z*e2011-11-23 08:11

13 楼

"single thread"

【在 r*****s 的大作中提到】

: 是啊，
: per thread/process的一般不叫node
: 用jni+C++也很奇怪
: 除非legacy code很难重写

S*h2011-11-23 08:11

14 楼

Yeah. They did not go into details about the system. I think what they
mean is that in each node, you only need a single thread c++ program. Since
everybody seems quite interesting it it, I will say everything I know. It
is a big Monte Carlo. Simulation 12M cases, with 300 senario for each case.
1 senario takes a node a time. A single thread C++ program runs in a node.
That is pretty much all I know. I would report back more if they hire me
and show me more. :P

z*e2011-11-23 08:11

15 楼

actually i think this is some kind of legacy like other people said
probably it will cost them a lot to reconstruct the whole program in java
thus they just use the single thread program to run the business
i asked a question in this board about how to deal with r language
which is also a single thread program...
the weird thing is they use 500 nodes to do the simulation?
what a waste

Since
.
.
me

【在 S****h 的大作中提到】

: Yeah. They did not go into details about the system. I think what they
: mean is that in each node, you only need a single thread c++ program. Since
: everybody seems quite interesting it it, I will say everything I know. It
: is a big Monte Carlo. Simulation 12M cases, with 300 senario for each case.
: 1 senario takes a node a time. A single thread C++ program runs in a node.
: That is pretty much all I know. I would report back more if they hire me
: and show me more. :P

S*h2011-11-23 08:11

16 楼

I understand for the underlying simulation, C++ is probably faster than Java
, at least historically. Java is probably more flexible in terms of manage
the parallel and plumbing.
What do you mean "waste" for the simulation?

g*g2011-11-23 08:11

17 楼

You don't know how heavy the computing is. Even single thread
can take the entire CPU. Plus the node can be a virtual one.

【在 z****e 的大作中提到】

: actually i think this is some kind of legacy like other people said
: probably it will cost them a lot to reconstruct the whole program in java
: thus they just use the single thread program to run the business
: i asked a question in this board about how to deal with r language
: which is also a single thread program...
: the weird thing is they use 500 nodes to do the simulation?
: what a waste
:
: Since
: .

z*e2011-11-23 08:11

18 楼

well i can guess
since he said the simulation is about Monte Carlo
i happen to know what this method about
thus i think it is a little bit waste

【在 g*****g 的大作中提到】

: You don't know how heavy the computing is. Even single thread
: can take the entire CPU. Plus the node can be a virtual one.

S*h2011-11-23 08:11

19 楼

Do you mean Monte Carlo is not CPU intensive? I am not sure how complicated
one path is. But it is probably fairly complicated. It seems fairly natural
to me that to perform one path in one thread in one node and parallel
different paths in different node.
C++ codes for real computation are developed by another subgroup. The
subgroup I am interviewed focuses on the java management code.

【在 z****e 的大作中提到】

: well i can guess
: since he said the simulation is about Monte Carlo
: i happen to know what this method about
: thus i think it is a little bit waste

z*32011-11-23 08:11

20 楼

因为用java的math lib比较烦，不直观，但是有比c++更好的选择，比如各种数学和统
计软件
回头再说

complicated
natural

【在 S****h 的大作中提到】

:
: Do you mean Monte Carlo is not CPU intensive? I am not sure how complicated
: one path is. But it is probably fairly complicated. It seems fairly natural
: to me that to perform one path in one thread in one node and parallel
: different paths in different node.
: C++ codes for real computation are developed by another subgroup. The
: subgroup I am interviewed focuses on the java management code.

S*h2011-11-23 08:11

21 楼

呵呵，这样的话，你这是要断了那个 C＋＋的全组的饭碗啊。 :P

【在 z*******3 的大作中提到】

: 因为用java的math lib比较烦，不直观，但是有比c++更好的选择，比如各种数学和统
: 计软件
: 回头再说
:
: complicated
: natural

r*l2011-11-23 08:11

22 楼

If the C++ code can effectively utilize the CPU, no additional multi-thread
thing you need to do in Java. Otherwise, either C++ team needs to create
more efficient code or they will lose their jobs.
In your scenario, I guess the CPUs are already fully utilized.

【在 S****h 的大作中提到】

: 呵呵，这样的话，你这是要断了那个 C＋＋的全组的饭碗啊。 :P