B*n
2 楼
难道是因为所有计算都是in memory的?看了databrick 的demo,每个cluster的内存都
是上千G的。
但内存大的话计算显然快呀,这idea不是很简单么?
新手,求科普,谢谢
是上千G的。
但内存大的话计算显然快呀,这idea不是很简单么?
新手,求科普,谢谢
z*g
8 楼
RDD can provide fault tolerance for in-memory intermediate result by only
storing very small amount of data on persistent storage. This is
particularly useful for iterative algorithms, since there is intermediate
result involved. Although in case there is not enough memory, Spark performs
exactly like Hadoop.
【在 s********k 的大作中提到】
: 一直没搞懂这个RDD,到底牛在什么地方
storing very small amount of data on persistent storage. This is
particularly useful for iterative algorithms, since there is intermediate
result involved. Although in case there is not enough memory, Spark performs
exactly like Hadoop.
【在 s********k 的大作中提到】
: 一直没搞懂这个RDD,到底牛在什么地方
z*g
9 楼
There is nothing new about in-memory. The key point is that RDD can achieve
fault tolerance for intermediate computation results without having to
writing the whole data back to disk.
fault tolerance for intermediate computation results without having to
writing the whole data back to disk.
相关阅读
同学们, 写书去吧?抛砖引玉讨论一下Reddit的长盛不衰一个system design题:content feed如果今天重新开始搞一个论坛,还有需求吗?zabbix, nagios, icinga, sensu这些监控软件有Windows版吗?Eclipse 出installer了Weighted Graph Challenge 一道面试题iphone 4巨慢, 但storage还有1.5G这个YOUTUBER 被微软 LAYOFF 以后全职 YOUTUBE人工智能下围棋超过人类, 是一个虚假结论, 纯属误导!Reddit 是不错,但是据说不盈利,快关了?请教一个设计问题。天啊噜 wordpress 的用户管理很弱 有没有插件啊?被docker气死了大龄转行请教今天都有谁去了那个AI frontier conferenceAI 对股市有何影响?Quote of computation我工作中遇到技术难题了,大家给我讲讲 (转载)当机了?