only average statistics - 未名空间MITBBS历史存档

国际科技财经博客移民网络热点娱乐民生时事公众号

Redian新闻

>未名空间

>DataSciences - 数据科学

only average statistics

only average statistics# DataSciences - 数据科学

j*g2014-05-20 07:05

1 楼

This is not my task. This is a task of my colleague and I don't like the way
he did it. I am interested in how to analysis this. We have very detailed
data and we want to find some correlation with another source of data. On
the other source, they only have average statistics, for example, average
price of a product by each district. We have the detailed price for each
store, but no district information. How can I find the correlation? Can I
fit the distribution with weighted average and then compare the distribution
?
Thanks

j*g2014-05-20 07:05

2 楼

另外，我们的data一定是bias的,而他们的不是

c*z2014-05-20 07:05

3 楼

maybe the only thing you can do is to aggregate to district level and
compare with them...

D*e2014-05-20 07:05

4 楼

Hierarchical modeling

d*n2014-05-20 07:05

5 楼

那你恐怕很难做东西。
如果两个分布已经叠加了，把他们分开也是很难的。现在即使告诉你分布而不仅仅是平
均数，让你对一系列数据分类，这也是一个不简单的过程。

【在 j*******g 的大作中提到】

: 另外，我们的data一定是bias的,而他们的不是

B*62014-05-20 07:05

6 楼

way
lz的意思是说不知道每个store对应的district？你指的数据的bias是说没有收集到所
有store的数据？
distribution

【在 j*******g 的大作中提到】

: This is not my task. This is a task of my colleague and I don't like the way
: he did it. I am interested in how to analysis this. We have very detailed
: data and we want to find some correlation with another source of data. On
: the other source, they only have average statistics, for example, average
: price of a product by each district. We have the detailed price for each
: store, but no district information. How can I find the correlation? Can I
: fit the distribution with weighted average and then compare the distribution
: ?
: Thanks

j*g2014-05-20 07:05

7 楼

I think this is interesting topic, but not for this particular scenario :)

【在 D***e 的大作中提到】

: Hierarchical modeling

j*g2014-05-20 07:05

8 楼

Our data is like 1/4 of the other data source and sample size is so large
that if data is not bias, we should be able to prove they come from the same
distribution, but I cannot prove that, plus given the way we collect our
data, I kind of believe our data is bias.

【在 B*******6 的大作中提到】

:
: way
: lz的意思是说不知道每个store对应的district？你指的数据的bias是说没有收集到所
: 有store的数据？
: distribution

B*62014-05-20 07:05

9 楼

算samples基于已知mean的分布likelihood?

same

【在 j*******g 的大作中提到】

: Our data is like 1/4 of the other data source and sample size is so large
: that if data is not bias, we should be able to prove they come from the same
: distribution, but I cannot prove that, plus given the way we collect our
: data, I kind of believe our data is bias.

h*72014-05-20 07:05

10 楼

用supervised clustering
train之后把mean做成initialization，clustering loop算一步得出membership即可

way
distribution

【在 j*******g 的大作中提到】

j*g2014-05-20 07:05

11 楼

this is close. but i don't think the information is enough to do
classification, since we only look at the price.

【在 h*****7 的大作中提到】

: 用supervised clustering
: train之后把mean做成initialization，clustering loop算一步得出membership即可
:
: way
: distribution

j*g2014-05-20 07:05

12 楼

the result of these will be very dense on the mean... i don't think it will
work

【在 B*******6 的大作中提到】

: 算samples基于已知mean的分布likelihood?
:
: same

h*72014-05-20 07:05

13 楼

not classification, it is called supervised clustering, quite different from
classification

【在 j*******g 的大作中提到】

: this is close. but i don't think the information is enough to do
: classification, since we only look at the price.

j*g2014-05-20 07:05

14 楼

I did some homework... Are you suggesting using the average statistics as '
teacher' to do clustering over our dataset? Thank you.

from

【在 h*****7 的大作中提到】

: not classification, it is called supervised clustering, quite different from
: classification

j*g2014-05-20 07:05

15 楼

I am thinking this is not going to work, if you use price difference as the
distance, the cluster will again becomes very dense on the price, it will
not cluster to the district

【在 h*****7 的大作中提到】

: 用supervised clustering
: train之后把mean做成initialization，clustering loop算一步得出membership即可
:
: way
: distribution