longgb246的博客

发表于 | 分类于 | 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
CREATE TABLE dev.tmp_check_sku_20161202
AS
SELECT
    a.item_sku_id,
    b.sku_id
FROM
    (SELECT item_sku_id,dt
    FROM 
    gdm.gdm_m03_item_sku_da
    WHERE dt= '2016-10-08'
    AND sku_valid_flag=1) A
    FULL OUTER JOIN
    (SELECT sku_id,dt
    FROM gdm.gdm_m08_item_stock_day_sum
    WHERE dt= '2016-10-08'
    ) B
    ON  a.item_sku_id=b.sku_id
阅读全文 »
longgb246的博客

发表于 | 分类于 |

1、设置动态分区

1
2
set hive.exec.dynamic.partition=true;
set hive.exec.dynamic.partition.mode=nonstrict;

2、设置最大动态分区

1
2
set hive.exec.max.dynamic.partitions=2000;
set hive.exec.max.dynamic.partitions.pernode=2000;
阅读全文 »
longgb246的博客

发表于 | 分类于 |

使用pickle模块将数据对象保存到文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
import pickle
 
data1 = {'a': [1, 2.0, 3, 4+6j],
         'b': ('string', u'Unicode string'),
         'c': None}
 
selfref_list = [1, 2, 3]
selfref_list.append(selfref_list)
 
output = open('data.pkl', 'wb')
 
# Pickle dictionary using protocol 0.
pickle.dump(data1, output)
 
# Pickle the list using the highest protocol available.
pickle.dump(selfref_list, output, -1)
 
output.close()

阅读全文 »