google-cloud-datalab - 谷歌数据实验室 : how to import pickle

Question

是否可以在 Google Datalab 中使用 %%storage 子句从 Google Storage 读取 pickle/joblib 模型？

这个问题与Is text the only content type for %%storage magic function in datalab有关

Answer 1

在原本为空的单元格中运行以下代码:

%%storage read --object <path-to-gcs-bucket>/my_pickle_file.pkl --variable test_pickle_var

然后运行以下代码:

from io import BytesIO    
pickle.load(BytesIO(test_pickle_var))

我使用下面的代码将 Pandas DataFrame 作为腌制文件上传到 Google Cloud Storage 并读取它:

from datalab.context import Context
import datalab.storage as storage
import pandas as pd
from io import BytesIO
import pickle

df = pd.DataFrame(data=[{1,2,3},{4,5,6}],columns=['a','b','c'])

# Create a local pickle file
df.to_pickle('my_pickle_file.pkl')

# Create a bucket in GCS
sample_bucket_name = Context.default().project_id + '-datalab-example'
sample_bucket_path = 'gs://' + sample_bucket_name
sample_bucket = storage.Bucket(sample_bucket_name)
if not sample_bucket.exists():
    sample_bucket.create()

# Write pickle to GCS
sample_item = sample_bucket.item('my_pickle_file.pkl')
with open('my_pickle_file.pkl', 'rb') as f:
    sample_item.write_to(bytearray(f.read()), 'application/octet-stream')

# Read Method 1 - Read pickle from GCS using %storage read (note single % for line magic)
path_to_pickle_in_gcs = sample_bucket_path + '/my_pickle_file.pkl'
%storage read --object $path_to_pickle_in_gcs --variable remote_pickle_1
df_method1 = pickle.load(BytesIO(remote_pickle_1))
print(df_method1)

# Read Alternate Method 2 - Read pickle from GCS using storage.Bucket.item().read_from()
remote_pickle_2 = sample_bucket.item('my_pickle_file.pkl').read_from()
df_method2 = pickle.load(BytesIO(remote_pickle_2))
print(df_method2)

注意:有一个 known issue哪里 %storage如果它是单元格中的第一行，则命令不起作用。将注释或 python 代码放在第一行。