python - 如何找到在 Graphlab SFrame 中保存时引发错误的特定行？

Question

我有一个 SFrame，其外观与 sf.print_rows(10) 类似:

+--------------+---------------+-------+-------------------------------+
|   Dataset    |     Domain    | Score |             Sent1             |
+--------------+---------------+-------+-------------------------------+
| STS2012-gold | surprise.OnWN |  5.0  | render one language in ano... |
| STS2012-gold | surprise.OnWN |  3.25 | nations unified by shared ... |
| STS2012-gold | surprise.OnWN |  3.25 | convert into absorbable su... |
| STS2012-gold | surprise.OnWN |  4.0  | devote or adapt exclusivel... |
| STS2012-gold | surprise.OnWN |  3.25 | elevated wooden porch of a... |
| STS2012-gold | surprise.OnWN |  4.0  | either half of an archery bow |
| STS2012-gold | surprise.OnWN | 3.333 | a removable device that is... |
| STS2012-gold | surprise.OnWN |  4.75 |      restrict or confine      |
| STS2012-gold | surprise.OnWN |  0.5  |     orient, be positioned     |
| STS2012-gold | surprise.OnWN |  4.75 | Bring back to life, return... |
+--------------+---------------+-------+-------------------------------+
+-------------------------------+-------------------------------+
|             Sent2             |        Sent1_tokenized        |
+-------------------------------+-------------------------------+
| restate (words) from one l... | [render, one, language, in... |
| a group of nations having ... | [nations, unified, by, sha... |
| soften or disintegrate by ... | [convert, into, absorbable... |
| devote oneself to a specia... | [devote, or, adapt, exclus... |
| a porch that resembles the... | [elevated, wooden, porch, ... |
| either of the two halves o... | [either, half, of, an, arc... |
| a supplementary part or ac... | [a, removable, device, tha... |
| place limits on (extent or... |    [restrict, or, confine]    |
|          be opposite.         |   [orient,, be, positioned]   |
|  cause to become alive again. | [Bring, back, to, life,, r... |
+-------------------------------+-------------------------------+
+-------------------------------+-----------+-----------+----------------------+
|        Sent2_tokenized        | Sent1_len | Sent2_len | NGRAM-cosChar2ngrams |
+-------------------------------+-----------+-----------+----------------------+
| [restate, (words), from, o... |     6     |     8     |      0.82090085      |
| [a, group, of, nations, ha... |     8     |     7     |      0.53250804      |
| [soften, or, disintegrate,... |     11    |     11    |      0.43274232      |
| [devote, oneself, to, a, s... |     10    |     8     |      0.47759567      |
| [a, porch, that, resembles... |     6     |     9     |      0.38885689      |
| [either, of, the, two, hal... |     6     |     12    |      0.55555556      |
| [a, supplementary, part, o... |     10    |     5     |      0.44963552      |
| [place, limits, on, (exten... |     3     |     6     |      0.27124449      |
|        [be, opposite.]        |     3     |     2     |      0.43528575      |
| [cause, to, become, alive,... |     8     |     5     |      0.37047929      |
+-------------------------------+-----------+-----------+----------------------+
+----------------------+----------------------+----------------------+
| NGRAM-cosChar3ngrams | NGRAM-cosChar4ngrams | NGRAM-cosChar5ngrams |
+----------------------+----------------------+----------------------+
|      0.74964917      |      0.71490469      |      0.67925959      |
|      0.36701702      |      0.28941438      |      0.23635427      |
|      0.25899951      |      0.21053227      |      0.17058877      |
|      0.26248718      |      0.20518234      |      0.14285714      |
|      0.17107978      |      0.12049505      |      0.09320546      |
|      0.40754381      |      0.24715577      |      0.11547005      |
|      0.21997067      |      0.17554945      |      0.15450786      |
|      0.13284223      |      0.09284767      |       0.048795       |
|      0.31426968      |      0.17149859      |      0.09449112      |
|      0.0632772       |      0.03402069      |         0.0          |
+----------------------+----------------------+----------------------+
+---------------------+---------------------+---------------------+---------------------+

[19097 rows x 134 columns]

但是当我尝试使用 sf.save('trainers.csv', format='csv') 将其保存到 csv 中时，它会抛出错误:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-23-f82bcb3fa197> in <module>()
----> 1 sts.save('trainers.csv', format='csv')

/usr/local/lib/python2.7/dist-packages/graphlab/data_structures/sframe.pyc in save(self, filename, format)
   2924                 self.export_json(url)
   2925             else:
-> 2926                 raise ValueError("Unsupported format: {}".format(format))
   2927 
   2928     def export_csv(self, filename, delimiter=',', line_terminator='\n',

/usr/local/lib/python2.7/dist-packages/graphlab/cython/context.pyc in __exit__(self, exc_type, exc_value, traceback)
     47             if not self.show_cython_trace:
     48                 # To hide cython trace, we re-raise from here
---> 49                 raise exc_type(exc_value)
     50             else:
     51                 # To show the full trace, we do nothing and let exception propagate

RuntimeError: Runtime Exception. Traceback (most recent call last):
  File "<ipython-input-5-e29b4d4eba06>", line 20, in <lambda>
ZeroDivisionError: division by zero

我打印了n号。一次一行，例如sf.print_rows(10)、sf.print_rows(100) 和 sf.print_rows(129) 处抛出错误:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-24-13550768dbcd> in <module>()
----> 1 sts.print_rows(129)

/usr/local/lib/python2.7/dist-packages/graphlab/data_structures/sframe.pyc in print_rows(self, num_rows, num_columns, max_column_width, max_row_width, output_file)
   2226         max_row_width = max(max_row_width, max_column_width + 1)
   2227 
-> 2228         printed_sf = self._imagecols_to_stringcols(num_rows)
   2229         row_of_tables = printed_sf.__get_pretty_tables__(wrap_text=False,
   2230                                                          max_rows_to_display=num_rows,

/usr/local/lib/python2.7/dist-packages/graphlab/data_structures/sframe.pyc in _imagecols_to_stringcols(self, num_rows)
   2250                 if t in image_column_names:
   2251                     printed_sf[t] = self[t].astype(str)
-> 2252         return printed_sf.head(num_rows)
   2253 
   2254     def __str_impl__(self, num_rows=10, footer=True):

/usr/local/lib/python2.7/dist-packages/graphlab/data_structures/sframe.pyc in head(self, n)
   2454         tail, print_rows
   2455         """
-> 2456         return SFrame(_proxy=self.__proxy__.head(n))
   2457 
   2458     def to_dataframe(self):

graphlab/cython/cy_sframe.pyx in graphlab.cython.cy_sframe.UnitySFrameProxy.head()

graphlab/cython/cy_sframe.pyx in graphlab.cython.cy_sframe.UnitySFrameProxy.head()

RuntimeError: Runtime Exception. Traceback (most recent call last):
  File "<ipython-input-5-e29b4d4eba06>", line 20, in <lambda>
ZeroDivisionError: division by zero

所以我做了一个sf.fillna(c, 0):

for c in sts.column_names():
    sts = sts.fillna(c, 0)

它抛出另一个错误:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-26-e63cf73308dd> in <module>()
      1 for c in sts.column_names():
----> 2     sts = sts.fillna(c, 0)

/usr/local/lib/python2.7/dist-packages/graphlab/data_structures/sframe.pyc in fillna(self, column, value)
   5652             raise TypeError("Must give column name as a str")
   5653         ret = self[self.column_names()]
-> 5654         ret[column] = ret[column].fillna(value)
   5655         return ret
   5656 

/usr/local/lib/python2.7/dist-packages/graphlab/data_structures/sarray.pyc in fillna(self, value)
   2439 
   2440         with cython_context():
-> 2441             return SArray(_proxy = self.__proxy__.fill_missing_values(value))
   2442 
   2443     def topk_index(self, topk=10, reverse=False):

/usr/local/lib/python2.7/dist-packages/graphlab/cython/context.pyc in __exit__(self, exc_type, exc_value, traceback)
     47             if not self.show_cython_trace:
     48                 # To hide cython trace, we re-raise from here
---> 49                 raise exc_type(exc_value)
     50             else:
     51                 # To show the full trace, we do nothing and let exception propagate

RuntimeError: Runtime Exception. Default value must be convertible to column type

如何查找在 Graphlab SFrame 中保存时引发错误的特定行？

如何修复这一行？我可以用 fillna() 替换行中有问题的列吗？我无法真正使用 dropna() 丢弃这些行，因为我需要跟踪有问题的行。

但即使使用 dropna()，我最终得到的是:

sf.dropna()
sf.save('trainers.csv', format='csv')

如何找到这些给我错误或 ZeroDivisionErrors 的行？以及如何纠正它们或用零填充这些列？

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-28-f82bcb3fa197> in <module>()
----> 1 sts.save('trainers.csv', format='csv')

/usr/local/lib/python2.7/dist-packages/graphlab/data_structures/sframe.pyc in save(self, filename, format)
   2924                 self.export_json(url)
   2925             else:
-> 2926                 raise ValueError("Unsupported format: {}".format(format))
   2927 
   2928     def export_csv(self, filename, delimiter=',', line_terminator='\n',

/usr/local/lib/python2.7/dist-packages/graphlab/cython/context.pyc in __exit__(self, exc_type, exc_value, traceback)
     47             if not self.show_cython_trace:
     48                 # To hide cython trace, we re-raise from here
---> 49                 raise exc_type(exc_value)
     50             else:
     51                 # To show the full trace, we do nothing and let exception propagate

RuntimeError: Runtime Exception. Traceback (most recent call last):
  File "<ipython-input-5-e29b4d4eba06>", line 20, in <lambda>
ZeroDivisionError: division by zero

奇怪的是，当我尝试使用以下命令迭代 SFrame 时，我无法迭代 SFrame:

for i in sf:
    print i

它抛出此错误:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-29-d2d0035d7bbe> in <module>()
----> 1 for i in sts:
      2     print i

/usr/local/lib/python2.7/dist-packages/graphlab/data_structures/sframe.pyc in generator()
   3712         def generator():
   3713             elems_at_a_time = 262144
-> 3714             self.__proxy__.begin_iterator()
   3715             ret = self.__proxy__.iterator_get_next(elems_at_a_time)
   3716             column_names = self.column_names()

graphlab/cython/cy_sframe.pyx in graphlab.cython.cy_sframe.UnitySFrameProxy.begin_iterator()

graphlab/cython/cy_sframe.pyx in graphlab.cython.cy_sframe.UnitySFrameProxy.begin_iterator()

RuntimeError: Runtime Exception. Traceback (most recent call last):
  File "<ipython-input-5-e29b4d4eba06>", line 10, in <lambda>
TypeError: 'NoneType' object is not iterable

事情变得更奇怪了，我无法使用 sf[num] 检索特定行，但我可以执行子 SFrame，然后检索特定的 num 行。所以这个:

print sf[25]

中断和抛出:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-62-6bc8898704c0> in <module>()
----> 1 print sts[25]

/usr/local/lib/python2.7/dist-packages/graphlab/data_structures/sframe.pyc in __getitem__(self, key)
   3595             ub = min(sf_len, lb + block_size)
   3596 
-> 3597             val_list = list(SFrame(_proxy = self.__proxy__.copy_range(lb, 1, ub)))
   3598             self._cache["getitem_cache"] = (lb, ub, val_list)
   3599             return val_list[key - lb]

graphlab/cython/cy_sframe.pyx in graphlab.cython.cy_sframe.UnitySFrameProxy.copy_range()

graphlab/cython/cy_sframe.pyx in graphlab.cython.cy_sframe.UnitySFrameProxy.copy_range()

RuntimeError: Runtime Exception. Traceback (most recent call last):
  File "<ipython-input-5-e29b4d4eba06>", line 10, in <lambda>
TypeError: 'NoneType' object is not iterable

但是当我尝试提取子集然后打印时，它起作用了。下面的代码检索之前使用上面的代码抛出错误的第 25 个元素:

x =  sf[:30]
print x[25]

前面带有 sf[25] 的代码抛出 NoneType 是否有原因？ sf[0] 到 sf[24] 有效，但任何高于 25 的值都无效。

显然，以这种方式迭代 SFrame 并将其转储为 str sorta 有效:

fout = open('superbad.txt', 'w')
sflen = len(sf)
i = 0
while i < sflen:
    m = i+100 if i+100 < sflen else sflen
    x = sf[i:m]
    for j in x:
        fout.write(str(j) +'\n\n')

这很奇怪。 为什么分块迭代并转储到字符串有效？

Answer 1

问题是运行应用时出现除零错误(在保存上方的某个位置)

RuntimeError: Runtime Exception. Traceback (most recent call last):
File "<ipython-input-5-e29b4d4eba06>", line 20, in <lambda>
ZeroDivisionError: division by zero

发生这种情况是因为惰性求值 ( https://en.wikipedia.org/wiki/Lazy_evaluation )。作为示例，假设我从具有单列的 SFrame 开始

sf = gl.SFrame({'x': range(10000, -1, -1)})
sf['x'].apply(lambda x: 1.0/x)

此时，SFrame 的最后一行包含 1.0/0 值，这是一个错误，但尚未对其进行评估。 save 方法会触发具体化，即数据中所有行的实际计算，然后导致错误发生。您可以通过调用 __materialize__

来触发此过程

sf.__materialize__()

这会导致发生以下错误。

RuntimeError: Runtime Exception. Traceback (most recent call last):
File "<ipython-input-55-5af90e232e2d>", line 1, in <lambda>
ZeroDivisionError: float division by zero

惰性评估和查询规划作为性能优化非常重要，也是 SFrame 快速且可扩展的原因之一。不幸的是，跟踪错误是它的烦恼之一，但是一旦您了解它的工作原理，您就会习惯它。

head() 函数不会触发完整的具体化，因此您可以在任意多的行上执行它，直到发现错误为止。