python - 如何在上传过程中不将整个文件加载到内存中

Question

正在使用Bottle创建上传 API。下面的脚本能够将文件上传到目录，但有两个问题需要解决。一是如何避免将整个文件加载到内存中，另一个是如何设置上传文件的最大大小？

是否可以连续读取文件并将已读取的内容转储到文件直到上传完成？ upload.save(file_path, overwrite=False, chunk_size=1024) 函数似乎将整个文件加载到内存中。在教程中，他们指出使用 .read() is dangerous .

from bottle import Bottle, request, run, response, route, default_app, static_file
app = Bottle()

@route('/upload', method='POST')
def upload_file():
    function_name = sys._getframe().f_code.co_name
    try:
        upload = request.files.get("upload_file")
        if not upload:
            return "Nothing to upload"
        else:
            #Get file_name and the extension
            file_name, ext = os.path.splitext(upload.filename)
            if ext in ('.exe', '.msi', '.py'):
                return "File extension not allowed."

            #Determine folder to save the upload
            save_folder = "/tmp/{folder}".format(folder='external_files')
            if not os.path.exists(save_folder):
                os.makedirs(save_folder)

            #Determine file_path    
            file_path = "{path}/{time_now}_{file}".\
                        format(path=save_folder, file=upload.filename, timestamp=time_now)

            #Save the upload to file in chunks            
            upload.save(file_path, overwrite=False, chunk_size=1024)
            return "File successfully saved {0}{1} to '{2}'.".format(file_name, ext, save_folder)

    except KeyboardInterrupt:
        logger.info('%s: ' %(function_name), "Someone pressed CNRL + C")
    except:
        logger.error('%s: ' %(function_name), exc_info=True)
        print("Exception occurred111. Location: %s" %(function_name))
    finally:
        pass

if __name__ == '__main__':
    run(host="localhost", port=8080, reloader=True, debug=True)
else:
    application = default_app()

我也尝试过执行 file.write 但情况相同。文件正在读取到内存中，并且挂起机器。

file_to_write = open("%s" %(output_file_path), "wb") 
while True:
    datachunk = upload.file.read(1024)
    if not datachunk:
        break
    file_to_write.write(datachunk)

与此相关，我看过属性MEMFILE_MAX其中有几个SO posts声称可以设置最大文件上传大小。我尝试过设置它，但似乎没有任何效果，因为所有文件(无论大小)都会经过。

请注意，我希望能够接收 Office 文档，该文档可以是带有扩展名的普通文档，也可以是带有密码的压缩包。

使用Python3.4和bottle 0.12.7

Answer 1

基本上，您想在循环中调用upload.read(1024)。像这样的东西(未经测试):

with open(file_path, 'wb') as dest:
    chunk = upload.read(1024)
    while chunk:
        dest.write(chunk)
        chunk = upload.read(1024)

(请勿在 upload 上调用 open；它已为您打开。)

This SO answer包含更多示例，说明如何读取大文件而不“吞食”它。