python - 通过 JSON 发送文件而不是上传到服务器，Django

Question

我有一个当前允许用户上传文件并将文件保存在网络服务器上的应用程序。我的客户现在决定使用第三方云托管服务来满足他们的文件存储需求。该公司有自己的 API 用于在他们的服务器上执行 CRUD 操作，所以我编写了一个脚本来测试他们的 API 并将一个文件作为 base64 编码的 JSON 负载发送到 API。该脚本运行良好，但现在我陷入了如何将此功能实现到 Django 中的确切方式。

json_testing.py

import base64
import json
import requests
import magic

filename = 'test.txt'

# Open file and read file and encode it as a base64 string
with open(filename, "rb") as test_file:
    encoded_string = base64.b64encode(test_file.read())

# Get MIME type using magic module
mime = magic.Magic(mime=True)
mime_type = mime.from_file(filename)

# Concatenate MIME type and encoded string with string data
# Use .decode() on byte data for mime_type and encoded string
file_string = 'data:%s;base64,%s' % (mime_type.decode(), encoded_string.decode())
payload = {
    "client_id": 1,
    "file": file_string
}
headers = {
    "token": "AuthTokenGoesHere",
    "content-type": "application/json",
}
request = requests.post('https://api.website.com/api/files/', json=payload, headers=headers)
print(request.json())

模型.py

def upload_location(instance, filename):
    return '%s/documents/%s' % (instance.user.username, filename)

class Document(models.Model):
    user = models.ForeignKey(settings.AUTH_USER_MODEL)
    category = models.ForeignKey(Category, on_delete=models.CASCADE)
    file = models.FileField(upload_to=upload_location)

    def __str__(self):
        return self.filename()

    def filename(self):
        return os.path.basename(self.file.name)

所以重申一下，当用户上传文件时，我不想将文件存储在网络服务器上的某个位置，而是希望对文件进行 base64 编码，以便我可以将文件作为 JSON 负载发送。关于什么是解决此问题的最佳方法的任何想法？

Answer 1

The simplest way I can put this is that I want to avoid saving the file to the web server entirely. I just want to encode the file, send it as a payload, and discard it, if that's possible.

来自django docs :

Upload Handlers

When a user uploads a file, Django passes off the file data to an upload handler – a small class that handles file data as it gets uploaded. Upload handlers are initially defined in the FILE_UPLOAD_HANDLERS setting, which defaults to:

["django.core.files.uploadhandler.MemoryFileUploadHandler", "django.core.files.uploadhandler.TemporaryFileUploadHandler"]

Together MemoryFileUploadHandler and TemporaryFileUploadHandler provide Django’s default file upload behavior of reading small files into memory and large ones onto disk.

You can write custom handlers that customize how Django handles files. You could, for example, use custom handlers to enforce user-level quotas, compress data on the fly, render progress bars, and even send data to another storage location directly without storing it locally. See Writing custom upload handlers for details on how you can customize or completely replace upload behavior.

相反的想法:

我认为您应该考虑坚持使用默认的文件上传处理程序，因为它们可以防止有人上传会占用服务器内存的文件。

Where uploaded data is stored

Before you save uploaded files, the data needs to be stored somewhere.

By default, if an uploaded file is smaller than 2.5 megabytes, Django will hold the entire contents of the upload in memory. This means that saving the file involves only a read from memory and a write to disk and thus is very fast.

However, if an uploaded file is too large, Django will write the uploaded file to a temporary file stored in your system’s temporary directory. On a Unix-like platform this means you can expect Django to generate a file called something like /tmp/tmpzfp6I6.upload. If an upload is large enough, you can watch this file grow in size as Django streams the data onto disk.

These specifics – 2.5 megabytes; /tmp; etc. – are simply “reasonable defaults” which can be customized as described in the next section.

---

request.FILES info :

#forms.py:

from django import forms

class UploadFileForm(forms.Form):
    title = forms.CharField(max_length=50)
    json_file = forms.FileField()

A view handling this form will receive the file data in request.FILES, which is a dictionary containing a key for each FileField (or ImageField, or other FileField subclass) in the form. So the data from the above form would be accessible as request.FILES[‘json_file’].

Note that request.FILES will only contain data if the request method was POST and the <form> that posted the request has the attribute enctype="multipart/form-data". Otherwise, request.FILES will be empty.

---

HttpRequest.FILES

A dictionary-like object containing all uploaded files. Each key in FILES is the name from the <input type="file" name="" />. Each value in FILES is an UploadedFile.

---

Upload Handlers

When a user uploads a file, Django passes off the file data to an upload handler – a small class that handles file data as it gets uploaded. Upload handlers are initially defined in the FILE_UPLOAD_HANDLERS setting, which defaults to:

["django.core.files.uploadhandler.MemoryFileUploadHandler", "django.core.files.uploadhandler.TemporaryFileUploadHandler"]

source code对于 TemporaryFileUploadHandler包含这个:

lass TemporaryFileUploadHandler(FileUploadHandler):
    """
    Upload handler that streams data into a temporary file.
    """
      ...
      ...
      def new_file(self, *args, **kwargs):
        """
        Create the file object to append to as data is coming in.
        """
        ...
        self.file = TemporaryUploadedFile(....)  #<***HERE

还有 source code对于 TemporaryUploadedFile包含这个:

class TemporaryUploadedFile(UploadedFile):
    """
    A file uploaded to a temporary location (i.e. stream-to-disk).
    """
    def __init__(self, name, content_type, size, charset, content_type_extra=None):
        ...
        file = tempfile.NamedTemporaryFile(suffix='.upload')  #<***HERE

还有 python tempfile docs这么说:

tempfile.NamedTemporaryFile(...., delete=True)
...
If delete is true (the default), the file is deleted as soon as it is closed.

类似地，两个默认文件上传处理程序中的另一个，MemoryFileUploadHandler , 创建类型为 BytesIO 的文件:

A stream implementation using an in-memory bytes buffer. It inherits BufferedIOBase. The buffer is discarded when the close() method is called.

因此，您所要做的就是关闭request.FILES[“field_name”]删除文件(无论文件内容存储在内存中还是磁盘上的/tmp 文件目录)，例如:

 uploaded_file = request.FILES[“json_file”]
 file_contents = uploaded_file.read()

 #Send file_contents to other server here.

 uploaded_file.close()  #erases file

如果出于某种原因你不希望 Django 写入服务器的 /tmp目录，那么您需要编写一个自定义文件上传处理程序来拒绝上传过大的文件。