python - 如何在文本分割器(langchain)之后将代码分配给文件？

Question

我正在使用 Langchain 的 RecursiveCharacterTextSplitter 来分割 python 文件。这样做我会丢失哪个 block 属于哪个文件的信息。之后如何跟踪各个 block 并将其分配给文件名？

def index_repo(repo_url):

    os.environ['OPENAI_API_KEY'] = ""

    contents = []
    fileextensions = [
        ".py", ]


    print('cloning repo')
    repo_dir = get_repo(repo_url)

    print(repo_dir)

    for dirpath, dirnames, filenames in os.walk(repo_dir):
        for file in filenames:
            if file.endswith(tuple(fileextensions)):
                try:
                    with open(os.path.join(dirpath, file), "r", encoding="utf-8") as f:
                        contents.append(f.read())

                except Exception as e:
                    pass


    # chunk the files
    text_splitter =  RecursiveCharacterTextSplitter.from_language(language=Language.PYTHON, chunk_size=5000, chunk_overlap=0)
    texts = text_splitter.create_documents(contents)

    return texts

Answer 1

create_documents(texts: List[str], metadatas: Optional[List[dict]] = None) → List[Document]

在元数据中添加文件信息并将其传递给create_documents。