python - 遍历 .tar.gz 目录并连接文件(不解压缩文件夹)

Question

我有一个包含大约 20000 个 tar.gz 目录的文件夹，每个目录包含一堆文件。我想进入源文件夹，遍历 tar.gz 目录(不解压)并连接文件，这样最后我将得到三个大文件。

例如我有一个根文件夹 pnoc 其中有 .tar.gz 目录，每个压缩文件夹有三个文件夹 - Kallisto, RSEM 和 Hugo。我已经解压了一个这样的目录，看起来像这样:

pnoc/
├── C021_0001_20140916_tumor_RNASeq.tar.gz
├── C021_0002_001113_tumor_RNASeq.tar.gz
├── C021_0003_001409_tumor_RNASeq.tar.gz
├── C021_0004_001418_tumor_RNASeq.tar.gz
├── C021_0005_001661_tumor_RNASeq.tar.gz
├── C021_0007_001669_tumor_RNASeq.tar.gz
├── C021_0008_001699_tumor_RNASeq.tar.gz
├── C021_0009_001766_tumor_RNASeq.tar.gz
├── C021_0010_001774_tumor_RNASeq.tar.gz
├── C021_0011_001786_tumor_RNASeq.tar.gz
├── C021_0012_001825_tumor_RNASeq.tar.gz
├── C021_0013_001872_tumor_RNASeq.tar.gz
├── CPBT_0001_1_tumor_RNASeq.tar.gz
├── CPBT_0003_1_tumor_RNASeq.tar.gz
├── CPBT_0004_1_tumor_RNASeq.tar.gz
├── CPBT_0005_1_tumor_RNASeq.tar.gz
├── CPBT_0006_1_tumor_RNASeq.tar.gz
├── CPBT_0007_1_tumor_RNASeq.tar.gz
├── CPBT_0008_1_tumor_RNASeq.tar.gz
├── CPBT_0009_1_tumor_RNASeq.tar.gz
├── IMPROPERLY_PAIRED.C021_0006_001666_tumor_RNASeq.tar.gz
└── pnoc-manifest


C021_0001_20140916_tumor_RNASeq
├── Kallisto
│   ├── C021_0001_20140916_tumor_RNASeq.abundance.h5
│   ├── C021_0001_20140916_tumor_RNASeq.abundance.tsv
│   └── C021_0001_20140916_tumor_RNASeq.run_info.json
└── RSEM
    ├── C021_0001_20140916_tumor_RNASeq.rsem.genes.norm_counts.tab
    ├── C021_0001_20140916_tumor_RNASeq.rsem.genes.raw_counts.tab
    ├── C021_0001_20140916_tumor_RNASeq.rsem.isoform.norm_counts.tab
    ├── C021_0001_20140916_tumor_RNASeq.rsem.isoform.raw_counts.tab
    ├── C021_0001_20140916_tumor_RNASeq.rsem_genes.results
    ├── C021_0001_20140916_tumor_RNASeq.rsem_isoforms.results
    └── Hugo
        ├── C021_0001_20140916_tumor_RNASeq.rsem.genes.norm_counts.hugo.tab
        ├── C021_0001_20140916_tumor_RNASeq.rsem.genes.raw_counts.hugo.tab
        ├── C021_0001_20140916_tumor_RNASeq.rsem.isoform.norm_counts.hugo.tab
        ├── C021_0001_20140916_tumor_RNASeq.rsem.isoform.raw_counts.hugo.tab
        ├── C021_0001_20140916_tumor_RNASeq.rsem_genes.hugo.results
        └── C021_0001_20140916_tumor_RNASeq.rsem_isoforms.hugo.results

所以我想将所有 *.abundance.tsv 连接在一个文件中，*.rsem.genes.norm_counts.tab 在第二个文件中，*.rsem_genes.hugo.results 在第三个文件中。最好和最有效的方法是什么？我可以使用任何东西 - R、Python 或 Bash。

$ find --version
find (GNU findutils) 4.5.11
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Eric B. Decker, James Youngman, and Kevin Dalley.
Features enabled: D_TYPE O_NOFOLLOW(enabled) LEAF_OPTIMISATION SELINUX FTS(FTS_CWDFD) CBO(level=2) 

谢谢!

Answer 1

使用 bash find 命令如下； exec 中的cat 命令应用于该命令返回的所有文件。 + 选项用于确保 shell 生成的 cat 实例不超过一个。

这里的 {} 表示返回查找命令的文件。引用更多关于 find -exec

find . -type f -name '*.abundance.tsv' -exec cat "{}" + >> ../AbundanceTSV.tsv
find . -type f -name '*.rsem.genes.norm_counts.tab' -exec cat "{}" + >> ../GenesNormCounts.tab
find . -type f -name '*.rsem_genes.hugo.results' -exec cat "{}" + >> ../HugoResults.results