bash - 在特定作业上运行 SHell 脚本时如何从 HDFS 获取最新的有效分区日期？

Question

我的任务是为特定的 Spark 作业实现所有分配的表。我需要根据要为分配给作业的所有表打印的时间戳和路径编写脚本。我需要获取与该作业关联的表关联的所有时间戳。

这是我开发的脚本。

#!/usr/bin/env bash
JOB_NAME=${1}
 inputDirListings=$(awk -F: -v key="$1" '$1==key {print $2}' test_paths.txt)
for dir in  $(echo $inputDirListings | tr "," "\n");
do
    path=$dir
    echo "dir is $path"
    cmd2='hdfs dfs -du -h $path'
    ev1=`eval $cmd2 | tail -1`
    echo "ev1 value is $ev1"

    hdfsPath=`echo $ev1 | cut -d";" -f3- `
    echo "partition is $hdfsPath"

    latestPartition=`echo $hdfsPath | grep -Eo '[[:digit:]]{4}-[[:digit:]]{2}-[[:digit:]]{2}'`
    echo "latest partition is $latestPartition"

    dt1="$(echo $ev1 | cut -d'=' -f2)"
    arr[i]=`date -d $dt1 +%Y%m%d`

    #---Getting minimum date from array---------
    max=${arr[0]}
    min=${arr[0]}

    for i in ${arr[@]}
    do
    if [[ $i > $max ]] ; then                           
    max=$i-1
    fi
    if [[ $i < $min ]] ; then
    min=$i
    fi
    echo "dt1"
    for (( c=$dt1; c<=$currDate; c++ ))
    do
        echo -n "$c "
        sleep 1
    done 
done
 echo "Max value is $max  , minimal value is $min"
dt2=`date -d $min +%Y-%m-%d`
done

我得到的输出最大值和最小值相同

例如:最大尺寸为 9999-12-31，最小尺寸为 9999-12-31

基本上，我需要获取 9999-12-31 之前的最新分区日期

Answer 1

您的代码仅将最后一个目录中的分区值存储到数组中，因为它在循环中每次都会被覆盖。

你的数组需要定义在循环外，i需要在循环内递增，需要把你的内循环拿出来，如下图:

#!/usr/bin/env bash
JOB_NAME=${1}
arr=()
i=0

inputDirListings=$(awk -F: -v key="$1" '$1==key {print $2}' test_paths.txt)
for dir in  $(echo $inputDirListings | tr "," "\n");
do
    path=$dir
    echo "dir is $path"
    cmd2='hdfs dfs -du -h $path'
    ev1=`eval $cmd2 | tail -1`
    echo "ev1 value is $ev1"

    hdfsPath=`echo $ev1 | cut -d";" -f3- `
    echo "partition is $hdfsPath"

    latestPartition=`echo $hdfsPath | grep -Eo '[[:digit:]]{4}-[[:digit:]]{2}-[[:digit:]]{2}'`
    echo "latest partition is $latestPartition"

    dt1="$(echo $ev1 | cut -d'=' -f2)"
    arr[i]=`date -d $dt1 +%Y%m%d`

    let "i++"
echo "Max value is $max  , minimal value is $min"
dt2=`date -d $min +%Y-%m-%d`
done

#---Getting minimum date from array---------
max=${arr[0]}
min=${arr[0]}

for i in ${arr[@]}
do
    if [[ $i > $max ]] ; then                           
        max=$i-1
    fi
    if [[ $i < $min ]] ; then
        min=$i
    fi
    echo "dt1"
    for (( c=$dt1; c<=$currDate; c++ ))
    do
        echo -n "$c "
        sleep 1
    done 
done