java - 无法读取 FileWriter 之后生成的 csv 文件并在 Java 中找到重复项

Question

我正在尝试读取从 FileWriter 创建的第一个 csv 文件。

第一个 csv 文件的输出是出现/出现超过 10 次的实体名称(列[1])的内容。

读取第一个 csv 文件后，我尝试检查列 [5](即推文 token )的重复项并写入，并将其添加到第二个 csv 文件中。我尝试使用 .contains 方法，它不会检查重复项。

Update: I have successfully read the file but not able to remove duplicates in EventDetectionToken().

这是代码:

import java.io.*;
import java.util.*;

public class EventDetectioncopy {
    public static void main(String[] args) throws FileNotFoundException, IOException{
        //1st csv file
        System.out.print("Enter a name for new Tweet Cluster sorting by name entity: ");
        BufferedReader scanName = new BufferedReader(new InputStreamReader(System.in));
        String newNamefile = scanName.readLine();

        //2nd csv file
        System.out.print("Enter a name for new Tweet Cluster sorting by tweet tokens: ");
        BufferedReader scanToken = new BufferedReader(new InputStreamReader(System.in));
        String newTokenfile = scanToken.readLine();

        try {
            eventDetectionName(newNamefile);
            eventDetectionToken(newNamefile, newTokenfile);

        }
        catch (FileNotFoundException e) {
            System.out.println(e);
        }
        catch (IOException e){

        }
    }

    public static void eventDetectionToken(String fileInput, String fileOuput) throws FileNotFoundException, IOException{
        FileWriter newCsv = new FileWriter(fileOutput + "." + "csv");
        BufferedWriter newCsvBW = new BufferedWriter(newCsv);
        BufferedReader reader = new BufferedReader(new FileReader(fileInput + ".csv"));
        String data;

        try{
            String temp = null;
            List<String> tempList = new ArrayList<String>();

            do
            {
                data = reader.readLine();
                String tweetToken = null;

                if(data != null)
                {
                    String[] splitText = data.split(",");
                    tweetToken = splitText[5];
                }

                if(temp != null)
                {
                    if(data == null || tweetToken.contains(tweetToken))
                    {
                        if(!(temp.equals(tweetToken)))
                        {
                            for (int i = 0; i < tempList.size(); i ++)
                            {
                                newCsvBW.append(tempList.get(i));
                                newCsvBW.append("\n");
                                System.out.println(tempList.get(i));
                            }
                         }

                        tempList.clear();
                        temp = tweetToken;
                    }
                }
                else
                {
                    temp = tweetToken;
                }
                tempList.add(data);
            }
            while(data != null);
        }
        finally
        {
            newCsvBW.close();
            reader.close();
        }
    }

    public static void eventDetectionName(String filename) throws FileNotFoundException, IOException{
        String csv = "1day/clusters.sortedby.clusterid.csv";
        FileWriter newCsv = new FileWriter(filename + "." + "csv");
        BufferedWriter newCsvBW = new BufferedWriter(newCsv);
        BufferedReader reader = new BufferedReader(new FileReader(csv));
        String data;

        try{
            String temp = null;
            List<String> tempList = new ArrayList<String>();
            List<Long> tempTime = new ArrayList<Long>();
            do 
            {
                data = reader.readLine();
                String nameEntity = null;
                if (data != null) 
                {
                    String[] splitText = data.split(",");
                    nameEntity = splitText[1];
                }
                if (temp != null) 
                {
                    if (data == null || !(nameEntity.equals(temp))) 
                    {
                        if (tempList.size() >= 10) 
                        {
                            for (int i = 0; i < tempList.size(); i++) 
                            {
                                newCsvBW.append(tempList.get(i));
                                newCsvBW.append("\n");
                                System.out.println(tempList.get(i));
                            }
                        }
                        tempList.clear();
                        temp = nameEntity;
                    }
                } 
                else 
                {
                    temp = nameEntity;
                }
                tempList.add(data);
            } 
            while (data != null);
        }
        finally
        {
            reader.close();
            newCsvBW.close();
        }

    }
}

下面是原始 csv 文件的一些内容:“clusters.sortedby.clusterid.csv”，在使用重复的推文 token 运行 EventDetectioncopy.java 之前(第 [5] 列):[clusterid]、[名称实体]、[tweetid]、[时间戳]、[用户 ID]、[推文 token ]、[推文文本]

1   rick ross   2.5582E+17  1.34983E+12 389746870   rick ross dice pineappl Rick Ross x diced pineapples
1   rick ross   2.5582E+17  1.34983E+12 56082039    dice pineappl uhhh rick ross voic   Diced Pineapples. UHHH *Rick Ross voice*
1   rick ross   2.55821E+17 1.34983E+12 870278689   rick ross trend Why is Rick Ross trending?
1   rick ross   2.55822E+17 1.34983E+12 379948188   lmfao rick ross grunt   Lmfao he did that rick ross grunt .
1   rick ross   2.55822E+17 1.34983E+12 276594374   play rick ross  they played w| rick ross !
1   rick ross   2.55822E+17 1.34983E+12 386219877   rick ross ugli  Rick Ross So Ugly ..
1   rick ross   2.55822E+17 1.34983E+12 53327754    wanna play rick ross belli  I Wanna Play in Rick Ross Belly..!
1   rick ross   2.55824E+17 1.34983E+12 19690034    rick ross dice pineappl ft wale amp drake video via laleak  Rick Ross - Diced Pineapples ft. Wale & Drake (Video) via @laleakers
1   rick ross   2.55825E+17 1.34983E+12 357250991   husband rick ross   where my husband rick ross î„…î‰
1   rick ross   2.55825E+17 1.34983E+12 53734179    throw rick ross kirko bangz *Throws Rick ross At Kirko Bangz*
1   rick ross   2.55825E+17 1.34983E+12 462179553   rick ross stay fresh    Rick Ross Stay Fresh!!!!
1   rick ross   2.55827E+17 1.34983E+12 46744853    offici music video dice pineappl rick ross drake wale   Official Music Video " Diced Pineapples" Rick Ross / Drake / Wale
1   rick ross   2.55829E+17 1.34983E+12 461725574   saw rick ross uhhh ifxckgaygirl dadd    i saw rick ross their .. uhhh @ifxckgaygirls dadd :p
1   rick ross   2.55832E+17 1.34983E+12 283244204   rick ross wavi fat guy  Rick Ross is a wavy fat guy
1   rick ross   2.55832E+17 1.34983E+12 528834435   rick ross dice pineappl Rick Ross - Diced Pineapples
1   rick ross   2.55835E+17 1.34983E+12 463279022   rick ross featur wale amp drake dice pineappl ricki ross experi downtim less 24 hour    Rick Ross featuring Wale & Drake – Diced Pineapples:   Ricky Ross experiences no downtime as less than 24 hours ...
1   rick ross   2.55835E+17 1.34983E+12 28460245    yuck lalasodiddi need husband rick ross take award home hiphiopaward    YUCK! RT @LalaSoDiddy: I need my husband Rick Ross to take some awards home #HipHiopAwards
1   rick ross   2.55836E+17 1.34983E+12 330811468   kingkennzi rick ross round  “@KingKennzie: Rick Ross is very round.†ðŸ
1   rick ross   2.55836E+17 1.34983E+12 124024753   rick ross titti Rick Ross Titties!
1   rick ross   2.55836E+17 1.34983E+12 765822380   rick ross titti tho Rick Ross and them titties tho!!!
2   tyler oakley    2.55821E+17 1.34983E+12 867420925   know someth trend new asktyl tyleroakley live   HOW DO YOU KNOW WHEN SOMETHING IS TRENDING? IM NEW TO THIS... #aSKTYLER
2   tyler oakley    2.55822E+17 1.34983E+12 504044044   asktyl get perfect quiff tyleroakley live   #AskTyler How do you get a perfect quiff :)?
2   tyler oakley    2.55822E+17 1.34983E+12 709347721   asktyl realli homework right now tyleroakley live   #asktyler i really should be doing homework right now
2   tyler oakley    2.55822E+17 1.34983E+12 171667747   obsess right now asktyl tyleroakley live    what is your obsession right now? #asktyler
3   wiz khalifa 2.5582E+17  1.34983E+12 588829718   dont like wiz khalifa look sexi I don't like Wiz Khalifa but he looks sexy.
3   wiz khalifa 2.55856E+17 1.34984E+12 502086440   feel like wiz khalifa right now I feel like wiz Khalifa right now..
3   wiz khalifa 2.55866E+17 1.34984E+12 446056049   like wiz khalifa hes ador realli look like hot cheeto man thingi    I like Wiz Khalifa he's adorable, but he really do look like the hot cheeto man thingy
3   wiz khalifa 2.55883E+17 1.34984E+12 67747115    np ne yo ft wiz khalifa dont make em like   #Np Ne-Yo ft. Wiz Khalifa - They don't make em like you 

Update: How can I remove the duplicates of it?

Answer 1

已编辑:它将删除所有重复项并仅保留一项。

public static void eventDetectionToken(String fileInput, String fileOuput)
        throws FileNotFoundException, IOException {
    FileWriter newCsv = new FileWriter(fileOuput + "." + "csv");
    BufferedWriter newCsvBW = new BufferedWriter(newCsv);
    BufferedReader reader = new BufferedReader(new FileReader(fileInput + ".csv"));
    String data;

    try {
        List<String> existanceTokens = new ArrayList<String>();

        do {
            data = reader.readLine();
            String tweetToken = null;

            if (data != null) {
                String[] splitText = data.split(",");
                tweetToken = splitText[5];

                if (!(existanceTokens.contains(tweetToken))) {
                    newCsvBW.append(data);
                    newCsvBW.append("\n");
                    existanceTokens.add(tweetToken);
                }
            }
        } while (data != null);
    } finally {
        newCsvBW.close();
        reader.close();
    }
}

<小时/>

但是，如果您想首先按 [名称实体] 创建包含重复项的 CSV 文件，然后基于此文件创建第二个包含 [tweet token] 重复项的 CSV 文件，则需要将 inputCSV 更改为 newNamefile 用于第二个 eventDetection 调用，如下所示:

    eventDetection(inputCSV, newNamefile, 1);
    eventDetection(newNamefile, newTokenfile, 5);

希望有帮助。