laravel - 如何在AWS Elastic Search中进行批量索引

Question

我有一个包含200,000条记录的数据库-在AWS Elastic Search中，我们想为这些文档建立索引，但是，(使用laravel console命令)一次为它们建立索引需要很长时间。
我们如何做一个大索引来加快速度。
这是我们当前的索引功能

public function handle()
{
    $this->init();
    $pageSize=10;
    $indexName = env('AWS_VIDEO_INDEX');
    $videos = Video::orderBy('id','desc')->chunk($pageSize, function ($vserverVideos) use ($indexName){
        foreach($vserverVideos as $video){
            /**
             * @var $video Video
             * var/class[]
             */
            $subtitles = $video->getSubtitles();
            $params = [
                'index' => $indexName,
                'id'=>$video->id,
                'body'  => [
                    'id'=>$video->id,
                    'title'=>$video->title,
                    'description'=>$video->description,
                    'jobStatus'=>$video->jobStatus,
                    'youtubeId'=>$video->youtubeId,
                    'thumbnail'=>$video->thumbnail,
                    'playlistId'=>$video->playlistId,
                    'category'=>$video->category,
                    'channelId'=>$video->channelId,
                    'publishedDate'=>$video->publishedDate,
                    'created_at'=>$video->created_at->toIso8601String(),
                    'updated_at'=>$video->updated_at->toIso8601String(),
                    'url'=>$video->url,
                    'subtitles'=>$subtitles,
                ]
            ];
            $response = $this->elasticClient->index($params);
            if ($response['result']==='updated'||$response['result']==='created'){
                $result = $response['result'];
                $numSubs = count($subtitles);
                $this->line("Indexed ($result) $video->title with $numSubs subtitles");
            }
        }
    });



    return 0;
}

Answer 1

再次不熟悉Laravel，但这是python中的一种方法，使用 elasticsearch 软件包，可以实现“批量索引”:

import psycopg2
conn = psycopg2.connect(host="<host>",database='<database>',user='<user>',password='<password>')
cur = conn.cursor()
# just fetching records from say, a postgres table
cur.execute("select * from public.music")
res = cur.fetchall()

marr = []
for r in res:
    mobj = {
        '_id': int(r[0]), 'routing': r[4],
        'created_at': r[1].strftime("%Y-%m-%d %H:%M:%S"), 'updated_at': r[2].strftime("%Y-%m-%d %H:%M:%S"),
        'sound_id': r[3], 'music_id': r[4], 'music_name': r[5], 'music_author': r[6], 'music_original': r[7],
        'play_url': r[8], 'cover_thumb': r[9], 'cover_medium': r[10], 'fetched_times': r[11], 'cover_large': r[12]
    }
    marr.append(mobj)

然后 :

from elasticsearch import Elasticsearch, helpers
import json 

client = Elasticsearch('http://<es-ip>:9200/') # replace with AWS ES domain accordingly
resp = helpers.bulk(client, uarr, index="music")

resp = helpers.bulk(客户，marr，index =“tk音乐”)