How to speed up indexing

[kskarthik]

Hello, I am exploring bleve to handle search for https://gitlab.com/kskarthik/go-ifsc & i am having a couple of questions

1. I have a csv slice which has 165785 records & bleve is taking a lot of time to index these one by one. Am i doing it wrong?

	for i := range csvSlice[1:] {
		index.Index(csvSlice[i][1], csvSlice[i])
	}

2. How do i access this map's value? I am new to Go, So excuse my limited knowledge 😄

map[:[Abhyudaya Co-operative Bank ABHY0065001 Abhyudaya Co-operative Bank IMPS MUMBAI MUMBAI MAHARASHTRA ABHYUDAYA BUILDING, KAMAL NATH MARG,NEHRU NAGAR,KURL
A-EAST,MUMBAI-400024 +919653261383 true true MUMBAI IN-MH true 400065001 true ]]

[Thejas-bhat]

bleve supports indexing batch wise (which is less expensive at the end of the day when compared with directly indexing docs separately). what can be done is create a batch using:

batch := index.NewBatch()

and then put the docs inside the batch using
err := batch.Index(docID, doc)

and periodically (every 1000 docs for eg) index this batch using

err := index.Batch(batch)

also, the batch variable itself can be reused for indexing the next batch of docs by reseting the batch using batch.Reset() before the next operation.

Iterating the map itself can be done using the range functionality (for k, v := range map { ... }). The Fields map I believe is keyed by the field name and the value is content of the field retrieved from a particular document (the docNum would be part of hit as "ID").

[kskarthik]

Great! I will try these suggestions! Thank you very much!

[kskarthik]

batch indexing worked like a charm! This is how i implemented the index https://gitlab.com/kskarthik/go-ifsc/-/blob/bleve/cmd/index.go

Now, I am stuck at the result handling stage. bleve gives the result as []interface{} but i need it as []string. How could i achieve this? Excuse me for another noob question.

Code: https://gitlab.com/kskarthik/go-ifsc/-/blob/bleve/cmd/root.go?ref_type=heads#L118

[kskarthik]

update: i could convert it with this code

	// convert the result []interface{} to []string
	var finalResult []string
	for _, val := range result.Hits[0].Fields {
		for _, v := range val.([]any) {
			finalResult = append(finalResult, v.(string))
		}
		return finalResult, nil
	}