performance of filtering a large table

[dnalor]

When filtering a table with many rows (tested with ~40000), cursor movement gets very sluggish (several secs to go to the next row).
Table setup is just table.WithFilterInputValue( somefilter)
Any idea how I can speed this up?

[Evertras]

This is likely being recalculated way too often. The workaround would be to filter the rows yourself and keep the cache of filtered rows to feed to the table, but there's likely something the table can do to better track/cache this. I haven't worked with such a large row count before, this would be a good area for improvement.

[Evertras]

I went to check this for myself, and a table with 40,000 rows will absolutely lock up if you're not using paging. However, if you're using paging, this is extremely quick when I tested for both the filtering and the rendering/scrolling. Are you using paging or trying to render all 40,000 rows on the screen directly?

[dnalor]

Yes, pagination is enabled:

	return  table.New([]table.Column{
			table.NewFlexColumn(columnKeyName, " name", 13).
			WithStyle( lipgloss.NewStyle().Align(lipgloss.Left)).WithFiltered(true),
			table.NewColumn(columnKeySize, "size", 10),
			table.NewColumn(columnKeyModTime, "time", 20).WithFiltered(true),
			table.NewColumn(columnKeyMode, "mode", 10),
		}).
		WithTargetWidth(int(parms.Columns)).
		WithMaxTotalWidth(int(parms.Columns)).
		WithHorizontalFreezeColumnCount(1).
		HeaderStyle(lipgloss.NewStyle().Foreground(lgColorSnazzyCyan).Bold(true)).
		Border( tableCustomBorderThin).
		WithSelectedText(" ", "✓").
		WithPageSize( int(parms.Rows) - termRowsReserve).
		SelectableRows(true).
		Focused(true).
		Filtered(true)

One thing I noticed is that performance depends on how many rows are displayed. On my mid-range desktop (AMD 5600), 10 rows are ok, but for 100, it takes ~2sec to move the cursor one row.

[Evertras]

Ahhh, I'm also seeing slowdown at higher page size counts. Even 30 is noticeable for me. This is a performance issue, I hadn't checked past 10.

[Evertras]

My initial suspicion was correct, there was a function that was doing a LOT of repeated calculations that was being used all over the place. I've turned it into a cached version and seen massive performance improvements when running the 40,000 line stress test with a page size of 100. Before I was also getting some nasty slowdown, but now it seems quite smooth.

Released the fix as v0.15.1, please let me know if this fixes performance for you as well or if we need to do more work here.

[dnalor]

It's definitely significantly faster.
There's a weird phenomenon, though:
Scrolling down is real-time (i.e. scrolling stops when cursor down is released)
Scrolling up is not real-time / lags (i.e. scrolling continues for a couple of rows after cursor up is released)

(terminal: kitty | terminal size: 250x100 | rows: 44000 | filtered down to: various (seems irrelevant))

[Evertras]

Interesting... I'm unable to reproduce that. I get the same apparent performance both up and down and switching between pages.

For the record, I'm using Alacritty at size 224x62 with 40000 rows, both with filtering and without. 🤔 I tried increasing the terminal height + page size to 130 as well and ended with the same result.

https://gist.github.com/Evertras/07b78eca90f2faee45292db434505577 I'm basically just altering the base filter example for testing this. If you run this file, do you still see the same behavior?

[dnalor]

(sorry for the late reply)
So, I tested this with my app in both alacritty and urxvt, and it occurs in both.
Then I ran your filterstress, and it doesn't exhibit this behavior.

However, with my app, it seems to depend on the unfiltered number of rows (eg. 900 (of 1300 total) -> not noticeable, 1400 (of 45000 total) -> noticeable)
It also doesn't occur if the table is not filtered, even with 44K rows
And it's only for upward movement - rather strange.

Anyway, I just released my app here, in case you want to check if you can replicate this on your setup.
If that's too much of a hassle, we can certainly close this issue, as the performance is now much better in any case.

[Evertras]

That's some curious behavior. I'd be interested to try checking it out at some point, if you could also provide a sample archive file where you're seeing this with your app?

Let's leave this open for now, and if it turns into too much of a goose chase then we can close this later.

[dnalor]

I'm testing with source packages I have in my OS cache.
Examples:
go1.20.2.src.tar.gz 13283 members / filtered by .go (99 pages) → fast
rust-1.65.0-x86_64-unknown-linux-gnu.tar.xz 35251 members / filtered by .html (354 pages) → down ok, up lags
firefox-102.9.0esr.source.tar.xz 334655 members / filtered by .cpp (95 pages) → generally too slow

I usually run everything on kitty 250x100, but I tried alacritty and even urxvt and noticed the same behavior.

[Evertras]

I'm able to replicate the problem when using your app and looking at the rust archive. A brief profiler check seems to indicate most of the work is being done in the table code, let me see if I can figure out what might be going on here...

[Evertras]

I'm still a little puzzled why up vs down was doing this, but it was DEFINITELY happening and going up was taking approximately double the total time as going down according to the profiler. Regardless, digging around has uncovered that we should be able to be a lot more aggressive with the caching. I've made some updates and tested them locally with your app and seen drastic improvements in performance. Please give v0.15.2 a try and let me know if it works for you as well.

[dnalor]

Performance is indeed significantly better. In fact, it's excellent - even for a filtered table with 330k rows, scrolling is now completely lag-free.
Thanks for fixing this!