high concurrency ( >8) query , postgresql(cstore_fdw) crash

[jyzhao2006]

I ran a test with 8 thread , and encountered server crash . when the thread number is 4, it is okay.

see the log below:

postgres=# LOG: server process (PID 29409) was terminated by signal 9: Killed
DETAIL: Failed process was running: select * from jaelog limit 1 offset $1
LOG: terminating any other active server processes
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

[pykello]

@jyzhao2006
To investigate the problem, I have some questions:

1. What was your table schema?
2. What was your data size?
3. What was your query?
4. How much main memory did you have?
5. Which OS distribution did you use?

Any additional information that you might think can be relevant is welcome :)

Update. I see from the log that you used the select * from jaelog limit 1 offset $1 as the query. What values of offset were you using?

[jyzhao2006]

Hi,thanks for your anther!

I found the problem. The postgresql was killed by linux oom killer when I test PG with 8 thread concurrency query.

The value I use in " select * from table limit 1 offset $1 " is generated Randomly.

[jyzhao2006]

Hi, I have other questions.

1. How do i do high concurrency query with cstore_fdw effectively? do you have some suggestions?
2. Do you have some high concurrency query test on cstore_fdw? how does it looks like?

Thank you very much!

[pykello]

@jyzhao2006

1. Since it seems that you had "out of memory problem", you can try decreasing stripe_row_count option to a lower value (say 50000) so memory usage also decreases. This may decrease performance. You can try different values for stripe_row_count to find the optimal one for you.
2. Not yet. We are planning to do another benchmark which will be more extensive than the previous one. We may have a high concurrency one in there.

[pykello]

@jyzhao2006 did reducing stripe_row_count solve the memory problem?

[jyzhao2006]

sorry, I forget give feedback because i am busy with other things these days.

Thanks for your advice, reducing stripe_row_count did decrease the memory usage so i can run more concurrent request.

I think cstore_fdw consume too much memory when concurrent query is high, can you optimize this?
I run the some case with infobright and it can execute hundreds of concurrent query by consuming not to much memory

[pykello]

I think we should be able to shrink the memory usage by 2x-3x, but more than it seems a bit difficult. I have to look into how infobright achieves this. One difference between cstore_fdw and infobright is that cstore_fdw currently only modifies the scanner node of PostgreSQL and other parts of PostgreSQL don't know / don't care that the scanner node is column store or a row store. While infobright is a column store from ground up, so they might be able to optimize better.

[jyzhao2006]

@pykello
Thanks for your answers! expect to see your improvement!