Getting corrupted zip files for twitter filter after stopping the filter

[rajatvij]

I am getting corrupt zip files for twitter filters after stopping the filter.

Steps to reproduce:

1. Create and save a new active filter in SFM admin.
2. De-activate the filter.
3. Run python manage.py organizedata
4. Go to the directory where zip file is stored and run gunzip -c <filename.gz>

We can see an unexpected end of file.

NOTE: I used docker to create filter.

[kerchner]

@rajatvij, is this occurring in a non-docker environment?

[edsu]

I suspect that Supervisor is sending a signal to the filterstream process to shut it down, and it's not being handled long enough to close and flush the gzip file that is being written to, leaving it in a corrupted state. One thing you might want to try is adding a close method to sfm.ui.models.RotatingFile and then handling the interrupt signal (SIGINT, SIGTERM, I'm not sure which) when streaming the filter in sfm.ui.management.commands.filterstream so that you can call it before exiting.