Add a send_state column to problems

[dracos]

In a similar vein to #3865 if we add such a column to the problem table we can get the following benefits:

• Can mark a report as 'skipped' (not going to be sent) without having to remove its bodies_str as we do now;
• Can mark a report as 'acknowledged' (want to be sent, but backend is refusing it and we have contacted them about it) so it's not repeatedly included in the 'stuck reports' output.
• Anything else I've missed?

Currently sending looks for reports: in an open state, with a bodies_str, not to a No-op body or No-op contact, currently unsent (either totally or with send_fail_body_ids set).
Could instead look for 'unprocessed' (and acknowledged?) - if no bodies_str/closed/fixed, mark as 'processed'; if No-op body/contact, do nothing; otherwise try and send - set sent if succeeded.

[davea]

I like this idea. Would help identify real problems amongst the noise. A couple of thoughts:

• A send_state_note text field would let admins add a description of why it had been acknowledged/link to FD ticket/etc
• Could break the acknowledged reports out into their own section on the admin front page maybe, or at least indicate their status on that table
• Would acknowledged stuck reports be counted by icinga?
• Should there be a limit to how long a report can be in the acknowledged state before automatically reverting? Otherwise there's a danger it'll just languish forever

[dracos]

More details on first bullet point - per some recent user support, potentially enable us to have Refused councils still show new reports in All reports (currently, e.g. the Tower Hamlets page only shows reports sent to Tower Hamlets because it goes off the bodies_str, it ignores reports in TH not sent there) in that it could record the bodies_str but set the state to skipped.

[chrismytton]

I've started a branch for this here: https://github.com/mysociety/fixmystreet/compare/add-send-state-to-problems

At the time of writing it's just got the DB migrations in there.

Next steps are to update the report sending code to check the value of the send_state attribute and handle appropriately.

[dracos]

"Could break the acknowledged reports out into their own section on the admin front page maybe, or at least indicate their status on that table" - they'll be greyed out if acknowledged.
"Would acknowledged stuck reports be counted by icinga?" - I'd go with no, point of acking is to keep the alert to unknowns.
"Should there be a limit to how long a report can be in the acknowledged state before automatically reverting?" - can revisit this, but I think no, we can't force things to fix. And we can keep an eye on admin front page.

[dracos]

Plus #4517, and adding send_state => 'processed' to GetServiceRequests.pm (so they weren't processed), and changing the index to be partial excluding hidden/unconfirmed and putting send_state first, and making sure unconfirmeds were still to be processed too