Committee members data is wrong / not updated

[shevyk]

e.g: the Legislative Committee
knesset. gov: http://main.knesset.gov.il/Activity/committees/huka/Pages/default.aspx

oknesset: https://oknesset.org/committee/5/

MKs Tali Polsakov and Michael Malkieli are members but are not on Oknesset
MKs Yoav ben Zur and Rachel Azaria appear as a members on oknesset but are not.

[alonisser]

@OriHoch @shevyk Is this automatically scrapped? I don't think so.. I think currently this is a manual update done by someone

[OriHoch]

@RavivBarzilay is working on getting this data form the new APIs

[OriHoch]

related issues:

• should add committee membership data from the new knesset API knesset-data-python#10
• datapackage should include committee membership details knesset-data-datapackage#7
• should have a scraper that updates committee membership data automatically knesset-data-django#4

[alonisser]

@OriHoch @shevyk There is a bigger modeling problem here, I keep nudging about , just scraping won't solve this.. the problem is that committee members can change during a knesset period, Also there are a couple of statuses besides membership (replacements for example) while our modeling does not allow for membership periods in committees , so currently we always display wrong data. either we display mks that were and currently are not or the other way around..

Solving this should handle the modeling

[OriHoch]

@alonisser
once the relevant tasks I wrote above will be done we will have access to the data from the new knesset api
the new knesset api supports this model - it provides start and end time for each person position
this will then be available in the datapackage zip and in the django models (in a new model, because current model as you say don't support it)
this can be then be used directly from the datapackage or via Re:Dash

Anything we should change in the above plan ^ ?
just trying to be practical here 👍

[alonisser]

Actually I'm still don't understand separating models from Oknesset.. I thought the datapackage thing was about separating data and passing scraped data in predefined package. Separating models and models logic from the app does not look very good for me, and the whole inlined imports on every instance method in Oknesset does not look very pythonic to me. I view the missing tests (I guess since there is no point of testing them outside of oknesset app) as a hint that this separation isn't correct, since tests are supposed to be package contained.

But since this is the current implementation and ori wants to be practical: We need a new model of committee membership (start, end, mk, committee, type) , and changing the views to get and display according to committee membership . From the top of my head I don't think we need to change Committe model fields but change adapt some instance methods do need to change, including admin and admin edit screens for adding/removing a member from committee.

I can write the the needed changes in Oknesset, once I know the data structure retrieved

[OriHoch]

would be best to discuss in person what's the goal with Knesset-data separation. the darapackage is only one goal, there are others

regarding the inline imports - you are absolutely correct, it's bad and supposed to be temporary. the end goal is to have knesset-data-django as a complete django package which contains the knesset data DB, different frontend implementations could use this, preferably via api (Open Knesset website being one such frontend)