[super]: Move upload/import to work over the API

[markmacgillivray]

Upload/import should send to the API, and should run on a separate process so that the user is not waiting for response/timeout. Instead, respond to the user with "this is happening, here is the collection link, check back later (very soon for most small collections)". Alternatively but a bit harder, set up the system so that a hit on the collection link shows a "i am processing response", then the user could be sent straight there anyway.

[markmacgillivray]

ref #155 - temporary fix of increasing nginx timeout

[epoz]

Suggestion - To simplify the parser pipeline:

A BibServer parser accepts the to-parse format on stdin and outputs valid BibJSON on stdout.
That is the bare minimum specification, all other options are format specific.
This could then also handle any site-specific scrapers that the BibServer community would want to create.

[rufuspollock]

The key point here, though is that parsers are not usually run by us but by community (though we perhaps preserve support for things like BibText and RIS). So, we don't really care about the "parser API" since how those parsers work will not affect us -- people will run parsers wherever -- writing to BibSoup over our API.

[markmacgillivray]

this should not be in the beta sprint. will be part of the next dev sprint. moving out

[markmacgillivray]

Etienne now working on altering the parser functionality and creating stand-alone async

[markmacgillivray]

see also #117

[markmacgillivray]

and the closed #141 for more info

[rufuspollock]

Where are details of what Etienne is working on or planning to do :-)

My 2c we want a convert.bibsoup.net service that does conversions / parsing. This can be used standalone or used for importers to bibsoup.net.

[epoz]

I am busy working on this: https://github.com/okfn/bibserver/wiki/AsyncUploadDesign
Have actually started coding on it, as a branch from main repo.
Will commit first cut soon.

[markmacgillivray]

Yep, Etienne has written his plan and we agreed it yesterday. Then import from bibsoup will cease to rely on an internal routing and will be able to use the converters via API.

[epoz]

See: 5454da1 for a running version. In this branch you can now do en upload from the web interface.