-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
C coded generators (explodes, readfile, _popenr) #1005
Conversation
And fix the Travis CI build, natch. |
@svnpenn If you want to kick the tires on this, go for it.
|
The privilege model where we assign privileges to particular modules and/or the top-level program is not perfect. Since jq function argument closures take no arguments, they can't be used to wrap around functions like A better privilege model would allow us to have modules that perform authorization based on the name of module(s) in the call stack, a la Java's model. |
Then we'd need to distinguish trusted modules from not, so we'd need something just like Java policies, probably a JSON file to load at runtime. Can we grow the privilege model slowly, so we don't have to develop the whole thing at once? Here's a thought:
And then we can add the first and second usage now, the third a bit later, and the last one later still. For the last one we'd have a local .json file indicating which modules get what privs, and we'd have a builtin Now, the Java model is quite good, but falls down due to lack of attention to authorization by developers, and complexity of writing local policies. So, it's not exactly a great model. In practice we've ended up with people running with all classes as trusted and never downloading code -- the applet model failed. So perhaps we should NOT copy Java, and we should stop at the first two or three usages above. I'm all ears as to better privilege models. |
@svnpenn Let me rephrase for users. Should users have to "bless" trusted modules manually, or even grant them specific privileges, and if so, should this be done only on the command-line, at module installation time, or either at the choice of the user? I'm of the opinion that most users won't really be in a position to manage fine-grained code trust. For most users it will be all-or-nothing. I can live with that, but I suppose it'd be nice to be able to limit trust when installing random modules from random repositories. Think of things like jqplay.org. It should allow you to run jq programs, but not to read/write files or run programs on their heroku. For jqplay a big switch is sufficient. For scripts with a she-bang we may need to be able to request privilege in the she-bang invocation, but the user should still be able to refuse it. If we ever support From the point of view of generality, I like the Java model, but from the point of view of usability, I think it's just best to KISS: one simple switch granting privilege to the entire program and the modules it uses. |
Suppose you download and install a jq program/module from some repository. Now you want to run it. But you don't want it to do entirely arbitrary things, like run "rm -rf /", add your host to a botnet, or similar other destructive things. Without I/O and shell-out builtins, jq is just a sandboxed filter, and the most harm it can do is consume memory and CPU -- that's not nothing, if it's a fair bit less severe a problem than being able to run arbitrary command, don't you think? Remember, various contributors really want to build a package repository ecosystem for jq. I like the idea myself. The easiest way to manage privilege then is to let the user decide whether to grant it to the whole jq program, modules and all, in each run ( |
The C-coded generators now consist of three functions: one to setup the state and return the first value, one to step to return the next value, and a reset function to cleanup the generator's state. The step function is new. This saves a few lines of code in |
Oy, I screwed up my history. EDIT: Fixed. |
24d34ed
to
9cb6b56
Compare
9cb6b56
to
ab780a8
Compare
A trampoline for C-coding jq functions that take and call closure arguments won't be that hard, and could be quite useful. It will require two new opcodes: one to enter the C-coded function, and one to re-enter when a closure it "calls" outputs a value, that way the outputs of the closures can be made available to the C-coded closure. A start at such a thing can be seen in the The idea is to make it possible to implement The same approach would allow for a C-coded Both could be implemented with something like file handles, of course. Note that we could even implement co-routines with this C-coded generic function scheme -- without handles. A C-coded EDIT: Symbolic names for coroutines ("handles") would be best though -- more user-friendly. Passing a value to a co-routine would be something like |
How about a variant where the input is sent to stdin like a normal pipe, and make |
I would really like to see something like this. Is there any ETA? |
Closing this in preference to #1843. |
NOTE: NOT DONE YET.
This is just a proof of concept. I'm able to read files, read from commands.
TODO:
_popenr
_popenr
(an arg-vector form should use posix_spawn(), or spawn() on Windows, and bypass the shell)readfile
and_popenr
that raise anerror
More importantly, how to handle something like
writefile(EXP)
orpopen(EXP)
? Even C-coded generators can't call jq expressions. So that means having file handles. We can avoid those forreadfile
. We've discussed how to do filehandles in a way that doesn't leak handles out, doesn't allow callers to slip in an incorrect handle, and doesn't have a dangling reference problem: make the C-coded functions to open/close handles and read from / write to handles be available only to privileged jq-coded functions that will use them in such a way as to never leak them:((Food for though: Some day it would be nice to be able to C-coded functions that take closure arguments. I imagine setting up a trampoline in the VM for a C-coded function to call a closure argument, but to avoid filling the C call stack the C-coded function would have to immediately return and get called again with the closure's output(s); state would be kept in a state structure pointed to by the C-coded function's jq VM frame. Calling closures in separate VMs could also be done, but arranging for them to be able to access their captured variables would be tricky. With such an extension there'd be no need for handles for simple I/O builtins.))
We'll still need handles, and ones that are exposed to jq callers, for co-routines, so we might want to bite this bullet sooner rather than later. I have an old branch with file handle support for the jq VM that I could revive. Still, for simple I/O builtins, we don't need to expose a concept of handles, and that seems appealing.