Make a function that runs cell blocks instead of copying cell contents into the console

[bcolsen]

I use lots of cell blocks to make figures and other things so my history gets really full of cell blocks. I don't need my Cell blocks in history or the 'up arrow' on IPython as they are already in my editor.

I tried to make cell blocks into temp files and the run the temp files with 'run file'. That seemed overly complex though.

Is there a way to run code on the IPython kernel without the front end knowing about it?

[CAM-Gerlach]

I agree, this would be nice. No idea about the best way to handle it, though. I do sometimes like the ability to rerun single lines or short selections previously sent from the editor, but never cells, and they both make the history log much larger (leading to, e.g. #7080 ) and the console much more cumbersome to navigate with up and down arrow. @ccordoba12 ?

[bcolsen]

I do sometimes like the ability to rerun single lines or short selections previously sent from the editor

I was thinking that F9 would stay the same as it is now for coping lines from the editor.

One crazy idea would be to make a temp copy of the current file with all the code commented out except the cell that needed to be run. That way my trace backs would show the right lines as well.

I might try this, but I haven't had a great time dealing with temp files on windows(I could delete them)

[CAM-Gerlach]

I was thinking that F9 would stay the same as it is now for coping lines from the editor.

I assume F9 is the default shortcut for run line/selection (of course, the keybindings for the various run commands are one of the first things I change in a Spyder install, since I loath the default ones). If you're not using cells and have to make long selections it could get annoying, but better safe then sorry there I'd think.

One crazy idea would be to make a temp copy of the current file with all the code commented out except the cell that needed to be run. That way my trace backs would show the right lines as well.

That's always been a tad annoying to me too, but I'm sure there's much more elegant ways to handle that—you'd just need to grab the line number of the first line of the cell being run, then offset all the traceback lines by that much, or (hackier) insert that many blank lines before the actual traceback. Furthermore, I'm not sure I fully understand how the above idea would solve the primary problem discussed in this thread.

I haven't had a great time dealing with temp files on windows(I could delete them)

Again, I'm almost sure that won't be necessary, since there's certainly a better solution than something that crude.

[ccordoba12]

Is there a way to run code on the IPython kernel without the front end knowing about it?

Yes, there is. We have a method called silent_execute that evaluates code in the kernel without showing anything in the console. In fact, the Variable Explorer/Console interaction is built around that.

Several people have asked for this feature already, so we could implement it for Spyder 4. However, we should show a message in the console to let people know something is being evaluated on it. It could be similar to the message we show when restarting a kernel, i.e. something like

Evaluating cell placed in foo.py:12-30

[bcolsen]

I've had a look at this in the past. You can assign me if no one else is on it.

[ccordoba12]

Done, thanks!

[bcolsen]

I split off the history saving part of this issue to #7233

I have been playing around with the making cells run in the console without showing all the cell content here are some option/challenges I came up with.

Goals for this issue:

1. Run a cell without taking up lines of console space
2. Fix the traceback to have line numbers that match the file

Option 1: Silent Execute

Silent execute allows for commands to be run in the console with no indication to the users. There is no "busy" status and there is no history. I was worried this wouldn't work at all because there would be no output from the cells at all but print statements still output to the console. The print statements however get printed awkwardly before the prompt.

A bit of a hack (but it works surprisingly well) is to execute a visible command right after the silent one. The visible command waits for the silent command so the "busy" status is activated, the users see the visible command and the stdout of the cell is even placed after the prompt.

For example I run this cell:

# %% run cell
x=10
for i in range(10):
    print(i,'cell printing')
    time.sleep(0.2)

I get:

In the example above the visible command is 'cell running...'; the ';' is just there to suppress output. The visible command can even be a custom function but it's not doing the cell execution.

The console stop button and keyboard interrupts even work.

Tracebacks all work as they should. The trace backs could have the right line numbers by adding in the right number of blank lines before execution.

Cons

1. Users could try to run the cell again by running the visible command, which would do nothing
2. This is an undocumented hack

Option 2: Make a runcell function

This would be a function like the runfile function in sitecustomize.py.

One challenge with this approach is how to send the cell contents to the function. This was where I was previously using temporary files, but I could just use the python file the cell is in

The call signature would be something like this:

runcell('/some/long/path/to/the/file.py', lines=(10,17))

This would require the file to be saved when the cell is executed. For me, this would be a feature, because I'm often just running cells and therefore rarely saving. However I can see some users might not appreciate the change.

This would also add all those extra sitecustomize tracebacks to errors in the cell execution.

Cons

1. More difficult to implement
2. Need to save file to runcell
3. Sitecustomize tracebacks

[jitseniesen]

My first impression is that option 1 is rather confusing to users with some magic going on behind the scene. I am especially thinking about when I use Spyder to teach Python and programming. Maybe we should submit a PR to qtconsole so that the message "Running cell" is displayed in a different style, to make it clear that it is not a normal Python command?

Regarding option 2: I agree that saving the file is a change in behaviour which not everybody will appreciate, but maybe this is not necessary. The console currently communicates with the variable explorer, so in principle it should also be possible for the console to retrieve the text from the editor. This would definitely be more difficult to implement.

The sitecustomize tracebacks are annoying, but for me not so much of an issue since runfile suffers from the same. We can later think about ways to ameliorate this issue. Pytest has options for modifying traceback printing so it should be possible.

@bcolsen You have thought a lot about it, but at least in what you write, you constrain yourself to what you think is possible. Do you have an ideal UI in mind? Maybe we should start from there and then think about how to achieve that.

[ccordoba12]

There's no need to save the contents of a file to run a cell on it because we can simply copy them. That's what we're doing right now to run a cell in our consoles. The problem with using a function is that it'll get registered in the history and so users could run it again, even when the limits of the cell have changed.

About tracebacks, we're using the Plain mode of them in our consoles because it's easier to parse to detect file path and make them clickable. But we could use the regular (enriched) IPython mode and fix the parsing issue with them.

[bcolsen]

@jitseniesen

My first impression is that option 1 is rather confusing to users with some magic going on behind the scene.

I agree. I think option two is better in that respect

@ccordoba12

There's no need to save the contents of a file to run a cell on it because we can simply copy them

I don't know a way to do this without having the code visible to the user. Does sitecustomize have way of pulling text or names from the editor namespace?

I might be able to send the code through user_expressions
https://ipython.org/ipython-doc/3/development/messaging.html

@jitseniesen

you constrain yourself to what you think is possible. Do you have an ideal UI in mind?

I don't want to over promise and under deliver, but I totally agree that I need help on this one :-)

My ideal interface is close to Option 2. There should be a function in the console that can run a cell from the file currently in focus in the editor. Ideally:

runcell('name of your cell')

This would be super simple from a user perspective. You can use a hot key, rerun the code by using up enter or just type it in. I wouldn't even mind that in my history.

From the programming side. I need to know the currently open file and then find the cell by name. There would also have to be a warning or an error if they have 2 cells with the same name.

Currently the spyder is performing this in the reverse direction, where the console just gets python code from the editor and runs it.

[bcolsen]

Option 3: Custom Messengers

I did my research on how to implement my "ideal interface" i.e. a function in the shell like this:

runcell('name of your cell')

To implement this I would need Spyder to reply back to the IPython kernel upon request with the contents of the cell block in the editor. According to my reading of:

https://jupyter-client.readthedocs.io/en/latest/messaging.html#messaging

The only existing socket that would do this is the stdin, but that would be quite the hack to use that socket for this. That leaves the option of writing our own custom messenger to talk to IPython

If we use a custom messenger we can send any JSON serialization objects back and forth to the kernel.

Are there any other uses for this in for a custom messenger in Spyder? (ie the ipython kernel needs to ask spider for information). Is it worth making a separate object to deal with this?

I guess the runfile command could save the file in the arguments or default to the current open file if it is run without arguments.

They use custom messages with Jupyter widgets in the notebook:
http://jupyter-notebook.readthedocs.io/en/stable/comms.html

[ccordoba12]

Comms are certainly more elegant than using silent_execute and we're probably going to need them to develop our planned improvements for debugging.

However, there's one important problem: qtconsole doesn't support Comms right now (only the notebook and JLab), so we'd need to solve that first.

[ccordoba12]

runcell('name of your cell')

How is this going to work if users don't name their cells?

[jitseniesen]

Here is another idea which is not perfect but might be not too hard to implement at the moment. What if, when the user wants to run a cell, we send a silent command to the IPython console which sets some private variable to the contents of the cell, and we then run the runcell command visibly, which uses that variable to retrieve the contents of the cell.

[bcolsen]

@ccordoba12
I think that Comms are definitely the right(least hacky) way to do it.

However, there's one important problem: qtconsole doesn't support Comms right now (only the notebook and JLab), so we'd need to solve that first.

I thought that might be the case. I don't think I can do that. It would be great to have an API through qtconsole to access to the IPython Comms

How is this going to work if users don't name their cells?

The function could actually just run the cell that the editor cursor is on, then the cell name wouldn't matter. Just run:

runcell()

Would work just like ctrl-enter does now in the editor. With comms we could even move the cursor to the next cell in the editor (like shift-enter) from the console with runcell('next') or a new function

@jitseniesen

Here is another idea which is not perfect but might be not too hard to implement at the moment. What if, when the user wants to run a cell, we send a silent command to the IPython console which sets some private variable to the contents of the cell, and we then run the runcell command visibly, which uses that variable to retrieve the contents of the cell.

This is a good idea. I originally dismissed adding a variable to the user namespace without them knowing, because I didn't want to break user code. However, I guess we could use a spyder. namespace or maybe IPython already has a reserved namespace for something like that. The function could look like:

runcell(spyder.cell)

It wouldn't update with the file in anyway but this wouldn't be much different than the current situation in that respect. I can do this :-)