Spyder is slow to start due to big history file

[impact27]

When starting, spyder reads a 10Mo text file in ~/.spyder-py3 called history.py.
That takes ~15s, which is a painfully noticeable start-up time.
Is there an option to limit how far back I want to load the history?

The easy work-around is for me to edit manually this file, but if an user is not aware of this, he might get annoyed by the start-up time.
I can go from 22s to 6s start-up time, which is not fast but way less annoying.

Also closing spyder takes almost 9s. I couldn't figure out why. If I have to restart spyder, usually I just kill it to avoid having to wait.

Python 3.6.5 64bits, Qt 5.10.1, PyQt5 5.10.1 on Darwin
OSX 10.13.4

[impact27]

For closing, I can go down to 2s by not having any ipython console opened, and down to <1s by changing self.process.waitForFinished(1000) to self.process.waitForFinished(100) in the close function of AsyncClient in plugin_client.py.

[impact27]

Upon further investigation, I would say that when a new client is added to ipythonconsole, a new QtWebEngineProcess is added but not closed when the client is removed.
If I open and then close 3 iPython tabs:

First I have 1 main Python process, 2 other python process and 2 QtWebEngineProcess.
If I close spyder now it will be fast.

Then I open 3 tabs. I get 3 extra python process, and 3 extra QtWebEngineProcess. Spyder will need to wait for some kind of timeout before closing (5-15 s usually).

The three tabs are closed. The 3 extra python process disappeared, but not the QtWebEngineProcess. Spyder is still slow to close.

[ccordoba12]

Python 3.6.5 64bits, Qt 5.10.1, PyQt5 5.10.1 on Darwin

Please downgrade to Qt 5.9 and try again. There are multiple reports that PyQt apps don't work well with this version.

[ccordoba12]

When starting, spyder reads a 10Mo text file in ~/.spyder-py3 called history.py.
That takes ~15s, which is a painfully noticeable start-up time.
Is there an option to limit how far back I want to load the history?

We have plans to improve this for Spyder 4, but I'm unsure what to do about Spyder 3. There's a config option to limit the number of lines of history.py, so I guess you're evaluating very long lines. Is that the case?

[impact27]

I can reproduce with: Python 3.6.5 64bits, Qt 5.9.1, PyQt5 5.9 on Darwin

[impact27]

My problem is that until I spent time looking into that, I didn't know why spyder was so slow to start. So this is not a good experience for an everyday user. I would probably try to show a message like: "Spyder is slow to start, here what you can do"?

My history is 225k lines, but the setting says 100 max. My understanding of that is:

When spyder is starting, mainwindow.py run_spyder is called. Somewhere in main.setup(), shell.py's load_history is called. This history is loaded from '/Users/quentinpeter/.spyder-py3/history_internal.py'. This file is empty, so the length is 0. Then the limit is applied, but 0 < 100 so it keeps on going.
Later, main.post_visible_setup() is called. When loading ipython, history.py add_history is called. This loads
'/Users/quentinpeter/.spyder-py3/history.py', which is 200k lines long. The length is never compared to the CONF.get('historylog', 'max_entries') settings, so this file gets to grow as much as it wants.

So either '/Users/quentinpeter/.spyder-py3/history_internal.py' and '/Users/quentinpeter/.spyder-py3/history.py' should be the same thing, or CONF.get('historylog', 'max_entries') should be applied on '/Users/quentinpeter/.spyder-py3/history.py'. I don't know which one is best.

[impact27]

This seems to be two unrelated issues. Should I open a new issue?

[ccordoba12]

My history is 225k lines, but the setting says 100 max. My understanding of that is:

Your understanding seems to be correct. Could you open a PR for this? It'd be greatly appreciated!

Should I open a new issue?

Yes, please open a new issue about the WebEngineProcess'es.

[impact27]

What does the history_internal.py file stores? Is the bug that history_internal.py should be the same as history.py or should I just apply the max_entries to history.py as well?

[impact27]

I opened #7091

[jitseniesen]

What does the history_internal.py file stores? Is the bug that history_internal.py should be the same as history.py or should I just apply the max_entries to history.py as well?

It's not my code, but I'm pretty sure that history_internal.py stores the history of the internal console (called InternalShell in code). I think the max_entries setting shold be appled to history.py as well.

[ccordoba12]

Yes, @jitseniesen is right.

[impact27]

The history is duplicated. Why is the history not just loaded from the ipython "history.sqlite"? Is there anyone not using the ipython console? Also this is a bit confusing because the "history log" is not related to what you can actually access by pressing "up" in the ipython console.

[ccordoba12]

Also this is a bit confusing because the "history log" is not related to what you can actually access by pressing "up" in the ipython console.

What do you mean buy this? We save the contents of every command introduced in our consoles in the history log.

[impact27]

If I delete the ~/.spyder-py3/history.py file, I can still access the history from the ipython console, but not from the history widget. Also the ipython history saves blocks, not lines. The history.py file doesn't know anything about blocks.

When you press up, ipython gets data from ~/.ipython/profile_default/history.sqlite. When a command is executed, it is saved in ~/.ipython/profile_default/history.sqlite and in ~/.spyder-py3/history.py. history.py is useless for ipython and might not be consistent with history.sqlite.

So while we do save every command in history.py, they are duplicated in history.sqlite. If I edit history.py to say "This is a test", I will only see "This is a test" in the history log, but pressing up in ipython will give me the previous commands I used.

My questions are therefore:

• Is history.py only used for the "History log" widget?
• Should the "History log" widget instead display the content of history.sqlite?
• Is anyone using a regular python console, as opposed to IPython?
• What is the best way to access ipython history? I don't want to read the file directly as the API might change.

[ccordoba12]

My answers:

Is history.py only used for the "History log" widget?

Yes.

Should the "History log" widget instead display the content of history.sqlite?

Yeah, that would be better. We could get rid of history.py and simply access that database to populate the History Log.

However, we haven't had time to do it but it'd be a very nice thing to do for Spyder 4.

Is anyone using a regular python console, as opposed to IPython?

No, the Python console was removed in 3.2

What is the best way to access ipython history?

No idea. @takluyver, could you give us a hand here? I think you are the one who knows best how to access history.sqlite.

[takluyver]

For read-only access, you want a HistoryAccessor object. The HistoryManager subclass has write functionality too.

This was one of the first bits I worked on in IPython :-)

[ccordoba12]

Thanks a lot Thomas!