ANSI Filter for ESC 7 and possibly other codes

[denisbondar]

Hi, everybody!

I have encountered a ZyXEL MGS3520 device that transmits ANSI ESC 7 (\e7) bytes 0x1b 0x37 at the end of every response.
Each CLI prompt ends in \e7, each echo also ends with at least one \e7, but it can be more if the command takes a long time to execute.

I noticed that the filter is designed for a small part of ESC-codes, so I suggest to either supplement it with 0x37 (i.e. just character 7) or to modify it so that it correctly clears all unnecessary ANSI-characters.

I saw closed PR #265 that had the ESC E symbol added to it. I suggest discussing adding either one more character 7 or adding all the necessary ESC codes at once. Let's discuss which ESC codes should be removed from buf.

Example of communication with ZyXEL MGS3520:

> User name:
< admin
> admin
< \n
> Password:
< secret
> ******
< \n
> Copyright (c) 1994 - 2021 ZyXEL Communications Corp.\nZyxelMGS3520-28# \e7
< cable-diagnostics 1,2,3
> cable-diagnostics 1,2,3 \e7\e7\e7\e7\e7\e7
< \n
...

[carlmontanari]

👋 hey @denisbondar thanks for raising this! yeah we should handle this better probably :)

does the pattern in scrapligo here look better/like it would handle what ya need (and hopefully more)? if "yes" -- would you be up for a PR to make that happen here?

thanks a bunch!

carl

[denisbondar]

@carlmontanari

In my opinion, the Channel._strip_ansi() function should not only delete ANSI characters, but also perform one important action: ESC[2K - erase the entire line.

Although, I haven't figured out how to implement multipage data leafing in scrapli for devices where there is no option to disable paginated output, but in this case with leafing the ESC[2K command deletes the line with a leafing prompt like:

CTRL+C ESC q Quit SPACE n Next Page ENTER Next Entry a All

By the way.
I think the topic with flipping is relevant for scrapli, as not everywhere there is an option to disable paginated output. I have not yet figured out how to realize it with available means.

As for the regular expression example you gave from the scrapligo project, it is difficult to understand at a quick glance, but if it is time-tested and works as it should, you can use it as a basis.

What do you think about the execution of the ESC[2K command?

[carlmontanari]

I think the topic with flipping is relevant for scrapli, as not everywhere there is an option to disable paginated output. I have not yet figured out how to realize it with available means.

I think you can probably handle this yourself but you're not going to (w/out patching methods at runtime which yay python we can do it :P) be able to do so and use the send_X methods since they'll be expecting "normal" behavior. To do it you'd probably just need a read loop running (this is actually more like scrapligo since its evolved from py version quite a bit) in the background and whenever you see "PRESS ENTER" or whatever the "more" prompt is you would obviously just do that + whatever handling to remove that prompt you want.

What do you think about the execution of the ESC[2K command?

I'd be open to seeing an example of this and testing it and seeing whats up, but as I understand it now (which maybe I dont understand it well!) id say there is a very low chance that would be something id merge.

[denisbondar]

I only found out about scrapli a few days ago. I tried to fix my old code due to the telnetlib abandonment announced in PEP 594 and none of the replacement options recommended in that PEP appealed to me. So I started looking for other options and found scrapli. In these few days I have not had much time to deal with anything, although I am actively doing it now.

I saw that you could override the sendCommand() method to change this behavior, but I also expected to see methods like read() in the base class that could also be overridden to implement pagination logic in them. This could work at the level of a custom driver class for absolutely any commands. But I haven't found yet how to implement it using existing features.

[carlmontanari]

here would be the kinda "mid level" read, and then you can also read directly from the transport here for example if you wanted

[denisbondar]

this is actually more like scrapligo since its evolved from py version quite a bit

Does this mean that the Python project has lost relevance and can be abandoned?

[carlmontanari]

Does this mean that the Python project has lost relevance and can be abandoned?

nah, both can exist. go is obviously just different than python in lots of ways so it started changing over time a bit.

[denisbondar]

Tested the regular expression from the scrapligo project for various binary strings from my telnet dumps. Everything worked correctly, including \e7 and \eE.

I forked the project and ran the tests first of all. So far I see a lot of failed tests. Tomorrow morning I will sort it out and create a PR.

[denisbondar]

Tried to figure out why some tests failed yesterday.
These are unit tests that depend on the presence of an ssh server on the host.

• test_socket_open_close_isalive
• test_socket_bool
• The other methods that use socket_transport cannot be executed correctly either, because they depend on the environment.

It seems to be incorrect in terms of the idea of unit testing.

[denisbondar]

@carlmontanari I created PR #302 based on this discussion.