invoke a pager to display results

[c-cube]

Just tried the tool, it's great (as is the blostpost btw). I'm usually using grep -r foo | less -R for searching (with an alias to get colored output), since there might be several pages of results; rg coloring is nice but is lost if piped into a pager. Would it make sense to invoke a pager automatically if the tool detects that its output is not piped?

[BurntSushi]

You might consider passing the -p flag to rg, which should cause it to retain its formatting.

I'm a little skeptical about actually calling a pager from within rg. That kind of seems like it's doing too much. (I will note though that git grep seems to do it, and it seems smart. It will invoke the pager automatically if there are lots of results.)

[c-cube]

Indeed, I will probably alias rg to rg -p. Still, automatic paging is quite nice :-)

[ayoshi]

BTW, there is a nice way to invoke pager from rust console CLI: https://gitlab.com/imp/pager-rs it does pager instantiation in a quite clever way

[BurntSushi]

@ayoshi That is pretty slick. It won't work on Windows though. Doing it only on *nix is an option if we decide to proceed.

[bluss]

git does it the same way I believe, makes sure that the pager is the parent process.

[JohnVillalovos]

+1 from me on having a pager. I like how "git grep" will page the results. Something like that would be nice.

I can do:

$ rg -p foo | less -r

Mostly works. But it can leave stray colors. As in my prompt becomes green.

[BurntSushi]

@JohnVillalovos If you use less -R, does that help with the stray colors?

[JohnVillalovos]

@BurntSushi Thanks, that does work :)

More typing than I would like. /me lazy ;) But it works!

[BurntSushi]

@JohnVillalovos Yeah, I have alias less="less -R" in my .bashrc. :-)

[bluss]

um I'm trying to make a shell script for that (better than alias, available in more contexts)

#!/bin/sh

rg -p "$@" | less -R

(chmod +x that thing and call it as rg test to try). This just shows a blank terminal and seems to eat memory. Warning: If you reproduce, be sure to Ctrl-C it quick. Am I doing something wrong or what's happening?

[bluss]

Oh.. oh.. 😫

any brown paper bags available?

Hint: I made it spawn itself recursively. sigh.

[kugel-]

I also support this feature, missing it badly from ag and git grep.

[windware-ono]

@bluss Perhaps as a function instead?
Got this in .zshrc, works well.

function rg()
{
   /usr/local/bin/rg -p "$@" | more -R
}

[bluss]

I use this in my bin directory, works well

#!/bin/sh

~/.cargo/bin/rg -p "$@" | less -RFX

[BurntSushi]

I like @bluss's approach here. I'm not strongly opposed to adding this into ripgrep itself, but it seems better to at least try and get by without doing so.

[ssokolow]

I just noticed the need for this and augmented my config-file wrapper to also pipe the output through less -RFX if it sees -p in the options.

https://github.com/ssokolow/profile/blob/master/home/.zshrc.d/rg

(Feel free to change that. I just needed something quick and didn't think I'd ever use -p for anything other than making the colors work when piped through less.)

I still think the need for the wrapper is a bit of a wart though, given that, in the abstract, such tools are competing on how convenient they can be and my wrapper feels fundamentally no different than "Oh? You want to ignore stuff in your .gitignore? Well, we offer a command-line flag to ignore things. You're free to write a wrapper." (Reading --type-add from a config file) or "Oh? You want colored output? Well, the colordiff people showed it can be done in a wrapper." (pager support)

[jason-s]

less doesn't seem to ANSI terminal control sequences on Windows. Would be really nice to have a built-in pager.

I guess what I don't understand behind the philosophy of trying to use external utilities for paging, is that ripgrep already takes care of some presentation issues through the use of ANSI coloring.

[BurntSushi]

Would be really nice to have a built-in pager.

This is a completely unreasonable request, so I'm just going to nip this one right in the bud: so long as I'm the maintainer, ripgrep will never ever never get a built-in pager.

[ssokolow]

less doesn't seem to ANSI terminal control sequences on Windows.

From what I remember, cmd.exe doesn't use ANSI sequences. Last time I looked at making one of my utilities do portable colorization, it seemed that cmd.exe expected you to do text colorization through a side-channel more similar to the "set pen attributes" APIs you find in things like Qt's QPainter.

(Which would mean that less would have to be color-aware enough to parse the input and convert it to an interleaved sequence of text blocks and "set color" API calls.)

[jason-s]

From what I remember, cmd.exe doesn't use ANSI sequences.

Well, rg itself uses colors just fine, so either it's using some other mechanism for Windows, or Windows does support ANSI terminal sequences.

I've tried updating less to the latest version I can find for Windows (v381 from UnxUpdates.zip in http://unxutils.sourceforge.net/ which does support -R and -r; my previous version only supported -r) but it still turns the ANSI sequences into visible gobbledygook.

I understand the need to keep things simple but it really impairs the usability of rg on Windows when there is a lot of output.

[BurntSushi]

I understand the need to keep things simple

Simplicity is only part of it. The other part of it is that you're asking me or someone else to build an entire cross platform pager into ripgrep, and then you're asking me to maintain that. It's an unreasonable request not only because it completely violates the principle of responsibility, but that's unreasonable because it would be an ungodly amount of really annoying work. On top of all of that, I don't even know that such a thing is possible in Windows.

---

On coloring... The standard way to color things in Windows is through the various console APIs. It's completely unlike ANSI escape sequences. Instead of putting special codes into the output text, you instead need to: 1) stop writing output, 2) ask the console to change the color to X, 3) write output again and 4) ask the console to change the color back to the setting before X.

With that said, Windows 10 has some support for ANSI escape sequences, but it needs to be explicitly enabled. Alternatively, the various MSYS2 terminals for cygwin support ANSI escape sequences.

I understand the need to keep things simple but it really impairs the usability of rg on Windows when there is a lot of output.

I've spent a ton of time making ripgrep work well on Windows, but I have to draw the line somewhere. I don't know exactly where it is, but it certainly doesn't include a built-in pager.

[jason-s]

Fair enough, that explains some of the complexity.

Maybe by the time I get a new laptop later this year and it's Windows 10 then this becomes a non-issue for me. I certainly don't defend Microsoft's decisions for brain-dead implementations of things that otherwise work very well in Unix-land.

Looks like

DOSKEY rgl=rg --color never $* ^| less

works for me in my crippled Windows 7 world....

[BurntSushi]

Maybe by the time I get a new laptop later this year and it's Windows 10 then this becomes a non-issue for me.

FWIW, I don't think ripgrep quite works yet with ANSI support on Windows 10. I think it really needs a champion to make it happen who knows more about how to make it work. You can force ripgrep to emit ANSI escapes with --color ansi---even on Windows---but I think you actually need to enable some config knob to make ANSI support work on Windows.

DOSKEY rgl=rg --color never $* ^| less

Oh, I thought you wanted paging with coloring. Disabling coloring is certainly one way of getting rid of the escapes. Although, I do find it interesting that you're seeing ANSI escapes at all in a Windows console. That shouldn't happen.

I certainly don't defend Microsoft's decisions for brain-dead implementations of things that otherwise work very well in Unix-land.

I won't lie, sometimes I think this way too. But I try hard not to, and try extra hard not to actually say it because it tends to just ruffle everyone's feathers. And Unix has its fair share of strange things too. :-)

[jason-s]

Well, I'd like paging with coloring if I can get it, but I need paging and if the way to do that is disabling coloring, then it works well enough for me.

Although, I do find it interesting that you're seeing ANSI escapes at all in a Windows console. That shouldn't happen.

No idea. I just use rg without options, and I get the coloring, but if I pipe it through less then I see all the ANSI-looking gobbledygook like... oh wait a minute, it disappears when I pipe it through less. I only get it when I use rg -p otherstuff | less. Sigh. Never mind.

[ssokolow]

I certainly don't defend Microsoft's decisions for brain-dead implementations of things that otherwise work very well in Unix-land.

It's not always Microsoft's fault. For example, they chose \ as a path separator when they updated DOS to support directories because they were already using CP/M-style / to denote command-line switches and I remember reading that the decision to follow CP/M on that front ultimately lay with IBM.

...and, the more layers of backward-compatibility abstraction you drop below, the more POSIX-compatible modern Windows tends to get. (I assume, purely out of practicality.)

For example, as far as I've been able to determine, DOS and Windows have accepted / as a path separator in APIs where it's not ambiguous since DOS 2.0 introduced directories. It's just the command-line parsing which requires paths to use \ to avoid ambiguity.

...and kernels of the Windows NT lineage don't use drive letters internally... they're just an artifact of the Win32 API subsystem and it's possible to specify paths using an internal, singly-rooted syntax. (Though, admittedly, that is the one notable exception to the / compatibility. You have to use \ in and after the escape prefix to bypass the Win32 path rules.)

(the NT kernel was written so that Win32, OS/2-compatibie, and POSIX-compliant APIs could be offered as pluggable subsystems and NTFS actually has a POSIX "personality" where, among other things, the only disallowed filename characters are \0 and /. WSL is just the latest in a lineage of POSIX subsystem modules for the NT kernel.)

[labianchin]

Here how I solved this, if others are looking:
I used to use ag before and defined an alias in bashrc/zshrc to "invoke ag with less":

alias agl="ag --pager='less -XFR'"

For rg I am now using:

rgl() {
  rg -p "$@" | less -XFR
}

[scorphus]

Sure! I'll take this one! 🥨🍻

[scorphus]

The crate pager is really nice and fits well. I'm gonna use it. Any objections, @BurntSushi? BTW, any further tips are more than appreciated.

[BurntSushi]

@scorphus I am skeptical. That crate hasn't been updated in 2 years. Here is what I would do:

1. Research how git grep spawns a pager. This is important because git grep is a widely used and popular tool. It is very likely that they've probably iterated towards an optimal solution. So let's benefit from their experience and practice by seeing if we can emulate their logic.
2. Scrutinize the pager crate to see if it matches git grep's behavior. Also, carefully review pager's internals to make sure it looks okay. I just did a brief scan and I can already see that its atty detection won't work on Windows, so that's probably a deal-breaker unless its API allows us to work around that.

The implementation of pager does contain unsafe, and normally I'd see that as a good reason to use a crate for it, because it becomes a rallying point for other people to join together and audit and fix bugs that arise. But if the crate isn't being maintained, then this stops being an advantage.

[wonderfulspam]

The pager crate has recently been updated but it still doesn't seem like it quite cuts the mustard, IMO.

Something that I would consider to be a constructive contribution here would be a deep dive on how other widely deployed cross platform software implements automatic paging functionality.

This is a very shallow (and opinionated) analysis rather than a deep dive, but perhaps it may be of use:

The git grep implementation is a little gnarly with the logic for deciding whether and how to spawn a pager residing at the bottom of a 300+ line function. I would not be surprised if there are more subtle bugs like this one lurking under the surface.

I would consider taking a look at bat's implementation. bat may not be as widely used as git grep but it does happen to be cross-platform, is written in Rust and, anecdotally, has provided a flawless paging experience for me in the past two years or so that it's been my daily driver. At the very least, @sharkdp may be able to provide some valuable pointers on the pitfalls and gotchas. See, for example, sharkdp/bat#1402, or sharkdp/bat#887 (comment) in which the maintainer of less even piped in to confirm that the bug in question existed in less, not bat.

Interestingly, bat is currently exploring adding a built-in pager which would, presumably, also be available as a library that might be of interest.

[scorphus]

Yeah! I've been considering bat's implementation as the one to follow. I've also been using it flawlessly for quite some time.

TBH, I didn't look too much into anything else, particularly git's one, although it was suggested — that code frightened me, to say the least. I didn't entirely believe it could really help ☺️

Thank you for the the links, @wonderfulspam! 🙌 Espcially the last one, I wasn't aware of the idea around a built-in parser for bat, that might be something to consider!

[sharkdp]

I would consider taking a look at bat's implementation. bat may not be as widely used as git grep but it does happen to be cross-platform, is written in Rust and, anecdotally, has provided a flawless paging experience for me in the past two years or so that it's been my daily driver. At the very least, @sharkdp may be able to provide some valuable pointers on the pitfalls and gotchas.

I think you already pointed out the most important things. I'll try to list a few points that come to my mind right now:

• Adding paging support to bat is something that turned out to be MUCH more painful than I imagined initially (take a look at the amount of issues linked in Discussion: a builtin pager for 'bat' sharkdp/bat#1053). If you take a look at our implementation, you can probably tell that it's not completely trivial to make this work consistently across platforms and less versions. The logic would be a lot simpler if ripgrep decides to only add a --pager=<cmd> option (instead of also adding PAGER handling, something like RIPGREP_PAGER, etc.).
• One of the most annoying things in bat is the interplay between the pager and bats own line wrapping and line-number output (especially if the terminal is resized). I guess this wouldn't be a problem for ripgrep.
• Another pain point is colorized output in combination with a pager. less can pipe-through ANSI codes when called with the -R option, but we had a lot of weird things happen. Especially on Windows.
• Making this an opt-in feature in ripgrep is definitely a good idea. From experience, I can tell that unknowing users will otherwise assume that the less frontend is part of the application itself. If they are not familiar with a pager, that can be confusing (how do I quit?). Also, they will report all kinds of bugs in less against your repo (e.g.: scrolling does not work).
• Even if users opt in with a --pager=<cmd> option in their config file, it might be a good idea to have a --paging=auto/never/always option (similar to what the --color=… option does). For bat, we even added a short option -P to disable the pager because it's used so often. Imagine you are fine-tuning a search pattern and run multiple ripgrep searches in sequence. You probably don't always want to quit the pager in-between.
• The explanation in https://github.com/sharkdp/bat#using-a-different-pager might be of interest (how we call less and why).

Interestingly, bat is currently exploring adding a built-in pager which would, presumably, also be available as a library that might be of interest.

If we ever make this happen, it would be available as a library. Yes. However, most of the features that we would want in bat are probably not that useful for ripgrep.

[sergeevabc]

Gosh, these stubborn developers… Guys, check out vgrep.
Basically, it’s a mix of Ripgrep and Less (with truly working colors), which we’d like to see here.

[God-damnit-all]

I would like to point out that if you're using PowerShell (regardless of platform), Out-Host -Paging seems to work just fine with ripgrep's --color always parameter. Its pager leaves much to be desired, though.

You can also set $PSStyle.OutputRendering to ANSI and then less -R should work fine, even with less-Windows.

ripgrep having its own pager would certainly be more convenient, though.

[dandavison]

delta has special support for acting as a pager for ripgrep: see the delta manual section. In addition to paging, this brings two other features:

• syntax-highlighted code in the search hits
• search hits can be formatted as terminal hyperlinks, which can be configured to open your text editor at the appropriate line (e.g. see open-in-editor)

You use it by piping rg --json output to delta. Here's an example in the delta repo using a light background and the GitHub color theme:

rg --json 'handle' | delta

EDIT Please see update below: #86 (comment)

[BurntSushi]

Wow, that is pretty slick! Nice work. @dandavison I've added a link to that from ripgrep's README: 595e784

PRs are welcome if you'd like to make wording tweaks!

[ryuheechul]

Thanks @dandavison's comment!
Now my alias has changed like below

# from this
alias rg='rg -p -i'

# to this function
rg ()
{
  env rg --json $@ | delta
}

And it works just like how bat works in terms of automatic paging!

[wbolster]

env rg --json $@ | delta

@ryuheechul, for correctness with all file names etc, you should be using this function body:

command rg --json "$@" | delta

(you could also detect whether the output goes to a tty and avoid calling delta in that case, but the exact desired behaviour is subjective.)

[ryuheechul]

@wbolster good points to think about and will be helpful as context when I encounter related issues :), thanks!

[krishnakumarg1984]

@BurntSushi I wonder if the cross-platform pager crate minus shall be of immediate help. minus is written both as a library and a binary crate, with the library specifically intended for being used as a pager integration with other applications. This is exactly the use case we have here, and seems like a perfect fit for the problem we have.

For more info, please read the minus README section on integration with applications as well as this blog post by its author detailing its design and purpose.

Hope that helps, and soon we won't hopefully need to check for & rely on the presence of an external tool delta for paged output from ripgrep.

[BurntSushi]

@krishnakumarg1984 I do not think it's a good idea to depend on external crates like that for core functionality. As soon as I ship something that is based on what minus provides, I've essentially signed up to provide that functionality forever, regardless of whether minus continues to be maintained or not. It also has a very large dependency tree. A significant fraction of ripgrep's dependency tree. There's probably some overlap, but that's not something that inspires confidence from my perspective. Instead, it just increases the risk of having to maintain something else.

[dandavison]

Quick update on delta's ripgrep support:

Delta now tries to exactly replicate ripgrep output format, so the idea is that piping ripgrep --json to delta should be a strictly additive improvement, bringing:

• paging
• syntax highlighting
• hyperlinks to open matching lines in an editor/IDE
• "n" and "N" keybindings to navigate between matches

The hyperlinks are particularly convenient if you're using an editor such as VSCode, Pycharm, or IntelliJ that install their own URL handlers for opening links.

Docs: https://dandavison.github.io/delta/grep.html

Please open an issue in the delta repo if delta is failing to emulate the visual appearance of ripgrep output that you would like.

In the image below, the line numbers are OSC8 terminal hyperlinks.

[ltrzesniewski]

@dandavison Nice work!

I couldn't find the URI format spec for the JetBrains IDEs, I hope you don't mind if I add the one you've found to #2483 (I'd like for ripgrep to support hyperlinks out of the box).

Did you by any chance find a list of the URI schemes used by the other JetBrains IDEs besides IntelliJ and PyCharm? If not, they should be rather easy to guess, but I can install them all just to make sure.

[BurntSushi]

I think while I had said my stance on this softened a while back, I'm ultimately going to pass on this. I think this is a good example of scope creep, and that ripgrep just does not benefit much from having a --pager flag when users can invoke a pager themselves (or write an alias to do it for them).

[txtsd]

I just wanted to show everyone how beautiful delta's output is! 💜

I came here fully prepared to ask for native support for paging via delta, and then I found this issue. I wanted to show everyone what beauty sparked that compulsion.

[gcflymoto]

Beautiful! What terminal are you using? Still see hyperlinks from latest delta broken in Konsole and kitty.

[txtsd]

Beautiful! What terminal are you using? Still see hyperlinks from latest delta broken in Konsole and kitty.

I use kitty! Hyperlinks worked when I tried them yesterday.

[UnsolvedCypher]

For fish users wanting to use Delta for your pager, you can add an abbreviation: abbr -a --set-cursor -- rg 'rg % | delta' . This will pipe your rg into delta but place your cursor before the pipe (where the % is) so you can type your query.