Implement lossless round-tripping of negative number literals

[cutsoy]

When writing -4.0f32 in Rust, it is usually parsed as two different tokens (a minus operator and a float literal).

But a procedural macro can also generate new tokens, including negative float literals:

#[proc_macro]
fn example_verbose(input: TokenStream) -> TokenStream {
    let literal = Literal::f32_suffixed(-42.0);
    quote! { #literal }
}

or even shorter

#[proc_macro]
fn example(input: TokenStream) -> TokenStream {
    let literal = -42.0f32;
    quote! { #literal }
}

Unfortunately, these currently cause Rust Analyzer to crash because it doesn't accept literals that start with a minus sign:

thread '<unnamed>' panicked at 'Fail to convert given literal Literal {
    text: "-42.0f32",
    id: TokenId(
        4294967295,
    ),
}', crates/mbe/src/subtree_source.rs:161:28

I think RA is right here, because the Rust Reference states that:

Note that the Rust syntax considers -1i8 as an application of the unary minus operator to an integer literal 1i8, rather than a single integer literal.

And the grammar for number literals does explicitly exclude the minus sign.

Unfortunately, the authors of proc_macro also acknowledge this:

Literals created from negative numbers may not survive round-trips through
TokenStream or strings and may be broken into two tokens (- and positive literal).

I submitted a similar issue to rust-analyzer here, but I don't think it's easy to fix with the current architecture (their "reparsing" logic only looks at the first token its lexer encounters, which is the minus).

It might be easier to fix this in the ToTokens implementation for number primitives here.

Would you be open to accept a PR that does that?

Currently, the work-around I am using is this newtype struct to wrap number literals and provide a custom ToTokens implementation:

use num_traits::Signed;
use proc_macro2::{Punct, Spacing, TokenStream};
use quote::ToTokens;

pub struct Number<T>(T);

impl<T> ToTokens for Number<T>
where
    T: Signed + ToTokens,
{
    fn to_tokens(&self, tokens: &mut TokenStream) {
        if self.0.is_negative() {
            Punct::new('-', Spacing::Alone).to_tokens(tokens);
            self.0.abs().to_tokens(tokens);
        } else {
            self.0.to_tokens(tokens);
        }
    }
}

fn usage() -> TokenStream {
    let example = Number(-2.0f32);
    quote! { #example }
}

The num-traits crate is not necessary if we implement this in the macros here. We could just call is_sign_negative(). Apart from that, I think the code could be trivially adapted.

Any thoughts?

[cutsoy]

I should have noticed that there's already an old pull request on this that links to an equally old issue in Rust itself.

I think the relevant bits are:

dtolnay reviewed on Mar 28, 2018

I would prefer not to work around this so that we are motivated to fix rust-lang/rust#48889 sooner rather than later.

dtolnay commented on Mar 12, 2018

👍 for what I think Nick is saying which is Literal::i32(-1) should be a perfectly legitimate single token.

I am assuming this applies to the same "nonacceptance" bug in RA as well?

[dtolnay]

Yeah, as you found, I don't plan to change this. This needs to be fixed in rust-analyzer to match rustc. The libproc_macro documentation is explicit that negatives are a legal input to the Literal constructors even if the resulting token it produces may get expanded into a pair of tokens by a later compilation stage.