Conversion to unsigned integers shall not produce range checks

[Araq]

This is a followup of:

nim-lang/Nim#12554

There are two conflicting designs at play here:

1. Nim's distinction between T(x) and cast[T](x) implies that runtime checks for T(x) are always anticipated and desired. This is what the spec for version 1 says and no special casing for unsigned is mentioned.

2. Unsigned numbers are heavily used in low level contexts and diverging too much from C's behaviour is confusing and dangerous. Arithmetic operations on unsigned wrap around because of this requirement.

In the past Nim followed (2) but starting with version 1 Nim followed its own spec. This RFC proposes to change the spec to reflect what was true in previous Nim versions.

Note that either (1) or (2) is fine from a design standpoint IMO but the legacy code relies on (2) (including Nim's own library) and now the compiler inserts time ticking bombs in the form of secret runtime checks that can result in exceptions being raised without any warnings. This is bad enough to revert it to the old behavior and change the spec so that it matches Nim's historical behavior.

[zah]

A narrowing conversion could always have two possible interpretations:

• "I know that the converted value is in range and I just want to make the compiler happy"
  (when the expectation here is violated, this can be considered a defect)

• "I want to truncate the higher bits of this value"

It's impossible for the language to be always right when the programmer hasn't stated one of these intentions precisely. I would argue that all types are affected by this and the language would be better if all types are handled in a consistent way. Introducing a specific operation such as truncate that encodes one of these intentions is the best solution IMO. Why should I learn that some types are checked and others aren't? How will I write generic code that works with any type?

[Clyybber]

IMO uint shouldn't be checked at all and we should introduce some sort of safe uint that has all normal checks enabled (also overflow checks). Could be a bit hard to implement since we have to do something that C can't afaik.

[krux02]

@zah that specific operation that you call truncate already exists. It is called cast.

[zah]

cast in considered an unsafe operation. We should always try to avoid it when a safer construct is available. In practical terms, If I want to review a particular codebase for unsafe code, I could search for all usages of cast and other similar constructs. My proposal is to have something else that's considered slightly more safe.

[mratsim]

The fact that cast allows casting between types of different sizes for integers is also questionable.
This was also not allowed for a very long time in the VM and semfold I expect.

It's still not allowed for objects for example:

type Foo64 = object
  data: uint64

type Foo8 = object
  data: uint8

let x = Foo64(data: 300'u64)
echo cast[Foo8](x) # Error: expression cannot be cast to Foo8=object

It probably requires another RFC but I'd rather have cast be only about reinterpreting the bit pattern of the type.

[mratsim]

A note on the title

conversion from unsigned to unsigned integers and raw bytes shall not produce range checks.

Signed -> unsigned conversion should still be checked for negative values.

[Araq]

conversion from unsigned to unsigned integers and raw bytes shall not produce range checks.

byte is an alias for uint8 in Nim.

Signed -> unsigned conversion should still be checked for negative values.

Uh but in the past byte(-1) did compile. Now you are adding more special rules, I don't like it.

[mratsim]

Signed -> unsigned conversion should still be checked for negative values.

Uh but in the past byte(-1) did compile. Now you are adding more special rules, I don't like it.

It did not, it throws Error: -1 can't be converted to byte for semfold and a runtime error for runtime values.

[cheatfate]

@mratsim i dont think signed to unsigned conversion must be checked. it can be converted without any problems even with sign bit. So:

uint8(-1) = 0xFF
uint16(-1) = 0xFFFF
uint32(-1) = 0xFFFF_FFFF
uint64(-1) = 0xFFFF_FFFF_FFFF_FFFF

[zah]

@cheatfate, I can see the habits of the C programmer here and the desire to cling to the familiar. Why should we write uint8(-1) when we have uint8.high?

[cheatfate]

@zah this is just an example of possible values which must be accepted and work properly.

[krux02]

Here this is the situation we were in (tested on 0.17):

echo high(uint64)   # invalid argument for high
echo uint64(-1)     # OK

echo high(uint32)   # OK
echo uint32(-1)     # conversion from int literal(-1) to uint32 is invalid

Range checks sometimes, sometimes not. No consistent way to get the high value of all integer types. All that has been changed is, now we are not half ass anymore. You can rely on the behavior. Also, high now consistently works on all types.

This is the same code in version 1.0.0

echo high(uint64)   # OK
echo uint64(-1)     # -1 can't be converted to uint64

echo high(uint32)   # OK
echo uint32(-1)     # -1 can't be converted to uint64

And about people who are afraid of the cast. It is now a well specified and safe operation. Casting between differently sized integer types is a conversion without range check. It does sign extension if needed, so that a -1'i8 (0xFF) can become a -1'i16 (0xFFFF). Casting between int and float does reinterpret the bit pattern and requires the types to be the exact same size. This behavior is specified and tested on both C backand as well as the VM.

I still hold my position. We don't need to change anything in the compiler or the spec. But we do need to better communicate this behavior for the users. Something like All interger type conversions including conversions to and from unsigend integers are range checked, if you wish for conversions that may not raise an an error, even when out of range, use the `cast` operator. would be sufficient I think.

[dryajov]

@mratsim i dont think signed to unsigned conversion must be checked. it can be converted without any problems even with sign bit. So:

uint8(-1) = 0xFF
uint16(-1) = 0xFFFF
uint32(-1) = 0xFFFF_FFFF
uint64(-1) = 0xFFFF_FFFF_FFFF_FFFF

Doesn't signed to unsigned and specifically the code above assume 2s complement hardware (no matter how prevalent it is this days)? Does Nim asume 2s complement hardware everywhere or is it up to the backend to do the "right thing"? Wouldn't this require this assumption to be explicit at the language spec level, i.e. don't repeat other's errors.

[dryajov]

IMO, the safer thing to do is to disallow implicit signed to unsigned conversion.

[Araq]

Nobody talks about implicit conversions here.

[Araq]

Does Nim asume 2s complement hardware everywhere or is it up to the backend to do the "right thing"? Wouldn't this require this assumption to be explicit at the language spec level, i.e. don't repeat other's errors.

2s complement hardware is pretty safe to bet on and a compiler can always emulate it on alien hardware.

[dryajov]

Nobody talks about implicit conversions here.

Ok I stand corrected. But sign conversion more vulnerable than unsigned overflow, for example:

var a: string = "boom!"
var b: string = "     "

copyMem(addr b[0], addr a[0], uint(-1)) # boom!!

The above contrived example builds fine on 0.19.6. It basically implies that uint can be used in place of int anywhere, including Naturals?