Use of CSVProvider with inferRows leads to UI delays

[cartermp]

See this comment: #6341 (comment)

We should investigate this setting with large CSVs and script files in the IDE.

@nhirschey do you have a repro that can be shared?

[nhirschey]

Sure, repro below. This isn't quite as bad as what I was seeing, but it shows the general effect.

#load @"../.paket/load/net47/FSharp.Data.fsx"

open System
open FSharp.Data

let n = 80
let header = [ for i = 1 to n do yield "A"] |> String.concat ","
let row = [ for i = 1 to n do yield "1.1"] |> String.concat ","

let [<Literal>] file = __SOURCE_DIRECTORY__ + "/test.csv"

seq { for i = 1 to 100_000 do yield row }
|> Seq.append [ header ]
|> fun lines -> IO.File.WriteAllLines(file, lines)

type TestCsv = CsvProvider<file>

// everything works fine.

TestCsv.GetSample().Rows
|> Seq.take 5
|> Seq.toList

// Now change InferRows to 0 so it uses the whole file, then try typing things.

type TestCsv2 = CsvProvider<file, InferRows = 0>

[cartermp]

Thanks! As mentioned in the previous mail, this is with the latest 3.1.1, which means it incorporates the fixes to type provider initialization that resulted in huge LOH allocations over time.

However, It think the root problem is we're still causing a re-evaluation of the file when it's obviously not different. @dsyme do you have any ideas here?

[nhirschey]

Is there any reason this would happen with .fsx and not .fs files? I'm using both interactively. Now on 16.0 (non preview 2019 I just downloaded today in an attempt to fix).

My original file using the large .csv crashes when in an .fsx. But when I put the same code in .fs, I am not getting the VS UI to hang.

[cartermp]

Script files are sort of like files drifting in space. In addition to have slightly different semantics, they aren't associated with any particular context in a project like .fs and .fsi files are. The way the tooling handles these is very different - scripts have their own workspace that manages files - so it could be that there's something we can do here in the VS tooling side of things.

[nhirschey]

For completeness, I'll add a comparison to VS 2017. In VS 2017, intellisense stops working but the UI does not freeze. I think the issue is whether the process inferring types in the csv file blocks the UI or not.

In VS 2017

1. Start with the type provider inferring columns using the default number of rows.

type TestCsv2 = CsvProvider<file>

2. Then try to get intellisense working (it will work).

TestCsv2.  // Intellisense pops up after the period.

3. Now set infer rows to use the full file.

type TestCsv2 = CsvProvider<file, InferRows = 0>

4. Then try to get intellisense working. Intellisense will not work, but the editor does not freeze.

TestCsv2.  // No intellisense, but the UI does not freeze.

In VS 2019

If you do the same thing in 2019, in step 4 the UI will freeze.

[nhirschey]

The UI hang happens with other type providers too.

Using the SAS type provider from FSharp.Toolbox.SAS, I am getting the same UI hang. It happens in the exact same way after typing the "." and waiting for the type to update, for example it hang when typing "x." in the lines below:

let x = new SasFileTypeProvider<file>()
x. /// hang while waiting for intellisense to work

However, once the UI comes back and there is one line of code referencing x (so the type is cached I guess), I no longer get hangs.

Using VS 2019 16.1.0, and this code is in a .fs file used interactively.

[nhirschey]

Scratch that. The SAS type provider hangs are caused by using CsvProvider with a problematic csv elsewhere in the .fs file.

If I comment out the code using CsvProvider, the SAS type provider does not cause the UI to hang. Trying to use the SAS provider must have caused the parser to try parsing the CSV for type info.

[cartermp]

@dsyme and @TIHan we should take a look at this too.

@nhirschey Just as FYI - we're looking at some issues with scripting today, as typing any key leads to re-parsing of all referenced scripts. It might be related.