[wsl2] filesystem performance is much slower than wsl1 in /mnt

[ioweb-gr]

I decided to open this as a separate issue because although it's related to the generic issue of filesystem performance it's directly related to WSL 2 while the other issues are for WSL 1 and it's showing very conflicting results.

• Your Windows build number: (Type ver at a Windows Command Prompt)
  Version 10.0.18917 Build 18917

• What you're doing and what's happening:
  I'm testing filesystem write speed in /mnt using dd command. Performing the following tests

WSL2

root@LUCIANO-PC:/home/# dd if=/dev/zero of=/mnt/e/testfile bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 25.939 s, 40.4 MB/s

WSL1

root@LUCIANO-PC:/home/# dd if=/dev/zero of=/mnt/e/testfile bs=1M count=20000
20000+0 records in
20000+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 47.4897 s, 442 MB/s

On / it's actually the reverse. WSL2 is more than 2 times faster than WSL1.

• What's wrong / what should be happening instead:
  I would expect the filesystem performance in /mnt to at least be on the same level but it's over 10 times slower.

Another interesting fact is that if I mount the same drive as a cifs share I get 3x performance

WSL 2 (cifs share)

root@LUCIANO-PC:/mnt/sambae# dd if=/dev/zero of=/mnt/sambae/testfile bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB, 9.8 GiB) copied, 84.001 s, 125 MB/s

An update of the current status because it's way too hidden in this thread.

Latest status report: #4197 (comment)_

[billziss-gh]

On / it's actually the reverse. WSL2 is more than 2 times faster than WSL1.

Can you post the results when doing the same test on a file on / for WSL1 and WSL2?

This is straight I/O on a single file where I would expect to see more or less the same perf between WSL1 and WSL2. My expectation was that WSL2 perf is better than WSL1 only when doing "namespace" operations (i.e. working on lots of small files, listing them, stat'ing them, etc.)

As for the results on /mnt/e they are not very surprising, since I/O has to go through both the Linux and Windows file system stack in likely a new and unoptimized piece of code.

[ioweb-gr]

I see,

I'm assuming you're still refering to / and not /mnt folders about namespace operations.
I've repeated the write tests in both environments a few times now and there's no consistent result.

WSL 2

root@LUCIANO-PC:/home/# dd if=/dev/zero of=/testfile bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB, 9.8 GiB) copied, 7.0873 s, 1.5 GB/s
root@LUCIANO-PC:/home/# dd if=/dev/zero of=/testfile bs=1M count=20000

20000+0 records in
20000+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 51.3779 s, 408 MB/s
root@LUCIANO-PC:/home/#
root@LUCIANO-PC:/home/# dd if=/dev/zero of=~/testfile bs=1M count=20000
^C12360+0 records in
12360+0 records out
12960399360 bytes (13 GB, 12 GiB) copied, 12.1766 s, 1.1 GB/s

WSL1

root@LUCIANO-PC:/home/# dd if=/dev/zero of=~/testfile bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB, 9.8 GiB) copied, 29.7096 s, 353 MB/s
root@LUCIANO-PC:/home/#

I'm thinking this has to do something with OS Caching because in task manager even though I can see the file being written, the disk usage write speed is unchanged while in wsl1 I can see it constantly at around 350mb/sec .

As another interesting fact, the vmmem usage while writing the file is increasing rapidly. By the 3rd copy action all my RAM is used up (32GB).
It goes back down a while after quitting my shell. So maybe the performance is bottlenecked by this bug as well.

[paultyng]

I'm also experiencing this after an upgrade to WSL2, only slow performance under /mnt, not /, slower than WSL1. I can post numbers as well if necessary. This is on Microsoft Windows [Version 10.0.18936.1000].

[definelicht]

I'm experiencing the same issue, /mnt is extremely slow compared to WSL1 (seconds to run git status on a relatively small repository).

[paultyng]

When I run a git status on a clone of https://github.com/hashicorp/terraform in /mnt I get output like this:

It took 6.97 seconds to enumerate untracked files. 'status -uno'
may speed it up, but you have to be careful not to forget to add
new files yourself (see 'git help status').

[ioweb-gr]

This is an example from the command

pv files.tar.gz | tar -zxf -

The speed is amazingly slow in wsl2 on /mnt

[definelicht]

Do we know what's at the root of this, and whether there are any other workarounds than downgrading?

[benhillis]

@definelicht - Yep we're working on improving the performance. In the meaning working out of your root file system (the ext4 volume) will have MUCH better performance.

[ioweb-gr]

Indeed after the last update I can already see much better performance on WSL2.

root@LUCIANO-PC:~# dd if=/dev/zero of=/mnt/e/testfile bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB, 9.8 GiB) copied, 41.277 s, 254 MB/s

That's on an SSD and it's like 6 times faster than before. Still slower then WSL1 but it's definitely improved.

--correction---
I see much better performance on sequential writes. Untar seems very slow still extracting data at bytes/sec while in wsl1 I can see speeds at kb/sec for the same part

[tuananh]

Indeed after the last update I can already see much better performance on WSL2.

root@LUCIANO-PC:~# dd if=/dev/zero of=/mnt/e/testfile bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB, 9.8 GiB) copied, 41.277 s, 254 MB/s

That's on an SSD and it's like 6 times faster than before. Still slower then WSL1 but it's definitely improved.

--correction---
I see much better performance on sequential writes. Untar seems very slow still extracting data at bytes/sec while in wsl1 I can see speeds at kb/sec for the same part

I got

1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 3.57894 s, 293 MB/s

but git status still taking more than 2 seconds on quite small repo

[definelicht]

I think this is a latency rather than a bandwidth issue, surfacing when a large number of files are accessed (e.g., when running git status).

[tuananh]

@definelicht - Yep we're working on improving the performance. In the meaning working out of your root file system (the ext4 volume) will have MUCH better performance.

can you share the cause of it? I'm curious

[gbatalski]

can confirm a very bad performance on /mnt (mvn install takes forever), on /tmp it takes only 2 minutes for the same project

[nono-pxbsd]

After switch configuration from WSL to WSL 2 (Build 18975), without any change on my distro, the server respond time incresed.

Configuration : Debian buster Apache2.X / PHP-FPM 7.3 / MySQL
Root : /
Web directories : /c/repositories/xxx

WSL 1 : 0.5 sec
WSL 2 : 8 sec

With Docker (Ubuntu 18.04) WSL Tech preview i could not test on WSL 1, but WSL 2 takes over 30 sec... I suppose the issue is the same.

[realtica]

This is my performance in a medium project:

GIT_TRACE_PERFORMANCE=1 git status -sb -uno

13:20:04.630972 read-cache.c:2267 performance: 0.017249300 s: read cache .git/index
13:20:07.766270 preload-index.c:146 performance: 3.135180100 s: preload index
Refresh index: 100% (3463/3463), done.
13:20:53.531359 read-cache.c:1575 performance: 48.900269600 s: refresh index
13:20:53.981829 read-cache.c:2267 performance: 0.016242500 s: read cache /mnt/e/p/keyed_au/.git/ modules/Files/media/nusoap/index
13:20:54.383135 read-cache.c:1575 performance: 0.401222700 s: refresh index
13:20:54.426399 diff-lib.c:251 performance: 0.000005000 s: diff-files
13:20:54.468000 unpack-trees.c:1554 performance: 0.000087700 s: traverse_trees
13:20:54.468266 unpack-trees.c:465 performance: 0.000164900 s: check_updates
13:20:54.468346 unpack-trees.c:1649 performance: 0.000438000 s: unpack_trees
13:20:54.468368 diff-lib.c:537 performance: 0.001873400 s: diff-index
13:20:54.569793 read-cache.c:2989 performance: 0.014679900 s: write index, changed mask = 2
13:20:54.586550 trace.c:477 performance: 0.736638600 s: git command: /usr/lib/git-core/git status --porcelai n=2 -uno
13:20:55.277184 read-cache.c:2267 performance: 0.031572000 s: read cache /mnt/e/p/keyed_au/.git/ modules/Files/modules/unittest/index
13:20:55.958706 read-cache.c:1575 performance: 0.681302500 s: refresh index
13:20:56.007094 diff-lib.c:251 performance: 0.000003500 s: diff-files
13:20:56.041768 unpack-trees.c:1554 performance: 0.000081200 s: traverse_trees
13:20:56.042014 unpack-trees.c:465 performance: 0.000162100 s: check_updates
13:20:56.042084 unpack-trees.c:1649 performance: 0.000399800 s: unpack_trees 13:20:56.042102 diff-lib.c:537 performance: 0.000454400 s: diff-index 13:20:56.126615 read-cache.c:2989 performance: 0.012095500 s: write index, changed mask = 2 13:20:56.141174 trace.c:477 performance: 1.261971900 s: git command: /usr/lib/git-core/git status --porcelain=2 -uno 13:20:56.148677 diff-lib.c:251 performance: 2.586112000 s: diff-files 13:20:56.190888 unpack-trees.c:1554 performance: 0.000022200 s: traverse_trees 13:20:56.191028 unpack-trees.c:465 performance: 0.000027800 s: check_updates 13:20:56.191106 unpack-trees.c:1649 performance: 0.000639200 s: unpack_trees 13:20:56.191175 diff-lib.c:537 performance: 0.001413100 s: diff-index
13:20:56.215068 trace.c:477 performance: 51.628126000 s: git command: git status -sb -uno

File System exFat, the result is the same is other file system...
WSL 2 git=51.62 sec
git for windows=0.8 sec

[paslandau]

I'm currently using the normal Docker Desktop with linux containers to run a PHP projekt during local development. The repo is in a normal Windows directory and gets mounted via volume bind in the docker container, so that I can easily manipulate the files through an IDE.

The same setup on a native linux machine has always been multiple times faster and the "problem" (from what I've read) has always been attributed to the Linux VM that is used on Docker Desktop.

Then WSL 2 was announced and I finally came around to set everything up. But when I ran our teststuite for the first time, I was pretty disappointed that the performance was actually much worse than before (40s on the Docker Deskop Linux VM setup vs 100s on the WSL2 setup). Subsequent investigation brought me to this thread.

From what I've read here so far:

Does it even make sense to switch to WSL 2 for local development yet if "bad performance" is your primary reason to do so?

A "no" would be a totally acceptable answer - I just don't want to waste any more time on this if it's simply not ready yet.

[stweedie]

I know that wsl2 is a preview, but is there any hope that accessing \wsl$ from windows or /mnt from wsl will reach acceptable speeds?

[gormulent]

I'd also like to ask about this, doing a git status on something in /mnt/c/.. takes about a minute, something seems wrong ..

[gbatalski]

same with 1903 (18990.1) absolutely useless...

[wusticality]

I am also seeing this, accessing Windows files from the Linux environment is several orders of magnitude slower in WSL2 than in WSL1.

[foriequal0]

But... We are still waiting to see the impact of the optimizations that were added to the 6.1 kernel in the end of 2022 phoronix.com/news/Linux-9p-10x-Performance, but WSL is still using kernel 5.15 for some reason. I haven't seen any benchmarks with a custom-built kernel, but I don't think it's enough, we probably need a similar patch on the Windows side as well.

6.1 kernel doesn't make a huge change.
NTFS mount on WSL2

5.15.153.1-microsoft-standard-WSL2
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 3.76512 s, 278 MB/s

6.1.21.2-microsoft-standard-WSL2+
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 3.73518 s, 281 MB/s

Compare to ext4 on WSL2

5.15.153.1-microsoft-standard-WSL2
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 0.361747 s, 2.9 GB/s

[LambdaScorpii]

Hi @benhillis, any news on the issue? Mounting an external drive has around 1/10 of the Performance it is on native windows. The external drive is necessary however. Is there a workaround to increase read/write performance on mounted drives in wsl2?

Thanks in advance for the help.

[jensgreven]

Hi @benhillis, any news on the issue? Mounting an external drive has around 1/10 of the Performance it is on native windows. The external drive is necessary however. Is there a workaround to increase read/write performance on mounted drives in wsl2?

Thanks in advance for the help.

Using ext4 file system on mounted drives will give you “near native” performance

[LambdaScorpii]

Hi @benhillis, any news on the issue? Mounting an external drive has around 1/10 of the Performance it is on native windows. The external drive is necessary however. Is there a workaround to increase read/write performance on mounted drives in wsl2?
Thanks in advance for the help.

Using ext4 file system on mounted drives will give you “near native” performance

Thanks for the quick reply 👍 I will try that.

[jensgreven]

To mount a drive on WSL startup

1. Use PowerShell to find out the number of the drive you want to mount:
   GET-CimInstance -query "SELECT * from Win32_DiskDrive"

2. Find out the uuid of the drive by mounting it with the --bare option

wsl --mount \\.\PHYSICALDRIVE4 --bare

and then querying devices with

sudo blkid

(You might want to use lsblkto find out which device you are looking for.)

3. Edit /etc/wsl.conf:

[boot]
systemd=true
 
[boot]
command="bash /mount.sh > /mount.log"

4. Create /mount.sh:

sudo bash -c "cat > /mount.sh" <<EOF
#!/usr/bin/env bash

/mnt/c/Users/<USERNAME>/AppData/Local/Microsoft/WindowsApps/wsl.exe -d Ubuntu-24.04 --mount \\\\.\\PHYSICALDRIVE4 --bare
echo "Physical drive mounted in WSL"

timeout=30
elapsed=0
interval=2

while [ ! -e /dev/disk/by-uuid/cd7c642e-0c89-4b6c-87e8-c839ca9c8be6 ]; do
    if [ $elapsed -ge $timeout ]; then
        echo "Timed out waiting for /dev/disk/by-uuid/5da64086-ff3a-4142-8d0b-5ec56d071298 to become ready."
        exit 1
    fi

    echo "Waiting for /dev/disk/by-uuid/5da64086-ff3a-4142-8d0b-5ec56d071298 to be ready..."
    sleep $interval
    elapsed=$((elapsed + interval))
done
echo "Device is ready"

mount -t ext4 -o defaults /dev/disk/by-uuid/5da64086-ff3a-4142-8d0b-5ec56d071298 /home/<USERNAME>/my_mounted_drive
echo "Device mounted to /home/jgreven/echo
EOF

(replace the UUID in the script with the one you found out in step 2 and adjust the mount point to your liking)

The idea of calling WSL to mount the drive from within WSL at bootup is absurd and genious :-)

(source: #6073)

[ioweb-gr]

Just keep in mind that drive then is removed fully from windows side and it's no longer accessible there directly

Only through wsl

[lukasf]

Is there any status update on this? I read in this discussion that a new 9P driver is available in the Linux kernel, which should increase performance nearly 10x:

#9412

It would be awesome if this could be integrated in WSL2 soon. The current WSL2 IO performance is just horrible, and the workarounds are too complicated for many use cases.

[iccfish]

Is there any status update on this? I read in this discussion that a new 9P driver is available in the Linux kernel, which should increase performance nearly 10x:

#9412

It would be awesome if this could be integrated in WSL2 soon. The current WSL2 IO performance is just horrible, and the workarounds are too complicated for many use cases.

I've tried kernel 6.13.0 and the performance was even worse. Reading from windows file system, WSL1 operates at 3.3GB/s, and WSL2 with 6.13.0 only get 90MB/s, I still hard to believe this.

[ioweb-gr]

How are you getting 3.3gb per sec, I'm capped at 250mb per sec on a disk which can easily hold 7gb per sec

[n0valis]

Since operating pytorch/triton is way faster on Linux I tried to install my stuff in the wsl. And indeed processing images with Flux is significantly faster. Unless you count how long it takes to load the 24GB models from /mnt/x/aimodels which takes as long as my grandma for taking a dump. I can't believe this is real in 2025, Cmon Microsoft get your big pants on and fix this.

[jensgreven]

Since operating pytorch/triton is way faster on Linux I tried to install my stuff in the wsl. And indeed processing images with Flux is significantly faster. Unless you count how long it takes to load the 24GB models from /mnt/x/aimodels which takes as long as my grandma for taking a dump. I can't believe this is real in 2025, Cmon Microsoft get your big pants on and fix this.

Put the models on the filesystem that redides in your vhdx file or mount an ext4 partition. in case your wsl gets too big you can easily move it to another drive with wsl —manage DISTRO —move DEST

I have put together some howtos under https://ai-telligence.info (work in progress)

[n0valis]

Put the models on the filesystem that redides in your vhdx file or mount an ext4 partition. in case your wsl gets too big you can easily move it to another drive with wsl —manage DISTRO —move DEST

Thank you Jens for your suggestion, I already read your posts above. That's very useful for a workaround.

But I can't do that, firstly I can not shift around a couple of hundreds of GB into my distro. They are fine where they are and they are needed there. Of course I can duplicate them, what I have done already for one or two models. But that is really a crap thing to do.

I just wanted to express my anger towards Microsoft to be so ignorant and not doing anything about it for ages.

[jensgreven]

Put the models on the filesystem that redides in your vhdx file or mount an ext4 partition. in case your wsl gets too big you can easily move it to another drive with wsl —manage DISTRO —move DEST

Thank you Jens for your suggestion, I already read your posts above. That's very useful for a workaround.

But I can't do that, firstly I can not shift around a couple of hundreds of GB into my distro. They are fine where they are and they are needed there. Of course I can duplicate them, what I have done already for one or two models. But that is really a crap thing to do.

I just wanted to express my anger towards Microsoft to be so ignorant and not doing anything about it for ages.

I feel you. I use StabilityMatrix for managing my models and Loras etc and thanks to the bad performance of WSL with files on a Windows FS I could not find a way to a) run StableDiffusion in a Docker Container and b) run StabilityMatrix for managing the Models. That‘s why I switched to Linux on the PC that stuff runs on and never regretted it.

[n0valis]

I feel you. I use StabilityMatrix for managing my models and Loras etc and thanks to the bad performance of WSL with files on a Windows FS I could not find a way to a) run StableDiffusion in a Docker Container and b) run StabilityMatrix for managing the Models. That‘s why I switched to Linux on the PC that stuff runs on and never regretted it.

Mayby I gonna try a seperate ext4 partition. Is there anything else you could tell me? I just create an ext4 partition somwhere and mount it from my wsl distro?

[jensgreven]

I feel you. I use StabilityMatrix for managing my models and Loras etc and thanks to the bad performance of WSL with files on a Windows FS I could not find a way to a) run StableDiffusion in a Docker Container and b) run StabilityMatrix for managing the Models. That‘s why I switched to Linux on the PC that stuff runs on and never regretted it.

Mayby I gonna try a seperate ext4 partition. Is there anything else you could tell me? I just create an ext4 partition somwhere and mount it from my wsl distro?

Two aspects come to mind:

1. In order for WSL to mount a partition the drive it resides on needs to be deactivated for windows (if i understand it correctly) so no other partition on that drive can be used in windows while any partition from it is mounted.
2. Mounting the partition in WSL seems to take a significant amount of time, so the automatic start of my docker containers (i used docker desktop) failed, because a bind mount there tried to mount a dir from a drive that was not yet available. i found no solution to that problem.

IIRC I have a script for mounting an ext4 partition on my website.

That “solution” may be sufficient for trying out stuff but is way too fragile for anything even remotely productive (like everything when it comes to WSL). but to be fair i think MS never promoted WSL as production-quality component or even explicitly advised not to use it for anything more serious than testing. You find a similar statement on the docker page that basically says: the only environment for production use of docker is linux.

[n0valis]

IIRC I have a script for mounting an ext4 partition on my website.

You probably mean mount.sh (few post above)

That “solution” may be sufficient for trying out stuff but is way too fragile for anything even remotely productive (like everything when it comes to WSL). but to be fair i think MS never promoted WSL as production-quality component or even explicitly advised not to use it for anything more serious than testing. You find a similar statement on the docker page that basically says: the only environment for production use of docker is linux.

Yeah that's a killer.

[jensgreven]

IIRC I have a script for mounting an ext4 partition on my website.

You probably mean mount.sh (few post above)

That “solution” may be sufficient for trying out stuff but is way too fragile for anything even remotely productive (like everything when it comes to WSL). but to be fair i think MS never promoted WSL as production-quality component or even explicitly advised not to use it for anything more serious than testing. You find a similar statement on the docker page that basically says: the only environment for production use of docker is linux.

Yeah that's a killer.

most important thing (that took me quite some time to find out) is escaping the backslashes, s make sure you have ‘\\.\PHSICALDRIVE7’

[okulev]

Has someone tried mounting Windows folder with Samba?

Mounting the NTFS filesystem using Linux’s samba / cifs support in the WSL 2 container gets you timings that are 4x faster than the built-in 9p NTFS access, but still between 0.25x and 2x slower than WSL 1’s built-in support for accessing Windows files. -- https://vxlabs.com/2019/12/06/wsl2-io-measurements/

[jensgreven]

Has someone tried mounting Windows folder with Samba?

Mounting the NTFS filesystem using Linux’s samba / cifs support in the WSL 2 container gets you timings that are 4x faster than the built-in 9p NTFS access, but still between 0.25x and 2x slower than WSL 1’s built-in support for accessing Windows files. -- https://vxlabs.com/2019/12/06/wsl2-io-measurements/

I tried CIFS/SMB mounts as well as NFS mounts and though they were not quite es slow it was still far from usable.

[trentbuck]

On 23 Jan 2025, at 23:14, n0valis ***@***.***> wrote: Put the models on the filesystem that redides in your vhdx file or mount an ext4 partition. in case your wsl gets too big you can easily move it to another drive with wsl —manage DISTRO —move DEST

You can also downgrade your wsl2 VM to a wsl1 chroot using the wsl command. If you don’t need any GUI stuff there’s no downside afaict. This is what I do. You can also use gtk broadway or qt webgl to export the GUI app to edge — no sound though.

[n0valis]

Two aspects come to mind:

1. In order for WSL to mount a partition the drive it resides on needs to be deactivated for windows (if i understand it correctly) so no other partition on that drive can be used in windows while any partition from it is mounted.

I think I found a solution that works for me. I created a dynamic vhdx, formated ext4 and used
Write-Output "\.\PhysicalDrive$((Mount-VHD -Path Y:\WSL\Debian\data.vhdx -PassThru | Get-Disk).Number)"
to get to the Physicaldrive #. Then mounted within wsl.
That gives me full speed of the NVME where it resides.