Error having multiple json Files in input directory

[en1cc]

Hey,
I encountered the following issue with more than one file in the input directory:

root@syseleveneigenbedarf-syseleven-ollispielt.dbl1.rwx-test-node01:~# ll fio-output/
total 96
drwxr-xr-x 2 root root 4096 Jun  6 09:37 ./
drwx------ 9 root root 4096 Jun  6 09:24 ../
-rw-r--r-- 1 root root 7482 Jun  6 09:57 cinder_randrw_10.json
-rw-r--r-- 1 root root 7495 Jun  6 09:39 cinder_randrw_1.json
-rw-r--r-- 1 root root 7488 Jun  6 09:41 cinder_randrw_2.json
-rw-r--r-- 1 root root 7491 Jun  6 09:43 cinder_randrw_3.json
-rw-r--r-- 1 root root 7482 Jun  6 09:45 cinder_randrw_4.json
-rw-r--r-- 1 root root 7495 Jun  6 09:47 cinder_randrw_5.json
-rw-r--r-- 1 root root 7486 Jun  6 09:49 cinder_randrw_6.json
-rw-r--r-- 1 root root 7486 Jun  6 09:51 cinder_randrw_7.json
-rw-r--r-- 1 root root 7484 Jun  6 09:53 cinder_randrw_8.json
-rw-r--r-- 1 root root 7496 Jun  6 09:55 cinder_randrw_9.json
-rw-r--r-- 1 root root 7485 Jun  6 09:17 cinder_randrw.json

root@syseleveneigenbedarf-syseleven-ollispielt.dbl1.rwx-test-node01:~# fio-plot -i fio-output/ -T "RANDRW on cinder blockstorage with ocfs2" -l -r randrw -d 256 -n 4 -f read
Traceback (most recent call last):
  File "/usr/local/bin/fio-plot", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/fio_plot/__init__.py", line 41, in main
    routing_dict[graphtype]["function"](settings, data)
  File "/usr/local/lib/python3.10/dist-packages/fio_plot/fiolib/bar2d.py", line 127, in chart_2dbarchart_jsonlogdata
    return_data = create_bars_and_xlabels(settings, data, ax1, ax3)
  File "/usr/local/lib/python3.10/dist-packages/fio_plot/fiolib/bar2d.py", line 99, in create_bars_and_xlabels
    ax1.set_xticklabels(labels=x_axis, fontsize=fontsize,)
  File "/usr/local/lib/python3.10/dist-packages/matplotlib/axes/_base.py", line 74, in wrapper
    return get_method(self)(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/matplotlib/_api/deprecation.py", line 297, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/matplotlib/axis.py", line 1969, in set_ticklabels
    raise ValueError(
ValueError: The number of FixedLocator locations (10), usually from a call to set_ticks, does not match the number of labels (1).

When there is only one file, i have no problem:

root@syseleveneigenbedarf-syseleven-ollispielt.dbl1.rwx-test-node01:~# fio-plot -i fio-output/ -T "RANDRW on cinder blockstorage with ocfs2" -l -r randrw -d 256 -n 4 -f read

 Saving to file RANDRW-on-cinder-blockstorage-with-ocfs2_2023-06-06_112541_mK.png

[louwrentius]

Thanks for the report. Would it be possible to share the resulting benchmark data here or through louwrentius*gmail.com?
I've double-checked on some test data I have and I don't encounter this issue.

[louwrentius]

I've never received the test data, so I'm closing this issue, feel free to reopen if you can provide me with some sample data so I can reproduce the problem.

[ABVitali]

Hi!
I'm sorry to comment on this old issue but I'm having the same. If you prefer that I create a separate issue, I'll do it.
I'm using the latest version of fio-plot (1.1.15), even if I tried several old versions too, and I generate the data with bench-fio (packed in the same 1.1.15 version).

This is the benchmark ini I used to generate the data

[benchfio]
target = /dev/sdb
output = benchmark
type = device
mode = randread,randwrite
size = 10G
iodepth = 1,2,4,8,16,32,64
numjobs = 1,2,4,8,16,32,64
block_size = 4k
direct = 1
engine = libaio
precondition = False
precondition_repeat = False
extra_opts = norandommap=1,refill_buffers=1
runtime = 60
destructive = True

and attached there is the list of files generated in the benchmark/sdb/4k folder:

Here I already see an issue that is that I miss many ".log" file. I saw that there was the same issue many versions ago, I don't know if I did anything wrong but I think there may be an issue here.
The other problem I'm having is that when I execute the command:

fio-plot -i benchmark/sdb/4k --source "test"  -T "test" -N -r randread -o test.png

I get this error:

Traceback (most recent call last):
  File "/home/marss/.local/bin/fio-plot", line 8, in <module>
    sys.exit(main())
  File "/home/marss/.local/lib/python3.10/site-packages/fio_plot/__init__.py", line 41, in main
    routing_dict[graphtype]["function"](settings, data)
  File "/home/marss/.local/lib/python3.10/site-packages/fio_plot/fiolib/bar2d.py", line 127, in chart_2dbarchart_jsonlogdata
    return_data = create_bars_and_xlabels(settings, data, ax1, ax3)
  File "/home/marss/.local/lib/python3.10/site-packages/fio_plot/fiolib/bar2d.py", line 99, in create_bars_and_xlabels
    ax1.set_xticklabels(labels=x_axis, fontsize=fontsize,)
  File "/home/marss/.local/lib/python3.10/site-packages/matplotlib/axes/_base.py", line 73, in wrapper
    return get_method(self)(*args, **kwargs)
  File "/home/marss/.local/lib/python3.10/site-packages/matplotlib/_api/deprecation.py", line 297, in wrapper
    return func(*args, **kwargs)
  File "/home/marss/.local/lib/python3.10/site-packages/matplotlib/axis.py", line 2025, in set_ticklabels
    raise ValueError(
ValueError: The number of FixedLocator locations (98), usually from a call to set_ticks, does not match the number of labels (1).

If you want I can send you an archive with the test data at the email address you wrote in the previous message.
Thank you for your work and patience to build and maintain this useful project.

[TjerkNan]

Thanks for your report, it's ok to reopen the issue. Feel free to share the test data as discussed, that would help.

[ABVitali]

Hi TjerkNan,
I'm very sorry for my late reply. Can I ask you if you want me to send the results at the email address indicated by @louwrentius or at a different address?
Thank you and again sorry for taking so long to answer

EDIT:
In the meanwhile I run new tests so my benchmark.ini is slightly different in the sense that I added different block_size instead of using only 4k. This is reflected in the results I will send you

[louwrentius]

No worries. Hi use louwrentius . gmail.com for email.

[kogakenji]

Hi! I've had the same issue:

.venv) root@x13-2:/home/inpe/fio/inpe# fio-plot -i inpe-poc/fio/4k/ -T "Title" -s https://louwrentius.com -l -r randread Traceback (most recent call last): File "/home/inpe/fio/inpe/.venv/bin/fio-plot", line 33, in <module> sys.exit(load_entry_point('fio-plot==1.1.15', 'console_scripts', 'fio-plot')()) File "/home/inpe/fio/inpe/.venv/lib/python3.10/site-packages/fio_plot/__init__.py", line 41, in main routing_dict[graphtype]["function"](settings, data) File "/home/inpe/fio/inpe/.venv/lib/python3.10/site-packages/fio_plot/fiolib/bar2d.py", line 127, in chart_2dbarchart_jsonlogdata return_data = create_bars_and_xlabels(settings, data, ax1, ax3) File "/home/inpe/fio/inpe/.venv/lib/python3.10/site-packages/fio_plot/fiolib/bar2d.py", line 99, in create_bars_and_xlabels ax1.set_xticklabels(labels=x_axis, fontsize=fontsize,) File "/home/inpe/fio/inpe/.venv/lib/python3.10/site-packages/matplotlib/axes/_base.py", line 74, in wrapper return get_method(self)(*args, **kwargs) File "/home/inpe/fio/inpe/.venv/lib/python3.10/site-packages/matplotlib/axis.py", line 2071, in set_ticklabels raise ValueError( ValueError: The number of FixedLocator locations (2), usually from a call to set_ticks, does not match the number of labels (1)

I'm using bench-fio:
[benchfio] target = /mnt/weka/fio output = benchmark type = directory mode = randread size = 4G iodepth = 1 numjobs = 1,2,4,8 filename_format=$jobnum/$filenum/iops.$jobnum.$filenum ;direct = 1 block_size=4k ;engine = libaio engine=posixaio ; precondition = False ; precondition_repeat = False ;extra_opts = norandommap=1,refill_buffers=1 runtime = 300 disk_util=0

and after executing bench-fio, I try to get the plot using:

fio-plot -i inpe-poc/fio/4k/ -T "Title" -s https://louwrentius.com -l -r randread

But I get this same error.

Do you have anything that clear this issue?

Thanks!

[louwrentius]

I'm not sure what's wrong here. Can you share your output files?

[kogakenji]

Hi @louwrentius,

I tested fio-plot and when I have only one .json output file it works. When I run bench-fio and it results in more than 1 output file fio-plot doesn't work.

(.venv) root@x13-2:/home/inpe/fio/inpe# fio-plot -i benchmark/fio/4k/ -T "Title" -s https://louwrentius.com -l -r randread Traceback (most recent call last): File "/home/inpe/fio/inpe/.venv/bin/fio-plot", line 33, in <module> sys.exit(load_entry_point('fio-plot==1.1.15', 'console_scripts', 'fio-plot')()) File "/home/inpe/fio/inpe/.venv/lib/python3.10/site-packages/fio_plot/__init__.py", line 41, in main routing_dict[graphtype]["function"](settings, data) File "/home/inpe/fio/inpe/.venv/lib/python3.10/site-packages/fio_plot/fiolib/bar2d.py", line 127, in chart_2dbarchart_jsonlogdata return_data = create_bars_and_xlabels(settings, data, ax1, ax3) File "/home/inpe/fio/inpe/.venv/lib/python3.10/site-packages/fio_plot/fiolib/bar2d.py", line 99, in create_bars_and_xlabels ax1.set_xticklabels(labels=x_axis, fontsize=fontsize,) File "/home/inpe/fio/inpe/.venv/lib/python3.10/site-packages/matplotlib/axes/_base.py", line 73, in wrapper return get_method(self)(*args, **kwargs) File "/home/inpe/fio/inpe/.venv/lib/python3.10/site-packages/matplotlib/_api/deprecation.py", line 297, in wrapper return func(*args, **kwargs) File "/home/inpe/fio/inpe/.venv/lib/python3.10/site-packages/matplotlib/axis.py", line 2025, in set_ticklabels raise ValueError( ValueError: The number of FixedLocator locations (2), usually from a call to set_ticks, does not match the number of labels (1).

and in the directory list files:

(.venv) root@x13-2:/home/inpe/fio/inpe# ls benchmark/fio/4k/ randread-1-1.json randread-iodepth-1-numjobs-1_clat.1.log randread-iodepth-1-numjobs-1_slat.1.log randread-1-2.json randread-iodepth-1-numjobs-1_iops.1.log randread-iodepth-1-numjobs-1_bw.1.log randread-iodepth-1-numjobs-1_lat.1.log

Please let me know if you can see those files:

randread-1-1.json
randread-1-2.json

Thanks!

[louwrentius]

@kogakenji

Thanks for sharing those test files. I'm not sure what went wrong, but those files seem to be almost identical and that's not what fio-plot expects.

I used your INI settings and ran it with a fresh fio + fio-plot install on a VM just to reproduce the issue.

[benchfio] 
target = /mnt/weka/fio
output = benchmark
type = directory
mode = randread
size = 100M
iodepth = 1
numjobs = 1,2,4,8 
filename_format=$jobnum/$filenum/iops.$jobnum.$filenum 
;direct = 1
;block_size=4k 
;engine = libaio engine=posixaio 
;precondition = False 
;precondition_repeat = False 
;extra_opts = norandommap=1,refill_buffers=1 
runtime = 10
disk_util=0 

Then I ran:

fio-plot -i benchmark/fio/4k/ -T "TEST" -N -n 1 2 4 8 -r randread

Notice the -N parameter because you used different 'numjobs' parameters.

This resulted in this test image:

Does this help you?

[kogakenji]

Hi @louwrentius,

Thanks for sharing those.
Unfortunately I ran the INI file you provided with bench-fio and tried to generate the plot with fio-plot, but had the same issue as before.

bench-fio run:

fio-plot run:

Here are the files generated by bench-fio:
4k.zip

Added to this I have those libraries in my environment:
contourpy==1.3.0
cycler==0.12.1
fio-plot==1.1.15
fonttools==4.54.1
Jinja2==3.1.4
kiwisolver==1.4.7
markdown-it-py==3.0.0
MarkupSafe==3.0.2
matplotlib==3.8.0
mdurl==0.1.2
numpy==1.26.4
packaging==24.1
pillow==11.0.0
pyan3==1.2.0
Pygments==2.18.0
pyparsing==3.2.0
python-dateutil==2.9.0.post0
rich==13.9.4
six==1.16.0
typing_extensions==4.12.2

Could you please share yours so that I can see if there's any difference?

I'm still trying to figure out what's happening on my side.

Thanks for your help!

[kogakenji]

Hi @louwrentius ,

I tried to generate with a different graph (bargraph3d) using:

fio-plot -i benchmark/fio/4k/ -T "Benchmark" -L -t iops -n 1 2 4 8 -r randread

and it works. At least it doesn't fail.

It isn't the graph I'd like to have though.

[louwrentius]

@kogakenji

I understand what's wrong, but not sure why. I do noticed that your fio version is 3.28 and that's quite old. Would you mind trying with a newer fio version?

I've compared my and your json files and I'm not so sure it has anything to do with fio output, probably something in the fio-plot code.

test.tgz

This is my generated test data.

[louwrentius]

@kogakenji

I know what the problem is. The numjobs parameter in your json output files are wrong, they are always of value 1, whereas my json files have correct increasing numbers 1,2,4,8.

[kogakenji]

@louwrentius Good catch!
Yeah, I can see that yours have those number differently. Potentially a fio problem.
I'm working on upgrading fio now. I'll let you know how it goes.
Thank you!

[kogakenji]

Hi @louwrentius ,

I've upgraded fio to versin 3.38, but I can see the same output from the json files (numjobs=1) in all files.
Here is the output json files:
benchmark.zip

maybe sth is happening in bench-fio?

[kogakenji]

@louwrentius ,

Sth else might be different from your environment and mine... not sure what.

my pip freeze:

contourpy==1.3.0 cycler==0.12.1 fio-plot==1.1.15 fonttools==4.54.1 Jinja2==3.1.4 kiwisolver==1.4.7 markdown-it-py==3.0.0 MarkupSafe==3.0.2 matplotlib==3.8.0 mdurl==0.1.2 numpy==1.26.4 packaging==24.1 pillow==11.0.0 pyan3==1.2.0 Pygments==2.18.0 pyparsing==3.2.0 python-dateutil==2.9.0.post0 rich==13.9.4 six==1.16.0 typing_extensions==4.12.2

and fio:

$ fio --version
fio-3.38

but editing the json files with the proper numjobs, it works perfectly. So, just sth that bench-fio is outputting numjobs always 1, but not sure how yours work and mine doesn't.

[kogakenji]

Hi @louwrentius,

I've created a work around by creating a shellscript to execute my .ini fio files 1 by 1 and then, with the resulting json files I execute fio-plot on top of that:

#!/bin/bash
# Iterate over all .ini files in the current directory
for ini_file in *.ini; do
    # Extract the base name using regex to match fio-bdc-<name>.ini
    if [[ "$ini_file" =~ fio-bdc-(.*)\.ini ]]; then
        base_name="${BASH_REMATCH[1]}"
       
        # Output to a .json file
        output_file="fio-bdc-${base_name}.json"
        sudo fio "$ini_file" --output="$output_file" --output-format=json
    else
        echo "Skipping $ini_file as it does not match the pattern fio-bdc-<name>.ini"
    fi
done

then:
put my resulting json files in, e.g. io-depth-1 folder.

and execute:

fio-plot -i iodepth-1 -T "One Example Title for I/O depth 1" -N -n 64 128 256 512  -r randread

Not ideal, but at least I could move on with my benchmark.

[louwrentius]

@kogakenji Thanks for the update, that's not an ideal solution, but I also noticed the error in your screenshot 'failed to write temporary file'. That sounds not ok.

[kogakenji]

@louwrentius. Yeah, I could see that error: 'failed to write temporary file' even though I'm running using root credentials. Note: this only happen when running bench-fio and not fio directly.