Performance Improvement

[oldhand7]

Hi, everyone, we want to ask how we can greatly speed up the processing times?
OS: Linux | Nvidia RTX: 4000 | GPU: 4GB | System Memory: 8GB
It took 3minutes 45 seconds to generate 10 images; and 8 minutes 51 seconds to generate 25 images for a user. Approximately 22.5 seconds per image.
Which means that our GPU can generate around 115,200 images per month.
Which supports 11,520 user requests at 10 images per user request or 4,608 user requests at 25 images per user request. I'd love to 4x - 5x that for this to be feasible.

Looking forward to hearing back from you. Thanks.

[mashb1t]

@oldhand7 possible measures to increase performance:

• use lesser steps / models with faster image inference time such as SDXL Turbo or performance mode LCM (quality may decrease though)
• use the Fooocus websocket API (js/pyhton) to prevent gradio and frontend from being the bottleneck OR only write a function for processing the AsyncTasks
  Swagger Ui ? or Api Documentation #1765 (reply in thread)
  add advanced parameter for disable_intermediate_results (progress_gallery) #1013
• get a better GPU (you've only mentioned the RTX 4000 series, but not the specific type of GPU. Example with RTX 4090: 1 image ~7s) OR use one or more cloud instance(s) with A10 GPU (AWS g5.xlarge, e.g. eu-central 1,4€/h) or better to highly increase model loading time due to being able to keep the model loaded in VRAM (using --always-gpu)
• implement multi-gpu support Parallel Running #2292
• use A1111 OR ComfyUI for parallel rendering

[rossaai]

@mashb1t I'm asking for performance improvements in Extreme Speed. I'm trying to build a realtime inference on top of Fooocus (I use it for normal image generation and i want to preserve compatibility)

Performance improvements

1. Disable Prompt Expansion (Fooocus V2). It saves ~700ms
2. Add high-vram

EXTRA_PARAMS = [
    "--always-high-vram",
    "--disable-offload-from-vram",
    "--disable-in-browser",
]
 sys.argv = EXTRA_PARAMS

3. Ignore image save. It saves around ~500ms
   

   Fooocus/modules/async_worker.py

   Lines 776 to 801 in 1c999be

   	for x in imgs:
   	d = [
   	('Prompt', task['log_positive_prompt']),
   	('Negative Prompt', task['log_negative_prompt']),
   	('Fooocus V2 Expansion', task['expansion']),
   	('Styles', str(raw_style_selections)),
   	('Performance', performance_selection),
   	('Resolution', str((width, height))),
   	('Sharpness', sharpness),
   	('Guidance Scale', guidance_scale),
   	('ADM Guidance', str((
   	modules.patch.positive_adm_scale,
   	modules.patch.negative_adm_scale,
   	modules.patch.adm_scaler_end))),
   	('Base Model', base_model_name),
   	('Refiner Model', refiner_model_name),
   	('Refiner Switch', refiner_switch),
   	('Sampler', sampler_name),
   	('Scheduler', scheduler_name),
   	('Seed', task['task_seed']),
   	]
   	for li, (n, w) in enumerate(loras):
   	if n != 'None':
   	d.append((f'LoRA {li + 1}', f'{n} : {w}'))
   	d.append(('Version', 'v' + fooocus_version.version))
   	log(x, d)

4. [TODO] Skip Fooocus Model Management

TO FIX

The huge problem is [Fooocus Model Management], generations take around 1300ms but the model management always takes 1200 ms, and every request or generation do same even if nothing changed (no controlnets, same offset lora)

Fooocus/modules/patch.py

Lines 439 to 445 in 1c999be

	def patched_load_models_gpu(*args, **kwargs):
	execution_start_time = time.perf_counter()
	y = ldm_patched.modules.model_management.load_models_gpu_origin(*args, **kwargs)
	moving_time = time.perf_counter() - execution_start_time
	if moving_time > 0.1:
	print(f'[Fooocus Model Management] Moving model(s) has taken {moving_time:.2f} seconds')
	return y

It's like a blocking point, as generations have. have to wait 1200ms to perform the next generation which only takes 1300ms

[mashb1t]

@rossaai we've recently discussed this in a request, see

• how to avoid model offloading/reloading #2311 (comment)
• how to avoid model offloading/reloading #2311 (comment)

I've also tried using Fooocus as real time inference for live drawing in #1043, but stopped due to mashb1t#11 (comment).
StreamDiffusion is just plain faster and doesn't have the burden of using a queue system like Fooocus does.

Hope this helps!

[rossaai]

Thanks for the reply and attach content. I really appreciate it

[rossaai]

@mashb1t I have some questions about how Fooocus works.

1. If there are no loras selected (lcm and offset) then the managemenet model is ignored, right?
2. Does the model management include only LoRAS or also controlnets?

Can you give me advice to achieve that goal in the best way, I want to make the inference as quickly as possible but with the possibility of loading custom LoRAS on demand, using styles and also controlnets and even inpaintings

I know that those requirements would be a big tradeoff, since that would hinder realtime, but I also don't want to make a realtime that only works with txt2img or img2img but is enriched with loras, styles and even controlnets

One of the solutions that come to mind are

1. Merge the Base Model + LCM to avoid model loading on each request
2. Reduce sampling steps to 4 instead of 8
3. Use a better GPU like A100

[mashb1t]

1. model management is always called when generating an image as the model can be different, but this can be mitigated by keeping it loaded in VRAM
2. CNs and IPs are always loaded, separately from LoRAs. See

   Fooocus/modules/async_worker.py

   Line 334 in 1c999be

   pipeline.refresh_controlnets([controlnet_canny_path, controlnet_cpds_path])

   
   Styles are never an issue as they extend the prompt (positive and negative).

if you're the only person using thr system to generate images i'd deactivate queueing for the generate button or in general to, depending on your needs, eliminate it as bottleneck.

All three of the mentioned solutions are valid + you can spawn more instances of Fooo us depending on GPU count to parallelize generation.
You can also directly try out SDXL Lightning with optimized 1, 2, 4 and 8 step inference LoRAs and models. See #2318 for more information and examples and https://github.com/mashb1t/Fooocus/blob/main/presets/lightning.json for my lightning preset with 8 steps.

[poor7]

1. Merge the Base Model + LCM to avoid model loading on each request
2. Reduce sampling steps to 4 instead of 8

There is already a combined 4-step model Juggernaut XL
https://civitai.com/models/133005/juggernaut-xl

[rossaai]

Yes! Using a 4-step process and the Euler-SG Uniform sampler yields the best performance without the need to reload LoRAs. Interestingly, if I increase the CFG beyond 1, the performance speed decreases to 3.4 iterations per second. However, if I maintain the CFG at 1, the performance speed is 6.7 iterations per second (tested on an A10G).

[oldhand7]

Hi, @rossaai , Thank you. Can you let me know more about the detail regarding the configuration in codebase? Thank you in advance.

[mashb1t]

@oldhand7 the sampler/scheduler can directly be set in the UI, no need for code changes.
Use Hyper-SD instead of Lightning for even better quality, already meeged into my fork and also to develop, next release 2.4.0 will be released soon.

[aparker21]

I am a complete noob to Fooocus. I have been using it for 2 or 3 months but I am still very new to it and learning to adjust some configurations and defaults in the config file. I just changed over from Windows 10 to Linux Mint on the same machine. On Win 10 Fooocus was doing about 7-15 seconds/iteration. Now in Linux Mint it is doing about 30 seconds/iteration. How can I speed this up? What did Win 10 do that Linux Mint is not and can I force Mint to work better? I see articles and links posted above and I will look through those, but I am not sure what applies to my situation and what does not. I am running a VERY slow machine and I know that and I accept that and am ok with it. I just want to make it as fast as it can possibly be or at least as fast as it was when I was using Windows 10. I am running a Ryzen 5 processor with 16G RAM and a GTX 1060 6G GPU. I do expect this to be on the slow side. Any help making it run closer to the 10 seconds per iteration or faster would be incredibly helpful. I normally run the Realistic model but also use the JuggernautXL_v8Rundiffusion. I get the same performance from both. I generallly run the 3 default Fooocus styles as well and I also us the Image Prompt with ImagePrompt and PyraCanny and/or Inpaint as well.

[mashb1t]

it's the latter, Fooocus will also auto-clear necessary resources when a new model / LoRA is loaded and handles vram management by itself. You may also experiment with setting command line arguments like --disable-offload-from-vram or --always-offload-from-vram, but the default is the recommended configuration.

[aparker21]

I thought that might be the case. I will keep the default. It would be nice if it did some sort of pare down so that while not actively running I can do other things and go back to it and not have to fully re-start. . . but I get why that might not make sense. I appreciate all the help! Looks like I need to upgrade my RAM a little bit (which I have always wanted to do anyway).

[aparker21]

While I have some helpful folks here: Can I tell Fooocus to use a different terminal program? I have Tilix installed and I keep track of Memory and Swap using "watch free -h" and "htop" and it would be nice to have all those with Fooocus in one Tilix window. Can I update a script to do that? If so, which one and where?

[mashb1t]

You can always start Fooocus in your terminal of choice by executing the startup command there. If you use the included .bat file, it will run in the Windows default terminal.

On Linux, commands can be executed as background process using an ampersand & at the end of the command, e.g echo "hello"&
https://www.digitalocean.com/community/tutorials/how-to-use-bash-s-job-control-to-manage-foreground-and-background-processes#starting-processes

sidenote: The Fooocus main thread spawns a worker thread for the queue, which will automatically run in the background but is not bound to a terminal.

[aparker21]

Ok. I have a desktop shortcut created in Linux that just says "Terminal=true". I wonder if I can somehow update that shortcut to point it to the terminal I want. I don't use the startup command and honestly have to research how to even do that.

I just answered my own question about the background process by reading the article you referenced. What a concept!!! Thank you for the help!! That is something I might have to look into. Though I don't really need it to run in the background since it is easy to pull up as many terminals as I want. It would just be nice to group my "watch free -h" and my "htop" as I confirm memory usage and processes running.