Flux.1 Pro vs SDXL for consistent character design in comics

I’ve been struggling to keep my protagonist’s face consistent across different panels for a webcomic I’m generating. I started with SDXL and used IP-Adapter, but I keep getting subtle shifts in eye shape and jawline that really break immersion when zoomed in. I recently tried Flux.1 Pro through a local ComfyUI setup, and the base quality is undeniably better, especially with text rendering, but I’m worried about the workflow complexity. Has anyone managed to get truly consistent character references working with Flux without resorting to heavy LoRA training for every single minor character? I tried using a simple reference image in the prompt, but it mostly just influences the style rather than locking the facial features. I’m considering training a small LoRA on my main character, but I only have about 20 high-quality reference images and I’m worried about overfitting or losing the natural variation I need for different expressions. Also, is the speed difference between Flux and SDXL on my 4090 significant enough to matter for drafting multiple pages a day? I find that the iterative prompting with Flux takes longer per generation, which slows down my sketching phase. If anyone has a specific ComfyUI node setup or workflow trick for maintaining character identity in Flux that doesn’t require a full LoRA, I’d love to hear it. I’m trying to avoid switching back to SDXL just because it’s easier to control, but the artifacts in SDXL are becoming too noticeable for my final output.