Fuji cameras really are not that big. I would go to the Camera Size Comparison website and have a look.
The great thing about Fuji is also the problem--what camera to buy? There are so many to chose from, but you can get one for every mood and outfit. And if you don't like the one you have, you can also buy another, which is totally inline with the spirit of GetDPI.
My move to Fuji was two fold. I wanted something smaller than my Pentax 645D system that would be more spontaneous, but also be a good formal camera when need be. I also like OVFs and rangefinder/viewfinder cameras. I have two X Pro2s. I also picked up an XF10 for a pocket camera, which gave me an additional focal lengths of 18mm. I carry all three cameras and four lenses in a Mountainsmith lumbar pack.
I find the new X-E4 compelling, but also as a second to my X Pros. With the 27mm, it makes a very compact system, which can also take other lenses. Having a Sony RX-1, the missing grip does not bother me. But then, I would not be using big lenses, which may be less appealing on a small body.
The X-S10 and X-T3 both have better articulating LCDs. I shoot a lot of verticals and so that is would be useful. But then, my X Pros have a fixed screen and I function with that, so it is not a deal breaker for me. Obviously, having IBIS is not that important for me either. But I do want dual card slots having lost images from a corrupt SD card previously--once was enough for me. The X-T3 has dual slots.
For a control layout, I actually like the Fuji implementation on the X-T3 (and X Pros). You want P, set the shutter speed dial and aperture to A, for A, set the shutter speed dial to A, for S, set the aperture to A. Basically, you simply tell the camera what be automatically controlled by the camera. Manual is just setting both shutter speed and f-stop. (The X-E4 has a P on the shutter speed dial to put the camera into fully automatic, but it also uses the A on the shutter speeds and aperture.) Basically, everything is marked and laided out for you, including ISO and exposure comp. The X-S10 is just blank, which makes it very customizable, but not obvious.
I would see if you can get your hands on some of these cameras.