The XU4 is fairly expensive for a single-board computer ($59 plus PSU, storage and case), and it's getting a bit long in the tooth with 32-bit and all, but it's probably still the nicest choice among the machines Hardkernel have to option. In particular, it's fairly fast, the eMMC option is so much better than SD, and these days, you can run mainline kernel on them instead of some 3.10 build nobody cares about anymore. (Well, in Debian's kernel, you don't get HDMI, though…) It's not nearly as widely supported as the Raspberry Pi, of course, and it doesn't have the crazy huge ecosystem, but it's definitely faster.
Debian doesn't officially support the XU4, but with only a small amount of non-free bits in the bootloader, you can get an almost vanilla image; Debian U-Boot (with GRUB!), Debian kernel, and a plain image that comes out of debootstrap with only some minor awkwardness for loading the device tree. My personal one runs sid, but stretch is a good start for a server and it's easy to dist-upgrade, so I haven't bothered making sid images. I probably will make buster images at some point, though.