Another plus one for Proton with your own domain.
Self hosting sounds good, but it’s fraught with mines that if you don’t know what you’re doing can take from “can’t send email because my domains been back listed” to “everything in my network is now sending spam to the entire world”. Sure, many folks self hosting sounds with no issues, but the price for configuring something wrong can be steep and IMO is just not worth the trouble and risks when there are good options for encrypted, privacy protecting email services for a reasonable price.
Something like LPS/TENS would fit your needs exactly, but it stopped being supported in 2021. It was essentially Tails but was built to be run from a wide verity of sources (we used to network boot onto PCs that had no hard drives at all, but had local network share access).