I find spending a bit more on batteries goes along way. Although the nominal voltage and size may be the same, better batteries have lower internal resistance, ie provide the same current with less voltage sag. This prevent the low bat detection from tripping prematurely.
Batteries have one advantage over over supplies: extremely low noise. Even an good LDO will bump up the noise floor, and a cheap lcsc part will do so too. Plus you’s want a reasonably low dropout and quiescent current, which also increases price. Maybe 10ct in volume is reasonable for such a part - and yes, that will absolutely eat the margin
Fascinating! Thank you for including a picture of the new design, using the ribs to reinforce the surface should improve things a lot with minimal material added.