The required or recommended length is mainly to do with how much (the publisher thinks) the average reader is willing to pay for a book. Everyone in the chain between the writer and reader has their costs and overheads, and a clear idea of how big a profit margin is acceptable. That imposes a limit on how much the publisher can spend on printing each copy of the book, which imposes a limit on how long the book can be before the spreadsheets and algorithms start to say, “We probably won’t make a profit on this.”
If a book is much shorter than the “recommended” length, the price that the reader is willing to pay goes down, but some of the costs stay the same, or don’t go down proportionately. That means the number of copies the publisher has to sell to make a profit goes up. That in turn is why publishers will let Stephen King or JK Rowling publish a novella, but won’t let you or me publish one - or not until we’ve got a track record of novels that sell well, anyway.
It’s not unheard-of for a book that’s much longer or much shorter than the average to get published, but it’s one more obstacle on a path that’s already strewn with them.
The rise of ebooks is apparently loosening restrictions on length, since some of the overheads for print don’t exist there.