I would say that the authors of some of the most impactful, most successful books do this. Because ultimately, novelty is key, and novelty often involves breaking taboos, whether cultural or those of the publishing business. The converse is not true, though: that breaking taboos automatically leads to a great book.
Just as you will probably not write a great book ‘to rule’, in the work-to-rule sense.
There are paradoxes here that all writers face - how to write what you think needs to be written, the story that your heart wants to tell, and follow the rules so that it passes gatekeeper filters and some people actually read it.
I’ve been mulling this a bit for the book I’m posting now, which I wrote for readers from 11 to 14 (transition from MG to YA) with crossover potential to older readers, and which initially had a 14 year-old MC. I know I have to give it a clear publishing category, and from the length, it has to be YA rather than MG. But according to the rules, the MC can’t be 14 in a YA or MG - she’d have to be 16 for YA (or at a stretch, 15). Yet, the story is about a tribal society in which people take on adult-like roles (marriage, raiding, etc.) earlier in life than is the norm in Western society. That meant content that some agents and publishers might consider too adult for YA. Does that mean it has to be called Adult? Still doesn’t seem right. I tweaked it so that now she’s 15.5, but it’s still a bit of an MG-YA hybrid. Still mulling it.
Anyway, I’m happy to notice more and more category-defying exceptions, though the point about debut authors needing to ‘play ball’ is well made.
My advice? Be aware of and respect the ‘rules’ and conventions, understand the business reasons for them (e.g. shelving categories), then write the story that your soul needs to write, without thinking of the rules. If you try to write to the rules, your story will be crap.
If you end up with something that could be great, but will find it hard to be accepted as a debut? No problem, put it aside and write another one, and keep doing that until either you write one that happens to fit a niche, or the market catches up with you Probably, once you’ve written three or four, you’ll realize the first one wasn’t quite as great as you first thought, and you can improve it.
Footnote: Before I’m justly accused of romanticizing the business of writing, I’ll point out that
- there are also examples of ‘great’ stories that were quite conventional and derivative, but great because of great execution. The Harry Potter series comes to mind.
- there are plenty examples of stories that are successful despite being derivative and having mediocre writing. They happen to be well-timed, and catch a wave of interest.