These genres moved the technologies forward.
A brief lesson in anime history: back in the 80s and 90s, anime was still drawn on cels, and not digitally. One of the most popular genres at the time was mecha, which required the artists to create the frames with utmost precision due to the abundance of details in all the robotic parts. And the studios were able to offer that: the Bubble Economy of the 80s was a big engine that moved the technologies forward, letting directors, writers, and animators go crazy.
Same thing with the shoujo, but the stylistic choice that forced the technologies to evolve wasn't the overly detailed giant robots, but on the contrary, soft, dreamy-looking, almost watercolor-like sequences that still required a lot of skill to portray on cels. Thankfully, the budgets also allowed that.
Then the bubble burst, but the studios somehow kept going, though they started to play it a bit safer. Until the crisis of 2008-2009 hit them and production companies decided that they couldn’t handle any more risks. Now we live in a world where capitalism has won and budgets are scarce. Anime studios don't take many leaps of faith, instead deciding to just play it safe and reuse whatever was already popular and will definitely bring in at least some money. The genres have a cycle of life, some rise in popularity and others fall, and currently neither shoujo nor mecha are on top of the hill.
These genres not only pushed the industry forward in terms of technology, but they also explored themes that other titles barely touched. Most of the time mecha titles are just barely concealed political dramas, filled with anti-world sentiments, exploring themes of empathy and the human psyche — and the last part is where it crossed over with shoujo, which has never really been synonymous with romance. The popular shoujo series of that time paved the way for many other works in other genres — for example, Kentaro Miura, the author of Berserk, quite often cites Rose of Versailles as one of his main inspirations. A lot of early isekai anime also dabbled in themes that were mostly associated with shoujo — 12 Kingdoms, for example.
But the safest bet for now is to cater to a broader audience, and in that audience the amount of people who can supposedly enjoy shoujo and mecha is not very high. Most people still believe that shoujo can be enjoyed only by the female demographic and that mecha lovers are even more scarce.
The safest bets don't push the technology forward, limiting the creative liberties that directors, writers, and animators are allowed to take.