Yeah... melding the physical with the metaphysical is troublesome at best. And that's even with audience suspension of belief.
Though, I will point out that some of the best fiction out there leaves some hanging questions - those 'what happened next?' or 'What is the meaning of that?' The main reason being is it stimulaties the imaginations of the audience.
Re: the ending of Macross Zero:
Shin's damaged VF-0 'soft' landed and he had his preverbial ''walk into the sunset."
Sara created the barrier around him because she had become one with the Tori-no-hito (birdman/'alien flower mecha') and was able to use her spiritia powers combined with its super technology to reach across time and space from where ever it folded to. (Given it's sorry state, I suspect that it probably folded away and expired - and Sara's powers were used while it was mid-fold.)
Was it something her father taught? Probably it was a combination of the Tori-no-hito's abilities and her spiritia singing powers. So yes, in part it was something her father taught.
They did. The Protodevlin in Macross 7. They fit your descriptions of, and requests for more horror fairly well. Though I will agree that they didn't come across very well in the anime...
In Macross 7, specifically Dynamite, they have reverse engineered anti-gravity to the point that it's included in civilian vehicles. As the anti-grav system on the SDF-1 Macross malfunctioned during it's first take-off, we can presume that it did take something like 35 to 40 years to work all the bugs out.
If a new Macross gets made, the question about having a darker tone or not really depends on how Big West feels the audience will react to it. In the end, it's all about making money, and shows like Macross 7 make more money than limited (or slow to be released) shows like Macross Plus and Macross Zero. (More time for marketing related toys/music/accessories/etc., potential of sequal OVAs and movies, and a 'lower' budget per episode, meaning a smaller start-up cost. In this regard, Shouji Kawamori is bad for a production, as he likes to spend lots of money in making them.)