Defeat is declared fail a level, most often because too many monsters reached your orb or tower. When you are defeated, you are prompted with different options in different games, but always the options to restart the battle, return to the map, or view a guide. There are multiple ways to fail a level, including:
Monsters reaching your orb or tower when you cannot banish them
Not finishing the event of an event level by the last wave or last monster
In Gemcraft Chasing Shadows, being defeated makes you lose all progress on a level, any achievements you got, and any talisman fragments, field tokens, and other items you found in the level. The defeat summary will tell you how much experience you earned, but you do not get any of it. Shadow cores found in the battle are kept though, making it not a complete loss of time.