While first to file has a solid claim, current use is also valid. This situation can come down to who is more scary. If they are a larger company they can use their lawyers and resources to show that they have already been using this mark in trade. And you can always use this official filing to bully other small companies into abandoning the mark.

As to your direct question, yes, the Virgin group can definitely come after you if they want to. Even if you have a strong case, a legal battle can bleed you dry and this is a common tactic of larger companies to bully and force smaller ones to do what they want in legal matters like this.