A:
>> Would it be possible for me to create a Custom Editor using VSPackage
Yes.
>>and then use it with VSTA?
No.
However, you should be able to create ToolWindows in VSTA using an In-Process Host and 'CreateToolWindow' off the DTE.
Also, with VSTA, you could create a custom editor in a host application and use VSTA's DTE to generate (DOM or CodeModel), compile, and run VB or C# code hidden behind the host application or alongside it.