I actually have the same problem myself. I have a *HACK* work around in place that lets me develop and testing UI for now (Until i actually want to figure out the real fix to the problem).
My temporary hack works by just creating a Application.xaml/cs file and directly calling Run() in the winForms application. Since there is no System.Windows.Application in the current app domain at the time, .Run() works. The below snippet is launched from
a injected MenuItem on the parent .Net 2.0 winforms application. To redisplay the main window after it's closed i think you would need to expose the bootstrapper in the application object and call DisplayRootView. I just havent tried this yet. I'm not
sure how friendly this would be to other applications trying to use the same trick since the Application object is a singleton and only 1 per AppDomain.
if (System.Windows.Application.Current == null)
MyApplication app = new MyApplication(); //The application defines the Caliburn framework normally.
//Do something else I haven't worked out yet.
If you find a better/cleaner solution let me know I'm in the same boat. =)