Thank you for your quick response.
I opened a ticket with Telerik requesting for some insight on RadMaskedTextBox if there are any hidden events/properties causing convention not getting recognized etc., and they suggested to try with RadMaskedTextInput instead and it worked with
here is the convention for RadMaskedTextInput:
ConventionManager.AddElementConvention<RadMaskedTextInput>(RadMaskedTextInput.ValueProperty, "Value", "ValueChanged");
ConventionManager.AddElementConvention<RadMaskedTextInput>(RadMaskedTextInput.MaskProperty, "Mask", "ValueChanged");
I will send this to VCarauelan who is responsible for Caliburn.Micro.Telerik nuget package.
Also there is a convention defined for RadWindow