The problem with Browser-Enabled Forms is that your code runs entirely on the server. As such, there is no way you can show a WinForms messagebox: even if you succeed, no one will ever see it.
Your best is to implemented modeless UI in the form intself using conditional formatting by showing/hiding an section in your form that contains the error. This is not ideal (you cannot block the rest of the form) but it's pretty much the only solution for browser-enabled forms (for all practical purposes).
Of course once you have a conditionally formatted section you can control its appearance with either code or rules/default values. Please be aware that since all code runs on the server side, using code for this will require your form to send data to the server which will decrease its performance.
Hope this clarifies the issue!
David Airapetyan / Software Services ArchitectQdabra® Software / Streamline data gathering to turn process into knowledge