I don't believe there's a built-in way to invoke the save prompt via rules.
One thing you can do is show a hidden section in the form that prompts the user to save when the Cancel button is clicked, providing a Submit+Close button for Yes, and a Close button for No.
To conditionally only use that section when the document has changes in it, you can either use qRules' IsDirty command, or this approach:
http://alecpojidaev.wordpress.com/2009/07/06/how-to-check-if-your-form-is-dirty/
Though I would suggest putting the InitialState field in a secondary data source instead of the main data source if you go with the latter approach.