Hi Qwer,
Great question. There are two mechanisms - Windows Shell and Internet Explorer.
When I was working at Microsoft, a developer on my team wrote the code to launch InfoPath. When clicking on an XML file on a web page, The XML document type is handled by the text/xml key under the CLASSES_ROOT\PROTOCOLS\Filters\ path in the registry. You can turn off the launch of InfoPath by changing the "text/xml" key to ".text/xml". Don't forget to change it back though. :)
If you install Office, the key will have a CLSID value that points to an InprocServer called MSOXMLMF.DLL. Office registers a handler for URLMON that launches InfoPath based on the mso-application PI in the XML. The actual code that launches InfoPath is in InLaunch.dll, which you can find under your installation for Office (e.g. C:\Program Files\Microsoft Office\OFFICE11). Keep in mind that Office requires Internet Explorer 6.0.
For the Windows Shell, InfoPath.Document points to a shell handler that launches InfoPath directly from Windows. Look under CLASSES_ROOT\InfoPath.Document.1 for various keys that the handler uses.
Hope that helps a little.
Patrick Halstead
Project Manager at Qdabra