In my experience attaching very large files in InfoPath is problematic. InfoPath can't compress the file attachment and simply inserts it into the form xml as a base 64 encoded string. Base 64 encoded strings might actually bloat the file size of the attachment so a 20mb file might be larger then 20mb stored in the form.
Also your SharePoint site can have an impact here, especially if you have high latency. Everytime a user opens the form the entire form and that 20MB plus attachment need to be downloaded.
One thing you might want to consider is to save the file attachments as links in the form and upload the file from InfoPath to a SharePoint library, this way there is a one time hit while the attachment is uploaded to SharePoint but the form will open very fast the next time because the file attachment is not actually saved in the form! We have a tool called "qRules" which has a command to do this for you called "SaveToSharePoint". We offer a free trial of qRules so you can try it out for yourself and see if it solves your issue.
Hope that helps
Ted