Hi Shraddha,
Unfortunately, with that many records, the data connection is bound to be slow.
One alternative is to have users search. The data connection would return only results that match the search term and load those into the dropdown. Sadly, I don't believe that InfoPath allows this out of the box. You would need to create your own solution, or use a third party tool, to filter the data on the server.
One such web service is DBXL, created by Qdabra Software, the company that sponsors this InfoPath forum. Here is a tutorial that explains the process.
Hope this helps!