The restore browser doesn’t talk to the remote side at all. Duplicati queries your local job specific sqlite database. The speed of the restore browsing depends on your computer performance, the number of files you protect, the data size, the number of versions you are retaining, your deduplication block size, and possibly some other factors.
If you know the path of what you want to restore, you can use the Search box that is above the folder/file browser. It should let you get to your file with a single database search.
As to your suggestion about finding the file first before you select the snapshot version/time, that is being worked on (see Display versions for a file and allow any version to be restored by seantempleton · Pull Request #4805 · duplicati/duplicati · GitHub) but I’m not sure when it will be complete.