For anyone else that is drawn to this subject line, I had what sounds like the same problem.
TL;DR: Try the āārestore-volume-cache-hintā option. It controls the amount of temp space for storing remote dblock, etc files locally. The default is 5GB (volume size x 100). The parameter will want a size suffix, eg ārestore-volume-cache-hint=20GB.
I periodically do a full, āno-local-db, restore of my backups. One of them was taking many hours (I let it run to completion one time and I think it ran to 40+hrs). Other backups might run to a few hours .. maybe, usually much less. I also use a dedicated temp directory (ātempdir) and I could see a LOT of file churn. My log files recorded many downloads of the same files; I stopped counting after 50 downloads for the same file!
For the record, the backup in question tips the scales at 60GB for remote storage with thousands of small files, most of which donāt change once created, but thereās fairly constant addition and removal.
Anyway, I saw the āārestore-volume-cache-hintā option in the source code and, basically, set it to the size of the backup (ie the total size of the dblock, etc, files).
Result? Restore took around 45min.
The default value for the restore volume cache is volume size * 100, or 5GB with a default volume size of 50MB. Once the cache size is exhausted, volumes are deleted by a āleast recently usedā (LRU) eviction policy (donāt quote me on that). Sounds reasonable.
The problem though is that the new restore process is āfile-basedā whereas the previous, legacy, process, is āvolume-basedā. Iāll explain (what follows is a very simplistic take on what kenkendk explained above but, I think, more clearly gets my point across).
The legacy process iterates through each volume in the backup, extracts all the file data it contains, which can be many files, and builds āscaffoldā files which contains the extracted pieces of each file. As each new volume is downloaded, data is used to incrementally patch the missing data in each file, until all the volumes are processed and, if all goes to plan, all your files are fully patched and restored. You can see the process in the logs if you enable verbose logging. And, if you watch the temp folder, youāll see a single volume being processed at a time.
The new process iterates through the list of files to restore and retrieves all volumes that contain data for that file. If all the fileās data is in a single volume, your golden. If, on the other hand, the data is spread over many volumes, youāre going to need to retrieve all those volumes. Say a volume contains a single 128KB (I think?) block from your file, the rest of the volume is basically hoping a file it has data for will be processed before it gets evicted. At 50MB/volume and 128KB/block, a single volume could potentially have data for 400 files. With large backups and small cache, you can see the problem.
Interestingly, the cache eviction policy already has an āevict if all files with data in this volume have been processedā step which operates even when the cache is barely used, but the cache size takes precedence and triggers the LRU qualification once exceeded.
I also tried the legacy restore process (ārestore-legacy=true) which came in at about 3.5hrs, miles ahead of the default restore configuration for this particular backup, but itās easy to see why the new process, an impressive improvement, was implemented.
I know this is a long post but Iāll also add that I think the new restore process is definitely worthwhile but I think it needs to be more aggressive in itās cache allocation, when required, and it would be good if there was some indication of when the cache was exhausted. Alternatively, some level of auto-tuning could be useful too.
Thanks for reading.
). This in turn resulted in temp stores being filled up, which lead to the introduction of the cache size limiting option.