sui-tool download-formal-snapshot fails due to request throttling

Technical Support
1

Hi all,

I’m trying to download a mainnet snapshot using sui-tool, but I keep running into issues that prevent me from completing the process. Here’s the command I’m using:

sui-tool download-formal-snapshot \
  --latest \
  --genesis /home/xdl/sui-fullnode/genesis.blob \
  --network mainnet \
  --snapshot-bucket s3://mysten-mainnet-formal/ \
  --snapshot-bucket-type s3 \
  --path /var/sui/db \
  --num-parallel-downloads 50

The problem is that I often hit request limits, and the download fails with an error like this:

Failed to get file MANIFEST with error: Generic { store: "S3", source: Reqwest { retries: 10, max_retries: 10, elapsed: 14.776050894s, retry_timeout: 180s, source: reqwest::Error { kind: Request, url: "http://169.254.169.254/latest/api/token", source: hyper_util::client::legacy::Error(Connect, TimedOut) } } }

After this happens, the tool exits. When I restart it, it deletes all previously downloaded data, which makes it impossible to finish the snapshot download under throttling conditions.

Is there a way to:

  • Resume downloads without starting over?
  • Avoid or work around the S3 rate limiting?

Any help would be greatly appreciated!