A big server makes a scan faster. It does not make it cheaper.
The scan needs a small machine, in the correct city, for a few hours. A large server that bills each month is the wrong shape for that work.
More cores buy speed, not savings
The scan is CPU-bound. It decodes about 40 to 60 MB/s on each vCPU. A machine with 4 times the cores finishes 4 times sooner, and costs 4 times more for each hour. The cost for each TB stays the same.
| vCPU | Hours for each TB |
|---|---|
| 4, measured | 1.1 to 1.7 |
| 8, projected | 0.6 to 0.9 |
| 16, projected | 0.3 to 0.4 |
The archive does not fit on your disk, and it does not have to
The archive is more than 600 TB. The engine streams each epoch through memory and writes nothing of it to disk. Our machine has 8 GB of RAM and a 75 GB disk. The disk holds only Parquet output, and each file uploads when it closes.
A server with tens of TB of NVMe gives you nothing here.
The location sets the speed
The archive origin is in Amsterdam, behind the Cloudflare CDN. From a machine in Amsterdam we measured about 20 Gbps. One large cloud provider tells customers to expect 300 to 500 Mbit/s for each server. At that rate the link, not the CPU, sets the speed, and an epoch takes several times longer.
Our machines run in Amsterdam only.
A monthly server bills when it is idle
A monthly server costs the same for 2 hours of work or for 700. It is cheaper than hourly machines only when it scans for most of the hours in the month.
Our machine exists for one job. We delete it when the job ends. If it hangs, a watchdog deletes it at twice the expected run time, and you get a credit.
Your server, or one request.
| Your own server | This service | |
|---|---|---|
| Cost when idle | The full monthly price | Nothing |
| Setup | Build the engine. Write the plugin, the filter and the decoder. | None. Select a preset, or send a filter and an IDL. |
| Hardware | Your choice. Bigger is not cheaper. | 4 dedicated vCPU, 8 GB RAM, in Amsterdam, for your job only |
| First result | After the setup, then the scan | After the scan: about 1 to 2 hours for each TB |
| Price | Machine, disk, your time, and every failed run | $2 for each TB of archive scanned (launch price; list price $5), firm before you pay |
| A failed run | You find it, fix it and pay again | A credit for a free re-run |
| Output | What you build | Decoded Parquet: instructions, events, inner instructions |
| Long ranges | One machine, one range at a time | Up to 13 TB in one job. Send several jobs for more. |
A machine is not a pipeline.
A server gives you CPU and a network port. The work is in the build, the filter, the decoder, the uploads and the watchdog. The worst errors in that work produce no error message.
When your own server is the correct choice
- You scan all day, every day.
- You need a live stream of new blocks, not history.
- You need stateful logic for each transaction that a filter cannot express.
- You already run Jetstreamer.
Skip the server.
Get a firm price for your range. The machine starts after you pay, and we delete it after the job.