Ninjastic
PostEdited versionsQuotes to this post
Post
Topic
Board Project Development
Re: List of all Bitcoin addresses ever used (OP rewritten, updates work again)
by
PrimeNumber7
on 22/12/2020, 03:28:37 UTC
Quote from: LoyceV on November 30, 2020, 12:26:10 PM
Quote from: PrimeNumber7 on November 29, 2020, 10:23:57 PM
I had used AWS as an example because I believed you used it for some of your other projects.
Correct, loyce.club runs on AWS (sponsored).

Quote
Yes, transferring data to the internet is very expensive. You can use a CDN (content delivery network) to reduce costs a little bit. 5 TB of data is a lot.
I highly doubt I'd find a cheaper deal Cheesy I hope not to use the full 5 TB though, I expect some overselling and don't want to push it to the limit.
I am not sure what level of access you have to the AWS account sponsoring your site. However, it is possible to setup a storage bucket so that anyone can access it, but that the requestors IP address is among the IP addresses of the same region the files are stored in. See this stack overflow discussion. You can also setup the storage bucket such that the requestor pays for egress traffic.


Quote from: LoyceV on November 30, 2020, 12:26:10 PM
Quote
The longer the list, the longer it will take to sort one additional line.
At some point a database might beat raw text sorting, but for now I'm good with this Smiley
Using a database will not solve this problem. There are some things a DB can do to make sorting go from O^2 to O^2/n, but this is still exponential growth.

You make the argument that your input size is sufficiently small such that having exponential complexity is okay, and you may have a point.



Quote from: LoyceV on November 30, 2020, 12:26:10 PM
Quote from: PrimeNumber7 on November 30, 2020, 03:42:59 AM
I was under the impression that traffic out of the AWS network (for AWS) will count as egress traffic, and will be billed accordingly.
AWS charges $0.09/GB, and especially since this one is sponsored, I don't want to abuse it. I love how stable the server is though, it has never been down.
AWS is very reliable. I would not expect much downtime when using AWS or other major cloud providers. Egress traffic is very expensive though.

Quote from: LoyceV on December 15, 2020, 11:28:03 AM
Downloads are fast, I've seen 20-100 MB/s. Enjoy Smiley

This works out to approximately a 24-minute download. I measured a download speed of ~125 Mbps using a colab instance.