Re: the fastest possible way to mass-generate addresses in Python

Quote from: arulbero on Today at 03:08:58 PM

I don't have a fast 'pub_to_addr' function

from 12.3 s to 3m and 9s to generate 16.4 M of addresses, 16.4 M / 190 s = about 86k addresses/s.

12 seconds to generate 16.4 M public keys, almost 3m to compute 16.4 M addresses.

Code:

I had already thought that halfway. Now with your updated version which also generates addresses it's much slower, the result is clearly different than just calculating the pubkey as you showed in your initial version.
I get about 70,000 addresses/sec with your updated version:
[code]
$ time python3 new_gen_batches.py > addresses.out

Quote

real   0m13,378s
user   0m13,347s
sys   0m0,012s

I realized that you output tuples of addresses using list(map). I suggest to adjust your program so it outputs a single address on each line so we compares apples by apples. I am not sure if the list output affects performance in some way, that's why I am pointing to it.

The example I originally showed using iceland2k14/libsecp256k1 was about half the speed. With it I can generate 1 million addresses using all cores (in my case it was 16 cores) and write to the file in 4.8 seconds. I have modified my initial program so that you can now also configure the cores under which the program is executed. So one can select specifically "1" as value, so that we also compare apples with apples.

Code:

#!/usr/bin/env python3
# 2022/Dec/26, citb0in
import concurrent.futures
import os
import numpy as np
import fastecdsa.keys as fkeys
import fastecdsa.curve as fcurve
import secp256k1 as ice

# how many cores to use
num_cores = 1
#num_cores = os.cpu_count()

# Set the number of addresses to generate
num_addresses = 1000000

# Define a worker function that generates a batch of addresses and returns them
def worker(start, end):
# Generate a NumPy array of random private keys using fastecdsa
private_keys = np.array([fkeys.gen_private_key(fcurve.P256) for _ in range(start, end)])

# Use secp256k1 to convert the private keys to addresses
thread_addresses = np.array([ice.privatekey_to_address(2, True, dec) for dec in private_keys])

return thread_addresses

# Use a ProcessPoolExecutor to generate the addresses in parallel
with concurrent.futures.ProcessPoolExecutor() as executor:
# Divide the addresses evenly among the available CPU cores
addresses_per_core = num_addresses // num_cores

# Submit a task for each batch of addresses to the executor
tasks = []
for i in range(num_cores):
start = i * addresses_per_core
end = (i+1) * addresses_per_core
tasks.append(executor.submit(worker, start, end))

# Wait for the tasks to complete and retrieve the results
addresses = []
for task in concurrent.futures.as_completed(tasks):
addresses.extend(task.result())

# Write the addresses to a file
np.savetxt('addresses_1M_with_singleCore.txt', addresses, fmt='%s')

Running this under one single core, now I get 26,5 sec for 1 million generated addresses. This is a rate of 37,735 keys/second. I am curious to see your program using all available cores.[/code]