Re: python script compare lines in 2 text files and output matches
i use windows so yes it has to be python
You can use bash utilities on Windows too. Or boot your computer from an Ubuntu LIVE DVD just to do this.Quote
yes it is very very slow for files above 1,000,000 lines
when i want to compare a files with +10,000,000 lines to other 2,000,000 lines (i had to cancel and close the script
)
It took me a while to create 2 test-files with 50,000 Bitcoin addresses, so I just copied the same addresses to the same files to make it 10 million and 2 million lines per file.when i want to compare a files with +10,000,000 lines to other 2,000,000 lines (i had to cancel and close the script
)The comm-code above took 9 seconds to find all matches (and my PC is not very fast). I strongly suggest to use the proper tools for the job
Update: it used 1.5 GB RAM to do this. If you have much more data to compare, it might reduce memory load if you sort the files first.