This produces four .csv files which the biggest one is nearly 100 GB of data. Due to the size, I can't load it on a python jupyter notebook.
You can read the data into some database that does what you need, or use things like
grep and
split to get only the lines you need.
I am wondering what common programs and strategies that people use to process this data.
Can you start by sharing what you're trying to accomplish? I usually pipe the data through some Linux command line tools to get what I need, you'll find some examples in those topics: