When your data doesn’t fit in memory: the basic techniques by Itamar Turner-Trauring Last updated 06 Jan 2023, originally created 08 Nov 2019 You’re writing software that processes data, and it works fine when you test it on a small sample file. But when you load the real data, your program crashes. The problem is that you don’t have enough memory—if you have 16GB of RAM, you can’t load a 100GB fi
![When your data doesn’t fit in memory: the basic techniques](https://cdn-ak-scissors.b.st-hatena.com/image/square/2a2f5e3d3a486fe74269acde754ce8ab1a0d579e/height=288;version=1;width=512/https%3A%2F%2Fpythonspeed.com%2Fassets%2Ftitles%2Fdata-doesnt-fit-in-memory.png)