You can use the Lambda Cloud Storage feature to save:
- Large datasets that you don’t want to re-upload every time you start an instance
- The state of your system, including software packages and configurations
Preserving the state of your system
For saving the state of your system, including:
- Packages installed system-wide using
apt-get - Python packages installed using
pip - conda environments
We recommend creating containers using Docker or other software for creating containers.
You can also create a script that runs the commands needed to re-create your system state. For example:
sudo apt install PACKAGE_0 PACKAGE_1 PACKAGE_2 && \
pip install PACKAGE_3 PACKAGE_4 PACKAGE_5
Run the script each time you start an instance.
If you only need to preserve Python packages and not packages installed system-wide, you can create a Python virtual environment.
You can also create a conda environment.