Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions devops/devops.rst
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ If you plan to use Conan in production in your project, team, or organization, t
using_conancenter
devops_local_recipes_index
backup_sources/sources_backup
download_cache
metadata
versioning/versioning
save_restore
Expand Down
40 changes: 40 additions & 0 deletions devops/download_cache.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
.. _devops_download_cache:

Download Cache
==============

The Conan download cache is a feature intended to cache downloaded artifacts and recipe files, avoiding recurrent downloads of the same files. This is especially useful in Continuous Integration (CI) environments where multiple jobs might require downloading the same packages, or for developers that switch environments frequently.

Configuration
-------------

The download cache is enabled by setting the ``core.download:download_cache`` configuration in the :ref:`global.conf<reference_config_files_global_conf>` file.

.. code-block:: text
:caption: global.conf

core.download:download_cache = /path/to/download_cache

This configuration points to an absolute path to a folder where the Conan packages and downloaded files will be stored *compressed* as they come from the remote server.

Conan has two configuration variables related to the download cache:

- ``core.download:download_cache``: To cache Conan artifacts (like ``conan_package.tgz`` and ``conan_export.tgz``) downloads.
- ``tools.files.download:download_cache``: To cache user downloads performed via the :ref:`download()<conan_tools_files_download>` or :ref:`get()<conan_tools_files_get>` tools in recipes.

By default, ``tools.files.download:download_cache`` defaults to the value of ``core.download:download_cache``. Thus, it is only necessary to define ``core.download:download_cache`` to enable caching for both Conan packages and user recipe downloads. If a different location is desired for user downloads, ``tools.files.download:download_cache`` can be explicitly set.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't seem correct, this doesn't exist. The conf is core.sources:download_cache and it doesn't default to core.download:download_cache.


Usage
-----

Once enabled, every time Conan needs to download a package artifact or a user file, it will first check if the file is present in the cache folder. If it is, it will be copied from the cache to the Conan cache or recipe folder, avoiding the network request. If it is not, it will be downloaded from the remote server and a copy will be stored in the download cache folder.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Once enabled, every time Conan needs to download a package artifact or a user file, it will first check if the file is present in the cache folder. If it is, it will be copied from the cache to the Conan cache or recipe folder, avoiding the network request. If it is not, it will be downloaded from the remote server and a copy will be stored in the download cache folder.
Once enabled, every time Conan needs to download a package artifact or a user file, it will first check if the file is present in the cache folder. If it is, it will be copied from the download cache to the Conan cache or recipe folder, avoiding the network request. If it is not, it will be downloaded from the remote server and a copy will be stored in the download cache folder.


The download cache is concurrency safe. Multiple concurrent Conan processes can share the same download cache folder simultaneously.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The download cache is concurrency safe. Multiple concurrent Conan processes can share the same download cache folder simultaneously.
The download cache is concurrency safe. Multiple concurrent Conan processes can share the same download cache folder simultaneously. The synchronization uses system file locks, so distributed synchronization is not guaranteed.


Download cache vs Backup sources
--------------------------------

While both the download cache and the :ref:`backup sources<conan_backup_sources>` features deal with caching downloaded files, they serve different purposes:

- The **download cache** is a local filesystem cache. Its main purpose is to speed up operations and save bandwidth by keeping a local compressed copy of the downloaded files. It is volatile and can be safely cleared at any time.
- The **backup sources** feature is designed to upload third-party source files (like ``.tar.gz`` from GitHub releases) to your own infrastructure (like an Artifactory server) to ensure traceability and reproducibility in case the original URLs go down.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- The **backup sources** feature is designed to upload third-party source files (like ``.tar.gz`` from GitHub releases) to your own infrastructure (like an Artifactory server) to ensure traceability and reproducibility in case the original URLs go down.
- The **backup sources** feature is designed to upload third-party source files (like ``.tar.gz`` from GitHub releases) to your own infrastructure (like an Artifactory server) to ensure traceability and reproducibility in case the original URLs go down. This feature uses the download cache functionality to store the downloaded sources, defaulting to a folder inside the Conan packages cache, but it can be also explicitly defined with the ``core.sources:download_cache`` configuration.

1 change: 1 addition & 0 deletions reference/config_files/global_conf.rst
Original file line number Diff line number Diff line change
Expand Up @@ -423,6 +423,7 @@ core.download:download_cache

Absolute path to a folder where the Conan packages will be stored *compressed*.
This is useful to avoid recurrent downloads of the same packages, especially in CI.
Please check the :ref:`download cache <devops_download_cache>` dedicated section for more information.

.. code-block:: text
:caption: *global.conf*
Expand Down
4 changes: 2 additions & 2 deletions reference/config_files/profiles.rst
Original file line number Diff line number Diff line change
Expand Up @@ -504,8 +504,8 @@ In this case, whatever version of ``cmake`` declared in recipes, will be replace

* This section should be added to the profile whose context is the one that requires the tool, i.e., if the
tool is required in the host context, then it should be added to the host profile, so that the requirement
itself can be replaced.

itself can be replaced. For example, if a ``zlib`` recipe in the host context has a ``tool_requires("cmake/xxx")``, a ``replace_tool_requires`` in the **host profile** will replace it.
* If what you want to replace are transitive dependencies of the tools that live inside ``tool_requires`` packages, those live in the **build context**. To replace them, you must add the replacements to the **build profile**. Both ``[replace_requires]`` and ``[replace_tool_requires]`` in the build profile will affect the build context in the same way, replacing ``requires`` and ``tool_requires`` of the tools themselves.

.. _reference_config_files_profiles_platform_requires:

Expand Down