Skip to content

docs: create dedicated documentation page for the download cache - #4470

Open
anushkagupta200615-jpg wants to merge 2 commits into
conan-io:develop2from
anushkagupta200615-jpg:develop2
Open

docs: create dedicated documentation page for the download cache#4470
anushkagupta200615-jpg wants to merge 2 commits into
conan-io:develop2from
anushkagupta200615-jpg:develop2

Conversation

@anushkagupta200615-jpg

Copy link
Copy Markdown
Contributor

Description

This PR addresses issue #4200 by adding explicit documentation for the Conan 2 download cache feature.

Previously, information regarding the download cache was only partially covered under the "backup sources" section and briefly mentioned in global_conf.rst. This PR introduces a dedicated page in the DevOps section (devops/download_cache.rst) to provide clear and central documentation.

Changes included:

  • Added download_cache.rst: Explains the download cache feature, how it operates, and how to configure it (core.download:download_cache and tools.files.download:download_cache).
  • Compared with Backup Sources: Clarified the distinct purposes of the local download cache versus the backup sources feature to avoid confusion.
  • Updated devops.rst: Registered the new documentation under the DevOps table of contents.
  • Updated global_conf.rst: Added a cross-reference link in the core.download:download_cache entry that points users directly to the new dedicated page for complete context.

Resolves #4200

@memsharded memsharded left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Many thanks for your contribution @anushkagupta200615-jpg

Overall it is looking good.

Please check the tools.files.download:download_cache issue, if this was generated with AI, please double check the output, this could be an hallucination or a confusion with Conan 1.

Also, there is a failure in the build:

/home/jenkins/workspace/conan_docs_PR-4470/devops/download_cache.rst:23:undefined label: 'conan_tools_files_download'

Comment thread devops/download_cache.rst
- ``core.download:download_cache``: To cache Conan artifacts (like ``conan_package.tgz`` and ``conan_export.tgz``) downloads.
- ``tools.files.download:download_cache``: To cache user downloads performed via the :ref:`download()<conan_tools_files_download>` or :ref:`get()<conan_tools_files_get>` tools in recipes.

By default, ``tools.files.download:download_cache`` defaults to the value of ``core.download:download_cache``. Thus, it is only necessary to define ``core.download:download_cache`` to enable caching for both Conan packages and user recipe downloads. If a different location is desired for user downloads, ``tools.files.download:download_cache`` can be explicitly set.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't seem correct, this doesn't exist. The conf is core.sources:download_cache and it doesn't default to core.download:download_cache.

Comment thread devops/download_cache.rst
Usage
-----

Once enabled, every time Conan needs to download a package artifact or a user file, it will first check if the file is present in the cache folder. If it is, it will be copied from the cache to the Conan cache or recipe folder, avoiding the network request. If it is not, it will be downloaded from the remote server and a copy will be stored in the download cache folder.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Once enabled, every time Conan needs to download a package artifact or a user file, it will first check if the file is present in the cache folder. If it is, it will be copied from the cache to the Conan cache or recipe folder, avoiding the network request. If it is not, it will be downloaded from the remote server and a copy will be stored in the download cache folder.
Once enabled, every time Conan needs to download a package artifact or a user file, it will first check if the file is present in the cache folder. If it is, it will be copied from the download cache to the Conan cache or recipe folder, avoiding the network request. If it is not, it will be downloaded from the remote server and a copy will be stored in the download cache folder.

Comment thread devops/download_cache.rst

Once enabled, every time Conan needs to download a package artifact or a user file, it will first check if the file is present in the cache folder. If it is, it will be copied from the cache to the Conan cache or recipe folder, avoiding the network request. If it is not, it will be downloaded from the remote server and a copy will be stored in the download cache folder.

The download cache is concurrency safe. Multiple concurrent Conan processes can share the same download cache folder simultaneously.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The download cache is concurrency safe. Multiple concurrent Conan processes can share the same download cache folder simultaneously.
The download cache is concurrency safe. Multiple concurrent Conan processes can share the same download cache folder simultaneously. The synchronization uses system file locks, so distributed synchronization is not guaranteed.

Comment thread devops/download_cache.rst
While both the download cache and the :ref:`backup sources<conan_backup_sources>` features deal with caching downloaded files, they serve different purposes:

- The **download cache** is a local filesystem cache. Its main purpose is to speed up operations and save bandwidth by keeping a local compressed copy of the downloaded files. It is volatile and can be safely cleared at any time.
- The **backup sources** feature is designed to upload third-party source files (like ``.tar.gz`` from GitHub releases) to your own infrastructure (like an Artifactory server) to ensure traceability and reproducibility in case the original URLs go down.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- The **backup sources** feature is designed to upload third-party source files (like ``.tar.gz`` from GitHub releases) to your own infrastructure (like an Artifactory server) to ensure traceability and reproducibility in case the original URLs go down.
- The **backup sources** feature is designed to upload third-party source files (like ``.tar.gz`` from GitHub releases) to your own infrastructure (like an Artifactory server) to ensure traceability and reproducibility in case the original URLs go down. This feature uses the download cache functionality to store the downloaded sources, defaulting to a folder inside the Conan packages cache, but it can be also explicitly defined with the ``core.sources:download_cache`` configuration.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems already in develop2, please rebase from develop2 branch, this should dissapear.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Explicitly document cache download

2 participants