Syncing Files from a Webserver or an S3 Bucket

The pulp-manifest tool is designed to generate PULP_MANIFEST files for the Pulp File plugin. pulp-manifest has been recently significantly enhanced with support for S3 buckets. This feature allows users to generate a PULP_MANIFEST directly from content in an S3 bucket. It's exciting to note that this valuable addition was a community contribution from ozanunsal. This post is about the pulp-manifest tool. Sync and Publish workflows for File repositories are documented separately.

How the S3 Support Works

To generate a PULP_MANIFEST for an S3 bucket, you simply use the command with an S3 path:

pulp-manifest s3://bucket-name/path/to/prefix/

When generating the manifest for S3 content, the tool computes a SHA256 digest by downloading the file content. Users can exclude specific files or directories that match a glob pattern by using the --exclude option.

pulp-manifest s3://bucket-name/path --exclude '*.log'