Source: staged

The staged push source allows the loading of content from any locally accessible directory referred to as the staging area. Content types and destinations are defined by the directory structure in the staging area, and additional metadata for content can be provided in a YAML or JSON file.

Supported content types:

  • RPMs

  • Advisories

  • Generic files

  • Content Gateway metadata

  • Complete directories

  • Amazon Machine Images (AMIs)

  • comps.xml files (yum repo metadata)

  • modulemd YAML streams (yum repo metadata)

  • productid files (yum repo metadata)

Staged source URLs

A staged source is obtained by URL:

staged:stagedir1[,stagedir2[,stagedir3[,...]]]

One or more staging directories must be provided (usually, exactly one). Each staging directory must use the Staging structure defined below.

Here are some examples of staged source URLs:

  • staged:/mnt/volume/my-staging-area - a single staging directory

  • staged:/mnt/shared/stage1,/mnt/shared/stage2 - two staging directories

Staging structure

Here is a brief overview of the structure of a staging directory:

root/
root/staged.yaml  # optional
root/destination/
root/destination/COMPS/*.xml
root/destination/ERRATA/*
root/destination/FILES/*
root/destination/CGW/*
root/destination/PRODUCTID/*
root/destination/MODULEMD/*
root/destination/RPMS/*.rpm
root/destination/AWS_IMAGES/*
root/destination/RAW/*

The staging directory consists of:

  • an optional file including some metadata for files in the staging area

  • a directory for each “destination” of content in this staging area

  • subdirectories for each type of content

  • files containing the content to be pushed or manipulated

These are explained in further detail below.

root/staged.yaml

This file provides additional metadata on the content in the staging area.

It is generally mandatory for file types which do not embed their own metadata (such as FILES and AWS_IMAGES), and is unused for file types such as RPMS which have all necessary metadata built-in to the file format.

Here’s an example to give an idea of the metadata controlled by this file:

header:
   version: "0.2" # current schema version - don't change

payload:
   files:
      # desired name, doesn't need to be same as staged filename
      filename: "RHEL-6.7-20150429.0-Client-i386-boot.iso"

      # path relative to staging directory
      relative_path: "rhel-6-desktop-beta-isos__6Client__i386/FILES/RHEL-6.7-20150429.0-Client-i386-boot.iso"

      # SHA256 checksum
      sha256sum: b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c

      # Additional attributes
      attributes:
         # human-readable description
         description: "Red Hat Enterprise Linux 6.7 Boot DVD"

Schema: staged provides a complete reference of the fields which can be set by this file.

For historical reasons, the file may also be provided as JSON and named pub-mapfile.json.

root/destination/

A directory exists for each discrete destination of content in this staging area.

The meaning of destination depends on the context in which the staging area is used. For example, if this staging area is being used to publish content into a Pulp server, each destination will be a Pulp repository name. This library does not enforce any semantics on these destination names.

Note that, if a single piece of content should be published to multiple destinations, the content needs to be placed in multiple destination directories. Symlinks are often used for this to avoid duplication.

root/destination/COMPS/*.xml

Each file in COMPS should be a comps XML file in the yum repo metadata format, defining package groups.

Will yield instances of CompsXmlPushItem.

root/destination/ERRATA/*

Each file in ERRATA should be a YAML or JSON dump containing advisory metadata.

The format is essentially the same as that produced by Errata Tool’s XML-RPC APIs. Schema: errata formally documents the schema.

Will yield instances of ErratumPushItem.

root/destination/FILES/*

Each file in FILES can be any type of file, to be published verbatim. In practice, this is often used for binary files such as ISO disc images or installers.

Files in this directory must have metadata included in staged.yaml.

For historical reasons, this directory may also be named ISOS.

Will yield instances of FilePushItem.

root/destination/CGW/*

Each file in CGW should be a valid YAML containing a list of definitions for objects in Content Gateway.

Will yield instances of CGWPushItem.

root/destination/PRODUCTID/*

Each file in PRODUCTID should be a PEM-formatted product certificate, as used by Satellite 6 / RHSM.

Will yield instances of ProductIdPushItem.

root/destination/MODULEMD/*

Each file in MODULEMD should be a valid YAML stream containing modulemd and/or modulemd-defaults documents.

Will yield instances of ModuleMdPushItem.

root/destination/RPMS/*.rpm

Each file in RPMS should be an RPM. Various metadata will be parsed from RPM headers.

For historical reasons, this directory may also be named SRPMS. It is not required that source RPMs are placed in the SRPMS directory.

Will yield instances of RpmPushItem.

root/destination/AWS_IMAGES/*

Each file in AWS_IMAGES should be an EBS snapshot for use as an Amazon Machine Image (AMI).

Files in this directory must have metadata included in staged.yaml.

Will yield instances of AmiPushItem.

root/destination/RAW/*

RAW should generally be interpreted as a request to recursively publish the entire directory tree as-is. Each file in RAW can be any type of file, to be published verbatim. In practice, RAW is used for kickstart trees, ostrees, and origin-only content.

Will yield instances of DirectoryPushItem.

Python API reference

class pushsource.StagedSource(url, threads=4, timeout=3600)[source]

Uses a directory with a predefined layout (a “staging directory”) as the source of push items.

__init__(url, threads=4, timeout=3600)[source]

Create a new source.

Parameters:

  • url (list[str]) –

    URL(s) of locally accessible staging directories, e.g. "/mnt/staging/my-content-for-push".

    These directories must follow the documented layout for staging areas.

  • threads (int) – Number of threads used for concurrent loading of files.

  • timeout (int) – Number of seconds after which an error is raised, if no progress is made during each step.