community-push

Upload an AMI to S3 and update RHSM.

usage: pubtools-marketplacesvm-community-push [-h] [--debug]
                                              [--rhsm-url RHSM_URL]
                                              [--rhsm-cert RHSM_CERT]
                                              [--rhsm-key RHSM_KEY]
                                              [--starmap-url STARMAP_URL]
                                              [--repo REPO]
                                              [--repo-file REPO_FILE]
                                              [--offline]
                                              [--credentials CREDENTIALS]
                                              [--azure-allow-draft-push]
                                              [--skip SKIP] [--pre-push]
                                              [--beta]
                                              [--container-prefix CONTAINER_PREFIX]
                                              source [source ...]

Positional Arguments

source

Source(s) of content to be pushed

Named Arguments

--debug, -d

Show debug logs; can be provided up to three times to enable more logs

Default: 0

--skip

skip given comma-separated sub-steps

Default: []

--pre-push

Pre-push mode: do as much as possible without making content available to end-users, then stop. May be used to improve the performance of a subsequent full push.

Default: False

--beta

Ship beta images instead of GA

Default: False

--container-prefix

prefix to storage container for upload

Default: “redhat-cloudimg”

RHSM service

--rhsm-url

Base URL of the RHSM API

--rhsm-cert

RHSM API certificate path (or set RHSM_CERT environment variable)

--rhsm-key

RHSM API key path (or set RHSM_KEY environment variable)

StArMap Service

--starmap-url

Base URL for the StArMap server.

Default: “https://starmap.engineering.redhat.com”

--repo

Override the StArMap mappings for push items. e.g: {‘name’: ‘foo’, ‘workflow’: ‘stratosphere’: [{‘aws-na’: {‘destinations’ [{‘destination’: ‘c39fd…’}, …]},…},…}]

Default: []

--repo-file

Override the StArMap mappings for push items with yaml file.

--offline

Do not connect to a real StArMap server, use a mock session instead. It requires –repo to be set.

Default: False

Cloud Service

--credentials

Path to the credentials files separated by comma or base64 encoded credentials separated by comma (or set CLOUD_CREDENTIALS environment variable)

Default: “”

--azure-allow-draft-push

Allow publishing to draft offers on Azure.

Default: False

Example

Push Example

A typical invocation of push would look like this:

pubtools-marketplacesvm-community-push \
  --starmap-url https://starmap.example.com \
  --credentials PATH_TO_CREDS_1.json,PATH_TO_CREDS_2.json \
  koji:https://koji.example.com/kojihub?vmi_build=build-example1,build-example2

using a staged sourced

pubtools-marketplacesvm-community-push \
  --starmap-url https://starmap.example.com \
  --credentials PATH_TO_CREDS_1.json,PATH_TO_CREDS_2.json \
  staged:/direct/path/to/folder

Credentials

This section describes the supported values for the --credentials command line argument.

Expected format

Overall format

Each credential file must be in the following format:

{
  "marketplace_account": "CLOUD_ALIAS",
  "auth":
  {
      "KEY": "VALUE", // ...
  }
}

The supported values for the key marketplace_account are described below:

Marketplace accounts (push)

Required when using the “marketplace” or “combined” workflow

• aws-na

• aws-emea

• azure-na

• azure-emea

Storage accounts (community-push)

Required when using the “community” or “combined” workflow

• aws-us-storage

• aws-us-gov-storage

• aws-china-storage

Auth format

The KEYs and VALUEs for the auth object are specific for each Cloud Provider implementation.

AWS Format

The following properties are supported for AWS auth:

• AWS_IMAGE_ACCESS_KEY: The AWS access key for uploading the images and import them as AMIs

• AWS_IMAGE_SECRET_ACCESS: The AWS secret key for uploading the images and import them as AMIs

• AWS_MARKETPLACE_ACCESS_KEY: The AWS access key for publishing a new product version with its AMI into the marketplace

• AWS_MARKETPLACE_SECRET_ACCESS: The AWS secret key for publishing a new product version with its AMI into the marketplace

• AWS_ACCESS_ROLE_ARN: IAM role Amazon Resource Name (ARN) used by AWS Marketplace to access the provided AMI. For details about creating and using this ARN, see Giving AWS Marketplace access to your AMI in the AWS Marketplace Seller Guide.

• AWS_GROUPS: The default sharing accounts for uploading and importing AMIs

• AWS_SNAPSHOT_ACCOUNTS: The default sharing snapshot accounts for uploading and importing AMIs

• AWS_REGION: The default region for uploading and importing AMIs

• AWS_S3_BUCKET: The S3 bucket name to upload the raw VM images

Example:

{
  "AWS_IMAGE_ACCESS_KEY": "******************",
  "AWS_IMAGE_SECRET_ACCESS": "***************",
  "AWS_MARKETPLACE_ACCESS_KEY": "******************",
  "AWS_MARKETPLACE_SECRET_ACCESS": "******************",
  "AWS_ACCESS_ROLE_ARN": "******************",
  "AWS_GROUPS": [],
  "AWS_SNAPSHOT_ACCOUNTS": [],
  "AWS_REGION": "us-east-1",
  "AWS_S3_BUCKET": "******************"
}

Azure Format

The following properties are supported for AWS auth:

• AZURE_TENANT_ID: The tenant ID for publishing a new product version

• AZURE_PUBLISHER_NAME: The publisher name for publishing a new product version

• AZURE_CLIENT_ID: The Azure’s AD Client ID to communicate with Microsoft APIs

• AZURE_API_SECRET: The Azure’s AD Client Secret to communicate with Microsoft APIs

• AZURE_STORAGE_CONNECTION_STRING: The Azure Storage Account connection string to upload VHD images

{
  "AZURE_TENANT_ID": "******************",
  "AZURE_PUBLISHER_NAME": "*************",
  "AZURE_API_SECRET": "******************",
  "AZURE_CLIENT_ID": "*******************",
  "AZURE_STORAGE_CONNECTION_STRING": "******************"
}

Examples

Example for AWS NA:

{
  "marketplace_account": "aws-na",
  "auth":
  {
      "AWS_IMAGE_ACCESS_KEY": "******************",
      "AWS_IMAGE_SECRET_ACCESS": "***************",
      "AWS_MARKETPLACE_ACCESS_KEY": "******************",
      "AWS_MARKETPLACE_SECRET_ACCESS": "******************",
      "AWS_ACCESS_ROLE_ARN": "******************",
      "AWS_GROUPS": [],
      "AWS_SNAPSHOT_ACCOUNTS": [],
      "AWS_REGION": "us-east-1",
      "AWS_S3_BUCKET": "******************"
  }
}

Example for Azure NA:

{
  "marketplace_account": "azure-na",
  "auth": {
      "AZURE_TENANT_ID": "******************",
      "AZURE_PUBLISHER_NAME": "*************",
      "AZURE_API_SECRET": "******************",
      "AZURE_CLIENT_ID": "*******************",
      "AZURE_STORAGE_CONNECTION_STRING": "******************"
  }
}

CLI parameter format

Once the credentials are properly crafted, then can be passed to the tooling in one of the following ways:

Credentials as files

It’s possible to storage each credential into a single JSON file and passing the path for all files using a comma delimited string for the parameter --credentials:

--credentials PATH_TO_CREDS_1.json,PATH_TO_CREDS_2.json,PATH_TO_CREDS3.json

The tooling will split the string by comma (,) and open/parse each credential file for loading its credentials.

Note that the parameter --credentials support only a single string argument. Avoid using space as bash may consider it two or more arguments.

Credentials as list of base64 string

Another way to pass the credentials is to encode each credential JSON into a base64 string and passing it as a list of encoded strings separated by comma (,).

Since comma will never be used on base64 encoding the tooling will first split the string into multiple substrings of base64 encoding, each one representing a single credentials.

## Convert each creds to a base64 encoded string
for i in PATH_TO_CREDS_1.json PATH_TO_CREDS_2.json PATH_TO_CREDS3.json; do
  cat $i | base64
done

## Join the output into a single line and then

--credentials BASE64_STRING_1,BASE64_STRING_2,BASE64_STRING_3

Note that the parameter --credentials support only a single string argument. Avoid using space as bash may consider it two or more arguments.

Examples

Using the path mode:

pubtools-marketplacesvm-push \
--credentials creds/aws-na.json,creds/aws-emea.json,creds/azure-na.json,creds/azure-emea.json \
...

Using the base64 mode:

pubtools-marketplacesvm-push \
--credentials ewogICAgIm1hcmtWNlX[...],ewogICAgIm1hcmtldH[...],ewogICAgIm1hcmtl[...],ewogICAgIm1hcmtldHBsYW[...] \
...

Supported Sources

The tooling supports the KojiSource , ErrataSource , and StagedSource from pushsource.

All the VM images in the given source path will be uploaded to the corresponding cloud provider and published to its marketplace.

The expected parameter for the VM images is vmi_build:

Example:

koji:https://koji.example.com/kojihub?vmi_build=build-example1,build-example2

staged:/direct/path/to/folder

Destination Mapping

The destination mapping is given by an internal service named StArMap throught the Starmap Client library.

The tool expects a valid endpoint to the StArMap service and/or a JSON string containing the list of mappings using the parameter --repo. Instead of using --repo you can instead use --repo-file and provide a yaml file to be parsed.

The following sections will cover all the details about the destination mappings.

NOTE: This tooling supports the input format for StArMap APIv2 only.

Mapping request modes

Online mode

In this mode the tooling will use the Starmap Client library to connect into a StArMap server to provide the mappings for each VMIPushItem into the proper marketplace destination.

To initiate the tooling into the “online mode” it’s just a matter of providing a valid StArMap server URL

--starmap-url https://starmap.base.url

The workflow will be executed like this:

# For each incoming "VMIPushItem" get its destination data from StArMap
# using the image name and version
curl -S https://starmap.base.url/api/v2/query?name=IMG_NAME&version=IMG_VER

# If mapping not defined => fail; else
# Upload and publish the VMs into the proper destinations
...

Offline mode

In this mode the tooling will use the Starmap Client library to load the mappings from the --repo argument and use these mappings to upload and publish each VMIPushItem into the proper marketplace destination. In this mode it will fail any VMIPushItem which doesn’t have a mapping specified on --repo.

Note that no network calls will be made to the StArMap server in this mode, so the --starmap-url can be any fake url.

To initiate the tooling into the “offline mode” you must specify the --repo mappings as well as the flag --offline:

--starmap-url https://some.fake.url \
--repo "{...}" \
--offline

The workflow will be executed like this:

# For each incoming "VMIPushItem" get its destination data from --repo
# using the image name and the desired workflow
# If mapping not defined => fail; else
# Upload and publish the VMs into the proper destinations
...

Hybrid mode

In this mode the tooling will use the Starmap Client library to load the mappings from the --repo argument and use these mappings to upload and publish each VMIPushItem into the proper marketplace destination. Howevever, if a certain mapping is not defined on repo it will request the data from the StArMap server.

Note that no network calls will be made to the StArMap server in this mode, so the --starmap-url can be any fake url.

To initiate the tooling into the “offline mode” you must specify the --repo mappings as well as valid StArMap server URL for fallback:

--starmap-url https://starmap.base.url \
--repo "{...}"

The workflow will be executed like this:

# For each incoming "VMIPushItem" get its destination data from --repo
# using the image name.
# If mapping not defined call the StArMap server
curl -S https://starmap.base.url/api/v2/query?name=IMG_NAME&version=IMG_VER
# If mapping is still not defined => fail; else
# Upload and publish the VMs into the proper destinations
...

Mapping Format

The value for the --repo argument expects a JSON list which represents a QueryResponseContainer data with one or more QueryResponseEntity objects as elements.

The StArMap APIv2 QueryResponseContainer has the following format:

[
    {
        "name": "str",
        "cloud": "str",
        "workflow": "str",
        "mappings": {"obj"},
        "billing-code-config": {},
        "meta": {}
    },
    {"..."}
]

• name: The name from the image’s NVR. It will be used to match the propre VM artifact.

• cloud: The name of the targeted cloud marketplace. It currently supports aws or azure.

• workflow: Can be either community or stratosphere and it’s used to match the tooling workflow.

• mappings: The clouds object with the marketplace name and its destinations and metadata

• billing-code-config (optional): An entry which is only mandatory for the community workflow. It describes the billing code configuration to be applied to the AMIs.

• meta (optional): An entry which aims to set the global metadata, which has less precedence than lower meta keys.

The mappings object has the following format:

{
   "MARKETPLACE_NAME": {
   "destinations": [
      {
        "architecture": "str",
        "destination": "str",
        "meta": {"obj"},
        "overwrite": "bool",
        "provider": "str",
        "restrict_version": "bool",
        "restrict_major": "bool",
        "restrict_minor": "bool",
        "tags": {}
       },
       {"..."}
   ],
   "provider": "str",
   "meta": {}
}

• MARKETPLACE_NAME: A string matching a single marketplace_account from credentials. E.g. aws-na. It’s used the retrieve the proper credentials to upload and publish into the given marketplace.

• destinations: A list of destination objects, which are described in the next sub-section.

• provider (optional): An entry for defining the provider. It’s mandatory of the community workflow only.

• meta (optional): An entry which aims to set the mid level metadata, which has higher precedence than global meta and lower than destination’s one.

Destination objects format:

• architecture: A string representing the VM image architecture to publish. E.g. x86_64.

• destination: A string representing a offer/plan destination for the image to be published. E.g. offer_1/plan_1.

• meta (optional): An object with any key/values which may be threated as complementary metadata for publishing on marketplaces. It has higher precedence than upper levels.

• overwrite: A boolean indicating whenever the image should replace the existing version (true) in the marketplace.

• provider (optional): A string meant to be used only on community workflow. It indicates the provider name (AWS, AGOV, ACN). It receives the value from mappings.provider.

• restrict_version: A boolean for AWS marketplace only which indicates whether a previous version need to be restricted after publishing.

• restrict_major (optional): An optional boolean indicating whether to restrict a major version. Only applicable if restrict_version is set to true.

• restrict_minor (optional): An optional boolean indicating whether to restrict a minor version. Only applicable if restrict_version is set to true.

• tags (optional): An object with any key/values to be applied as tags once the VM images are uploaded.

Examples

An Azure mapping for RHEL using the architecture x86_64:

curl -S 'https://starmap.base.url/api/v2/query?name=rhel-azure&version=8.0'

[
    {
        "name": "rhel-azure",
        "cloud": "azure",
        "workflow": "stratosphere",
        "meta": {"generation": "V2"}
        "mappings": {
            "azure-na": {
                "destinations": [
                    {
                        "architecture": "x86_64",
                        "destination": "rh-rhel-test/rh-rhel8-internal",
                        "overwrite": false,
                        "restrict_version": false,
                    },
                ],
                "meta": {"support_legacy": true}
            },
        }
    }
]

An AWS mapping for RHEL using the architecture x86_64:

curl -S 'https://starmap.base.url/api/v2/query?name=rhel-ec2'

[
    {
        "name": "rhel-ec2",
        "cloud": "aws",
        "workflow": "stratosphere",
        "mappings": {
            "aws-na": {
                "destinations": [
                    {
                        "architecture": "x86_64",
                        "destination": "d87bcebf-9cf4-47f5-9b5b-5470d4490f3d",
                        "overwrite": false,
                        "restrict_version": true,
                    }
                ]
            }
        },
        "meta": {
            "description": "Provided by Red Hat, Inc.",
            "ena_support": true,
            "marketplace_entity_type": "AmiProduct",
            "recommended_instance_type": "m5dn.2xlarge",
            "release": {
                "product": "Red Hat Enterprise Linux",
                "type": "ga",
                "variant": "Server"
            },
            "release_notes": "https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/{major_version}/html/{major_minor}_release_notes/index",
            "root_device": "/dev/sda1",
            "scanning_port": 22,
            "security_groups": [
                {
                    "from_port": 22,
                    "ip_protocol": "tcp",
                    "ip_ranges": [
                        "0.0.0.0/0"
                    ],
                    "to_port": 22
                }
            ],
            "sriov_net_support": "simple",
            "usage_instructions": "Access your instance via ssh using the default username \"ec2-user\" and the ssh key registered with AWS. This product provides access to multiple versions. When launching with 1-click launch, please pay attention to the version. You have the ability to select another version of the RHEL image (including RHEL 8 and newer) when launching from the full AWS Marketplace website.",
            "user_name": "ec2-user",
            "virtualization": "hvm",
            "volume": "gp2"
        }
    },
    {
        "name": "rhel-ec2",
        "workflow": "community",
        "cloud": "aws",
        "billing-code-config": {
            "rhel-access": {
                "codes": [
                    "xy-00000000"
                ],
                "image_name": "rhel",
                "image_types": [
                    "access"
                ],
                "name": "Access2"
            },
            "rhel-hourly": {
                "codes": [
                    "xy-00000001"
                ],
                "image_name": "rhel",
                "image_types": [
                    "hourly"
                ],
                "name": "Hourly2"
            }
        },
        "mappings": {
            "aws-us-storage": {
                "destinations": [
                    {
                        "architecture": null,
                        "destination": "us-east-1-access",
                        "overwrite": false,
                        "restrict_version": false,
                    },
                    {
                        "architecture": null,
                        "destination": "us-east-2-access",
                        "overwrite": false,
                        "restrict_version": false,
                    },
                    {
                        "architecture": null,
                        "destination": "us-west-1-access",
                        "overwrite": false,
                        "restrict_version": false,
                    },
                    {
                        "architecture": null,
                        "destination": "us-west-2-access",
                        "overwrite": false,
                        "restrict_version": false,
                    }
                ],
                "provider": "AWS"
            }
        },
        "meta": {
            "description": "Provided by Red Hat, Inc.",
            "ena_support": true,
            "release": {
                "product": "RHEL",
                "type": "ga",
                "variant": "BaseOS"
            },
            "root_device": "/dev/sda1",
            "sriov_net_support": "simple",
            "virtualization": "hvm",
            "volume": "gp3"
        }
    }
]