Creates a catalog in the Glue Data Catalog.

CfnCatalog

name

Allows third-party engines to access data in Amazon S3 locations that are registered with Lake Formation.

allowFullTableExternalDataAccess

A structure that specifies data lake access properties and other custom properties.

catalogProperties

An array of PrincipalPermissions objects for default database permissions.

createDatabaseDefaultPermissions

An array of PrincipalPermissions objects for default table permissions.

createTableDefaultPermissions

description

A FederatedCatalog structure that references an entity outside the Glue Data Catalog.

federatedCatalog

Specifies whether to overwrite child resource permissions with the default permissions.

overwriteChildResourcePermissionsWithDefault

A map of key-value pairs that define parameters and properties of the catalog.

parameters

An array of key-value pairs to apply to this resource.

targetRedshiftCatalog

The `AWS::Glue::Classifier` resource creates an AWS Glue classifier that categorizes data sources and specifies schemas.

For more information, see [Adding Classifiers to a Crawler](https://docs.aws.amazon.com/glue/latest/dg/add-classifier.html) and [Classifier Structure](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-classifiers.html#aws-glue-api-crawler-classifiers-Classifier) in the *AWS Glue Developer Guide* .

CfnClassifier

A classifier for comma-separated values (CSV).

csvClassifier

grokClassifier

jsonClassifier

xmlClassifier

The `AWS::Glue::Connection` resource specifies an AWS Glue connection to a data source.

For more information, see [Adding a Connection to Your Data Store](https://docs.aws.amazon.com/glue/latest/dg/populate-add-connection.html) and [Connection Structure](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-connections.html#aws-glue-api-catalog-connections-Connection) in the *AWS Glue Developer Guide* .

CfnConnection

The ID of the data catalog to create the catalog object in.

Currently, this should be the AWS account ID.

> To specify the account ID, you can use the `Ref` intrinsic function with the `AWS::AccountId` pseudo parameter. For example: `!Ref AWS::AccountId` .

catalogId

connectionInput

The `AWS::Glue::Crawler` resource specifies an AWS Glue crawler.

For more information, see [Cataloging Tables with a Crawler](https://docs.aws.amazon.com/glue/latest/dg/add-crawler.html) and [Crawler Structure](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-crawling.html#aws-glue-api-crawler-crawling-Crawler) in the *AWS Glue Developer Guide* .

CfnCrawler

The Amazon Resource Name (ARN) of an IAM role that's used to access customer resources, such as Amazon Simple Storage Service (Amazon S3) data.

role

targets

A list of UTF-8 strings that specify the names of custom classifiers that are associated with the crawler.

classifiers

Crawler configuration information.

This versioned JSON string allows users to specify aspects of a crawler's behavior. For more information, see [Configuring a Crawler](https://docs.aws.amazon.com/glue/latest/dg/crawler-configuration.html) .

configuration

The name of the `SecurityConfiguration` structure to be used by this crawler.

crawlerSecurityConfiguration

The name of the database in which the crawler's output is stored.

databaseName

Specifies whether the crawler should use AWS Lake Formation credentials for the crawler instead of the IAM role credentials.

lakeFormationConfiguration

A policy that specifies whether to crawl the entire dataset again, or to crawl only folders that were added since the last crawler run.

recrawlPolicy

For scheduled crawlers, the schedule when the crawler runs.

schedule

The policy that specifies update and delete behaviors for the crawler.

The policy tells the crawler what to do in the event that it detects a change in a table that already exists in the customer's database at the time of the crawl. The `SchemaChangePolicy` does not affect whether or how new tables and partitions are added. New tables and partitions are always created regardless of the `SchemaChangePolicy` on a crawler.

The SchemaChangePolicy consists of two components, `UpdateBehavior` and `DeleteBehavior` .

schemaChangePolicy

The prefix added to the names of tables that are created.

tablePrefix

Creates a custom pattern that is used to detect sensitive data across the columns and rows of your structured data.

Each custom pattern you create specifies a regular expression and an optional list of context words. If no context words are passed only a regular expression is checked.

CfnCustomEntityType

A list of context words.

If none of these context words are found within the vicinity of the regular expression the data will not be detected as sensitive data.

If no context words are passed only a regular expression is checked.

contextWords

A name for the custom pattern that allows it to be retrieved or deleted later.

This name must be unique per AWS account.

A regular expression string that is used for detecting sensitive data in a custom pattern.

regexString

AWS tags that contain a key value pair and may be searched by console, command line, or API.

The `AWS::Glue::Database` resource specifies a logical grouping of tables in AWS Glue .

For more information, see [Defining a Database in Your Data Catalog](https://docs.aws.amazon.com/glue/latest/dg/define-database.html) and [Database Structure](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-databases.html#aws-glue-api-catalog-databases-Database) in the *AWS Glue Developer Guide* .

CfnDatabase

The AWS account ID for the account in which to create the catalog object.

> To specify the account ID, you can use the `Ref` intrinsic function with the `AWS::AccountId` pseudo parameter. For example: `!Ref AWS::AccountId`

databaseInput

Sets the security configuration for a specified catalog.

After the configuration has been set, the specified encryption is applied to every catalog write thereafter.

CfnDataCatalogEncryptionSettings

The ID of the Data Catalog in which the settings are created.

Contains configuration information for maintaining Data Catalog security.

dataCatalogEncryptionSettings

The `AWS::Glue::DataQualityRuleset` resource specifies a data quality ruleset with DQDL rules applied to a specified AWS Glue table.

For more information, see AWS Glue Data Quality in the AWS Glue Developer Guide.

CfnDataQualityRuleset

Used for idempotency and is recommended to be set to a random ID (such as a UUID) to avoid creating or starting multiple instances of the same resource.

clientToken

A description of the data quality ruleset.

A Data Quality Definition Language (DQDL) ruleset.

For more information see the AWS Glue Developer Guide.

ruleset

A list of tags applied to the data quality ruleset.

An object representing an AWS Glue table.

targetTable

The `AWS::Glue::DevEndpoint` resource specifies a development endpoint where a developer can remotely debug ETL scripts for AWS Glue .

For more information, see [DevEndpoint Structure](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-jobs-dev-endpoint.html#aws-glue-api-jobs-dev-endpoint-DevEndpoint) in the AWS Glue Developer Guide.

CfnDevEndpoint

The Amazon Resource Name (ARN) of the IAM role used in this `DevEndpoint` .

roleArn

A map of arguments used to configure the `DevEndpoint` .

Valid arguments are:

- `"--enable-glue-datacatalog": ""`
- `"GLUE_PYTHON_VERSION": "3"`
- `"GLUE_PYTHON_VERSION": "2"`

You can specify a version of Python support for development endpoints by using the `Arguments` parameter in the `CreateDevEndpoint` or `UpdateDevEndpoint` APIs. If no arguments are provided, the version defaults to Python 2.

arguments

endpointName

The path to one or more Java `.jar` files in an S3 bucket that should be loaded in your `DevEndpoint` .

> You can only use pure Java/Scala libraries with a `DevEndpoint` .

extraJarsS3Path

The paths to one or more Python libraries in an Amazon S3 bucket that should be loaded in your `DevEndpoint` .

Multiple values must be complete paths separated by a comma.

> You can only use pure Python libraries with a `DevEndpoint` . Libraries that rely on C extensions, such as the [pandas](https://docs.aws.amazon.com/http://pandas.pydata.org/) Python data analysis library, are not currently supported.

extraPythonLibsS3Path

The AWS Glue version determines the versions of Apache Spark and Python that AWS Glue supports.

The Python version indicates the version supported for running your ETL scripts on development endpoints.

For more information about the available AWS Glue versions and corresponding Spark and Python versions, see [Glue version](https://docs.aws.amazon.com/glue/latest/dg/add-job.html) in the developer guide.

Development endpoints that are created without specifying a Glue version default to Glue 0.9.

You can specify a version of Python support for development endpoints by using the `Arguments` parameter in the `CreateDevEndpoint` or `UpdateDevEndpoint` APIs. If no arguments are provided, the version defaults to Python 2.

glueVersion

The number of AWS Glue Data Processing Units (DPUs) allocated to this `DevEndpoint` .

numberOfNodes

The number of workers of a defined `workerType` that are allocated to the development endpoint.

The maximum number of workers you can define are 299 for `G.1X` , and 149 for `G.2X` .

numberOfWorkers

The public key to be used by this `DevEndpoint` for authentication.

This attribute is provided for backward compatibility because the recommended attribute to use is public keys.

publicKey

A list of public keys to be used by the `DevEndpoints` for authentication.

Using this attribute is preferred over a single public key because the public keys allow you to have a different private key per client.

> If you previously created an endpoint with a public key, you must remove that key to be able to set a list of public keys. Call the `UpdateDevEndpoint` API operation with the public key content in the `deletePublicKeys` attribute, and the list of new keys in the `addPublicKeys` attribute.

publicKeys

The name of the `SecurityConfiguration` structure to be used with this `DevEndpoint` .

securityConfiguration

A list of security group identifiers used in this `DevEndpoint` .

securityGroupIds

subnetId

The type of predefined worker that is allocated to the development endpoint.

Accepts a value of Standard, G.1X, or G.2X.

- For the `Standard` worker type, each worker provides 4 vCPU, 16 GB of memory and a 50GB disk, and 2 executors per worker.
- For the `G.1X` worker type, each worker maps to 1 DPU (4 vCPU, 16 GB of memory, 64 GB disk), and provides 1 executor per worker. We recommend this worker type for memory-intensive jobs.
- For the `G.2X` worker type, each worker maps to 2 DPU (8 vCPU, 32 GB of memory, 128 GB disk), and provides 1 executor per worker. We recommend this worker type for memory-intensive jobs.

Known issue: when a development endpoint is created with the `G.2X` `WorkerType` configuration, the Spark drivers for the development endpoint will run on 4 vCPU, 16 GB of memory, and a 64 GB disk.

workerType

Creates a new AWS Glue Identity Center configuration to enable integration between AWS Glue and AWS IAM Identity Center for authentication and authorization.

CfnIdentityCenterConfiguration

The Amazon Resource Name (ARN) of the Identity Center instance associated with the AWS Glue configuration.

instanceArn

A list of Identity Center scopes that define the permissions and access levels for the AWS Glue configuration.

scopes

Indicates whether users can run background sessions when using Identity Center authentication with AWS Glue services.

userBackgroundSessionsEnabled

The `AWS::Glue::Integration` resource specifies an AWS Glue zero-ETL integration from a data source to a target.

For more information, see [zero-ETL integration supported by AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/zero-etl-using.html) and [integration structure](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-integrations.html) in the AWS Glue developer guide.

CfnIntegration

integrationName

The ARN for the source of the integration.

sourceArn

The ARN for the target of the integration.

targetArn

An optional set of non-secret key–value pairs that contains additional contextual information for encryption.

This can only be provided if `KMSKeyId` is provided.

additionalEncryptionContext

Selects source tables for the integration using Maxwell filter syntax.

dataFilter

The structure used to define properties associated with the zero-ETL integration.

For more information, see [IntegrationConfig structure.](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-integrations.html#aws-glue-api-integrations-IntegrationConfig)

integrationConfig

The ARN of a KMS key used for encrypting the channel.

kmsKeyId

Metadata assigned to the resource consisting of a list of key-value pairs.

The `AWS::Glue::IntegrationResourceProperty` resource type can be used to setup `ResourceProperty` of the AWS Glue connection (for the SaaS source), DynamoDB Database (for DynamoDB source), or AWS Glue database ARN (for the target).

ResourceProperty is used to define the properties requires to setup the integration, including the role to access the connection or database, KMS keys, event bus for event notifications and VPC connection. To set both source and target properties the same API needs to be invoked twice, once with the AWS Glue connection ARN as ResourceArn with SourceProcessingProperties and next, with the AWS Glue database ARN as ResourceArn with TargetProcessingProperties respectively.

CfnIntegrationResourceProperty

The connection ARN of the source, or the database ARN of the target.

resourceArn

The resource properties associated with the integration source.

sourceProcessingProperties

The structure used to define the resource properties associated with the integration target.

targetProcessingProperties

The `AWS::Glue::Job` resource specifies an AWS Glue job in the data catalog.

For more information, see [Adding Jobs in AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/add-job.html) and [Job Structure](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-jobs-job.html#aws-glue-api-jobs-job-Job) in the *AWS Glue Developer Guide.*

CfnJob

command

The name or Amazon Resource Name (ARN) of the IAM role associated with this job.

This parameter is no longer supported. Use `MaxCapacity` instead.

The number of capacity units that are allocated to this job.

allocatedCapacity

connections

The default arguments for this job, specified as name-value pairs.

You can specify arguments here that your own job-execution script consumes, in addition to arguments that AWS Glue itself consumes.

For information about how to specify and consume your own job arguments, see [Calling AWS Glue APIs in Python](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-python-calling.html) in the *AWS Glue Developer Guide* .

For information about the key-value pairs that AWS Glue consumes to set up your job, see [Special Parameters Used by AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html) in the *AWS Glue Developer Guide* .

defaultArguments

Indicates whether the job is run with a standard or flexible execution class.

The standard execution class is ideal for time-sensitive workloads that require fast job startup and dedicated resources.

The flexible execution class is appropriate for time-insensitive jobs whose start and completion times may vary.

Only jobs with AWS Glue version 3.0 and above and command type `glueetl` will be allowed to set `ExecutionClass` to `FLEX` . The flexible execution class is available for Spark jobs.

executionClass

The maximum number of concurrent runs that are allowed for this job.

executionProperty

Glue version determines the versions of Apache Spark and Python that AWS Glue supports.

The Python version indicates the version supported for jobs of type Spark.

For more information about the available AWS Glue versions and corresponding Spark and Python versions, see [Glue version](https://docs.aws.amazon.com/glue/latest/dg/add-job.html) in the developer guide.

Jobs that are created without specifying a Glue version default to the latest Glue version available.

A mode that describes how a job was created. Valid values are:.

- `SCRIPT` - The job was created using the AWS Glue Studio script editor.
- `VISUAL` - The job was created using the AWS Glue Studio visual editor.
- `NOTEBOOK` - The job was created using an interactive sessions notebook.

When the `JobMode` field is missing or null, `SCRIPT` is assigned as the default value.

jobMode

Specifies whether job run queuing is enabled for the job runs for this job.

A value of true means job run queuing is enabled for the job runs. If false or not populated, the job runs will not be considered for queueing.

If this field does not match the value set in the job run, then the value from the job run field will be used.

jobRunQueuingEnabled

logUri

This field specifies a day of the week and hour for a maintenance window for streaming jobs.

AWS Glue periodically performs maintenance activities. During these maintenance windows, AWS Glue will need to restart your streaming jobs.

AWS Glue will restart the job within 3 hours of the specified maintenance window. For instance, if you set up the maintenance window for Monday at 10:00AM GMT, your jobs will be restarted between 10:00AM GMT to 1:00PM GMT.

maintenanceWindow

The number of AWS Glue data processing units (DPUs) that can be allocated when this job runs.

A DPU is a relative measure of processing power that consists of 4 vCPUs of compute capacity and 16 GB of memory.

Do not set `Max Capacity` if using `WorkerType` and `NumberOfWorkers` .

The value that can be allocated for `MaxCapacity` depends on whether you are running a Python shell job or an Apache Spark ETL job:

- When you specify a Python shell job ( `JobCommand.Name` ="pythonshell"), you can allocate either 0.0625 or 1 DPU. The default is 0.0625 DPU.
- When you specify an Apache Spark ETL job ( `JobCommand.Name` ="glueetl"), you can allocate from 2 to 100 DPUs. The default is 10 DPUs. This job type cannot have a fractional DPU allocation.

maxCapacity

The maximum number of times to retry this job after a JobRun fails.

maxRetries

The name you assign to this job definition.

Non-overridable arguments for this job, specified as name-value pairs.

nonOverridableArguments

Specifies configuration properties of a notification.

notificationProperty

The number of workers of a defined `workerType` that are allocated when a job runs.

The maximum number of workers you can define are 299 for `G.1X` , and 149 for `G.2X` .

The name of the `SecurityConfiguration` structure to be used with this job.

The job timeout in minutes.

This is the maximum time that a job run can consume resources before it is terminated and enters TIMEOUT status. The default is 2,880 minutes (48 hours).

timeout

The type of predefined worker that is allocated when a job runs.

AWS Glue provides multiple worker types to accommodate different workload requirements:

G Worker Types (General-purpose compute workers):

- G.1X: 1 DPU (4 vCPUs, 16 GB memory, 94GB disk)
- G.2X: 2 DPU (8 vCPUs, 32 GB memory, 138GB disk)
- G.4X: 4 DPU (16 vCPUs, 64 GB memory, 256GB disk)
- G.8X: 8 DPU (32 vCPUs, 128 GB memory, 512GB disk)
- G.12X: 12 DPU (48 vCPUs, 192 GB memory, 768GB disk)
- G.16X: 16 DPU (64 vCPUs, 256 GB memory, 1024GB disk)

R Worker Types (Memory-optimized workers):

- R.1X: 1 M-DPU (4 vCPUs, 32 GB memory)
- R.2X: 2 M-DPU (8 vCPUs, 64 GB memory)
- R.4X: 4 M-DPU (16 vCPUs, 128 GB memory)
- R.8X: 8 M-DPU (32 vCPUs, 256 GB memory)

The AWS::Glue::MLTransform is an AWS Glue resource type that manages machine learning transforms.

CfnMLTransform

A list of AWS Glue table definitions used by the transform.

inputRecordTables

The name or Amazon Resource Name (ARN) of the IAM role with the required permissions.

The required permissions include both AWS Glue service role permissions to AWS Glue resources, and Amazon S3 permissions required by the transform.

- This role needs AWS Glue service role permissions to allow access to resources in AWS Glue . See [Attach a Policy to IAM Users That Access AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/attach-policy-iam-user.html) .
- This role needs permission to your Amazon Simple Storage Service (Amazon S3) sources, targets, temporary directory, scripts, and any libraries used by the task run for this transform.

The algorithm-specific parameters that are associated with the machine learning transform.

transformParameters

A user-defined, long-form description text for the machine learning transform.

This value determines which version of AWS Glue this machine learning transform is compatible with.

Glue 1.0 is recommended for most customers. If the value is not set, the Glue compatibility defaults to Glue 0.9. For more information, see [AWS Glue Versions](https://docs.aws.amazon.com/glue/latest/dg/release-notes.html#release-notes-versions) in the developer guide.

The number of AWS Glue data processing units (DPUs) that are allocated to task runs for this transform.

You can allocate from 2 to 100 DPUs; the default is 10. A DPU is a relative measure of processing power that consists of 4 vCPUs of compute capacity and 16 GB of memory. For more information, see the [AWS Glue pricing page](https://docs.aws.amazon.com/glue/pricing/) .

`MaxCapacity` is a mutually exclusive option with `NumberOfWorkers` and `WorkerType` .

- If either `NumberOfWorkers` or `WorkerType` is set, then `MaxCapacity` cannot be set.
- If `MaxCapacity` is set then neither `NumberOfWorkers` or `WorkerType` can be set.
- If `WorkerType` is set, then `NumberOfWorkers` is required (and vice versa).
- `MaxCapacity` and `NumberOfWorkers` must both be at least 1.

When the `WorkerType` field is set to a value other than `Standard` , the `MaxCapacity` field is set automatically and becomes read-only.

The maximum number of times to retry after an `MLTaskRun` of the machine learning transform fails.

A user-defined name for the machine learning transform. Names are required to be unique. `Name` is optional:.

- If you supply `Name` , the stack cannot be repeatedly created.
- If `Name` is not provided, a randomly generated name will be used instead.

The number of workers of a defined `workerType` that are allocated when a task of the transform runs.

If `WorkerType` is set, then `NumberOfWorkers` is required (and vice versa).

The tags to use with this machine learning transform.

You may use tags to limit access to the machine learning transform. For more information about tags in AWS Glue , see [AWS Tags in AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/monitor-tags.html) in the developer guide.

The timeout in minutes of the machine learning transform.

The encryption-at-rest settings of the transform that apply to accessing user data.

Machine learning
transforms can access user data encrypted in Amazon S3 using KMS.

Additionally, imported labels and trained transforms can now be encrypted using a customer provided
KMS key.

transformEncryption

The type of predefined worker that is allocated when a task of this transform runs.

Accepts a value of Standard, G.1X, or G.2X.

- For the `Standard` worker type, each worker provides 4 vCPU, 16 GB of memory and a 50GB disk, and 2 executors per worker.
- For the `G.1X` worker type, each worker provides 4 vCPU, 16 GB of memory and a 64GB disk, and 1 executor per worker.
- For the `G.2X` worker type, each worker provides 8 vCPU, 32 GB of memory and a 128GB disk, and 1 executor per worker.

`MaxCapacity` is a mutually exclusive option with `NumberOfWorkers` and `WorkerType` .

- If either `NumberOfWorkers` or `WorkerType` is set, then `MaxCapacity` cannot be set.
- If `MaxCapacity` is set then neither `NumberOfWorkers` or `WorkerType` can be set.
- If `WorkerType` is set, then `NumberOfWorkers` is required (and vice versa).
- `MaxCapacity` and `NumberOfWorkers` must both be at least 1.

The `AWS::Glue::Partition` resource creates an AWS Glue partition, which represents a slice of table data.

For more information, see [CreatePartition Action](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-partitions.html#aws-glue-api-catalog-partitions-CreatePartition) and [Partition Structure](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-partitions.html#aws-glue-api-catalog-partitions-Partition) in the *AWS Glue Developer Guide* .

CfnPartition

The AWS account ID of the catalog in which the partion is to be created.

> To specify the account ID, you can use the `Ref` intrinsic function with the `AWS::AccountId` pseudo parameter. For example: `!Ref AWS::AccountId`

The name of the catalog database in which to create the partition.

AWS Glue

L1 Constructs25

CfnCatalog

CfnClassifier

CfnConnection

CfnCrawler

CfnCustomEntityType

CfnDatabase

CfnDataCatalogEncryptionSettings

CfnDataQualityRuleset

CfnDevEndpoint

CfnIdentityCenterConfiguration

CfnIntegration

CfnIntegrationResourceProperty

CfnJob

CfnMLTransform

CfnPartition

CfnRegistry

CfnSchema

CfnSchemaVersion

CfnSchemaVersionMetadata

CfnSecurityConfiguration

CfnTable

CfnTableOptimizer

CfnTrigger

CfnUsageProfile

CfnWorkflow

Learn AWS the Practical Way

Quick Facts

Related Links

More AWS Glue deep dives