Skip to main content
On this page

Console resources

ConsoleGroup

  • API key(s): Admin API key
  • Managed with: Console UI CLI API Terraform
  • Labels support: Label support upcoming

Creates a Group with members and permissions in Console

---
apiVersion: iam/v2
kind: Group
metadata:
name: developers-a
spec:
displayName: "Developers Team A"
description: "Members of the Team A - Developers"
externalGroups:
- "LDAP-GRP-A-DEV"
members:
- member1@company.org
- member2@company.org
permissions:
- resourceType: TOPIC
cluster: shadow-it
patternType: PREFIXED
name: toto-
permissions:
- topicViewConfig
- topicConsume
- topicProduce

Groups checks:

  • spec.description is optional
  • spec.externalGroups is a list of LDAP or OIDC groups to sync with this Console Group
    • Members added this way will not appear in spec.members but spec.membersFromExternalGroups instead
  • spec.membersFromExternalGroups is a read-only list of members added through spec.externalGroups
  • spec.members must be email addresses of members you wish to add to this group
  • spec.permissions are valid permissions as defined in Permissions

Side effect in Console and Kafka:

  • Console
    • Members of the Group are given the associated permissions in the UI over the resources
    • Members of the LDAP or OIDC groups will be automatically added or removed upon login
  • Kafka
    • No side effect

ConsoleUser

  • API key(s): Admin API key
  • Managed with: Console UI CLI API Terraform
  • Labels support: Label support upcoming

Create a user with Platform permissions.

---
apiVersion: iam/v2
kind: User
metadata:
name: john.doe@company.org
spec:
firstName: "John"
lastName: "Doe"
permissions:
- resourceType: PLATFORM
permissions:
- taasView
- datamaskingView
- resourceType: TOPIC
cluster: shadow-it
patternType: PREFIXED
name: toto-
permissions:
- topicViewConfig
- topicConsume
- topicProduce
warning

Make sure you set permissions for this user, otherwise it won't have access to Platform functionality (such as Application Catalog or Data Policies) and Kafka resources.

Users checks:

  • spec.permissions are valid permissions as defined in Permissions

Side effect in Console and Kafka:

  • Console
    • User is given the associated permissions in the UI over the resources
  • Kafka
    • No side effect

KafkaCluster

Creates a Kafka Cluster Definition in Console.

  • API key(s): Admin API key
  • Managed with: Console UI CLI API Terraform
  • Labels support: Partial label support (no UI yet)
---
apiVersion: console/v2
kind: KafkaCluster
metadata:
name: my-dev-cluster
spec:
displayName: "My Dev Cluster"
icon: "kafka"
color: "#000000"
bootstrapServers: "localhost:9092"
ignoreUntrustedCertificate: false
properties:
sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret";
security.protocol: SASL_SSL
sasl.mechanism: PLAIN
schemaRegistry:
type: "ConfluentLike"
url: http://localhost:8080
security:
type: BasicAuth
username: some_user
password: some_password
ignoreUntrustedCertificate: false
kafkaFlavor:
type: "Confluent"
key: "string"
secret: "string"
confluentEnvironmentId: "string"
confluentClusterId: "string"
info

metadata.name, spec.displayName, spec.icon and spec.color work together to build the visual identity of the KafkaCluster throughout Console. Cluster identity

KafkaCluster checks:

  • spec.icon (optional, default kafka) is a valid entry from our Icon Sets
  • spec.color (optional, default #000000) is a HEX color for spec.icon
  • spec.ignoreUntrustedCertificate (optional, default false) must be one of [true, false]
  • spec.schemaRegistry.type (optional) must be one of [ConfluentLike, Glue]
  • spec.kafkaFlavor.type (optional) must be one of [Confluent, Aiven, Gateway]
warning

Conduktor CLI does not verify that your Kafka configuration (spec.bootstrapServers, spec.properties, ...) is valid. You need to check that in Console directly.

Schema Registry

This section lets you associate a Schema Registry to your KafkaCluster.

Confluent or Confluent-like Registry

spec:
schemaRegistry:
type: "ConfluentLike"
url: http://localhost:8080
ignoreUntrustedCertificate: false
security:
type: BasicAuth
username: some_user
password: some_password

Confluent Schema Registry checks:

  • spec.schemaRegistry.urls must be a single URL of a Kafka Connect cluster
    • Multiple URLs are not supported for now. Coming soon
  • spec.schemaRegistry.ignoreUntrustedCertificate (optional, default false) must be one of [true, false]
  • spec.schemaRegistry.properties (optional) is Java Properties formatted key values to further configure the SchemaRegistry
  • spec.security.type (optional) must be one of [BasicAuth, BearerToken, SSLAuth]

AWS Glue Registry

spec:
schemaRegistry:
type: "Glue"
region: eu-west-1
registryName: default
security:
type: Credentials
accessKeyId: accessKey
secretKey: secretKey

AWS Glue Registry checks:

  • spec.schemaRegistry.region must be a valid AWS region
  • spec.schemaRegistry.registryName must be a valid AWS Glue Registry in this region
  • spec.schemaRegistry.security.type must be one of [Credentials, FromContext, FromRole]

Credentials
Use AWS API Key/Secret to connect to the Glue Registry

    security:
type: Credentials
accessKeyId: AKIAIOSFODNN7EXAMPLE
secretKey: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

FromContext

    security:
type: FromContext
profile: default

FromRole

    security:
type: FromRole
role: arn:aws:iam::123456789012:role/example-role

Kafka provider

This section lets you configure the Kafka Provider for this KafkaCluster.

Confluent Cloud

Provide your Confluent Cloud details to get additional features in Console:

  • Confluent Cloud Service Accounts support
  • Confluent Cloud API Keys support
spec:
kafkaFlavor:
type: "Confluent"
key: "yourApiKey123456"
secret: "yourApiSecret123456"
confluentEnvironmentId: "env-12345"
confluentClusterId: "lkc-67890"

Aiven

Provide your Aiven Cloud details to get additional features in Console:

  • Aiven Service Accounts support
  • Aiven ACLs support
spec:
kafkaFlavor:
type: "Aiven"
apiToken: "a1b2c3d4e5f6g7h8i9j0"
project: "my-kafka-project"
serviceName: "my-kafka-service"

Gateway
Provide your Gateway details to get additional features in Console:

  • Interceptor support
spec:
kafkaFlavor:
type: "Gateway"
url: "http://gateway:8888"
user: "admin"
password: "admin"
virtualCluster: passthrough

Icon Sets

cloudBoltcloudRainbowcloudsnowflake
pooStormpoopboltumbrella
tennisBallrugbyBalltrafficConefaucet
basketShoppingboxscaleBalancedsunglasses
swordaxeBattlevialfeatherPointed
bombflagheartkey
fireExtinguisherfireFlameCurvedalienhelmetBattle
ghostrobotdogelephant
birdcrabcatSpaceplanetRinged
meteormoonspaceStationrocketLaunch
paperPlanecarSidebuildingColumnscastle
acornburgerLettucecroissantmug
cactusclovercameraCctvcalendar
alarmClockcompassgamepadModernserver
shieldBlankcomputerClassicdharmachakrakafka

KafkaConnectCluster

Creates a Kafka Connect cluster definition in Console.

  • API key(s): Admin API key
  • Managed with: Console UI CLI API Terraform
  • Labels support: Partial label support (no UI yet)
---
apiVersion: console/v2
kind: KafkaConnectCluster
metadata:
cluster: my-dev-cluster
name: connect-1
spec:
displayName: "Connect 1"
urls: "http://localhost:8083"
headers:
X-PROJECT-HEADER: value
AnotherHeader: test
ignoreUntrustedCertificate: false
security:
type: "BasicAuth"
username: "toto"
password: "my-secret"

KafkaConnectCluster checks:

  • metadata.cluster must be a valid KafkaCluster name
  • spec.urls must be a single URL of a Kafka Connect cluster
    • Multiple URLs are not supported for now. Coming soon
  • spec.ignoreUntrustedCertificate (optional, default false) must be one of [true, false]
  • spec.headers (optional) must be key-value pairs of HTTP Headers
  • spec.security.type (optional) must be one of [BasicAuth, BearerToken, SSLAuth]

KsqlDBCluster

  • API key(s): Admin API key
  • Managed with: Console UI CLI API
  • Labels support: Label support upcoming

Creates a ksqlDB Cluster Definition in Console.

---
apiVersion: console/v2
kind: KsqlDBCluster
metadata:
cluster: my-dev-cluster
name: ksql-1
spec:
displayName: "KSQL 1"
url: "http://localhost:8088"
ignoreUntrustedCertificate: false
security:
type: "BasicAuth"
username: "toto"
password: "my-secret"

KafkaConnectCluster checks:

  • metadata.cluster must be a valid KafkaCluster name
  • spec.url must be a single URL of a KsqlDB cluster
  • spec.ignoreUntrustedCertificate (optional, default false) must be one of [true, false]
  • spec.headers (optional) must be key-value pairs of HTTP Headers
  • spec.security.type (optional) must be one of [BasicAuth, BearerToken, SSLAuth]

Alerts

  • API key(s): Admin API key
  • Managed with: Console UI CLI API
  • Labels support: Label support upcoming

Creates an Alert in Console.

---
apiVersion: console/v3
kind: Alert
metadata:
name: messages-in-dead-letter-queue
group: support-team # will be the owner of the alert, can be either a user, a group or an appInstance
# user: user@company.org
# appInstance: my-app-instance
spec:
cluster: my-dev-cluster
type: TopicAlert
topicName: wikipedia-parsed-DLQ
metric: MessageCount
operator: GreaterThan
threshold: 0
destination:
type: Slack
channel: "alerts-p1"

Alert checks:

  • metadata.user|metadata.group|metadata.appInstance must be a valid user, group or appInstance
  • metadata.destination.type can be either Slack, Teams or Webhook
  • spec.cluster must be a valid KafkaCluster name
  • spec.type must be one of [BrokerAlert,TopicAlert,KafkaConnectAlert]
    • Check the section below for the additional mandatory fields needed for each spec.type
  • spec.metric is depending on the spec.type
    • Check section below
  • spec.operator must be one of [GreaterThan, GreaterThanOrEqual, LessThan, LessThanOrEqual, NotEqual]
  • spec.threshold must be a number
  • spec.disable (optional, default false) must be one of [true, false]

When spec.destination.type is Slack

  • spec.destination.channel must be a valid Slack channel id

When spec.destination.type is Teams

  • spec.destination.url must be a valid Teams webhook URL

When spec.destination.type is Webhook

  • spec.destination.url must be a valid URL
  • spec.destination.method must be one of [GET, POST, PUT, DELETE]
  • spec.destination.headers (optional) must be key-value pairs of HTTP Headers
  • spec.destination.authentification.type (optional) must be one of [BasicAuth, BearerToken]
    • when is BasicAuth spec.destination.authentification.username and spec.destination.authentification.password must be set
    • when is BearerToken spec.destination.authentification.token must be set

When spec.type is BrokerAlert

  • spec.metric must be one of [MessageIn, MessageOut, MessageSize, OfflinePartitionCount, PartitionCount, UnderMinIsrPartitionCount, UnderReplicatedPartitionCount]

When spec.type is TopicAlert

  • spec.metric must be one of [MessageCount, MessageIn, MessageOut, MessageSize]
  • spec.topicName must be a Kafka Topic that the owner can access

When spec.type is KafkaConnectAlert

  • spec.metric must be FailedTaskCount
  • spec.connectName must be a valid KafkaConnect Cluster associated to this spec.cluster Kafka Cluster
  • spec.connectorName must be a Kafka Connect Connector that the owner can access

When spec.type is ConsumerGroupAlert

  • spec.metric must be one of [OffsetLag, TimeLag]
  • spec.consumerGroupName must be a Kafka Consumer Group that the owner can access

DataMaskingPolicy

Coming soon

Partner Zones

  • API key(s): Admin API key
  • Managed with: Console UI CLI API
  • Labels support:Partial label support (no UI yet)

Create or update a Partner Zone.

---
apiVersion: console/v2
kind: PartnerZone
metadata:
name: external-partner-zone
spec:
displayName: External Partner Zone
description: An external partner to exchange data with.
url: https://partner1.com
partner:
name: John Doe
role: Data analyst
email: johndoe@partner.io
phone: 07827 837 177
cluster: cdk-gateway
serviceAccount: partner-external-partner
topics:
- name: topic-a
backingTopic: kafka-topic-a
permission: WRITE
- name: topic-b
backingTopic: kafka-topic-a
permission: READ
trafficControlPolicies:
maxProduceRate: 1e+06
maxConsumeRate: 1e+06
limitCommitOffset: 30

Partner Zone checks:

  • spec.displayName is Mandatory
  • spec.description, spec.url and spec.partner are optional context informations.
  • spec.cluster must be a valid Console cluster technical id with the Provider configured as Gateway.
  • spec.serviceAccount must be a Local Gateway Service Account. It doesn't need to exist before creating the Partner Zone. The service account will be created automatically.
  • topics[].name is the name of the topic as it should appear to your partner. This can be different from backingTopic.
  • topics[].backingTopic is the internal name of the topic that you want to share with your partner.
  • topics[].permission must be set to either READ or WRITE (which additionally grants READ).
  • trafficControlPolicies.maxProduceRate is optional. Sets the maximum rate (in bytes/s) at which the partner can produce messages to the topics per Gateway node.
  • trafficControlPolicies.maxConsumeRate is optional. Sets the maximum rate (in bytes/s) at which the partner can consume messages from the topics per Gateway node.
  • trafficControlPolicies.limitCommitOffset is optional. Sets the maximum number of commits requests (in requests/minute) that the partner can make per Gateway node.

Side effect in Console and Kafka:
Upon creation or update, the following fields will be available:

  • metadata.updatedAt field will be made available by consecutive get from the CLI/API.
  • metadata.status field will be made available by consecutive get from the CLI/API. Possible values are PENDING, READY or FAILED.
  • metadata.failedReason field will be populated in case of FAILED status.
  • The service account will be created if it doesn't exist and will be granted the permissions as declared in spec.topics
  • The traffic control policies will be applied to the service account.

HTTP security properties

HTTP security properties are used in KafkaCluster (Schema Registry), KafkaConnect, KsqlDBCluster.

Basic authentication

  security:
type: "BasicAuth"
username: "toto"
password: "my-secret"

Bearer token

  security:
type: "BearerToken"
token: "toto"

mTLS/client certificate

  security:
type: "SSLAuth"
key: |
-----BEGIN PRIVATE KEY-----
MIIOXzCCDUegAwIBAgIRAPRytMVYJNUgCbhnA+eYumgwDQYJKoZIhvcNAQELBQAw
...
IFyCs+xkcgvHFtBjjel4pnIET0agtbGJbGDEQBNxX+i4MDA=
-----END PRIVATE KEY-----
certificateChain: |
-----BEGIN CERTIFICATE-----
MIIOXzCCDUegAwIBAgIRAPRytMVYJNUgCbhnA+eYumgwDQYJKoZIhvcNAQELBQAw
RjELMAkGA1UEBhMCVVMxIjAgBgNVBAoTGUdvb2dsZSBUcnVzdCBTZXJ2aWNlcyBM
...
8/s+YDKveNdoeQoAmGQpUmxhvJ9rbNYj+4jiaujkfxT/6WtFN8N95r+k3W/1K4hs
IFyCs+xkcgvHFtBjjel4pnIET0agtbGJbGDEQBNxX+i4MDA=
-----END CERTIFICATE-----

Permissions

Permissions are used in groups and users and lets you configure all the access to any Kafka resource or Console feature. A permission applies to a certain resourceType, which affect the necessary fields as detailed below.

Topic permissions

# Grants Consume, Produce and View Config to all topics toto-* on shadow-it cluster
- resourceType: TOPIC
cluster: shadow-it
patternType: PREFIXED
name: toto-
permissions:
- topicViewConfig
- topicConsume
- topicProduce
  • resourceType: TOPIC
  • cluster is a valid Kafka cluster
  • patternType is either PREFIXED or LITERAL
  • name is the name of the topic or topic prefix to apply the permissions to
  • permissions is a list of valid topic permissions (See Table)
Available topic permissionsDescription
topicConsumePermission to consume messages from the topic.
topicProducePermission to produce (write) messages to the topic.
topicViewConfigPermission to view the topic configuration.
topicEditConfigPermission to edit the topic configuration.
topicCreatePermission to create a new topic.
topicDeletePermission to delete the topic.
topicAddPartitionPermission to add partitions to the topic.
topicEmptyPermission to empty (delete all messages from) the topic.

Subject permissions

# Grants View and Edit Compatibility to all subjects starting with sub-* on shadow-it cluster
- resourceType: SUBJECT
cluster: shadow-it
patternType: PREFIXED
name: sub-
permissions:
- subjectView
- subjectEditCompatibility
  • resourceType: SUBJECT
  • cluster is a valid Kafka cluster
  • patternType is either PREFIXED or LITERAL
  • name is the name of the subject or subject prefix to apply the permissions to
  • permissions is a list of valid subject permissions (See Table)
Available subject permissionsDescription
subjectCreateUpdatePermission to create or update the subject.
subjectDeletePermission to delete the subject.
subjectEditCompatibilityPermission to edit the subject compatibility settings.
subjectViewPermission to view the subject details.

ConsumerGroup permissions

# Grants View and Reset on all consumer groups starting with group-* on shadow-it cluster
- resourceType: CONSUMER_GROUP
cluster: shadow-it
patternType: PREFIXED
name: group-
permissions:
- consumerGroupView
- consumerGroupReset
  • resourceType: CONSUMER_GROUP
  • cluster is a valid Kafka cluster
  • patternType is either PREFIXED or LITERAL
  • name is the name of the consumer group or consumer group prefix to apply the permissions to
  • permissions is a list of valid consumer group permissions (See Table)
Available ConsumerGroup permissionsDescription
consumerGroupCreatePermission to create a new consumer group.
consumerGroupResetPermission to reset the consumer group.
consumerGroupDeletePermission to delete the consumer group.
consumerGroupViewPermission to view the consumer group details.

Cluster permissions

# Grants View Broker, Edit Schema Registry Compatibility, Edit Broker, View ACL, and Manage ACL on shadow-it cluster
- resourceType: CLUSTER
name: shadow-it
permissions:
- clusterViewBroker
- clusterEditSRCompatibility
- clusterEditBroker
- clusterViewACL
- clusterManageACL
  • resourceType: CLUSTER
  • name is the name of the cluster to apply the permissions to
    • Use * for all clusters
  • permissions is a list of valid cluster permissions (See Table)
Available cluster permissionsDescription
clusterViewBrokerPermission to view broker details.
clusterEditSRCompatibilityPermission to edit Schema Registry compatibility settings.
clusterEditBrokerPermission to edit broker configuration.
clusterViewACLPermission to view Access Control Lists (ACLs) for the cluster.
clusterManageACLPermission to manage Access Control Lists (ACLs) for the cluster.

KafkaConnect permissions

# Grants Create and Delete on all connectors starting with connector-* on shadow-it cluster and kafka-connect-cluster
- resourceType: KAFKA_CONNECT
cluster: shadow-it
kafkaConnect: kafka-connect-cluster
patternType: PREFIXED
name: connector-
permissions:
- kafkaConnectorCreate
- kafkaConnectorDelete
  • resourceType: KAFKA_CONNECT
  • cluster is a valid Kafka cluster
  • kafkaConnect is a valid Kafka Connect cluster
  • patternType is either PREFIXED or LITERAL
  • name is the name of the connector or connector prefix to apply the permissions to
  • permissions is a list of valid Kafka Connect permissions (See Table)
Available KafkaConnect permissionDescription
kafkaConnectorViewConfigPermission to view the Kafka Connect configuration.
kafkaConnectorStatusPermission to view the status of Kafka Connect connectors.
kafkaConnectorEditConfigPermission to edit the Kafka Connect configuration.
kafkaConnectorDeletePermission to delete connectors.
kafkaConnectorCreatePermission to create new connectors.
kafkaConnectPauseResumePermission to pause and resume connectors.
kafkaConnectRestartPermission to restart connectors.

KsqlDB permissions

# Grants all permissions on KsqlDB cluster ksql-cluster
- resourceType: KSQLDB
cluster: shadow-it
ksqlDB: ksql-cluster
permissions:
- ksqldbAccess
  • resourceType: KSQLDB
  • cluster is a valid Kafka cluster
  • ksqlDB is a valid Kafka Connect cluster
  • permissions is a list of valid KsqlDB permissions (See Table)
Available KafkaConnect PermissionsDescription
ksqldbAccessGrants all permissions on the KsqlDB Cluster.

Platform permissions

# Grants Platform permissions
- resourceType: PLATFORM
permissions:
- userView
- datamaskingView
  • resourceType: PLATFORM
  • permissions is a list of valid Platform permissions
Available Console permissionsDescription
clusterConnectionsManagePermission to add / edit / remove Kafka clusters on Console
certificateManagePermission to add / edit / remove TLS Certificates on Console
userManagePermission to manage Console users, groups and permissions
userViewPermission to view Console users, groups and permissions
datamaskingManagePermission to manage Data policies (masking rules)
datamaskingViewPermission to view Data policies
notificationChannelManagePermission to manage Integration channels
auditLogViewPermission to browse audit log
taasViewPermission to view Application Catalog
chargebackManagePermission to view Chargeback and manage its settings
sqlManagePermission to view indexed topics and create SQL queries