[source-mysql] Unable to Pass Source Check Job with [source-mysql] Connector Version 3.11.0

[HaJunYoo]

Connector Name

source-mysql

Connector Version

3.11.0

What step the error happened?

Configuring a new connector

Relevant information

We are currently using Airbyte deployed on EKS with Helm chart version 1.2.0.

When using the MySQL source connector version 3.11.0, we encounter an issue where the Connector Job - Check Job becomes stuck in the "Pending" state. During the Check Job phase, the logs show "----- START CHECK -----" but do not proceed any further. The job remains in this state indefinitely before eventually failing.

Downgrading to an earlier version (3.5.1) of the mysql source connector resolves the problem, allowing the Check Job to complete successfully.

Could you help us understand the root cause of this issue and provide any recommendations for resolving it while using the latest connector version?

Relevant log output

Autoscroll:On      FullScreen:Off     Timestamps:Off     Wrap:Off                                                                                    │
│ Unsetting empty environment variable 'DATA_PLANE_SERVICE_ACCOUNT_CREDENTIALS_PATH'                                                                                                                                                      │
│ Unsetting empty environment variable 'KEYCLOAK_INTERNAL_REALM_ISSUER'                                                                                                                                                                   │
│ Unsetting empty environment variable 'DATA_PLANE_SERVICE_ACCOUNT_EMAIL'                                                                                                                                                                 │
│ Unsetting empty environment variable 'AWS_SECRET_ACCESS_KEY'                                                                                                                                                                            │
│ Unsetting empty environment variable 'KEYCLOAK_CLIENT_ID'                                                                                                                                                                               │
│ Unsetting empty environment variable 'AWS_ACCESS_KEY_ID'                                                                                                                                                                                │
│ Unsetting empty environment variable 'CONTROL_PLANE_AUTH_ENDPOINT'                                                                                                                                                                      │
│ Unsetting empty environment variable 'S3_PATH_STYLE_ACCESS'                                                                                                                                                                             │
│ 2025-01-20 11:18:59,860 [main]    INFO    i.a.c.ApplicationKt(main):12 - Sidecar start                                                                                                                                                  │
│ 2025-01-20 11:19:01,664 [main]    INFO    i.m.c.e.DefaultEnvironment(<init>):168 - Established active environments: [worker-v2, control-plane, oss, local-secrets]                                                                      │
│ 2025-01-20 11:19:05,356 [pool-4-thread-1]    INFO    i.a.c.i.LineGobbler(voidCall):166 -                                                                                                                                                │
│ 2025-01-20 11:19:05,469 [pool-4-thread-1]    INFO    i.a.c.i.LineGobbler(voidCall):166 - ----- START CHECK -----                                                                                                                        │
│ 2025-01-20 11:19:05,469 [pool-4-thread-1]    INFO    i.a.c.i.LineGobbler(voidCall):166 -                                                                                                                                                │
│

Contribute

• Yes, I want to contribute

[marcosmarxm]

@HaJunYoo are you able to monitor the cpu/memory used by the check job? Maybe this can provide some insights about what is causing it to hang.

[HaJunYoo]

Issue occurred in the connector-sidecar container (mysql-check-job pod).

it consistently uses less than 250m CPU and 500Mi memory.
I have configured the helm values.yaml global environment variables for resource allocation as follows

# Resource requests and limits for the main job container
JOB_MAIN_CONTAINER_CPU_REQUEST: "0.5"     # Minimum CPU cores
JOB_MAIN_CONTAINER_CPU_LIMIT: "2"         # Maximum CPU cores
JOB_MAIN_CONTAINER_MEMORY_REQUEST: "1Gi"   # Minimum memory
JOB_MAIN_CONTAINER_MEMORY_LIMIT: "2Gi"    # Maximum memory

• Image: First 3 minutes after job start

• error log (container - connector-sidecar)

 Unsetting empty environment variable 'DATA_PLANE_SERVICE_ACCOUNT_CREDENTIALS_PATH'                                                                                                                   │
│ Unsetting empty environment variable 'KEYCLOAK_INTERNAL_REALM_ISSUER'                                                                                                                                │
│ Unsetting empty environment variable 'DATA_PLANE_SERVICE_ACCOUNT_EMAIL'                                                                                                                              │
│ Unsetting empty environment variable 'AWS_SECRET_ACCESS_KEY'                                                                                                                                         │
│ Unsetting empty environment variable 'KEYCLOAK_CLIENT_ID'                                                                                                                                            │
│ Unsetting empty environment variable 'AWS_ACCESS_KEY_ID'                                                                                                                                             │
│ Unsetting empty environment variable 'CONTROL_PLANE_AUTH_ENDPOINT'                                                                                                                                   │
│ Unsetting empty environment variable 'S3_PATH_STYLE_ACCESS'                                                                                                                                          │
│ 2025-01-23 01:47:22,873 [main]    INFO    i.a.c.ApplicationKt(main):12 - Sidecar start                                                                                                               │
│ 2025-01-23 01:47:24,766 [main]    INFO    i.m.c.e.DefaultEnvironment(<init>):168 - Established active environments: [worker-v2, control-plane, oss, local-secrets]                                   │
│ 2025-01-23 01:47:28,396 [pool-4-thread-1]    INFO    i.a.c.i.LineGobbler(voidCall):177 -                                                                                                             │
│ 2025-01-23 01:47:28,512 [pool-4-thread-1]    INFO    i.a.c.i.LineGobbler(voidCall):177 - ----- START CHECK -----                                                                                     │
│ 2025-01-23 01:47:28,513 [pool-4-thread-1]    INFO    i.a.c.i.LineGobbler(voidCall):177 -                                                                                                             │
│ 2025-01-23 01:56:28,497 [main]    WARN    i.a.c.ConnectorWatcher(waitForConnectorOutput):104 - Failed to find output files from connector within timeout of 9 minute(s). Is the connector still runn │
│ 2025-01-23 01:56:28,499 [main]    INFO    i.a.c.ConnectorWatcher(failWorkload):311 - Failing workload 09642960-1b9f-49a5-9ae4-c803e45c0824_d6585407-d2cf-4d7d-a477-35029073fa25_0_check.             │
│ 2025-01-23 01:56:28,520 [main]    INFO    i.a.c.ConnectorWatcher(exitFileNotFound):229 - Deliberately exiting process with code 2. 

Please let me know if there are any areas where I can take action or contribute to resolving this issue.