[source-tiktok-marketing] Null values in data ingestion fields

[fernando-heitor]

I am currently using the TikTok connector version 4.3.6. However, during data ingestion, it appears that the records are being loaded as null values.

After reviewing the GitHub repository, it seems that the Airbyte connector automatically defines the schema, which does not match the data provided by the API.

I conducted several tests using Postman, where I was able to retrieve the data correctly, passing the metrics and dimensions parameters without issues.

A clear example of the issue arises when querying the ad_groups_reports_daily endpoint via Airbyte. All records appear as null in the dimension fields. For instance, fields such as:

advertiser_id
campaign_id

When tested via Postman, the same fields return the correct values. Below is an example of the URL used during testing:

https://business-api.tiktok.com/open_api/v1.3/report/integrated/get/?advertiser_id=123456789&metrics=["advertiser_id","campaign_name","campaign_id","adgroup_name","placement_type","tt_app_id","tt_app_name","mobile_app_id","promotion_type","dpa_target_audience_type","conversion","cost_per_conversion","conversion_rate","real_time_conversion","real_time_cost_per_conversion","real_time_conversion_rate","result","cost_per_result","result_rate","real_time_result","real_time_cost_per_result","real_time_result_rate","secondary_goal_result","cost_per_secondary_goal_result","secondary_goal_result_rate"]&dimensions=["adgroup_id", "stat_time_day"]&report_type=BASIC&data_level=AUCTION_ADGROUP&start_date=2024-01-09&end_date=2024-01-10

Additionally, the YAML file containing the configurations for the Airbyte connector can be found at the following link:
Airbyte TikTok Connector Manifest.yaml

Schema default

This issue suggests a mismatch between the API response structure and the schema automatically generated by the Airbyte connector.

Would it be possible to make the selection of fields optional to avoid such issues? Allowing users to specify the desired fields for metrics and dimensions manually could provide more flexibility and prevent data mismatches.