Datapal

# Datapal Marketing Tools - The Backend Side Overview

# Introduction

This document provides an overview of the architecture and back-end components that build Unified Datapal Marketing tool.

# Architecture Overview

The tools are designed to view/modify/make update on marketing data, provided by marketing agencies (e.g True Media, Radar,...). It consists of the followings:

  1. Conversion Mapping Tool

    The Conversion Mapping Tool facilitates the mapping of marketing data to conversion events, allowing marketers to track the effectiveness of their campaigns and optimize their strategies based on conversion metrics. It provides an intuitive interface for defining mapping rules and visualizing the correlation between marketing activities and conversion outcomes.

  2. Custom Grouping Tool

    The Custom Grouping Tool enables marketers to create custom segments or groups based on specific attributes within the marketing data. It offers flexibility in segmenting campaigns, allowing marketers to tailor their targeting strategies. The tool provides features for defining grouping rules, managing segmentations, and analyzing segment performance.

  3. Data-Stride

  4. Vi-Motus

  5. Account Mapping Service: The Account Mapping Service is responsible for managing the relationship between organizations (equally marketing agencies) and marketing accounts (equally advertisers). It helps individual backends to show appropriate campaign data for different agencies.

alt text

Figure 1: Datapal Marketing Tools - The backend side overview

# Agency Onboarding

Agency is the concept from marketing companies like Radar, while Organization is the concept introduced by Datapal for product/client management.

In the scope of the tools, a marketing agency will is equivalent to an organization. Hence, in this document, we will use agency and organization interchangably.

# Custom Grouping Tool & Vi-Motus

The two tools have the same onboard process for new agency.

N.o Steps Person in-charge Notes
1 Prepare agency information, which includes agency name,
parent agency (if existing),...		 Agency Representative
2 Add the agency to Authentication Service * Datapal Authen admin tool (if exist)
* Or, Datapal Admin/Developers		 This step results in a unique org_id
3 Add agency's users to Authentication Service * Datapal OAuth2 login Button on UI (if exist)
* Datapal Authen admin tool (if exist)
* Or, Datapal Admin/Developers		 Add the users (requested by user directly or by the agency's representative)
to organization id org_id generated in step 2.
4 Provide BigQuery table ID where the tool will pull data.
Provide BigQuery table ID where the working data will locate in.		 Agency Representative Make sure the tables are granted to be used by Datapal.
5 Provide Account List for the agency. Agency Representative The list should be in a CSV file, having 5 columns: platform, account_name, account_id, client_code, description.
6 Tell the tool that the organization (step 3) will use data from the
BigQuery table ID (step 4, input table) and will filtered by the
account list (step 5). Also specify output table ID, where the working
data will locate in.		 * Admin tool (in development)
* Developers (manually)		 In case of doing this step manually, execute two below commands:

* Each tool has an admin endpoint to maintain the mapping between organization_id and working table.

- Vi-motus:
curl -X 'POST' 'https://<TOOL_API_URL>/admin/onboard/<ORGANIZATION_ID>' -H 'accept: application/json' -H 'Content-Type: application/json' -d '{"out_table": "project_id.dataset.the_output_table_id", "input_table": "project_id.dataset.the_input_table_id", "channel_table": "project_id.dataset.channel_table_id", "brand_table": "project_id.dataset.brand_table_id", "location_table": "project_id.dataset.location_table_id"}'
- Custom-Grouping Tool: It's quite similar to Vi-Motus, visit the tool's document page for more detail.

* Then init the account list in the Account Mapping Service. No matter how many time we call the API
for the same account list. The API will do de-duplication before insertion.

curl -X 'POST' 'http://<ACCOUNT_MAPPING_API_URL>/admin/migrate-accounts/<ORGANIZATION_ID>' -H 'accept: application/json' -H 'Content-Type: multipart/form-data' -F 'file=@<PATH_TO_ACCOUNT_CSV_FILE>;type=text/csv'
7 Schedule a job to transfer data from input table to output table ID * Datapal Admin/Developers * Schedule using Cloud Scheduler.
- Developers provide an API endpoint (specificalize for each tool to sync data). Admin/Develops will config
Cloud Run to call to the API periodically.
For example, Vi-Motus, schedule Cloud Scheduler to call to https://<TOOL_URL>/admin/view-to-table?view_id=<input_table_id>&table_id=<output_table_id>&org_id=<org_id>
* Schedule using BigQuery Query Scheduler.
In each code folder for every tool, developers include SQL code to synchronize data. Schedule a job to
execute this code using the BigQuery Query Scheduler.
8 Notify user to use the tool by accounts created in step 3 * Agency Representative
* Mail center

# Supplymentary

  1. Granting BigQuery Permissions

    In step 4 of the onboarding process. The agency has to grant write/read permission to Datapal, which is necessary for Datapal's tools to read and update data.

    alt text

    Figure 2: Granting BigQuery permissions for Datapal service account

    The recommendation is that the agency should create a blank dataset, then grant read/write permission to the dataset, Datapal will do the rest. Utimately, the agency will come to the dataset to see output and use the output data for its own way.

    Depends on which tool the agency want to use, the number of BigQuery tables to be granted could be variant, in particularly:

    Custom-Grouping Tool
    Table Initialization Required API Param Required Description
    Input View/Table Yes Yes The input view/table where data is synced to output table, on which the tool will work (read/update).
    Output Table No No The output table where the tool will work on. Usually, agency need having access to this table for post-processing on the output data (analyze, use in other tools such as Tableau).
    Benchmark Metric Table No No Where benchmark metrics are saved as input for drop-down menu(s) in the tool. It will be automatically initialized if not existing.
    Tactical Benchmark Metric Table No No Same to Benchmark Metric Table.
    Funnel Level Table No No Same to Benchmark Metric Table, but for saving funnel levels instead of benchmark metrics.
    Vi-Motus
    Table Initialization Required API Param Required Description
    Input View/Table Yes Yes The input view/table where data is synced to output table, on which the tool will work (read/update).
    Output Table No Yes The output table where the tool will work on. Usually, agency need having access to this table for post-processing on the output data (analyze, use in other tools such as Tableau).
    Channel Table No Yes Where channels are saved as input for drop-down menu(s) in the tool. It will be automatically initialized if not existing.
    Location Table No Yes Same to Channel Table but for saving locations.
    Brand Table No Yes Same to Channel Table but for saving brands and passcodes.