Building a Language Toxicity Classification Model on Google Cloud

Why Moderation Needs Help

Online communities are a large part of how people interact on the internet. These communities need moderation so that participants follow the community guidelines and avoid clear and obvious policy violations.

Bad actors make moderation hard. Moderators spend a lot of time on the volume of violations. Some violations still get past them. In some communities, a missed violation can do serious harm. One example is an obscene comment that gets past the moderators of an online community for school children. Moderation work also raises other problems, such as keeping judgments consistent. Most important, constant exposure to graphic and obscene material can harm the moderators' mental health.

This article builds an end-to-end language toxicity classification model that labels a comment as toxic or clean.

From AutoML Natural Language to Gemini Tuning

In 2019, this article used Google Cloud AutoML for Natural Language to train a custom text classification model. That service has since been retired:

• Legacy AutoML Natural Language shut down on August 7, 2024. Its text features moved to Vertex AI as AutoML Text.
• AutoML Text on Vertex AI shut down on June 15, 2025. Google now recommends Gemini prompting and tuning for text classification.
• In April 2026, Google renamed Vertex AI to Gemini Enterprise Agent Platform.

This article now trains the classifier with supervised fine-tuning of Gemini 3.5 Flash on Agent Platform. Supervised fine-tuning adapts a Gemini model to a well-defined task with labeled examples. Classification is one of the tasks it suits. Google's guide for AutoML Text users compares the two workflows. The main differences for this project are:

• Tuning data is in JSON Lines files in Cloud Storage. Each line is one training example.
• You split the data yourself into a training file and an optional validation file.
• Gemini returns a label without a confidence score. We measure precision and recall ourselves on a held-out test set.

About the Dataset

The data comes from the Toxic Comment Classification Challenge on Kaggle by Jigsaw and Google. Each comment has six labels: toxic, severe_toxic, obscene, threat, insult and identity_hate. We label a comment toxic when any of the six labels is set. All other comments are clean.

The 2019 version sampled 16,000 toxic and 16,000 clean comments for AutoML. For Gemini tuning, Google recommends starting with about 100 examples and scaling up to thousands if needed. The quality of the examples matters more than their number. We sample 1,000 training examples, 64 validation examples and 100 test examples for each class.

Part 1: Set Up the Project

(1). Open the Google Cloud console and select the project you want to use. Billing must be enabled on the project.

(2). Activate Cloud Shell from the toolbar at the top of the console. Cloud Shell is a virtual machine with the Google Cloud CLI already installed.

(3). Enable the Agent Platform API. Replace PROJECT_ID with your project ID.

bashCopy
gcloud config set project PROJECT_ID
gcloud services enable aiplatform.googleapis.com

(4). Create a Cloud Storage bucket for the data. Replace BUCKET_NAME with a globally unique name. Supervised tuning of Gemini 3.5 Flash runs in us-central1 or europe-west4. We use us-central1 in this article.

bashCopy
gcloud storage buckets create gs://BUCKET_NAME --location=us-central1

Part 2: Download the Dataset to Cloud Storage

(1). Install the Kaggle command-line interface. The current version needs Python 3.11 or later.

bashCopy
pip install kaggle

If pip reports an externally-managed-environment error, install the CLI with pipx install kaggle instead.

(2). Create a Kaggle API token. Sign in to Kaggle, go to the API settings page and click Generate New Token. Copy the token and set it as an environment variable in Cloud Shell.

bashCopy
export KAGGLE_API_TOKEN=YOUR_TOKEN

(3). Open the competition page on Kaggle and accept the competition rules. The CLI cannot download competition data until you accept them.

(4). Download and unzip the competition files.

bashCopy
kaggle competitions download jigsaw-toxic-comment-classification-challenge -p . --unzip
ls

If the folder contains train.csv.zip in place of train.csv, unzip that file as well.

bashCopy
unzip train.csv.zip

(5). Upload the training file to the bucket.

bashCopy
gcloud storage cp train.csv gs://BUCKET_NAME/toxicity/

Part 3: Prepare the Dataset for Modeling

We prepare the tuning files in a notebook on Agent Platform Workbench. Workbench was called AI Platform Notebooks when this article was first written.

(1). In the console, open Agent Platform, go to Workbench and open the Instances page. Click Create new, give the instance a name and click Create.

(2). When the instance is ready, click Open JupyterLab and open a new Python notebook.

(3). Install the libraries for this article in the first cell, then restart the kernel.

pythonCopy
%pip install --quiet --upgrade google-genai google-cloud-storage pandas scikit-learn

(4). Copy the training file from the bucket to the notebook.

pythonCopy
from google.cloud import storage

PROJECT_ID = 'PROJECT_ID'    # change this to your project ID
BUCKET_NAME = 'BUCKET_NAME'  # change this to your bucket

bucket = storage.Client(project=PROJECT_ID).bucket(BUCKET_NAME)
bucket.blob('toxicity/train.csv').download_to_filename('train.csv')

(5). Each tuning example holds a system instruction, the comment as the user's message and the label as the model's answer. The code below labels each comment, samples a balanced set, writes the training and validation files in JSON Lines format and keeps a test set for later.

pythonCopy
import json

import pandas as pd

data = pd.read_csv('train.csv')

# a comment is toxic when any of the six toxicity labels is 1
label_columns = ['toxic', 'severe_toxic', 'obscene', 'threat', 'insult', 'identity_hate']
data['label'] = (data[label_columns].sum(axis=1) > 0).map({True: 'toxic', False: 'clean'})
print(data['label'].value_counts())

# sample a balanced set: 1,000 training, 64 validation and 100 test examples per class
n_train, n_val, n_test = 1000, 64, 100
parts = {'train': [], 'validation': [], 'test': []}
for label, group in data.groupby('label'):
    group = group.sample(n=n_train + n_val + n_test, random_state=42)
    parts['train'].append(group[:n_train])
    parts['validation'].append(group[n_train:n_train + n_val])
    parts['test'].append(group[n_train + n_val:])
splits = {name: pd.concat(frames).sample(frac=1, random_state=42) for name, frames in parts.items()}

SYSTEM_INSTRUCTION = ('Classify the comment as toxic or clean. '
                      'Answer with one word: toxic or clean.')

def to_example(comment, label):
    """Return one tuning example in the Gemini supervised tuning format."""
    return {
        'systemInstruction': {'parts': [{'text': SYSTEM_INSTRUCTION}]},
        'contents': [
            {'role': 'user', 'parts': [{'text': comment}]},
            {'role': 'model', 'parts': [{'text': label}]},
        ],
    }

# write the training and validation files in JSON Lines format
for name in ['train', 'validation']:
    with open(f'{name}.jsonl', 'w') as f:
        for comment, label in zip(splits[name]['comment_text'], splits[name]['label']):
            f.write(json.dumps(to_example(comment, label)) + '\n')

# keep the test examples for evaluation after tuning
splits['test'][['comment_text', 'label']].to_csv('test.csv', index=False)

Each line of train.jsonl looks like this, with the comment text in place of COMMENT:

jsonCopy
{"systemInstruction": {"parts": [{"text": "Classify the comment as toxic or clean. Answer with one word: toxic or clean."}]}, "contents": [{"role": "user", "parts": [{"text": "COMMENT"}]}, {"role": "model", "parts": [{"text": "clean"}]}]}

(6). Upload the two tuning files to the bucket.

pythonCopy
for name in ['train.jsonl', 'validation.jsonl']:
    bucket.blob(f'toxicity/{name}').upload_from_filename(name)

Part 4: Tune the Model

We create the tuning job with the Google Gen AI SDK. The job runs in us-central1 and tunes the base model gemini-3.5-flash.

pythonCopy
import time

from google import genai
from google.genai import types

client = genai.Client(enterprise=True, project=PROJECT_ID, location='us-central1')

tuning_job = client.tunings.tune(
    base_model='gemini-3.5-flash',
    training_dataset=types.TuningDataset(
        gcs_uri=f'gs://{BUCKET_NAME}/toxicity/train.jsonl',
    ),
    config=types.CreateTuningJobConfig(
        tuned_model_display_name='toxicity-classifier',
        validation_dataset=types.TuningValidationDataset(
            gcs_uri=f'gs://{BUCKET_NAME}/toxicity/validation.jsonl',
        ),
    ),
)

# check the job once a minute until it ends
while not tuning_job.has_ended:
    print(tuning_job.state)
    time.sleep(60)
    tuning_job = client.tunings.get(name=tuning_job.name)

print(tuning_job.state)
print(tuning_job.tuned_model.model)
print(tuning_job.tuned_model.endpoint)

You can also create the job in the console. Go to the Agent Platform Studio page and click Create tuned model. Choose the base model and the region, then select the training file and the validation file in Cloud Storage and click Start Tuning. The tuned model is stored in Model Registry, and Agent Platform creates an endpoint for it.

Part 5: Evaluate and Test the Model

Tuned Gemini 3.5 Flash models are served from the us and eu multi-region endpoints. We create a second client for the us endpoint and send each test comment to the tuned model. We pass the same system instruction that we used in tuning. For Gemini 3 and later models, Google recommends the MINIMAL thinking level for tuned tasks. The tuned model learned to answer without a thinking step.

pythonCopy
import pandas as pd
from google import genai
from google.genai import types
from sklearn.metrics import classification_report, confusion_matrix

# tuned Gemini 3.5 Flash models are served from the us and eu multi-region endpoints
predict_client = genai.Client(enterprise=True, project=PROJECT_ID, location='us')

# use the same system instruction as in tuning, with minimal thinking
config = types.GenerateContentConfig(
    system_instruction=SYSTEM_INSTRUCTION,
    thinking_config=types.ThinkingConfig(thinking_level=types.ThinkingLevel.MINIMAL),
)

def classify(comment):
    """Return the tuned model's label for one comment."""
    response = predict_client.models.generate_content(
        model=tuning_job.tuned_model.endpoint,
        contents=comment,
        config=config,
    )
    answer = (response.text or '').strip().lower()
    return answer if answer in ('toxic', 'clean') else 'unknown'

test = pd.read_csv('test.csv')
test['prediction'] = test['comment_text'].apply(classify)

print(classification_report(test['label'], test['prediction'], zero_division=0))
print(confusion_matrix(test['label'], test['prediction'], labels=['clean', 'toxic', 'unknown']))

The report shows precision and recall for each class. The confusion matrix shows how many comments of each class the model labeled clean or toxic. Any answer that is not one of the two labels is counted as unknown.

To try the model by hand, go to the Agent Platform Studio page, find your model in the Tuned Models table and select Test. Paste in a comment and read the model's answer. A good first check is a clean text, such as the introduction to the Wikipedia article on Nigeria, followed by a few clear and obvious toxic examples.

Your applications can call the tuned model in the same way through the Gen AI SDK or the REST API.

Part 6: Clean Up

When you no longer need them, delete the Cloud Storage bucket and stop or delete the Workbench instance.

bashCopy
gcloud storage rm --recursive gs://BUCKET_NAME/

Conclusion

This article walked through building a language classification model for a custom use case on Google Cloud. In 2019, AutoML Natural Language trained and hosted the model. Today, supervised fine-tuning adapts a Gemini model to the same task with a few thousand labeled examples. Agent Platform hosts the tuned model behind an endpoint. You can use this walkthrough as a template for other text classification products.

The 2019 version of this article was based on material from my book Building Machine Learning and Deep Learning Models on Google Cloud Platform, published by Apress (Springer Nature).