Example 2

We illustrate how to evaluate a Transformer Network for classifying the trajectories of the MNIST stroke dataset. This examples seeks to partially reproduce the results reported in [1]

The example is structured as follows:: 1. Setup dependencies

2. Definition of parameters

3. Loading Data

4. Loading the model

5. Training and evaluation

6. References

Note

You can access the script of this example.

1. Setup dependencies

Import all the dependencies:

from tensorflow import keras
from pactus import Dataset
from pactus.models import TransformerModel

2. Definition of parameters

We define a random seed for reproducibility

SEED = 0

3. Loading Data

To load the MNIST stroke dataset we can simply do:

dataset = Dataset.mnist_stroke()

Then, we can create a train/test split as proposed on [1]:

train, test = dataset.cut(60_000)

4. Loading the model

Since transformers are able to deal with data of arbitrary length, there is no need to create a featurizer for this model, and we can directly use it:

model = TransformerModel(
    head_size=512,
    num_heads=4,
    num_transformer_blocks=4,
    optimizer=keras.optimizers.Adam(learning_rate=1e-4),
    random_state=SEED,
)

5. Training and evaluation

Training and evaluation can be conducted as follows:

# Train the model on the train dataset
model.train(train, dataset, epochs=150, batch_size=64, checkpoint=checkpoint)

# Evaluate the model on a test dataset
evaluation = model.evaluate(test)

# Print the evaluation
evaluation.show()

It may take a lot of time to complete depending on the hardware running the script.

6. References

[1] BAE, Keywoong; LEE, Suan; LEE, Wookey. Transformer Networks for Trajectory Classification. En 2022 IEEE International Conference on Big Data and Smart Computing (BigComp). IEEE, 2022. p. 331-333.