Skip to content

1D Models

k3im.cait_1d.CAiT_1DModel

An extention of Class Attention in Image Transformers (CAiT) reimplemented for 1D Data. The model expects 1D data of shape (batch, seq_len, channels)

Parameters:

Name Type Description Default
`seq_len`

number of steps

 required
`patch_size`

number steps in a patch

required
`num_classes`

output classes for classification

 required
`dim`

projection dim for patches,

 required
`dim_head`

size of each attention head

 required
`mlp_dim`

Projection Dim in transformer after each MultiHeadAttention layer

 required
`depth`

number of patch transformer units

 required
`cls_depth`

number of transformer units applied to class attention transformer

 required
`heads`

number of attention heads

 required
`channels`

number of features/channels in the input default 3

 required
`dropout_rate`

dropout applied to MultiHeadAttention in class and patch transformers

 required
Source code in k3im/cait_1d.py 
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
def CAiT_1DModel(
    seq_len:int,
    patch_size:int,
    num_classes:int,
    dim:int,
    dim_head:int,
    mlp_dim:int,
    depth:int,
    cls_depth:int,
    heads:int,
    channels:int=3,
    dropout_rate:float=0.0,
):
    """ 
    An extention of Class Attention in Image Transformers (CAiT) reimplemented for 1D Data. 
    The model expects 1D data of shape `(batch, seq_len, channels)`

    Args:
        `seq_len`: number of steps
        `patch_size`: number steps in a patch 
        `num_classes`: output classes for classification
        `dim`: projection dim for patches,
        `dim_head`: size of each attention head
        `mlp_dim`: Projection Dim in transformer after each MultiHeadAttention layer
        `depth`: number of patch transformer units
        `cls_depth`: number of transformer units applied to class attention transformer
        `heads`: number of attention heads
        `channels`: number of features/channels in the input default `3`
        `dropout_rate`: dropout applied to MultiHeadAttention in class and patch transformers
    """
    assert seq_len % patch_size == 0
    patch_dim = channels * patch_size
    i_p = layers.Input((seq_len, channels))
    patches = layers.Reshape((-1, patch_dim))(i_p)
    patches = layers.LayerNormalization()(patches)
    patches = layers.Dense(dim)(patches)
    patches = layers.LayerNormalization()(patches)
    pos_embedding = posemb_sincos_1d(patches)
    patches += pos_embedding
    dim = ops.shape(patches)[-1]
    patches = Transformer(dim, depth, heads, dim_head, mlp_dim, dropout_rate=dropout_rate)(patches)
    _, cls_token = CLS_Token(dim)(patches)
    cls_token = Transformer(dim, cls_depth, heads, dim_head, mlp_dim, dropout_rate=dropout_rate)(
        cls_token, patches
    )
    cls_token = ops.squeeze(cls_token, axis=1)
    o_p = layers.Dense(num_classes)(cls_token)
    return keras.Model(inputs=i_p, outputs=o_p)

options: show_signature: true

k3im.cct_1d.CCT_1DModel

Create a Convolutional Transformer for sequences.

Parameters:

Name Type Description Default
input_shape

A tuple of (seq_len, num_channels).

 required
num_heads

An integer.

 required
projection_dim

An integer representing the projection dimension.

 required
kernel_size

An integer representing the size of the convolution window.

 required
stride

An integer representing the stride of the convolution.

 required
padding

One of 'valid', 'same' or 'causal'. Causal is for decoding.

 required
transformer_units

A list of integers representing the number of units in each transformer layer.

 required
stochastic_depth_rate

A float representing the drop probability for the stochastic depth layer.

 required
transformer_layers

An integer representing the number of transformer layers.

 required
num_classes

An integer representing the number of classes for classification.

 required
positional_emb

Boolean, whether to use positional embeddings.

 False
Source code in k3im/cct_1d.py 
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
def CCT_1DModel(
    input_shape,
    num_heads,
    projection_dim,
    kernel_size,
    stride,
    padding,
    transformer_units,
    stochastic_depth_rate,
    transformer_layers,
    num_classes,
    positional_emb=False,
):
    """
    Create a Convolutional Transformer for sequences.

    Args:
        input_shape: A tuple of (seq_len, num_channels).
        num_heads: An integer.
        projection_dim: An integer representing the projection dimension.
        kernel_size: An integer representing the size of the convolution window.
        stride: An integer representing the stride of the convolution.
        padding: One of 'valid', 'same' or 'causal'. Causal is for decoding.
        transformer_units: A list of integers representing the number of units
            in each transformer layer.
        stochastic_depth_rate: A float representing the drop probability for the
            stochastic depth layer.
        transformer_layers: An integer representing the number of transformer layers.
        num_classes: An integer representing the number of classes for classification.
        positional_emb: Boolean, whether to use positional embeddings.

    """
    inputs = layers.Input(input_shape)

    # Encode patches.

    cct_tokenizer = CCTTokenizer1D(
        kernel_size,
        stride,
        padding,
        n_output_channels=[64, projection_dim],
        n_conv_layers=2,
    )
    encoded_patches = cct_tokenizer(inputs)

    # Apply positional embedding.
    if positional_emb:
        sequence_length = encoded_patches.shape[1]
        encoded_patches += PositionEmbedding(sequence_length=sequence_length)(
            encoded_patches
        )

    # Calculate Stochastic Depth probabilities.
    dpr = [x for x in np.linspace(0, stochastic_depth_rate, transformer_layers)]

    # Create multiple layers of the Transformer block.
    for i in range(transformer_layers):
        # Layer normalization 1.
        x1 = layers.LayerNormalization(epsilon=1e-5)(encoded_patches)

        # Create a multi-head attention layer.
        attention_output = layers.MultiHeadAttention(
            num_heads=num_heads, key_dim=projection_dim, dropout=0.1
        )(x1, x1)

        # Skip connection 1.
        attention_output = StochasticDepth(dpr[i])(attention_output)
        x2 = layers.Add()([attention_output, encoded_patches])

        # Layer normalization 2.
        x3 = layers.LayerNormalization(epsilon=1e-5)(x2)

        # MLP.
        x3 = mlp(x3, hidden_units=transformer_units, dropout_rate=0.1)

        # Skip connection 2.
        x3 = StochasticDepth(dpr[i])(x3)
        encoded_patches = layers.Add()([x3, x2])

    # Apply sequence pooling.
    representation = layers.LayerNormalization(epsilon=1e-5)(encoded_patches)
    weighted_representation = SequencePooling()(representation)

    # Classify outputs.
    logits = layers.Dense(num_classes)(weighted_representation)
    # Create the Keras model.
    model = keras.Model(inputs=inputs, outputs=logits)
    return model

k3im.convmixer_1d.ConvMixer1DModel

ConvMixer model for 1D data.

Parameters:

Name Type Description Default
`seq_len`

number of steps

 required
`n_features`

number of features/channels in the input default 3

 required
`filters`

number of filters in the convolutional stem

 required
`depth`

number of conv mixer blocks

 required
`kernel_size`

kernel size for the depthwise convolution

 required
`patch_size`

number steps in a patch

 required
`num_classes`

output classes for classification

 required
Source code in k3im/convmixer_1d.py 
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
def ConvMixer1DModel(
    seq_len=32,
    n_features=3,
    filters=256,
    depth=8,
    kernel_size=5,
    patch_size=2,
    num_classes=10,
):
    """ 
    ConvMixer model for 1D data.

    Args:
        `seq_len`: number of steps
        `n_features`: number of features/channels in the input default `3`
        `filters`: number of filters in the convolutional stem
        `depth`: number of conv mixer blocks
        `kernel_size`: kernel size for the depthwise convolution
        `patch_size`: number steps in a patch
        `num_classes`: output classes for classification

    """
    inputs = keras.Input((seq_len, n_features))

    # Extract patch embeddings.
    x = conv_stem(inputs, filters, patch_size)

    # ConvMixer blocks.
    for _ in range(depth):
        x = conv_mixer_block(x, filters, kernel_size)

    # Classification block.
    x = layers.GlobalAvgPool1D()(x)
    outputs = layers.Dense(num_classes)(x)

    return keras.Model(inputs, outputs)

k3im.eanet_1d.EANet1DModel

Create an External Attention Network for 1D data.

Parameters:

Name Type Description Default
`seq_len`

number of steps

 required
`patch_size`

number steps in a patch

 required
`num_classes`

output classes for classification

 required
`dim`

projection dim for patches,

 required
`depth`

number of patch transformer units

 required
`heads`

number of attention heads

 required
`mlp_dim`

Projection Dim in transformer after each MultiHeadAttention layer

 required
`dim_coefficient`

coefficient for increasing the number of heads

 required
`attention_dropout`

dropout applied to MultiHeadAttention in class and patch transformers

 required
`channels`

number of features/channels in the input default 3

 required
Source code in k3im/eanet_1d.py 
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
def EANet1DModel(
    seq_len,
    patch_size,
    num_classes,
    dim,
    depth,
    heads,
    mlp_dim,
    dim_coefficient=4,
    attention_dropout=0.0,
    channels=3,
):
    """
    Create an External Attention Network for 1D data.

    Args:
        `seq_len`: number of steps
        `patch_size`: number steps in a patch
        `num_classes`: output classes for classification
        `dim`: projection dim for patches,
        `depth`: number of patch transformer units
        `heads`: number of attention heads
        `mlp_dim`: Projection Dim in transformer after each MultiHeadAttention layer
        `dim_coefficient`: coefficient for increasing the number of heads
        `attention_dropout`: dropout applied to MultiHeadAttention in class and patch transformers
        `channels`: number of features/channels in the input default `3`

    """
    assert seq_len % patch_size == 0
    patch_dim = channels * patch_size
    i_p = layers.Input((seq_len, channels))
    patches = layers.Reshape((-1, patch_dim))(i_p)
    patches = layers.LayerNormalization()(patches)
    patches = layers.Dense(dim)(patches)
    patches = layers.LayerNormalization()(patches)
    pos_embedding = posemb_sincos_1d(patches)
    patches += pos_embedding
    patches = Transformer(
        dim=dim,
        depth=depth,
        heads=heads,
        mlp_dim=mlp_dim,
        dim_coefficient=dim_coefficient,
        attention_dropout=attention_dropout,
        projection_dropout=0,
    )(patches)
    patches = layers.GlobalAveragePooling1D(name="avg_pool")(patches)
    o_p = layers.Dense(num_classes)(patches)

    return keras.Model(inputs=i_p, outputs=o_p)

k3im.fnet_1d.FNet1DModel

Instantiate a FNet model for 1D data.

Parameters:

Name Type Description Default
seq_len

An integer representing the number of steps in the input sequence.

 required
patch_size

An integer representing the number of steps in a patch (default=4).

required
num_classes

An integer representing the number of classes for classification.

 required
dim

An integer representing the projection dimension.

 required
depth

An integer representing the number of transformer layers.

 required
channels

An integer representing the number of channels in the input.

 3
dropout_rate

A float representing the dropout rate.

 0.0
Source code in k3im/fnet_1d.py 
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
def FNet1DModel(
    seq_len, patch_size, num_classes, dim, depth, channels=3, dropout_rate=0.0
):
    """
    Instantiate a FNet model for 1D data.

    Args:
        seq_len: An integer representing the number of steps in the input sequence.
        patch_size: An integer representing the number of steps in a
            patch (default=4).  
        num_classes: An integer representing the number of classes for classification.
        dim: An integer representing the projection dimension.
        depth: An integer representing the number of transformer layers.
        channels: An integer representing the number of channels in the input.
        dropout_rate: A float representing the dropout rate.
    """
    assert seq_len % patch_size == 0
    num_patches = seq_len // patch_size
    patch_dim = channels * patch_size
    i_p = layers.Input((seq_len, channels))
    patches = layers.Reshape((-1, patch_dim))(i_p)
    patches = layers.LayerNormalization()(patches)
    patches = layers.Dense(dim)(patches)
    patches = layers.LayerNormalization()(patches)
    pos_embedding = posemb_sincos_1d(patches)
    patches += pos_embedding
    dim = ops.shape(patches)[-1]
    for _ in range(depth):
        patches = FNetLayer(dim, dropout_rate)(patches)
    patches = layers.GlobalAveragePooling1D(name="avg_pool")(patches)
    o_p = layers.Dense(num_classes)(patches)
    return keras.Model(inputs=i_p, outputs=o_p)

k3im.gmlp_1d.gMLP1DModel

Instantiate a gMLP model for 1D data.

Parameters:

Name Type Description Default
seq_len

An integer representing the number of steps in the input sequence.

 required
patch_size

An integer representing the number of steps in a patch (default=4).

 required
num_classes

An integer representing the number of classes for classification.

 required
dim

An integer representing the projection dimension.

 required
depth

An integer representing the number of transformer layers.

 required
channels

An integer representing the number of channels in the input.

 3
dropout_rate

A float representing the dropout rate.

 0.0
Source code in k3im/gmlp_1d.py 
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
def gMLP1DModel(
    seq_len, patch_size, num_classes, dim, depth, channels=3, dropout_rate=0.0
):
    """Instantiate a gMLP model for 1D data.

    Args:
        seq_len: An integer representing the number of steps in the input sequence.
        patch_size: An integer representing the number of steps in a
            patch (default=4).
        num_classes: An integer representing the number of classes for classification.
        dim: An integer representing the projection dimension.
        depth: An integer representing the number of transformer layers.
        channels: An integer representing the number of channels in the input.
        dropout_rate: A float representing the dropout rate.

        """
    assert seq_len % patch_size == 0
    num_patches = seq_len // patch_size
    patch_dim = channels * patch_size
    i_p = layers.Input((seq_len, channels))
    patches = layers.Reshape((-1, patch_dim))(i_p)
    patches = layers.LayerNormalization()(patches)
    patches = layers.Dense(dim)(patches)
    patches = layers.LayerNormalization()(patches)
    pos_embedding = posemb_sincos_1d(patches)
    patches += pos_embedding
    dim = ops.shape(patches)[-1]
    for _ in range(depth):
        patches = gMLPLayer(num_patches, dim, dropout_rate)(patches)
    patches = layers.GlobalAveragePooling1D(name="avg_pool")(patches)
    o_p = layers.Dense(num_classes)(patches)
    return keras.Model(inputs=i_p, outputs=o_p)

k3im.mlp_mixer_1d.Mixer1DModel

Instantiate a Mixer model for 1D data.

Parameters:

Name Type Description Default
seq_len

An integer representing the number of steps in the input sequence.

 required
patch_size

An integer representing the number of steps in a patch (default=4).

 required
num_classes

An integer representing the number of classes for classification.

 required
dim

An integer representing the projection dimension.

 required
depth

An integer representing the number of transformer layers.

 required
channels

An integer representing the number of channels in the input.

 3
hidden_units

An integer representing the number of hidden units in the MLP.

 64
dropout_rate

A float representing the dropout rate.

 0.0
Source code in k3im/mlp_mixer_1d.py 
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
def Mixer1DModel(
    seq_len,
    patch_size,
    num_classes,
    dim,
    depth,
    channels=3,
    hidden_units=64,
    dropout_rate=0.0,
):
    """Instantiate a Mixer model for 1D data.

    Args:
        seq_len: An integer representing the number of steps in the input sequence.
        patch_size: An integer representing the number of steps in a
            patch (default=4).
        num_classes: An integer representing the number of classes for classification.
        dim: An integer representing the projection dimension.
        depth: An integer representing the number of transformer layers.
        channels: An integer representing the number of channels in the input.
        hidden_units: An integer representing the number of hidden units in the MLP.
        dropout_rate: A float representing the dropout rate.
    """
    assert seq_len % patch_size == 0
    num_patches = seq_len // patch_size
    patch_dim = channels * patch_size
    i_p = layers.Input((seq_len, channels))
    patches = layers.Reshape((-1, patch_dim))(i_p)
    patches = layers.LayerNormalization()(patches)
    patches = layers.Dense(dim)(patches)
    patches = layers.LayerNormalization()(patches)
    pos_embedding = posemb_sincos_1d(patches)
    patches += pos_embedding
    for _ in range(depth):
        patches = MLPMixerLayer(num_patches, hidden_units, dropout_rate)(patches)
    patches = layers.GlobalAveragePooling1D(name="avg_pool")(patches)
    o_p = layers.Dense(num_classes)(patches)
    return keras.Model(inputs=i_p, outputs=o_p)

k3im.simple_vit_1d.SimpleViT1DModel

Create a Simple Vision Transformer for 1D data.

Parameters:

Name Type Description Default
`seq_len`

number of steps

 required
`patch_size`

number steps in a patch

 required
`num_classes`

output classes for classification

 required
`dim`

projection dim for patches,

 required
`depth`

number of patch transformer units

 required
`heads`

number of attention heads

 required
`mlp_dim`

Projection Dim in transformer after each MultiHeadAttention layer

 required
`channels`

number of features/channels in the input default 3

 required
`dim_head`

size of each attention head

 required
Source code in k3im/simple_vit_1d.py 
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
def SimpleViT1DModel(
    seq_len,
    patch_size,
    num_classes,
    dim,
    depth,
    heads,
    mlp_dim,
    channels=3,
    dim_head=64,
):
    """ Create a Simple Vision Transformer for 1D data.

    Args:
        `seq_len`: number of steps
        `patch_size`: number steps in a patch
        `num_classes`: output classes for classification
        `dim`: projection dim for patches,
        `depth`: number of patch transformer units
        `heads`: number of attention heads
        `mlp_dim`: Projection Dim in transformer after each MultiHeadAttention layer
        `channels`: number of features/channels in the input default `3`
        `dim_head`: size of each attention head
    """
    assert seq_len % patch_size == 0
    patch_dim = channels * patch_size
    i_p = layers.Input((seq_len, channels))
    patches = layers.Reshape((-1, patch_dim))(i_p)
    patches = layers.LayerNormalization()(patches)
    patches = layers.Dense(dim)(patches)
    patches = layers.LayerNormalization()(patches)
    pos_embedding = posemb_sincos_1d(patches)
    patches += pos_embedding
    patches = Transformer(dim, depth, heads, dim_head, mlp_dim)(patches)
    patches = layers.GlobalAveragePooling1D(name="avg_pool")(patches)
    o_p = layers.Dense(num_classes)(patches)

    return keras.Model(inputs=i_p, outputs=o_p)

k3im.vit_1d.ViT1DModel

Create a Vision Transformer for 1D data.

Parameters:

Name Type Description Default
`seq_len`

number of steps

 required
`patch_size`

number steps in a patch

 required
`num_classes`

output classes for classification

 required
`dim`

projection dim for patches,

 required
`depth`

number of patch transformer units

 required
`heads`

number of attention heads

 required
`mlp_dim`

Projection Dim in transformer after each MultiHeadAttention layer

 required
`channels`

number of features/channels in the input default 3

 required
`dim_head`

size of each attention head

 required
Source code in k3im/vit_1d.py 
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
def ViT1DModel(
    seq_len,
    patch_size,
    num_classes,
    dim,
    depth,
    heads,
    mlp_dim,
    channels=3,
    dim_head=64,
):
    """ Create a Vision Transformer for 1D data.

    Args:
        `seq_len`: number of steps
        `patch_size`: number steps in a patch
        `num_classes`: output classes for classification
        `dim`: projection dim for patches,
        `depth`: number of patch transformer units
        `heads`: number of attention heads
        `mlp_dim`: Projection Dim in transformer after each MultiHeadAttention layer
        `channels`: number of features/channels in the input default `3`
        `dim_head`: size of each attention head
    """
    assert seq_len % patch_size == 0
    patch_dim = channels * patch_size
    i_p = layers.Input((seq_len, channels))
    patches = layers.Reshape((-1, patch_dim))(i_p)
    patches = layers.LayerNormalization()(patches)
    patches = layers.Dense(dim)(patches)
    patches = layers.LayerNormalization()(patches)
    num_patches = ops.shape(patches)[1]
    patches = ClassTokenPositionEmb(num_patches, dim)(patches)
    patches = Transformer(dim, depth, heads, dim_head, mlp_dim)(patches)
    cls_tokens = patches[:, -1]
    o_p = layers.Dense(num_classes)(cls_tokens)

    return keras.Model(inputs=i_p, outputs=o_p)