Deep Learning — Practice

• Early stopping— choose —GeneralizationTraining speed/stability
• Weight decay— choose —GeneralizationTraining speed/stability
• Dropout— choose —GeneralizationTraining speed/stability
• Learning rate— choose —GeneralizationTraining speed/stability
• Shortcut / skip connection— choose —GeneralizationTraining speed/stability
• Xavier initialization— choose —GeneralizationTraining speed/stability
• ReLU activation functions— choose —GeneralizationTraining speed/stability
• Batch Normalization— choose —GeneralizationTraining speed/stability

• The reason people did not develop Multi-Layer Perceptrons (MLPs) further after 1969 was that they were going to be, in any case, linear classifiers, and because of this, Feed-Forward Neural Networks were introduced.TrueFalse
• In Hebbian learning, for each wrongly-predicted sample the weight $w_i$ is modified by adding a quantity proportional to the input (the perceptron/Hebbian update rule).TrueFalse
• Since the activation function of the perceptron model is a STEP or SIGN function, then the decision boundary is nonlinear.TrueFalse
• Hebbian learning is a supervised training procedure.TrueFalse
• The Perceptron model was presented originally in the book by Marvin Minsky and Seymour Papert, "Perceptrons: an introduction to computational geometry".TrueFalse

• Provided I use enough S-shaped (sigmoid or tanh) hidden neurons, a single hidden layer feed-forward neural network can approximate any continuous function on a compact set with arbitrary accuracy.TrueFalse
• There are tasks where applying single hidden layer neural networks straight to raw data shows no significant robustness or invariance, e.g., image or text classification.TrueFalse
• We can stack as many hidden layers of S-shaped (sigmoid or tanh) functions as we want and will always be able to learn the desired approximation for both regression and classification — the more the better.TrueFalse
• A single hidden layer feed-forward network can SOLVE both regression and classification with the desired accuracy, given enough S-shaped neurons in that layer.TrueFalse

• ☐it may lead to unstable training if set too high.
• ☐if set high, gradient descent moves faster toward the minimum of the loss.
• ☐if set low, training is stable and will converge.
• ☐if set low, it lets you fine-tune a pre-trained model while keeping its "knowledge".
• ☐it must be changed strictly monotonically during training, so the starting value cannot be reversed later.
• ☐it can reduce the effectiveness of ReLU-based networks if set too low.
• ☐modifying it during training by starting low and increasing leads to better minima.

• Maximum gradient of the sigmoid— choose —0.2510
• Maximum gradient of the hyperbolic tangent (tanh)— choose —0.2510
• Maximum gradient of the ReLU— choose —0.2510
• Minimum gradient of the sigmoid— choose —0.2510
• Minimum gradient of the hyperbolic tangent (tanh)— choose —0.2510
• Minimum gradient of the ReLU— choose —0.2510

• ☐Long Short-Term Memories have replaced Recurrent Neural Networks in sequential problems.
• ☐Cross-entropy has been introduced for classification tasks, while Least Squares is used in regression tasks.
• ☐Batch normalization is used to apply normalization at layer level.
• ☐Weight initialization is now based on the number of layer connections, such as Xavier or He initialization.
• ☐Convolution and pooling layers are used in image related tasks.
• ☐Rectified linear units and their evolutions (e.g. leaky ReLUs) have replaced sigmoids and tanhs in hidden layers.
• ☐Skip connections are used whenever possible or whenever it makes sense.

• word2vec embeddings represent discrete words and therefore cannot be added or subtracted.TrueFalse
• word2vec is unsupervised because it does not minimise a learning loss — it merely optimises vector distances in the embedding space.TrueFalse
• word2vec improved on one-hot encoding because it is sparse and therefore represents words with fewer parameters.TrueFalse
• word2vec is not trained in an unsupervised manner.TrueFalse
• word2vec is unsupervised because it implements a classical neural autoencoder architecture.TrueFalse

• ☐The weight initialization method
• ☐The patience in early stopping
• ☐The learning rate in batch gradient descent
• ☐The mask probability in a Dropout layer
• ☐The $\lambda$ value in weight decay
• ☐The kernel size in a convolutional neural network
• ☐The choice between sigmoid and hyperbolic tangent in binary classification tasks

• To build a multi-set encoder, one-hot items are sorted and fed one at a time to a zero-initialised LSTM whose final state encodes the multi-set; the same multi-set is therefore always encoded identically.TrueFalse
• A simple spellchecker can be built from a bidirectional LSTM to get a look-ahead effect and avoid the representation bottleneck.TrueFalse
• For sentence sentiment, a 2-layer stacked model made of two bidirectional LSTM layers followed by a sigmoid output is a legitimate design.TrueFalse
• To build a set encoder, one-hot items are fed one at a time to a zero-initialised LSTM; because the LSTM is permutation invariant, the same set is always encoded the same way.TrueFalse
• To reverse a sentence with a sequential model we cannot use an LSTM (generation must go forward autoregressively) and must instead use a bidirectional LSTM.TrueFalse

• ☐Multi-class classification
• ☐Regression (provided the output is normalized between 0 and 1)
• ☐Instance segmentation
• ☐Binary segmentation
• ☐Multi-label classification

• ☐The feature extraction layer could be implemented via a Conv2D layer followed by a MaxPooling2D layer.
• ☐Using a sliding window, you can use a feed-forward neural network with only dense layers and ReLUs.
• ☐The first hidden layer could be implemented with a Long Short-Term Memory layer.
• ☐The output could be a sigmoid trained in a supervised way with binary cross-entropy.
• ☐You can implement it in an unsupervised way with a time-series predictor trained only on sequences of the machine working properly.

• ☐Convolution and correlation in image processing are translation-invariant operations (ignoring boundaries and padding).
• ☐Correlation can be seen as a very rudimentary solution to template matching.
• ☐A trained linear classifier applied to an image computes correlation against one learned template per class; the template with the largest correlation defines the output class, up to a bias correction.
• ☐The convolutional backbone of a CNN can be seen as a learnable nonlinear filter scanning the whole image.
• ☐Training an ensemble of CNNs by changing the weight initialisation is a good strategy to reduce classification errors.
• ☐Batch normalization standardises each channel's activations, so it has no parameters because the operation is well defined per channel.
• ☐The deep-learning revolution is essentially that it became computationally possible to train networks with many layers.
• ☐Regression, classification and anomaly detection are all supervised learning tasks.
• ☐A Conv2D layer requires the number of input channels to equal the spatial size of the filter.
• ☐In binary classification you may have more than two classes, as long as the last layer has more than two output neurons.

• ☐A stack of Conv2D layers that all have linear activations is equivalent to a single Conv2D layer with filters of a suitable size and number.
• ☐Convolutional layers can be configured to reduce the spatial extent of the input volume.
• ☐A Conv2D without a nonlinear activation is a linear operation — the product of the vectorised input with a large sparse matrix whose nonzero entries lie along diagonals, with weights replicated every few rows and a mostly-constant bias.
• ☐Convolutional backbones that use skip connections (like ResNet-50) are poor candidates for fine-tuning because the skip connections prevent weight updates.
• ☐The Conv2D layers in LeNet use "same" padding to preserve the spatial extent of the activations.
• ☐In a trained CNN, each convolutional filter contributes to every neuron of the output volume.
• ☐Inference by LeNet corresponds to computing correlation against templates learned during training.
• ☐A good standard preprocessing is to subtract the average image computed over the whole dataset before performing cross-validation.

• Given a textual prompt describing a short video sequence, generate the corresponding video sequence.— choose —many to many (encoder–decoder, delayed)many to many (synchronous, per-step)one to manyone to onemany to one
• Given the video from a car's front camera (plus vehicle dynamics), suggest at each frame the current steering and speed command.— choose —many to many (encoder–decoder, delayed)many to many (synchronous, per-step)one to manyone to onemany to one
• During a conference call, translate simultaneously the audio spoken by the current speaker into the languages of the other attendants.— choose —many to many (encoder–decoder, delayed)many to many (synchronous, per-step)one to manyone to onemany to one
• Given a text in natural language describing what a C++ program should do, transform it into C++ code.— choose —many to many (encoder–decoder, delayed)many to many (synchronous, per-step)one to manyone to onemany to one
• On a noisy channel, get a stream of tokens and transform it continuously into a stream of corrected tokens (online error correction).— choose —many to many (encoder–decoder, delayed)many to many (synchronous, per-step)one to manyone to onemany to one
• Given an X-ray image, generate the report describing it and a possible diagnosis.— choose —many to many (encoder–decoder, delayed)many to many (synchronous, per-step)one to manyone to onemany to one
• Given a word as a one-hot encoding, generate a vector representing its embedding in a real-valued space.— choose —many to many (encoder–decoder, delayed)many to many (synchronous, per-step)one to manyone to onemany to one
• Given a C++ code, and just looking at the code, identify whether it can potentially generate a runtime error.— choose —many to many (encoder–decoder, delayed)many to many (synchronous, per-step)one to manyone to onemany to one

• ☐Class Activation Mappings are explainability tools for convolutional neural networks.
• ☐A positive aspect of CAMs is that you do not need to store any network activations, just the GAP outputs.
• ☐CAM are used to boost the performance of localization networks.
• ☐CAM suggests that the last layers of a CNN preserve localization properties.
• ☐Class Activation Mapping has a low computational overhead at inference time.
• ☐CAM are very flexible — they work in any CNN classifier of any architecture without retraining.
• ☐Class Activation Mappings are often used with AutoEncoders to spot anomalous regions in images.
• ☐CAM are difficult to implement in ResNet-like architectures because of the skip-connection blocks at the first layers.
• ☐To compute CAM the network needs a Global Average Pooling layer before a single dense output layer, and the number of channels entering the GAP must equal the number of classes.
• ☐CAM are typically at a lower resolution than the input image, so some upsampling (e.g. bilinear) is required.

• ☐R-CNN uses a non-trainable region-proposal algorithm to find candidate object locations, then crops, warps and classifies each region.
• ☐The RPN outputs objectness scores (probabilities) used to select the most likely locations of objects.
• ☐Anchors are used in the RPN to provide region proposals, which are then refined by the detection head.
• ☐A network doing both localisation and classification solves a multi-task problem, combining an MSE (box) loss and a cross-entropy (class) loss in a weighted sum.
• ☐YOLO is a single-shot network that directly predicts multiple bounding boxes and classes from the image, without a separate region-proposal stage.
• ☐Mask R-CNN performs instance segmentation by running an independent U-Net on each bounding box found by Faster R-CNN.
• ☐Fast R-CNN speeds up R-CNN by using separable convolutions, as in MobileNet.
• ☐When fine-tuning a detector you must change the RPN anchors to reflect the number of object classes.
• ☐Faster R-CNN is trained end-to-end in one standard optimisation, just like a localisation-and-classification network.

• ☐In a traditional n-gram language model, the joint probability of a sentence $P(w_1,\dots,w_T)$ is approximated as the product $\prod_t P(w_t \mid w_{t-n+1},\dots,w_{t-1})$.
• ☐In Continuous Bag-of-Words (CBOW), the model predicts a target word from the average (or sum) of the context-word embeddings.
• ☐In the Neural Network Language Model (Bengio et al., 2003), the conditional probability of the next word is computed with a softmax over the vocabulary.
• ☐In Skip-gram, the objective is to maximize the log-probability of the context words given the center word.
• ☐In the NNLM, the input to the hidden layer is obtained by concatenating the embedding vectors of the context words.
• ☐The computational complexity of NNLM training includes a term proportional to the vocabulary size $|V|$, due to the softmax over the full vocabulary.
• ☐In one-hot encoding, two semantically similar words have a large cosine similarity.

import torch
import torch.nn as nn

• ☐nn.ReLU()
• ☐nn.Conv2d(36, 36, kernel_size=3, padding=1)
• ☐nn.Upsample(scale_factor=2, mode='nearest')
• ☐nn.AdaptiveAvgPool2d((1, 1))
• ☐It is not possible to say, since we don't know what will happen in the next layers.
• ☐nn.Conv2d(64, 64, kernel_size=1, padding=1)
• ☐nn.MaxPool2d(kernel_size=3, stride=1, padding=1)

self.c0     = nn.Conv2d(3,   32, kernel_size=3, padding=1)
self.p1     = nn.MaxPool2d(2, 2)
self.c1     = nn.Conv2d(32,  64, kernel_size=3, padding=1)
self.p2     = nn.MaxPool2d(2, 2)
self.c2     = nn.Conv2d(64, 128, kernel_size=3, padding=1)
self.d      = nn.Dropout(0.2)
self.u1     = nn.Upsample(scale_factor=2)      # cat with r1 -> 192 in-ch
self.c3     = nn.Conv2d(192, 64, kernel_size=3, padding=1)
self.u2     = nn.Upsample(scale_factor=2)      # cat with r0 -> 96 in-ch
self.c4     = nn.Conv2d(96,  32, kernel_size=3, padding=1)
self.c5     = nn.Conv2d(32,  16, kernel_size=1, padding=0)
self.c6     = nn.Conv2d(16,   8, kernel_size=1, padding=0)
self.output = nn.Conv2d(8, num_classes, kernel_size=1)

• ☐It is not possible to perform transfer learning in convolutional backbones that use a GAP, as the locality of the extracted features is lost.
• ☐The ResNet architecture presents skip connections from the output of each convolutional layer to the latent representation.
• ☐The Inception Network presents multiple classification heads whose outputs are combined as an ensemble at inference to give a single prediction.
• ☐In the Inception Module, 1×1 convolutional layers are placed after the standard convolutional layer in each branch to adjust the number of channels.
• ☐In InceptionNet, multiple parallel branches are used to reduce the number of operations and the risk of overfitting.
• ☐The MLPconv layer in NiN performs nonlinear transformations of the input volume and replaces the convolutional layers; it can be implemented as an MLP sliding over the input volume.
• ☐AlexNet is an expanded LeNet using small filters to reduce parameters, and the vast majority of its learnable parameters are in the convolutional backbone.
• ☐To interpret a trained CNN classifier you can optimize both the input image and the conv2D weights to obtain an image that maximally activates an output neuron.
• ☐A plain CNN classifier with a GAP at the end of the convolutional backbone is a translation-invariant classifier.
• ☐A key intuition behind VGG is to replace conv2D layers having large filters with a cascade of conv2D layers having small filters.

• ☐Pooling layers can be placed everywhere inside a VGG-like CNN.
• ☐Pooling layers increase the receptive field.
• ☐Pooling layers cannot be used inside a residual block of plain convolutional layers.
• ☐Pooling layers always need to add a nonlinearity.
• ☐Pooling layers are the only way to reduce the spatial extent of activation maps.

• ☐Given an aerial image, predict at each pixel the yearly amount of rainfall, snow and UV radiation.
• ☐Suppress noise in natural (RGB) images.
• ☐Apply an artistic effect to portraits — the subject in colour, the background in tones of beige.
• ☐Given a city-map image, estimate at each pixel three pollution parameters (dioxide, monoxide, CO₂).
• ☐Given a photo of a chessboard mid-game, redraw the image highlighting which piece to move and where, by drawing circles.
• ☐Segment the input image into person, cat and background classes.
• ☐Given a picture of a building, estimate its height, width and depth.
• ☐Colourise a grayscale image of handwritten text, colouring correct words blue and typos red.
• ☐Output 1 if the image is AI-generated, 0 if genuine, and a value in between otherwise.
• ☐Predict at each pixel the probability of bone, tissue or background in an X-ray.

• ☐When moving a CNN classifier to a fully-convolutional CNN it is necessary to re-train the last convolutional layers, which correspond to the transformed dense layers of the classifier.
• ☐The semantic segmentation task can be framed as a pixel-wise classification over a fixed set of labels.
• ☐A classifier can be made fully-convolutional by turning its dense layers into convolutions; however, if it contains a flattening layer this is impossible, and the network must be modified and retrained (adding a GAP or downsampling to reach a 1×1 resolution).
• ☐To increase the spatial resolution of the activations you must use a non-trainable upsampling layer, and the only way to get trainable upsampling is to attach a convolutional layer to it.
• ☐A network performing coarse segmentation can be obtained by first training a CNN classifier and then transforming it into a fully-convolutional CNN.
• ☐A classifier containing a GAP can be moved to a fully-convolutional network that gives a coarse segmentation — the GAP maps to a convolutional layer whose filters are non-trainable, span the whole input volume, and are constant within each channel.
• ☐The U-Net can be trained using only image-wise labels, i.e. one class per image as in the ImageNet training set.
• ☐When training a semantic segmentation network it is important to adopt a weighted loss to compensate for class imbalance.
• ☐The last layer of a semantic segmentation network uses sigmoid activations to predict multiple output classes at each pixel.
• ☐A viable way to do instance segmentation is to segment the image with a U-Net and then take the isolated connected components as instances, assigning each a different label.

• ☐CAMs and Grad-CAMs provide saliency maps without having to re-train the network.
• ☐CAMs are leveraged in object-detection and instance-segmentation networks to give a coarse estimate of object locations, later refined by special layers.
• ☐Grad-CAM and CAM give the same saliency maps on a classifier with a single dense layer, and Grad-CAM additionally works on any CNN classifier.
• ☐Class Activation Maps were presented as the first form of explainability for CNN classifiers and required the CNN to have a single dense layer.
• ☐Multi-label predictions can be obtained with sigmoid output activations; the training labels are then not one-hot encoded, since the target may contain multiple ones.
• ☐Saliency maps are very useful for detecting biases in the data, as they visualise which region confuses the network the most, leading to a low confidence score.
• ☐CAMs interpret a classifier's output by highlighting which regions contribute the most to the loss-function term during training.
• ☐CAMs and Grad-CAMs are obtained as a weighted sum of the channels of the volume returned by the last convolutional layer.
• ☐CAMs are class-specific — you specify a class and the map highlights the distinctive patterns for it; querying another class needs no second inference, only re-aggregating the last volume's channels with different weights.
• ☐Grad-CAM is an evolution of CAM because it does not require the classifier to have a single dense layer at the output.

import torch
import torch.nn as nn
import torch.nn.functional as F
from torchsummary import summary

class Model(nn.Module):
    def __init__(self, input_shape):
        super().__init__()
        in_channels = input_shape[0]
        self.c1 = nn.Conv2d(in_channels, 64, kernel_size=4, stride=2, padding=1)
        self.mp1 = nn.MaxPool2d(kernel_size=2, stride=2)
        self.c2 = nn.Conv2d(64, 128, kernel_size=4, stride=2, padding=1)
        self.mp2 = nn.MaxPool2d(kernel_size=2, stride=2)
        self.c3 = nn.Conv2d(128, 256, kernel_size=1, padding=0)
        self.c4 = nn.Conv2d(256, 128, kernel_size=1, padding=0)
        self.do1 = nn.Dropout(0.2)
        self.c5 = nn.Conv2d(128, 256, kernel_size=1, padding=0)
        self.up2 = nn.Upsample(scale_factor=4, mode="nearest")
        self.c6 = nn.Conv2d(256, 128, kernel_size=4, padding=0)
        self.up3 = nn.Upsample(scale_factor=4, mode="nearest")
        self.c7 = nn.Conv2d(128, 64, kernel_size=4, padding=0)
        self.output_layer = nn.Conv2d(64, in_channels, kernel_size=1, padding=0)

    def forward(self, x):
        x = F.relu(self.c1(x)); x = self.mp1(x)
        x = F.relu(self.c2(x)); x = self.mp2(x)
        x = F.relu(self.c3(x)); x = F.relu(self.c4(x))
        x = self.do1(x)
        x = F.relu(self.c5(x))
        x = self.up2(x); x = F.pad(x, (1, 2, 1, 2))
        x = F.relu(self.c6(x))
        x = self.up3(x); x = F.pad(x, (1, 2, 1, 2))
        x = F.relu(self.c7(x))
        x = torch.sigmoid(self.output_layer(x))
        return x

INPUT_SIZE = (3, 32, 64)
model = Model(INPUT_SIZE)
summary(model, input_size=INPUT_SIZE)

• ☐Given a human face as input, estimate its age, weight and length of eyebrows.
• ☐Suppress noise in an image.
• ☐Classify the digits from 0 to 2 in a subset of the MNIST dataset.
• ☐Suppress blur in a grayscale image.
• ☐Determine which pixels of an image are covered by pedestrians, cars or background.
• ☐Given a weather-forecast map, estimate at each pixel the probability of rain, snow and strong winds.
• ☐Given a picture of a person, estimate the coordinates of the tip of the nose and the centers of the left and right hands.

class Model(nn.Module):
    def __init__(self, in_channels=3, num_classes=3, dropout_rate=0.5):
        super().__init__()
        self.conv0 = nn.Conv2d(in_channels, 16, kernel_size=3, padding='same')
        self.relu0 = nn.ReLU()
        self.mp0   = nn.MaxPool2d(kernel_size=2, stride=2)
        self.conv1 = nn.Conv2d(16, 32, kernel_size=3, padding='same')
        self.relu1 = nn.ReLU()
        self.mp1   = nn.MaxPool2d(kernel_size=2, stride=2)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=3, padding='same')
        self.relu2 = nn.ReLU()
        self.mp2   = nn.MaxPool2d(kernel_size=2, stride=2)
        self.conv3 = nn.Conv2d(64, 128, kernel_size=3, padding='same')
        self.relu3 = nn.ReLU()
        self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
        self.seq1 = nn.Sequential(nn.Dropout(dropout_rate), nn.Linear(128, num_classes))
        self.seq2 = nn.Sequential(nn.Dropout(dropout_rate), nn.Linear(128, 4))
    def forward(self, x):
        x = self.mp0(self.relu0(self.conv0(x)))
        x = self.mp1(self.relu1(self.conv1(x)))
        x = self.mp2(self.relu2(self.conv2(x)))
        x = self.relu3(self.conv3(x))
        x = self.avgpool(x)
        x = torch.flatten(x, 1)
        out_1 = self.seq1(x)
        out_2 = self.seq2(x)
        return out_1, out_2

• ☐Analyse an image of a car, tell whether it is a Ferrari, a McLaren or a Lamborghini, and predict the RGB colour of the car and its maximum speed.
• ☐Given a picture of a person, locate the centre of each eye and return whether the person is a kid, an adult, or an older person.
• ☐Classify an aerial image as containing a city, a forest, a body of water, or a mixture of the three, and count the number of cars, trees, whales and houses.
• ☐Tell whether an image contains a cat, a dog or a duck, and provide the bounding box around that animal.
• ☐Count the number of cars, pedestrians and cyclists in an image, and return the bounding box enclosing the largest of these.
• ☐Given a person in academia, determine whether they are teaching staff, admin staff, or a student, and return a circle enclosing the face plus the distance between the eyes in centimetres.
• ☐An object-detection network for images with exactly 4 persons that returns a bounding box around each person and whether they are sitting, standing or jumping.
• ☐A segmentation network that, given an RGB histological image, labels each pixel as cell, vessel or other tissue and draws a bounding box around the whole sample.
• ☐Predict the 3D dimensions (height, width, depth) of an appliance and classify it as a fridge, washing machine, oven, or coffee machine.
• ☐Given an RGB histological image, output whether the tissue is healthy, cancerous, or has a different disease and, for each suspicious region, estimate the area, centre and perimeter.

• Batch normalization— choose —Training performanceGeneralization
• Data augmentation— choose —Training performanceGeneralization
• Early stopping— choose —Training performanceGeneralization
• Weight decay— choose —Training performanceGeneralization
• Gradient descent with momentum— choose —Training performanceGeneralization
• Xavier weight initialization— choose —Training performanceGeneralization

• ☐A feed-forward classifier whose input is the vibration intensity of each sensor at a single time instant, one class per action; it can discriminate WHERE the touch/knock is, but cannot count knocks.
• ☐A 1-D convolutional network whose input is the multi-channel sensor signal (one channel per sensor) over a window of a few seconds, trained with categorical cross-entropy; it can both count knocks and discriminate their position.
• ☐A recurrent network taking the sequence of sensor recordings with a softmax output; it can count knocks but cannot discriminate the position of the touch/knock.

• ☐I regularise the model by adding the sum of the absolute values of the weights (L1 regularisation) to the loss.
• ☐If I normalise the target to the [-1, +1] range I can use a hyperbolic-tangent activation on the output.
• ☐I use the mean-absolute-error loss.
• ☐In a multi-valued regression I can use as many outputs as values to regress and a ReLU activation to force non-negative outputs.
• ☐I use the categorical cross-entropy loss.
• ☐I use the mean-absolute-error loss in order to force the outputs to be positive.
• ☐In a multi-valued regression I use as many outputs as values to regress with a softmax activation.

• Word embedding turns a one-hot word representation into a dense vector representation.TrueFalse
• Training a word embedding can be considered an unsupervised (self-supervised) learning task.TrueFalse
• In word2vec the embedding-space regularity (a constant difference vector for analogous concepts) is imposed explicitly by a dedicated loss at training time.TrueFalse
• Skip-gram and Continuous Bag-of-Words are two possible implementations of the word2vec model.TrueFalse

• ☐During training the network learns a set of templates the same size as the input image, and the output is defined by the correlation between these templates and the input.
• ☐If it is trained with a cross-entropy loss, the output neurons need a softmax (or sigmoid) activation.
• ☐This is a classifier over hand-crafted features.
• ☐Once trained, an input image is assigned to the class whose template is closest in Euclidean distance.
• ☐Because the image is unfolded into a vector, the same network can classify images with a different number of channels than those used in training.
• ☐It is equivalent to a 1-D CNN with a single convolutional layer that classifies images as time series.
• ☐This network is equivalent to two linear classifiers followed by nonlinearities, in cascade.

x = tfkl.Conv2D(16, kernel_size=3, padding='valid', activation='relu')(input_layer)
x = tfkl.MaxPooling2D(4)(x)
x = tfkl.Conv2D(32, kernel_size=5, padding='valid', activation='relu')(x)
x = tfkl.MaxPooling2D(3)(x)
x = tfkl.Conv2D(64, kernel_size=7, padding='valid', activation='relu')(x)
x = tfkl.MaxPooling2D(2)(x)
x = tfkl.Conv2D(128, kernel_size=9, padding='valid', activation='relu')(x)
x = tfkl.MaxPooling2D(1)(x)

• ☐In a U-Net the number of filters in the last layer must equal the number of classes.
• ☐A semantic-segmentation network can be trained with class-specific weights (assigned per pixel).
• ☐When training a segmentation network, random masking can be used to reproduce the stochastic effect of mini-batch selection.
• ☐Several semantic-segmentation networks have a contractive path and an expanding path.
• ☐A U-Net can be tested on images larger than those it was trained on.
• ☐The last layer of a segmentation network must be a Conv2D with 1×1 filters.
• ☐The U-Net is popular but can only be used on medical images.
• ☐Photometric augmentation cannot be used when training a U-Net, or the label would no longer match the image.
• ☐The U-Net steadily increases the number of filters at every layer to increase the spatial resolution of the activations.
• ☐Semantic-segmentation networks can be used to count the persons in an image.
• ☐It is not possible to perform transfer learning in a semantic-segmentation network.

input_layer = tfkl.Input(shape=input_shape)
c1 = tfkl.Conv2D(32, 5, padding='valid', activation='relu')(input_layer)
c1 = tfkl.BatchNormalization()(c1)
c1 = tfkl.AveragePooling2D(pool_size=(4, 4))(c1)
c1 = tfkl.Dropout(0.25)(c1)
s1 = tfkl.Conv2D(64, 1, padding='same', activation='relu')(c1)
s1 = tfkl.Conv2D(32, 5, padding='same', activation='relu')(s1)
s1 = tfkl.Add()([c1, s1])
c2 = tfkl.Conv2D(64, 5, padding='same', activation='relu')(s1)
c2 = tfkl.BatchNormalization()(c2)
c2 = tfkl.AveragePooling2D(pool_size=(4, 4))(c2)
c2 = tfkl.Dropout(0.25)(c2)
s2 = tfkl.Conv2D(128, 1, padding='same', activation='relu')(c2)
s2 = tfkl.Conv2D(64, 5, padding='same', activation='relu')(s2)
s2 = tfkl.Add()([c2, s2])
gmp = tfkl.GlobalMaxPooling2D()(s2)
fc1 = tfkl.Dense(128, activation='relu')(gmp)
fc1 = tfkl.BatchNormalization()(fc1)
fc2 = tfkl.Dense(64, activation='relu')(fc1)
output_layer = tfkl.Dense(output_number, activation='linear')(fc2)

• ☐Given a picture of a cat, locate the centre of the pupils and the centre of the tail.
• ☐Given a picture of a professor, estimate the age, the number of MSc and BSc courses in the first and second semester, and the salary.
• ☐Given a picture of a vehicle, return its three spatial dimensions (height, width, length) and the barycentre in 3-D space.
• ☐Given a histological image of many nuclei, perform instance segmentation.
• ☐Given an image of a newspaper page, count the characters, the words and the articles on the page.
• ☐Given a photo of a car registered in Italy, read its licence plate.
• ☐This is an object-detection network that localises objects of a single class.
• ☐Estimate the average colour of the input image.

• A network for autonomous driving that recognises where pedestrians and cars are located in the image.— choose —SupervisedUnsupervised
• An image restoration / inpainting model that, given an image with obscured pieces, restores the missing pixels.— choose —SupervisedUnsupervised
• A model that, from an image of a cow, tells whether the cow is of the Frisona breed or not.— choose —SupervisedUnsupervised
• A network that predicts the next "acqua alta" (high-water) event in Venice from a 20-year time series of past events.— choose —SupervisedUnsupervised

• We can have very deep Transformer architectures and still train them, because the residual connections in every Transformer block limit the vanishing-gradient issue.TrueFalse
• The positional-encoding mechanism based on sinusoidal and cosinusoidal functions of token positions is used by all Transformers whenever they encode position.TrueFalse
• Transformers require both an encoder and a decoder stage — "encoder-only" or "decoder-only" Transformers cannot be trained.TrueFalse
• Given the same input sequence twice in two separate executions, a Transformer produces the same output sequence both times.TrueFalse
• In Multi-Head Attention it is possible to "cross" queries/keys/values coming from the encoder and the decoder.TrueFalse
• Because self-attention is "permutation invariant", feeding a reversed sentence to a self-attention layer yields the same result as the forward sentence.TrueFalse

• ☐It could be implemented with a CNN that observes the trainee's face while asking the question and directly regresses the number of taps to produce.
• ☐It is a classical sequence-to-sequence model trained on pairs (question, tap sequence) with a cross-entropy loss.
• ☐It could be a recurrent network taking images as input (possibly pre-processed by convolutional layers) with a sigmoidal output over two symbols {tap, <eos>}.
• ☐It can be trained to decide the current output with binary cross-entropy, since the output is essentially "tap" or "don't tap".
• ☐It needs a sequence of images as input, so it is likely to include some convolutional layers.

• ☐Mixup augmentation feeds the network a convex combination of a pair of images, and the target is the label of the image that contributes most to the combination.
• ☐Augmentation layers are useful during training but must be removed at inference, because a test image cannot be augmented.
• ☐For VGG pre-trained on ImageNet, transfer learning here means running a few epochs on the new set while keeping trainable only VGG's own ImageNet MLP head.
• ☐It is good to perform transfer learning before fine-tuning, as long as there is enough training data.
• ☐Any photometric or geometric transformation can be plugged in during training and will be beneficial regardless.
• ☐To backpropagate through max-pooling you must keep track of the coordinates of the locations that propagated the maximum.
• ☐It is good to perform fine-tuning before transfer learning.

• ☐A CNN with a flattening layer can be made fully convolutional only if the flattening is directly connected to the output layer.
• ☐To make a CNN classifier with an N-neuron dense layer fully convolutional, you must add a convolutional layer with L/N filters, where N is the product of the filter's spatial dimensions.
• ☐When designing semantic-segmentation networks it is mandatory to include skip connections.
• ☐It is not possible to convert a trained CNN classifier that contains a GAP into a fully convolutional network.
• ☐A U-Net can be trained via the "fully-convolutionalisation" process, using only the image-level class label.
• ☐A trained CNN classifier that contains a flattening layer can be transformed into a fully convolutional network.
• ☐Segmentation networks are not meant to locate/separate individual object instances in the scene.
• ☐A trained CNN classifier made of conv, maxpool and dense layers can always be modified to return low-resolution heatmaps of class probabilities.

input_layer = tfkl.Input(shape=input_shape)
# Block 1
c1 = tfkl.Conv2D(64, 7, padding='same', activation='relu')(input_layer)
c1 = tfkl.MaxPooling2D()(c1)
c1 = tfkl.Dropout(0.2)(c1)
s1 = tfkl.Conv2D(128, 1, padding='same', activation='relu')(c1)
s1 = tfkl.Conv2D(64, 7, padding='same', activation='relu')(s1)
s1 = tfkl.Add()([c1, s1])
# Block 2
c2 = tfkl.Conv2D(128, 7, padding='same', activation='relu')(s1)
c2 = tfkl.MaxPooling2D()(c2)
c2 = tfkl.Dropout(0.2)(c2)
s2 = tfkl.Conv2D(256, 1, padding='same', activation='relu')(c2)
s2 = tfkl.Conv2D(128, 7, padding='same', activation='relu')(s2)
s2 = tfkl.Add()([c2, s2])
# Head
s2 = tfkl.GlobalAveragePooling2D()(s2)
s2 = tfkl.Dense(64)(s2)
s2 = tfkl.BatchNormalization()(s2)
s2 = tfkl.Dense(64)(s2)
output_layer = tfkl.Dense(output_number, activation='linear')(s2)

• ☐Counting the number of cats, dogs, traffic lights, bicycles, cars and persons in a Street-View picture.
• ☐Given a picture of a car, estimate its weight and its speed and localise it with a bounding box.
• ☐Given a picture of a human face, estimate the positions of the tips of the two ears and the tip of the nose.
• ☐Given a picture of a six-person family, estimate the age of each of the six members.
• ☐Given a picture taken from a driving car, classify every pixel as car, person, bicycle, truck, traffic light or background.
• ☐Detect persons and cats and return a bounding box for each one found.
• ☐Weather forecasting over the Lombardia region squeezed into a 224×112 image.
• ☐Instance segmentation of the cells in a biopsy image (counting up to six cells).