Exam practice
Natural Language Processing — Practice
Past-exam questions, filterable by chapter, year and difficulty. Answer for instant grading, reveal the worked solution, and watch your accuracy climb.
Seen 0 / 142 · 0 correct · 0% accuracy
Chapter
Year
Difficulty
- agent-tool-countAccording to the LLM agent prompt below, how many different tools are available for the agent to use?
Here are the names and descriptions for each tool: exponentiate(base, power) - computes base raised to the power compound_interest(principal, rate, years) - applies interest to principal play_radio_tool() - starts the radio pause_radio_tool() - pauses the radio if it is playing increase_radio_volume_tool() - increases the volume decrease_radio_volume_tool() - decreases the volume search_tool5(query) - performs a web search translate_tool6(input, language) - translates input into the language - anthropomorphismThe act of attributing human emotions and intentions to a computer program (such as a chatbot) is referred to as:
- bert-reduce-burdenYou are building a document-search system and train a BERT model to determine whether a document is relevant to a query. How might you reduce the computational burden when deploying this system without greatly degrading its performance?
- bidirectional-transformer-modelWhich of the following machine-learning models makes use of a bidirectional Transformer architecture to extract a feature representation of text?
- bio-taggingBegin-Inside-Outside (BIO) tagging is often used for:
- bow-correctWhich of the following statements about the Bag-of-Words (BOW) representation of a document is correct?
- bpe-definitionA byte-pair encoding:
- chatbot-dialog-evalWhen evaluating dialogue produced by a chatbot, ideally we would rate performance based on:
- clip-definitionCLIP is a model primarily used for:
- confusion-matrix-precisionA text classifier produces the following confusion matrix on the test set: | | Pred + | Pred − | |----------|--------|--------| | Actual + | 95 | 25 | | Actual − | 5 | 75 | What is the precision of the classifier?
- embedding-memory-w2vYou learn Word2Vec embeddings of size 512 over a vocabulary of four hundred thousand tokens. Roughly how much memory (in GB) is needed to store all the vectors using double-precision (64-bit) floats? (1 byte = 8 bits.)
- few-shot-jokesImagine providing the prompt below to a large language model. What type of learning is the model doing in this case?
"I'm afraid for the calendar. Its days are numbered. => not funny I only know 25 letters of the alphabet. I don't know y. => not funny What do you call a fish wearing a bowtie? SoFISHticated. => funny What do you call a factory that makes okay products? A satisfactory. => funny I asked my dog what's two minus two. He said nothing. =>" - glove-embeddingsGloVe embeddings:
- gpt-not-trueWhich statement about GPT (GPT-2, GPT-3, etc.) models is NOT true?
- gpt3-4-autoregressiveWhile we don't know the details of the GPT-3.5 and GPT-4 models on which ChatGPT is built, it is highly likely — given their names (Generative Pre-trained Transformer) — that:
- jailbreakingPrompting a chatbot in a certain way so as to cause it to respond in an unintended manner and operate outside of its intended operating conditions — e.g. so that it tells the user how to do something dangerous — is usually called:
- langchain-purposeThe LangChain software library is primarily used to build what type of applications?
- lda-usageLatent Dirichlet Allocation (LDA) is usually used to:
- lora-purposeLow-Rank Adaptation (LoRA) is a technique in which we:
- lstm-not-trueWhich statement about a Long Short-Term Memory (LSTM) network is NOT true?
- mel-spectrogram-axesThe Mel spectrogram is just a spectrogram which has:
- mt-problem-typeMachine translation is an example of what type of problem?
- multilingual-bert-poorYou train a multilingual-BERT-based classifier and find that it performs particularly poorly, with accuracy far below that of a linear classifier. What might you do to try to improve the model?
- nb-prior-spamA spam filter is built from 2,000 spam emails and 8,000 non-spam emails. In this collection, what is the prior probability of seeing a spam email?
- ngram-improve-estimatesTo improve the probability estimates of an n-gram language model, we could:
- ngram-limitation-falseWhich statement about the limitations of n-gram language models is FALSE?
- not-a-ranking-function-ndcgWhich of the following is NOT a common ranking function used in term-based information retrieval?
- one-shot-learning-exampleWhich of the following prompts to a language model would be considered an example of one-shot learning?
- opensource-llm-not-trueWhich of the following statements about current open-source LLM technology (in 2024) is NOT true?
- parse-tree-not-correctIn NLP, which of the following statements regarding a parse tree is NOT correct?
- perplexity-we-really-like-nlpThe probability of the sequence "we really like NLP" is exactly 1/16. Assuming word-level tokenisation, what is the perplexity of the sentence?
- post-attention-layernormIn a Llama-style Transformer block, what is the "post_attention_layernorm" doing?
- posting-listIn a traditional (lexical/term-based) search engine, a posting list contains:
- prefer-perplexityDo we prefer language models with higher perplexity or lower perplexity?
- preprocessing-common-stepWhich of the following is a common text pre-processing step in an NLP application?
- pronoun-reference-taskThe task of determining who or what is referred to by a pronoun in a sentence is called:
- quantisation-mistral-vramIf we apply 4-bit quantisation when loading a Mistral-7B LLM on a GPU, approximately how much video RAM would be needed just to hold the weights?
- rag-best-promptWhich of the following system prompts would be the best if you are building a RAG (Retrieval-Augmented Generation) application?
- rank-learning-signalsIn a web search engine, rank-learning algorithms combine many signals to rank webpages for a query. Which of the following could provide useful features to the rank learner?
- regex-emoticonsWhich of the following regular expressions matches emoticons (e.g. :-), :3, <3)?
- rlhf-usageIn NLP, Reinforcement Learning from Human Feedback is usually used to do what?
- seq2seq-attention-not-trueWhich statement about sequence-to-sequence models with attention is NOT true?
- seq2seq-techniqueWhich of the following techniques is often used for learning sequence-to-sequence models in NLP?
- seq2seq-trueWhich statement about sequence-to-sequence models is true?
- spectrogram-technique-not-usedWhich, if any, of the following techniques is NOT used to produce a spectrogram for analysing audio signals to understand spoken language?
- speech-act-directiveIn the taxonomy of speech acts defined by Bach and Harnish (1979), when someone advises / asks / orders / requests somebody, they are performing which type of speech act?
- spoken-agents-harderBuilding spoken interface agents is much harder than building chatbots because:
- statistical-lm-computesA statistical language model computes:
- stemming-reasonThe main reason for performing stemming before building a text classifier is to:
- stemming-vs-lemmatizationWhat is the difference between stemming and lemmatization?
- subword-embeddingsWhich of the following statements about sub-word embeddings is NOT usually true?
- t5-not-true-distilledWhich statement about the T5 (Text-To-Text Transfer Transformer) model is NOT true?
- test-time-computeIn the context of LLMs, test-time compute scaling usually refers to the process of:
- tfidf-cosine-rankingGiven the normalised tf-idf document vectors (dimensions 1..14, all other components zero): ``` A = [0, 0, 0.1, 0.1, 0, 0, 0.2, 0, 0, 0, 0.2, 0, 0, 0 ] B = [0.1, 0, 0, 0, 0, 0, 0.2, 0, 0, 0, 0.4, 0, 0, 0.3] C = [0, 0, 0.1, 0, 0.3, 0, 0.1, 0, 0, 0, 0.3, 0, 0, 0 ] D = [0, 0.1, 0, 0.2, 0, 0, 0, 0, 0, 0.3, 0, 0, 0, 0.1] E = [0.4, 0, 0.1, 0, 0, 0, 0.2, 0, 0, 0, 0.2, 0, 0, 0 ] ``` and the query q = [0, 0.25, 0.25, 0.25, 0.25, 0, …], rank the documents from most to least similar to the query using cosine similarity.
- top-k-vocab-randomWhen generating text with top-k sampling, setting k to the size of the vocabulary is equivalent to performing:
- transformer-removed-recurrenceIn order to speed up model training, the Transformer model REMOVED what part of the sequence-to-sequence-with-attention model architecture?
- tts-normalisation-extendedText normalisation is needed for a text-to-speech system in order to:
- vector-db-nn-searchProviding fast nearest-neighbour search in a document-embedding space is the task of what technology?
- w2v-not-propertyWhich of the following is NOT a property of Word2Vec word embeddings?
- w2v-trueWhich statement about Word2Vec is true?
- word-embedding-uses-trueWhich of the following statements about the uses of word embeddings is generally true?
- zero-one-few-shot-definitionZero-, one- and few-shot learning involves:
- ascii-utf8What are ASCII and UTF-8, and what is the difference between them?
- attention-coreferenceIn the classic visualisation where the word "it" attends strongly to "animal" or to "street" in "The animal didn't cross the street because it was too tired / too wide", which NLP task is the self-attention mechanism implicitly performing?
- bert-not-trueWhich statement about the BERT model is NOT true?
- bert-pairwise-tokensWhat special tokens are often used to fine-tune a BERT model for pairwise text-classification tasks?
- bm25-propertyA particularly useful property of the BM25 ranking function is that:
- bow-not-correctWhich of the following statements about a Bag-of-Words (BOW) representation of a document is NOT correct?
- bpe-transformer-effectWhat effect does Byte-Pair Encoding (BPE) achieve in Transformer models?
- chain-of-thoughtChain-of-thought prompting of an LLM involves:
- clip-multimodalFor multi-modal retrieval, which type of model can be used to first align the embedding spaces of text and images?
- clustering-task-typeWhat type of task is text clustering?
- confusion-matrix-accuracyA text classifier produces the following confusion matrix on the test set: | | Pred + | Pred − | |----------|--------|--------| | Actual + | 95 | 15 | | Actual − | 5 | 85 | What is the accuracy of the classifier?
- coref-resolutionCo-reference resolution is the task of:
- entity-linkageEntity linkage is the task of:
- gpt-stands-forIn NLP, the acronym GPT usually stands for:
- gpt-without-finetuningHow might a GPT model best be used without fine-tuning?
- gpt2-translate-emergentHow was it possible for a GPT-2 model to learn to translate text from English to French if it was trained on a corpus of (predominantly) English documents?
- hac-complete-linkageIn hierarchical agglomerative clustering, if complete-linkage is used, what does that tell you about the clusters that will be found?
- heaps-lawHeaps' law states that:
- lexical-search-before-rerankingWhat is the primary advantage of using a lexical search engine before reranking with BERT?
- llm-7b-12gb-gpuIs it possible to generate text from a 7-billion-parameter LLM on a computer with a 12 GB GPU card?
- llm-content-falseWhich of the following statements about content generated by an LLM is obviously FALSE?
- llm-to-chatbotTurning an LLM into a chatbot usually involves:
- llm-training-tokensState-of-the-art LLMs (in 2024) have typically been trained on how much text?
- logistic-regression-trueWhich of the following statements about the Logistic Regression (LR) classifier is true?
- mt-true-2024Which statement about machine-translation systems is TRUE (as of 2024)?
- nb-cheap-conditionalA spam filter is built from 1,000 spam emails and 10,000 non-spam emails. The word "cheap" appears in 54 of the spam emails. Using a document-frequency estimate with no smoothing, what is P("cheap" given spam)?
- not-a-ranking-function-bceWhich of the following is NOT a common ranking function used in term-based information retrieval?
- parse-tree-correctIn NLP, which of the following statements regarding a parse tree is correct?
- preprocessing-not-a-stepWhich of the following is NOT a text pre-processing step in an NLP application?
- qa-self-consistencyThe robustness (and hence performance) of an LLM-based question-answering system would likely be improved by:
- rag-not-trueWhich statement about Retrieval-Augmented Generation (RAG) models is NOT TRUE?
- regex-emailWhich one of the following regular expressions would match a student email of the form "first.lastname@mail.polimi.it"?
- rnn-not-trueWhich of the following statements about Recurrent Neural Networks (RNNs) is NOT true?
- self-attention-updateHow does a self-attention mechanism update the embedding vector at a position?
- seq2seq-attention-translationWhen used to build translation systems, sequence-to-sequence models with attention:
- seq2seq-long-text-problemWhat is the main problem with a simple sequence-to-sequence architecture when translating long texts?
- stopwordsStopwords are:
- stt-eval-measureWhich of the following measures would NOT be appropriate for evaluating a speech-to-text system?
- symmetry-positional-embeddingsWhat technique is used to break the symmetry with respect to the input token positions within a Transformer model?
- tokenization-definitionIn NLP, what is tokenization?
- tts-homographIn text-to-speech systems, certain words like "bass" can be problematic. Why?
- tts-normalisationText normalisation is needed for a text-to-speech system in order to:
- variance-ratio-criterionThe Variance Ratio Criterion (a.k.a. the Calinski–Harabasz index) is an example of:
- vector-db-rag-applicationWhat is the primary application of vector databases in the context of RAG models?
- word-embedding-not-propertyWhich of the following is NOT usually a property of word embeddings?
- word-normalisationWord normalisation is the process of:
- activation-not-usedWhich of the following functions is NOT commonly used as an activation function in neural networks?
- clustering-news-aggregatorNews aggregators (sites that gather headlines from many sources) can use clustering to:
- confusion-matrix-countsAfter training a text classifier and running it on the test set, we get the confusion matrix below. What are the true-positive (TP), true-negative (TN), false-positive (FP) and false-negative (FN) counts, taking "+" as the positive class?
Rows = actual class, columns = predicted class Pred + Pred - Actual + 2401 103 Actual - 142 3071 - crf-taskTraditionally, Conditional Random Fields (CRFs) were used in NLP to solve which of the following tasks?
- dnn-not-trueWhich of the following statements about Deep Neural Networks (DNNs) is NOT generally true?
- eliza-rogerianWhat trick did ELIZA (the chatbot) use to create meaningful conversations in an open domain with little or no domain knowledge?
- embedding-memory-gloveYou learn GloVe embeddings of size 512 over a vocabulary of one million tokens. Roughly how much memory (in GB) is needed to store all the vectors using double-precision (64-bit) floats? (1 byte = 8 bits.)
- few-shot-learning-exampleImagine the following prompts to a language model that all try to redefine addition in an Orwellian dystopia. Which of them is an example of few-shot learning?
- hac-single-linkageIn hierarchical agglomerative clustering, if single-linkage (minimum distance) is used, what does that tell you about the clusters that could be found?
- idf-interpretationAssuming terms occur approximately independently within documents, when we apply IDF term weighting to rank documents for a query, we effectively score each document by:
- info-theory-informationIn information theory, the amount of information learnt from an event is equal to:
- kmedoids-disadvantageThe main disadvantage of the k-Medoids algorithm with respect to k-Means is:
- multitask-finetuning-benefitIn multi-task learning we fine-tune a language model to perform many different tasks (with different prompts) at the same time. The aim/benefit of doing this is to get the model to:
- nb-good-gradeA Naive Bayes model predicts whether a student gets a "good" or "bad" grade from the fact that she described the exam as "easy" but "long". Given P(easy|good)=2/3, P(long|good)=1/3, P(easy|bad)=1/3, P(long|bad)=2/3, P(good)=3/4, what is P(good | easy, long)? (Recall P(y|x) is proportional to P(y)·∏ P(xᵢ|y).)
- ner-paris-hilton"Paris Hilton was photographed leaving the Paris Hilton" is an example of a sentence containing:
- ngram-not-a-techniqueA number of techniques improve the next-word probability estimates of a Markov (n-gram) language model. Which of the following is NOT one of those techniques?
- perplexity-i-love-nlpIf the probability of the sequence "I love NLP" is exactly 1/64 (word-level tokenisation), what is the perplexity of the sentence?
- phoneme-speaking-rateSome people speak faster than others, so a phoneme detector tuned to fast speakers will observe many repeated phonemes for slower speakers. How might we BEST overcome this when building a speech-transcription system?
- prosody-definitionIn NLP, prosody refers to:
- regex-count-funnyHow many (non-overlapping) times would the regular expression "f\w*ny?" match the string "While I get how you feel, I don't find this exam either fun or funny."?
- regex-count-theHow many (non-overlapping) times would the regular expression ".*th?e" match the string "In the tree, there were three extraterrestrials with no teeth"?
- regex-license-plateWhich, if any, of the following strings would the regular expression "[A-Z]{6}\d{2}[A-Z]\d{2}[A-Z]\d{3}[A-Z]" match?
- regex-not-a-limitationRegular expressions provide a powerful language for writing rules to extract content from text documents, but have various limitations. Which of the following would NOT be considered a limitation of regular-expression-based text extraction?
- regex-phoneWhich one of the following regular expressions would match the telephone number "+69 403 992 010"?
- seq2seq-not-possible-taskWhich of the following tasks could NOT be modelled as a sequence-to-sequence problem?
- speech-act-backchannelIn terms of speech acts and discourse analysis, when someone repeats back to the speaker part of what they have just said, what is usually the purpose of doing this?
- t5-not-true-encoder-decoderWhich statement about the T5 (Text-To-Text Transfer Transformer) model is NOT true?
- top-k-1-greedyWhen generating text with top-k sampling, setting k to 1 is equivalent to performing:
- topic-discovery-ldaWhich of the following techniques might be used to discover the topics discussed in a collection of documents?
- topk-renormalize-examA bigram model gives these next-token probabilities after "amazing" (highest to lowest): grace 4%, food 3%, location 2%, experience 2%, people 2%, place 2%, minds 2%, experiences 1%, places 1%, exam 1%, … Using top-k sampling with k = 10, what is the chance the model produces "exam" as the next token?
- topk-renormalize-funA trigram model gives these next-token probabilities after "too much" (highest to lowest): work 3%, effort 2.5%, fun 2%, food 1.5%, time 1%, money 0.5%, … Using top-k sampling with k = 5, what is the chance the model produces "fun" as the next token?
- wav2vec-whisper-trueWhich of the following statements about Wav2vec (2020) and Whisper (2022) is true?
- word-embedding-uses-not-trueWhich statement about the uses of word embeddings is NOT generally true?
- word-level-label-taskWhich of the following tasks involves assigning a categorical label to every word in a piece of text?