PS6
- #data-science
- #intro-data-science
- #recitation
- #nlp
- #bag-of-words
- #cosine-similarity
- #image-analysis
- #convolution
Toolkit
- 1
Preprocess a new description.
Take
"The BEST homemade Pasta with Fresh Basil and Creamy Tomato Sauce! Absolutely delicious.". Applypreprocess(). How many words before vs after? Which words were removed? Were any important words lost? - 2
Find similar meals.
Compute cosine similarity between listing 0's BoW vector and all other listings. Print the top 5 most similar meals with names, cuisines and similarity scores. Are they the same cuisine? Does the similarity make sense?
- 3
Classify a new listing.
Write a new meal description (1 to 2 sentences) for a cuisine of your choice, but not too obvious. Preprocess it, transform it with
bow_vectorizer.transform, predict the cuisine with the trained classifier. Does it get it right? If not, why? - 4
Create your own image.
Create a 24x24 grayscale image with a bright square (220) on a dark background (40), centred, roughly 10x10. Display it with
plt.imshow()and print the pixel values of a row that crosses the square edge. - 5
Filter the noisy image.
Apply the vertical edge filter to the noisy image (
img_noisy). Apply the blur filter to the noisy image first, then the edge filter. Display all three: edges on noisy, blurred, edges on blurred. Which gives cleaner edges? Why does blurring first help?
Recitation walkthrough
Part 1: text
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics.pairwise import cosine_similarity
from sklearn.linear_model import LogisticRegression
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from nltk.stem import WordNetLemmatizer
stop_words = set(stopwords.words("english"))
lemmatizer = WordNetLemmatizer()
def preprocess(text):
"""Full preprocessing pipeline for one text string."""
tokens = word_tokenize(text.lower()) # lowercase, then split into tokens
tokens = [t for t in tokens if t.isalpha() and t not in stop_words] # drop punctuation and stopwords
return " ".join([lemmatizer.lemmatize(t) for t in tokens]) # reduce words to their base form
df["clean_text"] = df["description"].apply(preprocess)
The pipeline is lowercase → tokenise → remove stopwords and non-alphabetic tokens → lemmatise.
bow_vectorizer = CountVectorizer(max_features=200)
X_bow = bow_vectorizer.fit_transform(df["clean_text"]) # rows = listings, columns = vocabulary words
The result is a sparse document-term matrix: each cell is how often a word occurs in a listing. Then a plain classifier works on it:
X_train, X_test, y_train, y_test = train_test_split(X_bow, df["cuisine"], test_size=0.2,
random_state=42, stratify=df["cuisine"])
clf = LogisticRegression(max_iter=1000, random_state=42).fit(X_train, y_train)
The notebook expects Israeli and Middle Eastern to be confused most (shared words like hummus and pita) and inspects clf.coef_ to list each cuisine's most important words.
Part 2: images
from scipy.signal import convolve2d
img = np.full((24, 24), 30, dtype=float) # dark table
for row in range(24):
for col in range(24):
if (row - 12)**2 + (col - 12)**2 <= 8**2: # circle of radius 8 at the centre
img[row, col] = 230 # bright plate
A grayscale image is a 2D array (0 black, 255 white). The plate edge is the jump from 30 to 230.
edge_v = np.array([[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]]) # vertical edges (Sobel-like)
edge_h = np.array([[-1, -2, -1], [0, 0, 0], [1, 2, 1]]) # horizontal edges
blur = np.ones((3, 3)) / 9 # average of the 3x3 neighbourhood
sharpen = np.array([[0, -1, 0], [-1, 5, -1], [0, -1, 0]]) # amplify differences from neighbours
edges_v = convolve2d(img, edge_v, mode="same", boundary="fill")
edge_magnitude = np.sqrt(edges_v**2 + convolve2d(img, edge_h, mode="same", boundary="fill")**2)
Convolution means: place the filter on the image, multiply each filter value by the pixel beneath it, sum, slide, repeat. The output is a feature map. mode="same" keeps the output size; boundary="fill" pads edges with zeros.
Photo-quality idea from the notebookMean edge magnitude is a quality score. Blurry photos have too few edges (low magnitude); noisy photos have too many spurious edges (artificially high magnitude).
Related: Lecture 7 - NLP and Image Analysis | Lecture 6 - Recommendation Systems | Lecture 8 - Intro to Deep Learning | Data Science