PS6

  1. 1

    Preprocess a new description.

    Take "The BEST homemade Pasta with Fresh Basil and Creamy Tomato Sauce! Absolutely delicious.". Apply preprocess(). How many words before vs after? Which words were removed? Were any important words lost?

  2. 2

    Find similar meals.

    Compute cosine similarity between listing 0's BoW vector and all other listings. Print the top 5 most similar meals with names, cuisines and similarity scores. Are they the same cuisine? Does the similarity make sense?

  3. 3

    Classify a new listing.

    Write a new meal description (1 to 2 sentences) for a cuisine of your choice, but not too obvious. Preprocess it, transform it with bow_vectorizer.transform, predict the cuisine with the trained classifier. Does it get it right? If not, why?

  4. 4

    Create your own image.

    Create a 24x24 grayscale image with a bright square (220) on a dark background (40), centred, roughly 10x10. Display it with plt.imshow() and print the pixel values of a row that crosses the square edge.

  5. 5

    Filter the noisy image.

    Apply the vertical edge filter to the noisy image (img_noisy). Apply the blur filter to the noisy image first, then the edge filter. Display all three: edges on noisy, blurred, edges on blurred. Which gives cleaner edges? Why does blurring first help?

Recitation walkthrough

Part 1: text

python
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics.pairwise import cosine_similarity
from sklearn.linear_model import LogisticRegression
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from nltk.stem import WordNetLemmatizer

stop_words = set(stopwords.words("english"))
lemmatizer = WordNetLemmatizer()

def preprocess(text):
    """Full preprocessing pipeline for one text string."""
    tokens = word_tokenize(text.lower())                                   # lowercase, then split into tokens
    tokens = [t for t in tokens if t.isalpha() and t not in stop_words]    # drop punctuation and stopwords
    return " ".join([lemmatizer.lemmatize(t) for t in tokens])             # reduce words to their base form

df["clean_text"] = df["description"].apply(preprocess)

The pipeline is lowercase → tokenise → remove stopwords and non-alphabetic tokens → lemmatise.

python
bow_vectorizer = CountVectorizer(max_features=200)
X_bow = bow_vectorizer.fit_transform(df["clean_text"])    # rows = listings, columns = vocabulary words

The result is a sparse document-term matrix: each cell is how often a word occurs in a listing. Then a plain classifier works on it:

python
X_train, X_test, y_train, y_test = train_test_split(X_bow, df["cuisine"], test_size=0.2,
                                                    random_state=42, stratify=df["cuisine"])
clf = LogisticRegression(max_iter=1000, random_state=42).fit(X_train, y_train)

The notebook expects Israeli and Middle Eastern to be confused most (shared words like hummus and pita) and inspects clf.coef_ to list each cuisine's most important words.

Part 2: images

python
from scipy.signal import convolve2d

img = np.full((24, 24), 30, dtype=float)            # dark table
for row in range(24):
    for col in range(24):
        if (row - 12)**2 + (col - 12)**2 <= 8**2:   # circle of radius 8 at the centre
            img[row, col] = 230                      # bright plate

A grayscale image is a 2D array (0 black, 255 white). The plate edge is the jump from 30 to 230.

python
edge_v = np.array([[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]])   # vertical edges (Sobel-like)
edge_h = np.array([[-1, -2, -1], [0, 0, 0], [1, 2, 1]])   # horizontal edges
blur    = np.ones((3, 3)) / 9                              # average of the 3x3 neighbourhood
sharpen = np.array([[0, -1, 0], [-1, 5, -1], [0, -1, 0]])  # amplify differences from neighbours

edges_v = convolve2d(img, edge_v, mode="same", boundary="fill")
edge_magnitude = np.sqrt(edges_v**2 + convolve2d(img, edge_h, mode="same", boundary="fill")**2)

Convolution means: place the filter on the image, multiply each filter value by the pixel beneath it, sum, slide, repeat. The output is a feature map. mode="same" keeps the output size; boundary="fill" pads edges with zeros.

Photo-quality idea from the notebook

Mean edge magnitude is a quality score. Blurry photos have too few edges (low magnitude); noisy photos have too many spurious edges (artificially high magnitude).

Related: Lecture 7 - NLP and Image Analysis | Lecture 6 - Recommendation Systems | Lecture 8 - Intro to Deep Learning | Data Science