TF-IDF turns documents into vectors where common words count for little and distinctive words count for a lot. It's still a strong baseline for search and text classification.
Implement a class TfidfVectorizer with:
fit(docs): tokenise by lowercasing and splitting on whitespace. Build vocabulary_, a dict from word to column index with words sorted alphabetically, and idf_ with the smoothed IDF: . Return self.transform(docs): a NumPy array of shape (len(docs), vocab_size). Each entry is the raw count of the term in the document times its IDF, and each row is then L2-normalised. Rows with no known words stay all zeros. Unknown words are ignored.fit_transform(docs).(These are scikit-learn's default settings.)