Can I control the way the CountVectorizer vectorizes the corpus in scikit learn?

2024/11/15 7:25:01

I am working with a CountVectorizer from scikit learn, and I'm possibly attempting to do some things that the object was not made for...but I'm not sure.

In terms of getting counts for occurrence:

vocabulary = ['hi', 'bye', 'run away!']
corpus = ['run away!']
cv = CountVectorizer(vocabulary=vocabulary)
X = cv.fit_transform(corpus)
print X.toarray()

gives:

[[0 0 0 0]]

What I'm realizing is that the CountVectorizer will break the corpus into what I believe is unigrams:

vocabulary = ['hi', 'bye', 'run']
corpus = ['run away!']
cv = CountVectorizer(vocabulary=vocabulary)
X = cv.fit_transform(corpus)
print X.toarray()

which gives:

[[0 0 1]]

Is there any way to tell the CountVectorizer exactly how you'd like to vectorize the corpus? Ideally I would like an outcome along the lines of the first example.

In all honestly, however, I'm wondering if it is at all possible to get an outcome along these lines:

vocabulary = ['hi', 'bye', 'run away!']
corpus = ['I want to run away!']
cv = CountVectorizer(vocabulary=vocabulary)
X = cv.fit_transform(corpus)
print X.toarray()[[0 0 1]]

I don't see much information in the documentation for the fit_transform method, which only takes one argument as it is. If anyone has any ideas I would be grateful. Thanks!

Answer

The parameter you want is called ngram_range. You pass in a tuple (1,2) to the constructor to get unigrams and bigrams. However, the vocabulary you pass in needs to be a dict with ngrams as keys and integers as values.

In [20]: print CountVectorizer(vocabulary={'hi': 0, u'bye': 1, u'run away': 2}, ngram_range=(1,2)).fit_transform(['I want to run away!']).A
[[0 0 1]]

Note the default tokeniser removes the exclamation mark at the end, so the last token is away. If you want more control over how the string is broken up into tokens, follow @BrenBarn's comment.

https://en.xdnf.cn/q/71971.html

Related Q&A

mod_wsgi process getting killed and django stops working

I have mod_wsgi running in daemon mode on a custom Linux build. I havent included any number for processes or threads in the apache config. Here is my config:WSGIDaemonProcess django user=admin WSGIPro…

Reindex 2nd level in incomplete multi-level dataframe to be complete, inserting NANs on missing rows

I need to reindex the 2nd level of a pandas dataframe, so that the 2nd level becomes a (complete) list 0,...,(N-1) for each 1st level index.I tried using Allan/Haydens approach, but unfortunately it on…

ImportError: cannot import name _gdal_array from osgeo

I create a fresh environment, install numpy, then install GDAL. GDAL imports successfully and I can open images using gdal.Open(, but I get the ImportError: cannot import name _gdal_array from osgeo er…

How do I insert a map into DynamoDB table?

I have the following line of code :table.put_item( Item={filename : key, status : {M : iocheckdict }})The iocheckdict looks like this:{A: One, C: Three, D: Four, B: Two, E: Five}So, when I am running t…

How to redirect django.contrib.auth.views.login after login?

I added django.contrib.auth.views.login everywhere in my webpage, for that I had to load a templatetag (that returns the AuthenticationForm) in my base.html. This templatetags includes the registration…

How to do windows API calls in Python 3.1?

Has anyone found a version of pywin32 for python 3.x? The latest available appears to be for 2.6.Alternatively, how would I "roll my own" windows API calls in Python 3.1?

Returning the outputs from a CloudFormation template with Boto?

Im trying to retrieve the list of outputs from a CloudFormation template using Boto. I see in the docs theres an object named boto.cloudformation.stack.Output. But I think this is unimplemented functi…

numpy.array of an I;16 Image file

I want to use TIFF images to effectively save large arrays of measurement data. With setting them to mode="I;16" (corresponding to my 16 bit data range), they yield 2MB files (~1000x1000 &quo…

Namespace packages and pip install -e

I have a ns.pkg2 package that depends on ns.pkg1 package. I make a fork of it, publish it to git and want to install my version into my virtualenv. I use pip install -e mygit and end up with ns.pkg in …

Python sys.argv out of range, dont understand why

I have a script that Ive been using for a some time to easily upload files to my server. It has been working great for a long time, but I cant get it to work on my new desktop computer. The code is sim…