Pandas: How to do analysis on array-like field?
<p>I'm doing analysis on movies, and each movie have a <code>genre</code> attribute, it might be several specific genre, like <code>drama</code>, <code>comedy</code>, the data looks like this:</p>

<pre><code>movie_list = [
    {'name': 'Movie 1',
    'genre' :'Action, Fantasy, Horror'},
    {'name': 'Movie 2',
    'genre' :'Action, Comedy, Family'},
    {'name': 'Movie 3',
    'genre' :'Biography, Drama'},
    {'name': 'Movie 4',
    'genre' :'Biography, Drama, Romance'},
    {'name': 'Movie 5',
    'genre' :'Drama'},
    {'name': 'Movie 6',
    'genre' :'Documentary'},
]
</code></pre>

<p>The problem is that, how do I do analysis on this? For example, how do I know how many action moviews are here, and how do I query for the category action? Specifically:</p>

<ol>
<li><p>How do I get all the categories in this list? So I know each contains how many moviews</p></li>
<li><p>How do I query for a certain kind of movies, like action?</p></li>
<li><p>Do I need to turn the <code>genre</code> into <code>array</code>?</p></li>
</ol>

<p>Currently I can get away the 2nd question with <code>df[df['genre'].str.contains("Action")].describe()</code>, but is there better syntax?</p>

<p>If your data isn't too huge, I would do some pre-processing and get 1 record per genre. That is, I would structure your data frame like this:</p>

<pre><code> Name    Genre
 Movie 1 Action
 Movie 1 Fantasy
 Movie 1 Horor
 ...
</code></pre>

<p>Note the names should be repeated. While this may make your data set much bigger, if your system can handle it it can make data analysis very easy.
Use the following code to do the transformation:</p>

<pre><code>import pandas as pd

def reformat_movie_list(movies):
    name = []
    genre = []
    result = pd.DataFrame()
    for movie in movies:
        movie_name = movie["name"]
        movie_genres = movie["genre"].split(",")
        for movie_genre in movie_genres:
             name.append(movie_name.strip())
             genre.append(movie_genre.strip())
    result["name"] = name
    result["genre"] = genre
    return result
</code></pre>

<p>In this format, your 3 questions become</p>

<ol>
<li><p>How do I get all the categories in this list? So I know each contains how many movies?</p>

<p>movie_df.groupby("genre").agg("count")</p></li>
</ol>

<p>see <a href="http://stackoverflow.com/questions/19384532/how-to-count-number-of-rows-in-a-group-in-pandas-group-by-object">How to count number of rows in a group in pandas group by object?</a></p>

<ol start="2">
<li><p>How do I query for a certain kind of movies, like action?</p>

<p>horror_movies = movie_df[movie_df["genre"] == "horror"]</p></li>
</ol>

<p>see <a href="http://stackoverflow.com/questions/11869910/pandas-filter-rows-of-dataframe-with-operator-chaining">pandas: filter rows of DataFrame with operator chaining</a></p>

<ol start="3">
<li>Do I need to turn the genre into array?</li>
</ol>

<p>Your de-normalization of the data should take care of it.</p>

python pandas statistics