Pandas python: Create function to merge two dataframes based on defined list of columns
<p>In the script that I am writing, I want to frequently repeat the same piece of code, where I create a "numerator" dataframe with one group by, and then a "denominator" dataframe with a different group by. I then merge the two together so that I have the numerator and denominator in one place. I am trying to create a function where all I have to pass to it is the list of fields I want included in the numerator and denominator.</p>

<p>Here is the function:</p>

<pre><code>  def calcfractions(self, df, numlist, denomlist):
    print("test 1")
    numlist.append(denomlist)
    selectlist = numlist
    selectlist.append("TeamID")
    selectlist.append("PlayerID")

    print("test 2")
    numdf = df[selectlist].groupby(numlist).agg({"PlayerID": "count"})
    denomdf = df[selectlist].groupby(denomlist).agg({"PlayerID": "count"})

    print("test 3")
    mergeddf = pd.merge(numdf, denomdf, on=denomlist)

    print("test 4")
    return mergeddf
</code></pre>

<p>Here is the script I'm trying to use it in:</p>

<pre><code> def team_pr(self, df1):
    numlist = ['PlayerLevel']
    denomlist = ['TeamName', 'Year']

    mergeddf = self.calcfractions(df1, numlist, denomlist)
    print(mergeddf.head(2))
</code></pre>

<p>However, when I run this, I only get to printing "test 2" in def calcfractions, something fails after that point. I think it might have to do with trying to append denomlist to numlist. Any thoughts?</p>

<p>EDIT: The script doesn't "fail", there is no error. It just ends.</p>

<p>So, after concocting my own dataframe with bogus values and trying to work through this, I have found that I run into a <code>ValueError: setting an array element with a sequence</code>. This is due to the fact that you are appending a list to a list and trying to use that as a column index in your df:</p>

<pre><code>numlist = ['PlayerLevel']
denomList = ['TeamName', 'Year']
numlist.append(denomlist) # as you suspected this is problematic:

print(numlist)
['PlayerLevel', ['TeamName', 'Year']]
</code></pre>

<p>Try this instead:</p>

<pre><code>numlist += denomlist
</code></pre>

<p>Is this entire provided snippet wrapped up in some <code>try: except:</code> clause somewhere? In any case, if this doesn't solve your problem, please provide us with a small version of your dataframe.</p>

<hr>

<p><strong>Edit:</strong> From the <a href="https://docs.python.org/3/tutorial/errors.html#handling-exceptions" rel="nofollow">docs on exceptions</a>: "The last except clause may omit the exception name(s), to serve as a wildcard. Use this with extreme caution, since it is easy to mask a real programming error in this way!" </p>

<p>Definitely look through the linked docs, but here is the gist of the immediate problem. It's considered poor practice to write try/except clauses like this:</p>

<pre><code>try:
    # do stuff
except:
    # do different/more stuff if original stuff fails
</code></pre>

<p>because the <code>except:</code> excepts <code>Exceptions</code> of <em>all</em> types. To parrot the quoted docs, <strong>this makes it seem like there is no error.</strong> Furthermore, as this entire question exemplifies, <strong>this makes it impossible to know what is exactly causing an error</strong> (if you even manage to detect one). In nearly all cases, you should have an expectation of what kind of error your code can throw, so your try/except's should look like:</p>

<pre><code>try:
    # do stuff here
except ValueError: # or whatever type of child of Exception()
    # do different/more stuff if original stuff fails
</code></pre>

<p>If you <strong>have</strong> to do a wildcard <code>except:</code> for some crazy reason, ideally refactor, so that such a thing isn't necessary, but at the very least, <code>print()</code> <em>some</em> kind of message indicating that the <code>try:</code> failed. </p>

<p>Generally, to avoid this problem (as it relates to wildcard or even specific exceptions), <strong>do your best to ensure that the try/except clause wraps as little code as is necessary to accomplish your goals.</strong> </p>

python pandas