Segfault while embedding Python using scrapy in C
<p>i'm currently trying to run some scrapy spider (in Python) from a C code but i'm constantly getting segfault while testing.</p>

<p>i have this code that allow me to run a simple python function from c :</p>

<pre><code>    int main() {
    PyObject *retour, *module, *fonction, *arguments;
    char *resultat;

    Py_Initialize();
    PySys_SetPath(".");
    module = PyImport_ImportModule("test");
    fonction = PyObject_GetAttrString(module, "hellowrld);

    arguments = Py_BuildValue("(s)", "hello world");
    retour = PyEval_CallObject(fonction, arguments);
    PyArg_Parse(retour, "s", &amp;resultat);
    printf("Resultat: %s\n", resultat);

    Py_Finalize();
    return 0;
}
</code></pre>

<p>if i call the hellowrld function that looks like this in test.py</p>

<pre><code>def hellowrld(arg):
    return arg + '!!'
</code></pre>

<p>it will work fine, but i'm trying to run the function runspider_with_url from this code :</p>

<pre><code>    # -*- coding: utf-8 -*-
import scrapy
from scrapy.crawler import CrawlerProcess
import lxml.etree
import lxml.html

class GetHtmlSpider(scrapy.Spider):
    name = "getHtml"
    def __init__(self, var_url=None, *args, **kwargs):
        super(GetHtmlSpider, self).__init__(*args, **kwargs)
        self.start_urls = [var_url]
    def parse(self,response):
        root = lxml.html.fromstring(response.body)
        print lxml.html.tostring(root)
def runspider_with_url(var_url):
    process = CrawlerProcess({
        'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
    })
    process.crawl(GetHtmlSpider,var_url=var_url)
    process.start()
    return "It works!!"
</code></pre>

<p>And when i'm trying to execute it, i keep getting segmentation fault errors</p>

<p>I tried to add this at the end of my python</p>

<pre><code>foo = runspider_with_url("http://www.google.com/")
print foo
</code></pre>

<p>This call works when i execute it in bash with the command : </p>

<pre><code>python -c 'import get_html; get_html.runspider_with_url("https://www.wikipedia.org")'
</code></pre>

<p>So i could ask my C program to execute the python with bash, and writte the result in a .txt, but i'd rather not.</p>

<p>Thanks</p>

<p>There is a segfault at <code>PyArg_Parse(retour, "s", &amp;resultat);</code> because the script raises an exception and <code>retour</code> is NULL. After wrapping that in some error detection, the problem was that <code>PySys_SetPath(".");</code> replaces the existing <code>sys.path</code> so things like <code>scrapy</code> could no longer be imported. So I fixed that with a quick call to python code that inserts in <code>sys.path</code>. Adding additional error handling along the way, you get</p>

<pre><code>#include &lt;stdio.h&gt;
#include &lt;stdlib.h&gt;
#include &lt;Python.h&gt;

int main() {
    PyObject *retour, *module, *fonction, *arguments;
    char *resultat;

    printf("starting\n");

    Py_Initialize();
    //PySys_SetPath(".");
    if(PyRun_SimpleString("import sys;sys.path.insert(0, '.')"))
    {
        printf("path expansion failed\n");
        return(2);
    }

    module = PyImport_ImportModule("test");
    if(module == NULL) {
        printf("import failed\n");
        PyErr_Print();
        return(2);
    }

    fonction = PyObject_GetAttrString(module, "runspider_with_url");
    if(fonction == NULL) {
        printf("could not find function\n");
        return(2);
    }

    arguments = Py_BuildValue("(s)", "hello world");
    if(arguments == NULL) {
        printf("arg parsing failed\n");
        return(2);
    }

    printf("Calling\n");
    retour = PyEval_CallObject(fonction, arguments);
    printf("Returned\n");

    // note: need to release arguments
    Py_DECREF(arguments);

    if(retour == NULL) {
        printf("It all went wrong\n");
        PyErr_Print();
        return(2);
    }

    PyArg_Parse(retour, "s", &amp;resultat);
    printf("Resultat: %s\n", resultat);

    Py_Finalize();
    return 0;
}
</code></pre>

python c scrapy embed