Sending content directly to scrapy pipeline
<p>I'm working with scrapy. In my current project I am capturing the text from pdf files. I want to send this to a pipeline for parsing. Right now I have:</p>

<pre><code>def get_pdf_text(self, response):
    in_memory_pdf = BytesIO(bytes(response.body))
    in_memory_pdf.seek(0)
    doc = slate.PDF(in_memory_pdf)
    item =OveItem()
    item['pdf_text']=doc
    return item
</code></pre>

<p>pipelines.py</p>

<pre><code>class OvePipeline(object):
    def process_item(self, item, spider):
       .......
        return item
</code></pre>

<p>This works ,but I think it would be cleaner just to yield the result directly and not have to attach the result to an item to get it to a pipeline, like:</p>

<pre><code>def get_pdf_text(self, response):
    in_memory_pdf = BytesIO(bytes(response.body))
    in_memory_pdf.seek(0)
    yield slate.PDF(in_memory_pdf)
</code></pre>

<p>Is this possible?</p>

<p>According to <a href="https://doc.scrapy.org/en/latest/topics/spiders.html#scrapy.spiders.Spider.parse" rel="nofollow">Scrapy documentation</a>, a spider callback has to either return a <code>Request</code> instance(s), dictionary(ies) or <code>Item</code> instance(s):</p>

<blockquote>
  <p>This method, as well as any other Request callback, must return an
  iterable of Request and/or dicts or Item objects.</p>
</blockquote>

<p>So, if you don't want to define a special "item" for the pdf content, simply wrap it into a dict:</p>

<pre><code>def get_pdf_text(self, response):
    in_memory_pdf = BytesIO(bytes(response.body))
    in_memory_pdf.seek(0)

    doc = slate.PDF(in_memory_pdf)

    return {'pdf_text': doc}
</code></pre>

python scrapy