google cloud dataflow can run locally but can't run on the cloud
<p>I can't run my pipeline locally, using the "DirectPipelineRunner" but when I use "BlockingDataflowPipelineRunner" to run on the cloud it always shows that "Failed to split source". I don't where is the problem of my pipeline.</p>

<p>I define my custom sources to read a lot of tgz file in my bucket. The following is my code.</p>

<pre><code>class My_Compressed_Source(filebasedsource.FileBasedSource):
    def read_records(self, file_name, range_tracker):
        import tarfile
        import re    
        start_offset = range_tracker.start_position()
        with self.open_file(file_name) as f:
            tar = tarfile.open(fileobj = f)            
            for member in tar.getmembers():
            f = tar.extractfile(member)
            content = f.read()                    
            # do some regular expression process to content 
</code></pre>

<p>And here is my pipeline</p>

<pre><code>pcoll = p | 'Read' &gt;&gt; beam.Read(My_Compressed_Source(args.input, splittable = False))
pcoll | beam.ParDo('parse tar member', Parse_Members())
p.run()
</code></pre>

<p>also, my input path of bucket is "--input gs://mybucket/source/*.tgz" and my job-id is "2016-10-14_09_21_43-4409773810679481086".
I am wondering should I set "splittable" to True. Or if there is something wrong. </p>

<p>Reading a tar file using StringIO as mentioned above cannot be recommended due to having to load all data into memory.</p>

<p>Seems like your original implementation didn't work since tarfile.open() using methods seek() and tell() which are not supported by fileio._CompressedFile object returned by filebasedsource.open_file().</p>

<p>I filed <a href="https://issues.apache.org/jira/browse/BEAM-778" rel="nofollow">https://issues.apache.org/jira/browse/BEAM-778</a> for this.</p>

python google-cloud-dataflow