Is there a way to read all files excluding a defined list of files in python apache beam?
<p>My use case is that I am batch processing files in a bucket that is constantly being updated with new files. I don't want to process csv files that have already been processed. </p>

<p>Is there a way to do that? </p>

<p>One potential solution I thought of, is to have a text file that maintains a list of processed files and then reads all csv files excluding the files in the processed list. Is that possible? </p>

<p>Or is it possible to read a list of specific files? </p>

<p>There's not a good built-in way to do this, but you can have one stage of your pipeline that computes the list of files to read as you suggested, the using a DoFn that maps a filename to the contents of the file.  See <a href="http://stackoverflow.com/questions/39578932/reading-multiple-gz-file-and-identifying-which-row-belongs-to-which-file">Reading multiple .gz file and identifying which row belongs to which file</a> for information about how to write this DoFn</p>

python google-cloud-dataflow dataflow apache-beam