Randomly distribute files into train/test given a ratio
<p>I am at the moment trying make a setup script, capable of setting up a workspace up for me, such that I don't need to do it manually. 
I started doing this in bash, but quickly realized that would not work that well.  </p>

<p>My next idea was to do it using python, but can't seem to do it a proper way.. My idea was to make a list (a list being a .txt files with the paths for all the datafiles), shuffle this list, and then move each file to either my train dir or test dir, given the ratio.... </p>

<p>But this is python, isn't there a more simpler way to do it, it seems like I am doing an unessesary workaround just to split the files. </p>

<p>Bash Code: </p>

<pre><code># Partition data randomly into train and test. 
cd ${PATH_TO_DATASET}
SPLIT=0.5 #train/test split
NUMBER_OF_FILES=$(ls ${PATH_TO_DATASET} |  wc -l) ## number of directories in the dataset
even=1
echo ${NUMBER_OF_FILES}

if [ `echo "${NUMBER_OF_FILES} % 2" | bc` -eq 0 ]
then    
        even=1
        echo "Even is true"
else
        even=0
        echo "Even is false"
fi

echo -e "${BLUE}Seperating files in to train and test set!${NC}"

for ((i=1; i&lt;=${NUMBER_OF_FILES}; i++))
do
    ran=$(python -c "import random;print(random.uniform(0.0, 1.0))")    
    if [[ ${ran} &lt; ${SPLIT} ]]
    then 
        ##echo "test ${ran}"
        cp -R  $(ls -d */|sed "${i}q;d") ${WORKSPACE_SETUP_ROOT}/../${WORKSPACE}/data/test/
    else
        ##echo "train ${ran}"       
        cp -R  $(ls -d */|sed "${i}q;d") ${WORKSPACE_SETUP_ROOT}/../${WORKSPACE}/data/train/
    fi

    ##echo $(ls -d */|sed "${i}q;d")
done    

cd ${WORKSPACE_SETUP_ROOT}/../${WORKSPACE}/data
NUMBER_TRAIN_FILES=$(ls train/ |  wc -l)
NUMBER_TEST_FILES=$(ls test/ |  wc -l)

echo "${NUMBER_TRAIN_FILES} and ${NUMBER_TEST_FILES}..."
echo $(calc ${NUMBER_TRAIN_FILES}/${NUMBER_OF_FILES})

if [[ ${even} = 1  ]] &amp;&amp; [[ ${NUMBER_TRAIN_FILES}/${NUMBER_OF_FILES} != ${SPLIT} ]]
    then 
    echo "Something need to be fixed!"
    if [[  $(calc ${NUMBER_TRAIN_FILES}/${NUMBER_OF_FILES}) &gt; ${SPLIT} ]]
    then
        echo "Too many files in the TRAIN set move some to TEST"
        cd train
        echo $(pwd)
        while [[ ${NUMBER_TRAIN_FILES}/${NUMBER_TEST_FILES} != ${SPLIT} ]]
        do
            mv $(ls -d */|sed "1q;d") ../test/
            echo $(calc ${NUMBER_TRAIN_FILES}/${NUMBER_OF_FILES})
        done
    else
        echo "Too many files in the TEST set move some to TRAIN"
        cd test
        while [[ ${NUMBER_TRAIN_FILES}/${NUMBER_TEST_FILES} != ${SPLIT} ]]
        do
            mv $(ls -d */|sed "1q;d") ../train/
            echo $(calc ${NUMBER_TRAIN_FILES}/${NUMBER_OF_FILES})
        done
    fi

fi   
</code></pre>

<p>My problem were the last part. Since i picking the numbers by random, I would not be sure that the data would be partitioned as hoped, which my last if statement were to check whether the partition was done right, and if not then fix it.. This was not possible since i am checking floating points, and the solution in general became more like a quick fix. </p>

<p><code>scikit-learn</code> comes to the rescue =)</p>

<pre><code>&gt;&gt;&gt; import numpy as np
&gt;&gt;&gt; from sklearn.cross_validation import train_test_split
&gt;&gt;&gt; X, y = np.arange(10).reshape((5, 2)), range(5)
&gt;&gt;&gt; X
array([[0, 1],
       [2, 3],
       [4, 5],
       [6, 7],
       [8, 9]])
&gt;&gt;&gt; y
[0, 1, 2, 3, 4]


# If i want 1/4 of the data for testing 
# and i set a random seed of 42.
&gt;&gt;&gt; X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
&gt;&gt;&gt; X_train
array([[4, 5],
       [0, 1],
       [6, 7]])
&gt;&gt;&gt; X_test
array([[2, 3],
       [8, 9]])
&gt;&gt;&gt; y_train
[2, 0, 3]
&gt;&gt;&gt; y_test
[1, 4]
</code></pre>

<p>See <a href="http://scikit-learn.org/stable/modules/generated/sklearn.cross_validation.train_test_split.html" rel="nofollow">http://scikit-learn.org/stable/modules/generated/sklearn.cross_validation.train_test_split.html</a></p>

<hr>

<p>To demonstrate:</p>

<pre><code>alvas@ubi:~$ mkdir splitfileproblem
alvas@ubi:~$ cd splitfileproblem/
alvas@ubi:~/splitfileproblem$ mkdir original
alvas@ubi:~/splitfileproblem$ mkdir train
alvas@ubi:~/splitfileproblem$ mkdir test
alvas@ubi:~/splitfileproblem$ ls
original  train  test
alvas@ubi:~/splitfileproblem$ cd original/
alvas@ubi:~/splitfileproblem/original$ ls
alvas@ubi:~/splitfileproblem/original$ echo 'abc' &gt; a.txt
alvas@ubi:~/splitfileproblem/original$ echo 'def\nghi' &gt; b.txt
alvas@ubi:~/splitfileproblem/original$ cat a.txt 
abc
alvas@ubi:~/splitfileproblem/original$ echo -e 'def\nghi' &gt; b.txt
alvas@ubi:~/splitfileproblem/original$ cat b.txt 
def
ghi
alvas@ubi:~/splitfileproblem/original$ echo -e 'jkl' &gt; c.txt
alvas@ubi:~/splitfileproblem/original$ echo -e 'mno' &gt; d.txt
alvas@ubi:~/splitfileproblem/original$ ls
a.txt  b.txt  c.txt  d.txt
</code></pre>

<p>In Python:</p>

<pre><code>alvas@ubi:~/splitfileproblem$ ls
original  test  train
alvas@ubi:~/splitfileproblem$ python
Python 2.7.12 (default, Jul  1 2016, 15:12:24) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import os
&gt;&gt;&gt; from sklearn.cross_validation import train_test_split
&gt;&gt;&gt; os.listdir('original')
['b.txt', 'd.txt', 'c.txt', 'a.txt']
&gt;&gt;&gt; X = y= os.listdir('original')
&gt;&gt;&gt; X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)
&gt;&gt;&gt; X_train
['a.txt', 'd.txt', 'b.txt']
&gt;&gt;&gt; X_test
['c.txt']
</code></pre>

<p>Now move the files:</p>

<pre><code>&gt;&gt;&gt; for x in X_train:
...     os.rename('original/'+x , 'train/'+x)
... 
&gt;&gt;&gt; for x in X_test:
...     os.rename('original/'+x , 'test/'+x)
... 
&gt;&gt;&gt; os.listdir('test')
['c.txt']
&gt;&gt;&gt; os.listdir('train')
['b.txt', 'd.txt', 'a.txt']
&gt;&gt;&gt; os.listdir('original')
[]
</code></pre>

<p>See also: <a href="http://stackoverflow.com/questions/8858008/how-to-move-a-file-in-python">How to move a file in Python</a></p>

python bash text-files file-handling train-test-split