Split Large File into Multiple Files

I was involved in a research project where I needed to provide a result set back to the requester. The file contained 7 million records which is big to send it over via an email. Even if I manage to send it to the requester via FTP or similar tools, the file is huge for anyone to open in a notepad or in a text editor. It is advisable to split a big file into many smaller files that are more manageable to view, edit or analyse.

If data is being pulled from a database, you can limit the number of records in your query while spooling to a file. If this is not an option then I would use Linux SPLIT command to split in a way that is more meaningful to a requester.  I am going to show split command on how to split a large file into smaller files. A file can be split in many ways, we will go over more than one way to slice a big file.

Split Syntax:
split [OPTION]... [FILE[PREFIX]]
Split based on file size [b]: This option (-b) splits a file into many smaller files of size included with option -b
split -b5000000 file_prefix_
This creates small files of size 500000 bytes.

Split based on the number of small files [n]: This option splits into the specified number of files mentioned with option -n50
split -n50 -e small_prefix
Creates 50 files out of

Split based on the number of lines on each file [l]: This option splits file put the number of lines specified in -l to each file
spit -l500 small_prefix
Creates multiple files with 500 lines of records on each file.

Awesome, you can now split a large file into many small files. I prefer to split mine based on the number of lines per file (-l). There are instances where you may be sick of managing multiple files and wanted to have a big single file. How do you accomplish this? cat is a Linux command which is short for concatenation that allows you to create files, view the content of files, and also let you combine multiple file into a single file. We have xaa, xab, xcc, and xde files that we need to combine into a single file.
cat x* >
cat xaa xab xcc xde >
Split and cat are Linux commands to split a file or concats multiple files. These commands have very powerful and I only showed you the most frequently used options. I would encourage you to man these commands and scan all the options these commands offer.  Hey windows fan, how you do you split and cat file, we would like to hear from you as Linux is not an option for everyone.

Interested in working with me? I can be reached at pbaniya04[at] for any questions, consulting opportunities or you may drop a line to say HELLO. Thank your again for visiting my blog and looking forward to serving you more.

Have a Database-ious Day!

No comments

Powered by Blogger.