Ad-Hoc Multifiles

One common problem in dealing with large amounts of data is that it often arrives in several files, perhaps over a period of time. The classic way of dealing with this is to have a programme which can identify all the files that possess a common record format and use this to feed the files to the appropriate processing.

Ab Initio has a rather elegant way of dealing with this problem. An ad hoc multifile is a variant of the standard file component that allows a graph to treat several serial files as a single item. Moreover, if you put some processing in your graph to identify and handle different record formats, the files don't even need to have the same record format. This approach can be used with output, intermediate, and lookup files as well as input files.

One method of defining an ad hoc multifile is to list the files explicitly. The process is pretty straight forward. Start by placing a file component on your graph (or selecting an existing one) then open the properties dialog and select the 'Description' tab. Now select 'Partitions' in the 'Data Location' area and click 'Edit' to open the 'Define multifile partitions' dialog. Click the 'New' button and type in the first file name. Repeat this process until you've entered all the file names that make up your compound multifile. When you're done, click the 'OK' button.

The good news is that the file component now behaves as if it were a n-way multifile system, where 'n' is the number of files you defined. Downstream components can even be set to use this as their layout. The bad news is that the Ab Initio shell commands such as m_ls and m_dump will not operate on your compound multifile, because they'll only see one element of it at a time.

Of course, if the above was the only way to use compound multifiles, they would have limited application. Luckily, Ab Initio is rather cleverer than that. Instead of listing the files individually, you can enter a wildcard. Thus you could have an entry like $AI_SERIAL/sales_*.dat_* and all the files matching that pattern would immediately become part of your ad hoc multifile.

You could go further still. Suppose you had a helper application that landed the files and then handed off to your graph. You could define your file name as a shell variable, such as ${SALES_DELTAS}, and your helper application could load that shell variable with a list of file names before it runs the graph. You could even put this processing in the start script of your graph or use a run programme component to invoke it.

Another option is to define the file name of your compound multifile as the output of a shell command, say $(ls $AI_SERIAL/sales_*.dat_* | grep -v FULL). This is nice because it allows you to restrict the selection in a complex manner but all the processing is visible in the file component.

With so much flexibility on offer, there are very few situations involving multiple file loads that can't be handled.