Python Sample Identification ETL

This sample shows the steps needed to implement a byte identifier ETL in Python.

Sample File Format

This sample will identify a fake file format we call Acme. Acme Corporation is a known supplier of nefarious tools and explosives. Their file format begins with a five byte magic value followed by eleven bytes in a data structure.

0000h: 88 77 66 55 00 11 22 33 44 55 66 77 88 99 AA BB
0010h: CC                                               Ì

Source Code

 1 import truxton
 2 
 3 def main():
 4 
 5   etl = truxton.etl()
 6   etl.name = "Acme Identifier"
 7   etl.description = "This ETL identifies files using the Acme method"
 8   etl.queue = "acme"
 9   etl.stage = 2
10   etl.id = 26572
11   etl.addtype(truxton.Type_Unknown)
12 
13   message = etl.getmessage()
14 
15   while message is not None:
16     if message.length >= 16 and message.signature == 0x55667788:
17       file_in_truxton = message.file()
18 
19       file_in_truxton.seek(4)
20 
21       if file_in_truxton.read(1) == 0:
22         file_in_truxton.changetype(10111)
23         message.file_type = 10111
24         message.route()
25 
26     message = etl.getmessage()
27 
28 if __name__ == "__main__":
29     main()

Code Walkthrough

Lines 5-11 setup the ETL. The message queue name will be "acme", we are an early stage and want to receive Type_Unknown files.

Line 13 starts the ETL logic and waits until a message arrives on the "acme" queue.

Line 16 looks at the data in the message to see if it is even possible for the file we have received to be an Acme file. The signature member contains the first four bytes of the file. Acme's format has a five byte signature which means we will have to read bytes from the file in order to perform a valid check. Opening a file is rather expensive and we want identification to be as fast as possible. By checking using signature to check the first four bytes, we can avoid unnecessarily incurring a performance hit by reading from a file we know can't possibly be Acme.

Line