Mialian,
That’s not c, sfederman’s still in Python at that point. If you replace the c.execute line in the Option #3 code with that suggested above it should work. Alternative try this method instead: https://www.polarmicrobes.org/?p=859.

sfederman,
I would like to implement your suggestion to (hopefully) speed up the process further, but I don’t understand exactly how to implement it. I am not used to c-code. Could you show where and how you would implement it in Jeff’s code!?

Thanks in advance!

Further modifications… if you use the above fgrep –max-count=1 method you will invariably incorrectly match some lines in the .dmp file (wherever your query gi appears in a target gi). Since –max-count=1 aborts the search after one match you will often not return the line you want. Use grep(z) -P “23435/t” instead. The -P option uses Perl regex, allowing the use of the \t character. This example would match:

23435 46
123435 893

but not:

234357 21456

This effectively reduces the .dmp file to a size that allows dictionary creation, but it will contain a number of erroneous matches (not a problem, because the dictionary will force exact matches downstream).

I want to let you know one adjustment I made in order to speed up the queries. When creating the database, I adjusted your script above slightly:

c.execute('''CREATE TABLE gi_taxid (
gi INTEGER PRIMARY KEY,
taxid INTEGER)''')

This makes the gi a primary key, and indexes this field in the file – greatly speeding up searches.