1. When user opens a file which is bigger than LOAD_SIZE (currently 5M), ask what to do
2. User can normally open the file, cancel, or use the new mechanism:
3. Load a chunk of LOAD_SIZE of SQL into memory
4. Split chunk with parseSQL into single queries
5. Run queries and go on with 3.
parseSQL is the bottleneck here, very CPU consuming, as it has to take care of different comment-styles and delimiters. So, the above strategy effected a good compromise regarding overall performance on different tests with worst case SQL files:
- "Wide" table exports with many big sized fields => long lines
- "Narrow" table exports with only one mini-sized field, extended INSERTs => short lines
Especially in the latter case it avoids to cause a hellfire of parseSQL-calls
Still seems to have some memory leaks somewhere.
