stats and frequency --output parameter discrepancy

[kulnor]

There is a slight behavioural discrepancy between the stats and frequency commands.

For stats, if I do not specify an --output parameter, a <basename>.stats.csv file is automatically created, and the results are also sent to stdout

For frequency, I must provide a --output parameter along with a filename to produce a CSV file, and the results in that case are not sent to stdout

This could be reconciled as follows:

• if no --output parameter is specified, the results are only sent to stdout
• if --output is specified without a filename, a CSV file is created as <basename>.stats.csv or <basename>.freqs.csv
• If --output is specified with a filename, it is used to save the CSV file (current behaviour)
• When --output is specified, results are not sent to stdout

Note that this could introduce a breaking change for current users or existing workflows that expect' stats' to produce a CSV file without the --output parameter.

[jqnatividad]

Thanks @kulnor for the report.

However, this is by design. The .stats.csv file is the "stats cache".

https://github.com/dathere/qsv/blob/master/docs/PERFORMANCE.md#stats-cache

Having the stats cache is essential to the "smart" commands - allowing them to use the stats cache to make them work faster/smarter - e.g. frequency can short-circuit compiling the enums for a column if its cardinality is equal to the number of rows, etc.

The stats cache is left in the same directory as the file as it has utility by itself, and even for very large datasets, its only in kilobytes.

I was thinking of doing the same for frequency and having a .freqs.csv file, but decided against it as frequency distribution is not as small as the stats cache.

I'm thinking of refactoring the stats cache (#2972), using the SHA256 of a file as the cache key and hide the files in a central directory (e.g. ~/.qsv-cache like I did with describegpt to cache LLM responses). However, computing the SHA256 of a very large file takes a few seconds, that's why I haven't gone ahead with it.

[kulnor]

Ah, thanks for the feedback, and it makes sense. Frequencies can certainly be quite large files.

I'm actually planning to leverage the .stats.csv file for other purposes as well (along with the frequencies). Storing the file in the same directory as the data is very helpful, so would not move it to the cache.

Having a default filename (e.g. foo.freqs.csv) for --output on frequency would be a small but nice quality of life enhancement.

Other options could include a --zip or --parquet parameter/flag (for frequency, stats, and potentially other commands that produce CSV outputs).

[jqnatividad]

I'll look into a default filename convention - perhaps - even making it suite-wide, i.e. if --output option is used but is empty, use the basename of the input file and the command as part of the extension - e.g. <basename>.freqs.csv, <basename>.sample.csv, etc.

As for producing --parquet files, qsv already has the ability to do so with to parquet and sqlp command's --format option.

It also has suite-wide auto compression/decompression using the Snappy streaming compression format:
https://github.com/dathere/qsv?tab=readme-ov-file#automatic-compressiondecompression

It can also auto decompress other compression formats since v4.0.0
https://github.com/dathere/qsv/releases/tag/4.0.0

I thought about auto-compression support for .zip and .gz, but I thought that can be easily composed using purpose-built command line tools for those formats, in keeping with the Unix philosophy, e.g.:

$ qsv frequency bigdata.csv | gzip > bigdata.freqs.csv.gz
$ qsv lens bigdata.freqs.csv.gz