Working with Files

DASCore contains two data structures which are useful for working with directories of fiber data.

FileSpool

The file spool is used to retrieve data from a directory of dascore-readable files. It has the same interface as other spools and is created with the dascore.spool function.

For example:

import dascore
from dascore import examples as ex

# Get a directory with several files
diverse_spool = dascore.get_example_spool('diverse_das')
path = ex.spool_to_directory(diverse_spool)

# Create a spool for interacting with the files in the directory.
spool = (
  dascore.spool(path)
  .select(network='das2')  # sub-select das2 network
  .select(time=(None, '2022-01-01'))  # unselect anything after 2022
  .chunk(time=2, overlap=0.5)  # change the chunking of the patches
)

# Iterate each patch and do something with it
for patch in spool:
  ...

Directory Indexer

The ‘DirectoryIndexer’ is used to track the contents of a directory which contains fiber data. It creates a small, hidden HDF index file at the top of the directory which can be efficiently queried for directory contents (it is used internally by the DirectorySpool).

import dascore
from dascore.io.indexer import DirectoryIndexer
from dascore import examples as ex

# Get a directory with several files
diverse_spool = dascore.get_example_spool('diverse_das')
path = ex.spool_to_directory(diverse_spool)

# Create an indexer and update the index. This will include any new files
# with timestamps newer than the last update, or create a new HDF index file
# if one does not yet exist.
indexer = DirectoryIndexer(path).update()

# get the contents of the directory's files
df = indexer.get_contents()

# This dataframe can be used to ascertain data availability, detect gaps, etc.
cable_id d_distance d_time data_category data_type dims distance_max distance_min file_format file_version instrument_id network path station tag time_max time_min
0 1.0 0 days 00:00:00.004000 distance,time 299.0 0.0 DASDAE 1 /DAS____big_gaps__random__2020_01_03T00_00_00_... big_gaps random 2020-01-03 00:00:07.996 2020-01-03 00:00:00.000
1 1.0 0 days 00:00:00.004000 distance,time 299.0 0.0 DASDAE 1 /DAS______random__2020_01_03T00_00_00_00000000... random 2020-01-03 00:00:07.996 2020-01-03 00:00:00.000
2 1.0 0 days 00:00:00.004000 distance,time 299.0 0.0 DASDAE 1 /DAS____smallg__random__2020_01_03T00_00_08_00... smallg random 2020-01-03 00:00:15.996 2020-01-03 00:00:08.000
3 1.0 0 days 00:00:00.004000 distance,time 299.0 0.0 DASDAE 1 /DAS______random__2020_01_03T00_00_15_99200000... random 2020-01-03 00:00:23.988 2020-01-03 00:00:15.992
4 1.0 0 days 00:00:00.004000 distance,time 299.0 0.0 DASDAE 1 das2 /DAS__das2____random__2020_01_03T00_00_00_0000... random 2020-01-03 00:00:07.996 2020-01-03 00:00:00.000