-
Notifications
You must be signed in to change notification settings - Fork 101
Description
I'm trying to train an FFN, and the first 2 steps (partition and build the coordinate file) seem to go fine, but training throws Key Value errors. On further inspection (using TFRecord Viewer), I get this error:
tensorflow.python.framework.errors_impl.DataLossError: corrupted record at 0.
Any help would be super appreciated!
TF version 1.13.2, and here are the exact calls I'm making:
For computing partitions:
python ../../ffn/compute_partitions.py \ --input_volume ../training_data_img.h5:label \ --output_volume training_data2.h5:af \ --thresholds 0.025,0.05,0.075,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9 \ --lom_radius 3,3,3 \ --min_size 10000
For building the TFRecord file:
python ../../ffn/build_coordinates.py \ --partition_volumes validation:training_data2.h5:af \ --coordinate_output tf_record_fileyw \ --margin 3,3,3
and for training:
python ../../ffn/train.py \ --train_coords tf_record_fileyw \ --data_volumes validation1:../training_data_img.h5:image \ --label_volumes validation1:../training_data_img.h5:label \ --model_name convstack_3d.ConvStack3DFFNModel \ --model_args "{\"depth\": 2, \"fov_size\": [2, 2, 2], \"deltas\": [2, 2, 2]}" \ --image_mean 72 \ --image_stddev 33