FilesystemMgt Deleter issues
Error Description: Filesystems on ONLINE_STORAGE Filesystem Group run out of space, even deleter criteria are specified and studies could be deleted.
Check basic configuration:
Deleter configured to work automatically?
Either ScheduleStudiesForDeletionInterval not NEVER and/or ScheduleStudiesForDeletionOnSeriesStored = true.
Deleter will check free disk space?
DeleterThresholds must not be NONE and have an adequate value.
Info
Current treshold value (dependent on daytime) can be seen in CurrentDeleterThreshold
Study ONLINE Lifetime?
DeleteStudyOnlyIfNotAccessedFor attribute is responsible for how long a study have to be ONLINE.
The value is dependent on online disk space and expected data volume.
Warning
If the value is to high (e.g. 365d and disk capacity is just for 180 days) no studies can be deleted!
Deleter criterias?
Which criterias are enabled? (all enabled criteria must be fulfilled)
Are all basic conditions for the criterias configured correctly?
(e.g. DeleteStudyOnlyIfExternalRetrieveable: Forward Service, Storage Commitment)
Available criterias:
DeleteStudyOnlyIfStorageNotCommited
DeleteStudyOnlyIfExternalRetrieveable
DeleteStudyOnlyIfCopyOnMedia
DeleteStudyOnlyIfCopyOnFileSystemOfFileSystemGroup
DeleteStudyOnlyIfCopyArchived
DeleteStudyOnlyIfCopyOnReadOnlyFileSystem
Check server.log
Logger level must be INFO
Ensure that 'org.dcm4chex.archive.mbean' and/or 'org.dcm4chex.archive.mbean.FileSystemMgt2Service' has at least INFO level
org.dcm4chex.archive
org.dcm4chex.archive.mbean
org.dcm4chex.archive.mbean.FileSystemMgt2Service
A 'Operation completed successfully without a return value.' result message means, that no logger level is set for this category and the level of the parent will be applied!
Message 'Check file system group .. for deletion of studies' does not exist?
Check ScheduleStudiesForDeletionInterval attribute in ONLINE_STORAGE FilesystemMgt service to be not NEVER.
Deleter disabled?
The Deleter is disabled when DeleteStudyIfNotAccessedFor is NEVER and DeleterThresholds is NONE!
Message 'Try to free ... of disk space on file system group ... ' does not exist?
Most likely the UsableDiskSpace is NOT below CurrentDeleterThreshold.
Set DeleterThreshold according to your needs.
Message 'Scheduled ... studies for deletion on file system group ...' does not exist?
- Any Transaction / Rollback Exceptions ?
Try to reduce batch size (ScheduleStudiesForDeletionBatchSize) and/or increase DeleteStudyOnlyIfNotAccessedFor (to limit DB query) - DCMEE-1275 (Deleter stops processing when the first collection of studies doesnt match delete criteria.) Issue?
Check if studies are ready to delete:
SQL examples shown here are for oracle DBS!
By enabling DEBUG logger level for 'org.jboss.ejb.plugins.cmp' (log4j service in jmx-console) you can find the queries of the deleter in server.log,
Find studies on filesystemgroup not accessed since NUM_DAYS
Set NUM_DAYS to corresponding value of DeleteStudyOnlyIfNotAccessedFor (one hour can be expressed as 1/24)
Simple query: (select of all FS groups!)
SELECT * FROM study_on_fs WHERE access_time < (sysdate - <NUM_DAYS>) AND rownum < 10;
Query of studies stored on ONLINE_STORAGE Filesystems:
SELECT * FROM study_on_fs, filesystem WHERE filesystem.pk = study_on_fs.filesystem_fk AND filesystem.fs_group_id = 'ONLINE_STORAGE' AND access_time > (sysdate - <NUM_DAYS>) AND rownum < 10;
As used by deleter:
SELECT * FROM STUDY_ON_FS t0_sof, SERIES t2_s, FILESYSTEM t1_sof_fileSystem, STUDY t7_sof_study WHERE ((t1_sof_fileSystem.fs_group_id = 'ONLINE_STORAGE' AND t1_sof_fileSystem.fs_status IN (0, 1) AND t2_s.series_status = 0 AND t0_sof.filesystem_fk=t1_sof_fileSystem.pk)) AND t7_sof_study.pk=t2_s.study_fk AND t0_sof.study_fk=t7_sof_study.pk AND t0_sof.access_time < (sysdate - <NUM_DAYS>);
Result of last statement is empty?
- Check configuration values
- Simply all studies are to young to die!
- STUDY_ON_FS entries are missing!
This may happen if a study is scheduled for deletion but not deleted (Deletion and all subsequent retries failed or DeleteOrders are removed from deleter queue (e.g. removeAllMessages() )In this case the missing STUDY_ON_FS entries must be added manually:SELECT DISTINCT study.pk, files.filesystem_fk FROM study, series, instance, files WHERE study.pk=series.study_fk AND series.pk=instance.series_fk AND instance.pk=files.instance_fk AND NOT EXISTS(SELECT study_on_fs.pk FROM study_on_fs WHERE study_on_fs.study_fk=study.pk);
<PK>...new study_on_fs.pk using auto numbering! e.g. Oracle: study_on_fs_pk_seq.nextvalINSERT INTO study_on_fs (PK, STUDY_FK, FILESYSTEM_FK, ACCESS_TIME) VALUES (<PK>,<STUDY_PK>,<FILESYSTEM_FK>,(sysdate - <NUM_DAYS>));
Check deleter criteria
In general all combinations of deleter criteria can be configured and act in conjunction (logical AND).
There is one exception: DeleteStudyOnlyIfStorageNotCommited and DeleteStudyOnlyIfExternalRetrieveable are mutually exclusive!
Study External retrieveable:
- Check configuration: Forward service, StorageCommitment activated (RequestStorageCommitFromAETitles attribute in QueryRetrieveScp service).
- Look for Exceptions, check C_MOVE, C_STORE and N_ACTION if all worked as expected.
- Check in DB:
All Instances of study must be external retrieveable:must be equal toSELECT COUNT(i.pk) FROM instance i, series, study WHERE i.series_fk = series.pk AND study_fk = study.pk AND series.study_fk = <STUDY_PK> AND i.ext_retr_aet IS NOT NULL;
SELECT num_instances FROM study WHERE pk = <STUDY_PK>;
Copy of Study available on Filesystemgroup:
- Check configuration: Copy service, (HSM if archived status is needed).
- Look for Exceptions
- Check in DB:
SELECT num_instances FROM study WHERE pk = <STUDY_PK>;
must be equal to
Copy of Study available on Media (CD / DVD / ...):
- Check configuration: Media Creation Managment Service (MCMScu) .
- Look for Exceptions, check C-STORE, N_CREATE and N_ACTION communication with Media Creation SCP if all worked as expected.
- Check in DB:
All Instances of the study must be also available on Media with media status COMPLETED(4):must be equal toSELECT count( t0_i.pk) FROM instance t0_i, series t3_i_series, study t1_i_series_study, media t2_i_media WHERE (t1_i_series_study.pk = <STUDY_PK> AND t2_i_media.media_status = 4 AND t0_i.series_fk=t3_i_series.pk AND t3_i_series.study_fk=t1_i_series_study.pk AND t0_i.media_fk=t2_i_media.pk)
SELECT num_instances FROM study WHERE pk = <STUDY_PK>;