FilesystemMgt Deleter issues

Owned by Franz Willer
Last updated: Jul 22, 2010Version comment

Error Description: Filesystems on ONLINE_STORAGE Filesystem Group run out of space, even deleter criteria are specified and studies could be deleted.

Check basic configuration:

Unknown macro: {table-row}
Unknown macro: {table-cell}

Open ONLINE_STORAGE FilesystemMgt service in jmx-console ( link:http://<host>:8080/jmx-console )

Unknown macro: {table-cell}

Links on this page will only work when dcm4chee is on localhost!

Deleter configured to work automatically?

Either ScheduleStudiesForDeletionInterval not NEVER and/or ScheduleStudiesForDeletionOnSeriesStored = true.

Deleter will check free disk space?

DeleterThresholds must not be NONE and have an adequate value.

Info

Current treshold value (dependent on daytime) can be seen in CurrentDeleterThreshold

Study ONLINE Lifetime?

DeleteStudyOnlyIfNotAccessedFor attribute is responsible for how long a study have to be ONLINE.
The value is dependent on online disk space and expected data volume.

Warning

If the value is to high (e.g. 365d and disk capacity is just for 180 days) no studies can be deleted!

Deleter criterias?

Which criterias are enabled? (all enabled criteria must be fulfilled)
Are all basic conditions for the criterias configured correctly?
(e.g. DeleteStudyOnlyIfExternalRetrieveable: Forward Service, Storage Commitment)

Available criterias:

DeleteStudyOnlyIfStorageNotCommited
DeleteStudyOnlyIfExternalRetrieveable
DeleteStudyOnlyIfCopyOnMedia
DeleteStudyOnlyIfCopyOnFileSystemOfFileSystemGroup
DeleteStudyOnlyIfCopyArchived
DeleteStudyOnlyIfCopyOnReadOnlyFileSystem

Check server.log

Logger level must be INFO

Ensure that 'org.dcm4chex.archive.mbean' and/or 'org.dcm4chex.archive.mbean.FileSystemMgt2Service' has at least INFO level
org.dcm4chex.archive
org.dcm4chex.archive.mbean
org.dcm4chex.archive.mbean.FileSystemMgt2Service

A 'Operation completed successfully without a return value.' result message means, that no logger level is set for this category and the level of the parent will be applied!

Message 'Check file system group .. for deletion of studies' does not exist?

Check ScheduleStudiesForDeletionInterval attribute in ONLINE_STORAGE FilesystemMgt service to be not NEVER.

Deleter disabled?

The Deleter is disabled when DeleteStudyIfNotAccessedFor is NEVER and DeleterThresholds is NONE!

Message 'Try to free ... of disk space on file system group ... ' does not exist?

Most likely the UsableDiskSpace is NOT below CurrentDeleterThreshold.
Set DeleterThreshold according to your needs.

Message 'Scheduled ... studies for deletion on file system group ...' does not exist?

Check if studies are ready to delete:

SQL examples shown here are for oracle DBS!

By enabling DEBUG logger level for 'org.jboss.ejb.plugins.cmp' (log4j service in jmx-console) you can find the queries of the deleter in server.log,

Find studies on filesystemgroup not accessed since NUM_DAYS

Set NUM_DAYS to corresponding value of DeleteStudyOnlyIfNotAccessedFor (one hour can be expressed as 1/24)

Simple query: (select of all FS groups!)

SELECT * FROM study_on_fs WHERE access_time < (sysdate - <NUM_DAYS>) AND rownum < 10;

Query of studies stored on ONLINE_STORAGE Filesystems:

SELECT * FROM study_on_fs, filesystem WHERE filesystem.pk = study_on_fs.filesystem_fk
AND filesystem.fs_group_id = 'ONLINE_STORAGE'
AND access_time > (sysdate - <NUM_DAYS>) AND rownum < 10;

As used by deleter:

SELECT * FROM STUDY_ON_FS t0_sof, SERIES t2_s, FILESYSTEM t1_sof_fileSystem, STUDY t7_sof_study
WHERE ((t1_sof_fileSystem.fs_group_id = 'ONLINE_STORAGE' AND t1_sof_fileSystem.fs_status IN (0, 1)
AND t2_s.series_status = 0 AND t0_sof.filesystem_fk=t1_sof_fileSystem.pk))
AND t7_sof_study.pk=t2_s.study_fk AND t0_sof.study_fk=t7_sof_study.pk AND t0_sof.access_time < (sysdate - <NUM_DAYS>);

Result of last statement is empty?

  • Check configuration values
  • Simply all studies are to young to die!
  • STUDY_ON_FS entries are missing!
    This may happen if a study is scheduled for deletion but not deleted (Deletion and all subsequent retries failed or DeleteOrders are removed from deleter queue (e.g. removeAllMessages() ) 
    SELECT DISTINCT study.pk, files.filesystem_fk FROM study, series, instance, files 
WHERE study.pk=series.study_fk AND series.pk=instance.series_fk AND 
instance.pk=files.instance_fk AND NOT EXISTS(SELECT study_on_fs.pk FROM study_on_fs WHERE study_on_fs.study_fk=study.pk);
    In this case the missing STUDY_ON_FS entries must be added manually: 
    INSERT INTO study_on_fs (PK, STUDY_FK, FILESYSTEM_FK, ACCESS_TIME) VALUES
(<PK>,<STUDY_PK>,<FILESYSTEM_FK>,(sysdate - <NUM_DAYS>));
    <PK>...new study_on_fs.pk using auto numbering! e.g. Oracle: study_on_fs_pk_seq.nextval

Check deleter criteria

In general all combinations of deleter criteria can be configured and act in conjunction (logical AND).
There is one exception: DeleteStudyOnlyIfStorageNotCommited and DeleteStudyOnlyIfExternalRetrieveable are mutually exclusive!

Study External retrieveable:

  • Check configuration: Forward service, StorageCommitment activated (RequestStorageCommitFromAETitles attribute in QueryRetrieveScp service).
  • Look for Exceptions, check C_MOVE, C_STORE and N_ACTION if all worked as expected.
  • Check in DB:
    All Instances of study must be external retrieveable:

    Selected attributes:

    DeleteStudyOnlyIfExternalRetrieveable=true

    SELECT COUNT(i.pk) FROM instance i, series, study
WHERE i.series_fk = series.pk AND study_fk = study.pk AND series.study_fk = <STUDY_PK> AND i.ext_retr_aet IS NOT NULL;
    must be equal to 
    SELECT num_instances FROM study WHERE pk = <STUDY_PK>;

Copy of Study available on Filesystemgroup:

  • Check configuration: Copy service, (HSM if archived status is needed).
  • Look for Exceptions
  • Check in DB:

    All Instances of study must be also available on NEARLINE_STORAGE:

    Selected attributes:

    DeleteStudyOnlyIfCopyOnFileSystemOfFileSystemGroup=NEARLINE_STORAGE

    SELECT count(DISTINCT t0_i.pk) FROM instance t0_i, files t3_f, series t5_i_series, study t1_i_series_study,
filesystem t2_f_fileSystem WHERE (((t1_i_series_study.pk=<STUDY_PK>) AND t2_f_fileSystem.fs_group_id = 'NEARLINE_STORAGE'
AND t0_i.series_fk=t5_i_series.pk AND t5_i_series.study_fk=t1_i_series_study.pk
AND t3_f.filesystem_fk=t2_f_fileSystem.pk)) AND t0_i.pk=t3_f.instance_fk;

    All Instances of study must be also available on NEARLINE_STORAGE Filesystem group *with* _ARCHIVED_ status:

    Selected attributes:

    DeleteStudyOnlyIfCopyOnFileSystemOfFileSystemGroup=NEARLINE_STORAGE
    DeleteStudyOnlyIfCopyArchived=true

    SELECT count(DISTINCT t0_i.pk) FROM instance t0_i, files t3_f, series t5_i_series, study t1_i_series_study,
filesystem t2_f_fileSystem WHERE (((t1_i_series_study.pk=<STUDY_PK>) AND t2_f_fileSystem.fs_group_id = 'NEARLINE_STORAGE'
AND t3_f.file_status = 2 AND t0_i.series_fk=t5_i_series.pk AND t5_i_series.study_fk=t1_i_series_study.pk
AND t3_f.filesystem_fk=t2_f_fileSystem.pk)) AND t0_i.pk=t3_f.instance_fk;

    All Instances of study must be also available on a *read-only* Filesystem in READONLY_STORAGE Filesystem group:

    Selected attributes:

    DeleteStudyOnlyIfCopyOnFileSystemOfFileSystemGroup=READONLY_STORAGE
    DeleteStudyOnlyIfCopyOnReadOnlyFileSystem=true

    SELECT count(DISTINCT t0_i.pk) FROM instance t0_i, files t4_f, series t3_i_series, study t1_i_series_study, filesystem t2_f_fileSystem 
WHERE (((t1_i_series_study.pk=<STUDY_PK>) AND t2_f_fileSystem.fs_group_id = 'READONLY_STORAGE' AND t2_f_fileSystem.fs_status = 2 
AND t0_i.series_fk=t3_i_series.pk AND t3_i_series.study_fk=t1_i_series_study.pk AND t4_f.filesystem_fk=t2_f_fileSystem.pk)) 
AND t0_i.pk=t4_f.instance_fk
    must be equal to 
    SELECT num_instances FROM study WHERE pk = <STUDY_PK>;

Copy of Study available on Media (CD / DVD / ...):

  • Check configuration: Media Creation Managment Service (MCMScu) .
  • Look for Exceptions, check C-STORE, N_CREATE and N_ACTION communication with Media Creation SCP if all worked as expected.
  • Check in DB:
    All Instances of the study must be also available on Media with media status COMPLETED(4):

    Selected attributes:

    DeleteStudyOnlyIfCopyOnMedia=true

    SELECT count( t0_i.pk) FROM instance t0_i, series t3_i_series, study t1_i_series_study, media t2_i_media 
WHERE (t1_i_series_study.pk = <STUDY_PK> AND t2_i_media.media_status = 4 
AND t0_i.series_fk=t3_i_series.pk AND t3_i_series.study_fk=t1_i_series_study.pk 
AND t0_i.media_fk=t2_i_media.pk)
    must be equal to 
    SELECT num_instances FROM study WHERE pk = <STUDY_PK>;