posted on Thursday, August 11, 2011 - 10:03 pm
> Neelam, I would advise caution in prescribing to vendors exactly how they should implement their data storage. The implementation of storage of this type of data carries a number of subtleties of security, reliability, etc.
Many current PACS systems archive their data onto a write-once, read-many (WORM) archive system - for example DVD-R discs. Once data is written to such an archive, it is set in stone and cannot be changed. This is, in practice, highly desirable, as it dramatically mitigates the risk of damage from malicious software (virus), software/hardware malfunction (a malfunctioning hard-drive controller card can irretrievably corrupt an online data store) and, indeed, deliberate tampering (which, one would hope would be infrequent). For this reason, many data storage vendors offer optional 'WORM' functionality, even if this is not an inherent function of the technology (e.g. WORM magnetic tape is available from some vendors - by the incorporation of electronics that block a tape drive from overwriting existing data)
While the technology exists for archiving on 2nd tier online storage, there may still be a role for optical/tape archive for disaster recovery or 3rd tier archiving.
My understanding, is that you wish to ease migration. That is a laudable goal. However, it is not necessary for this to dictate the storage method. Many PACS systems are able to 'reconstruct' DICOM objects in real-time, in order to ensure that objects exported to CD, or sent to 3rd party applications/workstations, etc. all have up-to-date metadata.
Perhaps a better solution would be to require vendors to provide a method of bulk exporting their data in Part 10 format for the purposes of migration. This could be extended so that the vendors should ensure that proprietary metadata, is translated into DICOM standard metadata (where such standards exist).
Mark, Thanks you make very valid points here. PACS has 2 databases
1. Dicom image archive-- files should be stored in standard DICOM part 10 file format. At the end of contract supplier must support bulk migration of data-- minimum of 1TB per day in part 10 format. Supplier must support an IHE data migration profile when it becomes available. 2. PACS Relational database stores metadata information regarding images (see section 4 of groupbspecification document). Minimum metadata fields are described in section 4 of spec doc. These are largely provided from RIS (demographic updates provided by PAS) These fields must be migrated from old PACS. At the end of contract, supplier must clarify how these data fields will be migrated out. If there are any requirement for a new vendor. Vendor must be explicit.
PACS vendor must be capable of displaying reports in CDA format. There is no requirement for reports to be migrated from 1 PACS to another. PACS must be capable of recieving/storing if reqd reports from RIS for display.
Sven Bollue has written a good post on relationship between image file archive & relational database. Thanks Sven.
I think you might have a hard time with the "1TB per day" part - the supplier has no control over the network infrastructure between some aspects of the data. Some of it may be in an archive for example or the trust may have changed it (the network design) over the years (from first design)?
With the best will in the world the supplier might be constrained by the performance of the trust network.
posted on Friday, August 12, 2011 - 08:56 am
> Sven Bollue has written a good post on relationship between image file archive & relational database. Thanks Sven.
My pleasure, Neelam. Always interesting discussions on this board!
I think Mark's suggestion goes to the root of the issue:
"Perhaps a better solution would be to require vendors to provide a method of bulk exporting their data in Part 10 format for the purposes of migration"
Way back in the thread archives the simple solution (Query/Retrieve) was criticised for being too slow. For very large migrations this may be valid - but Q/R is 'slow' (compared to more basic disk-to-network operations) for two reasons-
1. The underlying mechanisms weren't designed for the kind of performance challenges and expectations of today's environments. This however is secondary to:
2. Each transfer requires resources (CPU, Database) to construct an up-to-date version of the data in DICOM format from the data held in the database. Good database and server design and implementation can minimise this but the time taken to perform this is necessarily finite.
However - while data is updated in the database (rather than the file store), this step is inevitable for *any* mechanism (only exception below). A process of exporting part 10 objects would suffer the same performance issue.
There is the possibility (the exception to above) of taking the database itself and transforming it to the 'new' schema as a database-to-database operation. That may be a possibility for some vendors although I suspect not for all.
There is also the possibility (for the future) of specifying a standard database schema. As a developer I would I would be quite wary of that. Database technology is as much in flux as any other in ICT and a standards-based approach would necessarily limit the advantages gained by any innovation around, for example non-relational or NOSQL databases, never mind within the relational database world.
So - other than the exception above - all possible options for the migration at hand are subject to the same performance constraints and really, Q/R as the simplest option stands out I guess.
I don't think the two profiles mentioned are much help in migration - my understanding is that they relate to synchronisation of _already copied_ data. Perhaps Dave/David know of any effort to propose a specific migration profile?
posted on Friday, August 12, 2011 - 11:52 am
The limiting speed factor with Q/R method will usually be the slowest part of the network across which you are doing the operation.
If you're going across an external network link e.g N3, it's going to be slow.
If you're within a LAN with good Gbps connection between the boxes this should still be quite quick.
In fact I'm not sure this is actually going to be much slower than the method of recreating up to date part 10 files and writing these on mass to another box which would be connected similarly across that same LAN. In which case the Q/R may be quicker overall if it's done directly to the incoming PACS box as it avoids the need for the second step of data import. Q/R certainly has the advantage of being the quickest to set up. Any thoughts?
"I think you might have a hard time with the "1TB per day" part - the supplier has no control over the network infrastructure between some aspects of the data. Some of it may be in an archive for example or the trust may have changed it (the network design) over the years (from first design)? "
Each extra day or week of migration is costly- as you pay outcoming vendor, migration vendor & incoming vendor for migration. So it is in Trusts interests to ensure that networks are adequate by 2023 (by then we expect to get 50mbps home broadband ). Minimum of 1TB per day in 2023 should be possible dont you think
What we have today 1. DICOM q/r is standards based migration but slow. Hence will be costy in my view. 2. Bulk migration- quicker but relies on co-operation by outgoing vendor to provides links to the relational database. Each Trust will be required to make their own decisions about what works for them. Cost comparisn should be performed.
Dave & David are hopefully listening to this thread and hopefully we will see standards emerge in the future which will help ease the migration process. Having said that I think due to DICOM, PACS migration is still relatively technically straightforward ( if LSPs co-operate).
Martin Peacock is right that the elephant in the room is RIS. However, following the discussions on PACS-- I can actually see a path for RIS migration. 1. Reports to be migrated as CDA doc (instead of images you have reports) 2. Relational database for RIS-- to be migrated as HL7 messages Locally we are planning to leave the old RIS database as read only files (migrating only the above 2)
RIS migration is also key to continued innovation in RIS. Now that CFH/LSP days are over, we are seeing some very good new mordern RIS entrants into NHS marketplace.
Anyway, a profile for the bulk export of everything to external drives in an up-to-date form in a standard format may be useful to address two problems:
1. the new and old archive separated by a slow or over-burdened or expensive network (LSP over N3 ?)
2. the new and old archive on the same site with a fast connection, but a poorly performing Q/R implementation
I agree with those who said that the existing native format archive plus database design is not going to be changed any time soon, if ever, not because it is not possible to design a live archive to be both high performance and up to date and standard, but rather because this is too expensive for legacy implementations to change if that is not already how they built it.
The WORM medium argument is theoretically spurious, since new files can be written with the revised images with "good" headers, and the old ones deprecated (e.g., by IOCM KOS rejection objects), such that the final state of ingesting them all is correct (and could even contain a reconstructed audit trail of the changes), but that is not a common design.
But PACS vendors are not going to re-implement their designs or revisit fundamental design decisions anytime in our lifetime, and since disk for the purpose of transport is cheap, and much existing code could be reused to implement a "bulk data export" profile, I suspect such an additional feature might be much more attractive to product managers; i.e., it is low risk and they can charge for it.
However, the practicality of such a profile may depend on whether or not there are so many other overwhelming issues with migration (RIS unreconciled studies, uncorrected studies, lack of standard strings or codes, lack of strings or codes to drive destination features (like hanging protocols), other proprietary features not mappable to standard encoding, local conventions abusing standard fields, sporadic export failures of pixel data), that migration remains a "hard" problem and that each will remain a custom (aka. expensive) project. Or to put it another way, many PACS contain a lot of garbage, and migration forces the issue of dealing with it, many sites are heavily customized and then have to pay a price to replicate or undo that customization at migration time, and many PACS have bugs that manifest themselves at migration time.
At least the use of an intermediate standard "archive copy" might separate the export problems (making good data, mapping stuff to a standard form) from the import problems (unmapping standard stuff to the destination proprietary requirements, reconciling to the new RIS codes and identifiers, etc.).
Actually it occurs to me that you could split this into three steps, 1) the export of the standard archive from the old PACS, 2) a third-party service to clean it up, with knowledge of, but not the support/involvement of the old or new PACS, producing a revised standard archive, 3) the import of the (revised) standard archive contents into the new PACS.
The reason I reiterate all this is that if we are going to try and convince IHE to develop a new profile for migration, we need to have a very clearly articulated scope and well-defined and plausible use cases. Defining what might be out of scope is probably as important as enumerating what should be in scope.
To some extent this depends on just how high fidelity one expects the migration to be (e.g., can one live without the prior studies hanging "properly" on the new system, or even being recognized as "appropriate" priors if the new PACS is selective?).
Some of these may be easier problems in the UK because at least you guys have standard codes and strings for requested procedures, right ? (Not series though, I suppose). And with the NHS identifier you don't have a problem with patient identity mapping, right ?
In the NHS we are fortunate in this regard esp with recent implementations our PACS data is largely clean for each hospital 1. We use our unique PAS ID consistent (NHS no is a national unique ID but does not work for business as there is no real time availability) 2. We have national RIS exam codes (this work started by our group led by Keith Foord previous secretary of our Group) 3. All exams stored in our local PACS have a RIS entry. Most NHS hospitals will be similar. I really doubt we need a huge amount of data cleansing.
You talk about customization. I agree that more customization increases lock-in. One phenomenon which is rampant is a desire of many PACS vendors to perform some RIS functions. Dave Harvey alerted me to this when I simply presumed that DMWL provision was a role of PACS. This is a single big non standard implementation of PACS in the NHS. Some PACS vendors insist on providing this functionality. Customers need to be wary of this. RIS should provide DMWL. PACS DMWL provision provides an additional point of failure.
Our relevant prior call is based on RIS codes--so migration should not be a problem.
Keeping migration profile simple should be the goal. It should be good practice to ensure data stored is clean & consistent.
> > Reading this from the USA, I have to wonder why the current and possible future PACs/RIS vendors have not commented on any of your "ideas" after over a year of these postings about 2013? Any thoughts on that?
Majority PACS vendors read this forum. I know that . They need to be aware what the user community is saying.
However, the changes into products will only happy if they are commercially viable. David Clunie also mentions this.
Our forum I do believe has made a huge contribution to global PACS. 1. Desk-top Integration between different vendors of RIS & PACS 2. Pushing forward adoption of XDS-I by vendors etc
3 Things which are going to set PACS vendors apart in future are 1. upfront approach to plug-ins 2. upfront approach to migration at end of contract 3. Move away from hardware provision--workstations & storage Let us see which vendors can rise above the traditional approach of lock-ins.
We will see!!! Those who do go forward & change in 2013 will get some good deals & huge cost savings!!! Dont be fooled by discounts which may not be real.
> 1. DICOM q/r is standards based migration but slow. Hence will be costy.
Hi Neelam, I would caution against the inference that DICOM Q/R would by definition be costly - whilst your 3-vendor train of thought makes it seductive reasoning, recall that the revenue costs for connecting to the DataCentre are huge, so any mechanism to reduce the amount of time the old vendor is in use would have plenty of savings there to make the overall business case very cost effective.
Having an intermediate archive/VNA means in the context of data migration you'd only have 2 vendors at a time: old vendor+migration vendor; then migration vendor+new vendor, so the amount of time with the old vendor is minimised.
Of course, I know that with training on the new PACS and so on, it means the old and new systems would overlap obviously. But with a VNA for the transition period, the service/contractual overlap of the old and new vendors would be minimal, ie only that required operationally during project go-live. This arguably makes the DICOM Q/R approach carry cost savings; I think they are low-hanging fruit for Trusts who start early.
posted on Wednesday, August 17, 2011 - 03:19 pm
You are right. We should not discount any option.
Trusts need to get total cost of both options 1. DICOM QR--including cost of running VNA/new vendor (alongside old vendor) for the period of transfer 2. Bulk transfer---including cost of running new vendor (alongside old vendor) for duration of transfer.
This info has been sent to me about a recent real migration. I have removed the names of vendors involved.
"There has been a recent migration one of the largest hospitals in the Netherlands from a X PACS to a Y PACS. X PACS said it would take more than a year to migrate the 35 million images. The hospital contacted a migration vendor and they connected to the PACS X database to be able to retrieve the images from PACS X and send them to paces Y.
They tested for about a month to finetune the hardware and software. At the end they were able to receive, check, compress to EOL, etc about 140.000 images per hour (average) !!! The whole migration was done in about 2 months were half the time was used in checking data.
Migration vendor was using 6 computers each running 3 send tasks to get the images from the PACS X system, decompress them, verify the header, add the NHS number, etc and send them to PACS Y.
PACS Y were using a sql cluster for the database and 4 virtual windows 2008 servers under VMware (2 bare metals) to receive and compress the images.
These 4 servers were behind a virtual loadbalancer (loadbalancer.org)."
posted on Wednesday, August 17, 2011 - 04:46 pm
Just to be clear with respect to the "intermediate archive" proposal, I (at least) was not referring to having a true "VNA" (whatever you define that to be) in the middle.
I was talking about a low cost bunch of throw away disks and a file system that one could physically transport, with the software smarts provided by a the authoring system (whether that be a source PACS feature or a third-party provided service or tool).
Not to say that one couldn't use a "VNA" for this if you could afford to or needed it for another purpose (using "VNA" in the sense of what current vendors of "VNAs" might mean by it).
We need answer to the following questions regarding PACS data migration from CFH/LSP 1. What is the contractual position for handover of data by 2013? 2. What is the earliest date that data migration can commence? 3. What charges will there be from an LSP perspective for migrating the data off? 4. Would there be any contractual limitations in re-using the existing PACS hardware - servers and workstations? 5. If we decided to migrate at database level how would we obtain a database copy and at what cost? 6. Please confirm that the data could be migrated in DICOM standard part 10 file format. If not, what format will it be in? 7. Is there any proprietary compression applied to the data? 8. What kind of data migration will be supported - Bulk or DICOM Q/R? 9. For each type of migration method what is the anticipated period of time for the migration to complete?
I am wondering why a VNA would be an intermediate archive? One of the major selling arguments for VNAs is that they ease future PACS migrations because the data just stays in the VNA. The new PACS get's connected to the VNA and you are done. Of cause when you want to switch your VNA vendor you will eventually nevertheless have to migrate the data. But this decouples the PACS lifecycle from the archive lifecycle.
You just need to make sure to ask the VNA vendor how you can get the data into another archive when you want to kick him out. (Which might turn out to be more tricky than a PACS data migration because VNAs usually can store all kinds of data, not only DICOM.)
And of cause ask your PACS vendor if he can use a VNA as an archive. (And since most PACS vendors nowadays also offer a VNA try to find out if they are doing any tricks between their systems which will make an independed switch difficult if you choose to initially get the PACS and the VNA from the same vendor).
Don't quite understand what VNAs. However, I think we have debated this on another thread.
WHAT NEEDS BE MIGRATED 1. Image Archive---contains images as they were sent from modalities---DICOM format 2. Relational Database of PACS containing the metadata items described below: A. PATIENT DEMOGRAPHICS (synchronized with PAS) a. Name, b. DOB, c. Sex, d. PAS No., e. NHS No. (CHI number for Scotland)—NHS number may not be present in all exams sent to PACS B. REQUESTER—synchronized with RIS a. Name of Requester b. Grade of requester c. Contact number of requester d. Requesting **Responsible Consultant/GP (Team)—(Also RECIPIENT) e. Requesting Speciality/Department/GP surgery f. Requesting Institution g. Date & time of request made C. IMAGE DOCUMENT—synchronized with RIS & modalities a. Modality b. Exam Description---(National Exam Codes & Descriptions) c. Date & Time image acquired on modality d. Date & time of image sent from modality e. Date & time received on PACS f. Exam Room (where the exam has been performed) D. OPERATOR/IMAGE CREATOR –synchronized with RIS & modality a. Name of Operator b. Grade of Operator c. Contact number of Operator d. Performing Responsible Consultant e. Performing Department/Speciality--Radiology f. Performing Institution/NHS Trust E. REPORTER—synchronized with RIS a. Name of Reporter b. Grade of reporter c. Contact number of reporter d. Reporting Responsible Consultant e. Reporting Department/Speciality f. Reporting Institution/NHS Trust g. Date & time report verified F. WORKFLOW data fields a. Workflow status b. Workflow priority G. Study UID & Accession No.
posted on Tuesday, September 13, 2011 - 12:31 pm
Every Trust needs to ask LSP/CFH/Outgoing PACS vendors the following questions & ask for an upfront answer
1. Will read only access to the image storage system be allowed ?
2. In what format are the image files stored, is there any proprietary compression ?
3. Can database dumps be provided & what is the cost per dump ?
4. How are annotations & image flips recorded (private tags or other formats)
This is important for any new PACS vendor to make an assessment of how easy or difficult the migration process will be, and they will come up with indicative costs based on this.
It also will reflect the attitide of your existing vendor (to what extent they wish to tie you in)
posted on Tuesday, September 13, 2011 - 01:50 pm
FYI, as discussed earlier in this thread, I finally got around to putting in an IHE work item proposal for the next cycle for a profile for the intermediate archive export/import concept, which also addresses the annotation/flip (presentation state) issue. See: