Dave Clunie said: "Not to say that it is trivial to come up with such an intermediate form that contains absolutely everything that the recipient might need (reconciled identifiers, corrected/merged/split studies, annotations, measurements, presentation states, etc), in the form that they might need it. "
That is a serious challenge for a simple data dump. Many (most/all?) PACS do not store corrections or merge/splits in the actual data file but only in the database. Indeed one PACS I have been exposed to didn't store *any* headers with the image - all headers were maintained in the database. The DICOM stream is amended (hopefully) when studies are transferred via the DICOM services (like Query/Retrieve). So simply re-importing Part10 files ends up losing all of those interactions. Routing the data back through the server to do that for offline media make the option a little pointless - one may as well use query/Retrieve
_But_ commodity hardware is dirt cheap - a 6TB (RAID5) NAS is only just north of £1000 (if that). Manage the data/corrections with an open source archive and one has an easy stage 2 ("getting it in")
"Nowadays this might be more effectively addresses with an IHE "offline migration" profile, which might be an interesting idea."
Given this is a painful process for anyone and everyone transitioning from one vendor to another - I'd say certainly a good idea.
Yes, that was exactly my point ... the central store (or any PACS that supports DICOM export for migration) SHOULD have a mechanism for (quickly) populating an external archive with "correct" DICOM objects ("Part 10 files") that DO have the corrections from the database folded in; there is absolutely no point in serving up stale or incorrect stuff.
After all, as you say, they have to do this for CD export and responses to DICOM and XDS queries anyway (or should; this use to be one of the biggest safety issues with the early 3.x Centricity versions with the support for PGP added; the C-FIND and CD export stuff was "wrong"); so the logic is there somewhere inside, even if it is not efficiently implemented for bulk transfer.
I am suggesting that if the central store or PACS doesn't natively support this, or do it fast enough, then a third party solution installed at the central site that can grab the database and replicate such behavior (create a collection of "correct" DICOM "Part 10 files").
That way Q/R over the WAN (slow) is not required (or Q/R at all, if the 3rd party solution can extract from the database and proprietary/stale image archive faster). Then the extraction can be done at the central site, the disks carried to the recipient, and a bulk data import of "correct" DICOM data done. I dare say the NHS will require such bulk data transfer disks be encrypted, but that can be done in a manner that shouldn't slow reading and writing too much.
Hi David & Martin, Whilst I struggle with the technical details, I do agree with David about what is being said. Every DICOM or PACS archive should be capable of bulk transfer of DICOM data using DICOM part 10 standard --which also allows for CD creation. (is my understanding correct David?)
Unwillingnessness to handover data by PACS suppliers/LSPs could be attributed to 2 reasons 1. Lack of DICOM part 10 support 2. Unwillingness to handover data to customer @ end of contract
What you are saying David, requires a change in mindset in NHS. We should be expecting LSPs to handover data at the end of contract using bulk transfer using DICOM part 10 standard.
Perhaps we in the NHS should be asking LSP and CFH--" how are you going to hand over/release the data at the end of contract in June 2013 using DICOM part 10 standard and using bulk transfer". This actually moves the responsibility of migration from the customer/new vendor to the outgoing vendor.
Looking ahead into 2023. For our PACS specification for 2013 should include "At the end of contract vendor MUST enable bulk migration of data using DICOM part10 standard" David, is this worded correctly.
In essence, yes, this is like CD creation, but there is a huge difference in scale between creating one patient study CD as opposed to doing it for thousands or hundreds of thousands of patients and studies.
The key additions that would need to be in the requirements would in my opinion be something like (my changes and reordering highlighted):
"how are you going to hand over/release ALL OF the data (INCLUDING MERGED DATABASE CORRECTIONS AND ANNOTATIONS) at the end of contract in June 2013 using ONLY BULK TRANSFER ON REMOVABLE HARD DRIVES of STANDARD DICOM part 10 files, AND HOW LONG WILL IT TAKE FROM REQUEST TO DELIVERY"
or something like that.
This strategy presupposes of course that a) DICOM is sufficient to encode everything that needs to be migrated (and things like study read status may be a problem in this regard), and b) that the recipient (new PACS) is able to ingest this form (quickly).
PS. Also, the creation of a DICOMDIR to index all the files may be superfluous and/or impractical (too big), even though it is a requirement for strictly compliant media, though it does serve as a "manifest" of what is supposed to be on the disk.
posted on Sunday, July 24, 2011 - 01:15 pm
I think it is important to bring a sense of reality to this discussion on data migration. We at Purple Fish have carried out a lot of data migrations from lot of different sources. We work for both the Trusts, with their data and also for the PACS companies, providing consultancy and the migration service. In order to provide this service we need to understand the state of the data to be migrated, the study volumes and where it is held. We always base the data on the number of studies as this is the critical item needed for speed and accuracy. This includes the necessary DICOM standards in place by the source archive. We must remember that this information has real value and cannot be corrupted or lost. 3 methods of migration are commonly used by us to perform the task of moving the patient data to a new location, update accordingly, add XDS/XDSi tags and ensure that is correct with the correct patient.
DICOM Transfer Our experience is that most hold certain tags and other items in different data-bases and only when a DICOM QRU is performed these items are updated but not always. This is where having good software enables these things to be picked up at an early stage. At this stage the data can be ‘cleaned’ and enabled for the new location and/or PACS provider. XDS/XDSi tags can be created as the data passes through. This is the best way to ensure that the data has been transferred correctly and also see what has been missed in the DICOM structure. Up-to 50,000 studies can be transferred providing the source PACS has the necessary capacity to allow this action. Most we have worked with do not and will fail if this volume of data is pushed though. The software we use is sensitive to the source system and ensures that this should not happen.
Bulk Data Transfer This can be carried out providing the source provider gives the necessary information through their data-bases and related links. The skill is ensuring the data is correctly synchronised with the data-base. On a production environment this has many risks. DICOM part 10 enables this process to be easier but most PACS vendors that we have worked with do not support this in their archive. High volumes can be migrated but it does require the source vendor to give the necessary data-bases.
Media Transfer For tape and disk we use a third party to perform this task as it is highly specialised skill requiring hardware and software to suit. This requires the media to be removed to a workshop for action and a data-base provided by the source provider. The data can then be mapped and presented back as a DICOM object through a temporary server and storage which will placed in the DC to transfer the data back. We have seen up-to 15,000 studies transferred from DLT and over 25,000 from LTO.
Where do you start? We recommend that an assessment should be carried out. It takes us about 2 days to provide the necessary information on the data quality and also the hardware performance of the source system. We place a server on site to perform this function. From this we know every patient on the system, a rough guide to how long the migration should take and what resources will be necessary to perform the clean-up if required. Just looking at the source system will not give you in in-depth knowledge required before you start.
Where is the data going? We have 3 areas where we send data. First to a new PACS, changing the tags if required. Second into a holding migration server and storage so migration can start to take place now. Third to a Vendor Neutral Archive, which is being used extensively in the USA. This allows all the data to be managed centrally linking all the ‘oligies and provides best of breed applications to be used by the users. If you want any more information please contact me. I have placed a white paper on this message about data migration from Acuo Technologies who are the largest migration and VNA providers in the industry. Chris Bull email@example.com
I think it is important that when we specify for that our next PACS vendor we contract with in 2013 has a clear requirement to "HANDOVER" data at the end of their contract in a standard format, so that it can be ingested by the new incumbent supplier. So we actually move responsibilty of data migration from customer/ incumbent PACS vendor to the outgoing vendor.
What is emerging is that we need bulk transfer of data is essential for migration. Simple DICOM transfer is not suitable for end of contract migration of data.
There are 2 types of archives 1. Offsite or "Cloud"--connected by slow network WAN like N3. LSP archive is an example. This requires bulk transfer using media. LSPs should handover data on media in DICOM part 10 format, for the new vendor to ingest. 2. Onsite Archive-- within NHS Trusts---is there any bulk transfer standards to allow LSP/PACS suppliers to handover DICOM data in a standard format for ingestion into new PACS over ethernet/LAN? David, any help on this would be appreciated.
The real question is about whether LSP-CFH contract has a requirement for them to "handover" Trusts data( whether onsite or offsite) in a standard format or not....
Thanks Chris for summarising the Migration methodologies from a migration vendors perspective.
The theme you highlight is lack of DICOM part 10 support by majority of PACS vendors. Hence, a middleware migration software vendor is required to move data from 1 PACS to another.
The vendors in question in 2013 are Agfa & GE. Sectra will be in 2015. Can anyone tell me whether these 3 vendors suppliers support DICOM part 10 within their archive or not. This maybe be a deciding factor for Trusts who maybe looking at them for new/replacement contracts?
Personally, I think these 3 vendors are large vendors and I find it difficult to believe that they will obstruct the migration of data.
I can agree with Chris’ assessment from the point of view of another vendor of PACS Migration Services. Our staff’s background stems predominantly from Mitra broker days. To build on Chris’ comments I can suggest that there are basic means to take stock of your set up, for example, there are a variety of PACS Migration Surveys. I am happy to share the one we use with anyone that is interested.
The best possible position is to test the migration process because you never truly know all of the obstacles. That being said, it isn’t essential. It can reduce unpleasant surprises for all concerned, but it isn’t full proof. Please bear in mind that it takes a similar effort to set up testing as it does to carry out the migration process simply because you are pulling all of the same levers.
Neelam with a bulk transfer technique, this implies moving data to an intermediary if I understood the intended meaning correctly. So, once you have your data in 1 spot, there are 2 common choices to proceed to the target that I am aware of: 1. Use DICOM Commands 2. Get the Target PACS to open up a share – ideally you arrange to have a View of the source & target PACS’ schema – this can be a business level discussion to provide assurances to source and target vendors
Please excuse the commercial tone, my intention is to help out the discussion as industry category Cheers Alan McLaren MDI Solutions
I welcome all your comments as vendors. I welcome Davids comments as a standards expert. I am grateful for all your comments.
"Bulk transfer techniques means transferring data to an intermediary" I actually mean a quicker transfer rates -1/2 TB per day. I do not understand the technology behind this but what I do understand is that traditional DICOM push/QR is ineffective for large volume data migrations-- from listening to experiances.
My simple question is "why do we need intermediaries?" The answer I seem to be getting 1. PACS suppliers are unwilling to handover customer data in a gentlemanly fashion, so we need experts who can extract the data out..... 2. Data held in PACS archive is not held in standard file format--- PACS archive is not DICOM part 10 compliant.
Most of us use Agfa Impax 6.2 here. Good news is that does support DICOM part 10 as it support PDI profile.
Similar search maybe done for GE or Sectra or others.
My question is that if Accenture/ Agfa were to "handover" the data in DICOM part 10 file format do we really need an intermediary. Why would the incoming PACS supplier not be able to simply ingest the data supplied by the outgoing PACS supplier?
But being practical, I think 1. For 2013 get migration experts involved 2. For 2023 make sure there is a requirement for your supplier to "handover" data in a "standard" format.
PDI is NOT the answer for bulk data migration, only for individual patient and study data.
As I stated earlier, exporting large numbers of studies is a completely different problem, and claims of PDI support are no help at all in judging whether a PACS can export large volumes in a standard form at a reasonable speed.
There is no IHE profile for bulk data export (yet), but we could work on one during the next IHE cycle if there is enough support for doing it (i.e., workers as well as votes on the planning committee and priority over other tasks).
I do agree with your final summary though ... our objective should be essentially to put the migration experts and service providers out of business for the next round, by having all new PACS purchases support something standard natively. That said, I very much doubt that the migration service vendors feel threatened by this prospect, given the glacial pace of PACS feature deployment and the market forces that drive them.
1. Does anyone know what process LSPs are going to adopt to " handover" data or "allow" extraction of data from CDS or local archives. HOW 2. What rate of data extraction will be allowed? RATE 3. How long do they think will be required? DURATION 4. What will be the total costs from their side? COST 5. How will cost be calculated? Total cost for the process or PER DAY rate?
It is really interesting to hear how Data Migration will really work.
There is a lack of clarity about 1. what METHODOLOGY will be "allowed" to migrate the data out from LSP PACS (whether CDS or local PACS archive). Will it be a "snail rate" or "bulk rate". From a LSP business point of view--snail rate is better--longer the duration--more money to LSPs (more money drain from NHS) 2. What will be the DURATION for the entire data to be transferred? If a Trust was planning to move to a new PACS on 1st August 2013, when do they need to START transferring data to new PACS so that transfer is complete on 1st August 2013. Longer duration more money to LSPs from business perspective. But longer the duration---Longer duration for NHS 2 have to pay for 2 PACS (old one & new one) 3. COST from LSPs to allow data migration.
I think we are in a mess in the NHS with these LSP PACS contracts relating to data migration!!!!
However, lessons to be learnt here. Choose a PACS vendor who is upfront about discussions on data migration at the end of contract (even before you enter into discussions with them)!
If a Trust wants to move to a new PACS vendor at the June 2013, then it really needs to know how long the migration is going take.
If LSP says it will take 6months then Trusts needs to have a contract commencement date of Jan 2013 with new vendor---so that data can be migrated into the new PACS archive. Without clarity on FROM WHEN will data migration be allowed & HOW LONG will it take these decisions are very difficult.
posted on Monday, August 01, 2011 - 09:09 am
As I had said in my last posting the 2 types of migration have different issues and without a full assessment the best way to migrate the data could not be determined. The DICOM route can start at once as this has no need for any database transfers etc. from the source vendor just an AE and IP address. Again then need to store this information while waiting for the new PACS can be made to a temporary store or VNA. This can be in place from today as it is the easiest way to migrate the data off the current source and safest way as this is what the outside world would see using DICOM QRU. The key here is to ensure that there is little impact on the current source servers. As there is no delay in starting this, it can be faster overall. We also carryout direct media migration but the need to have good co-operation from the source vendor is required. Set-up time is the problem and data testing is essential to ensure that all the information is correctly transferred as on some systems this can be across several databases.
I agree with Chris: 2 routes available to customers: 1. DICOM route--DICOM push to a temporary store until a decision about new PACS vendor is made "Again then need to store this information while waiting for the new PACS can be made to a temporary store or VNA. " Temporary store is often called VNA. a. Tried & tested but slow b. Based on standards so your PACS supplier can refuse c. Careful: Ensure VNA is connected to PAS/PMI (? RIS) to ensure VNA is kept up to date with regards to demographics etc. d. Careful:Although cost of storage in VNA is not as much as PACS, it still will cost you maintaing duplicate storage in both VNA & old PACS until new PACS vendor is in place
2. Bulk migration a. From old PACS vendor to new PACS vendor directly b. Fast- 1-2 TB/day c. Careful: will need good co-operation from accenture/CSC/BT and Agfa/GE/Sectra
Whilst CFH claim that LSP are required to release data at the end of contract, let us see what will be allowed. I understand negotiations are on-going.
I do think lessons have been learnt. At the end of next contract PACS suppliers must be willing to handover data in a gentlemanly manner. I personally think it should be possible to transfer data from one PACS to another and data transfer for 7 years worth of data should not take more than 2 weeks. However, this must be supported by global standards.
Having read most of the posts on this thread and other similar ones, please forgive me if I'm repeating anything that I've missed but I haven't seen anything else suggesting this as a possible way forward. Having been a Project Manager for implementation of a large NHS Trust in the Southern Cluster and then as PACS Manager during the transition from one LSP to another I've seen from the inside some of the issues relating to the current system (in the Southern Cluster) and would like to suggest a possible way forward for those with limited resources in terms of time, money etc. 1. Ensure during the process there is enough local RAID in-place and configured not to have to ditch data on the local store on the current FIFO (or other sub-routine) basis. 2. Seek a copy to external device made at the data centre of everything on the secondary store as per when the data centre moved site. 3. Likewise, arrange with the RIS provider / LSP for a copy of the RIS database relating to your Trust (not the entire SHA). 4. Negotiate a direct contract with PACS and RIS providers to install a local RIS database and PACS secondary store on your own site(s) with suitable provision i.e. in different data centre to your primary store and perhaps with RIS failover. 5. Your PACS central hardware (servers) and distributed hardware (workstations) will already have been refreshed so you'd need to negotiate a buy-back from the LSP if you want to keep using them. 6. You may also wish to incorporate software upgrades in the contracts.
This will give you a stand-alone system that should at the very worst be no more expensive than you're currently paying but should, looking at current costs be much cheaper bearing in-mind management (LSP) overheads that will be removed and costs for refresh likewise. You should be able to continue as-is until some other Trusts have migrated and the complexities have been worked out. Expertise will have been developed to make the process smoother and give you more choice then as to options on how to change and if / who to change to.
I make no apologies for suggesting to people they effectively sit on the fence and allow others to take a more painful route in migrating to another provider. I think it would be impossible for all Trusts to change at the same time within a Cluster in terms of existing LSP and sub-contractors support as well as other PACS companies to install replacements at the same time. After all, the CfH program took place over 3-4years of install.
If people are worried about not having the latest and greatest software / options then my response is simply this: Better to have a reliable, working system that everyone knows how to use than move quickly to a problematic short term future that brings in the possibility of instability in an essential clinical system.
I also accept that going it alone as above means lack of data sharing. On the PACS front this has never happened as originally intended so nothing lost there. On RIS, you may not want to see loss of sharing across an SHA. In which case the above scenario could potentially be done SHA wide. However, I'd put money on their being disagreement in every SHA about which way to go and that each and every SHA would have at least one Trust not wanting to do the same as all the others.
I've had is suggested to me that some companies currently working as sub-contractors to the LSP won't support Trust by Trust (or SHA) contracting. I would disagree with this viewpoint. These are commercial bodies seeking to maintain or grow market share and their profits. Faced with loss of business I'm sure that an agreement could be reached.
So, I put it to you merely as a suggestion and look forward to comments. I have a parachute at the ready for when I'm shot down in replies!
The critical unknown here seems to be any concrete understanding/agreement on what the LSPs are prepared/obliged to support in terms of data transfer. I think the most useful next step is to get all LSP customers to "gang up" on the LSPs or whoever is negotiating with them on your behalf and get this defined/agreed. Without this you are working in the dark and while most of the schemes outlined here will work just fine you are at risk of over-complicating, over-spending or working within assumed constraints which may not be realised. Once this is clear there will be many options to actually accomplish the transfer and you can cut these according to your needs - possibly even including it in your tender documents for new suppliers so they can take the headache.
To pick up on Neelam's earlier point "2. Data held in PACS archive is not held in standard file format--- PACS archive is not DICOM part 10 compliant."
You will probably not learn much from DICOM compliance statements or connectathons about this. These concentrate on the "edges" of the PACS system. At these interfaces, the data is indeed wrappeed up in DICOM wrappers/formats - that is the basis of compliance. However within the database the vendor may choose to store the data in a different format (for example to speed up image access and rendering onto the screen). In this case the export process becomes one of taking the "native" format and re-wrapping it in the DICOM format before it can be passed to another system. It is this process which limits the rate at which you can transfer data to a new archive, not the speed of disks/networks etc. If your vendor has not written the routines to do this conversion with speed in mind it will run slowly and take much longer to extract the whole database.
That's where the value of the integrators/migration experts comes in - they add value (and hance can charge money!) by accelerating this process and reducing the load on you live PACS at the same time.
All that said, some vendors DO store full DICOM files in their database. In this case it will be faster and easier to migrate - any you may want to consider this in any new procurement you undertake to future proof your next PACS.
In summary, you _can_ work around your existing vendor by using things like Query/Retrieve - but you need to exercise caution around speed and load on a live PACS. Best to at least talk to the PACS vendor inc ase they can help!