UK Imaging Informatics Group
Testing a VNA for vendor neutrality PreviousNext
UK Imaging Informatics Group > Questions & Answers > PACS Procurement & Business Cases >
Message/Author
 Link to this message Neelam Dugar  posted on Saturday, March 17, 2012 - 09:05 am Edit Post Delete Post Print Post
I am going to throw the cat amongst the pigeons here--and perhaps upset both VNA & PACS vendors who read the forum.

Rather than take a vendor's word that they are vendor neutral, is there are a means of objectively testing their vendor neutrality.

1. Sending multiformat documents (if it works for a radiology image & radiology report--I think we can safely say it will work for any medical document/images) to the VNA from a 3rd party independent XDS source.
2. Retrieve documents/images using a 3rd party independent XDS/XDS-I consumer-Portal type concept.
3. Migration of the data from VNA to a 3rd party independent VNA---thus testing that it works & also find the time taken for a volume of data.

Is there any independent company that specialises in this type of independent testing & will work on behalf of the customer?

Can this testing be performed prior to signing of contracts, so that customers can ensure that they are indeed have ownership of the data in the VNA? Or would that be against the rules of procurement?
 Link to this message Jeremy Bikman  posted on Saturday, March 17, 2012 - 09:20 pm Edit Post Delete Post Print Post
Neelam, you are definitely asking the right questions. Some independent testing would be highly beneficial; however, I don't know of any firm that isn't already married at the hip to a vendor that could provide such a service.

VNA suppliers like Acuo, Teramedica, InsiteOne (Dell) are most likely more truly neutral than are the PACS vendors (for obvious reasons), but a few of them like Merge (through its acquisition of Emageon via Amicas) are probably pretty close.

Yet as you posited, until there is actual testing done there is no way to be 100 percent sure.

What I'd also want to know, if I were a hospital or imaging group, is how many clients of any VNA vendor are actually doing non-DICOM image storage and management and the volumes in this area.

Jeremy Bikman
jbikman@katalus.com
 Link to this message Chris Bull  posted on Saturday, March 17, 2012 - 10:22 pm Edit Post Delete Post Print Post
We are working with a few Trusts in scoping their requirements for RIS/PACS and data sharing and VNAs etc. The data sharing project includes the promotion and use of XDS/XDSi etc.
One of the recommendations we make is to run a week-long trial of some of the shortlisted suppliers under test conditions. The Newcastle team put together a defined test that would solve their problems with connectivity and data sharing. The vendor would have to prove that they could integrate using the real problem before they would be considered. This was 9 years ago!!!!
We will be writing the protocols for these tests for different Trusts to suit. I do think this is important if you are going to sign a contract with a supplier 5 to 7 years.
Highlight what problems you want this to solve and write the test plans and get the shortlisted vendors to prove they can do the job. This should be scored to suit.
If you need help get in contact.
 Link to this message Neelam Dugar  posted on Sunday, March 18, 2012 - 09:43 pm Edit Post Delete Post Print Post
Thanks Jeremy.I agree with the points you make. Maybe there is a business opportunity for someone here. There is no IHE migration profile .
"What I'd also want to know, if I were a hospital or imaging group, is how many clients of any VNA vendor are actually doing non-DICOM image storage and management and the volumes in this area"
XDS Registry/Repository within a VNA is something relatively new in my view (previously VNA vendors converted doc into DICOM images to store them). Hence, any customer buying a VNA to store medical doc MUST ensure that XDS Registry/Repostory is built in within the VNA.

Chris,
Most NHS Trusts/customers dont have the technical expertise in DICOM/IHE to evaluate VNA vendors through an elaborate testing process.

By the time the final VNA supplier is chosen--as part of the procurement process--- supplier would have already declared
1. They are an XDS Registry & repository --as per their IHE statement (and hence will allow any XDS source to send documents to it & also will allow any XDS consumer to display XDS doc/images within the repository)
2. They would have declared that they are going to allow the migration of data at the end of contract--at no additional cost & tools for migration will be in the hands of the customer.

However, before contracts are signed what should happen is that an independent evaluator chosen by the customer come in to test the assertions:
1. Act as an XDS source--for both images & documents
2. Act as an XDS consumer--for both images & documents
3. Act as a DICOM archive for data to be migrated into--to test the migration ability & speed of the VNA.
I cannot see why a "simple VNA Test Package" cannot be developed. I think this will be key to proof of VNA & XDS workability.

Customers should ask for testing of vendor neutrality before they sign contracts in my view.
 Link to this message Jamie Clifton  posted on Monday, March 19, 2012 - 02:58 pm Edit Post Delete Post Print Post
Neelam,
It is a good question (getting vendors to prove interoperability) and I am sure we would all be willing to be tested but my question would be arenít we already doing this with the IHE Connectathonís?

I understand that trusts donít have infinite resources to test solutions but donít the Connectathonís at least give a starting point to gauge if a vendor has demonstrated the capabilities required. We are spending time and effort to attend these events and I would be interested to know if anyone has used the results from a Connectathon as part of their appraisal?

Clearly there is one major problem and that is there isnít an IHE profile for VNA.
 Link to this message David Clunie  posted on Monday, March 19, 2012 - 04:06 pm Edit Post Delete Post Print Post
Jamie wrote "there isn't an IHE profile for VNA".

True, but there isn't an IHE profile for a "PACS" or a "RIS" either.

IHE defines profiles that address specific use-cases that involve specific actors (with funky generic names), some of which are grouped.

Thus, to the extent that a PACS or a VNA (whatever that is, or whatever the difference is between the two) matches (or more likely contains the functionality of) an actor, or a group of actors, then some profiles may be applicable, e.g., Scheduled Workflow (SWF) or Patient Information Reconciliation (PIR) for an Image Manager/Archive (IM/IA), or Multiple Image Manager Addition (MIMA), or XDS-I.b for an Imaging Document Source (IDS), etc. A PACS or VNA will likely need IRWF too, maybe ED, REM, SINR, NM Image, Mammo Image, CPI, KIN, PDI, ATNA, TCE and maybe even BIR. And that is before even starting on cardiology, etc.

I.e., both a PACS and an "VNA" might contain full IM/IA or IDS functionality wrt. any particular profile, or not. One question that often arises as to whether the IM and IA should be split, and if so what each Actor would be and what transactions would need to be defined between them.

The IHE approach of not being tied to a specific product type definition avoids the issue of nebulous definitions of such products.

For example, some folks seem to have convinced themselves that a "true" "VNA" now includes support of non-DICOM documents (despite the fact that the issue here started out as being primarily a PACS replacement exercise that may now have morphed into a enterprise archive acquisition, which is fine, as long as everyone understands that feature creep and generality add cost and may compromise efficiency and workflow); this requires additional actor/profile involvement (e.g., XDS.b, not just XDS-I.b, presupposing that XDS.b functionality is sufficient for the workflow supporting non-DICOM document handling). Not to mention some sort of reporting workflow if the non-DICOM object being considered is a report (and IHE reporting workflow is a bit of a mess), and you are planning on replacing your RIS too.

So, to the extent that whatever it is you actually want/need (whatever you call "it") has interface boundaries and capabilities that you can tie to IHE actors and profiles, then you should, and to the extent that there are no such actors or profiles identified, then you are on your own to define that functionality (or to extend IHE if you can be convincing enough and can round up the technical expertise), and to find a forum in which to test it.

Obvious gaps that we have talked about before include the absence of an IHE "migration" profile to address the transfer of DICOM images in bulk between two archives (which VNA advocates may argue is unnecessary because they are the ultimate immortal archive that will never fail to satisfy the customer or go out of business). There is also no IHE generic "tag morphing" profile.

So, in summary, you won't be seeing an IHE VNA "profile" per se, nor testing of it at a Connectathon, but to the extent that you can fully define the functionality that you want in terms of IHE profiles and be satisfied that Connectathon testing of these is adequate to convince you of its efficacy, then for any additional functionality, you are "on your own" to define and test it. Further, to the extent that you imagine a "meta workflow" that combines various different IHE profiles and your own customizations, you are on your own to define and test that too.

Or to put it another way, a buzzword compliant checklist of features is not going to guarantee you the functionality you need, especially if it involves integration of multiple systems from multiple vendors (e.g., n x PACS + VNA) as opposed to just buying a monolithic system that you just have to integrate with your neighbours for referrals and priors.

David

PS. Personally, if I were drinking the VNA Kool-Aid, I would want to inspect a site that had the VNA installed with the other things (like various PACS) that I was layering on top of it, and inspect functionality and performance in a comparable working environment to my own (the proof of the pudding being in the eating). From that experience, I would define my own (focused, risk-based) acceptance testing, with a strong emphasis on performance (something that is not measured at Connectathons), and contractual obligations on the vendors to mitigate deficiencies. Hypothetical compliance and interoperability based on standards being necessary but not sufficient. Bearing in mind of course that you can have only two (or maybe only one) of cheap, fast or good, and that a contract is not a guarantee.
 Link to this message Neelam Dugar  posted on Monday, March 19, 2012 - 10:29 pm Edit Post Delete Post Print Post
I knew I was about to cause some controversy here & I did :-).

1. I take Jamie's point that if VNA is XDS-b/XDS-Ib registry & repository complaint as per IHE statement, does a customer really need to test it. I personally think, customers should test their claims & also test performance--by sending images & documents from XDS source & reviewing it via a consumer. However, we do need someone to provide a customer with a XDS test package.....(anyone out there?)
2. Bulk DICOM Migration Profile(with tag-morphing included)--in my view this is standard which is lacking. I was really disappointed David that IHE members with voting rights did not support your proposal.

I agree that PACS & VNA do NOT have a standard definition. However, I think in IHE terms PACS is largely Image Archive/Image Manager & Image Display actors.

VNA is an evolving concept--which started off by addressing the issues of DICOM migration. However, I do believe that it has the potential to bring non-DICOM to the forefront with XDS indexing.

I really dont have an issue whether a VNA is supplied by a PACS supplier or not--as along as the customer knows what they wish from a VNA & the VNA delivers it.

My issue is that there is no independent package/product which a customer can use to test a VNA--both for speed, performance & also vendor neutrality.
 Link to this message Chris Bull  posted on Monday, March 19, 2012 - 11:27 pm Edit Post Delete Post Print Post
I agree with all the comments and concerns
Yes meeting standards is crucial and IHE connectathons do prove that software products are able to connect. However the local environment, the need for cross boundary working via integration to other Healthcare systems to allow for data sharing when needed and the ability to add any 'ology should also be included.
The two questions that should be asked in any assessments of new solutions are these -
How will this benefit the patient? and How will this make us more productive?
For Trust staff the delivery of high quality patient care must come first. Any new solutions will need to deliver the right functionality for patient care by having accurate and complete patient data, the right security and compliance with standards and NHS guidelines. We are generating enough information on workflow and systems to enable this to be input into the testing for the Trusts that we are engaged with.
Testing and end to end solution and what vendors claim will provide Trusts with some confidence before signing up for a 5 /7 year deal. This is risk mitigation as any issues can be identified prior to contracts being signed which surely is worth doing?
We did this 8 years ago when the Newcastle deal was made while I was working for my last company. They had 4 PACS system from different vendors that could not share data this was fixed in 3 hours using a true VNA then on to a DICOM workstation from any of those vendors. That was just DICOM then but I know the same problem exists today.
We,as a consultancy are being asked as an independent team to help in defining this process and the methods to test such a programme. Running a defined test within the Trusts environment will prove the complete range of the offered solution and allow true data sharing using the IHE protocols required before any contract is signed.
 Link to this message John Parker  posted on Tuesday, March 20, 2012 - 09:19 am Edit Post Delete Post Print Post
At the risk of upsetting a number of people (apologies in advance)......

My understanding of Connectathon's is that suppliers turn up with something they have developed, and see if they can talk/ communicate with a bunch of other people, who are there for the same thing.

It doesn't mean:
1. a suppliers current products will then be developed to adhere to profile X
2. any/some/all future products will adhere to profile X
3. mandating a whole load of IHE profiles will produce a effective VNA/PACS/ anything else....

I'm (absolutely)not trying to rubbish what happens at these events, but point out that the pointy end of these events is some way from the safe, risk assured systems we all need and want.

Perhaps the answer will be in specifying what the VNA will do in operational terms, because we are experts in this area, rather than in technical terms that few of us really understand...

How many VNA vendors would be willing to stand up and say they will take any data from any system, safely maintain and update (synch)that data, and serve it up in a timely way, to any requesting system, in a timely fashion(3 seconds anyone??)
-obviously this would need to be developed....

John
 Link to this message David Clunie  posted on Tuesday, March 20, 2012 - 12:50 pm Edit Post Delete Post Print Post
Neelam, I don't think that you are going to find an out of the box "independent package or product" to "test" everything you want; as Chris points out, some expertise is required to do this sort of thing, and if you don't choose to avail yourself of such expertise, then you are relying to some extent on "faith" in the various vendors (and contracts).

I would ask some more questions of you:

1. What "tests" did your organization do on the last PACS you procured, and if any, did they help ?

2. Which of the VNA products that you are considering have tested at IHE Connectathons, and what existing profiles did they test (bearing in mind John's caveats)?

3. What are your performance targets for your next system?

4. Where is all this non-DICOM content that you are concerned about coming from?

David
 Link to this message Neelam Dugar  posted on Wednesday, March 21, 2012 - 07:52 am Edit Post Delete Post Print Post
I would ask some more questions of you:

1. What "tests" did your organization do on the last PACS you procured, and if any, did they help ?
ND:Like most NHS Trusts we were told to take the centrally procured PACS. We just had to get on with it. So this whole concept is a new one for most of us.

2. Which of the VNA products that you are considering have tested at IHE Connectathons, and what existing profiles did they test (bearing in mind John's caveats)?
ND: i do not know. Does anyone have a list?

3. What are your performance targets for your next system?
ND:. Click to display-3secs for both DICOM & non-DICOM images & doc
1 TB migration within ----time (dont know what this should be)

4. Where is all this non-DICOM content that you are concerned about coming from?
ND: RIS for reports
Ordercomms for radiology requests
This will be the start.
We have already created a metadata set for these.


By "test package" I meant including the human technical expertise (not simply a software package) :-)
 Link to this message Nick Hollings  posted on Thursday, March 22, 2012 - 12:42 pm Edit Post Delete Post Print Post
A bit of S Cluster news - for amusement if nothing else.
we were on the point of acquiring a VNA to manage our end of contract PACS transition when we were informed by CfH that we were NOT allowed to retrieve our CDS data over N3. (Only Trusts with >5Tb are allowed to do this, apparently). A large neighbouring Trust attempted to do this anyway and got told off!
At the same time (conveniently) CfH & CSC announced the charge for transferring all our data in a van on hard drives had dropped from over £100k to £65k. Would we still want to procure a VNA knowing this? May be not (although there are other good reasons for having a VNA as others have posted.) The final twist in the tail is an eleventh hour note from CfH saying that they can only guarantee that the CDS HDD data will run on a GE PACS platform, thus committing us, in theory, to choosing said provider again! GE strongly deny this.
You couldn't make it up...
 Link to this message Nick Hollings  posted on Thursday, March 22, 2012 - 12:44 pm Edit Post Delete Post Print Post
sorry, that should be <5Tb data for N3 transfer.
 Link to this message Martin Peacock  posted on Thursday, March 22, 2012 - 02:15 pm Edit Post Delete Post Print Post
Without knowing the specifics of the GE storage there is a strong probability that it is at least hardware dependent - the data is likely RAIDed across disks so compatible controllers would be required. There is also metadata in the database that would need to be synchronised (as discussed in a different thread). So it is more than likely simple physical access to the disks is not so useful. Unless of course they are exporting the data as DICOM Part10 onto more portable disks (as discussed elsewhere also).

There are of course tools available to 'test' much of the same functionality as IHE Connectathons. It is possible to verify conformance with published standards/conformance statements. One could also take existing systems and test interop between those and a new vendor - and the likes of Chris (or indeed myself or many others) could run those through the wringer but..

1. There isn't an accepted definition of 'VNA' at this point, so 'Certified Neutral' is fairly meaningless.
2. Even then, the ambiguities in the definition(s) are more about what it is you the end user want to see - not the technical implementation. The technical side is what is tested in a Connectathon anyway. (although as John points out - what is tested in the latest Connectathon is NOT what is in the latest stable product)
3. You can only test against existing systems - not those that may form a part of the wider IT infrastructure 2-4-6-8 years down the motorway*.
4. It is likely at least some of those tests will require a fully complete implementation, with data migration complete and storage tiers balanced to specification. Once you are in a position to do that then neither buyer nor seller are going to be too keen about backing out of the contract - success or fail.

John suggests "Perhaps the answer will be in specifying what the VNA will do in operational terms". I am a believer in the notion of Output Based Specifications (OBS) for precisely that reason - you define what you want something to do rather than how you expect it to do it.



* Apologies to Tom Robinson**
** Further apologies to anyone not from the UK or under the age of 40.
 Link to this message Neelam Dugar  posted on Thursday, March 22, 2012 - 10:10 pm Edit Post Delete Post Print Post
Martin,

VNA Assertions about vendor neutrality:
1. "VNA ensures DICOM Data Migration at end of contract is in the hands of the customer"--Is DICOM part 10 adoption in VNA archive sufficient or does this need to be tested by migration test to DCM4CHEE?. Any views......
2. "Supports storage & display of non-DICOM images & doc"--Is XDS/XDS-I registry & repository adoption by VNA sufficient--or does the customer need to test this by having an XDS source & consumer from an independent provider to test this assertion?

Comments please.
 Link to this message Martin Peacock  posted on Monday, March 26, 2012 - 05:47 pm Edit Post Delete Post Print Post
Neelam.. I think there has been so much discussion about VNAs and the standards most appropriate for them to support that sight of the important principles has been lost a little.

<Opinion>
From the perspective of data migration, a VNA vendor should adhere to two basic principles:

  • The process of data migration should be clearly documented and anticipated at the start of the contract.
  • The tools, knowledge and skills required to do so should be included in the package and/or straightforward to come by on the open market.


By far the easiest way (but not the only way) to achieve the second of those would be to adhere to *some* of the relevant standards. So the question is, which standard? In the absence of an IHE Data Migration Profile (more on that below), there are 3 alternatives - each with pros and cons, and none of which were designed for this purpose (hence the need for a DM Profile):

  • DICOM Query/Retrieve. Carries the advantage that metadata changes in the database are re-incorporated into the data stream, but is typically implemented *not* with bulk transfer in mind, and the practicalities of managing the process can get untidy.
  • DICOM Part 10. Either as the native archive data format or as an export option. Conceptually the simplest option, and easiest to manage.
  • XDS. Designed for X-Enterprise sharing rather than data migration. It does come with the advantage that complicating factors such as unclean Patient IDs are addressed but requires first that the Enterprise as a whole has the PIX profile fully in place. Also it requires the assumption that the XDS facility is to include all images which may not necessarily be true.


Any of those *could* do the job, and with proper tools and skills (either in-house or 3rd party) each would probably do as good a job as the others. But knowing at the start of the contract which is to be planned for is probably the most important thing - and as you point out - to ensure that all of the pieces are in place, it would absolutely be wise to test the process with a dataset which reflects as much as possible your own environment. You probably can't predict precisely what modalities/vendors you will have in 7 or 10 years time but a broad sweep should be possible. Yes, DCM4CHEE would be an ideal testbed for just that but probably not in isolation. The test plan should also include whichever methodology is proposed to minimise/qualify effect of normal day-to-day operation during a migration process.

You could include performance metrics in the testing, but that would be largely irrelevant. at the end of a 7-10 year contract, you really should be running on different hardware to the start of the contract so any performance targets achievable at the SoC should really be trivial at EoC. And the test platform itself may well be entirely different again.

On XDS testing - there are limited advantages. All that could really be done is replicate the testing done in an IHE Connectathon. Part of due diligence should be to get vendor to explicitly state which Connectathon this version of the product participated in, and claims can be independently verified.

A thought on the IHE DM profile - clearly IHE generally within NHS is of strategic importance, and particularly w.r.t. the Radiology process over the next few years. Does NHS have an appropriate number of voting members?

A question for Nick.. is it an option to get LSP to allow 3rd party temporary access to the datacentre to transfer via sneakertransit-net on your behalf?

</Opinion>
 Link to this message Neelam Dugar  posted on Tuesday, March 27, 2012 - 11:18 pm Edit Post Delete Post Print Post
Thanks Martin. This is really helpful. These are some crucial points:
"1. The process of data migration should be clearly documented and anticipated at the start of the contract.
2. The tools, knowledge and skills required to do so should be included in the package and/or straightforward to come by on the open market
3. DICOM Part 10. Either as the native archive data format or as an export option. Conceptually the simplest option, and easiest to manage
4. You could include performance metrics in the testing, but that would be largely irrelevant. at the end of a 7-10 year contract"

I think it is important to say that NHS Trust should not expect to have in-house DICOM expertise to do all this analysis themselves. They should be budget in the requirement to buy in the DICOM expertise for procurement. Ensuring that data is easily migratable at the end of contract--should be the key vision for anyone replacing their PACS. Nick Hollings posting highlights the problems if we dont do this.
 Link to this message Martin Peacock  posted on Wednesday, March 28, 2012 - 11:03 am Edit Post Delete Post Print Post
I do agree that the skills needed to perform *ALL* analysis are not necessary, but some in-house skills I think are important and many Trusts are quite possibly underskilled.

PACS (never mind PACS/RIS) is probably one of the largest capital investments a Trust is likely to make - certainly in the IT area. A mid-size Trust looking to a 7-10 years contract will be investing anywhere from £5M-10M.

As an analogy - as the driver of a private vehicle I'm quite happy to take my car into a mechanic to get the red light that's barking at me from the dashboard fixed. But if I was running a company with £5M worth of rolling stock I would have a small team of dedicated mechanics looking after them.

I recently posted a blog on the value of local skills w.r.t Business Continuity Planning (I won't link - anyone interested can click through my profile), but the same goes for performance management, day-to-day troubleshooting and many other aspects of the service. Just yesterday AM-Europe had an article from ECR on the value of local skills (it also raises the question of the relationship between Radiology & IT but this thread is probably controversial enough :-) ).

And it shouldn't be at enormous cost - a CPD budget of less than £10k per person/year would be ample to maintain a skill (and awareness) base that would provide RoI many times over. In the context of an overall procurement thats peanuts.

But then all of that is probably straying offtopic.
 Link to this message Chris Bull  posted on Wednesday, March 28, 2012 - 10:47 pm Edit Post Delete Post Print Post
We are providing a consultancy service to various Trusts offering expert help in areas like PACS, RIS, data sharing and VNA projects. We have also been assisting and reviewing the necessary documentation for the Trusts, which ensures that they meet their requirements and get good value for money in their procurement. My advice is to ensure that you use the right experts, meaning those who are part of the IHE, DICOM, HL7 communities and actively take part in these groups. The purchasers (the Trusts) should ask the same of the suppliers and their involvement with IHE connectathons and their certification from these events. These are published on the IHE.org web sites.

The other area of concern is ensuring that all data migration activities ĎMUST BE DATA SAFEí in order to guarantee patient safety and data continuity. In our experience, those migrations carried out Ďon the cheapí without data cleansing puts patients at risk. This task will need to be repeated when a full VNA is installed and the patient data updated to it. Again take the advice from an expert and use a true VNA provider or migration specialist to perform these tasks.

On the topic of VNAs, I have written another article for RAD Magazine on this topic which I believe will be in the April 2012 Issue.
 Link to this message Neelam Dugar  posted on Monday, April 02, 2012 - 02:44 pm Edit Post Delete Post Print Post
Thanks Chris & Martin for your input.

Many NHS Trusts will be going through PACS procurement for the 1st time. However, PACS replacements will happen every 7 years or so it is important that Trusts learn from the process.

PACS Procurement expertise with procurement document drafting etc is now available to most Trusts.
However, DICOM expertise to analyse documents, testing process & help with final contract document to ensure there is no "vendor lock-in" would be helpful for Trusts.
 Link to this message Neelam Dugar  posted on Tuesday, April 03, 2012 - 10:40 pm Edit Post Delete Post Print Post
If Trusts are considering VNA as part of PACS procurement, 27th April is hugely important to hear experts like Dr. David Clunie, Fred Behlen, & Kevin Wilson talk about vendor neutrality & data ownership. Last few spaces are remaining. Please let Adele/Katie know if you wish to attend.
 Link to this message Chris Bull  posted on Sunday, April 29, 2012 - 10:34 pm Edit Post Delete Post Print Post
For information my article on VNAs and data sharing is published on page 19 of Rad Mag April 2012. Dr Dugar's article is on page 15 of the same issue.
 
Add Your Message Here
Post:
Username: Posting Information:
This is a private posting area. Only registered users may post messages here.
Password:
Options: Automatically activate URLs in message
Action: