posted on Tuesday, September 06, 2011 - 06:18 pm
I thought that developing this doc would help Trusts. Please feel free to comment on it. This is based on the Group Specification document. I would like to thank everyone who contributes on the forum as the knowledge base for developing this questionaire is the forum discussions.
posted on Thursday, September 08, 2011 - 09:20 pm
I do not much understanding of the procurement process but from what I hear from talking to some people that PQQ is the basis of performing a short-list of suppliers.
Some people tell me that PQQ is largely for assessing the company profile & experience in the market rather than functional spec and standards compliance. The worry is that if you dont get PQQ right, one of the suppliers may challenge you. Above questionnaire is more a functional spec questionnaire rather a company assessment one.
My worry is that at the end it is a balance between financial credibility of a supplier with functional requirements to make a PACS work in a clinical env. Mr. Jim Wood will be speaking at our November 7th Meeting (Dr. Strickland cannot make it). Jim comes with a lot of experience in OJEU & procurements.
posted on Monday, September 12, 2011 - 08:52 am
There are 2 main segments of the PQQ 1. Economic & financial 2. Technical & professional. "In assessing the answers to the following questions, the Authority will be seeking evidence of the Potential Provider’s suitability to perform the services in terms of economic and financial standing, technical and professional ability. Qualification criteria will be a combination of both financial and non-financial factors and will be in accordance with Regulations 23 to 26 of the Public Contract Regulations 2006."
Getting your PQQ right is hugely important.
Top ten tips regarding PQQ process
"1. Be very clear about what you want to achieve from the contract and make sure everything in the PQQ is designed to help you choose the right bidder(s.) Don’t ask for information that is not directly relevant or useful, and make sure any decisions on de-selection are proportionate.
2. Run the PQQ process, and the whole procurement, as a project. Develop a plan, with agree timescales, and clear responsibilities (e.g. who is going to score the PQQs) and then stick to it.
3. Develop your scoring model (how you are going to assess the responses to the PQQ) at the same time as you develop the PQQ itself, and always before you issue the PQQ. Have all this information ready and available when you advertise the opportunity. (See Procurement Policy Note (PPN) 04/091
4. Make sure all the questions you ask in the PQQ are directly relevant, including questions about wider policies (heath and safety, equalities etc.)
5. The complexity and size of the PQQ should be proportionate to the value and risk of the contract; put word limits on responses in the PQQ and Invitation To Tender (ITT) wherever appropriate. Consider the time and effort required by potential bidders – and for you to assess the response.
6. References, examples of previous relevant, successful work, can be very useful. Ensure that you treat both relevant public and private sector experience equally. (See (PPN 10/09) 2
7. Use plain English, make sure it is very clear what each question is asking for, and make the whole process of completing the PQQ as painless as possible for the supplier.
8. Keep the financial assessment as simple as possible and don’t exclude bidders unless their financial situation is clearly unsatisfactory for the specific contract needs.
9. Keep the scoring system simple, and if you are using weighted evaluation criteria, make sure the high weightings are given to what is really important. That will often be the contract specific rather than the ‘standard’ PQQ questions.
10. Take a sensible number of suppliers through to the tender stage – for above threshold contracts, the EU Rules (under the Restricted Procedure) require that no less than 5 to be taken through to tender stage. "
posted on Monday, September 12, 2011 - 08:56 am
OJEU Process timetable for Restrictive Procedure (competitive dialogue requires a dialogue phase with 2 -3 bidders after the response to ICD (Invitation to Competitive Dialogue)
1.3 Time table Date or
[Target Date] Activity
[ 200*] OJEU notice published with PQQ made available to Potential Providers.
[ 200*] Potential Provider Open Day [if appropriate]
[ 200*] PQQ Return Date [see Public Contract Regulations 2006 for timescales]
[ 200*] Evaluation of PQQs completed
[ 200*] Invitation to tender issued to qualified Potential Providers
[ 200*] Tender Return Date
[ 200*] Evaluation of tenders completed.
[ 200*] Alcatel 10 day standstill period
[ 200*] Contract Award.
[*For further guidance on the Alcatel Mandatory Standstill Period please visit the OGC website.
posted on Monday, September 12, 2011 - 10:38 pm
A. General & Economic Assessment; This will be common questions used by your Trust for each of the procurements: 1. Details of the Bidder Name, Address, Contact etc
2. Subcontracting or consortia
3. Capability a. Business, products, services provided by vendor b. staff numbers--with qualifications employed c. Customers with date of contract award with date of completion
4. Financial audited accounts, guarantees , bank accounts
5. Insurance Employers liabilty Insurance Public liabilty Insurance
6. Disputes/Legal Proceedings
B. Technical Assessment of Radiology PACS 1. Integration with PAS for demographic & ADT using HL7 (or IHE) 2. Integration with RIS using HL7 (and SWF IHE) 3. Desktop Integration with 3rd party RIS 4. Culling of image data based on local policy 5. Ability to reuse existing workstation hardware 6. Ability to work in a brokerless PACS enviornment (RIS providing the DMWL) 7. Ability to receive & store & display images in DICOM format from all radiology modalities--CR, CT, MRI, US, NM, PET-CT, Fluroscopy, etc 8. Abilty to integrate with 3rd party plug-ins--Orthotemplating, 3D, CAD etc 9. Abilty to store & display radiology reports (Is CDA display supported) 10. Ability to be part of an EPR by adopting XDS-I? 11. Ability to share images beyond organizational boundaries--CDs, DICOM Push, IEP & XCA-I 12. Support Teaching Files creation (with use of IHE profile) 13. Support Radiation Dose Monitoring 14. Are there any known conflicts with other sofware applications that are commonly used. 15. How does the vendor deal with Private DICOM tags? 16. At the end of contract, in what format is the data going to be handed back 17. What kind of support is provided for PACS 18. What IHE profiles are supported?
Some suggestions for a PQQ
posted on Wednesday, September 14, 2011 - 10:40 pm
PQQ for Radiology PACS & RIS.
Good PQQ is key to getting the right vendors for PACS & RIS. Try to keep PQQ if possible to yes/no which allows easier scoring of the responses.
For Competitive Dialogue OJEU a 1. OJEU advert 2. Develop a PQQ --a number of vendors may respond to PQQ. 3. Evaluate PQQ responses you will be required to invite a minimum of 5 to ICD (Invitation to Competitive Dialogue) 4. During the dialogue phase you will build your final OBS 5. Then invite tenders to your OBS. 6. Select vendor
2 main documents that Trusts need to concentrate on are 1. PQQ (will allow you to choose the 5 for dialogue) 2. OBS (vendors will tender based on OBS)
Jim Wood has a lot of experiance in this field will be speaking at the 7th Nov Meeting of our Group.
posted on Thursday, September 15, 2011 - 10:55 am
Slightly shorter PQQ which includes questions for both RIS & Radiology PACS. Comments both on-line & offline are gratefully recieved.
posted on Monday, September 19, 2011 - 10:49 pm
PQQ is the methodology for shortlisting vendors for inviting them to dialogue. For PACS it is possible we may have to shortlist 5 out of 20 or so. For RIS it maybe more difficult. Hence, having a good set of questions will help.
I enclose a set of PQQ questions which maybe used. We could leave the financial assessments to Finance departments but the technical/functional assessment will be down to us.
It maybe useful divide questions into--- essential & desirable. It is important to have a fair shortlisting process which can be scrutinised if required.
One of the forum readers suggested that PQQ questions should largely be Y/N answers with a 100-200 word limit for details on the technical response to each question. Let me know what you think.
posted on Tuesday, September 20, 2011 - 05:33 am
Is the PQQ attached here is too detailed? I dont know the legal view on what is allowed on PQQ or not.
If we went really basic and all we said we needed a system that was able to store & display images,we would have at least 20 vendors for PACS. Then the shortlisting criteria is simply based on balance sheets. This discourages innovative SMEs. This also encourages complacency towards standards adoption by large vendors. Hence, I personally think PQQ should include standards. However, I am not familiar with Procurement rules & regulations regarding PQQ. Any views?
posted on Tuesday, September 27, 2011 - 12:43 pm
There is a lot of scaremongering that goes on about the OJEU Process & possible challenges from vendors if you dont get it right or how difficult it can be. We are very fortunate to have Tony Corkett (Cloud 21) speaking at the Autumn 2011 meeting at Birmingham
4.30 Replace PACS & Save Money—Simplifying the OJEU Process——Mr. Tony Corkett (Cloud 21)
I would reccomend getting your IT head/Finance Directors/Procurement people to listen to the final 2 talks. They will be interesting & helpful to all NHS Trusts.
He will take us through the PQQ and what can/cant be included at PQQ stage.
OJEU provides the real test of the PACS market. Many Trusts are going ahead/recently gone with this process & will advice if asked.
Simplistically there are 2 steps 1. Selection criteria--PQQ 2. Award criteria--ITT/Dialogue
These must be decided upfront. I would suggest using IHE standards & clinical requirements to be used in selection criteria so that award can then be made largely on the price. Price cannot be used on selection criteria.
OJEU is supposed to create a fair and transparent process for procuring PACS/RIS solutions.
PQQ stage is the shortlisting stage. 1. The shortlisting criteria must be declared upfront. 2. Based on offline discussions with many from industry & outside we need to have some pass/fail questions & some desirable (weighted scores) 3. PQQ should not reflect a particular vendor bias. 4. If a vendor passes a PQQ stage. You cannot disqualify him at award stage using the criteria set in PQQ. For example: If PACS A is XDS-I compliant & PACS B is not However, XDS-I was a desirable feature on PQQ you cannot award PACS A the contract simply because of XDS-I. At award stage your ITT/award criteria MUST be different to selection/PQQ criteria. At selection/PQQ stage you cannot ask for prices. In my very simplistic view most awards will largely be based on price (hence getting a good break-down of current & future costs is very important)
Here goes my simple generic PQQ/selection questions for PACS with some questions having a pass/fail criteria with others having a weighted score (depends on how much a Trust values the criteria)? My request is if you are using these questions please get them okayed by your procurement team lawyers for the correct wording B. TECHNICAL ASSESSMENT OF RADIOLOGY PACS SOLUTION 1. PAS Integration: 1.1Does the solution have the ability to integrate with a PAS for demographic & ADT using HL7 messaging. Pass/Fail. 1.2 Is PIR Profile of IHE supported by the PACS? –Yes/No/Future development (weighted score) 2. RIS Integration: 2.1Does the PACS have ability to integrate with RIS using HL7 messaging? Pass/Fail 2.2 Is SWF profile of IHE supported by PACS? Yes/No/Future Development (weighted score) 3. Desktop Integration: 3.1 Does the PACS have the ability to support a Desktop Integration with a 3rd party RIS? Pass/Fail 4. Data Culling: Does the PACS allow for rule based image culling based on local policy—Pass/Fail 5. Workstation Hardware Reuse: Will the vendor allow for the Trust use its existing diagnostic workstation hardware? Pass/Fail 6. Brokerless PACS: Is the PACS able to work in a brokerless PACS enviornment (RIS providing the DMWL) ? Pass/Fail 7. DICOM Store & Display 7.1 Is the PACS ability to receive & store & display images in DICOM format from all radiology modalities--CR, CT, MRI, US, NM, PET-CT, Fluroscopy, etc Pass/Fail 7.2 Are Basic Image Review of IHE supported in PACS? Yes/No/Future Development(weighted score) , 7.3 Is Nuclear Medicine Display Profile in IHE supported in PACS? Yes/No/Future Development (weighted score) 7.4 Is Mammography Image Profiles of IHE supported in PACS? Yes/No/Future Development (weighted score) 7.5 Is DICOM Blending Presentation State Supported in PACS? Yes/No/Future Development (weighted score) 7.6 Is DICOM Enhanced Objects supported in PACS? Yes/No/Future Development (weighted score) 8. Plug-ins 8.1 Does the PACS have the ability to integrate with 3rd party plug-ins--Orthotemplating, 3D, CAD etc . Pass/fail 8.2 Is DICOM 118 standard supported? Yes/No/Future Development (weighted score) 9. Radiology Reports: 9.1 Does the PACS have the ability to display radiology reports with images? Pass/fail 9.2 Is CDA (Clinical Document Architecture) display for reports supported? Yes/No/Future Development (weighted score) 9.3 Is Displayable Report Profile of IHE supported? Yes/No/Future Development (weighted score) 10. EPR Contribution: 10.1 Will the PACS be able to be part of vendor neutral EPR? Pass/Fail 10.2 Does the PACS support XDS-I of IHE? Yes/No/Future Development (weighted score) 11. Image Sharing: 11.1 Does the PACS have ability to share images beyond organizational boundaries—CDs & DICOM Push/IEP, Pass/Fail 11.2 Does the PACS support XCA-I of IHE? Yes/No/Future Development (weighted score) 12. Teaching Files: Does the PACS support Teaching Files creation? Yes/No/Future Development (weighted score) Is TCE profile of IHE supported? Yes/No/Future Development (weighted score) 13. Radiation Dose Monitoring: Does the PACS support Radiation Dose Monitoring? Yes/No/Future Development (weighted score) Is REM Profile of IHE supported? Yes/No/Future Development (weighted score) 14. Software Conflicts: Are there any known conflicts with other software applications that are commonly used in NHS hospitals on PCs? Yes/No Details & comments if applicable. (weighted score) 15. Private DICOM Tags: Are Private DICOM tags commonly used? Yes/No (comments if applicable) (weighted score) 16. Post Contract Data Migration: 16.1Will PACS data be handed over/migration support provided to a new vendor at the end of contract? Pass/Fail 16.2 Will read-only access to image store be permitted? Yes/No/Future Development (weighted score) 16.3 Will a database dump be permitted? Yes/No/Future Development (weighted score) 18. url PACS Image launch: Is it possible to launch the exam from PACS via a url containing the accession no. from another software which contains the report. Pass/fail 19. Business continuity Model: Will vendor support a business continuity model for PACS support & service? Pass/Fail 20. Enterprise Storage Architecture: Would the PACS be able to deliver the clinical performance requirements whilst using the Enterprise Storage & Network Infrastructure? Yes/No/Future Development (comments if applicable)—weighted score 21. Audit trail/View log—Does the PACS provide a audit trail/view log for image access that is accessible by user community. Yes/No—weighted score
IHE profile questions in my view are vendor neutral questions. It will be a sad day for the PACS community if vendors challenge Trusts for using IHE in the PQQ stage for selection.
Jim Wood & Tony Corkett with their experiance will give help & advice on procurement process on 7th Nov. Meeting. Remember that PACS replacement in 2013 is definately going to save Trusts a huge amount of money. To get the clinical functionality right make sure we remain involved in selection & award stages.
posted on Tuesday, October 04, 2011 - 05:20 pm
Neelam- I might suggest adding to #21, the ability to collect and store ALL audit transaction data in perpetuity in XML format. This includes data at the user, patient, exam and workstation level. Furthermore, the specification should include the ability to export or connect to the audit database via ODBC for further analysis and reporting.
Thanks Mark. I think that is a very good suggestion. I will include this. I know you have a lot of experiance in this area. Can you help with some questions: 1. Do all PACS suppliers consistently store this data? 2. Where is this data stored in PACS? I presume it is a the relational database. 3. How is this data migrated at the end of contract? Easily or very difficult. 4. Is there a DICOM standard or IHE profile which specifies how this information must be stored in a vendor neutral way so that migration at end of contract becomes easy?
ATNA Profile of IHE: "Audit Trails are based on the production of audit records, that provide a record of actions such as queries, views, additions, deletions and changes that are processed within the Security Domain covered by ATNA."
The profile has very high support from big & small vendors alike.
Thanks Mike for the nudge.
posted on Wednesday, October 05, 2011 - 05:31 am
Thanks Mike. Updated questions: 21. Audit trail/View log— 21.1 Does the PACS record an audit trail of access---user, patient, exam & workstation? Pass/Fail 21.2 Is the audit trail visible at the user front end?. Yes/No—weighted score 21.3 Does the PACS support Audit Trail & Node Authentication Profile of IHE? Yes/No/future development –weighted score
I provide advice to clients on the aspects of the procurement process (not the legalies – I am not a lawyer) and am also the IHE-uk cochair. There is always more than one way to do this, but I have looked at PQQ v3 and think its too detailed for the PQQ stage and needs to be cut down a bit and focussed more on the fundamental expertise you want from companies – you don't want to be too prescriptive at the PQQ stage. Ideally you want the PQQ stage to be open, innovative and about discovery of options rather than closed questions. The closed technical questions are for stage 2.
Fundamentally there are 3 stages to any purchasing decision (although you can have more if needed):
1) Discovery of relevant suppliers: deslect from the EOI longlist with a PQQ, 2) Invitation to Tender : Get detailed compliant proposals from shortlisted suppliers wth an RFP (or ITT in old speak) 3) Select: Implement and communicate decision to all suppliers: explaining decision and helping the losers get over it
From an IHE perspective it is important to influence ALL stages for a PACS/PAS/ equipment procurement - esp PQQ, as being an IHE member company is a strategic decision, and not simply something that a company decides at the proposal stage.
stage 1) only want suppliers who have been to or are committed to attend IHE connectathons (globally), as interoperability is AN ESSENTIAL MUST HAVE for clinical safety, so shortlist on basis of 'strategic intent' as well as simply 'technical capability'. stage 2) need to ensure only relevant IHE profiles are referenced in specifications/ evidence of prior success useful but absence should not exclude proposals stage 3) need to ensure that suppliers who promise compliance actually deliver evidence of it through successful IHE connectaton tests or through the contract"
If IHE intent/capability is introduced from the PQQ outset, then you can't be challenged by suppliers at a later date for not making it clear to vendors that it was important. If you introduce IHE as a requirement at a later stage - then it is possible for that supplier to challenge, on the basis that the requirements have been significantly changed.
The only worry for buyers(and probably lawyers) is whether there is enough competition in the IHE market for the procurement to be legal - but the PQQ stage is designed to flush this out. That's why its important to include general commitment to IHE to allow for new entrants to the IHE market provided that procurement timescales allow for this , but not to put too many closed questions in the PQQ , which exclude companies who have not aquired the knowledge yet.
However, before you start all this . . . there is nothing to stop a Trust holding a general supplier workshop outside of the procurement process , for all parties to learn more about the features of the desired service vision . . .. complications . . . .risks . . . etc any invited party can participate in this process and any objections to including IHE capability in the PQQ can be openly discussed here.
Thanks Malcolm for your comments on the forum. They are really appreciated.
Let us understand the process. PQQ is the shortlisting stage. Out of the 20-50 responses you are required to bring it down to a managable number of 3-5. If PQQ stage is not objective enough decisions will be made on size of company & number of staff which although not overtly but does discriminate against SME.
At ITT or award stage the decision is largely based on the price offered for the detailed functional spec provided.
My view is that use IHE in the PQQ/selection stage to select vendors who have an interest in IHE. Award stage should be based on the price on offer.