What is an Engineering Audit Like? An ISO 13485 Medical Device Engineering QMS Case Study

I think I found my new look.

If there’s one thing that gets me fired up, it’s a good ol’-fashioned audit!

… Said nobody ever.

Sadly, audits are a part of the business world that tend not to be fun.

However, they also serve an important purpose, and are part of the transition from the “fun team lead” style of management into the “business leader” role is coming to grips with the unsexy stuff.

As part of my role at HelloBetter, I was tasked with leading engineering compliance and reporting for ISO 13485, an internationally recognised standard for Quality Management Systems (QMS) specifically for medical device manufacturers, within the broader context of the EU Medical Device Regulation (MDR).

For those who aren’t familiar with what an audit of this kind involves, the simple version is this:

1. Certain requirements exist as part of the standard, not just about how to effectively create and maintain software, but also how to document that compliance.
2. This documentation is referred to as the Quality Management System (QMS), which as the name implies serves as documentation for how you maintain quality as per the ISO.
3. The ISO isn’t only about software, but software is part of “Lifecycle Management”, tracking how a medical device (in this case) goes from design and development to distribution and surveillance.
4. A company required to meet the ISO should therefore create and follow a system that ticks all the boxes for things like patient safety, risk management, and any additional regulatory hurdles.
5. Typically once a year, an auditor is hired to enter the company and review the documentation alongside representatives from the company who guide them through the QMS.
6. Additionally, individuals (like me!) are pulled from different parts of the company to talk through their processes and documentation, and to show any examples that may be relevant.

My Experience with Engineering Compliance for ISO 13485

This was my first audit, and I took the role very seriously. Alongside our compliance officers, I was tasked with reviewing and editing several QMS documents that were either outdated or needed more detail.

In order to ensure that we were producing accurate documentation, this process lasted several months and included pulling in engineers from throughout the organisation.

Some examples of what we needed to document included:

• Branching conventions
• Coding style guide
• Evaluation of risk for libraries/packages (Software of Unknown Provenance or SOUPs)
• Quality assurance and testing processes
• Traceability

Two of these were areas that I hadn’t explored much before, and which proved to teach me a lot about ISO 13485 and EU Medical Device Regulation generally.

Evaluation of SOUPs

As an engineer at previous companies, my experience with external packages and libraries I wanted to use was really rather simple. It went like this:

npm install

That was it.

However, during my time at HelloBetter I was exposed to a much more strictly regulated software landscape. Here, it wasn’t as simple as just installing and using the package you need. Instead, attention needs to be paid to where this package comes from, whether it is actively maintained, and what level of risk comes with its inclusion in your code.

Image courtesy of xkcd: Dependency

To understand this risk, engineers and/or Engineering Managers need to pay attention not just to the node warnings at runtime, but also to the ongoing balance between software benefit and patient risk. In other words, is the convenience and ease of using this package worth the potential downside (e.g. hacking attempts, data leaks, full app breakdowns)?

For the sake of the audit, companies need to be able to demonstrate:

1. Who has the responsibility for evaluating this benefit/risk balance?
2. When or how often is that balance revisited/reevaluated?
3. What procedures are in place for any sudden vulnerabilities discovered in SOUPs?
4. Where are the SOUPs themselves documented?

Combined, these questions must be answerable in documentation and as a realistic part of the software development process.

Traceability

The other main theme is traceability. Let’s go back to what I said earlier about the role of software in ISO 13485 relating to “Device Lifecycle”. If we take the goal of the audit as being an evaluation of the entire process of a device, and we acknowledge software as part of that process, then the core question becomes this:

How can we see the device changes occur, from conception through to execution?

For ISO and medical devices in Europe generally, this means tracking software features both via software (e.g. Jira or other task-tracking tooling) and also via consistent software versioning.

On this last point, you don’t need to guess at how to structure your versioning, because there are also guidelines on how to make these numbers. Specifically, the UDI (Unique Device Identifier).

Luckily for me, implementing the new UDI requirements was part of my team’s responsibilities, meaning I’m now very familiar with the requirements.

In short, you need to be able to identify:

• Unique Device Identifier or UDI-DI: the static number for a specific device
• Product Identifier or UDI-PI: a dynamic identifier which changes with any code or course content change

In the case of physical devices, you can actually see this information displayed as a barcode. For example:

In our case, we are a software product, so our structure is a bit different.

We pulled BitBucket build versions to automatically update our UDI-PI and combined those with our UDI-DI for each course, and the mobile version numbers. Taken together with proper branching conventions and when synced with Jira, you could trace every ticket right through to the exact build both for development and user support.

Importantly for the auditors, this process allows them to trace exactly when and how any change is pushed to production, completely independently of any potential human documentation errors.

The Audit Itself

Naturally, I can’t go into too much detail about how the audit itself ran. I can say that it is a very thorough and slow process. Certainly there was a lot of sitting around and a lot of waiting.

When needed, we would answer questions regarding the requirements, and then show evidence that we were meeting those requirements. Whenever there were points of confusion or clarification was needed, I would answer to the best of my ability. If more detail was needed, we would call in one of our relevant engineers (e.g. for security or quality assurance) to dive deeper.

Tips for Audit Preparation

Some things which helped me to prepare were:

1. A ‘Documentation Taskforce’ made up of volunteer engineers, where we went through all existing QMS and working instructions for engineering.
2. Preparing a spreadsheet of each ISO 13485 topic relevant to engineering, including a link to the documentation and 2–3 links to relevant, recent examples.
3. Ensuring that all senior/lead developers were aware of the audit and kept their calendars clear in case we needed to invite them into the meeting for clarification.

More importantly than all of the above, is having an organisation-specific person or people who are highly competent and good at explaining what is needed for each of us representing our departments. In our case, this was the expert work of Dr Victor Stephani and Esther Blum. Without them, I would have been in the dark about the expectations and preparation required, and I cannot thank them enough for being my guides into the weird and wonderful world of audits!