My old colleague and good friend Tim Williams sent me the story below for inclusion in EMI Stories/Banana Skins, and I found it so interesting in so many ways that I decided to publish it as a Blog.
In so very many EMC standards and other meetings on EMC, we hear that the stability of the AC mains power frequency is either ‘not an EMC issue’ or is ‘never a real EMC issue’.
And it’s true that in a national power grid, the mains frequency has to be very stable so that all the various rotating sources of generation can be ‘phase-locked’ and thus able to contribute power to the network.
But Tim’s contribution below shows us that AC mains power frequency can be a very important EMC issue indeed, and thinking of this spurred me to write some more on this oft-overlooked issue which I’ve added later, after Tim’s contribution.
Unexpected interactions between EMI and safety software stranded a number of trains for several hours on 9th August 2019
From Tim Williams
I’d like to recommend that you read ORR’s (Office of the Rail Regulator) report into what happened on 9th August, i.e. the major power outage country-wide: https://orr.gov.uk/__data/assets/pdf_file/0017/42164/railway-power-disruption-on-2019-08-09-report.pdf
You may remember that one of the consequences of the outage was that all Thameslink’s smart new Siemens trains stopped moving and refused to budge, in many cases until a nice man with a laptop had been to do a hard reset on them – which took several hours, and despite the fact that the traction power to them never actually failed.
The ORR report above gives chapter and verse on why this happened. Section 3.3. in the report is the most relevant bit. Essentially:
- If the train power supply frequency drops below 49Hz, there is the chance that the trains motor drives can generate interference “in a range that can affect signaling circuits”.
(I can understand that this could happen, but if it can happen below 49Hz, why is it not a problem at 50Hz? Anyway…)
- As a result, Siemens designed the software to lock out the system when it experienced below-49Hz power. Clause 4.2 Note 2 in EN 50163 allows this, despite normally requiring operation down to 47Hz.
- During the power disruption on 9th August 2019 (well described in the Ofgem report: https://www.ofgem.gov.uk/system/files/docs/2019/09/eso_technical_report_-_final.pdf the grid frequency dropped as far as 48.8Hz for a short time (it was below 49Hz for about a minute)
- In fact, Siemens was in the process of updating the train’s software such that a below-49Hz lockout was permanent, i.e. requiring a technician to restart it. In previous software versions, although the lock-out occurred, the driver was able to re-start using what was known as “Battery Reset”.
By the 9th of August 2019, some trains had been updated and some had not.
So, in some cases the driver was able to re-start the train, but in the majority of cases the lockout was permanent and they had to wait for the nice man with a laptop – and this took a long time.
- The ORR report above says “It appears therefore that the collective response of Class 700 and 717 trains to the out-of-specification supply frequency was in accordance with the software design, but was not an explicit intention. Siemens accepts that the temporary reduction in frequency should not have been considered a situation that requires a permanent lock-out.”
So here is a lovely EMI Story for you: the unexpected interaction between EMI and safety software: a potential EMC issue which demanded a software safety fix – which in turn created substantial and unnecessary aggravation in the presence of an unusual grid-wide failure situation.
By the way, the Ofgem report that I also mentioned is a good and detailed explanation of why and how the grid burped.
Particularly, it shows what happens when a lightning strike hits a particular point on the 400kV network. It was fascinating for me to read this, and assuming I occasionally still do some design training I’ll be including this in the section on transient protection.
So, here’s what Tim’s story, above, spurred me to write about…
Our equipment isn’t always run only from national mains distribution networks (‘grid’) – sometimes it runs from a single power source – anything from portable generators, through generators sized in the hundreds of kW to power hospitals and other critical facilities, to generators in the MW range for powering offshore drilling platforms, remote mining sites, military bases, large towns or small cities where there is no national grid.
Because they have no need for synchronization with other rotating generators, these power sources and their distribution networks can have very much larger frequency variations. The worst example I know, is the ±100% frequency variations from a 10MW diesel generator on an offshore drilling rig.
That ±100% was not a typo – its nominal 50Hz operating frequency would collapse to zero when the 10MW drill motor was turned on, and increase to 100Hz when the drill motor was turned off! Of course, the drill motor was not fitted with a soft-start/stop device.
When we design off-line power converters making the erroneous assumption that the AC power grid will always be at 50Hz (or 60Hz) plus/minus < 1%, frequency reductions don’t have to be very large before the regular rectifier – storage capacitor circuits we use to create unregulated DC voltages for our power converters start to dip below the minimum input of the regulator, hence dip below their minimum DC outputs. For a worked example, see the section on: “How do dips and dropouts cause problems?’ in the free download https://www.emcstandards.co.uk/a-practical-guide-for-en-61000-4-11-testing-an.
Modern digital devices operate on low voltages with very small tolerances, and when their DC rails dip below their minimum specified voltages, even for a microsecond, they can go completely insane – the least of what they can do includes over-writing their program memories, reconfiguring their logic blocks, ‘losing’ their internal calibration and set-up data (e.g. limit settings), etc., – and may require power cycling at least, to restore normal operation.
The current fashion for low-dropout power converters/regulators to save power, reduce cost, and save the planet by increasing efficiency makes the horrible consequences of such momentary dips more likely, and the minimal dips and dropout tests in the generic and product-family EMC standards are not comprehensive enough to detect them all.
Accordingly, I always recommend the following for off-line power converter design:
- Design for continuous mains variations, of, say, ±3%, and verify/validate with a variable-frequency synthesized mains source. If your equipment is likely to be used on sites with portable or ‘islanded’ power generation, design and verify/validate for ±5% (or more, if site data suggests it). And, if your equipment is likely to be used in the third world, design and verify/validate for ±20% (or more, if site data suggests it).
- Pass all of the dips/dropouts tests in IEC 61000-4-11, instead of the very limited selection of them found in the generic and product-family EMC standards (see the free guidebook on IEC 61000-4-11 posted at https://www.emcstandards.co.uk/emc-testing).
- Design and verify/validate that your equipment complies with all the tests in IEC 61000-13, 14, 16, 27 and 28 (and/or -29 if it can be powered from a DC power distribution network, all free and posted at https://www.emcstandards.co.uk/emc-testing).
- Ensure by design that nothing that could possibly ever happen to the quality of the electrical power supply, whether AC or DC, could cause a failure in your equipment that could be potentially deadly or life-changing to any operators, patients, or third parties (see https://www.emcstandards.co.uk/emiemc-risk-management), over its complete life-cycle. Never ignore errors/fluctuations in the electrical power supply that could only be caused by people behaving totally stupidly even though they know better. Even people who are skilled, knowledgeable, senior experts can make totally stupid mistakes, as Chernobyl has proved to us all.
- Don’t rely on warnings in the manuals to only operate within < ±1% (or whatever) of the specified mains frequency. Everyone knows that no one ever reads manuals, so trying to defend yourself against a Product Liability lawsuit on the basis that the user should have read the manual and followed its requirements to the letter is going to need a lot of high-power and costly lawyering to stand any chance of success.