A discussion between Peter Thomas (PROFIBUS/PROFINET expert) and Keith Armstrong (EMC expert), in December 2017
Peter: Hello Keith. I have spent today visiting an industrial plant in the UK. Whilst there I took the opportunity to look at several areas from an EMI point of view. I wondered what you thought of the following:
I list below a few observations following yesterday’s informal review of your installations from an EMI (Electromagnetic Interference) point of view. These comments are not a detailed analysis of the installation but simply based on observation. They are also made from a future-proofing point of view in the knowledge that switching devices like variable speed drives (VSDs) will employ higher switching rates which will lead to greater emissions if not adequately controlled.
1. Vessel Control System
This consists of an industrial Programmable Logic Controller (PLC) that communicates with remote IO using the EthernetIP industrial Ethernet protocol. The following observations were made:
1.1: The EthernetIP network cables appear to be unscreened. In an environment containing several variable speed drives this is surprising. Rockwell provide advice on choosing the most appropriate cables for use with EthernetIP and generally recommend unscreened cable for office environments or light industrial installations only.
If this were a PROFINET installation, which is just another form of Industrial Ethernet, screened cables would have been mandatory.
Keith: Hello Peter. I often see this sort of thing. Either people don’t bother to read the EMC installation instructions for the equipment, or if they are quoting for a project they ignore the instructions to help quote the lowest price.
For example, I know a major company European manufacturer of VSDs with excellent EMC and excellent EMC installation instructions, but I’ve seen their own salespersons in the Near East quoting their VSDs for projects whilst ignoring all of their own company’s EMC installation guidance – to get the price down to try to get the job!
The emphasis of most buyers is on cost of purchase, whereas cost of ownership is arguably as important. The cost of ownership could even be overwhelmingly important, if the low cost of purchase was achieved at the expense of adequate EMC, so that whatever was purchased doesn’t work properly, or at all!
Also, unfortunately, because most Salespeople are paid commission on the raw value of a contract, they make money even if the project loses money for their employers!
(Most Salespeople could care less, which is why many of them move around so much from company to company.)
Peter: 1.2: Several of the motor cables between the drive and the motor appear to have at least one end of the cable screen disconnected, instead of being terminated at both ends. This will allow high frequency EMI to escape from the motor cables that may impact on other systems.
Keith: Of course it will! But the electrical installation business worldwide believes that the most important thing ever is to prevent “ground loops”, and connecting a cable screen at both ends causes a “ground loop”!
I’ve even heard of installers refusing to terminate screens at both ends despite the equipment supplier and the customer literally beseeching them to follow the equipment’s installation instructions to do so!
I have several case-studies where incorrect cabling (inc.: incorrect type of cable, not terminating screens at both ends, etc.) cost both supplier and customer many millions of GB Pounds, which could not be recovered from the company that performed the installation because they were not worth much and simply went bankrupt when they were sued.
Peter: 1.3: The international standard IEC 61000-5-2 (EMC – Earthing and Cabling) recommends a minimum vertical and horizontal separation of 15cm between sensitive cables like PROFIBUS, PROFINET, EthernetIP etc. and single phase power cables and a minimum of 30cm between them and 3 phase at all points along the length. Where cables cross they should do so at 90 degrees. This approach does not appear to have been taken.
Keith: IEC 61000-5-2 also says that cables that are more than one class apart should cross at 90 degrees and be spaced well-apart from each other.
People may not realise that, where stage payments are being made, a common reason for refusing to pay the final payment (often the one that contains all the supplier’s/installer’s profit on a project) is that the new cabinet/system/installation causes/suffers EMI, and the supplier had better fix it at his own cost. Quite likely, these claims are completely spurious and are just a delaying tactic for the final stage payment.
But where the claims of EMI are valid, and if an independent consultant such as you or me is called in to review the new cabinet/system/installation, the consultant is quite likely to point out what you have just done, whereupon the supplier/installer is in serious legal trouble.
This is because IEC 61000-5-2:1997 is the UK’s national standard on system/installation cabling and “earthing”, and if a UK company does not apply it when it is applicable they make their legal defence very difficult indeed!
The valid EMI might have nothing whatsoever to do with the system’s/installation’s cabling and “earthing”, but who is going to be able to say that with any certainty until the EMI problem has been bottomed and the real reason found?
And it is the supplier/installer who is being sued who has to pay for the independent expert to solve the EMI problem – difficult to afford their fees and costs when they are having to pay their legal team as well, and when their senior engineers are busy fighting this fire instead of working on new projects that will produce much-needed income!
It can easily become a ‘perfect storm’, that could quickly ruin any debt-financed company (when their Bank hears about the problem, they foreclose as quickly as they can to minimise their financial exposure), and I have seen ruin the wealth of all a company’s Directors to the point where their company was on the brink of bankruptcy.
And to think that companies are put at such huge risk by people fairly low-down the seniority ladder who imagine that by cutting corners they can save a few quid and their Directors are going to thank them for it! If Directors only knew what risks their companies were being exposed to, by people they employ who are supposed to know what they are doing, they would probably sack them all on the spot!
The larger problem, the ‘elephant in the room’ if you like, is that technical people aren’t taught about financial risk management, and management people aren’t taught anything technical (or have forgotten it).
The result is that in most companies it is impossible to have any meaningful debate about whether it is worth the risk to the company’s very existence, to save a few quid cutting corners by not fully applying IEC 61000-5-2:1997.
Another problem is that until they get snared by a legal complaint, most of us tend to assume that ‘the law’ tries to be fair. But in fact, even basic common sense and ‘what everyone knows’ (e.g. Ohms Law) immediately go out of the window the very second that lawyers get involved.
In reality, ‘the Law’ is just a less bloodthirsty way for people with more money to screw other people, than how they used to screw them over in previous centuries. The outcome is the same, with no respect at all for what’s fair or what’s right, only who has the bigger stick (i.e. the more expensive legal team).
I’ve seen all this for real. And it’s not pretty! In fact, it can be shockingly sickening and disgusting, even to engineers who think they are already very hardened, cynical, and world-weary.
Peter:
2. Water Processing Area
This consisted of several devices that connected to each other via a PROFIBUS DP network.
2.1: Several of the motor cables between the drive and the motor appear to have at least one end of the cable screen disconnected, instead of being terminated at both ends. This will allow high frequency EMI to escape from the motor cables that may impact on other systems.
Keith: Same comment as in 1.2, regarding not terminating cable screens at both ends.
Peter: 2.2: The screens of the PROFIBUS cables appeared to have been correctly terminated at both ends however several had currents > 80mA (40mA is a typical maximum). The accepted good practice of grounding the screen on entry / exit from a panel had not been followed.
Keith: Same comment as in 1.2, regarding not terminating cable screens at both ends.
Peter: 2.3: When terminating the screens of cables at both ends is there is a potential for the screens to unintentionally become part of an existing equipotential bonding system and in doing so carry much higher currents than they were designed for. This can be offset by ensuring that the screened cables run in close proximity to cable trays that are rf-bonded (Radio Frequency) at their joints using metal plates or braid straps as opposed to wires. This should be done along their entire length, and to the panels at each end. This approach makes the trays act as parallel earth conductors (PEC).
Keith: Absolutely correct! I would add that when one has created the minimum-possible mesh-bonded system/installation according to both IEC 61000-5-2:1997 and my ‘lambda/10 at the highest frequency of concern” rules, the more cross-bonding that happens the better everything gets. Without exception!
At one extreme is a so-called “single-point grounded” system with no (so-called) “ground loops” at all. Such systems almost never exist in reality, except for the very simplest installations. In real-life there is always one, usually more (so-called) “ground loops” occurring by accident and almost impossible to track down.
These accidental (and probably undiscoverable) “ground loops” can easily cause excessive cable screen currents, with excessive RF currents often arising at places where you wouldn’t expect them to be, anywhere on the site, because of uncontrolled resonances in the “ground loops”.
A small change to the system can cause these resonances to vanish, only to pop up somewhere completely different, anywhere on the site.
At the other extreme, is the meshed-common-bonding-network (‘MESH-CBN’) structure recommended by IEC 61000-5-2:1997 (especially when constructed using my ‘lambda/10 at the highest frequency of concern” rules), which suppresses almost all resonances by design, and has so many possible paths for 50/60Hz currents that no cable screen ever sees more than a few mA.
If we take an existing “single-point grounded” (supposedly) system and over a period of time modify it with the aim of eventually creating a ‘proper’ MESH-CBN, during those modifications all sorts of weird and wonderful EMI problems can arise, anywhere on the site, until we get to the minimum MESH-CBN, when the resonances are finally all brought under some kind of control.
From then on, as we improve the meshing, any remaining EMI problems caused by (so-called) “ground loops” get continually less and less until they become completely negligible. I’ve done this many times!
Peter:
3: Server Room
This room contains several floor-standing server cabinets, two of which are used for the storage of process-related data. There are specific standards associated with providing a low-impedance reference (ground) in IT Server rooms. This usually involves an underfloor mesh bonded network and a bonding ring conductor (BRC) around the perimeter of the room. Neither was found. Additionally, it was noticed that there was no panel to panel RF (radio frequency) bonding employed and where braid straps had been provided they had been left disconnected. The two panels associated with process data were manufactured by Schneider who have published a document that is consistent with IEC 61000-5-2:1997.
Keith: Same comments as earlier, regarding not following the relevant standards and/or not following suppliers EMC installation instructions.
Before 61000-5-2 was first published in 1997, Schneider published lots of great guidance that was based on its 1995 draft. Since then, I have had the great fortune to meet and work with the author of Schneider’s recommendations, Jacques Delaballe, who retired a few years ago but still represents Schneider on some EMC-related standards committees.
Peter:
4: Conclusions
From an EMI point of view and based purely on a very brief observational survey, I would consider that the installations may lack robustness and be at risk from intermittent / annoying glitches that could impact production now or in the future. It is clearly impractical to address all of the concerns on an existing installation so a more pragmatic / risk-based approach is recommended. This would require unobtrusive measurements to be taken around the equipment and the interconnecting cables and would need to be done during normal operation, i.e. with the process running. If issues were identified then local solutions could be implemented.
Keith: Absolutely correct! I would add that most large systems/installations suffer from problems that reduce their output rate or degrade their output quality. Such problems are often intermittent, with no obvious pattern to them, but (like a manufacturer with some % products being returned under warranty) they can be lived with.
In my experience, these are often caused by the kinds of poor EMC installation practices you describe above, and because their causes are unknown and they are therefore not controlled, a small change to the system/installation can cause them to ‘flare up’ and become show-stoppers.
This can happen whether the changes are intentional or not, for example they could just be the aging of some of the equipment (the EMC performance of equipment always degrades over time!).
Peter: In line with recommendations for the qualification of PROFINET networks, the same should apply to EthernetIP networks. Such an approach checks the low-level device to device communication prior to handover. Relying on the absence of PLC alarms / diagnostics is not considered good enough and could leave you with a network that is at risk from intermittent failure that will be difficult / time-consuming to resolve. This is a relatively simple task to undertake.
Keith: Yes, PROFIBUS / PROFINET are simply examples of datacommunications, and all datacommunications suffer from the same EMC issues. Some suffer more because their error correcting protocols are less robust, but they all suffer!
A problem with modern digital datacommunications is that their error correction hides EMI problems, until it is too late. For example, on an old analogue television you could always tell if it was suffering EMI from the noise in its picture, so you knew if there was a problem that needed to be solved before it got so bad that you couldn’t watch the programmes you wanted to. (Sometimes all that was needed was to bang on the wall to get the neighbour to turn off his electric drill!)
But with modern digital TVs the picture is always perfect, or else it isn’t there at all, so you have no idea whether EMI is present or not. I’m told that many people return their digital TVs to the shop they bought them from, saying that they are ‘broken’, only to find that the next one they are given is also ‘broken’, and the next and so on. They all work just fine in the shop, but not in their home.
I know one guy who, having had several replacement TVs from the shop, realised it must be something else so had a complete new aerial installation put in, even replacing the coaxial cable buried in the wall – a disruptive and costly exercise. But the new aerial had no effect at all, his new TV still wouldn’t work in his house (although it did in the shop). Eventually he found that all he had to do was move his DECT phone in its charging station a little further away from the TV!
Many industrial datacommunications could be on the point of not working at all, all they need is a little more noise (perhaps when the next legacy motor drive is replaced by a modern VSD) to become unacceptably unreliable. Some people call this the ‘digital cliff’ – the digital system works just fine despite EMI right up until it ‘falls off the cliff’ and stops working. I’ve seen exactly this scenario bring a 1GW electrical power generating station to a halt for several weeks. Very costly!
It’s all about the acceptable level of financial risk, and almost all installations where they have cut corners on good EMC design/construction (e.g. by not fully applying IEC 61000-5-2:1997) may be feeling pleased with themselves for ‘getting away with it’. They can’t know if they are – in effect – standing too close to the edge of a crumbling cliff whilst blindfolded.
Of course, simple/quick checks with clamp-on RF current probes and portable spectrum analysers can identify potential EMI issues, but doesn’t tell us how close the datacomm’s are to their cliff edge?
You were telling me the other day that PLC’s do not always reveal how much error correction is going on, but that PROFIBUS and PROFINET network analysers were available that could do so, revealing how close a system/installation is to the ‘cliff edge’ of unreliability due to excessive EMI. It seems to me that using a special datacomm’s analyser like yours, that reveals how overworked the error correction is, is something that all industrial installations should do for risk management reasons.
If a problem is found, some simple/quick EMI measurements will identify if the problem is likely to be caused by EMI, and whether any steps that are taken to try to reduce the level of that EMI are actually doing so.
Bio for Peter Thomas
[email protected]
www.controlspecialists.co.uk
Peter is a Chartered Engineer (C.Eng) / (Eur Ing), Chairman of the PROFIBUS and PROFINET International Training Centre Workgroup (PITC), a member of the PROFIBUS UK Steering Committee and an Instructor at the Application, Training & Engineering Centre of Endress + Hauser. He is a Certified PROFIBUS Engineer, Certified PROFIBUS Designer, a Certified PROFINET Engineer and a Siemens Certified Network Professional – Security.
He has 30+ years of practical experience in several areas of Process & Manufacturing Automation, the last 24 of which have been with Control Specialists Ltd. This experience includes Software Design, GAMP Validation, PID Loop Optimisation, Control System Support, Industrial Network validation and support (AS-I, PROFIBUS and PROFINET), developing and delivering Process Control, PROFIBUS and PROFINET training courses. He has worked for clients in the Pharmaceutical, Water, Chemical, Glass, Oil and Gas and Food industries and has worked on projects / delivered training courses in Australia, the US and several counties around Europe.
www.linkedin.com/in/petermthomas
Bio for Keith Armstrong
[email protected]
www.cherryclough.com
www.emcstandards.co.uk
Keith graduated from Imperial College, London,in 1972 with an Honours Degree in Electrical Engineering.
He has been a member of the IEE/IET since 1977 and a member of the IEEE since 1997. Appointed IET Fellow and IEEE Senior Member in 2010.
After working as an electronic designer, then as project manager and design department manager, Keith started Cherry Clough Consultants in 1990 to help companies reduce financial risks and project timescales through the use of proven good EMC engineering practices.
Over the last 27 years, Keith has provided design consultancy and training courses, and fixed hundreds of EMI problems, for well-over over 800 customers worldwide in a huge range of products and applications. He has presented many papers and published many articles and three books, on good EMC engineering techniques and on managing functional safety risks that can be caused by EMI.
Most of his articles and training courses have recently started to be made available through the new emcstandards.co.uk website.
Keith has chaired the IET’s Working Group on EMC and Functional Safety since 1997, and is the UK’s appointed expert to the IEC committees on 61000-1-2 (the basic standard on EMC for Functional Safety), 60601-1-2 (risk management of EMC for medical devices), and 61000-6-7 (generic standard on EMC for Functional Safety).
Since 2015 he has also chaired the IEEE Standards P1848 team creating: “IEEE Standard Practice for Techniques and Measures to Manage Functional Safety and Other Risks with Regard to Electromagnetic Disturbances”.
