Runaway servo motor

W

Thread Starter

wackadoodle

Hi,
I've been encountering some strange runaways with one of our machines. There is many like it but this one is the only one with this kind off problem. Every now and then (can be 1-2 months in between) it seems to be starting a servo motor 'by itself'. This causes dangerous conditions as this is a pretty big machine, though last time it looks like it hit the torque limit and shut off.

There doesn't seem to be any logic in or the PLC/motion controller that can be causing this (it has been changed and it's only happening to one of the eight machines)..

Can this be caused by electrical noise, maybe on the position feedback?
The servo packs and motors are omron yaskawa.

Any pointers and ideas are appreciated.
 
B

Bob Peterson

More modern systems tend to incorporate software that can detect such conditions and shut down the drive.

It's hard to determine just what goes wrong with stuff like this that is intermittent. It is easy to blame it on "noise" or "bad grounds", but IME that is not usually what it ends up being.

It can take a long time to find out just what it is.

You might want to start by adding some code to your PLC or motion controller to detect if the speed or position command goes outside of a normal range and shutdown. It could be that the program is making some kind of calculation and there is a calculation error. If next time it happens one of these limits trips it will give you some more clues.

It's not real hard to write some code to detect if there is a runaway servo either. A sliding window around the expected position will do it, although again, this is usually something that is already available in the PLC or motion controller, but might not be turned on or being used for anything.

I had a bad RTD module one time that would about once a month start to give an erroneous value. It was only about 10-20 degrees off, and the anomaly only lasted for a few minutes at a time. I had the guy on site add some code to trap the data and found that now and then the rate of change of the temperature would jump to a very high value. The process just could not do this. Once we found that was what it was, I had the maint guy change out the module and the problem did not recur.

Make sure that the program checks all values it uses for calculation purposes to make sure that the values are reasonable based on the current machine condition. Trap any obviously erroneous values and have the program record them so you can later on back track it.
 
J

James Ingraham

This is certainly a scary problem. I wish I could give you a nice succinct answer, but I'm afraid this one's going to be a tough one. Let's start with some important information about the setup.

a) What is model of the servo motor and servo drive? (Omron Yaskawa is gone now, but they were also pretty much Yaskawa's porduct anyway.) What power range are they, roughly? Are they AC or DC powered?

b) What is controlling this servo? PLC, a PC with motion controller, a stand-alone motion controller, etc. Again, brand and model would help.

c) How are the controller and drives connected? Mechatrolink? +/-10V analog with encoder feedback?

d) "starts by itself" and "runaway" can mean different things to different people. Can you be a little more specific about the circumstances and action? A video would be even better. Are the servos enabled when this happens, or is it enabling itself? Does it move in a "controlled" fashion to an unexpected point, or is it flying away as fast as possible? Does this happen while the machine is running or only when it is idle?

e) All servo systems should have a safety circuit that removes the power from the drives. Is this functioning correctly? Will the servo runaway with the safety circuit tripped? That would be VERY bad.

Some things to try, even without having more information:

1) Since you have 8 working machines, take the drive from the bad one and swap it with one of the good ones. Does the problem stay with the drive or with the machine?

2) Review the wiring, especially the safety circuit. Does the safety circuit trigger properly? Is power really removed from the drives? Are all connections solid?

3) Replace the wiring between the servo motor and the drive, preferably with manufactured cables. This could be an expensive and time-consuming process, depending on the size and type of machine we're talking about.

4) You sound like an end-user, not a machine builder. Have you spoken with the machine's manufacturer? Perhaps they have encountered this problem before.

5) Put some "traps" in the code. This could be hard or easy, depending on the setup. Basically, you want to watch every command that goes out to the servos.

6) I hate to bring this up, but a while back there was a question here on control.com about drives catching on fire. Everybody pitched in to try and come up with a rational explanation, but in the end it was sabotage. A camera caught a person deliberately setting fire to the drives with a lighter. A camera isn't a bad idea even if it isn't sabotage, because it may help you pin down the exact circumstances that trigger the problem.

Hope that helps.

-James Ingraham
Sage Automation, Inc.
 
C

curt wuollet

I think you're on the right track as bad position feedback will command motion. Actually, depending on just what you are doing, any feedback problem can cause weird (but logical) things to happen. You should check the basics first as I have this sort of infrequent even turn out to be power, cabling, etc. too. It's the worst type to troubleshoot, I use a sampling scope that freezes the inputs, etc. when the event occurs if you can find an alarm or something to trigger on.

Regards
cww
 
W
The problem is that this doesn't even happen once a month. Another big issue is that it's not placed on my site. So I won't be able to monitor it. I will be traveling over to the location soon though, and will try to troubleshoot as best I can. The machine operator is instructed to shut of the servo contactors when doing work in prior of running the main process
 
W
Indeed, it can create dangerous situations if the operator is caught with his hand doing prep work and it just starts running wild.

To answer your Qs:

a) AC powered SGDH-30DE , 24 V DC control.

b) Delta Tau Turbo UMAC(universal motion and automation controller) - this have been changed recently, and the problem was there before as well

c)connected through servo axis card (analog +-10V)

d) yeah I guess that one was unclear, as runaway can mean doing something else than commanded. Though in this case it starts running without anyone setting the command. It is in idle mode but the servo's are activated (run mode). I've only gotten second hand info on this, but from what I can understand it just starts accelerating up to a point were it hits a torque limit and shuts down.

e) The servos will trip on a torque limit, following error limit,end limit switches etc and will shut off.


1) I've already instructed them to swap the drive and see if thats problem. I guess we'll know that in a few months...(or maybe not since they are shutting off the servos manually between runs now..

2) the safety circuits will trigger, and the machine will shut down but it will move quite a bit before this happens. I will check the connections when I travel to the location.

3) I will check the cables as well when I go there, and change if I see any visible damages.

4) Indeed I am not the machine builder. It a custom made machine, made by colleagues and former colleagues. I recently just started here, working on automation, and I'm learning all about the machines from the problems there is.. This problem has not been heard of before, though there have been some problems earlier with other axes on the machine that created somewhat similar behaviour. However this seemed to be because of insufficient grounding on different parts of the machine and was fixed on later versions (included the one with problems).

5) I have hopes for this one. I will make some programs that logs critical parameters when this event happens again.

6) I like that you bring this up, the thing is here that this has now happened 3 times. At all events it was the same operator working the machine when it happens (one out of two or three working the machine in total). Though not sabotage I'm really considering the operator might be doing something wrong, even so it should not be possible for the operator to trigger such event from the HMI...

Thank you. I appreciate the thorough information.
 
J

James Ingraham

The SGDH-30DE is a fairly typical servo drive, and a pretty well regarded one at that. It's not current generation, but there's nothing about it that would inherently make me not trust it. (There are drives out there I wouldn't touch with a 10 foot pole.) It receives feedback from the motor, and then emulates encoder feedback out to the motion controller. If there's a problem with the encoder itself or the wiring from the motor to the drive I would expect the drive to fault without running away. (Not to say that it's impossible for the problem to lie there, it's just not my first guess.) The drive itself could be flaky or the wiring from the drive to the motion controller.

Again, the UMAC is well-respected motion controller, and fairly typical of stand-alone motion controllers. If has been replaced, the problem is probably not there.

The fact that you're using +/-10V analog with encoder feedback is where I suspect the problem lies. Either the feedback is not reporting correctly or the analog signal is getting messed up. I would suspect the feedback before the analog signal. I've never actually seen an analog signal "stuck on."

"It a custom made machine, made by colleagues and former colleagues."

Ah. No machine builder to yell at then. The Delta Tau people would probably tell you it's a servo problem, and the servo people would probably tell you it's a controller problem. Nevertheless, there may be a good technician in the area who knows both and could help. I'd offer my company's services, but (a) I'm guessing you're in Europe some where, since we never had Omron Yaskawa here in the States, and (b) we're not overly familiar with that particular set up.

"...seemed to be because of insufficient grounding on different parts of the machine and was fixed on later versions..."

Grounding issues can be particularly difficult to nail down. Let's hope that's not it.

"...I'm really considering the operator might be doing something wrong, even so it should not be possible for the operator to trigger such event from the HMI..."

Agreed. Don't forget the Therac-25.

Good luck.

-James Ingraham
Sage Automation, Inc.
 
Wackadoodle... Search Control.Com archives for Threads titled "EMI", "Interference", "Bite", "T/C", etc! One of the stories may ring a bell!

Regards, Phil Corso
 
Wackadoodle... for a specific "pointer"... try to correlate the time of the incident with de-energization of an AC or DC solenoid in proximity to the suspect circuit(s)!

Phil Corso
 
J

James Ingraham

Regarding Phil Corso's advice, I'm a bit mixed. Certainly, reading past articles on electrical noise could be helpful, if only for background.

As for "...try to correlate the time of the incident with de-energization of an AC or DC solenoid in proximity to the suspect circuit(s)," I doubt this one for a couple of reasons. First of all, the machine is at idle when this happens, so things shouldn't be triggering. Second, the problem is so rare. What solenoid could possibly be firing so rarely? Third, I would expect such a spike to be short. The noise should only occur for that short period of de-energization. The servo system ought to be able to recover itself if that's the case.

"...also investigate possibility of Walkie-Talkie RFI"

Again, I doubt this one. For one thing, only one machine has the problem. For another, the infrequency again. I've rarely seen RF noise hit a wire that badly, unless it wasn't shielded, wasn't properly grounded, or it's LOT of noise. Noise on a line is usually injected more directly than that. Servos are exceedingly noisy (they'll make your radio go to static, for example) and they're in the same cabinet with the encoder wires. Even then, we usually don't see a problem unless the noise propagates from the drive directly back on to the 24VDC control voltage.

I could be wrong, of course. I heard a story about a system failing every night about the same time. It turned out to be a light that came on. The electrical noise from the light blitzed the communications network. Another was a grounding rod on a CNC that was actually GENERATING random noise spikes, which traveled through the ground to a different machine's grounding rod and cause IT to flake out. It's just that these types of thing aren't my first guess.

The lack of reproducibility is going to make this one extra tough to pin down.

-James Ingraham
Sage Automation, Inc.
 
W
James,

Interesting story of the Therac-25.

Both the servo and the motion controller are as you say well respected, though I've heard about them both failing in strange ways, but that was some years ago with overheating in the electrical cabinet.

I talked to Delta Tau about it a while ago, and of course it was not possible that it was their product. I still haven't talked to Yaskawa, but I would guess they wouldn't blame their products either, but I'll contact them see if I can find an honest soul who knows about problems and faults.

Say that it is some problem with the encoder feedback, or grounding issues. For the two axes to run away at high speed it would take a 5-10V signal to be generated. I can't see this happening at least not from noise,but maybe if there still is a problem with the grounding.

Also, do you have any experience with EMI from smaller TIG and MIG welding devices?

Thanks
 
N

NetworkSprocket

You really need to go back to basics on this. A servo position loop compares setpoint position with actual position (usually in software), and . For it to "run away" it needs one element in this loop to fail.

(1) Position- an incremental encoder? Could the encoder output a condition which the position counter interprets as a fixed or continuously incrementing position? Could the input circuit be such that cable failure can be interpreted as a fixed or continuously incrementing position? You say that the actual controller has been replaced so that seems to be in the clear.

(2) Software - is that still operating in other respects, or is the controller completely crashed? If so, it could be noise as others have suggested. Can you log the setpoint and feedback counters so that when it does misbehave, you can see which is causing the runaway?

(3) Setup- is the gain so high (or is there an I term in the controller) such that a stuck input causes a large swing in the output?

(4) Amplifiers- can the output get stuck for some reason without the software knowing about it?

My guess for a first look would be the encoder cable- could there be an intermittent fault (short core to core or core to shield, one conductor open circuit, failed solder joint, bad pin etc.)? This might well show up as the counter toggling plus and minus one as the machine moves, the integrator doing the rest. Can you put an integration limit on to see if this is the cause of the problem, is there a software error limit that could stop it becoming too scary?
 
J

James Ingraham

"...do you have any experience with EMI from smaller TIG and MIG welding devices?"

Absolutely. Unless someone is actually welding ON THE MACHINE, however, I wouldn't expect it to cause this type of issue. And they shouldn't be welding on the machine if the machine isn't locked out.

-James Ingraham
Sage Automation, Inc.
 
W
>"...do you have any experience with EMI
>from smaller TIG and MIG welding
>devices?"
>
>Absolutely. Unless someone is actually
>welding ON THE MACHINE, however, I
>wouldn't expect it to cause this type of
>issue. And they shouldn't be welding on
>the machine if the machine isn't locked
>out.

Hehehe, no it's nearby, aprox 2 meters from the el. cabinet, but as you said, it shouldn't be a problem.
 
D
>"Absolutely. Unless someone is actually
>welding ON THE MACHINE, however, I
>wouldn't expect it to cause this type of
>issue."

I agree with the first statement - you can absolutely have noise problems because of a welder running. The second statement is likely to be true in Europe and North America, but I have seen some pretty awful wiring practices (both for machines and plant power) outside of these areas. One thing to do is to put a scope on the Delta Tau 5VDC signal (you can get to it on an encoder connector). Watch it while the welder is running, cycle some solenoids and contactors, fans, heaters, etc. This will not completely rule out intermittent noise but can at least give you an idea of how clean the system is. With the Delta Tau controller you can also put a Max Change filter on the encoder input - make it equal to your maximum velocity plus some fudge factor - maybe 25%. This will throw away any counts that come in over that max value.

Davis Gentry
Senior Applications Engineer
Delta Tau Data Systems
 
K

Ken Emmons Jr.


I have a few points/questions:

What type of cabling runs from the drive to the PMAC encoder input? I would expect a 4 twisted pair (index, A, B, and Power pairs) with overall shield. If its not twisted pairs you may have a problem. I agree with other posts about hooking an oscilloscope to the encoder lines (power and signals) and see if there is significant noise. You may want to build a breakout connector interface before you get to the site if possible so you can clip in with your scope easily.

The DAC connection between PMAC and drive should be a separate shielded cable just to be safe.

The PMAC has a decent plot and gather facilities, try to work with Delta Tau to see if you can capture your encoder signal and plot it when the error occurrs. The trick here will be in knowing when to trigger (maybe at any fault)? If the motor is sitting there and the encoder shows a huge step in position, well, there you have it.

In the end, replace the motor, drive, and wiring between the PMAC and Yaskawa. I've seen malfunctioning servo systems where the encoder was faulty. If it only happens once a month you will be troubleshooting this thing forever.
 
W

William Sturm

If the PMAC parameters for fatal following error are set correctly then the controller should disable the drive upon an excessive position error.  This assumes proper wiring and system design.  The position controller should always be able to disable the drives upon any serious error.  Has the PMAC power supply been changed?  Possibly if one of the DC rails disappears or drifts, the analog outputs could be affected
 
Top