How to Improve the Processing Precision of Machine Tools
A practical shop-floor guide for engineers and buyers who need tighter tolerances from the machines they already own. You will learn where error comes from, which checks pay off first, and when a machine cannot be fixed by setup alone.
Key takeaways
Thermal drift and machine geometry
Most precision problems on a CNC are not program errors. They are physical. The spindle grows as it warms, the ballscrew stretches, and the bed tilts a few micrometres as the coolant tank heats up. On a 500 mm part, a 5 °C rise in the spindle housing can push the tool 0.02–0.04 mm deeper into the cut. That is four to eight times a ±0.005 mm tolerance band.
Run a warm-up cycle before the first good part. A 20–30 minute spindle warm-up at 3,000–8,000 rpm brings the structure to a stable temperature. Many shops then run one or two dummy parts and measure them. The offset you find on the dummy is the offset you apply to the real run. This costs 30 minutes and saves a scrap batch.
Geometry is the other half. Squareness between X and Y, spindle-to-table parallelism, and Z-axis perpendicularity all shift with age and crashes. A granite square and a dial indicator on a 300 mm sweep will show squareness error down to 0.005 mm over that length. If the machine cannot hold 0.01 mm over 300 mm, no amount of cutter compensation will fix the part.
- 1Warm-up first20–30 minutes at running speed before the first inspection part.
- 2Measure the dummyCut one part, measure it, apply the offset to the run.
- 3Square the machineGranite square plus indicator, 300 mm sweep, 0.01 mm limit.
Spindle error and backlash on the axes
Spindle radial runout shows up as a bore that is out of round or a face that is not flat. Mount a 0.001 mm indicator on the spindle taper and turn it by hand. Below 0.005 mm total indicated runout (TIR) is good for a production mill. Between 0.005 and 0.015 mm you can still hold ±0.005 mm on many features if you take a spring pass, but not on a fine bore. Above 0.015 mm, the spindle needs service.
Backlash is a lost-motion error. It appears when the axis reverses direction. On a machine with 0.01 mm backlash, a contour that changes direction will leave a step or a flat. Check it by approaching the same point from both directions with an indicator and comparing the readings. Ball screw preload, thrust bearing wear, and loose coupling bolts are the usual causes.
Repeatability is the number that matters for production. Send the axis to the same point 20 times and record the spread. A machine that repeats to 0.002 mm but is 0.01 mm off in absolute position is easy to correct. A machine that repeats to 0.01 mm cannot hold a ±0.005 mm tolerance no matter how you offset it.
- 1TIR checkHand-turn the spindle with a 0.001 mm indicator on the taper.
- 2Backlash checkApproach one point from both directions, compare readings.
- 3Repeatability check20 moves to one point, record the spread.
Fixturing, tooling, and cutting data
A weak setup moves under cutting force. That movement looks exactly like machine error, so it is worth ruling out early. Clamp the part, then push it with a 0.01 mm indicator while you tighten the last clamps. If the reading changes, the part is being distorted before the cut even starts. Thin walls, long overhangs, and unsupported floors are the usual offenders.
Tool overhang is the second lever. A 4× diameter overhang is stiff. At 8× diameter the tool deflects roughly eight times as much under the same load. On a 12 mm end mill, going from 48 mm to 96 mm of stick-out can turn a 0.01 mm deflection into 0.08 mm. Use the shortest tool that reaches the feature, and prefer a necked tool over a long flute.
Cutting data decides whether the tool pushes or cuts. Too light a feed rubs the edge and work-hardens stainless. Too heavy a feed bends the tool and the fixture. For finishing aluminium, 0.05–0.15 mm per tooth at 0.2–0.5 mm radial depth is a safe starting range. For 304 stainless, drop to 0.03–0.08 mm per tooth and keep the tool in the cut instead of dwelling.
- 1Clamp checkWatch a 0.01 mm indicator while tightening the last clamp.
- 2Short overhangKeep to 4× diameter where the geometry allows.
- 3Finish data0.05–0.15 mm/tooth aluminium, 0.03–0.08 mm/tooth stainless.
Measurement, compensation, and when to stop
Measure on the machine before you unload. A probe or a dial indicator on a known datum lets you correct the offset while the part is still clamped. This removes the unload-measure-reload loop that eats a shift. For a ±0.005 mm feature, measure the same way the CMM will: same datum, same temperature, same contact force.
Temperature is the silent variable in measurement. A part that is 10 °C warmer than the gauge reads larger than it is. Let parts cool to room temperature before final inspection, or record the temperature and correct. For aluminium, the expansion is about 23 μm per metre per °C. On a 300 mm part, a 5 °C difference is 0.035 mm.
Finally, know when the machine is the limit. If repeatability, spindle TIR, and squareness are all inside spec and the part still drifts, the process needs a different method. Grinding, hard turning, jig boring, or a temperature-controlled room may be the honest answer. Pushing a worn mill past its capability produces scrap, not precision.
- 1Measure clampedCorrect the offset while the part is still in the fixture.
- 2Match the CMMSame datum, temperature, and contact force as final inspection.
- 3Let it coolAluminium moves about 23 μm/m per °C. Cool before measuring.
Seven steps to improve the processing precision of machine tools
Run these in sequence on the same shift. Each step takes minutes and rules out one error source.
- 1Warm up the spindleRun 20–30 minutes at 3,000–8,000 rpm with the coolant on. Do not cut a tolerance part cold.
- 2Check spindle TIRMount a 0.001 mm indicator on the taper. Hand-turn and record. Keep total indicated runout below 0.005 mm for production milling.
- 3Measure backlash on X, Y, ZApproach one point from both directions. More than 0.01 mm on any axis means screw preload or bearing work.
- 4Test repeatabilitySend each axis to the same point 20 times. A spread above 0.005 mm will not hold a ±0.005 mm tolerance, regardless of offset.
- 5Square the geometryUse a granite square and indicator over a 300 mm sweep. Correct or reject anything worse than 0.01 mm over that length.
- 6Stiffen the setupReduce tool overhang to 4× diameter, check clamp distortion with a 0.01 mm indicator, and support thin floors. Re-cut the dummy part.
- 7Compensate and verifyApply the measured offset, cut one part, measure it clamped, then confirm on the CMM after it cools to room temperature.
Which correction to use and when
Match the symptom to the likely cause before you touch the offsets.
| Symptom | Likely cause | First check | Practical fix |
|---|---|---|---|
| Bore out of round | Spindle radial runout | Indicator on spindle taper | Service spindle, or spring pass at 0.2 mm |
| Step at direction change | Axis backlash | Approach point from both sides | Screw preload, thrust bearings, coupling |
| Size drifts over a shift | Thermal growth | Part temperature vs shop temperature | Warm-up cycle, offset after first part |
| Size varies part to part | Poor repeatability | 20 moves to one point | Service axis before chasing offsets |
| Taper or out-of-square face | Geometry error | Granite square, 300 mm sweep | Level and tram, or move to a jig borer |
| Wall thickness varies | Fixture distortion | Indicator while clamping | Softer clamps, support the floor, rough then finish |
Common questions
How long should a spindle warm up before cutting tight parts?
Run 20–30 minutes at the speed you will actually cut at, with coolant on. A fixed 5-minute warm-up is not enough for a ±0.005 mm feature because the housing is still growing.
Cut one dummy part after the warm-up, measure it, and apply the offset to the production run. That single part absorbs the residual drift.
Can I improve the processing precision of machine tools without a laser interferometer?
Yes for most day-to-day work. A 0.001 mm dial indicator, a granite square, and a known-length gauge block cover spindle TIR, backlash, squareness, and repeatability.
A laser or ballbar is worth the cost when you need volumetric accuracy over a large travel, or when you suspect pitch error compensation has drifted.
What repeatability do I need to hold ±0.005 mm?
Aim for axis repeatability of 0.002 mm or better. If the machine only returns to 0.005 mm, the tolerance is consumed by the machine alone before tool wear and thermal effects.
Repeatability is more important than absolute accuracy because it can be corrected with offsets. Absolute position error of 0.01 mm is workable. A 0.01 mm spread is not.
Does coolant affect part size?
Yes. Flood coolant pulls heat out of the part and the spindle, which helps stability. But cold coolant on a warm part can shrink a bore between the cut and the measurement.
Keep coolant temperature steady, and measure parts after they reach room temperature. Aluminium moves about 23 μm per metre per °C.
When should I stop adjusting the machine and change the process?
When spindle TIR, backlash, squareness, and repeatability are all inside spec and the part still drifts, the machine is no longer the limit.
Grinding, hard turning, jig boring, or a temperature-controlled room may be the right move. A worn mill pushed past its capability makes scrap.
How do I know if the fixture is moving during the cut?
Set a 0.01 mm indicator against the part, take a light cut, and watch the needle. Movement during the pass means the setup is flexing, not the machine.
Clamp distortion shows up even earlier. Watch the indicator while you tighten the last clamp. If the needle moves, the part is already stressed.
Need parts held to ±0.005 mm?
Send your drawings and we will run a free DFM check, quote within 12 hours, and machine on 127 CNC machines with 100% inspection before shipment.
12-hour quote±0.005 mm tolerance100% inspectionNDA on request
More CNC process notes
We publish setup notes, tooling trials and inspection data from the factory floor.
