Precision CNC: Tight tolerance technology

• 1
  Tight tolerance CNC technology starts with the machine, not the drawing
• 2
  Thermal stability sets the floor on achievable tolerance
• 3
  Which features hold tight tolerance and which do not
• 4
  GD&T and datums carry more weight than the size callout
• 5
  You cannot hold a tolerance you cannot measure
• 6
  Step by step: how a tight tolerance job is actually run
• 7
  Tolerance bands and what each one demands

A tolerance is nothing more than an allowed band of variation around a nominal dimension. Tight tolerance CNC technology earns its name when that band shrinks to ±0.05 mm, ±0.01 mm, or in the hardest work, ±0.005 mm (about ±0.0002 in). Every error source in the shop now competes for the same few microns.

Three groups of error decide the outcome. Geometric error comes from the machine itself: squareness between axes, spindle runout, ball screw pitch error, thermal growth of the frame. Dynamic error comes from the cut: tool deflection, chatter, workpiece vibration, servo lag on fast corner moves. Setting error comes from how the part sits in the fixture and how hard the vise or chuck squeezes it. Each group can eat the whole band on its own.

That is why buying a tighter machine rarely fixes a tolerance problem by itself. A 16-machine five-axis cell with Ø400 mm rotary tables gives you the kinematic reach to hold a true position across five faces in one setup. It does not give you a stable 20 °C room, a rigid fixture, or a probe you trust. Those are separate purchases and separate disciplines.

The practical question is never "can you hit this number?" It is "which features on this print actually need the number, and what does the rest of the part do?" Answer that first and the process plan writes itself.

• 1
  GeometricMachine geometry, spindle, ballscrew, thermal growth
• 2
  DynamicTool deflection, chatter, servo lag during the cut
• 3
  SettingFixture stiffness, clamping force, datum choice

Aluminium expands about 23 × 10⁻⁶ per °C. On a 100 mm aluminium bore, a 5 °C shop swing moves the diameter roughly 11.5 μm, which is more than a ±0.005 mm band allows. Steel is better at around 12 × 10⁻⁶ per °C, but a 200 mm steel shaft still drifts 2.4 μm per °C. There is no fixture stiff enough to hold a dimension that the material itself is moving.

The response is not to freeze the whole building. It is to control the loop that matters. Rough the part, let it cool, then finish. Keep the finishing pass short so the spindle and axes do not accumulate hours of heat before the critical cut. Run coolant at a controlled temperature rather than ambient tap water. On long parts, measure at the same temperature the part will be measured at during final inspection.

In-process probing closes part of the loop. Touch off the datum, check a witness feature, and let the control apply a work offset before the finishing pass. This catches setup error and slow thermal drift, though it cannot correct a tool that is already flexing mid-cut.

The engineering meaning is simple: tight tolerance work is temperature work. Quotes that ignore thermal control are quotes for a different tolerance.

• 1
  Rough, cool, finishSeparate roughing from finishing to release residual heat
• 2
  Short finishing passesLimit spindle and axis heat build-up before critical cuts
• 3
  Probe the datumApply work offsets before finishing, not after

A turned OD on a rigid, short shaft is the easiest tight tolerance feature in machining. The tool is supported, the load is steady, and the measurement is a simple diameter. This is where ±0.005 mm is routine on a mill-turn center.

A deep pocket with a thin floor is the opposite. As the tool reaches deeper, the effective stiffness of the tool-holder-spindle chain drops with the cube of the length-to-diameter ratio. A Ø6 mm end mill at 40 mm of reach deflects many times more than the same tool at 15 mm. Add a 2 mm floor and the material moves away from the cutter. Holding ±0.01 mm on that floor is a process, not a pass.

Holes behave differently again. A drilled hole is a locating feature, not a precision feature; it wanders with the drill point. Ream, bore, or interpolate with a smaller cutter if the hole carries a fit. Bore diameter and roundness are the same measurement problem, so measure both, not just the diameter.

Thin walls, long slender shafts, and unsupported bosses all share one failure mode: the part deflects under cutting force and springs back after the tool leaves. The dimension looks right on the machine and wrong on the bench.

• 1
  Holds easilyShort turned ODs, shallow bores, features near a rigid datum
• 2
  Holds with processDeep pockets, thin floors, long reach, interrupted cuts
• 3
  Needs redesignWalls under 1 mm, cantilevered features, sharp internal corners

A ±0.005 mm limit on a hole diameter is a size statement. It says nothing about where the hole sits or whether it is round. That is the job of geometric tolerancing, and it is usually the part that drives cost. A true position call of Ø0.05 mm MMC across six holes on two faces forces one-setup machining and a CMM program, even though the individual diameters are loose.

Datum structure decides how the part is held. If the drawing calls A-B-C on three surfaces that cannot all sit flat in a vise at once, the machinist has to choose which one to trust, and the geometry drifts. A datum scheme that matches the function of the part, and that can be reproduced in a fixture, is worth more than a tighter number on any single feature.

Maximum material condition (MMC) is the engineer's friend here. It lets the tolerance grow as the feature departs from its maximum size, which matches how a clearance fit actually behaves. Applying MMC to bolt holes commonly removes the need for a tight position call and cuts setup time.

One warning that comes up often: stack-up. If five stacked parts each carry ±0.05 mm, the assembly can move ±0.25 mm before any process error is added. Tightening one part in the stack rarely solves an assembly problem that belongs to the stack.

• 1
  Position drives costTrue position across faces forces single-setup work
• 2
  MMC saves moneyBonus tolerance matches real clearance fit behavior
• 3
  Check the stackFive ±0.05 mm parts can stack to ±0.25 mm

Measurement uncertainty should be roughly one fifth to one tenth of the tolerance band. For ±0.005 mm, that means an instrument good to about ±0.001 mm, in a room at a known temperature, with the part soaked to that temperature. A caliper at ±0.02 mm cannot qualify a ±0.005 mm callout no matter how carefully it is read.

For round features, diameter alone is not enough. A three-point micrometer can read a lobed bore as perfectly round. Two-point and three-point measurements disagree on out-of-round parts, which is why a roundness tester or a CMM with enough points matters on bearing seats and hydraulic bores.

For prismatic parts, a CMM reports size, position, and form from one setup. We run 100% inspection before shipment, covering raw material check, in-process monitoring, and final inspection, with reports available on request. On repeat production, a first-article report locks the process; in-process checks then confirm it has not drifted.

The uncomfortable truth is that a shop without the right metrology will still ship parts. They will just be parts whose tolerance nobody can prove. Ask what instrument will verify the critical feature before you place the order.

• 1
  5:1 ruleInstrument uncertainty should be a fifth of the band or better
• 2
  Roundness, not just sizeThree-point mics can miss lobing on bores
• 3
  Ask the methodName the instrument for every critical feature