High precision processing: what actually holds a tolerance

• 1
  Key takeaways
• 2
  Where the error in high precision processing actually comes from
• 3
  Why temperature control decides whether the size holds
• 4
  What the machine envelope allows in high precision processing
• 5
  How probing and inspection close the loop
• 6
  When tight tolerance is the wrong call
• 7
  Feature type against sensible tolerance band

A tolerance is not a property of the machine. It is the sum of everything that happens between the spindle and the finished surface. Geometric error in the ways, spindle runout, tool deflection, fixture compliance, thermal growth and material springback all stack up. When a shop claims ±0.005 mm, what it really means is that the stack stays inside that band under controlled conditions.

Start with the machine. A five-axis machining center with a Ø400 mm rotary table carries different error behavior than a three-axis mill with a long bed. Rotary axes add positioning error at every index, so a part that needs four sides held to ±0.005 mm is easier on a five-axis machine that reaches them in one setup. Each re-clamp adds its own offset.

Tool deflection is the next term. A 12 mm end mill hanging 60 mm out of the holder bends under cutting load. In aluminum the deflection is small; in 4140 or 17-4PH it is not. Rough passes at high radial engagement push the tool away from the wall, and a spring pass at 0.1 mm radial depth removes the witness left behind. That is why the last pass matters more than the first.

Material behavior closes the loop. Aluminum 6061 moves with heat and cuts clean. Titanium TC4 (Ti-6Al-4V) work-hardens at the surface and pushes back. Inconel resists the cut and heats the tool instead of the chip. The same program that holds ±0.005 mm in 6061 may open to ±0.02 mm in Inconel without a change in strategy.

• 1
  Geometry errorSquareness and straightness of the axes set the floor for any part.
• 2
  DeflectionGrows with tool overhang and with material hardness.
• 3
  Thermal growthScales with part length and with the temperature difference.

Aluminum 6061 expands about 23 μm per meter per °C. A 300 mm part that warms 5 °C during a long cycle grows roughly 0.035 mm. That is seven times a ±0.005 mm band. No amount of machine accuracy fixes it. The shop has to control the temperature of the room, the coolant and the part itself.

In practice this means a temperature-controlled bay, coolant chilled to a set point, and a dwell before final measurement. We hold parts in the inspection room long enough for them to reach the reference temperature. A part measured hot reads small and then grows after the operator has already signed off.

The same physics works in reverse for long parts. A 4,000 mm aluminum extrusion can swing 0.1 mm end to end on a 10 °C day-night change. For long parts the useful question is not the tolerance on the drawing but the temperature at which that tolerance is defined. Ask for a reference temperature of 20 °C and the number becomes meaningful.

Steel is easier. 4340 expands about 12 μm per meter per °C, roughly half of aluminum. Titanium sits near 9 μm. So a mixed-material assembly can drift out of alignment even when every individual part was measured good in the same room.

• 1
  Aluminum 6061About 23 μm/m per °C. The most temperature-sensitive common material.
• 2
  4140 steelAbout 12 μm/m per °C. Half the drift of aluminum.
• 3
  Titanium TC4About 9 μm/m per °C. Low growth, but hard to cut.

Capability depends on size as much as on tolerance. A small part with dense features suits a compact machine with short travels, where the axes are stiff and the thermal loop is short. A large part needs a machine that can reach it, and long travels usually come with slightly larger geometric error.

We run 127 high-precision CNC machines across three plants, including 16 simultaneous five-axis machining centers, 12 four-axis mills, 27 three-axis machines and 16 mill-turn centers. Travels range from 500 × 310 × 200 mm on the compact side up to 4,000 × 400 × 150 mm for long parts. A part that fits a smaller machine usually holds tolerance more easily.

Five-axis matters most when a part has features on several faces or contoured surfaces that a three-axis setup would need to re-clamp. Re-clamping is where stacked error appears. Reaching five faces in one setup removes that term and often removes a fixture from the job.

Mill-turn centers handle parts that are mostly turned but have milled flats, slots or cross holes. Doing both in one spindle keeps the relationship between the bore and the milled face inside the same setup, which is usually the dimension that matters on a shaft or a housing.

• 1
  Compact500 × 500 × 450 mm and 500 × 310 × 200 mm travels.
• 2
  Medium750 × 1,150 × 550 mm and 600 × 600 × 600 mm travels.
• 3
  Long4,000 × 400 × 150 mm for extended parts.

Cutting a part is only half of high precision processing. The other half is knowing what you cut. On-machine probing measures the feature while the part is still in the fixture, before the setup is broken. If a bore is 0.008 mm small, the operator can adjust the offset and take another pass instead of scrapping the part or re-clamping it.

For a first article, we check raw material, monitor in process and inspect at the end. Reports are available on request. For production runs the sampling plan follows the feature criticality: a sealing face gets checked on every part, a clearance hole gets checked at intervals.

Surface finish is measured separately. A turned or milled finish typically lands at Ra 1.6–3.2 μm, a controlled finish at Ra 0.8–1.6 μm, and a fine finish at Ra 0.2–0.8 μm. Finish and tolerance are not the same requirement. A part can hold size and still fail a seal because the surface is too rough.

The qualification rate we see on mature jobs is 99.99%. That number is not luck. It comes from stable programs, controlled temperature and measuring before the setup is released, not after.

• 1
  In-process probeCatches drift while the datum is still valid.
• 2
  Final inspectionConfirms the shipped part, not the machine.
• 3
  Finish checkRa 0.2–0.8 μm, Ra 0.8–1.6 μm or Ra 1.6–3.2 μm as specified.

Tolerance costs money. Every tightened band adds a roughing strategy, a finishing pass, a probe cycle and sometimes a slower spindle. If a hole only needs clearance, ±0.1 mm is fine and the part ships faster. Save the tight bands for the features that carry function: a bearing seat, a sealing face, a mating bore, a location pin hole.

Some geometries resist tight tolerance regardless of effort. A deep pocket with a high aspect ratio flexes the tool and the wall. A thin web under 1 mm will deflect under clamping and release when the vise opens. In those cases the better answer is usually a design change, not a tighter machine.

Material also sets a practical floor. Aluminum and brass hold ±0.005 mm without drama. Stainless 316 and 17-4PH are workable. Titanium and Inconel hold it, but with more passes, more tool changes and a longer cycle. If the part is Inconel and the tolerance is cosmetic, relax it.

The useful engineering question is not how tight the shop can hold but which features actually need it. A drawing with five tight tolerances and forty loose ones is normal. A drawing where everything is ±0.005 mm usually means the function was not defined yet. We flag those in the DFM review, which comes back with the quote within 12 hours.

• 1
  Keep tightBearing seats, sealing faces, mating bores, dowel holes.
• 2
  RelaxClearance holes, cosmetic edges, non-mating surfaces.
• 3
  RedesignThin webs and deep narrow pockets.