Tax Residency for Retirees: Social Security, Medicare IRMAA & Domicile

For most retirees, your tax residency in retirement is decided by your domicile, the one place you treat as your permanent home, and by how many days you spend in any state that uses a 183/184-day residency test. Get your domicile and your day count right and you control which state taxes your pension, IRA withdrawals, and the portion of Social Security that's taxable. Federal tax on benefits doesn't change with your address, but state tax, Medicare paperwork, and even how you prove residency all turn on where you actually live.

Domicile vs. days: what decides a retiree's home state

Two separate rules can make a state your tax home, and retirees who split the year need to clear both. The first is domicile, your one true, permanent home, the place you intend to return to. You keep a domicile until you clearly establish a new one; you can't have two. The second is statutory residency: many states treat you as a resident if you keep a permanent home there and spend more than 183 days (often 184) in the state during the year, even if your domicile is technically elsewhere.

The classic snowbird trap is thinking a winter address solves the problem. If you sell the northern house, register to vote and drive in Florida, and spend most of the year there, you've likely changed domicile. But if you keep the old home and still spend more than half the year in the high-tax state, statutory residency can pull you right back in. Days and intent both matter, see the snowbird tracking guide for the full playbook.

How Social Security is taxed, and where state lines matter

At the federal level, up to 85% of your Social Security benefits can be taxable depending on your total income, and that calculation is the same no matter which state you live in. Your domicile changes nothing federally. What your home state decides is whether to tax benefits on top of the federal rule.

The large majority of states do not tax Social Security benefits at all, and several tax no income whatsoever. A small and shrinking number of states still tax some benefits, usually with generous income-based exemptions for retirees. Because the list changes from year to year, confirm the current treatment for any state you're considering before you move, the direction of travel has been toward fewer states taxing benefits.

Note the third row: a state taxes your IRA and 401(k) withdrawals based on where you live when you take them, not where you earned the money. Federal law bars states from taxing the retirement income of former residents (the so-called "source tax" ban), so a clean move to a no-income-tax state before you start large withdrawals can permanently change the outcome. This is one of the highest-leverage residency decisions a retiree makes.

Medicare, IRMAA, and why residency still matters

Medicare is a federal program, so your state of residence doesn't change your eligibility or your basic premiums. But residency affects Medicare in three practical ways retirees underestimate.

• IRMAA surcharges, higher earners pay an income-related adjustment on Part B and Part D premiums. It's based on your tax return (typically two years prior), so your state move won't lower it, but the income from IRA conversions and pension elections you make as a new resident can push you into a higher bracket. Plan large conversions around the lookback.
• Medicare Advantage and Part D networks are local, Advantage (Part C) plans and drug plans are tied to your service area. Move to a new state or even a new county and you may need to switch plans during a Special Enrollment Period. Original Medicare travels with you; Advantage often does not.
• Living abroad, Medicare generally does not cover care outside the US. Retirees who relocate overseas usually keep paying Part B premiums to preserve enrollment or drop it and rely on local/private coverage. Time abroad doesn't reduce premiums, and gaps can trigger lifetime late-enrollment penalties if you re-enroll later.

Does where you live change your Social Security COLA?

No. Your Social Security cost-of-living adjustment (COLA) is a single national figure set each year for the whole country, it does not vary by state or by the local cost of living. A retiree in Manhattan and a retiree in rural Mississippi get the same percentage increase. So you can't boost your benefit by moving to a high-cost area, and you don't lose COLA by moving somewhere cheaper.

Living abroad usually doesn't stop your benefits either, the Social Security Administration pays beneficiaries in most countries, with a short list of exceptions. The real lever for a retiree's spendable income isn't COLA; it's state income tax. Eliminating state tax on your pension and withdrawals is often worth far more than any cost-of-living difference, which is why so many retirees re-domicile to no-income-tax states.

How to make a retirement move actually stick

Changing domicile is about evidence, not a forwarding address. Auditors look for a consistent story across every part of your life. The strongest moves line up the paperwork and the day count:

• Register to vote in the new state and actually vote there.
• Get a new state driver's license or ID and register your vehicles there.
• Update your address with Social Security, Medicare, the IRS, banks, brokerages, and your doctors.
• Where available, file a declaration of domicile (Florida, for example, lets you record one) and update your estate documents to the new state.
• Spend more days in the new state than anywhere else, and keep your days in the old state well under its 183/184-day line.
• Move the "near and dear": pets, family heirlooms, and the things you'd keep at your true home.

That last day-count point is where most retirees slip. A part of a day in the old state often counts as a full day, and an auditor will reconstruct your year from credit-card and toll records. The defensible approach is to track every day as it happens. Tax Days logs each trip on your iPhone and counts your days against any state's residency line, the federal Substantial Presence Test, and foreign 183-day rules, so if you split the year between, say, New York and Florida, you know your count before the year closes, not after. You can also sanity-check a single trip with the 183-day calculator.

One last wrinkle: if you retire abroad, you may still owe US federal tax on your worldwide income as a citizen, and high-tax states such as California can keep claiming you until you've clearly cut your ties. A clean domicile change to a no-income-tax state before you move overseas often simplifies your state position considerably.